YUY2 problems

[shh]

Hi!
This is probably faq, but I didn't find anything using the search-function. Most likely I've used wrong search-words, but I really don't know *what* words whould match my problems. :roll:
I'm having some problems with DGDecode/avisynth and YUY2 on my computer and cannot locate, what's going wrong. Perhaps you can help me. :)
Showing an MPEG2-file via DGDecode/avisynth with ConvertToRGB32() is fine, ConvertToRGB24() also fine, showing it with ConvertToYUY2() is crap: Visible at the edges of the letters.
Switching from interlaced=true to interlaced=false doesn't help.
Changing to another iDCT-algo doesn't help.
Using the old DVD2AVIdg-versions (decodefix v1.00 and v1.3.0) with MPEG2Dec3dg.dll doesn't help either.
VirtualDubMod shows the M2V-file as it should be shown, though it crashes when I want to get File->FileInformation from it (why?).

My computer: Athlon-XP, WinXP with DirectX 9c, Radeon 9800pro with Catalyst 5.4 @32bit-desktop-depth.
DGDecode: v1.31b3 (same problem with v1.3.0)
AviSynth: v2.5.6rc3 (same problem with v2.5.5)
VirtualDub: v1.6.5 (same problem with v1.5.10)
Src-file: test.m2v (http://www.sysh.de/forum_files/test.zip)
Stream: MPEG-2 MP@ML VBR
Resolution: 720x576@25fps 16:9
Picture structure: Frame, DCT type: Frame, 10bit
Frame type: Progressive, tff
Quantscale: Nonlinear, Scan type: Alternate
manual intra matrix, manual non-intra matrix
SequenceExtension: MP@ML, chroma: 4:2:0, progressive_sequence: no, low_delay: no,escape_bit: no
No SequenceDisplayExtension: -> No special color description

Avisynth-script1 (correct):
LoadPlugin("D:\Video\AviSynth\plugins\DGDecode.dll")
mpeg2source("test.d2v")
ConvertToRGB32()

Avisynth-script2 (wrong):
LoadPlugin("D:\Video\AviSynth\plugins\DGDecode.dll")
mpeg2source("test.d2v")
ConvertToYUY2()

The registered plugin-directory contains only DGDecode.dll (respectively MPEG2Dec3dg.dll for my decodefix tests)

D2V-project file:
DGIndexProjectFile08
1
8 test.m2v

Stream_Type=0
iDCT_Algorithm=2 (1:MMX 2:SSEMMX 3:FPU 4:REF 5:SSE2MMX)
YUVRGB_Scale=1 (0:TVScale 1:PCScale)
Luminance_Filter=0,0 (Gamma, Offset)
Clipping=0,0,0,0 (ClipLeft, ClipRight, ClipTop, ClipBottom)
Aspect_Ratio=16:9
Picture_Size=720x576
Field_Operation=0 (0:None 1:ForcedFILM 2:RawFrames)
Frame_Rate=25000
Location=0,0,0,8B

c00 1 0 0 0 0 d2 f2 e2 f2 f2 e2 f2 f2 e2 f2 f2 e2 ff

FINISHED 0.00% FILM


Pictures (via avisynth-script -> VirtualDub):
correct (full-size (http://www.sysh.de/forum_files/rgb32.png)):
http://www.sysh.de/forum_files/rgb32cut.png
wrong (full-size (http://www.sysh.de/forum_files/yuy2.png)):
http://www.sysh.de/forum_files/yuy2cut.png

So, do you have the same decoding-weirdness on your computer?
Any suggestions what could help to correctly display/use YUY2?

[Boulder]

Which codec decodes YUY2 on your system (you can find it in the registry)? It's probably just a decoder issue, if you encoded that clip in CCE, everything would be fine. HuffYUV decodes YUY2 correctly, you could try installing it.

VDubMod is a bit buggy, it always crashes on elementary MPEG2 streams when you try to get the file information.

[shh]

YUY2 is actually decoded by xvidvfw.dll.
Switching to huffyuv.dll provides me the same results.

What do you mean with "just a decoder issue"?
Where does this issue come from?

Hää? :confused:
I've just encoded this "crappy" YUY2-output in CCE. But somehow the encoded file seems to be ok - as long as I use ConvertToRGB for playback...
How could this be? How can a broken(?) clip produce clean results?!?? :confused:
Are the color-planes smoothed somehow when I use ConvertToRGB?

[Boulder]

So the YV12 output looks crappy as well (the CCE encoded clip)? Which program did you use to playback?

What I mean by decoder issue is that sometimes colorspaces other than RGB show incorrectly in VDub(Mod). The fact that ConverttoRGB32() shows a correct video suggests this problem.

[shh]

> So the YV12 output looks crappy as well.

No. The CCE-encoded clip looks exactly like my source-clip.
When I Use YV12 output in the avisynth script, the result looks like I have used ConvertToRGB(). VirtualDub then takes xvidvfw.dll as a YV12 decoder, probably providing RGB.

Every time I use ConvertToRGB I get a "nice" video.
How can YUY2-output look such blocky compared to RGB? :confused:
I must ask again... Are the color-planes smoothed somehow when I use ConvertToRGB?

[Wilbert]

YUY2 is actually decoded by xvidvfw.dll.
Switching to huffyuv.dll provides me the same results.
Like Boulder says, the problem has to do with the YUY2->RGB conversion.

AviSynth uses a 1-2-1 kernel for this conversion. Does anybody know what XviD and Huffyuv use?

@shh,

Your clip is not accessible?

[shh]

Wilbert> Your clip is not accessible

Oh, apparently my server cannot provide raw .m2v files.
I've now made a .zip of it and fixed the link above.
Clip again: test.m2v (http://www.sysh.de/forum_files/test.zip)

Wilbert> AviSynth uses a 1-2-1 kernel for this conversion

As far as I understand that, avisynth then does some kind of smoothing when it does it's RGB-upscaling, right?
And for YV12->YUY2 no filter is used? Just plain vertical color-doubling?

[Wilbert]

As far as I understand that, avisynth then does some kind of smoothing when it does it's RGB-upscaling, right?
Yup. The luma of a pixel Y2_new is interpolated using (Y1_old+2*Y2_old+Y3_old)/4.

I think XviD and Huffyuv use nearest neighbour (pixel replication).

And for YV12->YUY2 no filter is used? Just plain vertical color-doubling?
No. XviD does that. What AviSynth does is explained here

http://www.avisynth.org/Sampling

[squid_80]

Don't the new versions of virtualdub (1.6+) have internal YUV handling? If so it seems very strange that the error would still be there when using 1.6.5.

[Wilbert]

Ok, i asked Avery when does VDub use its internal color conversion routines, and this was his response


It is done by VirtualDub either in the input or display routines. By default it is done in the input routines, but if you have fast recompress mode enabled or explicitly choose a YCbCr input format in Video > Color Depth, it is done in the display routines instead. Finally, if you have video filtering enabled, it will happen at the start of the video filter chain.


Argh.

VirtualDub has had internal YUY2 support for some time, at least as far back as 1.5.5. That code definitely point-samples chroma, as it's the same routine used by the MPEG decoder.

In 1.6, everything gets routed through the Kasumi library instead, and that is supposed to do chroma interpolation in all cases... but, unfortunately, it's broken and is still point-sampling chroma for UYVY>RGB and YUY2>RGB conversions. The conversion table is specifying a shortcut path that would be better not taken, as otherwise it would have done a two-step conversion through X8V8Y8U8, which does interpolate. One workaround is to force VirtualDub to use YV16 as the input format, which causes the chroma to be interpolated to planar 4:2:2 YCbCr before conversion to RGB. Alternatively, disable YCbCr input support in Options > Preferences > AVI, which will then cause a video codec to be used for the conversion instead; I definitely know that Huffyuv will interpolate chroma for YUY2.

I'll look into fixing this for 1.6.6.

Summarized:

1) VDub has internal YUV handling (YUY2 at least since 1.5.5), so the conversion is done by VDub.

2) YUY2 is still point-sampled (nearest neighbour) to RGB (in 1.6.5).

3) I don't know about VDubMod though.