MPEG2DEC3 distorts luminance

[SIF]

ABSTRACT

Switching form AviSynth 2.08 and MPEG2DEC to AviSynth 2.52 and MPEG3DEC3 I immediately realized that MPEG2DEC3 alters the luminance. What is black in the original (YUV = 16:0:0, RGB = 16:16:16) is returned as (5:0:0, RGB = 5:5:5). This is a bug. As you know the legal range according to CCIR-601 for the luminance is 16 - 235.


DETAILS

I realized this because my way of encoding is a little different from the mainstream. The main difference is that I only encode using legal TV formats, for PAL: 720x576, 704x576 and 352x288. Currently I use 352x288. My AVISynth processing steps look like this:

- Crop away any black borders in the source material. Crop horizontally at least 16 Pixels.

- Reduce the picture to 1/4 of the original size. This operation respects the pixel aspect ratio of the source.

- Expand the resized image (letterboxing) to 352x288, using black (YUV = 16:0:0, RGB = 16:16:16) as fill color.

---------------------
LoadPlugin("MPEG2DEC.DLL")
mpeg2source(" ... ")

top = ..
bottom = ..
left = ..
right = ..
width = 720 - (left+right)
height = 576 - (top + bottom)
Crop(left,top,width,height)

ReduceBy2()

topBorder = ... *16 + ...
bottomBorder = 288 - (height/2 + topBorder)
leftBorder = ...
rightBorder = ...
AddBorders(leftBorder, topBorder, rightBorder, bottomBorder, $101010)
---------------------

I left out (...) the real values in order to make the script more readable.

Now use this script and an appropriately modified version for AVISynth 2.5 and MPEG2DEC3. Start VirtualDub / VirtualDubMod and inspect the title or credits part of the movie, where the background is black. Everything is fine with AVISynth 2.08 and MPEG2DEC but with AVISynth 2.52 and MPEG2DEC3 the black background of the actual image is darker than the expanding letterbox bars. You can now copy the image to the clipboard and paste it into your favorite image processing program in order to perform further investigations.

[Kurosu]

1) Try limiter() at the end of your script
2) It could also be color conversions that aren't legit; those have proven to produce a lot of unexpected results, though Avisynth crew is perfectly aware of such cases and has done its best to reduce these problems

[SIF]

Hi Kurosu,

Originally posted by Kurosu
1) Try limiter() at the end of your script
2) It could also be color conversions that aren't legit; those have proven to produce a lot of unexpected results, though Avisynth crew is perfectly aware of such cases and has done its best to reduce these problems

1. Using limiter() I would only conceal the problem. My workaround is not using AVISynth 2.5 and MPEG3DEC until this issue is fixed.

2. Using MPEG3DEC3 results in no color space conversion at all. The source is in YV12, the script operates in YV12 and using VirtualDubMod and DivX 5.0.5 the whole transcoding process operates in YV12.

[Kurosu]

Hi, SIF (well choosen nick, I must admit :) )

Originally posted by SIF
2. Using MPEG3DEC3 results in no color space conversion at all. The source is in YV12, the script operates in YV12 and using VirtualDubMod and DivX 5.0.5 the whole transcoding process operates in YV12.
True, but that would assume the error solely lies in mpeg2dec3. Then what difference is there between mpeg2dec3 and mpeg2dec2? To me, it's mainly the YV12->YUY2 conversion.
I can't believe the YV12 data actually stored has Y<5, but I guess that mpeg2dec2 also decodes it as such, and that some limiter is called before outputting the YUY2 data. Maybe it is the .d2v project wrongly parsed and PC scale is always used instead of TV scale. But on a white screen, I'd suspect you also have Y>235 data...

[yaz]

@SIF:
i can't get it. if your source is yv12 then mpeg2dec will(!) convert it as it ouputs in yuy2. why do u think it's ok? maybe, there's a kinda auto-clipping involved (as in some other color tweaking filters) & u just think it's ok.
afaik, mpeg2dec3 does not do any yuv conversion & even it's not recommended with its functions (as the avs converters work well).
lumafilter (another func in the pack) is defaulted to luma offset -2 but u should call it to get the effect. so, i can't see what & how would do that strange conversion.
it does not mean that mpeg2dec3 is 'innocent' but it was checked very thoroughly during the development by a pretty hard crew. i don't think they left such a stupid bug behind. however, anything can happen:-) it'd be made clear, really.
btw, how do u check the yuv ranges of the source?

the bests
y

[SIF]

Hi yaz,

Originally posted by yaz
@SIF:

btw, how do u check the yuv ranges of the source?



One way, which does not use DVD2AVI, is using ffmpeg on GNU/Linux, MacOS or Windows. I used ffmpeg running under Cygwin:

ms@epia ~/pers/prg/cygwin/ffmpeg-0.4.7
$ ./ffmpeg -i /d/video/Trainspotting/VOB/Trainspotting_1.vob t.yuv
Input #0, mpeg, from '/d/video/Trainspotting/VOB/Trainspotting_1.vob':
Duration: 00:00:00.0, bitrate: N/A
Stream #0.0: Video: mpegvideo, 720x576, 25.00 fps, 4700 kb/s
Stream #0.1: Audio: ac3, 48000 Hz, 5:1, 384 kb/s
Output #0, rawvideo, to 't.yuv':
Stream #0.0: Video: rawvideo, yuv420p, 720x576, 25.00 fps, q=2-31, 200 kb/s
Stream mapping:
Stream #0.0 -> #0.0
Press [q] to stop encoding
frame= 18 q=0.0 size= 10935kB time=0.7 bitrate=124416.0kbits/s
q

ms@epia ~/pers/prg/cygwin/ffmpeg-0.4.7
$ dump t.yuv | less
t.yuv:

Addr 0 1 2 3 4 5 6 7 8 9 A B C D E F 0 2 4 6 8 A C E
-------- ---- ---- ---- ---- ---- ---- ---- ---- ----------------
00000000 1010 1010 1010 1010 1010 1010 1010 1010 ................
00000010 1010 1010 1010 1010 1010 1010 1010 1010 ................
00000020 1010 1010 1010 1010 1010 1010 1010 1010 ................
00000030 1010 1010 1010 1010 1010 1010 1010 1010 ................
00000040 1010 1010 1010 1010 1010 1010 1010 1010 ................
00000050 1010 1010 1010 1010 1010 1010 1010 1010 ................
00000060 1010 1010 1010 1010 1010 1010 1010 1010 ................
00000070 1010 1010 1010 1010 1010 1010 1010 1010 ................
00000080 1010 1010 1010 1010 1010 1010 1010 1010 ................
00000090 1010 1010 1010 1010 1010 1010 1010 1010 ................
000000a0 1010 1010 1010 1010 1010 1010 1010 1010 ................
000000b0 1010 1010 1010 1010 1010 1010 1010 1010 ................
000000c0 1010 1010 1010 1010 1010 1010 1010 1010 ................
000000d0 1010 1010 1010 1010 1010 1010 1010 1010 ................
000000e0 1010 1010 1010 1010 1010 1010 1010 1010 ................
000000f0 1010 1010 1010 1010 1010 1010 1010 1010 ................

Addr 0 1 2 3 4 5 6 7 8 9 A B C D E F 0 2 4 6 8 A C E
-------- ---- ---- ---- ---- ---- ---- ---- ---- ----------------
00000100 1010 1010 1010 1010 1010 1010 1010 1010 ................
00000110 1010 1010 1010 1010 1010 1010 1010 1010 ................
q

Quoting from the ffmpeg documentation:

--------------------------------------------------
The RAW YUV420P is a file containing RAW YUV planar, for each frame first come the Y plane followed by U and V planes, which are half vertical and horizontal resolution.

* You can output to a RAW YUV420P file:

ffmpeg -i mydivx.avi -o hugefile.yuv
--------------------------------------------------

So I let ffmpeg start decoding the beginning of a movie, which starts with black for the first few seconds. After decoding a few pictures I terminated ffmpeg. After that I dumped the raw yuv file to the console. As you can see the luminance values are $10 = 16. This result is the expected one according to the norm ITU-R 601.

[sh0dan]

Sounds strange!

Your RGB <-> YUV is not as AviSynth assumes. RGB(16,16,16) != YUV (16,128,128) (chrominance 128 is grey, 0 is green). In a YUV <-> RGB conversion CCIR 601 is considered, so RGB(0,0,0) = YUV(16,128,128). At least that's how all internal conversion routines work. This is why your boxes are not black (as you should specify $000000).

In AviSynth 2.5 - specify:

- MPEG2DEC3 version?
- Manually specify IDCT parameter in MPEG2DEC3.
- What does coloryuv(analyze=true) show?

[SIF]

SHORT ANSWER

See subject.


LONG ANSWER

1. VERIFICATION

I verified that both MPEG2DEC and MPEG2DEC3 do not distort luminance using a modified procedure I already described in this thread using the subject "how do u check the yuv ranges of the source?" I produced two Huffyuv encoded AVI files:

1. AVISynth 2.08 and MPEG2DEC 01-08-16, VIA Eden 533 MHz CPU
2. AVISynth 2.52 and MPEG3DEC3 03-07-27, V1.0.1.0, AMD K6-2 266 MHz CPU

I fed both files as input to ffmpeg in order to transcode to two raw YUV420P files. Dumping each file showed that in both cases the luminance has a value of 16 for the first black picture.

A slight problem showed up on step 2 on the K6-2 machine. AVISynth crashed. A fix was to specify idct=1 calling MPEG2Source().

What really puzzles me is that I can not reproduce the RGB values anymore I mentioned in my original post.


2. COLOR SPACE MISUNDERSTANDING

Originally posted by sh0dan
Sounds strange!

Your RGB <-> YUV is not as AviSynth assumes. RGB(16,16,16) != YUV (16,128,128) (chrominance 128 is grey, 0 is green). In a YUV <-> RGB conversion CCIR 601 is considered, so RGB(0,0,0) = YUV(16,128,128). At least that's how all internal conversion routines work. This is why your boxes are not black (as you should specify $000000).
...

Sh0dan, you are right! Obviously I needed a color space fresh up. I googled and the page Video for Linux Two - Image Data Formats (http://www.thedirks.org/v4l2/v4l2fmt.htm) made it clear. What we call YUY2 and YV12 are actually modified YUV formats called YCbCr. What led me in the wrong direction is that DVD2AVI, delivering RGB, does allow selecting between TV-Scale and PC-Scale. Now I do not see any right to use TV-Scale. RGB values should use the full range 0 to 255.

So I am sorry for the disturbance and I thank you all very much for the cognition. I think it is really time to start coding, because this is the definite proof of understanding.

@Kurosu:

Originally posted by Kurosu
Hi, SIF (well choosen nick, I must admit )

Thank you. You probably see that my English is German English, so I hope that "SIF" does not have any indecent meaning :)

[WarpEnterprises]

Using TV-range and TMPGenc's d2v-reader (which respects that flag) is the only way I know to read mpegs with values <16 or >240 WITHOUT clipping (you can create such an mpeg if you select "output YUV data..." in TMPG)

If I'm correct then mpeg2dec3 DOES NOT use the TV-scale-flag.
(maybe Nic/trbarry/sh0dan can comment this)

[int 21h]

Originally posted by WarpEnterprises
...If I'm correct then mpeg2dec3 DOES NOT use the TV-scale-flag.
(maybe Nic/trbarry/sh0dan can comment this)...

AFAIK, Mpeg2dec3, since it always operates in YV12, does not use the TV-scale-flag, because that flag only applies in RGB mode.

[Didée]

A sidenote:

Yesterday, I checked the deringing of mpeg2dec3's PP, by

interleave( source, source.blindPP(cpu2="ooooxx") )

and there it became obvious that deringing *lightens* the image a little bit.

- Didée

[yaz]

Originally posted by Didée
A sidenote: ... mpeg2dec3's PP ... deringing *lightens* the image a little bit.
another sidenote : lotsa filters change (at least) the Y values within a frame. just think of the techniques used for denoising, blurring, sharpening, ... in this meaning, a kinda Y(UV) distortion is always imposed. that's why YUV maintaining is always necessary. (imho!)

the bests
y

[WarpEnterprises]

But there must not occur a net change of Y!

[yaz]

Originally posted by WarpEnterprises
But there must not occur a net change of Y!
imho, 'should' not :-) but who knows? anyway, i meant here harming the tv-scale. that was dropped in at the top of the thread.

the bests
y