YV12 chroma upsampling bug investigation

[bleo]

This is a report of some of my findings on the YV12 chroma upsampling bug. There is some background information in these threads: dvd2avi/mpeg2dec3 interlaced/progressive output (http://forum.doom9.org/showthread.php?threadid=74056)
DGMPGDec 1.1.0 version definition (http://forum.doom9.org/showthread.php?threadid=76609)
Bug in dgmpegdec? (Bad Color upsample) (http://forum.doom9.org/showthread.php?threadid=76878)

The source material was Kill Bill Vol. 1, DVD PAL region 4, frames 96900 (1:04:36) - 97200 (1:04:48).

First investigation: Are the interlaced flags recorded in MPEG-2 video true?

BitRate Viewer reports VTS_01_1.VOB as

Num. of picture read: 26
Stream type: MPEG-2 MP@ML VBR
Resolution: 720*576
Aspect ratio: 16:9 Generic
Framerate: 25.00
Nom. bitrate: 9800000 Bit/Sec
VBV buffer size: 112
Constrained param. flag: No
Chroma format: 4:2:0
DCT precision: 10
Pic. structure: Frame
Field topfirst: Yes
DCT type: Field
Quantscale: Nonlinear
Scan type: Alternate
Frame type: Interlaced
Notes:

I am not completely sure, but I think it means the video is interlaced

The video was indexed using DGIndex 1.0.12. It reports the Frame Type as Interlaced

The video was frameserved with AviSynth 2.5.3 using the script:

MPEG2Source("C:\usr\KILLBILL_VOL1\KILLBILL_VOL1.d2v")
Trim(96900,97200)

When viewed in VirtualDub using the Helix YV12 codec, the chroma upsampling bug was apparent.

ConvertToYUY2(interlaced=false) was added to the avs script, and when viewed in VitrualDub (as YUY2), the chroma upsampling was correct

ConvertToYUY2(interlaced=true) was changed in the avs script, and when viewed in VirtualDub, the chroma upsampling bug was there

Finding: the interlaced flag as reported by BitRate Viewer and DGIndex is not necessarily correct

Second Investigation: Which YV12 format (progressive or interlaced) does Helix Producer assume as input?

The video was frameserved with AviSynth using the script:

MPEG2Source("C:\usr\KILLBILL_VOL1\KILLBILL_VOL1.d2v")
Trim(96900,97200)

The video was encoded using Helix Producer build 10.0.0.545 at 2.9 Mbit/s

When viewed using RealPlayer 10 the chroma upsampling was correct

Finding: although all combinations have not been tested, at this stage it is assumed that Helix Producer assumes progressive YV12 as input

Discussion:
If the interlaced flag cannot be trusted, then having DGDecode automatically upsample to YUY2 based on it will not be the 'be all end all' solution to the chroma upsampling bug.

It is also favourable to keep the video as YV12 throughout the encoding chain for speed and 'purity'. Therefore, we should investigate for each encoder whether it accepts progressive or interlaced YV12. Then we may ConvertToYUY2 if the source YV12 format and encoder are mismatched.

For example, if I determined through visual inspection that my source had interlaced YV12, regardless of what the interlaced flag was or whether it was a film or 50 field/sec video, then I should ConvertToYUY2(interlaced=true) before encoding to RealVideo 10.

[vispgraedde]

Originally posted by bleo
It is also favourable to keep the video as YV12 throughout the encoding chain for speed and 'purity'.

If you need to filter the video, staying in YV12 is from my experience a bad idea.

An example was when I added subs (yes, hardsubs) to a video in YV12 with the then latest vsfilter.dll, and I in the video could see visible artifacts that was not there if I applied the filter in YUY2 or RGB, and downsampled to YV12.

In general I've found that upsampling to YUY2 before filtering gives better results than filtering in YV12.

just my 2 cent

[bleo]

Fair enough point vispgraedde. Though I don't want to comment too much about which filters work properly in YV12, I hope that the artifacts that you saw were not due to incorrect chroma upsampling... :p Converting to YUY2 and then back to YV12 may have 'inverted' the chroma to the interlaced/progressive format that your encoder/viewer was made for.

By the way, I have just found two more PAL region 4 DVDs that appear to be flagged interlaced but show incorrect chroma upsampling with ConvertToYUY2(interlaced=true). So, this is not an isolated incident...

[trbarry]

We've long assumed that the repeat flags for force film are sometimes not reliably encoded. My own guess is the interlaced flag may not be much better. That's why I prefer to use Telecide or TomsMoComp in YV12 to make progressive material and then convert (if needed) to YUY2 as if it was progressive.

- Tom