Color Problems: Avisynth? CCE? Shifting reds

[DoctorM]

I am trying to convert an HDTV capture to DVD. For some reason the final video has screwed up reds.

My mpeg encoder is CCE SP v2.70 (though I can duplicate the problem in v2.67).

The only change I make to the colors are the standard converttoyuy2() before sending the video to the encoder.

Below are a screenshot after the avisynth script, and a screenshot of the final mpv file that CCE made.

I checked and CCE is set for 10-bit color and I don't know what else to do to fix this. Any advice would be appreciated.

http://img357.imageshack.us/img357/7100/encodedrv1.jpg
BEFORE ENCODING

http://img79.imageshack.us/img79/2770/rawvl8.jpg
AFTER CCE

(Note: If you save these to your desktop to look at, the files are named wrong. Raw=encoded and vice versa.)

[ToS_Maverick]

did you convert the colorspace from HD (709) to SD (601)?

[DoctorM]

While the source is x264, I'm guessing that hasn't been done before hand.
Doesn't Converttoyuy2() take care of colorspace conversion?
If not, I guess I didn't do this. More info please.

Edit 1: My googling tells me TRBarry's filter is what I need, I'll give that a try.

Edit 2: Tested BT709toBT601 filter... this is ridiculous.
When done I encoded a clip and compared 4 clips.
1) Original source (post avs)
2) CCE Encoded Original (of 1)
3) BT709toBT601 converted (post avs)
4) CCE Encoded BT709toBT601 (of 3)

Red Brightness levels:
Darker to Lighter
1 > 2=3 > 4

That is the original source has the deepest reds.
The CCE encode of the original and the BT209toBT601 filtered raw video have the same red levels (but lighter than the raw source).
Finally the colorspace converted then encoded to mpeg produces the brightest reds of the whole bunch, pushing the colors even farther off.

What is going on!

[DoctorM]

Wait, it gets worse.

I just decided to test re-encode a DVD... it's doing the same thing to the color!
Mind.... boggling... Having... meltdown....

Edit: Checked raw DVD and encoded DVD frames: they are a perfect match.
Avisynth is showing incorrect colors when previewing the clip before encoding.

That makes me feel better about my final output, but I'm using avisynth to color correct.
Again I ask, what gives?

[setarip_old]

@DoctorM

Just wanted to mention that the two examples you've posted appear to differ in the saturation of ALL colors, not just RED...

[Sharc]

Is your player using the same filter for playback of the *.avs and the *.m2v? If the filters are not the same you might get color differences.

[DoctorM]

Indeed, I noticed the colors on the tool box change a bit as well.
The blues not so much. The red is the worst offender.

Both screenshots were taken with VirtualDubMod.
The input source used DGAVCDecode/ffdshow/avisynth, the mpeg output is whatever VDubMod uses to decode mpeg with.

I tried using media player classic and setting ffdshow/libavcodec for both H.264 and mpeg1 to compare the clips again. I still see the colors shift.

So to rehash:
x264 -> Avisynth -> DVD Video
Good Colors -> Good Colors -> Wonky Colors

DVD -> Avisynth -> DVD Reencoded
Good Colors -> Wonky Colors -> Good Colors

Is it something as simple as mpeg colors are different than avi colors? That's obviously not true, but it's what I'm seeing.

Edit:
x264 -> Avisynth -> Lossless AVI Format -> DVD Video
Good Colors -> Good Colors -> Good Colors -> Wonky Colors

Edit 2:
Ironically I can minimize (but not eliminate) the color changes by using BT601 to BT709. Now THAT can't be right.

[PhillipWyllie]

@DoctorM: Could you post a sample of your source? I'll try and replicate your problem. What version of 2.70 are you using?

[DoctorM]

What is the preferred format? The original source was x264 indexed with dgavcindex.
For convenience I can send the clip uncompressed, or in a lossless like huffyuv or lagarith. (Converting to lossless avi doesn't have an effect on the color, just encoding to mpeg).

[PhillipWyllie]

The source x264, how you capped it. Do a binary chop if you need to. Failing that if the x264 opens up in VDub(whether it decodes it or not) you can do a direct-stream copy on an edited section. If you do anything else you'll have decoded the x264(which is lossy).

[DoctorM]

The file does not open in VDub, and I have no idea how to do a binary chop. I'll Google it later unless you can provide some quick info.

Now that I think about it, it would be pretty nice if VDubMod added DGAVCDecode's indexing algorithm.

[PhillipWyllie]

What am I thinking, VDub only likes video that's compatable with avi. :)
You'll have to put the x264 into a container like matroska. The mkvMergeGUI part of mkvtoolnix (http://www.bunkus.org/videotools/mkvtoolnix/) has the option of splitting the resulting mkv file. Either by time-code(s) or file size, both splitting the x264 by I-frame.

[DoctorM]

Ok, so opening it in VDubMod (which won't render), a direct stream copy runs briefly and then throws an error.

It was easier to open the file in MKVMerge and have it copy the file splitting on 100mb increments and canceling after the first part was finished.
I then extracted the .h264 file (if you prefer it muxed in an mkv container let me know).

http://rapidshare.com/files/118947746/IndyCarClip.h264.html

[PhillipWyllie]

That's cool if you demuxed the mkv using mkvextract, or some other reputable matroska demuxer. You didn't mention what version of CCE 2.70 you were running. You also didn't say how you got the original image( "1) Original source (post avs)" ).

[DoctorM]

The h264 stream was demuxed using mkvextract.

CCE SP 2.70.02.12 I also tried CCE SP 2.67.00.27 with similar results.

The "original source" screenshot was taken in virtualdubmod loading the avisynth script:
loadplugin("dgavcdecode.dll")
avcsource("hdmotegi.dga")
Spline64Resize(720,426).Addborders(0,27,0,27)
selectodd().assumefps(29.97)
converttoyuy2()

The .dga file was of course generated with DGAVCIndex (v1.0.0 Alpha 23).

[PhillipWyllie]

Cool, I'll try later on today when the encoding computer has finished it's chores. Could you check the source stream to see what the DAR is set at? What was the container of the captured stream? Although DAR has no bearing on your problem I'm, gonaa give you a little advice anyway. :)
You want to keep things mod8actually 16) when encoding to MPEG-2. The black borders you add, it's best to make sure they are divisable by 8, or at the very least divisable by 4.
CCE SP2 has a nice little feature that adds the black borders for you, although you are not using that version.
On getting the DAR from a captured HDTV stream I have no idea, let's hope that neuron2 passes by.

Edit:
I've done some encoding with SP2 and have found out a few things.
Getting the source image produced different results when using DGAVCDec(PC Scale) and VDubMod(didn't matter about colour-space conversion in the avs), the reds in the image produced by VDubMod was more vivid than DGAVCs image. To get the images the same I had to use converttorgb32(matrix="rec709") in the avs script. ToS_Maverick mentioned this in his reply earlier.

Opening up the resulting encodes in DGIndex and saving a snaphot(PC Scale) produced images that were very similar to the vdubmod images, regardless of the colour-space conversion used. I wouldn't say it was CCE.

The problem is we're trying to view yv12 or yuy2 images on a monitor which can only display RGB. A real test is to see how the different encodes display on a TV, unfortunately I don't have my PC hooked up to one.

[DoctorM]

Being from an HDTV source the pixel AR is 1:1 of course @ 1280x720. I think the AR and black bordering got funny because the capper cropped ESPN's score ticker from the bottom (80 lines).

I tried to keep a correct aspect ratio during the resizing, but 1280x640 although nice and even for HD produces bad numbers trying to take it to DVD (720x427). 424 & 432 would be the closest mod8 number but farther off than I would have liked.
In retrospect it probably would have produced slightly better results, but the encoding still turned out quite good.

IIRC using mod8 is to preserve the dct blocks, reduce the bitrate and prevent artifacting(?)
It actually didn't cross my mind that the 80 lines he cropped were mod16 letting him preserve the blocks.

Anyway, from what you are saying it SOUNDS like using VDubMod to decode the mpeg2 produces different results than if you use DGIndex. That would suggest that VDubMod's mpeg decoder isn't accurate... which is much easier to live with. It would also mean that although the video looked wrong in VDubMod, it would actually have been correct.

Btw, I tested some non-x264 content the same way (Xvid to mpeg2) and found the same color shifting while using VDubMod to index the mepg2 video. I'll have to test again with DGAVCDec to see if the results are a closer match.

Edit: Ok, I think I just caught up. You are saying that VDub's preview was different than DGAVCDec's preview of the source x264 (not that I ever compared), but after encoding to mpeg2 (no color conversion) the image matched VDub's preview of the x264?
Then you are saying you can't duplicate my problem?

[PhillipWyllie]

...Then you are saying you can't duplicate my problem?Yup, whatever VDubMod showed was what DGIndex showed. Each time the reds being fairly vivid, compared to DGAVCDec(source x264). The only time I got consistant results was when I used converttorgb32(matrix="rec709") in the script. This time the reds were not as vivid but roughly the same for DGAVCDec, VDubMod, and DGIndex.

I'll post the images below. The naming convention for the images are state_source_conversion. State being avs, endoded, or original. Source being the program generating the image (dgavcdec,dgindex, or vdubmod).Conversion being the colour-space conversion used in the avs script(yv12 for none,yuy2,or rgb32_rec709). You'll see that there's very little difference in the reds from vdubmod(before encoding) and dgindex(encoded). You'll also see that in the case of rgb32_rec709 that reds are fairly consistant throughout. DGAVCDec and DGIndex were both set to PC Scale under video for saving the images, with VDubMod colour-depth being set to 24bit(I dunno if that has a bearing).

The point being is when you compare YV12 video on an RGB monitor you have to know how the the YUV -> RGB conversion was made. It's better to test the video on a TV, whether you have to make the DVD first or connect your PC upto a TV using an apropriate video card.

http://rapidshare.com/files/119184179/redShift.rar.html

[DoctorM]

I'm sorry, I'm feeling kind of thick here. I just don't understand what you are explaining.
The best I get is that you can't replicate the problem and the only way you can is by artificially shifting the colors with matrix="rec709" (or a similar setting in dgavc's preview).

Otherwise you had matching deep reds in the input and outputs regardless of what you did.
Are you suggesting the more washed-out colors are in fact the accurate ones? If so, even though you can't DUPLICATE my problem, I don't actually HAVE a problem because although my preview colors were deeper they should have been lighter like my final output?

Edit: I guess the problem then becomes why are my preview colors wrong (and what is changing them once I encode the video)?

[PhillipWyllie]

Yes it's all about how you got your preview image(s). When I played the source x264 in VLC the resulting video snapshot was even more washed out looking(but it's hard to get VLC to stop where you want and it saves in png). When viewing YUV based video(which is clipped 16,235) on rgb medium(0,255) we have to control how that 16,235 -> 0,255 conversion is taking place. Do you interpolate(the more vivid reds), or do you not(a value of 16 in the yv12 video is 16 in the rgb video and so on)<-----converttorgb32(matrix="rec709").

You're right I didn't duplicate your situation as each encode was similar to vdubmod's interpretation, but those encodes were not like the source(which was more like your source). The only way was to put converttorgb32(matrix="rec709") in the script. This meant that any preview(that's rgb) would be faithfull to the source.

Another consideration is how the program in question that makes the images decodes the video it's snapshotting. DGAVCDec uses libavcodec(I think) for instance.

The only way to be doubly sure is to watch the source, and encoded on an analogue TV. Then you'd compare.

[DoctorM]

Which is all pretty interesting from an intellectual standpoint but really doesn't point me in a direction of my problem.
You were at least managing deep reds on the final encoding and I can't. I've tried previewing in DGIndex, VLC, PowerDVD, VDub and MPC.

So in all likelihood it is not how I am viewing the final video but how it is being encoded... which is back where I started. Perhaps SP2 handles colors better.

The next answer is I should try another mpeg encoder and compare the results.

Edit:
HCenc produces the same colors as CCE suggesting either a) CCE SP2 handles colors differently than other encoders (HCenc, CCE 2.67, CCE 2.70), or b) my system isn't displaying mpeg2 video correctly regardless of software or codecs involved.

Neither answer is particularly plausible. This is very confusing.