Help with ConvertToYV12 for game video

[Tuubi]

I have a smallish issue... I'd like to make sure that I do the colour space conversion properly. So here's the deal: I use AviSynth 2.58 and MeGUI (x264) for encoding. What do I need to do to get correct output for game captures? For capturing I use Dxtory and UtVideo RGB (ULRG) lossless codec.

After reading a lot I was under impression that ConvertToYV12(matrix="pc.709") would be the right one to use since the source is RGB and PC levels, right? However, it results in dark output. Using just the default ConvertToYV12() seems to give right results (but I'm still confused, doesn't it assume TV levels?).

So in short, assuming the source video is lossless RGB, what do I need to do for avisynth and x264 to get the correct output?

[IanB]

TV levels (16 to 235) and PC levels (0 to 255) only apply to YUV colour spaces. RGB is nearly always 0 to 255 and all Avisynth conversion filters assume this default.

Most real world components assume YUV signals are TV levels unless you explicitly do something to the contrary.

The default for ConvertToYV12() is to us matrix="Rec601". The alternative is to use matrix="Rec709". Some PC software in the absence of other information by default assumes SD size video is Rec601 and HD size video is Rec709. The actual change over values are implementation specific, some use greater than 576p, some use less than 720p, some do something else.

It is best to always tag your video encodes explicitly with the correct colour space information and not rely on dubious default behaviour.


If you use ConvertToYV12(matrix="pc.709") then you need to be able to tell all the components involved right up to the video output display that you are using a PC levels YUV signal, not a lot have this options.

[raffriff42]

It seems to need the default matrix, Rec601. Here's the script I used to determine this #avisynth
fpath="test DX-ULRG.avi"

## choose one:
#C=AviSource(fpath)
#C=DirectShowSource(fpath)
C=FFVideoSource(fpath) ## seems a little faster

## ** phase 1 - get decoder luma range

#return (C.IsRGB)
#\ ? C.RGBAdjust(analyze=true).Subtitle("RGB", align=2)
#\ : C.ColorYUV(analyze=true).Subtitle("YUV", align=2)
## (result == 0-255, RGB for all decoders)

## ** phase 2 - convert to YV12, evaluate (in MPC-HC + Haali)

#C=C.ConvertToYV12(matrix="PC.709").Subtitle("PC.709", align=1) ## result == blown out, matrix error
#C=C.ConvertToYV12(matrix="PC.601").Subtitle("PC.601", align=1) ## result == blown out
C=C.ConvertToYV12(matrix="Rec601").Subtitle("Rec601", align=1) ## result == good (range 16-235, YUV)

return (C.IsRGB)
\ ? C.RGBAdjust(analyze=true).Subtitle("RGB", align=2)
\ : C.ColorYUV(analyze=true).Subtitle("YUV", align=2)
__END__ Examine the LumaY, "LooseMinimum" & "LooseMaximum" overlays on a variety on source images. I get 0-255 on all decoders. On the other hand, UT Video 420/709 (ULH0) luma range varies, depending on chosen decoder. (EDIT AviSource and DirectShowSource results may also vary depending on the installed codecs on your system - another reason to use FFVideoSource whenever possible)

Note, my finding of "blown out" playback in MPC-HC may look perfect in your video editor. From what you report, I don't think so.
(EDIT when I say "blown out" here, I mean light areas are too light and dark areas are too dark. Caused by decoders expanding luma range when not needed, or by feeding 0-255 video to a player that expects only 16-235.)

Completely off topic - why record in RGB and immediately convert to YV12? Not judging or anything; maybe you have multiple workflows in parallel...

[Warperus]

Tuubi
You need Rec709 matrix if you encode it h264 (or mpeg-2 HD). It better to set color-matrix, transfer characteristics and color primaries settings in encoded file, even though defaults to these 2 cases are bt.709 with tv levels.
You need Rec601 matrix if you encode it in mpeg-4 part 2 (divx, xvid).

raffriff42
Dxtory, Bandicam, PlayClaw use BT.601 conversion matrixes for YUV recordings. If your video ends up in h264, you'll have to either convert colors bt.601->bt.709 (=colors degradation) or encode with non-conventional color matrix bt.601 indicating bt470bg in x264 settings (=filters not aware of color matrix settings will process video it incorrectly). It's safer to keep original colors and color resolution for video processing.
Also it's possible to process video in RGB and only after processing convert it into YV12.

[Tuubi]

@raffriff42
I record in RGB because I want to get lossless frames for, say, screenshots - it's more flexible. Also from my experience using YV12 in lagarith or UTVideo doesn't seem to do that good job when it comes to the quality. Although if I only want to shoot video and know that, then I could of course use just YV12 to begin with.

@Warperus
Rec709 really?

I've made some tests with VirtualDub.
Original AVI:
http://abload.de/img/default1dulw.png

Default ConvertToYV12:
http://abload.de/img/avs_defaultp3um3.png
Looks like the original in terms of tones and brightness.

rec709
http://abload.de/img/rec709xkukk.png
Colours change a bit.

pc.709
http://abload.de/img/pc7094muik.png
Clearly too dark.

Results with raffriff42's script (that's really handy!):
http://abload.de/img/scriptpmkdl.png

So I assume it's safe to use the default i.e. Rec601?

[raffriff42]

Agree with most of the theory mentioned, but I'm big on testing everything, because color space implementation varies so much for various applications.
BT.601 and BT.709 compatibility benchmark (http://forum.doom9.org/showthread.php?t=164378)

Thanks, I'm glad you like the script! I think it's pretty handy too!

After going mad squinting at screen grabs like those, I developed FRAFS Test Pattern (https://sourceforge.net/projects/frafstestpatt/), a Direct3D test pattern generator for calibrating Fraps & Dxtory etc. I use this in combination with ColorPic (http://www.iconico.com/colorpic/), a screen color probe, or the color probe in PhotoShop etc.

[Warperus]

Tuubi
I assume you load videos into VirtualDUB and take screenshots. Do you see the same picture in software player?

[Tuubi]

Tuubi
I assume you load videos into VirtualDUB and take screenshots. Do you see the same picture in software player?
With original capture? Yeah.

[Warperus]

VirtualDUB should get RGB from your YV12 file, so it uses yet another "default" YV12->RGB converter. Why not something like default ConvertToRGB (which uses Rec601 matrix by default)?
Please process some small video in avisynth+megui and load resulting file into VLC, MPC or other stand-alone software player.

[Tuubi]

VirtualDUB should get RGB from your YV12 file, so it uses yet another "default" YV12->RGB converter. Why not something like default ConvertToRGB (which uses Rec601 matrix by default)?
Please process some small video in avisynth+megui and load resulting file into VLC, MPC or other stand-alone software player.
I do not use avisynth when I work with virtualdub. I only use it to direct stream copying (i.e. cutting and maybe appending) and I use RGB files. The conversion only happens when I encode to h264 with x264.

[Tuubi]

By the way... Now that I've done some more testing. It looks like using rec709 actually gives better results (the same as source) when actually encoding to h264 (or VC-1 using Expression Encoder SP2) and actually playing the encoded video in WMP using Windows 8 built in codec or VLC. It also look correct on youtube (webm and h264 using flash, but ie10 and html5 h264 gives wrong results although still better than Rec601 version). Using avisynth to view the h264 in virtualdub gives different results than when watching the encoded video, in that case it's the opposite. this is complicated.

[TheSkiller]

Using avisynth to view the h264 in virtualdub gives different results than when watching the encoded video, in that case it's the opposite. this is complicated.
If you decode the Rec.709 h264 video in AviSynth you feed uncompressed YV12 to VirtualDub unless you include ConvertToRGB(matrix="rec709") in your script.

VirtualDub now calls for some Codec installed on your computer to do the YV12->RGB conversion which is needed to display the video within VDub. And this is where anything might happen, hence your results. For previewing purposes one really should feed nothing but raw RGB into VirtualDub.

If you do it right, there will be no differences at all between a correctly flagged 601 version and a correctly flagged 709 version (this does not apply to YouTube however, YouTube is dumb, ignores flags and assumes 709 for anything, no matter if HD or SD).

[Tuubi]

If you decode the Rec.709 h264 video in AviSynth you feed uncompressed YV12 to VirtualDub unless you include ConvertToRGB(matrix="rec709") in your script.

VirtualDub now calls for some Codec installed on your computer to do the YV12->RGB conversion which is needed to display the video within VDub. And this is where anything might happen, hence your results. For previewing purposes one really should feed nothing but raw RGB into VirtualDub.

If you do it right, there will be no differences at all between a correctly flagged 601 version and a correctly flagged 709 version (this does not apply to YouTube however, YouTube is dumb, ignores flags and assumes 709 for anything, no matter if HD or SD).
Indeed. That gives correct result. Thanks all, I think I got this.

[turbojet]

How about determining matrix/colorspace automatically by converting to common matrice/colorspaces and comparing rgb levels to the original. Is this possible?

[Tuubi]

I just added external encoders for virtualdub and that works now fine but there just one issue: It appears the colour space conversion get bugged somewhere. I've tried everything I know yet reds have clear orange tint. I'm using these settings for command line parameters: --crf 19 --preset slow --demuxer raw --input-res %(width)x%(height) --fps %(fpsnum)/%(fpsden) --colormatrix bt709 -o "%(tempvideofile)" -

The input video is RGB. I've tried changing the colour depth, used convert format filter but nothing changes. Any ideas?

[sneaker_ger]

What is the source?
What is the resolution of the source?
Are you doing any resizing?
In what player are you seeing the tint?
In what player do you see the original for comparison?
How are you feeding the source to VirtualDub?
Are you using AviSynth?
If so, what does the script look like?
What video processing mode are you using in VirtualDub?
What filters are you using in VirtualDub?
What input and output formats is VirtualDub showing for these filters?
What exactly have you selected as decompression and output format under Color Depth in VirtualDub?

[raffriff42]

Agree, :script:

After some testing, I would suggest:#avisynth
f="p:\ath\to\DXtory UT-RGB source.avi"
C=FFmpegSource2(f, atrack = -1)
#C=AviSource(f) ## use either source; there is no color difference
C=C.ConvertToYV12(matrix="Rec709") Preview in Vdub appears wrong (green+, red-) because it's 709, but encoded output (using external x264, flagged as Bt709) is correct, as viewed in MPC-HC w/ Haali renderer, color=auto, range=TV.
EDIT you may use Alias Format (Rec.709) for true colors in preview, but disable it before encoding.

[Tuubi]

No avisynth involved (when I talked about input video, I really meant pure 1080p RGB video without avisynth usage - heck, or png image sequence), tint happens in every player (WMP, VLC, Firefox, IE10). Pure RGB video in AVI container. No filters. If I use Avisynth and ConvertToYV12(matrix="Rec709") it shows a bit off in virtualdub but the encoding result is fine obviously. I just want to know what Virtualdub does when it send the raw video or what ever to x264. For some reason changing the video colour depth options makes no difference to encoded output. Neither does the convert format filter. It feels like it converts it to something...

From performance point of view it would probably be better to not use avinsyth if possible. Why doing the colour space conversion with avisnyth works but not when doing it with virtual dub? It looks like Rec709 in preview but then suddenly the output is clearly different when you use the external encoder option.

[raffriff42]

>I just want to know what Virtualdub does when it send the raw video or what ever to x264.
VirtualDub help - Processing: External encoder support
...
The color space is 4:2:0 YCbCr using Rec. 601 limited-range, 8-bit per channel encoding. The normal excursions are 16-235 for luma and 16-240 for chroma, but values from 0-255 may be encoded. Chroma positioning is per MPEG-2 conventions, with chroma samples coaligned with luma samples horizontally and centered vertically. Frames are always encoded as progressive scan.

Don't take this to mean you should convert to Rec601. Just pass the YUV-709 video, unchanged, to x264 and it will be handled correctly. (this is why I told you to ignore the preview window)

[Tuubi]

>I just want to know what Virtualdub does when it send the raw video or what ever to x264.


Don't take this to mean you should convert to Rec601. Just pass the YUV-709 video, unchanged, to x264 and it will be handled correctly. (this is why I told you to ignore the preview window)
The thing is... No matter what I do, I can't seem to do that without using Avisynth. It should output YUV-709 after converting the RGB video using the filter, right?

This is what happens.

RGB source, converted to Rec709 with virtualdub filter (works as intended as far as I can see):
http://abload.de/img/convertp3sl9.png

RGB source, converted to Rec709 with ConvertToYV12(matrix="Rec709") using avisynth:
http://abload.de/img/avsfnsju.png

With the first method the H264 output is this (result is the same without using the virtualdub conversion filter too):
http://abload.de/img/h264n8sgx.png
(Red is R:255 G:25 B:0)

With the second method the H264 output is this (correct in other words):
http://abload.de/img/h264_right5lsnv.png
(Red is R:255 G:0 B:0)

[raffriff42]

>Why doing the colour space conversion with avisnyth works but not when doing it with virtual dub?
Try opening your UT-RGB source in VirtualDub, full processing, no filters, color depth input=auto, output=RGB24, compress with x264vfw ("--colorprim bt709 --transfer bt709 --colormatrix bt709"). The results look OK to me.

Your original question seemed to be about Avisynth and external encoders, but these are not actually needed.

EDIT: This is what happens...Don't use 100% color bars; you can't tell if there is clipping going on.

[sneaker_ger]

Tuubi seems to be right and it does confirm to what you have quoted from the VirtualDub help. I am trying to output RGB input to 709 using external encoders but there seems to be no way to do that without resorting to AviSynth. It just ignores everything you set in Color Depth or the convert format filter or at least the desired result is net achieved. Of course you could switch to VfW, but the external encoder option has some advantages, like being able to use both VirtualDub 32 and x264 64 and having easy access to up-to-date builds. Also, not all encoders are available for VfW/ACM in the first place.

Maybe this question should be asked directly to the author in the VirtualDub forum, he might know the solution or could be asked to make the different color matrices work with the external encoders in a future version.

[Warperus]

AFAIK x264 uses BT.601 color matrix for RGB->YUV conversion if it receives RGB input and procudes YUV output, and there are no command line settings to change this matrix. So I don't think you'll have correct result unless you use avisynth.

[sneaker_ger]

VirtualDub does not support RGB output to external encoders in the first place so that is not applicable to the situation at hand.

[Tuubi]

>Why doing the colour space conversion with avisnyth works but not when doing it with virtual dub?
Try opening your UT-RGB source in VirtualDub, full processing, no filters, color depth input=auto, output=RGB24, compress with x264vfw ("--colorprim bt709 --transfer bt709 --colormatrix bt709"). The results look OK to me.

Your original question seemed to be about Avisynth and external encoders, but these are not actually needed.

EDIT: Don't use 100% color bars; you can't tell if there is clipping going on. BTW I created a test pattern for DirectX capture programs (http://sourceforge.net/projects/frafstestpatt/) (like Dxtory and Fraps) for diagnosing these color matrix issues. Try it. Color matrix errors are much easier to spot with proper color bars.
x264vfw is quite hacky (especially if you use it in AVI container) and I'd rather do it properly (I use NeroAAC for audio or FLAC if I use MKV container). Since I know how it would look like, I used solid pure red in this case to visualise the issue. When working with RGB source, the encoded result is always the same no matter what.

Tuubi seems to be right and it does confirm to what you have quoted from the VirtualDub help. I am trying to output RGB input to 709 using external encoders but there seems to be no way to do that without resorting to AviSynth. It just ignores everything you set in Color Depth or the convert format filter or at least the desired result is net achieved. Of course you could switch to VfW, but the external encoder option has some advantages, like being able to use both VirtualDub 32 and x264 64 and having easy access to up-to-date builds. Also, not all encoders are available for VfW/ACM in the first place.

Maybe this question should be asked directly to the author in the VirtualDub forum, he might know the solution or could be asked to make the different color matrices work with the external encoders in a future version.
Exactly. It does indeed look like I need to ask dev about this. By the looks of it everything is fine if the source is in desired colour space (i.e. do the conversion with avisynth) but if you try to do anything to it inside virtualdub it either is ignored or converted back to something before feeding it to external encoder.

[raffriff42]

Success! :cool: Source - UT Video RGB
Full processing mode
Filter 1: Convert Format 4:2:0 YV12 Rec.709
Filter 2: Alias Format color=Rec.601, range=no change
x264 --colorprim bt709 --transfer bt709 --colormatrix bt709

[sneaker_ger]

Great find, I totally didn't see the alias format filter up there. Can confirm it works.

[Tuubi]

Success! :cool: Source - UT Video RGB
Full processing mode
Filter 1: Convert Format 4:2:0 YV12 Rec.709
Filter 2: Alias Format color=Rec.601, range=no change
x264 --colorprim bt709 --transfer bt709 --colormatrix bt709


Perfect! That works. By the way, it looks like x264 can auto guess it right without manually changing the colour matrix stuff in this case. At least every player I've tried shows it right without them.

[sneaker_ger]

x264 does not tag the files if you don't use e.g. --colormatrix bt709. Players will auto-guess 709 for HD video, so it works either way, but flagging is recommended nonetheless.

[Tuubi]

x264 does not tag the files if you don't use e.g. --colormatrix bt709. Players will auto-guess 709 for HD files, so it works anyways.
I see, I probably leave it off for now or create two different settings for HD and SD content.

Now if only VirtualDub would have some kind of workaround for two pass encoding... Maybe in the future. It would be nice if it would be possible make it so that one could make 1st pass setting and 2nd pass setting and it woudl automatically do both before muxing. But anyway, this works wonderfully now.

[sneaker_ger]

Hmm... maybe you could create a 1st encoder set which only does first pass video encoding and a 2nd set which does the second pass + audio + muxing. External encoder seems to be allowed as a batch operation.

[Tuubi]

Hmm... maybe you could create a 1st encoder set which only does first pass video encoding and a 2nd set which does the second pass + audio + muxing. External encoder seems to be allowed as a batch operation.

Damn, totally forgot batch operation option - I need to toy with it.