Squeeze levels before or after deinterlace/resize?

[nhope]

I'm frameserving 1440x1080 anamorphic interlaced footage (HDV source) in RGB24 from Vegas Pro and deinterlacing/resizing in AviSynth before passing to MeGUI for x264 encoding. Here's my script:

AviSource("d:\fs.avi")
ConvertToYV12(interlaced=true, matrix="PC.709")
AssumeTFF
LanczosResize(1280,height)
QTGMC( Preset="faster" )
SelectEven()
LanczosResize(width,720)
ColorYUV(levels="PC->TV")

Destination is primarily YouTube/Vimeo/Facebook/JW Player.

I have to upload [16,235] levels because the browser's Flash Player is expanding levels to [0,255] on playback.

Some questions...

1. Is it better to squeeze the levels with ColorYUV at the end, or better to use matrix="Rec709" to start with and leave the last line out? Which is more lossy? My theory was that the resizing and deinterlacing would have more range to work with if I did the squeeze at the end, but maybe that is cancelled out by lossiness in the extra ColorYUV operation.

2. Any difference between ColorYUV(levels="PC->TV") and Levels(0, 1, 255, 16, 235, coring=false)? I read they are the same but I see small differences on the RGB parade scope in Vegas.

3. Does anyone know what matrix Flash Player/YouTube/Vimeo/Facebook/JW Player are using for standard definition (<1280 px wide)? I read that YouTube uses Rec709 at all resolutions but I don't know if that's still the case.

Thanks!

[Yellow_]

Vegas is an NLE, can it not do all this, it's basic stuff?

The whole process seems dodgy to me, right from the start frameserving RGB from Vegas, my first query would be how good's Vegas conversion to RGB and what levels is it using for the conversion, probably 16 - 235/240 anyway. Does Vegas frameserve YV12 or YUY2? Is there a Vegas plugin that will like debugmodes?

Would exporting from Vegas as uncompressed in an .avi master file will all your transitions and corrections baked in and then doing everything else via avisynth.? Gives you the option to encode to any other formats from that master later that way too.

[nhope]

Thanks for the reply, Yellow_

Vegas uses "studio RGB" internally. It cannot work with Y'CbCr natively. Y'CbCr[16,235] gets mapped to RGB[16,235].

I am indeed using Debugmode Frameserver and Vegas gives data to it as RGB32. Frameserver can serve that RGB32 or convert to RGB24 or YUY2. QTGMC does not support YUY2, although -Vit- is adding support for it at the moment, so the path of least conversions for the time being is to serve RGB and convert to YV12 in AviSynth.

I could squeeze the levels in Vegas in a number of ways prior to frameserving (e.g. levels filter on the video output with the preset "Computer RGB to Studio RGB") but I thought I may as well retain the full range and convert the levels in AviSynth.

I prefer to use Frameserver than encode an .avi master because it's faster and there's no huge file to deal with. Quality should be the same.

[Yellow_]

Vegas uses "studio RGB" internally. It cannot work with Y'CbCr natively. Y'CbCr[16,235] gets mapped to RGB[16,235].

But what does it do to the range outside of 16 - 235? That's where all the problems start. :-) One of three things, either it discards it, maps it through 0 - 255 or squeezes the lot into 16 - 235/240. Assume last option.

Have you acertained which it does? It could be your PC to TV levels mapping has already been done by Vegas going to RGB?

[nhope]

It maps the full range. Y'CbCr[0,255] gets mapped to RGB[0,255] and I see plenty of detail in Vegas' scopes outside of [16,235]. Although some seem to disagree (http://forum.doom9.org/showthread.php?t=159861).

[poisondeathray]

It maps the full range. Y'CbCr[0,255] gets mapped to RGB[0,255] and I see plenty of detail in Vegas' scopes outside of [16,235]. Although some seem to disagree (http://forum.doom9.org/showthread.php?t=159861).


native formats like hdv appear to retain superwhites/darks, but some input formats (like lagarith, huffyuv, ut, uncompressed YV12 from vdub) have those values clipped by vegas

to answer your question you have to explain what you did in vegas (if you corrected for levels or made adjustments in vegas), and what your hdv levels are when shot. If you've left black at RGB16 and white at RGB235 , then converting to YV12 using a PC matrix will leave black at Y' 16 and white at Y' 235. PC matrix means Y'[0,255] <=> RGB [0,255]. If you have small excursions - superbrights & darks then presumably you've already recovered them in vegas. In that case, you don't need coloryuv PC->TV , because that will squish the entire range, making the image washed out with poor contrast: white will now be at ~Y' 32 and black at ~ Y' 215 . Use histogram() to monitor Y' levels .

But if you've adjusted for that in vegas (ie. adjusted it so black is at RGB 0 and white RGB 255) , then your script makes sense

If you're not making adjustments or using global filters in vegas, then what david suggested makes sense. Just smart render out HDV, and make fine adjustments with smoothlevels in avisynth.



2. Any difference between ColorYUV(levels="PC->TV") and Levels(0, 1, 255, 16, 235, coring=false)? I read they are the same but I see small differences on the RGB parade scope in Vegas.


You can check out smoothlevels. It dithers and the histogram will be smoother with less quantization, but it's slower to process

[Yellow_]

It maps the full range. Y'CbCr[0,255] gets mapped to RGB[0,255] and I see plenty of detail in Vegas' scopes outside of [16,235]

Ok, I'm not saying your mistaken but are you sure. Have you done a controlled conversion :-) to RGB via Avisynth using a codec that passes the levels straight through like FFmpeg using FFmpegSource2, checking that your source does have levels outside of 16 - 235/240 with the Classic Histogram (ie Waveform), then ConvertToRGB with the correct range you saw in the waveform and correct matrix then compared that with the same frame of your RGB output from Vegas? :-)

The 'correct' conversion to RGB would be YCbCr 16 - 235 & 240 to 0 - 255 RGB.

If you don't mind me asking what codec you're using to decompress?

Although some seem to disagree (http://forum.doom9.org/showthread.php?t=159861).

Only scanned the thread but if PD is suggesting that a full range conversion isn't happening I'd tend to agree. :-)

**EDIT** Cross post with poisondeathray

[Gavino]

ConvertToYV12(interlaced=false, matrix="PC.709")
Since you are using QTGMC, I assume your source is interlaced, so you should be using interlaced=true.
2. Any difference between ColorYUV(levels="PC->TV") and Levels(0, 1, 255, 16, 235, coring=false)? I read they are the same but I see small differences on the RGB parade scope in Vegas.
There will be slight differences in chroma between the two.
Levels() on a YUV clip scales chroma by the same ratio as luma, so the above call does not map chroma [0, 255] exactly to [16, 240], but ColorYUV(levels="PC->TV") does.

[nhope]

Gavino, interlaced=false was an error. I'm not actually using that in action. I'm using interlaced=true for interlaced and interlaced=false for progressive. Sorry for that. I've corrected it.

Thanks for other replies. I will reply about those tomorrow.

[nhope]

Vegas is an NLE, can it not do all this, it's basic stuff?
I should have mentioned that the other reasons for my workflow are that Vegas' deinterlacing is very basic. It can only do simple blending or linear interpolation. Also I'm not sure how it resizes exactly but it's not as good as Lanczos. And finally I prefer x264 over its two H.264 codecs. Sony AVC is very fast but poor at low bitrates.

...to answer your question you have to explain what you did in vegas...
In my own personal case I usually have DV .avi or native HDV .m2t on the timeline, mixed with a few titles and graphics. Plenty of my DV and HDV goes up to 255. I'm generally conforming everything to within [16,235] on the timeline on a long-winded track/media/event basis, because I know Flash Player/YouTube/Vimeo/JW Player etc. is going to expand an uploaded AVC file from [16,235] to [0,255] on playback. However in some cases I want to leave everything at [0,255] in Vegas and squeeze the levels in my AviSynth script. I'm also refining a video-for-web-upload tutorial (http://www.bubblevision.com/underwater-video/Vegas-YouTube-Vimeo.htm) where I want to recommend a generic approach to achieve WYSIWYG path from camera to web by leaving full range levels in Vegas and squeezing them in AviSynth.

You can check out smoothlevels. It dithers and the histogram will be smoother with less quantization, but it's slower to process
Thanks for the tip. I'll take a look. Approximately how much slower are we talking about for SmoothLevels(preset="pc2tv") (bearing in mind that I'm a perfectionist and QTGMC is already very slow)? I will test it.

Levels() on a YUV clip scales chroma by the same ratio as luma, so the above call does not map chroma [0, 255] exactly to [16, 240], but ColorYUV(levels="PC->TV") does.
I'll prefer ColorYUV over Levels then, but will check out smoothlevels as well.

[2Bdecided]

All my DV and HDV has black at ~16 and white at ~255. That's just standard consumer video - has been for over a decade (there were similar tricks in the analogue world too). I re-map the whole lot in AVIsynth as it passes through. This makes for some strange decisions in vegas though (i.e. increasing any good video / stills to be "too bright").

That said, I now shoot mostly in cinemode. It does go all the way to 255, but the value that I want to be white in the final render can sometimes be as low as 150 (very low light), or 225 (typical) in the raw footage. Sometimes it's even 235, but that's with lots of light.

Cheers,
David.