UpScaling

[simple_simon]

I'm upscaling a bunch of dvd videos to 720p using Dogway's deep_resize script. I'm also converting the colorspace from 601 (SD)--> 709 (HD). Should i also be converting the color range? Is the standard color range for HD video Full or is it still supposed to be Limited? Also should I be converting to 16bits before applying filters and then dithering back to 8bits at the end? Or does that not matter? I've tried googling this but can't find any definite answers.

[Reel.Deel]

Consumer video is almost always limited. Converting to full range may lead to wrong levels on some hardware players, while most well known software players have no problem as long as it is flagged properly.

[Boulder]

Also should I be converting to 16bits before applying filters and then dithering back to 8bits at the end?

Depends on your destination format, and also the encoder.

[simple_simon]

Encoding with x264. Format will be archival mkv with no specific playback requirements/limitations.

I was thinking maybe the filters would perform more effectively & accurately with the wider bitdepth.

[Boulder]

Encoding with x264. Format will be archival mkv with no specific playback requirements/limitations.

I was thinking maybe the filters would perform more effectively & accurately with the wider bitdepth.

It is recommended to convert to high bitdepth for processing, that is true. You might want to test even converting to YUV420P32 before resizing. With x264, I think it's better to convert to the final bitdepth before feeding the source to the encoder. If you don't need to worry about playback limitations, downsample to 10 bits and go for the High10 profile. It will reduce banding in the final result and also compresses slightly better.

[simple_simon]

Is there any benefit to converting all the way up to YUV444P32 and then dithering back down to 8bit at the end of the script?

[DTL]

I'm upscaling a bunch of dvd videos to 720p using Dogway's deep_resize script. I'm also converting the colorspace from 601 (SD)--> 709 (HD). Should i also be converting the color range? Is the standard color range for HD video Full or is it still supposed to be Limited?

Full is only for old poor designed PC programs like 30 years old at times of Windows 3 without ability to process and display motion pictures data in 'limiited'. Also for some very limited by bits HDR formats like DolbyVision 10bit to have more dynamic range but less spatial quality.

"Is there any benefit to converting all the way up to YUV444P32 and then dithering back down to 8bit at the end of the script?"

It depends on your DVD content. May be unique for every title. For really best digital video quality it is required to operate with RGB linear (that is RGB 444 and linear transfer and bits about 10..12 minimum for SDR, 16bits or float32 are better). Only in RGB linear you can get best quality of linear scalers. But if your source and target formats are in badly distorted by design chroma-subsampled and non-linear transfered 420 from poor past you may not see significant difference.

[FranceBB]

With x264, I think it's better to convert to the final bitdepth before feeding the source to the encoder.


At this point, it's really a matter of taste.
When you feed x264 with a bit depth higher than its target (like 16bit planar from Avisynth targeting 10bit planar as High10), it will use the Sierra A2-4 error diffusion to dither down to its target bit depth (like from 16bit to 10bit planar).
Dithering inside Avisynth makes sense only if the user prefers a different dithering like Stucki or Atkinson (or for whatever reason ordered dithering), but given that a good 80% of people use the Floyd Steinberg Error Diffusion, it won't matter that much as both Sierra and Floyd Steinberg are two very well balanced algorithms. ;)
Me, personally, I use Floyd Steinberg for x262 ('cause it wouldn't be able to do anything other than rounding to 8bit), while I leave x264 and x265 do their thing as they're supposed to play nicely with themselves and know about how to encode the patterns in a more efficient way. :)

[Boulder]

Is there any benefit to converting all the way up to YUV444P32 and then dithering back down to 8bit at the end of the script?

I personally wouldn't go 4:4:4 with DVD sources.

When you feed x264 with a bit depth higher than its target (like 16bit planar from Avisynth targeting 10bit planar as High10), it will use the Sierra A2-4 error diffusion to dither down to its target bit depth (like from 16bit to 10bit planar).
I probably remembered this wrong for x264 -- x265 has the separate --dither switch for higher quality dithering but x264 does not. I remembered that it will just do a simple truncating but good if it doesn't :)

[FranceBB]

I probably remembered this wrong for x264 -- x265 has the separate --dither switch for higher quality dithering but x264 does not. I remembered that it will just do a simple truncating but good if it doesn't :)

So, x265 will use a very simple dithering unless --dither is used, while x264 doesn't have a switch and USED to always truncate years ago, but then Sierra A2-4 was introduced and since it doesn't have a switch that has become the default behavior for every time the bit depth is higher than the target bit depth. ;)
x262 still truncates, though, so you need to dither down to 8bit for that one.

[wonkey_monkey]

Format will be archival mkv

If it's archival... wouldn't it be best not to process it at all?

[DTL]

For the already damaged by 4:2:0 subsampled content the best scaler may be something content-adaptive non-linear and such class of scalers may still progress in quality. So may be better not to scale today by current versions of scalers and simply keep content as it is in archive for possibly better future.

[simple_simon]

If it's archival... wouldn't it be best not to process it at all?

Ideally. But I want to clean up mpeg-2 compression artifacts, remove noise, sharpen, antialias and then upscale. Cleaning up the video file and saving in the same resolution as the source seems to just reintroduce edge artifacts and noise when it gets upscaled at playback. Upscaling the file to at least 720p after filtering seems to retain the cleaned up video better at playback.


So this is my usual script for most dvd upscaling (minus source specific tweaks):

tfm(mode=1, cthresh=8, pp=7)
tdecimate(mode=1, dupthresh=1.4)
z_convertformat(pixel_type=last.pixeltype, colorspace_op="170m:709:709:full=>709:709:709:full").propSet("_ColorRange",1)
neo_fft3d(sigma=5.0,y=2)
smdegrain(prefilter=4,interlaced=false)
hqderingmod()
finedehalo()
#edgecleaner(strength=20)
maa2()
lsfplus(strength=60,preset="slow",soft=30)
propSet("_FieldBased",0).deep_resize(1280,720,grain=0,qual=2,gpuid=-1,show=false)
#gradfun2dbmod(str=0.0,thr=1.2)
addgrainc(3,0)
prefetch(2)


Since it appears x264 dithers to the necessary bitdepth, I don't have to worry about that. But what is the best way to convert to YUV420P32? And where in the script is the best place to do it? Before converting the colorspace?

[Boulder]

Since it appears x264 dithers to the necessary bitdepth, I don't have to worry about that. But what is the best way to convert to YUV420P32? And where in the script is the best place to do it? Before converting the colorspace?

You can use z_ConvertFormat for it, the pixel_type parameter is the one to use. See the Avsresize page for a neat table of values. I'd try converting to 32-bit float in the colorspace conversion call. If the subsequent filters don't support 32-bit float input, YUV420P16 is a good choice instead. Maybe then convert to 32 bits before deep_resize.

Shouldn't you use "colorspace_op="170m:709:709:l=>709:709:709:l" since you have a DVD source?

http://avisynth.nl/index.php/Avsresize

[simple_simon]

You can use z_ConvertFormat for it, the pixel_type parameter is the one to use. See the Avsresize page for a neat table of values. I'd try converting to 32-bit float in the colorspace conversion call. If the subsequent filters don't support 32-bit float input, YUV420P16 is a good choice instead. Maybe then convert to 32 bits before deep_resize.

Thanks. For some reason I thought pixel_type was for setting the input. Looks like I'm using filters before and after resizing that won't work with 32-bit so I think I'm just going to stick with converting to 16-bit. Everything works with that and I don't have to do any converting back and forth.

Shouldn't you use "colorspace_op="170m:709:709:l=>709:709:709:l" since you have a DVD source?

http://avisynth.nl/index.php/Avsresize

According to the documentation for avsresize, it's recommended to use full-->full even when dealing with limited color range in order to avoid rounding errors on the darkest/brightest pixels. The color range still remains limited if the input was limited. That's why I add the _propset for the color range at the end so that any subsequent filters won't get confused.

Go to the link you provided and scroll down near the bottom where it shows examples and look at the notes.

[PoeBear]

What's the best/modern way to dither for those that would like to try 444 when upscaling? Usually I keep things in 8-bit, because whenever I try to do YUV444P16, and apply my z_ConvertFormat dithering options at the end, it would produce visual flaws like light highlighting or haloing, that would use up more bitrate in CRF mode. I tried the 3-4 built-in dither options and they mostly looked the same. And even though the 8-bit path wasn't as pretty side-by-side, they don't produce any sort of extra noise and CRF bitrate goes down, so I stuck with it. But I've been playing with some upscalers, and AiUpscale+KrigBilateral really makes some DVDs look a lot nice, but it only outputs 4:4:4

[FranceBB]

According to the documentation for avsresize, it's recommended to use full-->full even when dealing with limited color range in order to avoid rounding errors on the darkest/brightest pixels.

Yes and no, it depends on your use case and it's not rounding errors, it's clipping.
This is what's gonna happen:

limited->limited

the range will be kept limited tv range, however if there are overshooting over 0.7V or undershooting under 0.0V they will be clipped out, no if, no but.
If, however, for some reason, you DO want to keep those superwhite and superblacks, then:

full->full

will be ok.
In other words, let's suppose you have an 8bit limited tv range video 16-235 and let's suppose that in a certain scene the cameraman recorded a river with some light reflections of the sun going over 235 at like 240.

If you use: limited->limited, those will be clipped thus getting a perfectly legal stream staying between 16 and 235. If you're a broadcaster and you have no one else checking the content afterwards, this is what you want.

If you use: full->full, the content will STILL be limited tv 16-235 range as it originally was, however, given that avsresize THINKS that it's dealing with a full pc range 0-255 content, it will preserve the overshooting, so the 240 values of the light reflection in the river in the fictional scenario I used as an example. However keep in mind that you DON'T WANT those extra high values to stay there, so you MUST bring them down yourself afterwards to have everything within the limited tv range threshold for real. This is because TVs and PC monitors are RGB Full PC Range and when you play a Limited TV Range flagged content, 16 is expanded to 0 and 235 is expanded to 255 anyway, so those values you preserved at 240 would be shifted over 255, but there's nothing other than pure white beyond 255 so they would be lost anyway.



I would show you a graph and an example using my VideoTek() and Levels() etc, but it's 22 minutes past midnight and I just watched Chelsea lose yet another game, but this time in the Champions League... and... I'm just not in the mood right now...

[poisondeathray]

Yes and no, it depends on your use case and it's not rounding errors, it's clipping.
This is what's gonna happen:

limited->limited

the range will be kept limited tv range, however if there are overshooting over 0.7V or undershooting under 0.0V they will be clipped out, no if, no but.


limited=> limited in avsresize does not clip ; if you have overshoots they are retained

For that "colormatrix" operation, you cannot "see" the difference f=>f vs. l=>l on "normal" content; it's only detectable by amplified differences

[simple_simon]

limited=> limited in avsresize does not clip ; if you have overshoots they are retained

For that "colormatrix" operation, you cannot "see" the difference f=>f vs. l=>l on "normal" content; it's only detectable by amplified differences

poisondeathray, your forum post is the one sited as a reference for why it's recommended to use full=>full instead of limited=>limited to avoid clipping. Are you saying that's no longer true?

[FranceBB]

@poisondeathray... interesting, I might be wrong, but I too somehow remember one post about this somewhere and it was yours. Maybe is my memory eluding me... But ok, if it's not clipping, even better, one reason more for me to stick with limited->limited :D
Thanks for the info. ;)

[DTL]

What's the best/modern way to dither for those that would like to try 444 when upscaling? Usually I keep things in 8-bit, because whenever I try to do YUV444P16, and apply my z_ConvertFormat dithering options at the end, it would produce visual flaws like light highlighting or haloing, that would use up more bitrate in CRF mode. I tried the 3-4 built-in dither options and they mostly looked the same. And even though the 8-bit path wasn't as pretty side-by-side, they don't produce any sort of extra noise and CRF bitrate goes down, so I stuck with it. But I've been playing with some upscalers, and AiUpscale+KrigBilateral really makes some DVDs look a lot nice, but it only outputs 4:4:4

In best way if you got some good looking 444 it is no more need to go to old poor and ugly chroma-subsampled 2:1 compression of 420. If your current playback workflow not support 444 - you may make temporal 420 copy for today displaying (with any required bitrate with CRF encoding and delete it after viewing to save storage space) and store 444 in archive for future use when your playback devices will support 444 playback from archive directly.

In even more better future it is good to use RGB linear archival storage to avoid some possible issues from transfer-function and quantization compression. So it is about 16bit (10..12 minimum for about 1000:1 linear range, but 16 is simply computer-typical with 8bit bytes granularity) linear for SDR and 32bit float for HDR linear.

[FranceBB]

Yeah, DTL is right, encoding it in 4:4:4 would be the preferred way if it's for archival purposes, however if you really have to go back 4:2:0 for hardware playback compatibility, the least painful way to do that is to resize the chroma using an appropriate resizing kernel made for downscales and in that case your best bet would be Sin squared kernel and thanks to DTL it has been integrated into Plugins_JPSDR's SinPowResizeMT(), however it's not in the Avisynth core, so it's not in ConverttoYUV420() as a selectable resizing kernel... :(

In terms of dithering from 16bit to a lower bit depth, if your encoder supports it (x264 or x265) let the encoder do it (in case of x265 you have to specify --dither), otherwise if you're using x262 or xvid, use the Floyd Steinberg Error Diffusion with dither=1 in ConvertBits() which is a well balanced algorithm. Don't go to Stuki or Atkinson using Dither_tools unless you really know what you're doing, PoeBear, but for the sake of information, I'll explain: Stuki will preserve the edges between objects and will therefore look sharper than other dithering algorithms, however it MIGHT still be prone to a bit of banding, while Atkinson is the other way around and there's no banding at all, but flat areas are averaged and therefore is blurrier. You can think about Floyd Steinberg as something in between.

Someone might make the argument that those dithering algorithms will generate patterns that will make life harder for encoders to encode the content due to the non correlation between blocks and macroblocks and he would be right, however nowadays encoders are pretty good at recognizing those patterns. Still, this is exactly why I said that if you have x264 or x265 doing the encode you should leave them handling the dithering: 'cause they will do it with a dithering algorithm that will play nice with the subsequent motion compensation tools and will therefore theoretically allow you to save bitrate while retaining the same quality and without having to go into something obvious like ordered dithering which makes life easier for the encoder but looks bad-ish to the human eye once your brain starts to recognize the static pattern.

[poisondeathray]

poisondeathray, your forum post is the one sited as a reference for why it's recommended to use full=>full instead of limited=>limited to avoid clipping. Are you saying that's no longer true?

I never wrote anything about "clipping". It never has clipped. But there are differences in some dark and bright pixels - but they are not "clipped" to limited range (almost undetectable without metrics or amplified differences). Probably this post:
https://forum.doom9.org/showthread.php?p=1906754&highlight=full#post1906754

If you test the newest avsresize, the behaviour is the same.

[DTL]

Yeah, DTL is right, encoding it in 4:4:4 would be the preferred way if it's for archival purposes, however if you really have to go back 4:2:0 for hardware playback compatibility, the least painful way to do that is to resize the chroma using an appropriate resizing kernel made for downscales and in that case your best bet would be Sin squared kernel and thanks to DTL it has been integrated into Plugins_JPSDR's SinPowResizeMT(), however it's not in the Avisynth core, so it's not in ConverttoYUV420() as a selectable resizing kernel... :(



The question is really complex enough.
First I open an issue at AVS+ development github https://github.com/AviSynth/AviSynthPlus/issues/337 about adding more resize kernels inside AVS+ core - it is anyway good practice to allow more users to use more different downsizers with 'conditioning' effect and not only very soft GaussResize.

Next is about 4:2:0 compression in colour systems: The digital production standards like Rec.601 and other directly not define anti-Gibbs conditioning at 4:4:4 to 4:2:0 compressor for UV downsampled channels (it is the task of end of chain display device). If you look at the provided lowpass filters graphs in these Recs you see it is not anti-Gibbs conditioning for 2:1 downsampling of source anti-Gibbs conditioned RGB/UV 4:4:4 channels (well - it is not easily seen from the graph curve but it is).
It is done because Rec.601 is defined for studio and other professional multi-generations conversions of 444->422->444 and so on and if we place conditioning filter in the 444->42x conversion it will degrades chroma sharpness in multi-generations conversion (because it applied to the valid Nyquist frequencies in 2:1 downscaled chroma).

Some hint: you can not apply any spatial transform to 42x->444 converted data before applying anti-Gibbs filtering to UV chroma in the 42x->444 convesion device or you may got additional chroma ringing or other distortions. The 'immediate' 42x->444 conversion is only applicable to non-spatial processing like YUV/RGB adjust, overlay and other non-including any resampling. So studio 4:2:2 SDI may pass main program mixer with internal 4:4:4 (YUV or RGB) processing (like RGB DSK overlay with graphics and channel branding) and be downconverted to 4:2:2 SDI without intermediate filtering (faster and chaeper in hardware).

The next is the most sad story about current motion pictures digital industry from the previous century (from the Rec.601) - the anti-Gibbs condition filter in the end of chain display device still not put to any standard. It mean the total digital moving pictures workflow documentation still incomplete. So every display (and PC player and PC hardware) vendor may or may not implement any possible filter to 4:2:2 and 4:2:0 decompression and playback device.
This cause the issue for archiving attempt with 4:2:0 compression - it may not playback well at the future 4:2:0 playback devices with sometime finally put to standard anti-Gibbs conditioning filter for UV channels for display device. So you may play with adjustments of current downconversion from 4:4:4 to 4:2:x but you only can control result at _existing_ display devices with some (unknown) vendor-defined conditioning filter (because display device need to convert to RGB linear for displaying). For some future display devices the transfer function of filter may be other and 4:2:0 content may need additional re-adjustment. So it is one of the point to keep 4:4:4 content as it is and not attempt to compress to 4:2:0 with still uncomplete documentation for 'reference displaying/decompression' for long archival storage.

[FranceBB]

there are differences in some dark and bright pixels - but they are not "clipped" to limited range (almost undetectable without metrics or amplified differences)

I see! It was my memory eluding me, then, my bad, really!
The "bad" thing, though, is that this is not an outlier, this is occurring to me more and more often in recent months, which, honestly, is a bit scary...

[DTL]

Also if user like to play with UV filtering at 4:2:0 content creation it possible in scripting form like
1. separate planes
2. apply lowpass filtering to UV planes
3. perform 2x downsampling
4. combine planes to YUV

For example: Current AVS ConvertToYV420() uses closer to industry standards for 'intermediate' UV chroma filtering (for application in multi-generations conversion with minimal chroma sharpness losses, with default 'bucubic' kernel and its internal default settings) and if user's display device do not provide additional and/or appropriate filtering and it cause visible chroma ringing or other distortions at the displayed image - user can apply additional filtering to UV of created 4:2:0 content. But it may cause additional chroma sharpness degradation (chroma blurring) when displaying at other devices.

Any filtering with existing kernels may be applied also without resampling with AVS or jpsdr's 'resize' processing providing slightly non-zero (like 0.00001f) float src_left 'shifting' param to force convolution processing with selected kernel and its settings.

Also may be about equal processing:
1. separate planes
2. apply lowpass filtering to UV planes
3. combine planes to YUV
4. use ConvertToTV420() with 'point' chromaresample to not touch existing filtering of UV planes performed in step 2.