Gamma-aware, dithered colorspace conversion and resizing

[PhrostByte]

Hello all, I've made a new filter that performs gamma-corrected colorspace conversion and Spline36 resizing in a single function.

Download version 8 for 32-bit:
http://sourceforge.net/projects/int64/files/ResampleHQ/ResampleHQ-v8.zip/download

Full Documentation:
http://svn.int64.org/viewvc/int64/resamplehq/doc/index.html?revision=341

Kernel visualization:
http://svn.int64.org/viewvc/int64/resamplehq/doc/kernels.html

Usage:
ResampleHQ(clip
[, int width]
[, int height]
[, string dstcolorspace]
[, string srcmatrix]
[, string dstmatrix]
[, bool dither]
[, float src_left]
[, float src_top]
[, float src_width]
[, float src_height]
[, string kernel]
[, float karg1]
[, float karg2]
[, float kblur_x]
[, float kblur_y]
[, string chroma_kernel]
[, float chroma_karg1]
[, float chroma_karg2])

Too many settings to list here, see documentation above for what they all do.

Note: Dithering YV12 and YUY2 output is not currently supported and will come at a later version. Interlaced formats are also not supported.

Here's an example from Avisynth's gamma-ignorant Spline36Resize:

http://svn.int64.org/viewvc/int64/resamplehq/doc/earthlights_small1.png

And now here's the gamma-aware ResampleHQ version:

http://svn.int64.org/viewvc/int64/resamplehq/doc/earthlights_small2.png

Note how the ResampleHQ image is much brighter overall, and shows faint light in completely new places around Canada, Alaska, western United States, and along coastlines.

This image was cherry-picked to show how off Avisynth's resizers can be with sharp contrast. Most video will show far less improvement.

[Dogway]

music to my ears! will test it soon.

[jmac698]

http://avisynth.org/mediawiki/External_filters#Resizers
Updated.

[Dogway]

Just tested.
I can see dither still is not optimal.
Also I wanted to ask if the chroma around some edges is normal, notice the reddish outline:
http://i212.photobucket.com/albums/cc35/Dogway/Misc/raw.png
http://i212.photobucket.com/albums/cc35/Dogway/Misc/spline64-1.png
http://i212.photobucket.com/albums/cc35/Dogway/Misc/ResampleHQ.png

[PhrostByte]

Thanks for trying it out. I see the red edges you're talking about, I'll check it out.

Dithering isn't even there yet. The code is finished but not yet integrated. Next build will have it.

[Archimedes]

What a lucky break. In the last view days I was thinking about the same issue (making a plugin for gamma corrected resizing). And now I see, somebody has already done the job. :)

Regarding the picture above, I can't reproduce red edges with ResampleHQ.

Spline36Resize >> (http://img833.imageshack.us/i/rawspline36resize.png/)
ResampleHQ >> (http://img692.imageshack.us/i/rawresamplehq.png/)

[Dogway]

I imagine it has to do with yuv content, and chroma subsampling. But maybe that is the expected behaviour, I was just wondering from my ignorance.
Talking about kernels any options for automttap3? Its a function so maybe it implies kernels from bilinear and lanczosresize.

[PhrostByte]

The red edges were due to a chroma conversion bug with Y'CbCr formats. I've fixed it already, am finishing up dithering before next release though. Some time today probably.

[pbristow]

This is good to see! :)

Once all the bugs are out of the colourspace/gamma correction & dither code, will you then be adding other options for the resize algorithm (e.g. Lanczos, or even good old Bilinear & Bicubic? :) )?

Is it modular enough that you could just add an option such as ' "resizer" = string ', and use it to call any existing AVIsynth resizer in between the input and output stages? That would open to door to things such as gamma-aware EDI, etc.

[Archimedes]

There is also a bug with the alpha channel (RGB32 input).

[PhrostByte]

This is good to see! :)

Once all the bugs are out of the colourspace/gamma correction & dither code, will you then be adding other options for the resize algorithm (e.g. Lanczos, or even good old Bilinear & Bicubic? :) )?

Is it modular enough that you could just add an option such as ' "resizer" = string ', and use it to call any existing AVIsynth resizer in between the input and output stages? That would open to door to things such as gamma-aware EDI, etc.

Other sinc kernels (Lanczos, Spline, etc.) will be added soon. Non-sinc algorithms will come later.

Unfortunately it's not possible to reuse other filters.

There is also a bug with the alpha channel (RGB32 input).

Actually didn't realize the alpha channel was used by anything. Easy to fix, at least. I'll add it to the list.

[PhrostByte]

v2 is up, download here (http://sourceforge.net/projects/int64/files/ResampleHQ/ResampleHQ-v2.zip/download).

I'll be busy the next couple days so I decided to put out what I've finished. All the bugs identified are fixed (please test to make sure!), and dithering works for RGB output. I unfortunately wasn't able to finish YV12 and YUY2 dithering in time, so that'll come later.

[Dogway]

Thanks for all the effort!
I will wait until you release the dither version for a current project.
And hopefully make this one my default resizer.

[leeperry]

music to my ears!
to mine as well :D

I'm currently forced to use this for upscaled SD through madVR:
ColorMatrix(mode="Rec.601->Rec.709",clamp=0,opt=0)

I already use SmoothLevels() for a proper TV>PC conversion, it does add dithering...and so does madVR, so I'm not sure that I've got any headroom for more dithering.

How is your plugin supposed to be different/better than the aforementioned command? I strictly want 601>709, and especially not something that messes w/ saturations :o

[PhrostByte]

How is your plugin supposed to be different/better than the aforementioned command? I strictly want 601>709, and especially not something that messes w/ saturations :o

TV.601->PC.709 doesn't require any linear blending, so my filter won't be able to do much for you. Two possible improvements which would be negligible:

If you're using both ColorMatrix and SmoothLevels together, then ResampleHQ will give you less rounding error.
Proper dithering does require linear blending, so ResampleHQ will give more correct results than anything that does non-linear dithering. I'd be (pleasantly) surprised if whatever dithering SmoothLevels does is performed linearly.


The brightness difference you see in those pictures is actually what correct resizing is supposed to look like. ResampleHQ doesn't mess with brightness, saturation, or anything like that at all -- everything else just does it wrong :)

[IanB]

Neat filter.

For those interested in the source code, it appears to be here (http://svn.int64.org/viewvc/int64/resamplehq/resamplehq/).

The dither seems to be error diffusion with a twist. r1 and l1 flip sign on each line. This is fine for l1 because it applies to the rerr2[] pointers of the next line, but r1 applies to the rerr1[] pointers of the current line, so the accumulating error is added to an already processed pixel :confused:

[PhrostByte]

the accumulating error is added to an already processed pixel :confused:

if you scan left->right, then l1 is an offset to the left pixel, and r1 is an offset to the right pixel.

it alternates between scanning left->right and right->left (see the calculation of x). I flip the sign on l1/r1 each line too, so r1 is always an offset to the pixel next in line.

it's called "serpentine scanning" and helps avoid some rare bad looking patterns.

[Mug Funky]

a lil' feature request?

could you possibly implement cropping routines similar to avisynth's resizers? these allow precise (subpixel) cropping of the image which is necessary in all manner of video resizing applications (ie field alignment, correct 702 width scaling from and to HD, etc).

thanks!

btw, very nice work :)

[IanB]

if you scan left->right, then l1 is an offset to the left pixel, and r1 is an offset to the right pixel.

it alternates between scanning left->right and right->left (see the calculation of x). I flip the sign on l1/r1 each line too, so r1 is always an offset to the pixel next in line.

it's called "serpentine scanning" and helps avoid some rare bad looking patterns.

Ah, yes I missed this code color::dimension_type x = (y & 1) ? (width - xx - 1) : xx;

You have obviously thought about this a lot. So on the Even lines you diffuse error with this kernel 1 [ - # 7 ]
16 [ 4 5 0 ]and on the Odd line with this kernel 1 [ 7 # - ]
16 [ 0 5 4 ]

What other advantages have you found instead of using the default Floyd and Steinberg kernel? 1 [ - # 7 ]
16 [ 3 5 1 ]

[PhrostByte]

a lil' feature request?

could you possibly implement cropping routines similar to avisynth's resizers? these allow precise (subpixel) cropping of the image which is necessary in all manner of video resizing applications (ie field alignment, correct 702 width scaling from and to HD, etc).

Definitely planned.

What other advantages have you found instead of using the default Floyd and Steinberg kernel? 1 [ - # 7 ]
16 [ 3 5 1 ]

A while ago I read a study that did an automated comparison of various dithering techniques, including all common error diffusion patterns and even randomized ones. It concluded that while the Floyd-Steinberg pattern was near perfect for regular scanning, it was actually sub-optimal for serpentine scanning. The pattern I'm using tested best in its class, outmatched only by a slow and complicated edge-directed binary search algorithm.

I tried it out myself on a few handfuls of images and found that it did help avoid certain bad patterns, so I stuck with it.

[madshi]

FWIW, most of these dithering algorithms are optimized for halftoning (8bit -> 1bit). I believe that for our use (high bitdepth -> 8bit) we have different needs. I would suggest to use the method proposed in the paper "Optimized Error Diffusion for Image Display" (PDF download available via google). This dithering method whitens the quantization error which should produce best results for our needs, IMHO.

But of course the optimal solution would be if AviSynth would be able to handle a full high bitdepth processing chain (16bit int or even 32bit float). This way dithering could be avoided altogether, or at least delayed to the very last step. E.g. dithering could be done to 10bit, with the final result fed to the new x264 10bit encoder. The current limitation of AviSynth to 8bit means that basically every processing step that internally produces higher bitdepth (resampling, color conversion, and many more) must do an internal dithering step for optimal results. If this is not done, rounding errors will add up. But even if it's done, dithering noise adds up. Having higher bitdepth support would be better in any case.

[PhrostByte]

FWIW, most of these dithering algorithms are optimized for halftoning (8bit -> 1bit). I believe that for our use (high bitdepth -> 8bit) we have different needs. I would suggest to use the method proposed in the paper "Optimized Error Diffusion for Image Display" (PDF download available via google). This dithering method whitens the quantization error which should produce best results for our needs, IMHO.

But of course the optimal solution would be if AviSynth would be able to handle a full high bitdepth processing chain (16bit int or even 32bit float). This way dithering could be avoided altogether, or at least delayed to the very last step. E.g. dithering could be done to 10bit, with the final result fed to the new x264 10bit encoder. The current limitation of AviSynth to 8bit means that basically every processing step that internally produces higher bitdepth (resampling, color conversion, and many more) must do an internal dithering step for optimal results. If this is not done, rounding errors will add up. But even if it's done, dithering noise adds up. Having higher bitdepth support would be better in any case.

I've also wondered if the classic error diffusion algorithms were optimal for high bit depths. From my tests it seems to perform acceptably, but if there's something better I definitely want to know. Interesting paper, I'll read it. Thanks.

I've been considering making a backwards-compatible extension to IClip -- it would have GetFrame16 and GetFrame32F member functions, and transparently down-convert if GetFrame was called by a legacy filter. All anyone would need to do is call dynamic_cast<IClipEx*>(clip) to test if a clip offers the extended formats. It'd be a fair amount of work, but I think rewarding in the end. Would people use this?

[jmac698]

Yes, I would use it. For now I'm using a script-compatible system where the extra low bits are in a separate clip. There's other filters that use deep color processing as well. Could you make it compatible to my tools for now? It would be the easiest approach.
http://forum.doom9.org/showthread.php?p=1467907#post1467907

[PhrostByte]

v3 is out. sub-pixel cropping, selectable resize kernels, and some small optimizations. big overhaul to support SSE touched a lot of code, so be on the lookout for bugs.

Download for 32-bit and 64-bit:
http://sourceforge.net/projects/int64/files/ResampleHQ/ResampleHQ-v3.zip/download

Full documentation:
http://svn.int64.org/viewvc/int64/resamplehq/doc/index.html?revision=255

[PhrostByte]

Yes, I would use it. For now I'm using a script-compatible system where the extra low bits are in a separate clip. There's other filters that use deep color processing as well. Could you make it compatible to my tools for now? It would be the easiest approach.
http://forum.doom9.org/showthread.php?p=1467907#post1467907

Unfortunately transparent compatibility is basically impossible -- all the filters you're depending on would have to explicitly support it. Until they did, the video they see would just appear like any other clip (low bit depth and non-linear colorspace), so you wouldn't be able to get much out of it.

Given the amount of filters we have that haven't been updated in ages and have had their authors disappear, I don't really know how big of an impact it would be able to make. But I guess we have to start somewhere?

[Dogway]

Perceived bugs:
-Omitting or setting dither=false, causes a crash.
-looks like dither only works when changing colorspace from yv12 to rgb (not on resizing etc)
-Strange black artifacts around contrasty edges when DSTCOLORSPACE is RGB (independent of input)
-Overall there's strong ringing, black outline plus bright ringing.

Tell me if you want me to upload my tests.

[PhrostByte]

Perceived bugs:
-Omitting or setting dither=false, causes a crash.
-looks like dither only works when changing colorspace from yv12 to rgb (not on resizing etc)
-Strange black artifacts around contrasty edges when DSTCOLORSPACE is RGB (independent of input)
-Overall there's strong ringing, black outline plus bright ringing.

Tell me if you want me to upload my tests.

Please do, I've tested all the possible colorspace conversions and can't replicate any of that.

[Dogway]

Dither=false crashes no matter what. I even emptied the plugin folder.
Here are my tests:
http://www.mediafire.com/?l6t0pf2mfto32xg

[PhrostByte]

Dither=false crashes no matter what. I even emptied the plugin folder.
Here are my tests:
http://www.mediafire.com/?l6t0pf2mfto32xg

Thanks! v4 is up with everything fixed.

The crashes were a bug from my SSE preparations, and the dithering artifacts were due to resizing creating out-of-range values (which is normal, I just didn't clamp them properly).

I believe the extra ringing is merely a result of linear blending. Regular Spline36 has it too, just not as intense -- just like with other images, the sharp contrast is brought out more with linear blending.

Download for 32-bit and 64-bit:
http://sourceforge.net/projects/int64/files/ResampleHQ/ResampleHQ-v4.zip/download

Full documentation:
http://svn.int64.org/viewvc/int64/resamplehq/doc/index.html?revision=257

[tritical]

I believe the extra ringing is merely a result of linear blending. Regular Spline36 has it too, just not as intense -- just like with other images, the sharp contrast is brought out more with linear blending.

If you are enlarging an image, working with linear values will absolutely create more ringing around edges simply because you are enlarging the difference between dark/bright pixels... so the overshoot of resampling functions with negative coefficients will be greater than if you work with gamma corrected values. Really, for enlarging there is no reason to believe that working with the linear values will give a better result (with standard fir interpolation functions).. because the resampling functions do not approximate the true image better when fit to the linear values than when fit to the gamma corrected values. At least I have never seen any evidence of this, and my tests show that it isn't true.

Shrinking or downsizing an image is different because you have to low-pass filter the image before decimation (or interpolation) in order to avoid aliasing. It is the low-pass filtering step that requires working in linear space if you want to approximate an optical blur. Avisynth's resizing functions don't separate those two steps. The low-pass filtering is accomplished by increasing the support width of the resampling function by a factor of old_width/target_width. Obviously, the larger the downsizing ratio the stronger the low-pass filtering, and therefore the larger the difference between working with gamma-corrected versus linear values.

Random thought: I don't know how flexible Avisynth's resampling code is, but it would be really nice if you could manually control the support width - maybe by supplying a user specified scaling factor >= 1.0 - instead of having it always scaled by a factor of max(old_width/target_width,1.0).

[*.mp4 guy]

Increasing support width will blur things to all hell. The algorithm avisynth uses should be able to do this easily, iirc, if the code itself is this capable, I have no idea.

[cretindesalpes]

Nice! It would be great if you could also add support for 16-bit data as output and/or input (for example, with the MSB part stacked onto the LSB), so the dithering could be done later.

[Dogway]

@PhrostByte: Yes, everything looks right now : D
Just a question, still images should be treated as... Rec.601?

Also I made a funny experiment, ported resampleHQ to automttap3:
http://i212.photobucket.com/albums/cc35/Dogway/Misc/th_automttap3HQ.png (http://i212.photobucket.com/albums/cc35/Dogway/Misc/automttap3HQ.png)

[PhrostByte]

Just a question, still images should be treated as... Rec.601?

The matrix is only used for YV12/YUY2 (it defines the mix of R,G,B that goes into Y, and the integer range for Y'CbCr). For everything else it is ignored. I'd assume still images would be RGB.

[Dogway]

ay! yes, what I had been thinking...

[tritical]

Increasing support width will blur things to all hell. The algorithm avisynth uses should be able to do this easily, iirc, if the code itself is this capable, I have no idea.

What I'm saying is that Avisynth's current method of scaling the support by max(old_width/target_width,1.0) is just a rule of thumb. There is no reason to believe that that value is optimal for every image, and it would be interesting if you could supply the scaling factor yourself (after looking at the code I think this would be very easy to do). However, what I'm really thinking about is if you were to perform low-pass filtering yourself before calling one of the built in resizers. In that case, you wouldn't want the filter support increased because you don't need low-pass filtering just interpolation. Anyways, getting off topic now.

[*.mp4 guy]

What I'm saying is that Avisynth's current method of scaling the support by max(old_width/target_width,1.0) is just a rule of thumb.

Avisynths current method is naively theoretically correct for purposes of keeping the frequency response of the output as close to the input as can be done using the current resizing kernel. The only way to do better is to break out the non-linearity. Though, as you mentioned, there are other reasons to make it adjustable. I just don't want people to think that the avisynth interpolators are arbitrary.

[PhrostByte]

What I'm saying is that Avisynth's current method of scaling the support by max(old_width/target_width,1.0) is just a rule of thumb. There is no reason to believe that that value is optimal for every image, and it would be interesting if you could supply the scaling factor yourself (after looking at the code I think this would be very easy to do). However, what I'm really thinking about is if you were to perform low-pass filtering yourself before calling one of the built in resizers. In that case, you wouldn't want the filter support increased because you don't need low-pass filtering just interpolation. Anyways, getting off topic now.

I guess you mean min(target_width/old_width, 1.0). I'll add an option to let you override it.

[tritical]

Avisynths current method is naively theoretically correct for purposes of keeping the frequency response of the output as close to the input as can be done using the current resizing kernel.
Under what [arbitrary] definition of close? Certainly it does not completely capture the only thing that really matters for image processing - the perceived quality of the person doing the resizing.

[SilaSurfer]

PhrostByte nice work!;) Just tried it.

@Dogway

How did you port resampleHQ to automttap3, and was that image resampled with it? Looks awasome.

[*.mp4 guy]

Under what [arbitrary] definition of close? If the avisynth linear interpolators could use an infinite sinc kernel, and and the input to the interpolator is a perfectly bandlimited signal capable of being properly sampled at the destination resolution, then the support width calculations used by avisynth guarantee perfect reconstruction at the destination resolution, and any other arbitrary resolution that does not cause sub-critical sampling to occur.

Certainly it does not completely capture the only thing that really matters for image processing - the perceived quality of the person doing the resizing.
In my opinion, it is overall the best compromise. Avisynth resizers certainly look much better then the abysmal interpolators used on most dvds; said dvds either being blurred to hell, with accompanying gibbs artifacts or have aliasing everywhere.

[Sapo84]

I think that YV12 output is a bit broken at the moment, and there are visible artefacts, in particular they are pretty obvious near reds border.
But, well, let's make screenshots speak.

ResampleHQ(1280,720,"YV12","TV.709","TV.709",false,198,22,-48,-164)
http://www.cbland.net/images4/ResampleHQYV12.png
around the red circle there are obvious problems with the resampler output

Spline36Resize(1280,720,198,22,-48,-164)
http://www.cbland.net/images4/SplineYV12.png
on the other hand Spline36's output is perfect

and if we convert to RGB32
ResampleHQ(1280,720,"RGB32","TV.709","TV.709",false,198,22,-48,-164)
http://www.cbland.net/images4/ResampleHQRGB32.png
there are no more artefacts

Spline36Resize(1280,720,198,22,-48,-164).ConverttoRGB32(matrix="Rec709")
http://www.cbland.net/images4/SplineRGB32.png
Spline is good as well.

Using different matrix/crop/dither did not change the outcome at all.

[PhrostByte]

I think that YV12 output is a bit broken at the moment, and there are visible artefacts, in particular they are pretty obvious near reds border.

Confirmed. I'll look for a solution. Chroma sub-sampling is evil.

[henryho_hk]

Sapo84, can we have the original picture for comparsion? And what program do you use to extract the PNG file from a YV12 AVS?

[Dogway]

Yes, I can confirm it, YV12 is totally broken. Even the next line outputs wrong results:
# converting from RGB to YV12
ResampleHQ(DSTCOLORSPACE="YV12",dither=false)

[Sapo84]

Sapo84, can we have the original picture for comparsion?
Sure.
http://www.cbland.net/images4/notresampled.png

And what program do you use to extract the PNG file from a YV12 AVS?
AvsPmod.
Now that you can even specify the matrix used to convert from YV12 to RGB32 during playback is basically the swiss knife of avisynth ^^

[PhrostByte]

This is turning out to be a tricky problem to solve. It occurs whenever sub-sampled YUV is converted between gamma-compressed and linear.

You can't go directly from YUV -> linear YUV. You need to do YUV -> RGB -> linear RGB -> linear YUV.

When it is sub-sampled, you run into the same problem run-of-the-mill renderers do: when converting to RGB the chroma isn't rescaled nicely, but simply point sampled.

I wanted to avoid rescaling gamma-compressed chroma, but I'm not sure if that's possible now. Any clever ideas?

[Mug Funky]

could the gamma resize be just done on the luma only? chroma exists in chroma-land where gamma makes no sense...

or perhaps the gamma can happen on the chroma channels using a correspondingly subsampled luma channel to convert them to intermediate chroma channels. that might work.

[PhrostByte]

could the gamma resize be just done on the luma only? chroma exists in chroma-land where gamma makes no sense...

or perhaps the gamma can happen on the chroma channels using a correspondingly subsampled luma channel to convert them to intermediate chroma channels. that might work.

Chromaticity is independent from luminance. Chroma is dependent on luma. Not the same thing :(

I thought of sub-sampling luma too, but I haven't tried it yet. Will report back in a bit :)

[madshi]

Resampling luma will change the luma and chroma value of every pixel. So if you want resampling to be done with proper quality, I don't think you have any chance of avoiding doing proper chroma upsampling as a first step. Of course that means that the output of your filter should probably be either RGB or YCbCr 4:4:4 because otherwise chroma will lose quality. For chroma upsampling I'd suggest a soft filter to avoid aliasing artifacts.