Color banding and noise removal

[-Vit-]

Manalyze on half size clip with pel=1
Mflow/blur/whatever on full clip with pel 2 or 4
That's the wrong way round. The idea behind adjustSubPel is that vectors measured in half-pixels (pel=2) on a half-size clip can be left unchanged and treated as measured in full pixels (pel=1) on a full size clip. Only the pel setting in the vectors changes. I was experimenting with this before I had to stop avisynth development for more important things. My initial observations were that it worked, but did not provide any advantage over scaling the vectors and leaving the pel alone.

Maybe you had a different idea...?

Also, vector scaling doesn't work with divide>0.
Yes, this is true and should be added to the documentation. I'd had enough after deciphering the internal layout/format of standard multilevel blocks/padding. I guess I'll add that final (rarely used) layer at some stage.

[Dogway]

I had a look to your workaround (http://forum.doom9.org/showthread.php?p=1505297#post1505297)to make yuy2 colorspace sources be processed by ditherpost, but I'm having a hard time to understand how it works.

Do you apply that code to an 8 bit interleaved yuy2 ?

I'm testing by comparing the next to its yv12 counterpart:

(interleaved 8 bit yuy2 source)

y = ConvertToYV12 ()
u = UToY ().ConvertToYV12 ()
v = VToY ().ConvertToYV12 ()
YToUV (u, v, StackVertical (y, y))

dfttest(lsb=true)
ditherpost(mode=0)

y = Crop (0, 0, 0, Height () / 2).ConvertToYUY2 ()
u = UToY ().ConvertToYUY2 ()
v = VToY ().ConvertToYUY2 ()
YToUV (u, v, y)

But there's something different in the chroma planes so I wouldn't call that lossless, maybe I'm doing something wrong?

[cretindesalpes]

Yes this code applies to standard YUY2. It works by copying the chroma planes from the YUY2 clip to the YV12. Therefore the overall height must be doubled, so I have to stack two original luma planes. The filtering will work only if each plane is processed separately because chroma and luma are not synchronized. Fortunately, this is the case with dfttest and DitherPost.

If you compare the result with a YV12 source, it's normal you get something different on the chroma planes: regular YV12 to YUY2 conversions are not lossless because of the chroma placement and the interpolation method. If you bypass the dfttest/DitherPost lines in your code, you should obtain the exact source clip.

[Dogway]

I almost finished implementing it to my script but there's something that doesn't work.
separatefields
interleaved2planar
super_search=MSuper(planar=true)
bv2=super_search.MAnalyse(isb = true, delta = 2)
fv2=super_search.MAnalyse(isb = false, delta = 2)
MDegrain1(super_search, bv2, fv2, thSAD=300,planar=true,lsb=true)

It only happens when I use separatefields with lsb=true. Even setting assumeframebased doesn't fix it and always throws an error. This is the base for interlaced YUY2 denoising but I don't know what else to try.

[cretindesalpes]

Indeed, there was a bug with planar=true and lsb=true. Fixed in Dither 1.13.2 (http://forum.doom9.org/showthread.php?p=1386559#post1386559).

[Stephen R. Savage]

cretindesalpes,

Where are these custom versions of dfttest and mvtools you use coming from? I can't find any information about a dfttest 1.9 or mvtools 2.5.14. Can you provide a link to the original versions of these filters or are they solely your creation?

[Dogway]

It works like charm now! I forgot to ask, do you think if by adding a blankclip to the below stacked luma can save processing for ditherpost?
YToUV (u, v, StackVertical (y, blank_clip))

[cretindesalpes]

Where are these custom versions of dfttest and mvtools you use coming from? I can't find any information about a dfttest 1.9 or mvtools 2.5.14. Can you provide a link to the original versions of these filters or are they solely your creation?
Yes, these are my own updated versions of these tools. It seems they are no longer updated by their authors (excepted occasional bugfixes for mvtools) so I took the liberty to continue their development. I hope it doesn't hurt anyone.

do you think if by adding a blankclip to the below stacked luma can save processing for ditherpost?
No, I don't think it will change anything.

[mastrboy]

Do the modified MVtools in the dither package contain the threading fixes from the QTGMC modded plugins package by -Vit- ?

Little annoying that there are multiple forks of mvtools2 out there now, the official one, one from the dither package, one from QTGMC and one from www.svp-team.com :(

[cretindesalpes]

Yes, this version has the QTGMC threading fixes.

[mastrboy]

Thanks for answering so fast and that's good to hear :)

(You should probably update your post: http://forum.doom9.org/showthread.php?p=1386559#post1386559 so anyone else looking for the same info don't have to browse 18 pages...)

[-Vit-]

Little annoying that there are multiple forks of mvtools2 out there now, the official one, one from the dither package, one from QTGMC and one from www.svp-team.com :(
I'll remove my version from the QTGMC thread and use cretindesalpes' version for the modded plugins shortly. One down...

[mastrboy]

Good idea. A second idea to narrow it down further would be if both of your changes would be merged with the official release... Fizick seems to be the only "active" developer according to the changelog at least, hopefully he would read this post and accept your changes for 2.5.11.4.. :P

[SSH4]

cretindesalpes Is it possible add to dfttest (or may be to mdegrain) option like sstring(ssx/y/z) for change sigma depending on luma? I know different ways for simulate this (use luma as mask and merging filtered and original not the best one) but all of them require two passes or more of dfttest or mdeg :(

[Dogway]

Could you confirm one thing?. Motion vectors are very different when you take into account chroma between yv12 and yuy2 (planar) sources.
Since chroma is subsampled in yv12 I guess it is upsampled before or after calculating its motion vectors to comply to luma motion vectors.
Does this happen according to chroma in YUY2 sources taking into account the way chroma is stored in this format (when planar=true)?

And since I'm here, would you recommend mod16 inputs for more correct SAD calculation?

[cretindesalpes]

Since chroma is subsampled in yv12 I guess it is upsampled before or after calculating its motion vectors to comply to luma motion vectors.
All SAD calculations are scaled depending on the plane subsampling. For example, if you specify blksize=16, luma SAD part will be calculated on 16x16 blocks, 4:2:2 chroma on 8x16 blocks and 4:2:0 chroma on 8x8 blocks.

Motion vectors are very different when you take into account chroma between yv12 and yuy2 (planar) sources.
How different? Do you think there is a bug related to YUY2?

And since I'm here, would you recommend mod16 inputs for more correct SAD calculation?
I don't think using mod16 (or more likely a mod-overlap size) does any significant difference, because motion estimation is always funny on the frame borders.

Is it possible add to dfttest (or may be to mdegrain) option like sstring(ssx/y/z) for change sigma depending on luma?
It looks possible at the first glance; I made a note in my to do list. An additional improvement would be using the pixel values of a second clip as sigma multipliers. For MDegrain, you can obtain an equivalent effect by changing the luma curve of the search clip. What I usually do: prepare the search clip using dfttest(lsb=true) with basic MC, Dither_lut16() to apply a gamma-like curve and make it full range, convert it back to 8 bit using DitherPost(mode=-1) and feed MSuper (for the final MAnalyse only, not the actual filtering) with it. I think you could do a similar approximation with dfttest by applying a curve on the input clip luma (result in 16 bits to keep accuracy), filter with lsb=true, lsb_in=true then by applying the inverse function. Obviously, it would affect only the luma plane, the chroma plane filtering would be kept unchanged whatever the corresponding luma value.

Fizick seems to be the only "active" developer according to the changelog at least, hopefully he would read this post and accept your changes for 2.5.11.4.. :P
He wasn't interested when I asked him.

[Dogway]

I'm not very sure that's why I ask you if you could have a look. On a simple temporal radius of 1 there is this such difference (in the images below). If I set chroma false both output are 100% identical. Maybe this is just a matter of chroma being in different formats but difference catched my attention enough to not mention here.
planar=true
chroma=true
plane=0

planar ? interleaved2planar : last
super_search=MSuper(planar=planar,chroma=chroma?true:(plane==0?false:true))
bv2=super_search.MAnalyse(isb = true, delta = 1,chroma=chroma)
fv2=super_search.MAnalyse(isb = false, delta = 1,chroma=chroma)
MDegrain1(super_search, bv2, fv2, thSAD=300,planar=planar,lsb=false,plane=plane)
planar ? planar2interleaved : last
http://img849.imageshack.us/img849/6548/yv12.th.png (http://img849.imageshack.us/img849/6548/yv12.png)http://img190.imageshack.us/img190/6876/yuy2.th.png (http://img190.imageshack.us/img190/6876/yuy2.png)

edit:
...Dither_lut16() to apply a gamma-like curve and make it full range...
Very good idea! Do you also degamma? I'm not sure if this is preferred.

It also would be quite useful to add an interlaced parameter to manalyse for when multi=true, so interleaved delta order becomes ...-6,-4,-2, 2, 4, 6...

In another matter, I realised some time ago that in order to create motion vectors you don't need thSAD, this is only required for MDegrain (and well mrecalculate...), maybe the chroma smearing could be reduced by playing with plane and thSAD parameters running 2 instances of MDegrain but using the same vectors... since you know the internals of mvtools2 could you confirm this?

[Yellow_]

Is it possible to blend with float values and if so with values greater than 0.0 to 1.0 with Dither tools?

If I have two frames of video (YV12) of the same scene but with two exposures say one is 1 EV greater than the other, ie: twice as bright. Is it possible to blend them into a 16bit linear light frame with brightness values representing 0.0 - 2.0 and so on for wider EV spacing.

[Stephen R. Savage]

Is it possible to blend with float values and if so with values greater than 0.0 to 1.0 with Dither tools?

If I have two frames of video (YV12) of the same scene but with two exposures say one is 1 EV greater than the other, ie: twice as bright. Is it possible to blend them into a 16bit linear light frame with brightness values representing 0.0 - 2.0 and so on for wider EV spacing.

Pixels are just numbers; you can assume them to have whatever intervals you want. For example, there's nothing stopping you from averaging two values on the interval of [0.0 - 1.0] and calling the new value [0.0 - 2.0]. That said, you obviously can't get a display to show values greater than 100% brightness.

Pseudo-code:

a = Source1.ConvertTo16Bits().GammaToLinear()
b = Source2.ConvertTo16Bits().GammaToLinear()
Average(a,b)
MoreCode()


These values are nominally on the interval [0.0 - 1.0], but since they are linear values, there's no difference from a [0.0 - 2.0] interval with 15-bit precision.

[Yellow_]

Pixels are just numbers; you can assume them to have whatever intervals you want. For example, there's nothing stopping you from averaging two values on the interval of [0.0 - 1.0] and calling the new value [0.0 - 2.0]. That said, you obviously can't get a display to show values greater than 100% brightness.

Pseudo-code:

a = Source1.ConvertTo16Bits().GammaToLinear()
b = Source2.ConvertTo16Bits().GammaToLinear()
Average(a,b)
MoreCode()




Ok, I've found the AverageM.dll and discussion here:

http://forum.doom9.org/showthread.php?p=726930#post726930

These values are nominally on the interval [0.0 - 1.0], but since they are linear values, there's no difference from a [0.0 - 2.0] interval with 15-bit precision.

Ok, :-) I'm working within Dither Tools stacked 16bit functions and blending like this:

Blending 8-bit pictures in linear light:

# Blending amount for the first clip
bl = 0.75
bls1 = String ( bl)
bls2 = String (1 - bl)

# 8-bit clips converted to linear 16-bit full range (gamma undone)
ug = " 16 - 0 max 1.41624 / 2.2 ^ "

# Redo the gamma, result in 16 bits YUV
rg = " 0.454545 ^ 362.5585 * 4096 +"

# Blend
Dither_lutxy8 (src1, src2,
\ expr ="x " + bls1 + " * y " + bls2 + " * + 256 *",
\ yexpr="x" + ug + bls1 + " * y" + ug + bls2 + " * +" + rg,
\ y=3, u=3, v=3)



Then I'm using Avs2yuv (really should be using Avs2pipemod) to export raw 48bit rgb to imagemagick like this:

Dither_convert_yuv_to_rgb (lsb_in=true, output="rgb48y")
r = SelectEvery (3, 0)
g = SelectEvery (3, 1)
b = SelectEvery (3, 2)
Dither_convey_rgb48_on_yv12 (r, g, b)

And writing 16bit OpenEXR's from IM for each exposure, but as EXR supports HDR I would like to merge the two exposures from each pair frames into each OpenEXR image for Tonemapping externally rather than export 8bit tiffs, jpg's or pngs and then manually set the EV spacing for them in a HDR app like Photomatrix or psftools to export HDR formats for tonemapping. Too many steps for video. :-)

But I'm unsure how to 'position' the what were 'normalised' video frames shot at various EV's (8bit 0 - 255 -> Dither Tools to 16bit), for piping the raw rgb data into IM with EV spacing they were shot at. ie: There are options in camera to shoot at 1 EV ie twice bright, up to 5EV.

Sorry if it sounds convoluted or plain daft. :-) Not sure if Average is the tool in this scenario?

[mandarinka]

For MDegrain, you can obtain an equivalent effect by changing the luma curve of the search clip. What I usually do: prepare the search clip using dfttest(lsb=true) with basic MC, Dither_lut16() to apply a gamma-like curve and make it full range, convert it back to 8 bit using DitherPost(mode=-1) and feed MSuper (for the final MAnalyse only, not the actual filtering) with it.

Could you post a code snippet for that, if it isn't a problem?
Mainly the Dither_lut16() part, I'd love to see the formula you use... I've been interested in luma-sensitive mdegraining for quite some time, but my own solution is crude at best (weighting together two clips...)

[cretindesalpes]

Mandarinka:

# 16-bit TV-range -> full range with extra curvature
# s0 = new slope at 0
# c = length of the curved part, > 0, no specific unit
#
# Matlab code:
# c = 1/16;
# s0 = 2;
# x = linspace (0, 1, 10001);
# y1 = (x * 219 + 16) / 256; % Scale [0 ; 1]
# t = ((y1*256 - 16) / 219);
# k = (s0-1)*c;
# y2 = ((1+c) - (1+c)*c ./ (t + c)) * k + t * (1 - k);
# plot (x, y1, x, y2); grid on;
Function fslg_remap_luma_search_clip (clip src, float "s0", float "c")
{
s0 = Default (s0, 2.0)
c = Default (c, 1.0/16)

k = (s0 - 1) * c
t = "x 4096 - 56064 / 0 1 clip"
e = String(k)+" "+String(1+c)+" "+String((1+c)*c)+" "+t+" "+String(c)
\ +" + / - * "+t+" 1 "+String(k)+" - * + 65536 *"

src.Dither_lut16 (
\ yexpr=e,
\ expr="x 32768 - 32768 * 28672 / 32768 +",
\ y=3, u=3, v=3
\ )
}

[mandarinka]

Thanks!

[Dogway]

if I set s0 to 1 the result is the same as if I did ((x-16)/219)*255 so s0 is to revert source gamma or just to make the luma tv2pc range process gamma aware? Wouldn't be 2.2 more appropiate? and what is c, kind of a multiplier?

[cretindesalpes]

No, this is not a gamma function, and actually it makes the data going even further from the linear-light scale. The purpose is to amplify the scale of the dark parts relative the the other parts. So it acts like a luma-based multiplier on the SAD. With a slope s0 = 2.0, thSAD for the darkest parts is implicitly divided by 2. The c parameter indicates the length of the transition between the dark and normal SAD.

Here are some graphs to show how the curve looks for different s0 and c values:

http://img20.imageshack.us/img20/3832/s0c.th.png (http://img20.imageshack.us/img20/3832/s0c.png)

Blue = original luma, green = mapped luma

Note that for high c or s0 values, the brightest parts have also their thSAD modified (in the other direction), because the slope of the curve near 1 tends to flatten.

[Dogway]

Very explicative! so you just supply the knots for curve "free playing" (while mapping tv to pc)
So I assume it can undo gamma? possibly with s0=2.2,c=1.0?

By doing that you miss the advantage of tweaking the darks in the same filter, although I always thought that to ungamma was sufficient for manalyse to not underperform in dark parts. YET human eyes are very sensitive to low luma levels and thus films and such have a notable bias for obscure scenes so it really might be necessary to have an special care in these parts.

I would have thought something ideal as:
tv to pc->ungamma->dark enhacement

Possibly is to ask too much but having an equal thSAD weighting through the whole luma range as a base for dark enhacement isn't such a bad idea.
Thanks for the functions everyday is Xmas day : P

[cretindesalpes]

Very explicative! so you just supply the knots for curve "free playing" (while mapping tv to pc)
So I assume it can undo gamma? possibly with s0=2.2,c=1.0?
As I said it cannot undo gamma, it's not a "power" function, moreover the general curve is inverted (so it's more like additional gamma). It is intended to be applied to the motion search clip only, not the real clip, so after motion estimation and SAD calculation, the dark parts of the real clip will be less filtered (similar to a thSAD reduction) than the bright ones.

—

On a completely unrelated note, I updated this script (http://forum.doom9.org/showthread.php?p=1519194#post1519194). It's helpful to dither a 16-bit clip to 10 bits and make x264-10bit encode the correct values, without requiring a patched build.

[Yellow_]

On a completely unrelated note, I updated this script. It's helpful to dither a 16-bit clip to 10 bits and make x264-10bit encode the correct values, without requiring a patched build.

Could you just clarify a little more? I'm a bit confused. :-)

Are you saying 16bit from previous version(s) of Dither tools to 10bit vanilla x264 encoder results is slightly greenish output?

And as such it was necessary to do some hack if not using a patched x264 encoder build?

But that previous hack is now not needed due to a new function in Dither , which is less of a hack but still function needs to be added to our scripts?

That when going from Dither tools 16bit to patched 10bit build nothing is needed?

[sneaker_ger]

Currently the 10 bit vanilla builds of x264 will convert 10 bit input to 16 bit and then dither down to 10 bit again. Some patched builds (like the newest from JEEB or Taro's) can skip this unnecessary, potentially harmful/unwanted, conversion.
I guess that script uses the inverse function of vanilla x264's 10 bit -> 16 bit -> 10 bit conversion, so that the conversion can do no harm and you can get the same result with a vanilla build as you would get with one of the patched builds. I don't know where you got the "greenish" idea from?

[cretindesalpes]

@Yellow_:

Yes vanilla x264 now scales and dither 16-bit input. However I don't know when it started to do this. I think the hacked x264 builds were mostly concerned about the 8-bit issue which was there from the beginning. Anyway the 16-bit path in the JEEB or 06_taro builds should also be free of unnecessary scaling.

@sneaker_ger:

Actually the 10->16->10 bit conversion is neutral. It's probably a (minor) waste of processing power, but it seems to work as expected. See the discussion here (http://x264.fushizen.eu/?p=393#comment-5113).

[Yellow_]

sneaker_ger the greenish tint was from the thread cretindesalpes pointed to via #377.

Thanks to you both for the explanation.

[sneaker_ger]

@sneaker_ger:

Actually the 10->16->10 bit conversion is neutral. It's probably a (minor) waste of processing power, but it seems to work as expected. See the discussion here (http://x264.fushizen.eu/?p=393#comment-5113).

Ah, I remember reading that in the changelog, but I didn't understand it at that time. So what's the point of using that script (aside from lowering processing power a bit)? Or was 10 bit output not supported at all prior?

[cretindesalpes]

My previous script had 10-bit dithering with 16-bit output, based on the assumption that the 6 lowest bits would just be dropped or rounded to the nearest value when entering x264.

[Boulder]

Does MScaleVect work with the vector clip created by using multi=true in MAnalyse? The docs don't say anything about that, but I assume that it works as if the source clip was a regular clip containing one set of vectors.

[mp3dom]

I'm asking just to avoid further problems and to be sure...
If a progressive 16bit YV16 stacked clip needs to be converted to a 16bit YV12 stacked clip, can I safely use a plain ConvertToYV12() or do I need to pay attention to something (mainly the LSB part)?
Thanks

[cretindesalpes]

Does MScaleVect work with the vector clip created by using multi=true in MAnalyse?
Hmm… I'm checking the code, and it might work, but there are things that must be fixed to be safe. Actually the flip parameter will not work at all.

If a progressive 16bit YV16 stacked clip needs to be converted to a 16bit YV12 stacked clip, can I safely use a plain ConvertToYV12() or do I need to pay attention to something (mainly the LSB part)?
Your guess is right, it will not work out of the box. The chroma planes will be resampled, so they definitely have to be processed in 16 bits. I'll probably submit soon a new function to convert between the YUV modes.

[Yellow_]

In the bit depth increase with Dither tools is it possible to 'extract' xvYCC extended gamut from suitable sources to 16bit RGB. As explained here: http://sutlib2.sut.ac.th/sut_contents/H95009/DATA/5637_2.PDF

[Overdrive80]

Hi, is possible to use MT with this filter??? I tried but show error that says; try vertical.....

[Stephen R. Savage]

Hi, is possible to use MT with this filter??? I tried but show error that says; try vertical.....

Internally, before each operation, Dither has to unpack a stacked frame into the interleaved version. Therefore, you need to use the "splitvertical" option in MT that processes the left+right sides of the frames simultaneously instead of the top+bottom. This is a terrible design decision in Dither and results in probably 10-20% performance loss shuffling bits around...

[mp3dom]

Your guess is right, it will not work out of the box. The chroma planes will be resampled, so they definitely have to be processed in 16 bits. I'll probably submit soon a new function to convert between the YUV modes.

Thanks for the explanation. On the contrary, may I assume that I can safely convert YV16 to YUY2 back and forth without too much trouble? They should be equal (apart the former being planar and the latter interleaved) and lossless.

[Overdrive80]

Internally, before each operation, Dither has to unpack a stacked frame into the interleaved version. Therefore, you need to use the "splitvertical" option in MT that processes the left+right sides of the frames simultaneously instead of the top+bottom. This is a terrible design decision in Dither and results in probably 10-20% performance loss shuffling bits around...

Thanks for your answer. It's a shame to lose that speed of process.

@Cretindealpes

With this script
dfttest(tbsize=1,sigma=10,lsb=true)

LSharpAAF(aatype="supaa") #With any aa success the same

gradfun3(smode=1,lsb_in=true,mode=0,ampo=1.2)

I get this in bottom image:

http://s17.postimage.org/yxeg24im7/snapshot.png

[cretindesalpes]

Boulder:

Dither 1.13.3 (http://forum.doom9.org/showthread.php?p=1386559#post1386559) released.
I think I fixed the MScaleVect + multi issue but I couldn't test all the possible cases. Send feedback if something goes wrong.

Yellow:

Sounds complicated but doable. It would require a custom source plug-in.

Stephen R. Savage:

The actual reason is that MT() requires a constant clip height for horizontal division, and a constant clip width for vertical division. Anyway, the Dither tools have been designed with heavy filtering in mind (dfttest, motion compensation, bilateral filtering, etc). The overhead caused by the stack16 format is probably several orders of magnitude lower than the load of the whole filter chain, so I don't really care. Clearly not perfect from the programmer's point of view, but much more user-friendly than a straightforward interleaved format.

mp3dom:

Yes, the 16-bit conversion between YV16 and YUY2 is lossless. Regarding your previous request, the YV16-YV12 converter is a bit complicated to design (like the YUV-RGB conversion was) and I haven't much time to spend on it. Unfortunately it won't be ready "soon", contrarily to what I said previously.

Overdrive80:

The result of your dfttest command is a stack16 clip. If you display it, you'll find that the bottom part of the picture is full of colored garbage. So if you try to apply an anti-aliasing filter on it, both parts of the picture will interfere. Moreover, there is a risk of messing up the pixel MSB/LSB consistency. Try to apply the AA filter in the 8-bit land, before dfttest or after GradFun3. If you really want to apply the AA filter on the stack16 clip, see post #250 (http://forum.doom9.org/showpost.php?p=1518953&postcount=250), but it looks overkill to me just for one filter, I wouldn't recommend that in this situation.

[Boulder]

Boulder:

Dither 1.13.3 (http://forum.doom9.org/showthread.php?p=1386559#post1386559) released.
I think I fixed the MScaleVect + multi issue but I couldn't test all the possible cases. Send feedback if something goes wrong.Thanks, I'll test it as soon as I have the time.

[Overdrive80]

@Cretindealpes

Thanks, i will try after gradfun3. ^^

[cretindesalpes]

For unpatched 10-bit x264 builds, straight 16-bit input will cause errors, unless you have scaled the original data in the inverse way x264 does. For 8-bit input, I don't know if things have changed or not (likely not).

Dither_convert_8_to_16 and other modified filters just shift bits to produce 16 bit output (an extension of the Rec 709 method to convert between 8- and 10-bit data), which is not the way x264 works. Therefore, to encode 10-bit video, you'll have to provide x264 with 10-bit data to be sure it won't rescale them. So the fslg_dither_to_n_bits script (or any equivalent) is required if you want to preserve the exact data scale. If you don't care about pre-dithering to 10 bits, the script becomes much simpler:
Dither_lut16 ("x 64 / 0.5 + 1023 min", y=3, u=3, v=3)
Dither_convey_yuv4xxp16_on_yvxx ()

[cretindesalpes]

If I understand correctly the x264 conversion code, the maximum value of the input scale (65535) is mapped onto the maximum value of the output scale (1023). So the error is roughly proportional to the value and remains below 1 quantum of the lowest bitdepth, in absolute value. That is -0.1 % for 10 bits.

[Keiyakusha]

It is not directly related to this thread but does anyone knows a way to open 16bit image sequence (png or tif) in avisynth for further processing? I imagine it should be opened as stacked YV24. (YV12 is a no go, I don't want subsampled chroma, upscaling image 2x with point resize before import is a no go too - they'll be too big, I don't have space or time for this.). So far with hacked ffms2 I only managed to open one image (not sequence) as stacked yv12. But PNG showed some corruption (tif is fine) and anyway it was far from what I need...

[atra dies]

Pardon my inexperience but I have several important questions:

What operations are more correctly done in linear scale light:
downsizing I assume, denoising (mdegrain)?, smoothlevels 16bit (altering y levels)? I remember someone said anything that averages pixels in another thread. I put mdegrain between a linear/gamma conversion and it looked OK although I did catch a blurring in some hay during a fast camera pan.

And when using bilinear to downsize, because every other filter rings or has a dark line around edges, is the lowpass an accurate calculation obtaining the highest sustainable frequency without aliasing or is it a conservative generic value that is blurring the image? I used fh and fv of 1.3 and things looked much better without much aliasing if at all. I know spline was much sharper but as I said there were dark lines around edges. I understand there will be undesirable things happening in certain patterns like brick or criss crossed clothing designs and have seen this in some modern dvds.

If converting yv12 to yuy2 with the bug fix script in 2.5.8, does the bilinear lowpass here or any other time when upsizing? I would assume it's just the downsize that the lowpass is applied but considering some bugs that have come to light like mpeg1 chroma placement, I am paranoid.

This lowpass is why everyone rags on bilinear but I find the spline useful only if I want to sharpen my clips and I usually don't like the outcome of sharpening, even LSF, when I view the final dvd. I probably oversharpened as a little goes a long way as far as that's concerned. For awhile there I was using spline48 then 144 and I do not like the effect those produced. Two tap spline is what I would prefer. Anything more warps some details because they are sharpened too much.

Something to watch out for: I noticed some ringing introduced by msuper when sharp is set above bilinear interpolation.

[tormento]

cretindesalpes, is it possible to have the previous version of dither? I am getting strange results with the newer and I'd like to compare them.

[cretindesalpes]

What operations are more correctly done in linear scale light:
downsizing I assume, denoising (mdegrain)?, smoothlevels 16bit (altering y levels)?
Although it's technically correct, denoising in linear light isn't necessary, because usually noise pixel values are close to each other. Operating in linear light is better when averaged pixel values are far. Moreover, the current modified MDegrain accepts only 8-bit input, and the conversion to 8-bit linear is not recommended because of the important resolution loss in the dark range.

And when using bilinear to downsize, because every other filter rings or has a dark line around edges, is the lowpass an accurate calculation obtaining the highest sustainable frequency without aliasing or is it a conservative generic value that is blurring the image?
The value is more or less arbitrary, but should be a decent compromise. It all depends on the interpolator frequency response shape. For example, the linear interpolator has a quite slow rolloff (huge aliasing excepted for highly oversampled data) and starts attenuating significantly the frequency content of the picture far before the Nyquist frequency. So if you increase fh and fv, you'll get a sharper picture but more aliasing. The visual result highly depends on the content nature. Anyway, for almost all filter (excepted sinc-based ones with a high number of taps), the default fh and fv values will bring you both aliasing and smoothing. Another thing to consider is that the ringing on the final video resulting from a large kernel is not necessarily bad. If the picture is scaled again during display, these ringing may contain valuable information that can help the display resizer to reconstitute fine details, instead of creating ringing and bluring where they didn't exist on the video. So it may look better at the native resolution, but worse when scaled.

If converting yv12 to yuy2 with the bug fix script in 2.5.8, does the bilinear lowpass here or any other time when upsizing?
I'm not sure to understand the question. From a signal processing point of view, the resizer works like this:
virtually upsample the picture to an infinite sampling frequency (like a continuous signal) by multiplying it with a Dirac impulse train (or grid, because we are in 2D). Muliplying is space domain is equivalent to convolving in frequency domain. Here, we get the picture spectrum mirrored and replicated. In one dimension: 0|¯\_|Fn becomes: ..._/¯¯¯\___/¯|¯\___/¯¯¯\_...
Convolve the signal with the normalised kernel impulse. This is where filtering happens, because the kernel is intended to be an approximation of a brickwall lowpass filter. fv and fh are implicitly divided by an additional factor in case of downscaling. Convolving in space domain is equivalent to multiplying spectrums in frequency domain, and we try here to clear everything above the desired cutoff frequency.
Then, the filtered signal is sampled again according to the new size, offsets, etc. This is where aliasing occurs, because the impulse spectrum may have a frequency content higher than both new and old nyquist frequencies, letting the rests of the mirrored parts fold and overlap where they shouldn't.
I don't know if I'm clear, but the important thing to understand it that there is always some kind of lowpass filtering during a scaling. Even upscaling.

cretindesalpes, is it possible to have the previous version of dither? I am getting strange results with the newer and I'd like to compare them.
Yes, they should be all available, just change the version numbers in the zip file url. What kind of strange result are you getting?