Color banding and noise removal

[ajp_anton]

No, it definitely works. The extra chroma resolution is clearly visible on video game footage, and even more so when loading the encoded video into Avisynth and comparing the chroma planes directly.
Maybe you forgot the "--output-csp i444", which is always needed for 4:4:4 encoding?

edit:
So the full command line would be
avs2pipe26mod -rawvideo script.avs | x264-10b - --demuxer raw --input-res wxh --input-depth 16 --input-csp i444 --output-csp i444 --fps x --crf x -o o.mkv

[cretindesalpes]

Oh you're right, I only had the --input-csp. Thanks for the tip! I'll add it to the documentation.

[Yellow_]

I'm getting weird ghosting exporting 16bit to 10bit x264 with 1.12.0 that with the same script worked fine on 1.10.0 using the following:

LoadPlugin("c:\Program Files (x86)\AviSynth 2.5\plugins\ffms2.dll")
LoadPlugin("c:\Program Files (x86)\AviSynth 2.5\plugins\mt_masktools_26.dll")

FFmpegSource2("MVI_8068.MOV")

trim(0,10)

Dither_convert_8_to_16 ()

Dither_convey_yuv4xxp16_on_yvxx()

Most of the image is intact but there is a semi transparent overlap starting from the left which travels across the frame as it plays, just seems to affect chroma and only certain colors, lots of grey appearing too.

I'll try to upload a image frame later.

[cretindesalpes]

Originally Posted by Yellow_

I'm getting weird ghosting exporting 16bit to 10bit x264 with 1.12.0 that with the same script worked fine on 1.10.0

This is weird, because dither.avsi hasn't changed from 1.10.0 to 1.12.0. Are you there wasn't any problem with 1.10.0?

Originally Posted by SilaSurfer

One request if I may, would it be possible to add to your function Dither_Add_Grain_16 to be able to add 3 layers of grain like what one would get with Grainfactory3 from Didée? (hope he's well) Thanks.

I might consider this later, but it's currently not in my priority task list.

[Yellow_]

Ok, my bad I left the -raw out of the avs2yuv command line. so dumb, sorry for confusion. :-(

[SilaSurfer]

Originally Posted by cretindesalpes

I might consider this later, but it's currently not in my priority task list.

Thanks cretindesalpes. No preasure. Keep on the good work.

[Dogway]

Several questions in Dither_convert_yuv_to_rgb could you implement rgb24? This in case it saves something because I never use alpha.
Also it's strange, when I convert yuv red color AFAIK (81, 90, 240) to RGB in TV range I get 238,14,14, is it ok?
Do you still have planned v&c algorithm, Im curious to know how it compares to bayer dithering

[cretindesalpes]

Originally Posted by Dogway

Several questions in Dither_convert_yuv_to_rgb could you implement rgb24?

OK.

Also it's strange, when I convert yuv red color AFAIK (81, 90, 240) to RGB in TV range I get 238,14,14, is it ok?

Are you sure ?

BlankClip (pixel_type="YV12", color_yuv=(81*256+90)*256+240)
Dither_convert_yuv_to_rgb (tv_range=true)

gives me the expected values around (255, 0, 0), and I obtain (238, 14, 14) with tv_range=false. Have you this problem with a specific setting?

Do you still have planned v&c algorithm, Im curious to know how it compares to bayer dithering

I haven't made any progress on this topic. I'm currently trying to multithread internally the MAnalyse/MRecalculate part of the MVTools because this is a major speed bottleneck for lots of scripts, but understanding how all this works is a quite complex and time-consuming task.

[Dogway]

No, its ok, it just looked like wanting to be 240,16,16. Im converting a bunch of frames for manual editing and wanted to be sure, but I will probably use tvrange true so no range conversion is done.


No problem on v&c, at your pace. Now that you are on mvtools2 it could be interesting if you could have a look on how to make it more flexible, like
mdegrain not relying on vectors/mask, so one could mix spatial denoiser and mdegrain through the mvtools motion mask, you can see my discussion in the mvtools thread. This could also enable a more orthodox way for "fake" lsb_in in mdegrain, and play a role on the masktools pipeline.
Multithreading the motion search stage is going to be a big leap, you might already know the SVP group are on the GPU direction with this, but I prefer your tools because they work rock steady. No need to hurry.

[Yellow_]

Is there anything that can be done to improve memory usage when using Dither_convey_yuv4xxp16_on_yvxx() -> AVS2yuv -> Imagemagick written 16bit image sequences? On larger video files the process fails, with out of memory errors by trying to do the whole video in memory I guess before writing out of IM. :-(

LoadPlugin("c:\Program Files (x86)\AviSynth 2.5\plugins\ffms2.dll")
LoadPlugin("c:\Program Files (x86)\AviSynth 2.5\plugins\mt_masktools_26.dll")

FFmpegSource2("MVI_8068.MOV")

Dither_convert_8_to_16 ()

Dither_convert_yuv_to_rgb(matrix="601", tv_range=false, cplace="MPEG2", chromak="bicubic", lsb_in=true, output="rgb48y")
Dither_y_gamma_to_linear (tv_range_in=false, tv_range_out=false, curve="709")
Dither_convey_rgb48_on_yv12 (SelectEvery (3, 0),SelectEvery (3, 1),SelectEvery (3, 2) )

Although I could probably batch write scripts to trim() so many frames into individual .avs files and batch convert, is there a more efficient way. Reduced bit depth to 12bit? But memory would just run out later in the process?

On another query, I see in the Dither.html that blending frames is possible, so is it possible to use Dither tools to do typical blend modes like overlay, multiply etc in linear light at 16bit depth? I'm interested in working this with new high bit depth tools in Avisynth now. http://forum.doom9.org/showpost.php?...&postcount=219 & here: http://www.dcs.gla.ac.uk/~jhw/filmy/, not so much simulating film grain, don't see much point in that unless mixing computer generated imagery with film scans, but the build up of layers.

[Dogway]

The interlaced parameter in ditherpost is for interlaced inputs or (separated)fields input? I wonder because it would be unpractical to work 16 bit in real interlaced material.
Also I have been wondering about the motion search in x264, isn't it fooled by the ordered dither as in mvtools2? or does it use another approach for it.

At last Im trying to optimize smdegrain depending on processed planes. But there are some behaviours I don't get to grasp.

Shouldn't this be true?


dfttest(lsb=true,U=true,v=true)
DitherPost(mode=-1,u=2,v=2)
==
dfttest(lsb=true,U=false,v=false)
DitherPost(mode=-1,u=3,v=3)

I also would think that in the first case UV output would be the MSB part of the clip which is 8 bit, but that also doesn't apply if I compare it to :

dfttest(lsb=true,U=false,v=false)
DitherPost(mode=-1,u=2,v=2)
dfttest(Y=false)

[cretindesalpes]

Originally Posted by Yellow_

Is there anything that can be done to improve memory usage when using Dither_convey_yuv4xxp16_on_yvxx() -> AVS2yuv -> Imagemagick

If I understand correctly your description of the problem, data accumulate in the pipe until Imagemagick decides to process them, probably after the pipe is closed at the AVS2yuv end. I don't know if replacing the pipe with an intermediate file would work, but you can try.

On another query, I see in the Dither.html that blending frames is possible, so is it possible to use Dither tools to do typical blend modes like overlay, multiply etc in linear light at 16bit depth?

It's possible to blend 8-bit pictures into a 16-bit frame using the masktools. However blending 16-bit pictures is another story, mainly because we lack functions that can combine two or three 16-bit clips.

Originally Posted by Dogway

The interlaced parameter in ditherpost is for interlaced inputs or (separated)fields input?

For interlaced input. It differs from progressive input only when using the error diffusion modes.

Also I have been wondering about the motion search in x264, isn't it fooled by the ordered dither as in mvtools2? or does it use another approach for it.

Sure but this is not a problem here if motion estimation is inaccurate in nearly flat areas. x264 isn't a temporal denoiser, it doesn't need to perfectly overlay fine details or textures. It has to take advantage of the data redundancy, and this is what an ordered dither can offer.

dfttest(lsb=true,U=true,v=true)
DitherPost(mode=-1,u=2,v=2)
==
dfttest(lsb=true,U=false,v=false)
DitherPost(mode=-1,u=3,v=3)

No, because in the first case, the U and V plane have been filtered by dfttest. Whatever the conversion to 8 bits done afterward, you'll never obtain the input back. Using u=2, v=2 in DitherPost will keep only the MSB (truncation) instead of rounding, like cropping the top half of a stack16 frame.

[Dogway]

Originally Posted by cretindesalpes

For interlaced input. It differs from progressive input only when using the error diffusion modes.

I see, although I don't know when would it apply since interlaced inputs are commonly filtered after separated fields, in any case I rarely use error diffusion.

Originally Posted by cretindesalpes

Using u=2, v=2 in DitherPost will keep only the MSB (truncation) instead of rounding, like cropping the top half of a stack16 frame.

Yes that makes sense, as I explained afterwards it would more likely keep the msb part when using u=2 v=2, but I compared it also to filtering chroma after ditherpost in 8 bits and it isn't the same so the question is whether the MSB part is like the result of filtering in 8 bit or it is something different.

dfttest(lsb=true,U=true,v=true)
DitherPost(mode=-1,u=2,v=2)
==
dfttest(lsb=true,U=false,v=false)
DitherPost(mode=-1,u=2,v=2)
dfttest(Y=false)

[Yellow_]

cretindesalpes, thanks. Investigating further I've found that Imagemagick has a function called -limit, which can be set to restrict how much memory IM can use before it has to write temp files to disk. Using this on the CLI:

identify -list resource

Gives info on what resources IM thinks is available. Then setting -limit memory some GiB and -limit map some GiB functions on the avs2yuv command line, to levels within those resources keeps memory errors away. :-)

[TheProfosist]

DitherPost dithers? a 16bit clip to 8bit. Then what does GradFun3 dither? What im trying to ask is how is using SmoothGrad+DithePost different from using GradFun3?

[AzraelNewtype]

Originally Posted by TheProfosist

DitherPost dithers? a 16bit clip to 8bit. Then what does GradFun3 dither? What im trying to ask is how is using SmoothGrad+DithePost different from using GradFun3?

DitherPost turns stacked 16bit, usually created by using a denoiser that is designed to take 8bit input and yield that special output format, into dithered single height 8bit. GradFun3 takes your 8bit video and dithers it to attempt to deband source that's already been filtered, in much the same way GradFun2 and its modifications did before it. Which you should use depends on your workflow.

[TheProfosist]

Originally Posted by AzraelNewtype

DitherPost turns stacked 16bit, usually created by using a denoiser that is designed to take 8bit input and yield that special output format, into dithered single height 8bit. GradFun3 takes your 8bit video and dithers it to attempt to deband source that's already been filtered, in much the same way GradFun2 and its modifications did before it. Which you should use depends on your workflow.

will DitherPost help with debanding at all? I know that SmoothGrad should by smoothing the gradients.

[Dogway]

TheProfosist: Gradfun3 basically is the same as:

Dither_convert_8_to_16()
put a low pass denoiser here...
ditherpost

with some masks, etc. Cretindesalpes explained it at some point in the thread.

Ditherpost just takes whatever 16 bit processing clip and converts it to 8 bit with the consequent dither so you don't see banding. So ditherpost in nature doesn't deband but rather prevents banding when doing filtering (among other things).

For example GradFun3 with smode=3 is the same code as above but with a bunch of smoothgrad() lines in between with varying settings. So I guess smoothgrad is also some kind of a very lowpass denoiser.

If you never filter in 16 bit, use gradfun3 to deband. If instead you filter in 16 bit (denoisers, resizers, etc) then before ditherpost add a smoothgrad() line, and I think you are good to go, I do this oftenly.

[TheProfosist]

Originally Posted by Dogway

If instead you filter in 16 bit (denoisers, resizers, etc) then before ditherpost add a smoothgrad() line, and I think you are good to go, I do this oftenly.

This is what I have been doing. Thanks for the info.

[SSH4]

Originally Posted by cretindesalpes

This is a planned feature. In the meantime… here is a quick and dirty hack:

# Assume stack16 clip here

n = 10	# Final bitdepth (range: 8 to 14)
msb = Dither_get_msb ()
lsb = Dither_get_lsb ()
hr = 2
m = String (Pow (2, 16 - hr - n) - 1)
l = msb.mt_lut ("x     "+m+"   &u", y=3, u=3, v=3)
h = msb.mt_lut ("x 255 "+m+" - &u", y=3, u=3, v=3)

s1 = String (n - 8)
ofs = String (Pow (2, 7 - hr))
a = StackVertical (l, lsb)
a = a.Dither_lut16 ("x "+s1+" <<u "+ofs+" 256 * +", y=3, u=3, v=3)

b = a.DitherPost ()

s2 = String (16 - n)
ofs2 = String (Pow (2, 15 - hr - n))
b_msb = b.mt_lut ("x "+s1+" >>u 255 &u "+ofs2+" - 128 +", y=3, u=3, v=3)
b_lsb = b.mt_lut ("x "+ofs+" - "+s2+" <<u 255 &u", y=3, u=3, v=3)

c = mt_adddiff (h, b_msb, y=3, u=3, v=3)
StackVertical (c, b_lsb)

Dither_convey_yuv4xxp16_on_yvxx ()

how about this kind dither LSB from 8bit to 2bit (16bit to 10bit)

# Assume stack16 clip here

a=Dither_get_lsb()
b=Dither_get_msb()
a.mt_lut("x 6 >>",u=3,v=3)
DitherPost (last,a,mode=6, u=3, v=3)
mt_lut("x 6 <<",u=3,v=3)

And why ampn not work on mode 2-5 ?

And what can give better encoding result (quality/size on same CRF) on x264-10bit encode:
• dithered (ordered or floid steinberg + static noise) 8-bit source;
• 16-bit (MSB/LSB)
• 10-bit dithered (probably with static noise too) from 16-bit
?