Color banding and noise removal

[Izuchi]

Is it possible to to serve high bit depth scripts to Virtualdub and create lossless 16-bit avi files which can then be encoded using x264 --input-depth 16? I think I know the answer, but just want to make sure.

[feisty2]

dither_out ()

[Reel.Deel]

Is it possible to to serve high bit depth scripts to Virtualdub and create lossless 16-bit avi files which can then be encoded using x264 --input-depth 16? I think I know the answer, but just want to make sure.

Why not just use Avs2YUV (http://avisynth.nl/index.php/Avs2YUV) or avs4x26x (http://forum.doom9.org/showthread.php?t=162656) to pipe high bit-depth video to x264?


dither_out ()
I see why you have 1,226 posts...

[Sparktank]

Shouldn't you be able to encode to x264 using avs2yuv/av4x26x ?
Using Dither_out()

The documentation gives example to encode 10bit:
Dither_quantize (bitdepth=10)
Dither_out ()

Which works if you set:
avs2yuv -raw "script.avs" -o - | x264-10bit --demuxer raw --input-depth 16

VDub should not really be needed. Unless you want a lossless intermediate.
Which would probably serve better if you used FFMPEG to create ffvhuff lossless intermediate.
I haven't really used VDub in a long time, unless I test with UTVideo Codec lossless intermediates.

ffmpeg -codec jpeg2000 -i "video.mxf" -map_metadata -1 -pix_fmt yuv444p12le -vcodec ffvhuff "output-ffvhuff-yuv444p12le.mkv"

ffmpeg supported (ffvhuff) pixel formats:
yuv420p yuv422p yuv444p yuv411p yuv410p yuv440p

gbrp gbrp9le gbrp10le gbrp12le gbrp14le gray gray16le

yuva420p yuva422p yuva444p

gbrap

ya8

yuv420p9le yuv420p10le yuv420p12le yuv420p14le yuv420p16le
yuv422p9le yuv422p10le yuv422p12le yuv422p14le yuv422p16le
yuv444p9le yuv444p10le yuv444p12le yuv444p14le yuv444p16le

yuva420p9le yuva420p10le yuva420p16le
yuva422p9le yuva422p10le yuva422p16le
yuva444p9le yuva444p10le yuva444p16le

rgb24 bgra

[Izuchi]

Yes, I was thinking of creating some lossless intermediary files, should have been more clearer. Thanks, Sparktank I'll give that a try.

[real.finder]

Thanks for your answer, cretindesalpes, I appreciate it. I've been slowly learning a bit here and a bit there and wanted to make sure I was putting the pieces together correctly. Another question, if I may. I updated Dither a few days ago, from v1.25.1 to v1.27.1 (I'm a bit slow) and in the process unwittingly added the avstp.dll to my plugins folder. I typically use the basic above script with SEt's MT AviSynth 2.6 like so:

SetMemoryMax(2048)
SetMTMode(3,7)
LoadPlugin("F:\[0]StandAloneApps\MeGUI-2500(core)2443(data)0.3.5(libs)[Portable]\tools\lsmash\LSMASHSource.dll")
LWLibavVideoSource("D:\Temp\[0000]ReEncTemp\TheUnbearableLightnessOfBeing[S]{1988}[720p]\OriginalVideo,OriginalAudio[ViGi].mkv")
SetMTMode(2)
SMDegrain(tr=3,thSAD=400,RefineMotion=True,Plane=0,Chroma=False,Lsb=True,Lsb_Out=True,PreFilter=2)
F=DitherPost(Mode=-1)
S=F.FastLineDarkenMod()
D=MT_MakeDiff(S,F).Dither_Convert_8_To_16()
Dither_Add16(Last,D,Dif=True,U=2,V=2)
GradFun3(Lsb_In=True,Lsb=True)
# DitherPost
Dither_Out()

This has always given me good results (except for the occasional corrupt source), but with the avstp.dll in my plugins folder the output video had nasty artifacts every few seconds. Once I figured out that the avstp.dll was the cause of the glitches I found my way to your thread (http://forum.doom9.org/showpost.php?p=1564516&postcount=1) and, after reading the "Short user manuel" I don't understand why the avstp.dll would cause so much trouble. To my obviously wrong understanding it should help. Or is the problem that SMDegrain and FastLineDarkenMod aren't AVSTP enabled?

was the artifacts be like http://i.imgur.com/cvt0nW9.jpg ?

[StainlessS]

Just thought I'de point out that FFTW library used by eg FFT3DFilter as fftw3.dll and dftTest as libfftw3f-3.dll (same dll, just different name)
has been updated several times since the one included in dither tools. (not sure what date last update was). http://www.fftw.org/index.html


Here some update history, current version v3.3.4

FFTW 3.3.4

* New functions fftw_alignment_of (to check whether two arrays are
equally aligned for the purposes of applying a plan) and fftw_sprint_plan
(to output a description of plan to a string).

* Bugfix in fftw-wisdom-to-conf; thanks to Florian Oppermann for the
bug report.

* Fixed manual to work with texinfo-5.

* Increased timing interval on x86_64 to reduce timing errors.

* Default to Win32 threads, not pthreads, if both are present.

* Various build-script fixes.

FFTW 3.3.3

* Fix deadlock bug in MPI transforms (thanks to Michael Pippig for the
bug report and patch, and to Graham Dennis for the bug report).

* Use 128-bit ARM NEON instructions instead of 64-bits. This change
appears to speed up even ARM processors with a 64-bit NEON pipe.

* Speed improvements for single-precision AVX.

* Speed up planner on machines without "official" cycle counters, such as ARM.

FFTW 3.3.2

* Removed an archaic stack-alignment hack that was failing with
gcc-4.7/i386.

* Added stack-alignment hack necessary for gcc on Windows/i386. We
will regret this in ten years (see previous change).

* Fix incompatibility with Intel icc which pretends to be gcc
but does not support quad precision.

* make libfftw{threads,mpi} depend upon libfftw when using libtool;
this is consistent with most other libraries and simplifies the life
of various distributors of GNU/Linux.

FFTW 3.3.1

* Changes since 3.3.1-beta1:

- Reduced planning time in estimate mode for sizes with large
prime factors.

- Added AVX autodetection under Visual Studio. Thanks Carsten
Steger for submitting the necessary code.

- Modern Fortran interface now uses a separate fftw3l.f03 interface
file for the long double interface, which is not supported by
some Fortran compilers. Provided new fftw3q.f03 interface file
to access the quadruple-precision FFTW routines with recent
versions of gcc/gfortran.

* Added support for the NEON extensions to the ARM ISA. (Note to beta
users: an ARM cycle counter is not yet implemented; please contact
fftw@fftw.org if you know how to do it right.)

* MPI code now compiles even if mpicc is a C++ compiler; thanks to
Kyle Spyksma for the bug report.


EDIT: Needs to be in system32 (or equiv on 64bit) as both fftw3.dll and ibfftw3f-3.dll (ibfftw3f-3.dll is name as supplied in zip).

[AzraelNewtype]

EDIT: Needs to be in system32 (or equiv on 64bit) as both fftw3.dll and ibfftw3f-3.dll (ibfftw3f-3.dll is name as supplied in zip).

It's SysWOW64 on 64-bit OS... for the 32-bit dlls. If you're doing 64-bit avs, you put the 64-bit versions into system32 still. I'm only spelling it out because it's hilariously confusing.

Thanks Microsoft.

[StainlessS]

Thank you AzraelNewtype.

It's hilariously confusing.

Thanks Microsoft.


Another MS sweetie,

According to MS, Your System drive is the one you boot to, and your Boot drive is the one your system is on !

Always good for a giggle, is MS.

[creaothceann]

Does it have to be in System/SYSWOW64, or can it also be in a directory included in %PATH%?

[Groucho2004]

Does it have to be in System/SYSWOW64, or can it also be in a directory included in %PATH%?
The latter also works.

[LouieChuckyMerry]

was the artifacts be like http://i.imgur.com/cvt0nW9.jpg ?

It's difficult to tell by your screenshot because I never encoded any anime with the AVSTP.dll issue, but it looks very similar. For my non-anime encodes, it looked like frost or ice (white artifacts) on the edges of images in the frame and was especially noticeable when there was action (camera pans, movement, etc.).

[real.finder]

It's difficult to tell by your screenshot because I never encoded any anime with the AVSTP.dll issue, but it looks very similar. For my non-anime encodes, it looked like frost or ice (white artifacts) on the edges of images in the frame and was especially noticeable when there was action (camera pans, movement, etc.).

the screenshot from pan too

and didn't happened in all video, it happened in ED in this frame only, and in avs 64 (with setmtmode) without AVSTP, and after I see it in the final encode I try to reproduce it but it didn't reproduced with many tries

[LouieChuckyMerry]

the screenshot from pan too and didn't happened in all video, it happened in ED in this frame only, and in avs 64 (with setmtmode) without AVSTP, and after I see it in the final encode I try to reproduce it but it didn't reproduced with many tries

If you can't reproduce it then perhaps it was just a one-time occurrence? I've experienced, very rarely, artifacts because the source file was corrupted, possibly a bad mux or a digital glitch.

[real.finder]

If you can't reproduce it then perhaps it was just a one-time occurrence? I've experienced, very rarely, artifacts because the source file was corrupted, possibly a bad mux or a digital glitch.

I tried with the same source and script

[bxyhxyh]

Hello, guys

If I use dither_resize16(1280,720,csp="YV24") on 1080p clip, this works ok.

I always resized chroma and luma separately before.
Are they same?

[Desbreko]

Yes, that'll be identical to resizing the chroma separately like this:

y = Dither_resize16(1280,720, u=1, v=1)
u = UToY8().Dither_resize16(1280,720, src_left=0.25)
v = VToY8().Dither_resize16(1280,720, src_left=0.25)
YToUV(u,v,y)

Resizing the chroma separately as Y8 clips is useful if you're using AviSynth's resizers or something else like nnedi3_rpow2 instead of Dither_resize16 on it, but otherwise, it's simpler and probably also faster to just use the csp option.

[fvisagie]

Resizing the chroma separately as Y8 clips is useful if you're using AviSynth's resizers or something else like nnedi3_rpow2 instead of Dither_resize16 on it, but otherwise, it's simpler and probably also faster to just use the csp option.

Please explain in what situations one would want to do one of these (separately resize or use the csp option), and why?

Thanks,
François

[Desbreko]

It's for when you want to downscale the luma of a video with subsampled chroma without losing any more of the chroma resolution. Specifically, a lot of anime is produced at 720p or thereabout and upscaled to 1080p for release on Blu-ray, so converting the 1080p YV12 video on the BD to 720p YV24 when you encode it lets you avoid wasting bits on upscaled luma.

[fvisagie]

Thanks.

[LouieChuckyMerry]

Hello :) . I've been using the following script on HD sources with good results:

SMDegrain(TR=1,ThSAD=100,RefineMotion=True,Plane=0,Chroma=False,Lsb=True,Lsb_Out=True)
LinearResize(1280,720,Lsb_In=True,Lsb_Out=True)
F=DitherPost(Mode=-1)
S=F.FastLineDarkenMod(Strength=24)
D=MT_MakeDiff(S,F).Dither_Convert_8_To_16()
Dither_Add16(Last,D,Dif=True,U=2,V=2)
GradFun3(Radius=16,Lsb_In=True,Lsb=True)
Dither_Out()

but now find myself wondering if I could improve speed and-or quality by doing the resizing before the denoising (this seems logical, at least to me, to make the denoising area smaller first). I've searched about for an answer but can't find anything of value. Please, would One Of You Who Knows More Than I explain to me which should come first and why (or point me to a thread that specifically discusses this issue). Thank you very much.

[creaothceann]

Resizing automatically removes noise simply because it removes pixels; on the other hand the remaining noise might then be harder to remove? Probably depends on the filter.

[StainlessS]

I would say no matter if upsize or downsize, denoise first, resize last.

Order of operations according to the old TmpGenc app.


1 ) Source range (trim)
2 ) Inverse Telecine
3 ) Ghost Reduction
4 ) Noise Reduction
5 ) Sharpen
6 ) Color Correction
7 ) Deinterlace
8 ) Clip frame (crop/Resize)
# 9 ) 3/2 Pulldown
10) FrameRate Conversion

[LouieChuckyMerry]

Resizing automatically removes noise simply because it removes pixels[emphasis LCM]; on the other hand the remaining noise might then be harder to remove? Probably depends on the filter.

Thanks for that; I've never thought of it that way but it makes perfect sense now that I read it.


I would say no matter if upsize or downsize, denoise first, resize last.

Order of operations according to the old TmpGenc app.


1 ) Source range (trim)
2 ) Inverse Telecine
3 ) Ghost Reduction
4 ) Noise Reduction
5 ) Sharpen
6 ) Color Correction
7 ) Deinterlace
8 ) Clip frame (crop/Resize)
# 9 ) 3/2 Pulldown
10) FrameRate Conversion


Interesting. What made me recently question my approach was remembering that MeGUI's default script has:

#deinterlace
#crop
#resize
#denoise

listed after the source. Hmmm, which application is correct? I've always understood (ha ha ha) that inverse telecine-deinterlace come first, when applicable, and have always done as such with no apparent problems; however, my dealings with interlacing have (thankfully) been infrequent and almost always involve seemingly endless trial-and-error and-or begging for help somewhere. That or I wait for the Blu-ray ;) . Given your above post, would you then reckon that my script:

SMDegrain(TR=1,ThSAD=100,RefineMotion=True,Plane=0,Chroma=False,Lsb=True,Lsb_Out=True)
LinearResize(1280,720,Lsb_In=True,Lsb_Out=True)
F=DitherPost(Mode=-1)
S=F.FastLineDarkenMod(Strength=24)
D=MT_MakeDiff(S,F).Dither_Convert_8_To_16()
Dither_Add16(Last,D,Dif=True,U=2,V=2)
GradFun3(Radius=16,Lsb_In=True,Lsb=True)
Dither_Out()

should be rearranged to:

SMDegrain(TR=1,ThSAD=100,RefineMotion=True,Plane=0,Chroma=False,Lsb=True,Lsb_Out=True)
F=DitherPost(Mode=-1)
S=F.FastLineDarkenMod(Strength=24)
D=MT_MakeDiff(S,F).Dither_Convert_8_To_16()
Dither_Add16(Last,D,Dif=True,U=2,V=2)
GradFun3(Radius=16,Lsb_In=True,Lsb=True)
LinearResize(1280,720,Lsb_In=True,Lsb_Out=True)
Dither_Out()

[StainlessS]

I am very definitely no expert,(nor hardly even a novice),
but maybe the FastLineDarken, GradFun and Dither stuff should be (I think) POST resize.
(they just were not a factor in TmpGenc days)

EDIT: Downsize, Better qual if denoise first, then resize down, faster if downsize first but with penalties in quality.
Upsize, no penalties in, denoise, then resize up.

EDIT: Qwik rule of thumb: the one that takes longest produces better quality (generally speaking, its a tradeoff).

[fvisagie]

3 ) Ghost Reduction
4 ) Noise Reduction
5 ) Sharpen
...
7 ) Deinterlace


I'm just not sure that mostly spatial operations should be applied to temporally separated fields. Better output should result from applying deinterlacing first, at least according to my understanding.

[StainlessS]

OK, do you have some general order of process, could be of interest to many ?

[fvisagie]

I'm even less of an expert than you, but from the discussions on the topic I've read and/or participated in, there seem to be two main anchor points around the rest of the processing.

The more hard-and-fast of the two is deinterlacing. When needed, in order to preserve field integrity deinterlacing should come before spatial and/or geometric transformation operations like resizing, [sub]titling, sharpening, denoising etc. Arguably, this group might also include exposure/colour correction that is non-linear and/or non-global, e.g local contrast enhancement. LCE may cause loss of coherence between adjacent fields.

The less hard-and-fast anchor point is in fact denoising. Preferably this is done last 1) to clean up after all fudging is complete, and 2) to prevent denoising from reducing detail needed in earlier processing.

However, that brings one to the topic of this whole thread - denoising can cause or worsen banding;). Although I have been fortunate enough not to need debanding much, circumstances like quality of input, severity of banding and strength of denoising needed seem to indicate the appropriate debanding-denoising sequence on a case-by-case basis. I.e., suck it and see ;).

[LouieChuckyMerry]

The more hard-and-fast of the two is deinterlacing. When needed, in order to preserve field integrity deinterlacing should come before spatial and/or geometric transformation operations like resizing, [sub]titling, sharpening, denoising etc. Arguably, this group might also include exposure/colour correction that is non-linear and/or non-global, e.g local contrast enhancement. LCE may cause loss of coherence between adjacent fields.

Thanks for the lucid explanation.


The less hard-and-fast anchor point is in fact denoising. Preferably this is done last 1) to clean up after all fudging is complete, and 2) to prevent denoising from reducing detail needed in earlier processing.

However, that brings one to the topic of this whole thread - denoising can cause or worsen banding;). Although I have been fortunate enough not to need debanding much, circumstances like quality of input, severity of banding and strength of denoising needed seem to indicate the appropriate debanding-denoising sequence on a case-by-case basis. I.e., suck it and see ;).

As creaothceann typed, "Resizing automatically removes noise simply because it removes pixels.". So it seems reasonable that one would denoise before resizing so that the resizer has a higher quality source for pixel selection(?). Of course, I'm even less expert than you ;) . The sharpening is typically after denoising (and resizing, if applicable) from what I've read (although the best sharpener is to not blur, which is where masking come into play; bummer that this slows things down so much). I think I'll stick with my original script with Denoise-->Resize-->Sharpen-->Deband (I'm 99.99% sure that debanding comes last :p ).

Thanks for the help.

[Boulder]

You could try sharpening while downsizing. Didée once came up with a brilliant suggestion to abuse BicubicResize: http://forum.doom9.org/showthread.php?p=1579385#post1579385

That's what I do when I go from 1080p to 720p, which is about 99% of the time. You just need to adjust b and c according to the source to compensate the loss of sharpness.

[fvisagie]

The sharpening is typically after denoising (and resizing, if applicable)

Indeed, that slipped my mind.

[cretindesalpes]

dither 1.27.2 (http://forum.doom9.org/showthread.php?p=1386559#post1386559):
Fixed Dither_crop16 which wasn’t passing the align parameter if there was no horizontal cropping. Thanks to Reel.Deel for making me check.
Fixed Dither_srgb_display with autopar=false and 1440×1080 input.
Fixed Dither_quantize with reducerange=true, some rare pixel values were wrongly quantized. Thanks to _08 for reporting this bug.
Fixed a picture corruption in Dither_bilateral16. Thanks to _08 for reporting what was actually a combination of two different bugs.

[Sparktank]

:cool: :thanks:

[LouieChuckyMerry]

You could try sharpening while downsizing. Didée once came up with a brilliant suggestion to abuse BicubicResize: http://forum.doom9.org/showthread.php?p=1579385#post1579385

That's what I do when I go from 1080p to 720p, which is about 99% of the time. You just need to adjust b and c according to the source to compensate the loss of sharpness.

Thanks for that. Would you recommend this method for downsizing animation?


cretindesalpes: thanks for the updated dither :thanks:

[Motenai Yoda]

the last version include the fix of this test build too?
http://forum.doom9.org/showpost.php?p=1731185&postcount=927

[cretindesalpes]

the last version include the fix of this test build too?
http://forum.doom9.org/showpost.php?p=1731185&postcount=927
Yes, but I probably forgot to report it in the history.

[Motenai Yoda]

@cretindesalpes what is the exact formula for the elasticity of limit_dif16?
AFAIU it's something like "x" up to threshold then "x * (((thr*elast) -x)/(thr*(elast -1)))" up to thr*elast

[asarian]

Using the following line,

Dither_convert_yuv_to_rgb (matrix="601", output="rgb48y", lsb_in=false)

Would still produce a lsb_out = true (16 bit) output clip, right?

Thanks.

[AzraelNewtype]

Probably not, though you could test it yourself in about 10s. If the height doesn't double it's still 8-bit.

[asarian]

Probably not, though you could test it yourself in about 10s. If the height doesn't double it's still 8-bit.

You're right: that was a simple enough test. :)

It *does* produce a 16-bit clip, btw (960p for a 480p source).

[real.finder]

official mvtools v2.5.11.20 has been released and pinterf did some fix too https://github.com/pinterf/mvtools/tree/mvtools-pfmod

can we have new mvtools mod16 ?

[Reel.Deel]

official mvtools v2.5.11.20 has been released and pinterf did some fix too https://github.com/pinterf/mvtools/tree/mvtools-pfmod

can we have new mvtools mod16 ?

pinterf's mod is based on mvtools v2.6.0.5 so it already includes mod16.

[real.finder]

pinterf's mod is based on mvtools v2.6.0.5 so it already includes mod16.

but it don't have official mvtools v2.5.11.20 changes

and not work with avs 2.5 (some people still use it)

[asarian]

but it don't have official mvtools v2.5.11.20 changes

and not work with avs 2.5 (some people still use it)

Hmm, so I just upgraded the wrong AviSynth mvtools?! LOL. Guys, please, don't confuse me. :) It's hard enough to keep everything apart as it is.

[pinterf]

I came up with the my mod mainly because of the 2.6 Api and to fix some memory exceptions of the x64 version. Really not big modifications. At about the same time Physic came back to business after many years and fixed some other things with the 2.5.11.20 version. But this is for 32 bit only. Note that the 2.6.0.5 version (and thus my mod also) is still 8 bit except the output of mdegrain. Looking at the complexity of the code I think it would be easier to backport the plugin from VapourSynth when we need a true (stacked :) )16 bit or even float! (Avisynth+ 2017 :) ) version.

[asarian]

Well, looks like your modded version works great! :) Thanks for all your hard work!

[James Freeman]

With this line:
Dither_convert_rgb_to_yuv(matrix="2020")
http://www.mediafire.com/convkey/e11a/3raelcy59pqjqq6zg.jpg

What can be the cause??

[Groucho2004]

With this line:

http://www.mediafire.com/convkey/e11a/3raelcy59pqjqq6zg.jpg

What can be the cause??
Either Avisynth cannot find dither.dll or you're missing some MS runtime libraries (or both).
You can troubleshoot your Avisynth install with this (http://forum.doom9.org/showthread.php?t=170647) tool.

Also, post your full script.

[James Freeman]

Either Avisynth cannot find dither.dll or you're missing some MS runtime libraries (or both).
You can troubleshoot your Avisynth install with this (http://forum.doom9.org/showthread.php?t=170647) tool.

Also, post your full script.

My Script:

AVISource("D:\renderin\signpost.avi",audio=false).AssumeFPS(25,1)
Dither_convert_rgb_to_yuv(matrix="2020")
That's it.

Am I supposed to copy the files from the Win32 directory to the same with the .avsi file?

[Groucho2004]

Am I supposed to copy the files from the Win32 directory to the same with the .avsi file?
Correct.