Color banding and noise removal

[foxyshadis]

For fear of being labeled more ignorant, how would I "view it at 100%"? I understand what you typed, but I'm not sure how to "upscale" the video.

Sorry about that; AvsPmod can do that, plus it makes life so much easier anyway. When you right-click on the preview image, there's a Bit Depth option and you need to select "Stacked yuv420p16" (or whatever format you're working in if it's something else), then you will see your true 16-bit output with no downscaling, and you can scroll around as you like.

[LouieChuckyMerry]

foxyshadis: thanks for that. I checked out AvsPMod some time ago but had trouble with it not responding on my older (4GB RAM) laptop when working with HD sources. Hopefully it'll be more friendly with an SD source.

Edit: happily it's much more responsive with an SD source, but when I right-click the preview window there's no "Bit Depth" option. It's version 2.5.1, which as far as I can tell is the latest. Granted I'm a bit slow sometimes, but searching the right-click options and menu bar options I don't see anything regarding "Bit Depth" or playback possibilities other than "YUV -> RGB".

EditEdit: hmmm, seems there's a newer version (I found it here (http://www.videohelp.com/software/AvsP/old-versions#download)), 2.5.1-r426-x86-04874ed, which has a "Bit depth" option when right-clicking on the preview window. Cool...

[Dogway]

cretindesalpes, I wanted to ask if you have any hints on to what causes the mdegrain DC artifacts. I used to get them when BTB or near it values, but recently I'm including some averaged PC levels autogain to smdegrain's prefilter and I get them on almost every flash (when swift dark-bright frames occur), this puzzles me because I don't remember getting this with similar flashes on TV levels. I know you used to tackle that, I'm fixing them with repair() but just wondering if this is a bug or by design.

[cretindesalpes]

Dogway: I’m not sure to understand your terminology… What is BTB?

If you process your search clip with dynamic levels, you’re going to encounter DC artefacts in the filtered clip. This is more or less “by design”, because MDegrain averages the content of the input frames. Averaging the content of frames of different brightness will give a frame of variable brightness, depending on the weight of each input frame. Because these weights (inverse SAD) vary from frame to frame and from block to block in a single frame, as result of MAnalyse, you’re likely getting blocking artefacts during flashes or fades. Even with subtle values, the effect is noticeable and annoying. In any case, this is not related to PC or TV levels.

If you want to counter that, you have to replace the DC (and the lowest frequencies) of the filtered clip with the one of the original clip. Generally you don’t want to change them while denoising.

[Sparktank]

What is BTB?

Blacker Than Black, IIRC.

WTW = Whiter Than White

[Boulder]

Which is the better way to go what comes to final quality:

1) Process normally in 8-bit domain (denoise etc. which I cannot translate to high bit depth stuff) --> convert to 16-bit video --> deband (GradFun3) --> output 16-bit data to 10-bit x264

2) Process normally in 8-bit domain (denoise etc. which I cannot translate to high bit depth stuff) --> convert to 16-bit video --> deband (GradFun3) --> convert back to 8-bit data (DitherPost) and output to 10-bit x264

3) Process normally in 8-bit domain (denoise etc. which I cannot translate to high bit depth stuff) --> deband (GradFun3) --> output 8-bit data to 10-bit x264

My guess is that number 1 is the way to go, but I'm not sure if x264 is capable of proper dithering back to 10 bits?

[Desbreko]

1 is what you want to do for 10-bit encoding. x264 uses high-quality dithering for its bit depth conversions.

Or if you want control over the dithering, you could get a patched build of x264 that will skip bit depth conversions when the input and output depths are the same and use Dither_quantize to dither to 10-bit in your script. I'm not sure if there are any up-to-date builds with that patch, though.

2 and 3 are the same since GradFun3 works in 16-bit and just calls DitherPost to convert its output back to 8-bit without lsb=true.

[Groucho2004]

denoise etc. which I cannot translate to high bit depth stuff
Just curious - why not?

[Boulder]

There's some stuff that my skills don't cover, such as the prefilter for the analysis clip which is Flux5framesT (probably cannot be translated to 16-bit stuff)

function Flux5framesT(clip c, int "th", int "thC", bool "chromamotion")
{
chromamotion = default(chromamotion, true)
th = default(th, 2)
thC = default(thC, chromamotion ? th : 0)
med = chromamotion ? ytouv(c.utoy8().median5t(), c.vtoy8().median5t(), c.median5t()) : c.median5t().mergechroma(c)
avg = c.temporalsoften(2, th, thC, 24, 2)
output = interleave(c, med, avg).clense(grey=!chromamotion).selectevery(3,1)
return output
}

Also MAnalyse requires 8-bit input anyway and there's the contrasharpening part after the denoising by MDegrain which makes my head spin. I've seen the 16-bit mod of contrasharpening but to me it looks like it's trying to be much too fancy compared to Didée's original, brilliant idea.

I've also tried comparing the gamma-aware resizing of Dither and ResampleHQ (I almost always downsize to 720p) and found out that the method by Dither doesn't scale well with SetMTMode. The performance sometimes fluctuates quite a lot and I haven't found a good reason for that.

[Groucho2004]

There's some stuff that my skills don't cover, such as the prefilter for the analysis clip which is Flux5framesT (probably cannot be translated to 16-bit stuff)
I see. In that case, as already suggested, denoise in 8 bit, convert to 16, deband, reduce to 10 bit, feed to x264 10 bit.

[Sapo84]

Also MAnalyse requires 8-bit input anyway and there's the contrasharpening part after the denoising by MDegrain which makes my head spin. I've seen the 16-bit mod of contrasharpening but to me it looks like it's trying to be much too fancy compared to Didée's original, brilliant idea.

MAnalyse requires 8-bit but MDegrain can output in 16-bit with lsb=true, so it shouldn't be a problem at all, unless you're really against the 16-bit mod of contrasharpening.

[feisty2]

In that case you should probably switch to vaporsynth, vaporsynth mvtools takes 16bits or even 32bits (floating point) precision and, no concerns about MT stuff no more

[Boulder]

Yeah, that's one option. I've tried to do the switch at least once but haven't had the time to properly familiarize myself with the syntax. Loading stuff is easy but there's always more than that :)

[feisty2]

I don't find vaporsynth script (Python) and avisynth script much different...
Both are simple scripting languages, not monster like C++...

[Boulder]

Once you get used to Python (and coding in general, after 20 years..) :) In fact, I've had to do some very small coding on Python at work to create some helper jobs so maybe I'll get around to it in the near future.

I have a feeling that the slowdown issue with Dither's gamma-aware resizing is cache related. I just need to find out which portion of the script causes the severe slowdown because directly compared, there is no problem with it.

[Groucho2004]

I've also tried comparing the gamma-aware resizing of Dither and ResampleHQ (I almost always downsize to 720p) and found out that the method by Dither doesn't scale well with SetMTMode. The performance sometimes fluctuates quite a lot and I haven't found a good reason for that.
Do you have avstp.dll in your plugin directory? The multi-threading of avstp and SET's AVS MT don't go well together. If you have the file, remove it from the auto-load directory.

[Boulder]

I do have it but it's set to single-thread mode with avstp_set_threads(1) in my script, that should cover it.

[Groucho2004]

I do have it but it's set to single-thread mode with avstp_set_threads(1) in my script, that should cover it.
I'd still try without it, I tested it with "avstp_set_threads(1)" and it's presence still influenced the results.

[LouieChuckyMerry]

I see. In that case, as already suggested, denoise in 8 bit, convert to 16, deband, reduce to 10 bit, feed to x264 10 bit.

Happy Friday! Do you reckon there's a noticeable difference between the above and simply feeding the 16-bit output to 10-bit x264?

[feisty2]

Happy Friday! Do you reckon there's a noticeable difference between the above and simply feeding the 16-bit output to 10-bit x264?

"reckon", aussie?
no, I guess, x264 got "Sierra-2-4A error diffusion", something similar to typical FS (ditherpost mode 6)

[Groucho2004]

"reckon", aussie?
Info on the origin of the word can be found here (http://www.oxforddictionaries.com/definition/english/reckon).
I was rather sure that this word is commonly used in AE as well. Maybe it just fell through the cracks when spelling dictionaries for phones were created. :D

[fvisagie]

commonly used in AE

I reckon quite a few Safricans also use it. So not a reliable indicator of origin.

[feisty2]

I was rather sure that this word is commonly used in AE as well.

not now... unless someone like 50 yrs old "country bumpkin" (no offense)

[LouieChuckyMerry]

"reckon", aussie?
no, I guess, x264 got "Sierra-2-4A error diffusion", something similar to typical FS (ditherpost mode 6)

I'm using "Dither_Out()" to feed 10-bit x264; would there still be no apparent difference? Apologies for my obtuseness.

Info on the origin of the word can be found here (http://www.oxforddictionaries.com/definition/english/reckon).
I was rather sure that this word is commonly used in AE as well. Maybe it just fell through the cracks when spelling dictionaries for phones were created. :D

Thanks, good to learn more ways to spread The Tao Of Reckon ;) .

I reckon quite a few Safricans also use it. So not a reliable indicator of origin.

I picked it up watching weekend westerns as an impressionable youth. Nice to know I'd feel at home in South Africa :D .

[feisty2]

I'm using "Dither_Out()" to feed 10-bit x264; would there still be no apparent difference? Apologies for my obtuseness.
you can either dither down to 10bits and x264 will take it and skip the bit depth conversion, or pipe that 16bits stuff directly to x264 and x264 will "Sierra-2-4A" it to 10bits
so, no apparent difference

[LouieChuckyMerry]

Thanks for the clarification, feisty2 :) .

[Desbreko]

Does vanilla 10-bit x264 skip the bit depth conversion now if you feed it 10-bit? Last I heard, it would convert to 16-bit first and then dither down to 10-bit.

[feisty2]

the internal bit depth will always be, 16bits, like, that's an elementary data type to computers
you got char, which is an 8bits integer, and can be used to do, stuff, to 8bits vids
and short, which is a 16bits integer, and in between, it's just, nothing, 10bits integer? never heard of it
8<bit depth<=16 just gotta be padded to 16bits first and truncated (or dithered) to the desired bit depth when all the internal processes are done

[sneaker_ger]

Does vanilla 10-bit x264 skip the bit depth conversion now if you feed it 10-bit? Last I heard, it would convert to 16-bit first and then dither down to 10-bit.
I think that has remained unchanged but does not matter anyways. At least that's how I always understood it.
x264 uses a variant on Sierra-2-4A error diffusion, a faster alternative to Floyd-Steinberg error diffusion. x264's implementation is explicitly designed such that if the input is upscaled within x264 to a higher bit depth, then downscaled using error diffusion, the result will be losslessly identical to the input.

[LouieChuckyMerry]

the internal bit depth will always be, 16bits, like, that's an elementary data type to computers
you got char, which is an 8bits integer, and can be used to do, stuff, to 8bits vids
and short, which is a 16bits integer, and in between, it's just, nothing, 10bits integer? never heard of it
8<bit depth<=16 just gotta be padded to 16bits first and truncated (or dithered) to the desired bit depth when all the internal processes are done

Interesting information to this ignorant and curious individual. Any idea why 10-bit x264 isn't instead 16-bit x264?

[feisty2]

H.264 got no profile with support to 16bits precision (High444pp supports bit depth up to 14bits)

[colours]

Does vanilla 10-bit x264 skip the bit depth conversion now if you feed it 10-bit? Last I heard, it would convert to 16-bit first and then dither down to 10-bit.

Must've been quite a few years ago when you last heard, then. I think bypassing the depth filter has been around since 2012/2013.

the internal bit depth will always be, 16bits, like, that's an elementary data type to computers
H.264 got no profile with support to 16bits precision (High444pp supports bit depth up to 14bits)

Wrong. Bit depth is not the size of the integer format used to contain data.

[feisty2]

Right, 8-15bits ain't got padded to 16bits, uint16_t is used for, it's just the smallest data type that covers the whole range of 9-16bits
My mind was floating somewhere else when I said that :D

[tormento]

>>>> dfttest 1.9.4 <<<< (http://ldesoras.free.fr/src/avs/dfttest-1.9.4.zip)
Mod16 version of dfttest.
>>>> MVTools 2.6.0.5 <<<< (http://ldesoras.free.fr/src/avs/mvtools-2.6.0.5.zip)
Mod16 version of MVTools 2
Any chance to get a x64 compile of them?

At least of modified MVTools.

Please. =P

[feisty2]

Go vaporsynthing

[tormento]

Go vaporsynthing

VS MVTools goes up to 3 frames and there is no official SMDegrain script. I need from 4 to 6 frames and dogway support ;)

[feisty2]

VS MVTools goes up to 3 frames and there is no official SMDegrain script. I need from 4 to 6 frames and dogway support ;)

http://forum.doom9.org/showthread.php?p=1736069#post1736069
see EDIT10

[tormento]

http://forum.doom9.org/showthread.php?p=1736069#post1736069
see EDIT10
LOL 24 frames. Thank you.

Seriously, I want to solve this issue on standard AviSynth or AviSynth+ before starting to learn new tricks.

[feisty2]

the source code is there, compile to x64 yourself

[tormento]

the source code is there, compile to x64 yourself

Hey Siri, compile MVtools in x64 flavour. :D

Compiling is beyond my capabilities.

[feisty2]

I don't know what you mean by "compile MVtools in x64 flavour."
How about a web search for it?

seriously... avs mvtools is real horrible (like, code monster), and no one wants to touch it....

[Reel.Deel]

Any chance to get a x64 compile of them?

At least of modified MVTools.

Please. =P

Both x64 plugins can be found here: http://avisynth.nl/index.php/AviSynth%2B#AviSynth.2B_x64_plugins

[tormento]

Both x64 plugins can be found here: http://avisynth.nl/index.php/AviSynth%2B#AviSynth.2B_x64_plugins

:thanks:

It's a pity AviSynth+ has a problem of image corruption.

[tormento]

cretindesalpes, I wanted to ask if you have any hints on to what causes the mdegrain DC artifacts. I used to get them when BTB or near it values, but recently I'm including some averaged PC levels autogain to smdegrain's prefilter and I get them on almost every flash (when swift dark-bright frames occur), this puzzles me because I don't remember getting this with similar flashes on TV levels. I know you used to tackle that, I'm fixing them with repair() but just wondering if this is a bug or by design.

I suppose it's similar to what I get with AVS+.

[Dogway]

I suppose it's similar to what I get with AVS+.
No, it's not. It's an "issue" related to mvtools2 on an unreleased version.
If I understood correctly your problem is with KNLmeansCL right?

[tormento]

No, it's not. It's an "issue" related to mvtools2 on an unreleased version.
If I understood correctly your problem is with KNLmeansCL right?
No, image corruption is related to AVS+ (32/64) and SMDegrain, prefilter 2 or 4.

[Dogway]

Those are quantile and KNLMeansCL, narrow down the culprit.

[tormento]

Those are quantile and KNLMeansCL, narrow down the culprit.

Other modes not tested, as 2 (software) and 4 (hardware) gives me similar if not identical corruption.

[LouieChuckyMerry]

tormento: I don't know if this is related, but I had corruption issues with SMDegrain that were due to having the AVSTP.dll in my plugins folder. As soon as I removed it the corruption stopped.

[tormento]

tormento: I don't know if this is related, but I had corruption issues with SMDegrain that were due to having the AVSTP.dll in my plugins folder. As soon as I removed it the corruption stopped.
AVSTP has been removed as I use MT.

Even tests with single thread were done without that library.

However: :thanks: