How does High Bit Depth (HBD) work with AVS+

[Hotte]

Hi,

Probably a simple question:
I am trying to understand the concept of interim HDR solutions (stacked lsb filters). I have read quite a lot but cannot get it to work.

I have a 10-bit YUY2 AVI (Canopus HQX) as Input and would like to apply SMDegrain in lsb-mode:

FFmpegSource2("myfile.avi", atrack=-1)
SMDegrain(tr=1,thSAD=300,lsb_in=true,lsb_out=true)

Tells me: "lsb-Flag only for 8-bit sources" ?!
Info() tells me, it is a 16-bit source.
How does that work ?

[FranceBB]

HDR has nothing to do with HBD.
HDR stands for High Dynamic Range and it basically means that there are logarithmic or pseudo-logarithmic curves in a signal to extend the "range" from the standard 100 nits to an higher one (and we're gonna leave it there).
What you are referring to is HBD.
HBD stands for "High Bit Depth" and it refers to the bit depth in which you're working with.
There are 3 possible ways to work with high bit depth in Avisynth:

- Planar
- Stacked
- Interleaved

Planar means regular high bit depth, namely you can work in 8bit, 10bit, 12bit, 14bit, 16bit planar and 32bit float. The picture is the way it is and has regular high bit depth.
In other words, 1920x1080 16bit planar is 1920x1080.

https://i.imgur.com/tsDIBBZ.png

Stacked means that MSB (most significant bit) and LSB (less significant bit) are "stacked" one on top of the other, so you end up with a picture which has double its height.
In other words, 1920x1080 16bit stacked becomes 1920x2160 as it has 8bit MSB + 8bit LSB stacked one on top of the other.

https://i.imgur.com/s4yznAb.png

Interleaved means that MSB and LSB are "interleaved" together one next to the other, so you end up with a picture which has twice its Width.
In other words, 1920x1080 16bit interleaved becomes 3840x1080 as it has 8bit MSB + 8bit LSB interleaved one next to the other.

https://i.imgur.com/s9g8ilr.png

Now, in your case, there's something... ehm... a bit peculiar.
Your source is YUY2... 10bit? O_o
I didn't know YUY2 existed outside 8bit as it certainly doesn't in Avisynth.
What I think happens, then, is that ffms2 converts YUY2 to 4:2:2 planar 10bit so you end up with a yuv422p10.
Once you have that, you have to work with SMDegrain in PLANAR, however with lsb_in=true and lsb_out=true you're telling SMDegrain to work in STACKED.
So, to keep your filterchain intact, you can do:


#Indexing
FFmpegSource2("myfile.avi", atrack=-1)

#From 10bit planar to 16bit planar
ConvertBits(16)

#From Planar to Stacked
ConverttoStacked()

#Denoise with 16bit stacked precision
SMDegrain(tr=1,thSAD=300,lsb_in=true,lsb_out=true)

#From 16bit Stacked back to 16bit planar
ConvertFromStacked()


Cheers,
Frank

[Hotte]

:thanks: Frank. That´s the way it works. Thank you also for being so kind to explain the matter so detailed and intelligible.

I could not find an exact specification of the Canopus HQX codec built into Edius NLE, but still with 8-bit export I am getting YUY2 and with 10-bit Export I am getting YUV422P16 out of FFMS2, so the ConvertBits(16) was not necessary.

Maybe you could help me one more time with my favourite though very simple detail-sharpener. For sbdy experienced it is probably a simple task to make this little filter 16-bit ready.

It used to work with BinomialBlur instead of MedianBlur, but there is no HBD-version of BinomialBlur available whereas MedianBlur2 supports HBD natively:


Function mySharpen(clip clp, int "strength", int "width")
{
i=clp.MedianBlur(radiusY=1, radiusU=1, radiusV=1)
j=subtract(i,clp).levels(0+strength,1,255-strength,0,255)
width=width*(128/strength)
k=j.levels(127-width,1,128+width,127-width,128+width
clp=subtract(clp,k)
Return clp
}


Due to your suggestions I chained this the following way:


clp = ConverttoStacked(clp)
clp = mySharpen(clp,strength=40,width=2)
clp = SMDegrain(clp,tr=1,thSAD=1000,lsb_in=true,lsb_out=true)
clp = ConvertFromStacked(clp)


Well, at least I am getting no errors, sharpening works and the results looks fine.
Still my impression is that this is no true 10/16-bit sharpened result anymore as I did not care for any lsb/msb matter.

What do you think ?

[real.finder]

why use lsb hack in SMDegrain when there HBD support since age?!

also lsb_in in SMDegrain is not real 16bit

[FranceBB]

Still my impression is that this is no true 10/16-bit sharpened result anymore as I did not care for any lsb/msb matter.

What do you think ?

That it's actually wrong as you're filtering the LSB part as if it was part of the video too, effectively screwing up the values. The only reason why to your eyes it looks good is that the filters are working ok-ish in the MSB 8bit part and that's the one you're seeing the difference on, but you're messing up with the lower LSB values.
I could try to scale the values for you, but it's late in the night and I'm gonna take the safe route, so here's my suggestion:


#Indexing
FFmpegSource2("myfile.avi", atrack=-1)

#From 10bit planar to 16bit planar
ConvertBits(16)

#From Planar to Stacked
ConverttoStacked()

#Denoise with 16bit stacked precision
SMDegrain(tr=1,thSAD=300,lsb_in=true,lsb_out=true)

#we save the 16bit stacked
s16=last

#we go down to 8bit planar and apply the sharpening
DitherPost(mode=-1)
mySharpen(strength=40,width=2)

#we take back the 16bit stacked denoised clip before sharpening
#and we overlay the sharpened 8bit details to the original
#16bit clip, hence preserving the 16bit precision
Dither_convert_8_to_16 ()
s16.Dither_limit_dif16 (last, thr=1.0, elast=2.0)

#From 16bit Stacked back to 16bit planar
ConvertFromStacked()


;)



why use lsb hack in SMDegrain when there HBD support since age?!

I was pretty sure planar was supported, but since it's not a filter I use and I didn't know for sure and he was using the stacked command, I took the safe route, 'cause deep inside we all know I'm lazy xD

[Hotte]

I tried this and it worked!

Thank you so much, Frank!

No I am starting to understand how dither works.

There are so many questions still left... I do not want to overtax your generosity. The problem with all this powerful tools is that it is difficult to understand the meaning of all this tech-dialogs and overly brief insider explanations. So if you could be so kind to give me some hints with the following questions:

1. My sharpener only uses internal filters (except for MedianBlur). I thought that Avs+ ist fully compatible with HBD. So why does the sharpener only work on the msb-part of the clip ?

2. If I am using internal filters like Lanczos4Resize(), Crop() or Overlay() within AVS+, do they keep the 16bit HBD or do I have to use dither tools to keep the 16bit ?

3. Is there a simple method to check if my 16-Bit chain is still intact, i.e. the 16bit are still there, e.g. a testclip in/out comparison or sth ?

4. If I overlay 8-bit-sharpened footage with s16 I understood that only the upper 8-bits of colour are being sharpend, not the lower 2 to 8 bits. Isn't there any visually perceptable effect to this ?

5. How can I find out if a filter or script is 16-bit compatible ? A good example is the script dehalo_alpha, which I use quite lot ?

Thank you for your time.

[Hotte]

Hi real.finder,

why use lsb hack in SMDegrain when there HBD support since age?!

Interesting! I tried to follow these instructions:
High Bit Depth Processing And Dithering

lsb_in bool = False
Set true if you use a 16 bits filter chain. *(EXPERIMENTAL, use under test environment)

lsb_out bool = False
Output to 16-bit instead of 8-bit. Use the helpers at the end of the SMDegrain() script to use Ditherpost() with YUY2 formats.

lsb bool = False
This enables 32 bit depth precision for denoising. You gain extra denoising accuracy, most noticeable in the prevention of banding in flat areas. Automatically set if either lsb_in or lsb_out are enabled.

mode int (-1-8, default 0)
This is the mode of DitherPost when lsb_out=False, as a dithering method must be chosen for the 32bit->8bit conversion. (Interlaced content is locked to mode=6) The default mode=0 will help you optimize the dithering for optimum encodings when no further non-edge processing is done. Use mode=6 (error diffusion) if further processing will be done.

Why does the filter have an lsb-hack if you can call it directly ? The following works. But I am not sure if I am getting true HBD. Or do I need to call it in a different way ?
FFmpegSource2("myFile.avi", atrack=-1) # loads as YUV422P16
SMDegrain(tr=1,thSAD=1000)
Return last

[Asmodian]

Why does the filter have an lsb-hack if you can call it directly ?

Legacy reasons, the hack was implemented well before the non-hack way existed.

[FranceBB]

I'll try to address all your points, don't worry.
I do this in my spare time, I like helping the community.
Besides, if people are confused about this, it means that we should once again get our hands on the wiki and improve it even further. :)
Ok, on to the points:

1) Avisynth internal functions in the core are all high bit depth aware, however they're planar high bit depth aware, so they don't work in stacked or interleaved. The reason why in your case they were messing around is that there's no way to tell them that your picture is not a weird aspect ratio one but rather a 16bit stacked one.

2) resizers work in planar high bit depth, so you must use them with regular high bit depth. If you're in stacked, convert and use the normal HBD, then convert back. Same goes for interleaved. For instance:

ConvertBits(16)
Spline36Resize(1280,720)

Works.

3) Uhm... If you work in planar it's easy, but if you work in stacked or interleaved it's tricky. This is because Avisynth officially doesn't know that you're working in stacked or interleaved as there are no metadata to tell the frameserver (and why would there be... After all they were invented before Avisynth supported regular HBD to overcome its limitations)

4) It's less precise, but at least it's only gonna be the sharpening which is less precise and you don't compromise the denoise precision. By the way, there are plenty of sharpeners that work in planar high bit depth.

5) Documentation. If it's stacked or interleaved compatible the author must tell you via the documentation. If it's planar high bit depth compatible, they should tell you in the documentation, but if they don't, you can easily just try a simple ConvertBits(16) and then apply the filter. If it returns an output, it's very likely that it works, if it doesn't recognize the colorspace and outputs an error, then it doesn't. We're trying to keep the Avisynth wiki updated (me included) but it's a mammoth effort and it's very difficult... For what it's worth, here you can find my "suggested filters for everything" and in which bit depth they work: https://github.com/FranceBB/FranceBB_Sample-Avisynth-Script I actually made it for myself but a friend of mine actually wanted it time ago so I put it online and apparently other people found it interesting too... Anyway, feel free to use it as a reference. ;)


Why does the filter have an lsb-hack if you can call it directly ? The following works. But I am not sure if I am getting true HBD.


I think that by calling it that way you're using the planar (regular) high bit depth version of that filter.
It's not uncommon for filters to support more than one high bit depth type. For instance, f3kdb supports both stacked and interleaved etc.
By the way, real finder is one of those I really appreciate and respect 'cause unlike other people he is very keen on not breaking support. You see, sometimes filters that were supporting 16bit stacked or interleaved and are modified and modernized to work with planar lose their ability to work with 16bit stacked and interleaved, hence breaking old scripts that used to rely on them. Real Finder, though, keeps everything and we're on the same page about this. :)
Sometimes you'll also find "neo" close to a plugin, it means that it's a newer version of the plugin, but it breaks compatibility (like different parameters etc) and as result the function name has been changed.


Avisynth is a wonderful yet hugely wide world, but all its peculiarities and its history made it the way it is today, namely the best frameserver in the world. :)

Some Vapoursynth users might argue to the last sentence, but hey, it's the Avisynth section xD

[real.finder]

Well, the Documentation is not updated, same as the wiki (maybe the wiki is kinda get small updates from time to time)

I cant do Documentation also I am not good in English, maybe another one can do update them

[Hotte]

Thank you so much Frank for throwing light into the HBD corner. I have now quite some homework to experiment with the hints you gave and will possibly come back to this thread if questions arise.

One was left for today: I have had a look at your wonderful recommendation list of filters and found BlindDeHalo3 as dehalo filter. It took me a while to find the HBD-version in some pastebin, the wiki link seemed to point to a non-HBD version. I tried out BDH3 and found that it removes halos well however at the cost of significant overall blurring. This is much less the case with dehalo_alpha if you are ready to accept some fine tuning. Also dehalo_alpha allows to differentiate between black and white halos and if I only need white halo filtering the risk of overall blurring is even smaller.

So I´d like to stick to dehalo_alpha which was designed to use YV12 as input. And here is where the HBD question drops in. If I code
clp = Converttoyuv422(clp)
clp = convertbits(clp,bits=16)
clp = dehalo_alpha(clp, rx=1.4, ry=1.4, darkstr=0.0, brightstr=1.3, lowsens=200, highsens=300, ss=2.2)


I am getting no errors and correct results from a visual pov. But is this still true 16 bit ? With your answer to my question 3) (How can I check if my 16-bit chain is intact?) you were saying: "If you work in planar it's easy". So how can I check ?

Thanks.

I added the code of the dehalo_alpha version I used:
function DeHalo_alpha(clip clp, float "rx", float "ry", float "darkstr", float "brightstr", float "lowsens", float "highsens", float "ss")
{
rx = default( rx, 2.0 )
ry = default( ry, 2.0 )
darkstr = default( darkstr, 1.0 )
brightstr = default( brightstr, 1.0 )
lowsens = default( lowsens, 50 )
highsens = default( highsens, 50 )
ss = default( ss, 1.5 )

LOS = string(lowsens)
HIS = string(highsens/100.0)
DRK = string(darkstr)
BRT = string(brightstr)
ox = clp.width()
oy = clp.height()
uv = 1
uv2 = (uv==3) ? 3 : 2

halos = clp.bicubicresize(m4(ox/rx),m4(oy/ry)).bicubicresize(ox,oy,1,0)
are = mt_lutxy(clp.mt_expand(U=uv,V=uv),clp.mt_inpand(U=uv,V=uv),"x y -","x y -","x y -",U=uv,V=uv)
ugly = mt_lutxy(halos.mt_expand(U=uv,V=uv),halos.mt_inpand(U=uv,V=uv),"x y -","x y -","x y -",U=uv,V=uv)
so = mt_lutxy( ugly, are, "y x - y 0.001 + / 255 * "+LOS+" - y 256 + 512 / "+HIS+" + *" )
lets = mt_merge(halos,clp,so,U=uv,V=uv)
remove = (ss==1.0) ? clp.repair(lets,1,0)
\ : clp.lanczosresize(m4(ox*ss),m4(oy*ss))
\ .mt_logic(lets.mt_expand(U=uv,V=uv).bicubicresize(m4(ox*ss),m4(oy*ss)),"min",U=uv2,V=uv2)
\ .mt_logic(lets.mt_inpand(U=uv,V=uv).bicubicresize(m4(ox*ss),m4(oy*ss)),"max",U=uv2,V=uv2)
\ .lanczosresize(ox,oy)
them = mt_lutxy(clp,remove,"x y < x x y - "+DRK+" * - x x y - "+BRT+" * - ?",U=2,V=2)

return( them )
}

[real.finder]

dehalo_alpha already got HBD update in my Avisynth Stuff (see signature)

edit:


It used to work with BinomialBlur instead of MedianBlur, but there is no HBD-version of BinomialBlur available whereas MedianBlur2 supports HBD natively:


Function mySharpen(clip clp, int "strength", int "width")
{
i=clp.MedianBlur(radiusY=1, radiusU=1, radiusV=1)
j=subtract(i,clp).levels(0+strength,1,255-strength,0,255)
width=width*(128/strength)
k=j.levels(127-width,1,128+width,127-width,128+width
clp=subtract(clp,k)
Return clp
}



I don't think MedianBlur can be a replacement for BinomialBlur, good BinomialBlur replacements I find are


Variance = 4.0
rad=round(Variance*2)
#~ sh_Padding(rad,rad,rad,rad).GBlur(rad,sqrt(Variance)).crop(rad,rad,-rad,-rad)
#~ BinomialBlur(Variance)
FastBlur(sqrt(Variance),iterations=2,gamma=false) #FastBlur(sqrt(Variance),iterations=rad,gamma=false)

[Hotte]

Hi real.finder & Frank,

@real.finder: Your filter repository is a real treasure chest. Should really find its way into the AVS-wiki!

With your help guys I was able to create my first filter: A rewrite of my little sharpener to become a real 16-bit YUV-Filter :-)

When I am doing some deep zooming-in and pixel peeping I can clearly see, that the colours are a tad clearer and more precise compared to the old YV12 version even if input footage was only 8-bit.

This simple unsharp-mask sharpener does a very good job in fine-sharpening. It increases micro-contrast without
creating edge halos (although it may emphasize noise as well as existing halos).

(This is a bit off-topic, but if anybody knows how I could limit amplification of micro contrast to areas of higher contrast only, we would have less amplifcation of noise).

iterat: 1-n to multiply the effect. Do not exagerate.
strength: 1-63
width: 0-127

function mySharpen_HBD(clip clp, int "iterat", int "strength", int "width")
{ i = iterat
s = float(strength * 256)
w = float(width * 256)

for (c=1,i,1)
{ clpBlur = FastBlur(clp,0.75,iterations=2,gamma=false)
clpMask = Subtract(clpBlur,clp)
clpMask = Levels(clpMask,s,1,65280-s,0,65280)
w = w * (32768/s)
clpMask = Levels(clpMask,32768-w,1,32768+w,32512-w,32768+w)
clp = Subtract(clp,clpMask)
}
return(clp)
}

[Hotte]

....
#From 10bit planar to 16bit planar
ConvertBits(16)

#From Planar to Stacked
ConverttoStacked()

#Denoise with 16bit stacked precision
SMDegrain(tr=1,thSAD=300,lsb_in=true,lsb_out=true)

#From 16bit Stacked back to 16bit planar
ConvertFromStacked()
....


@Frank, or any other expert out here, I have started to integrate all your hints into my regular automated workflow and stumbled upon these questions (and this is not about the known ability of SMDegrain to work in native HBD):

1. Does ConverttoStacked() / ConvertFromStacked() come with any loss in image quality if I do it several times ? I'd suspect: No, it is just a rearrangement of data, no information is lost. If yes, I'd have to optimize the automated workflow for as few conversions as possible.

2. Similar question: Does ConvertBits() come with any loss if the original bit-depth is not being underrun, e.g. my video is originally 10-bit, I convert to 16 and immediately after back to 10-bit. Any losses ?

3. Just to be sure: I learned that native avs+ filters are HBD compatible. But they work on all bitdepths - don't they ?

# last video is 10-bit
ConvertBits(16)
Spline36Resize(1280,720)
ConvertBits(10)

# ...These converts are not necessary. I could have done:
Spline36Resize(1280,720)


Thank you.

[poisondeathray]

1. Does ConverttoStacked() / ConvertFromStacked() come with any loss in image quality if I do it several times ? I'd suspect: No, it is just a rearrangement of data, no information is lost. If yes, I'd have to optimize the automated workflow for as few conversions as possible.



1) no; it's a lossless transform - and you can confirm with psnr or similar, I just checked - it's ok



2. Similar question: Does ConvertBits() come with any loss if the original bit-depth is not being underrun, e.g. my video is originally 10-bit, I convert to 16 and immediately after back to 10-bit. Any losses ?


no; it's lossless if there are no other operations applied such as dithering (Down bit depth conversion is often done with dithering). But and you can confirm with psnr or similar, I just checked it's ok

http://avisynth.nl/index.php/ConvertBits
By default dither=-1 (no dithering)



3. Just to be sure: I learned that native avs+ filters are HBD compatible. But they work on all bitdepths - don't they ?

# last video is 10-bit
ConvertBits(16)
Spline36Resize(1280,720)
ConvertBits(10)

# ...These converts are not necessary. I could have done:
Spline36Resize(1280,720)




I do not think "all" native filters are . You will get a warning message if something is not compatible

[Hotte]

Hi,

in this example DitherPost(mode=-1) is used to go from 16-bit stacked to 8-bit planar:

DitherPost(mode=-1)
mySharpen(strength=40,width=2)


Would
ConvertFromStacked()
ConvertBits(8) # ...with a dither option
mySharpen(strength=40,width=2)
be an appropriate alternative or has this any disadvantage ?

[poisondeathray]

Hi,

in this example DitherPost(mode=-1) is used to go from 16-bit stacked to 8-bit planar:


Would
ConvertFromStacked()
ConvertBits(8) # ...with a dither option
mySharpen(strength=40,width=2)
be an appropriate alternative or has this any disadvantage ?


quality wise those are identical

speed wise, ditherpost method should be slightly faster

but if all filters were native (not stacked workflow anywhere) this would be the fastest

You can use avsmeter64 to test speeds, and or ffmpeg pipe speeds or similar measurements. (And PSNR or similiar to test "quality" compared to a reference)

[Hotte]

Thanks, poisondeathray. Qualified as ever.

[FranceBB]

Thanks, poisondeathray. Qualified as ever.

Indeed he is, I have nothing to add. :)

Oh well maybe just a thing:
the only reason why I didn't use dithering in that instance is that you don't really need it as you won't be using the 8bit video. In the script I made you will take the differences in the sharpened 8bit video and overlay them to the original 16bit stacked video, so dithering would make it worse as it would make it overlay back sparsed data.
If we were going to go to 8bit for real and deliver an 8bit I would have used dithering then. Especially on broadcast quality mezzanine files, Floyd Steinberg error diffusion works great (for consumer-tier low bitrates, not so much). There are others like Stucki and Atkinson, but for consumer bitrates I deliver HBD to the encoder like x264/x265 and I let it dither.


Alright, I have nothing else to add for real now xD

[Hotte]

Thanks Frank. I am about to make my environment HBD-compatible. My avs+ conversion scripts are being generated automatically and at the moment the generator learns to deal with HBD. I am integrating the dithering option in the ConvertBits() function so I can simply switch it on or off for comparison.
The current idea is to have a "fast" mode which passes 8-bit through 8-bit filtering and a "slow" mode which filters everything in 16-bit and convertBits(8) down at the very end using dithering if the result has to be 8-bit.
This is because I have seen 8-bit material to (very slightly) benefit from 16-bit filtering (sounds 'bit like Jesus multiplying bread). I am very curious what the results will look like.

[miltonsoto658]

Hi guys, can someone please help me out?, i'm trying to denoise a video with this script in HBD but after i call ConvertFromStacked() i still get the video with double height i know the problem is in a.mdegrainn but i don't have any idea how to solve it without deleting "out16=true"

ConvertBits(16)
ConvertToStacked()
a=last
pre=minblur_sbr16 ().Dfttest (sstring="0.0:4.0 0.2:9.0 1.0:15.0",tbsize=1,lsb=true,lsb_in=true,quiet=true).edgecleanse16 ().Vinverse ()
super1=pre.msuper (pel=4,chroma=true,hpad=8,vpad=8,sharp=2,rfilter=4,levels=0)
super2=a.msuper (pel=4,chroma=true,hpad=8,vpad=8,sharp=2,rfilter=4,levels=1)
vmulti=super1.manalyse (multi=true,overlap=2,blksize=4,search=3,chroma=true,truemotion=false,delta=6,searchparam=16,pelsearch=16,dct=6)
a.mdegrainn (super2,vmulti,6,thSAD2=200,thSADC2=200,thSAD=1200,thSADC=400,limit=255,limitc=255,plane=4,out16=true)
ConvertFromStacked()

https://imgur.com/igtupKG.png

[FranceBB]

Ok, a couple of things: when you work in 16bit, either stacked, interleaved or planar, all filters MUST be 16bit aware and support the 16bit flavor you're working with.
Take a look at Vinverse for instance: it's chained there, but it's an 8bit planar filter with no clue whatsoever about 16bit...
This is just wrong as it's gonna filter out stuff in the LSB as if it was MSB!
Same goes for MSuper, Manalyse and MDegrain, they must know in which bit depth they're working on!

This works:


#Indexing
FFMpegSource2("\\mibctvan000\Ingest\MEDIA\temp\ACC47774-lc.mxf", atrack=-1)

#Going to 16bit stacked
ConvertBits(16)
a=last
ConvertToStacked()

#Blur with 16bit stacked precision
minblur_sbr16()

#Denoise with 16bit stacked precision
Dfttest(sstring="0.0:4.0 0.2:9.0 1.0:15.0",tbsize=1, lsb=true, lsb_in=true, quiet=true)

#Edge cleaning with 16bit stacked precision
edgecleanse16()

#Apply vinverse as 8bit planar filter
s16 = last
DitherPost(mode=-1)
Vinverse()

#Overlay 8bit vinversed clip to 16bit stacked video thus retaining 16bit stacked precision
Dither_convert_8_to_16 ()
s16.Dither_limit_dif16 (last, thr=1.0, elast=2.0)

#Going back to 16bit planar
ConvertFromStacked()
pre=last

#Degrain with 16bit planar precision
super1=pre.msuper (pel=4, chroma=true, hpad=8, vpad=8, sharp=2, rfilter=4, levels=0)
super2=a.msuper (pel=4, chroma=true, hpad=8, vpad=8, sharp=2, rfilter=4, levels=1)
vmulti=super1.manalyse (multi=true, overlap=2, blksize=4, search=3, chroma=true, truemotion=false, delta=6, searchparam=16, pelsearch=16, dct=6)
a.mdegrainn (super2, vmulti, 6, thSAD2=200, thSADC2=200, thSAD=1200, thSADC=400, limit=255, limitc=255, plane=4)


https://i.imgur.com/BGi4xro.png

[miltonsoto658]

Thank you so so much for your answer and your time to answer, it worked perfecty and cleared a lot of doubts i had

Hope you can help me with this too, i also have this script, i know KNLMEANSCL can work in stacked but not finedehalo, would there be any problem? or how how could this be adjust after the denoising script i asked before, thanks in advance!

s=last
knlmeanscl (d=0, a=15, s=0, h=50,stacked=true)
x=last
mt_makediff(x,x.blur(1.57))
mt_adddiff(s.blur(1.57),last)
y=last
halomask=x.finedehalo (showmask=1)
mt_merge (s,y,halomask)

[FranceBB]

Uhmmmm I'm not terribly familiar with FineDeHalo but from what I'm seeing it only supports yv12 8bit planar, which is not good.
Anyway, I checked and KNLMeansCL not only supports 16bit stacked but also 16bit planar, so it would be better to just stay in planar there.
The problem arises with finedehalo, though, 'cause it only really works in 8bit planar yv12, therefore you should be using FineDehaloHBD instead, which you can find here: https://forum.doom9.org/showpost.php?p=1846547&postcount=1
FineDehaloHBD works in 16bit planar, so you can just do:


#Indexing
FFMpegSource2("\\mibctvan000\Ingest\MEDIA\temp\ACC47774-lc.mxf", atrack=-1)

#Going to 16bit stacked
ConvertBits(16)
a=last
ConvertToStacked()

#Blur with 16bit stacked precision
minblur_sbr16()

#Denoise with 16bit stacked precision
Dfttest(sstring="0.0:4.0 0.2:9.0 1.0:15.0",tbsize=1, lsb=true, lsb_in=true, quiet=true)

#Edge cleaning with 16bit stacked precision
edgecleanse16()

#Apply vinverse as 8bit planar filter
s16 = last
DitherPost(mode=-1)
Vinverse()

#Overlay 8bit vinversed clip to 16bit stacked video thus retaining 16bit stacked precision
Dither_convert_8_to_16 ()
s16.Dither_limit_dif16 (last, thr=1.0, elast=2.0)

#Going back to 16bit planar
ConvertFromStacked()
pre=last

#Degrain with 16bit planar precision
super1=pre.msuper (pel=4, chroma=true, hpad=8, vpad=8, sharp=2, rfilter=4, levels=0)
super2=a.msuper (pel=4, chroma=true, hpad=8, vpad=8, sharp=2, rfilter=4, levels=1)
vmulti=super1.manalyse (multi=true, overlap=2, blksize=4, search=3, chroma=true, truemotion=false, delta=6, searchparam=16, pelsearch=16, dct=6)
a.mdegrainn (super2, vmulti, 6, thSAD2=200, thSADC2=200, thSAD=1200, thSADC=400, limit=255, limitc=255, plane=4)

#Additional filtering with 16bit planar precision
s=last
KNLMeansCL(d=0, a=15, s=0, h=50, stacked=false)
x=last
mt_makediff(x,x.blur(1.57))
mt_adddiff(s.blur(1.57),last)
y=last
halomask=x.FineDehaloHBD(showmask=1)
mt_merge(s,y,halomask)




Also another doubt i have is can this values be changed?

#Apply vinverse as 8bit planar filter
s16 = last
DitherPost(mode=-1)
Vinverse()

To

#Apply vinverse as 8bit planar filter
s16 = last
ConvertBits(8)
Vinverse()

and

#Overlay 8bit vinversed clip to 16bit stacked video thus retaining 16bit stacked precision
Dither_convert_8_to_16 ()
s16.Dither_limit_dif16 (last, thr=1.0, elast=2.0)

to

#Overlay 8bit vinversed clip to 16bit stacked video thus retaining 16bit stacked precision
ConvertBits(16)
s16.Dither_limit_dif16 (last, thr=1.0, elast=2.0)


No!! :scared:

You can't!
We're not just truncating to 8bit planar and get away with it!
No, no, that would be bad!
We're basically truncating to 8bit planar, filtering with 8bit planar, THEN go back to 16bit stacked and calculate the differences between the 16bit stacked frame before going down to filter and the 8bit planar brought back to 16bit stacked. This way, we're gonna be able to RETAIN the 16bit stacked precision ;)
In your suggestion, you're going to 8bit planar with ConvertBits(8) and that's fine, then you filter, then you go back to 16bit planar with ConvertBits(16) and... you call Dither_limit_dif16() which is a filter that expects 16bit STACKED instead! This is a big problem as it won't work!

In other words this is wrong:


ConvertBits(16)
s16.Dither_limit_dif16 (last, thr=1.0, elast=2.0)

and must be:

#Overlay 8bit vinversed clip to 16bit stacked video thus retaining 16bit stacked precision
Dither_convert_8_to_16 ()
s16.Dither_limit_dif16 (last, thr=1.0, elast=2.0)

otherwise, if you MUST use ConvertBits(), you could do:

#Overlay 8bit vinversed clip to 16bit stacked video thus retaining 16bit stacked precision
ConvertBits(16)
ConverttoStacked()
s16.Dither_limit_dif16 (last, thr=1.0, elast=2.0)

but... why would you wanna do that in 2 steps when you can do it in 1 with Dither_convert_8_to_16() ? Efficiency, my friend ehehehehehheh

this last question is cause i happened to have more pink-ish colors when using ConvertBits(16) and go back to 8bit with ditherpost(mode=-1) instead of using ConvertBits(8) or also doing this
LSMASHVideoSource("x",format="YUV420p8") (not sure about "Dither_convert_8_to_16 ()")

Not sure, but yeah it might be a bug in the old DitherPost().
If it makes you feel better, you can do it your way. :)

[kedautinh12]

FineDehalo latest ver
https://github.com/realfinder/AVS-Stuff/blob/Community/avs%202.5%20and%20up/FineDehalo.avsi