Stacked16 / Converttostacked

[anton_foy]

Hello!

I have now upgraded to Avisynth+ and want to use some 8-bit filters in 16-bit with Stack16 but when I run a script with "convertToStacked()" or "convertToDoubleWidth()", I get the error:
Avisynth open failure:
Script error: There is no function named 'converttostacked'.

When I run Version() it says:
Avisynth+ version 3.6.1 (r3300, 3.6, I386)
...so why are the Avisynth+ filters not working?
Here is my script:

FFmpegSource2("F:\video.mp4")
SetMemoryMax(2048)

coverttostacked()
###Levels###
Levels(0, 1, 255, 0, 235, coring=false, dither=false)

### DENOISE ###
fft3dfilter(sigma=1.5, bt=5, bw=32, bh=32, ow=16, oh=16, sharpen=0.0, plane=3).removegrain()
gradfun2db(0.4)

### TOPSHARP ###
shrp=sharpen(1.0)
vid1 = last
shine1 = vid1.Overlay(shrp, mode="blend", opacity=1.0)
emask1 = vid1.mt_edge(mode="prewitt",thy1=0,thy2=255).expand().fastgaussblur(10)
result1 = mt_merge(vid1,shine1,emask1, U=2,V=2)
merge(result1)

### SOFTEDGE ###
blr=expand().BinomialBlur(varY=4.5, varC=6.5, Y=3, U=2, V=2, useMMX=true)
vid = last
shine = vid.Overlay(blr, mode="blend", opacity=0.9)
emask = vid.mt_edge(mode="prewitt",thy1=0,thy2=255).expand().expand().expand().fastgaussblur(20)
result = mt_merge(vid,shine,emask, U=2,V=2)
return(result)

covertfromstacked(bits=12)

Thank you.

*EDIT* I also tried the Gradfun3() in place of gradfun2db but it says: Avisynth open failure:
Script error: There is no function named 'mt_expand_multi'...if this makes it any clearer.

[real.finder]

those not part of avisynth.dll but they are independent plugins that come with avisynthplus setup

Gradfun3 and mt_expand_multi are part of dither pack http://avisynth.nl/index.php/Dither_tools

[anton_foy]

Sorry I forget to mention that i already have the dither.dll and the dither.avsi and the mt_xxpand_multi.avsi loaded.
*edit* my bad with the mt_xxpand_multi.avsi at least gradfun3() is now working but still not the converttostacked()

[hello_hello]

Is convertstacked.dll loaded? I assume the installer puts it in the plugins folder, but did you download the "no-installer" version of Avisynth+?

[anton_foy]

Thanks but I thought this was an internal Avisynth+ filter?
http://avisynth.nl/index.php/ConvertStacked

Where is the link for convertstacked.dll?

*EDIT* I think I used the exe-file, installed it a few weeks ago but did not use it until now.

[anton_foy]

Aha, it seems that I had plugins in two places, one folder in "program files" and one in "avisynthrepository" AddAutoloadDir() refered to plug ins in the wrong place.
Thanks now it works to run it BUT I get the image all green striped and stretched out horisontally with this code:

SetFilterMTMode("DEFAULT_MT_MODE", 2)
FFmpegSource2("F:\video.mp4")
SetMemoryMax(2048)

ConvertBits(16, truerange=false)
ConvertToDoubleWidth()
###Levels###
Levels(0, 1, 255, 0, 235, coring=false, dither=false)

### DENOISE ###
fft3dfilter(sigma=1.5, bt=5, bw=32, bh=32, ow=16, oh=16, sharpen=0.0, plane=3).removegrain()
gradfun2db(0.4)

### TOPSHARP ###
shrp=sharpen(1.0)
vid1 = last
shine1 = vid1.Overlay(shrp, mode="blend", opacity=1.0)
emask1 = vid1.mt_edge(mode="prewitt",thy1=0,thy2=255).mt_expand().fastgaussblur(10)
result1 = mt_merge(vid1,shine1,emask1, U=2,V=2)
merge(result1)

### SOFTEDGE ###
blr=mt_expand().BinomialBlur(varY=4.5, varC=6.5, Y=3, U=2, V=2, useMMX=true)
vid = last
shine = vid.Overlay(blr, mode="blend", opacity=0.9)
emask = vid.mt_edge(mode="prewitt",thy1=0,thy2=255).mt_expand().mt_expand().mt_expand().fastgaussblur(20)
result = mt_merge(vid,shine,emask, U=2,V=2)
return(result)

ConvertFromDoubleWidth()
Prefetch(4)


And with this below I get the video with added green part on the bottom as if the "ConvertFromStacked" does not work?

SetFilterMTMode("DEFAULT_MT_MODE", 2)
FFmpegSource2("F:\video.mp4")
SetMemoryMax(2048)

ConvertBits(16, truerange=false)
ConvertToStacked()
###Levels###
Levels(0, 1, 255, 0, 235, coring=false, dither=false)

### DENOISE ###
fft3dfilter(sigma=1.5, bt=5, bw=32, bh=32, ow=16, oh=16, sharpen=0.0, plane=3).removegrain()
gradfun2db(0.4)

### TOPSHARP ###
shrp=sharpen(1.0)
vid1 = last
shine1 = vid1.Overlay(shrp, mode="blend", opacity=1.0)
emask1 = vid1.mt_edge(mode="prewitt",thy1=0,thy2=255).mt_expand().fastgaussblur(10)
result1 = mt_merge(vid1,shine1,emask1, U=2,V=2)
merge(result1)

### SOFTEDGE ###
blr=mt_expand().BinomialBlur(varY=4.5, varC=6.5, Y=3, U=2, V=2, useMMX=true)
vid = last
shine = vid.Overlay(blr, mode="blend", opacity=0.9)
emask = vid.mt_edge(mode="prewitt",thy1=0,thy2=255).mt_expand().mt_expand().mt_expand().fastgaussblur(20)
result = mt_merge(vid,shine,emask, U=2,V=2)
return(result)

ConvertFromStacked()
Prefetch(4)


Anyone knows why?

[FranceBB]

now it works to run it BUT I get the image all green striped and stretched out horizontally.

Anyone knows why?

Of course it is!
You're working in 16bit interleaved without telling filters that you're in 16bit interleaved!

After indexing, you're bringing everything to 16bit planar, then to 16bit interleaved (MSB and LSB interleaved one next to the other, aka DoubleWidth) and then you're using filters that expect a planar input, so that's no good.
Besides, I don't understand a few things:


Levels(0, 1, 255, 0, 235, coring=false, dither=false)


What are you trying to do? Map a Full Range into a Limited Range leaving blacks as they are and soft-clipping highlights only? Is it what you're trying to do?



fft3dfilter(sigma=1.5, bt=5, bw=32, bh=32, ow=16, oh=16, sharpen=0.0, plane=3)

The old FFT3D works in 8bit planar only, you might wanna take a look at Neo_FFT3D which works in 8/10/12/14/16bit planar: https://forum.doom9.org/showthread.php?t=181294
You're feeding it with 16bit interleaved which is wrong.



shrp=sharpen(1.0)
vid1 = last
shine1 = vid1.Overlay(shrp, mode="blend", opacity=1.0)
emask1 = vid1.mt_edge(mode="prewitt",thy1=0,thy2=255).mt_expand().fastgaussblur(10)
result1 = mt_merge(vid1,shine1,emask1, U=2,V=2)
merge(result1)
blr=mt_expand().BinomialBlur(varY=4.5, varC=6.5, Y=3, U=2, V=2, useMMX=true)
vid = last
shine = vid.Overlay(blr, mode="blend", opacity=0.9)
emask = vid.mt_edge(mode="prewitt",thy1=0,thy2=255).mt_expand().mt_expand().mt_expand().fastgaussblur(20)
result = mt_merge(vid,shine,emask, U=2,V=2)
return(result)

I gotta search for FastGaussBlur 'cause I'm not sure, but the rest of it (Sharpen and MaskTools) work just fine in 16bit planar. Again, you're feeding it with 16bit interleaved input which is wrong.
My suggestion is: go planar, use 16bit planar and you're probably gonna be fine with it. Don't mess around with 16bit Interleaved (or 16bit stacked) unless a specific filter requires it 'cause it can't work in planar. Besides, remember that while 16bit planar was introduced later, it's reported by Avisynth all the time so that filters are aware of it, but if you're working in 16bit stacked or 16bit interleaved, they'll both just be reported as 8bit planar to filters as they don't carry any metadata, they're literally 8bit MSB and 8bit LSB either stacked one on top of the other (16bit stacked aka double Height) or interleaved one next to the other (16bit interleaved, aka Double Width).


And with this below I get the video with added green part on the bottom as if the "ConvertFromStacked" does not work?


Same as 16bit interleaved, you're feeding 16bit interleaved to plugins that are expecting planar.



Side note: every time people come here confused about 16bit stacked, 16bit interleaved and 16bit planar I feel like we didn't make a very good job in the documentation/wiki... Perhaps I should write more about these... If only I had time... :(

[anton_foy]

@FranceBB

What are you trying to do? Map a Full Range into a Limited Range leaving blacks as they are and soft-clipping highlights only? Is it what you're trying to do?
I was putting levels in this way because before I switched to Avisynth+ it clipped my highlights if I used Levels(0, 1, 255, 0, 255, coring=false) and lost lots of information. But now when you tell me this I tried it and avs+ does not seem to clip it. Great!

Also I replaced FFmpegSource2 with LSMASHVideoSource now.

The old FFT3D works in 8bit planar only, you might wanna take a look at Neo_FFT3D which works in 8/10/12/14/16bit planar: https://forum.doom9.org/showthread.php?t=181294
You're feeding it with 16bit interleaved which is wrong.

Thanks for the recommendation with Neo_FFT3D. I tried KNLMEANSCL also and it works better (and faster) than the old FFT3D did.

Same as 16bit interleaved, you're feeding 16bit interleaved to plugins that are expecting planar.
Thanks, Im very newbie with these formats/colorspaces. Should I do it like this? LSMASHVideoSource("F:\video.mp4").converttoyv24()
At least it did not give me any error when I tried it now, so I guess fastgaussblur is okay.
Although when I export to Prores 4444 (Using Vdub) it is very slow but the result is worse than before (banding and creeping blotches)...hmm.

[FranceBB]

Uhm... Converttoyv24() isn't gonna help as it's 4:4:4 planar 8bit, while you wanted to use High Bit Depth.
I think this should work:

#Indexing
video=LWLibavVideoSource("F:\video.mp4")
audio=LWLibavAudioSource("F:\video.mp4")
AudioDub(video, audio)

#Loudness Correction
ResampleAudio(48000)
Normalize(0.22, show=false)

#Bringing everything to 16bit planar
ConvertBits(16)

#Soft-clipping highlights to Limited TV Range 0.7V
Levels(0, 1, 65280, 0, 60160, coring=false, dither=false)

#Clipping with 16bit planar precision
Limiter(min_luma=4096, max_luma=60160, min_chroma=4096, max_chroma=60160)

#Denoise with 16bit planar precision
neo_FFT3D(sigma=1.5, bt=5, y=3, u=3, v=3)

#Sharpening
shrp=sharpen(1.0)
vid1 = last
shine1 = vid1.Overlay(shrp, mode="blend", opacity=1.0)
emask1 = vid1.mt_edge(mode="prewitt",thy1=0,thy2=255).mt_expand().fastgaussblur(10)
result1 = mt_merge(vid1,shine1,emask1, U=2,V=2)
merge(result1)

#Soft-Edge
blr=mt_expand().BinomialBlur(varY=4.5, varC=6.5, Y=3, U=2, V=2, useMMX=true)
vid = last
shine = vid.Overlay(blr, mode="blend", opacity=0.9)
emask = vid.mt_edge(mode="prewitt",thy1=0,thy2=255).mt_expand().mt_expand().mt_expand().fastgaussblur(20)
result = mt_merge(vid,shine,emask, U=2,V=2)
return(result)

#Chroma Upscaling
ConverttoYUV444()

This should output a 16bit 4:4:4 planar and do all calculations in 16bit. If you wanna feed your AppleProRes 4444 XQ Encoder (I assume FFMpeg) with 10bit planar, you can just add:

#Dithering down to 10bit planar with the Floyd-Steinberg error Diffusion
ConvertBits(10, dither=1)

If you still have banding, before "Chroma Upscaling" and "Dithering" you can add:


neo_f3kdb(range=15, Y=45, Cb=30, Cr=30, grainY=0, grainC=0, sample_mode=2, blur_first=true, dynamic_grain=false, mt=true, keep_tv_range=true)


which also works in 16bit planar.


As a side historical note, f3kdb can work in 16bit stacked and interleaved as well, like:


#16bit stacked
ConverttoStacked()
f3kdb(range=15, Y=45, Cb=30, Cr=30, grainY=0, grainC=0, sample_mode=2, blur_first=true, dynamic_grain=false, opt=3, mt=true, keep_tv_range=true, input_mode=1, input_depth=16, output_mode=1, output_depth=16)
ConvertFromStacked()


#16bit interleaved
ConverttoDoubleWidth()
f3kdb(range=15, Y=45, Cb=30, Cr=30, grainY=0, grainC=0, sample_mode=2, blur_first=true, dynamic_grain=false, opt=3, mt=true, keep_tv_range=true, input_mode=2, input_depth=16, output_mode=2, output_depth=16)
ConvertFromDoubleWidth()


Anyway, let me know if the first script I posted works fine for you. (I didn't test it).

p.s since you mentioned, AppleProRes, I assume you're gonna send it to a studio, so I added a Safe Color Limiter to TV Range after the levels you wanted to use to soft clip highlights and a simple Loudness Correction to -24LUFS for the audio, although you can do loudness correction on ffmpeg as well with something like: -c:a pcm_s24le -ar 48000 -af loudnorm=I=-24:LRA=1:tp=-2 . ;)

[anton_foy]

@FranceBB: Many thanks for this!

Sadly it gives me an error:
Avisynth open failure: Script error: Invalid arguments to function 'ConvertBits'.
Although I modified the script, removed the audio part since I only need to use the image.

I also tried to remove video= from the indexing part but then I get this error:
Couldn't locate decompressor for format 'P016' (unknown)
VirtualDub requires a Video for Windows (VFW) compatible codec
to decompress video. DirectShow codecs, such as those used by
Windows Media Player, are not suitable.
I have FFMPEG and all the codecs installed though.

I tried another version of VirtualDub and it loads only a grey image and an error down below: Error reading source frame 0: Avisynth read error: CAVIStreamSynth
:System exception -Access Violation at 0x04A8A0C9


p.s since you mentioned, AppleProRes, I assume you're gonna send it to a studio, so I added a Safe Color Limiter to TV Range after the levels you wanted to use to soft clip highlights and a simple Loudness Correction to -24LUFS for the audio, although you can do loudness correction on ffmpeg as well with something like: -c:a pcm_s24le -ar 48000 -af loudnorm=I=-24:LRA=1:tp=-2 .

I am going to color grade it in Davinci Resolve and do some hard transformations with 3D LUTs, that is why I need to clean it up from noise and especially in the color channels. It is my own footage for a short film shot on Sony A6300 (3840x2160 resolution) :)


EDIT: I tried again and I unchecked "Use OpenGL" in VDub's Preferences->Display and now it loads the script but the right half of the image is green and VDub gets an error: "out of memory" if I step to the next frame. If I use "SetFilterMTMode("DEFAULT_MT_MODE", 2)" and lastly "Prefetch(4)" VDub crashes instantly when loading.

[FranceBB]

@FranceBB: Many thanks for this!

Sadly it gives me an error:
Avisynth open failure: Script error: Invalid arguments to function 'ConvertBits'.



Weird. You can try with:

ConvertBits(bits=10, dither=1)

It should work.


Although I modified the script, removed the audio part since I only need to use the image.

I also tried to remove video= from the indexing part but then I get this error:
Couldn't locate decompressor for format 'P016' (unknown)
VirtualDub requires a Video for Windows (VFW) compatible codec
to decompress video. DirectShow codecs, such as those used by
Windows Media Player, are not suitable.
I have FFMPEG and all the codecs installed though.
I tried another version of VirtualDub and it loads only a grey image and an error down below: Error reading source frame 0: Avisynth read error: CAVIStreamSynth
:System exception -Access Violation at 0x04A8A0C9


I see! It must be the RAM usage!
I bet you're using Avisynth+ x86, just like me.
By the way, I strongly suggest you to preview things using AVSPmod mod by Gispos: https://forum.doom9.org/showthread.php?t=175823

In this case, since all you need is video and you're working with 4K footages, I'll strongly suggest MPPipeline. This way, you'll be able to allocate up to 4GB per process (instead of 4GB in the whole frameserver), so you'll be able to get it working.

Here's the updated script:



MP_Pipeline("""
#Indexing
FFVideoSource("F:\video.mp4")

### platform: win32
### ###

#Bringing everything to 16bit planar
ConvertBits(16)

### platform: win32
### ###

#Soft-clipping highlights to Limited TV Range 0.7V
Levels(0, 1, 65280, 0, 60160, coring=false, dither=false)

### platform: win32
### ###

#Clipping with 16bit planar precision
Limiter(min_luma=4096, max_luma=60160, min_chroma=4096, max_chroma=60160)

### platform: win32
### ###

#Denoise with 16bit planar precision
neo_FFT3D(sigma=1.5, bt=5, y=3, u=3, v=3)

### platform: win32
### ###

#Sharpening
shrp=sharpen(1.0)
vid1 = last
shine1 = vid1.Overlay(shrp, mode="blend", opacity=1.0)
emask1 = vid1.mt_edge(mode="prewitt",thy1=0,thy2=60160).mt_expand().fastgaussblur(10)
result1 = mt_merge(vid1,shine1,emask1, U=2,V=2)
merge(result1)

#Soft-Edge
blr=mt_expand().BinomialBlur(varY=4.5, varC=6.5, Y=3, U=2, V=2, useMMX=true)
vid = last
shine = vid.Overlay(blr, mode="blend", opacity=0.9)
emask = vid.mt_edge(mode="prewitt",thy1=0,thy2=60160).mt_expand().mt_expand().mt_expand().fastgaussblur(20)
mt_merge(vid,shine,emask, U=2,V=2)

### platform: win32
### ###

#Debanding
neo_f3kdb(range=15, Y=45, Cb=30, Cr=30, grainY=0, grainC=0, sample_mode=2, blur_first=true, dynamic_grain=false, mt=true, keep_tv_range=true)

### platform: win32
### ###

#Chroma Upscaling
ConverttoYUV444()

### platform: win32
### ###

ConvertBits(bits=10, dither=1)

""")




But... there's a problem.
Although I changed the values for your MT calls to respect 16bit (namely from thy2=255 to thy2=60160) I keep getting weird results like:

https://i.imgur.com/TeAWtpu.png


Perhaps, someone who's better than me with MaskTools can chip in and fix the code.
The result above is by faking the input with color bars like:


ColorBars(848, 480, pixel_type="YV12")



So something is clearly wrong in my "scaled" values of MaskTools.




I am going to color grade it in Davinci Resolve and do some hard transformations with 3D LUTs, that is why I need to clean it up from noise and especially in the color channels. It is my own footage for a short film shot on Sony A6300 (3840x2160 resolution) :)

Ah, ok, got it. Out of curiosity, why you didn't shoot in SLog? I always shoot (or expect cameraman with a Sony to shoot) in either Slog3 or HLG to get as many nits as possible out of the camera stops. For what it's worth, I always found the biggest limitation of Sony cameras the fact that they're limited to 30fps in 4K and H.264 8bit... :(

[anton_foy]

So something is clearly wrong in my "scaled" values of MaskTools.
It works for me (in 8-bit at least), I did not try it with colorbars though but on my footage it looks like a fine extra shine / glow without any weird artifacts.
I use the newest Masktools2.

Ah, ok, got it. Out of curiosity, why you didn't shoot in SLog? I always shoot (or expect cameraman with a Sony to shoot) in either Slog3 or HLG to get as many nits as possible out of the camera stops. For what it's worth, I always found the biggest limitation of Sony cameras the fact that they're limited to 30fps in 4K and H.264 8bit...
I did shoot in SLog2 (SLog3 in 8-bit footage turns out horrible with banding and compression artifacts) so I want to use levels full range and now it seems to work.
On my A6300 I always shoot SLog2, 25fps in 4K :) When filtered in Avisynth it looks better compared to the Sony FS7 and even Arri mini (although 3.2K) in regards of resolution and natural sharpness and noise. And I work as a colorist on a daily basis with footage from these cameras. As for the 3D Luts in Resolve I use my own made 3D Luts to simulate various 35mm film stocks so I need as much image info and as clean footage as possible. They are matched perfectly to the color and SLog settings I use on my camera though.

In this case, since all you need is video and you're working with 4K footages, I'll strongly suggest MPPipeline. This way, you'll be able to allocate up to 4GB per process (instead of 4GB in the whole frameserver), so you'll be able to get it working.
Awesome if I can get this to work! I tried it with the AvsMod mod you linked but it gives me an error: MP_Pipeline: Unable to create slave process. Message: Script error: Script error: Invalid arguments to function 'MP_PrepareDownStreamClip'
Is it suppose to work with this version of AvsMod only or does it work with VDub aswell?

Thanks!

EDIT:
Aha I found out that I had "mcdegrain" left in my script and when I remove it the green half of the image disappear!
Binomialblur is the one f***ing up the blur on the colorbars, not masktools. When I replaced it with blur(1.58) x7 it looks almost as I want it. I would like to find a replacement for "mcdegrain" though.

[FranceBB]

I did shoot in SLog2 (SLog3 in 8-bit footage turns out horrible with banding and compression artifacts) so I want to use levels full range and now it seems to work.
On my A6300 I always shoot SLog2, 25fps in 4K :) When filtered in Avisynth it looks better compared to the Sony FS7 and even Arri mini (although 3.2K) in regards of resolution and natural sharpness and noise. And I work as a colorist on a daily basis with footage from these cameras. As for the 3D Luts in Resolve I use my own made 3D Luts to simulate various 35mm film stocks so I need as much image info and as clean footage as possible. They are matched perfectly to the color and SLog settings I use on my camera though.


Ah, I see. Fair enough then. :)



Aha I found out that I had "mcdegrain" left in my script and when I remove it the green half of the image disappear!
Binomialblur is the one f***ing up the blur on the colorbars, not masktools. When I replaced it with blur(1.58) x7 it looks almost as I want it. I would like to find a replacement for "mcdegrain" though.

Ah, great!
If you're looking for a 16bit planar temporal degrain, you can use something like:

super = MSuper(pel=2, sharp=1)
bv1 = MAnalyse(super, isb = true, delta = 1, overlap=4)
fv1 = MAnalyse(super, isb = false, delta = 1, overlap=4)
bv2 = MAnalyse(super, isb = true, delta = 2, overlap=4)
fv2 = MAnalyse(super, isb = false, delta = 2, overlap=4)
MDegrain2(super,bv1,fv1,bv2,fv2,thSADC=400, thSAD=400)

and tweak thSADC and thSAD according to your needs. (Raising 'em will get rid of more grain of course). As to the noise, I generally use DFTTest (which comes with the neo_DFTTest variant as well).

[anton_foy]

Thanks FranceBB! You helped me to get it into 16bits all the way, im very excited about this :D

If you're looking for a 16bit planar temporal degrain, you can use something like:
This code gives me a memory error: Out of memory. I must get that MP_Pipeline to work I guess...

As to the noise, I generally use DFTTest (which comes with the neo_DFTTest variant as well).
Do you think neo_DFTTest is a good choice for the Sony footage? I mean the biggest problem for me is the unclean blocky and noisy color channels while the luma do not need as much denoising (well sometimes in low light scenes).

I found that these lines below worked quite well to clean up the chroma but I have not yet tested more than two clips.
U = UToY8().vsTBilateral(diameterY=7, sdevY = 9.2, idevY = 9.0).vsttempsmooth(maxr=6,ythresh=150, strength=6)
V = VToY8().vsttempsmooth(maxr=6,ythresh=150, strength=6)
YToUV(U,V,last)