Porting SMDegrain to Native High Bit Depth

[MysteryX]

With Avisynth+ now supporting native high-bit-depth, and MaskTools2 and MvTools2 also supporting high-bit-depth, is anyone interested in modifying SMDegrain to work with full native high-bit-depth?

In the best world, you could simply call SMDegrain with a 16-bit native clip and it would process it all in HBD. In practice, it returns errors. Some side libraries (that probably have HBD replacements) haven't been ported to HBD yet, and a few minor issues can happen along the way.

It shouldn't be very hard but someone's got to do it, sooner or later.

I'll create this thread for those who want to chime in, and let's see where it goes from here.

[ChaosKing]

I think real.finder did that already? https://pastebin.com/Mz5taAgN

masktools 2.2.5 or up now required in avs26 and avs+ now work basically with new high bit depth in avs+

His Avisynth Stuff
https://forum.doom9.org/showthread.php?t=174121

[dipje]

I know vapoursynth is not avisynth but vapoursynth has a ported smdegrain script with high bitdepth in havsfunc. (I don't know if it's a direct port or takes some liberties ).
Could at least be useful to take a peek how it works in vpy .

[MysteryX]

I get

MDegrain2 : no valid DEGRAINLUMA function for 16x16, pixelsize=2, lsb_flag=1, level=2

[real.finder]

work fine for me, seems you have old mvtools2 and you try use lsb

[tormento]

Is there any convenience (and way) to process 8 bit video in 16 bit space and back?

[real.finder]

Is there any convenience (and way) to process 8 bit video in 16 bit space and back?


convertbits(16)
filter1
filter2
filter3
convertbits(8) #without dither, you can set the setting to do dither too

[MysteryX]

you try use lsb
Duh!

Works now... but performance isn't good at all!

8-bit

FPS (min | max | average): 1.874 | 101671 | 18.11
Memory usage (phys | virt): 757 | 775 MiB
Thread count: 34
CPU usage (average): 77%

16-bit

FPS (min | max | average): 0.288 | 47.37 | 3.171
Memory usage (phys | virt): 1059 | 1095 MiB
Thread count: 32
CPU usage (average): 76%

Something is wrong here.

[tormento]

convertbits(16)
filter1
filter2
filter3
convertbits(8) #without dither, you can set the setting to do dither too



How have I to set lsb, lsb_in, lsb_out and mode to let use SMDegrainMOD in full 16 bit mode and leave Avisynth to dither with convertbits(8)?
Is there a way to make SMDegrainMOD work in 16 bit AND output a 8 bit video?

[real.finder]

How have I to set lsb, lsb_in, lsb_out and mode to let use SMDegrainMOD in full 16 bit mode and leave Avisynth to dither with convertbits(8)?
[/LIST]

lsb=hack, just forget it in avs+ HBD

use the lines I post last time will do everything in full 16 bit


Is there a way to make SMDegrainMOD work in 16 bit AND output a 8 bit video?


yes with add another parameter to do that but why?! it make no sense, stay with lsb will be better and faster

[tormento]

use the lines I post last time will do everything in full 16 bit
Has mode to be set to -1?

[real.finder]

Has mode to be set to -1?

mode in SMDegrain? will do nothing without lsb things

convertbits() in avs+ has dither modes, you can set that when you back to 8 bit

[tormento]

mode in SMDegrain? will do nothing without lsb things

convertbits() in avs+ has dither modes, you can set that when you back to 8 bit

My default parameters are
SMDegrain (tr=4,PreFilter=4,thSAD=400,contrasharp=false,refinemotion=false,truemotion=true,plane=4,chroma=true,lsb=true,mode=0)
So I have to modify them as
SMDegrain (tr=4,PreFilter=4,thSAD=400,contrasharp=false,refinemotion=false,truemotion=true,plane=4,chroma=true)
right?

[real.finder]

My default parameters are
SMDegrain (tr=4,PreFilter=4,thSAD=400,contrasharp=false,refinemotion=false,truemotion=true,plane=4,chroma=true,lsb=true,mode=0)
So I have to modify them as
SMDegrain (tr=4,PreFilter=4,thSAD=400,contrasharp=false,refinemotion=false,truemotion=true,plane=4,chroma=true)
right?

yes, but even if you leave mode=0 it will do nothing without lsb

and set dither=0 in convertbits() will give you dither like mode=0

[tormento]

SetMemoryMax(8000)
SetFilterMTMode("DEFAULT_MT_MODE", 2)
SetFilterMTMode("ChangeFPS", 3)
SetFilterMTMode("DGSource", 3)
LoadPlugin("D:\eseguibili\media\DGDecNV\x64\DGDecodeNV.dll")
DGSource("E:\in\1_55 founder, The\founder.dgi")
CompTest(1)
ChangeFPS(last,last,true)
SMDegrain (tr=4,PreFilter=4,thSAD=400,contrasharp=false,refinemotion=false,truemotion=true,plane=4,chroma=true,lsb=true,mode=0)
Prefetch(6)
[2017-05-05][21:58:43] encoded 1680 frames, 12.38 fps, 3004.32 kb/s, duration 0:02:15.65
[2017-05-05][21:58:43] av2y [info]: E:\in\1_55FO~1\founde~1.avs: 1920x800, YV12, 8-bits, progressive, 24 fps, 1680 frames
[2017-05-05][21:58:43] Final file size is 25.1 MB bytes.

SetMemoryMax(8000)
SetFilterMTMode("DEFAULT_MT_MODE", 2)
SetFilterMTMode("ChangeFPS", 3)
SetFilterMTMode("DGSource", 3)
LoadPlugin("D:\eseguibili\media\DGDecNV\x64\DGDecodeNV.dll")
DGSource("E:\in\1_55 founder, The\founder.dgi")
CompTest(1)
ChangeFPS(last,last,true)
convertbits(16)
SMDegrain (tr=4,PreFilter=4,thSAD=400,contrasharp=false,refinemotion=false,truemotion=true,plane=4,chroma=true)
convertbits(8)
Prefetch(6)
[2017-05-05][22:01:59] encoded 1680 frames, 9.76 fps, 3343.63 kb/s, duration 0:02:52.06
[2017-05-05][22:01:59] av2y [info]: E:\in\1_55FO~1\founde~2.avs: 1920x800, YV12, 8-bits, progressive, 24 fps, 1680 frames
[2017-05-05][22:01:59] Final file size is 27.9 MB bytes.

SetMemoryMax(8000)
SetFilterMTMode("DEFAULT_MT_MODE", 2)
SetFilterMTMode("ChangeFPS", 3)
SetFilterMTMode("DGSource", 3)
LoadPlugin("D:\eseguibili\media\DGDecNV\x64\DGDecodeNV.dll")
DGSource("E:\in\1_55 founder, The\founder.dgi")
CompTest(1)
ChangeFPS(last,last,true)
convertbits(16)
SMDegrain (tr=4,PreFilter=4,thSAD=400,contrasharp=false,refinemotion=false,truemotion=true,plane=4,chroma=true)
convertbits(8,dither=0)
Prefetch(6)
[2017-05-05][22:06:02] encoded 1680 frames, 9.70 fps, 3164.08 kb/s, duration 0:02:53.15
[2017-05-05][22:06:02] av2y [info]: E:\in\1_55FO~1\founde~3.avs: 1920x800, YV12, 8-bits, progressive, 24 fps, 1680 frames
[2017-05-05][22:06:02] Final file size is 26.4 MB bytes.

SetMemoryMax(8000)
SetFilterMTMode("DEFAULT_MT_MODE", 2)
SetFilterMTMode("ChangeFPS", 3)
SetFilterMTMode("DGSource", 3)
LoadPlugin("D:\eseguibili\media\DGDecNV\x64\DGDecodeNV.dll")
DGSource("E:\in\1_55 founder, The\founder.dgi")
CompTest(1)
ChangeFPS(last,last,true)
convertbits(16)
SMDegrain (tr=4,PreFilter=4,thSAD=400,contrasharp=false,refinemotion=false,truemotion=true,plane=4,chroma=true)
convertbits(8,dither=1)
Prefetch(6)
[2017-05-05][22:09:02] encoded 1680 frames, 9.62 fps, 3132.65 kb/s, duration 0:02:54.63
[2017-05-05][22:09:02] av2y [info]: E:\in\1_55FO~1\founde~4.avs: 1920x800, YV12, 8-bits, progressive, 24 fps, 1680 frames
[2017-05-05][22:09:02] Final file size is 26.2 MB bytes.

Regarding plain bitrate, 8 bit + lsb is clearly a winner, both speed and final size.

Regarding visual quality, need time to decide :D

[tormento]

The Silence of the Lambs BD

SMDegrain (tr=4,PreFilter=4,thSAD=400,contrasharp=false,refinemotion=false,truemotion=true,plane=4,chroma=true,lsb=true,mode=0)
[2017-05-05][22:23:48] encoded 1708 frames, 9.44 fps, 4109.46 kb/s, duration 0:03:00.98
[2017-05-05][22:23:48] av2y [info]: E:\in\1_58SI~1\silenz~1.avs: 1920x1032, YV12, 8-bits, progressive, 24000/1001 fps, 1708 frames
[2017-05-05][22:23:48] Final file size is 34.9 MB bytes.

convertbits(16)
SMDegrain (tr=4,PreFilter=4,thSAD=400,contrasharp=false,refinemotion=false,truemotion=true,plane=4,chroma=true)
convertbits(8,dither=1)
[2017-05-05][22:29:18] encoded 1708 frames, 6.30 fps, 6812.83 kb/s, duration 0:04:31.24
[2017-05-05][22:29:18] av2y [info]: E:\in\1_58SI~1\silenz~2.avs: 1920x1032, YV12, 8-bits, progressive, 24000/1001 fps, 1708 frames
[2017-05-05][22:29:18] Final file size is 57.9 MB bytes.

SMDegrain (tr=6,PreFilter=4,thSAD=600,contrasharp=false,refinemotion=false,truemotion=true,plane=4,chroma=true,lsb=true,mode=0)
[2017-05-05][22:34:43] encoded 1708 frames, 7.59 fps, 3702.49 kb/s, duration 0:03:44.95
[2017-05-05][22:34:43] av2y [info]: E:\in\1_58SI~1\silenz~3.avs: 1920x1032, YV12, 8-bits, progressive, 24000/1001 fps, 1708 frames
[2017-05-05][22:34:43] Final file size is 31.5 MB bytes.

convertbits(16)
SMDegrain (tr=6,PreFilter=4,thSAD=600,contrasharp=false,refinemotion=false,truemotion=true,plane=4,chroma=true)
convertbits(8,dither=1)
[2017-05-05][22:40:39] encoded 1708 frames, 4.88 fps, 6839.06 kb/s, duration 0:05:49.72
[2017-05-05][22:40:39] av2y [info]: E:\in\1_58SI~1\silenz~4.avs: 1920x1032, YV12, 8-bits, progressive, 24000/1001 fps, 1708 frames
[2017-05-05][22:40:39] Final file size is 58.1 MB bytes.

Very strange results about bitrate difference between 8 bit and 16 bit noise reduction. Any idea?

[MysteryX]

The noise level has a large impact on the compress-ability of the video and output size.

[tormento]

The noise level has a large impact on the compress-ability of the video and output size.

Ok and why so much difference between processing in 8 or 16 bit?

[MysteryX]

OK I found the performance problem: MinBlur.

MinBlur(2,3,true)

8-bit

FPS (min | max | average): 5.066 | 146152 | 59.99
Memory usage (phys | virt): 155 | 153 MiB
Thread count: 29
CPU usage (average): 66%


16-bit

FPS (min | max | average): 0.961 | 66813 | 10.50
Memory usage (phys | virt): 222 | 220 MiB
Thread count: 29
CPU usage (average): 77%

Considering MedianBlur2's latest version was in 2014, it most certainly doesn't support native high-bit-depth, so I'm surprised it's not crashing, but it's probably not doing what we want.

[MysteryX]

real.finder, I believe you said MinBlur was only performed on 8-bit-depth?


input8h = defined(input8h) && isclip(input8h) ? input8h : lsb_in ? inputP. Ditherpost(mode=6, slice=slices) : nop()
input8y = defined(input8y) ? input8y : planar ? (lsb_in ? input8h.Dither_YUY2toInterleaved() : inputP) : inputP
input8 = defined(input8 ) ? input8 : lsb_in ? (planar ? input8y.Interleaved2planar() : input8h) : input8y


This converts to 8-bit only when using lsb, and native 16-bit remains in 16-bit. Call SMDegrain_prefilters.Info() in 16-bit and the result is corrupt with vertical bars.

[real.finder]

real.finder, I believe you said MinBlur was only performed on 8-bit-depth?


input8h = defined(input8h) && isclip(input8h) ? input8h : lsb_in ? inputP. Ditherpost(mode=6, slice=slices) : nop()
input8y = defined(input8y) ? input8y : planar ? (lsb_in ? input8h.Dither_YUY2toInterleaved() : inputP) : inputP
input8 = defined(input8 ) ? input8 : lsb_in ? (planar ? input8y.Interleaved2planar() : input8h) : input8y


This converts to 8-bit only when using lsb, and native 16-bit remains in 16-bit. Call SMDegrain_prefilters.Info() in 16-bit and the result is corrupt with vertical bars.

I didn't say that before, I know that, but I said that MinBlur is not always needed

[MysteryX]

ok, but the script with prefilter is broken for now; but it shouldn't be hard to fix if it only needs to be dithered back to 8-bit