Avisynth woes - ffmpeg hanging, denoising

[ENunn]

I'm having issues getting my script encoded. When I start encoding with ffmpeg, I get around 6fps. Could be better but, eh. I left it on overnight, the speed has now dropped to 0.8fps and has hanged. No CPU or GPU is being used. I've been plagued by this issue for months.

I thought the initial fix was to add setmemorymax(63448) to the top of my script but now it's not helping. I don't know what's causing it. I know my script is a bit heavy, but I can wait it out at 6fps, not when it's frozen.

Does anyone know how to fix the issue?

Script (Source file is FFV1):

setmemorymax(63448)
v = lwlibavvideosource("i:\virtualdub\tape transfers\angelcam g4 lot\tape 37.mkv", fpsnum=30000, fpsden=1001) #atrack=0, 
a = lwlibavaudioSource("g:\recordings\angelcam tape 37.w64")
AudioDub(v,a)
delayaudio(-0.150)
getleftchannel()
assumetff().converttoyuv422(matrix="rec601", interlaced=true).convertbits(10)
Levels(20, 1, 1045, 0, 1023, coring=false,dither=true).tweak(hue=-3, coring=false, dither=true).convertbits(8)
#turnRight().Histogram().TurnLeft()
Trim(68, 615963)
Trim(54, 81752)
normalize(0.8912)
CCD(interlaced=true, u=10, v=1)
qtgmcp(Preset="slow", Tuning="DV-SD", EZKeepGrain=1.0, sourcematch=3, tr0=2, tr1=2, tr2=3, lossless=2, MatchEnhance=1, NoiseProcess=1, NoiseRestore=1.0, Sigma=2.0, SLMode=4, subpel=4)
MergeChroma(aWarpSharp2(depth=20,threads=4))MergeChroma(aWarpSharp2(depth=10,threads=4))
santiag(strv=0, nns=4, nsize=5, threads=4)
SMDegrain(str=0.8, refinemotion=true)
cropresize(indar=4.0/3.0, resizewo=true)
ConvertToYv24(matrix="rec601",interlaced=false)
Crop(8, 0, -5, -15)
SuperRes(matrixin="rec601", 2, .43, 0, """edi_rpow2(2, nns=4, cshift="Spline16Resize")""", matrixout="rec709")
x = float(width) / float(height)
spline36resize(round(x * 2164),2164)
converttoyv12(matrix="rec709", interlaced=false)
prefetch(6)

ffmpeg command:

-c:v hevc_nvenc -gpu any -profile:v rext -tier high -g 30 -rc vbr -cq 12 -qmin 12 -qmax 12 -b:v 0K -preset p7 -c:a copy

Specs:
ASUS Prime Z390-A
Intel(R) Core(TM) i7-8700K CPU @ 3.70GHz
Zotac NVIDIA GeForce RTX 2060 6 GB
G.SKILL TridentZ RGB Series 64GB DDR4 3600MHz
Samsung 960 EVO 500GB M.2
2x WD Black 4TB

I have also tired to use h264_nvenc to no avail. CPU encoding gives me with ~3fps x264 slow and ~0.9fps with x265 slow. Expected, but I want to take advantage of the full speed of the script, which is why I've been using NVENC. Final output looks fine to me.

[Boulder]

Do you happen to have avstp.dll anywhere in your path? If so, remove it.

[Reel.Deel]

@ENunn

No offense here but your script is doing some questionable things. Try to streamline the script a bit.

For example:

• If your source is YUV420 do not do any chroma subsampling conversions;
• MergeChroma(aWarpSharp2(depth=20,threads=4))MergeChroma(aWarpSharp2(depth=10,threads=4)) can be called like this aWarpSharp2(depth=20,threads=4,chroma=2).aWarpSharp2(depth=10,threads=4,chroma=2) and it might be better to disable internal multithreading since AVS+ is already doing it;
• Try to combine deinterlacing and denoising in one step with QTGMC;
• Odd that you need additional anti-aliasing after denoising;
• SuperRes with EEDI3 I bet is slow, try using NNEDI3 or maybe W2xncnnvk;
• This has no effect on speed but ConvertToXX(matrix="rec709") or any other matrix has no effect on YUV, only on YUV<->RGB conversions.

Do you have a short sample you can share?

[FranceBB]

Originally Posted by Boulder

Do you happen to have avstp.dll anywhere in your path? If so, remove it.

Boulder is right, go to your plugins folder and delete avstp.dll as this is almost definitely caused by this known bug: https://github.com/pinterf/mvtools/issues/46

[kedautinh12]

QTGMC() will be faster than qtgmcp() in 8 bit
https://github.com/realfinder/AVS-St...0up/QTGMC.avsi

[ENunn]

Originally Posted by Boulder

Do you happen to have avstp.dll anywhere in your path? If so, remove it.

Originally Posted by FranceBB

Boulder is right, go to your plugins folder and delete avstp.dll as this is almost definitely caused by this known bug: https://github.com/pinterf/mvtools/issues/46

It's been removed. Thanks for the heads up!

Originally Posted by Reel.Deel

@ENunn
If your source is YUV420 do not do any chroma subsampling conversions;

Source is YUV422, I convert to 10-bit to fix levels without any major banding, then convert to YV24 to do pixel-accurate cropping, then convert to YV12 for the final output. I don't switch back and forth between color spaces.

Originally Posted by Reel.Deel

MergeChroma(aWarpSharp2(depth=20,threads=4))MergeChroma(aWarpSharp2(depth=10,threads=4)) can be called like this aWarpSharp2(depth=20,threads=4,chroma=2).aWarpSharp2(depth=10,threads=4,chroma=2) and it might be better to disable internal multithreading since AVS+ is already doing it;

Removed any mention of threads, thanks. I did see a slight difference in changing the line, and I'm not sure if I prefer it.

MergeChroma(aWarpSharp2(depth=20,threads=4))MergeChroma(aWarpSharp2(depth=10,threads=4))

aWarpSharp2(depth=20,threads=4,chroma=2).aWarpSharp2(depth=10,threads=4,chroma=2)

Originally Posted by Reel.Deel

Try to combine deinterlacing and denoising in one step with QTGMC;

That could work, but I do prefer SMDegrain's denoising over QTGMC's.

Originally Posted by Reel.Deel

[*]Odd that you need additional anti-aliasing after denoising;

I don't use aWarpSharp for AA, I use it to fix chroma/color errors, and remove some streaks in my tape transfers.

Without MergeChroma(aWarpSharp2):


With MergeChroma(aWarpSharp2):


I use Santiag for AA. Looks so much better than without it.

Without Santiag:


With Santiag:

Originally Posted by Reel.Deel

SuperRes with EEDI3 I bet is slow, try using NNEDI3 or maybe W2xncnnvk;

Changed it to NNEDI3. Thanks!

Originally Posted by Reel.Deel

This has no effect on speed but ConvertToXX(matrix="rec709") or any other matrix has no effect on YUV, only on YUV<->RGB conversions.

Ahh, OK. Just wanted to make sure everything was converted properly from 601 to 709.

Originally Posted by Reel.Deel

Do you have a short sample you can share?

My source? I'll send as soon as I can (got work), you probably won't have any issue with a small clip of a 40 minute source.

Originally Posted by kedautinh12

QTGMC() will be faster than qtgmcp() in 8 bit
https://github.com/realfinder/AVS-St...0up/QTGMC.avsi

Interesting. I'll get that replaced.

Thanks for the optimizations everyone! Will speed also decrease if my source is using FFV1? I know it's pretty slow to decode (wish there was a GPU accelerated version out there :P)

Unfortunately these usually take about 7hrs a piece, so I won't know if it's properly encoded until then. I'll post back once this is all done.

[kedautinh12]

About Santiag for AA. I think your sample same blocking not AA. Try Deblock scrips like Deblock_QED combine with SM_Degrain
https://github.com/Dogway/Avisynth-S...ckPack.avsi#L9

Example:

# Remove blocking and smooth out jagged edges (only applies where motion match fails)
mb=Deblock_QED(quant1=30, quant2=40,UV=1)
mb=mb.ex_smooth(1, mode="SG", limit=true, sharp=true)

# LFR to recover some smearing. limits=false to don't sharpen motion areas
SMDegrain(6, 400, prefilter=6, mfilter=mb, ContraSharp=true, RefineMotion=true, plane=4, LFR=300, limits=false, DCTFlicker=false)

[Reel.Deel]

Originally Posted by ENunn

My source? I'll send as soon as I can (got work), you probably won't have any issue with a small clip of a 40 minute source.

I'm not trying to test the encoding time of a full clip, it's more to see what the actual output of your script looks like and if can be made faster by means of streamlining your script. Things start to become slow when you use many filters, albeit sometimes needed, but there are times when you can achieve the same results with less.

And maybe try running the script in AVSMeter with SetLogParams("stdout", 4) at the very beginning of your script. The logging facility in AVS+ will automatically log errors, and will issue warnings and notes about potential problems, buggy plugins, suboptimal settings, etc.

[ENunn]

Originally Posted by kedautinh12

About Santiag for AA. I think your sample same blocking not AA. Try Deblock scrips like Deblock_QED combine with SM_Degrain
https://github.com/Dogway/Avisynth-S...ckPack.avsi#L9

Example:

Not sure what I like better to be honest.

mb=Deblock_QED(quant1=30, quant2=40,UV=1)
mb=mb.ex_smooth(1, mode="SG", limit=true, sharp=true)
SMDegrain(6, 400, prefilter=6, mfilter=mb, ContraSharp=true, RefineMotion=true, plane=4, LFR=300, limits=false, DCTFlicker=false)

santiag(strv=0, nns=4, nsize=5).SMDegrain(str=0.8, refinemotion=true)

Originally Posted by Reel.Deel

I'm not trying to test the encoding time of a full clip, it's more to see what the actual output of your script looks like and if can be made faster by means of streamlining your script. Things start to become slow when you use many filters, albeit sometimes needed, but there are times when you can achieve the same results with less.

Ahhhh, OK. Here's a sample.

Originally Posted by Reel.Deel

And maybe try running the script in AVSMeter with SetLogParams("stdout", 4) at the very beginning of your script. The logging facility in AVS+ will automatically log errors, and will issue warnings and notes about potential problems, buggy plugins, suboptimal settings, etc.

Ran it, I pasted the log in a text file, that's attached.

I did a test encode after removing threads/avstp and changing QTGMCp to QTGMC, I'm now encoding at 9fps. Definitely better than 6.

[Reel.Deel]

Originally Posted by ENunn

Ran it, I pasted the log in a text file, that's attached.

I did a test encode after removing threads/avstp and changing QTGMCp to QTGMC, I'm now encoding at 9fps. Definitely better than 6.

3 fps increase, that's a start

Paste your log on pastebin.com or something similar. Attachments here take forever and a day to get approved.

[ENunn]

Originally Posted by Reel.Deel

Paste your log on pastebin.com or something similar. Attachments here take forever and a day to get approved.

Here you go.

[kedautinh12]

copy this scripts to file mtmodes.avsi and place this file in your plugins folders
https://publishwith.me/ep/pad/view/r...cdWn4k9/latest

[Reel.Deel]

@ENunn

I took a look at the sample, along with the script and to be honest I think some of your settings for QTGMC are way overkill for the quality of the source. The chroma with some bands and the luma has some type of pattern on it. I think with the settings you use for QTGMC accentuates those artifacts and thus needing to use santiag afterwards to mitigate them. Is your intent to keep the video noisy?

Does something like this seem acceptable to you? https://imgsli.com/MTM0MDkw

[ENunn]

Originally Posted by kedautinh12

copy this scripts to file mtmodes.avsi and place this file in your plugins folders
https://publishwith.me/ep/pad/view/r...cdWn4k9/latest

Looks like I already had this, I replaced it with the one you sent. Thanks.

Originally Posted by Reel.Deel

I took a look at the sample, along with the script and to be honest I think some of your settings for QTGMC are way overkill for the quality of the source. The chroma with some bands and the luma has some type of pattern on it. I think with the settings you use for QTGMC accentuates those artifacts and thus needing to use santiag afterwards to mitigate them. Is your intent to keep the video noisy?

Nah, not at all. I need to find a good denoising settings for my ES15 transfers. I've been using the same settings for all my transfers, and they've all looked fine except for most of my ES15 captures.

I have to change my VCR to edit mode because with the TBC off, there's visual errors with it set to norm. Sadly this introduces a lot of noise. Doesn't help that the source is a 2nd gen tape (1st gen is probably lost and taped over by now).

Originally Posted by Reel.Deel

Does something like this seem acceptable to you? https://imgsli.com/MTM0MDkw

Are those new settings or my script? Looks definitely better than what I saw with my script.

An update for encoding: it took five hours but it eventually completely rendered! Thanks for the help in getting my encodes to stop hanging!

[kedautinh12]

Add SelectEven() beside QTGMC() will be faster. Example: QTGMC(Preset="Slow", Sharpness=0).SelectEven() but may cause error in trans screen. Only suggestion Not test with your sample

[Reel.Deel]

Originally Posted by kedautinh12

Add SelectEven() beside QTGMC() will be faster.

Adding SelectEven() does not make QTGMC faster, the processing speed still remains the same for any processes after/before SelectEven(). It's only 'faster' because there is half the frames to process, not sure if that is what he wants.

Originally Posted by ENunn

Are those new settings or my script? Looks definitely better than what I saw with my script.

It's my own frankenstein script (borrowed snippets from a few other scripts), I can post it later if you want. Unless you're done with this clip.

Here's the script that I used in case you're interested. No cropping or upscaling there. The script is a modification of a script Didée wrote a long time ago: https://forum.doom9.org/showthread.p...22#post1107122

FFVideoSource("tape 37-trim.mkv")
src = last
BWDIF(field=-2)#QTGMC(preset="fast")
o = last

mds()
spline36resize(o.width(),o.height()/2/4*4)
mds()
spline36resize(o.width(),o.height()/6/4*4)
mds()
spline36resize(o.width(),o.height())

o.mergechroma(last)
nostripes2 = last.minblur(1)#LSFplus(ss_x=1.0,ss_y=1.0,strength=200,soft=100,overshoot=5)

srchbase   = nostripes2.fft3dfilter(sigma=6,sigma2=4,sigma3=3,sigma4=2,bt=5,bw=16,bh=16,ow=8,oh=8,plane=4).fluxsmootht()
pre = Merge(srchbase, o, weight=.45)

BLK  = 16 # 8
ME1  = 5
ME2  = 2 # 8
_DCT = 5

sup1 = srchbase.MSuper(hpad=16, vpad=16, pel=2, levels=0,sharp=1)
sup2 = pre.MSuper(hpad=16, vpad=16, pel=2, sharp=0)
bv2  = MAnalyse(sup2,delta=2,truemotion=false,global=true,blksize=BLK,overlap=BLK/2,search=ME1,searchparam=ME2,isb=true, dct=_DCT)
bv1  = MAnalyse(sup2,delta=1,truemotion=false,global=true,blksize=BLK,overlap=BLK/2,search=ME1,searchparam=ME2,isb=true, dct=_DCT)
fv1  = MAnalyse(sup2,delta=1,truemotion=false,global=true,blksize=BLK,overlap=BLK/2,search=ME1,searchparam=ME2,isb=false,dct=_DCT)
fv2  = MAnalyse(sup2,delta=2,truemotion=false,global=true,blksize=BLK,overlap=BLK/2,search=ME1,searchparam=ME2,isb=false,dct=_DCT)

mdegrain2(sup1,bv1,fv1,bv2,fv2,thsad=200,thSCD1=256,thSCD2=96)
QTGMCp(src, EdiExt=last)

Interleave(o,last)
PointResize(width*2, height*2)



# helper function: vertical-only MinBlur with vertical radius 2

function mds(clip clp)
{
a  = clp.mt_luts(clp,mode="median",pixels="0 -2 0 -1 0 0 0 1 0 2",Y=2,U=3,V=3)
b  = clp.mt_convolution("0 1 0","1 2 1",y=2,U=3,V=3).mt_convolution("0 1 0","1 2 1",Y=2,U=3,V=3)
aD = mt_makediff(clp,a,Y=2,U=3,V=3)
bD = mt_makediff(clp,b,Y=2,U=3,V=3)
DD = mt_lutxy(aD,bD,"x 128 - y 128 - * 0 < 128 x 128 - abs y 128 - abs < x y ? ?",Y=2,U=3,V=3)
clp.mt_makediff(DD,Y=2,U=3,V=3)
return(last)
}

Needs MinBlur

[kedautinh12]

Yes, i mean faster is half the frames to process

[Reel.Deel]

Originally Posted by kedautinh12

Yes, i mean faster is half the frames to process

QTGMC().SelectEven() is your solution to everything. Third time I see you suggest that, in the last week

[kedautinh12]

Because it's faster

[ENunn]

Originally Posted by Reel.Deel

It's my own frankenstein script (borrowed snippets from a few other scripts), I can post it later if you want. Unless you're done with this clip.

Here's the script that I used in case you're interested. No cropping or upscaling there. The script is a modification of a script Didée wrote a long time ago: https://forum.doom9.org/showthread.p...22#post1107122

FFVideoSource("tape 37-trim.mkv")
src = last
BWDIF(field=-2)#QTGMC(preset="fast")
o = last

mds()
spline36resize(o.width(),o.height()/2/4*4)
mds()
spline36resize(o.width(),o.height()/6/4*4)
mds()
spline36resize(o.width(),o.height())

o.mergechroma(last)
nostripes2 = last.minblur(1)#LSFplus(ss_x=1.0,ss_y=1.0,strength=200,soft=100,overshoot=5)

srchbase   = nostripes2.fft3dfilter(sigma=6,sigma2=4,sigma3=3,sigma4=2,bt=5,bw=16,bh=16,ow=8,oh=8,plane=4).fluxsmootht()
pre = Merge(srchbase, o, weight=.45)

BLK  = 16 # 8
ME1  = 5
ME2  = 2 # 8
_DCT = 5

sup1 = srchbase.MSuper(hpad=16, vpad=16, pel=2, levels=0,sharp=1)
sup2 = pre.MSuper(hpad=16, vpad=16, pel=2, sharp=0)
bv2  = MAnalyse(sup2,delta=2,truemotion=false,global=true,blksize=BLK,overlap=BLK/2,search=ME1,searchparam=ME2,isb=true, dct=_DCT)
bv1  = MAnalyse(sup2,delta=1,truemotion=false,global=true,blksize=BLK,overlap=BLK/2,search=ME1,searchparam=ME2,isb=true, dct=_DCT)
fv1  = MAnalyse(sup2,delta=1,truemotion=false,global=true,blksize=BLK,overlap=BLK/2,search=ME1,searchparam=ME2,isb=false,dct=_DCT)
fv2  = MAnalyse(sup2,delta=2,truemotion=false,global=true,blksize=BLK,overlap=BLK/2,search=ME1,searchparam=ME2,isb=false,dct=_DCT)

mdegrain2(sup1,bv1,fv1,bv2,fv2,thsad=200,thSCD1=256,thSCD2=96)
QTGMCp(src, EdiExt=last)

Interleave(o,last)
PointResize(width*2, height*2)



# helper function: vertical-only MinBlur with vertical radius 2

function mds(clip clp)
{
a  = clp.mt_luts(clp,mode="median",pixels="0 -2 0 -1 0 0 0 1 0 2",Y=2,U=3,V=3)
b  = clp.mt_convolution("0 1 0","1 2 1",y=2,U=3,V=3).mt_convolution("0 1 0","1 2 1",Y=2,U=3,V=3)
aD = mt_makediff(clp,a,Y=2,U=3,V=3)
bD = mt_makediff(clp,b,Y=2,U=3,V=3)
DD = mt_lutxy(aD,bD,"x 128 - y 128 - * 0 < 128 x 128 - abs y 128 - abs < x y ? ?",Y=2,U=3,V=3)
clp.mt_makediff(DD,Y=2,U=3,V=3)
return(last)
}

Needs MinBlur

Hoooo boy. This helped out a lot!!! I modified and added stuff to the script, now I'm getting at least 12 fps and it looks so much better.

Old script:


New script:


Easily compare:
https://imgsli.com/MTM0MDk4

Thanks so much Real! Now to see what looks good for non ES15 tapes.