Speeding up my script

[gunfelix]

I'm using AviSynth+ on a 64-bit machine with i7 (8 threads) and 8GB of RAM. All plugins are 32-bit and so is the x264 build. I couldn't get 64-bit to work because not all plugins are available in 64-bit. Source is 1080p animation.

It takes a lot of time before it even begins encoding, and then it stays at about 0.01fps and completion is ~170 hours away. The output looks perfect (all noticable problems are gone). The main issue is probably MCTemporalDenoise, because if I remove that filter, it takes only a minute or two before it begins encoding, and at a more acceptable 0.4fps (still takes 30 hours), but the output looks bad.

When encoding without MCTemporalDenoise, it uses 1.4-1.8GB of memory and 40-60% CPU.

AviSynth+ code:
SetFilterMTMode("DEFAULT_MT_MODE", 2)
SetFilterMTMode("FFVideoSource", 3)
ffVideoSource("input.mkv")
MosquitoNR(strength=21)
SeparateFields()
SelectEven()
nnedi3(dh=true, nns=4)
Santiag()
Santiag()
Santiag()
Dehalo_alpha()
MergeChroma(aWarpSharp(depth=5), aWarpSharp(depth=10))
ChromaShift(c=2)
MCTemporalDenoise(settings="medium")
GradFun3(radius=16,smode=2)
LSFMod(defaults="slow")
Prefetch(8)

x264 commandline:
x264_x86.exe --profile high --level 4.0 --preset veryslow --crf 16 --tune animation --output output.mkv script.avs

Is it possible to speed it up? I know some of these filters have 64-bit versions, but I can't get them to work in combination with e.g. MCTemporalDenoise (32-bit)

[creaothceann]

It takes a lot of time before it even begins encoding
Are you encoding very large frames?

Anyway, what happens when you switch to single-threading mode? Maybe you can just encode 8 parts at once (1 week → 1-2 days).

[Groucho2004]

Why do you de-interlace when the clip is 1080p?

Anyway, I did a little single threaded test with your script (except MCTemporaldenoise) with a 1080i clip. I used DGDecNV as the source filter with AVS+ r1576. CPU is a i5-2500K Sandy Bridge, 4 cores @ 4GHz.

[General info]
Log file created with: AVSMeter 2.0.5 (x86)
Script file: F:\Test\1080i50.avs
Command line switches: -log
Avisynth version: AviSynth+ 0.1 (r1576, x86) (2.6.0.5)
Avisynth.dll linker/compiler version: 11.0 (VC 2012)
Avisynth DLL location: E:\Apps\Div\AviSynth.dll
AutoLoadDir 1 (x86): E:\Apps\VideoTools\AVSPlugins\AutoLoad


[Clip info]
Number of frames: 4817
Length (hh:mm:ss.ms): 00:03:12.680
Frame width: 1920
Frame height: 1080
Framerate: 25.000 (25/1)
Colorspace: YV12
Audio channels: n/a
Audio bits/sample: n/a
Audio sample rate: n/a
Audio samples: n/a


[Runtime info]
Frames processed: 187 (0 - 186)
FPS (min | max | average): 0.942 | 2.777 | 2.698
Memory usage (phys | virt): 1025 | 1176 MB
Thread count: 63
CPU usage (average): 55%
Time (elapsed): 00:01:09.465


[Script]
LoadPlugin("E:\Apps\VideoTools\DGDecNV\DGDecodeNV.dll")
DGSource("F:\Test\1080i50.dgi")
MosquitoNR(strength=21)
SeparateFields()
SelectEven()
nnedi3(dh=true, nns=4)
Santiag()
Santiag()
Santiag()
Dehalo_alpha()
MergeChroma(aWarpSharp(depth=5), aWarpSharp(depth=10))
ChromaShift(c=2)
GradFun3(radius=16,smode=2)
LSFMod(defaults="slow")

I'm not sure if it's the encoder that's responsible for your very low speed or if it's an issue with AVS+ (MT). I suggest you use AVSMeter to tune/check your script. If you include the encoder it's difficult to judge what slows down your process.
You probably don't have to multi-thread that script. For example, NNEDI3, which is internally multi-threaded, runs in at least 4(!) instances in your script and is the main CPU hog.

[gunfelix]

Thanks for the replies

Are you encoding very large frames?
Large frames? If you mean the resolution of the frames, it's 1920x1080. If you mean the size, source bitrate is about 10000kbit/s. Framerate is 25, duration about 23¾ minutes.


Anyway, what happens when you switch to single-threading mode?
Maybe you can just encode 8 parts at once (1 week → 1-2 days).

If I remove the multi-threading lines, it takes a few minutes more to load. I think I might be using multi-threading the wrong way.

Why do you de-interlace when the clip is 1080p?
To get rid of some horrible aliasing that was in the video. It was progressive and got deinterlaced, and that's the video I'm trying to fix with this script.

I'm not sure if it's the encoder that's responsible for your very low speed or if it's an issue with AVS+ (MT). I suggest you use AVSMeter to tune/check your script. If you include the encoder it's difficult to judge what slows down your process.
You probably don't have to multi-thread that script. For example, NNEDI3, which is internally multi-threaded, runs in at least 4(!) instances in your script and is the main CPU hog.

I grabbed a clip of the source file that's ~20 seconds and it gives me this:
E:\avs\>avsmeter.exe script.avs -log

AVSMeter 2.0.5 (x86)
AviSynth+ 0.1 (r1689, x86) (2.6.0.5)

Number of frames: 550
Length (hh:mm:ss.ms): 00:00:22.000
Frame width: 1920
Frame height: 1080
Framerate: 25.000 (25/1)
Colorspace: YV12

Frame (current | last): 15 | 549
FPS (cur | min | max | avg): 0.050 | 0.003 | 0.086 | 0.016
Memory usage (phys | virt): 2546 | 2795 MB
Thread count: 257
CPU usage (current | average): 25% | 29%

Time (elapsed | estimated): 00:16:41.745 | 09:33:54.984

This is without multithreading and in 32-bit. I switched from 2.5.8 to + because it didn't load at all on 2.5.8 after 1-2 hours.

[creaothceann]

Thread count: 257

I don't know if that's usual or not, but...

http://i.imgur.com/U5CxInV.jpg

[Groucho2004]

This is without multithreading and in 32-bit.
Highly unlikely with 257 threads.


Post your complete script
Downgrade AVS+ to r1576, r1689 was experimental and probably very buggy
Use the other tool in my signature (AVS Info Tool) and post the plugin report (Tools->Plugin info->Save plugin info report)


Edit:
I grabbed a clip of the source file that's ~20 seconds and it gives me this:
Is this with or without MCTemporalDenoise?

[Groucho2004]

Made another test with MCTemporalDenoise included:

[General info]
Log file created with: AVSMeter 2.0.5 (x86)
Script file: F:\Test\1080i50.avs
Command line switches: -log
Avisynth version: AviSynth+ 0.1 (r1576, x86) (2.6.0.5)
Avisynth.dll linker/compiler version: 11.0 (VC 2012)
Avisynth DLL location: E:\Apps\Div\AviSynth.dll
AutoLoadDir 1 (x86): E:\Apps\VideoTools\AVSPlugins\AutoLoad


[Runtime info]
Frames processed: 71 (0 - 70)
FPS (min | max | average): 0.205 | 1.269 | 1.120
Memory usage (phys | virt): 3246 | 3458 MB
Thread count: 63
CPU usage (average): 47%
Time (elapsed): 00:01:03.386


[Script]
setmemorymax(2500)
LoadPlugin("E:\Apps\VideoTools\DGDecNV\DGDecodeNV.dll")
DGSource("F:\Test\1080i50.dgi")
MosquitoNR(strength=21)
SeparateFields()
SelectEven()
nnedi3(dh=true, nns=4)
Santiag()
Santiag()
Santiag()
Dehalo_alpha()
MergeChroma(aWarpSharp(depth=5), aWarpSharp(depth=10))
ChromaShift(c=2)
MCTemporalDenoise(settings="medium")
GradFun3(radius=16,smode=2)
LSFMod(defaults="slow")


Note that I used SetMemoryMax(2500) which was necessary for this script, anything below and it would continuously slow down.
As you can see, the memory used is almost at the limit for AVS32 (4GB) so you would have to pipe this script to x264 (64 bit) with avs2yuv or similar.
You might also consider a less resource-chewing denoiser.

Edit:
Which i7 CPU exactly do you have? There are several generations with vastly different performance.

[gunfelix]

Highly unlikely with 257 threads.


Post your complete script
Downgrade AVS+ to r1576, r1689 was experimental and probably very buggy
Use the other tool in my signature (AVS Info Tool) and post the plugin report (Tools->Plugin info->Save plugin info report)


Edit:

Is this with or without MCTemporalDenoise?



Post your complete script
See the attachments (script.zip)

Downgrade AVS+ to r1576, r1689 was experimental and probably very buggy
Done

Use the other tool in my signature (AVS Info Tool) and post the plugin report (Tools->Plugin info->Save plugin info report)
I get the error message: "No autoload-directories specified in the registry."

Is this with or without MCTemporalDenoise
This is with

[gunfelix]

As you can see, the memory used is almost at the limit for AVS32 (4GB) so you would have to pipe this script to x264 (64 bit) with avs2yuv or similar.Is there a tutorial on how to pipe a script?

Which i7 CPU exactly do you have? There are several generations with vastly different performance.
Old one, i7-2600CPU@3.40GHz

[Groucho2004]

See the attachments (script.zip)
If you attach something here it can take days to be approved. It's just a script, post it here and use [code] tags, you did it in your first post.

I get the error message: "No autoload-directories specified in the registry."
Odd. That would mean that you didn't install Avisynth properly. Use the installer from avs-plus.net.

[Groucho2004]

Is there a tutorial on how to pipe a script?
It's simple.

Here (https://www.dropbox.com/s/ijql860s9cgmg6q/avs2yuv.7z?dl=0) is avs2yuv which you need.

The command line would be (using the same parameters from your first post):
avs2yuv.exe "source.avs" -o - | x264.exe - --profile high --level 4.0 --preset veryslow --crf 16 --tune animation --output "target.mkv" --demuxer y4m

[gunfelix]

LoadPlugin("C:\Program Files (x86)\AviSynth+\pluginsx32\ffms2.dll")
LoadPlugin("C:\Program Files (x86)\AviSynth+\pluginsx32\MosquitoNR.dll")
LoadPlugin("C:\Program Files (x86)\AviSynth+\pluginsx32\nnedi3.dll")
LoadPlugin("C:\Program Files (x86)\AviSynth+\pluginsx32\gradfun2db.dll")
LoadPlugin("C:\Program Files (x86)\AviSynth+\pluginsx32\aWarpSharp.dll")
LoadPlugin("C:\Program Files (x86)\AviSynth+\pluginsx32\masktools2-25.dll")
LoadPlugin("C:\Program Files (x86)\AviSynth+\pluginsx32\FFT3DFilter.dll")
LoadPlugin("C:\Program Files (x86)\AviSynth+\pluginsx32\ChromaShift.dll")
LoadPlugin("C:\Program Files (x86)\AviSynth+\pluginsx32\mvtools2.dll")
LoadPlugin("C:\Program Files (x86)\AviSynth+\pluginsx32\RemoveGrainSSE3.dll")
LoadPlugin("C:\Program Files (x86)\AviSynth+\pluginsx32\RepairSSE3.dll")
LoadPlugin("C:\Program Files (x86)\AviSynth+\pluginsx32\dither.dll")
Import("C:\Program Files (x86)\AviSynth+\pluginsx32\dither.avsi")
Import("C:\Program Files (x86)\AviSynth+\pluginsx32\LSFmod.v1.9.avsi")
Import("C:\Program Files (x86)\AviSynth+\pluginsx32\santiag.avsi")
Import("C:\Program Files (x86)\AviSynth+\pluginsx32\MCTemporalDenoise.v1.4.20.avsi")
Import("C:\Program Files (x86)\AviSynth+\pluginsx32\Dehalo_alpha_MT2.avsi")
Import("C:\Program Files (x86)\AviSynth+\pluginsx32\mt_xxpand_multi.avsi")
#SetFilterMTMode("DEFAULT_MT_MODE", 2)
#SetFilterMTMode("FFVideoSource", 3)
ffVideoSource("input.mkv")
MosquitoNR(strength=21)
SeparateFields()
SelectEven()
nnedi3(dh=true, nns=4)
Santiag()
Santiag()
Santiag()
Dehalo_alpha()
MergeChroma(aWarpSharp(depth=5), aWarpSharp(depth=10))
ChromaShift(c=2)
MCTemporalDenoise(settings="medium")
GradFun3(radius=16,smode=2)
LSFMod(defaults="slow")
#Prefetch(8)

Odd. That would mean that you didn't install Avisynth properly. Use the installer from avs-plus.net.I did, but I'll try to reinstall it.

[Groucho2004]

LoadPlugin("C:\Program Files (x86)\AviSynth+\pluginsx32\ffms2.dll")
LoadPlugin("C:\Program Files (x86)\AviSynth+\pluginsx32\MosquitoNR.dll")
LoadPlugin("C:\Program Files (x86)\AviSynth+\pluginsx32\nnedi3.dll")
LoadPlugin("C:\Program Files (x86)\AviSynth+\pluginsx32\gradfun2db.dll")
LoadPlugin("C:\Program Files (x86)\AviSynth+\pluginsx32\aWarpSharp.dll")
LoadPlugin("C:\Program Files (x86)\AviSynth+\pluginsx32\masktools2-25.dll")
LoadPlugin("C:\Program Files (x86)\AviSynth+\pluginsx32\FFT3DFilter.dll")
LoadPlugin("C:\Program Files (x86)\AviSynth+\pluginsx32\ChromaShift.dll")
LoadPlugin("C:\Program Files (x86)\AviSynth+\pluginsx32\mvtools2.dll")
LoadPlugin("C:\Program Files (x86)\AviSynth+\pluginsx32\RemoveGrainSSE3.dll")
LoadPlugin("C:\Program Files (x86)\AviSynth+\pluginsx32\RepairSSE3.dll")
LoadPlugin("C:\Program Files (x86)\AviSynth+\pluginsx32\dither.dll")
Import("C:\Program Files (x86)\AviSynth+\pluginsx32\dither.avsi")
Import("C:\Program Files (x86)\AviSynth+\pluginsx32\LSFmod.v1.9.avsi")
Import("C:\Program Files (x86)\AviSynth+\pluginsx32\santiag.avsi")
Import("C:\Program Files (x86)\AviSynth+\pluginsx32\MCTemporalDenoise.v1.4.20.avsi")
Import("C:\Program Files (x86)\AviSynth+\pluginsx32\Dehalo_alpha_MT2.avsi")
Import("C:\Program Files (x86)\AviSynth+\pluginsx32\mt_xxpand_multi.avsi")
#SetFilterMTMode("DEFAULT_MT_MODE", 2)
#SetFilterMTMode("FFVideoSource", 3)
ffVideoSource("input.mkv")
MosquitoNR(strength=21)
SeparateFields()
SelectEven()
nnedi3(dh=true, nns=4)
Santiag()
Santiag()
Santiag()
Dehalo_alpha()
MergeChroma(aWarpSharp(depth=5), aWarpSharp(depth=10))
ChromaShift(c=2)
MCTemporalDenoise(settings="medium")
GradFun3(radius=16,smode=2)
LSFMod(defaults="slow")
#Prefetch(8)

I did, but I'll try to reinstall it.

Did you install the 64 Bit version only and then add the 32 Bit DLL? Something is messed up.

I see that you're using some outdated plugins (masktools25, removegrain), I suggest that you use this (https://www.dropbox.com/s/rr0coy45wlzc4hr/plugins.7z?dl=0) plugin package. It will also make things faster in single threaded mode.

I ran a few (longer) test with your script and it turns out that it will never work with 32 Bit Avisynth. MCTemporaldenoise uses too much memory. You either hit the 4GB ceiling or it starts slowing down after a few minutes. You have to use a different denoiser.

[gunfelix]

Did you install the 64 Bit version only and then add the 32 Bit DLL? Something is messed up.

I see that you're using some outdated plugins (masktools25, removegrain), I suggest that you use this (https://www.dropbox.com/s/rr0coy45wlzc4hr/plugins.7z?dl=0) plugin package. It will also make things faster in single threaded mode.

Thanks for those, I installed both versions I think and added the 32-bit avisynth.dll.

I ran a few (longer) test with your script and it turns out that it will never work with 32 Bit Avisynth. MCTemporaldenoise uses too much memory. You either hit the 4GB ceiling or it starts slowing down after a few minutes. You have to use a different denoiser.

What would be another good denoiser? I used MCTemporalDenoise for motion compensated temporal smoothing because of some flickering in the source. I already tried TTempSmooth, but that resulted in smearing and it looks like there's no x64 version of MCTemporalDenoise

[Groucho2004]

What would be another good denoiser? I used MCTemporalDenoise for motion compensated temporal smoothing because of some flickering in the source. I already tried TTempSmooth, but that resulted in smearing and it looks like there's no x64 version of MCTemporalDenoise
If you have a decent video card you could try KNLMeansCL (http://forum.doom9.org/showthread.php?t=171379). It will also take the load off the CPU.

Edit:
I guess KNLMeansCL is not quite what you're looking for. How about MDegrain? It features motion compensation.

[gunfelix]

If you have a decent video card you could try KNLMeansCL (http://forum.doom9.org/showthread.php?t=171379). It will also take the load off the CPU.

Edit:
I guess KNLMeansCL is not quite what you're looking for. How about MDegrain? It features motion compensation.

If I replace MCTemporalDenoise(settings="medium") with MDeGrain2() (or 1 or 3), I get this error:
error: Script error: Invalid arguments to function 'MDeGrain2'.

According to this (http://avisynth.org.ru/mvtools/mvtools2.html), there should be default settings, so I'm not sure what to add

[Groucho2004]

The MVTools version I posted is from this post (http://forum.doom9.org/showthread.php?p=1386559#post1386559). Parameters have changed. Download it and check the included documentation.

[gunfelix]

MDeGrain2(thSAD=400, thSADC="thSAD", plane=4, limit=255, limitC="limit")
Still getting the same error

[Groucho2004]

MDeGrain2(thSAD=400, thSADC="thSAD", plane=4, limit=255, limitC="limit")
Still getting the same error
Try something like this (this is from the documentation, by the way):
super = MSuper(pel = 2, sharp = 1)
backward_vec2 = MAnalyse(super, isb = true, delta = 2, overlap=4)
backward_vec1 = MAnalyse(super, isb = true, delta = 1, overlap=4)
forward_vec1 = MAnalyse(super, isb = false, delta = 1, overlap=4)
forward_vec2 = MAnalyse(super, isb = false, delta = 2, overlap=4)
MDegrain2(super, backward_vec1,forward_vec1,backward_vec2,forward_vec2,thSAD=400)

[foxyshadis]

If you're going to use MDegrain, stick with SMDegrain (http://avisynth.nl/index.php/SMDegrain), it's so much simpler and more flexible. Also, Vapoursynth makes the whole 32-bit ceiling go away and it's quite clean once you get used to it, eliminating the swapping that's killing your speed, but that's a personal decision.

If you upload an chunk of the raw source, it might help in removing some of the other pain points.

[gunfelix]

If you upload an chunk of the raw source, it might help in removing some of the other pain points.

I've got different videos/series with the same problems, but these ones are the worst:
https://mega.co.nz/#F!94QH0JTb!SerVk1S_hc6hMdmTy4zBuA

Try something like this (this is from the documentation, by the way):
super = MSuper(pel = 2, sharp = 1)
backward_vec2 = MAnalyse(super, isb = true, delta = 2, overlap=4)
backward_vec1 = MAnalyse(super, isb = true, delta = 1, overlap=4)
forward_vec1 = MAnalyse(super, isb = false, delta = 1, overlap=4)
forward_vec2 = MAnalyse(super, isb = false, delta = 2, overlap=4)
MDegrain2(super, backward_vec1,forward_vec1,backward_vec2,forward_vec2,thSAD=400)


That works, speed is now better, around 0.5fps (chunk0.mkv)

[Runtime info]
Frames processed: 100 (0 - 99)
FPS (min | max | average): 0.381 | 6.930 | 0.545
Memory usage (phys | virt): 1373 | 1452 MB
Thread count: 128
CPU usage (average): 38%
Time (elapsed): 00:03:03.365

Entire logfile: https://mega.co.nz/#!o0RnxbiY!YnPzgSWl4CZEq4zAFojfLnhYiuuP1tyi-8XCA8xYd4s

If you're going to use MDegrain, stick with SMDegrain (http://avisynth.nl/index.php/SMDegrain), it's so much simpler and more flexible.

I'll try SMDegrain

Also, Vapoursynth makes the whole 32-bit ceiling go away and it's quite clean once you get used to it, eliminating the swapping that's killing your speed, but that's a personal decision.

Does that mean that with Vapoursynth the script with MCTemporalDenoise will run faster?

[Groucho2004]

You're still using at least some of the old plugins:
LoadPlugin("C:\Program Files (x86)\AviSynth+\pluginsx32\masktools2-25.dll")

Why don't you just use one of the predefined auto-load directories?

[feisty2]

just stick to nlmeans (knlmeanscl) if got a not that crappy video card
it's an even (theoretically) better denoise algorithm compared to mdegrain (motion compensation), and better, yeah, I mean quality wise kinda better
cuz:
1. motion compensation only works in temporal dimension, nlmeans works both spatially and temporally.
2. even if it's just temporal stuff we're talking about, nlmeans still wins, in ME&MC algo, every block ("pixel" if it's motion flow) only gets to be matched ONCE, nlmeans got no limit like that, every pixel within the search window gets to blend with the center pixel by a unique similarity weight, pixels could blend with other pixels multiple times

you could try to set device type to cpu and it would be A LOT slower than mdegrain cuz, as a killer level algorithm, it's also a killer of hardware consumption, the only reason knlmeanscl works that fast is, gpu works better than cpu as an image processor. (knlmeanscl works on gpu by default, mvtools works on cpu)

EDIT:
docs from TNLMeans
http://bengal.missouri.edu/~kes25c/nl3.pdf
http://bengal.missouri.edu/~kes25c/nl1.pdf
http://bengal.missouri.edu/~kes25c/nl2.pdf

[feisty2]

and also in case you got curious
the only even theoretically better than NLMeans algorithm we got in doom9 forum for now
http://forum.doom9.org/showthread.php?t=172172
but it's even slower than you can imagine, practically unusable to actual video processing.

[gunfelix]

@Groucho2004

Argh, forgot to hit Save.

@feisty2

SMDegrain is 0.01fps slower:
[Runtime info]
Frames processed: 100 (0 - 99)
FPS (min | max | average): 0.369 | 9.035 | 0.513
Memory usage (phys | virt): 1357 | 1436 MB
Thread count: 128
CPU usage (average): 39%
Time (elapsed): 00:03:14.745

TNLMeans is too slow for my pc (42 hours for 23m40s of video):
[Runtime info]
Frames processed: 100 (0 - 99)
FPS (min | max | average): 0.079 | 32.61 | 0.234
Memory usage (phys | virt): 1348 | 1426 MB
Thread count: 128
CPU usage (average): 15%
Time (elapsed): 00:07:08.188

KNLMeansCL:
When installing the OpenCL driver, I get:
This installation package is not supported by this processor type
https://downloadcenter.intel.com/search?keyword=4th+Generation+Intel%C2%AE+Core%E2%84%A2+Processors+with+Intel%C2%AE+Iris%E2%84%A2+Pro+Graphics+5200

[feisty2]

tnlmeans is a cpu filter, slow like hell, unusable like V-BM3D
knlmeanscl is kinda a usable NLMeans filter, I got no problem with it so far, but you could comment in its thread (http://forum.doom9.org/showthread.php?t=171379) if you failed to run it

[Groucho2004]

KNLMeansCL:
When installing the OpenCL driver, I get:
This installation package is not supported by this processor type
https://downloadcenter.intel.com/search?keyword=4th+Generation+Intel%C2%AE+Core%E2%84%A2+Processors+with+Intel%C2%AE+Iris%E2%84%A2+Pro+Graphics+5200
Don't you have a dedicated graphics card? The integrated GPU in your i7 might be too slow.

[gunfelix]

Don't you have a dedicated graphics card? The integrated GPU in your i7 might be too slow.

According to dxdiag my graphics cards is a Intel HD Graphics 2000.

[gunfelix]

tnlmeans is a cpu filter, slow like hell, unusable like V-BM3D
knlmeanscl is kinda a usable NLMeans filter, I got no problem with it so far, but you could comment in its thread (http://forum.doom9.org/showthread.php?t=171379) if you failed to run it

I think it's more of an issue with the OpenCL driver than a issue with KNLMeansCL itself.

[Asmodian]

According to dxdiag my graphics cards is a Intel HD Graphics 2000.

Drop any idea of GPU filters, an HD 2000 isn't going to handle it.