Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > Avisynth Usage

Reply
 
Thread Tools Search this Thread Display Modes
Old 29th June 2015, 12:43   #1  |  Link
gunfelix
Registered User
 
Join Date: Feb 2015
Posts: 34
Speeding up my script

I'm using AviSynth+ on a 64-bit machine with i7 (8 threads) and 8GB of RAM. All plugins are 32-bit and so is the x264 build. I couldn't get 64-bit to work because not all plugins are available in 64-bit. Source is 1080p animation.

It takes a lot of time before it even begins encoding, and then it stays at about 0.01fps and completion is ~170 hours away. The output looks perfect (all noticable problems are gone). The main issue is probably MCTemporalDenoise, because if I remove that filter, it takes only a minute or two before it begins encoding, and at a more acceptable 0.4fps (still takes 30 hours), but the output looks bad.

When encoding without MCTemporalDenoise, it uses 1.4-1.8GB of memory and 40-60% CPU.

AviSynth+ code:
Code:
SetFilterMTMode("DEFAULT_MT_MODE", 2)
SetFilterMTMode("FFVideoSource", 3)
ffVideoSource("input.mkv")
MosquitoNR(strength=21)
SeparateFields()
SelectEven()
nnedi3(dh=true, nns=4)
Santiag()
Santiag()
Santiag()
Dehalo_alpha()
MergeChroma(aWarpSharp(depth=5), aWarpSharp(depth=10))
ChromaShift(c=2)
MCTemporalDenoise(settings="medium")
GradFun3(radius=16,smode=2)
LSFMod(defaults="slow")
Prefetch(8)
x264 commandline:
Code:
x264_x86.exe --profile high --level 4.0 --preset veryslow --crf 16 --tune animation --output output.mkv script.avs
Is it possible to speed it up? I know some of these filters have 64-bit versions, but I can't get them to work in combination with e.g. MCTemporalDenoise (32-bit)
gunfelix is offline   Reply With Quote
Old 29th June 2015, 16:27   #2  |  Link
creaothceann
Registered User
 
Join Date: Jul 2010
Location: Germany
Posts: 360
Quote:
Originally Posted by gunfelix View Post
It takes a lot of time before it even begins encoding
Are you encoding very large frames?

Anyway, what happens when you switch to single-threading mode? Maybe you can just encode 8 parts at once (1 week → 1-2 days).
creaothceann is offline   Reply With Quote
Old 29th June 2015, 17:31   #3  |  Link
Groucho2004
 
Groucho2004's Avatar
 
Join Date: Mar 2006
Posts: 3,902
Why do you de-interlace when the clip is 1080p?

Anyway, I did a little single threaded test with your script (except MCTemporaldenoise) with a 1080i clip. I used DGDecNV as the source filter with AVS+ r1576. CPU is a i5-2500K Sandy Bridge, 4 cores @ 4GHz.

Code:
[General info]
Log file created with:                 AVSMeter 2.0.5 (x86)
Script file:                           F:\Test\1080i50.avs
Command line switches:                 -log
Avisynth version:                      AviSynth+ 0.1 (r1576, x86) (2.6.0.5)
Avisynth.dll linker/compiler version:  11.0 (VC 2012)
Avisynth DLL location:                 E:\Apps\Div\AviSynth.dll
AutoLoadDir 1 (x86):                   E:\Apps\VideoTools\AVSPlugins\AutoLoad


[Clip info]
Number of frames:                 4817
Length (hh:mm:ss.ms):     00:03:12.680
Frame width:                      1920
Frame height:                     1080
Framerate:                      25.000 (25/1)
Colorspace:                       YV12
Audio channels:                    n/a
Audio bits/sample:                 n/a
Audio sample rate:                 n/a
Audio samples:                     n/a


[Runtime info]
Frames processed:               187 (0 - 186)
FPS (min | max | average):      0.942 | 2.777 | 2.698
Memory usage (phys | virt):     1025 | 1176 MB
Thread count:                   63
CPU usage (average):            55%
Time (elapsed):                 00:01:09.465


[Script]
LoadPlugin("E:\Apps\VideoTools\DGDecNV\DGDecodeNV.dll")
DGSource("F:\Test\1080i50.dgi")
MosquitoNR(strength=21)
SeparateFields()
SelectEven()
nnedi3(dh=true, nns=4)
Santiag()
Santiag()
Santiag()
Dehalo_alpha()
MergeChroma(aWarpSharp(depth=5), aWarpSharp(depth=10))
ChromaShift(c=2)
GradFun3(radius=16,smode=2)
LSFMod(defaults="slow")
I'm not sure if it's the encoder that's responsible for your very low speed or if it's an issue with AVS+ (MT). I suggest you use AVSMeter to tune/check your script. If you include the encoder it's difficult to judge what slows down your process.
You probably don't have to multi-thread that script. For example, NNEDI3, which is internally multi-threaded, runs in at least 4(!) instances in your script and is the main CPU hog.

Last edited by Groucho2004; 29th June 2015 at 18:21.
Groucho2004 is offline   Reply With Quote
Old 29th June 2015, 18:54   #4  |  Link
gunfelix
Registered User
 
Join Date: Feb 2015
Posts: 34
Thanks for the replies

Quote:
Originally Posted by creaothceann View Post
Are you encoding very large frames?
Large frames? If you mean the resolution of the frames, it's 1920x1080. If you mean the size, source bitrate is about 10000kbit/s. Framerate is 25, duration about 23 minutes.

Quote:
Originally Posted by creaothceann View Post
Anyway, what happens when you switch to single-threading mode?
Maybe you can just encode 8 parts at once (1 week → 1-2 days).
If I remove the multi-threading lines, it takes a few minutes more to load. I think I might be using multi-threading the wrong way.

Quote:
Originally Posted by Groucho2004 View Post
Why do you de-interlace when the clip is 1080p?
To get rid of some horrible aliasing that was in the video. It was progressive and got deinterlaced, and that's the video I'm trying to fix with this script.

Quote:
Originally Posted by Groucho2004 View Post
I'm not sure if it's the encoder that's responsible for your very low speed or if it's an issue with AVS+ (MT). I suggest you use AVSMeter to tune/check your script. If you include the encoder it's difficult to judge what slows down your process.
You probably don't have to multi-thread that script. For example, NNEDI3, which is internally multi-threaded, runs in at least 4(!) instances in your script and is the main CPU hog.
I grabbed a clip of the source file that's ~20 seconds and it gives me this:
Code:
E:\avs\>avsmeter.exe script.avs -log

AVSMeter 2.0.5 (x86)
AviSynth+ 0.1 (r1689, x86) (2.6.0.5)

Number of frames:                  550
Length (hh:mm:ss.ms):     00:00:22.000
Frame width:                      1920
Frame height:                     1080
Framerate:                      25.000 (25/1)
Colorspace:                       YV12

Frame (current | last):         15 | 549
FPS (cur | min | max | avg):    0.050 | 0.003 | 0.086 | 0.016
Memory usage (phys | virt):     2546 | 2795 MB
Thread count:                   257
CPU usage (current | average):  25% | 29%

Time (elapsed | estimated):     00:16:41.745 | 09:33:54.984
This is without multithreading and in 32-bit. I switched from 2.5.8 to + because it didn't load at all on 2.5.8 after 1-2 hours.
gunfelix is offline   Reply With Quote
Old 29th June 2015, 20:20   #5  |  Link
creaothceann
Registered User
 
Join Date: Jul 2010
Location: Germany
Posts: 360
Quote:
Originally Posted by gunfelix View Post
Thread count: 257
I don't know if that's usual or not, but...

creaothceann is offline   Reply With Quote
Old 29th June 2015, 21:32   #6  |  Link
Groucho2004
 
Groucho2004's Avatar
 
Join Date: Mar 2006
Posts: 3,902
Quote:
Originally Posted by gunfelix View Post
This is without multithreading and in 32-bit.
Highly unlikely with 257 threads.
  1. Post your complete script
  2. Downgrade AVS+ to r1576, r1689 was experimental and probably very buggy
  3. Use the other tool in my signature (AVS Info Tool) and post the plugin report (Tools->Plugin info->Save plugin info report)

Edit:
Quote:
Originally Posted by gunfelix View Post
I grabbed a clip of the source file that's ~20 seconds and it gives me this:
Is this with or without MCTemporalDenoise?

Last edited by Groucho2004; 29th June 2015 at 21:40.
Groucho2004 is offline   Reply With Quote
Old 29th June 2015, 21:58   #7  |  Link
Groucho2004
 
Groucho2004's Avatar
 
Join Date: Mar 2006
Posts: 3,902
Made another test with MCTemporalDenoise included:

Code:
[General info]
Log file created with:                 AVSMeter 2.0.5 (x86)
Script file:                           F:\Test\1080i50.avs
Command line switches:                 -log
Avisynth version:                      AviSynth+ 0.1 (r1576, x86) (2.6.0.5)
Avisynth.dll linker/compiler version:  11.0 (VC 2012)
Avisynth DLL location:                 E:\Apps\Div\AviSynth.dll
AutoLoadDir 1 (x86):                   E:\Apps\VideoTools\AVSPlugins\AutoLoad


[Runtime info]
Frames processed:               71 (0 - 70)
FPS (min | max | average):      0.205 | 1.269 | 1.120
Memory usage (phys | virt):     3246 | 3458 MB
Thread count:                   63
CPU usage (average):            47%
Time (elapsed):                 00:01:03.386


[Script]
setmemorymax(2500)
LoadPlugin("E:\Apps\VideoTools\DGDecNV\DGDecodeNV.dll")
DGSource("F:\Test\1080i50.dgi")
MosquitoNR(strength=21)
SeparateFields()
SelectEven()
nnedi3(dh=true, nns=4)
Santiag()
Santiag()
Santiag()
Dehalo_alpha()
MergeChroma(aWarpSharp(depth=5), aWarpSharp(depth=10))
ChromaShift(c=2)
MCTemporalDenoise(settings="medium")
GradFun3(radius=16,smode=2)
LSFMod(defaults="slow")
Note that I used SetMemoryMax(2500) which was necessary for this script, anything below and it would continuously slow down.
As you can see, the memory used is almost at the limit for AVS32 (4GB) so you would have to pipe this script to x264 (64 bit) with avs2yuv or similar.
You might also consider a less resource-chewing denoiser.

Edit:
Which i7 CPU exactly do you have? There are several generations with vastly different performance.

Last edited by Groucho2004; 29th June 2015 at 22:01.
Groucho2004 is offline   Reply With Quote
Old 29th June 2015, 23:03   #8  |  Link
gunfelix
Registered User
 
Join Date: Feb 2015
Posts: 34
Quote:
Originally Posted by Groucho2004 View Post
Highly unlikely with 257 threads.
  1. Post your complete script
  2. Downgrade AVS+ to r1576, r1689 was experimental and probably very buggy
  3. Use the other tool in my signature (AVS Info Tool) and post the plugin report (Tools->Plugin info->Save plugin info report)

Edit:

Is this with or without MCTemporalDenoise?


Quote:
Originally Posted by Groucho2004 View Post
Post your complete script
See the attachments (script.zip)

Quote:
Originally Posted by Groucho2004 View Post
Downgrade AVS+ to r1576, r1689 was experimental and probably very buggy
Done

Quote:
Originally Posted by Groucho2004 View Post
Use the other tool in my signature (AVS Info Tool) and post the plugin report (Tools->Plugin info->Save plugin info report)
I get the error message: "No autoload-directories specified in the registry."

Quote:
Originally Posted by Groucho2004 View Post
Is this with or without MCTemporalDenoise
This is with
Attached Files
File Type: zip script.zip (624 Bytes, 14 views)
gunfelix is offline   Reply With Quote
Old 29th June 2015, 23:06   #9  |  Link
gunfelix
Registered User
 
Join Date: Feb 2015
Posts: 34
Quote:
Originally Posted by Groucho2004 View Post
As you can see, the memory used is almost at the limit for AVS32 (4GB) so you would have to pipe this script to x264 (64 bit) with avs2yuv or similar.
Is there a tutorial on how to pipe a script?

Quote:
Originally Posted by Groucho2004 View Post
Which i7 CPU exactly do you have? There are several generations with vastly different performance.
Old one, i7-2600CPU@3.40GHz
gunfelix is offline   Reply With Quote
Old 29th June 2015, 23:19   #10  |  Link
Groucho2004
 
Groucho2004's Avatar
 
Join Date: Mar 2006
Posts: 3,902
Quote:
Originally Posted by gunfelix View Post
See the attachments (script.zip)
If you attach something here it can take days to be approved. It's just a script, post it here and use [code] tags, you did it in your first post.

Quote:
Originally Posted by gunfelix View Post
I get the error message: "No autoload-directories specified in the registry."
Odd. That would mean that you didn't install Avisynth properly. Use the installer from avs-plus.net.
Groucho2004 is offline   Reply With Quote
Old 29th June 2015, 23:27   #11  |  Link
Groucho2004
 
Groucho2004's Avatar
 
Join Date: Mar 2006
Posts: 3,902
Quote:
Originally Posted by gunfelix View Post
Is there a tutorial on how to pipe a script?
It's simple.

Here is avs2yuv which you need.

The command line would be (using the same parameters from your first post):
Code:
avs2yuv.exe "source.avs" -o - | x264.exe - --profile high --level 4.0 --preset veryslow --crf 16 --tune animation --output "target.mkv" --demuxer y4m
Groucho2004 is offline   Reply With Quote
Old 30th June 2015, 08:17   #12  |  Link
gunfelix
Registered User
 
Join Date: Feb 2015
Posts: 34
Code:
LoadPlugin("C:\Program Files (x86)\AviSynth+\pluginsx32\ffms2.dll")
LoadPlugin("C:\Program Files (x86)\AviSynth+\pluginsx32\MosquitoNR.dll")
LoadPlugin("C:\Program Files (x86)\AviSynth+\pluginsx32\nnedi3.dll")
LoadPlugin("C:\Program Files (x86)\AviSynth+\pluginsx32\gradfun2db.dll")
LoadPlugin("C:\Program Files (x86)\AviSynth+\pluginsx32\aWarpSharp.dll")
LoadPlugin("C:\Program Files (x86)\AviSynth+\pluginsx32\masktools2-25.dll")
LoadPlugin("C:\Program Files (x86)\AviSynth+\pluginsx32\FFT3DFilter.dll")
LoadPlugin("C:\Program Files (x86)\AviSynth+\pluginsx32\ChromaShift.dll")
LoadPlugin("C:\Program Files (x86)\AviSynth+\pluginsx32\mvtools2.dll")
LoadPlugin("C:\Program Files (x86)\AviSynth+\pluginsx32\RemoveGrainSSE3.dll")
LoadPlugin("C:\Program Files (x86)\AviSynth+\pluginsx32\RepairSSE3.dll")
LoadPlugin("C:\Program Files (x86)\AviSynth+\pluginsx32\dither.dll")
Import("C:\Program Files (x86)\AviSynth+\pluginsx32\dither.avsi")
Import("C:\Program Files (x86)\AviSynth+\pluginsx32\LSFmod.v1.9.avsi")
Import("C:\Program Files (x86)\AviSynth+\pluginsx32\santiag.avsi")
Import("C:\Program Files (x86)\AviSynth+\pluginsx32\MCTemporalDenoise.v1.4.20.avsi")
Import("C:\Program Files (x86)\AviSynth+\pluginsx32\Dehalo_alpha_MT2.avsi")
Import("C:\Program Files (x86)\AviSynth+\pluginsx32\mt_xxpand_multi.avsi")
#SetFilterMTMode("DEFAULT_MT_MODE", 2)
#SetFilterMTMode("FFVideoSource", 3)
ffVideoSource("input.mkv")
MosquitoNR(strength=21)
SeparateFields()
SelectEven()
nnedi3(dh=true, nns=4)
Santiag()
Santiag()
Santiag()
Dehalo_alpha()
MergeChroma(aWarpSharp(depth=5), aWarpSharp(depth=10))
ChromaShift(c=2)
MCTemporalDenoise(settings="medium")
GradFun3(radius=16,smode=2)
LSFMod(defaults="slow")
#Prefetch(8)
Quote:
Odd. That would mean that you didn't install Avisynth properly. Use the installer from avs-plus.net.
I did, but I'll try to reinstall it.
gunfelix is offline   Reply With Quote
Old 30th June 2015, 09:17   #13  |  Link
Groucho2004
 
Groucho2004's Avatar
 
Join Date: Mar 2006
Posts: 3,902
Quote:
Originally Posted by gunfelix View Post
Code:
LoadPlugin("C:\Program Files (x86)\AviSynth+\pluginsx32\ffms2.dll")
LoadPlugin("C:\Program Files (x86)\AviSynth+\pluginsx32\MosquitoNR.dll")
LoadPlugin("C:\Program Files (x86)\AviSynth+\pluginsx32\nnedi3.dll")
LoadPlugin("C:\Program Files (x86)\AviSynth+\pluginsx32\gradfun2db.dll")
LoadPlugin("C:\Program Files (x86)\AviSynth+\pluginsx32\aWarpSharp.dll")
LoadPlugin("C:\Program Files (x86)\AviSynth+\pluginsx32\masktools2-25.dll")
LoadPlugin("C:\Program Files (x86)\AviSynth+\pluginsx32\FFT3DFilter.dll")
LoadPlugin("C:\Program Files (x86)\AviSynth+\pluginsx32\ChromaShift.dll")
LoadPlugin("C:\Program Files (x86)\AviSynth+\pluginsx32\mvtools2.dll")
LoadPlugin("C:\Program Files (x86)\AviSynth+\pluginsx32\RemoveGrainSSE3.dll")
LoadPlugin("C:\Program Files (x86)\AviSynth+\pluginsx32\RepairSSE3.dll")
LoadPlugin("C:\Program Files (x86)\AviSynth+\pluginsx32\dither.dll")
Import("C:\Program Files (x86)\AviSynth+\pluginsx32\dither.avsi")
Import("C:\Program Files (x86)\AviSynth+\pluginsx32\LSFmod.v1.9.avsi")
Import("C:\Program Files (x86)\AviSynth+\pluginsx32\santiag.avsi")
Import("C:\Program Files (x86)\AviSynth+\pluginsx32\MCTemporalDenoise.v1.4.20.avsi")
Import("C:\Program Files (x86)\AviSynth+\pluginsx32\Dehalo_alpha_MT2.avsi")
Import("C:\Program Files (x86)\AviSynth+\pluginsx32\mt_xxpand_multi.avsi")
#SetFilterMTMode("DEFAULT_MT_MODE", 2)
#SetFilterMTMode("FFVideoSource", 3)
ffVideoSource("input.mkv")
MosquitoNR(strength=21)
SeparateFields()
SelectEven()
nnedi3(dh=true, nns=4)
Santiag()
Santiag()
Santiag()
Dehalo_alpha()
MergeChroma(aWarpSharp(depth=5), aWarpSharp(depth=10))
ChromaShift(c=2)
MCTemporalDenoise(settings="medium")
GradFun3(radius=16,smode=2)
LSFMod(defaults="slow")
#Prefetch(8)
I did, but I'll try to reinstall it.
Did you install the 64 Bit version only and then add the 32 Bit DLL? Something is messed up.

I see that you're using some outdated plugins (masktools25, removegrain), I suggest that you use this plugin package. It will also make things faster in single threaded mode.

I ran a few (longer) test with your script and it turns out that it will never work with 32 Bit Avisynth. MCTemporaldenoise uses too much memory. You either hit the 4GB ceiling or it starts slowing down after a few minutes. You have to use a different denoiser.
Groucho2004 is offline   Reply With Quote
Old 30th June 2015, 12:23   #14  |  Link
gunfelix
Registered User
 
Join Date: Feb 2015
Posts: 34
Quote:
Originally Posted by Groucho2004 View Post
Did you install the 64 Bit version only and then add the 32 Bit DLL? Something is messed up.

I see that you're using some outdated plugins (masktools25, removegrain), I suggest that you use this plugin package. It will also make things faster in single threaded mode.
Thanks for those, I installed both versions I think and added the 32-bit avisynth.dll.

Quote:
Originally Posted by Groucho2004 View Post
I ran a few (longer) test with your script and it turns out that it will never work with 32 Bit Avisynth. MCTemporaldenoise uses too much memory. You either hit the 4GB ceiling or it starts slowing down after a few minutes. You have to use a different denoiser.
What would be another good denoiser? I used MCTemporalDenoise for motion compensated temporal smoothing because of some flickering in the source. I already tried TTempSmooth, but that resulted in smearing and it looks like there's no x64 version of MCTemporalDenoise
gunfelix is offline   Reply With Quote
Old 30th June 2015, 12:26   #15  |  Link
Groucho2004
 
Groucho2004's Avatar
 
Join Date: Mar 2006
Posts: 3,902
Quote:
Originally Posted by gunfelix View Post
What would be another good denoiser? I used MCTemporalDenoise for motion compensated temporal smoothing because of some flickering in the source. I already tried TTempSmooth, but that resulted in smearing and it looks like there's no x64 version of MCTemporalDenoise
If you have a decent video card you could try KNLMeansCL. It will also take the load off the CPU.

Edit:
I guess KNLMeansCL is not quite what you're looking for. How about MDegrain? It features motion compensation.

Last edited by Groucho2004; 30th June 2015 at 13:07.
Groucho2004 is offline   Reply With Quote
Old 30th June 2015, 13:46   #16  |  Link
gunfelix
Registered User
 
Join Date: Feb 2015
Posts: 34
Quote:
Originally Posted by Groucho2004 View Post
If you have a decent video card you could try KNLMeansCL. It will also take the load off the CPU.

Edit:
I guess KNLMeansCL is not quite what you're looking for. How about MDegrain? It features motion compensation.
If I replace MCTemporalDenoise(settings="medium") with MDeGrain2() (or 1 or 3), I get this error:
Code:
error: Script error: Invalid arguments to function 'MDeGrain2'.
According to this, there should be default settings, so I'm not sure what to add
gunfelix is offline   Reply With Quote
Old 30th June 2015, 14:12   #17  |  Link
Groucho2004
 
Groucho2004's Avatar
 
Join Date: Mar 2006
Posts: 3,902
The MVTools version I posted is from this post. Parameters have changed. Download it and check the included documentation.
Groucho2004 is offline   Reply With Quote
Old 30th June 2015, 14:40   #18  |  Link
gunfelix
Registered User
 
Join Date: Feb 2015
Posts: 34
Code:
MDeGrain2(thSAD=400, thSADC="thSAD", plane=4, limit=255, limitC="limit")
Still getting the same error
gunfelix is offline   Reply With Quote
Old 30th June 2015, 14:46   #19  |  Link
Groucho2004
 
Groucho2004's Avatar
 
Join Date: Mar 2006
Posts: 3,902
Quote:
Originally Posted by gunfelix View Post
Code:
MDeGrain2(thSAD=400, thSADC="thSAD", plane=4, limit=255, limitC="limit")
Still getting the same error
Try something like this (this is from the documentation, by the way):
Code:
super = MSuper(pel = 2, sharp = 1)
backward_vec2 = MAnalyse(super, isb = true, delta = 2, overlap=4)
backward_vec1 = MAnalyse(super, isb = true, delta = 1, overlap=4)
forward_vec1 = MAnalyse(super, isb = false, delta = 1, overlap=4)
forward_vec2 = MAnalyse(super, isb = false, delta = 2, overlap=4)
MDegrain2(super, backward_vec1,forward_vec1,backward_vec2,forward_vec2,thSAD=400)

Last edited by Groucho2004; 30th June 2015 at 14:49.
Groucho2004 is offline   Reply With Quote
Old 30th June 2015, 22:45   #20  |  Link
foxyshadis
ангел смерти
 
foxyshadis's Avatar
 
Join Date: Nov 2004
Location: Lost
Posts: 9,383
If you're going to use MDegrain, stick with SMDegrain, it's so much simpler and more flexible. Also, Vapoursynth makes the whole 32-bit ceiling go away and it's quite clean once you get used to it, eliminating the swapping that's killing your speed, but that's a personal decision.

If you upload an chunk of the raw source, it might help in removing some of the other pain points.
__________________
There are four boxes to be used in defense of liberty: soap, ballot, jury, and ammo. Please use in that order. ~ Ed Howdershelt
foxyshadis is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 14:13.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2018, vBulletin Solutions Inc.