Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > Avisynth Usage

Closed Thread
 
Thread Tools Search this Thread Display Modes
Old 29th April 2013, 12:01   #1501  |  Link
LigH
German doom9/Gleitz SuMo
 
LigH's Avatar
 
Join Date: Oct 2001
Location: Germany, rural Altmark
Posts: 4,937
As Taurus already stated, a "Distributor()" call is only recommended for very few exceptional uses, not for a normal 32-bit encoding process.

Most common reasons for a slowly decreasing speed are a) excess swap file usage (check your harddisk activity); b) excess threading (ProcessExplorer may help to reveal that); c) extreme RC-lookahead complexity (too slow x264 preset)
__________________

German doom9 / Gleitz video board
CQME – change the Matrix!
BeSweet 1.5b31 All In One | HeadAC3he 0.24a13

Rémoulade is spoiled
LigH is offline  
Old 29th April 2013, 12:18   #1502  |  Link
Boulder
Pig on the wing
 
Boulder's Avatar
 
Join Date: Mar 2002
Location: Hollola, Finland
Posts: 4,324
Encoding also gets slower when the bitrate is higher.

Whenever I need to deal with interlaced sources, I save the QTGMC pass to a lossless intermediate file (ffv1) with VirtualDub and then feed that to x264 in the script. Of course you need the extra diskspace but diskspace is quite cheap these days.
__________________
And if the band you're in starts playing different tunes
I'll see you on the dark side of the Moon...
Boulder is offline  
Old 29th April 2013, 20:25   #1503  |  Link
Johnnyas
Registered User
 
Join Date: Mar 2012
Posts: 15
Thanks for the speedy help! I included the distributor line just because it said 'this line may or may not be necessary' in the first example script I used when trying out the MT version. I removed it now!

I tried removing the old 2.5 version to see if I could get any more speed out of 2.6, so the versions I've been using are:
AviSynth: 32bit 2.6.0 Alpha 4 [130114]
SEt's AviSynth 2.6 MT mod from 20130309
QTGMC 3.33 (because I had problems using 3.32, I tried 3.33)
QTGMC 32-bit Plugins [Vit-Mod].zip - Note: I tried using QTGMC 32-bit Plugins [Vit-2.6] as well, but I got an error with MT_Masktools (access violation in mt_masktools-26.dll), so I tried this package

MediaInfo:
General
Format : AVI
Format/Info : Audio Video Interleave
Format profile : OpenDML
File size : 131 GiB
Duration : 1h 29mn
Overall bit rate : 210 Mbps
TCOD : 0
TCDO : 53547200000

Video
ID : 0
Format : Lagarith
Codec ID : LAGS
Duration : 1h 29mn
Bit rate : 210 Mbps
Width : 1 920 pixels
Height : 1 080 pixels
Display aspect ratio : 16:9
Frame rate : 25.000 fps
Color space : YUV
Chroma subsampling : 4:2:0
Bit depth : 8 bits
Bits/(Pixel*Frame) : 4.051
Stream size : 131 GiB (100%)

I tried modifying the script a bit, just removing the distributor line and reducing maxmem, but that didn't seem to help much. When I checked in after ~5 hours or so, it was down to around 0,76fps. I didn't see neither much swap file usage, nor a lot of evidence of excess threading - at least not from what I could see in Process Explorer.

Separating the qtgmc and the x264 process seems like a good idea, guess I'll do some research on how to do that I guess. I've been using MeGui, and I'm not very familiar with virtualdub, but I guess now is a good a time to learn as any.

edit: oh right, should also give ffvideosource a go.

Last edited by Johnnyas; 29th April 2013 at 20:31. Reason: forgetting stuff.
Johnnyas is offline  
Old 29th April 2013, 22:14   #1504  |  Link
Groucho2004
 
Join Date: Mar 2006
Posts: 3,525
Quote:
Originally Posted by Johnnyas View Post
I tried modifying the script a bit, just removing the distributor line and reducing maxmem, but that didn't seem to help much. When I checked in after ~5 hours or so, it was down to around 0,76fps. I didn't see neither much swap file usage, nor a lot of evidence of excess threading - at least not from what I could see in Process Explorer.
You could test the speed, memory and CPU usage of just the script (without encoder) with AVSMeter.
What's your x264 command line?
Groucho2004 is offline  
Old 30th April 2013, 07:29   #1505  |  Link
Johnnyas
Registered User
 
Join Date: Mar 2012
Posts: 15
Tried AVSMeter (neat program), and after close to 7,5 hours I got this:

FPS (min | max | average): 1.15 | 1.93 | 1.58
CPU usage (current | average): 25% | 25%
Thread count: 7
Physical Memory usage: 1497 MB
Virtual Memory usage: 2053 MB
Time (elapsed | estimated): 07:29:46.906 | 47:12:12.648

Though not exactly blazing, it seems to keep stable at least. CPU usage indicates it's only using one core though, even if Thread count says 7. Have no idea what that is about.
I've been running this on one pc. It's pretty similar to the other one, and performs just about the same.

The X264 command line (autogenerated by MeGui):
program --preset veryslow --crf 20 --qpmin 10 --qpmax 51 --output "output" "input"

On the other pc, I've been trying to get rendering working with Vit's 2.6 modded plugins, but I've been unable to. I think it's only producing errors on Very Slow, not Slower, but it didn't do that a few days ago.

The problem I got was an access violation connected to mt_masktools-26.dll
Tried using mt_masktools-26-for-2.6alpha4.7z (based on vit 2.6), which fixed the access violation with mt_masktools, but then I got an access violation with RemoveGrainSSE2.dll I think. Still trying to get it back to a working state.

Last edited by Johnnyas; 30th April 2013 at 09:42. Reason: clarified where I was working with the 2.6 modded plugins
Johnnyas is offline  
Old 30th April 2013, 07:51   #1506  |  Link
LigH
German doom9/Gleitz SuMo
 
LigH's Avatar
 
Join Date: Oct 2001
Location: Germany, rural Altmark
Posts: 4,937
You are certainly using AviSynth 32-bit. If its calling application is not "Large Address Aware", then both together will only be able to address at most 2 GB of virtual memory; fortunately, a current x264 (or avs4x264mod.exe in case you want to run the 64-bit x264 via this pipe bridge) should be compiled with LAA flag, providing access to 4 GB at most. But I am not sure if AviSynth plugin DLLs compiled without this compiler mode may cause issues... It may be advisable to limit threads in SetMTMode and in functions which fork on their own (see parameter EDIThreads in QTGMC).

Your AVSMeter call (which ignores the video it receives) already surpassed this 2 GB limit (2053 MB) for the script alone; x264 will need even more RAM for its encoding algorithms, including a look-ahead buffer for frame type decisions. The slower the preset, the more frames it looks ahead and needs to store in memory.
__________________

German doom9 / Gleitz video board
CQME – change the Matrix!
BeSweet 1.5b31 All In One | HeadAC3he 0.24a13

Rémoulade is spoiled
LigH is offline  
Old 30th April 2013, 09:52   #1507  |  Link
Johnnyas
Registered User
 
Join Date: Mar 2012
Posts: 15
Yes, I'm using 32bit AviSynth. There were too many plugins and filters that didn't work with the 64bit version the last time I set up AviSynth/QTGMC

EDIThreads is at 1 already, so reducing that would be difficult
QTGMC( Preset="Very Slow", EdiThreads=1 )

The last script I tried, which I reused for AVSMeter, had SetMemoryMax(1200) I think. I reduced this to 1024 and tried again, and got:
Physical Memory usage: 1320 MB
Virtual Memory usage: 1876

Better at least. Will see if I can get a compression job started with this. I tried 1024 earlier, but I'm unsure of which of the plugin packs I was testing it with at the time.
Johnnyas is offline  
Old 30th April 2013, 09:58   #1508  |  Link
Boulder
Pig on the wing
 
Boulder's Avatar
 
Join Date: Mar 2002
Location: Hollola, Finland
Posts: 4,324
You can easily go down to SetMemoryMax(512) for testing, I'm quite sure it won't affect performance in any way.
__________________
And if the band you're in starts playing different tunes
I'll see you on the dark side of the Moon...
Boulder is offline  
Old 30th April 2013, 10:41   #1509  |  Link
Johnnyas
Registered User
 
Join Date: Mar 2012
Posts: 15
I might be having problems because of MeGui, and that it's not using the correct components or something like that..
BUT that shouldn't cause problems for VirtualDub, right! I've tried for a while to get VirtualDub to produce a lossless deinterlaced file with QTGMC set on Very Slow, but it crashes with a Kernel Base error (this was also with the SetMemoryMax at 1024). It seems to work on Slower, but not Very Slow. Don't really know how to fix it at this point, have tried reinstalling quite a few times :/
Johnnyas is offline  
Old 30th April 2013, 10:59   #1510  |  Link
Boulder
Pig on the wing
 
Boulder's Avatar
 
Join Date: Mar 2002
Location: Hollola, Finland
Posts: 4,324
One other thing to try with the lossless intermediate is to forget the multithreading in Avisynth but split the encoding in multiple parts. I have a quad-core i5 so I split the video in three parts using Trim in the scripts. That is, create three scripts that handle three different parts of the video and encode in three VDub sessions simultaneously. Then join the intermediate files in the script that is fed to x264. I have found out that this is often the only way to make it work because the memory usage for HD sources is huge.
__________________
And if the band you're in starts playing different tunes
I'll see you on the dark side of the Moon...
Boulder is offline  
Old 30th April 2013, 11:40   #1511  |  Link
Johnnyas
Registered User
 
Join Date: Mar 2012
Posts: 15
I'm not able to test lowering SetMemoryMax further right now, as I started a new attempt earlier (which is running still of course). Splitting up into several parts is probably a good idea. Should the splitting be done on 'black' (as in transitions in the video), or would a split just between two frames wherever be unnoticeable?
Johnnyas is offline  
Old 30th April 2013, 11:47   #1512  |  Link
Boulder
Pig on the wing
 
Boulder's Avatar
 
Join Date: Mar 2002
Location: Hollola, Finland
Posts: 4,324
It should be unnoticable despite the point where you split it. I always divide the number of frames by three and create the Trim commands according to that. I don't know if you should place the Trim command before or after QTGMC to have a bit-identical output compared to just one script (Avisynth parses the script kind of backwards) - just remember that after QTGMC, the number of frames is doubled.
__________________
And if the band you're in starts playing different tunes
I'll see you on the dark side of the Moon...
Boulder is offline  
Old 30th April 2013, 14:43   #1513  |  Link
Johnnyas
Registered User
 
Join Date: Mar 2012
Posts: 15
Tried SetMemoryMax(512) - but that didn't go too well. I got up to 0.3-0.4 fps, and after a couple of hours it crashed.
Trying SetMemoryMax(800) on that pc now, and it's currently at 0.85 fps. That is QTGMC + X264 compression.

On the other pc, I'm using normal AviSynth (non multitasking), I've divided into four Avisynth scripts with trim, and have four VirtualDub sessions running at the same time. Combined, the four sessions run at around 4 fps. Looking at the Resource monitor, it's at ~100% usage across all cores (though it's very responsive, I'm typing this on the same pc), disk IO has been between 3 and 9 MB/sec, and I'm using about a third of my available ram.

The scripts I'm running now are like this:
1:
AVISource("F:\render\video.avi", audio=false).AssumeFPS(25,1)
trim(0, 35000)
QTGMC( Preset="Very Slow" )

2:
AVISource("F:\render\video.avi", audio=false).AssumeFPS(25,1)
trim(35001, 70000)
QTGMC( Preset="Very Slow" )
etc.

Unless this fails spectacularly, I think we have a solution I'm happy with.
Thanks for the input pals!
Johnnyas is offline  
Old 1st May 2013, 10:01   #1514  |  Link
ChibiBoi
Registered User
 
Join Date: Dec 2004
Posts: 50
I'm trying to use QTGMC in progressive mode because AnimeIVTC is giving me this really bad shimmering on the video footage when set to mode=2, and it works great at removing the shimmering! But QTGMC seems to be adding little blended fields from previous frames back into the footage, even though AnimeIVTC has gotten rid of them. I have two comparison shots, notice the extra horizontal lines coming from the previous frame when QTGMC is incorporated:
AnimeIVTC vs. AnimeIVTC + QTGMC

I've tried adjusting the parameters for QTGMC, such as setting sourcematch=1 or srchclippp=0 individually, and they help, but there is still blending occurring.

Code:
setmtmode(2)
AVISource("C:\Users\Public\Recorded TV\sms test 2.avi", audio=false).AssumeFPS(30000,1001)
assumetff()
converttoyv12(interlaced=true)
animeivtc(mode=2)
qtgmc(inputtype=1)
lanczos4resize(640,480)
ChibiBoi is offline  
Old 1st May 2013, 15:17   #1515  |  Link
kolak
Registered User
 
Join Date: Nov 2004
Location: UK
Posts: 2,183
Quote:
Originally Posted by Johnnyas View Post
I'm not able to test lowering SetMemoryMax further right now, as I started a new attempt earlier (which is running still of course). Splitting up into several parts is probably a good idea. Should the splitting be done on 'black' (as in transitions in the video), or would a split just between two frames wherever be unnoticeable?
Scene change would be a good place for splitting.
kolak is offline  
Old 1st May 2013, 15:37   #1516  |  Link
SamKook
Registered User
 
Join Date: Mar 2011
Posts: 212
Since a couple frames are used before and after for analysis, it would be better to overlap the split by something like 50 frames on both side(when applicable) with a temporary lossless encode and then remove the overlap for the final encode. That way it's as if you never splitted it.
SamKook is offline  
Old 1st May 2013, 15:54   #1517  |  Link
kolak
Registered User
 
Join Date: Nov 2004
Location: UK
Posts: 2,183
At the scene change motion vectors are reset no?
kolak is offline  
Old 1st May 2013, 16:23   #1518  |  Link
SamKook
Registered User
 
Join Date: Mar 2011
Posts: 212
Most likely now that you mention it(but I'm no expert so I don't know for sure).

So my method would be for lazy people who won't bother to take the time to split it in the right places.
SamKook is offline  
Old 1st May 2013, 17:07   #1519  |  Link
Boulder
Pig on the wing
 
Boulder's Avatar
 
Join Date: Mar 2002
Location: Hollola, Finland
Posts: 4,324
I wouldn't be surprised if Avisynth was smart enough to use the frames that are not in the actual output range. I think the logic is that it requests frames as needed and not as the Trim statement says.
__________________
And if the band you're in starts playing different tunes
I'll see you on the dark side of the Moon...
Boulder is offline  
Old 1st May 2013, 17:52   #1520  |  Link
ChibiBoi
Registered User
 
Join Date: Dec 2004
Posts: 50
Quote:
Originally Posted by Boulder View Post
I wouldn't be surprised if Avisynth was smart enough to use the frames that are not in the actual output range. I think the logic is that it requests frames as needed and not as the Trim statement says.
I think this might be right. I just tried the splitting method for multithreading last night and simply divided the number of frames in each segment by the number of threads and it worked like a charm, I appended the segments together using mkvmerge and the transitions between threads was flawless as if I encoded everything as one segment.
ChibiBoi is offline  
Closed Thread

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 20:40.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2017, vBulletin Solutions Inc.