QTGMC Deinterlacing Script (v3.32)

[LigH]

As Taurus already stated, a "Distributor()" call is only recommended for very few exceptional uses, not for a normal 32-bit encoding process.

Most common reasons for a slowly decreasing speed are a) excess swap file usage (check your harddisk activity); b) excess threading (ProcessExplorer may help to reveal that); c) extreme RC-lookahead complexity (too slow x264 preset)

[Boulder]

Encoding also gets slower when the bitrate is higher.

Whenever I need to deal with interlaced sources, I save the QTGMC pass to a lossless intermediate file (ffv1) with VirtualDub and then feed that to x264 in the script. Of course you need the extra diskspace but diskspace is quite cheap these days.

[Johnnyas]

Thanks for the speedy help! I included the distributor line just because it said 'this line may or may not be necessary' in the first example script I used when trying out the MT version. I removed it now!

I tried removing the old 2.5 version to see if I could get any more speed out of 2.6, so the versions I've been using are:
AviSynth: 32bit 2.6.0 Alpha 4 [130114]
SEt's AviSynth 2.6 MT mod from 20130309
QTGMC 3.33 (because I had problems using 3.32, I tried 3.33)
QTGMC 32-bit Plugins [Vit-Mod].zip - Note: I tried using QTGMC 32-bit Plugins [Vit-2.6] as well, but I got an error with MT_Masktools (access violation in mt_masktools-26.dll), so I tried this package

MediaInfo:
General
Format : AVI
Format/Info : Audio Video Interleave
Format profile : OpenDML
File size : 131 GiB
Duration : 1h 29mn
Overall bit rate : 210 Mbps
TCOD : 0
TCDO : 53547200000

Video
ID : 0
Format : Lagarith
Codec ID : LAGS
Duration : 1h 29mn
Bit rate : 210 Mbps
Width : 1 920 pixels
Height : 1 080 pixels
Display aspect ratio : 16:9
Frame rate : 25.000 fps
Color space : YUV
Chroma subsampling : 4:2:0
Bit depth : 8 bits
Bits/(Pixel*Frame) : 4.051
Stream size : 131 GiB (100%)

I tried modifying the script a bit, just removing the distributor line and reducing maxmem, but that didn't seem to help much. When I checked in after ~5 hours or so, it was down to around 0,76fps. I didn't see neither much swap file usage, nor a lot of evidence of excess threading - at least not from what I could see in Process Explorer.

Separating the qtgmc and the x264 process seems like a good idea, guess I'll do some research on how to do that I guess. I've been using MeGui, and I'm not very familiar with virtualdub, but I guess now is a good a time to learn as any.

edit: oh right, should also give ffvideosource a go.

[Groucho2004]

I tried modifying the script a bit, just removing the distributor line and reducing maxmem, but that didn't seem to help much. When I checked in after ~5 hours or so, it was down to around 0,76fps. I didn't see neither much swap file usage, nor a lot of evidence of excess threading - at least not from what I could see in Process Explorer.
You could test the speed, memory and CPU usage of just the script (without encoder) with AVSMeter (http://forum.doom9.org/showthread.php?t=165528).
What's your x264 command line?

[Johnnyas]

Tried AVSMeter (neat program), and after close to 7,5 hours I got this:

FPS (min | max | average): 1.15 | 1.93 | 1.58
CPU usage (current | average): 25% | 25%
Thread count: 7
Physical Memory usage: 1497 MB
Virtual Memory usage: 2053 MB
Time (elapsed | estimated): 07:29:46.906 | 47:12:12.648

Though not exactly blazing, it seems to keep stable at least. CPU usage indicates it's only using one core though, even if Thread count says 7. Have no idea what that is about.
I've been running this on one pc. It's pretty similar to the other one, and performs just about the same.

The X264 command line (autogenerated by MeGui):
program --preset veryslow --crf 20 --qpmin 10 --qpmax 51 --output "output" "input"

On the other pc, I've been trying to get rendering working with Vit's 2.6 modded plugins, but I've been unable to. I think it's only producing errors on Very Slow, not Slower, but it didn't do that a few days ago.

The problem I got was an access violation connected to mt_masktools-26.dll
Tried using mt_masktools-26-for-2.6alpha4.7z (based on vit 2.6), which fixed the access violation with mt_masktools, but then I got an access violation with RemoveGrainSSE2.dll I think. Still trying to get it back to a working state.

[LigH]

You are certainly using AviSynth 32-bit. If its calling application is not "Large Address Aware", then both together will only be able to address at most 2 GB of virtual memory; fortunately, a current x264 (or avs4x264mod.exe in case you want to run the 64-bit x264 via this pipe bridge) should be compiled with LAA flag, providing access to 4 GB at most. But I am not sure if AviSynth plugin DLLs compiled without this compiler mode may cause issues... It may be advisable to limit threads in SetMTMode and in functions which fork on their own (see parameter EDIThreads in QTGMC).

Your AVSMeter call (which ignores the video it receives) already surpassed this 2 GB limit (2053 MB) for the script alone; x264 will need even more RAM for its encoding algorithms, including a look-ahead buffer for frame type decisions. The slower the preset, the more frames it looks ahead and needs to store in memory.

[Johnnyas]

Yes, I'm using 32bit AviSynth. There were too many plugins and filters that didn't work with the 64bit version the last time I set up AviSynth/QTGMC

EDIThreads is at 1 already, so reducing that would be difficult
QTGMC( Preset="Very Slow", EdiThreads=1 )

The last script I tried, which I reused for AVSMeter, had SetMemoryMax(1200) I think. I reduced this to 1024 and tried again, and got:
Physical Memory usage: 1320 MB
Virtual Memory usage: 1876

Better at least. Will see if I can get a compression job started with this. I tried 1024 earlier, but I'm unsure of which of the plugin packs I was testing it with at the time.

[Boulder]

You can easily go down to SetMemoryMax(512) for testing, I'm quite sure it won't affect performance in any way.

[Johnnyas]

I might be having problems because of MeGui, and that it's not using the correct components or something like that..
BUT that shouldn't cause problems for VirtualDub, right! I've tried for a while to get VirtualDub to produce a lossless deinterlaced file with QTGMC set on Very Slow, but it crashes with a Kernel Base error (this was also with the SetMemoryMax at 1024). It seems to work on Slower, but not Very Slow. Don't really know how to fix it at this point, have tried reinstalling quite a few times :/

[Boulder]

One other thing to try with the lossless intermediate is to forget the multithreading in Avisynth but split the encoding in multiple parts. I have a quad-core i5 so I split the video in three parts using Trim in the scripts. That is, create three scripts that handle three different parts of the video and encode in three VDub sessions simultaneously. Then join the intermediate files in the script that is fed to x264. I have found out that this is often the only way to make it work because the memory usage for HD sources is huge.

[Johnnyas]

I'm not able to test lowering SetMemoryMax further right now, as I started a new attempt earlier (which is running still of course). Splitting up into several parts is probably a good idea. Should the splitting be done on 'black' (as in transitions in the video), or would a split just between two frames wherever be unnoticeable?

[Boulder]

It should be unnoticable despite the point where you split it. I always divide the number of frames by three and create the Trim commands according to that. I don't know if you should place the Trim command before or after QTGMC to have a bit-identical output compared to just one script (Avisynth parses the script kind of backwards) - just remember that after QTGMC, the number of frames is doubled.

[Johnnyas]

Tried SetMemoryMax(512) - but that didn't go too well. I got up to 0.3-0.4 fps, and after a couple of hours it crashed.
Trying SetMemoryMax(800) on that pc now, and it's currently at 0.85 fps. That is QTGMC + X264 compression.

On the other pc, I'm using normal AviSynth (non multitasking), I've divided into four Avisynth scripts with trim, and have four VirtualDub sessions running at the same time. Combined, the four sessions run at around 4 fps. Looking at the Resource monitor, it's at ~100% usage across all cores (though it's very responsive, I'm typing this on the same pc), disk IO has been between 3 and 9 MB/sec, and I'm using about a third of my available ram.

The scripts I'm running now are like this:
1:
AVISource("F:\render\video.avi", audio=false).AssumeFPS(25,1)
trim(0, 35000)
QTGMC( Preset="Very Slow" )

2:
AVISource("F:\render\video.avi", audio=false).AssumeFPS(25,1)
trim(35001, 70000)
QTGMC( Preset="Very Slow" )
etc.

Unless this fails spectacularly, I think we have a solution I'm happy with.
Thanks for the input pals!

[ChibiBoi]

I'm trying to use QTGMC in progressive mode because AnimeIVTC is giving me this really bad shimmering on the video footage when set to mode=2, and it works great at removing the shimmering! But QTGMC seems to be adding little blended fields from previous frames back into the footage, even though AnimeIVTC has gotten rid of them. I have two comparison shots, notice the extra horizontal lines coming from the previous frame when QTGMC is incorporated:
AnimeIVTC (http://i39.tinypic.com/63r70o.jpg) vs. AnimeIVTC + QTGMC (http://i44.tinypic.com/2w7ogo8.jpg)

I've tried adjusting the parameters for QTGMC, such as setting sourcematch=1 or srchclippp=0 individually, and they help, but there is still blending occurring.

setmtmode(2)
AVISource("C:\Users\Public\Recorded TV\sms test 2.avi", audio=false).AssumeFPS(30000,1001)
assumetff()
converttoyv12(interlaced=true)
animeivtc(mode=2)
qtgmc(inputtype=1)
lanczos4resize(640,480)

[kolak]

I'm not able to test lowering SetMemoryMax further right now, as I started a new attempt earlier (which is running still of course). Splitting up into several parts is probably a good idea. Should the splitting be done on 'black' (as in transitions in the video), or would a split just between two frames wherever be unnoticeable?

Scene change would be a good place for splitting.

[SamKook]

Since a couple frames are used before and after for analysis, it would be better to overlap the split by something like 50 frames on both side(when applicable) with a temporary lossless encode and then remove the overlap for the final encode. That way it's as if you never splitted it.

[kolak]

At the scene change motion vectors are reset no?

[SamKook]

Most likely now that you mention it(but I'm no expert so I don't know for sure).

So my method would be for lazy people who won't bother to take the time to split it in the right places.

[Boulder]

I wouldn't be surprised if Avisynth was smart enough to use the frames that are not in the actual output range. I think the logic is that it requests frames as needed and not as the Trim statement says.

[ChibiBoi]

I wouldn't be surprised if Avisynth was smart enough to use the frames that are not in the actual output range. I think the logic is that it requests frames as needed and not as the Trim statement says.

I think this might be right. I just tried the splitting method for multithreading last night and simply divided the number of frames in each segment by the number of threads and it worked like a charm, I appended the segments together using mkvmerge and the transitions between threads was flawless as if I encoded everything as one segment.

[Johnnyas]

The clip(s) are getting the x264 treatment right now. Will check the transition points to see if the trimpoints are noticeable. If they are, I'll try it with the +/- 50 frames first I guess - would take the work out of finding a good spot to do it :)

Right now, it looks like the combined time to process both operations will be around 43 hours. Thats much faster than any of my attempts with MT would have finished on (by at least a day actually) :cool:

[kolak]

For QTGMC deinterlacing on many core CPU you can try MP_Pipeline with branching option. It's slower than splitting into many processes, but all in one go and piece :)
Also- anything above fast for HD is overkill, very fast is enough in most cases.

[Groucho2004]

Also- anything above fast for HD is overkill, very fast is enough in most cases.
That's a very good point. I only noticed now that Johnnyas is using "very slow" which is painful even with the fastest CPUs for HD resolutions. I also have seen very good results with "Medium" and "Fast" presets. Not to mention that the memory requirements will most likely be lower.

[SamKook]

Also- anything above fast for HD is overkill, very fast is enough in most cases.

That is far from true if you care about the size of the resulting file. Slower presets will make a noticeably smaller file if using CRF with x264.

For HD, I usually use very slow paired with the source match and lossless settings in qtgmc and veryslow with x264. I also use setmemorymax(1200) which I found was the most stable on my pc with those settings and split the encode in 4 with setmtmode set to 2 threads each. I also use avs2pipemod to pipe the avs to 64bit x264, mostly for the added stability piping provides since it doesn't seem faster than using the 32bit version.

It usually take a little less than a day to make the lossless encode to FFV1 and a bit over a day for the final 10bit x264 one(so 8bit would be close to half the time) for a 2 hour movie.

[Groucho2004]

That is far from true if you care about the size of the resulting file. Slower presets will make a noticeably smaller file if using CRF with x264.
Hm, I was under the impression that we were talking about QTGMC presets...

[SamKook]

Hm, I was under the impression that we were talking about QTGMC presets...

I was, I simply mentioned CRF x264 because it doesn't produce a fixed file size. My statement would have made no sense with a fixed bitrate.

Either way, both slower QTGMC and x264 settings affect file size(or quality at fixed bitrate) by a lot. I wish I would have kept my result from the last time I tested this.

[kolak]

That is far from true if you care about the size of the resulting file. Slower presets will make a noticeably smaller file if using CRF with x264.

For HD, I usually use very slow paired with the source match and lossless settings in qtgmc and veryslow with x264. I also use setmemorymax(1200) which I found was the most stable on my pc with those settings and split the encode in 4 with setmtmode set to 2 threads each. I also use avs2pipemod to pipe the avs to 64bit x264, mostly for the added stability piping provides since it doesn't seem faster than using the 32bit version.

It usually take a little less than a day to make the lossless encode to FFV1 and a bit over a day for the final 10bit x264 one(so 8bit would be close to half the time) for a 2 hour movie.

Well- it's your workflow.
I would not waste 3x more time for even 25% smaller file size. I assume it's not close to 25% is it?
If it works for you than fine :)

[SamKook]

It indeed depends on your priorities. If I were encoding a lot of them, then I might reconsider how much time I'm willing to spend on it.
I don't care at all how long it takes as long as I can get all I can out of it. Everything is automated anyway once I've tweaked my settings for a particular video and it doesn't affect anything else I do on the computer at the same time so it doesn't bother me.
I've even tried to encode a bluray without splitting it with crazy(and not so good) settings and 21 days later, it finished. I won't do that ever again, but it was a fun experiment.

And I also doubt it's as much as 25%, but I remember being surprised by how much it was. I'll have to try it again when I have the time.

[Johnnyas]

The files I'm encoding are our home movies, and I encode them to keep like this. Since these will be my archive files, I definately want to do it on the best quality possible (within reason). I know there isn't too much difference between the slower presets on HD (well, at least from medium and slower), but a I'm encoding to keep so, it's not a one-time watch thing. Also, as SamKook, I've seen files get smaller when I've had both QTGMC AND the x264 encoder set at Very Slow, as opposed to Slower on QTGMC/Very Slow on the x264 encoder.

[Johnnyas]

Just an update here. Have done some testing, and the final renders on this movie at least - and I can't see the points where I join the movies at all, and that was with trimming from the exact frames (as opposed to doing some frames before/after). Very happy with the final results of this, so once again - thanks all!

[Chainmax]

I am trying to use QTGMC with the plugin pack that uses the updated Avisynth 2.6 interface, and these filters from it:

mt_masktools-26

removegrainsse2

mvtools2

fft3dfilter

Gave out an access violation error at an address and attempted to read from 0x00000000. Also, I got an error with nnedi3 that said arg0 needed to be a clip. Replacing these five DLLs with the ones from the original Modded Plugins Package seems to have solved the issues.

What could be the cause of these error messages?

[Selur]

What could be the cause of these error messages?
you replaced the five dlls with the ones from the modded plugins package :)
(read this thread and you will see that a bunch of people, me included, have problems with the modded plugins)

[LigH]

And if you are using MT 2.60a4:

Therefore the "latest" version is a patch for mt_masktools26.dll by 06_taro (http://forum.doom9.org/showthread.php?p=1619023#post1619023) for AviSynth MT 2.60 alpha 4, not 3.

The frame cache interface changed between 3 and 4.

[aegisofrime]

Hi. Rather a noobish question here.

I have a video that MeGUI detects as being of varying field order. How do I handle this with QTGMC? :thanks:

[LigH]

Serious answer:

Never trust an automatical detection. There is always a chance that the samples catch frames with too little changes to detect field changes reliably.

If there is a doubt about the structure, check the video manually by stepping frame-by-frame through a bobbed result (bobbing interpolates fields to whole frames for a convenient display), preceded with both AssumeTFF and AssumeBFF successively, counting how many fields are mostly equal or different over at least 30 fields in a row.

QTGMC is usually only recommendable if you are certain that the material is plain interlaced. For telecined material, use an IVTC plugin instead. And special standard conversions (PAL/NTSC/Film; censorship cover-up) might require custom solutions even...

[GMJCZP]

After calling the following function:

# Anti-aliasing with contra-sharpening by Didée (thanks to Didée, modded for this case with nnedi3)
# Is required VerticalCleaner
function DAAMod(clip c)
{
nn = c.nnedi3(field=-2)
dbl = mt_average(selecteven(nn),selectodd(nn),U=3,V=3)
dblD = mt_makediff(c,dbl,U=3,V=3)
shrpD = mt_makediff(dbl,dbl.removegrain((width(c)>1100) ? 20 : 14),U=3,V=3).verticalcleaner(mode=2)
DD = shrpD.repair(dblD,9)
return dbl.mt_adddiff(DD,U=3,V=3)
}

I get the following error message:

Avisynth open failure:
nnedi3: arg 0 must be a clip!

The nnedi3 calling is in red.
I was using version 2.6-Vit-nnedi3 (part of plugin pack for testing) and I had to return to the tritical's original version to run the script. I'm using the latest version of Avisynth 2.6.0 MT of SEt 2013.03.09.

[LigH]

The problem might be outside this function, you may be calling DAAMod() with something that isn't a clip (e.g. in a context where the clip is not 'last', but a specific variable), and its parameter is passed on to NNEDI3().

[Gavino]

The problem might be outside this function, you may be calling DAAMod() with something that isn't a clip
But in that case you would expect to get the error "Invalid arguments to function DAAMod" and it would never get as far as calling nnedi3.

[GMJCZP]

The problem might be outside this function, you may be calling DAAMod() with something that isn't a clip (e.g. in a context where the clip is not 'last', but a specific variable), and its parameter is passed on to NNEDI3().

As I said before, the script works with the old version of nnedi3. I downloaded again the -Vit- modded plugin package of QTGMC (not 2.6 version), but I have not tested the nnedi3 version modded.

-0-
On the other hand, I did something funny. I hung up the VirtualDub is processing a script more than once when appear 5 minutes (I have so configured in my PC) the screen saver. I had the idea of not carrying on the script the old version of nnedi3 (not was calling him!, only put #Load... nnedi3.dll) and incredibly the problem was solved. Coincidence?

[pila13]

Hello, i need small help with qtgmc + JDL_ApplyRange

I need use this filter only on xxxx of frames, and recive error -

"expected a, or ) "

JDL_ApplyRangeOld(6428, 6466, "QTGMC(Preset="placebo").selecteven()") This is example of how i try to use this plugin. Can any one help me ? thnak you.

[Boulder]

Have you tried triple quotes: JDL_ApplyRangeOld(6428, 6466, """QTGMC(Preset="placebo").selecteven()""") ?

[LigH]

You cannot embed a string with "single double-quotes" in a string with "single double-quotes"; to circumvent this limitation, there are also strings with """triple double-quotes""".

JDL_ApplyRangeOld(6428, 6466, """QTGMC(Preset="placebo").selecteven()""")
__

Apart from that: A preset named "placebo" uses this name for a specific purpose ... to make you feel bad using it, because it is rather a waste of time than "the optimum".

[pila13]

that one current moment, need placebo. this is hwy i don't want use it on a whole title, rest i will sue with medium.

thank you for help !

I tyr "" "", but didn't thoguht about """ """. (*´ー｀*)

[LigH]

Everything is documented, in the internet (http://avisynth.org/mediawiki/Script_variables) as well as in your local installation.

[tormento]

I want to convert 1080i50 to 1080p25. Is there a way to have no noise reduction thus making it possible for me to use SMDegrain later?

[fvisagie]

I haven't used this mode, but your best bet is probably the "Noise Bypass / Denoising" section in the documentation. The following seems to be key for your objective:
Important: You must use a non-zero value in one of EZDenoise, EZKeepGrain or NoiseProcess or no noise processing will occur

Cheers,
Francois

[aldix]

set tr1 and tr2 to 0

[Trademark]

The original post seriously needs example scripts so it can help newer people like me..

I'm trying to use QTGMC with AVSProxy for AVIDemux. I took all the right steps to install my plugins for the deinterlacer, but I'm stuck on the scripting part of it, what am I doing wrong?



AviSource(C:\Users\Owen\Desktop\deinterlacetest.avi)
QTGMC(Preset="slower")

[LigH]

The name of the AVI file is a string literal and belongs between double quotes.

I would assume that the error message would have given a hint.

[manono]

The original post seriously needs example scripts so it can help newer people like me.
No it doesn't, as your problem has nothing to do with QTGMC. This filter is among the best documented of all AviSynth filters and there are examples galore in the included doc. As LigH wondered, didn't you get an error message? Next time open a script in VDub to test and it'll tell you something helpful if there's something wrong with the script.

Anyway, as LigH also suggested, add quotation marks:

AviSource("C:\Users\Owen\Desktop\deinterlacetest.avi")
QTGMC(Preset="slower")

And maybe it's OK, but I never do anything at all from the desktop.