QTGMC Deinterlacing Script (v3.32)

[Reel.Deel]

RGTools requires AviSynth 2.6 (latest Alpha 5 (http://forum.doom9.org/showthread.php?t=168764) or Set's latest MT version). It also requires the Microsoft Visual C++ 2012 Redistributable Package (vcredist_x86.exe) (http://www.microsoft.com/en-us/download/details.aspx?id=30679), also be aware that if you use RGTools you cannot process YUY2 with QTGMC (it only works with planar colorspaces; it does have the 'planar' parameter but it's only so it won't break the script).

For MaskTools2, take a look at the updated version by the awesome tp7 (https://github.com/tp7/masktools/releases). (same requirements as RGTools)

[jriker1]

RGTools requires AviSynth 2.6 (latest Alpha 5 (http://forum.doom9.org/showthread.php?t=168764) or Set's latest MT version). It also requires the Microsoft Visual C++ 2012 Redistributable Package (vcredist_x86.exe) (http://www.microsoft.com/en-us/download/details.aspx?id=30679), also be aware that if you use RGTools you cannot process YUY2 with QTGMC (it only works with planar colorspaces; it does have the 'planar' parameter but it's only so it won't break the script).

For MaskTools2, take a look at the updated version by the awesome tp7 (https://github.com/tp7/masktools/releases). (same requirements as RGTools)

Check and check. Though I am using YUY2 colorspace. Not sure if YV12 would cause the script to execute or not and it's crashing because of YUY2. Really trying to get this 3 and a half hour YUY2 script to deinterlace before going to try some of my YV12 content with other plugins.

JR

[Reel.Deel]

For YUY2, you should really stick to the original plugins. You also have to use mt_masktools-25.dll (for YUY2). Other than that I don't really know.

[StainlessS]

Tis a shame that some feel the need to drop support for YUY2, it was I think supported before YV12 in Avisynth
and some older filters only work in YUY2 not planar, also Layer is YUY2 or RGB32 only in v2.6.

EDIT:
It's annoying having to swap colorspaces multiple times when some filters only support a subsection of possibilities.
YUY2 is a tad more irksome to code for, but not so very much.

[Reel.Deel]

To be fair, those who dropped YUY2 do support YV16. Thanks to YUY2 there's ugly hacks like interleaved2planar/planar2interleaved (mainly all of kassandro's plugins) but MVTools2 is also affected by this ugly hack (hence the planar parameter).

[jriker1]

OK, I'm on 2.6 of AVISynth and can go back to 2.5.8 but will revert to the original QTGMC files but here's a funny one. I'm using MagicYUV to try encoding and Virtual dub crashes because of the lagarith module. Huh?

Is this all happening because of AVISynth and the plugins or should I be running this script with something other then VDub?

Even though I can't use it (won't open in Premiere Pro), also going to try using HuffyUV as it never seemed to crash for me with YUY2. See if in this case if it does or not.

EDIT: With original files still crashes. Even faster. Like 2 minutes in. Again complaining about Lagarith for whatever reason. Uninstalling it..

Edit: Uninstall lagarith now can't do anything. Darnit. Now I know what's kind of going on. When I originally imported the video into the system I did so using Lagarith. Guess I'm going to have to install Lagarith again, open the file itself in VirtualDub, and do a fast recompress in another codec before I can continue.

JR

[Reel.Deel]

As I've been away, could someone update me on all the filter updates that have affected QTGMC recently? I noticed that masktools-2.6 doesn't work anymore. And now there is this RGTools. Anything else?

It would be nice to update the first post to be correct for current usage.

Wow, I thought about sending you a PM about this very thing.
Since QTGMC is probably the most known/used AviSynth script it would be nice to see the first post updated with current information. Also, how about making v3.33 the official version? Since it's released I've used it exclusively and have not had any problems.

Here's all of the changes that I know of:

AddGrainC (http://forum.doom9.org/showthread.php?t=111849) - added support for additional planar colorspaces.

dfttest (http://forum.doom9.org/showpost.php?p=1386559&postcount=3) - has been updated a few times, it now supports YV16/YV24 and also includes "-Vit-'s threading fixes".

EEDI3 v0.9.2 (http://forum.doom9.org/showpost.php?p=1652667&postcount=476) - updated faster version.

MaskTools2 b1 (http://forum.doom9.org/showthread.php?t=169832) - a more "mature" version; works correctly with the latest avs 2.6. Read more here (https://github.com/tp7/masktools#masktools-2). A while back 06_taro (http://forum.doom9.org/showpost.php?p=1619023&postcount=649) released a fixed mt_masktools-26.dll since the official one no longer works with the latest Alpha 4/5.

MVTools v2.6.0.5 (http://forum.doom9.org/showpost.php?p=1386559&postcount=3) - has not been updated in a while so I'm sure you know about this one. In your plugins package you're distributing v2.5.11.2, the latest official version is v2.5.11.3

nnedi3 for AviSynth 2.6 (http://forum.doom9.org/showthread.php?t=170083) - supports all planar colorspaces.

nnedi3ocl (http://forum.doom9.org/showthread.php?t=169766) - OpenCL rewrite of nnedi3. I know it's not used in QTGMC but would be nice if you could :)

RGTools (http://forum.doom9.org/showthread.php?t=169832) - a rewrite of RemoveGrain, Repair, and VerticalCleaner all in one plugin. It also includes some other filters not used in QTGMC.

Regarding QTGMC, I think that covers everything you might of missed...

Notes:
Your plugin pack using the updated Avisynth 2.6 interface (http://forum.doom9.org/showpost.php?p=1568142&postcount=1126) is now outdated, it does not work with the latest 2.6; last version it worked with was 2.6 Alpha 3.

**Wishful thinking :p**
- a parameter to let the user pass a custom pre-filtered clip for better motion analysis

[fvisagie]

I have some YUY2 footage. AVISynth script works fine in Windows XP 32-bit. But on my Windows 7 box 64-bit no joy. Note using AVISynth 2.5.8 32-bit on both and both "work", just not with YUY2 colorspace.

It seems you haven't returned to this issue in the ensuing discussion. In case it's still relevant and in case I forget by the time you return to this:

What is your script, specifically what media file filter do you use? The problem you describe could very easily result from a. using a platform-dependent filter (e.g. DirectShow() and derivatives, and I think the AVI filters use VfW system codecs), and b. having platform differences. Some potential platform differences that come to mind are XviD (can take over some colourspace decoding), Intel... Indeo(?) codecs I think, Helix codec, etc.

[Music Fan]

Edit: Uninstall lagarith now can't do anything. Darnit. Now I know what's kind of going on. When I originally imported the video into the system I did so using Lagarith. Guess I'm going to have to install Lagarith again, open the file itself in VirtualDub, and do a fast recompress in another codec before I can continue.

JR
In my opinion, Lagarith is not the problem. On my pc, it doesn't crash, whatever I use it in yuy2 or YV12.
It works in Virtual Dub with Windows XP 32 bit and Win 7 pro 64.

[qwerty1983]

Sorry to interrupt the discussion. I want to clarify some things about the denoising aspects of qtgmc.

I read that the script denoises by default using MDegrain.

1) If i use QTGMC(NoiseProcess=1, NoiseRestore=0.0, Denoiser="dfttest",Sigma=4.0 ) does this mean that :
a) dfttest will do some additional denoising (to that done by MDegrain)
or b) dfttest will replace MDegrain as a denoiser ??


2) If i use QTGMC(NoiseProcess=2, GrainRestore=0.5, NoiseRestore=0.1, Sigma=2.5 ) does this mean that :
MDegrain will do the default denoising. The amount of noise will be calculated (by what filter? fft3dfilter? can i use dfttest?) but not removed. Then some of the noise is restored back. Is this correct?

3) I don't understand this example from the qtgmc 'read me' :
QTGMC(NoiseProcess=1, NoiseRestore=1.0, Sigma=2.0 ) # Retain *all* the noise / fine detail of a less noisy source
What's the point of denoising and then restoring 'all' the noise back again? What does 'all' mean? I'm missing something here.

[jriker1]

In my opinion, Lagarith is not the problem. On my pc, it doesn't crash, whatever I use it in yuy2 or YV12.
It works in Virtual Dub with Windows XP 32 bit and Win 7 pro 64.

That's good or bad to hear. :)

My Windows 7 64-bit system is pretty clean. I use it strictly for video editing so only have the following installed:


xiph.org for FLAC support
Virtual Clone Drive
Rovi TotalCode for Adobe Premiere
Adobe Premiere CC
QuickTime 7
Neat Video for Premiere
Haali Media Splitter
Microsoft Expressions Encoder 4

And now AVISynth 2.6 (32-bit), Lagarith and MagicYUV Codec.

For reference opening my AVI (Lagarith) directly in VirtualDub and selecting Fast Recompress, setting the Codec to MagicYUV caused a crash 35% into the conversion again with an "Assumption" as VirtualDub likes to say, that Lagarith caused it.

So the above wasn't even using AVISynth. Hmmm. For reference my YUY2 based script is fairly simple when I am using it.

SetMTMode(5,8)
import ("D:\Program Files (x86)\AviSynth 2.5\plugins\QTGMC-3.32.avsi")
SetMTMode(2)
AVISource("D:\Conversion\2 - Working\CH\CH.avi", pixel_type="YUY2")
AssumeTFF()
QTGMC(preset="Very Slow")

The script I don't have issues with which is in the YV12 colorspace plus mpeg2 based is a bit different:

SetMTMode(5,8)
LoadPlugin("D:\Program Files (x86)\AviSynth 2.5\plugins\dgdecode.dll")
import ("D:\Program Files (x86)\AviSynth 2.5\plugins\QTGMC-3.32.avsi")
SetMTMode(2)
V = MPEG2Source("D:\Conversion\2 - Working\HHGTTG\HHGTTG-4.d2v")
A = NicAC3Source("D:\Conversion\2 - Working\HHGTTG\HHGTTG-4 T80 2_0ch 192Kbps DELAY 0ms.ac3")
AudioDub(V, A)
AssumeTFF()
QTGMC(preset="Very Slow")

I have never seen the mpeg2 one fail on me though my conversions are shorter in length.

Edit. I am also as mentioned above on AVISynth 2.6 32-bit now and strictly the base plugins. Here is what is in the folder:

AddGrainC.dll
dfttest.dll
DGDecode.dll
EEDI2.dll
eedi3.dll
FFT3DFilter.dll
mt_masktools-25.dll
mvtools2.dll
NicAudio.dll
nnedi.dll
nnedi2.dll
nnedi3.dll
QTGMC-3.32.avsi
RemoveGrainSSE2.dll
RepairSSE2.dll
SSE2Tools.dll
TDeint.dll
VerticalCleanerSSE2.dll
yadif.dll


JR

[Music Fan]

I'm not an avisynth specialist and it's hard to tell what's going on with your pc, but all I can say is that I use neither MTMode nor 64 bit avisynth, only 32 bit. My avisynth version is 2.60 alpha 5.

[LigH]

Wrong order:

SetMTMode(5,x)
# no calculations to be parallelized, just Import or LoadPlugin
SetMTMode(2)
# Source plugins possibly not compatible with mode 2
# other filters to be parallelized


Right order:

# no calculations to be parallelized, just Import or LoadPlugin
SetMTMode(5,x)
# Source plugins possibly not compatible with mode 2
# maybe some Preroll to linearize frame requests
SetMTMode(2)
# other filters to be parallelized


Consider using less than 8 threads, because each thread needs its own amount of RAM. Also be aware that EDI inside QTGMC forks threads too.

[-Vit-]

Wow, I thought about sending you a PM about this very thing.
Since QTGMC is probably the most known/used AviSynth script it would be nice to see the first post updated with current information. Also, how about making v3.33 the official version? Since it's released I've used it exclusively and have not had any problems.

Here's all of the changes that I know of:...
Thanks for all that info, when I get the time I'll update everything.

**Wishful thinking :p**
- a parameter to let the user pass a custom pre-filtered clip for better motion analysis
- a parameter to disable chroma processing in QTGMC (useful in certain scenarios)
First one you can do already. In the docs the section "External Linkage" lists the clips/vectors that you can either inject into or extract from QTGMC innards. Here's how to send in your own filter clip:

QTGMC_srcClip = your filtered clip
QTGMC( PrevGlobals="Reuse" )

Second point - what would be in the chroma channel of the output? Zeros, 128 (or whatever you need for greyscale, can't remember - rusty), undefined? Any of that you can do yourself. Or do you want speed? Then use the ChromaMotion, ChromaEdi, ChromaNoise settings.

[jriker1]

I'm not an avisynth specialist and it's hard to tell what's going on with your pc, but all I can say is that I use neither MTMode nor 64 bit avisynth, only 32 bit. My avisynth version is 2.60 alpha 5.

Thanks for the info. I believe if I went your path the conversion would work as I have tried eliminating the MT commands which I guess makes AVISynth run without multi threaded support and after a couple hours it was still running. It's a trade off of 1 1/2 days of deinterlacing vs 9 hours with MT support enabled. Large investment in time to tie up the computer for the first step of my conversion to digital for this content. Guess there is the argument that a slow conversion is better than one that doesn't work at all, but that's really slow.

There is also an old plugin for Premiere Pro x64 that is out there to allow import of AVS files. Requires a 64-bit version of AVISynth and all it's plugins to work though. Not sure how it works with Premiere CC version or if it would help resolve my problem. Guess I'd like to see if I could do the same thing but without VirtualDub in the equation if things would work.

JR

[-Vit-]

I read that the script denoises by default using MDegrain
Yes and no. MDegrain is used to perform a binomial temporal blur that is the core of the algorithm. The primary use of MDegrain in this script is to deinterlace, not to denoise. But of course it does denoise as well. Which is why default QTGMC does some denoising. You can slightly control the final pass of MDegrain denoising by setting TR2=1,2 or 3, which in some cases might be enough if all you want is a stronger or weaker denoise.

However, for more precise control over noise, QTGMC provides a completely separate set of noise controls. In fact the noise in your clip can take a completely separate processing path than the clean underlying image. These denoising controls do not use MDegrain, they use fft3dfilter or dfttest.

Retain *all* the noise / fine detail of a less noisy source
What's the point of denoising and then restoring 'all' the noise back again?
(1) A light grain is sometimes desirable, part of the intent of a video and you do not want to remove it. (2) Even good denoising sometimes removes fine detail, which you probably don't want to lose.
Because QTGMC's core process (using MDegrain) will always denoise, that means you are guaranteed to lose grain and some fine detail using the defaults. So to retain those features you can denoise at the start, keep the removed noise, do the QTGMC processing, then add the noise back after. And of course I use the word noise to mean grain, fine detail and noise.

Whether you want to do this or not will depend on your source (graininess, detail, noise) and your personal preference.

1) If i use QTGMC(NoiseProcess=1, NoiseRestore=0.0, Denoiser="dfttest",Sigma=4.0 ) does this mean that :
a) dfttest will do some additional denoising (to that done by MDegrain)
or b) dfttest will replace MDegrain as a denoiser ??

(a) sort of. First dfttest will denoise, then MDegrain will do it's job for deinterlacing on the cleaned image.



2) If i use QTGMC(NoiseProcess=2, GrainRestore=0.5, NoiseRestore=0.1, Sigma=2.5 ) does this mean that :
MDegrain will do the default denoising. The amount of noise will be calculated (by what filter? fft3dfilter? can i use dfttest?) but not removed. Then some of the noise is restored back. Is this correct?

No, fft3dfilter will be used to denoise, set Denoiser="dfttest" if you want that instead. The amount of noise to remove is the Sigma=2.5. Some of the noise /grain is restored at the end of the script (so the MDegrains don't wipe it out).

Consider using the EZDenoise or EZKeepGrain settings.

[Boulder]

Thanks for the info. I believe if I went your path the conversion would work as I have tried eliminating the MT commands which I guess makes AVISynth run without multi threaded support and after a couple hours it was still running. It's a trade off of 1 1/2 days of deinterlacing vs 9 hours with MT support enabled. Large investment in time to tie up the computer for the first step of my conversion to digital for this content. Guess there is the argument that a slow conversion is better than one that doesn't work at all, but that's really slow.

There is also an old plugin for Premiere Pro x64 that is out there to allow import of AVS files. Requires a 64-bit version of AVISynth and all it's plugins to work though. Not sure how it works with Premiere CC version or if it would help resolve my problem. Guess I'd like to see if I could do the same thing but without VirtualDub in the equation if things would work.

JRWhat comes to QTGMC crashing when using SetMTMode, I'd say that you are asking for trouble :)

I'm always repeating myself but the old trick of splitting the lossless encode into x parts, x being the number of threads, and then joining them in the AVS script for the final encode is a non-crashing workaround. In fact, it is also faster than trying to use the same number of threads with QTGMC.

[StainlessS]

@Boulder,
I dont use MT versions, but I do sometimes do the same as you suggest (but I only do it in two parts [on dual core]).

EDIT: Dont think I could be bothered with all that MT nonsense.

[Boulder]

Yes, it speeds up processing in all cases where you can utilise a multicore processor. Squeezing out all I can, I've ended up piping to 64-bit ffmpeg with three parallel encodes, it leaves a nice headroom for other activies on a quad-core.

[Music Fan]

I believe if I went your path the conversion would work as I have tried eliminating the MT commands which I guess makes AVISynth run without multi threaded support and after a couple hours it was still running. It's a trade off of 1 1/2 days of deinterlacing vs 9 hours with MT support enabled.
When you don't use MT, does your CPU reach 100 % (or the same CPU consumption than with MT, admitting MT does not reach 100 %) ?
If the CPU works less without MT (or does not reach 100% even with MT), there is a trick to use CPU at 100 % : divide virtually the video (with trim function) in 2 or 3 parts, each one in a single avs script, and encode them together in lossless (Lagarith for instance) with Virtual Dub.
The scripts are exactly the same, except the trim line (the part of the video each script concerns).
Then join the files (possibly with avisynth) and encode the complete file in x264 or what you want.

edit : Boulder was faster than me ;)

[jriker1]

When you don't use MT, does your CPU reach 100 % (or the same CPU consumption than with MT, admitting MT does not reach 100 %) ?
If the CPU works less without MT (or does not reach 100% even with MT), there is a trick to use CPU at 100 % : divide virtually the video (with trim function) in 2 or 3 parts, each one in a single avs script, and encode them together in lossless (Lagarith for instance) with Virtual Dub.
The scripts are exactly the same, except the trim line (the part of the video each script concerns).
Then join the files (possibly with avisynth) and encode the complete file in x264 or what you want.

edit : Boulder was faster than me ;)

With MT set to (5,8) I am utilizing about 70% of my CPU. Without any MT settings 14%.

Sounds like, if I'm reading this right, the suggestion is to have multiple AVS files that contain a piece of the video using trim. Run each one separately thru separate instances of VirtualDub until I max out my CPU, and then splice them back together at the end?

JR

[Boulder]

Yes, that's how you do it if you want to avoid problems.

[jriker1]

Thanks. One last question on this topic and off I go to try. If I want to break it into 30 minute segments (3 hour footage, 6 running at once), what would that look like. The trim command seems to be based on frames, not minutes, which makes sense as more precise, so not sure how this should be calculated. Original footage for reference is 29.97fps. Will be 59.94fps when done.

import ("D:\Program Files (x86)\AviSynth 2.5\plugins\QTGMC-3.32.avsi")
AVISource("D:\Conversion\2 - Working\CH\CH.avi", pixel_type="YUY2")
AssumeTFF()
QTGMC(preset="Very Slow")

If someone could provide me an example of the AVS file for the first 30 minutes and then for the second 30, and as an example from that point to the end, sure I can figure out the rest in between. And assuming Trim does both audio and video and not just video. Note assuming this video has a total of 385937 frames per mediainfo. Guessing conceptually something like this:

0,64322
64323,128645
128646,192967
192968,257289
257290,321612
321613-end

Thanks.

Edit. Would that be:

Trim(0,64322) in the first script
Trim(64323,128645) in the second script
Trim(321613,0) in the last?

Edit2: A lot of people say when using AVISource("D:\capture\clipo1.avi")+AVISource("D:\capture\clip02.avi") type commands they always have audio sync issues. Is this generally a problem and is there a better way to merge the resulting clips? Read about a double plus (++) for an aligned split but not sure if that works better.

JR

[Music Fan]

Would that be:

Trim(0,64322) in the first script
Trim(64323,128645) in the second script
Trim(321613,0) in the last?
Yes, it seems ok. You can also have a few pictures in common in 2 consecutive scripts to see clearly that no frame is missing after encoding, for example 0,64322 - 64320,128645 and cut these frames with trim when joining files.
Make a test (open the scripts in VDub) before encoding.
For a 14% CPU usage, you can make 7 or 8 scripts.

Edit2: A lot of people say when using AVISource("D:\capture\clipo1.avi")+AVISource("D:\capture\clip02.avi") type commands they always have audio sync issues. Is this generally a problem and is there a better way to merge the resulting clips? Read about a double plus (++) for an aligned split but not sure if that works better.
In my opinion, the simplest solution is to treat separately the sound, just mux it with video after joining parts.

[hello_hello]

jriker1,
I also use the "split encodes and run them simultaneously" method rather than MT AVISynth. I don't use intermediate lossless files though. Mostly I don't think there's any advantage to doing so. I just encode with the x264 encoder and append the encoded video. If that's what you do, and in case nobody's mentioned it, it pays to add --stitchable to the x264 command line (even when the same encoder settings are used). It'd be very disappointing to spend hours and hours encoding only to discover you can't append the encoded video.

I agree it's easier to treat the sound separately (as a single file), encode the video in sections, append the sections and add the whole audio track as you normally would. MKVMergeGUI makes it pretty easy, assuming you're happy with MKV. You can split the encoding job into individual scripts exactly as you described and append the encoded video when it's done. ie

Trim(0,64322) in the first script
Trim(64323,128645) in the second script
.......
Trim(321613,0) in the last

I tend split the video on scene changes. That way if the first or last frame of an encoded section looks a little different (a little less noise removal etc) it's not in the middle of a scene. Not that I think it's likely to, but just in case. Plus that's where the encoder would most likely put a keyframe anyway. MeGUI has an AVS Cutter under it's Tools menu which lets you add multiple trims to a script with a preview (then I make multiple copies of the script and delete all but one of the trims from each script in order to split the encoding job between them), or there's a standalone version (http://www.videohelp.com/tools/AVSCutter), or there's AvsPmod (http://www.videohelp.com/tools/AvsP).

[GMJCZP]

@ jriker1

In my experience it is safer not to put the Trim snugly, ie, put a gap of some frames, because if you use scripts like Stab() will give you problems.

Edit: the solution of Music Fan is the more solid for the audio.

[Music Fan]

In my experience it is safer not to put the Trim snugly, ie, put a gap of some frames, because if you use scripts like Stab() will give you problems.
Your sentence is confusing, I guess you mean it's better to put a gap of some frames (as I suggested).

[GMJCZP]

Your sentence is confusing, I guess you mean it's better to put a gap of some frames (as I suggested).

For example:

Trim(0,100)
Trim(90,210)
Trim(190,310)
Trim(290,0)

Overlap the frames.

[Music Fan]

Yes, I understood, but I guess you mean that it's a good solution and not a bad solution.

[GMJCZP]

Indeed, Stab() or other more complex function with motion estimation possibly malfunction if you put consecutive frames. Is more safe overlap the frames, it is annoying but "no pain, no gain".

[jriker1]

@ jriker1

In my experience it is safer not to put the Trim snugly, ie, put a gap of some frames, because if you use scripts like Stab() will give you problems.

Edit: the solution of Music Fan is the more solid for the audio.

That sounds challenging. So is the thought to open the files in an NLE and overlap them until they line up?

I extracted the audio with virtualdubmod from the original AVI. I think instead of running this thru again splicing them all together and re-encoding yet again, will just load each individual files into Adobe Premiere pro and line them up, add the separate audio into the timeline. Since I need to run NeatVideo on it probably makes more sense and eliminate the time for an extra encoding round to merge them all together even if lossless.

JR

[GMJCZP]

That sounds challenging. So is the thought to open the files in an NLE and overlap them until they line up?

I extracted the audio with virtualdubmod from the original AVI. I think instead of running this thru again splicing them all together and re-encoding yet again, will just load each individual files into Adobe Premiere pro and line them up, add the separate audio into the timeline. Since I need to run NeatVideo on it probably makes more sense and eliminate the time for an extra encoding round to merge them all together even if lossless.

JR

The only thing to worry is that using Trim the result is identical to a single process.
I guess in Adobe Premiere (I've never used) in the edit line (per frame) look for cuts for each video.
If you keep separate audio should have no problems.

[hello_hello]

Would it be normal for QTGMC to be slower when working with non-mod16 video than if it's mod16?

I've been re-encoding some old video which is mostly 720x576 and running QTGMC in progressive mode to clean it up a little.
QTGMC(InputType=1, Ezdenoise=1)

I noticed every so often one of the encodes would take much longer. A little investigating seems to indicate it's due to a mod4 width. If I resize to mod16 in the script before QTGMC the speed goes back to normal, but that's not ideal as I'm also resizing after QTGMC.

For the record, I found the culprit. It was mt_masktools-26.dll causing the speed problem. I ran a bunch of test encodes on a video with a mod4 width and a mod2 height. I ran all the encodes using mt_masktools-26.dll (dated 31/12/2010) from the "QTGMC 32-bit Plugins.zip" file and I repeated a couple of them with mt_masktools-26.dll from "QTGMC 32-bit Plugins [Vit-Mod].zip". I didn't repeat them all as it appears speed-wise those two versions of mt_masktools-26.dll are the same.
I also ran all the encodes with mt_masktools-26.dll (dated 09/03/2013) from here (http://forum.doom9.org/showpost.php?p=1619023&postcount=649), and also with masktools2.dll from here (https://github.com/tp7/masktools/releases) (x86 version).

These are the speed results (speed noted at the same point in the encode each time courtesy of MeGUI). I used the following in the script.

QTGMC(InputType=1, EzDenoise=1)

Avisynth 2.6.0.4:
mt_masktools-26.dll (dated 31/12/2010) - 3.88fps
mt_masktools-26.dll (dated 09/03/2013) - 6.00fps
masktools2.dll - 8.50fps

Avisynth+ 2.6.0.5:
mt_masktools-26.dll (dated 31/12/2010) - 3.97fps
mt_masktools-26.dll (dated 09/03/2013) - 6.09fps
masktools2.dll - 8.71fps

After resizing the video to mod16 (width and height) the speed difference narrowed quite a bit. Slowest was still the combination of Avisynth 2.6.0.4 and mt_masktools-26.dll (dated 31/12/2010) at 7.34fps. Fastest was Avisynth+ 2.6.0.5 and masktools2.dll at 8.90fps

[hello_hello]

For example:

Trim(0,100)
Trim(90,210)
Trim(190,310)
Trim(290,0)

Overlap the frames.

I'd be tempted to disagree with that. Surely it depends on the order things are placed in the script?

For instance if a video has 10,000 frames and you use the following in a script:

QTGMC()
Trim(500,9000)

Wouldn't the frames before number 500 and after 9000 still be used by QTGMC if required?

Or as another example, what if I add IVTC to a script like this, then follow it with trim:

tfm(order=-1).tdecimate()
Trim(500,9000)

Obviously the frame count and frame numbers after tdecimate are going to be dramatically different, but my argument would be Trim() is applied after the filter proceeding it. There's no need for overlapping frames to get tdecimate to work correctly, so I'm not sure why you'd need them for any other filter.
It'd be the same principle as navigating when playing video. If you don't stop on a keyframe the decoder might need to decode the frames around the one you requested in order to display it correctly, but that's what it'll do.

Well.... it's possible when it comes to a filter such as SRestore which sometimes works on a "pattern" of frames, it might pay to overlap them, but I've never seen the need for QTGMC. Someone may be able to correct me if I'm wrong but it seems fairly obvious to me when I add QTGMC to a script in MeGUI's script creator and hit the preview button the result is QTGMC doing it's thing correctly regardless of the frame I've been previewing at the time. I can navigate to any random frame in the video and see QTGMC is working as it should.

Anyway.... I've used QTGMC followed by Trim() lots and lots of times without overlapping frames, encoded with the x264 encoder and appended the encoded video without a lossless intermediate file in between and I've never noticed any ill-effects as a result. I always put the trim points on scene changes, but that's mainly because it's where the encoder would most likely place a keyframe anyway, and you're not forcing it to use one in the middle of a scene by splitting the video there.

[Boulder]

IIRC Avisynth is smart enough to provide frames outside of your Trim statement if they are requested by a filter. Hence there should be no need for any overlap when trimming.

[fvisagie]

For the record, I found the culprit. It was mt_masktools-26.dll causing the speed problem.

That's quite a significant difference. Thanks for looking into this, and esp. reporting on it.

[poisondeathray]

Hmmmm just some quick preliminary tests, but I'm getting different (quality) outputs with those different mt_masktools-26.dll 's (that you can see with naked eye , not microscopic ssim differences)

[hello_hello]

Hmmmm just some quick preliminary tests, but I'm getting different (quality) outputs with those different mt_masktools-26.dll 's (that you can see with naked eye , not microscopic ssim differences)

De-interlacing or running QTGMC in progressive mode as I was?
I didn't actually compare the output myself. I'll run some more test encodes and look at them this time.

I forgot to mention earlier.... I'm using QTGMC 3.33d (mod). It's the version from here (http://www.mediafire.com/?gpil7zkz24lq4). I probably should try a different version of QTGMC also, to make sure that doesn't change anything.

Edit 1: I tried a couple of my earlier encodes with QTGMC 3.32 and it didn't make any difference to the speed.

Edit 2: I ran a few more test encodes to compare the output. QTGMC 3.33d (mod) again. If there's any difference between the mt_masktools-26.dll versions I tested earlier and masktools2.dll in respect to quality, I'll confess I can't see it. I tried the three of them with QTGMC in progressive mode again QTGMC(InputType=1, EzDenoise=1) and also with an interlaced source QTGMC(). The output was lossless huffyuv/AVI. Both source videos were mod16.
I couldn't see any difference in the output and the file sizes were identical down to the byte. That's a bit of a relief really, :) but so far I can't replicate the quality differences you're seeing.

[poisondeathray]

Actually not QTGMC. It was just some quick tests since I got so excited by your % FPS gains (!!), but I noticed right away something was up. At first I thought it was something stupid like source filter issues, but it's not. I'll try to do some more systematic tests and on different computers to see if there is a real issue. Hopefully it was a false alarm

[jriker1]

Few items. Thank you for the suggestion to run multiple scripts. Brilliant. Ran thru with no issues so guess AviSynth MT is out for me even though it eliminates the need to manually create a bunch of scripts, however in the end this process works for me and enabling MT doesn't.

Second and third items. If I'm just doing:

QTGMC(preset="Very Slow")

Is there any denoising going on? I tend to do this part today in NeatVideo so hate to think I'm denoising twice.

And is there any changes going on from the original YUY2 colorspace during the encoding process?

Thanks.

JR

[GMJCZP]

I'd be tempted to disagree with that. Surely it depends on the order things are placed in the script?

For instance if a video has 10,000 frames and you use the following in a script:

QTGMC()
Trim(500,9000)

Wouldn't the frames before number 500 and after 9000 still be used by QTGMC if required?

I think you misunderstood my post, if you use two or four cores or how many are there must always be a first Trim(0, x) and a final Trim(y, 0). Furthermore Trim should go at the end of script. How will you divide a cake if it is not cooked?

Based on the advice of Boulder, I seems a good idea to use the number of Trim equal to the number of cores. I use VirtualDubMod to encode (lossless) and I usually put in "Process thread priority" "Lower", so if you are using four Trims will be four open instances of VirtualDubMod for each script and Trim associated. I do not say that if someone has eight cores to be used eight Trims, but maybe six is good.

Edit: so far I have processed with overlap of 200 frames, but a matter of personal preferences.

Edit 2: @Boulder: Stab(), at least, is not usable without gap in frames (bad luck only for me?). This function needs some frames to better estimate the stabilization, something like MDegrain and backward vectors or SCSelect (RemoveDirt).

[hello_hello]

I think you misunderstood my post, if you use two or four cores or how many are there must always be a first Trim(0, x) and a final Trim(y, 0). Furthermore Trim should go at the end of script. How will you divide a cake if it is not cooked?

There's lots of ways to add Trim() to a script, but in the context of this discussion where a single Trim() is being used to split an encode over multiple copies of a script, it makes sense to put it at the end.

I don't think I misunderstood your post. For example, when de-interlacing a 10,000 frame video with QTGMC while splitting the encode between two scripts, I'd put something like this at the end of the first one:
Trim(0,4999)
and this at the end of the second:
Trim(5000,0)

No overlap required.

Based on the advice of Boulder, I seems a good idea to use the number of Trim equal to the number of cores.

Possibly if you're processing with QTGMC and outputting lossless intermediate files, but I rarely do that myself. I process with QTGMC and encode directly with the x264 encoder, then append the encoded video when it's done. "Overlapping" wouldn't be practical. The x264 encoder tends to use a bit of CPU power, so I find QTGMC's standard settings + x264 keep a dual core pretty busy, and two simultaneous encodes will keep a quad core busy. It depends on the resolution and QTGMC/x264 settings though.
Currently I'm using QTGMC to de-interlace a PAL DVD with the default settings and a bit of noise removal, it's being resized to 640x480 and gradfun3() is also included. I've split the encode into two scripts with Trim() and I'm using the x264 slow preset. With both encodes running simultaneously, CPU usage (quad core) is sitting on around 85%.
I could possibly have split the script into three parts and run them together but it leaves me a bit of CPU headroom for other things and I'm not in a major hurry.

[GMJCZP]

I'll try to explain better. It is using a number of Trim so you get the same number of identical scripts, except the Trim cited going to the end of them. The scripts will run in parallel, simultaneously.

Now, if you previewing every piece of video and are equal or not usases Trim then no problem, encodes everything at once but considering that SetMTMode is risky.

[poisondeathray]

Actually not QTGMC. It was just some quick tests since I got so excited by your % FPS gains (!!), but I noticed right away something was up. At first I thought it was something stupid like source filter issues, but it's not. I'll try to do some more systematic tests and on different computers to see if there is a real issue. Hopefully it was a false alarm

I can't reproduce this difference now.... I can't figure out what caused it or what went wrong. Sorry for the false alarm

[hello_hello]

I'll try to explain better. It is using a number of Trim so you get the same number of identical scripts, except the Trim cited going to the end of them. The scripts will run in parallel, simultaneously.

I'm referring to making identical copies of the same script while adding trim to the end of each, with trim being added so each script encodes a different section of the video. Each script is then encoded at the same time.

If we're not talking about the same thing, then I guess I'm missing the obvious, but I don't agree it's necessary for the trims at the end of each script to "overlap".

[fvisagie]

I can't reproduce this difference now.... I can't figure out what caused it or what went wrong.

So, also from your testing,
masktools2.dll from here (https://github.com/tp7/masktools/releases)
seems good to go?

[hello_hello]

I'm using it now (masktools2.dll from here (https://github.com/tp7/masktools/releases)). No difference quality-wise compared to the various flavours of mt_masktools-26.dll.... at least when it came to my limited test encodes earlier.... but this PC has been running two encodes simultaneously with QTGMC and masktools2.dll for most of the day today without any issues so far.

[poisondeathray]

So, also from your testing,

seems good to go?


If I find a difference I'll report it. That newer dll had been posted a year and half ago, someone would have reported something already by now if something was wrong so I think it's ok

[jriker1]

Few items. Thank you for the suggestion to run multiple scripts. Brilliant. Ran thru with no issues so guess AviSynth MT is out for me even though it eliminates the need to manually create a bunch of scripts, however in the end this process works for me and enabling MT doesn't.

Second and third items. If I'm just doing:

QTGMC(preset="Very Slow")

Is there any denoising going on? I tend to do this part today in NeatVideo so hate to think I'm denoising twice.

And is there any changes going on from the original YUY2 colorspace during the encoding process?

Thanks.

JR

Hey folks, in addition to my above inquiries when you get a chance, has anyone written something already to automate creation of multiple avs files for "emulating" multi-threading? Doesn't seem to hard. Would imagine except for those looking for visual scene changes to trim on, it would be a matter of:


Create an AVS file to run the whole file singularly
Program asks for the file.
Program asks how many "threads" to run.
Program reads in number of frames on video say with MediaInfo
Program creates copies of original AVS file with -1, -2, -3 at the end based on how many threads were requested.
Appropriate trim statements added to each file.


Batch file is then created to launch all the VDub instances could be added but gets more complicated with Codec selection and stuff. Could just launch each VDub and the AVS file and then the user has to pick the rest and Save AVI. Curious if this exists as I could use something like this with the quantity of videos I will need this for so will write it if it's not here. GUI or command line. Not sure yet.

JR

[Groucho2004]

Batch file is then created to launch all the VDub instances could be added but gets more complicated with Codec selection and stuff. Could just launch each VDub and the AVS file and then the user has to pick the rest and Save AVI.
Why do you need VDub? What is your target format?