QTGMC Deinterlacing Script (v3.32)

[Groucho2004]

Originally Posted by BeNooL

Pro-tip: you can remux the partial output before a crash occurred.

- remux partial output with mkvmerge and cutoff 5-10Mb to get a clean cut
- get the number of frames
- trim original avs script of the same amount
- resume encoding of remainder part of video
(- repeat if new crash occurs)

Once you have all your parts, append them to each other in mkmerge. Voila.

That's how I dealt with some problematic video streams.

That's the most bizarre approach to dealing with the problems that Avisynth MT with QTGMC present.
I'm puzzled that someone would go through this ordeal instead of running the process single-threaded which would overall probably amount to the same time spent.

[Boulder]

What I usually do with QTGMC-related things is that I either 1) run the script with one thread and use cretindesalpes's MVTools build (in the Dither package) which includes internal multithreading or 2) run multiple one-thread encodes simultaneously and combine them later. SD sources seem to cope better with multithreaded QTGMC scripts than HD ones, probably due to memory usage issues.

[kypec]

Originally Posted by Groucho2004

That's the most bizarre approach to dealing with the problems that Avisynth MT with QTGMC present.
I'm puzzled that someone would go through this ordeal instead of running the process single-threaded which would overall probably amount to the same time spent.

My thoughts exactly. I still wonder why anyone dedicates so much time and effort to make MT mode usable with certain scripts when they could most likely had the encode finished in single-threaded rock-solid Avisynth in that equal time. I gave up on MT in AVS long ago and never had urge to cope with its instabilities again. Instead, I start my single-threaded script and let it run overnight or even two if necessary.

[BeNooL]

Originally Posted by Groucho2004

That's the most bizarre approach to dealing with the problems that Avisynth MT with QTGMC present.
I'm puzzled that someone would go through this ordeal instead of running the process single-threaded which would overall probably amount to the same time spent.

Originally Posted by kypec

My thoughts exactly. I still wonder why anyone dedicates so much time and effort to make MT mode usable with certain scripts when they could most likely had the encode finished in single-threaded rock-solid Avisynth in that equal time. I gave up on MT in AVS long ago and never had urge to cope with its instabilities again. Instead, I start my single-threaded script and let it run overnight or even two if necessary.

I'm equally surprised seeing these answers.

How is remuxing partial output to a new file lengthy ? It takes a matter of seconds/maybe couple of minutes if you are going for bluray sizes.
Getting frame number is a matter of reading a value (think mediainfo)
How hard is it to add Trim(0,x) at the end of an avs ?
Putting it back together is again a matter of 1 additional click in mkvmerge (select previous part, append instead of add)


Going single threaded on the other hand could easily slow things down by a factor of 3-5 times. Something like going from 4 hour encode to something in the range of 12-20 hours. I seriously believe you can do the above steps in less than 6-16 hours.
This is also only for dealing with the one problematic video stream out there that you only encounter maybe once a year. Not worth messing your plugins directory if it works fine 99% of the time.

Anyway this was a reply to Octo-Puss. I hope you are going to give

SetMTMode(5,6)

a go. it has been suggested to you twice.

[Octo-puss]

I did. And I said it kept crashing like Windows ME. I gave up, will see what happens in few months when AVS is hopefully out of alpha stage.

[Boulder]

The stability won't change a bit. The instability is due to the nature of the whole multithreading thingie, and that is out of the Avisynth developers scope. If you want to, you could try the SVP team's Avisynth build which does memory handling in a different way.

[Asmodian]

Originally Posted by BeNooL

I'm equally surprised seeing these answers.

I also always use non-MT Avisynth for QTGMC, it isn't that the cut+mux step takes a long time but that you do not know when it will need to be done without constantly checking it.

I like being able to set an encode up and when I come back in ~24h it is done, not crashed 2h into the encode and sitting idle for 22h.

Avisynth MT has been unstable, similar to the stability is has now, for years now.

[Groucho2004]

Originally Posted by BeNooL

I'm equally surprised seeing these answers.

How is remuxing partial output to a new file lengthy ? It takes a matter of seconds/maybe couple of minutes if you are going for bluray sizes.
Getting frame number is a matter of reading a value (think mediainfo)
How hard is it to add Trim(0,x) at the end of an avs ?
Putting it back together is again a matter of 1 additional click in mkvmerge (select previous part, append instead of add)

So, how do you know when it will crash? Are you sitting in front of the screen and wait for it? Do you have a crash alarm clock?

I'm just baffled thinking about what kind of mind set one must have to use a software setup that will inevitably crash at random and that needs wacky workarounds to produce results.

[Octo-puss]

I am quite afraid that what (theoretically, due to the crashes :P) would take say 30 hours to encode with MT version of AVS could take a week with the original. I need to use the PC at times you know

[BeNooL]

Originally Posted by Groucho2004

So, how do you know when it will crash? Are you sitting in front of the screen and wait for it? Do you have a crash alarm clock?

I'm just baffled thinking about what kind of mind set one must have to use a software setup that will inevitably crash at random and that needs wacky workarounds to produce results.

It's more a matter of being sure you're getting somewhere than it is a matter of time.
Random crashes are per definition not reproductable. You could run an encode 10th of times, fiddle with settings and still not be sure you're going to get the full video encoded without a crash.

I'll reiterate as well: this is only for the one odd video stream encountered once in a while. QTGMC is stable 99% of the time on my system.

[Groucho2004]

Originally Posted by BeNooL

It's more a matter of being sure you're getting somewhere than it is a matter of time.

Now you're not only contradicting what you wrote in your last post (that time is very important to you), you're also proving my point - that it is more important to get results.

[SamKook]

Once you've found the setmemorymax settings that works with your type of source video and the number of thread to use with them, you won't get any crash with avisynth MT and QTGMC.

For HD sources and high settings, it might not be worth it though. I have to lower it to 2 threads for it to not crash when using the placebo or veryslow preset and I usually split it in 4 to fill my CPU so it might be faster to simply split it in 7-8 and run it single threaded instead.
For SD, I've never had it crash ever since I found the right settings to use with my pc and it's much faster.

[Music Fan]

Originally Posted by SamKook

I usually split it in 4 to fill my CPU

What encoder do you use ?
I know that Virtual Dub can be opened several times at the same time (allowing simultaneous encodings) but it does not encode in AVC and I wonder if there are AVC encoders that can be opened several times at the same time.

[SamKook]

Originally Posted by Music Fan

What encoder do you use ?
I know that Virtual Dub can be opened several times at the same time (allowing simultaneous encodings) but it does not encode in AVC and I wonder if there are AVC encoders that can be opened several times at the same time.

I use mencoder to encode all parts(+ 50 frames on each side to make sure it has everything it needs to analyze for the last frame I'll actually use) to a lossless codec and then encode the full thing(minus the overlap) after with x264(I have a batch script that does it all for me automatically).

I was worried that simply encoding straight to the final result and merging the different parts together would hurt quality a bit, but mencoder is able to encode to AVC from what I read(it might require something extra though, I never tried it so I'm not sure) so you should be able to use it directly.

[Music Fan]

Thanks, but for the x264 part of the process, do you make simultaneous encodings or you do it only for the lossless part (which uses QTGMC) ?

[Octo-puss]

I tried to encode the same thing with single threaded version of AVS and it seems it would "only" take about one day longer than with MT version, which is... acceptable I guess. That's with very slow preset.

[SamKook]

Originally Posted by Music Fan

Thanks, but for the x264 part of the process, do you make simultaneous encodings or you do it only for the lossless part (which uses QTGMC) ?

I only make one encode for the x264 part. Once the QTGMC part is done for the lossless sections, it's plenty fast to feed the encoder without even needing to use mt.



I encoded once a bluray using single threaded avisynth, QTGMC @ placebo and x264 also @ placebo and it took 21 days to finish. With avisynth mt you can cut the time to about half and with the process I explained above it's done in about 3 days.

I since changed both settings to veryslow since x264 doesn't really benefit from placebo and veryslow give a smaller output and QTGMC actually produce better result with it when using the Source Match and Lossless options from what -Vit- once told me. It now takes me about a day for a 2h bluray.

[Asmodian]

I guess it is all about your mindset when encoding. When busy I will gladly trade 24h of CPU time for 1h of my time.

When not busy fiddling with Avisynth MT has provided hours of fun.

[WorBry]

Aw, come on guys, don’t burst my bubble. I’ve only recently upgraded to a six-core AMD FX PC, making MT processing, that was once an ethereal mystery, into which even angels long to gaze, now a tangible reality.

For 10 years, yes 10 years, I struggled with de-interlacing SD video on an ancient single core machine. Starting, way back, with TDeint and TomsMoComp. Then being enthralled with Scharfis_Brain’s ground-breaking mo-comped MVBob routines and exhilarated by Didee’s detail-squeezing McBob. Could bob-deinterlaced video look any better than this? It was a joy to set a 5 minute home video running over night, and in the morning I would excitedly rush to see the fruits of my labour…..only to find it had not completed.

I was there, right there, at the moment Didee pulled a fledgling TGMC out of his magical hat as a solution for interline shimmer/twitter, and I was hungry for more. But then as the years wore on, the long hours spent pouring over fine tunings and stiching together re-processed video segments began to take their toll. I reached a crisis point. Maybe it was family, maybe it was work, but I could go on no longer. De-interlacing video is a young mans sport and an utter foolishness, I had to conclude for sanity’s sake. So, with a dejected, heavy heart I resigned myself to the hum drum world of stuttery single progressive video, glancing only with painful regret as TGMC grew into the adorned, multi-threaded QTGMC we know today.

So, dear friend, you see, I’ve paid my dues, I still bare the scars…..I deserve multi-threaded QTGMC de-interlacing. And it is with a sense of long lost joy, that I find myself fiddling with MT settings and gazing with intense concern at the Task Manager. I have arrived…..I am home.

OK, so that’s story time over. Just a couple of questions for you thread-worn QTGMC-ers:

1. So, I’m now sitting with the MT AVISynth dll in my System 32 folder and Vit’s set of MT optimized filters in my QTGMC avs folder (for direct plug-in loading). Say now, if the excitement proves too much, and I want to resort to normal (Non-MT) QTGMC processing, and simply omit the MT-defining script entries (SetMemoryMax, SetMTMode etc), is it certain that the MT-optimized filters will function as they should, or would it be better to revert to the regular filter set?

2. Contemplating Boulder’s earlier comment:

Originally Posted by Boulder

What I usually do with QTGMC-related things is that I either 1) run the script with one thread and use cretindesalpes's MVTools build (in the Dither package) which includes internal multithreading or 2) run multiple one-thread encodes simultaneously and combine them later. SD sources seem to cope better with multithreaded QTGMC scripts than HD ones, probably due to memory usage issues.

Regarding the MVTools mod that includes hyper-threading; is there a modified QTGMC version available that allows this to be set in the function parameters, or would I need to work that in myself? And then, how best to balance MVTools hyper-threading against the Edithreads setting? Which has the greater impact on performance?

And, sorry for my ignorance (I've been out of the game for a good while), but how does one run multiple one-thread encodes simultaneously on a single PC?

Just want to examine the available MT options.

Cheers,

[Boulder]

I think the multithreaded MVTools uses the number of logical cores by default. If you want to set a different value, use avstp_set_threads(x) in which x is the number of threads. I have a file named avstp_set_threads.avsi in my plugins folder and it sets threads to 1 by default since I use SetMTMode successfully about 99% of the time. I'd expect that the MVTools functions get a better boost from multithreading, but I've usually left edithreads to default since I've not noticed any harm there.