MT Failure With Batch Encoding

[Danette]

I frequently batch encode (x264) 20-40 files with MeGUI’s One-click. After some difficulties, which were due to an incorrect AvisynthWrapper in MeGUI, I’ve gotten back on track. However, a new problem has appeared, when using MT:

Randomly, but usually after 15-20 successfully encoded files, a file will start taking over a day to encode, when it should be less than 25 minutes (like all previous files in the batch). So, I have to abort and restart, at the point of the previously failed file, then the file will encode with the normal time and the rest of the files will do so, as well.

There are no error messages indicated by MeGUI. Additionally, if I change the Prefetch to 4, this problem occurs much more frequently. If I slow it down to 2 threads, I might as well not use MT at all.

SetFilterMTMode("DEFAULT_MT_MODE",2)
MPEG2Source("C:\xxxx.d2v", cpu=0)
TFM(slow=2)
Decimate()
AssumeTFF()
QTGMC(InputType=1,Preset="Medium”)
CropResize(0,0, 0, 0, 0, -0,InDAR=15.0/11.0,ResizeWO=True,Resizer="Lanczos4Resize")
GradFun3()
Prefetch(3)

MeGUI version is v2928 (with AvisynthWrapper v1.0.2717.2), AviSynth+ v3.7.2, filters and dependencies all seem to be current.

[VoodooFX]

Lately I have some random failures with MeGUI batch encode too. I didn't use custom filters in avs script.

[Danette]

Lately I have some random failures with MeGUI batch encode too. I didn't use custom filters in avs script.

Did that involve MT? I haven't had an issue when not using it.

[Danette]

Well, with the lack of responses, I can only conclude that either others have not had this problem or that they do, and simply don't know what to do about it.

For the benefit of others that may find this post in searching a similar issue, this is a continuing problem with every batch encode that I do, with all of the above mentioned specifics. It does seem that MT use can be an unstable activity, but it is fast ...when it does work. I have noted that some other well-respected members have commented that MT is not all that reliable. So, I do continue to use it, but have to check, occasionally, to make sure that it hasn't locked-up on a file, requiring aborting and restarting.

[Blankmedia]

or that they do, and simply don't know what to do about it.

This, I made my own batcher in python that relaunch a thread for the failed encode. Also when I use setcachemode (1) sometimes the process avs2yuv or ffmpeg to y4m won't start for hours on a 23 minutes episode.

Dividing my script in multipart seems to help.

In your case I'd do

first pass -> source + decimation (to y4m)

Second pass -> y4m + qtgmc clean up + resize. (To x264 or x265)

But I don't know enough as to why. I saw that request linear was a cunt in MT and remove it from my script, but I think it's also in qtgmc.

On my good desktop megui keep erroring on me for whatever reason so I haven't played with it in a while.

[Danette]

I’ve found some methods that help stabilize performance. I am in the process of converting all of my DVD's to x264, which means that the dedicated computer (Win 7, 64-bit) is running constantly and will do so for many more months. This gives me a good base for trial and error with MT.

On my machine, it seems that running at close to 100% CPU capability (3-4 threads) leaves no room for flexibility. If I try to do other things, while encoding with MT, within an hour or two the encode fails, as described above, slowing to <1 fps. Even the automated turning off of the display will cause this. So, leaving the monitor always on has extended the time before failure.

Next thing that I tried was to move Avisynth+ and MeGUI to 64-bit versions. This also extended the time to failure, as well as increasing speed by about 15% vs. 32-bit versions.

Doing these two things roughly doubled the time before failure. I have been running 3 and 4 threads on my 4-core machine, with about equal speed results. Running 2 threads (utilizing about 80% of the CPU) gave me no failures, but added about 25% to the time for each encode.

I then reached close to no failures by varying between 3 and 4 threads. It seems that DVD studio packages (a single TV series comprising all episodes) perform differently between 3 and 4 threads. For example, if one set stumbles on 4 threads, I switch it to 3 threads and it processes without difficulty. Unfortunately, I have no idea why this happens, which means that I can never know which number of threads to start with and rely upon luck that I have chosen the right number. I do start with 4 threads, which, so far, has resulted in the most number of successes.

[johnmeyer]

Yup: reduce the number of threads. Also play around with the maximum memory setting.

[Shinkiro]

Something similar was discussed here https://github.com/pinterf/mvtools/issues/46
and as I understood, nothing has changed since then.
If I return avstp.dll, I also occasionally get an encoding hang at an arbitrary place