Setmtmode causing runtime error

[txporter]

I am hoping that there is an easy way to diagnose this problem, but I haven't turn up the answer with google yet.

I am getting two different errors when trying to encode using HCenc in conjunction with Setmtmode. If I do not specify the max avsmemory, then the encode fails (usually within the first few percent) as a Microsoft Visual C++ Runtime Error. If I set the max memory to 512mb, it gets further into the encode but then I get an APPCRASH that says MPEG2 encoder stopped working. I watched it a few times and the encode starts out at ~55fps for the first few percent and then starts slowing down to where the average fps is <10fps and then it crashes.

If I use MT.dll rather than setmtmode, everything is fine.

Computer: Q6600, 4gb ram, Vista64
Avisynth: 2.5.8 MT (SEt build - July '09)
plugins all in avisynth/plugin directory (not directly called in scripts)
Hcenc encode settings: -cq_maxbitrate 6 -maxbitrate 9000 -progressive

HC.ini
*PROFILE best
*AUTOGOP 12
*AVSMEMORY 512 (was not set at beginning)
*INTRAVLC 2
*MATRIX hclow
*LUMGAIN 2

setmtmode script that causes crash:

setmtmode(2)
MPEG2Source("I:\ToConvert\video.d2v",cpu=3)

TFM().TDecimate()
Undot().FluxSmoothT(3)
LSFMOD()

TextSub("I:\ToConvert\video.srt")
Distributor()

MT script that works fine:

MPEG2Source("I:\ToConvert\video.d2v",cpu=3)

MT("
TFM().TDecimate()
Undot().FluxSmoothT(3)
LSFMOD()
",threads=4)

TextSub("I:\ToConvert\video.srt")

For setmtmode scripts, I have tried starting with mode5 and then switching to mode2 after mpeg2source. Also, tried using mode5 for subtitles. All combos I have tried still cause a crash.

I am fairly certain that this is not a HCenc issue since I was seeing similar crashes with setmtmode when trying to use the WMCMD.vbs for VC-1 encoding.

[txporter]

Ok, this appears to be an issue with vsfilter.dll. If I totally remove the filter call, I can run with setmtmode(2).

I thought I had read that setmtmode(5) turns off multi-threading. Is that not the case? I tried the script below and it still errors out. Can I put Distributor() above Textsub()? Is there a version of vsfilter that works with setmtmode?

setmtmode(2)
MPEG2Source("I:\ToConvert\video.d2v",cpu=3)

TFM().TDecimate()
Undot().FluxSmoothT(3)
LSFMOD()

setmtmode(5)
TextSub("I:\ToConvert\video.srt")
Distributor()

update:

Well, simply putting the Distributor() call above the Textsub function seems to be working (over 27% now and hasn't crashed). Not sure if Distributor() is always supposed to be at the end of the script or not. /shrug

Working:
setmtmode(2)
MPEG2Source("I:\ToConvert\video.d2v",cpu=3)

TFM().TDecimate()
Undot().FluxSmoothT(3)
LSFMOD()

Distributor()
TextSub("I:\ToConvert\video.srt")

[txporter]

:(

Error'd out at 85% complete this time. This time it said that Fluxsmooth.dll was the problem. Not sure what the problem is. Guess I am back to using MT for the time being.

[yup]

txporter!
Change to
setmtmode(5)
MPEG2Source("I:\ToConvert\video.d2v",cpu=3)
setmtmode(2)
TFM().TDecimate()
Undot().FluxSmoothT(3)
LSFMOD()

Distributor()
TextSub("I:\ToConvert\video.srt")
MPEG2Source work only with setmtmode(5).
yup

[txporter]

Thanks, yup. Will give that a whirl tonight.

[txporter]

Tried these combos, all cause memory to shoot up to 1+ GB and app crash. I was messing with a 1080i clip because they are more resource intensive and seem to cause crashes faster. I have something wrong with my setup on quad core (or just Vista64), but I don't seem to have these same issues on my XP laptop (dual core) from work. Although samples at work were all short (~2min clips) so it is possible that I didn't see issue because of that. However, there was no noticeable slowdown in encoding fps on laptop.
setmtmode(2) 1. <---moved to setmtmode(5) based on yup suggestion
MPEG2Source("video.d2v",cpu=3)
1. <---inserted setmtmode(2)
TFM()
TDecimate()
Undot()
3. <---tried adding setmtmode(5)
FluxsmoothT(5)
3. <---turned setmtmode(2) back on
LSFMOD()
2. <---moved Distributor() here (because MT didn't like height change)
Bicubicresize(1280, 720)
Distributor() 2. <---removed this line
TextSub("video.srt")

[XhmikosR]

I've had issues with LSFMod too. After removing it from my AviSynth script I get no crashes.

[txporter]

Thanks, I will give that a try. I suppose that I might have to use some combination of MT and setmtmode. I seem to be having more trouble than seems normal. Dunno.

[txporter]

I've had issues with LSFMod too. After removing it from my AviSynth script I get no crashes.

:goodpost: That seems have been the problem. I haven't encoded anything all the way through with LSFmod taken out of setmtmode, but it seems to have fixed the memory issue, fps issues, etc. Awesome!

[XhmikosR]

Well it would be awesome if we could find the exact plugin that's causing the problem... Hopefully someone could give as some help on that.

[txporter]

Based on the MT wiki, it appears that LimitedSharpenFaster works with setmtmode(2). It must be whatever is different between the two scripts. First one that jumps out is VariableBlur. Not positive though. It also mentions on that page that LimitedSharpenFaster works faster with MT than with setmtmode(2), so we probably aren't giving anything up by running LSFmod with setmtmode(5) and MT().

[XhmikosR]

I don't use the mode which needs VariableBlur; I don't even have it in my plugins dir because I know it has a memory leak or something. Although IIRC neuron2 had fixed this problem... Can you try with LSF with SetMTMode?

[txporter]

I don't use the mode which needs VariableBlur; I don't even have it in my plugins dir because I know it has a memory leak or something. Although IIRC neuron2 had fixed this problem... Can you try with LSF with SetMTMode?

Ok, tried LimitedSharpenFaster (instead of MOD) with setmtmodes. It causes a crash unless I setmtmode(5) before LSF.

script: (1080i sample, encoding with HCenc 0.24beta[latest], avsmemory set to 512mb in HC.ini)
setmtmode(5,0)
MPEG2Source("video.d2v",cpu=3)
setmtmode(2) 5. <---MT("
TFM()
TDecimate()
Undot().FluxsmoothT(5)
2. <---Add setmtmode(5)
LimitedSharpenFaster(Smode=4, strength=100) 4. <---MT, threads=4
3. <---Move setmtmode(5) from 2. to here
5. <---", threads=4)
Bicubicresize(1280, 720)
TextSub("video.srt")
Distributor()

1. Run script as shown - memory goes to 1.7GB and crash
2. Add setmtmode(5) above LSF - runs with ~700mb memory usage at 7.9 fps
3. Shifted setmtmode(5) below LSF - crash (1.7GB mem)
4. Moved setmtmode(5) back to 2. and MT, threads=4 LSF - runs with ~700mb memory usage at 9.5fps
5. Remove all setmtmode calls (and Distributor()) and run with MT for TIVTC, Undot, FluxsmoothT and LSF) - runs with ~700mb memory usage at 8.4fps.
6. Tried approach 3. with Smode=3 - crash (1.7gb mem)
7. Tried approach 3. with Smode=2 - crash (1.7gb mem)
8. Tried approach 3. with Smode=1 - crash (1.7gb mem)

Conclusion, LimitedSharpenFaster also does not work with setmtmodes.

Filter info:
DGDecode - 11/26/09 - 1.5.7
TIVTC - 1/17/08 v1.05
Undot - 6/7/04 -??? not sure, older timestamp than one on warpenterprise
Fluxsmooth - 6/22/06 - ??? also later than v1.1 (2004)
Removegrain - 10/24/07 - v1.0pre-release
Masktools2 - 12/22/08 - v2.0a36
Warpsharp - 11/4/07 - ??? alos later than shown on warpenterprise
Do not have masktool (v1) in plugin folder

Most of these filters were taken from XVID4PSP5 plugin directory and moved to Avisynth 2.5.8 plugin folder.

[XhmikosR]

OK then most probably it's the RemoveGrain package. I suppose you use the SSE2 versions too?

[txporter]

Just standard RemoveGrain.dll.

[XhmikosR]

I'm using the SSE2 version of the RemoveGrain dlls... I'll try LSF when I have some time and post my results.

[XhmikosR]

@txporter: Can you try with AviSynth 2.5.7 and the modified avisynth.dll by tsp? I was testing TempGaussMC_beta2, and it always crashed unless I used SetMTMode(5). That was with AviSynth 2.5.8, SEt's mod. Now with AviSynth 2.5.7 and tsp's avisynth.dll I get no crashes (still encoding) even with SetMTMode(2).

EDIT: VirtualDub still crashed pointing to mvtools2.dll this time.

[txporter]

I am probably going to let this one go for now. I do most of my recodes with HCenc, and his 1-pass VBR sampling methodology simply doesn't work with setmtmode. So regardless of whether I can get LSF/LSFmod to work under setmtmode(2), I wouldn't be able to run it through 1-pass VBR. It is faster to run through 1-pass VBR with only MT calls then running through 2-pass with setmtmode. I will probably need to start toying with this again if Hank can make modifications to allow it to work (or to allow a different input script for 1st pass).

I don't really understand why a temporal filter like FluxsmoothT works with setmtmode anyhow. Perhaps each thread still pulls multiple consecutive frames for the analysis??

[XhmikosR]

Well, I wouldn't bother too but there's definitely something wrong that's why I (and you) get crashes.

[txporter]

I am not certain that there is somewhere wrong, per se, but LSF/LSFmod don't work with setmtmodes at least. Perhaps they need consecutive frames to work?

[txporter]

@XhmikosR- How many cores is your processor? My home machine is 4 cores and that is what I was posting about in this thread. It has all the issues with crashing with setmtmode(2) and non-working 1-pass VBR on HCenc. I just tried this script on my work laptop (T2400 CoreDuo, 4GB ram, Windows XP SP3) and it completes without an memory issues or crashes or anything. Can you try your script with 2 threads if you have more than that by default?

setmtmode(5)
MPEG2Source("D:\Temp\HD\video.d2v",cpu=3)
setmtmode(2)
SelectEven()
TDecimate()
Undot()
FluxsmoothT(5)
LSFMOD()
Distributor()

1-pass HCenc, 2min 720p clip, no multi-threading - 4.7fps, total time = 12:54, bitrate=5046, avg Q = 7.632
1-pass HCenc, 2min 720p clip, MT - 5.8fps, total time = 10:56, bitrate=5050, avg Q = 7.683
1-pass HCenc, 2min 720p clip, setmtmode(2) - 6.0fps, total time = 11:41, bitrate=5046, avg Q = 7.632

Sampling takes longer with setmtmode, but doesn't totally stall out (~2:40 with/without MT, ~3:40 with setmtmode). Interestingly, the final bitrate and Quant of the encode with no multi-threading and setmtmode(2) are identical. The encode with MT calls has slightly higher bitrate and slightly higher quant (worse).

[XhmikosR]

I'm using 4 cores. I'll try later with 2. The weird thing is I didn't have such issues in the past. I've been using SEt's MT build since the day he posted it here at doom9.

[Gavino]

I don't really understand why a temporal filter like FluxsmoothT works with setmtmode anyhow. Perhaps each thread still pulls multiple consecutive frames for the analysis??
Yes, each thread works on a different output frame, but apart from that, each instance of the filter works as normal (it doesn't know any different), fetching the input frames it requires.

[txporter]

Yes, each thread works on a different output frame, but apart from that, each instance of the filter works as normal (it doesn't know any different), fetching the input frames it requires.

Thanks! That makes a lot more sense now. That is also the reason that I get identical encodes with and without setmtmode multi-threading.

[Dogway]

Recently Im having many crashes with the same "visual c++ runtime error". And some different results playing with the multithreading options.

This code:

SetMTmode(5)
setmemorymax(768)
DGDecode_MPEG2Source("source.d2v")
SetMTmode(2)
ColorMatrix(dest=2,hints=false,interlaced=true,threads=2)

AssumeFPS(30000,1001)
tempgaussmc_beta2(2,2,3,EdiMode="NNEDI2", qual=3)
SRestore(frate=23.976)

MT("""tnlmeans(ax=11,ay=10,az=4,sx=2,sy=2,bx=1,by=1,h=1.0,a=0.5,sse=true)""",4,2)

DeHalo_alpha(rx=1.3, ry=1.3, darkstr=0.4, brightstr=1.0, ss=1.0, lowsens=50,highsens=65)
LSFmod(defaults="slow", strength=20, edgemode=0)

...seems to work (I only let a few minutes running) if I comment the tnlmeans line, or remove the MT portion of it.


But this other one:

SetMTmode(5)
Import("C:\Archivos de programa\AviSynth 2.5\plugins\AAA.avs")
MPEG2Source("VIDEO_TS.d2v", cpu=0)
SetMTmode(2)
c=trim(5230,6150)
c=c.mdegrainc2().AAA().LSFmod(defaults="fast")
c

...also crashes. CPU process varies from 60% to 90%, and RAM increases fastly before the crash.


The common point here is LSFmod. In one works in the other crashes. At the same point in the first script, the crash is due to mixing setMTmode with MT. So my question is, why this all happens?? How will I know if my script is not gonna crash?

I mean, I have read everything, TGMC (http://forum.doom9.org/showthread.php?p=1374393#post1374393) better to not use setmtmode, srestore, (http://forum.doom9.org/showthread.php?p=1379587#post1379587) recommended setmtmode(5), tnlmeans (http://forum.doom9.org/showthread.php?p=1382343#post1382343) (here (http://www.unkar.org/read/pc11.2ch.net/avi/1228216385) too) not real benefit from setmtmode but incompatible if MT, LSFmod (http://forum.doom9.org/showthread.php?p=1378271#post1378271) is also inconsistent with setmtmode by reading this thread. So a question comes to my mind, why bother multithreading? can anybody convince me?

btw Im using command line x264 rev.1471-nl-hrd if it matters, I do have visual c++ 2005 and .net framework 3.5