smdegrain crashing, how to solve?

[zerowalker]

Well didn't know what to write for the title.
But it's pretty simple, smdegrain is crashing the application which uses it (avs2avi, virtualdub etc), although virtualdub doesn't crash, it continues as if it was working, the the framenumber is stuck so it's just going infinite.

I tried many things, Avisynth+, Avisynth 2.6, made some faster settings for smdegrain, but it always crashes anyway.

I am not sure what to do, as far as i know smdegrain is just a simplification of the MDegrain, so in turn would mean it's MVTools that's the culprit?

I have updated MVTools to the latest: 1.11.4.5

So, any suggestions?
As this doesn't seem to be an Avisynth related problem, but rather the Plugin.

[feisty2]

you need MVTools 2.6.05 not mvtools 1.x

[zerowalker]

Oh, well got that one as well.

[zerowalker]

No ideas at all?

I thought of trying it on 64bit, but noticed that Dithering isn't supported yet.

[osgZach]

Is it running out of RAM maybe? Tried using SetMemoryMax(int) ?

[zerowalker]

It shouldn't be, it's at 400mb or something when it crashes (differs a bit, but not close to 1.2gb at all).

[Reel.Deel]

Post your full script that causes the crash and if possible a small sample. There's a handful on unknown variables that may help in helping you (source filter, resolution, colorspace, exact Avisynth version, etc).

[zerowalker]

Nothing to post really, Avisource, RGB, 640x480, Avisynth+ and Avisynth 2.6 Alpha (both of them tested).

Tried to save with different Codecs, UT and MagicYUV same results.

Tried saving with VirtualDub, same results.

Can't really provide a sample as it's a bit private sadly.
But it's nothing special, a normal 640x480 RGB file, it was Lagarith Originally, but resaved it with ffmpeg to UT just in case if that was the problem (Lagarith can have corruption on decode, never happened to me though).

So i am totally lost really.

[Reel.Deel]

Are you trying to process RGB as RGB or are you converting to YV12 for processing? Also you're talking about SMDegrain (http://forum.doom9.org/showthread.php?p=1523572#post1523572) and not SMDegrain Mod (http://doom10.org/index.php?topic=2178.0), correct?

[zerowalker]

I am converting it to YV12, thanks for mentioning that!

And i am also using: SMDegrain Mod (Didn't know it was called Mod, sorry for that).

[Guest]

Nothing to post really You can at least give us your full script, can't you?

[zerowalker]

You can at least give us your full script, can't you?

Here it is:

Avisource("Z:\TestGraph - ffmpeg.avi")
ConvertToYV12(matrix="Rec601", interlaced=false)
smdegrain(tr=3,lsb=true,thsad=300,lsb_out=true)

[Groucho2004]

It might be helpful to run the script through AVSMeter and check the memory usage there. Also, have you tried different source filters (LSMASH, FFMS2)? I vaguely remember some trouble with Avisource and a lossless file.

[zerowalker]

Yes it is, and yes i have.

Though i have smdegrain lsb on.
But i have tried without the Dithering to 16bit and just the normal (2 stacked pictures) output.

Will do that, i guess you mean, run till it crashes, and post the CMD output?

EDIT:

I have not tried using anything other than Avisource.

I have tried 2 versions of the same file, or rather one filtered and one original. The thing though is that one was UT Video Codec, other MagicYUV, so tried both just in case there was a decoding issue.

[Groucho2004]

Yes it is, and yes i have.

Will do that, i guess you mean, run till it crashes, and post the CMD output?

Sorry, our posts crossed and I edited mine.

Yes, use the -l switch to create a log with AVSMeter and post that.

[Guest]

I have not tried using anything other than Avisource. You should do that.

[Groucho2004]

The problem might be avstp which allocates memory beyond what is set with setmemorymax. I just ran a quick test with your script and an sd source and the memory usage was > 800 MB (4 threads) after a few minutes.

[Guest]

Is avstp built in to SMDegrainMod, or how did you guess he is using it?

[Groucho2004]

Is avstp built in to SMDegrainMod, or how did you guess he is using it?
The modified version of mvtools2 and dither.dll are using it. So yes, it's basically built in.

Edit: It seems that simply removing or renaming avstp.dll disables multi-threading. Should be worth a try for the OP.

[Guest]

I see. Thank you very much.

[Reel.Deel]

So yes, it's basically built in.


I thought it's only used if avstp.dll is loaded?
From the Avisynth user point of view, an AVSTP-enabled plug-in requires the avstp.dll file to be installed. Put it in the usual AviSynth 2.5\plugins\ directory, or load it manually with LoadPlugin("path\avstp.dll") in your script.

Edit: did not see your edit.

[zerowalker]

The problem might be avstp which allocates memory beyond what is set with setmemorymax. I just ran a quick test with your script and an sd source and the memory usage was > 800 MB (4 threads) after a few minutes.

Hmm, weird, mine uses less, or perhaps it's just in the crash it uses left (it's been purged) etc.

What can be done then, as i guess SetMemory won't work as you say it goes beyond it.

Though i have never used it on my script, as i thought it was an old MT thing.

[Groucho2004]

What can be done then
Did you try any of the things I suggested (run the script through AVSMeter, change the source filter, remove avstp.dll)?

[foxyshadis]

I'd also suggest installing WinDbg (http://msdn.microsoft.com/en-us/library/windows/hardware/ff551063%28v=vs.85%29.aspx) to try to isolate the crash. Even if you can't get it all the way, at least you can get an idea of the module and stack at that point, and repeated runs might tease it out.

First set the symbol server (http://support.microsoft.com/kb/311503) in the file menu, with a local path, then either Open Executable or Attach to a Process (if it's already started). Press F5 (or Debug->Go) and it'll run, then when it crashes it'll break back into WinDbg instead. Type !analyze -v and see what pops up. It should give you some details of the stack trace and which dll it was in when it crashed.

From there if the source code is available you (or someone) can rebuild it as a debug version to more closely examine the problem, but that's a bigger undertaking.

Alternately, if you can reliably make it crash with a more public video, one re-encoded by utvideo, that would be a big help. It doesn't sound like out-of-memory is the real issue.

[zerowalker]

Did you try any of the things I suggested (run the script through AVSMeter, change the source filter, remove avstp.dll)?

I tried AVSMeter this night (sadly i can't just try things fast, as it takes forever for the crash to occur).

and well, i am not sure i did it correctly, cause i didn't see any log, but here is the CMD output.


AVSMeter 1.7.7 (AVS 2.6, x86) by Groucho2004
AviSynth 2.60, build:Sep 18 2013 [17:36:36]

Number of frames: 317766
Length (hhh:mm:ss.ms): 002:57:16.518
Frame width: 640
Frame height: 960
Framerate: 29.875 (1250000/41841)
Interlaced: No
Colorspace: YV12

Frames processed: 317766 (0 - 317765)
FPS (min | max | average): 3.589 | 19.73 | 9.646
CPU usage (average): 64%
Thread count: 8
Physical Memory usage (peak): 560 MB
Virtual Memory usage (peak): 562 MB
Time (elapsed): 009:09:01.708


I'd also suggest installing WinDbg to try to isolate the crash. Even if you can't get it all the way, at least you can get an idea of the module and stack at that point, and repeated runs might tease it out.

First set the symbol server in the file menu, with a local path, then either Open Executable or Attach to a Process (if it's already started). Press F5 (or Debug->Go) and it'll run, then when it crashes it'll break back into WinDbg instead. Type !analyze -v and see what pops up. It should give you some details of the stack trace and which dll it was in when it crashed.

From there if the source code is available you (or someone) can rebuild it as a debug version to more closely examine the problem, but that's a bigger undertaking.

Alternately, if you can reliably make it crash with a more public video, one re-encoded by utvideo, that would be a big help. It doesn't sound like out-of-memory is the real issue.

I have actually made a Debug check with Visual Studio, but i didn't undestand anything of it, and none wrote back at Avisynth+ about it, so i thought it wasn't of any use.
I can make use of WinDbg though, will do it on the next test after i get a reply about AVSMeter as it's not a fast procedure.

I don't really get what you mean with one re-encoded by utvideo though.
I have re-encoded the file i am using from Lagarith to uvideo with ffmpeg already, just to prevent Lagarith from being the decoding bug culprit.

[Groucho2004]

I tried AVSMeter this night (sadly i can't just try things fast, as it takes forever for the crash to occur).

and well, i am not sure i did it correctly, cause i didn't see any log, but here is the CMD output.


AVSMeter 1.7.7 (AVS 2.6, x86) by Groucho2004
AviSynth 2.60, build:Sep 18 2013 [17:36:36]

Number of frames: 317766
Length (hhh:mm:ss.ms): 002:57:16.518
Frame width: 640
Frame height: 960
Framerate: 29.875 (1250000/41841)
Interlaced: No
Colorspace: YV12

Frames processed: 317766 (0 - 317765)
FPS (min | max | average): 3.589 | 19.73 | 9.646
CPU usage (average): 64%
Thread count: 8
Physical Memory usage (peak): 560 MB
Virtual Memory usage (peak): 562 MB
Time (elapsed): 009:09:01.708
You have to read the answers, I wrote that you have to use the "-l" switch to create a log file.

Anyway, looks like nothing crashed or got stuck so the frame server part is working.
Now you have to tell us what you do with the script in VirtualDub. What's the target format?

[zerowalker]

I did use -l, avsmeter -l "..." .

I normally use avs2avi, but if i do it with VD, i simply save it as, "Fast Recompression" and MagicYUV (only tried it once, but doubt the format itself matters in this case).

It won't crash VD, but the frames will get stuck and the timer will go on, so it looks like all is fine, but it's not.

[Groucho2004]

I did use -l, avsmeter -l "..." .In that case there should be a log file in the same directory as the script.
It won't crash VD, but the frames will get stuck and the timer will go on, so it looks like all is fine, but it's not.
I've never seen anything like that. Maybe someone else had a similar experience.

[zerowalker]

Oh, i got the log, i thought it was in the AVSMeter directory;P

Looking at the log, it just seems like all went fine, so i don't really get it.

https://www.sendspace.com/file/bi1229

I can only think that the issue occurs when giving the frames to an Encoders.
But as i have tried UT and MagicYUV i don't see how that can be the reason either...

[zerowalker]

Okay i have no idea, but i saved with VirtualDub again, and now it works:S

I am pretty sure it's the same settings as before, except one thing i have changed which improved the speed but reduced quality, but it still crashed with it.
So well, gonna do it again with that setting just to make sure.

Don't get why it's working though, been trying every night for days in different conditions.

[Guest]

except one thing i have changed What is that thing that you changed?

[zerowalker]

It was a motion estimation calculation "Refine Motion".

But i remove it awhile ago as i thought it was the culprit as it was so slow.
So i have tried many times with it off, so that's why i am surprised, pretty sure i have tried VD with it off.
Must have as i didn't try VD until later on.

So, just confused:(

[zerowalker]

Okay this is just embarrassing, i actually forgot the whole point why i did the test again.
I actually tried changing from Avisource to DSS2 (I know DirectShow isn't frame accurate, so i guess this one should be safe?).

So it's most likely it, will try again with the precise same settings except DDS2 -> Avisource, just to make sure.

[zerowalker]

Okay tried again, Avisource and all with VD, and it didn't fail.
So again, no clue why it works now, it's the same file, same settings, same VD.

But well, i should be happy that it's working, but it's rather disturbing not to know what the cause was.

[Groucho2004]

But well, i should be happy that it's working, but it's rather disturbing not to know what the cause was.
Could be a hardware issue, faulty memory, overheating, ...
In case you have not done so already, run some stress tests:
Prime95 (32 (http://files.extremeoverclocking.com/file.php?f=103)) (64 (http://files.extremeoverclocking.com/file.php?f=205))
Linx (https://github.com/sanekgusev/LinX-old/releases/tag/0.6.5)
Memtest (http://www.memtest.org/)

Encoding with x264 is also a very good stability test.

[zerowalker]

Done very much of those.
I do have my PC OCed.

But i did my previous tests without OC, and this test with OC, both worked, and many failed with and without OC.

It's just, confusing.

[foxyshadis]

Horray for the Heisenbug! Multithreading tends to be a minefield of them, but sometimes you just never quite find out why.