Encoding slowdown

[StainlessS]

I guess that you discovered that the log does not appear in avs file directory, until completed.
(Info for other users).

[VideoFanatic]

See my previous post. I've attached the log.

[Groucho2004]

See my previous post. I've attached the log.I can't see anything out of the ordinary in the log. The memory consumption is actually very moderate, probably because it's SD resolution.
You have to elaborate a bit on your statement above "but if I also use the McTemporalDenoise Low preset then the encode always fails."
What fails?
Any error message?
How long does it run before it fails?
What tool do you use for encoding?

[Groucho2004]

I guess that you discovered that the log does not appear in avs file directory, until completed.
(Info for other users).
Correct. The data is collected in memory and written to the log file just before the program terminates.

[VideoFanatic]

Fails in MeGUI x264 or Simple x264 Launcher. McTemporalDenoise Low and any QTGMC preset results in the encoding failing. But McTemp Medium and and QTGMC preset does NOT crash. In MeGUI for example, it doesn't actually crash. What happens is after an hour or so the video simply stops encoding. I can still see the video "encoding" in MeGUI but the CPU usage falls to 20% so the CPU isn't being used. I can see in the encoding window that the remaining time goes up instead of down and the encoding speed gradually goes down to zero where it then crashes. It just says something like the avsx264.exe crashed.

Any idea what's causing the crash? What extra filters does McTemporalDenoise low use that medium doesn't?

What's strange about this is that on a h264 720 x 576i source I can use McTemp Low and QTGMC SuperFast and it does NOT crash.

[Groucho2004]

What happens is after an hour or so the video simply stops encoding.
In that case you should run the whole script through AVSMeter and check if it slows down in a similar fashion.

As for troubleshooting I would increase the memory for Avisynth from the default (512MB) to 1000~1500. Put "SetmemoryMax(1000)" at the beginning of your script. Increase the value as needed, but not beyond 1500.

[VideoFanatic]

I'll try running the whole script through AVSMeter. I've never seen SetMemoryMax work before. Doesn't seem to do anything as I still got crashes. Are you sure it goes at the very start of my script? The docs say to put it after the first setmtmode line.

[VideoFanatic]

It seems that now I'm even getting crashes on McTemporalDenoise medium. Not sure why. I'm running the whole video in AVSMeter to see if it crashes and I'll post the results once it's finished. The script runs fine in non-MT and doesn't crash. I encoded loads of videos with this script in MT in the past and it rarely crashed. Now it seems to crash all the time and I don't know why.

I also notice that CPU usage is around 98% with the script when I'm sure it used to be around 75% before since it's only using 6 out of 8 cores as specified in my script.

[VideoFanatic]

At 1 hour 14 minutes the AVSMeter window froze - it stopped updating the time elapsed / estimated. Pressing Escape did nothing and it looks like the log is only written when you press Escape.

[Groucho2004]

At 1 hour 14 minutes the AVSMeter window froze - it stopped updating the time elapsed / estimated. Pressing Escape did nothing and it looks like the log is only written when you press Escape.
There's a very high number of useless nnedi3 threads being spawned because nnedi3 features internal multi-threading. Try reducing them like this:
setmtmode(5,6)
mpeg2source("M:\Video.d2v")
setmtmode(2)
McTemporalDenoise(settings="low", interlaced=true)
AssumeTFF()
QTGMC(Preset="Slow", EDithreads=1)
Crop(10,2,-10,-8)
nnedi3_rpow2(2, cshift="Spline36Resize", fwidth=720, fheight=576, threads=1)
SeparateFields() SelectEvery(4,0,3) Weave()
In my test, this reduced the number of threads from 93 to 39 and it was actually slightly faster.
If that does not work, reduce the number of threads in SetMTMode().

Edit1: I just tried the script above and it also freezes after a while.

Edit2: Excluding McTemporalDenoise from MT (i.e. moving "setmtmode(2)" after MCTD) seems to make the whole thing stable.

[VideoFanatic]

Thanks. Yes moving setmtmode makes it stable but that's only because it's basically running in non-MT and you get a slow speed!

So do you think it's Nnedi causing these problems? Is there nothing else I can do to get this script to run stable? It's weird. I thought maybe it was my PC that might have been the problem so I restored a backup image and it worked perfectly for several days but now the problem is back!

[Groucho2004]

Thanks. Yes moving setmtmode makes it stable but that's only because it's basically running in non-MT and you get a slow speed!
When I tested this it wasn't that much slower, maybe 20%. MCTD makes use of avstp.dll which comes with mvtools2 2.6.0.5. Make sure you have it in your auto-load directory.

Also, the CPU cycles that Avisynth isn't using can be used by the encoder, at least to some extent.

[VideoFanatic]

I already had those files in the directory. I have an 8 core 3.9 Ghz CPU. I'm getting maybe 3fps when avisynth has the slowdown. I normally get up to 7fps. I don't know what else I can do to fix this issue.

[Groucho2004]

I already had those files in the directory. I have an 8 core 3.9 Ghz CPU. I'm getting maybe 3fps when avisynth has the slowdown. I normally get up to 7fps. I don't know what else I can do to fix this issue.
OK, here's what I tried (with my i5 2500K, 4 cores @4GHz):

Script(basically your exact script which will eventually slow down and freeze):
LoadPlugin("E:\Apps\VideoTools\DGDec\DGDecode.dll")
Import("E:\Apps\VideoTools\AVSPlugins\qtgmc.avsi")
setmtmode(5,6)
MPEG2Source("F:\DVD_Interlaced\test.d2v", idct = 4).loop(100)

setmtmode(2)
McTemporalDenoise(settings="low", interlaced=true)
AssumeTFF()
QTGMC(Preset="Slow", EDithreads=1)
Crop(10,2,-10,-8)
nnedi3_rpow2(2, cshift="Spline36Resize", fwidth=720, fheight=576, threads=1)
SeparateFields() SelectEvery(4,0,3) Weave()


Result of running 2000 frames with AVSMeter:
Frames processed: 2000 (0 - 1999)
FPS (min | max | average): 1.710 | 91783 | 12.63
Memory usage (phys | virt): 855 | 1014 MB
Thread count: 39
CPU usage (average): 99%


When I move the setmtmode(2) below MCTD() I get this:
Frames processed: 2000 (0 - 1999)
FPS (min | max | average): 1.677 | 35.16 | 10.11
Memory usage (phys | virt): 679 | 759 MB
Thread count: 35
CPU usage (average): 76%

So, the second script runs at about 80% of the first one, uses less CPU/memory resources and won't freeze.
When you include the encoder in the process, the difference in speed will be negligible (actually, the second one will always be faster since the first one never gets to the end :rolleyes:).

[VideoFanatic]

Yes your speed was correct if you use Low for McTemporalDenoise. I was talking about using it with Medium. Speed in non-MT is too slow for Medium. Also I checked on my TV to compare the picture difference between QTGMC Slow and Super Fast and there was no difference so I'm using Super Fast. I'm encoding 2 videos at once in non-MT and I'm getting around 4fps each. I've encoded several videos without problems. But today when encoding 2 videos at once, one of the encoding has the slowdown again where the time remaining goes up instead of down. So it seems this bug happens even in non-MT! I'm encoding it again as hopefully this will just be a rare bug in non-MT instead of all the time like in MT!

[kuchikirukia]

Ok, currently trying for a better fix to this. Also here:
http://forum.doom9.org/showthread.php?t=173308

Best I've come up with so far that doesn't involve 2 encodes is:

Trim(4,-1) ++ Trim(1,0)

Blips frame 4 at the beginning, then continues on at frame 1 (skipping 0).
Anyone have a better idea? Trim(4,-1) ++ Trim(0,0) does work (avisynth behaves), so can we output the first trim to null?
Or can we use the first script and overlay frame 0 onto the out-of-place 4 instead?

EDIT:
Ok, someone tell me if this is stupid, because it seems to work:

clip1 = trim(4,-1)
clip2 = trim(0,-1)
Overlay(clip1, clip2, mode="blend", opacity=1.0) ++ trim(1,0)

-------------------------------------------------------------------------------------------------------------

Test 1:

setmemorymax(900)
LWLibavVideoSource("d:\video\00004.m2ts",threads=2)
QTGMC(preset="slower")

result:

http://s8.postimg.org/frh026tx1/Untitled.png

Test 2:
setmemorymax(900)
LWLibavVideoSource("d:\video\00004.m2ts",threads=2)
QTGMC(preset="slower")
clip1 = trim(4,-1)
clip2 = trim(0,-1)
Overlay(clip1, clip2, mode="blend", opacity=1.0) ++ trim(1,0)

result:

http://s8.postimg.org/hurf9utpx/Untitled1.png

Edit 2: 3000 frames in and it's still behaving! 4.3fps and no destroying and creating of MSVCR120 threads.

[StainlessS]

Its kinda ridiculous that you have to do this at all, but does the below have +ve effect ?


Colorbars
ShowFrameNumber

Trim(Trim(4,-1)++Trim(0,0),1,0) # -1 is 1 frame (Length=1 is v2.6 only)


Edit: or the even more ridiculous

Colorbars
ShowFrameNumber

Trim(Trim(4,-1)++Trim(0,-1)++Trim(1,0),1,0)


or


Colorbars
ShowFrameNumber

Trim(Trim(0,-1)++Trim(1,0),0,0)


Or wrap in a function and install in plugins (so you dont have to remember what to do)


Colorbars
ShowFrameNumber

Function PlayNice(clip c) {c Return Trim(Trim(0,-1)++Trim(1,0),0,0)}

PlayNice

[kuchikirukia]

I don't get what that's supposed to do. It gives me almost 108k frames of color bars and qtgmc doesn't seem to act on it. It doesn't break the input filter, but I don't think it would without the trim.

Did you mean without the Colorbars?
Frame 0 shows up as Frame 00000, it's the right frame, they're in order, and it runs...
... broken.

http://s12.postimg.org/881y3h2m5/Untitled.png

Oh, and I went and replaced the "length=1" in mine with "-1" to make it pre-2.6 safe. Confirmed it still works. Previous post edited.



Edit: or the even more ridiculous

Trim(Trim(4,-1)++Trim(0,-1)++Trim(1,0),1,0)


Broken.

or


Trim(Trim(0,-1)++Trim(1,0),0,0)


Broken.

Or wrap in a function and install in plugins (so you dont have to remember what to do)


Function PlayNice(clip c) {c Return Trim(Trim(0,-1)++Trim(1,0),0,0)}

PlayNice


Pretty sure that will be broken because it starts at frame 0.
Yup.

It needs to be >=4. Once we call a trim of (4,x) it doesn't seem to care if we go back to frame 0. But it doesn't like to start at 0.

[StainlessS]

Yeh, the colorbars and ShowFrameNumber should be replaced with whatever you want, I dont have your clip and wanted to verify that it works proper.

Try the edits also.

[kuchikirukia]

Hm, I suppose I don't need to specify that opacity and mode since they're the default.

Should it use something other than clip1 and clip2 as names, or should that be ok?

[StainlessS]

The other stuff is just to test function returns correct frames.

Colorbars.ShowFrameNumber.ConvertToYV12

Function PlayNice(clip c) {Return Overlay(c.trim(4,-1),c.trim(0,-1)) ++ c.trim(1,0)}

PlayNice


EDIT: Maybe add it as first line inside QTGMC

something like


Function QTGMC(clip Input, ...) {
Input = Playnice(Input)
}


or

Function QTGMC(clip Input, ...) {
Input = Overlay(Input.trim(4,-1), Input.trim(0,-1)) ++ Input.trim(1,0)
}


Fixed error, added stuff in BLUE.

[kuchikirukia]

PlayNice works. Is that just what I wrote just condensed to a single line?

I just get syntax errors trying the rest.

[StainlessS]

Is that just what I wrote just condensed to a single line?

Yes.

Sorry, missed out the Overlay( bit for some reason, fixed in post above (I hope).

EDIT: And you do understand that
Function QTGMC(clip Input, ...) {

is supposed to be the function description for QTGMC and '...' just means I missed out most of it, ie
you only ADD either

Input = Playnice(Input)

or

Input = Overlay(Input.trim(4,-1), Input.trim(0,-1)) ++ Input.trim(1,0)

as the first line inside of the QTGMC function.

[kuchikirukia]

Yeah, no clue.
Throwing Playnice into QTGMC and calling QTGMC().playnice is as far as I can get.

[kuchikirukia]

Ok, 4 frames doesn't seem to quite be enough. I just had a QTGMC + smdegrain script (that wanted at least setmemorymax(1900)) where I was getting slowdowns and corruption ~1000 frames in. I've bumped it to:

Function PlayNice(clip c) {Return Overlay(c.trim(100,-1),c.trim(0,-1)) ++ c.trim(1,0)}

And that's working.

I'll move this to my main machine and run some tests to see where the cutoff seems to be.

E1: Ok, even 100 isn't enough to stop MSVCR120 from going Christmas. It just stops it from slowing down as much.
E2: 1000 doesn't do it either. Though trim(999,0) without PlayNice is working.
E3: Trim(4,0) works. So it seems the PlayNice function jumping right back to frame 0 is partially reinserting the original problem. Its jump ahead mitigates whatever's going on, but doesn't fix it.

E4: Ok, apparently PlayNice doesn't play nice with smdegrain(lsb=true). It's actually introducing this slowdown in the script I'm using.
e5: ... if placed after?

Ok, I can't make heads or tails of this.


Function PlayNice(clip c) {Return Overlay(c.trim(1,-1),c.trim(0,-1)) ++ c.trim(1,0)}
Causes christmas lights if called before or after smdegrain(lsb=true), but minimal slowdown

Function PlayNice(clip c) {Return Overlay(c.trim(4,-1),c.trim(0,-1)) ++ c.trim(1,0)}
Works if called before smdegrain(lsb=true). Christmas + slowdown if placed after.

Function PlayNice(clip c) {Return Overlay(c.trim(10,-1),c.trim(0,-1)) ++ c.trim(1,0)}
Christmas + slowdown if placed before or after smdegrain(lsb=true).

And what's going on with this:

assumetff()
QTGMC(preset="medium",sharpness=0.9)
smdegrain(lsb_in=false,lsb=true,lsb_out=false,tr=4,prefilter=3,thSAD=150,contrasharp=true,refinemotion=true,plane=0,chroma=false,interlaced=false)
trim(20,0)

2.66 fps

https://s31.postimg.org/qk02zijx3/Untitled1.png (https://postimg.org/image/qk02zijx3/)


assumetff()
trim(10,0)
QTGMC(preset="medium",sharpness=0.9)
smdegrain(lsb_in=false,lsb=true,lsb_out=false,tr=4,prefilter=3,thSAD=150,contrasharp=true,refinemotion=true,plane=0,chroma=false,interlaced=false)


2.79 fps

https://s31.postimg.org/akhffynvb/Untitled.png (https://postimg.org/image/akhffynvb/)

e6: except Trim before QTGMC breaks it. 100 frames in I'm getting green screens.

e7: Ok, what I'm seeing is that PlayNice doesn't necessarily play nice if used after other filters or in conjunction with another trim command that doesn't start with 0. So it should be used QTGMC().playnice() and only when starting at the beginning of a video.

[ravewulf]

I don't know if this will be of any use to anyone, but I figured I'd post what works for me (using MeGUI's present syntax, but manually filling in the x's):

SetMemoryMax(x)
SetMTMode(3,x)
<input>
SetMTMode(2)
QTGMC(EdiThreads=1)
<crop>
<resize>
<denoise>
SetMTMode(1)
GetMTMode(false) > 0 ? distributor() : last

Basically ALWAYS use EdiThreads=1 and let SetMTMode handle the threading. I've also just started experimenting with removing AVSTP and sometimes that helps.

SetMTMode(3,x) for most source filters, but SetMTMode(5,x) for hardware decoders (DGIndexNV). I tend to use # threads = 1.5x # of cores (6 threads for a quad core processor) when going to huffyuv for further processing and 0 (aka same number of threads as cores) if I'm going to x264. Tweak memory up as needed (I've gone as high as 2000 as for me the amount needed to prevent crashes seems to increase as both the length of the video and the number of threads increases). Although I imagine I'll probably tweak my methodology once I upgrade past a quadcore processor. Lastly,

SetMTMode(1)
GetMTMode(false) > 0 ? distributor() : last

is only for going to ffmpeg to encode as huffyuv or previewing in AvsPmod (uses ffmpeg), do not use with x264/avsmeter or other programs that recognize AviSynth MT and add the Distributor call internally.

[real.finder]

So far the new QTGMC mod is behaving itself with that sample for me. Even when using preset="slow".

Have I mentioned that for this sample at least (and for me), the other fix, aside from Trim(50,0) was to use MP_Pipeline? It fixed the problem for these particular encodes, but I'm fairly sure there's been a few encodes since where it didn't. Why MP_Pipeline changes things in relation to slowdowns, I have no idea.

I'm copying the numbers from MeGUI's log file, and keep in mind this PC's old so it's slow. I've listed the encoding speed for comparing QTGMC versions.

QTGMC 3.33s (new version) - 3.89fps.
QTGMC 3.33s with Preset="slow" - 3.11fps
QTGMC 3.33s with MP_Pipeline - 4.66fps

QTGMC 3.33d (original version) - around 1fps (I didn't bother running the whole encode)
QTGMC 3.33d with Preset="slow" - around 1fps (I didn't bother running the whole encode)
QTGMC 3.33d with MP_Pipeline - 4.69fps

I did add xflateMaTo parameter to use masktools2 deflate and inflate, so it worth doing some test again with last avs+ and plugins updates

I think it maybe also worth check for deinterlace quality with xflateMaTo=ture vs xflateMaTo=false