Avisynth+ - Page 119

tormento · 30th August 2016, 18:39

Quote:

Originally Posted by Groucho2004

I just found it odd that you put such a high value considering the low memory requirements of that particular script.

You may have to increase the default (512 MB) if you have a script with a large temporal range but you would test with small increments, 256 ~ 512.

It comes from my SMDegrain 6 frames span script + KNLMeans.

No idea of how AviSynth+ x64 deals with memory. I have plenty of it. I read that too much SetMemory could be negative too but it referred to old AviSynth. If any of the active devolver could clarify this aspect, would be very useful.

shekh · 30th August 2016, 18:56

Tried to wrap VD filter through avs with avsplus-r1858-pfmod

Quote:

Avisynth open failure:
rgb_levels has an invalid parameter string (bug in filter)

the script:

Quote:

AviSource("E:\vd_dev\data\cfhd-high.avi")
LoadPlugin("E:\vd_dev\vd\VDubFilter.dll")
LoadVirtualdubplugin("E:\vd_dev\vd\plugins32\rgb_levels.vdf", "rgb_levels")

filter (nothing really special): https://sourceforge.net/projects/vdf...s/version%202/

arg_list from filter definition: "ddddddddd" (9 floats)

What`s wrong?

ultim · 30th August 2016, 19:48

The memory and cache management in Avs+ is completely different from Avs 2.6. It has been redesigned and rewritten from scratch, though I was uncomfortably constrained by the existing API which resulted in half-assed code for memory reclamation in some (hopefully not too common) cases. Anyway, the caches in Avs+ are highly adaptive and self-learning, and do not use any caching hints like 2.6 in general to perform their jobs well. Also, as long as there is enough memory to satisfy all memory requests, even uncapped caches should not cause significant slowdowns, because caches will only grow if needed, not as long as there is free memory.

The default setting to SetMemoryMax() can be up to 1GB in Avs+, but may be lower depending on your amount of RAM. I know in earlier Avs 2.6 alpha/2.5 it was necessary to keep SetMemoryMax() low because if too high it resulted in funny caching behavior, leaving users with phenomena like slowdowns and/or unreasonably high memory usage. Avs+ should not suffer from anything like that, and SetMemoryMax() really just sets an (approximate) upper cap. If your script does not need that much memory though, a high upper limit (even if too high) does zero harm. In Avs+, the only use of constraining memory is to prevent paging. The justification is that often it is faster to recompute frames than to trash your rotational paging HDD (ofc if you have an SSD you'll have to reevaluate this claim). Another (similar) justification is, even if you had enough RAM, encodes often take many hours and sometimes even days, and if you want to keep using your computer in the background without hitting the paging limit, you might want to constrain the memory consumption of the encoding process.

But this is all TL;DR. The important takeaway is that in Avs+, a high SetMemoryMax() only hurts if otherwise your script causes excessive paging to a slow hard disk. Pro tipp: If you enable logging in Avs+, it will automatically notify you if you can speed up your script by setting your SetMemoryMax() higher

Edit: The optimal setting of SetMemoryMax() in Avs+ is anywhere in the range where it is higher than what your script wants to use, but lower than the value where it would cause paging. This range is often very broad, so only those will have trouble setting it who have either low RAM, or extremely hungry scripts. This is just a "rule of thumb" though, and its applicability will depend on your script and hardware.

TheFluff · 30th August 2016, 20:02

Speaking of which, does Avs+ reuse framebuffers like the OG Avisynth does?

ultim · 30th August 2016, 20:40

Quote:

Originally Posted by TheFluff

Speaking of which, does Avs+ reuse framebuffers like the OG Avisynth does?

Depends. Probably. Dunno what you mean exactly by "reuse".

ultim · 30th August 2016, 20:49

Quote:

Originally Posted by rean

I see AviSynth+ is based on the memory allocation code of legacy AviSynth...

Nope, not at all. See above.

TheFluff · 30th August 2016, 21:34

Quote:

Originally Posted by ultim

Depends. Probably. Dunno what you mean exactly by "reuse".

(this is from memory, might be completely wrong)
OG Avisynth refcounted vfb's and when the refcount reached zero, instead of freeing the buffer it was kept around and a subsequent call to NewVideoFrame for a suitably sized frame could use that buffer instead of allocating a new one.

I'm asking because as far as I know VS didn't bother with this and I seem to remember Myrsloik stating that simple free/new malloc for "small buffers" worked "well enough". I'm curious how much the buffer reuse technique helps, especially for larger video frame sizes.

ultim · 30th August 2016, 22:01

Quote:

Originally Posted by TheFluff

(this is from memory, might be completely wrong)
OG Avisynth refcounted vfb's and when the refcount reached zero, instead of freeing the buffer it was kept around and a subsequent call to NewVideoFrame for a suitably sized frame could use that buffer instead of allocating a new one.

I'm asking because as far as I know VS didn't bother with this and I seem to remember Myrsloik stating that simple free/new malloc for "small buffers" worked "well enough". I'm curious how much the buffer reuse technique helps, especially for larger video frame sizes.

Yes that technique is still there, and admittedly I intend to keep it around. Not because of some speed argument, but because it improves reliability. Simply put, it prevents memory fluctuation. It ensures that once Avs+ allocated a buffer, the same size can be allocated again after the previous instance is dereferenced. If Avs+ was to free each buffer whose refcount falls to zero, there'd be a chance that due to some other thread or application also allocating memory, Avisynth wouldn't be able to get a new buffer anymore, even though its total memory consumption didn't increase. This means without this technique there is a higher chance that your encode goes OutOfMemory in the middle, only because some other process or application thread had a sudden (maybe even temporary) memory spike.

Groucho2004 · 30th August 2016, 22:11

Quote:

Originally Posted by ultim

Yes that technique is still there, and admittedly I intend to keep it around. Not because of some speed argument, but because it improves reliability. Simply put, it prevents memory fluctuation. It ensures that once Avs+ allocated a buffer, the same size can be allocated again after the previous instance is dereferenced. If Avs+ was to free each buffer whose refcount falls to zero, there'd be a chance that due to some other thread or application also allocating memory, Avisynth wouldn't be able to get a new buffer anymore, even though its total memory consumption didn't increase. This means without this technique there is a higher chance that your encode goes OutOfMemory in the middle, only because some other process or application thread had a sudden (maybe even temporary) memory spike.

Thank you for that insight.

bilditup1 · 31st August 2016, 03:18

Moving from r1858_pfmod to r2172, I've been getting the following error when attempting to preview a fairly simply script in AvsPmod:

Code:

"Only a single prefetcher is allowed per script." (path_to_script.avs, line 13)

This is the script, which isn't particularly complex:

Code:

Import("C:\blu\MeGUI\tools\avisynth_plugin\reel.deel\mt_modes_latest.avsi")
LoadPlugin("C:\blu\MeGUI\tools\dgindex\DGDecode.dll")
LoadPlugin("C:\blu\MeGUI\tools\avisynth_plugin\jpsdr\x86\Release_Intel_W7_Core2_SSE4.2\nnedi3.dll")	
LoadPlugin("C:\blu\MeGUI\tools\avisynth_plugin\tp7\RgTools.dll")			
LoadPlugin("C:\blu\MeGUI\tools\avisynth_plugin\tp7\MaskTools2.dll")
LoadPlugin("C:\blu\MeGUI\tools\avisynth_plugin\pinterf\MvTools2.7.0.22d.dll")		
Import("C:\blu\MeGUI\tools\avisynth_plugin\real_finder\QTGMC.avsi")
DGDecode_mpeg2source("d2v-path-here") # 1080i30 mpeg2 src
AssumeBFF()
QTGMC(Preset="Fast", ShowSettings=False)
Spline36Resize(1280,720)
Trim(0,3000)
Prefetch(4)

I checked the scripts I was importing - reel.deel's mt modes list and QTGMC - but Prefetch() is never called in them.
This error message must be a recent change, as in r2161, AvsPmod just crashes entirely, and when attempting to encode, the thread simply stalls and makes no progress (the same thing).
Removing the Prefetch call allows the script to be processed, in both r2161 and r2172, but I don't think mt is invoked, as it runs at st speeds (6.5fps vs 8.91).
I thought this was a problem with jpsdr's nnedi3, which I had to update to r25/r26 from r22 in order to avoid a memory violation with these new releases. But I reduced to script to merely this:

Code:

LoadPlugin("C:\blu\MeGUI\tools\dgindex\DGDecode.dll")
DGDecode_mpeg2source("d2v-path-here") # 1080i30 mpeg2 src
Prefetch(4)

And still get the same error. Is this somehow a quirk of dgdecode? Is mt broken? Is it enabled differently now? Or is something else at work here?

ED: It is in fact DGDecode.dll. Using DGDecodeIM.dll and a dgi instead did not present this error - not only that but the encode goes some 50% faster (from 9 to 13.5 fps) than the original script did under r1858_pfmod and DGDecode (though the first time I tried it, it quit after one frame with a typical memory access error.)
Does anybody know what's going on with DGDecode? I guess it could be considered legacy software by now, and presume that maintaining compatibility with it probably is not a priority, but this strikes me as odd behavior all the same.

ED2: Part of the higher speed and memory access errors I had been experiencing appears to be because I had erroneously switched to using the XP compatible build of nnedi3. The W7_SSE4.2 build still goes faster than before but less dramatically (12fps instead of 13.5fps, and trying the XP build again just now I only managed to reach 12.6fps, weirdly), and you pay for it with the danger of a memory problem. I've also experienced what I believe are memory issues when using nnedi3 with jpsdr's ResampleMT (latest) - iow encode just stops in the middle but without quitting or throwing an error, just sticks at a certain frame - though not consistently: when it does work then it brings the speed back up to 13.5 even whilst using the W7_SSE4.2 builds of both it and nnedi3. (Will also post about it in jpsdr's threads, I suppose, if someone hasn't already pointed this out.)

ED3: Groucho2004's build of DGDecode has the same problem as neuron2's.

ultim · 31st August 2016, 05:42

Quote:

Originally Posted by bilditup1

Moving from r1858_pfmod to r2172, I've been getting the following error when attempting to preview a fairly simply script in AvsPmod:

Code:

"Only a single prefetcher is allowed per script." (path_to_script.avs, line 13)

This should be a regression in Avs+ in the recent build. I'll take care of it by the weekend.

bilditup1 · 31st August 2016, 06:32

Quote:

Originally Posted by ultim

This should be a regression in Avs+ in the recent build. I'll take care of it by the weekend.

Wow, thanks for quick reply. Looking fwd to fix. Unfortch I've just discovered that TDecimate does not work with MT enabled, whether using tritical's original TIVTC.dll or groucho2004's build. Works fine in r1858_pfmod, quits with "internal error during prebuffering!" in r2085, r2161, r2172.

tormento · 31st August 2016, 08:34

Quote:

Originally Posted by ultim

The default setting to SetMemoryMax() can be up to 1GB in Avs+

So, having 16+ GB of RAM and a x64 environment, which is the maximum value we can set?

Is it of any use or we can write scripts without allocating memory?

Groucho2004 · 31st August 2016, 08:48

Quote:

Originally Posted by bilditup1

I've been getting the following error when attempting to preview a fairly simply script in AvsPmod:
...
This is the script, which isn't particularly complex:

Code:

Import("C:\blu\MeGUI\tools\avisynth_plugin\reel.deel\mt_modes_latest.avsi")
LoadPlugin("C:\blu\MeGUI\tools\dgindex\DGDecode.dll")
LoadPlugin("C:\blu\MeGUI\tools\avisynth_plugin\jpsdr\x86\Release_Intel_W7_Core2_SSE4.2\nnedi3.dll")	
LoadPlugin("C:\blu\MeGUI\tools\avisynth_plugin\tp7\RgTools.dll")			
LoadPlugin("C:\blu\MeGUI\tools\avisynth_plugin\tp7\MaskTools2.dll")
LoadPlugin("C:\blu\MeGUI\tools\avisynth_plugin\pinterf\MvTools2.7.0.22d.dll")		
Import("C:\blu\MeGUI\tools\avisynth_plugin\real_finder\QTGMC.avsi")
DGDecode_mpeg2source("d2v-path-here") # 1080i30 mpeg2 src
AssumeBFF()
QTGMC(Preset="Fast", ShowSettings=False)
Spline36Resize(1280,720)
Trim(0,3000)
Prefetch(4)

The script is far from simple, have a look at the content of qtgmc.avsi. As for your stability problems (aside from a AVS+ issue), see also here.

shekh · 31st August 2016, 08:56

Quote:

Originally Posted by shekh

Tried to wrap VD filter through avs with avsplus-r1858-pfmod

Avisynth open failure:
rgb_levels has an invalid parameter string (bug in filter)
...

same with 2172

burfadel · 31st August 2016, 11:27

Ultim, I hope I wasn't 'out-of-line' PM'ing you earlier with the script sample, I just wanted to avoid stoking the fire of the other threads regarding upsampling, changing resolutions etc., seeing as the script did involve some manipulation in that regard. I hope that it does prove useful though in what I mentioend regarding single thread and multithreaded modes etc, and also how it can run quite fast.

Is my assumption wrong in that any script either becomes all multithreaded on the avisynth side, even with MT mode 3 on the plugins, or single threaded on the Avisynth side when prefetch is not stated, but still multithreaded invidividual plugins if the plugins are multithreaded? If the former isn't right, it does seem to exhibit that behaviour.

bilditup1 · 31st August 2016, 11:49

Quote:

Originally Posted by Groucho2004

The script is far from simple, have a look at the content of qtgmc.avsi.

Yes, I'm aware that QTGMC itself is complex. I meant that my snippet was legible, straightforward and didn't have much of my own more-likely-to-be-faulty logic in it. In any case, QTGMC's presence was ultimately irrelevant.

Wilbert · 31st August 2016, 18:52

Quote:

Originally Posted by tormento

https://mega.nz/#!dxowVRCR!j5OaMcXwt...RXapah-UKQJlP0

What Groucho2004 said. You have to be patience. I imported your script in Virtualdub and compressed it to Xvid but it took me 10 hours on my slow laptop (2.2 GHz). I used plain AviSynth though, but that probably doesn't matter. Btw, it consists of 324 images (iirc).

Groucho2004 · 31st August 2016, 22:37

Quote:

Originally Posted by Wilbert

Btw, it consists of 324 images (iirc).

Should be 434.

qyot27 · 1st September 2016, 00:04

Quote:

Originally Posted by AzraelNewtype

There is nothing ordinary about a nearly CD sized animated gif.

I've noticed this happening on image hosting sites that offer a gif version and a webm version of whatever the animated image sequence is. The gifs in those cases are sometimes outlandishly sized, because I assume what's going on is the image host offers the ability to upload video, so the user uploads a video file, and then the image host compresses it to webm and to gif. But the gif's resolution hasn't been reduced, so you end up with ridiculous 1920x1080 animated gifs alongside a smaller, higher quality normal video file.

I keep asking myself what the use case of that could possibly be vs just using a video host, but I've never come up with a satisfactory answer. Because even meme sites like these gif posts probably are intended for should really have no issues embedding video files in 2016, and if your computer is so old that it can't decode one of those VP8 or VP9 webm files, is your browser going to somehow not choke on a 1080p gif?

30th August 2016, 19:48	#2363 \| Link
ultim AVS+ Dev Join Date: Aug 2013 Posts: 359	The memory and cache management in Avs+ is completely different from Avs 2.6. It has been redesigned and rewritten from scratch, though I was uncomfortably constrained by the existing API which resulted in half-assed code for memory reclamation in some (hopefully not too common) cases. Anyway, the caches in Avs+ are highly adaptive and self-learning, and do not use any caching hints like 2.6 in general to perform their jobs well. Also, as long as there is enough memory to satisfy all memory requests, even uncapped caches should not cause significant slowdowns, because caches will only grow if needed, not as long as there is free memory. The default setting to SetMemoryMax() can be up to 1GB in Avs+, but may be lower depending on your amount of RAM. I know in earlier Avs 2.6 alpha/2.5 it was necessary to keep SetMemoryMax() low because if too high it resulted in funny caching behavior, leaving users with phenomena like slowdowns and/or unreasonably high memory usage. Avs+ should not suffer from anything like that, and SetMemoryMax() really just sets an (approximate) upper cap. If your script does not need that much memory though, a high upper limit (even if too high) does zero harm. In Avs+, the only use of constraining memory is to prevent paging. The justification is that often it is faster to recompute frames than to trash your rotational paging HDD (ofc if you have an SSD you'll have to reevaluate this claim). Another (similar) justification is, even if you had enough RAM, encodes often take many hours and sometimes even days, and if you want to keep using your computer in the background without hitting the paging limit, you might want to constrain the memory consumption of the encoding process. But this is all TL;DR. The important takeaway is that in Avs+, a high SetMemoryMax() only hurts if otherwise your script causes excessive paging to a slow hard disk. Pro tipp: If you enable logging in Avs+, it will automatically notify you if you can speed up your script by setting your SetMemoryMax() higher Edit: The optimal setting of SetMemoryMax() in Avs+ is anywhere in the range where it is higher than what your script wants to use, but lower than the value where it would cause paging. This range is often very broad, so only those will have trouble setting it who have either low RAM, or extremely hungry scripts. This is just a "rule of thumb" though, and its applicability will depend on your script and hardware. __________________ AviSynth+ Last edited by ultim; 30th August 2016 at 19:56.

31st August 2016, 03:18	#2370 \| Link
bilditup1 Registered User Join Date: Feb 2004 Location: NYC Posts: 124	DGDecode_mpeg2source no longer works in MT Moving from r1858_pfmod to r2172, I've been getting the following error when attempting to preview a fairly simply script in AvsPmod: Code: "Only a single prefetcher is allowed per script." (path_to_script.avs, line 13) This is the script, which isn't particularly complex: Code: Import("C:\blu\MeGUI\tools\avisynth_plugin\reel.deel\mt_modes_latest.avsi") LoadPlugin("C:\blu\MeGUI\tools\dgindex\DGDecode.dll") LoadPlugin("C:\blu\MeGUI\tools\avisynth_plugin\jpsdr\x86\Release_Intel_W7_Core2_SSE4.2\nnedi3.dll") LoadPlugin("C:\blu\MeGUI\tools\avisynth_plugin\tp7\RgTools.dll") LoadPlugin("C:\blu\MeGUI\tools\avisynth_plugin\tp7\MaskTools2.dll") LoadPlugin("C:\blu\MeGUI\tools\avisynth_plugin\pinterf\MvTools2.7.0.22d.dll") Import("C:\blu\MeGUI\tools\avisynth_plugin\real_finder\QTGMC.avsi") DGDecode_mpeg2source("d2v-path-here") # 1080i30 mpeg2 src AssumeBFF() QTGMC(Preset="Fast", ShowSettings=False) Spline36Resize(1280,720) Trim(0,3000) Prefetch(4) I checked the scripts I was importing - reel.deel's mt modes list and QTGMC - but Prefetch() is never called in them. This error message must be a recent change, as in r2161, AvsPmod just crashes entirely, and when attempting to encode, the thread simply stalls and makes no progress (the same thing). Removing the Prefetch call allows the script to be processed, in both r2161 and r2172, but I don't think mt is invoked, as it runs at st speeds (6.5fps vs 8.91). I thought this was a problem with jpsdr's nnedi3, which I had to update to r25/r26 from r22 in order to avoid a memory violation with these new releases. But I reduced to script to merely this: Code: LoadPlugin("C:\blu\MeGUI\tools\dgindex\DGDecode.dll") DGDecode_mpeg2source("d2v-path-here") # 1080i30 mpeg2 src Prefetch(4) And still get the same error. Is this somehow a quirk of dgdecode? Is mt broken? Is it enabled differently now? Or is something else at work here? ED: It is in fact DGDecode.dll. Using DGDecodeIM.dll and a dgi instead did not present this error - not only that but the encode goes some 50% faster (from 9 to 13.5 fps) than the original script did under r1858_pfmod and DGDecode (though the first time I tried it, it quit after one frame with a typical memory access error.) Does anybody know what's going on with DGDecode? I guess it could be considered legacy software by now, and presume that maintaining compatibility with it probably is not a priority, but this strikes me as odd behavior all the same. ED2: Part of the higher speed and memory access errors I had been experiencing appears to be because I had erroneously switched to using the XP compatible build of nnedi3. The W7_SSE4.2 build still goes faster than before but less dramatically (12fps instead of 13.5fps, and trying the XP build again just now I only managed to reach 12.6fps, weirdly), and you pay for it with the danger of a memory problem. I've also experienced what I believe are memory issues when using nnedi3 with jpsdr's ResampleMT (latest) - iow encode just stops in the middle but without quitting or throwing an error, just sticks at a certain frame - though not consistently: when it does work then it brings the speed back up to 13.5 even whilst using the W7_SSE4.2 builds of both it and nnedi3. (Will also post about it in jpsdr's threads, I suppose, if someone hasn't already pointed this out.) ED3: Groucho2004's build of DGDecode has the same problem as neuron2's. Last edited by bilditup1; 31st August 2016 at 05:31. Reason: ED2

30th August 2016, 20:02	#2364 \| Link
TheFluff Excessively jovial fellow Join Date: Jun 2004 Location: rude Posts: 1,100	Speaking of which, does Avs+ reuse framebuffers like the OG Avisynth does?

31st August 2016, 11:27	#2376 \| Link
burfadel Registered User Join Date: Aug 2006 Posts: 2,229	Ultim, I hope I wasn't 'out-of-line' PM'ing you earlier with the script sample, I just wanted to avoid stoking the fire of the other threads regarding upsampling, changing resolutions etc., seeing as the script did involve some manipulation in that regard. I hope that it does prove useful though in what I mentioend regarding single thread and multithreaded modes etc, and also how it can run quite fast. Is my assumption wrong in that any script either becomes all multithreaded on the avisynth side, even with MT mode 3 on the plugins, or single threaded on the Avisynth side when prefetch is not stated, but still multithreaded invidividual plugins if the plugins are multithreaded? If the former isn't right, it does seem to exhibit that behaviour.