Avisynth+

[real.finder]

and you can use MP_Pipeline and load 1576 dll in it
and use the script any where you want regardless of the version you have installed in the system

[NightSprinter]

This is odd, running a Core i7 920 (4 cores, 8 threads) with everything done as posted on the tutorial. However, when I enable prefetch, I get an error along the lines of "I don't know what 'zb' means". I'm using the latest AVS+ installer from the wiki page, and the filter script chain I'm using is the CRT simulation filter that's on this forum as well. I'm including the script in .txt format

[Reel.Deel]

This is odd, running a Core i7 920 (4 cores, 8 threads) with everything done as posted on the tutorial. However, when I enable prefetch, I get an error along the lines of "I don't know what 'zb' means". I'm using the latest AVS+ installer from the wiki page, and the filter script chain I'm using is the CRT simulation filter that's on this forum as well. I'm including the script in .txt format

Line 269 of the crt_display.avsi:
zb = mt_lut (y=0)
From your script it seems you're using MaskTools V2.0a48. This version is incompatible with AviSynth 2.6/AviSynth+, use the update version here (https://github.com/tp7/masktools/releases).

[NightSprinter]

I tried using the version you linked to, but it appears to be the same file if not smaller than the one included with AviSynth+. Using the linked file gives me the exact same error.

[real.finder]

I am playing back and forth between Avisynth 2.6.0MT and AviSynth+ 1825 MT, testing speed and image quality.

I went thru the following image corruption.

AviSynth output:
http://i.imgur.com/8MGljqrm.jpg (http://imgur.com/8MGljqr)

from script:
SetMTMode(3)
SetMemoryMax(2048)
LoadPlugin("D:\eseguibili\media\DGDecIM\DGDecodeIM.dll")
DGSourceIM("E:\in\2_01 favoloso mondo di Amelie, Il\amelie.dgi",engine=1)
SetMTMode(2)
CompTest(1)
ChangeFPS(last,last,true)
Crop(0,132,0,-140)
SMDegrain (tr=4,PreFilter=4,thSAD=400,contrasharp=false,refinemotion=false,plane=4,chroma=true,lsb=true,mode=6)


AviSynth+ output:
http://i.imgur.com/PXXdVqam.jpg (http://imgur.com/PXXdVqa)
from script:
SetFilterMTMode("DEFAULT_MT_MODE", 2)
SetFilterMTMode("DGSourceIM", 3)
SetMemoryMax(2048)
LoadPlugin("D:\eseguibili\media\DGDecIM\DGDecodeIM.dll")
DGSourceIM("E:\in\2_01 favoloso mondo di Amelie, Il\amelie.dgi",engine=1)
CompTest(1)
ChangeFPS(last,last,true)
Crop(0,132,0,-140)
SMDegrain (tr=4,PreFilter=4,thSAD=400,contrasharp=false,refinemotion=false,plane=4,chroma=true,lsb=true,mode=6)
Prefetch(4)
Any idea?

it will be better if you put a sample

I remember your problem after I remember http://forum.doom9.org/showpost.php?p=1675980&postcount=718 and play with that sample with http://www.mediafire.com/download/iw507kav586cp6t/ffms2-r734-libav-e6160bd.7z and reproduce it, with the last ffms2 can't reproduce, maybe this has relation to your problem

[tormento]

and reproduce it, with the last ffms2 can't reproduce, maybe this has relation to your problem

As you can see I don't use ffms2 but DGSourceIM

[martin53]

This filter
ColorBars(pixel_type="RGB32")
LoadVirtualDubPlugin("...\MSU_SmartDeblock_0.8.vdf","MSU_Smart_DeblockingVDF", 0)
MSU_Smart_DeblockingVDF(1, 1, 1, 1)
loads thanks to VDubFilter.dll (3/23/2015), but then crashes with System exception - Access Violation
(...vdcrash.avs, line 3)

Does anyone know of help? I feel I need MSU_SmartDeblock, and this keeps me once more from moving to Avisynth+.

The system is a Win7 uptimate x64 with only x86 Avisynth + plugins. The script and everything works with latest Avisynth 2.6 and Avisynth 2.6 MT.
Oh, and Avisynth+ is r1815-MT

[Groucho2004]

The system is a Win7 uptimate x64 with only x86 Avisynth + plugins. The script and everything works with latest Avisynth 2.6 and Avisynth 2.6 MT.
Oh, and Avisynth+ is r1815-MT
Works fine with the stable version of AVS+ (r1576).

[jihyo]

no function SetMTMode :( (and some others)

[Groucho2004]

no function SetMTMode
No, that's specific to SEt's MT builds.
Documentation for AVS+ can be found here (http://avisynth.nl/index.php/AviSynth%2B).

[jihyo]

No, that's specific to SEt's MT builds.

Got it.

Who can help me to update the code with SetFilterMTMode of AS+?
SetMTMode(6,#CPUS * 2)
SetMTMode(2)
LoadPlugin("#PROGRAMDIR\AviSynth\plugins\InterFrame-2.8.2\Dependencies\svpflow1.dll")
LoadPlugin("#PROGRAMDIR\AviSynth\plugins\InterFrame-2.8.2\Dependencies\svpflow2.dll")
Import("#PROGRAMDIR\AviSynth\plugins\InterFrame-2.8.2\InterFrame2.avsi")
InterFrame(Preset="Medium", Tuning="Animation", GPU=true, Cores=#CPUS)
SetMTMode(1)
GetMTMode(false) > 0 ? distributor() : last

[foxyshadis]

Get this script (http://publishwith.me/ep/pad/view/ro.rDkwcdWn4k9/latest), save it and import it. Remove all the *MTMode lines from the script you posted, and you're done. Avs+ handles the rest.

[qyot27]

Get this script (http://publishwith.me/ep/pad/view/ro.rDkwcdWn4k9/latest), save it and import it. Remove all the *MTMode lines from the script you posted, and you're done. Avs+ handles the rest.
The GetMTMode/Distributor line needs to be replaced with Prefetch(# of threads) to actually enable MT.

[siella]

I've got this error messega "There is no function named 'SSRC'"

[Groucho2004]

I've got this error messega "There is no function named 'SSRC'"
In AVS+, this function has been moved to an external plugin (shibatch.dll). When you install AVS+, this plugin would be copied to the auto-load directory an you should not receive that message.
You have to elaborate a bit in order troubleshoot this.
- Did you use the AVS+ installer?
- Which version are you using, 64 or 32 bit?
- Which program throws the error message?

[siella]

In AVS+, this function has been moved to an external plugin (shibatch.dll). When you install AVS+, this plugin would be copied to the auto-load directory an you should not receive that message.
You have to elaborate a bit in order troubleshoot this.
- Did you use the AVS+ installer?
- Which version are you using, 64 or 32 bit?
- Which program throws the error message?

Thanks i found avisynth+ with ssrc from megui

[Octo-puss]

So just out of curiosity, is this project dead and left abandoned in unfinished state as it appears?

[qyot27]

More like extended hiatuses. The same thing happened between March 2014 and February 2015.

IMO, this could be ameliorated by getting the support for GCC and Linux banged out, since that would expand the ability for collaboration. Properly getting it to play nice with GCC's braindead handling of intrinsics is something I've attempted to do a couple times but gotten sidetracked and haven't gone back.

[Octo-puss]

It's just that the last commit on Github is from december 2013, so one naturally assumes the project is dead :) If it's not the case, good.

[qyot27]

No, the last commit on Github is from March 2015. The master branch isn't the current development branch; MT is.

[THEAST]

I used the MT build 1689 for a while alongside with QTGMC without any issues and encoded a lot of videos. Later I switched to latest MT build 1825 but didn't encode anything for a while. Recently I started encoding again and realized that all my of encodes run into a deadlock very early into the job. After going back and force between builds 1689, 1779 and 1825, I noticed that even though the exact same script runs fine under 1689, it deadlocks under 1779 and 1825, while all of the three versions give more or less the same speed (checked with AVSMeter). Anybody has any idea about this? My scripts look like this:

1689:
SetMemoryMax(8000)
SetFilterMTMode("", 2)
#SetFilterMTMode("DEFAULT_MT_MODE", 2)
LoadPlugin("DGDecode.dll")
DGDecode_mpeg2source("video.d2v", info=3)
LoadPlugin("ColorMatrix.dll")
ColorMatrix(hints=true, threads=0)
TRIM(x,x)
crop(x,x,x,x)
QTGMC( Preset="Very Fast", x,x,x,x,x,x ).SelectEven()
Prefetch(6)

1779 & 1825:
SetMemoryMax(8000)
#SetFilterMTMode("", 2)
SetFilterMTMode("DEFAULT_MT_MODE", 2)
LoadPlugin("DGDecode.dll")
DGDecode_mpeg2source("video.d2v", info=3)
LoadPlugin("ColorMatrix.dll")
ColorMatrix(hints=true, threads=0)
TRIM(x,x)
crop(x,x,x,x)
QTGMC( Preset="Very Fast", x,x,x,x,x,x ).SelectEven()
Prefetch(6)

[thescrapyard]

Reduce your memory, maybe your running out of memory. 8GB I think is a bit much, I use no more than 2GB (2048). Unless you have 32GB of memory. I've got 16GB and are now running with an i7 slightly overclocked

Try reducing the number of threads as I think each thread has its own memory requirements, maybe start at 2 threads and work up until it deadlocks again

[Groucho2004]

Recently I started encoding again and realized that all my of encodes run into a deadlock very early into the job. After going back and force between builds 1689, 1779 and 1825, I noticed that even though the exact same script runs fine under 1689, it deadlocks under 1779 and 1825
Start reading here (http://forum.doom9.org/showthread.php?p=1736090#post1736090) for a potential cause for your issue.

[THEAST]

Reduce your memory, maybe your running out of memory. 8GB I think is a bit much, I use no more than 2GB (2048). Unless you have 32GB of memory. I've got 16GB and are now running with an i7 slightly overclocked

Try reducing the number of threads as I think each thread has its own memory requirements, maybe start at 2 threads and work up until it deadlocks again

I have 16 GB of memory, I don't think Avisynth x86 can use more than 2 GB memory anyway, that number I am using is mostly for cosmetics. In my experience, on the latest builds, the code deadlocks even with two threads, it might just take longer. The only way to get it to work is to remove the "prefetch" line in which case the avisynth part of the work will run in single-threaded mode. Even in that case, the encode starts at something like 8 FPS but the speed gradually goes down until it becomes 1 FPS or so.

Start reading here (http://forum.doom9.org/showthread.php?p=1736090#post1736090) for a potential cause for your issue.

That doesn't seem to be a multi-threaded test, at least based on the reported CPU usage values. Still, it confirms my observations regarding the encoding speed going down with time using the latest builds. Do you guys recommend sticking with avisynth+ for MT or switching to normal Avisynth and SET's MT dll? I might try build r1765 later and see if the deadlock also happens there. I have a feeling this issue started when the syntax for setting default MT mode changed from "SetFilterMTMode("", 2)" to "SetFilterMTMode("DEFAULT_MT_MODE", 2)".

[Groucho2004]

I have 16 GB of memory, I don't think Avisynth x86 can use more than 2 GB memory anyway, that number I am using is mostly for cosmetics.
Cosmetics? WTF? RTFM.

All credit to ultim and the rest of the AVS+ dev team but the latest builds, particularly with MT and QTGMC(), are not stable. If you really have to multi-thread QTGMC(), use SEt's AVS MT.

Since you're using the "very fast" QTGMC() preset, I don't think you even need Avisynth(+) MT. Some of the filters are already internally multi-threaded. Your bottleneck is most likely the encoding.

For reference, here (http://forum.doom9.org/showthread.php?p=1758254#post1758254) are some benchmarks with SEt's AVS MT.

[THEAST]

Cosmetics as in that amount is supposed to be some absolute maximum that avisynth+ is never going to reach; trying to follow the "better safe than sorry" approach, even though it might not make much sense here. My script seems to use 400-500 MB of memory with 6 threads, on build 1689.

Regarding QTGMC, I am not actually using the Very Fast preset out of the box. The preset just acts as a baseline and I have like 15-20 of QTGMC's options manually set (I removed them here for the sake of brevity), which I will modify based on the source or encoding speed constraints. The final result is something along the lines of slow or very slow.

I remember I did some comparison between Avisynth+ MT and SEt's MT build a while back and the former was quite a bit faster and hence, I stuck with Avisynth+. It was before Avisynth v2.6.0 final, though. Maybe I should take a look at SEt's MT build's performance again.

[Groucho2004]

Cosmetics as in that amount is supposed to be some absolute maximum that avisynth+ is never going to reach; trying to follow the "better safe than sorry" approach
Had you read the documentation for SetMemoryMax (http://avisynth.nl/index.php/Internal_functions/SetMemoryMax#SetMemoryMax) you would know that your interpretation of the purpose of that function is completely wrong. You should hardly ever have to set it to more than about 1000.

[THEAST]

Had you read the documentation for SetMemoryMax (http://avisynth.nl/index.php/Internal_functions/SetMemoryMax#SetMemoryMax) you would know that your interpretation of the purpose of that function is completely wrong. You should hardly ever have to set it to more than about 1000.

Hmm, okay, I will change the value that I am using, thanks for the tip.

[pinterf]

Hello,

Last week I was starting to play with avs+ source, specifically with the MT branch because I was interested in the background of the slowdown issue.
The issue was probably solved, but before I upload test dlls I would like to ask some questions.

Question#1

I'm working with VS2015 on 64 bit Windows 7.
I have cloned the MT branch, run cmake and built x86. After minor modifications I succeeded.
Then tried to build x64 but this configuration did not exist, so I created one from VS Configuration manager. Linking was not successful, because the machine type of obj files did not match the linking target, even if I set it explicitly in the project properties (under Linker, target machine type).
LNK1112: module machine type 'x64' conflicts with target machine type 'X86'
I think that the generated cmake configuration is overriding this setting.

The only workaround I found was the following:

Compile for x86:
download cmake x86
delete CMakeCache.txt from project root
run cmake-gui from c:\Program Files (x86)\cmake\bin
choose directories
configure
Specify the generator for this project: Visual Studio 14 2015
(appears /machine=x86 in xxxxx_LINKER_FLAGS)
generate
Start visual studio, only Win32 is available: build.

Compile for x64:
download cmake x64
delete CMakeCache.txt from project root
run cmake gui from c:\Program Files\cmake\bin
choose directories
configure
Specify the generator for this project: Visual Studio 14 2015 Win64
(appears /machine=x64 in xxxxx_LINKER_FLAGS)
generate
Start visual studio, only x64 is available: build.

My question is that how can I tell cmake to generate the solution for both Win32 and x64 targets?

Question#2
Forgive me, I am new to git and these open source things (it was a whole day for me to see only MT branch as a separete source :) ), what is the usual workflow if I'd like to participate in the project?
Fork the project to my git account, and make modification inside it? Clone it?
How is it done if I put a lot of debug mess in my source (and don't want to remove it on my side) but the original project does not need them? Can the owner of the main project merge only specific lines?

Question#3
When I build the dll's they have the very same version (build number, etc. as the latest 'official' release. Version.h is autogenerated by some magic cmake thing.
How can I specify this version manually? E.g. 1825-test instead of 1825?

Thank you and sorry for the lame questions :)

[LigH]

Just a small remark: hg (Mercurial) is not completely equal to git.

[Reel.Deel]

Hello,

Last week I was starting to play with avs+ source, specifically with the MT branch because I was interested in the background of the slowdown issue.
The issue was probably solved, but before I upload test dlls I would like to ask some questions.


Awesome, I'll gladly help out with testing. Hopefully someone comes along and answer your other questions.

[qyot27]

My question is that how can I tell cmake to generate the solution for both Win32 and x64 targets?
There's no need to. AviSynth+'s build system is via CMake, not MSVC project files. CMake doesn't work this way (to my knowledge). It certainly would cause problems for the other generators - NMake and Ninja - that you can use with MSVC's compilers.

And you don't need 64-bit CMake to do this either, since the Win64-specific generator calls in the correct MSVC toolchain regardless of what CMake is.

Question#2
Forgive me, I am new to git and these open source things (it was a whole day for me to see only MT branch as a separete source :) ), what is the usual workflow if I'd like to participate in the project?
Fork the project to my git account, and make modification inside it? Clone it?
How is it done if I put a lot of debug mess in my source (and don't want to remove it on my side) but the original project does not need them? Can the owner of the main project merge only specific lines?
This is the usual for git projects:
Clone project
Create new branch with descriptive name
Make modifications, commit changes (broken up topically, usually).
Submit pull request (Github) or patchset (on mailing list oriented projects) to upstream.
Upstream reviews patches, suggests changes, and either allows the user to fix those in the patchset or sometimes upstream does it after merging it.
If you want to 'put a lot of debug mess in my source (and don't want to remove it on my side)', I'd suggest keeping a branch specifically *for* that, and having a pull request-ready branch with the actual changes but without the debug stuff.

Also remember that AviSynth+ is extraordinarily unlikely to accept compiler-specific stuff, what with sporadic efforts to move beyond MSVC for compilation. See the coding guidelines (http://avs-plus.net/coding-guidelines.php) for more general information on what should or should not be acceptable.

Question#3
When I build the dll's they have the very same version (build number, etc. as the latest 'official' release. Version.h is autogenerated by some magic cmake thing.
How can I specify this version manually? E.g. 1825-test instead of 1825?
Don't do that. Make sure the CMake versioning generator is working correctly and allow it to do its job. If you want to keep your branched DLLs straight, I'd put the branch name in the filename of the archive you pack the DLLs in.

[qyot27]

On a quasi-related note, I've been meaning to check the build instructions I posted before (http://forum.doom9.org/showpost.php?p=1643929&postcount=7) against VS Community 2015, but haven't gotten around to it. They still should work, just change the -G parameter (and optionally the -T parameter if XP support still matters to you) to the correct name.

[pinterf]

This is the usual for git projects:
Clone project
Create new branch with descriptive name
Make modifications, commit changes (broken up topically, usually).
Submit pull request (Github) or patchset (on mailing list oriented projects) to upstream.
Upstream reviews patches, suggests changes, and either allows the user to fix those in the patchset or sometimes upstream does it after merging it.
If you want to 'put a lot of debug mess in my source (and don't want to remove it on my side)', I'd suggest keeping a branch specifically *for* that, and having a pull request-ready branch with the actual changes but without the debug stuff.


Thank you, suppose you mean "fork" in the first step?

Regarding the multi target make file, I have read your instructions, but I was hoping that CMake evolved in the last one or two years to do that.

Just recognized that VS2015 has git integration.
Now it points to the official MT repository, tell me that I cannot do any harm (commit, false merge) to that, wherever I click with my mouse to :)

[LigH]

No, "clone" means "get a copy from the remote repository to your local working directory" in git slang (CVS or SVN use "checkout", IIRC).

[pinterf]

After two weeks of learning curve, I have uploaded a new avs+ build, both 32 and 64 bit MT dlls.

This build is primarily for addressing the huge slowdown issue of recent MT builds.

Please give it a try.

AviSynth+ r1828-pfmod (http://www.mediafire.com/download/csrdoqj5tesi640/avsplus1828-pfmod.7z)

I have no slowdowns (none or very minor), so longer clips could be tested: QTGMC fast with a 92 minutes DV-AVI source.

Some notes.

The artificial test case of colorbars is not quite good, although it allowed me to find out the main cause of the slowdown. Colorbars is using a static source framebuffer that is kept referenced throughout the whole qtgmc process, so its ref_count will never be released. During the qtgmc script the framebuffer-reusing logic had to iterate many hundred thousand (!) entries until it would find a free entry or see that no free buffer exists. After thousands of frames, one failed lookup needed almost 0.03 seconds (normally it should lookup 1000-10000x faster!), and it got linearly worse.

Todo: "hack" or parameterize colorbar to be able to generate new frames: copy the pre-generated pattern into a brand new frame instead of just linking to it.

Having no slowdown I realized the next problem: framebuffers sometimes are not released.

Example.
The first 17800 frames of an AVI source needs approx. 200 MBytes of framebuffers to deinterlace with Qtgmc, number of framebuffers is 300. Suddenly avs+ cannot find any buffers that has reference count 0 and starts allocating new framebuffers. Reaching the 20000 frames limit (that is 40000 frames after Qtgmc) we need already 500MB, so with SetMemoryMax(512) the script is soon exiting.

And later the memory consumption goes up in waves.

40525 frames (81050 QTGMC): 973 framebuffers, 613MB
49381 frames (99600 QTGMC): 1178 framebuffers, 745MB
56725 frames (113450 QTGMC): 1255 framebuffers, 816MB
61046 frames (122092 QTGMC): 1348 framebuffers, 876MB
65084 frames (130168 QTGMC): 1508 framebuffers, 980MB
68988 frames (137978 QTGMC): 1736 framebuffers, 1128MB
114512 frames (129024 QTGMC): 2121 framebuffers, 1376MB
etc...

Good news: my 1h32min 720x576 dv-avi video - that has finally 278000 50p frames - just succeeded with the SetMemoryMax(2048) setting. Needed approx. 1700MB memory.

But it can be done with much less memory.
Whick plugin or avs core logic is getting those frames stuck?
Have to figure out, how. Seems it is not related to MT.

Script (+having rgtools, mvtools2, masktools2.dll and nnedi3.dll in the plugins64 directory)

SetFilterMTMode("DEFAULT_MT_MODE",2)
SetFilterMTMode("AviSource",3)
SetMemoryMax(2048)
#colorbars(width = 640, height = 480, pixel_type = "yv12").killaudio().assumefps(25, 1).trim(0, 29999)
Avisource("c:\tape13\Videos\Tape02_digitall_Hi8.avi").killaudio().assumefps(25,1)#.trim(0, 29999)
AssumeBFF()
QTGMC(Preset="Fast")
prefetch(3)

[ryrynz]

Gosh, might we finally have an MT build stable enough to use as a replacement to SEt's MT? Or perhaps sometime soon? Nice to see some progress finally!
Will see how it fares, thanks for the considerable contribution!

[LigH]

Your efforts are very appreciated, pinterf.

:thanks:

[pinterf]

One note, my mod was built with VS2015, v140_xp toolset.
I have no "virgin" machine, so it works out-of-box for me. It may require Visual C++ Redistributable for Visual Studio 2015 Update 1 (https://www.microsoft.com/en-us/download/details.aspx?id=49984) for you.

[Reel.Deel]

After two weeks of learning curve, I have uploaded a new avs+ build, both 32 and 64 bit MT dlls.

This build is primarily for addressing the huge slowdown issue of recent MT builds.

Please give it a try.

AviSynth+ r1828-pfmod (http://www.mediafire.com/download/csrdoqj5tesi640/avsplus1828-pfmod.7z)

I have no slowdowns (none or very minor), so longer clips could be tested: QTGMC fast with a 92 minutes DV-AVI source.
.....


Thanks pinterf! Any comments on what was the problem, or how did you fix it? Just curious if it had anything to do with this (http://forum.doom9.org/showthread.php?p=1736551#post1736551).

I ran the same test (http://forum.doom9.org/showthread.php?p=1736090#post1736090) as I did when the issue was first discovered. So far it's looking good, r1576 is still faster but that has always been the case. Here's the results:

[General info]
Log file created with: AVSMeter 2.1.5 (x86)
Script file: M:\AviSynth+ Testing\Regression\AviSynth+ r1828 pfmod.avs
Avisynth version string: AviSynth+ 0.1 (r1828, MT-pfmod, i386)
Avisynth file version: 0.1.0.0
Avisynth Interface Version: 6
Avisynth MT support: Yes
Avisynth.dll linker/compiler version: 14.0
Avisynth.dll location: C:\Windows\SysWOW64\AviSynth.dll
Avisynth.dll time stamp: 2016-03-29, 07:50:15
PluginDir+ (HKLM, x86): C:\Program Files (x86)\AviSynth+\plugins+
PluginDir2_5 (HKLM, x86): C:\Program Files (x86)\AviSynth+\plugins


[Clip info]
Number of frames: 43596
Length (hh:mm:ss.ms): 00:12:07.327
Frame width: 720
Frame height: 480
Framerate: 59.940 (60000/1001)
Colorspace: YV12



[Runtime info]
Frames processed: 43596 (0 - 43595)
FPS (min | max | average): 5.967 | 174.4 | 19.99
Memory usage (phys | virt): 166 | 175 MB
Thread count: 38
CPU usage (average): 10%
Efficiency index: 1.999
Time (elapsed): 00:36:21.112


[Script]
SetMemoryMax(1024)
LWLibavVideoSource("S:\ITAS_Chappelle\VIDEO_TS\VTS_02_1.demuxed.m2v")
AssumeTFF()
QTGMC(Preset="slow")

http://s22.postimg.org/dzout8kfj/Avi_Synth_r1828_pfmod.png

[pinterf]

The FrameRegisty concept is a brilliant idea which helps avs core to reuse released frames and their video frame buffers (vfb) to recycle thus keeping the memory consumption low.

Why allocate new framebuffer when we can use an existing one that no process uses anymore.

When a frame and its vfb is created, their pointers are inserted in a list. That is FrameRegistry. For faster lookup it is "indexed" based on the requested size.

For subframes, only the frame object is created, but its vfb is just linked to the parent's vfb.

How the recycle logic works?

Both frames and their vfbs are reference counted.

A frame can be re-used if its refcount _and_ its vfb's refcount is zero. So when a new frame is created, first we lookup in the list of the frames with vfb's that have the required size.

We are linearly searching through the list, until we find a frame with zero refcount, then we check if it's vfb is zero or not. And that is where there is a problem in real-life.

QTGMC process creates many-many subframes. All these subframes share the same vfb. Creating subframes is fast. All we need is "abusing" pitch and rowsize.

To imagine: this is a small example how multiple frames share the same vfb. In the colorbars example, there are thousands of frame entrys for a single vfb (e.g. 300000+ frames if the frame count is big enough!)
Personally it took me some hours to understand what a subframe is, but now I like the concept :)
frame 000000000AF050A0 RowSize=720 Height=576 Pitch=736 Offset=16
frame 000000000B1771F0 RowSize=720 Height=288 Pitch=1472 Offset=752
frame 000000000B177270 RowSize=360 Height=144 Pitch=768 Offset=534928
frame 000000000B1772F0 RowSize=360 Height=144 Pitch=768 Offset=424336
frame 000000000B177370 RowSize=720 Height=288 Pitch=1472 Offset=752
frame 000000000B1773F0 RowSize=720 Height=288 Pitch=1472 Offset=16
frame 000000000B177470 RowSize=360 Height=144 Pitch=768 Offset=534544
frame 000000000B1774F0 RowSize=360 Height=144 Pitch=768 Offset=423952
frame 000000000B177570 RowSize=720 Height=288 Pitch=1472 Offset=16
frame 0000000013FF17B0 RowSize=720 Height=288 Pitch=1472 Offset=752
frame 0000000013FF1830 RowSize=360 Height=144 Pitch=768 Offset=534928
frame 0000000013FF18B0 RowSize=360 Height=144 Pitch=768 Offset=424336
frame 0000000013FF1930 RowSize=720 Height=288 Pitch=1472 Offset=752
frame 0000000013FF19B0 RowSize=720 Height=288 Pitch=1472 Offset=16
frame 0000000013FF1A30 RowSize=360 Height=144 Pitch=768 Offset=534544
frame 0000000013FF1AB0 RowSize=360 Height=144 Pitch=768 Offset=423952
frame 0000000013FF1B30 RowSize=720 Height=288 Pitch=1472 Offset=16
frame 0000000013FF2E30 RowSize=720 Height=288 Pitch=1472 Offset=752
frame 0000000013FF2EB0 RowSize=360 Height=144 Pitch=768 Offset=534928
frame 0000000013FF2F30 RowSize=360 Height=144 Pitch=768 Offset=424336
frame 0000000013FF2FB0 RowSize=720 Height=288 Pitch=1472 Offset=752
frame 0000000013FF3030 RowSize=720 Height=288 Pitch=1472 Offset=16
frame 0000000013FF30B0 RowSize=360 Height=144 Pitch=768 Offset=534544
frame 0000000013FF3130 RowSize=360 Height=144 Pitch=768 Offset=423952
frame 0000000013FF31B0 RowSize=720 Height=288 Pitch=1472 Offset=16
frame 0000000013FF44B0 RowSize=720 Height=288 Pitch=1472 Offset=752
frame 0000000013FF4530 RowSize=360 Height=144 Pitch=768 Offset=534928
frame 0000000013FF45B0 RowSize=360 Height=144 Pitch=768 Offset=424336
frame 0000000013FF4630 RowSize=720 Height=288 Pitch=1472 Offset=752
frame 0000000013FF46B0 RowSize=720 Height=288 Pitch=1472 Offset=16
frame 0000000013FF4730 RowSize=360 Height=144 Pitch=768 Offset=534544
frame 0000000013FF47B0 RowSize=360 Height=144 Pitch=768 Offset=423952
frame 0000000013FF4830 RowSize=720 Height=288 Pitch=1472 Offset=16
frame 00000000067F15B0 RowSize=720 Height=288 Pitch=1472 Offset=752
frame 00000000067F1630 RowSize=360 Height=144 Pitch=768 Offset=534928
frame 00000000067F16B0 RowSize=360 Height=144 Pitch=768 Offset=424336
frame 00000000067F1730 RowSize=720 Height=288 Pitch=1472 Offset=752
etc...


So frame count exceeds vfb count by a huge margin.

Most of the frames then are released.
Finally we have a frame list, in which (r1825MT frequent worst case) the frame refcount is already zero, but their (very same) vfb's refcount is not. Frames get into the queue, they are finally released, but their vfb is not.

The lookup process iterates in the frame list linearly through hundreds of thousands lines, but may find not a single frame with vfb refcount zero.

The lookup time is growing seriously, nearing to the 1/10 second range, instead of the initial microseconds.

So I took a reverse approach.

In the source I named it FrameRegistry2, I still left the original FrameRegistry declaration (but do not use it anymore).

Instead of frame list, let's have a list of vfb's, and under them, a frame list that are referencing this vfb.

When a vfb is not free (refcount > 0), we can freely ignore checking all the frames owning it. No need for iterating this huge frame list, only the vfb list. It is true, that once a vfb has zero refcount, all frames owning it has also zero refcount. Thus we can simply reuse the first frame and its free vfb, and delete all the other frame objects previously owning the specific vfb. (This can still be made faster, in this r1828 release it's not optimal)

The complexity of the search is still linear, but we are iterating through some hundred of vfb's instead of hundreds of thousands of frames.

typedef std::multimap<size_t, VideoFrame*> FrameRegistryType;

typedef std::list<VideoFrame*> VideoFrameArrayType;
typedef std::map<VideoFrameBuffer *, VideoFrameArrayType> FrameBufferRegistryType;
// P.F. tried vector instead of list http://baptiste-wicht.com/posts/2012/12/cpp-benchmark-vector-list-deque.html
// but in our case it is slower than std::list
typedef std::map<size_t, FrameBufferRegistryType> FrameRegistryType2; // P.F.


It's a long story but I hope it cleared some of the technical background.

P.S.
another very useful thing I learned

std::chrono::time_point<std::chrono::high_resolution_clock> t_start, t_end;
t_start = std::chrono::high_resolution_clock::now();
..process to profile..
t_end = std::chrono::high_resolution_clock::now();
std::chrono::duration<double> elapsed_seconds = t_end - t_start;

[stax76]

Awesome to see some progress, maybe somebody can also help with making QTGMC work with StaxRip x64 and Windows 10. In managed (.NET) x64 hosts under Windows 10 mvtools2 causes a access violation. I can give a reward of 50€/56$ for making it work.

edit:

mvtools2 issue was fixed by pinterf!

[StainlessS]

Pinterf, excellent work and explanation of the problem, made for very interesting reading.
Go sit yourself down and have a well earned glass of port. :)

[Groucho2004]

One note, my mod was built with VS2015, v140_xp toolset.
I have no "virgin" machine, so it works out-of-box for me. It may require Visual C++ Redistributable for Visual Studio 2015 Update 1 (https://www.microsoft.com/en-us/download/details.aspx?id=49984) for you.Unfortunately, WinXP compatibility is broken. It chokes when initializing IScriptEnvironment. I tried the usual tools, VirtualDub, mpc-hc, AVSMeter.

Neither Dependency Walker nor PE Explorer reveal any incompatibilities with XP. Weird.

[qyot27]

Just one comment on the commits (https://github.com/pinterf/AviSynthPlus/commits/MT-pfmod): AviSynth+ uses (preferably 4x) spaces for indentation; no tabs. This is stated in the coding guidelines I linked to earlier.

[pinterf]

Thank you qyot27, I will check it.

[pinterf]

Unfortunately, WinXP compatibility is broken. It chokes when initializing IScriptEnvironment. I tried the usual tools, VirtualDub, mpc-hc, AVSMeter.

Neither Dependency Walker nor PE Explorer reveal any incompatibilities with XP. Weird.
Dunno, I found this topic:
http://forum.doom9.org/showthread.php?t=172400

[Groucho2004]

Dunno, I found this topic:
http://forum.doom9.org/showthread.php?t=172400
No, that's not it. I have the latest re-distributable packages installed.

[pinterf]

Help. Windows 10 is too smart, and is trying to protect me from myself :)
I have troubles to put properly avisynth.dll into c:\windows\system32.
Once I had there an r1825 version, then I copied the r1828 over it.
Avsmeter64 is still showing avisynth+ version r1825.
Then I purged avisynth.dll from c:\windows\system32.
And a (not so big) surprise: avs scripts are still running fine using a ghost r1825. Where is this dll cache? How can I tell windows 10 to clear all historical use of avisynth.dll?

[stax76]

Help. Windows 10 is too smart, and is trying to protect me from myself :)
I have troubles to put properly avisynth.dll into c:\windows\system32.
Once I had there an r1825 version, then I copied the r1828 over it.
Avsmeter64 is still showing avisynth+ version r1825.
Then I purged avisynth.dll from c:\windows\system32.
And a (not so big) surprise: avs scripts are still running fine using a ghost r1825. Where is this dll cache? How can I tell windows 10 to clear all historical use of avisynth.dll?

I've never heard or experienced something like this, maybe ProcessExplorer could help since it can show the paths of DLLs used by a process.