Doom9's Forum - View Single Post

pinterf · 29th March 2016, 11:55

After two weeks of learning curve, I have uploaded a new avs+ build, both 32 and 64 bit MT dlls.

This build is primarily for addressing the huge slowdown issue of recent MT builds.

Please give it a try.

AviSynth+ r1828-pfmod

I have no slowdowns (none or very minor), so longer clips could be tested: QTGMC fast with a 92 minutes DV-AVI source.

Some notes.

The artificial test case of colorbars is not quite good, although it allowed me to find out the main cause of the slowdown. Colorbars is using a static source framebuffer that is kept referenced throughout the whole qtgmc process, so its ref_count will never be released. During the qtgmc script the framebuffer-reusing logic had to iterate many hundred thousand (!) entries until it would find a free entry or see that no free buffer exists. After thousands of frames, one failed lookup needed almost 0.03 seconds (normally it should lookup 1000-10000x faster!), and it got linearly worse.

Todo: "hack" or parameterize colorbar to be able to generate new frames: copy the pre-generated pattern into a brand new frame instead of just linking to it.

Having no slowdown I realized the next problem: framebuffers sometimes are not released.

Example.
The first 17800 frames of an AVI source needs approx. 200 MBytes of framebuffers to deinterlace with Qtgmc, number of framebuffers is 300. Suddenly avs+ cannot find any buffers that has reference count 0 and starts allocating new framebuffers. Reaching the 20000 frames limit (that is 40000 frames after Qtgmc) we need already 500MB, so with SetMemoryMax(512) the script is soon exiting.

And later the memory consumption goes up in waves.

40525 frames (81050 QTGMC): 973 framebuffers, 613MB
49381 frames (99600 QTGMC): 1178 framebuffers, 745MB
56725 frames (113450 QTGMC): 1255 framebuffers, 816MB
61046 frames (122092 QTGMC): 1348 framebuffers, 876MB
65084 frames (130168 QTGMC): 1508 framebuffers, 980MB
68988 frames (137978 QTGMC): 1736 framebuffers, 1128MB
114512 frames (129024 QTGMC): 2121 framebuffers, 1376MB
etc...

Good news: my 1h32min 720x576 dv-avi video - that has finally 278000 50p frames - just succeeded with the SetMemoryMax(2048) setting. Needed approx. 1700MB memory.

But it can be done with much less memory.
Whick plugin or avs core logic is getting those frames stuck?
Have to figure out, how. Seems it is not related to MT.

Script (+having rgtools, mvtools2, masktools2.dll and nnedi3.dll in the plugins64 directory)

Code:

SetFilterMTMode("DEFAULT_MT_MODE",2)
SetFilterMTMode("AviSource",3)
SetMemoryMax(2048) 
#colorbars(width = 640, height = 480, pixel_type = "yv12").killaudio().assumefps(25, 1).trim(0, 29999)
Avisource("c:\tape13\Videos\Tape02_digitall_Hi8.avi").killaudio().assumefps(25,1)#.trim(0, 29999)
AssumeBFF()
QTGMC(Preset="Fast")
prefetch(3)

29th March 2016, 11:55	#1336 \| Link
pinterf Registered User Join Date: Jan 2014 Posts: 2,314	After two weeks of learning curve, I have uploaded a new avs+ build, both 32 and 64 bit MT dlls. This build is primarily for addressing the huge slowdown issue of recent MT builds. Please give it a try. AviSynth+ r1828-pfmod I have no slowdowns (none or very minor), so longer clips could be tested: QTGMC fast with a 92 minutes DV-AVI source. Some notes. The artificial test case of colorbars is not quite good, although it allowed me to find out the main cause of the slowdown. Colorbars is using a static source framebuffer that is kept referenced throughout the whole qtgmc process, so its ref_count will never be released. During the qtgmc script the framebuffer-reusing logic had to iterate many hundred thousand (!) entries until it would find a free entry or see that no free buffer exists. After thousands of frames, one failed lookup needed almost 0.03 seconds (normally it should lookup 1000-10000x faster!), and it got linearly worse. Todo: "hack" or parameterize colorbar to be able to generate new frames: copy the pre-generated pattern into a brand new frame instead of just linking to it. Having no slowdown I realized the next problem: framebuffers sometimes are not released. Example. The first 17800 frames of an AVI source needs approx. 200 MBytes of framebuffers to deinterlace with Qtgmc, number of framebuffers is 300. Suddenly avs+ cannot find any buffers that has reference count 0 and starts allocating new framebuffers. Reaching the 20000 frames limit (that is 40000 frames after Qtgmc) we need already 500MB, so with SetMemoryMax(512) the script is soon exiting. And later the memory consumption goes up in waves. 40525 frames (81050 QTGMC): 973 framebuffers, 613MB 49381 frames (99600 QTGMC): 1178 framebuffers, 745MB 56725 frames (113450 QTGMC): 1255 framebuffers, 816MB 61046 frames (122092 QTGMC): 1348 framebuffers, 876MB 65084 frames (130168 QTGMC): 1508 framebuffers, 980MB 68988 frames (137978 QTGMC): 1736 framebuffers, 1128MB 114512 frames (129024 QTGMC): 2121 framebuffers, 1376MB etc... Good news: my 1h32min 720x576 dv-avi video - that has finally 278000 50p frames - just succeeded with the SetMemoryMax(2048) setting. Needed approx. 1700MB memory. But it can be done with much less memory. Whick plugin or avs core logic is getting those frames stuck? Have to figure out, how. Seems it is not related to MT. Script (+having rgtools, mvtools2, masktools2.dll and nnedi3.dll in the plugins64 directory) Code: SetFilterMTMode("DEFAULT_MT_MODE",2) SetFilterMTMode("AviSource",3) SetMemoryMax(2048) #colorbars(width = 640, height = 480, pixel_type = "yv12").killaudio().assumefps(25, 1).trim(0, 29999) Avisource("c:\tape13\Videos\Tape02_digitall_Hi8.avi").killaudio().assumefps(25,1)#.trim(0, 29999) AssumeBFF() QTGMC(Preset="Fast") prefetch(3)