AviSynth 2.5.6 RC 2 Release Candidate [October 7th]

[Fizick]

1. So, CACHE_RANGE=0 is really bad? But some time ago i saw recomendation use it.
2. I look to MipSmooth code. It use CACHE_RANGE=1 (it has temporal mode too).
3. I think, we must ask about cache other plugins writers too (Kassandro ? - He uses CACHE_NOTHING in Removegrain )

[IanB]

Originally Posted by Fizick

1. So, CACHE_RANGE=0 is really bad? But some time ago i saw recomendation use it.

It was a complicated way to say CACHE_NOTHING and in debug builds it would cause a sanity assertion. I do not know of the reference, maybe they were making the recomendation for a very specific case. The new cache more or less treats this as default caching, it probably trips over a boundary condition and causes the current frame to be protected for 1 cycle.

2. I look to MipSmooth code. It use CACHE_RANGE=1 (it has temporal mode too).

So this would dodge the debug build sanity assertion. It did provide a 1 frame cache, almost but not quite CACHE_NOTHING. Not a really well thought out choice.

3. I think, we must ask about cache other plugins writers too (Kassandro ? - He uses CACHE_NOTHING in Removegrain )

It seems I need to protect more against silly values.

A point to consider, for a purely linear script, one with no temporal filters or tricky cross dependancies, the cache is absolutely useless. If no frame is ever fetched a second time there is nothing to cache. So for this case dicky cache setting won't matter. And as the cache does take a small number of cpu cycles to manage disabling it must yield a very small overall speed improvement.

But for temporal filters and scripts with cross dependancies i.e. uses stack*(), merge*(), weave(), layer(), overlay(), etc the gain from having an effective cache is immense. The key point for a temporal filter author is they know with certainty the access pattern of there filter and they can give the cache important information in the form of a hint i.e. "I am going to fetch the same frames 7 times, as a priority please keep my last 7 GetFrame results."

As for cross dependancies this is purely in the domain of the script jockey. A filter author cannot begin to guess the many and wonderous ways a script will build very complicated cross dependancies using their filter, so they should do nothing to inhibit the successful action of the cache.

This is the major advance with the new cache, the old cache just cached everything and hoped that any subsequent fetches happened before a VFB got reused. The new cache tracks repeated requests for frames and promotes only candidate frames, so only frames likely to be requested a second time get cached. Single use VFB's do not polute the cache. Also protected VFB's from a CACHE_RANGE request do not join the cache thus also do not polute the main cache but they do reduce the overall size of the main cache, so the range specified should be as accurate as possible. To small and you probably cycle thru all the protected frames before getting to use them, to big and you rob the main cache.

IanB

[Fizick]

I make some new test with custom versions of DegRainmedian (with different CACHE_RANGE). I use two or more DegrainMedian, with conbination with DeFreq, fft3dfilter, fluxsmooth, Reverse.
Here is results:
1. For Avisynth 2.5.5 I have big slowdown with CACHE_RANGE=0 or =1.
2. For Avisynth 2.5.5 CACHE_Range=2 works perfectly. There no any difference with CACHE_RANGE=3 or more.
3. For latest 2.5.6 (14 july) I have no any slowdown for ANY CACHE_RANGE (0,1,2,3)

What is the most indicative script for cache settings?
(V1, V2, V3 does not show nothing)

[IanB]

Originally Posted by Fizick

I make some new test with custom versions of DegRainmedian (with different CACHE_RANGE). I use two or more DegrainMedian, with conbination with DeFreq, fft3dfilter, fluxsmooth, Reverse.
Here is results:
1. For Avisynth 2.5.5 I have big slowdown with CACHE_RANGE=0 or =1.

Classic example of cache cycling, each frame is destroyed just before it can be reused.

2. For Avisynth 2.5.5 CACHE_Range=2 works perfectly. There no any difference with CACHE_RANGE=3 or more.

Enough is enough, more is NOT better, just wastefull.

3. For latest 2.5.6 (14 july) I have no any slowdown for ANY CACHE_RANGE (0,1,2,3)

CACHE_RANGE no longer excludes the standard cache, if the hint is wrong it is no longer catastrophic. However you take your chances, if a given script places extreme demands on the cache then a misleading or wrong hint will bite you, to small and you get cache buffer cycling and purging, to large and you deplete the main cache for other filters.

What is the most indicative script for cache settings?
(V1, V2, V3 does not show nothing)

AviSource("blah.avi")
A=Trim(20, 0) # Adjust "20" to tune degree of cache abuse
B=Trim(10, 0) # optional extra cache abuse
FilterThatGivesCacheHint()
StackHorizontal(Last, A, B)

This style of script exposes how weak the old cache was, even without a filter giving bad hints it is easy to tune for total devistation, particularly if "blah.avi" is a keyframe challenged file. The true test is to use SetMemoryMax() to see how low can you go before it collapses.

IanB

[Fizick]

IanB,
thanks for some clarification.

But I decided to add some info.
I check ALL plugins source availiable (all from warpenterprises site and my own collection).
Here is the FULL list of plugins that have SetCacheHints explicit call. (please add more if you know).

Colormatrix by Wilbert: (CACHE_RANGE,1) (if use hints from DGDecode)
Convolution3D by Vlad: (CACHE_RANGE, 2)
Defreq by Fizick: (CACHE_RANGE,0)
Defliker by Fizick: (CACHE_RANGE,cachecapacity), cachecapacity = lag*2+2
DeGrainMedian by Fizick: (CACHE_RANGE,2)
DePanEstimate: (CACHE_RANGE,range+1) range is radius
DePanInterleave: (CACHE_RANGE,num) num = prev + next + 1, Invoked Interleave: (CACHE_RANGE,num)
DePan: (CACHE_RANGE,cacherange) cacherange = 1+2*abs(int_offset)
Despot since v3.3.2 by Fizick: (CACHE_RANGE, 2)
GetDups by Fizick: (CACHE_RANGE,0) (it is wrong for temporal but works somehow ?!)
Fluxsmooth by SansGrip: (CACHE_RANGE, 1) for YV12
fft3dfilter by Fizick: spatial (CACHE_RANGE,0), temporal (CACHE_RANGE,2)
LeakKernelDeint by Leak: (CACHE_RANGE,2)
MipSmooth by shodan: (CACHE_RANGE,1)
Removedirt06 by Kassandro: some internal (cache ? CACHE_RANGE : CACHE_NOTHING, cache) cache=true or false
Removedirt09 by Kassandro: more complex internal using, some clips with (CACHE_RANGE, 0), some with user (?) defined (CACHE_RANGE, cache)
Removegrain06 by Kassandro: spatial (CACHE_NOTHING, 0), temporal (CACHE_RANGE, 2);
today Kassandro tell me, that recent Removegrain uses spatial (CACHE_RANGE, 0)
TFM by tritical: (CACHE_RANGE, 1)
TDecimate by tritical: (CACHE_RANGE, 1)
Vaguedenoiser since v.0.30 by Fizick: (CACHE_RANGE,0)

(There are also some plugins without sources.)
I see, that I am in good team .
So, ALL plugins with explicit CACHE_RANGE used wrong settings?
May be this settings must be considered not as wrong but as quite correct?
I say about spatial (CACHE_RANGE,0) and temporal (CACHE_RANGE,2).

IMHO, Avisynth must detect plugin's CACHE_RANGE=0 special case and consider it as the mark of spatial only plugin.
Temporal plugins above used mostly CACHE_RANGE=1 or =2.
I do not see, that is more compatible with diameter (=3) than radius (=1).

Here is my crazy suggestion:
May be Avisynth must consider plugin's CACHE_RANGE value as a DISTANCE between boundary frames?
Consider 3 frames (100, 101, 102) to be cached. Distance between frame 100 and frame 102 is 102-100=2.
If diameter (number of frames) = 3, then distance= (diameter-1) , and diameter=distance+1.
So, Avisynth will use internal CACHE_diameter=CACHE_RANGE+1
For CACHE_RANGE=1 we will have diameter=2 (it is qute good)
For CACHE_RANGE=2 we will have diameter=3 (it is perfect).
For CACHE_RANGE=0 we will have diameter=1 or it may be considered as special case (converted to CACHE_ALL - not-protected VFB?).
So, we must not recomplile all plugins. Compatibility will be quite good.
Is it enough crazy idea?

[IanB]

Fizick,

That is pretty close to what I was already thinking, although I hadn't thought about the diameter=distance+1, this is good idea, particularly for small values. As you have found out the new cache doesn't slow down with bad values like the old one so it is no longer much of an issue.

Unfortunatly we can't fix it for previous versions, which is where it hurts the most. However for previous versions doing something like Crop(0,0,0,0).FilterWithBadCacheHints() should side step the problem if there ever is one.

IanB

[Wilbert]

Some throw error bugs:

1) Tweak. Feeding a RGB32 clip to Tweak, opening it in VDubMod, doesn't throw the correct error:



Although i don't see anything strange in the code.

2) Tweak, SwapUV (and others). Feeding a RGB32 clip to filters which don't accept RGB, using the scripteditor in VDubMod, doesn't throw the correct error:



This happens with all exceptions, not only feeding it with the wrong color format.

I have a W2K SP4 if that matters anything (and latest CVS).

[IanB]

Damn missing exception table code bug. I'll have to do another pass thru all the asm listings, fortunatly I'm getting good at finding these. I'll try to find time this week, then we can start doing 2.5.6-RC1, the DirectShowSource rework looks ready to go.

IanB

Edit: I've searched all the .asm's and fix another half dozen SEH bugs. I just commit'd the source fixes to CVS. 2.5.6-RC1 here we go.

[Wilbert]

@IanB,

Bug (1) above is corrected, but (2) still doesn't show the correct error. Just open a correct script in VDubMod, open the script editor, mess up, F5 (refresh), and you will get the

Cannot open file "":
The system cannot find the path specified.

error, instead of the correct exception.

Of course it might be a problem in VDubMod, but it used to work.

[sh0dan]

I created patch for Temporalsoften to enable RGB32 mode, and loosen restriction on YUY2 to MOD4. All modes where already supported by the code, but I seem to have forgotten to enable them.

Not committing since this (maybe) isn't in time for 2.5.6.

http://zenaria.com/kpo/temporalsoften.patch

[IanB]

@Sh0dan,

Might as well commit it, it's just constructor parameter validation.

It will be a few evenings before I can find/fix bug (2) above.

IanB

[Wilbert]

Sh0dan, is it true that this scenechange setting is not implemented for RGB32?

[sh0dan]

@Wilbert: Not really. I did one for the conditional filters. It's basicly the same as a regular scenechange - it just masks out the last byte, so that garbage in the 'A' channel doesn't confuse the math.

It should be fairly straigtforward to implement it into temporalsoften.

[Wilbert]

Despite the small issues above, i had some time and made a new release:

AviSynth 2.5.6 RC 1 Release (September 4th):

Additions:
* RGB32 mode in TemporalSoften
* *Resize(), src_height and src_width when negative work as in crop.
* Added options to DirectShowSource (seekzero, timeout and pixel_type).
* Added AudioDubEx(), blindly accepts video and audio streams.
* Added Load_Stdcall_Plugin(), alias for LoadCPlugin() (won't disappear when avisynth_c.dll is loaded).

Bugfixes:
* Masked "Evaluate: System Exception - Access Violation" in :- FadeIO*(), RGBAdjust(), Tweak(), Lanczos*Resize() and GaussResize().
* Fixed rounding in YUY2 turnleft/right chroma.
* Fixed AVSC_USE_STDCALL declaration in avisynth_c.h (was ACSC_USE_STDCALL).
* Fixed BlankClip(clip) now competely duplicates the donor clip's VI including parity.
* Fixed AssumeTFF/BFF() to correctly update internal parity state.
* Fixed Animate audio switching.
* Fixed aligned UVpitch from width rounding.

Changes:
* The avisynth_c plugin entry point is now officially "avisynth_c_plugin_init@4" (don't include @4 anywhere the compiler does it for you), this is not actually a change due to a bug in avisynth_c.h, which incidently caused it to be this already.
* Info() now autoselects a smaller font to fit info in small frames.
* Info() now distinguishes between "assumed" field parity and field parity.
* Animate now selects parity through the filter chain.

As usual download from Sourceforge.

[Wilbert]

@Wilbert: Not really. I did one for the conditional filters. It's basicly the same as a regular scenechange - it just masks out the last byte, so that garbage in the 'A' channel doesn't confuse the math.

It should be fairly straigtforward to implement it into temporalsoften.

Ok, i wasn't sure you already implemented it, because you added RGB32 mode

[tritical]

Was the "fZeroRead" not being reset in VC_audio.cpp problem ever fixed? I couldn't find any mention of it in cvs. All it seems to need is for fzeroread to be set to false in AudioStreamSource::Seek if it is currently set to true and samples < end_samp.

[IanB]

Opps, No, cracks, slipage thru thereof.

IanB

Edit: 12th Sep, Done! Into CVS soon(ish).

[Mug Funky]

i seem to be getting a 1 gig barrier with "2vuy" files and avisynth. to clarify, i load the avi into virtualdub and it shows 2975 frames. i load the same avi through avisynth and i get 1535 frames (that's right on the 1 gig threshold for uncompressed yuy2). this happens even if i change the fourcc of the file, or use the fourCC option in avisource (i force to uyvy, as that's the format the files are in).

also, by default loading these files as uyvy seems to return YV12 with progressively downsampled chroma. forcing yuy2 b0rks the luma. right now i'm loading like this:

y=AVISource("F:\GZ - son of godzilla cap test\GZ.avi",fourcc="uyvy",pixel_type="yv12").converttoyuy2()
uv=AVISource("F:\GZ - son of godzilla cap test\GZ.avi",fourcc="uyvy",pixel_type="yuy2")

mergechroma(y,uv).crop(0,0,720,480)

the crop is in there because the original file is 720x486, which is not mod4 and therefore doesn't behave nicely.

providing samples might be a problem as this only seems to affect 2vuy files that are over 1 gig... i've done heaps of DV stuff with avisource and there's been no problem. sodding blackmagic-design haven't got back to me about their crappy tarted-up uncompressed codecs that only return RGB24 while storing 4:2:2, so it looks like reverse-engineering is the only solution until i get a PC with a decklink card in it.

[IanB]

Mug Funky,

Ah, I've always wanted to use Avery's sparse avi file tool and you present the perfect opportunity.

In VirtualDub under tool->Create sparseAVI, Select source AVI File -> {one of your files}, Select sparse AVI file -> {some good name}. You now have the structure of your AVI container without any content, it should be quite small, stick it somewhere I can get it.

Now for some content, use VirtualDub to chop a few frames off the front (say <1Mb) of the same file you sparsed above, also stick it somewhere I can get it.

And just for good measure decode 1 frame (with useful colour picture content and from the segment chopped above) and present it as a lossless compressed image (bmp/gif/tiff NOT jpeg) somewhere I can get it.

Also what, where, how, why, which, .... about codec and application producing the AVI files.

IanB

[Mug Funky]

hmm. sparse avi file, huh? sounds interesting. if you're around melbourne i could just hand a DVD to you though... i'll give that sparse trick a shot when i'm less busy

on the provenance of these avi files. i'm pretty sure they're not completely compatible avi files. here's how they were made:

- a blackmagic quicktime file was captured on a mac through a decklink card. 8-bit uyvy uncompressed, with the fourCC "2vuy". quicktime doesn't offer much in the way of capturing formats, and i don't want to risk dropped frames just yet.

- the mov was then transmuxed into an avi with ffmpeg like so:

ffmpeg -i xxxx.mov -vcodec copy -an xxxx.avi

this copies the video stream as is, and ditches the audio (it's 4 mono channels... i'll figure this one out later ).

- the avi is then loaded into avisynth as uyvy rather than 2vuy, because the stock codec for this only returns RGB24 (this is the stupidest thing i've ever seen). see my last post for the exact loading script.

thanks for your help