AviSynth 2.5.5 CVS Releases. [August 26th]

[IanB]

@Bidoche,

A little chicken before the egg, If we knew when a total==0 was about to happen we could do a lot better than poison the values, and if we don't know, which way do we nudge + or -?

Bumping denominators to dodge divide by zero conditions will always gets you in the long run.:)

IanB

[IanB]

Dear all,

Some of the updates I submitted didn't make the last CVS version cut.

- Fix VideoFrameBuffer cache corruption.
- General Convolution updates:
* support legacy syntax

I will have to wait for the next version. :cool:

Sorry for any inconvienience. :o

IanB

[sh0dan]

Just did a silent update - all the advertised features should now be in place! :cool:

[Bidoche]

@IanB

We don't need to know when it will happen.
When it does we just retry with slightly different margins value.

Like this :

[...]
if ( total == 0.0 )
throw PatternZeroTotalException(); //specific custom exception
[...]

//in code calling the pattern

try { //creating the pattern }
catch( PatternZeroTotalException& )
{
marginleft += 0.01;
//retry again
}

[Wilbert]

Bugreport from our friend mf:

c1 = MPEG2Source("F:\From_hell\from_hell.d2v")
c2 = BlankClip(c1)
ConditionalFilter(c1, c1, c2, "AverageLuma()", "morethan", "100", true)

always returns true (also when it is false), and also c1 is show in that case.


c1 = MPEG2Source("F:\From_hell\from_hell.d2v")
c2 = BlankClip(c1)
ConditionalFilter(c1, c1, c2, "AverageLuma()", ">", "100", true)

works though.

[IanB]

@Bidoche,

Yes, this would be a really cool :cool: OO implementation, I quite like it in concept, but the whole concept of dodging errors, like divide by zero, in this way leave me a little unconfortable.

The key assumption here is that the programmer understands the relationship between the input value he is bumping and the output value he is trapping against. If they are tightly, exactly and obviously related this works extremely well. If they are even slightly independant then no amount of bumping can guarrantee avoiding the condition in all cases.

I recently fixed a bug in someones code where this type of thinking had been used inside a loop. You guessed it, the code never exited, it bumped forever. :p

As it turns out bumping here wouldn't have helped as Sh0dan found the code was running off the edge of the picture by a lot. The whole deal with this deficiency in the resizer code is to understand where the total==0.0 can come from and adjust the code to avoid the problem totally.

Thankyou for pushing this point because it has made think about the consequences of trapping illegal values so ruthlessly. Sh0dan has the right answer by trapping only in debug mode, in fact we probably should leave the trap code active until we go to release candidate.;)

IanB:D

[sh0dan]

Originally posted by Wilbert
Bugreport from our friend mf:

c1 = MPEG2Source("F:\From_hell\from_hell.d2v")
c2 = BlankClip(c1)
ConditionalFilter(c1, c1, c2, "AverageLuma()", "morethan", "100", true)

always returns true (also when it is false), and also c1 is show in that case.

Hmm... It should actually throw an error, since it should be "greaterthan". I'll see why it doesn't.

Edit: Fixed.

[Manao]

Any resizing to non mod8 resolutions in YV12 shift the chroma horizontally ( by 20 - 30 pixels I guess ). I never saw that bug, but I tried several version between 2.54 and latest 2.55 alpha, and all show that behavior.

[ARDA]

I couldn't reproduce in my machines. Which decoder are you using? I remember some old xvid decoder's bug (mod4)ShOdan helped to fix. Which avysinth date version?.
I hope that can help you to find your problem.
ARDA

[Manao]

Damned, that's why I kept saying me "I already saw that thing somewhere..." I was the one reporting that bug and I totally forgot it. Thanks ARDA, the guilty is XviD 1.0.1. I'll PM Koepi about it.

[yaz]

the 'zero coeff. total' (outlined here (http://forum.doom9.org/showthread.php?s=&threadid=79603)) has gone but now i have 'Unrecognized exception' or(!) 'Access violation ... attempted 0x0000000' depending on what resolution i set. now it's not on loading but on encoding/playing back the script. i'm sure it's still the resizer but ... what ? (when i'm back to 170604 everything worx fine) pls ... is it me or avs ?
thx
y

[IanB]

@Yaz,

The usual please....

Copy of your script, parameters of your input file(s) (height, width, codec, etc)

IanB

[yaz]

Originally posted by IanB
Copy of your script, parameters of your input file(s) (height, width, codec, etc) sorry, but 'crossposting & replication is forbidden'. that's why i dropped only the link above. there u can find all details i have.
shortly, i've used this/that script with the same dvd source for months wout any problem. the first crash came with 120704. pls find what i found via the link.
at the moment (1400704) there's no problem when loading the script but it crashes vdm when encoding a/o mpc when playing back with errmsg i wrote. the crashpoint has no chracteristic feature (at least i haven't found any) & it does not even seems to be reproducible (i mean, i can't locate a specific frame triggering it, rather it occurs 'after some time').
when switched back to 170604 everything goes fine, so i'm sure it's avs and not me :-)
thx
y

[sh0dan]

I've re-checked the code that has changed, and I'm not sure the problem is in the resizer. It seems like something somewhere is being given a zero-pointer. It _could_ be related to recent cache changes.

Are anyone capable of reproducing this problem WITHOUT using a resizer?

[IanB]

BlankClip(100000, 720, 480, pixel_type="YV12")
Crop(12, 12, -12, -12, align=true)
LanczosResize(528, 288)Won't even load in AviSynth_120704.exe (Build:Jul 5 2004 [14:29:59])

Works A-Okay in AviSynth_140704.exe (Build:Jul 15 2004 [16:39:25])

Note - also a bug in installer! reinstalling looses Notepad association.

IanB

[tritical]

I'm wondering if anyone ever found out anything more about getting "Evaluate: unrecognized exception" instead of the actual error msg when calling env->ThrowError() in a filter constructor. I saw earlier in this thread ColorYUV, and the resampling filters had the same problem. In a filter I'm working on I call throwerror in the constructor and this happens. I ran a release build through debugging real quick and it seems it hits an access violation trying to write somewhere after the first exception is thrown, but I haven't been able to track down the exact problem location. If I build the filter in debug configuration it works correctly, probably since the catch all (catch(...)) in evaluate gets ifdef'd out when in debug mode... but also, the write access violation doesn't occur in debug mode. Funny thing is, it seems totally random... I can cause the problem to occur and not occur on my current filter simply by removing or adding one line from the
AVSValue __cdecl Create_Filtername(AVSValue args, void* user_data, IScriptEnvironment* env)
function. And the most confusing thing is, the line is just:

char pdir[80];

I don't even have to reference the variable or use it, and it will still cause the problem to happen. It isn't a problem with conflicting names or variables as it occurs with any name I give the variable. Anyways, just wondering if anyone knew what was going on, its not very important but is rather annoying :devil:. I'm using avisynth 2.5.5 build from march 18th.

[stickboy]

Originally posted by tritical
Funny thing is, it seems totally random... I can cause the problem to occur and not occur on my current filter simply by removing or adding one line from the
AVSValue __cdecl Create_Filtername(AVSValue args, void* user_data, IScriptEnvironment* env)
function. And the most confusing thing is, the line is just:

char pdir[80];

I don't even have to reference the variable or use it, and it will still cause the problem to happen.Maybe you're overflowing some buffer somewhere and corrupting the stack.

Have any sample code that demonstrates your problem?

[yaz]

Originally posted by sh0dan
... Are anyone capable of reproducing this problem WITHOUT using a resizer? ... resizer still has some problems (140704 does not accept 'any' frame-size values even if mod4) but i had no time for a thorough & consistent test. maybe tonight.
maybe i try to do a 'filter-drop test' (dropping them 1by1) but ... time is not on my side these days.
just a simpleton question. if avisynth's once broken like i wrote above, can i go on with a job-queue (taking the next script aso aso)or is it better to restart the system ?
thx
y

[tritical]

Well, I downloaded the latest source from the cvs and built avisynth.dll to try debugging it, but the problem seems to be gone using this version and I can't reproduce it. Should have tried grabbing the latest version to begin with, sorry.

[Wilbert]

@Sh0dan,

All links to pictures in the html docs are working now.

Before this we had only one pictures folder, but that way, either the chm file didn't link the pics correctly (due to bug in HTML Workshop) or the html files themselves didn't link them correctly.

So, I put the pictures folder in every language folder and updated the installer (contrary to what I thought months ago, the pics are included once).

[sh0dan]

@All: I think the problems in the current build isn't strickly related to the resizers. I'll release a test version.

@Wilbert: The documentation looks extremely good! I noticed the installer has grown ~300k to about 1.7Meg. I see that duplicated files are not included several times, so I cannot see where the 300 are going. Is it because of the new images in externalfilters?

[Wilbert]

The new images are 323 KB. So, that could be (I guess). I will compress them a bit more ...

[IanB]

Originally posted by sh0dan
@All: I think the problems in the current build isn't strickly related to the resizers. I'll release a test version.@Sh0dan,

Yes, I have been trying to find why constructors can't reliably env->ThrowError(...) in non-debug releases. So far not much luck, but I have had 1 (unreproducable) instance of the stack being overwritten. I'll keep on looking, as this is what I suspected was happening.

As for the memory management changes, I commented out "Plan B" mainly to give the reclaimation code (which previously failed miserably) a very hard workout (it is currently the only way to reuse frame buffers). Previously for purely linear scripts "Plan B" would be used exclusively and never see the problem in the reclaimation code, this may now be exposing other hidden problems. I have given it a fairly strong workout on both my old PII-MMX 400 and current P4, but it needs to be severly beaten with P-III, HT & AMD's processors as well.

I would rather find and fix any bugs in the memory management now but if you feel it is compounding other problems and making them hard to find then please uncomment the "Plan B" code.

IanB

P.S. A linear script has a single path from input to output. A non-linear script has forked branches like doing a separatefields() with selecteven() and selectodd() branches.

Edit: Sorry "Plan B" not "Method 2"

[IanB]

I have found using Plan C exclusively causes heap fragmentation and hence the uncontrolled growth of memory usage. So in the CVS tree I have restored plan B memory management and have also decreased the memory overdraft from 25% to 12.5%. I will keep pondering for a clean way to get the cache to update the frame buffer pool for LRU.:p

I hopefully have also fixed handling of memory > 1.6GB (a signed int overflow problem).;)

No luck finding the stack trasher :devil: yet. When it strikes it damages the exception handler data in the stack frame, usually causing an access violation in the kernel. Debug versions seem immune.:mad:

IanB

[DDogg]

I hesitate to call this a report as there is no hard data, but I have found that the newest build (AviSynth_140704.exe) causes CCE 2.67.00.27 to 'drop out' and prematurely end encoding at seemingly random stages of the encode.

Initially I thought I was pushing my CPU too hard. So, I ran Prime95 and it was solid. After scratching my head a bit I installed an older version of AviSynth I had in my dvd2svcd directory (avisynth253.exe). After that all encodes finished properly.

The good thing is I can replicate this seemingly at will by changing back to the later build. When new builds come out I can test to see if this situation still exists. I'm also happy to test any test builds if needed.

The script I was using is a resizing and cropping script build by FACAR which is r6d2's derivative of Gerti's semi-auto crop and resize. Since it is so long I will not post it unless requested, or I can answer a PM request with the script if desired.

Perhaps it is something else interacting with the newer build. If time permits I will try to pin things down a tad better.

[Leak]

Originally posted by DDogg
I hesitate to call this a report as there is no hard data, but I have found that the newest build (AviSynth_140704.exe) causes CCE 2.67.00.27 to 'drop out' and prematurely end encoding at seemingly random stages of the encode.

Initially I thought I was pushing my CPU too hard. So, I ran Prime95 and it was solid. After scratching my head a bit I installed an older version of AviSynth I had in my dvd2svcd directory (avisynth253.exe). After that all encodes finished properly.

I guess you're having the same problem that I've reported in this thread (http://forum.doom9.org/showthread.php?threadid=79504):

Actually, I was doing some tests today trying out a plugin I'm just working on, and when I was encoding a 24 minutes episode halfway through VirtualDubMod just disappeared on me. Well, a while after starting the job a second time I took a look at Process Explorer's output and noticed that VirtualDubMod's virtual size was over 1.4GB - okay, so this is just the address space the application takes up and not really the memory it uses, but I guess it's what tore VDubMod down after reaching the 2GB mark - IIRC that's the limit for each application's address space (unless you do some extra work to get it to give you 3GB)...

Grab Process Explorer at http://www.sysinternals.com/, start it and click on "View/Select columns...", then switch to the "Process Performance" tab and tick "Virtual Size".

Then just start encoding and watch the virtual size of your encoding process increase steadily, until it reaches 2GB - where it'll die on you.

np: Alter Ego - Transphormer (Transphormer)

[malkion]

I can report similar findings with the AviSynth_140704 build. One attempt at encoding had VDMod disappear halfway through. A 2nd attempt resulted in an internal error, as did a third. Rolling back to AviSynth_170604 results in no errors.

Script:

directshowsource
crop(4,0,-0,-0).lanczosresize(640,480)
mipsmooth(preset="animelq")
deen("a3d")
convolution3d("animebq")
fluxsmooth()
limiter()

I'm sure Sh0dan and IanB are ironing out the memory management issues mentioned above. No fancy resizing on this job, just a simple compression encode.

[IanB]

Dear all,

My bad, all version since I knobbled Plan B, Sun Jul 11 07:36:02 2004 UTC, will eventually exhaust the process address space limit (2GB).

The reason is Plan C releases only the frame buffer not the managing structures. When a new one is requested stupid windows heap allocater grabs a fresh block from the top of memory rather than reusing any of the previously freed chunks (it apparently want memory contiguously free at the end of the heap to reuse it). After a while the heap looks like swiss cheese. Avisynth strictly speaking never uses more than "memory_max"+25% but it is not counting all the holes between inuse blocks. When the inuse memory plus all the unused holes try to exceed 2GB it is very fatal.

Anyway no one seems to have reported the frame buffer corruption caused by Plan C I originally targeted the fix at, and you have all tested it to the point of heap collapse, so it's not all bad news.

IanB

[morsa]

can't some kind of "defragmenter" be included?

[Leak]

Originally posted by IanB
The reason is Plan C releases only the frame buffer not the managing structures. When a new one is requested stupid windows heap allocater grabs a fresh block from the top of memory rather than reusing any of the previously freed chunks (it apparently want memory contiguously free at the end of the heap to reuse it).

Yeah, without a garbage collector you'll get that... :(

Hmmm... do those structures themselves have fixed sizes? If not, instead of freeing them you could add them to a pool (i.e. an array or linked list) and take them out again and reset them when a new one is requested and the pool isn't empty, only really freeing objects when the pool exceeds some limit - overloading the new and delete operators makes this rather easy in C++. That would also eliminate the overhead allocating memory in the swiss cheese is going to bring with it...

Just thought for food.

np: Bigg Jus - Plantation Rhymes (Southern Hospitality Mix) (Black Mamba Serums v2.0)

[Bidoche]

I am not sure it's really Windows who is to blame.
But rather the C runtime coming with VC6.

[IanB]

@morsa,
> can't some kind of "defragmenter" be included?

@Leak
> instead of freeing them you could add them to a pool....

"Plan B" is exactly that it looks thru the existing list of FrameBuffers for one the right size and if it is not in use it reuses it. There are very few bug reports I can pair with the "Plan C" corruption I found, so "Plan B" must have been working well enough:D

Unfortunatly the list it scans is NOT in LRU order but in first created order, this tends to encourage Murphy's law and a buffer about to be needed usually is selected. I have a few schemes I am pondering to improve this. When I find my stack trasher, I'll commit some proper time to this.

@Bidoche,
> ...the C runtime coming with VC6.
True, but a little harsh. I am yet to find a heap manager that is totally immune from combing (or swiss cheese);)


IanB

[sh0dan]

Hi!

Mostly a fix for the crash-issues that were present in the latest version. Though remember this _is_ beta-stage software. The other stuff is minor fixes, and a rather large addition to TCPDeliver - requiering both client and server upgrades.

- Fixed problems causing random crashes and excessive memory problems in July 13th binary.
- Fixed errors in NON-MMX ColorKeyMask.
- Fixed Loop() when called with no video (stickboy)
- Added AssumeFPS(clip [, sync_audio=true/false]) (stickboy again).
- AssumeFPS: Added sanity check to ensure that denominator isn't zero.
- Fixed error not being thrown in Conditionalfilter on an invalid operator.
- Added first version of compression for TCPdeliver. LZO, Huffman and GZIP Huffman algorithms added. Docs here (http://www.avisynth.org/TCPDeliver).


Hey - also 3002nd post! :D

[sh0dan]

Silent update:
- Fixed MergaLuma not giving correct results in some cases.
- Fixed DirectShowSource not releasing Graph/Files on unload/reload.

[Kurtnoise]

I've a bug after installing the last beta (29/07/04 version).

When I load an avs script in VDM, I've an error :


Runtime Error !

Program: C:\Montage_Video\VDM\VirtualDubMod.exe

abnormal program termination


Whereas if I install the last alpha, everything is ok...:scared: Any idea ??

[Leak]

Originally posted by Kurtnoise13
I've a bug after installing the last beta (29/07/04 version).

When I load an avs script in VDM, I've an error.

Whereas if I install the last alpha, everything is ok...:scared: Any idea ??

Well, since the same version works for me (and probably quite a few other people), you might want to do the basic exercises for that case - try a script only containing "Version()", remove all plugins from your plugin folder, if it works then put them back one by one until the crash happens again, etc...

EDIT: Oh yeah, since I always forget that step - :script:, like Boulder said... :)

np: Dntel - Barbara Morgenstern Remix ((This Is) The Dream Of Evan And Chan)

[Boulder]

You should also post the script you've used to help the devs;)

[Kurtnoise]

Yeah...Leak, you're right. Problem solved to removing the plugins from the plugin folder ;) Error Source was Antiblink.dll...

Thank you anyway :cool:

[yaz]

Originally posted by sh0dan
... Silent update ...
great great great ... it's stable again :-) been tested for 3 days but no more occasional crash (so far ... or should i be more trustful ? :-)
thx a lot
y

[sh0dan]

August 17th:

I currently cannot browse the CVS, so this is the changes I remember:


- Added: ImageReader/ImageSource now accepts relative paths in all configurations.
- Fixed: ImageReader now no longer reports errors/info upside-down.
- Added: DirectShowSource now requests interlaced material to avoid internal (crappy) WMV9 deinterlacer. (Thanks to Russel Lang)
- Fix: Huge stability fix by IanB. This should remove a bunch of "Evaluate: Unregnized exception", and crashes on reload in vdub.
- TDPDeliver: Simplification and cleanup. Planar pitches handled more consistently.
- Fix: DirectShowSource properly relasing filters on unload.
- Added: BlankClip now has color_yuv, that allows setting and YUV color for YUV clips.
- Added a huge list of color presets (http://www.avisynth.org/ColorPresets), that can be used instead of colors.

[Didée]

Fine, fine, fine :)
Big thanks to all contributors.

- Fix: Huge stability fix by IanB. This should remove a bunch of "Evaluate: Unregnized exception", and crashes on reload in vdub.
Ah. Last weeks, there were times where I was spending more time on "where IS the damn error, my application disappears without message" than on actual scripting ...

***

Quick question:
Yesterday I wanted to process a pile of images through avisynth, but resize the input with different resizers from ffdshow.

Made two scripts:
- loader_script: "ImageReader(...)
- proc_script: "DirectShowSource("loader_script.avs") ... etc."

Problem:
Loading the proc_script through graphedit, ffdshow was linked into the DS chain as expected.
Loading the same script in VdubMod, ffdshow was not kicking in.

Where's my mistake :confused:

- Didée

[Wilbert]

@Sh0dan,

I committed two color_preset files (the short avsi one, and the long txt one). I thought perhaps the long one put too much variables in your memory. But, I guess you can use 1024 variables, so it shouldn't matter ???

[sh0dan]

Originally posted by Wilbert
@Sh0dan,

I committed two color_preset files (the short avsi one, and the long txt one). I thought perhaps the long one put too much variables in your memory. But, I guess you can use 1024 variables, so it shouldn't matter ???
The Variable table should be a linked list, with no size restrictions, beside available memory. Have I missed this setting somewhere?

[Wilbert]

Which setting? Is there a maximal number of variables which can be defined (or does that depend on the available amount of free memory)?

[Dreassica]

I'm getting scriptclip errormessages on all my usual qmf scripts i use with this latest version. They all worked fine with the previous builds.
errormsg is:

evaluate Unrecognised exception
([scriptclip] line 1}
([scriptclip] line 1}

[sh0dan]

Originally posted by Wilbert
Which setting? Is there a maximal number of variables which can be defined (or does that depend on the available amount of free memory)?
I just wondered where you got '1024' from?

@Dreassica: Could I get a link to the scripts?

[Dreassica]

U mean mine, or the qmf script from homie fr?
Cuz theres an example of that on avisynth.org

anyway, my script used is (using qmf 1.5b1):

function Low_Motion_Filter(clip c)
{
c = Deen(c,"w3d",1,4,4,11,12,0.25,4)
c = unfilter(c,0,-1)
c = undot(c)
return c
}
function Medium_Motion_Filter(clip c)
{
c = Deen(c,"w3d",1,8,7,6,6,0.35,6)
c = undot(c)
c = unfilter(c,-1,-2)
return c
}

function High_Motion_Filter(clip c)
{
c = Deen(c,"w3d",2,12,11,3,4,0.45,1)
c = unfilter(c,-2,-3)
c = undot(c)
return c
}
mpeg2source("F:\VIDEO_TS\10.d2v")
doubleweave().TPRIVTC("F:\VIDEO_TS\\10\10.tpr")
crop(10,4,696,476,align=true)
QMF(chroma=true,threshold_lm=2.9,threshold_hm=11.3)
bicubicresize(640,480,0,0.5)
tweak(cont=1.02,sat=1.3)
limiter()

[sh0dan]

@Dreassica: Thanks!

@IanB: Seems like there is still some optimizations bugs left. :(

I have do diable: "Global Optimizations" and "Full Optimization", otherwise we get a crash, in code very similar to what you experienced. I can however turn on "Frame-Pointer Omission" without experiencing problems. Your reaction?

[Wilbert]

I just wondered where you got '1024' from?
This turned out the be the maximal no of arguments of a function :)

[You Know]

YDifferenceFromPrevious
UDifferenceFromPrevious
VDifferenceFromPrevious

YDifferenceToNext
....

does not work anymore


Evaluate: Unrecognised exception!
([scriptclip] line 1)


all code post on help and web page (www.avisynth.org) fails ... for example

clip = AVISource("clip.avi",audio=false)
ScriptClip(clip, "Subtitle(String(YDifferenceFromPrevious))")


I come back to 2.5.5 beta 2 and works perfect, please fix
:(