avs file to large?

[GoldBuggie]

Hi

I have a avs file which I have used for a previous encode. But this particular source that I'm working on has blends to the max. So...what I did was using 'FreezeFrame' to clean things up and get a lesser blended file. Problem now is that when i use the FreezeFrame commands I get a 'Avisynth open failure: Avisynth: script open failed!' when I open the avs file in latest build of VDubMod. The program also crashes on some attemps when i open it.

I tried commenting out the FreezeFrame commands and everything worked. From where I'm sitting it seems that the avs file gets to big when I include the necessary commands i need. Since I have a very very large file with the FreezeFrame commands(393k).

Does anyone have any idea how i can get this to work?

The smalles script that i tried just to get things as easy as possible and locating the fault is this:

loadplugin(...)
setmemorymax(512)
Mpeg2Source(...)
Telecide(...)
Import("...freeze.txt")


Thank You

[Guest]

What is in freeze.txt?

[GoldBuggie]

Only

.
.
.
FreezeFrame(269,271,272)
FreezeFrame(275,275,276)
FreezeFrame(277,278,279)
FreezeFrame(280,281,292)
FreezeFrame(291,291,292)
FreezeFrame(299,348,349)
FreezeFrame(350,350,351)
FreezeFrame(358,358,360)
FreezeFrame(359,359,360)
FreezeFrame(362,362,364)
FreezeFrame(363,363,364)
.
.
.

alot of those lines

[stickboy]

Unfortunately, there are some limits to the number of filters you can chain together. You could try calling SetMemoryMax to increase the memory limit, but I'm not sure that would help.

The best thing to do would be to split up your FreezeFrame calls, generate a temporary AVI file, use the remaining FreezeFrame calls on that new AVI, repeating as necessary.

[GoldBuggie]

Thanks for the input!

Since i didn't want to take away all the work i've put into creating that freezeframe file i decided to split it into 2 avifiles.

I had to create a little script that created 2 freezeframe files, which is very easy. The second file just needs the numbers subtracted by the beginning of the trim number.

Then I did this(in short):

...
Trim(18501,37852)
Import("freeze02.txt")

and a
...
Trim(0, 18500)
Import("freeze01.txt")

Which gives me 2 avi files. Then i created another avs file which joined those 2 avi files and with that file I am now doing decimate, filtering etc.

[opieant]

I just ran into a similar problem today with a utility I'm using to generate AVS files using a large number of calls to the Trim function. One Trim call is made for each frame of output. If it helps or means anything, the limit seems to be 16,065 times that this can be called. 16,066 or more (at least with the source material I'm using, a 16374 frame DV clip at 59.94FPS, 720x480) causes VirtualDub or WMP or close upon loading the AVS without reporting an error. Using SetMemoryMax makes no difference. Would be nice to be able to disable whatever limits are causing this in a future version of AVISynth. Splitting up the results will have to be my solution for now as well.

[stickboy]

Why do you need to call Trim on each frame?

[Bidoche]

@opieant

You seem to think we chose to enforce those limits, but we didn't.

There are due to hardware limits :

When a filter chain becomes too deep, a frame requests descending it will end with a stack overflow....

And there is not much we can do against that.

Other than trying to prevent filter chain depth to grow.

3.0 does that in the FreezeFrame case, it won't crash.

[opieant]

stickboy: May as well explain what I'm doing. I posted in the HDTV forum (http://forum.doom9.org/showthread.php?s=&threadid=77177) about the fact that the HiCon32 demo has a nasty habit of discarding frames. Actually, it seems to be working on a field-by-field basis from the input, but that doesn't change the fact that when it throws duplicate information away and the audio track is added back later, the varying rate of video playback throws the audio and video way out of sync). The video will always finish before the audio, and changing the frame rate of playback to "stretch" out the video will not work. Due to the program's frame limit (3000), I'm already working with smaller clips totaling 8187 frames. When I put the output files from HiCon32 back together I should have a total of 16,374 frames (double 8187 frames) but only 14,546 frames come out. To correct this, I have a set of bitmaps added along the left side (using Donald Graft's Logo filter) of the input video that make a white rectangle appear to be moving up the screen field by field, appearing in one of eight possible locations, then returning to the bottom and repeating until the clip is over. After HiCon32 processes these clips, I can Crop each of the regions where the rectangles can appear and use WriteFile to make an output file containing the AverageLuma values for each one of the rectangles on each output frame. Then, a tool I wrote uses the luminance value logs to determine which fields were discarded (or in very rare cases duplicated) and how many times the related output frame needs to be duplicated in order to end up with 16,374 frames all in the right places. Once it has processed the luminance logs fully, the script full of Trim functions is produced with one "Trim(FrameNum, -1)"+ for each instance of a frame. This works fine for smaller clips, so I guess I'll be sticking with those for now.

Bidoche: I assumed it was failing due to a safety measure in the software and that there was some finite limit set that nobody was expected to reach. That's why I suggested the ability to disable the restriction. Sorry about that.

I'm just curious, would having some kind of segment marker be a way to solve this problem? By this, I mean that in the script some kind of directive could be added that tells AVI Synth to only have the filter chain be made from the current directive to the next directive, or to the end of the file if on the last one. I'm not sure how these chains really work so I don't know if this could cause problems (besides a delay when a transition from one segment to another is made since the old chain would be cleared from the stack and the next one would take its place).

Also (on a different but project-related note), do you know why AverageLuma seems to be limited to clips with a width of 16 pixels or more? I get 1.#INF00 as all of my luminance values if I crop the region for my moving rectangle bitmaps and set it to crop more than 704 pixels out of 720 from the right side. Since the bitmaps must remain in the output video from HiCon32 (because they have a small effect on what is discarded and what is not), I only want them to be covering up unimportant parts of the video (an empty or noisy area). Using narrower rectangles for appropriate videos is not a problem by itself, but with a minimum width of 16 for the cropped region more and more of the values from AverageLuma (percentage-wise) will be based on changing video rather than solid regtangles with predictable luminances. The greater the percentage of normal video present in the cropped region, the more difficult it becomes to predict whether a rectangle is present or not, and this interferes with the ability to detect which fields were discarded accurately.

To All: Thanks for responding, and hopefully I can at least get one of my problems solved this week :-)

[esby]

Maybe adding internally a filter that does several freezeframes would solve partially the problem.
And simplifying the filter chain is several freezeframes are chained...`

The best solution would be to detect a freezeframe while we were going to add it, and thus, passing the new parameter to the (previous) existing trim filter instance.`

That would mean to modificate trim in order to operate with a non limited number of range to 'clone'...

esby

[stickboy]

Originally posted by Bidoche
When a filter chain becomes too deep, a frame requests descending it will end with a stack overflow....

And there is not much we can do against that.Another possibility might to be to use a heap-allocated stack, but that'd probably be a lot of work...

[Bidoche]

@esby

Maybe adding internally a filter that does several freezeframes would solve partially the problem.
And simplifying the filter chain is several freezeframes are chained...`Simplifying is the word, that's exactly what I do
But not with a grouping freezeframe filter.

Actually there is no FreezeFrame filter in 3.0, it's built in term of Trim and Splice and use their refactoring abilities ( Trim can trim inside the Splice and Splice is self-absorbing )


@stickboy

By definition, the stack is not heap-based...
Besides, that wouldn't change the fact that its size is fixed.

[rhombus]

FreezeFrame(269,271,272)
FreezeFrame(275,275,276)
FreezeFrame(277,278,279)
FreezeFrame(280,281,292)
FreezeFrame(291,291,292)
FreezeFrame(299,348,349)
FreezeFrame(350,350,351)
FreezeFrame(358,358,360)
FreezeFrame(359,359,360)
FreezeFrame(362,362,364)
FreezeFrame(363,363,364)
.
.
.

alot of those lines

If you are planning to do a lot of this type of stuff, it looks to me as if you need a new plugin that does custom mapping between the source frame and your output frame.

You could rewrite the tool that you use to produce "freeze.txt" so that it would instead produce a file suitable for input to ConditionalReader. I'd suggest that you input the offset from your output frame to the source frame, so the above spnippet would become:

TYPE int
DEFAULT 0
269 3
270 2
271 1
275 1
277 2
278 1
...
etc


The basic design for the plugin can be found by studying the code for the internal function SelectEvery, replacing the GetFrame function body with something like


inline PVideoFrame __stdcall GetFrame(int n, IScriptEnvironment* env)
{
// varname contains the name of the global variable that ConditionalReader writes to
AVSValue offset = env->GetVar{varname);
return child->GetFrame(n + offset.AsInt(), env);
}


If you don't have VC++ 6.0, I'm sure that one of us can cobble something together that does what you want.

-Ron

[stickboy]

Originally posted by Bidoche
By definition, the stack is not heap-based...
Besides, that wouldn't change the fact that its size is fixed.I don't mean the stack; I meant a stack. That is, your own, heap-allocated stack data structure that you can use to keep track of function calls.

If it's heap-allocated, its size is still limited, sure, but at least there's more room, and you have more control over it by calling SetMemoryMax and by buying more RAM, right?
Originally posted by rhombus:
If you don't have VC++ 6.0, I'm sure that one of us can cobble something together that does what you want.I probably could have something quick-and-dirty done tonight.

(BTW, not having VC++ 6 isn't an excuse anymore; Microsoft released their VC++ 2003 compiler for free (http://forum.doom9.org/showthread.php?s=&threadid=74608).)

[stickboy]

Ho hum, here's my lame implementation of what rhombus suggested. (It's not exactly the same; I think absolute frame numbers are easier to manage than offsets.)

[link removed, see below]

This is temporary; don't get too attached to this... I expect to redo a lot of it (including the input file format and maybe even the filter name) pretty soon.

[stickboy]

Okay, here's a real version.

RemapFrames (http://www.avisynth.org/stickboy/RemapFrames.zip)