Variable frame sizes

[mg262]

I'm encoding non-video information into video frames (in the same way that e.g. motion vectors are placed in clips by motion detection filters). The amount of information can vary substantially from frame to frame. Until recently I had just thought of allocating the worst-case amount, although that results in considerable space wastage as the average-case size is a fraction of the worst-case. But I think I may have just thought of a totally obscene workaround...

Keep the frame size very small (e.g. 8x8)* and use the optional align argument in NewVideoFrame to vary the pitch. So if I needed 8000 bytes of space for a particular frame, I could use this:

env->NewVideoFrame(vi, 1000)

-- i.e. request alignment of 1000 in order to get a 1000x8 block into which to write. (It's not nice, but it's probably better than the alternatives in this case.) What I wanted to ask was, is there any reason why this wouldn't work?

*The number should perhaps be changed to avoid excessively wide frames, which can cause problems for VirtualDub. Also I've found that allocating YV12 frames smaller than 4k pixels results in allocation becoming slower and slower with each frame allocated...

Thanks!
M.

[Guest]

I'm encoding non-video information into video frames OK, I'll bite: why are you doing that?

[Bidoche]

What you need is the ability to attach extra info to a frame, not to put it into the padding space...

[mg262]

The situation has actually cropped up several times. The first is that for a particular filter I want to have streams of booleans, integers, etc., i.e. one value per frame. So for example average luma could be regarded as a stream of integers or floats. This is relatively straightforward and makes it easy to do certain things which are a little tortuous in the current approach. (A slow example is to sort all the frames in a clip by some script-specifiable property...) This one is of course done in bounded space (i.e. constant frame size).

A slightly more complicated example is when storing motion vectors, but in a case where you might want to store e.g.

Frame 10 -> frame 11
Frame 10 -> frame 12
Frame 10 -> frame 13
Frame 10 -> frame 14
Frame 10 -> frame 15
Frame 10 -> frame 16
Frame 10 -> frame 17

Frame 11 -> frame 12
Frame 11 -> frame 13
Frame 11 -> frame 14
Frame 11 -> frame 15
Frame 11 -> frame 16
Frame 11 -> frame 17
Frame 11 -> frame 18

- i.e. for all frames within a certain distance of each other. Obviously, these are over determined but resolving them with each other (using e.g. least squares regression) gives you better motion estimates than you would had originally (at some cost in time). I have tried this and it works. The key point is that the script decides which motion vectors to store, and then calls a resolve function which collapses them to one motion vector per frame. This is not seriously constrained by being required to run in bounded space (bound known at filter construction time), but if variable size is allowed a script can choose to adaptably calculate different amounts of the motion vectors in different places so that the e.g. still scenes are stabilised more effectively than moving scenes.

Neither of these was actually the problem that really required variable frame size, which I'm building with a friend was visiting at the moment (and which is slightly crazy, so I'm rather sensibly overruled from writing about it until we see if it works!) But if it works you should hopefully see it before too long...

Edit: A simpler example (perhaps not to implement!) is to take a stream which encodes MPEG frame type, i.e. I, P, B...

[mg262]

What you need is the ability to attach extra info to a frame, not to put it into the padding space...Yes, that would be nice ;)

But with 2.5...

I know that I won't be able to save the resulting streams through e.g. VirtualDub (which is something I was doing with the kinds of streams I mention in my last post), but their main function is to flow out of one filter and into another to allow modularity.

Incidentally, some of the streams I'm working with (conceptually) need to be calculated from one clip and then used with another, so it's not quite the same as attaching extra information to a frame. (Although of course I could attach information to an empty video clip to have create an essentially independent stream, and I would if the functionality were at hand...)

Looking forward to 3.0!

edit: it seems to work...

[esby]

What is stopping you from using external files?
Can't you just write in a text file the data you want to pass with another filter.
If your data require the two filter to be in the same chain, and in the same pass, then you'll have to do something a bit more complicate, maybe using some client server application with local sockets to have the filter communicate between them.
It is just an idea I am suggesting.

esby

[mg262]

Thank you for the ideas! Here's something taken from a real script:

save
graph = ConstructFloodGraph(seeds,15,50)
sizemask = black.ColourGraphBySize(graph,0,100)
smallgraph=graph.MaskGraph(sizemask)
absorbedgraph=AbsorbByColour(smallgraph,save,400000, 7, 70)
mergedgraph=MergeGraphByColour(absorbedgraph,save, 0, 70)
#mergedgraph=MergeGraphByColour(graph,save, 255, 2*255*255)
ColourGraphByAverage(save, mergedgraph, save)
DrawGraphShowingAverage(graph,save)


All the arguments in bold need to be passed between filters, and they are not videos. So for the reasons that you mention using text files would beawkward... and also slow.

Client/server functionality... well it would require substantially more work, and would involve a separate cache, raising the difficulty of splitting work and memory between the explicit cache and the AVISynth one, which is difficult to do well as each side has only half the picture. Plus it would introduce a dependency on whatever technology was used for sockets, which I prefer to avoid.

[esby]

I can assume that all the function you written are yours.
So you have another possibility.
Try hacking the avs core,
adding a Graph class to avisynth.h and a Graph GetGraph(int i,env...) method to IClip.
And have your filters uses it to transmit the information you need.
That has some inconvenient, but it might work.

esby

[mg262]

-- the current approach is working... but I wrote that at the end of the post above in an edit, so it wasn't very obvious (sorry)! The example you saw is a real fragment of a real script that is working; I just need to tidy up a few things before posting a filter.

Hacking AVISynth might work, but it would also mean that no one else could use the filters... at the moment I just have to add GraphFromFrame and/or FrameFromGraph calls in a few places... probably a dozen in a couple of thousand lines of code, so it's not too inconvenient. In fact, in the case where the object you're encoding is small enough to be copied safely (or the adoption of an use counter with the associated overhead of heap allocation is acceptable), as with all the things I mentioned above, you can set up a context which lets you write something like this (note the parts in bold):

class ScaleMotion : public MotionFilter
{
Motion motion;
double scale;
public:
ScaleMotion(PClip _child, Motion _motion, double _scale,IScriptEnvironment* env)
: MotionFilter(_child, env), motion (_motion), scale (_scale) {}
virtual MotionPt GetPt(int n, IScriptEnvironment* env);
};

AVSValue __cdecl Create_ScaleMotion(AVSValue args, void* user_data, IScriptEnvironment* env)
{
return new ScaleMotion(args[0].AsClip(), Motion(args[0].AsClip()), double (args[1].AsFloat()), env);
}

... again that's taken from a real example, where Motion is a type associating a motion vector with each frame.