Proposal for AviSynth 2.6

[sh0dan]

I just saw that Avery Lee supports Y8 in his latest vdub, and this just spawned an idea for a meaningfull update to 2.5. Since the 2.5 is getting very stable and fast due to Ian's incredible refactoring job, I think we might look a bit further ahead and take a small leap forward in the core. I hope my inactivity doesn't disqualify me completely

Design goal proposals:
*) Full compatibility with existing 2.5 plugins.
*) Full compatibility with API. Meaning 2.5 filters should recompile with new avisynth.h without modification.
*) New features must be implementable within reasonable timeframe.

Proposed additions:
*) Y8 planar format.
*) YUV 4:4:4 planar format.
*) All planes delivered to filters are 16 byte adress aligned and have mod 16 pitch.
*) Thread-safe avisynth.h.
*) Add MaskTools to the core.

Possible additions:
*) Official Win64 support.
*) Intelligent frame prefetching on multicpu/core.

Why:
*) Y8 planar format. We are doing a lot of masks and similar stuff. Not always having two chroma channels to worry about would yeild good speedups.
*) YUV 4:4:4. Not being able to do YUV without having subsampled chroma is just annoying.
*) MaskTools Simply a spectacular package, with really good generic tools used by many script fuctions. Would also be great for developers to be able to use the filters by using env->Invoke.


Implementation:

Y8 planar format.
Add new videotype specifier and make the adjustments for VideoInfo and VideoFrame struct methods.

VideoInfo->IsPlanar() should return true. VideoInfo->IsY8() should be added.

VideoFrame->GetHeight(), GetWidth, GetPitch should all return 0 if a chroma channel is needed. GetReadPtr/GetWritePtr could either return NULL, or an empty buffer of X bytes if there are stupid filters out there.

As for filter implementation, it is mostly trivial changes needed for Y8 support for 90% of all filters. In filters where it was obvious no changes should be needed, as they only check vi->IsPlanar() and request the proper sizes for each plane.

Converters to and from this format are trivial. For RGB conversions we could start out by using an intermediate Y8<->YUY2<->RGB, just as YV12 currently does.

YUV 4:4:4 planar format.
Similar as Y8. I haven't been able to find any official name for this format. Proposed name 'YV24'.

Since there is no official name for this we might automatically convert it to another format (YUY2) when exporting to an application, and put in a variable switch to force YUVP export, for the experts.

Converters are not 100% trivial, but as a starting point there are converters to/from all current formats in the "Overlay" filter.

16 byte alignment
Already prepared AFAICT. We can push this in a 2.6 upgrade.

Thread safety.
Already done AFAICT. Guess we owe TSP some thanks - now it just needs to be compiled into all filters. "2.6" would be the perfect excuse.

Add MaskTools to the core
If Manano allows, it would make a great addition to the core, since it has now proven itself stable in the field. Should still be a separate plugin with Manao in complete control.

Official Win64 support.
I would very much like this to happend, but it requires sooo much work. The current version is a great start, but I'm not sure it's in a realistic timeframe. Realisticly it might not be a 2.6.0 priority, but a slow transition towards this would be great.

Intelligent frame prefetching on multicpu/core.
**) Proposal 1 (simple)
Make async interface, where frame generation is handled by a second thread.
Right before returning frame X to the application the second thread can immediately begin producing frame X+1. I don't think this will help vdub much, but other apps are not as intelligent.

**) Proposal 2 (advanced)
When application requests frame X, generate frame X and X+1 in two separate threads. Avisynth could buffer a number of frames ahead this way. Much more risky, since it can only use threadsafe plugins.

Again not a 2.6.0 priority, but it should be in our plans.


What's left out
*) More than 8bit per component.
*) More managed audio.
*) Your favorite annoyance???!?

Step 1, however, is IMO getting 2.5.6 out the door. I don't know about the new DSS, but otherwise the core seems VERY stable in the latest beta!

Thoughts? And when do we get started?

Edit 1: Clarified Masktools.
Edit 2: YUVP -> YV24

[mg262]

*) Y8 planar format.
*) YUV 4:4:4 planar format.

Yes and yes.

MaskTools would be nice, assuming @manao didn't mind and that it didn't e.g. slow down his frequent and useful minor updates. The name clash for Invert would need to be resolved... I think that the one in the core is far less useful than the one in MaskTools.

If the pitch issue has been resolved in such a way as not to break anything it would be nice to have that in... at least I often find that I can't be bothered to put certain things into SSE2 because of the 8/16 issue (but probably just means I'm lazy ).

Incidentally, thank you for organising that post to make it so clear and easy to read.

Edit: one other thing to think about is scripts which require older versions of MaskTools (particularly mfToon)... I can't remember why this is; it may just be because of the removal of a function (YV12layer?) that could just be added back? [I realise that you never said that all the MaskTools functions should necessarily be moved into the core.]

On the thread safety count, if it is added into AVISynth 2.6, a note specifying what new filters should/should not do would be really useful.

As you say, Win64 support seems to be on a different timescale to the other additions, and I think it may raise the difficulty that a relatively small proportion of the current user base is in a position to test it. To me, doesn't seem to fit in with the coherent set of core additions that you suggested... but then I don't have a 64-bit processor, so I'm probably biased!

[Wilbert]

Thoughts?

Sounds great!

I very much would like to see ImageMagick support: http://forum.doom9.org/showthread.php?s=&threadid=90131

[ARDA]

@sh0dan

First of all glad to see you back with many ideas.
More than a suggestion I have some questions that could open a small
change and could maybe drive to some improvement.To point.

Would it be possible to make a smaller core, that means, to compile
many of the filters that are nowadays included, in a separate project
as you have done with directshow.
What for?
In that way any change made over any filter will not need a whole avisynth recompile.
And if some change in the core not obliged to recompile all the filters.That could also
allow many authors to continue its development separately without problems.
I don't know if that could create problems with plugins directory; or any other I dont get
to forsee.
But thinking in the movements to future 64 bits version (now I have a laptop with ML37 turion);
I've thought working over a smaller core should be easier,and step by step recompile filters.
This is something that has been over my head many times, and the poor skills I have dont
allow me to go deeper in this idea.

Anyway any improvement and changes are always welcome, including bugs that makes
things more interesting.

Once more thanks
ARDA

[sh0dan]

@mg262: I clarified the Masktool bit. Of course Manao should still maintain complete control!

@ARDA: Some people are already hitting the ~70 plugin limit (imposed by windows DLL limitations AFAIK). So a complete splitup isn't the best. But I see your point - especially when porting is an issue.

[stickboy]

Originally Posted by sh0dan

Add MaskTools to the core
If Manano allows, it would make a great addition to the core, since it has now proven itself stable in the field. Should still be a separate plugin with Manao in complete control.

Yeah, I agree that in this case it'd be better to simply include it as a separate DLL like DirectShowSource.

@ARDA: Some people are already hitting the ~70 plugin limit (imposed by windows DLL limitations AFAIK). So a complete splitup isn't the best. But I see your point - especially when porting is an issue.

Exactly what's the deal with this limit? Is it that 70 DLLs can't be loaded simultaneously? Can AviSynth use a lazy loading scheme?

[Manao]

Hi guys,

What you plan to do would be great. I'll ask Kurosu ( original author of the masktools ) but he should agree.

However, at the moment, I'm rewriting the set of filters, to ease its portability ( linux -> no inline asm ) / maintenability ( avs 2.5 / avs 3.0 -> need a core totally independant from avisynth ). I'm half through that process.

I'll use this opportunity to break some things too. Mainly some filters default behaviors which sometimes aren't especially smart, and perhaps some other things will change ( frame boundary's behavior of edgemasks to help mmxing those which aren't ), or won't be supported anymore ( float dedgemask, special modes that can be emulated with yv12lut, float convolution, ... ), because they clutter greatly the code.

Filters should be able to support 4:0:0, 4:2:0 and 4:4:4 easily.

one other thing to think about is scripts which require older versions of MaskTools (particularly mfToon)... I can't remember why this is; it may just be because of the removal of a function (YV12layer?)

It's indeed YV12Layer, which I never supported. YV12Lutxy can do everything YV12Layer was doing, so I removed YV12Layer ( which was buggy anyway ). mf never cared to update its script, but I can bug him if it bothers some.

[Didée]

Originally Posted by Manao

YV12Lutxy can do everything YV12Layer was doing, so I removed YV12Layer ( which was buggy anyway ).

If you would (find the time to) cut out the exact formulas that were used by YV12Layer (seeing is better than guessing), I could quickly put the according LUTs into a custom function that rebuilds YV12Layer through YV12Lutxy.

[Manao]

add : ((y - x) * level) / 256 + x
add_no_cr : ((128 - x) * level) / 256 + x
sub : ((255 - y - x) * level) / 256 + x
sub_no_cr : ((128 - x) * level) / 256 + x
fast : (x + y) / 2 + 1
mul_cr : ((y - x) * level) / 256 + x
mul_no_cr : ((128 - x) * level / 65536 + x
mul_y : (((y * x) / 256 - x) * level) / 256 + x

Explanations :
- x, y : as for yv12lutxy.
- no_cr : chroma processing uses no_cr if (!chroma)

Now, as you can see, some formulas seem plain wrong ( but taken from the code ), which is why I never bothered trying to understand them. But :
* 'fast' creates an offset of 0.5
* mul_no_cr divides two times by 256, instead of one. ( but that's obviously a bug in the C version, since the mmx one seems correct and divides only once by 256 )

[IanB]

Originally Posted by sh0dan

Design goal proposals:
*) Full compatibility with existing 2.5 plugins.
*) Full compatibility with API. Meaning 2.5 filters should recompile with new avisynth.h without modification.
*) New features must be implementable within reasonable timeframe.

I was thinking about doing a NULL api transformation by ripping all the baked in code out of avisynth.h

Interface version 2 is the current baked in code, interface version 3 would be 100% source compatible but with no baked in code. Later with no baked in code we get to change the way things work internally without anybody needing to know or care or recompile. Also this would open the gate for David's avisynth3 to support version 3 api pluggins without a recompile. Also we should loose all the public variables and implement get/set methods, unfortunatly this is hard to make source compatible i.e. vi.width=640 --> vi.SetWidth(640) but for the get methods we could define suitable textual replacments ie "#define width GetWidth()" perhaps some C++ namespace trickery could get a cleaner result.

Proposed additions:
*) Y8 planar format.
*) YUV 4:4:4 planar format.

I was also thinking of a 4:2:2 planar format mapped straight from YUY2. This would make it easy to enlist the resizer core to do the hack work of resampling the chroma. i.e. just provide 444<->RGB and 422<->YUY2 and build everything else from the resizers.

Also we could just decouple the planesize relationships so we internally support any planar format one could dream up. We provide some standard templates like YV12, Y8, YUV24, YUV16 and YUV9 so filter authors can quickly check if they support the current shape. For output we simply Throw an exception if the shape is not well defined, like we do with non-mod2 now. This of course means no more hookey code that assume width/2 or other evil incantations.

Possible additions:
*) Official Win64 support.

The killer is what to do with the 1000's of lines of __ASM we have now, this code is the reason avisynth is fast. Perhaps softwire will get their 64 bit support out real soon now. But on a positive note we can at least aim to stop doing all the evil unportable address <-> int assumptions. And it is a chance to try out the C code versions.

Implementation:
...
VideoInfo->IsPlanar() should return true. VideoInfo->IsY8() should be added.

VideoFrame->GetHeight(), GetWidth, GetPitch should all return 0 if a chroma channel is NOT needed. GetReadPtr/GetWritePtr could either return NULL, or an empty buffer of X bytes if there are stupid filters out there.

Unfortunatly we don't have a "this is a new video format you do not support" state. So to maintain source compatibilty we might need a VideoInfo->IsPlanarEx() to get to the new formats, i.e. IsPlanar() only returns true for YV12 and maybe throws an exception for extended planar formats.

It is probably cleanest to just Throw exceptions when an old filter tries to process new formats, this forces the user to knowledgably "cast" the format appropriatly.

16 byte alignment
Already prepared AFAICT. We can push this in a 2.6 upgrade.

If we get rid of the baked in code this becomes much easier. The upcoming implementation is really a bag on the side, chroma 16/luma 32 just so the baked in code is kept happy.

I was also thinking of including diagnostic filters that randomises and test the alignment relationship so we can QA suspected plugins for illegal assumptions.

Thread safety.
Already done AFAICT. Guess we owe TSP some thanks - now it just needs to be compiled into all filters. "2.6" would be the perfect excuse.

I have parked this in a branch pending a 2.5.7/2.6 release. Again the baked in code makes this ugly to implement.

Step 1, however, is IMO getting 2.5.6 out the door. I don't know about the new DSS

I had hoped to be doing release candidates by now, the main core is ready, however I have been sucked into the DirectShow quicksand, it is proving very chalanging making it come to heal. (Bad dog!) But it should be worth waiting for. I hope to have it house trained by next weekend.

Also the rumblings about YUY2 <-> YV12 conversions, I am currently leaving it alone. I will probably do an interim 2.5.7 maintenance release to put that out (along with any latent bugs).

IanB

[mg262]

unfortunatly this is hard to make source compatible i.e. vi.width=640 --> vi.SetWidth(640) but for the get methods we could define suitable textual replacments ie "#define width GetWidth()" perhaps some C++ namespace trickery could get a cleaner result.

I I think this probably isn't the right solution in this case, but it's an interesting idea so...:



There is the option of having a function vi.Width() that allows the following sort of code to be written:

i = vi.Width();
vi.Width() = 340;

Which works by returning an object that behaves like a reference ... and i.e.

Pseudowidthreference Width();
Const_Pseudowidthreference Width() const;

And Const_Pseudowidthreference and Pseudowidthreference are built in such a way as to perform the appropriate get/set behaviour.

[MfA]

Would be nice if each frame had a pointer for some custom data.

[IanB]

Also there is stuffing with operator = And there has to be a balance between source API compatibility and devious unmaintainable implementation.

Keep the ideas flowing

IanB

[MfA]

That's cryptic. I can see how the data needs a destructor, but the operator = doesn't do a deep copy right? Just add add a class definition for custom data, with a pointer to the data, an ident string and a destructor. Add a pointer for custom data to the videoframe. Null the reference in the videoframe smartpointer init. If custom data is present call the destructor in release if reference count drops to 0. Pretty straightforward.

(I want to make some HDR filters, and string them together ... could always just store a pointer in the actual buffer I guess, fugly but it works.)

[IanB]

@Mfa, Sorry the reference was to Mg262's post, but I obviously inspired some illumination.

Many moons ago I had some thoughts on the per frame user data issue, I concluded a simple pointer wasn't adequate (pluggins would fight for it). I think I proposed some sort of simple keyed access i.e. value=frame->GetUserData(key); frame->SetUserDate(key, value);

IanB

[Bidoche]

Originally Posted by sh0dan

*) All planes delivered to filters are 16 byte adress aligned and have mod 16 pitch.

On that point, I would like to suggest to go one step further and add guards on the memory block.
That is to say extra space before the 1st scanline and behind the last one.

The point is to allow to make simpler code that doesn't have to worry about edge conditions.

For example, the asm version of RGB32 to RGB24 process 4 pixels at once, ie it reads 16 bytes and writes 12 in an inner loop.
With a guard of 16 bytes it could be done without handling the 3 edge cases.
At worst it would output 9 bytes of garbage, but the guard would guarantee its safetiness.

It's a change I am currently considering for 3.0.

Originally Posted by sh0dan

YUV 4:4:4 planar format.
Similar as Y8. I haven't been able to find any official name for this format. Proposed name 'YUVP'. Easier to distinguish from YUY2 than 'YUV'. 'P' is for planar.

I am using 'YV24' in 3.0.
If a consensus is found for another name, I will change it.

[Bidoche]

Originally Posted by IanB

Also we could just decouple the planesize relationships so we internally support any planar format one could dream up. We provide some standard templates like YV12, Y8, YUV24, YUV16 and YUV9 so filter authors can quickly check if they support the current shape. For output we simply Throw an exception if the shape is not well defined, like we do with non-mod2 now. This of course means no more hookey code that assume width/2 or other evil incantations.

My solution in 3.0 is these methods :

//inside ColorSpace class (but it can be inside VideoInfo in 2.5)
char * GetPlaneList() const;   //each char identifies a plane, '~' alone for interleaved
bool HasPlane(char plane) const;
void ToPlaneDim(int& x, int& y, char plane) const;  //converts frame dimension to plane dimension

Then everything should rely on these and never make assumptions.

[mg262]

Originally Posted by Bidoche

I am using 'YV24' in 3.0.

This is not a major point, but it's not very clear whether YV24 means 4:4:4 or a 16-bit version of YV12... or at least, I wasn't sure until I read this post.

[sh0dan]

Originally Posted by IanB

I had hoped to be doing release candidates by now, the main core is ready, however I have been sucked into the DirectShow quicksand, it is proving very chalanging making it come to heal. (Bad dog!) But it should be worth waiting for. I hope to have it house trained by next weekend.

Yeah - I remember implementing audio and seeking in DSS into the original code. Lot's of guessing and strange behaviour.

Could it be a solution to release 2.56 with the unmodified DSS (with the VFR fix) - after all 2.56 is miles better than 2.55, and DSS is easy upgradeable anyway.

Originally Posted by IanB

Unfortunatly we don't have a "this is a new video format you do not support" state. So to maintain source compatibilty we might need a VideoInfo->IsPlanarEx() to get to the new formats, i.e. IsPlanar() only returns true for YV12 and maybe throws an exception for extended planar formats.

Not sure I get this. 2.5 compiled plugins will only work with YV12, but if they are provided with an updated avisynth.h it should be able to return true for all 2.6 supported plugins.
Old (recompiled) filters checking isYV12() will (naturally) only support YV12. Old (recompiled) filters checking isPlanar() should process new formats - which should be ok.

Generic planar support sounds like a good idea. Y8/YV16/YV24 sound like good ideas. Using the resizer for chroma upsampling seem like a good idea. One concern. How do we indicate chroma placement on arbitrary planar formats?

The reason I proposed sticking with the baked code, and fixed formats is that it is a change that I could do myself. It's quick - for us to implement - and for developers to adopt. I would work - even if it's suboptimal.
It would make a great update for 95% of all users and developers. Basicly it's a "0.x" update, and not a "x.0" update.

I'm justifying the conservative thinking by trying to do something that we can do in a forseeable future.

[Mug Funky]

while on the subject of masktools... is it safe to use on HT systems? the version i'm using (not sure if it's the latest - the docs say version 1.4.16) doesn't work on hyperthreaded P4's. i've disabled HT on my machine, but there's other machines here that are often needed, and turning it off on all machines isn't a good option (and it does give a legitimate speed gain when encoding and using Spruce at the same time, as the latter uses different instructions to the former).

i'm all for merging it into avisynth, and Y8 looks exciting too. i guess i just want to be sure that it'll continue to work on HT machines.

btw, the new resizers are awesome. really good idea.