YUV12 in avisynth

[Marc FD]

Hi
I've heards some things about YUV12 support in avisynth, but there is nothing really serious.
(I'd really like my new filters, who are YUV12 native, to work in avisynth)
What do you think of starting YUV12 devellopement ?
What is really needed ?

- Conversion routines :
RGB32/24->YUV12 in ConvertToYUV12 (current YUV-RGB code to adapt)
YUV12->RGB32/24 in ConvertToRGB (current YUV-RGB code to adapt)
YUY2->YUV12 in ConvertToYUV12 (really trivial. deinterleave)
YUV12->YUY2 in ConvertToYUY2 (really trivial. interleave)

- YUV12 archictecture
Avisynth would provide Y,U and V Pointers and Pitchs, when using YUV12.

- YUY2->YUV12 code adaptation.
Porting YUY2 code to YUV12 may not be very easy, because of the different archictecture, but it's not too hard to adapt C code.
For ASM-optimised code, 2D processing is easier (in my point of view) to code in YUV12 than in YUY2. especially YUY2 1D horizontal processing, who is really a pain.

Of course, if the avisynth team need coders, i'm volunteer

Cheers,
MarcFD

[Nic]

Id highly recommned the XviD color conversion routines if you want to & from YV12

-Nic

[sh0dan]

@marc:
I've created an Avisynth_2_1 branch in the CVS, but I haven't put anything up yet, since I still have to rewrite audio.cpp for the new audio handling (it doesn't quite compile yet). I'll notify you all here, when a proposal is finished.
YV12 is definately a 2.1 priority IMO - but my time is limited ATM.
YUY2->YV12 seems trivial, but others have done it elsewhere.

@nic:
Yes - the Xvid routines would be preferable, if they are portable (nasm->masm) without too many problems - they are VERY well-written.

[vlad59]

I never read avisynth core sources before today so please forgive me if I say dumb things

My thoughts about adding YV12 colorspace support in avisynth :
- To make things easier we could keep only one VideoFrameBuffer per VideoFrame and have 3 new pointers for Y, U and V plane. offset and pitch don't need to be duplicated because U and V pitch and offset can be calculated with Y pitch and offset. For width and height, it's the same.

- The first avisynth function to port is Avisource (to make test) and it should be better for now to only decode in YV12 if the parameter is set.

I'll make some tests. I have a lot to learn in vfw programming so it can be long

[sh0dan]

Originally posted by vlad59

- To make things easier we could keep only one VideoFrameBuffer per VideoFrame and have 3 new pointers for Y, U and V plane. offset and pitch don't need to be duplicated because U and V pitch and offset can be calculated with Y pitch and offset. For width and height, it's the same.

U-V pitch should be the same, but should not be calculated from Y, since pitch also is dependant on other things, as memoryalignment, etc. If Y-pitch is is 8byte-aligned, UV-pitch will only be 4byte aligned, which will result in penalties. Solutions could be:

- GetPitch() returns Ypitch + (UVpitch<<16);
Hacky, but it would work.
- Implement GetUVPitch();
Probably the most reliable.

Regarding data, can it not be assumed that
U = *(data + height* Ypitch)
V = *(V + UVpitch*(height/2))

This would how it would be returned from vfw (and hopefully any MPEG-decoder) - or am I mistaking?

- The first avisynth function to port is Avisource (to make test) and it should be better for now to only decode in YV12 if the parameter is set.

Sounds like a great idea! I'll also get on it - we should get as many opinions on this as possible, since it's a very big change, that many people will have to work with it!

[Marc FD]

Originally posted by sh0dan

Regarding data, can it not be assumed that
U = *(data + height* Ypitch)
V = *(V + UVpitch*(height/2))

This would how it would be returned from vfw (and hopefully any MPEG-decoder) - or am I mistaking?

no, i don't think so, because of alignement too.
but it might be correct. i'll check that.

BTW, what do you think of :
overload GetReadPtr()/GetWritePtr()
with GetReadPtr(int channel) / GetWritePtr(int channel)
in avisynth.h, add a enum {Ychannel, Uchannel, Vchannel} or 3 define.

GetReadPtr(Uchannel) would give you a pointer to the U block, ect...

[vlad59]

@MarcFD

Yes your idea to overload GetReadPtr()/GetWritePtr() is great and with that we could imagine to have pointer to H/S/L plane one day.

About Sh0dan formulas :
U = *(data + height* Ypitch)
V = *(V + UVpitch*(height/2))

Just to be sure I'm not wrong
Width and height must be mod 4 so we really must care about memory alignement and make 16bits aligned buffer when possible to allow further SSE2 optimizations. Now I think (But I can be wrong, I haven't read all avisynth core code) alignement is done by the pitch wich will always be mod 8 or mod 16.
So this formula should be correct.
U = *(data + height* Ypitch)
V = *(V + UVpitch*(height/2))

For vfw I have no idea of how it work.

[trbarry]

I maybe haven't thought this through properly but you might get more compatibility with less Avisynth code changes if you just arranged the buffer like:


yyyy ... yyyy ...
yyyy ... yyyy
yyyy ... yyyy
yyyy ... yyyy
uuuu ... vvvv
uuuu ... vvvv ...

That is, only one pointer to the data and only one pitch and line count, like now. I believe 4:2:0 is only valid for even pixel widths and heights so it would be easy to calculate where the U and V blocks begin.

But I am thinking only a way to handle 4:2:0 planar data. I don't know what a YV12 file format looks like so maybe this isn't compatible.

Also be aware that a conversion from 4:2:0 to 4:2:2 (or RGB) needs to now whether it is a frame or field picture, that is, whether it was interlaced. Ignoring this is one of the sources of chroma delay causing so much flap with DVD players and decoders recently. So that info has to be dragged along with the data as long as it stays in 4:2:0 format.

- Tom

edit: Oops, forgot about the multiple of 8 thing mentioned above for performance. So it's better to waste the space and use:

yyyy...
uu...
vv...

[Marc FD]

i just etudied some cases :

- In XviD :
the IMAGE struct holds 3 pointers : y,u and v
stride is used for Y and stride>>1 or stride/2 for U and V

- In DVD2AVI MPEG-2 Decoder (MPEG2Dec)
it uses an array of 3 pointers. ex : src[0] is Y src[1] is U and src[2]
stride is used for Y and stride2 for UV. but stride2 is everywhere computed this way : stride2 = stride/2

But in these cases it's an internal architecture.
( i see avisynth more like a video processing platform )
so avisynth should handle YUV12 then cleanest way.

even 100 bytes spoiled for each frame is really ridiculous regarding the quantity of data we are processing, no ?

[-h]

even 100 bytes spoiled for each frame is really ridiculous regarding the quantity of data we are processing, no ?

Most applications will pass YV12 data to you in the form (Yplane with Ystride), (Uplane with Ystride/2), (Vplane with Ystride/2). You can enforce some larger U/V strides internally, it'll just mean a bunch of extra memcpy()'s per every frame and a cache hit by doing so.

-h

[Marc FD]

of course. i never said the opposite

the idea is, avisynth should use Ypitch/Ywidth/Yheight and UVpitch/UVwidth/UVheight, like ffdshow does !!
+ 3 pointers for Y,U and V.

with this toolbox, avisynth filters could be adpated to any configuration :
- Y, U and V aligned (classic) or Y, U and V non aligned !
- UVstride == Ystride/2 or UVstride != Ystride/2
or any other possibility.
the *source filter will fill this with the specific info.
then the filter will work the cleanest way possible.
when we alloc a new image, then we could align the Y,U and V blocks for optimal performance. All blocks should be 16 bytes aligned when
possible in avisynth.

BTW, a usefull debuging function to add in avisynth i had : KillCache()

PVideoFrame KillCache::GetFrame(n,IScriptEnv env)
{
  PVideoFrame src = child->GetFrame(n);
  PVideoFrame dst = env->NewVideoFrame(vi);
  env->BitBlt(<src to dst>); // (i'm too lazy to write the whole code)
  return dst;
}

this way, debugging of filters is much easier :

*source().Crop,trim,resize,ect....
KillCache()
SuspiciousOrNotStableFilter() #mainly mine ;)

because the blocks are as thin as they can be, any slight memory acces violation will be sanctionned by an exeption.
and a frame processed through KillCache should always be aligned.
adding a align check in end of it seems a good idea.

and, while i'm here, i've a (silly?) idea for avisynth 2.1 :
adding a AVSvalue in the VideoFrame Struct.
any filter could acces it by GetFrameInfo.
this way we can do cheap filter communication.
of course it's only to give some usefull hints (decomb patterns, scenes-changes, this kind of info...) not to store big masks.
maybe an extention of AVSvalue (i think of a Ptr type) could help to handle masks gracefully.

What do you think ?

[WarpEnterprises]

Could somebody please sum up the advantage of a third color format being used in AviSynth?

[trbarry]

Could somebody please sum up the advantage of a third color format being used in AviSynth?

Don't know about others but my own reasons are fairly simple.

1) Color conversions are potentially lossy, and slow.

2) Most of my video input comes from DVD's or HDTV and both of those are already in YV12 format.

3) Most of my video output goes to MPEG2 or MPEG4 (Divx or Xvid) also in YV12 format.

4) Some algorithms run faster or are easier to write when working with planar 4:2:0 data.

- Tom

[Guest]

4) Some algorithms run faster or are easier to write when working with planar 4:2:0 data.

Indulging in understatement, eh Tom? YUY2 is a giant pain in the you-know-what. Let's go planar!

[vlad59]

As -h said, I think we must stay simple and choose an already working solution. Extra optimizations can come later.

[sh0dan]

Fast response (I'll do a more detailed later)

- YV12 is easier and faster to process than YUY2, since it's planar and is source of MPEG2 decoding, and destination format of MPEG4 anyway. And it's faster because it has half the UV-information of YUY2.

- Pitch/Stride is mostly (by default) 8-byte-aligned , but that's easy to change. AVI always delivered in 4-byte-aligned frames. Perhaps the default alignment should be autodetected, and set to 16 if a P4 is detected. It seems like most apps use UV-pitch=Ypitch/2, which makes sense. but all frames are BitBlitted on output anyway, so output wouldn't mind if we have two different pitches.
Both pros and cons.
Two pitches = A little faster, less compatible (when converting filters from other apps).
One pitch = A little slower, easier integration of existing filters (ffdshow for instance).


- Seperate YUV pointers. Not IMO! They are easy to calculate, and seperate pointers, where they point to different memory-locations would involve copying the data when YV12 is delivered, and general confusion. Seperate pointers would also make YV12 too much a special case.
We could make helper functions that return the UV-pointers, but they should always be placed after Y with the correct pitch.

[trbarry]

Perhaps the default alignment should be autodetected, and set to 16 if a P4 is detected.

I think that if the default alignment in Avisynth was always 16 bytes then everything would run faster even on Athlon's and P3's, because data would more often be in a single cache line.

With memory sizes the way they are these days nobody gets enough small memory chunks that it is even worth while worrying about wasting a few bytes to get the alignment. So we should maybe just align everything in Avisynth to at least 16 bytes. Life would be simpler.

- Tom

[Marc FD]

Originally posted by sh0dan

One pitch = A little slower, easier integration of existing filters (ffdshow for instance).

ffdsow uses 2 pitches, i pointed it out some posts before.

- Seperate YUV pointers. Not IMO! They are easy to calculate, and seperate pointers, where they point to different memory-locations would involve copying the data when YV12 is delivered, and general confusion. Seperate pointers would also make YV12 too much a special case.
We could make helper functions that return the UV-pointers, but they should always be placed after Y with the correct pitch.

maybe i'm wrong , but :

1) The decoder i know the much in _deep_ detail is the MPEG2 decoder of DVD2AVI :
I'm sure of one thing : it uses 3 pointers, and it _NEEDS_ 3 pointers because each block as it own malloc (assuming they are contiguous is dangerous) and the output is done in the "assembleframe" function (the name speaks for itself) who just assemble the 3 blocks in one YUY2 or RGB32 block.

2) ffdshow filters :
3ptrs, 2 heigths, 2 widths, 2 pitchs.
see "TimgFilter"
it uses XviD (see below) for assembling/csp convertions.

3) XviD internally :
3ptrs, 1 heigth, 1 width, 2 pitchs.
see "IMAGE", "FRAME" and "DEC_FRAME"
it assembles them in "image_output",
depending of the colospace

4) ffdshow using XviD :
see "TmovieSourceXviD::getframe"
ffdshow uses XVID_CSP_USER to avoid any
assembling. this way, ffdshow is able to directly use the
internal image of XviD.
colorspace convertion is simply ignored.

5) in Nic's PP :
3 ptrs, 1 heigth, 1 width, 1 pitch.

Seperate pointers would also make YV12 too much a special case.

But IT IS a special case. RGB32,RGB24,YUY2 are NOT planar formats.
to handle a planar format, you need OF COURSE a new archictecture.

i think it's why VitualDub/Avisynth are currently not using it.

[sh0dan]

I think we're getting to the point, where I think it's arguments for the sake of arguments.

The compromise:

- 2 pitches. int GetUVPitch() Gets implemented into VideoFrame.
- VideoFrameBuffer hold ONE (1) pointer to data, as always, but gets a new function to return other pointers. Implemented as const BYTE* GetChannelReadPtr(byte channel) const { return data+(stuff); }. That way full compatibility is retained.

Any objections?

[Marc FD]

Originally posted by sh0dan
- 2 pitches. int GetUVPitch() Gets implemented into VideoFrame.
- VideoFrameBuffer hold ONE (1) pointer to data, as always, but gets a new function to return other pointers. Implemented as const BYTE* GetChannelReadPtr(byte channel) const { return data+(stuff); }. That way full compatibility is retained.
[/B]

what is (stuff) ??

BTW, i absolutly knows nothing about avisynth internals, but if the 3 blocks aren't contiguous, how do you free/copy the data