yuv->rgb colorspace conversion question

[hanfrunz]

Hello,

i am just writing a new filter which emulates an Tektronix WFM600 Vectorscope and Diamonddisplay. The Vectorscope works well right now and looks pretty nice. But i have Problems with the diamond display...
I need a fast routine which converts yuv data to rgb without clipping the rgb output.

example:
y=233, u=233, v=233 --> r=420, g=126, b=464

i need these illegal values (r=420 is normaly cropped to 255) to show them in the diamond-display.

Can i use the original routines from avisynth for that purpose?
If this is possible how would i do it, because i do not understand these mmx-asm-stuff :(

btw: it is very complicated to port an avisynth filter to dscaler? because then we had an very cheap alternative to professional waveformers/vectorscopes.

thanks,
hanfrunz

[sh0dan]

The biggest problem would be that all the filters spit out 8 byte pixel color component - which of course means that values >255 cannot be represented.

What's your destination format? 16 bit RGB?

A solution could of course be to drop one bit precision, and then scale all values to x*2 afterwards, but it would still not be easy to modify the existing routines, and of course quality would suffer.


As for dscaler, it would probably need some work, as all AviSynth ASM is written for the MASM assembler, and dscaler uses NASM. What is the conversion function you need to use? Do you have the algorithm as C?

[hanfrunz]

Hi sh0dan,
the output is still yuy2. I need the values to put a pixel in the scale of the diamond display. So the conversion is only for internal use to get the coordinates of a pixel.

see here for detailed information:
ftp://ftp.tek.com/mbd/manuals/video_audio/25W_15609_0.pdf

right now i am using this very slow formula, i found somewhere, sometime. I know there are faster and better ones out there but i used this for testing, because it was there :)

r=1.164383*(y-16)+1.596027*(u-128)+.5;
g=1.164383*(y-16)-0.812968*(u-128)-0.392*(v-128)+.5;
b=1.164383*(y-16)+2.017232*(v-128)+.5;

there will be a first release soon, when it's less buggy, slow and ...
hanfrunz

[sh0dan]

Well - ditching floats would be the first thing to do ;)

int r = (int)(255.0*1.164383+0.5)*(y-16)+(int)(255.0*1.596027+0.5)*(u-128)+(127);

perhaps even:

int r = ((int)(256.0*255.0*1.164383)*(y-16)+(int)(256.0+255.0*1.596027)*(u-128)+(int)(256.0*0.5*255.0)+128)>>8;

(For better precicion - the compiler should be able to optimize all float expessions out, so you should not have any speed penalty).

(Sorry if I am completely off, but I like to have some test material when doing proper rounding) ;)

[trbarry]

As for dscaler, it would probably need some work, as all AviSynth ASM is written for the MASM assembler, and dscaler uses NASM. What is the conversion function you need to use? Do you have the algorithm as C?

DScaler mostly uses VC6 inline asm. It works only in YUY2 but accepts frames in much the same format as Avisynth. TomsMoComp shares a great amount of code between the two projects as I first wrote it in Avisynth but with the intention of running in both, which it does.

- Tom