Speed and quality of ConvertToYUY2?

[Trixter]

(I'm posting this here because I figure people who have looked at/written the source code would know best)

I have 3+ hours of footage to process; I'm using avisynth to splice together 30 different .AVIs into a single .avs to feed to CCE. My source is 0-255 RGB32. I've got two options:

1. Letting CCE do the 0-255 -> 16-235 conversion
2. Do it myself with ConvertToYUY2()

CCE is closed source; avisynth is not. What would your professional recommendation be? Should I just let CCE do the conversion, or should I ConvertToYUY2 before feeding to CCE? Which would be faster? Which would result in better conversion quality?

[IanB]

I'm biased, but ...

Avisynth's RGB to YUY2 routines are highly optimised fast MMX code that are mathematically exact, however there is some data loss intrinsic in going to 8 bit YUV from 8 bit RGB.

Theoretically going from 8 bit RGB to 10 Bit YUV would avoid this loss, I don't know if CCE internal routines do a 10 bit conversion. Ask in the CCE forum. TMpgEnc for example works directly with 8 bit RGB data to produce 10 Bit mpeg2 streams and is slightly slower because of this.

A data stream once it has been converted to 8 bit YUV should suffer no further losses in precision as long as the YUV->RGB and RGB->YUV routines are mathematically exact. i.e. all the loss of precision happens on the very first RGB to YUV conversion.

So if your data has ever been 8 bit YUV in it's life then there is no issue. Using Avisynth routines will be as fast as possible and there will be no futher data loss.

Note most AVI codecs actually store data internally as 8 bit YUV and convert to RGB32 for output.

[Trixter]

Thanks very much for the explanation. And yet, I couldn't leave well enough alone, so I did a speed/quality test with both ConvertToYUY2() conversion and CCE conversion, and also with/without NTFS file compression (my RGB32 raw source compresses quite well with it). The results were surprising:

compression, avisynth conversion: 3m16s
raw, avisynth conversion: 3m19s
compression, CCE conversion: 3m27s
raw, compression, CCE conversion: 3m24s

Fastest: NTFS file compression+avisynth conversion

Highest quality: CCE conversion

When CCE did the colorspace conversion, it must have had the benefit of the 10-bit colorspace (I intentionally set DC bits to 10 for the test), as the following pictures show less banding in the CCE conversion:



...than in the ConvertToYUY2() conversion:



The CCE conversion has 221 levels of luma (and no chroma), which is damn near perfect given the 0-255->16-235 range adjustment. The ConvertToYUY2() conversion, however, is 330 different colors, which I'm at a loss to explain except to say that it's due to rounding or truncation errors in the calculations.

[Trixter]

Sorry to bring this thread back up, but I'm now in a situation where I need to do processing with avisynth and the conversion is most definitely losing precision in the original image. If I'm doing something wrong, please point it out to me, as the resulting loss is actually visible onscreen.

I have a UYUV source that is converted to RGB32 when I load it into our broadcast environment. In a waveform monitor, it looks like this:



After the following simple pal->ntsc transcode script:

AVISource("\assets\mc2\pal\arte.avi")
ConvertToYUY2

TDeint(order=1,mode=1,type=0,sharp=false,AP=50,APtype=0) # Use TDeint to turn 25i into 50p
Lanczos4Resize(720, 480)
ConvertFPS(59.94,zone=80)

SeparateFields()
SelectEvery(4, 0, 3) # 4,1,2 is for odd field first; 4,0,3 is for even field first
weave
AddBorders(0,2,0,4) # we have to pad to 486 lines for use in our broadcast environment

...the same section now has several values quantized, like this:



I've got to eliminate this loss somehow. Is the loss due to some math rounding error somewhere, or maybe the colorspace conversion matrix? Any help/suggestions are appreciated...

[scharfis_brain]

use AVISynths function histogram() setp by step after each filter to see which one is causing the quantisation

[Trixter]

I can't do that since histogram() requires YUY2, and it's convertoyuy2 itself that is under suspicion. Because of this, I can't take measurements before every step of the process.

[scharfis_brain]

you may also try the different available matrices for colorspace conversions.

see the AVISynth documentation for a more detailed description.

[Trixter]

Originally Posted by scharfis_brain

you may also try the different available matrices for colorspace conversions.

While there's still loss, this did indeed help. I apologize for bothering everyone; I should have figured that out before posting.

In any case, I wrapped the stuff that needed YUY2 thusly:

ConvertToYUY2(matrix="PC.601",interlaced=true)

TDeint(order=1,mode=1,type=0,sharp=false,AP=50,APtype=0)
Lanczos4Resize(720, 480) # Resize to NTSC resolution
ConvertFPS(59.94,zone=80) # ChangeFPS(59.94) adds/drops fields; ConvertFPS(59.94) blends fields

ConvertToRGB(matrix="PC.601",interlaced=true)

The PC.601 matrix does not compress/expand the range arbitrarily so I have more precision left over after the process. I have to force the conversion back to RGB with the same matrix, otherwise it scales using a different matrix and I get clipped ranges.

[scharfis_brain]

if you convert between yuy2 and rgb the interlaced paramter is useless, because there is no difference in vertical resolution.

the interlaced parameter is only needed if you are converting to or from YV12.

btw.: we really need some higher quantised colorspaces

[canuckerfan]

hello, don't mean to hijack this thread, but I have a quick question. would going from yv12 yo yuy2 within cce give a more accurate conversion at the expense of speed compared to converttoyuy2() function?

[scharfis_brain]

don't ever feed encoders that cannot handle YV12 natively with YV12. This includes ProCoder, CCE and TMPGenc.

The only encoders I know of, that are handling YV12 the correct way are HcEnc and QuEnc.

[Trixter]

Originally Posted by scharfis_brain

btw.: we really need some higher quantised colorspaces

No, lower! Where's my YUV 4:4:4?

What I really need from avisynth are channel precisions higher than 8 bits per channel (ie. RGB24/32). I have some 10-bit HD footage I'd like to process with avisynth but am limited to doing it in the editor because avisynth cronks everything down to 8 bits per channel. Is such support in the planning stages?

[foxyshadis]

Check out the Avisynth 2.6 pre-alpha released a month or so back. Support for YV16 (4:2:2 planar), YV24 (4:4:4), and YV12/16/24@16. The latter naming I just came up with, hope it fits. >.> No 10 or 12 bit; I'm not sure but it'd probably just retard speed, being all unaligned.

[Trixter]

Originally Posted by foxyshadis

No 10 or 12 bit; I'm not sure but it'd probably just retard speed, being all unaligned.

Speed isn't the issue ;-) It's quality, of course. I have AVI codecs that can handle more than 8 bits per channel, and an editor that can handle it as well... just wishing avisynth would handle it too someday.

Converting 10-bit to 8-bit it doesn't sound like a big deal, but footage that is just barely noisy becomes much noisier because you drop the 2 least significant bits... smooth gradients become posterized... all that stuff. For your own test, take a 24-bit image and convert it to 18-bit and you'll see the same types of effects.

[foxyshadis]

Well, I rather confusingly tried to say that 2.6 has 16-bit colorspaces to play with, just not 10 or 12 intermediate ones. I've been waiting for them as eagerly as you. =p

[Trixter]

Ah, thanks for clearing that up. :-) Unfortunately, I'm using mvbob, mvtools, fft3dfilter, and other stuff that won't work unless it's on YUY2 so I'm still somewhat screwed -- for now.

[Wilbert]

Well, I rather confusingly tried to say that 2.6 has 16-bit colorspaces to play with, just not 10 or 12 intermediate ones.

No, the formats in 2.6 are 8 bit. 16 bit precision is planned for 3.0, but these formats are not implemented yet.

[IanB]

The precision will actually be 15 bits. SSE/MMX 16 bit word instructions favour the data being signed. And it will be a long way off, all the functions will need to have 15 bit versions written. In 3.0 the C code should be easy (template classes), the SSE/MMX will not.

[foxyshadis]

Oh really? Here I was thinking it was coming in 2.6. Oh well, what I get for not reading close enough.