Loseless H264 files get corrupted in Vegas

[Echo7]

Hi,
I'm using Handbrake to encode Lossless H264 MP4 files (sources are FRAPS recordings). I've been trying for days now to open them in Vegas 11, but I can't get them to show correctly. Here's what they look like once imported:

http://img208.imageshack.us/img208/3336/vegasm.jpg

These are my video settings in Handbrake:

http://img812.imageshack.us/img812/8456/handbrakek.jpg

I noticed that when I change the quality setting to anything above 0 Vegas imports the files correctly. Of course in this case the compressions isn't lossless anymore which isn't what I want (I'm archiving game footage for later editing).

I tried 264VFW and the AVI files it generated were fine in Vegas, but they have another problem of slightly off green hues so I can't use them. Besides from searching I understand that AVI with H264 is not recommended because of compatibility problems.

Any ideas?
Thanks!

[LoRd_MuldeR]

The "lossless" mode of x264 uses Predictive Lossless Coding, which is available only in the "High 4:4:4" profile of H.264 - even when the colorspace used is not 4:4:4.

Consequently you will need a H.264 decoder that supports the "High 4:4:4" profile, such as an up-to-date libavcodec (e.g. ffdshow-tryouts or FFMS2). I suspect this does not apply Vegas' decoder :(

(I remember that you'll see similar artifacts with lossless H.264 using older CoreAVC versions)

[Audionut]

Vegas doesn't support x264 lossless. You're better off using huffy or something else.

[Echo7]

All right thank you, so I guess there is no choice but to use use a different lossless codec for editing in Vegas. Too bad because lossless H264 has a much better compression ratio than other codecs I tried. Oh well... :)

[vivan]

You can create dummy AVI with avisynth script and open it in Vegas... It will work, but H.264 is too slow for comfortable video editing.

[Manao]

Try to encode without 8x8 DCT first, since x264 is not compliant in lossless when 8x8 DCT is enabled. If the error in Vegas is still present, then Vegas doesn't support high 4:4:4 predictive, just plain old (and deprecated) high 4:4:4, and you're out of luck (unless you want to use a really old version of x264).

[Audionut]

Have you tried opening the fraps encode in vegas. I can't recall, but I thought vegas supported it.

[Warperus]

Vegas works with fraps encodes just fine. There is one big feature you have to remember though - fraps decodes to computer levels, while 8-bit projects in Vegas are mixture of studio levels and computer levels. Default preview in Vegas shows in computer levels, while vegas mpeg2 and avc codecs as well as x264vfw expect studio levels on timeline, so final picture is not always what you see in preview.

By the way, fraps reencodes in x264 face the same problem. As far as I remember, x264 (without avs scripts) uses inappropriate color conversion for fraps. You can use rgb+fullrange mode, but it might be problematic for later playback.

Default settings of x264 are hardly compatible with comfort editing in Vegas if you use straight approach. Sometimes they don't work at all, sometimes they produce great lags in motions, sometimes picture is corrupted. May be BD compliant streams are more vegas-friendly, but it's generally easier to avoid x264 input altogether.
I've heard of few lossless codecs popular in vegas - Lagarith, HuffYUV. Intermediate files can be as well in uncompressed avi (RGB, RGBA), SonyYUV.
But, more than "true lossless" variants Vegas users like visually lossless codecs. For example, geniune Sony MXF, XDCAM or CineForm codecs.