Looking for advice on archive-ish quality encodings

[chilledinsanity]

I create videos from computer video content (RGB) and I'm looking for a method of backing them up for what I'm calling archive-ish quality. I'm NOT looking for lossless encoding. I'd rather save some more space than get every last little detail I won't notice. I still would like the detail to be extremely high however, and I'm guessing there's a lot of room for having something super high quality, but still save quite a bit of space compared to lossless encoding

I was hoping to do something with x264 storage. I've been experimenting with encoding for that and I've been very happy with the details on it. My biggest problem however, is the color loss I get from having to convert RGB content to 8 bit YUV in x264 (using convert(YV12) in Avisynth beforehand). In some areas I can't tell the difference, in other areas I really can.

I read some about 10-bit color encoding and thought this might be a good compromise for color retention / storage space. Unfortunately, I can't get it to work properly at all. I downloaded the 10-bit encoder from x264.nl for encoding and tried using the latest beta builds of CCCP or VLC Player for playback. Here are some commands I was trying:

With an uncompressed RGB AVI:
x264 --input-csp rgb --output-csp i422 --crf 0 -o test.mkv test.avi

With a 4:2:2 YUV Lagarith encoded AVI:
x264 --input-csp i422 --output-csp i422 --crf 0 -o test.mkv test.avi

In both cases, it would create the file, but the results were completely corrupted and wouldn't playback properly on anything (I could see the correct shapes, but it was an absolute mess). I'm guessing I'm doing something wrong, but I don't know what it is.

So the short version is what would people recommend I do to back up RGB content for archive purposes with minimal quality / color loss but with a decent size savings (compared to lossless)?

[Ghitulescu]

Archiving means 100% original. :)

[chilledinsanity]

Well that's why I said archive-ish. What would the proper term be for higher quality and filesize than anything someone would distribute publicly, but not a 100% copy? For example, I have a 5 minute 1280x720 video I made that I'm fine with storing at 1-2GB, but I don't want to devote some 10-20GB to it if I'm making it lossless.

[nibus]

I'm betting the 10-bit encodes are not playing back properly for you. Try installing a new beta of ffdshow and MPC-Home Cinema:
http://www.xvidvideo.ru/ffdshow-tryouts-project-x86-x64/
http://www.xvidvideo.ru/media-player-classic-home-cinema-x86-x64/

Or try using an 8-bit version of x264. It will be a more compatible format to deal with, and with a really low CRF Value you shouldn't have to worry about color banding.... so I would probably use it instead. You don't need any beta codecs to deal with 8-bit.

I think CRF 12-15 is basically identical to the source (and overkill), but I suppose you can go as low as you want. I usually use CRF 16-17 for nearly complete transparency.

[chilledinsanity]

The files still don't play using the files you recommended, and I tried SMPlayer (a frontend for Mplayer) to test it out as well and it still plays completely corrupted.

As for the CRF values, I did testing and found the detail levels very good using lower numbers, but the color remains inaccurate in all of them. It's not a matter of banding so much as the color not being quite the same as the RGB source.

[hello_hello]

Well that's why I said archive-ish. What would the proper term be for higher quality and filesize than anything someone would distribute publicly, but not a 100% copy?

As far as I'm aware, and I did look it up, there's no definition of "archiving" which mandates the archived copy be the original.
In fact, there's a definition of "archive" which describes exactly what you're doing, so I wouldn't worry too much.

http://www.thefreedictionary.com/archived
ar·chive
2. Computer Science
b. A file containing one or more files in compressed format for more efficient storage and transfer.

There you go. Even the dictionary definitions move with the times....

[hello_hello]

As for the CRF values, I did testing and found the detail levels very good using lower numbers, but the color remains inaccurate in all of them. It's not a matter of banding so much as the color not being quite the same as the RGB source.

Are you converting from HD to SD, or the other way around?

[chilledinsanity]

Are you converting from HD to SD, or the other way around? Neither, I'm converting from uncompressed original to something that is compressed, but still very high quality.

[hello_hello]

Do you have any way of knowing which colorimetry was used when creating the original uncompressed video? I assume the video you're working with is RGB?
I "think" both Xvid and x264 use R.601 when encoding RGB (but don't quote me on that) so depending on the resolution it may be your encode is displaying using the wrong colorimetry. Or possibly the wrong colorimetry was used when creating the original RGB video.
What's the resolution?

[AnonCrow]

For example, I have a 5 minute 1280x720 video I made that I'm fine with storing at 1-2GB, but I don't want to devote some 10-20GB to it if I'm making it lossless.
Various lossless codecs would actually get a 20 GB RAW file (~5 minutes @ 720P) down into 3-6 GB range , depending on the type of source.

For archive-ish purposes, when using x264, might be wiser to encode with QP rather than CRF (and set ip and pbratios to 1.0) to get mathematically constant quality instead of visually constant quality.

[chilledinsanity]

hello_hello: I'm not sure of the colorimetry. I processed it in Premiere then export that as a uncompressed RGB AVI since I don't trust Premiere to output in much else. I'm surprised it makes a difference regarding color, but the resolution is 1280x720 and the framerate is 30fps, not 29.97. I'm willing to jump through some hoops if I need to prepare my footage better, but I'm not sure what I should be doing.

AnonCrow: Well one video I created is 26 minutes long and compresses to 20GB on Lagarith, it would be nice if it didn't have to be so huge. I've been very happy with how the detail levels scale given the space in CRF, regardless, it's only the color preservation I'm concerned with. Encoding at CRF 30 or CRF 0 doesn't seem to affect that. It is sounding like for RGB I have to choose either between lossless compression or else accept the color loss using x264.

[Ghitulescu]

I'm not sure of the colorimetry. I processed it in Premiere then export that as a uncompressed RGB AVI since I don't trust Premiere to output in much else.

What is the source for your clip? Otherwise, the default colorimetry is AdobeRGB.

[Asmodian]

Wow it would be odd to see video in AdobeRGB. One would think Premiere would output video that looked normal on a standard sRGB/BT.709 or BT.601 screen since they should not be worrying about CMYK printing. :rolleyes:

Are there any tools to convert AdobeRGB to sRGB video? All the avisynth tools only have BT.709, BT.601, or SMPTE 240M, right? I guess AdobeRGB is sort of close to BT.601. I hope you can you pick the color space when exporting from Premiere?

[hello_hello]

chilledinsanity,
Basically when YCbCr video is being displayed on your monitor (i.e. when converted to RGB) it will be converted using a particular colorspace. There's basically two of them to worry about.... R.601 for standard definition and R.709 for high definition. If you're encoding YCbCr using Xvid or x264 then usually you don't have to think about it as there's no colorspace conversion taking place. Unless of course you convert from high definition to standard definition etc as then you need to convert the colors to compensate for the different colorspace used when playing back the encoded video.

If there's any conversion to RGB taking place (or in your case you've converted to RGB and now you're re-encoding it) then as long as you don't go from HD to SD etc the encode should still display using the same colors as the original (before you converted to RGB) as long as you use the same colorspace when converting to RGB and "back again" (even if it's technically the wrong one). The RGB video you created however, may not be displaying using the same colors as the original video if the wrong colorspace was used when you converted to RGB.
(There's also the AdobeRGB thing previously mentioned which I don't know much about, except for the fact it's a different colorimetry again to sRGB. Whether Premier uses AdobeRGB instead of sRGB for video by default I don't know, but I kind of doubt it).

Your description of the colors not displaying accurately when you encode sounds like it's likely to be a colorimetry issue. If it's a R.601 v R.709 problem, usually it's most obvious in reds, which tend to look a bit darker, while blues and greens look a bit brighter (I think) or of course it can be the other way around. Sometimes (depending on the colors in the video) the difference is pretty much unnoticeable. I wouldn't think you should see a difference in colors simply because you encoded using x264. I don't think "8 bit" is your problem.

To convert the colorimetry when encoding, you'll need this AVIsynth plugin: http://avisynth.org.ru/docs/english/externalfilters/colormatrix.htm
Try simply adding something like this to your script:

LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\ColorMatrix.dll")
ColorMatrix(mode="Rec.709->Rec.601")

Or you can try it the other way around:

LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\ColorMatrix.dll")
ColorMatrix(mode="Rec.601->Rec.709")

Of course the above is just something to try to see if it fixes the color problem. It could be a colorimetry issue caused by some other colorimetry conversion I'm not aware of. The million dollar question would be, are you 100% certain your uncompressed RGB AVIs are actually displaying using the original/correct colors themselves? I don't suppose you still have the original compressed video to compare them to?

Sorry that turned into a bit of an essay, but hopefully it helps point you in the right direction.

[chilledinsanity]

I'll experiment with the colimetry settings in Avisynth like you've suggested. The difference is relatively subtle in most situations, but I did notice it in screenshot comparisons.

The original footage is outputted from videogame footage from the Source engine. I don't have my original encodings anymore, but it's relatively simple for me to create a new test to compare how h264 compares against the original footage and what if any, color changes there are along the way. So I'm not 100% certain of the conversion, but that's a good point. I'll try and run some tests soon and doublecheck the results.

Thanks, these scripts give me something to work with.

[naoan]

if you're using fraps and force RGB capture, try converttoyv12(matrix="rec601"), and set this two x264 parameter : --colormatrix bt470bg --fullrange off

[chilledinsanity]

I did some more testing, this time with lots of red content to see the differences more clearly. I can say with certainty now, the quality loss occurs as soon as I try to convert the content to any other colorspace. If it get converted to YV12 or YUY2 via avisynth or selecting the option in Lagarith encoded, the quality goes down immediately. In my original clip I have a sharp contrast between red and black with clear details. Using avisynth to convert to a different colorspace, it causes reds to become more pixellated. Using a different colorspace via Lagarith encoding, they become more blurred.

Your description of the colors not displaying accurately when you encode sounds like it's likely to be a colorimetry issue. If it's a R.601 v R.709 problem, usually it's most obvious in reds, which tend to look a bit darker, while blues and greens look a bit brighter (I think) or of course it can be the other way around. Sometimes (depending on the colors in the video) the difference is pretty much unnoticeable. I wouldn't think you should see a difference in colors simply because you encoded using x264. I don't think "8 bit" is your problem. The shade of the center reds stays exactly the same, it's mainly the detail that goes all to hell. 8 bit probably isn't my problem, but getting RGB content properly stored seems to be.

Try simply adding something like this to your script: Neither of your examples worked, because in both cases it said "input must be YV12 or YUY2" when I tried to run it on my uncompressed clip. If I convert to either of those colorspaces using any method so far, there's CLEAR quality loss in red content.

The million dollar question would be, are you 100% certain your uncompressed RGB AVIs are actually displaying using the original/correct colors themselves? I don't suppose you still have the original compressed video to compare them to? I did some more tests, and yes, the uncompressed RGB AVIs look exactly like the original footage after being exported from Premiere.

if you're using fraps and force RGB capture, try converttoyv12(matrix="rec601"), and set this two x264 parameter : --colormatrix bt470bg --fullrange off It's not fraps, it's exported from the engine itself. As for the first command, I get clear detail loss in reds as soon as I enter in the script in Avisynth, I didn't attempt to do the x264 encoding afterwards.

[Asmodian]

That sounds like you have RGB24 video and don't want it going yuv420p.

x264 supports RGB input and output, try that or yuv444p (YV24 in avisynth). You also might be happy with yuv422p in which the color is only subsampled horizontally.

Edit: This command line should work:

x264 --input-csp rgb --output-csp rgb --crf 0 -o test.mkv test.avi

or with your 4:2:2 YUV Lagarith encoded AVI (I think it is yuv422p, if not try one of the other 422 spaces):

x264 --input-csp yuv422p --output-csp i422 --crf 0 -o test.mkv test.avi

[chilledinsanity]

That sounds like you have RGB24 video and don't want it going yuv420p.

x264 supports RGB input and output, try that or yuv444p (YV24 in avisynth). You also might be happy with yuv422p in which the color is only subsampled horizontally.

Edit: This command line should work:

x264 --input-csp rgb --output-csp rgb --crf 0 -o test.mkv test.avi

or with your 4:2:2 YUV Lagarith encoded AVI (I think it is yuv422p, if not try one of the other 422 spaces):

x264 --input-csp yuv422p --output-csp i422 --crf 0 -o test.mkv test.avi Yes, preserving the colorspace would be ideal, I didn't know that was a viable option. As for your commands, I actually tried them before making this post, I forgot to mention it. The results are pretty much the same as 10 bit encoding (I did these encodes on the 8 bit encoder), they're completely corrupted. Here's what happens during playback:

MPC-HC + any renderer: Grey screen with occasional garbage patterns shown

SMplayer: Same as above

VLC stable version: Entire picture can be made out and seen, but color is completely distorted, totally pink and green.

VLC latest nightly build: entirely green screen


Results are the same for the i422 conversion. i420 is the only thing that likes to work at all for me in playback.

[Asmodian]

have you tried ffdshow or lav video?

[chilledinsanity]

Yes, LAV filters produce the pink / green distorted colors (though still a playable image) like that shown in VLC when playing back a RGB colorspace x264 encoding.

LAV filters seem to clear up the more major problems I was having with 10-bit encodings interestingly enough. The colors are still off, but nothing so extreme as what I was seeing before.


EDIT:

Ugh, this is driving me crazy. Now I have two problems:

1. Same as before, I'm trying to minimize the color loss (or better yet, retain) when converting from a RGB24 source.

2. For videos I'm releasing publicly as i420 for greater compatibility, I AM seeing a color shift when doing x264 encoding even on YV12 material. There's a clear difference in the color for a red scene in an AVI that's already been processed via ConvertToYV12() versus the final MKV encoding using yv12 input and i420 output settings. I'll try hello_hello's colorimetry settings to see if that helps at all.


EDIT 2:

Well for problem #2 at least, hello_hello's solution does seem to work. ColorMatrix(mode="Rec.601->Rec.709") fixes the problem after it's been encoded to x264. This makes the colors look worse when opening the AVS up in virtualdub, but then after being encoded in x264, the end result looks closer to the original YV12 conversion. It still doesn't help me with problem #1, but I'm glad there's a solution for this part. Thanks for the help on this part.

[Asmodian]

It looks like fraps is using BGR24/32 instead of RGB. Should fix your pink / green issue.

http://www.mplayerhq.hu/DOCS/codecs-status.html

I assume any more subtle color issues are due to Rec.709 vs Rec.601.

[chilledinsanity]

It looks like fraps is using BGR24/32 instead of RGB. Should fix your pink / green issue. I'm not using fraps, like I said in my earlier post, but I can experiment with this. Besides, what I'm using is getting exported from Premiere. Unless Premiere uses BGR, it shouldn't be doing this. When I play back the original clip in MPC-HC, under "properties" it reports the colorspace as RGB, but I'll doublecheck to see if BGR makes a difference.

EDIT:

Okay, after some testing I'm getting some bizarre results. I tried changing it to BGR for the source and that DOES make it work in MPC-HC with the MadVR renderer, however now all the footage has a slightly purple-ish tint to it. It still doesn't work on VLC Player or using LAV filters. Those continue to have the pink / green output.

I also did another test where I tried converting the RGB footage to i444 instead of RGB. This DOES playback and retains all the detail the original does, but the color is off and looks slightly more orange than before.


Incidentally if it helps at all, I don't actually need the RGB x264 footage to playback properly on a player, that's just a way of me testing it. What I do need it to do is be able to use some software to convert it back to uncompressed AVI at a later date if I need to go back to it for editing. Unfortunately with the RGB footage so far, the same errors occur when I try using mencoder to convert the footage.