overallocating bits for "archival" video?

[fibbingbear]

I recently got into video encoding. If I had infinite storage and infinite bandwidth, I would just leave everything as uncompressed AVI or huffyuv. Unfortunately, the real world is not so nice :-p

Although I can achieve transparency with x264 around 1000-1500kbps, I'm wondering if I should overallocate for "archival" purposes. The idea is that, sometime down the road, the format might be obsolete and the x264 codec no longer maintained / binaries built for Windows 20XX. I know that transcoding is generally frowned upon, even if you transcode at transparency settings, since artifacts will appear... however, if the codec becomes obsolete, I may have no choice.

I've done some audio transcoding experiments at transparency, and you can hear artifacts after a few transcodes, unless you overallocate. Not that audio is the same as video, but it does provide a guideline.

So my question to the encoding experts on this forum: is it better to say, "lossy is lossy, I'll stick with ~1500kbps and take my chances with the future" or to say "I'll overallocate to provide some buffer, even though it'll increase the amount of storage I use, but decrease the chance of artifacts if I have to transcode."

Btw, I looked around to find similar posts and couldn't find any. Sorry if this question has already been answered :-(

[Blue_MiSfit]

It all depends.

If you want to maintain everything, don't transcode. Keep the source material, be it DV, MPEG-2, whatever.

If you want to maintain most - compress with CRF at a low value - 18 or so.

Don't get locked into assuming 1500kbit is enough for any SD source. It may be overkill for some, and totally inadequate for others. Bitrate != quality. It's all subjective, and highly dependent on the source.

~MiSfit

[fibbingbear]

Hey Blue Misfit. Thanks for the info on CRF / quality based encoding --- I had completely overlooked that. Not to mention the whole "bitrate != quality".

I did a bit of research on it on doom9 and the Internet, and I had a few questions.

1.) Zep suggests to use -2,-2 for the deblocking in CRF:
http://forum.doom9.org/archive/index.php/t-111982.html

On Sharktooth's custom profiles thread, he mentions to use 0,0 for movies and 1,1 through 3,3 for anime/cartoon:
http://forum.doom9.org/showthread.php?t=101813

And finally, in MeGUI itself, it defaults to -2,-1 for the CRF profile.

Why such a disagreement between the three sources? I understand some of it is subjective, but it seems like there isn't even agreement on what's an appropriate range.

2.) Let's say I'm encoding anime. I do CRF at 18 and find out that, for this particular anime, it averages 900kbps over the resulting file. Should I ditch this file and go back to run the specialized 2-pass anime x264 profile, at this bitrate, over the original source? Since the anime profile is (presumably) tweaked for anime, it seems like it should capture more information at the same bitrate.

The mplayer forums had a lengthy discussion about this, but it didn't seem to me that there was a concensus opinion. Some people implied that if you trust the CRF bitrate, no need to encode a second time. Other people said that 2pass is better than CRF and it's not bad to use CRF to get an idea of your target bitrate. The relevant forum threads are:
http://lists.mplayerhq.hu/pipermail/mplayer-users/2007-January/065203.html
http://lists.mplayerhq.hu/pipermail/mplayer-users/2007-January/065236.html
http://lists.mplayerhq.hu/pipermail/mplayer-users/2007-January/065237.html

3.) What's the likely chance that, sometime in the future, codecs like x264 will no longer be supported / maintained / etc? Before I start encoding with it, it'll be nice to know how long it'll live... I don't want to make the mistake of picking a "betamax" format that'll need to be transcoded to something else.

Thanks for the help

[Dark Shikari]

Hey Blue Misfit. Thanks for the info on CRF / quality based encoding --- I had completely overlooked that. Not to mention the whole "bitrate != quality".

I did a bit of research on it on doom9 and the Internet, and I had a few questions.

1.) Zep suggests to use -2,-2 for the deblocking in CRF:
http://forum.doom9.org/archive/index.php/t-111982.html

On Sharktooth's custom profiles thread, he mentions to use 0,0 for movies and 1,1 through 3,3 for anime/cartoon:
http://forum.doom9.org/showthread.php?t=101813

And finally, in MeGUI itself, it defaults to -2,-1 for the CRF profile.

Why such a disagreement between the three sources? I understand some of it is subjective, but it seems like there isn't even agreement on what's an appropriate range.There really hasn't been that much testing of deblocking. I'd basically say to put it higher for anime and lower for sources with very fine detail. But remember that deblocking is weaker anyways at lower quantizers!
2.) Let's say I'm encoding anime. I do CRF at 18 and find out that, for this particular anime, it averages 900kbps over the resulting file. Should I ditch this file and go back to run the specialized anime x264 profile, at this bitrate, over the original source? Since the anime profile is (presumably) tweaked for anime, it seems like it should capture more information at the same bitrate.The only real form of "tweaking for anime" there is consists of using lots of reference frames. You can "tweak for anime" just as well using CRF.
The mplayer forums had a lengthy discussion about this, but it didn't seem to me that there was a concensus opinion. Some people implied that if you trust the CRF bitrate, no need to encode a second time. Other people said that 2pass is better than CRF and it's not bad to use CRF to get an idea of your target bitrate. The relevant forum threads are:
http://lists.mplayerhq.hu/pipermail/mplayer-users/2007-January/065203.html
http://lists.mplayerhq.hu/pipermail/mplayer-users/2007-January/065236.html
http://lists.mplayerhq.hu/pipermail/mplayer-users/2007-January/065237.html2pass is not measurably better than CRF, in general.
3.) What's the likely chance that, sometime in the future, codecs like x264 will no longer be supported / maintained / etc? Before I start encoding with it, it'll be nice to know how long it'll live... I don't want to make the mistake of picking a "betamax" format that'll need to be transcoded to something else.x264 creates H.264/AVC standard output. This will likely be the main standard for at least 10 years to come, and will be decodable for 20 or more (given that MPEG-2 is still popular today).

[fibbingbear]

Hey Dark Shikari. Thanks for the information!

You mention that 2-pass is generally not better than CRF. Are there rare occasions when 2-pass would produce a better quality video than CRF (assuming the resulting files are the same size)?

[Dark Shikari]

Hey Dark Shikari. Thanks for the information!

You mention that 2-pass is generally not better than CRF. Are there rare occasions when 2-pass would produce a better quality video than CRF (assuming the resulting files are the same size)?I've seen CRF be better and 2pass be better--but the difference is usually measurable only numerically, not visually.

[Shinigami-Sama]

I've seen CRF be better and 2pass be better--but the difference is usually measurable only numerically, not visually.

quoted for truth

in general I'd say QRF is the better choice, we have super cheap DVD media now, so it doesn't really matter if your episodes are +-75megs of each other, movies a little bigger in flux, but have a much nicer quality flow in the long run.

also
if you really want to try and hit a specific size ddog has come up with a nice way to run some eval passes on the material to guess its size

[fibbingbear]

Thanks for the info Shinigami-Sama.

I've been searching around and finding a lot of posts about archival lossy video. There were a few threads that had a lot of discussion, but no resolution.

To re-iterate, what I mean by "archival lossy video" is, "transparent and can survive a decent number of transcodings at equally high quality." Basically, I'd like to encode this video once and not worry if the format becomes obsolete (i.e., if the format does, I can transcode to another format and not see artifacts as long as the transcodes are infrequent), or if I need to transcode for a new device (i.e., if the iTouch of the year 2050 only supports futuristic codecs).

For example, ogg / mp3 is typically transparent around -q5 and -V2 (corresponds to ~160kbps and 192kbps on most music), but if you encode back and forth a few times, you can hear artifacts. If you encode at -q8 and -V0 (about 256kbps for both), it can remain transparent for quite a few iterations.

Some questions:
1.) At high quality, are any of the modern codecs the right choice? I've seen debates on these forums about h.264 having problems with grain/dust, even at high bitrates. I saw some people advocating xvid and even mpeg2, arguing that at high bitrates, everything looks the same so may as well go for what's fast (and doesn't over-compress).

2.) Is encoding at ultra high quality/bitrates even right? I'm capturing off of an s-video cable, so the signal is not perfect to begin with. I saw some people on the forums advocating lots of post-processing (temporal smoothers and such) and then encoding at a high rate --- the smoothers will cause the file size to decrease. I also saw some people argued that quality loss is quality loss, so using Q2 for Xvid or CRF18 is fine, and anything less (like Q1 or CRF14 and below) is purely wasteful.

Thanks for the help. :thanks:

[Shinigami-Sama]

Thanks for the info Shinigami-Sama.

I've been searching around and finding a lot of posts about archival lossy video. There were a few threads that had a lot of discussion, but no resolution.

To re-iterate, what I mean by "archival lossy video" is, "transparent and can survive a decent number of transcodings at equally high quality." Basically, I'd like to encode this video once and not worry if the format becomes obsolete (i.e., if the format does, I can transcode to another format and not see artifacts as long as the transcodes are infrequent), or if I need to transcode for a new device (i.e., if the iTouch of the year 2050 only supports futuristic codecs).

For example, ogg / mp3 is typically transparent around -q5 and -V2 (corresponds to ~160kbps and 192kbps on most music), but if you encode back and forth a few times, you can hear artifacts. If you encode at -q8 and -V0 (about 256kbps for both), it can remain transparent for quite a few iterations.

Some questions:
1.) At high quality, are any of the modern codecs the right choice? I've seen debates on these forums about h.264 having problems with grain/dust, even at high bitrates. I saw some people advocating xvid and even mpeg2, arguing that at high bitrates, everything looks the same so may as well go for what's fast (and doesn't over-compress).

2.) Is encoding at ultra high quality/bitrates even right? I'm capturing off of an s-video cable, so the signal is not perfect to begin with. I saw some people on the forums advocating lots of post-processing (temporal smoothers and such) and then encoding at a high rate --- the smoothers will cause the file size to decrease. I also saw some people argued that quality loss is quality loss, so using Q2 for Xvid or CRF18 is fine, and anything less (like Q1 or CRF14 and below) is purely wasteful.

Thanks for the help. :thanks:



how about this
keep a virtual machine image loaded with all the codecs you've used for your archive so you can easily change them back to what ever if you ever need to

and I really doubt that in the next 20 years you'll have to worry about it
by that time you'll be able to pick what ever for 3/5$ in walmart bargin bins anyways