HD-DVD, Blue-Ray, the old jitters...

[ukb007]

Hi, all.

Doom9 the great Roamer and chronicler of impending dooms in 10/29's newssheet informs us about the First Blood in the New DVD War (http://www.businessweek.com/bwdaily/dnflash/oct2004/nf20041027_5297_db002.htm). It gives me the jitters.

1. I had an exciting movie-collection in VHS. My wife has - I don't know when - unobtrusively freed space by getting rid of them ('good riddance' she said)

2. My VCD collection - she is now eying the dusty shelves maliciously.

3. My DivX/XviD collection - getting less and less visited lately.

4. My DVD collection - in pristine and sparkling condition as of now - that's what I am having the jitters about.

Okay, the VHS/Betamax fiasco is about to be re-enacted, now in a more epic scale perhaps. But what will happen to my dear movie collections? The phrase 'Collection of a Lifetime' has no meaning any longer...in my lifetime I have already made several collections of a lifetime.

Yeah, progress. How inevitable. How thrilling. How exciting. And, imagine, how sickening.

But, again, perhaps we will get another exciting compression program, more efficient that the current XviD that will compress the HD-DVD/BlueRay movies into one DVD-5 or DVD-9. The idea cheers me up. Another bout of exciting activities - encoding ...encoding ... encoding...

Until something else turns up.

Regards.

[Sycho]

good luck trying to crack bluray and hddvd, plus both have the abilty to use mpeg4 based codecs so encoding to divx or xvid will not gain much

[Mug Funky]

hmm... that's interesting. i wonder when we hit the point of diminishing returns in the lossy compression game?

of course, if HD-DVD is capable of mpeg-4 AVC, there's not much we can do to get better compression in the near future.

but codecs like Snow and Dirac seem interesting.

CPU power aside, i wonder just how efficient lossy compression can get? is there a theoretical limit, and if so, are we anywhere near it, or are we depressingly close to it?

[Neo Neko]

There is an absolute limit. We are currently nowhere near it though. :P We are limited by the power of our processors and the lack of complexity in our algorithms. I think with current production technology we are reaching a point of diminishing returns. But technology being currently researched will continue to push those boundaries.

Think if we could completely simulate every nuance of the human voice. Once that is done it would be a simple process to create a set of instructions to nearly losslessly recreate it. Think MIDI only far more complex. It is not compression as we traditionally view it. More like 3d gaming but eventually the two will grow in similarity. ;)

[unmei]

I don't expect a big problem with backing up Blue-Ray disks
There is maybe 4 to 5 times more space on them than DVD9, but most movies i have aren't filling that at all ..there a plenty of DVD9s with 5 or 6 GB filled only ..so for one i don't expect 30GB movies as standard too soon.
But even if so, the movies don't get longer, just a higher resolution and i for my own use don't see the point in backing up in a higher resolution than my TFT has..
But of course others might want to keep the full resolution then, but how long do you think it will take from a non-negligable part of the movies being that big to the customer Blue-Ray burners to appear?
And finally, h264 is really not the end of all wisdom. It is a standard that has been worked on for a long time. During that time research on other compression methods, also beside wavelets, has not stopped.

[ukb007]

A 30 GB movie will offer something extra only in terms of resolution. Currently the celluloid has the highest picture-quality, I guess. I can't think of any way that can be surpassed, at least in the forseeably near future. Even if that's the limit, we still have a long way to go.

I imagine, though, that if 30 GB movies are available that offer picture quality six times better than the 5 GB ones, the TFT resolution will hardly be a hindrance; it will have to go up simultaneously, or something else will take its place. Progress occurs on many fronts at the same time, and often advance on one spurs growth on others.

One thing, however, is certain. Soon (maybe sooner that I can imagine at this moment) it will be goodbye to my DVD collection. God, the vinyle record...the VHS cassette...the VCD...the LaserDisc...the DVD - the iterations are taking less and less time.

Oh yeah. The future is an exciting place. I know, 'cause I have seen it. I come a long way from the past.

Regards.

[virus]

Originally posted by Neo Neko
There is an absolute limit. We are currently nowhere near it though.
I'm a bit less optimist on that, at least for video encoding. There is an absolute limit, but we are relatively close to it with H.264 I think.

Take an excellent AVC High profile implementation, one that can exploit all the features of the standard to the maximum, or nearly: well, I don't think we will be able to improve too much over that "perfect AVC" codec. There won't be a new standard in the future that provides the same perceptual quality as H.264 at ~1/3 bitrate, like H.264 does with MPEG-2. Never, not even using wavelets.

Personally I think that a bitrate reduction of 20-30% over H.264 High profile is possible (with a huge jump in complexity, though), and maybe even a bit more on low bitrate using wavelets (say 40-50%). Each new standard will add less and less improvements over the old ones, and many years will be needed to develop and refine the new tools. Bottom line: don't hold your breath for breakthrough innovations in this field :)

cheers
virus

[guada]

Hello everyone,

I have a question
what it is the "Old Jitters".
In the domain of the audio, I know but there I don't see.
Thanks.

[Trahald]

While this is a general statement... watch your 'nevers' they are just bait to a hungry technician ;)

The growing size of media may not be a big push to expand compression knowledge. but the bitrate starved area of streamed images (terrestria*tv&portables*/inet/sat/cable) will push the envelope. some new thing that hasnt been thought of yet and/or tweaking of something that exists. you end up like the people who had old posts of themselves saying consumer affordable dual layer dvd recordables are impossible.

[guada]

I didn't understand everything, but thank you for you commentary.
English is not my language of predilection, but I benefit to inform me in the world of the video.
To soon ;)

[virus]

Originally posted by Trahald
you end up like the people who had old posts of themselves saying consumer affordable dual layer dvd recordables are impossible.
There's a subtle but fundamental difference that makes this analogy sound like "apples vs. oranges" :)

Nobody has never mathematically proven that dual layer DVDs were impossible... but someone did mathematically prove that there's a limit in compression which cannot be overcome, no matter what you do or how hungry you are. :) We're just trying to approach that limit using advanced codecs; it's the practical way of computing it, since computing the limit using math formulas would be almost impossible.

For video encoding, right now I don't think that we are so far from it...
(BTW that limit has a name, it's called "entropy rate" - it's more than 50 years old but still pretty alive and well :D)

virus

[RedBeard]

My only statement is while blueray and HDDVD will be gobbled up by the high end users, I dont think the general public will be quick to jump on the bandwagon after investing so much in dvd the "ultimate" solution. and without mass support through purchases I dont think we will see the end of the dvd collections.

HOWEVER, I do think the technology will be used to make smaller, more portable disks, imagine a CVD "compact versatile disk" that will hold 5 movies the size of a mini disk or the Movie stick like the usb pen drives that holds 10 movies to be watched on any computer or standaloe player with the "New" usb 4.o port

[akupenguin]

Originally posted by virus
(BTW that limit has a name, it's called "entropy rate")
Entropy is the absolute limit for lossless compression. But entropy is hard to determine. While CABAC is ideal for its probability model, it's nowhere near the best possible. Even if one were allowed only to change the entropy coder, I would estimate at least a 1.5x improvement over H.264. For example, PAQAR really is that much better than H.264's order-0 CABAC. It just takes 100s of MB of RAM and runs at ~5kbps on today's CPUs. (Hmmm... while PAQAR is unfeasible, what would happen if I ripped out CABAC and stuck in 7zip? It can easily decompress video bitrates in realtime, though encoding would be slow.)

But we're talking of lossy compression, and there are lots of areas to improve here. With no constraints on complexity of algorithm or speed, the ideal lossy compression would be just the script, which an AI directs and renders for you in realtime.
More realistically, once we develop a decent psychovisual model, I can image some sort of texture library, like a MIDI/MOD for video.
And my aesthetic sense tells me that there just has to be a better way than B-frames and block-based motion. e.g. even H.264 with an ideal encoder falls far short of compressing a pan as a single image plus a motion path.

[virus]

Originally posted by akupenguin
Entropy is the absolute limit for lossless compression.
It is a limit for lossy, too, since every lossy codec out there features an entropy coder at the end of the encoding chain. Of course with lossy you've got a very important additional degree of freedom, because you can replace the original input with a different one that look "similar enough" and it's more compressible. But still, the limit applies, after the decorrelation/noise shaping techniques that you're going to apply before entropy coding.

While CABAC is ideal for its probability model, it's nowhere near the best possible.
It's difficult to tell. With an arithmetic coder, you've already optimized one of the two steps that determine its efficiency: the ability to match the entropy associated with a specific (arbitrary) probability model. There's still another step: estimating the probability model and having it match the source as closely as possible. Here's a source for improvement, but how much?

You know, there's a lot of stuff you can do: using different contexts, switching from a binary to a multisymbol coder, using higher order models, or maybe even go for something different, like vector quantization. But the coder efficiency is mainly determined by the level of residual correlation in the encoded signal.

After a well designed MC step (with QPel precision) with adaptive MB size most of the inter-coded residual blocks/frames are almost uncorrelated. We're pretty close to white noise... and here using a higher order model would bring a very, very limited improvement (probably just a few percent, or less) because there's simply nothing more to compress; almost no higher order correlation to exploit. The more you remove, the less remains to be removed :)

That's why I don't feel so optimist about a revolutionary, brand-new entropy coder. In many cases, an arithcoder can shrink only a 5-10% over a RLE+Huffman combo, if not less. H.264 designers claimed that CABAC performs on average only 10% better than CAVLC which is based on cheap and fast Golomb codes. I mean, maybe zero-order CABAC is just what we need. We can refine it adding a lot of complexity, but we're already in the region of (very) diminishing returns... maybe it would be better to concentrate on different aspects of the codec, like MC. Personally I don't foresee any improvement of 40-50% simply by replacing the entropy coder: today we're already discussing of improvements with a magnitude of a few percent points... ;)

More realistically, once we develop a decent psychovisual model, I can image some sort of texture library, like a MIDI/MOD for video.
My personal vision of a future codec (but we're only guessing here :)) is centered about adaptivity. We've already a lot of adaptive stuff (MB size, ME parameters, a context-based entropy encoder, custom weight matrices, and so on). I think those concepts would be pushed to the limit, in order to squeeze the remaining bits. There will be a combination of many little improvements (say, no more that 2-3% for each new "tool") but not a real breakthrough, a radical innovation - no, not even wavelets IMHO.

Maybe the input parameters will include the type of source (noisy, clean hi-def, anime - or maybe even some more sophisticated params) and the codec will select different algorithms, each suitable for a different type of source. Maybe even different algorithms for each frame type/slice type (adaptivity!). Of course, psychovisual improvements offer some room for improvement, too.

But I have the feeling that we'll arrive quite soon (some years, maybe 5, or 10, at most 15) at a point where we would be better spend our time trying to enlarge the connection bandwidth or the media storage size instead of wasting ages to squeeze that last 5% out of our encodings :)

just my 2 cents
(seeing the size of the post, I'd say 10 cents :D - sorry for being so chatty, but I happen to enjoy these discussions)

virus

[Joe Fenton]

With the introduction of 64-bit computers and low-cost high volume harddrives, I think VQ will make a resurgence. Think of a codec using a 200G codebook mapped in virtual memory on a 64-bit processor - VQ has only been limited at this point by the size of the codebooks used and the ability to access them efficiently. Huge drives and 64-bit CPUs will make this a viable codec.