Ideas for New Concepts for Video Compression

[God Radio]

Even assuming you can extract the models and textures entirely from the remaining frames with no sideband data, and even assuming motion can be perfectly interpolated, that still only cuts the bitrate by about 20%, because you cut B-frames and keep I/P-frames. So this line of extrapolation doesn't go anywhere. Lets try from the other end:
How much space does it take to store the models, textures, and motion of a modern 3D CGI movie? Well, that wouldn't be very helpful either, as those probably don't involve any compression, and are designed to render at very high resolution. How much space does it take to store the models, textures, and motion of a modern 3D CGI movie, if models are subsampled and textures lossily compressed until the resulting movie is similar precision to a decent 1080p h264? Furthermore, there's a large difference between taking an existing movie and compressing it, vs designing the movie with compression in mind a la .kkrieger




Ahhh I think that I understand at last(maybe?):

Everyone is focused on a single do everything tool.

What I am trying to say is that when a wireframe is used then we go for a multiple tool set when doing the renders to optimize bandwith.

For example an adjustable wrench is never going to outperform a dedicated set of tools.

The same goes for focusing on a single codec to do everything.

Can we try four or six codecs(open source) with each optimized for a single purpose so that using all of them we get better performance?

If we had a codec just for I/P frames and another codec just for B frames we get much better performance.

I am sorry but my explanations are not too good.............hmmm let me try and post an example:

http://www.businesswire.com/portal/site/home/news/sections/?ndmViewId=news_view&newsLang=en&newsId=20080410005215

Broadcast International has patented the unique CodecSys approach of using multiple expert codecs rather than single, general-purpose codecs, which are used in all other video encoding solutions. By automatically switching between specialized codecs, choosing the best one for a particular frame or video sequence, CodecSys is able to reduce video bandwidth requirements more than 80% for HD video over all types of delivery platforms including broadcast, cable, IPTV, wireless and the Internet.

[akupenguin]

What I am trying to say is that when a wireframe is used then we go for a multiple tool set when doing the renders to optimize bandwith.
If multiple tools are good, then put them all in the same codec, and let them switch per block or even use all at the same time (as an overcomplete representation if there's no better way to reconcile them).
This is a perfectly valid technique, and is used in e.g. Bink with its 16 different coding algorithms. The result was decent and fast, but nothing special even when compared against others in its own generation.

If we had a codec just for I/P frames and another codec just for B frames we get much better performance.
The only difference between a B-frame and a P-frame is that one is motion interpolation and the other is motion extrapolation. How could there possibly be a coding method that's good at one and not at the other?
Intra I can see, though anything that's good at I-frames should also be good at large intra regions in P-frames if you can reconcile the transitions to inter.

Broadcast International has patented the unique CodecSys approach of using multiple expert codecs rather than single, general-purpose codecs, which are used in all other video encoding solutions. By automatically switching between specialized codecs, choosing the best one for a particular frame or video sequence, CodecSys is able to reduce video bandwidth requirements more than 80% for HD video over all types of delivery platforms including broadcast, cable, IPTV, wireless and the Internet.

It is generally safe to assume that any claim of "revolutionary" video compression improvements is bullshit.
If you need a reason for that, consider: a 1.5x improvement in compression is more than enough to publish; that's maybe as much H.264 gains over ASP (number pulled out of thin air, but you get the idea). So no company would wait until they had a 5x improvement before they started up the hype engine. You never heard of them before, right? Ergo, they don't really have a 5x improvement, either they have cooked tests which don't correctly represent their codec, or they have nothing at all and are just trying to scam people and run. Both of which are very common; we see articles like this every month or so.

[MfA]

Investment reporting is so disgusting, they are completely spineless. A company which hasn't once released R/D data for it's compression compared to the industry standards and none of them call them out on it.

As long as you have enough impressive suits, stupid investors and customers tech reporting will praise you into high heaven (giving rise to more stupid investors and customers). Kind of depressing they got IBM PHBs to pour money into it, they were desperate for a high profile Cell application I guess ... and well, codecsys was getting great press! Sigh.

[benwaggoner]

Broadcast International has patented the unique CodecSys approach of using multiple expert codecs rather than single, general-purpose codecs, which are used in all other video encoding solutions. By automatically switching between specialized codecs, choosing the best one for a particular frame or video sequence, CodecSys is able to reduce video bandwidth requirements more than 80% for HD video over all types of delivery platforms including broadcast, cable, IPTV, wireless and the Internet.
Yeah, I saw their demo at NAB last year (didn't bother this year).

Their demos weren't even DEINTERLACED. They were encoding interlaced as progressive! And they had obvious I-frame popping on codec transitions (as you'd expect)!

And it's hard to think how this could even kind of work. Maybe there's been a 500% improvement in codecs from H.261 to H.264 High Profile, MAYBE. From Cinepak, sure. But there's never a 500% difference between current generation codecs. Let's say we're comparing VP7, VC-1, RV10, and H.264 (current decent codecs with real-time implementations). One can imagine in some cases that one might do a little better than the other with the same content. But if you're going to do parallel encoding with stream switching, you could also just encode with different parameters and pick the one with the best SSIM or something. If I was doing this in VC-1, I might try Adaptive Deadzone Off/Conservative/Aggressive, A few different B-frame patterns, Overlap On/Off, a few DQuant modes. With H.264, some different loop filter strengths would be an obvious first step. Of course, when you figure out what options are useful there, you can probably get most of that heuristically by doing a lookahead encode, or a subsampled first pass.

And productizing the decoder? Ugh. Existing decoder ASIC's can't even switch between encoded frame sizes on the fly correctly, no way they'd be able to do that. So it'd be a PC-only solution.

[IgorC]

The other important thing is metrics.
Maybe H.264 and ASP could be improved for visual quality even more if there will be more realistic metric then SSIM and already out of game PSNR. Many times developers just almost blindly optimized their codec to metrics values because psycho algos could requires public test which are expensive.


Just looking at x264's development and I see how useless those SSIM values are especially at high bitrates when it comes to psychovisual improvements (VAQ, film grain optimization etc.)

[MfA]

Metrics are never going to be much use for grain ... whether it's desirable to have noise which is only approximately in the right place or whether to simply remove it (saving bits which can be used elsewhere in the process). So much room for personal preference there, some people care more than others for artist intent too.

About personal preference ... in one paper I read the other day they actually found two distinct classes of people in their testing of H.264's deblocking. Even with the relatively advanced loop filter in H.264 some people still preferred it off (maybe there should be a switch in h.264 decoders to output frames before the deblocking is applied?).

[Manao]

maybe there should be a switch in h.264 decoders to output frames before the deblocking is applied?There already is. But not a lot of decoder supports it (ffmpeg doesn't for example)

[MfA]

Hmm, I didn't mean a bitstream flag ... simply a tick button in for instance ffdshow to output the frame first and only then apply the loop filter (I'm not very quality conscious either way though, so it would be lost effort for me).

[benwaggoner]

About personal preference ... in one paper I read the other day they actually found two distinct classes of people in their testing of H.264's deblocking. Even with the relatively advanced loop filter in H.264 some people still preferred it off (maybe there should be a switch in h.264 decoders to output frames before the deblocking is applied?).
The H.264 loop filter is arguably too wide (in terms of number of pixels touched), especially with 4x4 blocks, and yields too much softening.

The wide For moderate-high bitrates, the narrower looper VC-1 is better at preserving detail, although it'll allow more blocking to be visible at lower bitrates without postprocessing filters (hence the postprocessing filter use in WMP at low bitrates).

[Dark Shikari]

The H.264 loop filter is arguably too wide (in terms of number of pixels touched), especially with 4x4 blocks, and yields too much softening.Its only wide at low bitrates... where one needs it to be wide.

I agree its slightly too strong for 4x4 blocks, but VC-1's filter is so weak as to be nearly useless; it is certainly not a good example to point to as a "good" deblocker!

[benwaggoner]

Its only wide at low bitrates... where one needs it to be wide.

I agree its slightly too strong for 4x4 blocks, but VC-1's filter is so weak as to be nearly useless; it is certainly not a good example to point to as a "good" deblocker!
I think nearly useless is a bit of an exaggeration, but no argument that H.264 helps low bitrates degrade into more visually acceptable softness instead of blockiness. The VC-1 Overlap filter can help quite a bit in cases where the Loop filter isn't enough. And of course, there's the option of postprocessing which most lower bitrate VC-1 decoding gets access to.

[*.mp4 guy]

Its only wide at low bitrates... where one needs it to be wide.

I agree its slightly too strong for 4x4 blocks, but VC-1's filter is so weak as to be nearly useless; it is certainly not a good example to point to as a "good" deblocker!

Too weak is often a better compromise then too strong in many respects. Especially considering that any kind of deblocking can do nothing but remove information from the video, you can't get back the information that was discarded, which caused the blocking to occur, all any deblocker ever does is blur the picture and hope that the blur is less obvious then the blocks were. Inloop deblocking however can be use full in that it increases coding efficiency, but as far as actually improving the picture, its really just a question of preference: to blur or to block.

[Dark Shikari]

Too weak is often a better compromise then too strong in many respects. Especially considering that any kind of deblocking can do nothing but remove information from the video, you can't get back the information that was discarded, which caused the blocking to occur, all any deblocker ever does is blur the picture and hope that the blur is less obvious then the blocks were. Inloop deblocking however can be use full in that it increases coding efficiency, but as far as actually improving the picture, its really just a question of preference: to blur or to block.There's an SEI message specifically for the purpose of displaying the image before deblocking is done.

Nevermind that fact, the primary cause of deblocking being too strong is encoders using overly high quantizers, rounding up coefficients to try to simulate grain/detail when they should be using a lower quant instead, and then wondering why their fake detail is blurred so much :rolleyes:

This is one of the primary benefits of AQ...

[benwaggoner]

...now if only there was a codec implementation that could combine a narrower loop filter with an adaptive deadzone and differential quantization, it'd be able to preserve detail at higher quants and hence lower bitrates :)...

[Dark Shikari]

...now if only there was a codec implementation that could combine a narrower loop filter with an adaptive deadzone and differential quantization, it'd be able to preserve detail at higher quants and hence lower bitrates :)...Higher quants do not necessarily mean lower bitrates. The only kind of detail you'll get at high quantizers is "fake" detail which can be far better generated using something like film grain modelling.

If you want more detail in a block, lower the damn quantizer. If its using too many bits, your encoder isn't efficient enough.

[benwaggoner]

Higher quants do not necessarily mean lower bitrates. The only kind of detail you'll get at high quantizers is "fake" detail which can be far better generated using something like film grain modelling.
...if you have decoders that include grain modeling. But getting new decoder features deployed is extremely expensive. And if if it's there, it's not guarantee it'll be used; I don't know that ANY HD DVD title ever shipped that used the grain modeling feature, even though it was mandatory.

Sometimes "fake detail" that gives the right perceptual experience is better than a perceptual lack of detail.

If you want more detail in a block, lower the damn quantizer. If its using too many bits, your encoder isn't efficient enough.
Well, that really depends on the context. It isn't always possible to have enough bits to get the quants low enough in lots of cases. Blu-ray, sure, but VOD, streaming, IPTV, etcetera are pretty much by definition bit-starved, so anything that can improve quality at the ragged edge of compression is a good thing.

[Dark Shikari]

Well, that really depends on the context. It isn't always possible to have enough bits to get the quants low enough in lots of cases. Blu-ray, sure, but VOD, streaming, IPTV, etcetera are pretty much by definition bit-starved, so anything that can improve quality at the ragged edge of compression is a good thing.I'm not talking about higher bitrates; bitrate is not directly dependent on quantizer.

My point is that if you have a constant bit budget, you're better off using those bits on lower quantizer blocks with higher relative lambdas than on higher quantizer blocks with lower relative lambdas. Adaptive quantization can be used to spread the quantizers accordingly. "Use a lower quantizer" does not mean "raise the bitrate." It means to raise your lambda.

If an encoder is trying to retain detail at an absurdly high quantizer by throwing bits everywhere instead of using a sane quantizer (at the same bitrate), its developers really have no right to complain about the deblocker being too strong.

[*.mp4 guy]

If an encoder is trying to retain detail at an absurdly high quantizer by throwing bits everywhere instead of using a sane quantizer (at the same bitrate), its developers really have no right to complain about the deblocker being too strong.

True, especialy considering that quantizers are only usefull as a relative value, and are completely meaningless without lots of ancillary information; cqms, deadzones, rounding thresholds, various forms of rd, block decimation/prediction decisions, relative frame biases, etc. all make quantizer values essentially meaningless on their own. Infact, the only thing that can be predicted from quantization values relatively accurately, is inloop deblocking.

[Birdy1]

Another idea:
Instead of I, P and B frames, use I, P and B macroblocks.
In quiet parts of a scene, mostly B blocks are used. In parts with much movement, mostly P blocks are used. And when something new pans in to the screen, I blocks are used.
This way, the encoder can pick the best solution per macroblock, instead of the best solution per frame (which is a compromise).

[Dark Shikari]

Another idea:
Instead of I, P and B frames, use I, P and B macroblocks.
In quiet parts of a scene, mostly B blocks are used. In parts with much movement, mostly P blocks are used. And when something new pans in to the screen, I blocks are used.
This way, the encoder can pick the best solution per macroblock, instead of the best solution per frame (which is a compromise).Already done. P-frames allow both I and P blocks, and B-frames allow all three block types.

[Birdy1]

didn't know that, thanks :)

[akupenguin]

But it raises an interesting question. P vs B adaption isn't just about block type, but rather order. Is there any sane way to use different frame orders in different parts of the frame? e.g predict the left half of frame 1 (P), then all of frame 2 (P, parts referring to 0 and parts to 1), then the right half of frame 1 (B). Easy without motion, but...

[MfA]

With more advanced motion compensation is there really any need for b-frames at all though? I've always found subsampled scanning patterns very inelegant compared to linear ones, on average they just increase the distance (and reduce the correlation) to already coded samples.

[akupenguin]

I see no significant difference in the importance of B-frames in 16x16 fullpel vs h.264 partitioned qpel or obmc. Thus I have no reason to think that further improvements in motion compensation would render B-frames obsolete.

[MfA]

Not talking about higher density motion fields, that's tangential. Instead better multihypothesis combination modes, temporal processing along the motion path to improve R/D ... etc.

PS. can H.264 use implicit weighted averaging with forward only prediction? (If so, how about x264?)

[Manao]

Implicit weights requires a bidirectionnal coding mode, thus a bpicture. But it doesn't require L0 & L1 reference to be respectively forward and backward. x264 with two refs will have a forward reference in its L1 list, so it can use implicit weight with forward only prediction.

One advantage of bframe that exists however good the MC is, is that it allows to handle sub-macroblock movements (small panning for example) perfectly : you code the appearing macroblocks only once (in the pframe), and all bframes in between can just copy & paste. Were you to code all those frames as P, you would have to put texture bits in appearing macroblocks each time.

[MfA]

You could add an extra border of blocks as overscan for pans and fill those with data from the future frames when appropriate. Pel based residual coding instead of block based would also help.

Manao, so how do I get x264 to use unidirectional b-frames?

[Manao]

By default, the first reference frame in the L1 list will be a future frame in x264. If you want to change that, you need to have a look at x264_reference_build_list() in encoder/encoder.c

You could add an extra border of blocks as overscan for pans and fill those with data from the future frames when appropriateBut if you're ready to require future frame(s) to decode the current, why not go the bframe way (which is more generic and propose an efficient bidirectionnal mode, plus useful direct interpolation for MVs) ?Pel based residual coding instead of block based would also helpIndeed, but then block based approach for residual usually works better.

[MfA]

But if you're ready to require future frame(s) to decode the current
I meant you would fill the overscan area (when appropriate) at the encoder. It would copy from the future frames, not the decoder.

[Birdy1]

Has this been done before:

Use spatial intra-frame motion vectors to code I-frames. This way the encoder can take advantage of similarities within the I-frame.

[Manao]

http://forum.doom9.org/showthread.php?p=1124809#post1124809

[Dark Shikari]

Has this been done before:

Use spatial intra-frame motion vectors to code I-frames. This way the encoder can take advantage of similarities within the I-frame.Already proposed (http://ftp3.itu.int/av-arch/video-site/0701_Mar/VCEG-AE11.zip)...

[brunogm]

Hi, I'm a computer science student, recently i found about curvelets and wedgelets (families of wavelets) and one thing that the authors claimed was that "If we´ve been using the structure of operating wavelets in the past 20 years, now we only have to add new bases to have a significant improvement"

So after looking on Dirac and Snow, i realized that if we make the Wavelet base in Snow changeable in the future it will be fast and easy to improve it as soon as new Wavelet bases are found. Also i want to help in this and make that my graduation project.

Cheers.

[MfA]

The curvelet/wedgelet/ridgelet/whatevernonwavelet-let transforms are not just variation on wavelet bases used with separable horizontal/vertical filtering like in Snow/Dirac ... the transforms are quite fundamentally different AFAICS.

[akupenguin]

Already proposed (http://ftp3.itu.int/av-arch/video-site/0701_Mar/VCEG-AE11.zip)...
2x2 block ESA at decode time, and this is the fast texture synthesis?

[Dark Shikari]

2x2 block ESA at decode time, and this is the fast texture synthesis?:p

You should have seen their method of avoiding coding motion vectors; they do an actual motion search and code the motion vector with respect to the one calculated by the decoder... I think they were proposing a full search too :rolleyes:

[burfadel]

:p

You should have seen their method of avoiding coding motion vectors; they do an actual motion search and code the motion vector with respect to the one calculated by the decoder... I think they were proposing a full search too :rolleyes:

Well that would reduce encoding impact I guess... but wouldn't that increase cpu use quite noticeably for 1080p content?

[akupenguin]

Well that would reduce encoding impact I guess... but wouldn't that increase cpu use quite noticeably for 1080p content?
That's nothing. The hypothetical algorithm (that wasn't actually in the linked pdf) involves ESA over the whole frame. Effectively merange=infinity. You'd be lucky to decode QCIF in realtime, and CPU cost grows with the square of resolution (i.e. fourth power of width).

[bratao]

Ok, a concept from a person that known nothing about video encoding:
-The encoder/decoder have a table of all possible combination of , for example 4x4 video blocks.
That may be:2^384 possible combinations
So , for each possibility is attributed a decimal number.

The all possibility table dont need to be compiled on runtime, it can be pre-compiled and distributed with the encoder/decoder.

For a 16x16 image(16x 4x4 blocks), this unoptimized algorithm should create a gziped loseless with 200 Bytes.
A optimized jpeg encoder gzipped , cannot do the same image with less than 2000 Bytes.

So, that´s make any sense ?

[Irakli]

-The encoder/decoder have a table of all possible combination of , for example 4x4 video blocks.
That may be:2^384 possible combinations

That's simply impossible. 2^384 is approximately 3.94 x 10^115 :eek:. This will require literally infinite amount of time to implements and literally infinite amount of memory to store all these combinations.

[MfA]

Also, there is no compression ...

[squid_80]

He's talking about bits there. 2^384 = 384 bits = 48 bytes (16 pixels@24bpp). Hardly infinite. :)

I don't know the ins and outs of jpegs but I'm guessing the cause is overhead, possibly tables of some sort? Try making the same comparison with a practical resolution...

[Dark Shikari]

The idea of VLC-coding blocks of pixels (and quantizing them) is called vector quantization and is anything but a new idea.

[Sergey A. Sablin]

Ok, a concept from a person that known nothing about video encoding:
-The encoder/decoder have a table of all possible combination of , for example 4x4 video blocks.
That may be:2^384 possible combinations
So , for each possibility is attributed a decimal number.

the number of possible combinations will be (2^8)^16, if we're talking about 8-bit color depth. which is effectively 2^128. ie attributed number is 128 bit long, which is equal to the uncompressed size of 4x4 pixel block -> no compression. (and mind that distribution of probabilities of 1's and 0's in your attributed number will be more closer to uniform than in original 4x4 pixel block, so you'll hardly compress any of these numbers)

For a 16x16 image(16x 4x4 blocks), this unoptimized algorithm should create a gziped loseless with 200 Bytes.
the size will be 256 bytes - absolutely same as uncompressed. jpeg has overhead as was already pointed - start codes, tables etc. and actually it wasn't designed to compress images of such small size. try to calculate a bit bigger sizes, 1024x768 for example.

[brunogm]

Well wavelets are a diffiult thing to find information so here is a good text that explain the daubechie and the more recent wavelets for image compression.

http://www.sci.sdsu.edu/compsciwork/IIIPASI/DOCUMENTS/COURSE%20NOTES/waveletPASI-IIICPereyra.pdf

Skip the math if you prefer, the important is the explanation of families of wavelets including Daubechies and curvelets.

Maybe aimprovement on Snow cna be achieved by using more recent and detailed work on wavelets. I'm willing to help but for now i will study what i lack from the source of x264.

Thanks!