Ideas for New Concepts for Video Compression

[3NC0D3_Y0_A$$]

Ok (this may be a bit noobish), So we know various video codec standards have different 'concepts' to encode video and reduce bitrate whilst maintaining quality to the human eye. I realize that a lot of video codec development is down to improving the efficiency or 'eloquance' of these video compression techniques and the languages they are written in.

The H.264 video codec standard has the following compression techniques:

CAVLC/CABAC, Loop/Deblocking Filter, Variable Block Sizes/Macroblock Partitions, Multiple Reference Frames, Weighted Prediction, Rate Distortion Optimisation (RDO).

What other video compression 'concepts' or techniques can you think of?

Its unlikely that anyone is generating a new video compression standard anytime soon, but it still would be interesting to see what ideas people can come up with.

Just for starters:
Perhaps encoding the edges of a still object into an equation for a shape (that outlines it) and also the inside of the shape if the colour is uniform. (this would be useful for distance objects in still shots)

[nm]

Its unlikely that anyone is generating a new video compression standard anytime soon, but it still would be interesting to see what ideas people can come up with.
New open standards are ratified quite often and some companies have their own closed "standards". As an example of an upcoming open (and royalty-free) standard, Dirac Pro is being standardized as SMPTE VC-2.

If you're interested in new techniques and innovations, I'd suggest checking out the abstracts of video compression research papers from the last five years or so.

[Inventive Software]

The only standard they could realistically go into is Snow, which has yet to be given a final specification.

[nm]

The only standard they could realistically go into is Snow, which has yet to be given a final specification.
Who could go? SMPTE? I seriously doubt Michael Niedermayer or other FFmpeg developers would ever consider going through such a standardization process for Snow. RFC maybe.

[3NC0D3_Y0_A$$]

Who could go? SMPTE? I seriously doubt Michael Niedermayer or other FFmpeg developers would ever consider going through such a standardization process for Snow.

Snow? can you explain what this means please?

[nm]

It's a wavelet codec in FFmpeg/libavcodec, written by Michael Niedermayer. See: http://en.wikipedia.org/wiki/Snow_(codec)

[3NC0D3_Y0_A$$]

It's a wavelet codec in FFmpeg/libavcodec, written by Michael Niedermayer. See: http://en.wikipedia.org/wiki/Snow_(codec)

cheers.

[Inventive Software]

ffmpeg's website has a notice that says a specification will be written up for Snow eventually, if there is enough interest. Until then, it's basically a free-for-all based on current code. I think it has CABAC and OBMC and inloop deblocking. Basically borrowing ideas from other codecs, and developing new techniques to make a usable codec to attack H.264 is I think Snow's final aim, but at the moment, speed is less than desirable.

[IgorC]

CAVLC/CABAC, Loop/Deblocking Filter, Variable Block Sizes/Macroblock Partitions, Multiple Reference Frames, Weighted Prediction, Rate Distortion Optimisation (RDO).

Don't worry. Guys are already preparing H.265.


I'm not a professional of video compression but afaik central concept of lossy compression is transform (between temporal to (complex) frequency domains ) like DCT.


There is no proof that wavelet transform is better than (M)DCT.
Jpeg2000 (wavelet) is worse than intra H.264 based on DCT.


afaik it is planned that 2008 will be last year of changing reference source of H.264, new profiles etc... And there will be first papers about H.265 in 2009.

[Dark Shikari]

There is no proof that wavelet transform is better than (M)DCT.But unlike DCT, it can be used in combination with OBMC to avoid blocking altogether.
Jpeg2000 (wavelet) is worse than intra H.264 based on DCT.That's basically because its intra prediction is FIR instead of IIR.

[MfA]

OBMC is not completely blocking free.

I think the improvements which can be gained by OBMC and wavelet coding in the end are only slight. IMO motion coding is the lowest hanging fruit for big improvements at the moment, motion motion compensation and per pixel lossy coding of the motion field.

[Dark Shikari]

OBMC is not completely blocking free.

I think the improvements which can be gained by OBMC and wavelet coding in the end are only slight.IIRC total residual can be decreased by about 25% using OBMC.

[Manao]

Some of the high points that are being considered for inclusion in h265 are :
- 80+ prediction mode just for intra, which greatly increase intra efficiency
- intra frame motion vectors
- free macroblock partitionning for inter : instead of fixed 16x8, 8x16, 8x8... partitions, you can cut a block/macroblock in two by defining a cutting line however you want (not just vertical/horizontal, but any angle too)
- better syntax (for example, better motion vector prediction)

[akupenguin]

That's basically because its intra prediction is FIR instead of IIR.
Which is in turn because I have yet to see anyone make IIR intra prediction compatible with wavelets, nor have I thought of a way myself.

@MfA:
Want to run Snow and point out a blocking artifact to me?

@Inventive Software:
Snow doesn't have inloop deblocking because it doesn't need it. OBMC+Wavelet doesn't produce blocking artifacts, so doesn't need to be deblocked.

[Dark Shikari]

Which is in turn because I have yet to see anyone make IIR intra prediction compatible with wavelets, nor have I thought of a way myself.Why is this? Isn't intra prediction entirely spatial, with wavelets merely being a way to encode the residual?

[NerdWithNoLife]

If you're interested in new techniques and innovations, I'd suggest checking out the abstracts of video compression research papers from the last five years or so.
Goodness, did you ever read those papers on DCT, quantization, or compression domain transcoding?

That's a real quick way to feel like an idiot.

[akupenguin]

H.264 spatial intra prediction predicts from the decoded pixels of the neighboring blocks. In any transform with overlapping basis vectors, the neighbors haven't been decoded yet. Plus you need a continuous prediction function if you want to retain the block-free nature.

If you're interested in new techniques and innovations, I'd suggest checking out the abstracts of video compression research papers from the last five years or so.
Don't try this at home, kids. The signal-to-noise ratio in video compression research is really low, so if you don't already have the intuition to cull the BS papers, reading isn't too useful.

[Zarxrax]

Just for starters:
Perhaps encoding the edges of a still object into an equation for a shape (that outlines it) and also the inside of the shape if the colour is uniform. (this would be useful for distance objects in still shots)

This is quite similar to the way flash animation works. I've long thought that something like this could be very effective for anime content, but I have yet to see any sort of proof of concept idea for it. There are some tools out there that have become fairly decent at vectorizing a still image, but I have yet to see anything that can intelligently vectorize a video stream. Perhaps the mathematics behind it is just too complex.

[Inventive Software]

Why is this? Isn't intra prediction entirely spatial, with wavelets merely being a way to encode the residual?

I think the wavelet transform is done before anything else, then OBMC is done on top of that.

[Birdy1]

Is there something like a 3 dimensional wavelet? If so, a scene could be encoded as a 3D wavelet, where the z-axis is the time axis.
No search algorithms, no motion vectors, no residual to encode. Memory usage would be huge however.

[Phanton_13]

Is there something like a 3 dimensional wavelet? If so, a scene could be encoded as a 3D wavelet, where the z-axis is the time axis.
No search algorithms, no motion vectors, no residual to encode. Memory usage would be huge however.Yes there are 3D wavelets and even there are 3D DCT, but at the same time this techniques are in a heavy mined patent land.

[akupenguin]

The problem with 3D wavelet is that it sucks. No motion compensation means you have to code all motion as residual, which is much more expensive than mvs.
What you can do is combine motion compensation with a temporal wavelet, i.e. at each wavelet prediction/update step use motion compensated samples from the neighboring frames rather than the original pixels. This is called "MC-lifted wavelet". But it turns out that wide basis vectors like 9/7 aren't very useful, and MC-lifted 5/3 wavelet is very similar in both structure and compression quality to H.264 B-pyramid.

[Manao]

One possibility that is considered for h265 is 3D DCT on consecutive bframes : you compensate the three bframes as usual, but code the residual of all three bframes with a 3D DCT.

It seems promising, but I don't know how it compares to b-pyramids.

[foxyshadis]

Some of the high points that are being considered for inclusion in h265 are :
- 80+ prediction mode just for intra, which greatly increase intra efficiency
- intra frame motion vectors
- free macroblock partitionning for inter : instead of fixed 16x8, 8x16, 8x8... partitions, you can cut a block/macroblock in two by defining a cutting line however you want (not just vertical/horizontal, but any angle too)
- better syntax (for example, better motion vector prediction)

Is there any effort directed toward 16x16 or larger DCT? Or multiscale residual coding (ala jpeg 2000) with DCT or wavelet? Both benefit low-bitrate video (or anime, I suppose) but have little practical use for sharp/noisy high-bitrate, and both would probably be a large speed hit.

Haha, speaking of speed hits, I'd love to see a multiscale intra coder with the scaler based on NNEDI. =p

[MfA]

MC-wavelet's and SVC's claim to fame isn't really performance, but scaleability (which is ultimately useless IMO).

[Manao]

foxyshadis : if I'm not mistaken, 16x16 DCT is being considered too.

As for SVC, it stands better chances than previous attempts at scalability (Mpeg2/4 FGS), because IPTV can profit from it. That it has better chances of being actually used doesn't mean of course it'll be effectively used in the end.

[akupenguin]

MC-wavelet's and SVC's claim to fame isn't really performance, but scaleability (which is ultimately useless IMO).
MC-wavelet gives you temporal scalability, and does it no better than B-pyramid. In fact, pyramid might be better; normal framerates are too low for a temporal lowpass to look good, so you're better off just dropping frames than using a wavelet subband.

[benwaggoner]

MC-wavelet's and SVC's claim to fame isn't really performance, but scaleability (which is ultimately useless IMO).
Why do you say that?

My customer are constantly asking about scalability. It's really hard to stream content reliably and in high quality over the public internet without it. Bandwidth, between users and even mid-session for longer content, is just too variable.

Move Networks has had tremendous success with abc.com and other sites by delivering practical scalability, even though it isn't using any kind of intraband or layered compression.

The real question is if we can do a "FGS" style scalability that doesn't cause such a cumulative reduction in compression efficiency that it's easier to just encode independant bands and switch between them (whcih is what all the real-world systems have used up to now, like Intelligent Streaming and SureStream).

[moogie]

Probably not very new or un-obvious... but i have been playing around with the idea of non-block based compression. i.e. pixel based.

I am currently using "least squares" to generate a equation which approximates the colour of a pixel per frame.

This means that there is an equation for each pixel.

Each pixel can(and probably will) have multiple equations which will be used sequentially. When a pixel is unable to be modelled within certain error bounds then the next equation is used. (the number of frames to use each equation is recorded in the bit stream).

Bit savings can be achieved when multiple frames (per pixel) can be modeled per equation. The actual frames necessary for compression to occur can vary as I can reduce the accuracy of the constants for the equation. i.e. instead of a 32 bit float per constant... 24 bits.

for example

Assuming an 8 bit grey scale video

i have 20 frames of a pixel which can be modeled with given error bounds using an equation consiting of 3 terms (or order 2)... lets say -1230 * x ^ 2 + 0.3315 x - 88

Further analysis has determined that the equation's terms can be encoded in 16 bits without going beyond the error bounds.

thus
8 bits for the equations frame length
+ 2 bits to record how many bytes per equation term.
+ 16 x 3 for the equation.

gives 26 bits for 20 frames. uncompressed it would be 20 x 8 = 160 bits


That example was a little contrived but from my experiments it does compress resaonably well. I doubt that it will have any where near the compression of modern codecs but as my first attempt at a lossy video codec i dont think it is too bad :P

My next step is to encode a colour video using this scheme.

some other possible enhancement ideas:

-previous frame pixel matching (similar to block matching)
-allow equations with different number of terms per video

a nice benefit of this system is that the decoding of the video should be quite fast.

[MfA]

The real question is if we can do a "FGS" style scalability that doesn't cause such a cumulative reduction in compression efficiency that it's easier to just encode independant bands and switch between them (whcih is what all the real-world systems have used up to now, like Intelligent Streaming and SureStream).
Even if you could avoid the reduction in compression efficiency, what are the advantages? A little saving's in storage? Storage is extremely cheap. There has to be a real advantage to bother with scaleable codecs in the first place, not just a lack of disadvantages.

[akupenguin]

a nice benefit of this system is that the decoding of the video should be quite fast.
Are you sure? If each pixel depends on the previous, then you can't use SIMD. So even if you reduce the number of theoretical arithmetic operations compared to MC+DCT, it may still be slower than a block-based codec.

[Manao]

The real question is if we can do a "FGS" style scalability that doesn't cause such a cumulative reduction in compression efficiency that it's easier to just encode independant bands and switch between them (whcih is what all the real-world systems have used up to now, like Intelligent Streaming and SureStream).SVC claims to be as efficient as AVC, while being scalable (I don't know whether scalable here encompass FGS, or if it's something more coarse).
Even if you could avoid the reduction in compression efficiency, what are the advantages?Bandwidth. And generate x streams at difference x bitrates requires x AVC encoders, while only one SVC encoder.

If I take my example, my ISP is providing some TV channels at both low bitrates (1.5mbits h264) and high bitrates (3mbit mpeg2). The purpose is to increase the number of people being able to watch videos. With SVC, they could also maximize the quality each individual receive (because a user has a max bitrate ranging from 1mbit to 22mbit, depending on the lossyness of its connection), and they could do it with only one encoder per channel.

[akupenguin]

And generate x streams at difference x bitrates requires x AVC encoders, while only one SVC encoder.
But one SVC encoder is slower than one AVC encoder. At the very least, you have to run macroblock mode decision (with a limited number of mdoes), dct, and miscellaneous bookkeeping at each layer, plus some motion refinement at each change in spatial resolution. Most of motion estimation can be shared between layers, but that's true of multiple AVC encoders too, if they want to take advantage of it. The only speed difference guaranteed by the change in bitstream format is less time spent in CABAC.

[moogie]

Are you sure? If each pixel depends on the previous, then you can't use SIMD. So even if you reduce the number of theoretical arithmetic operations compared to MC+DCT, it may still be slower than a block-based codec.

true, i should have qualified that statement :)

At the moment the algorthim does not depend on the previous when decoding... the act of decoding is as simple as inputing the current frame count into each pixel's current equation.. e.g. at frame 10 with the equation -2.5 * x^2 + 43 * x + 33, the pixel value would be -2.5 * 10^2 + 43 * 10 + 33 = 213. And at frame 11 the value will be 203.5

[akupenguin]

So you use no spatial prediction at all, and a temporal transform with basis vectors 1, t, t^2, etc. In that case, everything I dissed about 3D wavelet still applies, except that you don't get the benefits of spatial wavelet either.

[moogie]

That was just an example. the optimum number of terms in the equation might be more or less than this example for a given sequence of frames.

I am not familiar with 3-d wavelets so i cannot comment on that but will read up on them.

Yes, at the moment only temporal information is predicted... It is only early days for this video compression algorithm. I have only been developing this idea for a couple of days!

It should be easy to modify the algorithm such that it predicts differences instead of the actual pixel value. This would allow some form of spatial prediction to be utilised as well.

[benwaggoner]

But one SVC encoder is slower than one AVC encoder. At the very least, you have to run macroblock mode decision (with a limited number of mdoes), dct, and miscellaneous bookkeeping at each layer, plus some motion refinement at each change in spatial resolution. Most of motion estimation can be shared between layers, but that's true of multiple AVC encoders too, if they want to take advantage of it. The only speed difference guaranteed by the change in bitstream format is less time spent in CABAC.
Beyond encoding seperate streams, layered video lets you proritize deliver of base streams hgher, so that you can deliver those in advance.

Imagine you're doing SVC with four layers for a real-time VOD playback.

You burst ahead the base layer 60 seconds, the second layer 30 seconds, the third 15, and the fourth 10. If the user gets a change in banwidth and can only receive the base layer, that's buffered ahead the most, so they don't have any "buffering" message for stream switching. Also, if they're only getting say the bottom two layers and they get more bandwidth, they just need to start pulling down layer 3, and can keep what they've already cached.

Traditional stream-switching is painful, since often by the time you know you need to switch, you don't have enough time to get enough of the new stream to avoid a disruption in playback.

I'd love to see this be workable in practice.

[Birdy1]

Another idea:
Macroblocks are translated (geometric operation) from one frame to the other using a motion vector.
There are a lot more geometric operations you can do to a macro block, like rotate, scale, skew, warp, change brightness. This way you can get a better match from frame to frame, resulting is less residual.

[Dark Shikari]

Another idea:
Macroblocks are translated (geometric operation) from one frame to the other using a motion vector.
There are a lot more geometric operations you can do to a macro block, like rotate, scale, skew, warp, change brightness. This way you can get a better match from frame to frame, resulting is less residual.Already been done. Nokia MVC (a failed proposal codec for the H.264 standard) used an affine (linear) transformation, which thus would have allowed skewing. Changing of brightness can also sort of be done with existing H.264 through weighted prediction.

Rotate is problematic because it can't really be SIMD'd, so its a lot slower. Same problem with generic non-linear warping.

[MfA]

Rotate is problematic because it can't really be SIMD'd, so its a lot slower.
Modern budget video cards could do it without breaking a sweat (even with the interpolation inside the shader to make it work exactly as intended).

[akupenguin]

And you prefer the latency of sending a task to the video card and back for each macroblock? Or you plan to implement only DXVA/equivalent, with no software decoder at all? "Sorry, you don't have a GeForce 8xxx, you can't play this video."

[MfA]

And you prefer the latency of sending a task to the video card and back for each macroblock?
Prefer it over what? (Of course you would not want to do this stuff at that kind of granularity, only send parameters for entire slices/frames and read back the same.)

[God Radio]

But what if thanks to increased cpu power you can?

We use prediction to drop frames and enable compression.

The problem is that the more compression and prediction requires more and more cpu power, hence X264 and H264 usage is very cpu hungry versus mpeg-2.

So we can get even more compression by doing a prediction of two frames(two seconds) instead of a single frame representing a second.

But.

To enable this level of error prediction means the cpu usage is going to want an 8 core cpu like the PS3 and maybe even twice that.( if we go with five to ten second predictions)

As far as I know we can get better compression, better error correction/detection/prediction but its at the cost of better and better hardware.

I know nothing therefore please feel free to correct me on this.

[akupenguin]

So what would you do to two frames that would compress them better than predicting one at a time? There certainly are areas that would benefit from increases in processing power allowing more complex algorithms (e.g. OBMC, arbitrary motion shapes, non-translational MC), but higher order temporal prediction isn't one of them. As far as I know, temporal transforms are not limited by processing power, but rather by the fact that no one has yet come up with an idea fundamentally better than P/B frames.

[God Radio]

One frame gets a regular X264 or H264 job task.

The second frame gets turned into a wireframe.(eventually the whole movie gets the wireframe for transmission).

This cuts the required bandwith by 1/2.

The fun part is turning that wireframe back into a regular scene since it does take considerable cpu power to do so.(i.e. game CGI when running off the in game engine instead of pre-rendred).

I expect someone sooner or later to try this with either a cartoon or anime to cut bandwith cost when distributing programing.

With better cpu power available there is no real reason to not be able to turn an entire cartoon, anime and sooner or later CGI movies like Apple-Seed into wire frames to be de-compressed and skinned(rendered) at the destination point.

Since most movies these days begin life as wireframes before final ok to be rendered into the final product that we see.

It sounds doable to me but then I barely know what end to point the movie camera at when we are filming.

[MfA]

You are still just doing motion compensation, if compensating every second frame with it works well compensating every frame with it works even better.

[God Radio]

Think of it like building a house where you predict where the next line of bricks is going but you still have to set a flag position for every line of bricks, every door, window, etc.

The simple way is just have the entire blueprint with no set position flags needed.

The wireframe is the blueprint.
The render is the final product working from the wireframe.
The color palette is just a tiny data stream that tells the render engine what colors go where the same as sending a multicolor fax where you only send the color positions and color pallette as a tiny data stream.

Pre-rendered data( even compressed) uses a huge amount of space versus on the fly renders.

The downside is that the more complex or realistic the render the more cpu power is required.

So we make do with prediction but soon we will begin to try the on the fly render for at least some cartoons.

Have to go.

Thanks for the interesting conversation.

[nm]

Your idea sounds like Machinima (http://en.wikipedia.org/wiki/Machinima), which allows the scenes to be rendered at the viewer's computer, although large-scale distribution is usually done with video files. If the game/video engine and the data files are distributed instead of the resulting video file, complex scenes may still require quite large amounts of texture data. On the other hand, output pixel resolution is only limited by the viewer's equipment and the game-based medium allows varying levels of viewer interaction.

[akupenguin]

One frame gets a regular X264 or H264 job task.
The second frame gets turned into a wireframe.(eventually the whole movie gets the wireframe for transmission).
This cuts the required bandwith by 1/2.
Even assuming you can extract the models and textures entirely from the remaining frames with no sideband data, and even assuming motion can be perfectly interpolated, that still only cuts the bitrate by about 20%, because you cut B-frames and keep I/P-frames. So this line of extrapolation doesn't go anywhere. Lets try from the other end:
How much space does it take to store the models, textures, and motion of a modern 3D CGI movie? Well, that wouldn't be very helpful either, as those probably don't involve any compression, and are designed to render at very high resolution. How much space does it take to store the models, textures, and motion of a modern 3D CGI movie, if models are subsampled and textures lossily compressed until the resulting movie is similar precision to a decent 1080p h264? Furthermore, there's a large difference between taking an existing movie and compressing it, vs designing the movie with compression in mind a la .kkrieger

[Leak]

Your idea sounds like Machinima (http://en.wikipedia.org/wiki/Machinima), which allows the scenes to be rendered at the viewer's computer, although large-scale distribution is usually done with video files.
Made me think more of demos, actually - but Debris (http://www.pouet.net/prod.php?which=30244) would probably still take the cake at 177kB download vs. 197MB hi-res video... :D

np: Battles - Atlas (Mirrored)

[God Radio]

Even assuming you can extract the models and textures entirely from the remaining frames with no sideband data, and even assuming motion can be perfectly interpolated, that still only cuts the bitrate by about 20%, because you cut B-frames and keep I/P-frames. So this line of extrapolation doesn't go anywhere. Lets try from the other end:
How much space does it take to store the models, textures, and motion of a modern 3D CGI movie? Well, that wouldn't be very helpful either, as those probably don't involve any compression, and are designed to render at very high resolution. How much space does it take to store the models, textures, and motion of a modern 3D CGI movie, if models are subsampled and textures lossily compressed until the resulting movie is similar precision to a decent 1080p h264? Furthermore, there's a large difference between taking an existing movie and compressing it, vs designing the movie with compression in mind a la .kkrieger




Ahhh I think that I understand at last(maybe?):

Everyone is focused on a single do everything tool.

What I am trying to say is that when a wireframe is used then we go for a multiple tool set when doing the renders to optimize bandwith.

For example an adjustable wrench is never going to outperform a dedicated set of tools.

The same goes for focusing on a single codec to do everything.

Can we try four or six codecs(open source) with each optimized for a single purpose so that using all of them we get better performance?

If we had a codec just for I/P frames and another codec just for B frames we get much better performance.

I am sorry but my explanations are not too good.............hmmm let me try and post an example:

http://www.businesswire.com/portal/site/home/news/sections/?ndmViewId=news_view&newsLang=en&newsId=20080410005215

Broadcast International has patented the unique CodecSys approach of using multiple expert codecs rather than single, general-purpose codecs, which are used in all other video encoding solutions. By automatically switching between specialized codecs, choosing the best one for a particular frame or video sequence, CodecSys is able to reduce video bandwidth requirements more than 80% for HD video over all types of delivery platforms including broadcast, cable, IPTV, wireless and the Internet.

[akupenguin]

What I am trying to say is that when a wireframe is used then we go for a multiple tool set when doing the renders to optimize bandwith.
If multiple tools are good, then put them all in the same codec, and let them switch per block or even use all at the same time (as an overcomplete representation if there's no better way to reconcile them).
This is a perfectly valid technique, and is used in e.g. Bink with its 16 different coding algorithms. The result was decent and fast, but nothing special even when compared against others in its own generation.

If we had a codec just for I/P frames and another codec just for B frames we get much better performance.
The only difference between a B-frame and a P-frame is that one is motion interpolation and the other is motion extrapolation. How could there possibly be a coding method that's good at one and not at the other?
Intra I can see, though anything that's good at I-frames should also be good at large intra regions in P-frames if you can reconcile the transitions to inter.

Broadcast International has patented the unique CodecSys approach of using multiple expert codecs rather than single, general-purpose codecs, which are used in all other video encoding solutions. By automatically switching between specialized codecs, choosing the best one for a particular frame or video sequence, CodecSys is able to reduce video bandwidth requirements more than 80% for HD video over all types of delivery platforms including broadcast, cable, IPTV, wireless and the Internet.

It is generally safe to assume that any claim of "revolutionary" video compression improvements is bullshit.
If you need a reason for that, consider: a 1.5x improvement in compression is more than enough to publish; that's maybe as much H.264 gains over ASP (number pulled out of thin air, but you get the idea). So no company would wait until they had a 5x improvement before they started up the hype engine. You never heard of them before, right? Ergo, they don't really have a 5x improvement, either they have cooked tests which don't correctly represent their codec, or they have nothing at all and are just trying to scam people and run. Both of which are very common; we see articles like this every month or so.

[MfA]

Investment reporting is so disgusting, they are completely spineless. A company which hasn't once released R/D data for it's compression compared to the industry standards and none of them call them out on it.

As long as you have enough impressive suits, stupid investors and customers tech reporting will praise you into high heaven (giving rise to more stupid investors and customers). Kind of depressing they got IBM PHBs to pour money into it, they were desperate for a high profile Cell application I guess ... and well, codecsys was getting great press! Sigh.

[benwaggoner]

Broadcast International has patented the unique CodecSys approach of using multiple expert codecs rather than single, general-purpose codecs, which are used in all other video encoding solutions. By automatically switching between specialized codecs, choosing the best one for a particular frame or video sequence, CodecSys is able to reduce video bandwidth requirements more than 80% for HD video over all types of delivery platforms including broadcast, cable, IPTV, wireless and the Internet.
Yeah, I saw their demo at NAB last year (didn't bother this year).

Their demos weren't even DEINTERLACED. They were encoding interlaced as progressive! And they had obvious I-frame popping on codec transitions (as you'd expect)!

And it's hard to think how this could even kind of work. Maybe there's been a 500% improvement in codecs from H.261 to H.264 High Profile, MAYBE. From Cinepak, sure. But there's never a 500% difference between current generation codecs. Let's say we're comparing VP7, VC-1, RV10, and H.264 (current decent codecs with real-time implementations). One can imagine in some cases that one might do a little better than the other with the same content. But if you're going to do parallel encoding with stream switching, you could also just encode with different parameters and pick the one with the best SSIM or something. If I was doing this in VC-1, I might try Adaptive Deadzone Off/Conservative/Aggressive, A few different B-frame patterns, Overlap On/Off, a few DQuant modes. With H.264, some different loop filter strengths would be an obvious first step. Of course, when you figure out what options are useful there, you can probably get most of that heuristically by doing a lookahead encode, or a subsampled first pass.

And productizing the decoder? Ugh. Existing decoder ASIC's can't even switch between encoded frame sizes on the fly correctly, no way they'd be able to do that. So it'd be a PC-only solution.

[IgorC]

The other important thing is metrics.
Maybe H.264 and ASP could be improved for visual quality even more if there will be more realistic metric then SSIM and already out of game PSNR. Many times developers just almost blindly optimized their codec to metrics values because psycho algos could requires public test which are expensive.


Just looking at x264's development and I see how useless those SSIM values are especially at high bitrates when it comes to psychovisual improvements (VAQ, film grain optimization etc.)

[MfA]

Metrics are never going to be much use for grain ... whether it's desirable to have noise which is only approximately in the right place or whether to simply remove it (saving bits which can be used elsewhere in the process). So much room for personal preference there, some people care more than others for artist intent too.

About personal preference ... in one paper I read the other day they actually found two distinct classes of people in their testing of H.264's deblocking. Even with the relatively advanced loop filter in H.264 some people still preferred it off (maybe there should be a switch in h.264 decoders to output frames before the deblocking is applied?).

[Manao]

maybe there should be a switch in h.264 decoders to output frames before the deblocking is applied?There already is. But not a lot of decoder supports it (ffmpeg doesn't for example)

[MfA]

Hmm, I didn't mean a bitstream flag ... simply a tick button in for instance ffdshow to output the frame first and only then apply the loop filter (I'm not very quality conscious either way though, so it would be lost effort for me).

[benwaggoner]

About personal preference ... in one paper I read the other day they actually found two distinct classes of people in their testing of H.264's deblocking. Even with the relatively advanced loop filter in H.264 some people still preferred it off (maybe there should be a switch in h.264 decoders to output frames before the deblocking is applied?).
The H.264 loop filter is arguably too wide (in terms of number of pixels touched), especially with 4x4 blocks, and yields too much softening.

The wide For moderate-high bitrates, the narrower looper VC-1 is better at preserving detail, although it'll allow more blocking to be visible at lower bitrates without postprocessing filters (hence the postprocessing filter use in WMP at low bitrates).

[Dark Shikari]

The H.264 loop filter is arguably too wide (in terms of number of pixels touched), especially with 4x4 blocks, and yields too much softening.Its only wide at low bitrates... where one needs it to be wide.

I agree its slightly too strong for 4x4 blocks, but VC-1's filter is so weak as to be nearly useless; it is certainly not a good example to point to as a "good" deblocker!

[benwaggoner]

Its only wide at low bitrates... where one needs it to be wide.

I agree its slightly too strong for 4x4 blocks, but VC-1's filter is so weak as to be nearly useless; it is certainly not a good example to point to as a "good" deblocker!
I think nearly useless is a bit of an exaggeration, but no argument that H.264 helps low bitrates degrade into more visually acceptable softness instead of blockiness. The VC-1 Overlap filter can help quite a bit in cases where the Loop filter isn't enough. And of course, there's the option of postprocessing which most lower bitrate VC-1 decoding gets access to.

[*.mp4 guy]

Its only wide at low bitrates... where one needs it to be wide.

I agree its slightly too strong for 4x4 blocks, but VC-1's filter is so weak as to be nearly useless; it is certainly not a good example to point to as a "good" deblocker!

Too weak is often a better compromise then too strong in many respects. Especially considering that any kind of deblocking can do nothing but remove information from the video, you can't get back the information that was discarded, which caused the blocking to occur, all any deblocker ever does is blur the picture and hope that the blur is less obvious then the blocks were. Inloop deblocking however can be use full in that it increases coding efficiency, but as far as actually improving the picture, its really just a question of preference: to blur or to block.

[Dark Shikari]

Too weak is often a better compromise then too strong in many respects. Especially considering that any kind of deblocking can do nothing but remove information from the video, you can't get back the information that was discarded, which caused the blocking to occur, all any deblocker ever does is blur the picture and hope that the blur is less obvious then the blocks were. Inloop deblocking however can be use full in that it increases coding efficiency, but as far as actually improving the picture, its really just a question of preference: to blur or to block.There's an SEI message specifically for the purpose of displaying the image before deblocking is done.

Nevermind that fact, the primary cause of deblocking being too strong is encoders using overly high quantizers, rounding up coefficients to try to simulate grain/detail when they should be using a lower quant instead, and then wondering why their fake detail is blurred so much :rolleyes:

This is one of the primary benefits of AQ...

[benwaggoner]

...now if only there was a codec implementation that could combine a narrower loop filter with an adaptive deadzone and differential quantization, it'd be able to preserve detail at higher quants and hence lower bitrates :)...

[Dark Shikari]

...now if only there was a codec implementation that could combine a narrower loop filter with an adaptive deadzone and differential quantization, it'd be able to preserve detail at higher quants and hence lower bitrates :)...Higher quants do not necessarily mean lower bitrates. The only kind of detail you'll get at high quantizers is "fake" detail which can be far better generated using something like film grain modelling.

If you want more detail in a block, lower the damn quantizer. If its using too many bits, your encoder isn't efficient enough.

[benwaggoner]

Higher quants do not necessarily mean lower bitrates. The only kind of detail you'll get at high quantizers is "fake" detail which can be far better generated using something like film grain modelling.
...if you have decoders that include grain modeling. But getting new decoder features deployed is extremely expensive. And if if it's there, it's not guarantee it'll be used; I don't know that ANY HD DVD title ever shipped that used the grain modeling feature, even though it was mandatory.

Sometimes "fake detail" that gives the right perceptual experience is better than a perceptual lack of detail.

If you want more detail in a block, lower the damn quantizer. If its using too many bits, your encoder isn't efficient enough.
Well, that really depends on the context. It isn't always possible to have enough bits to get the quants low enough in lots of cases. Blu-ray, sure, but VOD, streaming, IPTV, etcetera are pretty much by definition bit-starved, so anything that can improve quality at the ragged edge of compression is a good thing.

[Dark Shikari]

Well, that really depends on the context. It isn't always possible to have enough bits to get the quants low enough in lots of cases. Blu-ray, sure, but VOD, streaming, IPTV, etcetera are pretty much by definition bit-starved, so anything that can improve quality at the ragged edge of compression is a good thing.I'm not talking about higher bitrates; bitrate is not directly dependent on quantizer.

My point is that if you have a constant bit budget, you're better off using those bits on lower quantizer blocks with higher relative lambdas than on higher quantizer blocks with lower relative lambdas. Adaptive quantization can be used to spread the quantizers accordingly. "Use a lower quantizer" does not mean "raise the bitrate." It means to raise your lambda.

If an encoder is trying to retain detail at an absurdly high quantizer by throwing bits everywhere instead of using a sane quantizer (at the same bitrate), its developers really have no right to complain about the deblocker being too strong.

[*.mp4 guy]

If an encoder is trying to retain detail at an absurdly high quantizer by throwing bits everywhere instead of using a sane quantizer (at the same bitrate), its developers really have no right to complain about the deblocker being too strong.

True, especialy considering that quantizers are only usefull as a relative value, and are completely meaningless without lots of ancillary information; cqms, deadzones, rounding thresholds, various forms of rd, block decimation/prediction decisions, relative frame biases, etc. all make quantizer values essentially meaningless on their own. Infact, the only thing that can be predicted from quantization values relatively accurately, is inloop deblocking.

[Birdy1]

Another idea:
Instead of I, P and B frames, use I, P and B macroblocks.
In quiet parts of a scene, mostly B blocks are used. In parts with much movement, mostly P blocks are used. And when something new pans in to the screen, I blocks are used.
This way, the encoder can pick the best solution per macroblock, instead of the best solution per frame (which is a compromise).

[Dark Shikari]

Another idea:
Instead of I, P and B frames, use I, P and B macroblocks.
In quiet parts of a scene, mostly B blocks are used. In parts with much movement, mostly P blocks are used. And when something new pans in to the screen, I blocks are used.
This way, the encoder can pick the best solution per macroblock, instead of the best solution per frame (which is a compromise).Already done. P-frames allow both I and P blocks, and B-frames allow all three block types.

[Birdy1]

didn't know that, thanks :)

[akupenguin]

But it raises an interesting question. P vs B adaption isn't just about block type, but rather order. Is there any sane way to use different frame orders in different parts of the frame? e.g predict the left half of frame 1 (P), then all of frame 2 (P, parts referring to 0 and parts to 1), then the right half of frame 1 (B). Easy without motion, but...

[MfA]

With more advanced motion compensation is there really any need for b-frames at all though? I've always found subsampled scanning patterns very inelegant compared to linear ones, on average they just increase the distance (and reduce the correlation) to already coded samples.

[akupenguin]

I see no significant difference in the importance of B-frames in 16x16 fullpel vs h.264 partitioned qpel or obmc. Thus I have no reason to think that further improvements in motion compensation would render B-frames obsolete.

[MfA]

Not talking about higher density motion fields, that's tangential. Instead better multihypothesis combination modes, temporal processing along the motion path to improve R/D ... etc.

PS. can H.264 use implicit weighted averaging with forward only prediction? (If so, how about x264?)

[Manao]

Implicit weights requires a bidirectionnal coding mode, thus a bpicture. But it doesn't require L0 & L1 reference to be respectively forward and backward. x264 with two refs will have a forward reference in its L1 list, so it can use implicit weight with forward only prediction.

One advantage of bframe that exists however good the MC is, is that it allows to handle sub-macroblock movements (small panning for example) perfectly : you code the appearing macroblocks only once (in the pframe), and all bframes in between can just copy & paste. Were you to code all those frames as P, you would have to put texture bits in appearing macroblocks each time.

[MfA]

You could add an extra border of blocks as overscan for pans and fill those with data from the future frames when appropriate. Pel based residual coding instead of block based would also help.

Manao, so how do I get x264 to use unidirectional b-frames?

[Manao]

By default, the first reference frame in the L1 list will be a future frame in x264. If you want to change that, you need to have a look at x264_reference_build_list() in encoder/encoder.c

You could add an extra border of blocks as overscan for pans and fill those with data from the future frames when appropriateBut if you're ready to require future frame(s) to decode the current, why not go the bframe way (which is more generic and propose an efficient bidirectionnal mode, plus useful direct interpolation for MVs) ?Pel based residual coding instead of block based would also helpIndeed, but then block based approach for residual usually works better.

[MfA]

But if you're ready to require future frame(s) to decode the current
I meant you would fill the overscan area (when appropriate) at the encoder. It would copy from the future frames, not the decoder.

[Birdy1]

Has this been done before:

Use spatial intra-frame motion vectors to code I-frames. This way the encoder can take advantage of similarities within the I-frame.

[Manao]

http://forum.doom9.org/showthread.php?p=1124809#post1124809

[Dark Shikari]

Has this been done before:

Use spatial intra-frame motion vectors to code I-frames. This way the encoder can take advantage of similarities within the I-frame.Already proposed (http://ftp3.itu.int/av-arch/video-site/0701_Mar/VCEG-AE11.zip)...

[brunogm]

Hi, I'm a computer science student, recently i found about curvelets and wedgelets (families of wavelets) and one thing that the authors claimed was that "If we´ve been using the structure of operating wavelets in the past 20 years, now we only have to add new bases to have a significant improvement"

So after looking on Dirac and Snow, i realized that if we make the Wavelet base in Snow changeable in the future it will be fast and easy to improve it as soon as new Wavelet bases are found. Also i want to help in this and make that my graduation project.

Cheers.

[MfA]

The curvelet/wedgelet/ridgelet/whatevernonwavelet-let transforms are not just variation on wavelet bases used with separable horizontal/vertical filtering like in Snow/Dirac ... the transforms are quite fundamentally different AFAICS.

[akupenguin]

Already proposed (http://ftp3.itu.int/av-arch/video-site/0701_Mar/VCEG-AE11.zip)...
2x2 block ESA at decode time, and this is the fast texture synthesis?

[Dark Shikari]

2x2 block ESA at decode time, and this is the fast texture synthesis?:p

You should have seen their method of avoiding coding motion vectors; they do an actual motion search and code the motion vector with respect to the one calculated by the decoder... I think they were proposing a full search too :rolleyes:

[burfadel]

:p

You should have seen their method of avoiding coding motion vectors; they do an actual motion search and code the motion vector with respect to the one calculated by the decoder... I think they were proposing a full search too :rolleyes:

Well that would reduce encoding impact I guess... but wouldn't that increase cpu use quite noticeably for 1080p content?

[akupenguin]

Well that would reduce encoding impact I guess... but wouldn't that increase cpu use quite noticeably for 1080p content?
That's nothing. The hypothetical algorithm (that wasn't actually in the linked pdf) involves ESA over the whole frame. Effectively merange=infinity. You'd be lucky to decode QCIF in realtime, and CPU cost grows with the square of resolution (i.e. fourth power of width).

[bratao]

Ok, a concept from a person that known nothing about video encoding:
-The encoder/decoder have a table of all possible combination of , for example 4x4 video blocks.
That may be:2^384 possible combinations
So , for each possibility is attributed a decimal number.

The all possibility table dont need to be compiled on runtime, it can be pre-compiled and distributed with the encoder/decoder.

For a 16x16 image(16x 4x4 blocks), this unoptimized algorithm should create a gziped loseless with 200 Bytes.
A optimized jpeg encoder gzipped , cannot do the same image with less than 2000 Bytes.

So, that´s make any sense ?

[Irakli]

-The encoder/decoder have a table of all possible combination of , for example 4x4 video blocks.
That may be:2^384 possible combinations

That's simply impossible. 2^384 is approximately 3.94 x 10^115 :eek:. This will require literally infinite amount of time to implements and literally infinite amount of memory to store all these combinations.

[MfA]

Also, there is no compression ...

[squid_80]

He's talking about bits there. 2^384 = 384 bits = 48 bytes (16 pixels@24bpp). Hardly infinite. :)

I don't know the ins and outs of jpegs but I'm guessing the cause is overhead, possibly tables of some sort? Try making the same comparison with a practical resolution...

[Dark Shikari]

The idea of VLC-coding blocks of pixels (and quantizing them) is called vector quantization and is anything but a new idea.

[Sergey A. Sablin]

Ok, a concept from a person that known nothing about video encoding:
-The encoder/decoder have a table of all possible combination of , for example 4x4 video blocks.
That may be:2^384 possible combinations
So , for each possibility is attributed a decimal number.

the number of possible combinations will be (2^8)^16, if we're talking about 8-bit color depth. which is effectively 2^128. ie attributed number is 128 bit long, which is equal to the uncompressed size of 4x4 pixel block -> no compression. (and mind that distribution of probabilities of 1's and 0's in your attributed number will be more closer to uniform than in original 4x4 pixel block, so you'll hardly compress any of these numbers)

For a 16x16 image(16x 4x4 blocks), this unoptimized algorithm should create a gziped loseless with 200 Bytes.
the size will be 256 bytes - absolutely same as uncompressed. jpeg has overhead as was already pointed - start codes, tables etc. and actually it wasn't designed to compress images of such small size. try to calculate a bit bigger sizes, 1024x768 for example.

[brunogm]

Well wavelets are a diffiult thing to find information so here is a good text that explain the daubechie and the more recent wavelets for image compression.

http://www.sci.sdsu.edu/compsciwork/IIIPASI/DOCUMENTS/COURSE%20NOTES/waveletPASI-IIICPereyra.pdf

Skip the math if you prefer, the important is the explanation of families of wavelets including Daubechies and curvelets.

Maybe aimprovement on Snow cna be achieved by using more recent and detailed work on wavelets. I'm willing to help but for now i will study what i lack from the source of x264.

Thanks!