can xvid be better than h.264 by better development?

[freedom999]

since i heard that h.264 is better than xvid i thought that we
will lost some of quality because we very newly are visiting some of xvid dvd players and its very hard to wait sometime for see the better quality of h.264 than xvid in dvd players that supports h.264
i want to know that is it possibe to xvid be harder development
be better than h.264?

[Kostarum Rex Persia]

Answer is: very unlikely.

[Hurricane Neddy]

since i heard that h.264 is better than xvid i thought that we
will lost some of quality because we very newly are visiting some of xvid dvd players and its very hard to wait sometime for see the better quality of h.264 than xvid in dvd players that supports h.264
i want to know that is it possibe to xvid be harder development
be better than h.264?

No. The more advanced features of the h.264 specification mean that it will be superior to XviD unless the h.264 codec being used is just a horrible implementation of the standard. There really isn't anything that could be developed for XviD that would make it better than h.264 without going beyond the mpeg-4 ASP spec which would make it no longer compliant to the standard.

[Sirber]

Answer is: very unlikely.
:goodpost: :helpful: :stupid:

but xvid can be better at high bitrates, depending on the source.
it's also easier on the hardware.

[freedom999]

thanks

[Prettz]

:goodpost: :helpful: :stupid:

but xvid can be better at high bitrates, depending on the source.
it's also easier on the hardware.
If you get better quality from Xvid at high bitrates then you must be doing something wrong with your H.264 codec (or your H.264 encoder is crappy). H.264's linear quantization means the lowest quant levels are way, way higher quality than ASP's quant 2. And remember that you can disable in-loop post-processing and prevent b-frames from being references in H.264.

[Dark Eiri]

I don't think so.
Maybe XviD AVC if it will be released, anyway.

[Teegedeck]

If you get better quality from Xvid at high bitrates then you must be doing something wrong with your H.264 codec (or your H.264 encoder is crappy). No, I think Sirber indeed is right. I've tested it extensively some weeks ago and neither disabling deblocking nor switching off high-profile features nor using high-bitrate AVC CQMs can change that x264 quantizes away filmgrain. I haven't tried disabling B-frames completely for x264, yet, but I think this is just a difference based on codec design. It won't make a difference anymore as soon as the source doesn't contain filmgrain (I believe) but today's DVDs all contain some degree of filmgrain. So I would say, yes, XviD does look better at low compression. That doesn't mean it is in any way 'better' than x264. Just different in a way that comes in helpful when you want to make 'transparent' backups of DVDs.

[Kopernikus]

If you get better quality from Xvid at high bitrates then you must be doing something wrong with your H.264 codec (or your H.264 encoder is crappy). H.264's linear quantization means the lowest quant levels are way, way higher quality than ASP's quant 2. And remember that you can disable in-loop post-processing and prevent b-frames from being references in H.264.

H.264 quantizer is not linear but exponential.

[GodofaGap]

No, I think Sirber indeed is right. I've tested it extensively some weeks ago and neither disabling deblocking nor switching off high-profile features nor using high-bitrate AVC CQMs can change that x264 quantizes away filmgrain.
Can you post examples of this? I'm not sure what you mean with filmgrain.

It seems very unlikely since H264 can relatively use much lower quants than ASP (below ASP's q1).

[Sagittaire]

There are problematic setting for grain with H264:
- Inloop for blocking fight
- partition for ringing fight

But for reproduce original noise with these setting actived, H264 can use "film grain modeling" during the encoding (x264 don't use this setting).

Anyway if you use really low quantizer then H264 will reproduce grain very well.

And don't forget that grain for ASP is not always original grain. High quantisation produce DCT residual noise even if the source is really clean without grain.

[GodofaGap]

High quantisation produce DCT residual noise even if the source is really clean without grain.
This is what I think too. Very often I think that the thing perceived as grain is just MPEG2 compression noise.

[Teegedeck]

But for reproduce original noise with these setting actived, H264 can use "film grain modeling" during the encoding (x264 don't use this setting).That would be cheating. ;) And the problem is that filmgrain makes it very hard for a codec to distinguish between 'noise' and actual detail. There are two working solutions: a) use a really good denoiser which can 'distinguish' between detail and grain, b) keep the grain. Solution a) costs a lot of time, solution b) a lot of space.Anyway if you use really low quantizer then H264 will reproduce grain very well.Not as far as I'm concerned. Not even a very-high-bitrate matrix at quantizer=2 in XviD does that. It takes that plus a lowering of XviD's VHQ and ME precision. I haven't found an equivalent setting in x264, yet.And don't forget that grain for ASP is not always original grain. High quantisation produce DCT residual noise even if the source is really clean without grain.
This is what I think too. Very often I think that the thing perceived as grain is just MPEG2 compression noise.When I talk about film grain I mean lowly quantized drama. Encoded with low-coefficient MPEG-2 CQMs. Definitely some of that noise is compression-induced but it still contains actual detail.

I prefer keeping it instead of investing the time into proper denoising. (Also, for some reason, I want my backup to look like the original.) As such film grain is a phenomenon of low-action drama, the filesaving of an XviD encode over the original still is about 60%. These sort of movies generally seem very compressible - precisely for the reason that film grain usually is dropped.

[Sharktooth]

@teegedeck:
http://www.webalice.it/f.corriga/temp/grain_example.png
x264 with custom matrices can keep grain... even at not-so-low quants...
but Noise and Grain are are 2 different things...

[Teegedeck]

I've re-tested this afternoon, switched off b-frames (a first-timer here) and deblocking and used your AVC HR matix; and yes: film grain at the same size as XviD with SixOfNine at quant=2 with 1 b-frame ME precision = 4 and VHQ=3 (meaning x264 at quantizer=16). So if you excuse me for a second I'm gonna eat my words now.

. . .

All I still need to find out is whether dropping b-frames really did make that difference.

[Sharktooth]

oh... no probs...
b-frames surely will make a difference... however they can be tweaked a bit to minimize the difference and spare some disk space and maybe an higher quantizer could be achieved without loosing too much.
The picture above is @ quant 21 and there are very few places where grain was smoothed/deblocked.
Also EQM AVC-HR is a "compromise"... there could be better matrices for keeping grain.

[Soulhunter]

Can you post examples of this? I'm not sure what you mean with filmgrain.
Sorta grainy (http://soulhunter.chronocrossdev.com/data/Scan01.jpg) / Not so grainy (http://soulhunter.chronocrossdev.com/data/Scan02.jpg)

Read also this (http://forum.doom9.org/showthread.php?t=93744), this (http://forum.doom9.org/showthread.php?p=841700#post841700) and this! (http://soulhunter.chronocrossdev.com/index.html#012)


Bye

[CruNcher]

For very high D5 Source (and yeah even good Mpeg-2 HD) most of the times you need no inloop deblocking and can very good preserve film grain (like the one thats coming from different Kodak Prints) but x264 is bad (compared to other Encoders) when the grain/noise is visible in static areas (mv 0), it skips to less and to uniform the grain/noise gets transfered into blocks (extreme shape changes) and this is percepted as heavy flickering you can see this for example in Spiderman 2 in the Caffe Scene background (closeup) Parker talks to Mary Jane. For me not the DCT Noise of ASP is the problem in Visual Quality terms, but the ringing is a bigger problem imho and that's non existant in H.264. And i would say the development cycle of H.264 is alot faster then ASP, allready from the fact that Hollywood adopted it what they didn't with ASP. Also DivX Inc is trying to go with the Hype by simulating the Sharpness of H.264 for HD content with Post Processing (Sharpening Filter @ Decoding), shows that they fight against an enemy that most likely is gonna crush them if they don't adapt fast. XviD is allready doing that with XviD AVC so DivX AVC (or DivX own Video Codec) isn't so far away i think :)

[Prettz]

No, I think Sirber indeed is right. I've tested it extensively some weeks ago and neither disabling deblocking nor switching off high-profile features nor using high-bitrate AVC CQMs can change that x264 quantizes away filmgrain. I haven't tried disabling B-frames completely for x264, yet, but I think this is just a difference based on codec design. It won't make a difference anymore as soon as the source doesn't contain filmgrain (I believe) but today's DVDs all contain some degree of filmgrain. So I would say, yes, XviD does look better at low compression. That doesn't mean it is in any way 'better' than x264. Just different in a way that comes in helpful when you want to make 'transparent' backups of DVDs.
I should have noted that I was thinking more in the theoretical with my post. x264 and Nero might not be able to beat Xvid for high-bitrate quality at the moment, but they certainly will be able to not too long from now (x264 will at least). There's nothing in H.264's technical specs that prevent it from reaching the quality level of ASP.

H.264 quantizer is not linear but exponential.
Is that true? I had heard that one of the advantages of H.264 over ASP was that it uses a linear quant scale of 1 to 64 instead of ASP's logarithmic scale of 1 to 32 (where the quality difference between quant 2, 3, and 4 is so very severe).
...because if H.264 doesn't do this like I thought it did... I think it should cause it's a great idea!

There are two working solutions: a) use a really good denoiser which can 'distinguish' between detail and grain, b) keep the grain. Solution a) costs a lot of time, solution b) a lot of space.
"Time?" What is this "time" you speak of? ;)

[foxyshadis]

Most likely the primary contributors to x264 have simply concentrated much harder on the low- and mid-rate performance of the codec, once HD-DVD and BluRay start to pick up I'm sure the top will be optimized too - I know Ateme's already been working on that for some time.

AVC's q scale is 1-51. (And 0=lossless.) How the question's answered depends on which part is defined as linear vs logarithmic: ASP is linear because coefficient[i] is divided by matrix[i]*quantizer, so its effect on the output follows a 1/x curve, close enough to exponential. AVC, lke MPEG-2, is logarithmic and divides by something like matrix[i]*log(quant), so the effect on the output is much more linear for most quants, but isn't entirely.

[Sharktooth]

i played a lot with ateme encoder and its film grain modelling capabilities. i was not satisfied though. however the FGM was standardized AFTER we get the latest beta encoder from ateme, so i suppose something is changed in newer versions but we didnt get any updates...

[Prettz]

AVC's q scale is 1-51. (And 0=lossless.) How the question's answered depends on which part is defined as linear vs logarithmic: ASP is linear because coefficient[i] is divided by matrix[i]*quantizer, so its effect on the output follows a 1/x curve, close enough to exponential. AVC, lke MPEG-2, is logarithmic and divides by something like matrix[i]*log(quant), so the effect on the output is much more linear for most quants, but isn't entirely.
Ok that makes sense. I'd heard this thing about "linear vs logarithmic" several years ago actually, and nothing about it since then, so I'd forgotten the details. I'm relieved to know I didn't just make the whole thing up myself.

[Manao]

It's linear vs exponantial, and it's matrix[i]*2^(Q/6) ( hence, the Q = Q + 6 --> size = size / 2 ). Mpeg2 allows both exponantial & linear scale.

An exponantial scale is better because lowering / raising the quantizer means lowering / raising the size by 13%, whatever the quantizer. For a linear scale, however, Q1 is twice as big as Q2, but there are hardly no difference between Q15 and Q16. Basically, the only usefull quantizers are Q1-10. The available bitrate range is quite lower with a linear scale too : Q1-Q31 has a size ratio of 31, while for AVC Q0/Q51 has a size ratio of 2^(51/6) ~ 362 ( the exponantial scale of mpeg allowed a range of 112 )

[foxyshadis]

Right, just dividing by log(q) wouldn't be that useful, now hopefully I'll just remember that formula in the future... I know you pointed it out once before.

At the high quant end it seems to end up non-linear, but I guess that's just the efficiencies of cabac when practically every coeff is 0.

[aabxx]

I'm somewhat disappointed with part 10 TBH. I encode a lot of noisy vhs material without any preprocessing apart from deinterlacing, and xvid looks clearly better than the 3 part 10-implementations I've tried (x264, nero and mainconcept). And yes, I experimented with a lot of settings, of course. I'm only talking about at mid-to-high-ish bitrates though, I've done no testing at low bitrates.

I don't doubt that part 10 on a whole is a superior codec, but the current well-known implementations just don't seem (to me anyway) very good with noisy material. It makes me wonder whether some properties of part 10 just makes it more unsuitable to encoding noisy material than part 2? Either that, or they (those implementing) have not given enough attention to noisy material... most people don't actually encode very noisy material like I do so it won't matter to them, I suppose.

[Sharktooth]

Noisy materials are problematic for part 10 coz the codec actually acts like a denoiser.
A correct encoding would require:
1- Remove noise.
2- analyze grain and store it parametrically (the encoder or some other software must support it)
3- remove grain.
4- encoding the movie
5- re-adding parametric grain (the decoder must support it)

Noise just ruins the image quality, you can remove it with filters.
While Grain is just a different thing and may have his "fashion" on a movie.

[akupenguin]

That method ignores the fact that the codec acts as a denoiser.
Better:
1- remove noise
2- remove grain
3- encode the movie
4- decode the movie and compare it to step 1. analyse the difference as parametric grain.
5- mux the grain parameters into the movie
6- add the parametric grain at playback (the decoder must support it)

[Sagittaire]

That method ignores the fact that the codec acts as a denoiser.
Better:
1- remove noise
2- remove grain
3- encode the movie
4- decode the movie and compare it to step 1. analyse the difference as parametric grain.
5- mux the grain parameters into the movie
6- add the parametric grain at playback (the decoder must support it)

"film grain modeling" for x264 perhaps ... ???

[Sharktooth]

or just an avisynth filter that outputs the grain parameters into a file so x264 can add it to the bitstream.

[Prettz]

or just an avisynth filter that outputs the grain parameters into a file so x264 can add it to the bitstream.
There's one other issue here, though. When you very effectively remove the noise and film grain from a source before encoding, the encoder is going to produce a lot of ringing artifacts around borders (and they will all be a lot more noticeable and prominent even if there isn't more of them than before).

How do you deal with this effectively without over-smoothing the picture or destroying tiny pixel-scale details that were left in the encode? The more noise you remove from the source before encoding, the more difficult this becomes.

[Manao]

When you very effectively remove the noise and film grain from a source before encoding, the encoder is going to produce a lot of ringing artifacts around bordersHuh ? since when ? It doesn't produce any more ringing. However, the ringing is more visible, because there is less noise overall.

[Prettz]

Huh ? since when ? It doesn't produce any more ringing. However, the ringing is more visible, because there is less noise overall.
Oh it sure does when you're talking about animation. With more noise left in the source, what would have been ringing in a macroblock now encodes some of the noise that was right outside the edge border. And the surrounding macroblocks have similar high-frequency coefficients. It's not entirely accurate to say that it's just now "more visible", because in this case, the high-frequency coefficients are actually encoding (noise) detail that was in the source; IMO it's not "ringing artifacts" anymore.

Maybe I'm being overly pedantic here?

[Manao]

We're speaking of 4x4 blocks here. An edge in a 4x4 block takes the whole block. Noise or no noise, it doesn't matter.

Even if the blocks were bigger, noise doesn't change in any way the strength of the ringing ( that's mathematically speaking ). However, noise hides the ringing. That's one of the thing that explains why adding noise during the postprocessing improves the video.

[Sagittaire]

For same bitrate: remove noise -> better compressibility for source -> smaller overall quantizer -> less ringing ... ???

[Prettz]

We're speaking of 4x4 blocks here. An edge in a 4x4 block takes the whole block. Noise or no noise, it doesn't matter.

Even if the blocks were bigger, noise doesn't change in any way the strength of the ringing ( that's mathematically speaking ). However, noise hides the ringing. That's one of the thing that explains why adding noise during the postprocessing improves the video.
Hmmmm. I was talking more about Xvid, but I see what you're getting at. However, the one test I've done of an anime with H.264 was using Nero 6 Recode's Normal profile encoder, which was also limited to the MPEG matrix. In that case, the same rules still ended up applying. I can easily see things turning out differently with a High profile encoder and a reasonable quant matrix.

Also, remember that this edge noise is being encoded over many frames, and motion prediction then comes into play. In the kind of anime sources I've got experience with, *I believe* you actually end up reducing all the predicted residuals when you have a bit of noise left over throughout the picture (but not too much noise).


For same bitrate: remove noise -> better compressibility for source -> smaller overall quantizer -> less ringing ... ???
With some of the anime DVDs I've ripped (*to Xvid*) this is not entirely how it works. The movies come out with P frames all quant 2, B frames all quant 3. Reducing noise improved the final quality only up to a certain point, at which point more noise removal made the overall picture quality worse.

If you want to remove even more noise, you have to soften the image as well, something I generally avoid as much as possible. Hell, when I did the anime Metropolis I had to denoise so much to get good compressability that I was forced to use MSharpen after denoising to keep the cel borders throughout the movie from being turned to mush by the encoder.

[Manao]

*I believe* you actually end up reducing all the predicted residuals when you have a bit of noise left over throughout the picture (but not too much noise)That can only happen if what you call noise is actually the source ringing itself. That ringing will get motion compensated in the source and perhaps also in the encoded picture, with the same motion vector. So, indeed, leaving it might reduce the residual, for inter pictures. But the intra one that brought the detail in would have encoded the ringing.

Any other kind of noise definitely won't improve the residual ( since it would actually amount to add random noise to the residual, that can't help )

[R3Z]

@teegedeck:
http://www.webalice.it/f.corriga/temp/grain_example.png
x264 with custom matrices can keep grain... even at not-so-low quants...
but Noise and Grain are are 2 different things...

Could you also post the original please ? I have found that at 576p x264 is unable to keep the majority of grain even at 3MBps.

[akupenguin]

However, the one test I've done of an anime with H.264 was using Nero 6 Recode's Normal profile encoder, which was also limited to the MPEG matrix. In that case, the same rules still ended up applying.
There is no "mpeg matrix" in h264.
The difference between "mpeg quant" and "h263 quant" in mpeg4asp is not a question of the matrix used (although mpeg mode allows cqm, both modes use a flat matrix by default), the difference is in the dequant algorithms. And h264's dequant algorithm is different from both the h263 and mpeg algorithms.
If you want to explicitly refer to "what h264 uses in the absence of a cqm (e.g. in main profile)", the term is the "flat matrix".

[Sharktooth]

Could you also post the original please ? I have found that at 576p x264 is unable to keep the majority of grain even at 3MBps.
I did that SS quite some time ago. I dont even remember what DVD of the serie is that frame on. I just remember the encoded file bitrate was around 1600/1700kbps and the original was very grainy.
However if you cant keep grain that measn you're using wrong settings...

[Prettz]

That can only happen if what you call noise is actually the source ringing itself.
Well of course there's lots of ringing. Deringing the source can be a big deal in ripping anime movies. But there's more high-frequency noise is on the blocks adjacent to the block that encodes an edge. And because of film noise, the values of the edge pixels themselves also fluxuate. If every block has high-frequency noise in it, the codec can get away with not cancelling it out or adding it back in the residual and just letting the left-over noise approximate whatever noise was supposed to be at the new location (even if the high frequencies describing noise are different from what should actually be there, it's indistinguishable when you watch the encoded result, and the codec seems to do a good job of taking advantage of this fact).

I was thinking of the DVD of Metropolis all while writing this paragraph. You should see how messy the picture is in that movie. Denoising and smoothing it out to a very clean picture and then adding noise at playback time would definitely not work for it.

There is no "mpeg matrix" in h264.
The difference between "mpeg quant" and "h263 quant" in mpeg4asp is not a question of the matrix used (although mpeg mode allows cqm, both modes use a flat matrix by default), the difference is in the dequant algorithms. And h264's dequant algorithm is different from both the h263 and mpeg algorithms.
If you want to explicitly refer to "what h264 uses in the absence of a cqm (e.g. in main profile)", the term is the "flat matrix".
Errrr, well, then why does Nero 6.6's Recode tell you that the quant matrix being used is "MPEG"???
Where does the matrix x264 calls "JVT" fit into this, anyway?

[Manao]

Nero 6.6 doesn't use custom quantization for h264, since it doesn't support high profile. "Mpeg" is used in the Mpeg4 encoder I guess.

As for the noise, a random noise will either reduce the edge in the residual or increase it. But the probabilities for either one of them to happen are the same. So if the noise on a macroblock actually help reduce the ringing on that macroblock, it may ( and will ) increase it on another. So I still disagree : noise reduces the impression of ringing, not the actual ringing. Hence, the most efficient way to fight ringing is to denoise during the preprocessing and to add noise on the postprocessing.

[akupenguin]

Where does the matrix x264 calls "JVT" fit into this, anyway?
The jvt matrix is just a cqm. It is mentioned in the standard, but has no special status compared to any other cqm. Actually, the standard calls it the "default" matrix, but it is not the default, so I renamed it.

[Prettz]

So I still disagree : noise reduces the impression of ringing, not the actual ringing. Hence, the most efficient way to fight ringing is to denoise during the preprocessing and to add noise on the postprocessing.
It really depends on the source, particularly with anime. And remember that I'm not arguing you shouldn't denoise, just that you might not want to denoise to the point that there is no more noise in the picture at all. Sometimes totally cleaning up the picture and still having the encoded result look decent is impossible; you have to keep a low level of noise in order to have a watchable result at your target bitrate.

One reason might be because of the source's colors and light levels; i.e. dark scenes with loads of browns and reds on cels and the backgrounds, and very soft cel borders with lots of ringing due to not enough bitrate in the DVD. You're going to be accepting massive ringing in your encode there whether you like it or not; if you remove all the noise rather than keeping a little of it, you're going to lose compressibility as the codec has to try much much harder to fix edge artifacts during motion (because everything around them is flat). Adding some random noise over this during playback is not going to hide the ringing, but it is going to drastically change the look of the picture from what you would have gotten with lighter denoising (that's bad).

[akupenguin]

if you remove all the noise rather than keeping a little of it, you're going to lose compressibility as the codec has to try much much harder to fix edge artifacts during motion (because everything around them is flat).

No, removing noise makes it easier on the codec. Always. Compare:
* codec has to encode a sharp edge surrounded by perfectly flat areas.
* codec has to encode a sharp edge surrounded by perfectly flat areas, and it also has to encode some noise.

Adding some random noise over this during playback is not going to hide the ringing, but it is going to drastically change the look of the picture from what you would have gotten with lighter denoising (that's bad).
It will change the look for the better: you get real random noise if you add it on playback, as opposed to lossily compressed noise if you leave it (or add it) before encoding.

[Prettz]

* codec has to encode a sharp edge surrounded by perfectly flat areas, and it also has to encode some noise.

Does. Not. Compute.


It will change the look for the better: you get real random noise if you add it on playback, as opposed to lossily compressed noise if you leave it (or add it) before encoding.
The type of noise you will get will be totally inappropriate to a mid-range bitrate anime encode. It will look ridiculous.
Edit: Nevermind, you took that quote out of context completely. With what I was talking about, the lossily compressed noise is there no matter what, because the source cannot be completely denoised without destroying the picture and/or making it so soft the encoder destroys it for you. Adding noise to this will not hide serious artifacts and will make your encode look rather goofy as well.

[R3Z]

I did that SS quite some time ago. I dont even remember what DVD of the serie is that frame on. I just remember the encoded file bitrate was around 1600/1700kbps and the original was very grainy.
However if you cant keep grain that measn you're using wrong settings...

Even using Insane HQ profiles in megui with 3Mbps bitrate and using a custom matrix provided by you (thankyou :)) the majority of grain is not kept even using deblocking values of -3,-3.

Is it wrong of me to post my own settings in a topic in the AVC forum to get critique ? I do search and read as many threads as i can but sometimes you just cant find an answer.

Cheers,

Jase

[Teegedeck]

I think the problem is that there is a wide variety of what we call 'grain'. Heavy grain is quite easily kept as Sharktooth demonstrated to me recently; XviD and x264 should be able to keep it without much tweaking. On the other hand there is something that I call grain that is only perceived as a slight, flickering noise - if you're talking about that sort of grain - I also find it hard to preserve.

In any case you should disable deblocking.

[Sharktooth]

I want to stress the fact the fine noise is not grain, and obviously less compression efficient codecs (MPEG-4 ASP, MPEG-2 or even MPEG-1) can keep that noise much better than AVC.
The same topics appeared when first MPEG-4 codecs (divx and xvid) appeared. People was criticizing the noise removal nature of the codec.
However noise can just be re-added during playback. While grain is much more difficult to reproduce.

[Didée]

and obviously less compression efficient codecs (MPEG-4 ASP, MPEG-2 or even MPEG-1) can keep that noise much better than AVC.
To me at least, it is not so obvious why the more efficient codec would remove more of the input signal than the less efficient codec.

[Sharktooth]

Well, the more you compress, the more (theoretically invisible but practically sometimes visible) details you remove.
It's lossy compression after all...