Maybe an idea to improve the blocky in x264

[huang_ch]

All,
I've met the blocks when encoding x264 these days, and did a lot of search in this forum but all I got is to either try AQ/Matrix/BlockBuster or direct saying is a weakness of all MPEG4 codec. In my opinion, AQ/Matrix/BlockBuster is all just like a patch, not a direct solution to the blocks. So I'm trying to start a thread to have some brainstorming on how could we reduce the blocks in x264.

First, let me test my understanding on how the blocks occurs, please help to check.:thanks:
To my understanding, there's a threshold in the encoder, which is used to test if an area(4x4/8x8/16x16...) contains mostly "similar" colors(chroma?luma?), the "similar" may be controlled by the threshold, so if the area is tested to be mostly "similar" it will treat it as a flat area, so this area is compressed a lot, thus, one block appears. And if nearby areas all meet this criteria they all encoded into a flat block, but if the nearby flat block is not in a same color, that's those blocks we don't want to see.

Above is my understanding of how the blocks in x264 happens, if this is right, I'm thinking of a way to at least help to reduce the blocks. The key point is the threshold, which used to test whether a block should be treated as a flat block or not. So if x264 could expose this threshold, we may tune it by ourselves by simply lowering the threshold. This may be pretty simple, but don't know how good it could be.

Another idea is a bit complex. All I hear is to add noises to cheat x264 that it is not a flat block. But by doing so, I just feel like somewhat disabling the "encode as a flat block" feature in x264, if we really want to disable it, we can simply set the threshold described above to 0, but why x264 implement such feature? To my understanding, this kind of flat block is used to compress those really flat block which only contains a small amount of noise, so those edges/bandings are incorrectly treated as the benefit target of using "encode as flat blocks" feature.

So I'm thinking of that is there any more precise flat block detection algorithm? Only detect those really flat block or flat+some noises block to encode as a flat block, but those edges/banding/gradient should not be chosen as the target to perform flat block encoding. Could the flat+noise become more flat like after perform a strong denoise filter? While the edges/banding/gradient just become a bit smooth after the strong denoise, so these two could be distinguished. Or maybe can use some patterns to detect the edges/banding/gradient, for these types have a special point is that their color difference is not random(a simple example could like: if we convert them to 2D is like 111222 or 112211, not 121121)

This is all my thoughts about the blocks in x264, willing to accept criticism. :)

[akupenguin]

Nowhere in the x264 sourcecode does there appear a line like "if(flatness < threshold)". The flattening behavior is an implicit side-effect of the quantization process. Quantization tends to make each block closer to the prediction of that block, and some of the intra prediction modes are flat. The quantization threshold is exposed (--deadzone-intra), but it affects all intra blocks whether or not they're flat.

The reason noise sometimes helps is purely psychovisual: the blocking artifacts are still there, you just don't notice them if there's some noise on top.

That said, it is possible to optimize the quantization process for reduced blocking, but it involves writing a new variant of trellis, not just tuning a number.

[Audionut]

Well the AQ built into sharktooth's builds is technically a patch. But only because it hasn't been commited.

Matrices are not patches. You will never find the perfect matrix for every encoding. Hence why you can specify your own matrix.


I'm not sure but perhaps the deadzone setting's could influence flat area blocking also. <--- edit: see akupengiun's post above.

[DarkZell666]

As a hint for you, the default CQM in x264 is literally called "flat", lol.

Normally "flat" blocks shouldn't appear visibly simply because they ARE "flat" (try spotting a black drawing on a black surface: you can't ;)). Those blocks appear visibly because x264 has mistaken a nearly-flat block for a definitely-flat block, and this is what the --no-fast-pskip option was designed to avoid (if you add this option, x264 will check a bit more precisely the flatness of some blocks).

Some of *.mp4 guy's CQM's are specifically designed to avoid blocking too: the values contained in a CQM are some parameters to the quantisation process, so they affect the visual aspect of macroblocks on the screen.

AQ is another good thing, since it tricks x264 into quantizing some areas less agressively (and those areas are precisely those who are "sensitive" to blocking: low-contrast and flat areas :p) Note: I could be wrong on this one, because the AQ patch has been revisited/rewritten by several people.

So, until know, I believe all the advice which has been given to you makes sense :p A good combination of these three "tricks" should be enough to get rid of all the blocking.

[akupenguin]

--no-fast-pskip can only help if the previous frame already encoded the area without blocking so that there's a good image to predict from. Which is kinda a circular dependency.

[shon3i]

--no-fast-pskip can only help if the previous frame already encoded the area without blocking so that there's a good image to predict from. Which is kinda a circular dependency.
I saw in some other avc encoder (i can't remeber but i think is elecard), to have no fast b/i skip option aslo. Is that possible.

Aslo i think that AQ can only help in this situations, but defintly AQ in x264 can't make a good thing.

[DarkZell666]

but defintly AQ in x264 can't make a good thing.There are different ways of implementing AQ.

Two of them consist of :
- Degrading visually negligible information a bit more than it is already (QP=QP+1)
- Enhancing typically difficult material (QP=QP-1)

x264 imposes this -1/+1 limitation (dunno if the standard is limited in this regard, sometimes +2/-2 would look better).

AQ is a good concept, but the question is "what" to distort. AQ simply means: use different QP's for different parts of the picture. Nothing says that AQ *should* work like XviD's lumi/darkmasking at all.

[akupenguin]

Haali's AQ for x264 will decrease QP by up to a factor of 2, it's not limited to +/-1. It will only decrease QP, but then ratecontrol has to increase the nominal QP to compensate, so it's still a question of taking quality away from one location and putting it in another.

otoh, the mpeg4asp standard is limited to +/-2, that's not a decision made just by XviD. (relative to the previous mb. so the absolute value can still vary between 1 and 31 within one frame, just not too quickly)

[Sagittaire]

In fact AQ and psy optimisation are differents things.

1) You can use RC with block level control or with frame level control. At this time x264 use only frame level for RC. Rate Control equation is "blurcplx^(1-qcomp)", "blurcplex" indicate the texture complexity for the frame; imply less bits for high texturing frame. You can make that for the block level too with RC equation not at the frame level but at the block level with the same equation and in this case "blurcplex" is the texture complexity for the block; imply less bits for high texturing block. This particular AQ mode from RC plugin is "spatial masking". Imagine a scene with grass (high texture complexity) and blue sky (low texture complexity), in this case AQ will raise the quantizers of the grass, thus decreasing its quality and spend more bits on the sky.


2) You can use lumimasking and darkmasking at the frame level or at the block level too. And you can make that too with the RC equation with this theorical equation for example:

(textcplx+lumicplx+darkcplx)^qcomp
high texturing, high lumi level or/and high dark level imply less bit for the frame quantisation level or the block quantisation level.


3) You can use RDO for block level quantisation decision too. Libavcodec make that for example.

[akupenguin]

x264 does block-level RC if and only if VBV is enabled.
And in case it wasn't obvious, that's "less bits than proportional to the complexity". Highly textured frames/blocks still get more bits than simple frames/blocks, just not enough to represent the extra texture at the same quality.

[BITS]

i have no idea why i'm posting because i have no idea on what you all are talking about. i'm still researching if x264 is something i need to move to from XviD, when i tried it last, it was not. But all these speculations make me feel, from a level of production/broadcast, that x264 is still a very long way off from being finalized? i know it's not yet, and still under heavy development (i guess), but when is the ETA for end users who just want confident quality out of a stream to arrive?

i see all the cli parameters, not knowing what they mean is no big deal since i can RTFM, but is now a good time to RTFM? i mean, if it the encoder is still in infancy, using it now could only be for a purely historical reason to just say "i used it back when...".

i'm am very sorry for just throwing this in this thread, so i understand if i get "flamed", but i'm very excited about x264. i haven't been this excited since XviD starting "wow'ing" me. With these daily builds (for i assume windows only) being a every day thing, i wonder to myself "well what will be available tom.?".

i don't know really. i guess with a post like this revealing certain _CURRENT_ limitations of the encoder, i feel not so excited. Don't get me wrong, i plan on in the next couple weeks sitting down and researching everything to start trying, just i'd like to have a ~estimate of what i will be seeing in results now vs. say a year from now.

Also, regardless of implementation right now, what are some of the big things left in x264 to be developed further in regards to quality that are known currently?

Again, sorry.

Also, props penguin. What you do is for free, and today i know how hard it can be to find time to do anything at all, let alone for free.

[akupenguin]

Sorry, my policy is not to speculate on future developments. I only discuss new features when they're reasonably done, or when I'm collaborating.

x264 will never be finalized, only abandoned (hopefully not soon).

[nm]

i don't know really. i guess with a post like this revealing certain _CURRENT_ limitations of the encoder, i feel not so excited. Don't get me wrong, i plan on in the next couple weeks sitting down and researching everything to start trying, just i'd like to have a ~estimate of what i will be seeing in results now vs. say a year from now.
Why would you need such an estimate? If you test x264 now and see that it is quite excellent already, then it is and you can use it. If the encoder gets even better in the future, thats good, but I don't see why it should matter that much now. And if it doesn't get better, you are free to do something about it ;)

[Sagittaire]

And in case it wasn't obvious, that's "less bits than proportional to the complexity". Highly textured frames/blocks still get more bits than simple frames/blocks, just not enough to represent the extra texture at the same quality.

imply less bits for high texturing frame (... if you compare with the constant quantizer encoding scenario)

[Off Topic]
I try to make RDO for bframe decision in x264:

Example: encoding with 3 adaptatives bframes
I compare PiPPPP, PiBPPP, PiBBPP and PiBBBP at constant quantizer for each Pi frame and I choose the best RDO profil

Good or bad idea ... ???
I will post my result on doom9.
[/Off Topic]

[Manao]

PiPPPP, PiBPPP, PiBBPP and PiBBBPYou must compare all the possibilities, i.e IPPPP, IBPPP, IPBPP, IPPBP, IBBPP, IBPBP, IPBBP, IBBBP. And that doesn't include checks with the last P transformed as a B... A trellis may solve that, but it'll be slow anyway ( with 3 bframes max, you will encode each frame 10 times ).

Moreover, if i'm not mistaken, Loren set the thresholds of his bframe decision algorithm by comparing it to such a trellis, so :
- the current decision shouldn't be that bad
- the trellis may be implemented somewhere on his computer. The fact that it didn't get commited ( if he did implement it ) would let me think it wasn't worthwhile.

Finally, there's much more to gain I think by doing a trellis on the quantizer decision ( CF, a patch somewhere on the ML ).

[Sagittaire]

You must compare all the possibilities, i.e IPPPP, IBPPP, IPBPP, IPPBP, IBBPP, IBPBP, IPBBP, IBBBP. And that doesn't include checks with the last P transformed as a B... A trellis may solve that, but it'll be slow anyway ( with 3 bframes max, you will encode each frame 10 times ).


Not if you test each consecutive I,P frame like that:

if P1 B2 B3 B4 P5 is the best
then you make next test for P5 X6 X7 X8 P9

if P1 B2 B3 P4 P5 is the best
then you make next test for P4 X5 X6 X7 P8

if P1 B2 P3 P4 P5 is the best
then you make next test for P3 X4 X5 X6 P7

if P1 P2 P3 P4 P5 is the best
then you make next test for P2 X3 X4 X5 P6

With that you test all the possibility I think ...

Libavcodec seem use something like that for bframe decision (vb_strategy) and the gain is not small for the ASP codec. For better speed decision libavcodec can downscale the source (with brd_scale).

Example with the ASP codec at q4:
Classic decision mode done 1117 Kbps and 43.43 dB
RD decision mode done 1081 Kbps and 43.46 dB
More than 3% for the size gain here ... ;-)

[Manao]

With that you test all the possibility I think ...No you don't. Suppose P1 B2 P3 B4 P5 is better than P1 B2 B3 B4 P5, but P1 B2 B3 B4 P5 is better than P1 P2 P3 P4 P5, P1 B2 P3 P4 P5 and P1 B2 B3 P4 P5. You won't ever get to test P1 B2 P3 B4 P5, since P1 B2 B3 B4 P5 will get chosen.

[akupenguin]

Libavcodec seem use something like that for bframe decision (vb_strategy) and the gain is not small for the ASP codec.
That's because libavcodec's non-rd frame type decision (vb_strategy=1) is really dumb compared to the algorithm used by xvid and x264.

Yes I used a rd trellis when developing B-adapt. But it was never a patch for x264, only a separate script that runs 10 passes with various forced frame types and then generates a fake 2pass statsfile for the final encode. (And that was before pyramid. If I did it now there would be 14 encodes of each frame.)

[Sagittaire]

That's because libavcodec's non-rd frame type decision (vb_strategy=1) is really dumb compared to the algorithm used by xvid and x264.

Yes I used a rd trellis when developing B-adapt. But it was never a patch for x264, only a separate script that runs 10 passes with various forced frame types and then generates a fake 2pass statsfile for the final encode. (And that was before pyramid. If I did it now there would be 14 encodes of each frame.)

I make little with the 100 first frames from my HPII trailer: for this part, RDO for bframes decision say that 3 bframe is the best way and by far.

I make encoding with x264:
1) actual bframe decision with 3 adaptative bframe
x264 [info]: SSIM Mean Y:0.9806651
x264 [info]: PSNR Mean Global:45.321 kb/s:531.00

2) 0 bframe
x264 [info]: SSIM Mean Y:0.9784828
x264 [info]: PSNR Mean Global:44.713 kb/s:530.37

3) 1 no adaptative bframe
x264 [info]: SSIM Mean Y:0.9814026
x264 [info]: PSNR Mean Global:45.494 kb/s:534.80

4) 2 no adaptative bframes
x264 [info]: SSIM Mean Y:0.9820615
x264 [info]: PSNR Mean Global:45.711 kb/s:528.16

5) 3 no adaptative bframes
x264 [info]: SSIM Mean Y:0.9827040
x264 [info]: PSNR Mean Global:45.811 kb/s:525.13


It's my first test with rdo decision for bframe and the result is incredible (perhaps bad example with fade for x264 ... ???). More than 16% for the size gain here.

D:\Mes dossiers\Codec\x264>x264.exe --bframe 3 --b-pyramid --b-rdo --bime --weightb --ref 5 --mixed-refs --direct auto --deblock -1:-1 --crf 22 --qcom
p 0.75 --min-keyint 100 --ipratio 1.25 --pbratio 1.30 --partitions "all" --8x8dct --me "umh" --subme 7 --no-fast-pskip --no-dct-decimate --trellis 2 -
-progress -o x264-adapt.mp4 Encodage.avs
avis [info]: 720x304 @ 25.00 fps (100 frames)
x264 [info]: using cpu capabilities MMX MMXEXT SSE 3DNow!
mp4 [info]: initial delay 2 (scale 25)
x264 [info]: slice I:1 Avg QP:20.00 size: 86 PSNR Mean Y:67.94 U:71.41 V:69.09 Avg:68.55 Global:68.55
x264 [info]: slice P:55 Avg QP:21.64 size: 4129 PSNR Mean Y:46.06 U:48.28 V:48.56 Avg:46.57 Global:46.10
x264 [info]: slice B:44 Avg QP:23.84 size: 840 PSNR Mean Y:44.15 U:46.89 V:47.26 Avg:44.81 Global:44.42
x264 [info]: mb I I16..4: 99.9% 0.1% 0.0%
x264 [info]: mb P I16..4: 17.9% 16.9% 1.1% P16..4: 46.7% 7.5% 1.7% 0.0% 0.0% skip: 8.1%
x264 [info]: mb B I16..4: 0.1% 0.0% 0.0% B16..8: 18.0% 1.1% 1.8% direct: 2.4% skip:76.5%
x264 [info]: 8x8 transform intra:44.7% inter:88.0%
x264 [info]: direct mvs spatial:95.5% temporal:4.5%
x264 [info]: ref P 93.2% 3.4% 1.7% 0.8% 0.9%
x264 [info]: ref B 93.1% 4.7% 1.4% 0.4% 0.4%
x264 [info]: SSIM Mean Y:0.9806545
x264 [info]: PSNR Mean Y:45.439 U:47.901 V:48.189 Avg:46.018 Global:45.317 kb/s:528.34

encoded 100 frames, 5.23 fps, 529.53 kb/s

D:\Mes dossiers\Codec\x264>x264.exe --bframe 3 --no-b-adapt --b-pyramid --b-rdo --bime --weightb --ref 5 --mixed-refs --direct auto --deblock -1:-1 --
crf 21.3 --qcomp 0.75 --min-keyint 100 --ipratio 1.25 --pbratio 1.30 --partitions "all" --8x8dct --me "umh" --subme 7 --no-fast-pskip --no-dct-decimat
e --trellis 2 --progress -o x264HP-3bf.mp4 Encodage.avs
avis [info]: 720x304 @ 25.00 fps (100 frames)
x264 [info]: using cpu capabilities MMX MMXEXT SSE 3DNow!
mp4 [info]: initial delay 2 (scale 25)
x264 [info]: slice I:1 Avg QP:19.00 size: 107 PSNR Mean Y:69.02 U:71.41 V:68.84 Avg:69.31 Global:69.31
x264 [info]: slice P:26 Avg QP:21.58 size: 6276 PSNR Mean Y:45.50 U:48.14 V:48.22 Avg:46.12 Global:45.92
x264 [info]: slice B:73 Avg QP:23.38 size: 793 PSNR Mean Y:44.70 U:47.53 V:47.96 Avg:45.42 Global:45.06
x264 [info]: mb I I16..4: 100.0% 0.0% 0.0%
x264 [info]: mb P I16..4: 17.6% 25.8% 4.0% P16..4: 34.0% 6.9% 2.9% 0.1% 0.0% skip: 8.6%
x264 [info]: mb B I16..4: 0.5% 0.4% 0.0% B16..8: 16.7% 0.8% 1.5% direct: 3.3% skip:76.7%
x264 [info]: 8x8 transform intra:50.2% inter:83.0%
x264 [info]: direct mvs spatial:98.6% temporal:1.4%
x264 [info]: ref P 87.2% 6.2% 3.2% 1.9% 1.6%
x264 [info]: ref B 95.6% 3.3% 0.5% 0.2% 0.4%
x264 [info]: SSIM Mean Y:0.9811857
x264 [info]: PSNR Mean Y:45.150 U:47.928 V:48.239 Avg:45.839 Global:45.318 kb/s:442.38

encoded 100 frames, 6.45 fps, 443.57 kb/s