Ateme H.264 Beta - Quality Feedback

[bobololo]

This new thread is dedicated to the quality feedback only.

Please report here everything related to quality comments about the tools, optimal settings, prefered configurations, etc. Anything that could help us to improve the encoder quality wise should come here

-- bobololo.

With the release of Beta 3, the Ateme AVC encoder is maintaining a fairly stable state and now quality tuning can really begin. As such, bobololo requested that I produce an impartial review of the preceding quality-related discussion.

Please note that in this summary I have attempted to convey the tester’s view as accurately as I could. If I have made any misinterpretations or otherwise, don’t hesitate to contact me.

Initial findings were that not only did this codec have huge potential but it has a huge advantage over all other h.264 codecs to date in that it is both fast AND maintains high quality. One user notes that Ateme AVC is more watchable than x264. Motion more fluid and natural, blocking artefacts less pronounced when in-loop deblocking not used. Link

The codec appears to be very scalable coping with a full range of bit rates and resolutions in most combinations. There were many findings of high quality at HD resolutions with surprisingly low bit rates or very good quality with high bit rates.

Some people noted that an output similar to XviD with constant Q2 with something like half the file size. Link Another user noted similar results with a HD source. Link Notice that even when really stressing the codec at 1000kbps for this HD clip, Ateme AVC still holds its own and maintains top-rank quality. Only WMV9 can keep up at this bit rate and XviD has fallen behind.

With beta 2 a comparable looking clip to ~Q3 XviD was made with Q27 in Ateme's AVC producing a bit rate saving of ~40%. Link Again with beta 2 we have some impressive quality at very high bit rates. Link And yet more comparisons made, this time between XviD Q4 with the 6of9 HVS matrix and Ateme AVC revealed that Q20 was similarly transparent with a 15-30% bit rate reduction.

Constant quant isn't really recommended however as it doesn't deliver the best quality at a certain file size which partially invalidates some further tests by Teegedeck of XviD with 6of9 HVS at ~Q4-Q5 versus "-qp 23 -adaptdeblock -deblock 3 -qual extra" show similar sized files with generally similar quality aside from a few artefacts in the AVC clip. Link

Moving on from this the lower resolution, low bit rate findings were also very spectacular. Two 300kbps encodes of the same clip at 640x256 with 48kbps HE AAC audio can still look good. The former clip uses -deblock adapt, the latter -cartoon -deblock -2. The update was made as the latter setting gave a more agreeable output to JohnV on his TFT. Link Link

Across four clips RBF found that Ateme AVC gave a more detailed output with less artefacts than VP6.2/VSS h.264/XviD at 400kbps with a resolution of 720x352. RBF had only used -qual normal for these clips, so it should be noted that -qual extra would make them look even better. Link

One user even thought that their AVC encode looked better than the original! Their source was quite blocky on some mist and as such the deblocking filter dealt with this to the users liking. In beta two we had an incredible clip at 400kbps. Link

There was a lot of feedback about various settings and options and a number of bugs and suggestions for improvement. Remembering that this IS a beta codec we all expected this. One of the main issues with beta 1 and 2 has been the b-frame skipping decision being too aggressive and producing an output with jumpy blocks and jerky motion. This has been fixed for beta 3 and will be very welcome among the testers. The problem only occurred at low bit rates and seemed to be compensated when using high bit rates.

There has been a large amount of discussion regarding the level of deblocking that looks good with certain resolutions/quantisers/bit rates and this discussion continues. Beta 1 had an adaptive mode but the strength of it was completely out of the user's hands. Beta 2 introduced a graded adaptive deblocking method to allow tuning to suit the individual. This gave a good improvement but some suspect that there is something wrong with the implementation and that it sometimes acts too aggressively when not needed and sometimes too weakly when needed. Hopefully this will be looked at soon.

Deblocking is certainly a very powerful feature and, when tweaked, will be a valuable asset to this codec and other h.264 codecs. Teegedeck has an adaptive deblocking behaviour suggestion, as different quants require different levels of deblocking to maintain good quality. Quoted values were - qp 19: none, qp 20: -3, qp 21: -2, qp 22: 0, qp 23: 3. Link

Not surprisingly, at low bit rates, increasing quality level DOES increase quality level drastically. Sometimes the best/extra quality improvements were debatable but most often extra was clearly better. At high bit rates (704x288 @ 1200kbps) one user found that -qual normal gave a better quality suggesting that it was sharper than good/best/extra.

Cartoon mode seems to work better all round, not just in cartoons, as it takes chroma information into account for various decisions. It may well be recommended to always use it if speed is not the primary consideration.

There were various comments about the rate control, both good and bad and sometimes a mix of the two. Bad quality in the last 10 seconds of a clip was noticed and it was reported by babayaga that this was to prevent an oversized output. This would not normally be noticeable in a full film situation as the end is very dark with credits but this method will be removed (due to 'unpopular demand' I suppose).

I personally noticed in my clip that low motion quality was much better than high motion quality and as such suggested an adjustment to the bias. The two-pass rate control was considered to not give constant quality according to some PSNR test comparisons made against the main rival codecs. There was a surprisingly bad quality output produced from a very high motion anime clip that was handled much better by other codecs. I suspect this is due to the aforementioned b-frame skipping but would be dealt by good rate control if the entire anime episode were to be encoded rather than just the start credits. The rate control in beta 3 has been improved so I look forward to testing this.

Hardly anyone could spot any difference between using one reference or multiple references; as such it seems to be a waste of time. Comments suggest that psycho visual level 1 helps with fades and faces and psycho visual level 2 looks about the same as psy1 but possibly slightly less quality due to removed detail. Poor quality was produced from a sharp/grainy/very detailed slow motion source with unnatural contrasts. Link

Soulhunter noticed some strange artefacts on an 8000kbps 'Shuttle' encode but these have been fixed for beta 3.

Sagittaire tested some settings with SSIM and gave his opinions on which looked best to him. Link

Finally, some blocks noticed on still areas and backgrounds but there is reduced ringing compared to XviD and good detail preservation noted. Link

All in all it's been a very busy week or so at Ateme. They seem responsive to our feedback and very helpful with any queries, also the speed of development is shocking. Most of all I would like to thank them for producing a high quality next-generation codec.

[Sagittaire]

@ superdump
waaaaouu ... good job

@ Andrey

Originally posted by Andrey
On low bitrates - definitely.
Inloop filtering produce really unexpected (in good meaning) results.

Still can not do good medium-to-high bitrate (~1Mbit) encode - too blurry. Seems that deblocking must be disabled for such a bitrates...
Will check it...

SSIM for deblock adapt (beta 2)

source HPII 640*272
~1700 Kbps with XviD q2 default setting

bitrate	          600	  800	 1000	
deblock off	75.26	79.59	83.12
adapt -6	75.46	79.68	83.11
adapt -4	75.46	79.68	83.11
adapt -2	75.47*	79.68	83.11
adapt 0	        75.84	79.76*	83.11
adapt 2	        76.45	80.21	83.24*
adapt 4	        76.96	80.75	83.57
adapt 6	        76.71	80.88	83.92

* Deblock activation treshold

Deblock for H264 is "adaptative". For low bitrate (high quant) treshold activation is low and for high bitrate (low quant) treshold activation is high. I think that for very high bitrate deblock is in practice off ...

[Sirber]

On my test clip, it's still heavily squary but it's less blicking. I can say it improved

encavc.exe -i test.avs -o test.mp4 -qual extra -rcmode 2pass -br 600000 -psy 2 -maxb 3 -cartoon -ref 3 -adaptdeblock

[RadicalEd]

Happy to report that the B-frame adjustments in beta-3 did the trick; they now work flawlessley with animation. Quality is excellent, surpassing XviD and, in a closer race, WMV9. Real10 is still king of the hill in my initial tests, but I'll get back to that later when the -deblock 6 encode is finished.

Current settings are: defaults + -qual extra -psy 2 -deblock 4 -adaptdeblock -ref 10 -cartoon -rcmode 2pass

I just tested to see what the RC was like now. It's much better.



Of course, the red line is beta 3.

[Sirber]

It seems to use less in the begining and lots more at the end.

[everwicked]

Originally posted by superdump
I just tested to see what the RC was like now. It's much better.



Of course, the red line is beta 3.

Is the filesize close to identical?

The lines are almost identical so that could mean 2 things:
- the encoder used more bits
- the coding effiency increased since beta2

Originally posted by everwicked
Is the filesize close to identical?

The lines are almost identical so that could mean 2 things:
- the encoder used more bits
- the coding effiency increased since beta2

Filesizes are almost identical.

1.0.1.17: 21.5 MB (22,647,187 bytes)
1.0.1.19: 21.6 MB (22,653,868 bytes)

I used identical settings for both encodes, and 1.0.1.17 had the b-frame fix.

[Sagittaire]

@ Superdump and everwicked



if you observe in detail H264 beta 3 is just a little below for 75% of the frames then passes largely above for the last 25%: it's simply a very better RC repartition. I think Average and Overall PSNR will be very better for beta 3 in this case. Visually the end will be also very better quality ... ;-)

Originally posted by Sagittaire
@ Superdump and everwicked



if you observe in detail H264 beta 3 is just a little below for 75% of the frames then passes largely above for the last 25%: it's simply a very better RC repartition. I think Average and Overall PSNR will be very better for beta 3 in this case. Visually the end will be also very better quality ... ;-)

Yes, yes, I can see that. I was just posting it to point out that the changed RC was working properly.

[SeeMoreDigital]

It's all getting rather exciting now.

I wish I could take part more in these tests but my main "encoding" PC is not behaving it's self properly and I have not had the time to format and start again!

Still, I wonder if bobololo could confirm whether the new NVE release will include a seeking fix for H.264/AVC in NeVideo.ax filter. And if NeAudio.ax filter will be accessible via all software players.

Plus... can we see some downloadable sample encodes in this thread please?


Cheers everyone

[bobololo]

Originally posted by SeeMoreDigital
Still, I wonder if bobololo could confirm whether the new NVE release will include a seeking fix for H.264/AVC in NeVideo.ax filter. And if NeAudio.ax filter will be accessible via all software players.

Plus... can we see some downloadable sample encodes in this thread please?

This is not a official confirmation, but I don't see any reason why the seek issue with ShowTime filters wouldn't be fixed by Ahead. I don't know when but hopefuly it should be done in the next update.

Regarding the samples, well a few clips were already posted. Did you catch them ?

[SeeMoreDigital]

Originally posted by bobololo
Regarding the samples, well a few clips were already posted. Did you catch them ?

I got just two.

One by your good self and another by plonk420


Cheers

[babayaga]

Originally posted by superdump
Yes, yes, I can see that. I was just posting it to point out that the changed RC was working properly.

There was a significant change in the RC that might help "unexpected clips" to be much better, especially at the end.

At the same time, the strengh of the servo loops has been reduced since some of you prefer to have a more constant quality at the expense of a slightly higher size error.
We expect that the size error wil be kept bellow something like 150kB but there was not enough time to test extensively.

On some "pathological clips" (much too low target bitrate) like the one provided by Sirber, the quantiser climbs up to 51 and the RC is very confused. In this case there is a large oversize/undersize.
Handling such situation is on the way :-)

[Bulletproof]

Originally posted by RadicalEd
Happy to report that the B-frame adjustments in beta-3 did the trick; they now work flawlessley with animation. Quality is excellent, surpassing XviD and, in a closer race, WMV9. Real10 is still king of the hill in my initial tests, but I'll get back to that later when the -deblock 6 encode is finished.

Current settings are: defaults + -qual extra -psy 2 -deblock 4 -adaptdeblock -ref 10 -cartoon -rcmode 2pass

I believe the current cap on -ref is 5, I think someone from nero/ateme said that in the main thread. That should probably be written in the next beta's encavc.txt

[Sagittaire]

At the same time, the strengh of the servo loops has been reduced since some of you prefer to have a more constant quality at the expense of a slightly higher size error.

Acceptable error for me: 0.5% for maxi error target size

Bitrate 500 Kbits/s 1000 Kbits/s 1500 Kbits/s 2000 Kbits/s
size    88 092 Ko   176 185 Ko   264 277 Ko   352 370 Ko
+/-        440 Ko       880 Ko     1 320 Ko     1 760 Ko

XviD    87 920 Ko   175 928 Ko   264 232 Ko   352 360 Ko ... :)
RV10    88 362 Ko   176 397 Ko   264 780 Ko   353 382 Ko ... :)
VP6     89 542 Ko   178 532 Ko   266 106 Ko   354 322 Ko ... :o
WMV9    88 362 Ko   176 482 Ko   264 550 Ko   352 730 Ko ... :)
DivX5   88 614 Ko   176 972 Ko   265 094 Ko   353 100 Ko ... :)
DivX3   88 518 Ko   176 516 Ko   264 516 Ko   352 520 Ko ... :)

PS: test in progress for H264 ...

[RadicalEd]

Originally posted by Bulletproof
I believe the current cap on -ref is 5, I think someone from nero/ateme said that in the main thread. That should probably be written in the next beta's encavc.txt

I recall it being said that 5 is the practical maximum and gains above that were insignificant, but as far as I know there's no actual limit.

[bobololo]

Originally posted by SeeMoreDigital
I got just two.
One by your good self and another by plonk420

JohnV also posted some good stuff, check at the first page of the initial beta feedback thread.

[bobololo]

Originally posted by RadicalEd
I recall it being said that 5 is the practical maximum and gains above that were insignificant, but as far as I know there's no actual limit.

The max value supported by the spec is 16. But actually values higher than 5 don't help very much.