[20-06-2005] Metric Benchmark Challenge ...

[redfordxx]

I made H.264 test encodes, I write approx info, coz I am not home and I will redo tests more exact and relevant.
Source 1440x816 anamorph WMV9Pro cca 6Mbps, blocky, noisy,2076frames
Preprocessing SmoothD deblock & denoise THEN -->1280x544
Nero 7 Premium Recode --> SSIM=cca 85 (visually more blocky)
x264 Sharktooth's v368 --> SSIM=cca 88
SSIM with luma, no deblock inloop (slow machine=2600+;-), 3bframes, cca4200kbps
quite in contrast with results on 1st page
Do you think I should make tests w/o bframes too to make it more relevant?

[EDIT] video 1:26, CPU time for playback BSP Overlay mixer: Nero 0:43, x264 0:51

[Ice =A=]

Good work!
But why should no bframes make it more relevant?

[redfordxx]

From some discussion somewhere I realized that b-frames are HVS feature and they may look good and sharp but differ from original frame siginificantly at the same time. So maybe good to know whether the b frames are the origin of the Neros lower performance

[stephanV]

B-frames mostly improve PSNR/SSIM and I'm not sure if it is (purely) an HVS feature.

[redfordxx]

B-frames mostly improve PSNR/SSIM and I'm not sure if it is (purely) an HVS feature.
Search for it, I am sure there is a discussion like "whether metrics are relevant and for which features". And Sharktooth mentioned it as a HVS like feature and I agree because reasons I mentioned post above.

[stephanV]

Aah, that famous thread. :)

I think Manao mentioned there that b-frames are usually better RD wise too and thus should improve PSNR and i guess normally SSIM too.

[redfordxx]

RD wise What is RD wise?should improve PSNR and i guess normally SSIM too.hmm I agree coz I remember some of my older experiences too (when properly set the amount)

Anyway, few days ago I thought I will choose Nero because the playback is faster (20%), according to this challenge is metrics better and read some visual opinions that it is better.
But now, 85 SSIM vs. 88 is significant difference and I should reconsider. Maybe the chart in this thread is old and x264 improved and Manao wrote Nero is not Highprofile anymore ...

[stephanV]

What is RD wise?
RD = rate distortion. Very simpy put it is bitrate/PSNR. RD optimisations are aiming to optimise that number.


But now, 85 SSIM vs. 88 is significant difference and I should reconsider. Maybe the chart in this thread is old and x264 improved and Manao wrote Nero is not Highprofile anymore ...
This chart shows Ateme's beta encoder, not the encoder in Nero Recode.

[redfordxx]

This chart shows Ateme's beta encoder, not the encoder in Nero Recode.Well I was suspicious I didn't understand the whole truth with the fact, that Nero is using Ateme now.
So, to end up my confusion, please where can I get the Ateme's encoder?

[Manao]

http://forum.doom9.org/showthread.php?p=731493#post731493

[redfordxx]

Hope this is suitable thread for this post...

I have seen often recommendations when making 3pass encoding to make (now speaking about ME) low quality 1st pass and hi qual 2nd and 3rd pass. Then I thought that maybe it is better to make 1st and 3rd hi qual and 2nd lo quality. I am running tests on Harry Potter now and I will prove the hypothesis ASAP.

Reason: (correct me if anything wrong)
IIUC the quality of encoding depends on following:
- optimally assign frame types
- optimally distribute the bitrate (curve)
- effectively use references instead of DCT

I believe
- frame types are assigned in 1st pass - nowhere else.
- quality of bitrate curve of any pass is dependent of the quality of bitrate curve in input stats file
- qual of refs etc. depends on the last pass itself

Conclusion:
I think the best quality/time ratio (provided we decided for 3pass) we can reach with following strategy:
Focus 1st pass setting on correct I,P,B frames distribution, nothing else
Focus 2nd pass setting on bitrate curve approx (forget frame types, the are already solved)
3rd pass with full quality (or maybe some advanced trade/off in settings related to frametypes distribution if there are some)
The proposed settings tweaking is for more experienced users than me who understand the x264 --- I appreciate help with your opinions.

I have read somewhere that the 3pass style "fast slow slow" makes 2.9 pass quality iin 2.1pass time. I think it is 2.8pass quality for the 2.9pass quality is reached with the "slow fast slow" sheme... just joking ;) Nevertheless, I believe (expressing with this logic) it is possible (with advanced settings) to reach the 2.8pass or so quality in less than 2pass time. It is maybe worth thinking...

[Manao]

- frame types are assigned in 1st pass - nowhere else.Wrong ( but it's a correct assertion for XviD ). Anyway, encoding quality should hardly influence frame type decision.

And also, a third pass is mainly useless as soon as the video is long enough ( long --> > 1000 frames ), so I wouldn't bother with it. The proper way is : first pass as fast as you can, second pass as slow as you can afford.

[redfordxx]

I have finished my tests:
To see the difference I made it really extreme ME dia vs. esa. I had two kinds of settings e (stands for esa) and d (stands for dia) as shown below--keyint 250 --ref 16 --mixed-refs --bframes 3 --b-bias 0 --pbratio 1.3 --b-pyramid --nf --subme 7 --b-rdo --weightb --trellis 2 --analyse all --8x8dct --me esa --merange 32
--keyint 250 --ref 16 --mixed-refs --bframes 3 --b-bias 0 --pbratio 1.3 --b-pyramid --nf --subme 7 --b-rdo --weightb --trellis 2 --analyse all --8x8dct --me dia --merange 16In following I will code the filenames with these letters like:
eee stands for 3pass file with all passes made with ME esa settings
dee stands for 3pass file with 1st pass made with ME dia settings
edx stands for 2pass file with 2nd pass made with ME dia settings

So, it was HPII without some mainly black scenes, 1000kbps,2714frames.

Assuming eee is the best we can reach, I compared all other encodes to eee with SSIM.


Here are the results:
ddx 81,45
edx 81,76
dex 83,22
eex 83,97
dde 85,79
dee 86,38
ede 88,04

So here, the middle fast makes better than first fast. Of course, this is only one example, but still...

[EDIT] Damn, now I got it: the B frames distribution is different... so this is not very relevant

[redfordxx]

frame types are assigned in 1st pass - nowhere else.
Wrong ( but it's a correct assertion for XviD ).Then on what condition it is changed? In all of my test encodings I have never seen that frametypecount changed between passes.
It changed only depending on 1st pass quant, bitrate, Bs, bias, me...

And also, a third pass is mainly useless as soon as the video is longAgree

[Manao]

Wrong ( but it's a correct assertion for XviD ).Mmm, indeed I'm wrong ( I checked the source code ). Strange, because I definitely seem to remember that frame types could be changed. So indeed, frame type depends on quality ( since ME during frame type decision is made with the ME settings chosen, on half resolution frames ). From what I read however, ref shouldn't play a part, but subme & me, merange will )

[redfordxx]

Sorry, too much information encoded in too short text...(for me)
So indeed, frame type depends on qualityQuality of what? 1st pass?
(since ME during frame type decision is made with the ME settings chosen, on half resolution frames ).Do I understand correctly, that when deciding the frame type, ReducedByTwo resolution frame is used? That implies especially important subme?
ref shouldn't play a part,
So ref can be 1 in 1st pass. In fact, it seems logical because I frame decision is important on scene change, which is two neighbour frame comparision. Bframe decision is I believe whether the previous and next frame are similar enough to create Bf. Is Bframe made only by references or DCT data also?

[DeathTheSheep]

-DivX6.1
-XviD1.1 Release1.1 build
-VP7.0.9
-AtemeHP b3

Do ya think these guys will make it into the testing arena? ;)

[IgorC]

When xvid.org will say final 1.1 it will be while Xvid 1.1 is not still realesed.
Xvid 1.1 new SSIM here result 78.02 while old Xvid beta1/2 was 77.96 and Divx6 old 77.99

Last x264 should be also checked. It's very close to Ateme beta now but still slow. There were a hundred improvements since then.


Indeed you can do yorself. Videotrailer and ssim/opsnr.dll are here for download.

[Sagittaire]

Actually I prepare another test:

video codec
- Ateme AVC MP (Nero) and HP
- x264 MP and HP
- XviD 1.1 b2
- DivX 6.1
- WMV9 VCM
- VP7 7.0.9

resolution
- portable resolution ~ 320*240
- DVD resolution ~ 720*400
- HD resolution ~ 1280*720

source
- Ice Age Trailer
- HPIV trailer
- King Kong Trailer

Quality Test
- metrix test : APSNR, OPSNR and SSIM
- screenshoot
- little blind test

Speed test
and for the first time a real speed test ...
- graph fps = fct (PSNR)
- graph fps = fct (SSIM)
IMO with very surprising result ... ;)

[redfordxx]

Actually I prepare another test:
source
- HPIV trailer
Isn't it better to test HPII for "backward compatibility"? I mean to see the improvement over time...

[temporance]

M. Sagittaire,

I like to try to replicate (parts of) codec shootouts so would like to know where you found your sources. Presumably they're downloaded HD trailers. Can you upload them somewhere like you did with the HP sources from your last comparison?

Look forward to seeing some of your results :)

[redfordxx]

Presumably they're downloaded HD trailers.
I don't know exactly the sources S. mentioned, but if interested in general, look here (http://www.highdefforum.com/showthread.php?t=6537).

[Sagittaire]

IMO the best HD source are here (http://www.apple.com/trailers/)

-> Very better quality than HD-WMV sources
-> Very better than *.TS HDTV 720p/1080i sources
-> Real Full HD progressive source

You can make that with these sources:
http://forum.surdvd.com/viewtopic.php?t=43651

[MuTeK]

Sagittaire
Please, clear PM.

[redfordxx]

Speed test
and for the first time a real speed test ...
- graph fps = fct (PSNR)
- graph fps = fct (SSIM)
IMO with very surprising result ... ;)
What is the best method to measure encoding speed (of x264) to eliminate avisynth script slow down?

Related question: Presuming, each input frame is decoded only once during the encoding process and then kept in memory until not used anymore?

[temporance]

I don't know exactly the sources S. mentioned, but if interested in general, look here (http://www.highdefforum.com/showthread.php?t=6537).
What codec was used for these source? I'm not familiar with .mov's.

[redfordxx]

What codec was used for these source? I'm not familiar with .mov's.various, they are not all the same..
usually inside the *.exe are wmv

[temporance]

Sorry, I meant to ask Sagittaire about the trailers at http://www.apple.com/trailers/ - what codec did Apple use?

[Sagittaire]

H264 for HD Quick Time trailer