Help x264 development: blind test

[audyovydeo]

Little joke ... what say PSNR ... ???

Tu m'etonnes !
But it's no joke. When DS has gathered enough feedback data, I also want to see each clip's metrics.

cheers
a/v

[Dark Shikari]

You can measure the PSNR yourself by comparing with the BlackPearl clip source I linked earlier.

[chornobyl]

G>D>A>C>F>E>B
G - smooth, few quantisation artifacts
D - not so smooth, more q a
A - more q a
C - lot of q a
F - some grain,not much q a
E - grainy some q a
B - very grainy lot of quant artifacts
viewed on crt screen in mpc

[Sagittaire]

You can measure the PSNR yourself by comparing with the BlackPearl clip source I linked earlier.

Well I will wait the final blind test result ...

[vmrsss]

side by side binary comparison:

C>D>A>E>F>G>B

found difficult to tell (C and D) and (A and E) apart.

I have low confidence in this judgement, I am well-know for begin semi-blind.

[ChaosKing]

watched in VLC player in Realtime
best to worst:
G D C A F E B

[Tagert]

Best to worst:
G
A
D
C
F
E
B

Watched in MPC HC.
19inch LCD.
First few seconds of each clip.

[yuki]

My 2 cents, from best to worst :

D>G>A>E>C>F>B

[le_canz]

Files played with MPC-HC (b1056)
Sony Bravia KDL-32W4000 (32 inches TV connected to the PC, HDMI)

From best to worse :

D - G - C - A - F - E - B

D : Clean image, lot of nice details. Very stable.
G : Almost as good as D, but with a bit less details.
C : Clean image, not bad.
A : Seems close to C... Couldn't tell which is better.
F : Seems close to E, maybe a bit better
E : A bit of grain, artifacts from compression begin to be annoying
B : Seems to have a lot of (ugly) grain, but it generates artifacts on edges mostly, it's annoying with a large screen.

Didn't spent much time for this blind test, hope it helps :)

[Raptus]

D > G > C > A > F > E > B

G is very pleasant for having few artifacts, but retains less detail then D. I'm curious about what was tried with B and E. I can imagine that with higher bitrate and/or different source the preferences could be very different...

[juGGaKNot]

bla

d
a
c
f
g
e
b

22 1080P screen, fullscreen.

Real playback, no stop.
The same conditions when I watch video normally on CRT 17.
From best to worse

1. D - good balance between details and smooth/not-blockness
2. A - very similar to D but more details/blocks
3. C - very detailed and more blocky. Even more than D
4. G,F - G is too washed
5. E - artifacts are masked to grain
6. B - very strong grain artifacts

Same taste lol.

My rank:
D>A~G>F>C>E>B
G - have the weakest grain retention but very good stability without blocking

Almost same.

[burfadel]

No point retaining detail if you introduce artifacts, another look over it D came out on top as the jumpy blocks weren't too numerous over G, think it could be the mood your in too. The higher details are only relevant in face pics and fairly still places of the image etc, where details are very noticeable. Having a non-stable image is much worse than a slight reduction in detail.

[smok3]

burfadel: mood + type of monitor, i expect people with CRT's to prefer 'sharper' encodes.

[mikeytown2]

Dark, would you mind spilling the beans on what this test means for the future of x264? Settings, your preference, overall stats from this thread, ect...

[Dark Shikari]

Dark, would you mind spilling the beans on what this test means for the future of x264? Settings, your preference, overall stats from this thread, ect...

The winners overall were, in order, low-psy QNS, current psy-RD output from x264, and the current without psy-RD.

The losers were known before I posted the thread really--the high-psy-RD QNS tests.

Many conclusions come from this and related tests:

1. Psy-RD is not as useful at low bitrates (still useful, but not as much). This is seen because the no-psy-RD test managed to compete relatively well with the psy-RD one.
2. Psy-RD is way too strong as it currently is, but for a good reason. Without QNS, one needs to exaggerate the differences in complexity between various mode options in order to make psy-RD effective. Psy-RD is definitely not an optimal psy metric and is mostly empirical: optimizing for it with QNS shows some serious vulnerabilities.
3. QNS is incredibly powerful and win, but cannot work without a truly phenomenal psy metric.
4. One particular issue with QNS+psy-RD (not shown here) is that low frequencies can compensate for high frequencies. This is a good thing to some extent (trying to prevent it made things worse), but in some cases has catastrophic results with overly high psy-RD strengths.

In general, I'm finding that psy-RD is very powerful in practice despite having so many failings in theory--until you throw QNS at it. Then, sometimes, it performs incredibly well, and other times, it totally fails.

[Sagittaire]

letter correspondance please ...

[Dark Shikari]

Psy-RD 1.0 + QNS: B
Psy-RD 0.8 + QNS: E
Psy-RD 0.6 + QNS: F
Psy-RD 0.4 + QNS: A
Psy-RD 0.2 + QNS: D
Psy-RD 0.0 + QNS: G (practically equivalent to psy-RD 0 no QNS)
Psy-RD 1.0 no QNS: C

[Sagittaire]

G and D ... my eyes work very well ... :D

[Audionut]

Auto sensing mode. Raise or lower Psy-RD with bitrate.

edit: on a frame by frame basis.

[Dark Shikari]

Auto sensing mode. Raise or lower Psy-RD with bitrate.

edit: on a frame by frame basis.x264 already does that. It raises psy-RD based on lambda ;)

The problem isn't that psy-RD needs to be lowered at very low bitrates--it's just that it's less useful overall, probably.

[Audionut]

x264 already does that. It raises psy-RD based on lambda ;)

To what extent? If using CRF 18, could I lower Psy-RD to 0.4 and expect x264 to raise it as needed/necessary.

The problem isn't that psy-RD needs to be lowered at very low bitrates--it's just that it's less useful overall, probably.

Judging by the results here, I would have thought it was safe to assume the Psy-RD needs to be lowered on very low bitrates.

[Dark Shikari]

To what extent? If using CRF 18, could I lower Psy-RD to 0.4 and expect x264 to raise it as needed/necessary.No, it's implicit in the algorithm. The strength is an extra weighting factor.Judging by the results here, I would have thought it was safe to assume the Psy-RD needs to be lowered on very low bitrates.How can you say that given that we didn't test anything between 0.0 and 1.0? :rolleyes:

[burfadel]

I'm guessing anything above quality wise (lower crf) than say, 26 would benefit from say, QNS + psy-rd 0.5 for example?

A crf test of 24 would be good, as it is still space friendly but leans towards a more quality oriented bitrate. Just for comparison reasons.

[Audionut]

How can you say that given that we didn't test anything between 0.0 and 1.0? :rolleyes:

Psy-RD 1.0 + QNS: B
Psy-RD 0.8 + QNS: E
Psy-RD 0.6 + QNS: F
Psy-RD 0.4 + QNS: A
Psy-RD 0.2 + QNS: D
Psy-RD 0.0 + QNS: G (practically equivalent to psy-RD 0 no QNS)
Psy-RD 1.0 no QNS: C

?....

[nurbs]

Only C and G were with psy-rd and no QNS.

[Sagittaire]

OPSNR = HVS here ... ?

[Sagittaire]

OPSNR = HVS here ... ?

yes ...