Help x264 development: blind test

[Dark Shikari]

At some point in researching psy optimizations, I have to actually have people other than me look at things.

This test is at 300kbps. Note that the source is supposed to have grain here, so don't be silly and complain about the source material. Part of the test is grain retention. Of course, if you don't like the way in which the encoder retained the grain, that's a valid complaint.

Download (.mkv) (http://mirror05.x264.nl/Dark/force.php?file=./blind_test_mkv.tar)
Download (.h264) (http://mirror05.x264.nl/Dark/force.php?file=./blind_test.tar)

Instructions:

1. Don't look at any of the responses below this post until after posting your rating (I've padded the post with spaces so you don't accidentally see the next post). This is so your opinion isn't biased by seeing the ratings of others.

2. Rank the files in order from best to worst. Feel free to post specific comments about specific files. While just knowing best/worst is useful, it's most helpful if you do the full ranking.

3. You can analyze things in any way you want, but say how you did it. For example, if you just watched them all and then ranked them without freezeframing, say so. You could also do a rank of visual quality from watching, and then a second rank from freezeframing, if you wanted.

4. Don't try to pinpoint which video represents what settings; you're probably wrong and this won't help the test.

Also, note this will probably be one of many tests; I'm investigating various different psy optimizations combined with QNS, which allows me to optimize quantization for any arbitrary psy metric.










.

[Kurtnoise]

2. Rank the files in order from best to worst.
Maybe, a range between [0-10] or [0-20] might be better instead of worst/best ?




I'll post my rank later...

[Dark Shikari]

Maybe, a range between [0-10] or [0-20] might be better instead of worst/best ?That could work, you could do that in addition to the ranking.

[smok3]

mkv, playback, no stoping, order from best to worse:

A
B,C,D
G
F
E
(i wouldn't know how to rank them)

[mikeytown2]

I think the best is C
Using VLC as a player

[leeperry]

best is A, worst is B

1024*768 upscale/LSF/GrainFactory3()

[Dust Signs]

MKV playback with VLC without freezing. As I always do when watching videos, I enlarged the video to full screen (1920x1200) - I know that this also enlarges the artefacts.

From best to worse:
D
G
C
A
F
E
B (I really dislike the artefacts caused by the grain (?) in the last three)

Dust Signs

[J_Darnley]

I used DGAVCIndex and loaded the clips into an avs file, used Stack*() loaded into VDub and played and frame-stepped my way through the clips. My rankings, best to worst, are:
D - Good, smooth
G - Good, slightly smoother than above
A - Nice detail retention
C - Similar to above
F - Similar to above but more gritty with many more chess board patterns
E - Poor and similar to below but not as strong
B - Just no! Too much crap all over the place

[Conquerist]

Watched scaled to 1360*664 on a 30" LCD, rather close (~6 feet) to the screen.

Details
A: moderate mosquito noise
B: oversharpened, large "spots" grain in areas with high detail
C: sharp, hardly any mosquito noise
D: not bad, but looks too smooth (denoised) compared to the previous ones
E: blocking!
F: great retention of detail, but still noticable mosquito noise
G: no artifacts that ruin the image, but way too smooth. Looks denoised or encoded at a lower resolution.

C>D>F>G>A>E>B

[Chengbin]

IMO

A better than B, D better than A, b=e, but b and e was terrible, f was slightly better than b and e. G was great, maybe slightly better than D
I watched this on a 22'' monitor, full screen. Note I prefer a smooth image.

[Guest]

Please don't obscure the results. It's not necessary as Dark has given the warning in the first post not to scroll down. It reduces the usability of the thread for people who actually want to see what all the different opinions are!

[IgorC]

Real playback, no stop.
The same conditions when I watch video normally on CRT 17.
From best to worse

1. D - good balance between details and smooth/not-blockness
2. A - very similar to D but more details/blocks
3. C - very detailed and more blocky. Even more than D
4. G,F - G is too washed
5. E - artifacts are masked to grain
6. B - very strong grain artifacts

[Chengbin]

Please don't obscure the results. It's not necessary as Dark has given the warning in the first post not to scroll down. It reduces the usability of the thread for people who actually want to see what all the different opinions are!

The point of us doing that is to prevent people from seeing it, and if they want to see it, they can just highlight it and easily see it. It is like the scratchcards.

[Dark Shikari]

The point of us doing that is to prevent people from seeing it, and if they want to see it, they can just highlight it and easily see it. It is like the scratchcards.I agree with Neuron2; you shouldn't scroll down until you've done the test or decided not to do it ;)

[JEEB]

Oh, this was a fun but not-so-easy ride. After watching all the clips just playbacked I pretty much couldn't decide on anything - some of them had more blocking, others had less. Other than that I couldn't make any comments.

After that, I took out ffms2 with Avisynth and started comparing frame-to-frame:

C , in my opinion is the "winner" (although there is no 'winner' or 'loser' here in reality, just a bad choice of words) here for my eyes. Albeit it might be conserving less 'detail' with its blocking scheme, it still retains more than most of the others.

B is more like a spiritual comrade of the more blocking block, resulting in somewhat more blocks on some frames, and quite more detail on others, as it tries to keep the grain.

After that come E, F, A, D and G.

After a second checkup with MPC-HC I find that my opinions haven't changed much. As a note I do prefer slight blur to blocking on low-bitrate encodes, but I tried not to look at it only from such a point of view.

[Audionut]

Only watched each one once from A to G. edit: at fullscreen on a 26" at 1.5 feet.

B is ****ing horrid.
F for me did the best job of retaining details on the faces without looking like B.
C is not as good as F.
A,D,E,G are all much of a much-nes at first glance.

[Keiyakusha]

Compared via Avisynth Interleave() + Virtual Dub

G-ood
D
A-verage
C
F
E
B-ad

[DeeGee]

Just looked the videos(mkv) time after time and again and again...
(MPC - HC, shader "16-235 -> 0-255", fullscreen on 1920x1200 S-IPS panel)

I guess this is the order for me:
D: nice balance between blur and detail.
G: quite blurry, lacks detail, but at least no blocks.
A: Some blocks
E: look previous.
F: look previous.
C: blocks + "block stretching" during pan.
B: very grainy/blocky.

[Sharktooth]

it seems to me the bitrate is a bit too low for real ranking.
you know i like grain but none of the encodes were able to retain it correctly.
at this point i can only say the F, G and D encodes just look better than the others coz there are less moving blocks... but they're also the "flattest" ones (D being the flattest)
so i know in a higher bitrate scenario, encoding that way would retain much less grain than i'd like...

EDIT: ok, i messed up the letters. now it's correct (finally).

[*.mp4 guy]

A acceptable quality, impressive performance on a ddificult source, This is what the other clips are compared to
B inconsistent quality, preference for retaining noise above actual detail (a c, d, and g keep more detail imo), looks bitrate starved (IE, the metric apears sound enough, but it is at too low a bitrate to function well, If the others can scale up noise retention at high bitrates to mach this one, then it is completely outclassed)
C slightly more consistent quality then a, both artifacting/detail retention apear slightly better
D Most consistent quality yet, no strange blocking in places that don't make sense, though there is a slight increase in a certian type of blocking (hard to explain), blocking is lower overall compared to a/c, most stable picture, while it does not retain more detail then c/a, I like the way it is retained more, except for one issue, temporal detail loss, as a scene progresses, detail is gradualy lost until a "detail floor" is hit, unfortunately this floor is a bit under the detail retained relativelty consistently throughout a/c, overall the detail is about equivelent, and I like the placement of it better.
E The best way to describe this is that it is to d as b is to a, I would surmise that b/e are what you get when you turn the psy metrics strength up to 11 (sorry, couldn't resist)
F vary sharp, just barily manages to be consistent enought to be usable, unfortunately, it consistently has too many artifacts, and it suffers the temporal detail loss to a larger extent then d, perhaps only because it has a bit more to lose at times
G Very consistent, least artifacts, very bland, detail retention always just slightly less then D at its worst.

Ranking: G/D C A F E B,

G is the safe bet, and what I would use it at low bitrates, or on dificult sources, such as this, D is also fairly safe, and brings a nice extra levels of detail, its what I would use on good sources at bitrates where I can expect little/no blatant artifacting, C (or even B) is what I would use If I needed maximum grain retention, there are some sources where you just cant lose grain, and these would do very well on those, but are fairly useless for regular use due to the sacrifices made to achieve the grain retention (mostly speaking of B).

ranking on 24" trinitron CRT, fullscreen playback in MPC via overlay mixer, paused during fast movement to accurately asses blocking, watched in full motion the rest of the time.

[pelle412]

The clips A, C, D, and G are visually superior to B, E, and F. I can't rank them individually as I am unable to discern differences down to that level by watching the clips from start to finish. I did not step through frame by frame.

I do like grain retention but if it consumes too much bitrate at the expense of overall video quality then it's not preferrable to me.

[Kurtnoise]

That could work, you could do that in addition to the ranking.
ok...

btw, sorry to ask this but could we have also the source (or a lossless clip) to have a reference to compare.

[Dark Shikari]

ok...

btw, sorry to ask this but could we have also the source (or a lossless clip) to have a reference to compare.Linkage (http://www.mediafire.com/?mymhmje0iki)

[roozhou]

My rank:
A=G>D>C>F>E>B

P.S. The aspect ratio is wrong! You should have sar set to 32:27.

[Reuf Toc]

Real playback, no stop. CRT 17".

A : The better for grain retention IMO. Unfortunately some moving blocks
G : Grain/noise is gone but good details retention and good stability
D : A little too smooth but good stability. Almost no gibbs and halo
F : Too pronounced gibbs effect. Flickering of the picture due to block moving
C : Same as previous
E : Same as previous but blocking in flat area worse
B : No comment

[Emulgator]

.mkv viewed in MediaPlayerClassic+Haali+ffdshow.

Embedded PAR looks wrong to me too - but I get the idea.

80% viewed resized to fullscreen (1920x1200TFT), 20% of time viewed at 1:1.
Some freeze frame views, but I do not weigh these too high at this low bitrate...

My personal MPEG-4 artifact of main concern at this bitrate:
frozen (blocked) zones or zones moving with contours containing smeared or artificial detail.

Ranking from top to bottom:

The (to me) more useful sample group:
D > C ~ A(both C and A have quite good grain retention, but a bit more zoning artifacts than D, I feel) > G
(G=special case:the haze became blocky, the rest of G is quite good at this low bitrate)

The lesser sample group:
> F > E > B.
But at the given resolution some viewers may like to see the artificial detail.

P.S. now that I saw the source I see that a little contouring was already in the source.

[MasterNobody]

My rank:
D>A~G>F>C>E>B
G - have the weakest grain retention but very good stability without blocking

[Blue_MiSfit]

Played back on a 24" Dell 2408WFP, using stackvertical with AviSynth on media player classic with haali renderer

Here's my ranking:
Top 3:

1) D
2) F
3) A

In short I found that the other clips which tried to retain grain (and facial detail etc) failed to do so, at the expense of making flat backgrounds block, or by making motion smear too much.

Clip D seemed to be the most consistent / stable to me. Consistency more than anything else is what I look for at low bitrates. Of course, I would subsample the heck out of this source after applying mdegrain3 :)

Not enough bits to have a hope :) Interesting test though. I'd like to know what the info from these tests means to you D_S.

~MiSfit

[Mr VacBob]

G>A>D>C>E>F>B with some uncertainty in the middle.

Watched them one after the other on my (good) laptop screen in Perian. I looked for smearing/flicker in motion first (on the left at 00:03-8) and then good static grain afterwards.

[vucloutr]

just watched them, no freezeframing.

A: good, some artificial artifacts, faces show quite some detail
B: too much artificial artifacts -> ugly
C: good, like A but a little more artificial artifacts
D: faces look good, but too much artifact movement, i would prefer A,C
E: too much artificial artifacts -> ugly
F: not as much artificial artifacts as B,E but where those occure, there they occur too aggressive. (hard to describe) -> not ugly but also not good
G: best, not as much detail in faces as the other but this clip shows the least artificial artifacts and artifact movement.

dunno if it comes from playing too much Q3 ^.^
but every tiny bit of movement jumps into my eye
and if there's somewhere movement where there shouldn't be any movement
it distracts me.

B < E < F < D < C < A < G

[Amato]

I watched the videos then made screenshots of the same frame (the keyframes at 2.0s and 3.1s) to compare them (edit: most comparing was done with the screenshots).

The very strange patterns in many of them (mostly in B and E, but a limited amount in A) were unpleasant. I assume it's trying to retain the grain here but it doesn't have enough bitrate to even make a good try. Maybe that setting would work better at a higher bitrate, where it does have enough bitrate to try to retain the grain, but it didn't seem to work well here. G and C were probably the best. C had more detail but (seemingly) more distortion. A seemed to roughly have the bad parts of both G and C (distortion and lack of detail), and D was similar (but didn't seem to have as much lost detail). B seemed to be the most intense with the strange patterns and so I would rank it as the worst. E was the second worst, with the distortions not as intense as B.

From best to worst: C, G, F, D, A, E, B

[sho_t]

best D>C>F>G>E>A>B worst

I valued smoothness. The detail is usually valued, but bitrate･･･

[Marius-the-Mad]

Greetings from a lurker. :)


Well, that was interesting. :)


1. D: I like it. Not a lot of grain (only in one scene really), but much less blocky and noticeably more detailed than C and G. Good backgrounds, good details overall and the details are quite evenly distributed. The best, most detailed sample.

2. C: I almost scored it as third best. Somewhat blocky and noisy. Not enough bitrate to keep the grain - or at least it looks like it. Just dirty a bit the bitrate-starved way. Unstable, but quite acceptable. Second place.

3. G: Quite good actually, but blurry. All the details are gone, but it's reasonably stable and reasonably blockless. Alas, it's just TOO blurry - I wanted to tie C and G together, but a part of Johnny's sword just disappears near the sixth second of the video, and that's not good. We can't speak about grain retention also, but I just like this sample.

4. F: Dirty AND blurry. Tries to keep the grain, doesn't manage to do it good enough. Worse than G, because it artifacts just a bit too much (ex. the background stairs around 5-6s).

5. A: Washed out, blocky, keeps some grain.

6. E: Tries to keep the grain, but sacrifices quality elsewhere. Not good. The grain it preserves mostly looks like compression artifacts to me. :(

7. B: This has some grain, but suffers from severe detail pumping in the background (12th second - 15th second). Too unstable and too blocky.


Scoring method: I just watched them, quite a lot of times. :D I did frame by frame comparison for G vs. C and D vs. G, but only to make sure I saw what I thought I saw.

Decoder: ffdshow with normal libavcodec. Output colorspace: RGB32 with HQ YV12->RGB conversion enabled (as drivers' colorspace conversion usually bites).

[burfadel]

Looking at the samples some are definitely worse than others, but what I did is picked probably the most demanding bitrate wise part of the picture and compared them betwee the clips. Once ranked, I looked at the whole video again as a comparison and find that the results are the same.

I compared B, D, and G mostly, as they are the ones that stand out. I chose certain points in the video as looking at them, there are sections of the video where there is very small difference between the clips, so might as well choose points where there are large differences and highlight those.

The first part of the clip I chose was right at the beginning, between 00:00:01 and 00:00:02:

B is just terrible. Thats not good grain retention, its horrid to put it nicely. Its not even grain, its more like pebbles?!
G is the one I like the most. The face at the beginning is not blocky, but the picture in other areas isn't smoothed too much.
D is quite good also, seems to have more detail but of a lower quality.

B between timestamp 00:00:03 and 00:00:07 also has 'jumpy' blocks.
G at the same point has the least 'jumpy blocks
D has some 'jumpy' blocks

At timestamp 00:00:08, just at the bottom left of the post in the centre of the screen, but above the working arms (the grindstone worker thing, can't remember the name of it), there is artifacting in all clips. G produces the least artifacts at that point, clip D is also good but the point seems a little more 'oily'.

In terms of G & D, at the face shot between 00:00:12 and 00:00:14, D produced a more detailed image. However, even though G was slightly less detailed the detail was of a higher quality.

I think its a possibility that G is better at lower bitrates, at higher bitrates D may produce a better image. The reason why I hypothesise that is because D does have a higher grain retention, and its only a lower quality grain than G in these clips due to bitrate contraints.

EDIT:
I did both watching test and a frame by frame test for the above timestamps and came to the same conclusion.

[cy]

Watched several times full screen on 1600x1200 lcd pc monitor at 1-1.5 feet)
No pixel peeping or still frame comparison.

From best to worst:

A-C-D (in no particular order, can't make my mind up)
G (although a bit too smooth and lacking in detail)
F ( their faces look a bit too much like "crap" sometimes)
E (blocky...more artefacts..etc...don't like it)
B (horrid)

Edit: actually...after watching some more...i'll put G up there with A-C-D in the top 4 ...it seems the most stable.
Also, i think...at these low bitrates...good grain retention is futile..

(Btw, what a silly movie...)

[aaar9800]

Watched the clips whole and then went frame-by-frame to rate D, C and A:

D - Best.
C - Close to D, but grain looks artificially sharper.
A - Grain is lost very quickly into scenes.
G - Stable and smooth.
F - Like A, tends to lose grain, but in-frame detail also looks uneven.
E - Better than B.
B - Worst, those artifacts are just too much.

[buba king]

Without comparing stills :devil:, my untrained eyes tell me:
(From best to worst)



G
F
C D (Can't decide between tease two)
A
E
B (Moving blocks suck)


Edit:
Ok i see the other posters added more information. I have not edited my ranking!

I think the reason i ranked G as the best as it has the least amount of blocks that move around. I really hate that blocks move.

[Rumbah]

G>D>A>C>F>E>B

G is the most consistent one. Not that much details, but very calm.

D is in good balance between consistency and details.

A is ok, but suffers from the degrading face, Orlando's face is first very detailed/grainy, then it slowly decrades to smoothness

Beginning with C, the picture gets unsettled (unruhig in German, opposite to calm). With C it's still tolerable and ok to watch.

F is similar to C but with more grain/unsettled

E and B are horrible, too much ringing, the grain looks artificial, very unconsistent, after a few frames some detailed/grainy blocks get smoothed, very irritating


In general I like it smooth, especially at these low bitrates. The grain never looks like normal grain, it always looks artificial and hurts the picture. That's why I like G, it is very consistent, not that detailed, but almost no grainy blocks that turn smooth or warp around. But D does a very good job at being consistent and retain details.

Ah, I forgot, I just watched the videos several times in fullscreen decoded with ffdshow and High Quality YV12 to RGB conversion.

[JohannesL]

G is my favorite.

[fields_g]

A>D>C>F>G>E>B

Easy to pick E + B as being my least favorite. The Halos and artifacts just hurt my eyes.
A and D were easily my favorites.

[kosmonaut]

G
D
A
C
F
E
B

Nothing special done, just watched them all twice through and went with overall look.

[sumawo13]

Best to Worst.

G
A
C
D
B
F
E

[audyovydeo]

G>A>D>C>E>F>B


LCD 17"
straight viewing in alphabetical order :
1. at clip's resolution
2. at full screen

this test is particularly interesting to me because I have trouble, with my own clips, finding visual differences between crf20 and crf24 stuff. So it's a great chance to compare my visuality with a wide public.

cheers
audyovydeo

[Yoshiyuki Blade]

D.
C.
G.
A.
F.
E.
B.

I simply watched each clip in real-time. Repeatedly, and in no particular order after going in order the first time.

EDIT: Even though the ones I favor seem to have the least grain retention, the overall consistency looks more visually pleasing than the ones that attempt to preserve grain detail at this bitrate.

[Who Am I ?]

G>D>A>C>E>F>B

B sucked the most , G was the most smooth whilst having retained the most details , D isnt that far away as well but sort off wierd artefacts ( when he seath's his sword and walks towards him ) , pixelation ( well not such a good bitrate , still as compared to others ) .

[kumi]

Ranking: G > D > A > F > C > E > B

24" calibrated LCD2490WUXi
Played back with MPC. I mostly focused on face details and face stability.

[cogman]

I came up with
G
D
C
A
F
E
B
With G being the best and B being the worst.

I ranked this by look, completely subjectively. B seemed to have way too many artifacts while G seemed like a much cleaner recording. I didn't stop to check individual frames but instead just went through each film.

[Ramir Gonzales]

What do we win if we give the correct answer ? :confused:

[Sagittaire]

My eyes want in order at low quality: max temporal stability (block flicking), low blocking level, low ringing level.

My eyes for this blind test says:
G > D >> all other

all other are really under G & D with lower quality in high motion, higher ringing level, higher flicking level. A low quality I prefer definitively bluring with low blocking, ringing, flicking treadoff. Good detail level with high blocking, ringing, flicking level is simply ... how say ... beurg.

Little joke ... what say PSNR ... ???

[Comatose]

In order from best (left) to worst (right), like you asked.
D (http://img1.hugeup.com/f/04242009/4fb6ba70.png) (more detailed than G and C, but clean enough (looks more uniform than the others?)) > G (http://img1.hugeup.com/f/04242009/2e37c2e9f43.png) (too little detail, but pretty clean) > A (http://img1.hugeup.com/f/04242009/4fb29db74.png) (too much distortion) > C (http://img1.hugeup.com/f/04242009/87af56a8db.png) (too little detail and still ugly) > F (http://img1.hugeup.com/f/04242009/10afcb2.png) (too much distortion) > E (http://img1.hugeup.com/f/04242009/efa6e9fd4.png) (too much distortion) > B (http://img1.hugeup.com/f/04242009/2f188949f0.png) (too much distortion, gigantic ugly grain)

I compared frame 38, didn't watch. Afterwards I went and watched them to try and compare the temporal stability. G seemed to be the best in that regard, but D isn't too far away... and personally, I would prefer it to look like D and be more unstable than look like G and be a bit more stable. I judged the temporal stability by looking at the other guy's back (not Depp) in scene 4, when he walks backwards.