Small, double blind test, which video do you prefer

[*.mp4 guy]

There are two videos in this zip file (http://www.mediafire.com/?5x2fwacubd9). You must choose one or the other, It doesn't matter how sure you are, just pick the one you like, no matter how minutely, more then the other. Pm me with 1- which video you prefer, and any information on why. 2- Your general impression of the comparative quality of the videos. Thats it, nice and simple.

I'm only interested in unbiased subjective feedback, so.

Do Not check the ssim, psnr, or in any other way try to analyze the files, other then viewing them. That would make your results useless. If you do check the files before watching them, and still want to participate, atleast make a note in the pm.

Do Not post your opinion in this thread, I will post the replies I get to this thread, with the names of the participants removed, and reveal any differences, if there are any, between the files, in 2 weaks or so.

As a final note, don't worry about analyzing the video minutely for hours, I would rather have more results, then few, however very well studied results are.

[RunningSkittle]

its only double blind if you dont know the settings :)

[Dark Shikari]

I could tell by the appearance of frame 24 alone (without even looking at settings or the bitstream or any stream analysis) what the encoding difference between the two streams was :p

Also, since its not a difference between the two streams: direct=none? ;)

[IgorC]

Well, don't you know *.mp4 guy? It's all about cqm :)

[cyberbeing]

I like the trailer-09227-250.mkv better. It seems a bit smoother and cleaner overall with less blocking.

The trailer-78712-250.mkv seems a bit more detailed but at the expense of more blocking artifacts.

[*.mp4 guy]

I almost forgot to update this thread, I didnt get many votes, but there is a general consensus, so some usefull conclusions can be reached.

First thank you, to everyone who sent in a reply, heres what they thought:

1:
"After a very quick watch:

1) I prefer trailer-09227-250.mkv
2) trailer-78712-250.mkv has more blocks

After watching the videos a couple of times side by side, I think these are the kind of things that I really don't like about trailer-78712-250:

[edit:link to embeded image added (http://img176.imageshack.us/img176/1892/sample01bu3.png)]
(blocky face)

Both encodes are full of blocks, the perceived quality is very similar and trailer-78712-250 seems sharper at times, but my eyes also tend to spot the blocks in trailer-78712-250 much faster (for example the blocky face) and that's more annoying about it."

2:
"I prefer trailer-09227-250.mkv. I think it is because it has less jagged edges in higher-contrast areas. This seems most noticeable along faces and jawlines. The video may also have been smoothed a bit more (not sure). (trailer-78712-250.mkv was more jagged.)

I watched trailer-78712-250.mkv first, then trailer-09227-250.mkv, and then each of them several times to help prevent myself from automatically liking the most recently viewed video.

Overall, though, I felt the bitrate was lower than I'd encode for myself."

3:
"09 is much better, with overall more accurate representation of shapes and less blurring of detail.

Unsurprisingly, after looking at the frames, my suspicions were confirmed by the stream; 78 uses a CQM, while 09 doesn't."

4:
"1- I preferred the trailer-09227-250. While the differences where somewhat hard to notice, this one looked slightly less blocky and seemed to have a few artifacts.

2 - In general, I didn't like the quality of either video all that much, both had a "Smugged" feel to them, like the bitrates where much lower then they should have been. The second video (the one I didn't prefer) had some places with fairly noticeable blocks, or jagged edges that really made me feel the other was better (The part with the balloon bike is where I noticed it the most.

Hope this helps."

5:
"I prefer:
trailer-78712-250.mkv

I find the artifacts more distracting in the other video, there seem to be more temporal 'flickering' artifacts that catch the eye."

6:
"I like the trailer-09227-250.mkv better. It seems a bit smoother and cleaner overall with less blocking.

The trailer-78712-250.mkv seems a bit more detailed but at the expense of more blocking artifacts."

obviously, 09227 was prefered over 78712, with the general overall opinion being that 78712 had more blocking and flickering, some people felt that 78712 was sharper, while one person thought it was blurier, everyone agreed that it had the most artifacts.

Replier #3 is correct, 78712 used a cqm, incidently this is why direct=none was used, as it can react differently to different matrices, and I wanted to limit the tested parameters to just the matrices.

The cqm in 78712 was an experimental one, that I was rather ambiguous about. It was inteded to have a better perceptual ringing/sharpness ratio then the standard matrix (IE, at the same sharpness, less ringing, or at the same ringing, more sharpness, as sharpness vs ringing can be adjusted via deblocking, it is not sufficient to just make a matrix that has less ringing or more sharpness then the flat matrix, it also has to have a better overall performance when taking deblocking into account). This matrix was supposed to shunt ringing artifacts into lower frequency bands, and thus look perceptualy better then the standard matrix when used with aq, by both having lower ringing around sharp transients (such as lines) which are primarily made out of high frequencies, and still retain the high quality gradients and textures from the aq, since deblocking does not have to be raised to reduce ringing.

The standard matrix is generally reguarded as optimal when considering "ringing" or "misquito noise" artifacts, the matrix used in 78712 was an attempt to make a matrix that was subjectively more optimal in this sense then the flat matrix. Unfortunately I did not get enough replies to ascertain whether it was percieved as being better in this area, though it is clear that overall it performs worse then the standard, flat matrix.

[elguaxo]

thanks for the update *.mp4 guy

The standard matrix is generally reguarded as optimal when considering "ringing" or "misquito noise" artifacts, the matrix used in 78712 was an attempt to make a matrix that was subjectively more optimal in this sense then the flat matrix.

One thing I know about my eyes: when it comes to lower bitrate encodes (where artifacts are expected), I really don't mind about some ringing or mosquito noise. So I guess that's why I didn't notice the improvement in that area when I first compared your clips.

[*.mp4 guy]

thanks for the update *.mp4 guy



One thing I know about my eyes: when it comes to lower bitrate encodes (where artifacts are expected), I really don't mind about some ringing or mosquito noise. So I guess that's why I didn't notice the improvement in that area when I first compared your clips.

Whether it offers any consistent improvement in that area, I am not entirely sure, which was the reason for this test. Usually I can tell easily enough If a matrix is doing what I want, but I've always had my doubts about this one, particularly because it may not be possible to make a matrix that consistently apears to have less ringing artifacts then the flat matrix. It does look better on some content to me then the flat matrix, but It isn't really up to the standard of the other matrices I've worked on, as far as achieving the look I was after; If it did well in a double blind test, in which I didn't participate, I was going to add it to the matrix thread, but at this point I think I'm going to have to accept that this is probably the best I can manage in this area, and it really isn't good enough.

What I'm saying, is that you probably didn't see an improvement, because for you, there wasn't one.

If people want to experiment with the matrix, thats fine, but i really can't recomend it for regular use, at this point, and I don't want to bias subjective opinion of it by giving it an endorsement, though If anyone wan;t to run more tests with it, I would be interested in their opinion.

[Dark Shikari]

I don't think matrices alone can do what you're aiming for; if you want to reduce ringing you will probably have to combine a matrix with a novel quantization method, such as QNS.

[shae]

How are QMs "designed"? Is there some tool with useful adjustment options that can show the result right away?

And since there are multiple QMs used in h264 (ASP only had one at a time, no?), isn't it even more of a problem, particularly the delta only QMs?

[*.mp4 guy]

I design cqm's using Ligh's cqm editor, and the ffdshow dct function (it aplieas a full dct->matrix->quantization->idct operation, which lets you see how different frequency weights will look on intra blocks, relatively accurately, in real-time). The ffdshow dct filter uses mpeg2/4-asp quantization routines iirc, so its not quite the same as those used in X264, but its close enough to be helpful.

[Didée]

Interesting. ffdshow's DCT filter (which is an implementation of trbarry's DCTFilter() for Avisynth, btw) operates directly on image data. The counterpart in x264 operates on delta images (residual after spatial prediction / temporal compensation). That's two completely different pairs of shoes ... different enough to make the observations of DCTFilter's results totally meaningless.
The somewhat useful approach would be to use MVTools to make a mo-comp of some previous frame to the actual, build the difference, apply DCTFilter on that difference, then add the DCT'ed difference to the initial frame. This way, the results have some significance.

[Dark Shikari]

Interesting. ffdshow's DCT filter (which is an implementation of trbarry's DCTFilter() for Avisynth, btw) operates directly on image data. The counterpart in x264 operates on delta images (residual after spatial prediction / temporal compensation). That's two completely different pairs of shoes ... different enough to make the observations of DCTFilter's results totally meaningless.Also add in the fact that H.264 doesn't even use the DCT to begin with.

[*.mp4 guy]

Interesting. ffdshow's DCT filter (which is an implementation of trbarry's DCTFilter() for Avisynth, btw) operates directly on image data. The counterpart in x264 operates on delta images (residual after spatial prediction / temporal compensation). That's two completely different pairs of shoes ... different enough to make the observations of DCTFilter's results totally meaningless.
The somewhat useful approach would be to use MVTools to make a mo-comp of some previous frame to the actual, build the difference, apply DCTFilter on that difference, then add the DCT'ed difference to the initial frame. This way, the results have some significance.

I design cqm's using Ligh's cqm editor, and the ffdshow dct function (it aplieas a full dct->matrix->quantization->idct operation, which lets you see how different frequency weights will look on intra blocks, relatively accurately, in real-time). The ffdshow dct filter uses mpeg2/4-asp quantization routines iirc, so its not quite the same as those used in X264, but its close enough to be helpful.

Also add in the fact that H.264 doesn't even use the DCT to begin with.

I design cqm's using Ligh's cqm editor, and the ffdshow dct function (it aplieas a full dct->matrix->quantization->idct operation, which lets you see how different frequency weights will look on intra blocks, relatively accurately, in real-time). The ffdshow dct filter uses mpeg2/4-asp quantization routines iirc, so its not quite the same as those used in X264, but its close enough to be helpful.


To sum up, I already knew all of that, and in fact If you both had read my post thoroughly instead of jumping to conclusions, I beleive its clear, that I am, infact, not a complete moron. I realise that spatial, and intra prediction change the nature of the image data being compressed by the transform, used in the h.264/mpeg4-avc standard, which is an integer add and subtract only aproximation of the dct, which is in turn only an aproximation of the KLT when aplied to a certain data set. I use it, along with extensive use of actual test-encodes, and it is usefull, though obviously, and I already stated this, not an exact match for what will happen in X264.

[Dark Shikari]

To sum up, I already knew all of that, and in fact If you both had read my post thoroughly instead of jumping to conclusions, I beleive its clear, that I am, infact, not a complete moron.You underlined intra, but that is actually where H.264 diverges most from previous formats--if anything, intra would be where such a tool is almost certain to be totally useless because it doesn't take into account spatial prediction. Inter is hardly different from previous formats at all, at least from a simplistic perspective, and so old-style tools are far more likely to be useful there.

Now, one could certainly make a tool that used x264 analysis functions to choose an optimal spatial prediction and then test CQMs as such, but the pure-intra DCT that tool uses basically doesn't exist in H.264.

[*.mp4 guy]

I see you replied while I was editing my previous post for clarity, I am aware of that.

[edit] To be more clear on how i use it, I use it to asses changes in frequency weights directly effect the image, to give me an idea of how to effect different quality trade-offs, IE, If I wan't to achieve X look, Do Y to Z frequencies, I don't use it in anyway as an indicator of real world performance, I usee it to try to hash out a basic concept of how to try to do a certain thing at the most basic level, and then work on refining the idea later.

[shae]

Sounds like a lot of work.

How were the default QMs in the various standards devised?

Wouldn't it be very beneficial to have a tool specifically tailored for the job? With what mp4 guy says it sounds comparable to programming with a hex editor instead of C.