VAQ Blind Tests - Please vote

[Razorholt]

I put together a little blind test here (http://69.46.26.200/?ow=Razorholt&ss=BT1)

The source is from the trailer "Married life" - 1080p / 23.98fps / MOV format - that can be found here (http://www.apple.com/trailers/sony/marriedlife/). I chose that trailer because I wanted to start testing Dark Shikari's VAQ on a low motion video first.

x264 settings
--pass 2 --bitrate 640 --stats ".stats" --keyint 14 --min-keyint 2 --ref 5 --mixed-refs
--no-fast-pskip --bframes 16 --b-pyramid --b-rdo --bime --weightb --direct auto --subme 7
--analyse p8x8,b8x8,i4x4,i8x8 --8x8dct --vbv-maxrate 25000 --qcomp 0.55 --me esa --merange 32
--threads auto --thread-input --progress --no-dct-decimate

Please take a look at the pictures - as well as the close frames - and vote for the ones you like the most. Since comparing frames only isn't really a good idea I have also added the video segments.

Let's find 1001 voters for the test to be as accurate as possible.


Thank you.

- Dan

[Snowknight26]

Curious as to why you chose not to show the source.. as the whole objective of reencoding (in most cases) is to get it as close to the source as possible, no?

[Razorholt]

The source is a 1080p file that I resized (using spline64) down to 592x320.

[Mr VacBob]

Voted, but I think this UI could be better - there's a lot of options to choose from, and I can't remember which one looks the best out of 11 if I can only see one at once. There are various things you can do for that (use ranked pair voting, reduce the number of choices) but it would really help if it could just show more than one choice from the same clip at once.

I don't think it's really useful to show the source for people here, but if you want to get normal people to know what's going on you could put up the frame in the source and a low anchor (100kbit xvid), maybe.

(And you could also use it as an ABX test, just to make sure they aren't blind)

[Lele-brz]

Very good test! I voted.
My only doubt is that with so many scene changes it cannot be considered a low-motion video, or I'm wrong?

[burfadel]

I always set b-frames to 16 as well (b-frame decision is one area that x264 could be improved with, from what I've read anyways), but having a key frame interval of 14 with 16 bframes seems a little pointless? I guess its just easier setting it to 16 for everything rather than changing it?

[Inventive Software]

What I've found is that when I set a number of refs and 16 bframes, my encode slows down dramatically. I decrease the limit on bframes, and it speeds up. So some optimisation could be done there.

[burfadel]

Wouldn't it only slow down at higher b-frame settings if more b-frames are being used? I thought x264 rarely used a high number of b-frames? I just set it to 16 because it gives the encoder the most freedom.

[fields_g]

Glad you're doing this!

I'm going to take my time throughout the day to do this comparison. Like to throw some comments out.

Since there isn't a really good comparison player other there, I think I am going to rate the screenshots first, then verify with clips. I don't know javascript that well, but is there a way to have the keyboard switch the images? Much better than looking away to see what you are clicking and returning attention.

I don't know if this is a really big deal, but the images are JPEGs instead of PNGs.

Again, thanks for the effort!

[Razorholt]

I don't know javascript that well, but is there a way to have the keyboard switch the images? Much better than looking away to see what you are clicking and returning attention.
You can use TAB + Enter if you want. Note that the page I wrote could be improved. I did it in less than an hour but if someone has some good ideas to make it look and feel better -> PM please.

I don't know if this is a really big deal, but the images are JPEGs instead of PNGs.

Does it really matter? :confused:


Thanks for participating:)

-Dan

[fields_g]

The source is a 1080p file that I resized (using spline64) down to 592x320.

I just noticed that the Jpegs are 600x400. Was this intended?

Edit: Just saw your request for PMs.... will do from now on. :)

[Razorholt]

I just noticed that the Jpegs are 600x400. Was this intended?

Yes. I played each clip Fullscreen and took screenshots that I cropped.

[lexor]

I don't understand, is First/Second/Third supposed to rate which of the numbers 1-11 better overall, or is it supposed to be for each of the 3 images? Like the first image is best on 1, second is best on 3 and third is best on 4? (I haven't done the comparison, I'm just giving an example)

And when you say you took screenshots, do you mean you used Save Image in MPC or you used Print Screen. Because some of them don't have Pierce's head in the same proportion, and I'm pretty sure AQ wouldn't do that (maybe AviSynth's Image Writer would've been the best way to go here?). Either way the comparison seems skewed since comparison frames aren't the orginal size of the video, that means they were unscaled on playback, so we are rating the interpolation filter more so than AQ here.

[IgorC]

One encoded sample hardly representates a whole situation around VAQ. There should be at least 10-20 diferent samples.
The results aren't constant for diferent videos.

Before to do this kind of tests it should be well prepared:
discussed with devs, participants, include all conditions to a builded model of test, realibity of possible results etc...

[Avenger007]

I'm seeing --keyint 14 in the settings, which means an I-frame almost every half-second.

Two questions:
1) Why use a short I-frame interval on low motion video, it seems pointless and a waste of bits?
2) Would such a short I-frame interval skew the VAQ results in any way?

I would most likely use --keyint 250.

[Razorholt]

One encoded sample hardly representates a whole situation around VAQ. There should be at least 10-20 diferent samples.
The results aren't constant for diferent videos.

This test offers 3 frames + video clip. That should be enough to get an idea on what looks good and what looks less impressive to any average testers. I'm not saying that this test is perfect, I just want to get an idea on where the VAQ development stands - as far as the "perception" is concerned.

Before to do this kind of tests it should be well prepared:
discussed with devs, participants, include all conditions to a builded model of test, realibity of possible results etc...

I wanted to get started on it this year ;)

[Razorholt]

2) Would such a short I-frame interval skew the VAQ results in any way?

That doesn't skrew anything. I kept that settings because I need it for the streaming player I'm using on a private project. I can surely run the next tests with keyint set to 250 if you want. :)

[Razorholt]

I don't understand, is First/Second/Third supposed to rate which of the numbers 1-11 better overall, or is it supposed to be for each of the 3 images? Like the first image is best on 1, second is best on 3 and third is best on 4? (I haven't done the comparison, I'm just giving an example).

Each Image (11 in total) includes 3 separate frames. Compare the 3 frames from within each say Image 1 with the 3 frames within Image 2, etc... Then, select the best Image - or the best group of frames if you want. I also recommend to watch the short clips in order to confirm your choice.

[fields_g]

If we are going to be doing this sort of thing in the future, we need to test how to test! I commend Razorholt for putting time into designing a working proposal. This is a good first attempt. Complete and come back after to view results and discuss how we should proceed with further tests.

[jethro]

If we are going to be doing this sort of thing in the future, we need to test how to test! I commend Razorholt for putting time into designing a working proposal. This is a good first attempt. Complete and come back after to view results and discuss how we should proceed with further tests.

I also like this test very much. It makes it easy to compare encoding results.
But really a clarification on the site is necessary about how to vote.
(I understood the first, second and third dropdowns as 'choose 1st best, 2nd best and 3rd best frame' as it seemed most logical. Also I already made a vote before I noticed there are 3 frames to compare not just one. So I submitted a second vote, taking 3 frames of each setting into account.)

Each Image (11 in total) includes 3 separate frames. Compare the 3 frames from within each say Image 1 with the 3 frames within Image 2, etc... Then, select the best Image - or the best group of frames if you want.

Unfortunately, this isn't clear for me.
I think a good way of doing it is just ranking each frame from best to worst - sth like (maybe more points for the best looking):

(best looking)
1st - 10 points
2nd - 9 points
3rd - 8 points
...
11th - 0 points
(worst)

...and after summing up the points from every frame from each tester we would have our overall result.

[Ranguvar]

Does it really matter? :confused:
Yes, it does... JPEG is a compressed format... PNG is lossless. You don't want to re-compress each frame with JPEG, as it will look different.

Oh well. Voted.

[foxyshadis]

If it's visually lossless (like 98-100% quality on most jpeg quality scales), that's good enough for a comparison, since you're not going to be editing the file or anything.