VAQ 2.0 Alpha Testing

[Razorholt]

@Avenger007: Can you please include the file size?

Thanks,
- Dan

[Avenger007]

@Avenger007: Can you please include the file size?

Thanks,
- Dan

Clip 1:
huffyuv: 43,212KB
Encoded: 2,151KB +- 3KB

Clip 2:
huffyuv: 31,145KB
Encoded: 1,099KB +- 3KB

Clip 3:
huffyuv: 221,532KB
Encoded: 8,076KB +- 8KB

[Sagittaire]

Warning: ALPHA BUILD, ONLY USE FOR TESTING VAQ2!

x264.808.vaq2.modified.exe (http://files.x264.nl/VAQ2/x264.808.vaq2.modified.exe) (gcc 3.4.6 fprofiled)

General thread:
http://forum.doom9.org/showthread.php?t=130364

x264_vaq2_alpha.01.r808.diff
http://forum.doom9.org/showthread.php?p=1120023#post1120023
x264.gaussian.cplxblur.01.diff
Dark Shikari: - gaussian cplxblur: gives a tiny improvement in 2pass ratecontrol
x264_me-prepass_DeathTheSheep.01.diff
http://forum.doom9.org/showthread.php?p=1093523
x264_2pass_vbv.6.diff
http://thread.gmane.org/gmane.comp.video.x264.devel/3093/focus=3550
x264_hrd_pulldown.04_interlace.diff
- HRD and pulldown for HD compatibility, updated patch for interlacing
http://forum.doom9.org/showthread.php?p=1047919#post1047919

Link to x264 patches collected: http://files.x264.nl/x264_patches/

Warning: ALPHA BUILD, ONLY USE FOR TESTING VAQ2!

build completely buggy ... produce very bad quality and completely different result if I compare with Dark Shikary build.

[Avenger007]

build completely buggy ... produce very bad quality and completely different result if I compare with Dark Shikary build.

Do you mind giving some figures to elaborate on "very bad quality and completely different result"? :sly:

I used the alpha build for the results I posted. :cool:

[Poopoo]

Didn't Dark Shikari say that higher SSIM values don't really guaranty a better encoding ?

[Blue_MiSfit]

That's always true. Stuff like AQ and RDO tend to lower metrics, but they usually improve subjective quality. Tricky to evaluate! :)

~MiSfit

[CruNcher]

Hmm Dark is it normal that with your 777 AQ test build Multithreading doesn't work? (AMD64) and with bob0rs it works?

[Dark Shikari]

Hmm Dark is it normal that with your 777 AQ test build Multithreading doesn't work? (AMD64) and with bob0rs it works?Yes, my pthreads broke again and I'm too lazy to fix it

[Razorholt]

Can someone please confirm that there is no visual difference between mode=0 and mode=1 (whatever strength you set).

Compare pictures (http://70.86.69.186/~matrix2/compare.php)

x264
--pass 2 --bitrate 640 --stats ".stats" --keyint 14 --min-keyint 2
--deadzone-inter 10 --deadzone-intra 6 --ref 5 --mixed-refs
--no-fast-pskip --bframes 16 --b-pyramid --b-rdo --bime --weightb
--direct auto --subme 7 --analyse p8x8,b8x8,i4x4,i8x8 --8x8dct
--vbv-maxrate 25000 --qcomp 0.55 --me esa --merange 32 --threads auto
--thread-input --progress --no-dct-decimate


EDIT: Image comparison page updated for better use. Should I include SSIM?

Thanks,
- Dan

[Dark Shikari]

Can someone please confirm that there is no visual difference between mode=0 and mode=1 (whatever strength you set).Mode=0 is also known as off. :p

Just leave it on mode 2 for now!

[Razorholt]

Yes, I was aware of mode=0 being the off switch. Apparently, so is mode=1 :D

ooookay... mode=2 it is. :)

[Avenger007]

@DS:
Is there a range for quality factor (bits per pixel) that VAQ1 and VAQ2 tend to work best?
Also, is there any artifacts I can look for to tell if the qf I'm using is too low?

[Dark Shikari]

@DS:
Is there a range for quality factor (bits per pixel) that VAQ1 and VAQ2 tend to work best?
Also, is there any artifacts I can look for to tell if the qf I'm using is too low?"Quality factor" (BPP) is a basically meaningless measure. I would avoid using it for any purpose.

[Avenger007]

I speculated as much. I'm guessing it's because bitrate distribution is far from being evenly distributed across the frame even without VAQ.
But what about my second question, are there any artifacts in particular that I can look for to suggest I'm using too low bitrate for VAQ to work effectively?

[Dark Shikari]

I speculated as much. I'm guessing it's because bitrate distribution is far from being evenly distributed across the frame even without VAQ.
But what about my second question, are there any artifacts in particular that I can look for to suggest I'm using too low bitrate for VAQ to work effectively?There is no real thing as "too low a bitrate" for VAQ; VAQ will start failing at the same sort of bitrates x264 in general fails at.

[Razorholt]

@DS: I noticed no difference between Trellis 1 and Trellis 2when VAQ is on. Could it be due to VAQ, somehow?

Look at Image 8 (Trellis1) vs. Image 9 (Trellis2) (http://70.86.69.186/~matrix2/compare.php)


Thanks,
- Dan

[Dark Shikari]

@DS: I noticed no difference between Trellis 1 and Trellis 2when VAQ is on. Could it be due to VAQ, somehow?

Look at Image 8 (Trellis1) vs. Image 9 (Trellis2) (http://70.86.69.186/~matrix2/compare.php)


Thanks,
- DanOdd.

I would be surprised if I broke trellis with VAQ2. :p

[burfadel]

The picture looks the same, is the file size & SSIM the same? In this case it is an accurate comparison, as same data=same numbers! The Trellis 2 file in CRF should be fractionally smaller? The header of the file may change I believe, but the change in header alone won't change the output filesize.

[Razorholt]

The picture looks the same, is the file size & SSIM the same? In this case it is an accurate comparison, as same data=same numbers! The Trellis 2 file in CRF should be fractionally smaller? The header of the file may change I believe, but the change in header alone won't change the output filesize.

I used 2 Pass but I'll try with CRF later.

Trellis1
x264 [info]: SSIM Mean Y:0.9790550
x264 [info]: PSNR Mean Y:43.634 U:48.093 V:49.790 Avg:44.499 Global:41.274 kb/s:643.04

encoded 3433 frames, 2.59 fps, 643.66 kb/s
desired video bitrate of this job: 640 kbit/s - obtained video bitrate (approximate): 645 kbit/s

Trellis2
x264 [info]: SSIM Mean Y:0.9794983
x264 [info]: PSNR Mean Y:43.761 U:48.219 V:49.940 Avg:44.628 Global:41.405 kb/s:643.12

encoded 3433 frames, 2.34 fps, 643.74 kb/s
desired video bitrate of this job: 640 kbit/s - obtained video bitrate (approximate): 645 kbit/s

[Razorholt]

Is it me or on the screenshot 2 (http://70.86.69.186/~matrix2/compare.php?ss=2) the No AQ looks better? :confused:

[Avenger007]

Is it me or on the screenshot 2 (http://70.86.69.186/~matrix2/compare.php?ss=2) the No AQ looks better? :confused:

Yes, that appears true. All I can say is: try a lower strength or look at frames close to it.

[Razorholt]

Yes, that appears true. All I can say is: try a lower strength or look at frames close to it.

Didn't you mention earlier that "Lower strength values (0.75) generally don't seem to help much.".

Anyway, I'm encoding @ strength=0.5 and strength=0.75 as you suggested. Let's see...

[Dark Shikari]

Is it me or on the screenshot 2 (http://70.86.69.186/~matrix2/compare.php?ss=2) the No AQ looks better? :confused:AQ has a tendency to move bits between frames. As a result, single-frame comparisons are often totally worthless.

[Avenger007]

So does lower strength translate into less bits moved across frames or is there no significant correlation?

[Razorholt]

OK! My bad on Trellis... it was a glitch :D

I also uploaded Previous/Next Frames: check again (http://70.86.69.186/~matrix2/compare.php?ss=2)

[burfadel]

That make sense, since the PSNR & SSIM are higher, hence that actual data stream is different with your Trellis 2 settings!

[stax76]

In case anybody is looking for a GUI, StaxRip has now AQ controls.

http://forum.doom9.org/showthread.php?p=1121319#post1121319

http://planetdvb.net/non_drupal/forum_images/greenshot_2008-04-04_14-12-24.png

[Sharktooth]

Under Rate control? Wrong place or name. It should be (in) Quantization.

[stax76]

It's under rate control documented. :confused:

[Sharktooth]

Maybe coz it actually "moves bits". Well, then i guess it's ok to leave it under rate control.

[Razorholt]

Yes, that appears true. All I can say is: try a lower strength or look at frames close to it.

Check the Image 7 and Image 8 (http://70.86.69.186/~matrix2/compare.php?ss=2)for lower strength.

The No AQ (Image 1) still looks better to me :( - Even by comparing the close frames. Why is that?

[IgorC]

Compare frame to frame is useless.

Why?

AQ has a tendency to move bits between frames. As a result, single-frame comparisons are often totally worthless.

[Avenger007]

Check the Image 7 and Image 8 (http://70.86.69.186/~matrix2/compare.php?ss=2)for lower strength.

The No AQ (Image 1) still looks better to me :( - Even by comparing the close frames. Why is that?
I really don't know :confused:. In my clips, VAQ off resulted in noticeably blurred textured details on the skin (but then again some blurred images looked visually a little better (less unnecessary/noise-like detail) than the original, go figure :D).
However, your No AQ image has a lot more textured detail than the VAQ2 ones.
I also tested metric=0 and compared it with the original, which did give slightly better quality over metric=3 for strength>1 contradicting SSIM in some clips, but that was relatively rare and not the norm. So that's the only advice I can offer.

Also, could it be that the jpg screenshots are loosing too much detail? Just a thought. :p

Compare frame to frame is useless.
Would you mind telling us how to go about doing this testing? :rolleyes:
Playing (30 fps) through some of the encoded clips hardly shows noticeable differences as opposed to, say, a football clip. ;)

[Razorholt]

Also, could it be that the jpg screenshots are loosing too much detail?

If this is the case it shouldn't reverse the results. I'm currently running another batch with CRF 24 instead of 2 Pass. Let's see how it goes.

[IgorC]

Would you mind telling us how to go about doing this testing? :rolleyes:
Playing (30 fps) through some of the encoded clips hardly shows noticeable differences as opposed to, say, a football clip. ;)

If you don't see those artifacts then why care about it?

I can see artefacts even after high block +2:+2. But I wouldn't use +2:+2. During playback I don't see artefacts that I see watching 1 frame per minute. They are 24-30 frames per second. (each frame by 1/30 sec = 0,033 sec = 33 msec each frame). Would you spot artefacts on that speed?
That is all about psy AQ (VAQ in this case). Balance between artefacts/blur and detail for real playback.

blind ABX is a good method to compare. Unique usefull? More probably.

[patrick_]

I'm not sure if I should post this here or in a StaxRip thread, but it's directly related to AQ.

I'm compressing an anime (from DVD) 22:15 minutes. It's an old anime. The image shakes a little, but there's hardly any noise.
I use staxrip compressibility check to set quality at 55%.

I've tried with x264 build 736 (so no AQ) and build 807 @ AQ 1.5. If AQ uses more bits in flat areas, the same 55% should requiere a higher bitrate shouldn't it?

Numbers:
0.0 AQ 1230 bps
1.0 AQ 1211 bps (-1,46%)
1.5 AQ 1028 bps (-16,4%)
2.0 AQ 883 bps (-28,2%)

It's hard to tell (subjectively) which one has better quality. In certain areas it's AQ 0.0 in others it's AQ 1.5 (I didn't compress at 1.0 or 2.0)

screenshots (AQ 0.0 and 1.5):
1A (http://www.megadescarga.es/mi1.png)
1B (http://www.megadescarga.es/mi2.png)

2A (http://www.megadescarga.es/mi3.png)
2B (http://www.megadescarga.es/mi4.png)

In my opionion AQ 0.0 is sharper.
AQ 0.0 and 1.5 block in certain areas, but different ones.
Shades are better at 0.0 (strange, because AQ should improve shades).

I guess I'm missing something about the influence of AQ on the filesize, and I hope someone can help me out.

[Deinorius]

It would be much better, if you compare at equal bitrates.

From your source (and the pictures you posted) my conclusion is, that you get nearly same quality with about 15 % less bitrate which means: You can use the same bitrate (as without AQ) or a lower crf value (Constant Quality) to get a better quality.

[patrick_]

I've used a lot StaxRip, and in my experience quality > 55% isn't worth it. I compress my DVDs to take the videos with me and because harddisk space is limited, I care a lot about filesize. So I'm looking for a way to maintain quality, while reducing size.

[Inventive Software]

OK guys, once I've finished my current encode, I'm gonna thrash the Elephants Dream SD source through both the current x264 trunk and x264 VAQ2 enabled.

Plan is: all partitions, 8x8dct, hadamard (tesa), subme 7, 16 bframes, 16 refs, b-optimizations, trellis 1, direct auto, me-prepass and no-fast-pskip. Target bitrate is 500 Kbits 2 pass, and I'll post the PSNR and SSIM values from both encodes. Any suggestions as to what else is useful/not useful? Any as to what I can throw away that won't affect quality for the first pass?

[Inventive Software]

(Note: AQ settings were the defaults for each build.)


VAQ1 (revision 808 trunk, x264.nl) x264 settings

Pass 1: --bitrate 500 --pass 1 --bframes 16 --ref 16 --subme 3 --partitions none --me dia --merange 32 --direct auto --8x8dct --mixed-refs --me-prepass --b-rdo --weightb --b-pyramid --bime --trellis 1 --stats "ED500KbpsVAQ1.stats" --threads 3 --progress --output "P1VAQ1.264" "Elephants Dream.avs"

Pass 2: --bitrate 500 --pass 3 --bframes 16 --ref 16 --subme 7 --partitions all --me tesa --merange 32 --direct auto --8x8dct --mixed-refs --me-prepass --b-rdo --weightb --b-pyramid --bime --trellis 1 --stats "ED500KbpsVAQ1.stats" --threads 3 --progress --output "Elephants Dream 500 Kbps VAQ1.mp4" "Elephants Dream.avs"


VAQ2 (revision 808 trunk + VAQ2 patch, x264.nl, download (http://forum.doom9.org/showthread.php?p=1120033#post1120033)) x264 settings

Pass 1: --bitrate 500 --pass 1 --bframes 16 --ref 16 --subme 3 --partitions none --me dia --merange 32 --direct auto --8x8dct --mixed-refs --me-prepass --b-rdo --weightb --b-pyramid --bime --trellis 1 --stats "ED500KbpsVAQ2.stats" --threads 3 --progress --output "P1VAQ2.264" "Elephants Dream.avs"

Pass 2: --bitrate 500 --pass 3 --bframes 16 --ref 16 --subme 7 --partitions all --me tesa --merange 32 --direct auto --8x8dct --mixed-refs --me-prepass --b-rdo --weightb --b-pyramid --bime --trellis 1 --stats "ED500KbpsVAQ2.stats" --threads 3 --progress --output "Elephants Dream 500 Kbps VAQ2.mp4" "Elephants Dream.avs"


No AQ (revision 808 trunk, x264.nl, same as the VAQ1 build)

Pass 1:--bitrate 500 --pass 1 --bframes 16 --ref 16 --subme 3 --partitions none --me dia --merange 32 --direct auto --8x8dct --mixed-refs --me-prepass --b-rdo --weightb --b-pyramid --bime --trellis 1 --aq-mode 0 --stats "ED500KbpsNoAQ.stats" --threads 3 --progress --output "P1NoAQ.264" "Elephants Dream.avs"

Pass 2: --bitrate 500 --pass 3 --bframes 16 --ref 16 --subme 7 --partitions all --me tesa --merange 32 --direct auto --8x8dct --mixed-refs --me-prepass --b-rdo --weightb --b-pyramid --bime --trellis 1 --aq-mode 0 --stats "ED500KbpsNoAQ.stats" --threads 3 --progress --output "Elephants Dream 500 Kbps No AQ.mp4" "Elephants Dream.avs"


2nd Pass SSIM Mean (According to x264)

VAQ1: 0.9741147, no AQ delta: 0.0009015

VAQ2: 0.9741536, no AQ delta: 0.0009404

No AQ: 0.9732132


2nd Pass PSNR Mean (According to x264)

VAQ1: Y: 41.588, U: 54.541, V: 54.094, Avg: 42.742, Global: 40.737, kb/s: 500.49, no AQ Global delta: -0.457

VAQ2: Y: 41.318, U: 54.578, V: 54.174, Avg: 42.487, Global: 40.416, kb/s: 500.46, no AQ Global delta: -0.778

No AQ: Y: 43.303, U: 54.978, V: 54.363, Avg: 44.375, Global: 41.194, kb/s: 500.74


Download links

No AQ: http://rapidshare.com/files/106110969/Elephants_Dream_500_Kbps_No_AQ.mp4

VAQ1: http://rapidshare.com/files/106111590/Elephants_Dream_500_Kbps_VAQ1.mp4

VAQ2: http://rapidshare.com/files/106112192/Elephants_Dream_500_Kbps_VAQ2.mp4

I will upload the videos in the morning so you can see for yourself. Screenshots are pretty useless, since VAQ works by shifting bits between frames. ;) Also metrics can only tell you so much. ;) The AviSynth script loads the Lagarith lossless-encoded PNG original multiple image source. I chose Lagarith partly because of speed, but mostly because it decodes quickly, and is lossless (duh).

Now, if you'll pardon me, I need sleep before I suffer burnout. Please wait until morning (roughly 8 hours) for a reply from anything remotely making sense. :D Any more different encodes you'd like with different AQ settings, please state which settings you want, and I'll set it to go whilst I'm at uni. :)

Update 9th April: PSNR and SSIM results are up! :) They appear to show that VAQ2 is slightly better at the job. ;)

I did find it's slower though, the first pass for VAQ2 took 10.37 fps compared to 11.57 fps for VAQ1, and the second pass for VAQ2 was 1.23 fps compared to 1.24 fps for VAQ1. "No AQ" was fastest overall, first pass was 12.59 fps, second pass was 1.27 fps. Those are x264's reported values, I'll calculate what it actually was later on when I've had substantially more caffeine. :D Videos will be uploaded to Rapidshare so you can see for yourself. :)

I hope this is helpful, Dark Shikari. I've wanted to do this since this popped into the forum just before the holidays. :D

2nd Update 9th April: Videos are uploaded. :) Please download and compare, as I can't seem to spot any difference between VAQ2 and VAQ1. No AQ looks a little fuzzy.

[IgorC]

Inventive Software, talking about your encodes:

The most pronounced difference between VAQ and No AQ that I could noticed was during period between 4:12 and 4:47.
For me VAQ 1 and 2 did better job than No AQ (on monotone and soft gradients structures like light emitions ). I hardly can say which VAQ is better.

[MasterNobody]

For those who interested I create the modification of VAQ2 in which I try to optimize some parameters of AQ.

Changes from original VAQ2:
- added aq-mode = 3 (hybrid of modes 1 and 2); dynamic aq-sensitivity which is calculated by algorithm similar to aq-mode = 1 but for all already encoded frames (not only the current frame), this results that at first it is close to aq-mode = 1 but more frames it encodes more close it becomes to aq-mode = 2
- remove dependence of qcomp from aq-strength (more flexibility of qcomp option gives greater possibilities for optimizations)
- disable AQ in CQP mode (it contradicts to title of the mode)
- retrieve aq-sensitivity option; its meaning is same but range of possible values was changed (I recommend to keep it in range from 6 to 12); it is used only in aq-mode = 2; for calculation of new aq-sensitivity from old you can use this function: <new aq-sensitivity> = ln(<old aq-sensitivity>^4 / 2)
- change default values of AQ options:
* decrease aq-strength from 1.0 to 0.6
* set aq-sensitivity equal to 10 (original VAQ2 use value about 8.5)
* set aq-mode equal to 3 instead of 2
* don't change aq-metric (keep it equal to 3)
- remove warning about using new AQ metrics
- other small or cosmetic changes in code

Here is diff (http://stashbox.org/103931/x264_vaq2mod.01.r821.diff).

[MythCreator]

x264.821.vaq2.modified.experimental.exe (http://www.fs2you.com/files/da20d414-0cd1-11dd-bb5d-00142218fc6e/)

General thread:
http://forum.doom9.org/showthread.php?t=130364

x264_vaq2mod.01.r821.diff
http://forum.doom9.org/showthread.php?p=1126723#post1126723
x264.gaussian.cplxblur.01.diff
Dark Shikari: - gaussian cplxblur: gives a tiny improvement in 2pass ratecontrol
x264_me-prepass_DeathTheSheep.01.diff
http://forum.doom9.org/showthread.php?p=1093523
x264_2pass_vbv.7.diff
http://thread.gmane.org/gmane.comp.video.x264.devel/3093/focus=3748
x264_hrd_pulldown.04_interlace.diff
- HRD and pulldown for HD compatibility, updated patch for interlacing
http://forum.doom9.org/showthread.php?p=1047919#post1047919
x264_fix_win_stdin.diff
http://forum.doom9.org/showthread.php?p=1120065#post1120065

Link to x264 patches collected: http://files.x264.nl/x264_patches/



make frofiled in GCC 4.4.0 20080331 experimental,totally for experiment & test

[CruNcher]

@MasterNobody
That sounds really good especially loosing the chain of qcomp from it sounds like AQ 0.46 on steroids
tough i quiet don't understand (* don't change aq-metric (keep it equal to 3)) 3 is bloody slow and unusable in production yet and the visual gain is questionable vs the speed lose @ the moment compared to 1 :D

[Razorholt]

@MasterNobody: You fixed --aq-mode 1 ?

[Razorholt]

hmm... I just tried vaq2mod.01 with GrainOptimizer (thanks again Darky) and I got this:

http://69.46.26.197/~doompics/razorholt/images/snapshot20080418103824.bmp


It was OK with the previous version of VAQ2 tho.

EDIT: I used --aq-mode 3



Thanks,
- Dan

[MasterNobody]

hmm... I just tried vaq2mod.01 with GrainOptimizer (thanks again Darky) and I got this:

http://69.46.26.197/~doompics/razorholt/images/snapshot20080418103824.bmp


It was OK with the previous version of VAQ2 tho.

EDIT: I used --aq-mode 3



Thanks,
- Dan

AQ couldn't cause such artifacts (it only changes QP for whole macroblock, but on your image artifacts are only in part of macroblocks [one pixel-height lines]), so I think this bug is caused by something else (possible variants: other patches, changes from 808 to 821, compiler error, GrainOptimizer [I don't know how it works and could it cause such artifacts], other AviSynth filters)

@MasterNobody: You fixed --aq-mode 1 ?If you mean my comment about x264_autosense_aq here (http://forum.doom9.org/showthread.php?p=1120023#post1120023) than yes.

[Razorholt]

@MythCreator: Can you please release a version with both x264_vaq2mod.01.r821.diff and the new FGO patch from Darky -as well as the usual diffs?


Thanks a lot!

- Dan

[DeathTheSheep]

With the exception of disabling VAQ in CQP mode, I think that AQ bugmaster mod has some good ideas.

CruNcher, you mentioned it was like "0.46 on steroids;" care to elaborate a bit?

[MasterNobody]

With the exception of disabling VAQ in CQP mode, I think that AQ bugmaster mod has some good ideas.
I disable AQ in CQP because of 3 things:
1) it contradicts to the title of the mode: Constant QP
2) in CQP mode min and max QP (which also used for limitation of AQ macroblocks QP) are limited by --ipratio and --pbratio (not --qpmin and --qpmax) and so for I-frames AQ could only increase QP for some macroblocks and for B-frames it could only decrease QP
3) for same result you can use CRF with --qcomp 1 (which gives bit-identical result without AQ, and with AQ it doesn't have limitations described in item 2).