Variance AQ Megathread (AQ v0.48 update--defaults changed)

[fields_g]

Dreassica,

"--aq-strength=0.7" is wrong.
Get rid of the "=" sign.
"--aq-strength 0.7" is your answer.

[Dreassica]

Dreassica,

"--aq-strength=0.7" is wrong.
Get rid of the "=" sign.
"--aq-strength 0.7" is your answer.

Tried it, still doesn't start rendering. It's certainly not aq causing this, If I take it out from custom commandline in Megui, it still does this.

[Dark Shikari]

Just as a fyi, those tests I did earlier used the same AQ strength & sensitivity in both passes, and the tests with static sensitivity actually had the quants go up in the 2nd pass.This is because x264 measures only framewide quantizers, not block quantizers, and as such with adaptive quantization the quantizer readings can be extremely misleading.

With AQ, the average quantizers will generally be much lower than x264 displays.

[fields_g]

Dreassica,

Try using RAWAVC as your container. Also let us know if there is something weird in your "Standard Error Stream" in your log.

[salehin]

Here is a set comparisons.. I hope this helps. The source was BBC HD, Power of the Planet h264 1080p- encoded around 480 frames in 720p. Please note that the clip doesn't contain too many dark sequences.

If possible, please advise other possible ways of improving it.

aq_test with DarkShikari's v.45 aq 0.9, sens 20:

Avs Scipt

SetMemoryMax(128)
clip1=dgdecode_mpeg2source("J:\temp\crf test.d2v",info=3).ColorMatrix(hints=true,interlaced=true).tfm(order=0).tdecimate(hybrid=1).ConverttoYV12()

a=clip1.crop(2, 8, -2, -2)
clip2=DeGrainMedian(a, limitY=2,limitUV=3,mode=1)

GrainOptimizer(clip2).LimitedSharpenFaster(Strength=150).Spline36Resize(1280, 720)


Job commandline: "C:\Program Files\megui\tools\x264\x264.exe" --pass 1 --bitrate 4800 --stats "K:\planet test\aq_test.stats" --bframes 3 --b-pyramid --direct auto --filter -3,-3 --subme 1 --analyse none --me dia --threads auto --thread-input --sar 1:1 --cqmfile "C:\Program Files\megui\Custom Matrices\Prestige.cfg" --progress --no-dct-decimate --output NUL "K:\planet test\aq_test.avs" --aq-strength 0.9 --aq-sensitivity 20
--[Information] [1/20/2008 11:52:46 AM] Encoding started
Standard output stream
Standard error stream
avis [info]: 1280x720 @ 25.00 fps (489 frames)
x264 [info]: using SAR=1/1
x264 [info]: using cpu capabilities: MMX MMXEXT SSE SSE2 Cache64
x264 [info]: slice I:6 Avg QP:21.33 size:123021 PSNR Mean Y:46.63 U:48.01 V:49.72 Avg:47.22 Global:46.50
x264 [info]: slice P:265 Avg QP:25.22 size: 35899 PSNR Mean Y:40.57 U:46.54 V:47.86 Avg:41.86 Global:41.33
x264 [info]: slice B:218 Avg QP:26.15 size: 11726 PSNR Mean Y:40.46 U:45.89 V:47.69 Avg:41.70 Global:41.32
x264 [info]: mb I I16..4: 36.1% 0.0% 63.9%
x264 [info]: mb P I16..4: 28.7% 0.0% 0.0% P16..4: 62.6% 0.0% 0.0% 0.0% 0.0% skip: 8.7%
x264 [info]: mb B I16..4: 3.4% 0.0% 0.0% B16..8: 24.3% 0.0% 0.0% direct:35.0% skip:37.3%
x264 [info]: final ratefactor: 25.61
x264 [info]: direct mvs spatial:79.8% temporal:20.2%
x264 [info]: SSIM Mean Y:0.9598064
x264 [info]: PSNR Mean Y:40.597 U:46.270 V:47.804 Avg:41.855 Global:41.367 kb/s:5238.34
encoded 489 frames, 1.18 fps, 5238.76 kb/s

-[Information] Log for job36 (video, aq_test.avs -> aq_test with DS v.45 aq 0.9, sens 20.mkv)
--[Information] [1/20/2008 11:59:52 AM] Started handling job
--[Information] [1/20/2008 11:59:52 AM] Preprocessing
Job commandline: "C:\Program Files\megui\tools\x264\x264.exe" --pass 2 --bitrate 4800 --stats "K:\planet test\aq_test.stats" --ref 5 --mixed-refs --no-fast-pskip --bframes 3 --b-pyramid --b-rdo --bime --weightb --direct auto --filter -3,-3 --subme 7 --trellis 1 --analyse all --8x8dct --me umh --threads auto --thread-input --sar 1:1 --cqmfile "C:\Program Files\megui\Custom Matrices\Prestige.cfg" --progress --no-dct-decimate --output "K:\planet test\aq_test with DS v.45 aq 0.9, sens 20.mkv" "K:\planet test\aq_test.avs" --aq-strength 0.9 --aq-sensitivity 20
--[Information] [1/20/2008 11:59:59 AM] Encoding started
Standard output stream
Standard error stream
avis [info]: 1280x720 @ 25.00 fps (489 frames)
x264 [info]: using SAR=1/1
x264 [info]: using cpu capabilities: MMX MMXEXT SSE SSE2 Cache64
x264 [info]: slice I:6 Avg QP:20.33 size:106716 PSNR Mean Y:46.13 U:48.86 V:50.38 Avg:46.97 Global:46.73
x264 [info]: slice P:265 Avg QP:22.55 size: 34281 PSNR Mean Y:41.89 U:47.37 V:48.66 Avg:43.13 Global:42.73
x264 [info]: slice B:218 Avg QP:23.57 size: 9752 PSNR Mean Y:41.32 U:46.51 V:48.34 Avg:42.54 Global:42.35
x264 [info]: mb I I16..4: 1.8% 93.3% 4.9%
x264 [info]: mb P I16..4: 0.1% 5.2% 0.2% P16..4: 54.9% 24.2% 9.7% 0.9% 0.5% skip: 4.2%
x264 [info]: mb B I16..4: 0.0% 0.4% 0.0% B16..8: 36.8% 1.2% 3.8% direct: 2.9% skip:54.9%
x264 [info]: 8x8 transform intra:93.5% inter:82.6%
x264 [info]: direct mvs spatial:80.3% temporal:19.7%
x264 [info]: ref P 69.0% 15.2% 7.6% 4.9% 3.4%
x264 [info]: ref B 83.4% 10.2% 2.9% 2.2% 1.3%
x264 [info]: SSIM Mean Y:0.9635958
x264 [info]: PSNR Mean Y:41.687 U:47.005 V:48.535 Avg:42.914 Global:42.586 kb/s:4846.87
encoded 489 frames, 0.41 fps, 4847.54 kb/s
--[Information] Final statistics
Desired video bitrate: 4800 kbit/s
Obtained video bitrate (approximate: 4850 kbit/s

oooooooooooooooo

aq_test with x264 r719 SVN aq 0.3.mkv

Applied patches (by techhouse):
x264_2pass_vbv.diff
x264_aq-brdo.diff
x264_faster-dia.diff
x264_fp-eta.01.r680.diff
x264_hrd_pulldown.diff
x264_me-prepass.diff
x264_satd_fpel.11.diff
x264_thread_pool.r680.diff

** note: it's not a cef built- i named the file incrrectly**
Avs Scipt: same

Job commandline: "C:\Program Files\megui\tools\x264\x264.exe" --pass 1 --bitrate 4800 --stats "K:\planet test\aq_test.stats" --bframes 3 --b-pyramid --direct auto --filter -3,-3 --subme 1 --analyse none --me dia --threads auto --thread-input --sar 1:1 --cqmfile "C:\Program Files\megui\Custom Matrices\Prestige.cfg" --progress --no-dct-decimate --output NUL "K:\planet test\aq_test.avs" --aq-strength 0.3

--[Information] [1/20/2008 12:25:38 PM] Encoding started
Standard output stream
Standard error stream
avis [info]: 1280x720 @ 25.00 fps (489 frames)
x264 [info]: using SAR=1/1
x264 [info]: using cpu capabilities: MMX MMXEXT SSE SSE2 Cache64
x264 [info]: slice I:6 Avg QP:21.00 size:121756 PSNR Mean Y:46.78 U:48.09 V:49.83 Avg:47.35 Global:46.77
x264 [info]: slice P:265 Avg QP:24.60 size: 36377 PSNR Mean Y:40.74 U:46.70 V:48.04 Avg:42.03 Global:41.48
x264 [info]: slice B:218 Avg QP:25.56 size: 12186 PSNR Mean Y:40.62 U:46.02 V:47.85 Avg:41.86 Global:41.50
x264 [info]: mb I I16..4: 39.0% 0.0% 61.0%
x264 [info]: mb P I16..4: 27.2% 0.0% 0.0% P16..4: 62.2% 0.0% 0.0% 0.0% 0.0% skip:10.6%
x264 [info]: mb B I16..4: 3.1% 0.0% 0.0% B16..8: 23.9% 0.0% 0.0% direct:36.1% skip:36.9%
x264 [info]: final ratefactor: 25.19
x264 [info]: direct mvs spatial:80.7% temporal:19.3%
x264 [info]: SSIM Mean Y:0.9606545
x264 [info]: PSNR Mean Y:40.765 U:46.414 V:47.976 Avg:42.022 Global:41.523 kb/s:5327.96
encoded 489 frames, 1.22 fps, 5328.38 kb/s
[1/20/2008 12:32:40 PM] Job completed
--[Information] [1/20/2008 12:32:40 PM] Postprocessing
---[Information] Deleting intermediate files
-[Information] Log for job37 (video, aq_test.avs -> aq_test with cef709 aq 0.3.mkv)
--[Information] [1/20/2008 12:32:40 PM] Started handling job
--[Information] [1/20/2008 12:32:40 PM] Preprocessing

Job commandline: "C:\Program Files\megui\tools\x264\x264.exe" --pass 2 --bitrate 4800 --stats "K:\planet test\aq_test.stats" --ref 5 --mixed-refs --no-fast-pskip --bframes 3 --b-pyramid --b-rdo --bime --weightb --direct auto --filter -3,-3 --subme 7 --trellis 1 --analyse all --8x8dct --me umh --threads auto --thread-input --sar 1:1 --cqmfile "C:\Program Files\megui\Custom Matrices\Prestige.cfg" --progress --no-dct-decimate --output "K:\planet test\aq_test with cef709 aq 0.3.mkv" "K:\planet test\aq_test.avs" --aq-strength 0.3
--[Information] [1/20/2008 12:32:56 PM] Encoding started
Standard output stream
Standard error stream
avis [info]: 1280x720 @ 25.00 fps (489 frames)
x264 [info]: using SAR=1/1
x264 [info]: using cpu capabilities: MMX MMXEXT SSE SSE2 Cache64
x264 [info]: slice I:6 Avg QP:20.17 size:102501 PSNR Mean Y:46.06 U:48.76 V:50.33 Avg:46.90 Global:46.66
x264 [info]: slice P:265 Avg QP:22.35 size: 34217 PSNR Mean Y:41.94 U:47.40 V:48.69 Avg:43.18 Global:42.79
x264 [info]: slice B:218 Avg QP:23.34 size: 9897 PSNR Mean Y:41.38 U:46.53 V:48.35 Avg:42.60 Global:42.40
x264 [info]: mb I I16..4: 2.2% 93.1% 4.6%
x264 [info]: mb P I16..4: 0.1% 5.5% 0.2% P16..4: 53.4% 23.4% 9.9% 1.0% 0.5% skip: 5.9%
x264 [info]: mb B I16..4: 0.0% 0.4% 0.0% B16..8: 35.7% 1.2% 3.9% direct: 3.1% skip:55.8%
x264 [info]: 8x8 transform intra:94.1% inter:83.1%
x264 [info]: direct mvs spatial:83.0% temporal:17.0%
x264 [info]: ref P 68.9% 15.2% 7.6% 4.9% 3.4%
x264 [info]: ref B 83.5% 10.3% 2.8% 2.1% 1.2%
x264 [info]: SSIM Mean Y:0.9639119
[info]: PSNR Mean Y:41.741 U:47.029 V:48.558 Avg:42.964 Global:42.642 kb/s:4842.55
encoded 489 frames, 0.46 fps, 4843.22 kb/s
Final statistics
Desired video bitrate: 4800 kbit/s
Obtained video bitrate (approximate: 4846 kbit/s

[Dreassica]

Dreassica,

Try using RAWAVC as your container. Also let us know if there is something weird in your "Standard Error Stream" in your log.

No on both counts.

--crf 16 --ref 16 --bframes 3 --b-pyramid --b-rdo --bime --weightb --direct auto --filter 1,1 --subme 7 --trellis 1 --analyse all --8x8dct --me esa --threads 8 --thread-input --progress --no-psnr --no-ssim --output "output" "input"

[Dark Shikari]

Unfortunately one cannot tell much from logs in this sort of situation; its really all about how it looks.

[Yoshiyuki Blade]

Tried it, still doesn't start rendering. It's certainly not aq causing this, If I take it out from custom commandline in Megui, it still does this.

As fields_g stated, select RAWAVC as your container (it will output as a .264 file). Also make sure pthreadGC2.dll is in the same folder as Dark's build of x264. If its not there, copy pthreadGC2.dll from the "ffmpeg" folder in the MeGUI directory, which is also in the "tools" folder like "x264"

I've been running a few anime encodes with the v0.45 and it seems great. At the same settings and bitrate with --aq-strength 1.0 --aq-sesitivity 25, there was much less blockyness with v0.45, although it seems to come at the cost of overall quality. On v0.44 the scenes look nicer, but it looks rather nasty in dark areas. Simply swapping out build v0.44 with v0.45 with the exact same configuration reduced blockyness by a lot, but the overall crispness is lost. Perhaps it needs more bitrate.

[Dreassica]

Pasting the commandline in a dos prompt gives me the error that pthreadGC2.dll can't be found. Odd as standard x264 seems to work fine.

EDIT

Downloaded said dll and put it in same dir as x264.exe, NOW it works finally.

[Dark Shikari]

Pasting the commandline in a dos prompt gives me the error that pthreadGC2.dll can't be found. Odd as standard x264 seems to work fine.That's because incompetent me can't seem to get a static build working. Get PthreadGC2.dll here (http://mirror05.x264.nl/Dark/force.php?file=./pthreadGC2.dll).

[Dreassica]

That's because incompetent me can't seem to get a static build working. Get PthreadGC2.dll here (http://mirror05.x264.nl/Dark/force.php?file=./pthreadGC2.dll).

Yea found that out on my own. Rendering finally now. :)

[Yoshiyuki Blade]

EDIT

Downloaded said dll and put it in same dir as x264.exe, NOW it works finally.

Great, now double-check and make sure RAWAVC is the output container instead of mp4, or itll error on the 2nd pass. I wasted several hours of testing by not doublechecking :D.

[salehin]

Unfortunately one cannot tell much from logs in this sort of situation; its really all about how it looks.
Here they are
With DarkShikari's v.45: aq_test with DS v.45 aq 0.9, sens 20.mkv (http://www.sendspace.com/file/vdx2rn)
*Without DarkShikari: aq_test with x264 r719 SVN aq 0.3.mkv (http://www.sendspace.com/file/rke9ij)

*Applied patches in the svn:


x264_2pass_vbv.diff
x264_aq-brdo.diff
x264_faster-dia.diff
x264_fp-eta.01.r680.diff
x264_hrd_pulldown.diff
x264_me-prepass.diff
x264_satd_fpel.11.diff
x264_thread_pool.r680.diff

[Dark Shikari]

Here they are
With DarkShikari's v.45: aq_test with DS v.45 aq 0.9, sens 20.mkv (http://www.sendspace.com/file/vdx2rn)
*Without DarkShikari: aq_test with x264 r719 SVN aq 0.3.mkv (http://www.sendspace.com/file/rke9ij)

*Applied patches in the svn:Looking at the quantizer distribution its clear neither of these actually used my AQ. This isn't surprising, given that "x264_aq-brdo.diff" isn't my patch, its Haali's.

[salehin]

eek!! i'll remove the svn from the x264 folder and post new link later

Update: Here is the proper one. Only your v.45 is in the MeGUIs x264 folder- renamed as x264.exe

(Proper) aq_test with DS v.45 aq 0.9, sens 20.mkv (http://www.sendspace.com/file/jyavnu)

Log

Job commandline: "C:\Program Files\megui\tools\x264\x264.exe" --pass 1 --bitrate 4800 --stats "K:\planet test\aq_test.stats" --bframes 3 --b-pyramid --direct auto --filter -3,-3 --subme 1 --analyse none --me dia --threads auto --thread-input --sar 1:1 --cqmfile "C:\Program Files\megui\Custom Matrices\Prestige.cfg" --progress --no-dct-decimate --output NUL "K:\planet test\aq_test.avs" --aq-strength 0.9 --aq-sensitivity 20

[1/20/2008 2:37:33 PM] Encoding started
avis [info]: 1280x720 @ 25.00 fps (489 frames)
x264 [info]: using SAR=1/1
x264 [info]: using cpu capabilities: MMX MMXEXT SSE SSE2 Cache64
x264 [info]: slice I:6 Avg QP:24.83 size:118889 PSNR Mean Y:45.43 U:47.34 V:49.11 Avg:46.12 Global:44.64
x264 [info]: slice P:270 Avg QP:29.72 size: 35574 PSNR Mean Y:39.36 U:45.62 V:46.89 Avg:40.67 Global:39.95
x264 [info]: slice B:213 Avg QP:30.15 size: 11561 PSNR Mean Y:39.46 U:44.99 V:46.81 Avg:40.70 Global:39.96
x264 [info]: mb I I16..4: 29.2% 0.0% 70.8%
x264 [info]: mb P I16..4: 29.3% 0.0% 0.0% P16..4: 62.8% 0.0% 0.0% 0.0% 0.0% skip: 7.9%
x264 [info]: mb B I16..4: 3.9% 0.0% 0.0% B16..8: 25.1% 0.0% 0.0% direct:33.7% skip:37.3%
x264 [info]: final ratefactor: 29.88
x264 [info]: direct mvs spatial:77.9% temporal:22.1%
x264 [info]: SSIM Mean Y:0.9580282
x264 [info]: PSNR Mean Y:39.480 U:45.366 V:46.880 Avg:40.751 Global:39.990 kb/s:5227.32
encoded 489 frames, 1.36 fps, 5226.36 kb/s
[1/20/2008 2:43:45 PM] Job completed


Log for job41 (video, aq_test.avs -> aq_test with DS v.45 aq 0.9, sens 20.mkv)
[1/20/2008 2:43:47 PM] Started handling job
[1/20/2008 2:43:47 PM] Preprocessing
Job commandline: "C:\Program Files\megui\tools\x264\x264.exe" --pass 2 --bitrate 4800 --stats "K:\planet test\aq_test.stats" --ref 5 --mixed-refs --no-fast-pskip --bframes 3 --b-pyramid --b-rdo --bime --weightb --direct auto --filter -3,-3 --subme 7 --trellis 1 --analyse all --8x8dct --me umh --threads auto --thread-input --sar 1:1 --cqmfile "C:\Program Files\megui\Custom Matrices\Prestige.cfg" --progress --no-dct-decimate --output "K:\planet test\aq_test with DS v.45 aq 0.9, sens 20.mkv" "K:\planet test\aq_test.avs" --aq-strength 0.9 --aq-sensitivity 20
[1/20/2008 2:44:01 PM] Encoding started
avis [info]: 1280x720 @ 25.00 fps (489 frames)
x264 [info]: using SAR=1/1
x264 [info]: using cpu capabilities: MMX MMXEXT SSE SSE2 Cache64
x264 [info]: slice I:6 Avg QP:23.67 size:105442 PSNR Mean Y:45.08 U:48.10 V:49.69 Avg:45.97 Global:45.58
x264 [info]: slice P:270 Avg QP:26.36 size: 33967 PSNR Mean Y:40.84 U:46.60 V:47.84 Avg:42.10 Global:41.61
x264 [info]: slice B:213 Avg QP:27.25 size: 9408 PSNR Mean Y:40.55 U:45.73 V:47.58 Avg:41.77 Global:41.46
x264 [info]: mb I I16..4: 1.9% 94.1% 4.0%
x264 [info]: mb P I16..4: 0.2% 6.6% 0.1% P16..4: 50.8% 28.8% 9.2% 0.3% 0.1% skip: 3.8%
x264 [info]: mb B I16..4: 0.0% 0.4% 0.0% B16..8: 39.1% 1.2% 4.0% direct: 3.1% skip:52.2%
x264 [info]: 8x8 transform intra:95.5% inter:81.8%
x264 [info]: direct mvs spatial:71.4% temporal:28.6%
x264 [info]: ref P 70.1% 14.4% 7.4% 4.8% 3.4%
x264 [info]: ref B 82.8% 10.4% 3.1% 2.3% 1.5%
x264 [info]: SSIM Mean Y:0.9639779
x264 [info]: PSNR Mean Y:40.766 U:46.242 V:47.749 Avg:42.004 Global:41.577 kb/s:4829.26
encoded 489 frames, 0.44 fps, 4828.30 kb/s
Final statistics
Desired video bitrate: 4800 kbit/s
Obtained video bitrate (approximate: 4831 kbit/s

[DeathTheSheep]

So you're saying you managed to successfully apply the satd and me-prepass patches to svn-r719? A lot of the variables in me.c don't even refer to the quite same things anymore... casting pointers into ints and whatnot, that's what the compiler complains of.

In other news, water does seem to be somewhat wet and slippery.
No, I don't believe you. (Why would you say that of all things?) This doesn't have anything to do with an "epic failtrain," does it? :p

[Sagekilla]

Well now, this is quite an oddity.. I tested --aq-strength 1 with sensitivity of 0 and 20 on a randomly picked scene. (Which also happened to be a very dark, flat scene.) To my surprise.. Sensitivity of 20 more than doubled the bitrate used under crf mode! And this is with the 0.45 AQ.

G:\Movies\Battlestar Galactica>x264_aq --keyint 1000 --crf 18 --ref 6 --mixed-re
fs --no-fast-pskip --bframes 16 --bime --weightb --b-pyramid --b-rdo --8x8dct --
subme 7 --me umh --trellis 1 --aq-strength 1 --aq-sensitivity 20 --threads auto
--thread-input --progress --output "video.264" --pass 1 --stats "stats.log" "sou
rce.avs"
avis [info]: 864x480 @ 23.98 fps (720 frames)
x264 [info]: using cpu capabilities: MMX MMXEXT SSE SSE2 SSE3 3DNow!
x264 [info]: slice I:3 Avg QP:14.67 size: 61484 PSNR Mean Y:50.83 U:52.00
V:53.61 Avg:51.36 Global:51.34
x264 [info]: slice P:351 Avg QP:17.30 size: 28768 PSNR Mean Y:49.25 U:50.53
V:52.70 Avg:49.85 Global:49.71
x264 [info]: slice B:366 Avg QP:19.85 size: 9467 PSNR Mean Y:47.62 U:49.54
V:51.63 Avg:48.35 Global:48.30
x264 [info]: mb I I16..4: 21.5% 69.4% 9.1%
x264 [info]: mb P I16..4: 4.7% 12.5% 2.2% P16..4: 35.2% 27.5% 17.0% 0.0% 0
.0% skip: 0.9%
x264 [info]: mb B I16..4: 0.5% 1.4% 0.5% B16..8: 33.8% 3.8% 10.1% direct:
5.8% skip:44.1%
x264 [info]: 8x8 transform intra:64.0% inter:47.0%
x264 [info]: ref P 73.5% 12.2% 5.6% 3.5% 2.6% 2.6%
x264 [info]: ref B 88.4% 7.9% 1.6% 0.9% 0.6% 0.6%
x264 [info]: SSIM Mean Y:0.9915203
x264 [info]: PSNR Mean Y:48.426 U:50.032 V:52.158 Avg:49.096 Global:48.940 kb/s:
3662.19

encoded 720 frames, 2.98 fps, 3661.24 kb/s


G:\Movies\Battlestar Galactica>x264_aq --keyint 1000 --crf 18 --ref 6 --mixed-re
fs --no-fast-pskip --bframes 16 --bime --weightb --b-pyramid --b-rdo --8x8dct --
subme 7 --me umh --trellis 1 --aq-strength 1 --aq-sensitivity 0 --threads auto -
-thread-input --progress --output "video2.264" --pass 1 --stats "stats.log" "sou
rce.avs"
avis [info]: 864x480 @ 23.98 fps (720 frames)
x264 [info]: using cpu capabilities: MMX MMXEXT SSE SSE2 SSE3 3DNow!
x264 [info]: slice I:3 Avg QP:14.67 size: 45170 PSNR Mean Y:49.09 U:50.68
V:52.41 Avg:49.72 Global:49.71
x264 [info]: slice P:351 Avg QP:17.30 size: 11716 PSNR Mean Y:47.14 U:49.62
V:51.95 Avg:47.99 Global:47.89
x264 [info]: slice B:366 Avg QP:19.85 size: 3612 PSNR Mean Y:45.90 U:48.76
V:50.91 Avg:46.80 Global:46.77
x264 [info]: mb I I16..4: 23.0% 68.2% 8.8%
x264 [info]: mb P I16..4: 5.4% 7.5% 2.4% P16..4: 51.1% 20.4% 7.8% 0.0% 0
.0% skip: 5.4%
x264 [info]: mb B I16..4: 0.5% 0.9% 0.2% B16..8: 39.4% 1.4% 3.3% direct:
1.8% skip:52.5%
x264 [info]: 8x8 transform intra:50.6% inter:65.8%
x264 [info]: ref P 69.6% 14.1% 6.9% 3.8% 2.9% 2.7%
x264 [info]: ref B 86.2% 8.4% 2.4% 1.2% 1.0% 0.8%
x264 [info]: SSIM Mean Y:0.9874901
x264 [info]: PSNR Mean Y:46.517 U:49.186 V:51.424 Avg:47.392 Global:47.289 kb/s:
1483.77

encoded 720 frames, 3.59 fps, 1482.82 kb/s


Any thoughts on this?

[salehin]

So you're saying you managed to successfully apply the satd and me-prepass patches to svn-r719? A lot of the variables in me.c don't even refer to the quite same things anymore... casting pointers into ints and whatnot, that's what the compiler complains of.


No, I don't believe you. (Why would you say that of all things?) :p

Sorry DtS- i don't know what or how to reply, as I'm no expert into this sort of thing. I just got that patched svn from another place. The accompanied text file mentions that those patches are included in that x264

techhouse's x264 (http://x264.tk/)

[DeathTheSheep]

Ah, I see. Then would you direct me to this place of mystery? :) Thanks.

Sagekilla: Yes, raising the sensitivity tends to increase the bitrate quite a bit. Try sensitivity 30 and be shocked. (EPICSHOCK.) Lowering the sensitivity (not 0, that's automatic I presume) can actually lower the bitrate significantly, because the threshold acts like a centerpoint--if the input is above this centerpoint, it gets more data data, and vise versa.

[bob0r]

If you want to test AQ, i guess you should use x264 svn + AQ patch only:
x264.721.dark.aq.rdrc.0.45.exe (http://files.x264.nl/AQ/x264.721.dark.aq.rdrc.0.45.exe) (pthreads/mp4 = yes, made with make fprofiled)

[DeathTheSheep]

You said it, bob0r. Most of the patches in that build were broken anyway--even calling fpel cmp satd would crash the program, and for good reason. The old patch would increment the memory addresses of some of the variables in the new revision, obviously leading to a crash even before the first keyframe could be composed. And it's no easy fix--the mechanism has been practically redesigned, not to mention rearranged. I'm here with 681, custom built, march core2, cyborg gcc 4.3 [rev6], custom makefprofiled commandlines, tweaked function grouping priorities, etc, which is still pretty darn fast and works like a charm, save for a non-determinism or two with MT.

But like I've indicated earlier, combining some of the old patches with this new AQ leads to a synergistic effect, yielding a significantly greater quality increase than any used separately.

[Sagekilla]

Dark Shikari, I can confirm that your new AQ does work well on animated video. I'm encoding Dark Fury right now and the quality is very good across clip so far.

[Atak_Snajpera]

I've converted 1080p anime and it looks very well. Size even lower than without AQ

[jvt40]

Both clips looked fine (watching from bit outdated 17" screen :p) , difference running them side by side was Proper DS was slightly warmer (more red!!) , could be seen on jungle colour , sky and mountains , which is more truthful is matter of opinion .

eek!! i'll remove the svn from the x264 folder and post new link later

Update: Here is the proper one. Only your v.45 is in the MeGUIs x264 folder- renamed as x264.exe

(Proper) aq_test with DS v.45 aq 0.9, sens 20.mkv (http://www.sendspace.com/file/jyavnu)

[Dark Shikari]

Well now, this is quite an oddity.. I tested --aq-strength 1 with sensitivity of 0 and 20 on a randomly picked scene. (Which also happened to be a very dark, flat scene.) To my surprise.. Sensitivity of 20 more than doubled the bitrate used under crf mode! And this is with the 0.45 AQ.This is why I said nonzero sensitivity isn't for CRF mode, because it doesn't maintain bitrate.

[Morte66]

This is why I said nonzero sensitivity isn't for CRF mode, because it doesn't maintain bitrate.

But that's much the same as the AQ in the current x264 mainstream build. --crf 18 --aq-strength 0.3 --aq-sensitivity 5 gives something like double the bitrate of plain crf 18, depending on the material, and it's about enough to control blocking/banding on a revealing display.

Nobody knows what bitrate is going to come out of crf anyhow, that's the point of it. You encode to a target quality, not a target size. AQ changes the definition of "quality" from PSNR to something more concerned with subjective perception of smooth areas.

[CruNcher]

Exactly that's also my fear as my tests show exactly this behaveiour, no doub't it has much better detail preservation @ low bitrates, but it comes at a price for more unbalanced quality between the whole final thing. I think exactly that's what the goal should be with a AQ "balanced percepted visual quality". Under normal viewing conditions for this lossy stuff it's hard anyway to see slight blocking in motion, but banding is imidietly visible under any viewing condition. Sure you can try to use a Higher Compression factor but this again comes most of the times @ a heavy speed loss (Encoding/Decoding).

[Dark Shikari]

Exactly that's also my fear as my tests show exactly this behaveiour, no doub't it has much better detail preservation @ low bitrates, but it comes at a price for more unbalanced quality between the whole final thing. I think exactly that's what the goal should be with a AQ "balanced percepted visual quality". Under normal viewing conditions for this lossy stuff it's hard anyway to see slight blocking in motion, but banding is imidietly visible under any viewing condition. Sure you can try to use a Higher Compression factor but this again comes most of the times @ a heavy speed loss (Encoding/Decoding).Wait, you're criticizing my AQ for banding, even though banding is what it handles far better than any previous algorithm?

:rolleyes:

This is getting pointlessly retarded. Its as if you ran out of valid criticisms for my AQ (of which there are many), because your comments seem just completely nonsensical. Lowering quantizers in areas with banding problems is not going to make the banding worse!

[CruNcher]

http://forum.doom9.org/showthread.php?p=1089048#post1089048

look @ those scenes in the samples you've got and tell me what your New AQ does wrong then :)
Possible that i visually interpreting those results wrong as banding but they look like as this for me because of the fine gradients of noise that suddenly become blocks

[Dark Shikari]

http://forum.doom9.org/showthread.php?p=1089048#post1089048

look @ those scenes in the samples you've got and tell me what your New AQ does wrong then :)
Possible that i visually interpreting those results wrong as banding but they look like as this for me because of the fine gradients of noise that suddenly become blocksThis is because you used automatic sensitivity, which intentionally doesn't allow bits to be moved between frames. That means if a frame does genuinely need more bits, and you can't just move the bits from elsewhere in the frame to solve the quality problem, it won't get fixed.

Use a static sensitivity with twopass, and try again.

[CruNcher]

But i don't want to use twopass, because of the Speed reason i mentioned above :) and i tried every combination possible with ABR and your New AQ and it results allways in those Visual results it doesn't change for this particular spots (and those are really nasty they ruin the complete visual interpretation of this subjectively).
Psy optimizations aren't about Metrics or that you have super duper detail preservation in one part of the final result, they are about the balanced Look & Feel of the whole thing. That's why im also not really impressed @ the moment with the Parkrun results seeing this now, i mean it should be consistent improvement and not only for 1 spot and such a scene as Parkrun is a very special thing because also it's only 1 part scene and not 1 part of something much bigger around it :)

[Dark Shikari]

But i don't want to use twopass, because of the Speed reason i mentioned above :) and i tried every combination possible with ABR and your New AQ and it results allways in those Visual results it doesn't change for this particular spots.Then use CRF with static sensitivity. Just remember the filesize isn't guaranteed to stay where you want it--much like Haali's AQ, of course.

Try sensitivity 20 if you want to try to stay near the original filesize.

[Inventive Software]

Remind me... what's RCRD? And how much of a quality benefit does SATD with this patch give over a vanilla build?

[Dark Shikari]

Remind me... what's RCRD? And how much of a quality benefit does SATD with this patch give over a vanilla build?RCRD is a ratecontrol method that finds the optimal quantizer distribution, RD-wise.

SATD patch hasn't been updated to be compatible with r717+ yet. Benefit would be the same as before.

[Inventive Software]

Thanks for the info clear-up. :)

[Dark Shikari]

Link to 1080p sample encoded with the new AQ added to the original post.

[DeathTheSheep]

Could you also upload an RCRD sample? I'm sure people would be more eager to try it out if they see some results justifying the speed loss. After all, a lot of work and #x264dev chat has apparently gone into it.

In a few days' time, we'll have working satd too, and with very well-tweaked thresholds, might I add. Or shouldn't I spoil the surprise? One can hope prepass will be tackled by someone, too (heck, if nobody's up to fixing prepass by that time, I'll look into it myself, though whether or not I can possibly whip that on to x264's new code I'm obviously quite unsure about...let's hope the problem isn't too deeply rooted). This will give the community a much better reason to use RDRC--an insane option among insane options--since they'll have others to stack on top of it to mitigate their fears.

[Dark Shikari]

Could you also upload an RCRD sample? I'm sure people would be more eager to try it out if they see some results justifying the speed loss. After all, a lot of work and #x264dev chat has apparently gone into it.

In a few days' time, we'll have working satd too, and with very well-tweaked thresholds, might I add. Or shouldn't I spoil the surprise? One can hope prepass will be tackled by someone, too (heck, if nobody's up to fixing prepass by that time, I'll look into it myself, though whether or not I can possibly whip that on to x264's new code I'm obviously quite unsure about...let's hope the problem isn't too deeply rooted). This will give the community a much better reason to use RDRC--an insane option among insane options--since they'll have others to stack on top of it to mitigate their fears.RDRC sample (http://momupload.com/files/69785/touhou.mp4.html) with AQ.

[DeathTheSheep]

MOMupload? MOM?! Boy oh boy and by golly george, there's a mom upload now of all things? And what's more, I see you used them over mediafire...ouch. :\

What, a 100MB upload limit to the former?

[Dark Shikari]

MOMupload? MOM?! Boy oh boy and by golly george, there's a mom upload now of all things? And what's more, I see you used them over mediafire...ouch. :\

What, a 100MB upload limit to the former?Correct, size limit :p

[DeathTheSheep]

Well, 180KB/sec, that's nice. (Results after the break, ladies and gentlemen!) ;)

[edit]
Oh my god, is that you playing the game? I watched it two and half times already... This is MAD SKILLZ to the max. Tell me it's not you playing the game...tell me...

As for RCRD, this seems like an inordinately complex source, and all that rapid motion (and R4D D061NG sk111z) were handled...erm, well. But since I haven't made any such video game encodes before (or even seen many of them), I'm not quite sure what to expect here. Maybe a test run using CruNcher's infamous test clip is in order, eh?

But seriously. This game is 73H 1337 H4x0rz. And the soundtrack kicks Major General Asce.

[CruNcher]

Hmm could you guys please post wich of these looks better to you on the first sight and then also your GFX Card, Monitor ,Operating System, Player Application,Decoder and used Renderer as information. Thx in Advance :)

Please don't look framewise just subjectively rate them on the first sight :)

http://mirror05.x264.nl/CruNcher/force.php?file=./1.mkv
http://mirror05.x264.nl/CruNcher/force.php?file=./2.mkv
http://mirror05.x264.nl/CruNcher/force.php?file=./3.mkv

Btw: If you want to see allways correct representation results in Windows with a Directshow Player use CoreAVC as Decoder :)

[Dark Shikari]

Oh my god, is that you playing the game? I watched it two and half times already... This is MAD SKILLZ to the max. Tell me it's not you playing the game...tell me...Hell no, I got that off Stage6 (download link, for your own test encodes--source = 12.5 megabit DivX (http://www.stage6.com/user/SeaWeeze/video/1652397/Touhou-10-Mountain-of-Faith---Extra-Reimu-B)). I can't even beat Perfect Cherry Blossom on Normal difficulty let alone Ludicrous or Extra!

Note that its not as bad as it looks, as the hitbox in the game (like all similar games in the genre) is extremely small.
As for RCRD, this seems like an inordinately complex source, and all that rapid motion (and R4D D061NG sk111z) were handled...erm, well. But since I haven't made any such video game encodes before (or even seen many of them), I'm not quite sure what to expect here. Maybe a test run using CruNcher's infamous test clip is in order, eh?

But seriously. This game is 73H 1337 H4x0rz. And the soundtrack kicks Major General Asce.The game series is called Touhou Project, and you can get it (with the English patch) and play it on non-insane difficulty :p There's 10 games in the series now, Perfect Cherry Blossom is the easiest to find of them.

I love the Touhou clip because its absolute murder for any video encoder.

The main benefit of AQ in that clip is in the backgrounds (which were blurred to death without it). The main benefit of RCRD was more intelligent I/B-frame offset decision.

[CruNcher]

Holly Bible :D
http://s3.directupload.net/images/080123/qon9i85i.png <- oldaq
http://s6.directupload.net/images/080123/bzj6hm86.png <- newaq

There is still Darkness also with it (in the real meaning) :) (in motion it can hurt sometimes especialy if it's directly in the ROI as here)
http://s5.directupload.net/images/080123/4fpl8zs5.png <- oldaq
http://s5.directupload.net/images/080123/omgt9f7o.png <- newaq

When this blocking problems gonna get fixed i jump around all night and day (when it's possible to balance it out efficiently without destroying it's purpose (takeing bits from one area allocateing to another)) :D
http://s5.directupload.net/images/080123/hgbfhnta.png <- oldaq
http://s1.directupload.net/images/080123/jnbkrecp.png <- newaq

Let's love it again ;)
http://s5.directupload.net/images/080123/m5aava2r.png <- oldaq
http://s5.directupload.net/images/080123/xh6fd92z.png <- newaq

Old AQ ( --aq-strength 1.0 --aq-sensitivity 15)

x264 [info]: final ratefactor: 32.15
x264 [info]: 8x8 transform intra:19.1% inter:49.5%
x264 [info]: direct mvs spatial:98.5% temporal:1.5%
x264 [info]: ref P 82.8% 11.2% 6.0%
x264 [info]: ref B 91.0% 9.0%
x264 [info]: SSIM Mean Y:0.9780284
x264 [info]: PSNR Mean Y:44.630 U:45.375 V:48.696 Avg:45.190 Global:44.796 kb/s:
2944.99

encoded 9336 frames, 8.66 fps, 2952.16 kb/s

New AQ (--aq-strength 1.0 --aq-sensitivity 20)

x264 [info]: final ratefactor: 34.11
x264 [info]: 8x8 transform intra:20.9% inter:50.6%
x264 [info]: direct mvs spatial:98.2% temporal:1.8%
x264 [info]: ref P 84.3% 10.2% 5.6%
x264 [info]: ref B 91.8% 8.2%
x264 [info]: SSIM Mean Y:0.9787061
x264 [info]: PSNR Mean Y:43.929 U:45.177 V:48.162 Avg:44.574 Global:43.988 kb/s:
3014.86

encoded 9336 frames, 8.53 fps, 3022.31 kb/s

New AQ (--aq-strength 0.5 --aq-sensitivity 20)

x264 [info]: final ratefactor: 30.68
x264 [info]: 8x8 transform intra:18.1% inter:51.1%
x264 [info]: direct mvs spatial:98.7% temporal:1.3%
x264 [info]: ref P 84.3% 10.1% 5.6%
x264 [info]: ref B 93.1% 6.9%
x264 [info]: SSIM Mean Y:0.9802665
x264 [info]: PSNR Mean Y:45.205 U:45.745 V:49.185 Avg:45.725 Global:45.209 kb/s:
3020.03

encoded 9336 frames, 8.72 fps, 3027.55 kb/s

New AQ (--aq-strength 0.1 --aq-sensitivity 20)

x264 [info]: final ratefactor: 28.51
x264 [info]: 8x8 transform intra:15.4% inter:51.4%
x264 [info]: direct mvs spatial:98.9% temporal:1.1%
x264 [info]: ref P 83.8% 10.4% 5.8%
x264 [info]: ref B 93.8% 6.2%
x264 [info]: SSIM Mean Y:0.9804187
x264 [info]: PSNR Mean Y:45.724 U:45.997 V:49.670 Avg:46.197 Global:45.673 kb/s:
3023.08

encoded 9336 frames, 8.72 fps, 3030.47 kb/s

New AQ (--aq-strength 0.1)

x264 [info]: final ratefactor: 28.40
x264 [info]: 8x8 transform intra:15.4% inter:51.4%
x264 [info]: direct mvs spatial:98.9% temporal:1.1%
x264 [info]: ref P 83.8% 10.4% 5.8%
x264 [info]: ref B 93.8% 6.2%
x264 [info]: SSIM Mean Y:0.9804481
x264 [info]: PSNR Mean Y:45.729 U:45.999 V:49.670 Avg:46.200 Global:45.682 kb/s:
3026.98

encoded 9336 frames, 8.67 fps, 3034.38 kb/s

New AQ (--strength 0.5)

x264 [info]: final ratefactor: 28.24
x264 [info]: 8x8 transform intra:16.8% inter:51.3%
x264 [info]: direct mvs spatial:98.7% temporal:1.3%
x264 [info]: ref P 84.1% 10.2% 5.7%
x264 [info]: ref B 93.4% 6.6%
x264 [info]: SSIM Mean Y:0.9804409
x264 [info]: PSNR Mean Y:45.506 U:45.891 V:49.456 Avg:45.998 Global:45.485 kb/s:
3033.67

encoded 9336 frames, 8.64 fps, 3041.13 kb/s

New AQ (--strength 1.0)

x264 [info]: final ratefactor: 28.58
x264 [info]: 8x8 transform intra:19.0% inter:50.8%
x264 [info]: direct mvs spatial:98.5% temporal:1.5%
x264 [info]: ref P 84.2% 10.1% 5.6%
x264 [info]: ref B 92.6% 7.4%
x264 [info]: SSIM Mean Y:0.9798093
x264 [info]: PSNR Mean Y:44.864 U:45.574 V:48.843 Avg:45.410 Global:44.888 kb/s:
3044.61

encoded 9336 frames, 8.60 fps, 3052.14 kb/s

All in all this is unbeliveable Detail Preservation (my wildest dreams since Grasprite come true) (ignoring it's side effects, for sure also partly coused by the insane bitrate) :)

[nm]

Hmm could you guys please post wich of these looks better to you on the first sight and then also your GFX Card, Monitor ,Operating System, Player Application,Decoder and used Renderer as information. Thx in Advance :)

Please don't look framewise just subjectively rate them on the first sight :)

http://mirror05.x264.nl/CruNcher/force.php?file=./1.mkv
http://mirror05.x264.nl/CruNcher/force.php?file=./2.mkv
http://mirror05.x264.nl/CruNcher/force.php?file=./3.mkv

Order from best to worst: 1, 2, 3
The first sample is noticeably better than the others. I saw difference between 2 and 3 only by scaling up the video to see the details.

Gfx card: NVIDIA GeForce4 Ti4200
Monitor: 21" CRT (resolution 1600x1200)
OS: Linux (NVIDIA binary blob X.org drivers)
Player/Decoder: MPlayer/libavcodec
Renderer: OpenGL output (same results with XVideo)

[Boardlord]

Hi all!

Has anyone tried the AQ on a concert footage? I tried to backup David Gilmour's "Remember that night" concert. I thought it would be a good test for Dark's AQ. I am sorry in advance that I have no pictures, but I am writing from my workplace, as I won't have net at home till March.
On the problems: the picture looked good, *very* little blocking, I was amazed! But, there were randomly appearing little black dot-like artifacts in the picture for a few secs, which then vanished, reappeared and so on. Tried it with Trellis, on-off (although Dark said Trellis doesn't matter). Aq strength was 0.8, sensitivity was tried at auto and 20 - same result. I used bob0r's compile. Setting weren't extreme (ref:5, bframes:3 (spatial), subme: 6, threads: 3, b-pyramid, no fast pskip, the rest were megui defaults)If needed, I'll post some pictures tomorrow, along with an x264 log. Of course the black dots were not present when AQ was not used... but blocks were :(

[salehin]

Hi all!

Has anyone tried the AQ on a concert footage? I tried to backup David Gilmour's "Remember that night" concert. I thought it would be a good test for Dark's AQ. I am sorry in advance that I have no pictures, but I am writing from my workplace, as I won't have net at home till March.
On the problems: the picture looked good, *very* little blocking, I was amazed! But, there were randomly appearing little black dot-like artifacts in the picture for a few secs, which then vanished, reappeared and so on. Tried it with Trellis, on-off (although Dark said Trellis doesn't matter). Aq strength was 0.8, sensitivity was tried at auto and 20 - same result. I used bob0r's compile. Setting weren't extreme (ref:5, bframes:3 (spatial), subme: 6, threads: 3, b-pyramid, no fast pskip, the rest were megui defaults)If needed, I'll post some pictures tomorrow, along with an x264 log. Of course the black dots were not present when AQ was not used... but blocks were :(

Try with bobr's again by using weak aq (not more than 0.50) and sensitivity of 20-25. Compare to Dark's build with aq 1.0 and sens .25 (posted in the 1st post), it gave me better results. However, note that my source contains very little dark sequence (see sample tests with other settings- links posted above)

update: perhaps you can use one of the CQMs (see the custom matrix bit (http://forum.doom9.org/showpost.php?p=1056555&postcount=3)) to see if it improves the transparency

[Morte66]

@DS

I thought you might be interested in this clip from 3 Colours White (http://www.mediafire.com/?dm1gmc1d2fl) (ffv1 in avi). It's been cropped/levelled/deblocked/denoised/debanded from DVD. It's my "worst reasonable case for AQ" material. I expect an encoder to handle it gracefully, if it's to get used for sight-unseen DVD backup.

It's assumed that it will be played back with Deband or GradFunkMirror() in ffdshow, which means it just has to avoid aggregating small blocks into bigger blocks.

[Dark Shikari]

@DS

I thought you might be interested in this clip from 3 Colours White (http://www.mediafire.com/?dm1gmc1d2fl) (ffv1 in avi). It's been cropped/levelled/deblocked/denoised/debanded from DVD. It's my "worst reasonable case for AQ" material. I expect an encoder to handle it gracefully, if it's to get used for sight-unseen DVD backup.

It's assumed that it will be played back with Deband or GradFunkMirror() in ffdshow, which means it just has to avoid aggregating small blocks into bigger blocks.
That was easy (http://www.mediafire.com/?5d1fywox22w). Just 600 kilobits, CRF mode, was plenty.

On this topic, it seems CRF mode seems to work fine with AQ if you use a reasonable sensitivity (20).

[Blue_MiSfit]

Hmm could you guys please post wich of these looks better to you on the first sight and then also your GFX Card, Monitor ,Operating System, Player Application,Decoder and used Renderer as information. Thx in Advance :)

Please don't look framewise just subjectively rate them on the first sight :)

http://mirror05.x264.nl/CruNcher/force.php?file=./1.mkv
http://mirror05.x264.nl/CruNcher/force.php?file=./2.mkv
http://mirror05.x264.nl/CruNcher/force.php?file=./3.mkv

Btw: If you want to see allways correct representation results in Windows with a Directshow Player use CoreAVC as Decoder :)


I like them in order. 1 is best followed by 2 and 3. Double checked.

8800GT, BenQ 24" LCD, Windows XP, Media Player Classic, CoreAVC, Haali Renderer.

Did the comparison at full screen (Scaled to 1920x1200). While 2 and 3 seemed to have more details in high frequency areas, the flat areas lost detail, and had irregular noise.

1 seemed to have the most even picture, and if there was any detail loss in the high frequency areas, it didn't distract me at all. The loss of detail in flat areas in the other versions WAS very distracting.

Excellent work! Let's see those settings!

~MiSfit