x264 "Macroblock Tree Ratecontrol" testing (committed)

[Dark Shikari]

http://i32.tinypic.com/2ev802e.jpg (http://i29.tinypic.com/34fiq9w.png)

Warning: (not really) Highly Experimental

It's done and committed! Grab the latest x264 from x264.nl!

Patch RC1 (http://pastebin.com/f87abd3f)
Download RC1 (Windows, old) (http://mirror05.x264.nl/Dark/x264_mbtree_rc1.exe)
Download RC1 (Windows debug build, old) (http://mirror05.x264.nl/Dark/x264_mbtree_rc1_debug.exe)
Patch v0.13 (http://pastebin.com/f6622f24d)
Download 0.13 (Windows) (http://mirror05.x264.nl/Dark/x264_mbtree13.exe)
Download 0.13 (Windows debug build) (http://mirror05.x264.nl/Dark/x264_mbtree13_debug.exe)
Patch v0.11 (http://pastebin.com/fe144569)
Download 0.11 (Windows, old) (http://mirror05.x264.nl/Dark/x264_mbtree11.exe)
Download 0.11 (Windows debug build, old) (http://mirror05.x264.nl/Dark/x264_mbtree11_debug.exe)
Patch v0.09 (old) (http://pastebin.com/f364aedf1)
Download 0.09 (Windows, old) (http://mirror05.x264.nl/Dark/x264_mbtree.exe)
Download 0.09 (Windows debug build, old) (http://mirror05.x264.nl/Dark/x264_mbtree_debug.exe)

Updates:

Changes in RC1:

1. Statsfile format changed; now takes only 2 bytes per macroblock. Varying-endianness now supported. Error handling improved.

2. More commandline validation fixes.

3. Clip propagation factors just in case they overflow.

4. Change default lookahead to 40 to save some memory.

5. It's now called --rc-lookahead, not --lookahead, at Akupenguin's request.

6. Various other things, just read the damn commit message in the patch.

Changes in 0.13:

1. Bugs fixed in the VBV patch. It's still separate, as per below.

2. B-adapt 1 now supported.

3. All B-adapt code is now merged into MB-tree code instead of duplicated. B-adapt 1 should be slightly better than it was before (even without MB-tree). In general, all three B-adapt modes now use the same basic framework.

4. MB-tree strength is now adjustable through qcomp. Raising qcomp lowers MB-tree strength; it works the same way that qcomp did before MB-tree, with qcomp 1 meaning constant frame quantizer. This lets you experiment a bit more; there's some possibility that the default qcomp 0.6 is not the psy-optimal value--it might be higher.

5. New --no-psy option, which disables all psy optimizations in x264 that don't improve PSNR or SSIM. This includes psy-RD/trellis, but also the recently-added chroma lambda offset and the magical "num_frames = j" line that causes MB-tree to pretend that non-scenecut I-frames don't exist.

6. "Fast" preset is now slower, "faster" preset added to cover the gap.

7. MB-tree is now default on presets "fast" and higher. Lookahead values are smaller for faster presets.

Changes in 0.11:

1. VBV now supported. Includes a separate VBV patch which fixes most VBV issues that MB-tree has; this will be committed separately, as it doesn't require MB-tree. It should also improve 1-pass VBV (especially CBR) performance on clips with lots of static content, e.g. anime, CGI.

Do note that CBR doesn't benefit much from MB-tree though, especially with small buffer sizes.

2. Lots of trivial parameter-handling fixes, e.g. lookahead automatically disabled on second pass.

FAQ:

How do I use it?

It's on by default. --no-mbtree turns it off.

What doesn't it work with yet?

--b-pyramid

How can I tweak it?

--lookahead; default is 50, higher takes more memory and is slower, but should give better results. It's probably overkill for most sources. If you get crashes immediately after the lookahead fills up, you probably ran out of memory (max is 2GB for x264 on 32-bit Windows).

--qcomp; works the same as it did before with actual qcomp, higher numbers lower MB-tree strength.

How much does it cost, speed-wise?

A bit. Not too much though. Speed cost is linearly proportional to --lookahead.

How can I help test it?

Use it.

Make sure you don't get any odd errors or warnings, such as asserts failing, crashes, etc. Try various combinations of the relevant settings (b-adapt, bframes, mbtree, lookahead, pass 1/2, crf, keyint, scenecut) that I might not have thought of.

Try it on various sources; see if you can find places where, with --tune ssim, at the same bitrate as without mbtree, it gives worse SSIM (or, equally, places where Global PSNR goes down with tune psnr).

Make sure it doesn't fall apart visually either.

Also, note that CRF is redefined again by this patch--I set it so that it kept the same bitrate on Blackpearl as before, but it may vary for different videos.

What does it actually do?

It tracks the propagation of information from future blocks to past blocks across motion vectors. It could be described as localizing qcomp to act on individual blocks instead of whole scenes. Thus instead of lowering quality in high-complexity scenes (like x264 currently does), it'll only lower quality on the complex part of the scene, while for example a static background will remain high-quality. It also has many other more subtle effects, some potentially negative, most probably not.

This helps at all bitrates, but can even help at phenomenally low bitrates where the video would otherwise fall apart completely, like in this 67kbps anime clip (http://mirror05.x264.nl/Dark/Flash/lowbitrateanime.html).

Should I go around distributing this patch and getting everyone I know to use it for their everyday encodes?

No. The commandline options here may change in the future, as may the statsfile format and various other aspects of the patch, so it's not a good idea to make this patch something that gets put in ordinary x264 builds. Furthermore, I might have broken something unrelated to mbtree.

Why does it give different output even without mbtree?

B-adapt (all modes) was slightly modified to work better with mbtree, so the frametype decisions may change somewhat.

What improvement does it give?

I've gotten up to a 70% SSIM increase.

You're joking, right?

No.

Seriously? How the hell can you get a 70% quality increase?

Magic.

OK, you're just making this u--wait a minute, what's this new statsfile, and why is it using up all my hard disk space?

Oh, that? (http://i26.tinypic.com/2s6odxt.jpg) Kyahahahahahaha. (http://mirror05.x264.nl/Dark/ahaha.wav)

[Snowknight26]

You know you want to provide a 64-bit Windows binary. :D
Any reason why the exe is so tiny?

I have to say, clever picture..

[Dark Shikari]

You know you want to provide a 64-bit Windows binary. :Dtry this (http://imk.cx/junk/x264_mbtree.7z)Any reason why the exe is so tiny?kkrunchy (http://www.farbrausch.de/~fg/kkrunchy/)I have to say, clever picture..Kuukunen made that.

[Audionut]

Magic.

Black?

[Snowknight26]

kkrunchy (http://www.farbrausch.de/~fg/kkrunchy/)
Farbrausch, should have known. I dare say I'm more impressed by that than the mbtree. :p

Funny how the QP for the mbtree version is twofold that of a normal build:


C:\temp\Encoding\test>cd ..

C:\temp\Encoding>avs2yuv -raw test/test.avs - | x264_mbtree_x64 --crf 19 --mbt
ree --bframes 4 --b-adapt 2 --ref 4 --b-pyramid --direct auto --deblock -3:-3 -
-subme 10 --trellis 2 --analyse all --no-dct-decimate --me umh --ssim --fps 2400
0/1001 --output "test/test.mbtree.mkv" - 480x320
x264 [info]: 480x320 @ 23.98 fps
x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 Cache64
x264 [info]: profile High, level 2.1
test/test.avs: 480x320, 10000000/417083 fps, 481 frames
x264 [info]: slice I:240 Avg QP:20.82 size: 79
x264 [info]: slice P:241 Avg QP:24.08 size: 79
x264 [info]: consecutive B-frames: 100.0% 0.0% 0.0% 0.0% 0.0%
x264 [info]: mb I I16..4: 100.0% 0.0% 0.0%
x264 [info]: mb P I16..4: 90.4% 0.0% 0.0% P16..4: 9.2% 0.0% 0.0% 0.0% 0
.0% skip: 0.4%
x264 [info]: 8x8 transform intra:0.0% inter:-1.$%
x264 [info]: coded y,uvDC,uvAC intra:0.0% 0.1% 0.0% inter:0.0% 0.0% 0.0%
x264 [info]: ref P L0 0.0% 100.0%
x264 [info]: SSIM Mean Y:1.0000000
x264 [info]: kb/s:15.2

encoded 481 frames, 151.83 fps, 16.87 kb/s

C:\temp\Encoding>avs2yuv -raw test/test.avs - | x264.x64 --crf 19 --bframes 4
--b-adapt 2 --ref 4 --b-pyramid --direct auto --deblock -3:-3 --subme 10 --trel
lis 2 --analyse all --no-dct-decimate --me umh --ssim --fps 24000/1001 --output
"test/test.mkv" - 480x320
x264 [info]: 480x320 @ 23.98 fps
x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 Cache64
x264 [info]: profile High, level 2.1
test/test.avs: 480x320, 10000000/417083 fps, 481 frames
x264 [info]: slice I:240 Avg QP:10.04 size: 83
x264 [info]: slice P:241 Avg QP:10.01 size: 91
x264 [info]: consecutive B-frames: 100.0% 0.0% 0.0% 0.0% 0.0%
x264 [info]: mb I I16..4: 100.0% 0.0% 0.0%
x264 [info]: mb P I16..4: 90.4% 0.0% 0.0% P16..4: 9.2% 0.0% 0.0% 0.0% 0
.0% skip: 0.4%
x264 [info]: 8x8 transform intra:0.0% inter:-1.$%
x264 [info]: coded y,uvDC,uvAC intra:0.0% 0.1% 0.0% inter:0.0% 0.0% 0.0%
x264 [info]: ref P L0 0.0% 100.0%
x264 [info]: SSIM Mean Y:1.0000000
x264 [info]: kb/s:16.7

encoded 481 frames, 162.39 fps, 18.08 kb/s

C:\temp\Encoding>pause
Press any key to continue . . .

And what did you mean by doesn't work with --b-pyramid? Different outputs seems to show that it does something differently.

[Dark Shikari]

Black?Of course. (http://i31.tinypic.com/2qd4nif.jpg)

[IgorC]

--lookahead; default is 50
Is lookahead measured in frames? 50 frames?

Great patch. Untill now I can say I get +4% of bitrate improvement for the same OPSNR and +20% for the same SSIM (foreman test sequence).

[Dark Shikari]

Farbrausch, should have known. I dare say I'm more impressed by that than the mbtree. :p

Funny how the QP for the mbtree version is twofold that of a normal build:


C:\temp\Encoding\test>cd ..

C:\temp\Encoding>avs2yuv -raw test/test.avs - | x264_mbtree_x64 --crf 19 --mbt
ree --bframes 4 --b-adapt 2 --ref 4 --b-pyramid --direct auto --deblock -3:-3 -
-subme 10 --trellis 2 --analyse all --no-dct-decimate --me umh --ssim --fps 2400
0/1001 --output "test/test.mbtree.mkv" - 480x320
x264 [info]: 480x320 @ 23.98 fps
x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 Cache64
x264 [info]: profile High, level 2.1
test/test.avs: 480x320, 10000000/417083 fps, 481 frames
x264 [info]: slice I:240 Avg QP:20.82 size: 79
x264 [info]: slice P:241 Avg QP:24.08 size: 79
x264 [info]: consecutive B-frames: 100.0% 0.0% 0.0% 0.0% 0.0%
x264 [info]: mb I I16..4: 100.0% 0.0% 0.0%
x264 [info]: mb P I16..4: 90.4% 0.0% 0.0% P16..4: 9.2% 0.0% 0.0% 0.0% 0
.0% skip: 0.4%
x264 [info]: 8x8 transform intra:0.0% inter:-1.$%
x264 [info]: coded y,uvDC,uvAC intra:0.0% 0.1% 0.0% inter:0.0% 0.0% 0.0%
x264 [info]: ref P L0 0.0% 100.0%
x264 [info]: SSIM Mean Y:1.0000000
x264 [info]: kb/s:15.2

encoded 481 frames, 151.83 fps, 16.87 kb/s

C:\temp\Encoding>avs2yuv -raw test/test.avs - | x264.x64 --crf 19 --bframes 4
--b-adapt 2 --ref 4 --b-pyramid --direct auto --deblock -3:-3 --subme 10 --trel
lis 2 --analyse all --no-dct-decimate --me umh --ssim --fps 24000/1001 --output
"test/test.mkv" - 480x320
x264 [info]: 480x320 @ 23.98 fps
x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 Cache64
x264 [info]: profile High, level 2.1
test/test.avs: 480x320, 10000000/417083 fps, 481 frames
x264 [info]: slice I:240 Avg QP:10.04 size: 83
x264 [info]: slice P:241 Avg QP:10.01 size: 91
x264 [info]: consecutive B-frames: 100.0% 0.0% 0.0% 0.0% 0.0%
x264 [info]: mb I I16..4: 100.0% 0.0% 0.0%
x264 [info]: mb P I16..4: 90.4% 0.0% 0.0% P16..4: 9.2% 0.0% 0.0% 0.0% 0
.0% skip: 0.4%
x264 [info]: 8x8 transform intra:0.0% inter:-1.$%
x264 [info]: coded y,uvDC,uvAC intra:0.0% 0.1% 0.0% inter:0.0% 0.0% 0.0%
x264 [info]: ref P L0 0.0% 100.0%
x264 [info]: SSIM Mean Y:1.0000000
x264 [info]: kb/s:16.7

encoded 481 frames, 162.39 fps, 18.08 kb/s

C:\temp\Encoding>pause
Press any key to continue . . .It looks like your source failed, because your input was all black ;)

Is lookahead measured in frames? 50 frames?Yes.

[Snowknight26]

It looks like your source failed, because your input was all black ;)

It's not actually. (http://stfcc.org/misc/test.mp4)

Crash after 100 frames:

C:\temp\Encoding\test>cd ..

C:\temp\Encoding>avs2yuv -raw test/test.avs - | x264_mbtree_x64 --crf 19 --mbt
ree --lookahead 500 --bframes 16 --b-adapt 2 --ref 4 --b-pyramid --direct auto
--deblock -3:-3 --subme 10 --trellis 2 --analyse all --no-dct-decimate --me umh
--ssim --fps 24000/1001 --output "test/test.mbtree.mkv" - 1920x816
x264 [info]: 1920x816 @ 23.98 fps
x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 Cache64
x264 [info]: profile High, level 4.0
test/test.avs: 1920x816, 10000000/417083 fps, 1378 frames
Output error: wrote only 1785423 of 2350080 bytes

C:\temp\Encoding>avs2yuv -raw test/test.avs - | x264.x64 --crf 19 --bframes 16
--b-adapt 2 --ref 4 --b-pyramid --direct auto --deblock -3:-3 --subme 10 --tre
llis 2 --analyse all --no-dct-decimate --me umh --ssim --fps 24000/1001 --output
"test/test.mkv" - 1920x816
x264 [info]: 1920x816 @ 23.98 fps
x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 Cache64
x264 [info]: profile High, level 4.0
test/test.avs: 1920x816, 10000000/417083 fps, 1378 frames
250 frames: 2.03 fps, 7743.86 kb/s
Other one is still going.. samples a bit big so hopefully someone can provide a debug build before I resort to uploading it.

[Dark Shikari]

All crashes that I've seen so far that occurred with very large --lookahead values traced back perfectly to x264_malloc, so you should check with gdb before reporting such issues as bugs.

[Snowknight26]

I have to say, the bitrate difference can be enormous sometimes:
C:\temp\Encoding>avs2yuv -raw test/test.avs - | x264_mbtree_x64 --crf 19 --mbt
ree --bframes 0 --b-adapt 0 --ref 0 --direct spatial --deblock -3:-3 --subme 1 -
-trellis 0 --analyse all --me umh --ssim --fps 24000/1001 --output "test/test.mb
tree.mkv" - 1920x816
x264 [info]: 1920x816 @ 23.98 fps
x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 Cache64
x264 [info]: profile High, level 4.0
test/test.avs: 1920x816, 10000000/417083 fps, 1378 frames
x264 [info]: slice I:20 Avg QP:14.02 size:131728
x264 [info]: slice P:1358 Avg QP:15.01 size: 37773
x264 [info]: mb I I16..4: 24.8% 47.2% 28.0%
x264 [info]: mb P I16..4: 22.4% 18.9% 1.6% P16..4: 35.4% 8.6% 1.8% 0.1% 0
.0% skip:11.2%
x264 [info]: 8x8 transform intra:44.2% inter:41.4%
x264 [info]: coded y,uvDC,uvAC intra:35.8% 87.5% 74.4% inter:13.7% 58.9% 19.6%
x264 [info]: SSIM Mean Y:0.9929551
x264 [info]: kb/s:7506.7

encoded 1378 frames, 30.29 fps, 7506.94 kb/s

C:\temp\Encoding>avs2yuv -raw test/test.avs - | x264.x64 --crf 19 --bframes 0
--b-adapt 0 --ref 0 --direct spatial --deblock -3:-3 --subme 1 --trellis 0 --ana
lyse all --me umh --ssim --fps 24000/1001 --output "test/test.mkv" - 1920x816
x264 [info]: 1920x816 @ 23.98 fps
x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 Cache64
x264 [info]: profile High, level 4.0
test/test.avs: 1920x816, 10000000/417083 fps, 1378 frames
x264 [info]: slice I:21 Avg QP:10.66 size:187241
x264 [info]: slice P:1357 Avg QP:11.45 size: 73091
x264 [info]: mb I I16..4: 23.5% 37.2% 39.3%
x264 [info]: mb P I16..4: 20.8% 17.8% 5.0% P16..4: 32.7% 10.9% 4.0% 0.4% 0
.1% skip: 8.4%
x264 [info]: 8x8 transform intra:40.7% inter:31.8%
x264 [info]: coded y,uvDC,uvAC intra:51.6% 88.9% 81.5% inter:36.5% 73.8% 41.9%
x264 [info]: SSIM Mean Y:0.9953622
x264 [info]: kb/s:14353.1

encoded 1378 frames, 26.12 fps, 14353.34 kb/s

[Audionut]

Well you're shooting for crf 19 and getting AVG QP's of 10.

[Dark Shikari]

I have to say, the bitrate difference can be enormous sometimes:Or... maybe it's a bug?

Upload a sample of the output.

[Snowknight26]

http://stfcc.org/misc/test.mbtree.10mb.mkv
http://stfcc.org/misc/test.10mb.mkv

[Dark Shikari]

http://stfcc.org/misc/test.mbtree.10mb.mkv
http://stfcc.org/misc/test.10mb.mkvNope, it seems like MB-tree is in the right here; x264 was lowering the quantizers way too much before for its own good.

[Snowknight26]

All crashes that I've seen so far that occurred with very large --lookahead values traced back perfectly to x264_malloc, so you should check with gdb before reporting such issues as bugs.

Couldn't get 64-bit x264+avs2yuv working with gdb but nevertheless, seems you were right:


C:\temp\Encoding\test>cd ..

C:\temp\Encoding>gdb --args x264_mbtree_debug --crf 19 --mbtree --lookahead 500
--bframes 16 --b-adapt 2 --ref 4 --b-pyramid --direct auto --deblock -3:-3 --su
bme 10 --trellis 2 --analyse all --no-dct-decimate --me umh --ssim --fps 24000/1
001 --output "test/test.mbtree.mkv" "test/test.avs"
GNU gdb 6.8
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "i686-pc-mingw32"...
(gdb) run
Starting program: C:\temp\Encoding/x264_mbtree_debug.exe --crf 19 --mbtree --loo
kahead 500 --bframes 16 --b-adapt 2 --ref 4 --b-pyramid --direct auto --deblock
-3:-3 --subme 10 --trellis 2 --analyse all --no-dct-decimate --me umh --ssim --f
ps 24000/1001 --output test/test.mbtree.mkv test/test.avs
[New thread 4380.0x10c0]
Error: dll starting at 0x76f10000 not found.
Error: dll starting at 0x768d0000 not found.
Error: dll starting at 0x76f10000 not found.
Error: dll starting at 0x76e10000 not found.
warning: DllMain: hModule=0x10000000, ulReason=1, lpReserved=0x00000000, gRefCnt
= 0

warning: DllGetClassObject() CLSID: CAVIFileSynth

warning: 00943450->CAVIFileSynth::CAVIFileSynth()

warning: 00943450->CAVIFileSynth::AddRef() gRefCnt=1, m_refs=1

warning: 00943450->CAVIFileSynth::QueryInterface() {00000001-0000-0000-c000-0000
00000046} (IClassFactory)

warning: 00943450->CAVIFileSynth::AddRef() gRefCnt=2, m_refs=2

warning: 00943450->CAVIFileSynth::Release() gRefCnt=1, m_refs=1

warning: DllGetClassObject() result=0x0, object=00943458

warning: 00943450->CAVIFileSynth::CreateInstance()

warning: 009434A8->CAVIFileSynth::CAVIFileSynth()

warning: 009434A8->CAVIFileSynth::AddRef() gRefCnt=2, m_refs=1

warning: 009434A8->CAVIFileSynth::QueryInterface() {00000000-0000-0000-c000-0000
00000046} (IUnknown)

warning: 009434A8->CAVIFileSynth::AddRef() gRefCnt=3, m_refs=2

warning: 009434A8->CAVIFileSynth::Release() gRefCnt=2, m_refs=1

warning: 00943450->CAVIFileSynth::CreateInstance() result=0x0, object=009434A8

warning: 009434A8->CAVIFileSynth::AddRef() gRefCnt=3, m_refs=2

warning: 009434A8->CAVIFileSynth::Release() gRefCnt=2, m_refs=1

warning: 00943450->CAVIFileSynth::Release() gRefCnt=1, m_refs=0

warning: 00943450->CAVIFileSynth::~CAVIFileSynth(), gRefCnt = 1

warning: 009434A8->CAVIFileSynth::QueryInterface() {00000000-0000-0000-c000-0000
00000046} (IUnknown)

warning: 009434A8->CAVIFileSynth::AddRef() gRefCnt=2, m_refs=2

warning: 009434A8->CAVIFileSynth::Release() gRefCnt=1, m_refs=1

warning: 009434A8->CAVIFileSynth::QueryInterface() {00020025-0000-0000-c000-0000
00000046} (unsupported!)

warning: 009434A8->CAVIFileSynth::QueryInterface() {0000010b-0000-0000-c000-0000
00000046} (IPersistFile)

warning: 009434A8->CAVIFileSynth::AddRef() gRefCnt=2, m_refs=2

warning: 009434A8->CAVIFileSynth::QueryInterface() {00020020-0000-0000-c000-0000
00000046} (IAVIFile)

warning: 009434A8->CAVIFileSynth::AddRef() gRefCnt=3, m_refs=3

warning: 009434A8->CAVIFileSynth::Load("test/test.avs", 0x0)

warning: 009434A8->CAVIFileSynth::Release() gRefCnt=2, m_refs=2

warning: 009434A8->CAVIFileSynth::Release() gRefCnt=1, m_refs=1

warning: 009434A8->CAVIFileSynth::GetStream(*, 73646976(vids), 0)

[New thread 4380.0xd1c]
warning: DllMain: hModule=0x10000000, ulReason=2, lpReserved=0x00000000, gRefCnt
= 1

[New thread 4380.0x13c8]
[New thread 4380.0xb7c]
warning: DllMain: hModule=0x10000000, ulReason=2, lpReserved=0x00000000, gRefCnt
= 1

warning: DllMain: hModule=0x10000000, ulReason=2, lpReserved=0x00000000, gRefCnt
= 1

[New thread 4380.0x13f8]
[New thread 4380.0x10bc]
[New thread 4380.0x8b4]
warning: DllMain: hModule=0x10000000, ulReason=2, lpReserved=0x00000000, gRefCnt
= 1

[New thread 4380.0xb24]
[New thread 4380.0x80]
warning: DllMain: hModule=0x10000000, ulReason=2, lpReserved=0x00000000, gRefCnt
= 1

warning: DllMain: hModule=0x10000000, ulReason=2, lpReserved=0x00000000, gRefCnt
= 1

[New thread 4380.0x644]
warning: DllMain: hModule=0x10000000, ulReason=2, lpReserved=0x00000000, gRefCnt
= 1

warning: DllMain: hModule=0x10000000, ulReason=2, lpReserved=0x00000000, gRefCnt
= 1

[...]


warning: DllMain: hModule=0x10000000, ulReason=3, lpReserved=0x00000000, gRefCnt
= 2

[New thread 4380.0xfb8]35.18 fps, 0.00 kb/s, eta 0:00:36
warning: DllMain: hModule=0x10000000, ulReason=2, lpReserved=0x00000000, gRefCnt
= 2

warning: DllMain: hModule=0x10000000, ulReason=3, lpReserved=0x00000000, gRefCnt
= 2

[New thread 4380.0xac8]35.08 fps, 0.00 kb/s, eta 0:00:36
warning: DllMain: hModule=0x10000000, ulReason=2, lpReserved=0x00000000, gRefCnt
= 2

warning: DllMain: hModule=0x10000000, ulReason=3, lpReserved=0x00000000, gRefCnt
= 2

[New thread 4380.0xed0]34.99 fps, 0.00 kb/s, eta 0:00:36

Program received signal SIGSEGV, Segmentation fault.
x264_malloc (i_size=6983680) at common/common.c:729
729 common/common.c: No such file or directory.
in common/common.c
(gdb) bt
#0 x264_malloc (i_size=6983680) at common/common.c:729
#1 0x00424229 in x264_frame_new (h=<incomplete type>) at common/frame.c:69
#2 0x00427bdb in x264_frame_pop_unused (h=<incomplete type>)
at common/frame.c:959
#3 0x0040de2c in x264_encoder_encode (h=<incomplete type>, pp_nal=0x28fa78,
pi_nal=0x28fa7c, pic_in=0x28fc60, pic_out=0x28fa80)
at encoder/encoder.c:1454
#4 0x00402fdd in Encode_frame (h=0x70cb2a0, hout=0x942e28, pic=0x28fc60)
at x264.c:1043
#5 0x00403335 in Encode (param=0x28fd00, opt=0x28fce0) at x264.c:1127
#6 0x004013c0 in main (argc=32, argv=0x9431a8) at x264.c:113
(gdb)

[imk]

Here's my quick tests:

http://pastie.org/569295
http://imk.cx/temp/dead_space_mbtree0.mkv (13,94 MB)
http://imk.cx/temp/dead_space_mbtree1.mkv (13,95 MB)

SSIM
0.9728932 -> 0.9740763

Here's two comparison shots:

http://imk.cx/temppics/t_00000060_0.png.png (http://imk.cx/temppics/00000060_0.png) http://imk.cx/temppics/t_00000060_1.png.png (http://imk.cx/temppics/00000060_1.png)

Dark_Shikari let me know that single frame comparisons are useless for ratecontrol comparisons, but with the above shot, you can see that it brings out some more detail in the darker areas.

[thewebchat]

Honestly, the only difference I see in that screenshot is that the position of the blocking artifacts shifted.

[Audionut]

Pass 2 is slower with --b-adapt 2 :confused:

[benwaggoner]

What does it actually do?[/b]

It tracks the propagation of information from future blocks to past blocks across motion vectors. It could be described as localizing qcomp to act on individual blocks instead of whole scenes. Thus instead of lowering quality in high-complexity scenes (like x264 currently does), it'll only lower quality on the complex part of the scene, while for example a static background will remain high-quality. It also has many other more subtle effects, some potentially negative, most probably not.
Cool. When you had me check out that Touhou clip a while ago, I figured something like this would be what was required. Have you tried it with this build?

The only decent looking encode of that I ever got was using PEP and ROI encoding, specifying a max QP for the static parts of the image so that the crazy motion parts wouldn't starve the background of bits and drive up their QP so high. Once encoded at a decent QP on the I-frame, they would be pretty much skip blocks the rest of the GOP and so didn't take up material bandwidth anyway.

Cool to see something that could achieve similar results without manual ROI and with moving images.

[Dark Shikari]

Cool. When you had me check out that Touhou clip a while ago, I figured something like this would be what was required. Have you tried it with this build?That's what I got the 70% on. I got ~50% with some other clips though, like anime and Akiyo.
The only decent looking encode of that I ever got was using PEP and ROI encoding, specifying a max QP for the static parts of the image so that the crazy motion parts wouldn't starve the background of bits and drive up their QP so high. Once encoded at a decent QP on the I-frame, they would be pretty much skip blocks the rest of the GOP and so didn't take up material bandwidth anyway.

Cool to see something that could achieve similar results without manual ROI and with moving images.It does even better than that, because, for example, when a bullet passes a background area, it will instantly restore the quality of said background area once the bullet leaves.

It naturally does a whole lot of these sort of things in addition to the obvious effect of locally optimal qcomp.

[Zachs]

Would this help in scenes with foreign dialogue subtitles on moving background too?

That's one area I can see macroblocking issues with the current version of x264 on my encodes.

[TheImperial2004]

Sorry for my noobness :)

I use AviDemux so I need to compile libx264 DLL with this patch ... Can somebody point me to the right direction ?

Edit : Found It !

[Dark Shikari]

Would this help in scenes with foreign dialogue subtitles on moving background too?

That's one area I can see macroblocking issues with the current version of x264 on my encodes.Your best bet there is to just lower AQ strength, but this might help there as well; I haven't tried it though.

Or do it right and use softsubs ;)

[Zachs]

Or do it right and use softsubs

It's some foreign dialogue hardsubbed in the source (BDRip).

And, it's only for a few seconds of the movie.

Is this patch going to be incompatible with vbv-buf max / b-pyramid etc. or is it only temporarily as they get implemented?

[Audionut]

Doesn't seem to increase ssim on this test.

source = ep 5 of band of brothers. Blu-ray source.

avs2yuv -raw e:\test.avs - | x264_mbtree_x64 --preset slower --tune ssim --pass 1 -B 6000
--level 4.1 -m 10 --mbtree --lookahead 240 --keyint 240 --min-keyint 24 --ssim --psnr --fps 24000/1001 -o e:\mbtree_ssim.mkv - 1280x720

x264 [info]: 1280x720 @ 23.98 fps
x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 Cache64
x264 [info]: profile Main, level 4.1
e:\test.avs: 1280x720, 24000/1001 fps, 5726 frames

x264 [info]: slice I:147 Avg QP:24.78 size: 76029 PSNR Mean Y:39.09 U:50.35 V:50.53 Avg:40.68 Global:40.03
x264 [info]: slice P:2508 Avg QP:27.14 size: 41571 PSNR Mean Y:36.36 U:49.38 V:49.50 Avg:38.01 Global:37.59
x264 [info]: slice B:3071 Avg QP:29.23 size: 19866 PSNR Mean Y:35.66 U:49.43 V:49.60 Avg:37.32 Global:36.91
x264 [info]: consecutive B-frames: 10.4% 37.2% 33.9% 18.4%
x264 [info]: mb I I16..4: 45.1% 0.0% 54.9%
x264 [info]: mb P I16..4: 32.0% 0.0% 0.0% P16..4: 50.9% 0.0% 0.0% 0.0% 0.0% skip:17.1%
x264 [info]: mb B I16..4: 7.2% 0.0% 0.0% B16..8: 33.4% 0.0% 0.0% direct:19.3% skip:40.2% L0:28.4% L1:40.2% BI:31.4%
x264 [info]: final ratefactor: 22.88
x264 [info]: direct mvs spatial:99.9% temporal:0.1%
x264 [info]: coded y,uvDC,uvAC intra:62.8% 12.0% 0.5% inter:36.8% 2.1% 0.0%
x264 [info]: SSIM Mean Y:0.9443182
x264 [info]: PSNR Mean Y:36.054 U:49.430 V:49.581 Avg:37.709 Global:37.258 kb/s:
5910.50

encoded 5726 frames, 19.40 fps, 5910.66 kb/s

avs2yuv -raw e:\test.avs - | x264_mbtree_x64 --preset slower --tune ssim --pass 2 -B 6000
--level 4.1 -m 10 --mbtree --lookahead 240 --keyint 240 --min-keyint 24 --ssim --psnr --fps 24000/1001 -o e:\mbtree_ssim.mkv - 1280x720

x264 [info]: 1280x720 @ 23.98 fps
x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 Cache64
e:\test.avs: 1280x720, 24000/1001 fps, 5726 frames
x264 [info]: profile High, level 4.1

x264 [info]: slice I:147 Avg QP:25.90 size: 68481 PSNR Mean Y:39.34 U:50.24 V:50.31 Avg:40.92 Global:40.44
x264 [info]: slice P:2508 Avg QP:27.95 size: 42077 PSNR Mean Y:37.73 U:49.76 V:49.87 Avg:39.35 Global:39.09
x264 [info]: slice B:3071 Avg QP:29.86 size: 20635 PSNR Mean Y:36.61 U:49.74 V:49.88 Avg:38.26 Global:38.01
x264 [info]: consecutive B-frames: 10.4% 37.2% 33.9% 18.4%
x264 [info]: mb I I16..4: 27.3% 62.8% 9.9%
x264 [info]: mb P I16..4: 9.8% 24.8% 4.6% P16..4: 27.2% 11.8% 6.1% 0.6% 0.1% skip:15.0%
x264 [info]: mb B I16..4: 2.2% 6.0% 1.8% B16..8: 32.2% 3.4% 3.0% direct: 9.3% skip:42.1% L0:36.5% L1:39.4% BI:24.0%
x264 [info]: 8x8 transform intra:62.5% inter:66.9%
x264 [info]: direct mvs spatial:99.6% temporal:0.4%
x264 [info]: coded y,uvDC,uvAC intra:66.6% 15.2% 0.3% inter:33.7% 1.7% 0.0%
x264 [info]: ref P L0 76.7% 9.8% 5.5% 2.7% 2.0% 1.4% 1.1% 0.8%
x264 [info]: ref B L0 86.1% 6.2% 3.1% 1.7% 1.2% 0.9% 0.7%
x264 [info]: SSIM Mean Y:0.9542913
x264 [info]: PSNR Mean Y:37.169 U:49.757 V:49.887 Avg:38.804 Global:38.500 kb/s:
5994.88

encoded 5726 frames, 5.04 fps, 5995.05 kb/s

avs2yuv -raw e:\test.avs - | x264_mbtree_x64 --preset slower --tune ssim --pass 1 -B 6000
--level 4.1 -m 10 --keyint 240 --min-keyint 24 --ssim --psnr --fps 24000/1001 -o e:\mbtree_ssim1.mkv - 1280x720

x264 [info]: 1280x720 @ 23.98 fps
x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 Cache64
x264 [info]: profile Main, level 4.1
e:\test.avs: 1280x720, 24000/1001 fps, 5726 frames

x264 [info]: slice I:147 Avg QP:24.65 size: 76321 PSNR Mean Y:39.08 U:50.49 V:50.66 Avg:40.68 Global:40.13
x264 [info]: slice P:2541 Avg QP:26.56 size: 42741 PSNR Mean Y:36.76 U:49.58 V:49.68 Avg:38.40 Global:37.69
x264 [info]: slice B:3038 Avg QP:29.29 size: 18494 PSNR Mean Y:35.55 U:49.47 V:49.60 Avg:37.22 Global:36.56
x264 [info]: consecutive B-frames: 10.9% 38.9% 31.3% 18.8%
x264 [info]: mb I I16..4: 42.4% 0.0% 57.6%
x264 [info]: mb P I16..4: 32.7% 0.0% 0.0% P16..4: 51.4% 0.0% 0.0% 0.0% 0.0% skip:15.9%
x264 [info]: mb B I16..4: 8.1% 0.0% 0.0% B16..8: 33.2% 0.0% 0.0% direct:19.3% skip:39.4% L0:28.1% L1:40.6% BI:31.3%
x264 [info]: final ratefactor: 21.04
x264 [info]: direct mvs spatial:99.9% temporal:0.1%
x264 [info]: coded y,uvDC,uvAC intra:65.6% 12.1% 0.4% inter:36.5% 2.0% 0.0%
x264 [info]: SSIM Mean Y:0.9453912
x264 [info]: PSNR Mean Y:36.180 U:49.546 V:49.663 Avg:37.833 Global:37.094 kb/s:
5895.91

encoded 5726 frames, 28.19 fps, 5896.07 kb/s

avs2yuv -raw e:\test.avs - | x264_mbtree_x64 --preset slower --tune ssim --pass 2 -B 6000
--level 4.1 -m 10 --keyint 240 --min-keyint 24 --ssim --psnr --fps 24000/1001 -o e:\mbtree_ssim1.mkv - 1280x720

x264 [info]: 1280x720 @ 23.98 fps
x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 Cache64
x264 [info]: profile High, level 4.1
e:\test.avs: 1280x720, 24000/1001 fps, 5726 frames

x264 [info]: slice I:147 Avg QP:25.02 size: 71456 PSNR Mean Y:39.81 U:50.62 V:50.79 Avg:41.39 Global:40.54
x264 [info]: slice P:2541 Avg QP:27.02 size: 43147 PSNR Mean Y:38.30 U:50.13 V:50.28 Avg:39.91 Global:39.28
x264 [info]: slice B:3038 Avg QP:29.33 size: 19403 PSNR Mean Y:36.72 U:49.97 V:50.12 Avg:38.37 Global:37.80
x264 [info]: consecutive B-frames: 10.9% 38.9% 31.3% 18.8%
x264 [info]: mb I I16..4: 27.2% 62.0% 10.8%
x264 [info]: mb P I16..4: 9.6% 25.9% 5.2% P16..4: 27.3% 11.9% 6.1% 0.5% 0.1% skip:13.3%
x264 [info]: mb B I16..4: 2.9% 6.8% 1.8% B16..8: 32.1% 3.3% 3.1% direct: 9.7% skip:40.3% L0:35.7% L1:38.8% BI:25.6%
x264 [info]: 8x8 transform intra:62.5% inter:64.8%
x264 [info]: direct mvs spatial:99.7% temporal:0.3%
x264 [info]: coded y,uvDC,uvAC intra:69.8% 15.8% 0.3% inter:34.7% 1.5% 0.0%
x264 [info]: ref P L0 77.4% 9.4% 5.3% 2.6% 1.9% 1.4% 1.1% 0.8%
x264 [info]: ref B L0 86.2% 6.2% 3.1% 1.7% 1.2% 0.9% 0.7%
x264 [info]: SSIM Mean Y:0.9564886
x264 [info]: PSNR Mean Y:37.499 U:50.060 V:50.204 Avg:39.131 Global:38.457 kb/s:
5999.04

encoded 5726 frames, 4.99 fps, 5999.21 kb/s

[Dark Shikari]

It's some foreign dialogue hardsubbed in the source (BDRip).

And, it's only for a few seconds of the movie.

Is this patch going to be incompatible with vbv-buf max / b-pyramid etc. or is it only temporarily as they get implemented?B-pyramid should not be too hard to add, but it won't really work ideally until we get MMCO. Which should be soon, Trahald's working on it.

VBV may be a trickier issue; it technically "works" but the row-based prediction completely falls apart with MB-tree despite the support code I wrote for it, causing serious quality loss, especially in 1-pass VBV. It'll take some investigation to figure out exactly what's wrong and whether it could be fixed or not.

[tetsuo55]

This sounds like the best thing since sliced bread!

[juGGaKNot]

Love sliced bread

Have to test this, will edit.

[ocal5]

Seems wonderfull, perfect for stand-up shoots ;-) Thanks !

[Audionut]

Same source as ssim test.

avs2yuv -raw e:\test.avs - | x264_mbtree_x64 --preset slower --tune psnr --pass 1 -B 6000
--level 4.1 -m 10 --mbtree --lookahead 240 --keyint 240 --min-keyint 24 --ssim --psnr
--fps 24000/1001 -o e:\mbtree_psnr.mkv - 1280x720

x264 [info]: 1280x720 @ 23.98 fps
x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 Cache64
x264 [info]: profile Main, level 4.1
e:\test.avs: 1280x720, 24000/1001 fps, 5726 frames

x264 [info]: slice I:147 Avg QP:26.28 size: 74245 PSNR Mean Y:39.48 U:50.13 V:50.23 Avg:41.05 Global:40.11
x264 [info]: slice P:2508 Avg QP:28.16 size: 43247 PSNR Mean Y:37.05 U:49.27 V:49.30 Avg:38.67 Global:38.21
x264 [info]: slice B:3071 Avg QP:29.71 size: 19382 PSNR Mean Y:36.09 U:49.38 V:49.45 Avg:37.74 Global:37.31
x264 [info]: consecutive B-frames: 10.4% 37.2% 33.9% 18.4%
x264 [info]: mb I I16..4: 50.4% 0.0% 49.6%
x264 [info]: mb P I16..4: 29.1% 0.0% 0.0% P16..4: 47.6% 0.0% 0.0% 0.0% 0.0% skip:23.3%
x264 [info]: mb B I16..4: 6.1% 0.0% 0.0% B16..8: 31.0% 0.0% 0.0% direct:18.4% skip:44.6% L0:28.0% L1:38.6% BI:33.5%
x264 [info]: final ratefactor: 24.69
x264 [info]: direct mvs spatial:99.9% temporal:0.1%
x264 [info]: coded y,uvDC,uvAC intra:58.7% 10.4% 0.6% inter:35.0% 1.8% 0.0%
x264 [info]: SSIM Mean Y:0.9409480
x264 [info]: PSNR Mean Y:36.598 U:49.352 V:49.404 Avg:38.235 Global:37.740 kb/s:5992.75

encoded 5726 frames, 20.85 fps, 5992.92 kb/s

avs2yuv -raw e:\test.avs - | x264_mbtree_x64 --preset slower --tune psnr --pass 2 -B 6000
--level 4.1 -m 10 --mbtree --lookahead 240 --keyint 240 --min-keyint 24 --ssim --psnr
--fps 24000/1001 -o e:\mbtree_psnr.mkv - 1280x720

x264 [info]: 1280x720 @ 23.98 fps
x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 Cache64
e:\test.avs: 1280x720, 24000/1001 fps, 5726 frames

x264 [info]: profile High, level 4.1
x264 [info]: slice I:147 Avg QP:26.91 size: 66830 PSNR Mean Y:39.68 U:49.96 V:49.95 Avg:41.22 Global:40.37
x264 [info]: slice P:2508 Avg QP:28.31 size: 42914 PSNR Mean Y:38.39 U:49.49 V:49.43 Avg:39.97 Global:39.63
x264 [info]: slice B:3071 Avg QP:29.79 size: 20087 PSNR Mean Y:37.07 U:49.56 V:49.52 Avg:38.70 Global:38.41
x264 [info]: consecutive B-frames: 10.4% 37.2% 33.9% 18.4%
x264 [info]: mb I I16..4: 32.2% 58.9% 8.8%
x264 [info]: mb P I16..4: 12.1% 21.2% 4.2% P16..4: 24.4% 11.2% 6.0% 0.9% 0.2% skip:19.9%
x264 [info]: mb B I16..4: 2.3% 4.9% 1.7% B16..8: 28.4% 3.4% 3.0% direct: 9.7% skip:46.6% L0:35.2% L1:36.8% BI:28.0%
x264 [info]: 8x8 transform intra:56.5% inter:65.8%
x264 [info]: direct mvs spatial:99.4% temporal:0.6%
x264 [info]: coded y,uvDC,uvAC intra:60.0% 14.8% 0.4% inter:33.2% 1.7% 0.0%
x264 [info]: ref P L0 76.4% 9.9% 5.5% 2.8% 2.0% 1.4% 1.2% 0.8%
x264 [info]: ref B L0 86.3% 6.2% 3.1% 1.6% 1.1% 0.9% 0.7%
x264 [info]: SSIM Mean Y:0.9502261
x264 [info]: PSNR Mean Y:37.714 U:49.538 V:49.494 Avg:39.320 Global:38.947 kb/s:
6000.78

encoded 5726 frames, 5.80 fps, 6000.96 kb/s

avs2yuv -raw e:\test.avs - | x264_mbtree_x64 --preset slower --tune psnr --pass 1 -B 6000
--level 4.1 -m 10 --keyint 240 --min-keyint 24 --ssim --psnr --fps 24000/1001 -o e:\mbtree_psnr1.mkv - 1280x720

x264 [info]: 1280x720 @ 23.98 fps
x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 Cache64
x264 [info]: profile Main, level 4.1
e:\test.avs: 1280x720, 24000/1001 fps, 5726 frames

x264 [info]: slice I:147 Avg QP:26.08 size: 71885 PSNR Mean Y:39.27 U:50.16 V:50.24 Avg:40.85 Global:40.16
x264 [info]: slice P:2541 Avg QP:27.71 size: 42358 PSNR Mean Y:37.24 U:49.28 V:49.32 Avg:38.86 Global:38.07
x264 [info]: slice B:3038 Avg QP:29.44 size: 19722 PSNR Mean Y:36.14 U:49.25 V:49.29 Avg:37.79 Global:37.09
x264 [info]: consecutive B-frames: 10.9% 38.9% 31.3% 18.8%
x264 [info]: mb I I16..4: 47.7% 0.0% 52.3%
x264 [info]: mb P I16..4: 30.1% 0.0% 0.0% P16..4: 48.2% 0.0% 0.0% 0.0% 0.0% skip:21.6%
x264 [info]: mb B I16..4: 6.9% 0.0% 0.0% B16..8: 31.4% 0.0% 0.0% direct:19.1% skip:42.7% L0:27.3% L1:39.1% BI:33.7%
x264 [info]: final ratefactor: 22.74
x264 [info]: direct mvs spatial:99.9% temporal:0.1%
x264 [info]: coded y,uvDC,uvAC intra:62.1% 10.2% 0.5% inter:35.8% 1.8% 0.0%
x264 [info]: SSIM Mean Y:0.9418719
x264 [info]: PSNR Mean Y:36.712 U:49.286 V:49.328 Avg:38.343 Global:37.561 kb/s:
5966.45

encoded 5726 frames, 29.07 fps, 5966.62 kb/s

avs2yuv -raw e:\test.avs - | x264_mbtree_x64 --preset slower --tune psnr --pass 2 -B 6000
--level 4.1 -m 10 --keyint 240 --min-keyint 24 --ssim --psnr --fps 24000/1001 -o e:\mbtree_psnr1.mkv - 1280x720

x264 [info]: 1280x720 @ 23.98 fps
x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 Cache64
e:\test.avs: 1280x720, 24000/1001 fps, 5726 frames
x264 [info]: profile High, level 4.1

x264 [info]: slice I:147 Avg QP:25.33 size: 70603 PSNR Mean Y:40.20 U:50.64 V:50.73 Avg:41.75 Global:40.64
x264 [info]: slice P:2541 Avg QP:27.18 size: 42189 PSNR Mean Y:38.69 U:49.89 V:49.95 Avg:40.27 Global:39.57
x264 [info]: slice B:3038 Avg QP:28.90 size: 20269 PSNR Mean Y:37.24 U:49.87 V:49.92 Avg:38.87 Global:38.24
x264 [info]: consecutive B-frames: 10.9% 38.9% 31.3% 18.8%
x264 [info]: mb I I16..4: 33.0% 58.3% 8.7%
x264 [info]: mb P I16..4: 13.9% 21.5% 4.5% P16..4: 24.6% 11.0% 5.6% 0.6% 0.1% skip:18.2%
x264 [info]: mb B I16..4: 3.1% 5.6% 1.6% B16..8: 28.2% 3.4% 3.0% direct:11.2% skip:43.9% L0:34.4% L1:37.1% BI:28.5%
x264 [info]: 8x8 transform intra:54.3% inter:63.1%
x264 [info]: direct mvs spatial:99.6% temporal:0.4%
x264 [info]: coded y,uvDC,uvAC intra:63.7% 18.8% 0.5% inter:35.3% 2.0% 0.0%
x264 [info]: ref P L0 78.3% 9.1% 5.1% 2.5% 1.9% 1.3% 1.1% 0.7%
x264 [info]: ref B L0 86.4% 6.1% 3.0% 1.6% 1.1% 0.9% 0.8%
x264 [info]: SSIM Mean Y:0.9521406
x264 [info]: PSNR Mean Y:37.958 U:49.897 V:49.954 Avg:39.569 Global:38.837 kb/s:
6001.42

encoded 5726 frames, 6.12 fps, 6001.59 kb/s

[juGGaKNot]

x264_x86_r1195_juGGaKNot (http://www.mediafire.com/download.php?dndzz0z0m4n)
GCC 4.4.0 generic, fprofiled, patched
Source: x264 GIT (git://git.videolan.org/x264.git)

Applied patches :

x264_win_zone_parse_fix_05.diff (http://www.mediafire.com/download.php?kqymujywhtw)
x264_Macroblock_Tree_Ratecontrol_01.diff (http://www.mediafire.com/download.php?zwezgdnhizy)

Please check Doom9.org patches thread (http://forum.doom9.org/showthread.php?t=130364), Doom9.org Macroblock tree Ratecontrol thread (http://forum.doom9.org/showthread.php?t=148686) and GIT shortlog (http://git.videolan.org/gitweb.cgi?p=x264.git;a=shortlog) for more info.

Compiled by juGGaKNot (http://forum.doom9.org/member.php?u=144865) on August 03-2009, 11:20:00 GMT with GCC 4.4.0 on Windows XP SP-2 32-bit.

Platform: X86
System: MINGW
asm: yes
avis input: yes
mp4 output: no
pthread: yes
debug: no
gprof: no
PIC: no
shared: no
visualize: no

Testing wise : ( a custom build not the generic one i uploaded )

--preset juggaknot (http://en.pastebin.ca/1516449)
vs
--preset juggaknot (http://en.pastebin.ca/1516451) --mbtree --lookahead 100.

no mbtree :
Writing library : x264 core 68 r1195 5d75a9b
Encoding settings : cabac=1 / ref=10 / deblock=1:1:1 / analyse=0x3:0x133 / me=tesa / subme=10 / psy_rd=1.0:0.0 / mixed_ref=1 / me_range=32 / chroma_me=1 / trellis=2 / 8x8dct=1 / cqm=0 / deadzone=21,11 / chroma_qp_offset=-2 / threads=3 / nr=0 / decimate=0 / mbaff=0 / bframes=4 / b_pyramid=0 / b_adapt=2 / b_bias=0 / direct=3 / wpredb=1 / keyint=250 / keyint_min=25 / scenecut=40 / lookahead=50 / rc=2pass / mbtree=0 / bitrate=4000 / ratetol=1.0 / qcomp=1.00 / qpmin=10 / qpmax=51 / qpstep=4 / cplxblur=20.0 / qblur=0.5 / vbv_maxrate=20000 / vbv_bufsize=20000 / ip_ratio=1.10 / pb_ratio=1.10 / aq=2:1.00

mbtree :
Writing library : x264 core 68 r1195 5d75a9b
Encoding settings : cabac=1 / ref=10 / deblock=1:1:1 / analyse=0x3:0x133 / me=tesa / subme=10 / psy_rd=1.0:0.0 / mixed_ref=1 / me_range=32 / chroma_me=1 / trellis=2 / 8x8dct=1 / cqm=0 / deadzone=21,11 / chroma_qp_offset=-2 / threads=3 / nr=0 / decimate=0 / mbaff=0 / bframes=4 / b_pyramid=0 / b_adapt=2 / b_bias=0 / direct=3 / wpredb=1 / keyint=250 / keyint_min=25 / scenecut=40 / lookahead=100 / rc=2pass / mbtree=1 / bitrate=4000 / ratetol=1.0 / qpmin=10 / qpmax=51 / qpstep=4 / cplxblur=20.0 / qblur=0.5 / ip_ratio=1.10 / aq=2:1.00

No mbtree :
Encoding X264 1st Pass :

avis : 1184x666 @ 40.00 fps (223 frames)
x264 [warning]: width or height not divisible by 16 (1184x666), compression will
suffer.
x264 [info]: using SAR=1/1
x264 [info]: using cpu capabilities: MMX2 SSE2Slow
x264 [info]: profile High, level 4.0
x264 [info]: slice I:2 Avg QP:24.81 size: 12698
x264 [info]: slice P:103 Avg QP:21.88 size: 19737
x264 [info]: slice B:118 Avg QP:23.60 size: 9047
x264 [info]: consecutive B-frames: 14.0% 31.7% 40.7% 9.0% 4.5%
x264 [info]: mb I I16..4: 58.0% 37.5% 4.4%
x264 [info]: mb P I16..4: 9.7% 21.0% 1.4% P16..4: 43.0% 10.4% 8.1% 0.1% 0
.1% skip: 6.1%
x264 [info]: mb B I16..4: 1.3% 2.5% 0.3% B16..8: 40.9% 1.8% 2.3% direct:
6.7% skip:44.4% L0:49.1% L1:46.6% BI: 4.2%
[I]x264 [info]: final ratefactor: 21.43
x264 [info]: 8x8 transform intra:63.5% inter:82.5%
x264 [info]: direct mvs spatial:98.3% temporal:1.7%
x264 [info]: coded y,uvDC,uvAC intra:46.6% 40.1% 9.2% inter:24.7% 14.8% 2.1%
x264 [info]: ref P L0 63.9% 12.0% 9.0% 3.8% 2.5% 2.3% 2.5% 1.7% 1.2% 1.
1%
x264 [info]: ref B L0 71.9% 10.1% 7.6% 2.3% 2.1% 2.0% 1.4% 1.3% 1.4%
x264 [info]: kb/s:4485.6

encoded 223 frames, 0.26 fps, 4488.52 kb/s

Encoding X264 2nd Pass :

avis [info]: 1184x666 @ 40.00 fps (223 frames)
x264 [warning]: width or height not divisible by 16 (1184x666), compression will
suffer.
x264 [info]: using SAR=1/1
x264 [info]: using cpu capabilities: MMX2 SSE2Slow
x264 [info]: profile High, level 4.0
x264 [info]: slice I:2 Avg QP:23.30 size: 13949
x264 [info]: slice P:103 Avg QP:23.50 size: 16439
x264 [info]: slice B:118 Avg QP:25.47 size: 7667
x264 [info]: consecutive B-frames: 14.0% 31.7% 40.7% 9.0% 4.5%
x264 [info]: mb I I16..4: 53.0% 42.3% 4.7%
x264 [info]: mb P I16..4: 10.7% 19.1% 1.1% P16..4: 44.2% 7.5% 8.3% 0.1% 0
.1% skip: 9.0%
x264 [info]: mb B I16..4: 1.3% 2.3% 0.2% B16..8: 38.3% 1.5% 1.8% direct:
6.0% skip:48.6% L0:49.8% L1:46.5% BI: 3.7%
x264 [info]: 8x8 transform intra:60.7% inter:86.5%
x264 [info]: direct mvs spatial:98.3% temporal:1.7%
x264 [info]: coded y,uvDC,uvAC intra:41.5% 34.6% 6.7% inter:21.3% 12.0% 1.5%
x264 [info]: ref P L0 63.8% 12.1% 9.1% 3.9% 2.5% 2.2% 2.6% 1.8% 1.1% 1.
0%
x264 [info]: ref B L0 70.1% 11.2% 7.7% 2.3% 2.3% 2.1% 1.5% 1.4% 1.3%
x264 [info]: kb/s:3768.0

encoded 223 frames, 0.30 fps, 3770.96 kb/s

Muxing X264 :

AVC-H264 import - frame size 1184 x 666 at 40.000 FPS
Import results: 223 samples - Slices: 2 I 103 P 118 B - 1 SEI - 2 IDR
IsoMedia import - track ID 1 - Audio (SR 44100 - 2 channels)
Saving C:\x264\Movie_2D\Movie_X264_2D.mp4: 0.500 secs Interleaving

Encoding started at 11:20 AM and finished at 11:47 AM

mbtree :
Encoding X264 1st Pass :

avis [info]: 1184x666 @ 40.00 fps (223 frames)
x264 [warning]: width or height not divisible by 16 (1184x666), compression will
suffer.
x264 [info]: using SAR=1/1
x264 [info]: using cpu capabilities: MMX2 SSE2Slow
x264 [info]: profile High, level 4.0
x264 [info]: slice I:2 Avg QP:24.39 size: 13165
x264 [info]: slice P:103 Avg QP:21.71 size: 20739
x264 [info]: slice B:118 Avg QP:24.23 size: 8159
x264 [info]: consecutive B-frames: 14.0% 31.7% 40.7% 9.0% 4.5%
x264 [info]: mb I I16..4: 59.7% 35.2% 5.1%
x264 [info]: mb P I16..4: 10.6% 20.4% 1.3% P16..4: 42.2% 11.6% 8.2% 0.1% 0
.1% skip: 5.5%
x264 [info]: mb B I16..4: 1.3% 2.4% 0.3% B16..8: 39.8% 1.7% 2.0% direct:
6.0% skip:46.4% L0:51.9% L1:43.9% BI: 4.2%
x264 [info]: final ratefactor: 17.72
x264 [info]: 8x8 transform intra:61.3% inter:82.0%
x264 [info]: direct mvs spatial:98.3% temporal:1.7%
x264 [info]: coded y,uvDC,uvAC intra:44.3% 38.5% 9.0% inter:24.0% 14.8% 2.7%
x264 [info]: ref P L0 62.1% 12.8% 9.4% 4.0% 2.5% 2.4% 2.7% 1.9% 1.3% 1.
1%
x264 [info]: ref B L0 71.4% 9.9% 8.3% 2.2% 2.1% 2.0% 1.4% 1.3% 1.4%
x264 [info]: kb/s:4484.7

encoded 223 frames, 0.29 fps, 4487.58 kb/s

Encoding X264 2nd Pass :

avis [info]: 1184x666 @ 40.00 fps (223 frames)
x264 [warning]: width or height not divisible by 16 (1184x666), compression will
suffer.
x264 [info]: using SAR=1/1
x264 [info]: using cpu capabilities: MMX2 SSE2Slow
x264 [info]: profile High, level 4.0
x264 [info]: slice I:2 Avg QP:23.41 size: 14636
x264 [info]: slice P:103 Avg QP:22.56 size: 18846
x264 [info]: slice B:118 Avg QP:25.26 size: 7304
x264 [info]: consecutive B-frames: 14.0% 31.7% 40.7% 9.0% 4.5%
x264 [info]: mb I I16..4: 50.4% 44.5% 5.1%
x264 [info]: mb P I16..4: 11.3% 19.7% 1.1% P16..4: 43.2% 9.4% 8.3% 0.1% 0
.1% skip: 6.9%
x264 [info]: mb B I16..4: 1.3% 2.3% 0.2% B16..8: 37.4% 1.5% 1.8% direct:
5.7% skip:49.7% L0:52.3% L1:43.8% BI: 3.9%
x264 [info]: 8x8 transform intra:60.3% inter:85.3%
x264 [info]: direct mvs spatial:99.2% temporal:0.8%
x264 [info]: coded y,uvDC,uvAC intra:41.6% 35.3% 6.9% inter:22.1% 13.2% 2.3%
x264 [info]: ref P L0 61.9% 13.1% 9.3% 4.1% 2.6% 2.2% 2.7% 1.8% 1.2% 1.
0%
x264 [info]: ref B L0 70.1% 10.5% 8.5% 2.2% 2.3% 2.2% 1.5% 1.4% 1.3%
x264 [info]: kb/s:4064.2

encoded 223 frames, 0.30 fps, 4067.11 kb/s

Muxing X264 :

AVC-H264 import - frame size 1184 x 666 at 40.000 FPS
Import results: 223 samples - Slices: 2 I 103 P 118 B - 1 SEI - 2 IDR
IsoMedia import - track ID 1 - Audio (SR 44100 - 2 channels)
Saving C:\x264\Movie_2D\Movie_X264_2D.mp4: 0.500 secs Interleaving

Encoding started at 11:50 AM and finished at 12:15 PM

I guess the test is invalid, the non mbtree test has vbv and the mbtree one no ....

but

x264 [info]: slice I:2 Avg QP:24.81 size: 12698
x264 [info]: slice P:103 Avg QP:21.88 size: 19737
x264 [info]: slice B:118 Avg QP:23.60 size: 9047
x264 [info]: final ratefactor: 21.43

vs

x264 [info]: slice I:2 Avg QP:24.39 size: 13165
x264 [info]: slice P:103 Avg QP:21.71 size: 20739
x264 [info]: slice B:118 Avg QP:24.23 size: 8159
x264 [info]: final ratefactor: 17.72

Is normal ? final ratefactor smaller by 4 almost ?

Also the mbtree build seems to ignore

param->rc.f_qcompress = 1.0f;
param->rc.f_pb_factor = 1.1f;

Does not show up in the mediainfo window, ipratio does.

[Dark Shikari]

Also the mbtree build seems to ignore

param->rc.f_qcompress = 1.0f;
param->rc.f_pb_factor = 1.1f;

Does not show up in the mediainfo window, ipratio does.Yes, because those parameters don't apply to mb-tree ratecontrol.

[juGGaKNot]

what about the results ?

but

x264 [info]: slice I:2 Avg QP:24.81 size: 12698
x264 [info]: slice P:103 Avg QP:21.88 size: 19737
x264 [info]: slice B:118 Avg QP:23.60 size: 9047
x264 [info]: final ratefactor: 21.43

vs

x264 [info]: slice I:2 Avg QP:24.39 size: 13165
x264 [info]: slice P:103 Avg QP:21.71 size: 20739
x264 [info]: slice B:118 Avg QP:24.23 size: 8159
x264 [info]: final ratefactor: 17.72

Is normal ? final ratefactor smaller by 4 almost ?

[Dark Shikari]

what about the results ?

but

x264 [info]: slice I:2 Avg QP:24.81 size: 12698
x264 [info]: slice P:103 Avg QP:21.88 size: 19737
x264 [info]: slice B:118 Avg QP:23.60 size: 9047
x264 [info]: final ratefactor: 21.43

vs

x264 [info]: slice I:2 Avg QP:24.39 size: 13165
x264 [info]: slice P:103 Avg QP:21.71 size: 20739
x264 [info]: slice B:118 Avg QP:24.23 size: 8159
x264 [info]: final ratefactor: 17.72

Is normal ? final ratefactor smaller by 4 almost ?Yes, read again what I said about CRF in the original post.

If you want to make a meaningful comparison, at a minimum compare SSIM or something like that?

[tetsuo55]

Does any other h264 encoder have a feature like this?
Does this have any influence on the decode difficulty?

EDIT:

And that really low bitrate anime example is simply amazing...

[Dark Shikari]

Does this have any influence on the decode difficulty?Almost surely not.

[Audionut]

Here is a silly question!!

How fast does the motion have to be, to be affected by this patch?

[juGGaKNot]

lookahead 100 only taked 335 mb on my xp
lookahead 200 crashed ( 236 frames source )

---pass 1 --slow-firstpass --preset slow --tune ssim --mbtree --lookahead 100 --ssim --bitrate 2000 :

avis [info]: 1184x666 @ 40.00 fps (223 frames)
x264 [warning]: width or height not divisible by 16 (1184x666), compression will
suffer.
x264 [info]: using cpu capabilities: MMX2 SSE2Slow
x264 [info]: profile High, level 4.0
x264 [info]: slice I:2 Avg QP:27.50 size: 8520
x264 [info]: slice P:101 Avg QP:25.02 size: 10155
x264 [info]: slice B:120 Avg QP:27.42 size: 4161
x264 [info]: consecutive B-frames: 12.2% 29.9% 48.9% 9.0%
x264 [info]: mb I I16..4: 63.8% 32.6% 3.6%
x264 [info]: mb P I16..4: 27.5% 19.0% 1.1% P16..4: 26.8% 3.2% 1.9% 0.0% 0
.0% skip:20.5%
x264 [info]: mb B I16..4: 6.0% 5.2% 0.3% B16..8: 23.9% 0.6% 0.5% direct:
1.9% skip:61.6% L0:47.9% L1:49.5% BI: 2.6%
x264 [info]: final ratefactor: 22.40
x264 [info]: 8x8 transform intra:40.8% inter:85.4%
x264 [info]: direct mvs spatial:98.3% temporal:1.7%
x264 [info]: coded y,uvDC,uvAC intra:25.6% 26.8% 2.6% inter:12.1% 5.1% 0.2%
x264 [info]: ref P L0 64.9% 12.5% 11.1% 4.9% 6.5%
x264 [info]: ref B L0 74.5% 11.0% 9.6% 4.9%
x264 [info]: SSIM Mean Y:0.9773563
x264 [info]: kb/s:2212.8

encoded 223 frames, 3.34 fps, 2213.70 kb/s

Encoding X264 2nd Pass :

avis [info]: 1184x666 @ 40.00 fps (223 frames)
x264 [warning]: width or height not divisible by 16 (1184x666), compression will
suffer.
x264 [info]: using cpu capabilities: MMX2 SSE2Slow
x264 [info]: profile High, level 4.0
x264 [info]: slice I:2 Avg QP:25.46 size: 9403
x264 [info]: slice P:101 Avg QP:25.84 size: 9277
x264 [info]: slice B:120 Avg QP:28.17 size: 3858
x264 [info]: consecutive B-frames: 12.2% 29.9% 48.9% 9.0%
x264 [info]: mb I I16..4: 73.3% 22.4% 4.2%
x264 [info]: mb P I16..4: 29.6% 16.7% 0.9% P16..4: 25.5% 2.9% 2.0% 0.0% 0
.0% skip:22.4%
x264 [info]: mb B I16..4: 6.7% 5.2% 0.2% B16..8: 22.6% 0.5% 0.4% direct:
1.8% skip:62.4% L0:49.0% L1:48.5% BI: 2.4%
x264 [info]: 8x8 transform intra:36.7% inter:85.5%
x264 [info]: direct mvs spatial:100.0% temporal:0.0%
x264 [info]: coded y,uvDC,uvAC intra:22.5% 25.2% 2.1% inter:11.2% 4.4% 0.2%
x264 [info]: ref P L0 64.7% 13.3% 10.7% 5.0% 6.3%
x264 [info]: ref B L0 74.5% 11.1% 8.9% 5.4%
x264 [info]: SSIM Mean Y:0.9766552
x264 [info]: kb/s:2035.7


--pass 1 --slow-firstpass --preset slow --tune ssim --lookahead 100 --ssim --bitrate 2000 :

avis [info]: 1184x666 @ 40.00 fps (223 frames)
x264 [warning]: width or height not divisible by 16 (1184x666), compression will
suffer.
x264 [info]: using cpu capabilities: MMX2 SSE2Slow
x264 [info]: profile High, level 4.0
x264 [info]: slice I:2 Avg QP:26.19 size: 8551
x264 [info]: slice P:104 Avg QP:24.79 size: 10073
x264 [info]: slice B:117 Avg QP:27.73 size: 3856
x264 [info]: consecutive B-frames: 13.6% 36.2% 33.9% 16.3%
x264 [info]: mb I I16..4: 62.4% 33.8% 3.7%
x264 [info]: mb P I16..4: 27.1% 20.2% 1.2% P16..4: 26.7% 3.0% 1.8% 0.0% 0
.0% skip:20.0%
x264 [info]: mb B I16..4: 5.7% 4.8% 0.2% B16..8: 24.6% 0.6% 0.5% direct:
1.6% skip:62.0% L0:45.7% L1:52.2% BI: 2.1%
x264 [info]: final ratefactor: 24.91
x264 [info]: 8x8 transform intra:42.1% inter:85.8%
x264 [info]: direct mvs spatial:98.3% temporal:1.7%
x264 [info]: coded y,uvDC,uvAC intra:26.8% 27.5% 2.7% inter:11.9% 4.7% 0.2%
x264 [info]: ref P L0 68.0% 11.7% 10.1% 4.7% 5.6%
x264 [info]: ref B L0 76.7% 10.4% 8.8% 4.1%
x264 [info]: SSIM Mean Y:0.9747903
x264 [info]: kb/s:2175.2

encoded 223 frames, 3.78 fps, 2176.15 kb/s

Encoding X264 2nd Pass :

avis [info]: 1184x666 @ 40.00 fps (223 frames)
x264 [warning]: width or height not divisible by 16 (1184x666), compression will
suffer.
x264 [info]: using cpu capabilities: MMX2 SSE2Slow
x264 [info]: profile High, level 4.0
x264 [info]: slice I:2 Avg QP:23.63 size: 10095
x264 [info]: slice P:104 Avg QP:25.44 size: 9294
x264 [info]: slice B:117 Avg QP:28.00 size: 3643
x264 [info]: consecutive B-frames: 13.6% 36.2% 33.9% 16.3%
x264 [info]: mb I I16..4: 66.0% 29.6% 4.4%
x264 [info]: mb P I16..4: 29.0% 18.0% 1.0% P16..4: 25.6% 2.8% 1.9% 0.0% 0
.0% skip:21.7%
x264 [info]: mb B I16..4: 6.1% 4.7% 0.2% B16..8: 23.5% 0.5% 0.4% direct:
1.6% skip:63.0% L0:47.0% L1:50.9% BI: 2.1%
x264 [info]: 8x8 transform intra:38.3% inter:85.9%
x264 [info]: direct mvs spatial:99.1% temporal:0.9%
x264 [info]: coded y,uvDC,uvAC intra:23.9% 26.2% 2.3% inter:11.2% 4.2% 0.2%
x264 [info]: ref P L0 66.6% 12.5% 10.2% 4.8% 5.9%
x264 [info]: ref B L0 74.3% 11.1% 9.9% 4.7%
x264 [info]: SSIM Mean Y:0.9751063
x264 [info]: kb/s:2027.6

encoded 223 frames, 4.72 fps, 2028.59 kb/s

[tetsuo55]

Here is a silly question!!

How fast does the motion have to be, to be affected by this patch?i was wonder the same, image a scene like the killa sample (the one from BBC where the flock of birds fly over the moving and reflective sea), would this patch "kill" it or help it.

[Audionut]

lookahead 200 crashed ( 236 frames source )

http://forum.doom9.org/showthread.php?p=1311000#post1311000

All crashes that I've seen so far that occurred with very large --lookahead values traced back perfectly to x264_malloc, so you should check with gdb before reporting such issues as bugs.

Couldn't get 64-bit x264+avs2yuv working with gdb but nevertheless, seems you were right:

[Firebird]

First test. Source is anime (Suzumiya Haruhi-chan Op)
(Tested on AMD athlon XP 1700, lol)

"D:\Program Files\megui\tools\x264\x264.exe" --profile high --crf 18.0 --keyint 240 --min-keyint 24 --ref 8 --no-fast-pskip --bframes 5 --b-adapt 2 --no-weightb --direct auto --subme 9 --trellis 2 --psy-rd 0.3:0 --partitions p8x8,b8x8,i4x4,i8x8 --me umh --thread-input --aq-strength 0.5 --psnr --ssim --output "E:\haruhi\haruhi op.mkv" "E:\haruhi\haruhi op.avs"

avis [info]: 720x480 @ 23.98 fps (1476 frames)
x264 [info]: using cpu capabilities: MMX2
x264 [info]: profile High, level 3.1

x264 [info]: slice I:34 Avg QP:17.37 size: 25881 PSNR Mean Y:51.48 U:53.35 V:52.78 Avg:51.45 Global:48.29
x264 [info]: slice P:667 Avg QP:20.04 size: 11522 PSNR Mean Y:47.32 U:49.74 V:49.10 Avg:47.77 Global:45.49
x264 [info]: slice B:775 Avg QP:23.48 size: 5286 PSNR Mean Y:45.22 U:47.57 V:46.88 Avg:45.76 Global:43.84
x264 [info]: consecutive B-frames: 12.8% 43.0% 20.6% 10.3% 10.1% 3.3%
x264 [info]: mb I I16..4: 41.1% 29.0% 29.9%
x264 [info]: mb P I16..4: 6.9% 8.1% 5.2% P16..4: 30.2% 10.2% 6.9% 0.0% 0.0% skip:32.4%
x264 [info]: mb B I16..4: 1.3% 1.6% 0.7% B16..8: 27.2% 4.3% 4.5% direct: 4.5% skip:55.9% L0:41.5% L1:47.4% BI:11.1%
x264 [info]: 8x8 transform intra:38.5% inter:47.9%
x264 [info]: direct mvs spatial:99.4% temporal:0.6%
x264 [info]: coded y,uvDC,uvAC intra:47.5% 68.4% 47.8% inter:16.9% 22.8% 9.0%
x264 [info]: ref P L0 59.8% 13.8% 8.2% 4.9% 4.4% 3.6% 2.4% 2.9%
x264 [info]: ref B L0 68.3% 12.8% 7.1% 4.9% 2.9% 2.6% 1.5%
x264 [info]: SSIM Mean Y:0.9933523
x264 [info]: PSNR Mean Y:46.313 U:48.684 V:48.016 Avg:46.798 Global:44.586 kb/s:1645.44
encoded 1476 frames, 1.69 fps, 1645.67 kb/s

With mbtree:

avis [info]: 720x480 @ 23.98 fps (1476 frames)
x264 [info]: using cpu capabilities: MMX2
x264 [info]: profile High, level 3.1

x264 [info]: slice I:34 Avg QP:16.10 size: 31193 PSNR Mean Y:52.56 U:54.01 V:53.48 Avg:52.51 Global:50.16
x264 [info]: slice P:660 Avg QP:20.11 size: 12298 PSNR Mean Y:47.53 U:49.85 V:49.34 Avg:48.00 Global:46.47
x264 [info]: slice B:782 Avg QP:24.08 size: 6384 PSNR Mean Y:45.86 U:48.21 V:47.57 Avg:46.40 Global:45.32
x264 [info]: consecutive B-frames: 12.0% 43.6% 20.2% 10.8% 9.7% 3.7%
x264 [info]: mb I I16..4: 41.3% 28.6% 30.2%
x264 [info]: mb P I16..4: 6.4% 8.0% 5.5% P16..4: 28.7% 10.0% 6.8% 0.0% 0.0% skip:34.7%
x264 [info]: mb B I16..4: 1.3% 1.7% 0.9% B16..8: 25.4% 4.7% 4.7% direct: 4.3% skip:57.0% L0:43.6% L1:44.5% BI:11.9%
x264 [info]: 8x8 transform intra:38.6% inter:46.9%
x264 [info]: direct mvs spatial:98.1% temporal:1.9%
x264 [info]: coded y,uvDC,uvAC intra:48.5% 68.8% 48.7% inter:16.9% 22.0% 9.6%
x264 [info]: ref P L0 56.5% 15.0% 9.0% 5.3% 4.8% 4.1% 2.1% 3.3%
x264 [info]: ref B L0 68.1% 13.1% 7.1% 4.6% 3.5% 2.3% 1.3%
x264 [info]: SSIM Mean Y:0.9944401
x264 [info]: PSNR Mean Y:46.763 U:49.077 V:48.494 Avg:47.259 Global:45.874 kb/s:1841.39
encoded 1476 frames, 1.55 fps, 1841.62 kb/s

[DarkZell666]

@Firebird : don't use CRF for doing such tests, go for two-pass (either bitrate or filesize, it doesn't matter), otherwise the results are meaningless x_x". Especially since Dark Shikari mentionned CRF was redefined. This means for any given CRF value, the expected bitrate will change for the same source. You won't know if the SSIM increase is due to the CRF redifinition or to the MB Tree Ratecontrol algo, and you won't know if the +0.0011 SSIM increase was worth the 204kbps increase.

[Firebird]

Sorry, my fault.

The same source, with very low bitrate.

"D:\Program Files\megui\tools\x264\x264.exe" --profile high --pass 2 --bitrate 272 --stats "E:\haruhi\haruhi op.stats" --keyint 240 --min-keyint 24 --ref 8 --no-fast-pskip --bframes 5 --b-adapt 2 --no-weightb --direct auto --subme 9 --trellis 2 --psy-rd 0.0:0 --partitions p8x8,b8x8,i4x4,i8x8 --me umh --thread-input --aq-strength 0.5 --psnr --ssim --output "E:\haruhi\haruhi op.mkv" "E:\haruhi\haruhi op cl 264.avs"

First pass:

avis [info]: 720x480 @ 23.98 fps (1476 frames)
x264 [info]: using cpu capabilities: MMX2
x264 [info]: profile Main, level 3.0

x264 [info]: slice I:34 Avg QP:39.30 size: 4834 PSNR Mean Y:36.49 U:43.22 V:42.39 Avg:37.44 Global:32.02
x264 [info]: slice P:680 Avg QP:42.17 size: 1988 PSNR Mean Y:30.98 U:38.60 V:37.64 Avg:32.21 Global:29.17
x264 [info]: slice B:762 Avg QP:44.72 size: 726 PSNR Mean Y:29.43 U:37.02 V:35.92 Avg:30.67 Global:28.19
x264 [info]: consecutive B-frames: 13.9% 43.1% 19.6% 12.5% 7.6% 3.3%
x264 [info]: mb I I16..4: 84.9% 0.0% 15.1%
x264 [info]: mb P I16..4: 24.6% 0.0% 0.0% P16..4: 15.8% 0.0% 0.0% 0.0% 0.0% skip:59.6%
x264 [info]: mb B I16..4: 3.8% 0.0% 0.0% B16..8: 13.1% 0.0% 0.0% direct: 4.3% skip:78.9% L0:41.4% L1:56.8% BI: 1.7%
x264 [info]: final ratefactor: 39.49
x264 [info]: direct mvs spatial:96.6% temporal:3.4%
x264 [info]: coded y,uvDC,uvAC intra:12.0% 40.7% 17.8% inter:1.5% 7.3% 1.3%
x264 [info]: SSIM Mean Y:0.8703792
x264 [info]: PSNR Mean Y:30.307 U:37.893 V:36.858 Avg:31.533 Global:28.676 kb/s:268.94
encoded 1476 frames, 7.77 fps, 269.17 kb/s

Second pass:

avis [info]: 720x480 @ 23.98 fps (1476 frames)
x264 [info]: using cpu capabilities: MMX2
x264 [info]: profile High, level 3.1

x264 [info]: slice I:34 Avg QP:35.68 size: 5261 PSNR Mean Y:39.15 U:44.12 V:43.20 Avg:39.67 Global:35.20
x264 [info]: slice P:680 Avg QP:39.40 size: 2009 PSNR Mean Y:34.56 U:40.01 V:38.85 Avg:35.51 Global:33.56
x264 [info]: slice B:762 Avg QP:41.76 size: 726 PSNR Mean Y:33.13 U:38.31 V:36.86 Avg:34.08 Global:32.43
x264 [info]: consecutive B-frames: 13.9% 43.1% 19.6% 12.5% 7.6% 3.3%
x264 [info]: mb I I16..4: 41.5% 47.9% 10.6%
x264 [info]: mb P I16..4: 10.0% 9.1% 1.1% P16..4: 24.7% 3.3% 1.6% 0.0% 0.0% skip:50.2%
x264 [info]: mb B I16..4: 1.2% 0.8% 0.0% B16..8: 19.5% 0.5% 0.3% direct: 0.5% skip:77.1% L0:49.2% L1:49.8% BI: 1.1%
x264 [info]: 8x8 transform intra:45.0% inter:77.3%
x264 [info]: direct mvs spatial:75.7% temporal:24.3%
x264 [info]: coded y,uvDC,uvAC intra:14.6% 32.7% 12.8% inter:2.2% 5.2% 0.7%
x264 [info]: ref P L0 54.4% 16.7% 10.5% 5.6% 5.5% 2.9% 2.1% 2.4%
x264 [info]: ref B L0 66.4% 13.3% 7.4% 3.8% 4.4% 2.9% 1.7%
x264 [info]: SSIM Mean Y:0.9407777
x264 [info]: PSNR Mean Y:33.928 U:39.227 V:37.926 Avg:34.866 Global:32.970 kb/s:272.70
encoded 1476 frames, 3.02 fps, 272.93 kb/s

With mbtree:

First pass:

avis [info]: 720x480 @ 23.98 fps (1476 frames)
x264 [info]: using cpu capabilities: MMX2
x264 [info]: profile Main, level 3.0

x264 [info]: slice I:33 Avg QP:38.49 size: 5475 PSNR Mean Y:36.39 U:43.41 V:42.54 Avg:37.50 Global:33.86
x264 [info]: slice P:670 Avg QP:43.89 size: 1958 PSNR Mean Y:30.81 U:38.71 V:37.73 Avg:32.07 Global:29.60
x264 [info]: slice B:773 Avg QP:46.00 size: 752 PSNR Mean Y:29.75 U:37.36 V:36.18 Avg:31.01 Global:28.85
x264 [info]: consecutive B-frames: 13.7% 40.5% 21.2% 11.9% 9.4% 3.3%
x264 [info]: mb I I16..4: 81.6% 0.0% 18.4%
x264 [info]: mb P I16..4: 23.4% 0.0% 0.0% P16..4: 14.1% 0.0% 0.0% 0.0% 0.0% skip:62.5%
x264 [info]: mb B I16..4: 3.7% 0.0% 0.0% B16..8: 13.2% 0.0% 0.0% direct: 3.9% skip:79.1% L0:43.4% L1:54.6% BI: 2.1%
x264 [info]: final ratefactor: 40.39
x264 [info]: direct mvs spatial:95.2% temporal:4.8%
x264 [info]: coded y,uvDC,uvAC intra:12.3% 41.9% 18.5% inter:1.4% 6.8% 1.2%
x264 [info]: SSIM Mean Y:0.8832674
x264 [info]: PSNR Mean Y:30.382 U:38.107 V:37.027 Avg:31.632 Global:29.246 kb/s:269.44
encoded 1476 frames, 7.06 fps, 269.68 kb/s

Second pass:

avis [info]: 720x480 @ 23.98 fps (1476 frames)
x264 [info]: using cpu capabilities: MMX2
x264 [info]: profile High, level 3.1

x264 [info]: slice I:33 Avg QP:37.28 size: 5388 PSNR Mean Y:37.39 U:43.97 V:43.14 Avg:38.41 Global:35.35
x264 [info]: slice P:670 Avg QP:40.64 size: 2023 PSNR Mean Y:34.47 U:40.15 V:39.04 Avg:35.42 Global:33.84
x264 [info]: slice B:773 Avg QP:43.18 size: 745 PSNR Mean Y:33.24 U:38.47 V:37.01 Avg:34.20 Global:32.76
x264 [info]: consecutive B-frames: 13.7% 40.5% 21.2% 11.9% 9.4% 3.3%
x264 [info]: mb I I16..4: 35.9% 52.8% 11.3%
x264 [info]: mb P I16..4: 9.7% 9.1% 1.3% P16..4: 23.9% 3.2% 1.5% 0.0% 0.0% skip:51.2%
x264 [info]: mb B I16..4: 1.3% 0.9% 0.0% B16..8: 18.9% 0.6% 0.3% direct: 0.5% skip:77.5% L0:50.6% L1:48.4% BI: 1.0%
x264 [info]: 8x8 transform intra:46.3% inter:77.9%
x264 [info]: direct mvs spatial:76.3% temporal:23.7%
x264 [info]: coded y,uvDC,uvAC intra:14.0% 33.0% 13.1% inter:2.2% 5.2% 0.8%
x264 [info]: ref P L0 54.0% 16.4% 10.9% 6.1% 5.2% 2.7% 2.0% 2.6%
x264 [info]: ref B L0 65.6% 13.9% 7.7% 4.0% 4.2% 2.9% 1.8%
x264 [info]: SSIM Mean Y:0.9440372
x264 [info]: PSNR Mean Y:33.892 U:39.353 V:38.070 Avg:34.849 Global:33.264 kb/s:274.15
encoded 1476 frames, 2.96 fps, 274.39 kb/s

[Chengbin]

Seems like a wonderful patch.

Can you explain "redefined" CRF? At the same CRF value, which build give the better quality?

Your links don't work (that directs to mirror05.x264.nl)

[Dark Shikari]

Seems like a wonderful patch.

Can you explain "redefined" CRF? At the same CRF value, which build give the better quality?

Your links don't work (that directs to mirror05.x264.nl)mirror05 is down, blame jarod. There's an alternative download link a few posts down.

"Redefining CRF" means that CRF's measure of quality changes, so some videos may have higher or lower bitrates at the same CRF.

[juGGaKNot]

x264_x86_r1195_juGGaKNot (http://www.mediafire.com/download.php?dndzz0z0m4n)
GCC 4.4.0 generic, fprofiled, patched
Source: x264 GIT (git://git.videolan.org/x264.git)

Applied patches :

x264_win_zone_parse_fix_05.diff (http://www.mediafire.com/download.php?kqymujywhtw)
x264_Macroblock_Tree_Ratecontrol_01.diff (http://www.mediafire.com/download.php?zwezgdnhizy)

Please check Doom9.org patches thread (http://forum.doom9.org/showthread.php?t=130364), Doom9.org Macroblock tree Ratecontrol thread (http://forum.doom9.org/showthread.php?t=148686) and GIT shortlog (http://git.videolan.org/gitweb.cgi?p=x264.git;a=shortlog) for more info.

Compiled by juGGaKNot (http://forum.doom9.org/member.php?u=144865) on August 03-2009, 11:20:00 GMT with GCC 4.4.0 on Windows XP SP-2 32-bit.

Platform: X86
System: MINGW
asm: yes
avis input: yes
mp4 output: no
pthread: yes
debug: no
gprof: no
PIC: no
shared: no
visualize: no

Your links don't work (that directs to mirror05.x264.nl)

You can try that ...

[popper]

just a random thought,when i looked at your tree picture in the OP i got to wondering if you have a way to visualise and track pattens in the algorithm(s) your using,
and that in turn reminded me of this old news post
http://www.physorg.com/news138893926.html
" Model helps computers sort data more like humans
August 25th, 2008
http://www.physorg.com/newman/gfx/news/modelhelpsco.jpg
MIT associate professor Josh Tenenbaum and his former student, Charles Kemp, have developed a computer algorithm that can select the best type of structure to fit a set of data. Such structures, shown here, include linear order, rings and clusters. Image courtesy / Charles Kemp

....
The model considers a range of possible data structures, such as trees, linear orders, rings, dominance hierarchies, clusters, etc. It finds the best-fitting structure of each type for a given data set and then picks the type of structure that best represents the data.

..."


i just wonder if anyone knows any more about the Tenenbaum/Kemp code algorithm since last year, and if any of its usable here in x264 POC code!

its good to see some experimentation in the algorithms used here and i wonder what drives the devs to try one algorithm against another, and if they measure and trial a given set of POC code just to see if its faster,better,more accurate than another set etc, and if so has there been any push/atemp to see visual pattens as per the above modelhelpsco.jpg appearing in any of these datasets/tests that may lead to even better ways to improve and add new code to the main codebase later...

[Dark Shikari]

Updated version in topic to 0.11. VBV now supported.

[G_M_C]

Updated version in topic to 0.11. VBV now supported.

That's faster than i hoped for. I make clips (AVCHD) for my STD, and i wanted to wait for VBV to added before i started testing. Lets see if it works on my STD :)

[juGGaKNot]

x264_x86_r1195_juGGaKNot (http://www.mediafire.com/download.php?mfy1vjjqaym)
GCC 4.4.0 generic, fprofiled, patched
Source: GIT

Applied patches :

x264_win_zone_parse_fix_05.diff (http://www.mediafire.com/download.php?kqymujywhtw)
x264_hrd_pulldown.15_interlace.diff (http://www.mediafire.com/download.php?ejhnmitmjt3)
x264_Macroblock_Tree_Ratecontrol_011.diff (http://www.mediafire.com/download.php?nlmj3d4t3yd)

Please check Doom9.org patches thread (http://forum.doom9.org/showthread.php?t=130364), Doom9.org Macroblock tree Ratecontrol thread (http://forum.doom9.org/showthread.php?t=148686) and GIT shortlog (http://git.videolan.org/gitweb.cgi?p=x264.git;a=shortlog) for more info.

Compiled by juGGaKNot (http://forum.doom9.org/member.php?u=144865) on August 04-2009, 12:20:00 GMT with GCC 4.4.0 on Windows XP SP-2 32-bit.

Platform: X86
System: MINGW
asm: yes
avis input: yes
mp4 output: yes
pthread: yes
debug: no
gprof: no
PIC: no
shared: no
visualize: no

[imk]

http://imk.cx/junk/x264_mbtree.7z
Updated with v0.11 of patch.

[G_M_C]

http://imk.cx/junk/x264_mbtree.7z
Updated with v0.11 of patch.

Wich patches did u use ? HRD by any chance ?

[IgorC]

I remember there was a fixed version of SSIM avisynth plugin 0.24a. (with lumimask true)

dunno about behavior x264's internal SSIM.

[imk]

Wich patches did u use ? HRD by any chance ?

No. Since this is for testing mbtree, the only patches I used were my ICC patch and the mbtree patch.

[Chengbin]

This patch significantly changed CRF (unless compressibility has significantly gone up).

I encoded both videos with CRF 18, same settings.

With mbtree:1366Kbps
Without mbtree:878Kbps

This was done with anime.

So the new CRF 18 should be 15-16?

[Dark Shikari]

This patch significantly changed CRF (unless compressibility has significantly gone up).

I encoded both videos with CRF 18, same settings.

With mbtree:1366Kbps
Without mbtree:878Kbps

This was done with anime.

So the new CRF 18 should be 15-16?Again, as I've said at least a dozen times now, the patch redefines CRF, so different sources will get higher or lower bitrates!

Your source is not every source, and I am not going to customize x264 to match your source at the expense of everything else.

[G_M_C]

Wich patches did u use ? HRD by any chance ?

x264_x86_r1195_juGGaKNot (http://www.mediafire.com/download.php?mfy1vjjqaym)
GCC 4.4.0 generic, fprofiled, patched
Source: GIT

Applied patches :

x264_win_zone_parse_fix_05.diff (http://www.mediafire.com/download.php?kqymujywhtw)
x264_hrd_pulldown.15_interlace.diff (http://www.mediafire.com/download.php?ejhnmitmjt3)
x264_Macroblock_Tree_Ratecontrol_011.diff (http://www.mediafire.com/download.php?nlmj3d4t3yd)

Please check Doom9.org patches thread (http://forum.doom9.org/showthread.php?t=130364), Doom9.org Macroblock tree Ratecontrol thread (http://forum.doom9.org/showthread.php?t=148686) and GIT shortlog (http://git.videolan.org/gitweb.cgi?p=x264.git;a=shortlog) for more info.

Compiled by juGGaKNot (http://forum.doom9.org/member.php?u=144865) on August 04-2009, 12:20:00 GMT with GCC 4.4.0 on Windows XP SP-2 32-bit.

Platform: X86
System: MINGW
asm: yes
avis input: yes
mp4 output: yes
pthread: yes
debug: no
gprof: no
PIC: no
shared: no
visualize: no

Again, as I've said at least a dozen times now, the patch redefines CRF, so different sources will get higher or lower bitrates!

[...]

Well my first test is allmost done. Made a 20 minutes reencode of a docu i capped from BBCHD. Will put it on BD9, so i've used juGGaKNot build with HRD.

For testing purposes i've enabled 2-pass mode with both --ssim and --psnr. Will paste results tomorrow, after i've done test no2 (the one without --mbtree (which will be the only difference in the commandline/source etc. used).

This test will also test if --mbtree is accepted by STD's like my Panasonic BD30 (and incidentily if the mbtree patch coincides with the other patches juGGaKNot used, like the hrd-patch).

[Chengbin]

Again, as I've said at least a dozen times now, the patch redefines CRF, so different sources will get higher or lower bitrates!

Your source is not every source, and I am not going to customize x264 to match your source at the expense of everything else.

That's not what I meant. What I was trying to say is, if this patch reduces bitrate by 35% with the same CRF value, it should say something like lower CRF value by (say 2) to compensate for this patch, not just vaguely stating it redefines CRF.

[Dark Shikari]

That's not what I meant. What I was trying to say is, if this patch reduces bitrate by 50% with the same CRF value, it should say something like lower CRF value by (say 2) to compensate for this patch, not just vaguely stating it redefines CRF.Please stop responding to my posts without reading them.

[moviefan]

@Chengbin: What DS (as I understood) is saying is that CRF most probably hasn't shifted by a constant value but it has entirely changed and this change cannot easily be described on basis of one encoding sample. You simply need to find a new CRF that meets your demands, but I think you always have to if you don't want to waste bitrate or lack quality in the end.

[Firethe1]

Hi,
its sad that this is my first post, but x264_mbtree keeps crashing with one input file ...
The crash occurs around frame 2-20 ...
Already tried to reduce lookahead to 0.

commandline:
C:\Dokumente und Einstellungen\Fire>"D:\$Test\temp\x264_mbtree11.exe" --lookahea
d 0 --pass 1 --mbtree --ref 2 --bitrate 500 --qcomp 0.95 --bframes 3 --b-adapt 2
--mixed-refs --no-fast-pskip --weightb --direct auto --subme 10 --trellis 2 --p
artitions all --8x8dct --me umh --merange 32 --nr 50 --stats "D:\$Test\temp\test
2.avs.mbtree.stats" --threads 1 --thread-input -o "D:\$Test\temp\test2.avs.mbtre
e.mkv" "D:\$Test\temp\test2.avs"
avis [info]: 1280x720 @ 23.98 fps (1750 frames)
x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 Cache64
x264 [info]: profile Main, level 3.1
[0.2%] 4/1750 frames, 43.01 fps, 0.00 kb/s, eta 0:00:40

even this cmd crashes: (works with latest x264.exe from x264.nl 32 Bit)
C:\Dokumente und Einstellungen\Fire>"D:\$Test\temp\x264_mbtree11.exe" -o D:\$Tes
t\temp\test2.avs.mkv D:\$Test\temp\test2.avs
avis [info]: 1280x720 @ 23.98 fps (1750 frames)
x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 Cache64
x264 [info]: profile High, level 3.1
[0.1%] 2/1750 frames, 32.26 fps, 0.00 kb/s, eta 0:00:54

gdb: (hopefully i did it right ... but i dont think so xD)
D:\$Test\temp>gdb.exe D:\$Test\temp\x264_mbtree11_debug.exe
GNU gdb (GDB) 6.8.50.20090804
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "i386-pc-mingw32".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
This binary was built by Equation Solution <http://www.Equation.com>...
(gdb) run -o a.mkv test2.avs
Starting program: D:\$Test\temp\x264_mbtree11_debug.exe -o a.mkv test2.avs
[New Thread 4472.0x12e4]
warning: Can not parse XML library list; XML support was disabled at compile tim
e
[New Thread 4472.0x1248]
[New Thread 4472.0x1208]
[New Thread 4472.0x264]
[New Thread 4472.0x858]
[New Thread 4472.0xf90]
warning: HEAP[x264_mbtree11_debug.exe]:
warning: Invalid Address specified to RtlFreeHeap( 00240000, 00FAF174 )


Program received signal SIGTRAP, Trace/breakpoint trap.
0x7c91120f in ?? ()
(gdb) bt
#0 0x7c91120f in ?? ()
#1 0x7c97c201 in ?? ()
#2 0x0022ebe8 in ?? ()
#3 0x7c97c63e in ?? ()
#4 0x00faf16c in ?? ()
#5 0x00240000 in ?? ()
#6 0x00faf174 in ?? ()
#7 0x0022ec5c in ?? ()
#8 0x7c97d826 in ?? ()
#9 0x00240000 in ?? ()
#10 0x00faf16c in ?? ()
#11 0x7c97d9dc in ?? ()
#12 0x00240000 in ?? ()
#13 0x00faf174 in ?? ()
#14 0x40000060 in ?? ()
#15 0x00000040 in ?? ()
#16 0x00f37e48 in ?? ()
#17 0x003e0000 in ?? ()
#18 0x003e0000 in ?? ()
#19 0x007d0037 in ?? ()
#20 0x0101ecec in ?? ()
#21 0x0022ebd8 in ?? ()
#22 0x0022ebdc in ?? ()
#23 0x0022ed10 in ?? ()
#24 0x00014320 in ?? ()
#25 0x00faf16c in ?? ()
#26 0x00240000 in ?? ()
#27 0x7c97d994 in ?? ()
#28 0x7c959e1c in ?? ()
#29 0x00010000 in ?? ()
#30 0x0022ebfc in ?? ()
#31 0x7c937764 in ?? ()
#32 0x0022ed34 in ?? ()
#33 0x7c91e900 in ?? ()
#34 0x7c97d9b8 in ?? ()
#35 0x00000001 in ?? ()
#36 0x0022ed44 in ?? ()
#37 0x7c959e1c in ?? ()
#38 0x00240000 in ?? ()
#39 0x50000061 in ?? ()
#40 0x00faf174 in ?? ()
#41 0x00240000 in ?? ()
#42 0x00faf174 in ?? ()
#43 0x40000060 in ?? ()
#44 0x40000060 in ?? ()
#45 0x7c92003d in ?? ()
#46 0x00efa744 in ?? ()
#47 0x00efa740 in ?? ()
#48 0x00240000 in ?? ()
#49 0x0022ed60 in ?? ()
#50 0x7c91e900 in ?? ()
#51 0x7c937768 in ?? ()
#52 0xffffffff in ?? ()
#53 0x7c937764 in ?? ()
#54 0x7c007553 in ?? ()
#55 0x003e0000 in ?? ()
#56 0x0022ebf0 in ?? ()
#57 0x7c92003d in ?? ()
#58 0x0022ed84 in ?? ()
#59 0x7c91e900 in ?? ()
#60 0x7c937768 in ?? ()
#61 0xffffffff in ?? ()
#62 0x7c937764 in ?? ()
#63 0x7c937553 in ?? ()
#64 0x00240000 in ?? ()
#65 0x40000060 in ?? ()
#66 0x7c92003d in ?? ()
#67 0x00e98f4c in ?? ()
#68 0x0022edd8 in ?? ()
#69 0x00e95038 in ?? ()
#70 0x0022ec40 in ?? ()
#71 0x003e0000 in ?? ()
#72 0x003e0000 in ?? ()
#73 0x7c91e900 in ?? ()
#74 0x7c920040 in ?? ()
#75 0xffffffff in ?? ()
#76 0x7c92003d in ?? ()
#77 0x7c001432 in ?? ()
#78 0x7c001463 in ?? ()
#79 0x0022ec4c in ?? ()
#80 0x00240000 in ?? ()
#81 0x0022ede0 in ?? ()
#82 0x7c91e900 in ?? ()
#83 0x7c937768 in ?? ()
#84 0xffffffff in ?? ()
#85 0x7c937764 in ?? ()
#86 0x7c007553 in ?? ()
#87 0x003e0000 in ?? ()
#88 0x0022ec70 in ?? ()
#89 0x7c92003d in ?? ()
#90 0x0022ee04 in ?? ()
#91 0x7c91e900 in ?? ()
#92 0x7c937768 in ?? ()
#93 0xffffffff in ?? ()
#94 0x0022ee14 in ?? ()
#95 0x7c937553 in ?? ()
#96 0x00240000 in ?? ()
#97 0x40000060 in ?? ()
#98 0x00faf174 in ?? ()
#99 0x0022eea8 in ?? ()
#100 0x00f37e90 in ?? ()
#101 0x00f08ca8 in ?? ()
#102 0x0022ed44 in ?? ()
#103 0x0022f218 in ?? ()
#104 0x0022f218 in ?? ()
#105 0x0022ed54 in ?? ()
#106 0x000113d8 in ?? ()
#107 0x0022f218 in ?? ()
#108 0x0022ecd8 in ?? ()
#109 0x77be2070 in ?? ()
#110 0x0022f218 in ?? ()
#111 0x7c91e900 in ?? ()
#112 0x7c920040 in ?? ()
#113 0xffffffff in ?? ()
#114 0x7c92003d in ?? ()
#115 0x774cd01c in ?? ()
#116 0x00240000 in ?? ()
#117 0x00000000 in ?? ()

AVS:
DirectShowSource("X:\test.mp4",audio=false)
LoadPlugin("D:\***\UnDot.dll")
UnDot()
ConvertToYV12()
SelectRangeEvery(5000,250)

AVS plays fine in mpc ... and this is the first file which crashes ...

If you need more information/i should try something then tell me ^^

[Dark Shikari]

Don't use Avisynth input during debugging; use raw YUV input only.

[Firethe1]

thx for the quick response ... could you also tell me how i get my avs input to a raw yuv file? thx ;)

found avs2yuv.exe .. tried like this: avs2yuv.exe test2.avs -raw -o test2.yuv
put also crashes ... maybe its avisynth ...
AppName: avs2yuv.exe AppVer: 0.0.0.0 ModName: avisynth.dll
ModVer: 2.5.7.0 Offset: 000168f0

Updated to Avisynth 2.5.8 with directshowsource plugin patch and it works ... sry ;)

[Dark Shikari]

thx for the quick response ... could you also tell me how i get my avs input to a raw yuv file? thx ;)

found avs2yuv.exe .. tried like this: avs2yuv.exe test2.avs -raw -o test2.yuv
put also crashes ... maybe its avisynth ...It probably is Avisynth ;)

[Yoshiyuki Blade]

Tests on some anime I have are looking good so far!

Even at lower bitrates, an aq and psy-rd value of 1.0 doesn't create as much horrible artifacts as before.

Testing with --tune animation now.

[Chengbin]

Please stop responding to my posts without reading them.

@Chengbin: What DS (as I understood) is saying is that CRF most probably hasn't shifted by a constant value but it has entirely changed and this change cannot easily be described on basis of one encoding sample. You simply need to find a new CRF that meets your demands, but I think you always have to if you don't want to waste bitrate or lack quality in the end.

I understood him well.

If one encoding says with the same CRF value an encoding dropped 35% in bitrate than the unpatched, that means the same CRF value gives lower bitrate encodes. Of course with different sources the percentage will differ, but I seriously doubt it will vary much, let alone 35%.

How about as a suggestion, when CRF changes, can you tell us what is the new "CRF 18"?

[Sagekilla]

Why don't you do some testing? Go pick out a bunch of clips, test them with and without mbtree.

Find what the average change in crf is, then you'll know. I know you think that it might not vary much, but the only way you can really prove that is by testing more than one source.

Otherwise, I could pull out a source and say "Hey, I think this changes the crf upwards by 5" which could be completely wrong.

I'd do testing myself, but I don't have the time for it at the moment (Going away on vacation soon). If anyone is willing to do the testing, you or otherwise, it would be useful for everyone.

[nm]

If one encoding says with the same CRF value an encoding dropped 35% in bitrate than the unpatched, that means the same CRF value gives lower bitrate encodes. Of course with different sources the percentage will differ, but I seriously doubt it will vary much, let alone 35%.
From the first post (emphasis mine):
Also, note that CRF is redefined again by this patch--I set it so that it kept the same bitrate on Blackpearl as before, but it may vary for different videos.

[Chengbin]

Why don't you do some testing? Go pick out a bunch of clips, test them with and without mbtree.

Find what the average change in crf is, then you'll know. I know you think that it might not vary much, but the only way you can really prove that is by testing more than one source.

Otherwise, I could pull out a source and say "Hey, I think this changes the crf upwards by 5" which could be completely wrong.

I'd do testing myself, but I don't have the time for it at the moment (Going away on vacation soon). If anyone is willing to do the testing, you or otherwise, it would be useful for everyone.

CRF has been changed twice in I think 2 months (AutoVAQ (from the time it officially been adopted in x264, I didn't take it seriously before), and mbtree).

If I did testing (which I did for AutoVAQ with all kinds of sources), by the time I finished with AutoVAQ testing (which is a few weeks ago), my data is obsolete because of mbtree. Who knows when another change could happen. Maybe an update to mbtree will change CRF again. I think you can understand why I'm asking.

Just to keep the thread on topic, are there any compression improvements with this patch? Or is it visuals only?

[wyti]

plz learn to read...
if metrics increase it means that's a compression improvement.
metrics almost always go down when it's visually (aq PSNR goes down, psyrd / trellis both go down...)

And the change at the crf value may happend until the commit of this patch, this is a TEST patch, and everything that is true today may be wrong tomorow. It's not for real encoding, but to see what it does, where it fails, where it does great and improve this path before he got commited

[Sagekilla]

Yes, mbtree should improve quality, visually and metrically speaking.

It might not be as easy to judge in crf mode given the wide fluctuations we've seen, but it should be optimal when it comes out of testing.

[Dark Shikari]

Original post updated with v0.13.

[Zachs]

@DS

In your OP, you said that mb-tree is on by default. And --preset fast and faster also turns it on. That means, other presets have it turned off?

[Dark Shikari]

@DS

In your OP, you said that mb-tree is on by default. And --preset fast and faster also turns it on. That means, other presets have it turned off?Yes, "faster", "veryfast", and "ultrafast" have it off.

Fast and above (medium, slow, etc) have it on.

[Zachs]

@DS

Thanks.

BTW, is mb-tree with b-pyramid any closer for testing?
How much does b-pyramid add to encodes anyway?

[Dark Shikari]

@DS

Thanks.

BTW, is mb-tree with b-pyramid any closer for testing?
How much does b-pyramid add to encodes anyway?I probably will commit without pyramid support, then add pyramid support when Trahald finishes MMCO-related stuff.

Currently I don't think MB-tree with pyramid is a good idea as long as we allow referenced B-frames to end up in the reference list.

[G_M_C]

Well my first test is allmost done. Made a 20 minutes reencode of a docu i capped from BBCHD. Will put it on BD9, so i've used juGGaKNot build with HRD.

For testing purposes i've enabled 2-pass mode with both --ssim and --psnr. Will paste results tomorrow, after i've done test no2 (the one without --mbtree (which will be the only difference in the commandline/source etc. used).

This test will also test if --mbtree is accepted by STD's like my Panasonic BD30 (and incidentily if the mbtree patch coincides with the other patches juGGaKNot used, like the hrd-patch).

As promised. I give the log of pass 2 of both encodes. Used same commandline on both encodes, only removed/added --mbtree.

Commandline used:
<<x264>> --threads auto --thread-input --psnr --ssim "%IN_TITLE%.avs" --stats "%IN_TITLE%.stats" --output "%OUT_TITLE%.264" --pass 2 --bitrate 4450 --vbv-bufsize 30000 --vbv-maxrate 24000 --level 4.1 --deblock -1:-1 --keyint 25 --min-keyint 1 --nal-hrd --fps 25 --aud --sar 1:1 --bframes 3 --b-adapt 2 --ref 6 --no-fast-pskip --ipratio 1.3 --pbratio 1.2 --direct auto --subme 10 --trellis 2 --psy-rd 1.0:0.2 --partitions all --me umh --merange 24 --mvrange 511 --aq-strength 1.0 --aq-mode 1 --mbtree

Encode without --mbtree
avis [info]: 1280x720 @ 25.00 fps (35001 frames)
x264 [info]: using SAR=1/1
x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 FastShuffle SSE4.1 Cach
e64
x264 [info]: profile High, level 4.1
x264 [info]: slice I:2212 Avg QP:19.00 size: 73712 PSNR Mean Y:46.71 U:50.33
V:50.92 Avg:47.62 Global:46.51
x264 [info]: slice P:12247 Avg QP:21.08 size: 32098 PSNR Mean Y:43.76 U:47.87
V:48.92 Avg:44.78 Global:43.59
x264 [info]: slice B:20542 Avg QP:22.86 size: 10825 PSNR Mean Y:43.02 U:47.50
V:48.58 Avg:44.09 Global:43.19
x264 [info]: consecutive B-frames: 4.1% 18.0% 57.1% 20.8%
x264 [info]: mb I I16..4: 8.2% 84.8% 7.0%
x264 [info]: mb P I16..4: 1.8% 14.8% 0.6% P16..4: 44.0% 20.5% 13.5% 0.2% 0
.2% skip: 4.5%
x264 [info]: mb B I16..4: 0.5% 1.9% 0.1% B16..8: 42.1% 1.6% 2.4% direct:
9.2% skip:42.3% L0:35.6% L1:48.1% BI:16.3%
x264 [info]: 8x8 transform intra:84.6% inter:74.8%
x264 [info]: direct mvs spatial:76.3% temporal:23.7%
x264 [info]: coded y,uvDC,uvAC intra:80.9% 80.7% 46.4% inter:30.4% 32.1% 3.9%
x264 [info]: ref P L0 78.9% 11.5% 5.4% 2.2% 1.3% 0.7%
x264 [info]: ref B L0 85.2% 8.9% 3.6% 1.5% 0.9%
x264 [info]: SSIM Mean Y:0.9784036
x264 [info]: PSNR Mean Y:43.516 U:47.805 V:48.844 Avg:44.553 Global:43.481 kb/s:
4448.60

encoded 35001 frames, 4.39 fps, 4448.63 kb/s
]

Encode with --mbtree
avis [info]: 1280x720 @ 25.00 fps (35001 frames)
x264 [info]: using SAR=1/1
x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 FastShuffle SSE4.1 Cach
e64
x264 [info]: profile High, level 4.1
x264 [info]: slice I:2211 Avg QP:18.28 size: 89332 PSNR Mean Y:47.28 U:50.78
V:51.30 Avg:48.17 Global:47.36
x264 [info]: slice P:12255 Avg QP:21.25 size: 32438 PSNR Mean Y:43.83 U:47.96
V:48.98 Avg:44.85 Global:43.92
x264 [info]: slice B:20535 Avg QP:24.79 size: 8944 PSNR Mean Y:43.01 U:47.53
V:48.62 Avg:44.07 Global:43.39
x264 [info]: consecutive B-frames: 4.2% 17.4% 59.0% 19.5%
x264 [info]: mb I I16..4: 10.8% 79.8% 9.4%
x264 [info]: mb P I16..4: 1.9% 15.0% 0.6% P16..4: 43.2% 19.4% 13.4% 0.2% 0
.3% skip: 6.0%
x264 [info]: mb B I16..4: 0.4% 1.8% 0.1% B16..8: 40.3% 1.4% 1.8% direct:
6.0% skip:48.2% L0:37.7% L1:47.7% BI:14.6%
x264 [info]: 8x8 transform intra:82.3% inter:74.7%
x264 [info]: direct mvs spatial:72.8% temporal:27.2%
x264 [info]: coded y,uvDC,uvAC intra:80.6% 80.1% 48.2% inter:26.1% 26.6% 3.5%
x264 [info]: ref P L0 78.7% 11.7% 5.5% 2.2% 1.3% 0.7%
x264 [info]: ref B L0 86.5% 8.1% 3.3% 1.3% 0.8%
x264 [info]: SSIM Mean Y:0.9790297
x264 [info]: PSNR Mean Y:43.564 U:47.884 V:48.917 Avg:44.603 Global:43.742 kb/s:
4449.59

encoded 35001 frames, 4.41 fps, 4449.62 kb/s

I saw a rise in both SSIM and PSNR, but not shockingly high percentages. But i did notice a difference of 2 points in the avg QP used for b-frames. At the same time I-frames got a slightly lower QP, and the QP for P stayed about the same.

But other than that, all went as if nothing changed. My STD ran these files as usual (no change noticed), and visual quality was about the same (with a slight plus on the --mbtree side).

Will run another encode with some different parameters while i'm at work (with/without VAQ). I'll post about those when i get back from work.

[tetsuo55]

Currently I don't think MB-tree with pyramid is a good idea as long as we allow referenced B-frames to end up in the reference list.Do you mean the that with that other patch b-frames will be freed from reference frames?

This might raise b-frame and b-pyramid compatibility with standalone devices! (or break them)

[juGGaKNot]

x264_x86_r1195_juGGaKNot (http://www.mediafire.com/download.php?5izgtetmjqz)
GCC 4.4.0, modified, fprofiled, patched
Source: GIT

Applied patches :

x264_win_zone_parse_fix_05.diff (http://www.mediafire.com/download.php?kqymujywhtw)
x264_hrd_pulldown.15_interlace.diff (http://www.mediafire.com/download.php?ejhnmitmjt3)
x264_Macroblock_Tree_Ratecontrol_013.diff (http://www.mediafire.com/download.php?jyhmmrnzotn)
x264_juGGaKNot_preset_01.diff (http://www.mediafire.com/download.php?5zwqzy0ymn2) ( This just adds a preset, juggaknot, download it to see the settings )

It is fprofiled on a uncompressed 300 frames source with very high motion

Please check Doom9.org patches thread (http://forum.doom9.org/showthread.php?t=130364), Doom9.org Macroblock tree Ratecontrol thread (http://forum.doom9.org/showthread.php?t=148686) and GIT shortlog (http://git.videolan.org/gitweb.cgi?p=x264.git;a=shortlog) for more info.

Compiled by juGGaKNot (http://forum.doom9.org/member.php?u=144865) with GCC 4.4.0 on Windows XP SP-2 32-bit.

Platform: X86
System: MINGW
asm: yes
avis input: yes
mp4 output: yes
pthread: yes
debug: no
gprof: no
PIC: no
shared: no
visualize: no

[imk]

CRF has been changed twice in I think 2 months (AutoVAQ (from the time it officially been adopted in x264, I didn't take it seriously before), and mbtree).

This is what it means to use bleeding edge.

Changes happen, they happen often, and they break compatibility.

Using the latest revisions from git will guarantee changes are happening, let alone a brand new patch.

If you want everything to stay consistent for longer periods of time, stick to a specific revision or version.

[juGGaKNot]

x264_x86_r1195_juGGaKNot_CCG_4.4.1 (http://www.mediafire.com/download.php?3wtyqmzzj1m)
GCC 4.4.1, modified, fprofiled, patched
Source: GIT

Applied patches :

x264_win_zone_parse_fix_05.diff (http://www.mediafire.com/download.php?kqymujywhtw)
x264_hrd_pulldown.15_interlace.diff (http://www.mediafire.com/download.php?ejhnmitmjt3)
x264_Macroblock_Tree_Ratecontrol_013.diff (http://www.mediafire.com/download.php?jyhmmrnzotn)
x264_juGGaKNot_preset_01.diff (http://www.mediafire.com/download.php?5zwqzy0ymn2) ( This just adds a preset, juggaknot, download it to see the settings )

It is fprofiled on a uncompressed 300 frames source with very high motion

Please check Doom9.org patches thread (http://forum.doom9.org/showthread.php?t=130364), Doom9.org Macroblock tree Ratecontrol thread (http://forum.doom9.org/showthread.php?t=148686) and GIT shortlog (http://git.videolan.org/gitweb.cgi?p=x264.git;a=shortlog) for more info.

Compiled by juGGaKNot (http://forum.doom9.org/member.php?u=144865) with GCC 4.4.1 on Windows XP SP-2 32-bit.

Platform: X86
System: MINGW
asm: yes
avis input: yes
mp4 output: yes
pthread: yes
debug: no
gprof: no
PIC: no
shared: no
visualize: no

[rack04]

Here are my results using v 0.13 of MB Tree patch.

Pass 1:
--preset fast --tune animation --pass 1 --slow-firstpass --bitrate 1000 --level 4.1 --keyint 24 --min-keyint 1
--vbv-bufsize 30000 --vbv-maxrate 24000 --thread-input --direct auto --b-adapt 2 --nal-hrd --ssim --psnr

SSIM/PNSR:
avis [info]: 1920x1080 @ 24.00 fps (400 frames)
x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 Cache64
x264 [info]: profile High, level 4.1

x264 [info]: slice P:155 Avg QP:33.65 size: 5986 PSNR Mean Y:41.42 U:45.49 V:46.99 Avg:42.49 Global:41.27
x264 [info]: slice B:226 Avg QP:36.38 size: 1917 PSNR Mean Y:41.33 U:45.80 V:47.55 Avg:42.45 Global:41.26
x264 [info]: consecutive B-frames: 7.6% 35.2% 22.8% 24.1% 3.9% 6.3%
x264 [info]: mb I I16..4: 28.8% 68.2% 3.0%
x264 [info]: mb P I16..4: 3.1% 5.0% 0.2% P16..4: 12.2% 0.9% 0.5% 0.0% 0.0% skip:78.2%
x264 [info]: mb B I16..4: 1.1% 0.0% 0.0% B16..8: 4.7% 0.2% 0.1% direct: 0.5% skip:93.4% L0:51.8% L1:44.8% BI: 3.4%
x264 [info]: final ratefactor: 31.95
x264 [info]: 8x8 transform intra:60.4% inter:82.6%
x264 [info]: direct mvs spatial:98.7% temporal:1.3%
x264 [info]: coded y,uvDC,uvAC intra:38.3% 54.2% 18.5% inter:1.7% 1.8% 0.1%
x264 [info]: ref P L0 67.1% 16.0% 9.3% 7.5%
x264 [info]: ref B L0 78.8% 13.1% 8.1%
x264 [info]: SSIM Mean Y:0.9695521
x264 [info]: PSNR Mean Y:41.434 U:45.704 V:47.348 Avg:42.529 Global:41.318 kb/s:1238.14

encoded 400 frames, 2.59 fps, 1238.63 kb/s

Pass 2:
--preset slow --tune animation --pass 2 --bitrate 1000 --level 4.1 --keyint 24 --min-keyint 1
--vbv-bufsize 30000 --vbv-maxrate 24000 --thread-input --trellis 2 --sar 1:1 --aud --nal-hrd --ssim --psnr

SSIM/PNSR:
avis [info]: 1920x1080 @ 24.00 fps (400 frames)
x264 [info]: using SAR=1/1
x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 Cache64
x264 [info]: profile High, level 4.1

x264 [info]: slice P:155 Avg QP:35.74 size: 4759 PSNR Mean Y:40.53 U:44.54 V:46.20 Avg:41.60 Global:40.84
x264 [info]: slice B:226 Avg QP:38.61 size: 1726 PSNR Mean Y:40.80 U:44.88 V:46.79 Avg:41.89 Global:41.09
x264 [info]: consecutive B-frames: 7.6% 35.2% 22.8% 24.1% 3.9% 6.3%
x264 [info]: mb I I16..4: 33.7% 64.6% 1.7%
x264 [info]: mb P I16..4: 3.4% 3.8% 0.1% P16..4: 10.9% 0.8% 1.0% 0.0% 0.0% skip:79.9%
x264 [info]: mb B I16..4: 0.5% 0.8% 0.0% B16..8: 6.9% 0.1% 0.1% direct: 0.2% skip:91.5% L0:45.6% L1:53.5% BI: 0.9%
x264 [info]: 8x8 transform intra:60.2% inter:93.7%
x264 [info]: direct mvs spatial:93.4% temporal:6.6%
x264 [info]: coded y,uvDC,uvAC intra:33.4% 48.8% 14.5% inter:1.1% 1.2% 0.1%
x264 [info]: ref P L0 62.0% 18.8% 10.8% 8.3%
x264 [info]: ref B L0 84.5% 9.7% 5.8%
x264 [info]: SSIM Mean Y:0.9666423
x264 [info]: PSNR Mean Y:40.736 U:44.759 V:46.566 Avg:41.811 Global:41.023 kb/s:984.06

encoded 400 frames, 3.75 fps, 984.44 kb/s

Pass 1:
--preset fast --tune animation --pass 1 --slow-firstpass --bitrate 1000 --level 4.1 --keyint 24 --min-keyint 1
--vbv-bufsize 30000 --vbv-maxrate 24000 --thread-input --direct auto --b-adapt 2 --nal-hrd --ssim --psnr --no-mbtree

SSIM/PNSR:
avis [info]: 1920x1080 @ 24.00 fps (400 frames)
x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 Cache64
x264 [info]: profile High, level 4.1

x264 [info]: slice P:167 Avg QP:31.99 size: 6805 PSNR Mean Y:41.16 U:45.00 V:46.80 Avg:42.20 Global:40.61
x264 [info]: slice B:214 Avg QP:34.39 size: 2249 PSNR Mean Y:41.01 U:45.17 V:47.09 Avg:42.10 Global:40.51
x264 [info]: consecutive B-frames: 13.4% 29.9% 23.6% 23.1% 5.2% 4.7%
x264 [info]: mb I I16..4: 33.0% 64.4% 2.6%
x264 [info]: mb P I16..4: 2.9% 5.4% 0.2% P16..4: 12.2% 1.0% 0.5% 0.0% 0.0% skip:77.8%
x264 [info]: mb B I16..4: 1.2% 0.0% 0.0% B16..8: 5.3% 0.2% 0.1% direct: 0.6% skip:92.6% L0:51.2% L1:44.6% BI: 4.2%
x264 [info]: final ratefactor: 33.01
x264 [info]: 8x8 transform intra:59.5% inter:81.6%
x264 [info]: direct mvs spatial:98.6% temporal:1.4%
x264 [info]: coded y,uvDC,uvAC intra:38.6% 53.2% 15.0% inter:2.0% 1.9% 0.1%
x264 [info]: ref P L0 69.0% 15.3% 8.8% 6.9%
x264 [info]: ref B L0 79.1% 12.7% 8.2%
x264 [info]: SSIM Mean Y:0.9642562
x264 [info]: PSNR Mean Y:41.117 U:45.110 V:46.981 Avg:42.183 Global:40.593 kb/s:1211.68

encoded 400 frames, 2.58 fps, 1212.12 kb/s

Pass 2:
--preset slow --tune animation --pass 2 --bitrate 1000 --level 4.1 --keyint 24 --min-keyint 1
--vbv-bufsize 30000 --vbv-maxrate 24000 --thread-input --trellis 2 --sar 1:1 --aud --nal-hrd --ssim --psnr --no-mbtree

SSIM/PNSR:
avis [info]: 1920x1080 @ 24.00 fps (400 frames)
x264 [info]: using SAR=1/1
x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 Cache64
x264 [info]: profile High, level 4.1

x264 [info]: slice P:167 Avg QP:33.94 size: 5149 PSNR Mean Y:40.49 U:44.43 V:46.34 Avg:41.56 Global:40.13
x264 [info]: slice B:214 Avg QP:35.98 size: 1948 PSNR Mean Y:40.87 U:44.95 V:46.98 Avg:41.96 Global:40.44
x264 [info]: consecutive B-frames: 13.4% 29.9% 23.6% 23.1% 5.2% 4.7%
x264 [info]: mb I I16..4: 34.4% 64.1% 1.5%
x264 [info]: mb P I16..4: 2.9% 4.0% 0.1% P16..4: 10.7% 1.0% 1.5% 0.0% 0.0% skip:79.9%
x264 [info]: mb B I16..4: 0.5% 0.8% 0.0% B16..8: 7.9% 0.1% 0.1% direct: 0.2% skip:90.4% L0:44.1% L1:54.6% BI: 1.3%
x264 [info]: 8x8 transform intra:61.7% inter:94.0%
x264 [info]: direct mvs spatial:95.8% temporal:4.2%
x264 [info]: coded y,uvDC,uvAC intra:35.1% 49.0% 12.5% inter:1.2% 1.2% 0.1%
x264 [info]: ref P L0 64.3% 17.8% 10.1% 7.7%
x264 [info]: ref B L0 84.9% 9.4% 5.8%
x264 [info]: SSIM Mean Y:0.9614927
x264 [info]: PSNR Mean Y:40.741 U:44.744 V:46.714 Avg:41.820 Global:40.333 kb/s:983.32

encoded 400 frames, 3.49 fps, 983.57 kb/s

[moviefan]

@rack04: Are you sure you encoded the mbtree one with bitrate 1000? Average bitrate is almost 6000 in the end...

[rack04]

@rack04: Are you sure you encoded the mbtree one with bitrate 1000? Average bitrate is almost 6000 in the end...

Looks like you're right. I'll re-encode and edit my post. Results have been updated. Sorry for the mistake.

[moviefan]

Looks like you're right. I'll re-encode and edit my post. If that is still an option.

Sure, I'd love to see some further benchmarks about the mbtree patch. Thanks in advance!

[imk]

http://imk.cx/junk/x264_mbtree.7z -- Updated with v0.13 of patch.

Patches used:
x264_hrd_pulldown.15_interlace.diff
x264_icc_03_win.diff
x264_win_zone_parse_fix_05.diff
mbtree patch v0.13

[IgorC]

Is it normal that avinaptic informs (lookahead default)?:

0.9 patch version
1st pass - lookahead=50
2d pass - lookahead=50

0.13 patch version
1st pass - lookahead=50
2d pass - lookahead=0

OPSNR is identical and SSIM is almost: 0.9v - .9926303, 0.13v - .9926298.

[wyti]

Changes in 0.11:

1. VBV now supported. Includes a separate VBV patch which fixes most VBV issues that MB-tree has; this will be committed separately, as it doesn't require MB-tree. It should also improve 1-pass VBV (especially CBR) performance on clips with lots of static content, e.g. anime, CGI.

Do note that CBR doesn't benefit much from MB-tree though, especially with small buffer sizes.

2. Lots of trivial parameter-handling fixes, e.g. lookahead automatically disabled on second pass.

So completely normal.

[nurbs]

Yes; see first post, changes in 0.11, point 2.

edit: beaten again :(

[Dark Shikari]

I will fix it later to read "lookahead" from the first pass statsfile rather than print zero.

[juGGaKNot]

OT : dark

new download link for version kkrunchya2 ?

a1 works fine, any updates on a2 ? link is down on the site and only other google result is an asian gui 2.3 but it needs registration ..

[Daiz]

I guess I might post these images as well. Done with patch 0.11. Click the images to open them in new tabs. Also, I totally forgot to include --ssim and --psnr in the command line options, but the images speak for themselves...

Source is a 5000-frame clip from a lossless H.264 stream that was encoded from an MPEG-2 Transport Stream.

Without mbtree:
x264 --pass 2 --stats "mbtree0.log" --bitrate 300 --psy-rd 1.0:0.2 --b-adapt 2 --bframes 3 --ref 15 --mixed-refs
--b-pyramid --me umh --subme 10 --merange 32 --partitions all --direct auto --trellis 2 -o "mbtree0.mkv" "18l.avs"

pass 1 - encoded 5000 frames, 68.38 fps, 290.88 kb/s
pass 2 - encoded 5000 frames, 16.38 fps, 299.97 kb/s

http://img40.imageshack.us/img40/3988/x264300kbpsmbtree0.png (http://img40.imageshack.us/img40/3988/x264300kbpsmbtree0.png)
Frame type B, size 646

With mbtree:
x264t --pass 2 --stats "mbtree1.log" --bitrate 300 --psy-rd 1.0:0.2 --b-adapt 2 --bframes 3 --ref 15 --mixed-refs
--mbtree --me umh --subme 10 --merange 32 --partitions all --direct auto --trellis 2 -o "mbtree1.mkv" "18l.avs"

pass 1 - encoded 5000 frames, 54.05 fps, 289.55 kb/s
pass 2 - encoded 5000 frames, 17.30 fps, 300.30 kb/s

http://img29.imageshack.us/img29/6396/x264300kbpsmbtree1.png (http://img29.imageshack.us/img29/6396/x264300kbpsmbtree1.png)
Frame type B, size 494

I have to say the difference is quite incredible.

[Chengbin]

That's amazing.

I want to do some testing with real life videos. How do I get a log from x264?

Also, is there a page explaining what SSIM and PSNR are and stuff?

[Fishman0919]

That's amazing.

I want to do some testing with real life videos. How do I get a log from x264?

Also, is there a page explaining what SSIM and PSNR are and stuff?


LINK (http://mewiki.project357.com/wiki/X264_Settings#psnr) to MeWiki for x264. Click on SSIM and PSNR to see details about them

[Snowknight26]

ffmpeg -i "test/LosslessTouhou.mkv" -vcodec rawvideo -f rawvideo -an -pix_fmt yuv420p - | x264.x64 --pass 1 --bitrate 50 --stats "test/test.stats" --bframes 5 --b-adapt 2 --direct auto --deblock -3:-3 --subme 4 --analyse none --me hex --fps 60 --output NUL - 640x480
ffmpeg -i "test/LosslessTouhou.mkv" -vcodec rawvideo -f rawvideo -an -pix_fmt yuv420p - | x264.x64 --pass 2 --bitrate 50 --stats "test/test.stats" --bframes 5 --b-adapt 2 --ref 5 --no-fast-pskip --b-pyramid --direct auto --deblock -3:-3 --subme 10 --trellis 2 --analyse all --8x8dct --me umh --ssim --fps 60 --output "test/test.mkv" - 640x480

ffmpeg -i "test/LosslessTouhou.mkv" -vcodec rawvideo -f rawvideo -an -pix_fmt yuv420p - | x264_mbtree13 --pass 1 --bitrate 50 --stats "test/test.stats" --bframes 5 --b-adapt 2 --direct auto --deblock -3:-3 --subme 4 --analyse none --me hex --fps 60 --output NUL - 640x480
ffmpeg -i "test/LosslessTouhou.mkv" -vcodec rawvideo -f rawvideo -an -pix_fmt yuv420p - | x264_mbtree13 --pass 2 --bitrate 50 --stats "test/test.stats" --bframes 5 --b-adapt 2 --ref 5 --no-fast-pskip --b-pyramid --direct auto --deblock -3:-3 --subme 10 --trellis 2 --analyse all --8x8dct --me umh --ssim --fps 60 --output "test/test.mbtree.mkv" - 640x480

http://i26.tinypic.com/vq15kg.png

http://i27.tinypic.com/2cz7sxj.png

Guess which is which.

Edit: Out of curiosity, does mbtree work with the lossless mode?

[Dark Shikari]

--bitrate 50ｗｗｗｗｗｗｗ

[Snowknight26]

Unfortunately the QP @ 50kbps was 51 for the first pass so the 2nd pass had to converge to ~400kbps. Even @ --bitrate 500 the average QP was ~49.

[Chengbin]

While doing tests with the crowdrun footage, I came across this warning on the second pass. This warning came many many times.

"MB-tree frametype (xxx) doesn't match actual frametype 0."
"x264_encoder_encode failed"

BTW, I wrote " --stats "c:\mbtree.log" to try to get a log file from x264. It gave me a stats file from the first pass. What do I write to get a log of the output? I don't have quick enough fingers to print screen the output the second it finishes.

[Dark Shikari]

While doing tests with the crowdrun footage, I came across this warning on the second pass. This warning came many many times.

"MB-tree frametype (xxx) doesn't match actual frametype 0."
"x264_encoder_encode failed"

BTW, I wrote " --stats "c:\mbtree.log" to try to get a log file from x264. It gave me a stats file from the first pass. What do I write to get a log of the output? I don't have quick enough fingers to print screen the output the second it finishes. 2> output.txt

It sounds like you specified the wrong input passfile (for example).@Snowknight26

Yes, mbtree does work with lossless.No, it doesn't, stop answering questions that you don't know about!

[popper]

Unfortunately the QP @ 50kbps was 51 for the first pass so the 2nd pass had to converge to ~400kbps. Even @ --bitrate 500 the average QP was ~49.

LOL at last, almost a visually good playable vga AVC over dialup ISP upload rates.

you used a 640x480 , try a CIF 352 × 288 PAL or widescreen 16/9 version and you might make it even better :)

http://en.wikipedia.org/wiki/Common_Intermediate_Format
"The CIF "image sizes" were specifically chosen to be multiples of macroblocks (i.e. 16 × 16 pixels) due to the way that discrete cosine transform based video compression/decompression is handled. So, by example, a CIF-size image (352 × 288) corresponds to 22 × 18 macroblocks."

[Chengbin]

No, it doesn't, stop answering questions that you don't know about!

Then why did x264 not crash when I used mbtree with lossless?

EDIT: On a second thought, there is no point of mbtree (or qcomp) in lossless mode anyway.

Sorry for spreading misinformation.

[Dark Shikari]

Original post updated with RC1. If there are no issues, MB-tree will be committed tomorrow.

[Daiz]

Sure is fast development in here ;)

[Shinigami-Sama]

Sure is fast development in here ;)

this is what happens when Dark finds out he can compress Touhou better

[tetsuo55]

Maybe he will find away to reach transparency for Touhou at less than 50kbit/s :P

[shon3i]

maybe here ;) :P http://forum.doom9.org/showthread.php?p=1311821#post1311821

[juGGaKNot]

x264_r1195_mbtree_RC1_GCC_4.4.1_juGGaKNot (http://www.mediafire.com/download.php?mjnolnongex), generic, fprofiled, patched
Source: GIT

Applied patches :

x264_win_zone_parse_fix_05.diff (http://www.mediafire.com/download.php?kqymujywhtw)
x264_Macroblock_Tree_Ratecontrol_RC1.diff (http://www.mediafire.com/download.php?hf2mnbzwomw)

It is fprofiled on a uncompressed 300 frames source with very high motion

Please check Doom9.org patches thread (http://forum.doom9.org/showthread.php?t=130364), Doom9.org Macroblock tree Ratecontrol thread (http://forum.doom9.org/showthread.php?t=148686) and GIT shortlog (http://git.videolan.org/gitweb.cgi?p=x264.git;a=shortlog) for more info.

Compiled by juGGaKNot (http://forum.doom9.org/member.php?u=144865) with GCC 4.4.1 on Windows XP SP-2 32-bit.

Platform: X86
System: MINGW
asm: yes
avis input: yes
mp4 output: yes
pthread: yes
debug: no
gprof: no
PIC: no
shared: no
visualize: no

Small side note : HRD 15 does not patch right with tree rc1 on gcc 4.4.1/4.4.0.

$ git reset --hard
HEAD is now at 5d75a9b Fix another 10L in QPRD

$ patch -p1 < /d/msys/d_win.diff
patching file encoder/ratecontrol.c
Hunk #1 succeeded at 659 (offset 21 lines).

$ patch -p1 < /d/msys/d_tre.diff
(Stripping trailing CRs from patch.)
patching file encoder/ratecontrol.c
Hunk #3 succeeded at 961 (offset 5 lines).
Hunk #5 succeeded at 1274 (offset 5 lines).
(Stripping trailing CRs from patch.)
patching file common/common.c
(Stripping trailing CRs from patch.)
patching file common/common.h
(Stripping trailing CRs from patch.)
patching file common/frame.c
(Stripping trailing CRs from patch.)
patching file common/frame.h
(Stripping trailing CRs from patch.)
patching file common/osdep.h
(Stripping trailing CRs from patch.)
patching file encoder/analyse.c
(Stripping trailing CRs from patch.)
patching file encoder/encoder.c
(Stripping trailing CRs from patch.)
patching file encoder/ratecontrol.c
Hunk #14 succeeded at 799 (offset 5 lines).
Hunk #16 succeeded at 1159 (offset 5 lines).
Hunk #18 succeeded at 1197 (offset 5 lines).
Hunk #20 succeeded at 1269 (offset 5 lines).
(Stripping trailing CRs from patch.)
patching file encoder/ratecontrol.h
(Stripping trailing CRs from patch.)
patching file encoder/slicetype.c
(Stripping trailing CRs from patch.)
patching file x264.c
(Stripping trailing CRs from patch.)
patching file x264.h

$ patch -p1 < /d/msys/d_hrd.diff
patching file common/bs.h
patching file common/common.c
Hunk #1 succeeded at 148 (offset 3 lines).
Hunk #3 succeeded at 598 (offset 9 lines).
patching file common/common.h
Hunk #1 succeeded at 332 (offset 44 lines).
patching file common/frame.c
Hunk #1 succeeded at 118 (offset 9 lines).
patching file common/frame.h
patching file common/set.h
patching file encoder/encoder.c
Hunk #2 succeeded at 753 (offset 28 lines).
Hunk #3 succeeded at 935 (offset 4 lines).
Hunk #4 succeeded at 968 (offset 28 lines).
Hunk #5 succeeded at 1390 (offset 4 lines).
Hunk #6 succeeded at 1424 (offset 28 lines).
Hunk #7 succeeded at 1422 (offset 4 lines).
Hunk #8 succeeded at 1605 (offset 34 lines).
Hunk #9 succeeded at 1637 (offset 4 lines).
Hunk #10 FAILED at 1733.
Hunk #11 FAILED at 1783.
Hunk #12 succeeded at 1829 (offset 36 lines).
2 out of 12 hunks FAILED -- saving rejects to file encoder/encoder.c.rej
patching file encoder/ratecontrol.c
Hunk #1 succeeded at 69 with fuzz 1 (offset 1 line).
Hunk #2 succeeded at 90 (offset 1 line).
Hunk #3 succeeded at 146 (offset 6 lines).
Hunk #4 succeeded at 323 with fuzz 2 (offset 37 lines).
Hunk #5 FAILED at 1126.
Hunk #6 FAILED at 1208.
Hunk #7 succeeded at 1376 (offset 83 lines).
Hunk #8 succeeded at 1342 (offset 37 lines).
2 out of 8 hunks FAILED -- saving rejects to file encoder/ratecontrol.c.rej
patching file encoder/ratecontrol.h
Hunk #1 FAILED at 34.
1 out of 1 hunk FAILED -- saving rejects to file encoder/ratecontrol.h.rej
patching file encoder/set.c
patching file encoder/set.h
patching file x264.c
Hunk #2 succeeded at 360 (offset 6 lines).
Hunk #4 succeeded at 477 (offset 10 lines).
patching file x264.h
Hunk #1 succeeded at 201 (offset 1 line).
Hunk #2 succeeded at 267 (offset 1 line).
Hunk #3 succeeded at 295 (offset 3 lines).

Same goes if i patch HRD first, tree fails.

[wyti]

Tested this RC on a (very) short game footage (around 500 frames)

Here is the logs without MBtree / Mbtree 0.13 / Mbtree RC1

Without :
D:\Encodage>x264 --slow-firstpass --keyint 1500 --min-keyint 2 --bframe 16 --b-a
dapt 2 --ref 16 --no-deblock --bitrate 1024 --qpmin 1 --qpmax 51 --ipratio 1.4 -
-aq-mode 2 --pass 1 --stats x264_stats.log --qcomp 0.66 --partitions all --direc
t auto --me tesa --merange 64 --subme 10 --psy-rd 0:0 --trellis 2 --no-fast-pski
p --threads 1 --ssim --psnr --thread-input --non-deterministic -o "TestSans.264"
"VideoFinal.avs"
avis [info]: 640x480 @ 60.00 fps (575 frames)
x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 FastShuffle SSE4.2
x264 [info]: profile High, level 3.2
x264 [info]: slice I:1 Avg QP:39.66 size: 8327 PSNR Mean Y:26.73 U:32.76
V:33.70 Avg:28.03 Global:28.03
x264 [info]: slice P:251 Avg QP:31.90 size: 2915 PSNR Mean Y:32.39 U:35.04
V:35.93 Avg:33.13 Global:31.79
x264 [info]: slice B:323 Avg QP:37.40 size: 905 PSNR Mean Y:29.86 U:33.72
V:34.72 Avg:30.83 Global:30.15
x264 [info]: consecutive B-frames: 12.0% 27.2% 38.2% 18.8% 1.7% 2.1% 0.0% 0.
0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0%
x264 [info]: mb I I16..4: 41.5% 38.5% 20.0%
x264 [info]: mb P I16..4: 3.0% 1.5% 0.4% P16..4: 24.2% 5.2% 6.5% 0.3% 0
.2% skip:58.7%
x264 [info]: mb B I16..4: 0.6% 0.3% 0.0% B16..8: 22.8% 0.8% 1.0% direct:
1.1% skip:73.4% L0:39.9% L1:55.1% BI: 5.0%
x264 [info]: final ratefactor: 29.08
x264 [info]: 8x8 transform intra:31.1% inter:44.5%
x264 [info]: direct mvs spatial:92.6% temporal:7.4%
x264 [info]: coded y,uvDC,uvAC intra:23.6% 36.0% 11.8% inter:5.0% 5.1% 1.0%
x264 [info]: ref P L0 74.4% 10.5% 5.3% 2.0% 1.5% 1.3% 1.0% 0.6% 0.5% 0.
5% 0.4% 0.4% 0.4% 0.4% 0.4% 0.3%
x264 [info]: ref B L0 75.4% 8.2% 4.2% 2.2% 1.7% 1.4% 1.3% 1.0% 0.8% 0.
7% 0.7% 0.5% 0.6% 0.6% 0.7%
x264 [info]: SSIM Mean Y:0.8545813
x264 [info]: PSNR Mean Y:30.958 U:34.295 V:35.244 Avg:31.830 Global:30.790 kb/s:
861.73

encoded 575 frames, 0.48 fps, 862.23 kb/s

D:\Encodage>x264 --keyint 1500 --min-keyint 2 --bframe 16 --b-a
dapt 2 --ref 16 --no-deblock --bitrate 1024 --qpmin 1 --qpmax 51 --ipratio 1.4 -
-aq-mode 2 --pass 2 --stats x264_stats.log --qcomp 0.66 --partitions all --direc
t auto --me tesa --merange 64 --subme 10 --psy-rd 0:0 --trellis 2 --no-fast-pski
p --threads 1 --ssim --psnr --thread-input --non-deterministic -o "TestSans.264"
"VideoFinal.avs"
avis [info]: 640x480 @ 60.00 fps (575 frames)
x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 FastShuffle SSE4.2
x264 [info]: profile High, level 3.2
x264 [info]: slice I:1 Avg QP:29.16 size: 38265 PSNR Mean Y:34.20 U:36.24
V:36.56 Avg:34.81 Global:34.81
x264 [info]: slice P:251 Avg QP:32.97 size: 3217 PSNR Mean Y:31.49 U:34.55
V:35.40 Avg:32.34 Global:31.98
x264 [info]: slice B:323 Avg QP:37.01 size: 1142 PSNR Mean Y:30.09 U:34.02
V:34.95 Avg:31.10 Global:30.75
x264 [info]: consecutive B-frames: 12.0% 27.2% 38.2% 18.8% 1.7% 2.1% 0.0% 0.
0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0%
x264 [info]: mb I I16..4: 19.2% 37.3% 43.5%
x264 [info]: mb P I16..4: 2.6% 1.5% 0.5% P16..4: 21.6% 6.3% 8.1% 0.3% 0
.2% skip:59.0%
x264 [info]: mb B I16..4: 0.7% 0.3% 0.1% B16..8: 23.4% 1.2% 1.4% direct:
1.5% skip:71.4% L0:43.3% L1:47.7% BI: 9.1%
x264 [info]: 8x8 transform intra:32.9% inter:39.0%
x264 [info]: direct mvs spatial:77.4% temporal:22.6%
x264 [info]: coded y,uvDC,uvAC intra:32.5% 40.4% 15.8% inter:5.7% 4.8% 0.7%
x264 [info]: ref P L0 71.9% 11.2% 6.0% 2.4% 1.7% 1.4% 1.2% 0.6% 0.6% 0.
6% 0.4% 0.4% 0.5% 0.4% 0.5% 0.3%
x264 [info]: ref B L0 73.9% 9.4% 4.7% 2.3% 1.8% 1.4% 1.2% 0.9% 0.7% 0.
6% 0.7% 0.5% 0.5% 0.6% 0.6%
x264 [info]: SSIM Mean Y:0.8685379
x264 [info]: PSNR Mean Y:30.709 U:34.255 V:35.146 Avg:31.651 Global:31.251 kb/s:
1013.81

encoded 575 frames, 0.49 fps, 1014.33 kb/s

With 0.13
D:\Encodage>x264MbTree --slow-firstpass --mbtree --lookahead 500 --keyint 1500 -
-min-keyint 2 --bframe 16 --b-adapt 2 --ref 16 --no-deblock --bitrate 1024 --qpm
in 1 --qpmax 51 --ipratio 1.4 --aq-mode 2 --pass 1 --stats x264_stats.log --qcom
p 0.66 --partitions all --direct auto --me tesa --merange 64 --subme 10 --psy-rd
0:0 --trellis 2 --no-fast-pskip --threads 1 --ssim --psnr --thread-input --non-
deterministic -o "TestAvec.264" "VideoFinal.avs"
avis [info]: 640x480 @ 60.00 fps (575 frames)
x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 FastShuffle SSE4.2 Slow
_mod4_stack
x264 [info]: profile High, level 3.2
x264 [info]: slice I:1 Avg QP:27.86 size: 43860 PSNR Mean Y:34.17 U:36.62
V:36.87 Avg:34.87 Global:34.87
x264 [info]: slice P:247 Avg QP:33.95 size: 2590 PSNR Mean Y:33.11 U:35.62
V:36.52 Avg:33.83 Global:32.80
x264 [info]: slice B:327 Avg QP:39.54 size: 851 PSNR Mean Y:30.71 U:34.54
V:35.51 Avg:31.69 Global:30.95
x264 [info]: consecutive B-frames: 12.2% 24.4% 38.2% 18.8% 4.4% 2.1% 0.0% 0.
0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0%
x264 [info]: mb I I16..4: 15.7% 39.4% 44.9%
x264 [info]: mb P I16..4: 2.7% 1.7% 0.3% P16..4: 20.8% 5.8% 5.0% 0.2% 0
.1% skip:63.5%
x264 [info]: mb B I16..4: 0.6% 0.3% 0.0% B16..8: 21.9% 0.8% 0.8% direct:
0.9% skip:74.8% L0:44.4% L1:50.0% BI: 5.5%
x264 [info]: final ratefactor: 29.73
x264 [info]: 8x8 transform intra:34.6% inter:45.2%
x264 [info]: direct mvs spatial:93.9% temporal:6.1%
x264 [info]: coded y,uvDC,uvAC intra:30.7% 40.9% 17.0% inter:4.4% 4.4% 0.6%
x264 [info]: ref P L0 71.0% 11.1% 6.2% 2.5% 1.8% 1.4% 1.3% 0.7% 0.6% 0.
6% 0.5% 0.4% 0.5% 0.5% 0.6% 0.3%
x264 [info]: ref B L0 75.9% 7.8% 4.0% 2.3% 1.8% 1.4% 1.3% 1.0% 0.8% 0.
6% 0.6% 0.6% 0.5% 0.5% 0.8%
x264 [info]: SSIM Mean Y:0.8788224
x264 [info]: PSNR Mean Y:31.746 U:35.009 V:35.946 Avg:32.611 Global:31.656 kb/s:
802.87

encoded 575 frames, 0.58 fps, 803.37 kb/s

D:\Encodage>x264MbTree --mbtree --lookahead 500 --keyint 1500 -
-min-keyint 2 --bframe 16 --b-adapt 2 --ref 16 --no-deblock --bitrate 1024 --qpm
in 1 --qpmax 51 --ipratio 1.4 --aq-mode 2 --pass 2 --stats x264_stats.log --qcom
p 0.66 --partitions all --direct auto --me tesa --merange 64 --subme 10 --psy-rd
0:0 --trellis 2 --no-fast-pskip --threads 1 --ssim --psnr --thread-input --non-
deterministic -o "TestAvec.264" "VideoFinal.avs"
avis [info]: 640x480 @ 60.00 fps (575 frames)
x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 FastShuffle SSE4.2 Slow
_mod4_stack
x264 [info]: profile High, level 3.2
x264 [info]: slice I:1 Avg QP:23.13 size: 71401 PSNR Mean Y:38.43 U:39.30
V:39.44 Avg:38.72 Global:38.72
x264 [info]: slice P:247 Avg QP:33.52 size: 3199 PSNR Mean Y:34.16 U:36.51
V:37.37 Avg:34.85 Global:33.78
x264 [info]: slice B:327 Avg QP:38.27 size: 1118 PSNR Mean Y:31.71 U:35.37
V:36.32 Avg:32.67 Global:31.77
x264 [info]: consecutive B-frames: 12.2% 24.4% 38.2% 18.8% 4.4% 2.1% 0.0% 0.
0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0%
x264 [info]: mb I I16..4: 4.8% 39.9% 55.3%
x264 [info]: mb P I16..4: 2.3% 1.7% 0.4% P16..4: 20.0% 6.7% 5.6% 0.3% 0
.1% skip:62.9%
x264 [info]: mb B I16..4: 0.7% 0.3% 0.1% B16..8: 23.5% 1.2% 1.3% direct:
1.3% skip:71.7% L0:45.9% L1:45.0% BI: 9.1%
x264 [info]: 8x8 transform intra:36.7% inter:39.2%
x264 [info]: direct mvs spatial:76.8% temporal:23.2%
x264 [info]: coded y,uvDC,uvAC intra:36.0% 44.4% 20.3% inter:5.6% 4.8% 0.8%
x264 [info]: ref P L0 69.0% 11.7% 6.4% 2.7% 2.0% 1.6% 1.4% 0.8% 0.7% 0.
6% 0.6% 0.5% 0.5% 0.5% 0.6% 0.4%
x264 [info]: ref B L0 74.3% 9.3% 4.4% 2.4% 1.8% 1.4% 1.2% 0.9% 0.7% 0.
6% 0.6% 0.5% 0.6% 0.6% 0.7%
x264 [info]: SSIM Mean Y:0.8957169
x264 [info]: PSNR Mean Y:32.775 U:35.869 V:36.773 Avg:33.617 Global:32.530 kb/s:
1024.41

encoded 575 frames, 0.58 fps, 1024.94 kb/s

With RC1
D:\Encodage>x264MbTree --slow-firstpass --mbtree --rc-lookahead 500 --keyint 15
0 --min-keyint 2 --bframe 16 --b-adapt 2 --ref 16 --no-deblock --bitrate 1024 -
qpmin 1 --qpmax 51 --ipratio 1.4 --aq-mode 2 --pass 1 --stats x264_stats.log --
comp 0.66 --partitions all --direct auto --me tesa --merange 64 --subme 10 --ps
-rd 0:0 --trellis 2 --no-fast-pskip --threads 1 --ssim --psnr --thread-input --
on-deterministic -o "TestAvec2.264" "VideoFinal.avs"
avis [info]: 640x480 @ 60.00 fps (575 frames)
x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 FastShuffle SSE4.2
x264 [info]: profile High, level 3.2
x264 [info]: slice I:1 Avg QP:29.35 size: 36309 PSNR Mean Y:33.08 U:35.84
V:36.29 Avg:33.86 Global:33.86
x264 [info]: slice P:247 Avg QP:34.36 size: 2469 PSNR Mean Y:32.18 U:35.02
V:35.94 Avg:32.98 Global:32.29
x264 [info]: slice B:327 Avg QP:38.99 size: 929 PSNR Mean Y:30.25 U:34.25
V:35.24 Avg:31.26 Global:30.74
x264 [info]: consecutive B-frames: 12.2% 24.4% 38.2% 18.8% 4.4% 2.1% 0.0% 0
0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0%
x264 [info]: mb I I16..4: 17.1% 38.0% 44.9%
x264 [info]: mb P I16..4: 2.7% 1.7% 0.4% P16..4: 20.6% 5.5% 6.2% 0.2%
.1% skip:62.6%
x264 [info]: mb B I16..4: 0.7% 0.3% 0.0% B16..8: 22.6% 0.8% 0.9% direct
1.1% skip:73.6% L0:43.6% L1:50.7% BI: 5.7%
x264 [info]: final ratefactor: 29.30
x264 [info]: 8x8 transform intra:34.8% inter:45.7%
x264 [info]: direct mvs spatial:93.3% temporal:6.7%
x264 [info]: coded y,uvDC,uvAC intra:31.5% 42.3% 18.1% inter:4.3% 4.3% 0.6%
x264 [info]: ref P L0 72.1% 11.0% 5.9% 2.3% 1.7% 1.4% 1.2% 0.6% 0.6% 0
5% 0.5% 0.4% 0.5% 0.5% 0.5% 0.3%
x264 [info]: ref B L0 76.0% 8.0% 4.0% 2.2% 1.7% 1.4% 1.3% 1.0% 0.8% 0
6% 0.7% 0.5% 0.6% 0.5% 0.7%
x264 [info]: SSIM Mean Y:0.8724616
x264 [info]: PSNR Mean Y:31.083 U:34.585 V:35.546 Avg:32.003 Global:31.341 kb/s
792.96

encoded 575 frames, 0.51 fps, 793.48 kb/s

D:\Encodage>x264MbTree --mbtree --rc-lookahead 500 --keyint 15
0 --min-keyint 2 --bframe 16 --b-adapt 2 --ref 16 --no-deblock --bitrate 1024 -
qpmin 1 --qpmax 51 --ipratio 1.4 --aq-mode 2 --pass 2 --stats x264_stats.log --
comp 0.66 --partitions all --direct auto --me tesa --merange 64 --subme 10 --ps
-rd 0:0 --trellis 2 --no-fast-pskip --threads 1 --ssim --psnr --thread-input --
on-deterministic -o "TestAvec2.264" "VideoFinal.avs"
avis [info]: 640x480 @ 60.00 fps (575 frames)
x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 FastShuffle SSE4.2
x264 [info]: profile High, level 3.2
x264 [info]: slice I:1 Avg QP:24.03 size: 65239 PSNR Mean Y:37.82 U:38.71
V:38.86 Avg:38.12 Global:38.12
x264 [info]: slice P:247 Avg QP:34.39 size: 3053 PSNR Mean Y:33.07 U:35.50
V:36.49 Avg:33.80 Global:33.19
x264 [info]: slice B:327 Avg QP:37.83 size: 1234 PSNR Mean Y:31.27 U:34.92
V:35.95 Avg:32.22 Global:31.59
x264 [info]: consecutive B-frames: 12.2% 24.4% 38.2% 18.8% 4.4% 2.1% 0.0% 0
0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0%
x264 [info]: mb I I16..4: 4.2% 40.4% 55.4%
x264 [info]: mb P I16..4: 2.5% 1.8% 0.5% P16..4: 19.7% 6.5% 5.8% 0.3%
.1% skip:62.9%
x264 [info]: mb B I16..4: 0.7% 0.4% 0.1% B16..8: 23.9% 1.2% 1.4% direct
1.5% skip:70.8% L0:45.2% L1:44.7% BI:10.1%
x264 [info]: 8x8 transform intra:36.4% inter:39.2%
x264 [info]: direct mvs spatial:71.9% temporal:28.1%
x264 [info]: coded y,uvDC,uvAC intra:37.3% 45.4% 21.1% inter:5.6% 4.7% 0.8%
x264 [info]: ref P L0 69.3% 11.8% 6.4% 2.6% 2.0% 1.5% 1.3% 0.7% 0.7% 0
6% 0.6% 0.5% 0.6% 0.5% 0.6% 0.4%
x264 [info]: ref B L0 74.9% 9.0% 4.4% 2.3% 1.7% 1.3% 1.2% 1.0% 0.8% 0
6% 0.6% 0.5% 0.6% 0.5% 0.7%
x264 [info]: SSIM Mean Y:0.8909777
x264 [info]: PSNR Mean Y:32.054 U:35.176 V:36.186 Avg:32.911 Global:32.213 kb/s
1020.89

encoded 575 frames, 0.50 fps, 1021.43 kb/s

I know i use strange (insane ? placebo ?) settings and the bitrate is very low (this source need more) but RC needs to be a little more accurate than 2 times more.
No-deblock is on purpose, i'ts ugly to see but i know where it takes bits and where it gives them.

But it's sad that we have a worse RC1 than the 0.13.
Will test on others sources to verify if RC1 is always worse.

[Trahald]

Ill fix hrd patch when mbtree is commited. (unless someone else beats me to it)

[burfadel]

Have you tried a higher rc-lookahead for comparison? 0.13 uses a value of 50, rc1 uses 40. Try again with 50 and see if the results align.

[wyti]

already tried, read my logs, in top of them there is the command line is used.
I've tested the pretty insane value of 500 for both. Maybe RC1 don't like too high values ?

[Chengbin]

I don't know how to interpret this data, so I'll leave it up to you guys to decode.

no mbtree

pass 1 (http://i26.tinypic.com/mslrmv.png)

http://i26.tinypic.com/mslrmv.png

pass 2

http://i27.tinypic.com/17rcqf.png

mbtree

pass 1

http://i30.tinypic.com/301zdyc.png

pass 2

http://i32.tinypic.com/10fbypf.png

no mbtree http://i27.tinypic.com/17rcqf.png

http://i26.tinypic.com/r2pzzs.png

mbtree http://i25.tinypic.com/10gjhxu.png

http://i25.tinypic.com/10gjhxu.png

[Japhsoncross]

i found mbtree gives quite good visual quality on game recordings.

but, i noticed mbtree doesn't treat fade in/out well.
my avs script:

blankclip(width=320, height=128, color=$FFFFFF, length=300, fps=30)
Subtitle("Hello", size=120)
FadeIO(150)
converttoyv12()


without mbtree

R:\>start /low /b /wait x264_mbtree_rc1.exe --keyint 300 --min-keyint 1 --bframes 6 --ref 16 --deblock 0:0 --bitrate 50 --pass 1
--stats "stats.log" --partitions all --direct auto --weightb --me esa --merange 12 --subme 10 --mixed-refs --8x8dct --no-fast-p
skip --no-dct-decimate --threads auto --thread-input --output NUL "111111.avs" --trellis 2 --b-adapt 2 --aq-mode 2 --no-mbtree
avis [info]: 320x128 @ 30.00 fps (302 frames)
x264 [info]: using cpu capabilities: MMX2 SSE2Fast FastShuffle SSEMisalign LZCNT
x264 [info]: profile Main, level 1.2
x264 [info]: slice I:6 Avg QP:27.83 size: 80
x264 [info]: slice P:197 Avg QP:30.94 size: 337
x264 [info]: slice B:99 Avg QP:32.34 size: 14
x264 [info]: consecutive B-frames: 50.7% 20.3% 2.0% 6.8% 5.1% 8.1% 7.1%
x264 [info]: mb I I16..4: 99.4% 0.0% 0.6%
x264 [info]: mb P I16..4: 57.6% 0.0% 0.0% P16..4: 31.2% 0.0% 0.0% 0.0% 0.0% skip:11.2%
x264 [info]: mb B I16..4: 0.0% 0.0% 0.0% B16..8: 0.1% 0.0% 0.0% direct: 0.2% skip:99.7% L0:18.2% L1:45.5% BI:36.4%
x264 [info]: final ratefactor: 34.90
x264 [info]: direct mvs spatial:19.2% temporal:80.8%
x264 [info]: coded y,uvDC,uvAC intra:4.8% 7.5% 2.4% inter:19.5% 9.4% 3.6%
x264 [info]: kb/s:54.3

encoded 302 frames, 327.55 fps, 54.90 kb/s

R:\>start /low /b /wait x264_mbtree_rc1.exe --keyint 300 --min-keyint 1 --bframes 6 --ref 16 --deblock 0:0 --bitrate 50 --pass 2
--stats "stats.log" --partitions all --direct auto --weightb --me esa --merange 12 --subme 10 --mixed-refs --8x8dct --no-fast-p
skip --no-dct-decimate --threads auto --thread-input --output "1195m.Test.mkv" "111111.avs" --psnr --ssim --trellis 2 --b-adapt
2 --aq-mode 2 --no-mbtree
avis [info]: 320x128 @ 30.00 fps (302 frames)
x264 [info]: using cpu capabilities: MMX2 SSE2Fast FastShuffle SSEMisalign LZCNT
x264 [info]: profile High, level 2.1
x264 [info]: slice I:6 Avg QP:20.29 size: 296 PSNR Mean Y:70.12 U:70.44 V:93.22 Avg:70.65 Global:57.59
x264 [info]: slice P:197 Avg QP:29.27 size: 291 PSNR Mean Y:39.95 U:40.46 V:45.77 Avg:40.56 Global:39.79
x264 [info]: slice B:99 Avg QP:27.78 size: 27 PSNR Mean Y:43.13 U:43.69 V:48.40 Avg:43.71 Global:41.95
x264 [info]: consecutive B-frames: 50.7% 20.3% 2.0% 6.8% 5.1% 8.1% 7.1%
x264 [info]: mb I I16..4: 85.1% 8.8% 6.1%
x264 [info]: mb P I16..4: 38.9% 4.8% 1.1% P16..4: 22.4% 2.2% 1.7% 0.0% 0.0% skip:28.8%
x264 [info]: mb B I16..4: 0.4% 0.0% 0.0% B16..8: 8.4% 0.0% 0.0% direct: 0.1% skip:91.1% L0: 7.3% L1:83.2% BI: 9.5%
x264 [info]: 8x8 transform intra:10.6% inter:84.8%
x264 [info]: direct mvs spatial:22.2% temporal:77.8%
x264 [info]: coded y,uvDC,uvAC intra:3.0% 7.2% 2.4% inter:19.6% 10.3% 4.1%
x264 [info]: ref P L0 94.1% 3.3% 0.9% 0.1% 0.2% 0.0% 0.6% 0.0% 0.3% 0.1% 0.1% 0.0% 0.1% 0.1% 0.1% 0.1%
x264 [info]: ref B L0 79.2% 20.8%
x264 [info]: SSIM Mean Y:0.9875633
x264 [info]: PSNR Mean Y:41.593 U:42.114 V:47.573 Avg:42.188 Global:40.489 kb/s:49.07

encoded 302 frames, 65.95 fps, 49.74 kb/s


with mbtree

R:\>start /low /b /wait x264_mbtree_rc1.exe --keyint 300 --min-keyint 1 --bframes 6 --ref 16 --deblock 0:0 --bitrate 50 --pass 1
--stats "stats.log" --partitions all --direct auto --weightb --me esa --merange 12 --subme 10 --mixed-refs --8x8dct --no-fast-p
skip --no-dct-decimate --threads auto --thread-input --output NUL "111111.avs" --trellis 2 --b-adapt 2 --aq-mode 2 --mbtree
avis [info]: 320x128 @ 30.00 fps (302 frames)
x264 [info]: using cpu capabilities: MMX2 SSE2Fast FastShuffle SSEMisalign LZCNT
x264 [info]: profile Main, level 1.2
x264 [info]: slice I:6 Avg QP:35.92 size: 80
x264 [info]: slice P:198 Avg QP:48.34 size: 157
x264 [info]: slice B:98 Avg QP:49.27 size: 14
x264 [info]: consecutive B-frames: 51.0% 20.3% 3.0% 5.4% 5.1% 8.1% 7.1%
x264 [info]: mb I I16..4: 99.3% 0.0% 0.7%
x264 [info]: mb P I16..4: 58.8% 0.0% 0.0% P16..4: 4.5% 0.0% 0.0% 0.0% 0.0% skip:36.7%
x264 [info]: mb B I16..4: 0.0% 0.0% 0.0% B16..8: 0.0% 0.0% 0.0% direct: 0.0% skip:100.0% L0:-1.$% L1:-1.$% BI:-1.$%
x264 [info]: final ratefactor: 1.#J
x264 [info]: direct mvs spatial:0.0% temporal:100.0%
x264 [info]: coded y,uvDC,uvAC intra:3.1% 19.0% 8.7% inter:1.0% 1.9% 0.4%
x264 [info]: kb/s:26.2

encoded 302 frames, 264.68 fps, 26.78 kb/s

R:\>start /low /b /wait x264_mbtree_rc1.exe --keyint 300 --min-keyint 1 --bframes 6 --ref 16 --deblock 0:0 --bitrate 50 --pass 2
--stats "stats.log" --partitions all --direct auto --weightb --me esa --merange 12 --subme 10 --mixed-refs --8x8dct --no-fast-p
skip --no-dct-decimate --threads auto --thread-input --output "1195mbt.Test.mkv" "111111.avs" --psnr --ssim --trellis 2 --b-adap
t 2 --aq-mode 2 --mbtree
avis [info]: 320x128 @ 30.00 fps (302 frames)
x264 [info]: using cpu capabilities: MMX2 SSE2Fast FastShuffle SSEMisalign LZCNT
x264 [info]: profile High, level 2.1
x264 [info]: slice I:6 Avg QP:36.05 size: 78 PSNR Mean Y:56.39 U:66.80 V:92.42 Avg:57.22 Global:47.00
x264 [info]: slice P:198 Avg QP:30.51 size: 280 PSNR Mean Y:39.92 U:41.00 V:45.74 Avg:40.56 Global:39.42
x264 [info]: slice B:98 Avg QP:33.28 size: 32 PSNR Mean Y:40.55 U:42.39 V:47.90 Avg:41.30 Global:39.17
x264 [info]: consecutive B-frames: 51.0% 20.3% 3.0% 5.4% 5.1% 8.1% 7.1%
x264 [info]: mb I I16..4: 45.4% 54.6% 0.0%
x264 [info]: mb P I16..4: 31.9% 6.9% 0.6% P16..4: 22.9% 1.5% 1.5% 0.0% 0.0% skip:34.6%
x264 [info]: mb B I16..4: 0.5% 0.5% 0.0% B16..8: 13.3% 0.0% 0.0% direct: 0.0% skip:85.7% L0: 5.5% L1:87.2% BI: 7.3%
x264 [info]: 8x8 transform intra:20.4% inter:81.0%
x264 [info]: direct mvs spatial:0.0% temporal:100.0%
x264 [info]: coded y,uvDC,uvAC intra:1.5% 4.6% 1.4% inter:18.5% 9.7% 4.4%
x264 [info]: ref P L0 92.4% 3.6% 1.1% 0.2% 0.3% 0.1% 0.6% 0.1% 0.5% 0.1% 0.2% 0.0% 0.4% 0.1% 0.1% 0.3%
x264 [info]: ref B L0 81.2% 16.2% 0.4% 0.0% 0.4% 1.1% 0.0% 0.0% 0.7%
x264 [info]: SSIM Mean Y:0.9744154
x264 [info]: PSNR Mean Y:40.454 U:41.967 V:47.369 Avg:41.133 Global:39.405 kb/s:46.89

encoded 302 frames, 65.30 fps, 47.57 kb/s

[poisondeathray]

but, i noticed mbtree doesn't treat fade in/out well.


I've only done 1 test so far, but I noticed this too qualitatively where every other frame seems similar or maybe slightly better, it's definitely worse on fades during the entire fade sequence (if you interleave() 2 encodes). I'll do some more tests on other sources, and if this is a consistent observation I'll post some examples

[Daiz]

I can confirm that fades aren't treated well - at the same bitrate or even at higher bitrates, fades ended up worse-looking with mbtree on than with mbtree off. It's a shame since the mbtree encodes look better everywhere else.

[Dark Shikari]

This seems to be inherent in the algorithm and I'm not entirely sure how to resolve it... it was a problem in RDRC as well, in the exact same way.

I can't replicate the problem too much though; my testing shows that fades, at least with b-adapt 2 and aq-mode 1, are usually good enough. They might be significantly worse without b-adapt 2 though.

They also eat up an extraordinary amount of bits with or without MB-tree; hopefully weightp will fix this.

[squid808]

This seems to be inherent in the algorithm and I'm not entirely sure how to resolve it... it was a problem in RDRC as well, in the exact same way.

I can't replicate the problem too much though; my testing shows that fades, at least with b-adapt 2 and aq-mode 1, are usually good enough. They might be significantly worse without b-adapt 2 though.

They also eat up an extraordinary amount of bits with or without MB-tree; hopefully weightp will fix this.

Could you elaborate a bit on the nature of the inherent problem with the algorithm?

[Dark Shikari]

Could you elaborate a bit on the nature of the inherent problem with the algorithm?MB-tree lowers the quality on things that aren't referenced much in the future.

Without weighted prediction, fades are nearly all intra blocks...

[Zachs]

Speaking of which, how's weightp coming along? IIRC that's part of the SoC'09 isn't it?

[Dark Shikari]

Speaking of which, how's weightp coming along? IIRC that's part of the SoC'09 isn't it?Very well. We have a number of algorithms so far; the result will probably be something like this:

--weightp 0 (off)
--weightp 1 (blind weighting, equivalent to searching one more ref frame in terms of speed cost)
--weightp 2 (normal weighting, meant to catch fades)
--weightp 3 (K-means, :awesome: weighting, slow)

[Japhsoncross]

Very well. We have a number of algorithms so far; the result will probably be something like this:

--weightp 0 (off)
--weightp 1 (blind weighting, equivalent to searching one more ref frame in terms of speed cost)
--weightp 2 (normal weighting, meant to catch fades)
--weightp 3 (K-means, :awesome: weighting, slow)

OMG, i just can't wait ;)

[Revgen]

no mbtree http://i27.tinypic.com/17rcqf.png

http://i26.tinypic.com/r2pzzs.png

mbtree http://i25.tinypic.com/10gjhxu.png

http://i25.tinypic.com/10gjhxu.png

If you look closely at the foliage farthest in the background, you can see more details preserved with mbtree. Very slight. But it's there.

[Fr4nz]

If you look closely at the foliage farthest in the background, you can see more details preserved with mbtree. Very slight. But it's there.

Yes. This is true also for the ground...look at the grass.

[G_M_C]

[...]
I can't replicate the problem too much though; my testing shows that fades, at least with b-adapt 2 and aq-mode 1, are usually good enough.
[...]

Speaking of that; I've done some more testing and I notice encodes using --mbtree with AQ-mode 2 tend to have less quality then when AQ-mode 1 is used (best noticable in dark scenes / more artifacting). Has anyone else seen this too ?

[nakTT]

Noob question here,

Am I good to go by just downloading the latest build 1198 from Jarod's http://x264.nl/ website? Sorry for the noob question, I just can't bear letting you all have fun with the "Macroblock Tree Ratecontrol" without me.:D

:thanks:

[bob0r]

Noob question here,

Am I good to go by just downloading the latest build 1198 from Jarod's http://x264.nl/ website? Sorry for the noob question, I just can't bear letting you all have fun with the "Macroblock Tree Ratecontrol" without me.:D

:thanks:

commit bb66c482242a0747823661b212114c1a2f015fe3 r1197
Author: Jason Garrett-Glaser <darkshikari@gmail.com>
Date: Tue Aug 4 17:46:33 2009 -0700

Macroblock-tree ratecontrol
On by default; can be turned off with --no-mbtree.

[juGGaKNot]

commit bb66c482242a0747823661b212114c1a2f015fe3 r1197
Author: Jason Garrett-Glaser <darkshikari@gmail.com>
Date: Tue Aug 4 17:46:33 2009 -0700

Macroblock-tree ratecontrol
On by default; can be turned off with --no-mbtree.

higher values on --rc-lookahead ( than default 40 ) will make use of mbtree

lower values or --qcomp ( than default 0.6 ) will make use of mbtree

gl

[nakTT]

commit bb66c482242a0747823661b212114c1a2f015fe3 r1197
Author: Jason Garrett-Glaser <darkshikari@gmail.com>
Date: Tue Aug 4 17:46:33 2009 -0700

Macroblock-tree ratecontrol
On by default; can be turned off with --no-mbtree.
Thanks for the reply,

Actually I have read that before posting my question. I just wanted to be absolutely sure that it will worked by just swap the old build with a new one (because the GUI on MeGUI is not yet updated, so might not be able to play with the lookahead and the like) as I don't want to end up wasting the ecoding time with a nonworking package. BTW, I'm using MeGUI 1051. Any thought?

:thanks:

[Yoshiyuki Blade]

Thanks for the reply,

Actually I have read that before posting my question. I just wanted to be absolutely sure that it will worked by just swap the old build with a new one (because the GUI on MeGUI is not yet updated, so might not be able to play with the lookahead and the like) as I don't want to end up wasting the ecoding time with a nonworking package. BTW, I'm using MeGUI 1051. Any thought?

:thanks:

Either way, MeGUI has a section to input custom commands. So if mb tree didn't happen to be the default, you could just put in --mbtree and --rc-lookahead X. In this case, you can just put in --rc-lookahead X since mbtree is on by default now :).

[nakTT]

I guess I might post these images as well. Done with patch 0.11. Click the images to open them in new tabs. Also, I totally forgot to include --ssim and --psnr in the command line options, but the images speak for themselves...

Source is a 5000-frame clip from a lossless H.264 stream that was encoded from an MPEG-2 Transport Stream.

Without mbtree:
x264 --pass 2 --stats "mbtree0.log" --bitrate 300 --psy-rd 1.0:0.2 --b-adapt 2 --bframes 3 --ref 15 --mixed-refs
--b-pyramid --me umh --subme 10 --merange 32 --partitions all --direct auto --trellis 2 -o "mbtree0.mkv" "18l.avs"

pass 1 - encoded 5000 frames, 68.38 fps, 290.88 kb/s
pass 2 - encoded 5000 frames, 16.38 fps, 299.97 kb/s

Frame type B, size 646

With mbtree:
x264t --pass 2 --stats "mbtree1.log" --bitrate 300 --psy-rd 1.0:0.2 --b-adapt 2 --bframes 3 --ref 15 --mixed-refs
--mbtree --me umh --subme 10 --merange 32 --partitions all --direct auto --trellis 2 -o "mbtree1.mkv" "18l.avs"

pass 1 - encoded 5000 frames, 54.05 fps, 289.55 kb/s
pass 2 - encoded 5000 frames, 17.30 fps, 300.30 kb/s

Frame type B, size 494

I have to say the difference is quite incredible.
After comparing the pictures, All I can say is that it was amazing!!!

I can now officially say goodbye to any other encoder now. Unless some miracle happen in the future.

[nakTT]

Either way, MeGUI has a section to input custom commands. So if mb tree didn't happen to be the default, you could just put in --mbtree and --rc-lookahead X. In this case, you can just put in --rc-lookahead X since mbtree is on by default now :).
Many thanks for the guidance.

:thanks:

[shon3i]

Ill fix hrd patch when mbtree is commited. (unless someone else beats me to it)
Your turn :)

http://mirror01.x264.nl/x264/changelog.txt

And why nal-hrd is not commited yet?

[bob0r]

:readguid:Your turn :)

http://mirror01.x264.nl/x264/changelog.txt

And why nal-hrd is not commited yet?

Unless the full blu-ray specs are published, they won't add/make patches for blu-ray encoding.

[Forteen88]

What doesn't it work with yet?

--b-pyramidWhat happens if I enable --b-pyramid with mbtree enabled? Is --b-pyramid automatically forced-disabled until I disable mbtree?

BTW, thanks for this great new code, DS (and Pengvado).

EDIT: Ok, thnx for the answer J_Darnley. I was too lazy to test it out myself :-P

[J_Darnley]

Yes, B-pyramid is disabled.

[rack04]

What are you thoughts on the size of the statsfile? I understand this is a necessary file but I just completed 1st pass and the size of the statsfile is a little over 500mb. Being a default option I wonder if the size of this file might pose a problem.

[Dark Shikari]

What are you thoughts on the size of the statsfile? I understand this is a necessary file but I just completed 1st pass and the size of the statsfile is a little over 500mb. Being a default option I wonder if the size of this file might pose a problem.If you have room for your output video, you surely have room for the statsfile ;)

[rack04]

If you have room for your output video, you surely have room for the statsfile ;)

Good point. I was just surprised by the size. Thanks.

[Raere]

My completely unscientific test:

Encoded the season 1 Stargate Atlantis opening (crf 25)

no MB tree (b-pyramid on)


x264 [info]: SSIM Mean Y:0.9642903
x264 [info]: PSNR Mean Y:42.396 U:49.296 V:50.054 Avg:43.650 Global:39.904 kb/s:997.48
encoded 1456 frames, 6.24 fps, 997.64 kb/s

[memory usage at finish was 719MB]

MB tree (rc-lookahead 40)


x264 [info]: SSIM Mean Y:0.9655772
x264 [info]: PSNR Mean Y:42.021 U:48.948 V:49.683 Avg:43.276 Global:40.204 kb/s:931.08
encoded 1456 frames, 6.10 fps, 931.24 kb/s

[memory usage at finish was 790MB]

MB tree (rc-lookahead 48)


x264 [info]: SSIM Mean Y:0.9655575
x264 [info]: PSNR Mean Y:42.015 U:48.950 V:49.683 Avg:43.271 Global:40.199 kb/s:931.86
encoded 1456 frames, 6.08 fps, 932.01 kb/s

[memory usage at finish was 785MB]

MB tree (rc-lookahead 64)


x264 [info]: SSIM Mean Y:0.9655707
x264 [info]: PSNR Mean Y:42.021 U:48.956 V:49.676 Avg:43.276 Global:40.204 kb/s:931.64
encoded 1456 frames, 6.12 fps, 931.80 kb/s

[memory usage at finish was 839MB]

[wyti]

mbtree requires aq >= 1, but does aq-strength (if != 0) affect the intensity of the mbtree ?

[Dark Shikari]

mbtree requires aq >= 1, but does aq-strength (if != 0) affect the intensity of the mbtree ?No, but AQ does affect the internal logic of MB-tree (not in a plus or minus strength way, but more subtly).

[imk]

I did an encode of Touhou last night.

There's two viewable versions:
http://imk.cx/videos/INExtraStage.html (Uses Flash)
http://imk.cx/videos/INExtraStage.html5 (Uses HTML5 <video> tag -- Only works with Chrome or Safari, as Firefox only supports Theora)

The MP4 is directly downloadable from here:
http://imk.cx/videos/files/INExtraStage.mp4 (289 MB)


Encoding settings (using r1195 with RC1 patch):
--preset placebo --tune touhou --crf 25

[cacepi]

OMG, i just can't wait ;)

Why wait?

GSOC project to get p-frames on x264 (http://repo.or.cz/w/x264/x264-p-frames.git)

[kemuri-_9]

Does mb-tree have a grudge against extremely short clips (like 36 frames)? I can never get it to encode. It would finish, but the resulting .h264 file is 0 bytes.

there's a known issue that if the frame delay is greater than the length of the input source, it will generally fail to work.
this delay value is dependent on a few settings:

if( h->param.i_bframe_adaptive == X264_B_ADAPT_TRELLIS )
h->frames.i_delay = X264_MAX(h->param.i_bframe,3)*4 + h->param.i_threads - 1;
else
h->frames.i_delay = h->param.i_bframe + h->param.i_threads - 1;
if( h->param.rc.b_mb_tree )
h->frames.i_delay = X264_MAX( h->frames.i_delay, h->param.rc.i_lookahead );


with the default value of rc-lookahead at 40, it's longer than your clip's length,
so either reduce rc-lookahead or don't use mbtree until this is corrected.

[Dark Shikari]

This will hopefully be fixed soon. Harass akupenguin about it.

[Chengbin]

akupenguin, where are you? (No just kidding)

Looks like I shouldn't have deleted my comment. I deleted it because it's not working with longer clips as well. It works with another clip (not that short though). I'm investigating this issue. I'll try rebooting the computer.

Urghh, I really should do further testing before asking. It does work with longer clips.

Also (there I go again), even with --no-mbtree, it doesn't encode the 36 frame clip. Even with an older version of x264. How did I encode it before???

[Revgen]

Got 83.70957253 SSIM with regular x264.

Got 83.86610669 SSIM with MBTree.

So technically it's an improvement.

Used Me-GUI settings program --profile high --pass 2 --bitrate 1320 --stats ".stats" --keyint 240 --min-keyint 24 --b-adapt 2 --b-pyramid --direct auto --subme 9 --trellis 2 --partitions all --thread-input --output "output" "input" on a 1 hour 7 minute black and white movie.

Couldn't really see any difference watching it with my own eyes. Perhaps it'll be better once B-pyramid support comes along.

[Kurtnoise]

Could you update this too ?

FAQ:

How do I use it?

It's on by default. --no-mbtree turns it off.

What doesn't it work with yet?

--b-pyramid

How can I tweak it?

--rc-lookahead; default is 40, higher takes more memory and is slower, but should give better results. It's probably overkill for most sources. If you get crashes immediately after the lookahead fills up, you probably ran out of memory (max is 2GB for x264 on 32-bit Windows).

--qcomp; works the same as it did before with actual qcomp, higher numbers lower MB-tree strength.



Maybe, adding this also can help some people for --rc-lookahead:
min = 0
max = max GOP size (default 250)

and for mb-tree:
if (mb-tree on) and (aq-mode = 0) then -aq-mode = 1
if (mb-tree on) and (b-pyramid on) then b-pyramid is off (x264 displays a warning anyway)





:thanks:

[burfadel]

rc lookahead of 250 is probably a bit excessive, maybe 150 max... although thats still high and may not prove beneficial over say, 80. Either way as DS has said its clipped to keyint anyway.

[juGGaKNot]

Default is 40/changes per profile

max is keyint

why clip it to 80 ?

[ocal5]

Another real life video test, with 1080i59,97 "ideal" picture : 1 sequence, man moving inside stable picture

WITHOUT

--bitrate 1200 --no-mbtree --interlaced --me umh --ssim --psnr
http://img188.imageshack.us/i/without.png/ image 630
http://dl.free.fr/qYKmpjUQz encoded video
SSIM Mean Y:0.8531129
PSNR Mean Y:32.314 U:42.239 V:42.956 Avg:33.865

WITH

--bitrate 1200 --interlaced --me umh --ssim --psnr
http://img33.imageshack.us/i/withj.png/ image 630
http://dl.free.fr/jxU0sG6Cs encoded video
SSIM Mean Y:0.8838190
PSNR Mean Y:33.609 U:42.781 V:43.630 Avg:35.125

Difference is so impressive that I doubt there is only "this" mbtree who do difference. But at least, as Dark Shikari explained in first message the theory, it seems real and not only for animes !

[CruNcher]

finaly lookahead funny that it's sooner finished then weighted p frames :D

btw tetsuo lookahead was allready in Atemes Encoder since about 2-3 years now :)

if it's the same thing visually it should show especialy @ parkrun without AQ heavy improvements i guess

[ocal5]

tetsuo lookahead

Sorry, what is that curious thing?

[Guest]

tetsuo is a member here and a comma was omitted:

btw tetsuo, lookahead ...

[CruNcher]

yeah CruNcher and punctuation ;)

[popper]

Another real life video test, with 1080i59,97 "ideal" picture : 1 sequence, man moving inside stable picture

WITHOUT

--bitrate 1200 --no-mbtree --interlaced --me umh --ssim --psnr
http://img188.imageshack.us/i/without.png/ image 630
http://dl.free.fr/qYKmpjUQz encoded video
SSIM Mean Y:0.8531129
PSNR Mean Y:32.314 U:42.239 V:42.956 Avg:33.865

WITH

--bitrate 1200 --interlaced --me umh --ssim --psnr
http://img33.imageshack.us/i/withj.png/ image 630
http://dl.free.fr/jxU0sG6Cs encoded video
SSIM Mean Y:0.8838190
PSNR Mean Y:33.609 U:42.781 V:43.630 Avg:35.125

Difference is so impressive that I doubt there is only "this" mbtree who do difference. But at least, as Dark Shikari explained in first message the theory, it seems real and not only for animes !

thats really impressive, without even looking at it on the big screen or zooming in, its so clearly better around the man's hair and the statue's head against the sky in the second png.

is that due to the first interlaced sequence we have seen MTR tested here, or is it also the same for a Progressive version of that sequence too ?

[vucloutr]

Strength of MB-tree adjustments can be tweaked using qcompress; higher values mean lower MB-tree strength.
what does higher or lower MB-tree strength mean?
could someone be so kind to explain this on an example?

[ocal5]

is that due to the first interlaced sequence we have seen MTR tested here, or is it also the same for a Progressive version of that sequence too ?

This sequence is interlaced, how do you want me to make it progressive ? Not sure that improvement is due to interlacing. More because it's a shoot choosen because to be ideal regarding the theory of this patch.

I have a somewhat not linked question :

This picture is before I frame, and the second one after (time related). (cropped)

http://img197.imageshack.us/img197/2017/beforeid.jpg (http://img197.imageshack.us/i/beforeid.jpg/)
http://img197.imageshack.us/img197/7545/afterio.jpg (http://img197.imageshack.us/i/afterio.jpg/)

I've try with slow settings, 2 pass, -i 1... everytime, there is a "shock" on the video starting the "I" frame. Background is sharp but before key frame it's blur.

Seems insame in my mind, but... is there a way for the codec to relate B&P frames to the comming I frame ? Or, is the only solution to put an I frame after (in this case) zooming ?

[Dark Shikari]

finaly lookahead funny that it's sooner finished then weighted p frames :D

btw tetsuo lookahead was allready in Atemes Encoder since about 2-3 years now :)Ateme's encoder does not have MB-tree or anything remotely similar.

Its lookahead is for ratecontrol.

[Amefurashi]

Version 1201 seems to have some problems. Maybe it's just me...

:scared:

CLI is the following:

x264.exe --pass 1 --slow-firstpass --bitrate 4642 --stats "xxx.stats"
--level 4.1 --keyint 240 --min-keyint 24 --ref 5 --mixed-refs --bframes 5
--b-pyramid --b-adapt 2 --weightb --direct auto --deblock -2:-1 --subme 9
--trellis 2 --psy-rd 1.0:0 --partitions all --8x8dct --vbv-bufsize 50000
--vbv-maxrate 50000 --me umh --merange 24 --threads auto --thread-input
--aq-strength 1.0 --no-dct-decimate --no-mbtree --output NUL yyy.avs
pause

Everything went fine. Second pass (VERY SAME options, except for --pass 2 and --output "zzz.mp4") goes this way:


x264.exe --pass 2 --slow-firstpass --bitrate 4642 --stats "xxx.stats" --lev
el 4.1 --keyint 240 --min-keyint 24 --ref 5 --mixed-refs --bframes 5 --b-pyramid
--b-adapt 2 --weightb --direct auto --deblock -2:-1 --subme 9 --trellis 2 --psy
-rd 1.0:0 --partitions all --8x8dct --vbv-bufsize 50000 --vbv-maxrate 50000 --me
umh --merange 24 --threads auto --thread-input --aq-strength 1.0 --no-dct-decim
ate --no-mbtree --output "zzz.mp4" yyy.avs
avis [info]: 1280x532 @ 23.98 fps (141409 frames)
x264 [warning]: width or height not divisible by 16 (1280x532), compression will
suffer.
x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 FastShuffle SSE4.1 Cach
e64
x264 [info]: profile High, level 4.1
mp4 [info]: initial delay 2002 (scale 24000)
[0.0%] 20/141409 frames, 33.06 fps, 0.00 kb/s, eta 1:11:17
F:\TEST
>pause


It suddendly stops when the encoding begins. Ideas? :confused:




EDIT: nevermind, just read the bug thread... MP4 output is broken in rev1201. My bad >_<

[bob0r]

Version 1201 seems to have some problems. Maybe it's just me...

:scared:

It suddendly stops when the encoding begins. Ideas? :confused:

Confirmed (and fixed?): http://forum.doom9.org/showthread.php?p=1312586#post1312586

[Amefurashi]

Confirmed (and fixed?): http://forum.doom9.org/showthread.php?p=1312586#post1312586

Thanks bob0r, MKV output works fine ;)

[CruNcher]

@Dark_Shikari
i get this error message continuously till the end after the first 250 frames (keyframe interval setting) with mbtree

x264 [warning]: specified frame type (3) is not compatible with keyframe interval

--crf 29 --preset fast --profile high -b 0 --b-adapt 0 --direct temporal --aq-mode 0 --ssim --psnr --scenecut -1 --nf --partitions i8x8,i4x4,p8x8

without mbtree memory consumption was @ the end 110 mb with mbtree 293 mb

[Dark Shikari]

@Dark_Shikari
i get this error message continuously till the end after the first 200 frames with mbtree on parkrun

x264 [warning]: specified frame type (3) is not compatible with keyframe interval

--crf 29 --preset fast --profile high -b 0 --b-adapt 0 --direct temporal --aq-mode 0 --ssim --psnr --scenecut -1 --nf --partitions i8x8,i4x4,p8x8

without mbtree memory consumption was @ the end 110 mb with mbtree 293 mbFixed.

[martinfrombern]

great patch :) here is how it looks on a source i'm currently working with. common parameters for both encodes --bitrate 9000 --aq 0.9 --psy-rd 1.0:0.1 --ref 4 --me umh --merange 32 (...). Here come the results

source / mbtree / no mbtree
http://thumbnails4.imagebam.com/4475/49ae4144744750.gif (http://www.imagebam.com/image/49ae4144744750) http://thumbnails13.imagebam.com/4475/27ee6344744753.gif (http://www.imagebam.com/image/27ee6344744753) http://thumbnails14.imagebam.com/4475/c0546244744755.gif (http://www.imagebam.com/image/c0546244744755)

as you can see, mbtree encode is significantly better, there are no spots on and around muzzle, and it is almost transparent to the source. Inspecting some other screenshots it's visible that mbtree is also better in noise/grain retention.

[poisondeathray]

With all the good, there is usually some bad....Just providing more observations of the trend towards "worse fades" These are just illustrative single frame examples , but EVERY frame in the fade sequences are clearly worse using mbtree. Detail seems to be missing, white clouds in the planet earth example, detail in the background buildings in the TDK example, more blurry, etc... I've done 2 other encodes (not posted here), and the fades seem to exhibit the same behaviour, I'll report back if I find anything different but so far it seems to be a reproducible trend
(Open them in tabs in your browser and flip back & forth)


planet earth trailer 1080i , 1740 frames
source available here: http://www.sendspace.com/file/cr48oe
avcsource,yadif(order=1),lanczosresize(720,400)
r1201, ref 5, 3b-frames, no b-pyramid, subme9, psy1:0, b-adapt2, 2pass 1000kbps

original
http://i26.tinypic.com/1ftrgj.png
mbtree
http://i32.tinypic.com/oa645h.png
no mbtree
http://i25.tinypic.com/n54kua.png



dark knight apple movie trailer 480p , 3042 frames
source available from Apple
r1201, ref 5, 3b-frames, no b-pyramid, subme9, psy1:0, b-adapt2, 2pass 1000kbps

original
http://i25.tinypic.com/wlvrz8.png
mbtree
http://i25.tinypic.com/29dtxys.png
no mbtree
http://i26.tinypic.com/nff9g3.png

[burfadel]

Thats a known side effect of mb-tree, weighted p-frames which is currently being worked on is meant to resolve that issue (as stated by DS).

[martinfrombern]

yep, seems like a signicant blur there :(

[poisondeathray]

Thats a known side effect of mb-tree, weighted p-frames which is currently being worked on is meant to resolve that issue (as stated by DS).

Thanks, I read that, and I know DS is already aware of this issue from earlier in the thread but I just promised to post some examples earlier and never got around to it till this weekend

BTW, I didn't mean to be negative or anything, I'm just providing some feedback/observations. I think these guys are doing an awesome job, and keep up the great work!

[popper]

poisondeathray, if you took the time to actually read the whole thread, you will realise fades are a known problem, and DS already said it will improve later when the other works done, read it, and keep in mind this is still a testing addition even though its committed for Non D9 readers to also see it and get it for testing.

[martinfrombern]

poisondeathray, if you took the time to actually read the whole thread, you will realise fades are a known problemi believe he pretty much knows that, because he was the one who mentioned fades in the first place. so i believe he's not the one who should actually read the whole thread

[Snowknight26]

I'm sure that only really happens in bitrate starved situations anyway.

[poisondeathray]

I'm sure that only really happens in bitrate started situations anyway.

You're probably right Mr. Touhou 50bits/s ! :D

[Fr4nz]

Did you mean "starved"? Well, for 480p 1000kbit/s is not so "starved"... :D

[CruNcher]

This is amazing just amazing parkrun @ 2 mbits 1pass and so great visually and that @ fantastic speeds i almost can't believe it (ok still some temporal fluctuations but hey 720p @ 2 mbits) :D if you compare how it looked like before in terms of artifacting (all the chroma problems colored lines that moved behind motion gone) this is imho a huge step for people knowing the complexity of this source :) not entirely sure what really improved it that much but mbtree alone can't be the only reason :)

And all this @ very sane compression settings :D

[Snowknight26]

Just tried doing a small 1080p sample encode of my Run Fatboy Run Blu-ray. To achieve the same SSIM (of whatever it was, I forget) using the same almost-placebo settings, the encode without mbtree was ~10000kbps while the one with it on was ~6000kbps. Visually identical to me at least.

[Chengbin]

Hey Dark Shikari

How much quality increase do you expect if I increase rc-lookahead to 100? Can I expect 1-2%?

[Sagekilla]

I'd imagine this would be one of those settings that, like ref frames, depend highly on the source in question. I don't think situations where one frame can have effects as far out as 100 frames away are very common, except maybe in anime. So I don't think the quality difference will be significant from 50 -> 100.

Theoretically, a larger lookahead should be better but in practice you might not even seen that difference. In anime you might, though.

[martinfrombern]

I did my tests on poisondeathray's planet earth trailer sample with various bitrates. Here come the results...
264 settings are
x264 --level 3.1 --ref 6 --mixed-refs --no-fast-pskip --bframes 4 --b-adapt 2 --b-pyramid --weightb --direct auto --deblock -3:-3 --subme 9 --trellis 2 --partitions p8x8,b8x8,i4x4,i8x8 --8x8dct --me umh
--merange 32 --threads 6 --aq-strength 1.0 --psy-rd 1.0:0.0 --thread-input --no-dct-decimate --frames 1740 --fps 30000/1001
Screenshots
source / bitrate 500 mbtree / bitrate 500 nombtree / bitrate 1000 mbtree / bitrate 1000 nombtree / bitrate 1500 mbtree / bitrate 1500 nombtree / bitrate 2000 mbtree / bitrate 2000 nombtree
http://thumbnails14.imagebam.com/4491/235e3d44907080.gif (http://www.imagebam.com/image/235e3d44907080) http://thumbnails15.imagebam.com/4491/47939f44907082.gif (http://www.imagebam.com/image/47939f44907082) http://thumbnails8.imagebam.com/4491/9ef89744907064.gif (http://www.imagebam.com/image/9ef89744907064) http://thumbnails14.imagebam.com/4491/dd7c6f44907070.gif (http://www.imagebam.com/image/dd7c6f44907070) http://thumbnails17.imagebam.com/4491/dfd99444907071.gif (http://www.imagebam.com/image/dfd99444907071) http://thumbnails15.imagebam.com/4491/fb4afb44907073.gif (http://www.imagebam.com/image/fb4afb44907073) http://thumbnails15.imagebam.com/4491/b1d2de44907074.gif (http://www.imagebam.com/image/b1d2de44907074) http://thumbnails8.imagebam.com/4491/ca737344907075.gif (http://www.imagebam.com/image/ca737344907075) http://thumbnails16.imagebam.com/4491/538d2944907078.gif (http://www.imagebam.com/image/538d2944907078)
As can be seen, nombtree version is better with all bitrates :(

[shon3i]

Well, comparing to source mbtree to me looks much closer :)

[donis]

As can be seen, nombtree version is better with all bitrates :(True.

Is mbtree really worth to go official? Will all that mbtree adaptations affect picture quality
(in comparison with latest x264 builds w/o mbtree and it's adoptations) even when it will be disabled?

Will make few tests myself, and report back.

[juGGaKNot]

mbtree is comited, download any revision over 1200 and its on by default

--no-mbtree to not use it.

[Chengbin]

Guys, please read the thread.

mb-tree is NOT optimized for fades (yet). The upcoming weight-p will correct this.

[Betsy25]

At this moment, It would certainly make more logic to make --no-mbtree default though.

[donis]

At this moment, It would certainly make more logic to make --no-mbtree default though.Yup, I also think that way. :)

[Chengbin]

Give me a good reason?

mb-tree improves video quality on all video except fading. So you think mb-tree sucks just because of 1 scenario?

[Kurtnoise]

why a such claim ?

Anyway, you're free to not use the last updates. It's up to you after all...

[jdobbs]

I've noticed some odd behaviour that I thought I'd report. It has to do with resulting filesizes of CRF encodes when the mbtree option is enabled.

I have a routine that does short encodes of samples from a source in order to find the closest CRF to obtain correct filesizing (a "quick" first pass). I noticed it stopped working correctly when mbtree was enabled. It failed to converge on a CRF value. Disabling mbtree returns the routine to normal behavior. So I investigated and noticed that at some point it hits a strange situation in which a lower CRF value actually created a smaller (rather than larger) file. Here is an example (all using the same source):

CRF=22.36 --> Filesize: 1,276,648
CRF=22.20 --> Filesize: 1,302,169
CRF=22.14 --> Filesize: 1,304,863
CRF=22.13 --> Filesize: 1,303,447* (smaller than 22.14)
CRF=22.12 --> Filesize: 1,304,684* (smaller than 22.14)
CRF=22.11 --> Filesize: 1,304,552* (smaller than 22.14)
CRF=22.10 --> Filesize: 1,303,587* (smaller than 22.14)
CRF=22.09 --> Filesize: 1,559,003
CRF=22.08 --> Filesize: 1,557,601* (smaller than 22.09)
CRF=22.07 --> Filesize: 1,558,434* (smaller than 22.09)
CRF=22.06 --> Filesize: 1,558,228* (smaller than 22.09)

Since it is sample frames the source is small, in this case 96 frames. It's no big deal and I can work around it, but it seems a little odd that decreasing the CRF would result in a lowered filesize, so I though I'd report it.

Is this an expected side effect of mbtree?

[elguaxo]

@jdobbs

Also, note that CRF is redefined again by this patch--I set it so that it kept the same bitrate on Blackpearl as before, but it may vary for different videos.

"Redefining CRF" means that CRF's measure of quality changes, so some videos may have higher or lower bitrates at the same CRF.

[Dark Shikari]

It's probably just luck; at such tiny increments anything could cause that to happen.

MB-tree just increases the likelihood by significantly increasing the amount of quantizer modification.

[jdobbs]

Cool. :cool: Thanks. I did notice that in the results between the two (no-mbtree and default settings) -- mbtree shows a significantly lower CRF to get the same filesize.

[IgorC]

I did my tests on poisondeathray's planet earth trailer
Have you link to source or upload it somewhere like http://www.mediafire.com/ ?

[rack04]

Have you link to source or upload it somewhere like http://www.mediafire.com/ ?

http://forum.doom9.org/showthread.php?p=1312714#post1312714

[IgorC]

Oh, Thanks. Didn't see it was the same.

[Zachs]

I did notice with aq-mode 2, fades are good enough (as dark has mentioned). On non-fade scenes, quality is visually much closer to source vs no-mbtree.
aq-mode is 1 by default.

[Forteen88]

I did notice with aq-mode 2, fades are good enough (as dark has mentioned). On non-fade scenes, quality is visually much closer to source vs no-mbtree.
aq-mode is 1 by default.Where did he write that? Not here:
"my testing shows that fades, at least with b-adapt 2 and aq-mode 1, are usually good enough."
https://forum.doom9.org/showthread.php?p=1312014#post1312014

EDIT: @DS, okey I was a bit confused about that. Yeah, I prefer better image overall but tiny less good fades. Although I'll wait a while for weightp until I do new encodes.

[Dark Shikari]

Where did he write that? Not here:
"my testing shows that fades, at least with b-adapt 2 and aq-mode 1, are usually good enough."
https://forum.doom9.org/showthread.php?p=1312014#post1312014I think he was just referring to the fact that I said fades were good enough. I didn't do much testing with the different AQ modes though.

[Zachs]

Thanks Dark.

Darn, my post *was* confusing...

What I meant to say was, while fades are good enough (as Dark has mentioned), I did notice aq-mode 2 to be closer to source on fades than mode 1. And comparing non-fade scenes of the same encode between the 2 modes with mb-tree enabled, my eyes tell me they're about the same. aq-mode is set at 1 by default so it may be worth giving aq-mode 2 a try when using mb-tree.

[MasterNobody]

Fades can also look awful with --mbtree and --aq-mode 2. Here is my test (63.2 Mb): http://www.mediafire.com/?yjojzmznzzt
Also in this test I tried to compensate this effect by adding --aq-mode 3 based on the same idea as --aq-mode 2 but with fixed curve center (so more bias to dark scenes).
Used acronyms:
base: --aq-mode 0 --mbtree --b-adapt 2 --psy-rd 0:0 --psnr --ssim
std: --aq-mode 1 --no-mbtree --b-adapt 2 --psy-rd 0:0 --psnr --ssim
auto: --aq-mode 2 --no-mbtree --b-adapt 2 --psy-rd 0:0 --psnr --ssim
new: --aq-mode 3 --no-mbtree --b-adapt 2 --psy-rd 0:0 --psnr --ssim
mbtstd: --aq-mode 1 --mbtree --b-adapt 2 --psy-rd 0:0 --psnr --ssim
mbtauto: --aq-mode 2 --mbtree --b-adapt 2 --psy-rd 0:0 --psnr --ssim
mbtnew: --aq-mode 3 --mbtree --b-adapt 2 --psy-rd 0:0 --psnr --ssim
So you can compare --mbtree with --no-mbtree and between different AQ modes.
It was encoded on Core i7 so --threads auto == --threads 12

P.S. I even though --aq-mode 3 help in this test (and get the highest SSIM) its usefulness highly depend from source. Here are results for different sources: http://stashbox.org/594950/test.xls
P.P.S. Lossless source: http://www.mediafire.com/?kmzz5mdznr2

[G_M_C]

I think we'll get the the real step forward when adaptive p frames are introduced, especially on the subject of fades and dark scenes.

I've seen on the GSOC project page that the majority of set goals were achieved already, more goeals than DS hoped for at first instance, reading his comment on that page. If i understand that page correctly the code-optimalisation fase was undergoing. After that maybe the testing can be done more public "alpha / beta / RC" testing phase (like the testing of mbtree done in this thread).

Before that i dont think discussing fades beeing good, bad or ugly is usefull. Cause it's allready know that fades are not handled optimally.

[nixo]

I'm having a bit of a problem hitting the desired bitrate using mbtree. Source is Elephant's Dream and I'm using the unpatched 32bit build of rev. 1206 from x264.nl.
Perhaps I'm wrong but it doesn't appear to be a question of saturation.
Options used: -B 8000 --level 4.0 --preset veryslow --tune animation -b 3 -I 24 -i 1 --no-progress --vbv-bufsize 31250 --vbv-maxrate 25000

With mbtree:

Pass1:
avis [info]: 1920x1080 @ 24.00 fps (15691 frames)
x264 [warning]: VBV bitrate (25000) > level limit (20000)
x264 [warning]: VBV buffer (31250) > level limit (25000)
x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 Cache64
x264 [info]: profile Main, level 4.0
x264 [info]: slice I:836 Avg QP:14.77 size:182341
x264 [info]: slice P:6827 Avg QP:18.22 size: 48533
x264 [info]: slice B:8028 Avg QP:21.68 size: 18782
x264 [info]: consecutive B-frames: 13.9% 28.2% 41.6% 16.3%
x264 [info]: mb I I16..4: 43.8% 0.0% 56.2%
x264 [info]: mb P I16..4: 20.1% 0.0% 0.0% P16..4: 36.9% 0.0% 0.0% 0.0% 0.0%
skip:43.0%
x264 [info]: mb B I16..4: 3.4% 0.0% 0.0% B16..8: 18.2% 0.0% 0.0% direct:
8.8% skip:69.7% L0:34.8% L1:41.5% BI:23.7%
x264 [info]: final ratefactor: 17.59
x264 [info]: direct mvs spatial:76.6% temporal:23.4%
x264 [info]: coded y,uvDC,uvAC intra:42.2% 44.5% 18.6% inter:14.3% 10.7% 0.6%
x264 [info]: kb/s:7764.6

encoded 15691 frames, 8.23 fps, 7765.44 kb/s

Pass2:
avis [info]: 1920x1080 @ 24.00 fps (15691 frames)
x264 [info]: using SAR=1/1
x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 Cache64
x264 [warning]: target: 8000.00 kbit/s, expected: 7659.36 kbit/s, avg QP: 22.2821
x264 [info]: profile High, level 4.0
x264 [info]: slice I:836 Avg QP:15.62 size:184470
x264 [info]: slice P:6827 Avg QP:19.99 size: 46680
x264 [info]: slice B:8028 Avg QP:23.57 size: 19026
x264 [info]: consecutive B-frames: 13.9% 28.2% 41.6% 16.3%
x264 [info]: mb I I16..4: 21.8% 53.4% 24.7%
x264 [info]: mb P I16..4: 5.5% 10.2% 1.7% P16..4: 23.6% 9.5% 6.3% 0.8% 0.4%
skip:42.1%
x264 [info]: mb B I16..4: 0.7% 1.5% 0.4% B16..8: 27.1% 2.2% 2.6% direct:
3.6% skip:62.0% L0:47.1% L1:39.1% BI:13.8%
x264 [info]: 8x8 transform intra:56.6% inter:53.6%
x264 [info]: direct mvs spatial:70.0% temporal:30.0%
x264 [info]: coded y,uvDC,uvAC intra:49.5% 46.7% 22.0% inter:14.1% 7.5% 0.9%
x264 [info]: ref P L0 72.2% 14.9% 8.3% 4.7%
x264 [info]: ref B L0 76.8% 15.5% 7.7%
x264 [info]: kb/s:7655.6

encoded 15691 frames, 1.40 fps, 7656.33 kb/s

Without mbtree:

Pass1:
avis [info]: 1920x1080 @ 24.00 fps (15691 frames)
x264 [warning]: VBV bitrate (25000) > level limit (20000)
x264 [warning]: VBV buffer (31250) > level limit (25000)
x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 Cache64
x264 [info]: profile Main, level 4.0
x264 [info]: slice I:836 Avg QP:15.67 size:146565
x264 [info]: slice P:7300 Avg QP:17.30 size: 48504
x264 [info]: slice B:7555 Avg QP:20.16 size: 20117
x264 [info]: consecutive B-frames: 19.4% 25.9% 37.2% 17.4%
x264 [info]: mb I I16..4: 42.9% 0.0% 57.1%
x264 [info]: mb P I16..4: 19.8% 0.0% 0.0% P16..4: 40.5% 0.0% 0.0% 0.0% 0.0%
skip:39.7%
x264 [info]: mb B I16..4: 3.7% 0.0% 0.0% B16..8: 20.9% 0.0% 0.0%
direct:11.5% skip:64.0% L0:34.4% L1:43.4% BI:22.2%
x264 [info]: final ratefactor: 18.90
x264 [info]: direct mvs spatial:84.6% temporal:15.4%
x264 [info]: coded y,uvDC,uvAC intra:45.8% 48.9% 20.5% inter:15.6% 13.8% 0.8%
x264 [info]: kb/s:7691.7

encoded 15691 frames, 8.97 fps, 7692.55 kb/s

Pass2:
avis [info]: 1920x1080 @ 24.00 fps (15691 frames)
x264 [info]: using SAR=1/1
x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 Cache64
x264 [info]: profile High, level 4.0
x264 [info]: slice I:836 Avg QP:15.24 size:160784
x264 [info]: slice P:7300 Avg QP:17.46 size: 48311
x264 [info]: slice B:7555 Avg QP:20.90 size: 22024
x264 [info]: consecutive B-frames: 19.4% 25.9% 37.2% 17.4%
x264 [info]: mb I I16..4: 19.0% 56.0% 25.1%
x264 [info]: mb P I16..4: 3.8% 11.0% 2.2% P16..4: 24.6% 11.3% 7.0% 0.7% 0.3%
skip:39.2%
x264 [info]: mb B I16..4: 0.8% 1.6% 0.4% B16..8: 28.7% 2.5% 3.0% direct:
4.5% skip:58.4% L0:45.1% L1:40.2% BI:14.7%
x264 [info]: 8x8 transform intra:60.9% inter:53.3%
x264 [info]: direct mvs spatial:73.9% temporal:26.1%
x264 [info]: coded y,uvDC,uvAC intra:54.2% 54.4% 28.2% inter:15.6% 9.6% 1.0%
x264 [info]: ref P L0 72.4% 15.3% 8.2% 4.1%
x264 [info]: ref B L0 77.0% 15.8% 7.2%
x264 [info]: kb/s:7996.1

encoded 15691 frames, 1.30 fps, 7996.89 kb/s

--
Nikolaj

[SquallMX]

Hi, i using MBTree for first time, and the Maxbitrate is not respected by x264 and causes buffer underflows.

Non MBTree (x264 1195)

"D:\Archivos de programa\megui\tools\x264\x264.exe" --profile high --pass 2 --bitrate 5900 --stats "D:\Temp\BR\DBE2.stats" --level 4.1 --keyint 48 --min-keyint 4 --b-adapt 2 --direct auto --deblock -2:-1 --psy-rd 0.8:0.2 --partitions p8x8,b8x8,i4x4,i8x8 --qpmax 40 --ipratio 1.1 --pbratio 1.2 --vbv-bufsize 15000 --vbv-maxrate 15000 --me umh --thread-input --aq-strength 0.8 --ssim --output "D:\Temp\BR\DBE.mkv" "D:\Temp\BR\DBE.avs" --mvrange 511 --aud --nal-hrd --sar 4:3

Max bitrate acording to DGAVCIndex: 14.736

MBTree (x264 1201)

"D:\Archivos de programa\megui\tools\x264\x264.exe" --profile high --pass 2 --bitrate 5900 --stats "D:\Temp\BR\DBE2.stats" --level 4.1 --keyint 48 --min-keyint 4 --b-adapt 2 --direct auto --deblock -2:-1 -mbtree --psy-rd 0.8:0.2 --partitions p8x8,b8x8,i4x4,i8x8 --qpmax 40 --ipratio 1.1 --pbratio 1.2 --rc-lookahead 47 --aq-mode 2 --vbv-bufsize 15000 --vbv-maxrate 15000 --me umh --thread-input --aq-strength 0.8 --ssim --output "D:\Temp\BR\DBE2.mkv" "D:\Temp\BR\DBE.avs" --mvrange 511 --aud --nal-hrd

Max bitrate acording to DGAVCIndex: 23.718
x264 [warning]: VBV underflow (-99323 bits)

Any Help? :thanks:

[Dark Shikari]

Stop setting qmax to 40; you're crippling the ability of VBV to do its job.

[SquallMX]

Stop setting qmax to 40; you're crippling the ability of VBV to do its job.

:thanks:, I will try again using the default value (51) but according to AVINaptic de Max Quant used in the stream is 29, i will post the results later.

[Dark Shikari]

:thanks:, I will try again using the default value (51) but according to AVINaptic de Max Quant used in the stream is 29, i will post the results later.Avinaptic prints frame quantizers, not block quantizers, the latter of which QPmax controls.

[juGGaKNot]

start "encode" /b /low /wait "%mypath%\bin\x264.exe" --pass 1 --preset veryslow --level %mylevel% --bitrate %btratex264% --stats "%mypath%\%mpath%\T1\%mymovie%.stats" --log-file "%mypath%\%mpath%\x264_1pass.log" --keyint %kint% --min-keyint %mint% --deblock -2;-2 --fullrange on --trellis 2 --mbtree --ref 4 --bframes 4 --merange 32 --ipratio 1.1 --pbratio 1.1 --vbv-bufsize 20000 --vbv-maxrate 20000 --qcomp 1.0 --aq-mode 1 --aq-strength 0.5 --sar 1:1 --aud --nal-hrd --slow-firstpass --output NUL %myavs%
echo.
echo %langi%
echo.
start "encode" /b /low /wait "%mypath%\bin\x264.exe" --pass 2 --preset veryslow --level %mylevel% --bitrate %btratex264% --stats "%mypath%\%mpath%\T1\%mymovie%.stats" --log-file "%mypath%\%mpath%\x264_2pass.log" --keyint %kint% --min-keyint %mint% --deblock -2;-2 --fullrange on --trellis 2 --mbtree --ref 4 --bframes 4 --merange 32 --ipratio 1.1 --pbratio 1.1 --vbv-bufsize 20000 --vbv-maxrate 20000 --qcomp 1.0 --aq-mode 1 --aq-strength 0.5 --sar 1:1 --aud --nal-hrd --output "%mypath%\%mpath%\T1\%mymovie%.264" %myavs%
echo.

Encoding X264 1st Pass :

avis [info]: 1136x656 @ 30.00 fps (756 frames)
x264 [info]: using SAR=1/1
x264 [info]: using cpu capabilities: MMX2 SSE2Slow
x264 [info]: cabac=1 ref=4 deblock=1:-2:-2 analyse=0x3:0x133 me=umh subme=10 psy
=1 psy_rd=1.0:0.0 mixed_ref=1 me_range=32 chroma_me=1 trellis=2 8x8dct=1 cqm=0 d
eadzone=21,11 chroma_qp_offset=-2 threads=3 nr=0 decimate=1 mbaff=0 bframes=4 b_
pyramid=0 b_adapt=2 b_bias=0 direct=3 wpredb=1 keyint=300 keyint_min=30 scenecut
=40 rc=cbr mbtree=0 bitrate=4000 ratetol=1.0 qcomp=1.00 qpmin=10 qpmax=51 qpstep
=4 vbv_maxrate=20000 vbv_bufsize=20000 ip_ratio=1.10 pb_ratio=1.10 aq=1:0.50
x264 [info]: profile High, level 4.0
x264 [info]: slice I:12 Avg QP:27.03 size: 22740
x264 [info]: slice P:300 Avg QP:25.28 size: 23924
x264 [info]: slice B:444 Avg QP:27.49 size: 16857
x264 [info]: consecutive B-frames: 5.0% 41.1% 22.6% 19.9% 11.4%
x264 [info]: mb I I16..4: 27.9% 67.7% 4.5%
x264 [info]: mb P I16..4: 4.1% 19.1% 1.6% P16..4: 36.4% 15.2% 8.1% 0.2% 0
.1% skip:15.2%
x264 [info]: mb B I16..4: 0.9% 4.6% 0.5% B16..8: 48.1% 3.6% 4.6% direct:
12.6% skip:25.2% L0:45.5% L1:44.4% BI:10.1%
x264 [info]: final ratefactor: 27.51
x264 [info]: 8x8 transform intra:76.2% inter:83.4%
x264 [info]: direct mvs spatial:99.8% temporal:0.2%
x264 [info]: coded y,uvDC,uvAC intra:67.3% 62.1% 27.1% inter:36.6% 29.0% 2.3%
x264 [info]: ref P L0 70.1% 14.8% 9.5% 5.6%
x264 [info]: ref B L0 75.2% 15.4% 9.4%

encoded 756 frames, 1.40 fps, 4741.16 kb/s

Encoding X264 2nd Pass :

avis [info]: 1136x656 @ 30.00 fps (756 frames)
x264 [info]: using SAR=1/1
x264 [info]: using cpu capabilities: MMX2 SSE2Slow
x264 [info]: cabac=1 ref=4 deblock=1:-2:-2 analyse=0x3:0x133 me=umh subme=10 psy
=1 psy_rd=1.0:0.0 mixed_ref=1 me_range=32 chroma_me=1 trellis=2 8x8dct=1 cqm=0 d
eadzone=21,11 chroma_qp_offset=-2 threads=3 nr=0 decimate=1 mbaff=0 bframes=4 b_
pyramid=0 b_adapt=2 b_bias=0 direct=3 wpredb=1 keyint=300 keyint_min=30 scenecut
=40 rc=2pass mbtree=0 bitrate=4000 ratetol=1.0 qcomp=1.00 qpmin=10 qpmax=51 qpst
ep=4 cplxblur=20.0 qblur=0.5 vbv_maxrate=20000 vbv_bufsize=20000 ip_ratio=1.10 p
b_ratio=1.10 aq=1:0.50
x264 [info]: profile High, level 4.0
[16.1%] 122/756 frames, 3.88 fps, 1143.08 kb/s, eta 0:02:43

is this normal ?

[Dark Shikari]

qcomp=1 is synonymous with mbtree=0.

[stpdrgstr]

Guys, sorry to look dumb, but, what's a good value of lookahead with 2 gb of ram and a 24 minutes video @ 24 fps?

I was playing with mbtree in a 1 minute video at the same frame rate, using 250 of lookahead, but I don't know what would be safe, yet "the max" for a longer one.

[nakTT]

Guys, sorry to look dumb, but, what's a good value of lookahead with 2 gb of ram and a 24 minutes video @ 24 fps?

I was playing with mbtree in a 1 minute video at the same frame rate, using 250 of lookahead, but I don't know what would be safe, yet "the max" for a longer one.
Yeah I also wonder the same thing. How about 2 hours long movies @25fps with 4GB RAM? Any simple formula that we, layman can use as a general guideline?

[Dark Shikari]

Yeah I also wonder the same thing. How about 2 hours long movies? Any simple formula that we, layman can use as a general guideline?General advice for laymen?!

Don't touch the bloody thing you gits!

[nakTT]

General advice for laymen?!

Don't touch the bloody thing you gits!
Sorry, wrong word I guess :D. I mean for not so layman. Come on, some advice please.

[Comatose]

The new CRF (with mbtree) is rather unstable with the new source, while previously CRF would produce encodes of similar sources at about 200 kbps within each other.

Mouryou no Hako (grainy, HD anime):

Episode 2 - source (http://i32.tinypic.com/32zrvv6.png)
Episode 2 - CRF 18.5 (before mbtree, b-pyramid enabled) = ~7000 kbps (http://i31.tinypic.com/16bxdgk.png)
Episode 2 - CRF 13 (with mbtree) = ~6900 kbps (http://i28.tinypic.com/2yvpjzk.png)

Episode 4 - CRF 18.5 (mbtree disabled, b-pyramid enabled) = ~7250 kbps
Episode 4 - CRF 12.85 (with mbtree) = ~5600 kbps

Is it likely that episode 4 just included that much more areas which mbtree is effective with (considering grain is heaviest in dark scenes and lightest in bright scenes, and episode 4 had more bright scenes than episode 2)?

edit: At episode 2, mbtree is actually worse at times. Added links to episode 2 above. This is compared to an "old" encode, though, before mbtree was introduced, so it uses the old b-adapt, etc. I'll encode again, this time using the new version with --no-mbtree. Check the bottom left and to the (your) right of the woman's rightmost arm. The earlier encode was not perfect in this regions either, but it actually got worse. Also, there's artifacting to the left of the close guy's neck

[Astrophizz]

Yeah I also wonder the same thing. How about 2 hours long movies @25fps with 4GB RAM? Any simple formula that we, layman can use as a general guideline?

I think it would be the number of frames your ram can hold so (max lookahead) <= (free ram)/[(# of pixels)x(bits per pixel in raw format - remember YUV12)]

[Dark Shikari]

edit: At episode 2, mbtree is actually worse at times. Added links to episode 2 above. This is compared to an "old" encode, though, before mbtree was introduced, so it uses the old b-adapt, etc. I'll encode again, this time using the new version with --no-mbtree. Check the bottom left and to the (your) right of the woman's rightmost arm. The earlier encode was not perfect in this regions either, but it actually got worse. Also, there's artifacting to the left of the close guy's neckFor the umpteenth time, this is intentional. MB-tree redistributes bits, therefore some parts are guaranteed to get worse.

[Comatose]

Very kind response... Considering the purpose of the thread (you know, TESTING/FEEDBACK), you'd think posting something like that would be met with more than that...

[nakTT]

I think it would be the number of frames your ram can hold so (max lookahead) <= (free ram)/[(# of pixels)x(bits per pixel in raw format - remember YUV12)]
Thanks for the reply bro,

I think I know how to calculate number of pixel. But bits per pixel in raw format? I'm not quite sure. Care to elaborate? (or of there is a simpler guide).

:thanks:

[DarkZell666]

Thanks for the reply bro,

I think I know how to calculate number of pixel. But bits per pixel in raw format? I'm not quite sure. Care to elaborate? (or of there is a simpler guide).

:thanks:

YV12 is 12 bits/pixel (I'm not sure if it's a coincidence or what though :p).

[Selur]

hmmm,..

1080p = 1920*1080 = 2 073 600 pixel
Yv12 = 12bit per pixel
-> 2073600 * 12 = 24 883 200 bit per frame

1 GB free RAM = 1024 * 1024 *1024 Byte = 8*1 073 741 824 bit = 8 589 934 592 bit
8589934592 / 24883200 = 345.210...

-> did I go wrong somewhere or should this mean that a rc-lookahead of 345 per GB of free RAM would be fine?

[DarkZell666]

Thanks for the reply bro,

I think I know how to calculate number of pixel. But bits per pixel in raw format? I'm not quite sure. Care to elaborate? (or of there is a simpler guide).

:thanks:

hmmm,..

1080p = 1920*1080 = 2 073 600 pixel
Yv12 = 12bit per pixel
-> 2073600 * 12 = 24 883 200 bit per frame

1 GB free RAM = 1024 * 1024 *1024 Byte = 8*1 073 741 824 bit = 8 589 934 592 bit
8589934592 / 24883200 = 345.210...

-> did I go wrong somewhere or should this mean that a rc-lookahead of 345 per GB of free RAM would be fine?

It looks right on the paper, except it doesn't account for the overhead-per-frame of x264 itself (without lookahead), influenced partly by the number of b-frames and reference frames IMHO (dunno how many Mbytes/frame x264 uses to store MV's, weights, DCT data, etc. though ...).

[kemuri-_9]

did I go wrong somewhere or should this mean that a rc-lookahead of 345 per GB of free RAM would be fine?
let's not forget that rc-lookahead is capped at 250.
but overall the majority of memory usage consumed by x264 is in the allocation of frames.

frames are relatively complex in their allocation and have a number of factors relating to what parameters were given on the command line (the cpu's cacheline split comes into play here too) in addition to the standard resolution/ mb count factors.

look at the code for x264_frame_new (http://git.videolan.org/gitweb.cgi?p=x264.git;a=blob;f=common/frame.c;hb=HEAD) to get the picture.

by no means is it as simple as what you all have been thinking :rolleyes:

[Selur]

wasn't my thinking the formula was presented by Astrophizz and I knew it was to simple but should at least help to give a general size.
to be on the save side 5 MB free save per frame should be fine unless there's something huge hiding in the code somewhere ;)

[akupenguin]

to be on the save side 5 MB free save per frame should be fine unless there's something huge hiding in the code somewhere ;)
There is. hpel and lowres planes bring the pixel data itself up to (1920+64)*(1088+64)*5.5 = 12MB per frame allocated. And metadata isn't negligible either, especially when it's O(n^2) in max_bframes.

[Chengbin]

Is decoding affected with mbtree? I found seeking much slower with mbtree encodes on my Archos 5.

[Selur]

@akupenguin: huha, that's a huge one ;)

[G_M_C]

Am i right in thinking that features like this mbtree make switching to 64-bits systems/OS etc. more interesting ? Cause i havent switched to 64 bits yet, but mbtree has the potential of breaking the 2 Gb process limit (with high resolutions and/or large values for lookahead).

[wyti]

Yeah you're right, tired it on 1080p with little insane values and crash :p

[LoRd_MuldeR]

Am i right in thinking that features like this mbtree make switching to 64-bits systems/OS etc. more interesting ?

Switching to 64-Bit OS is always a good thing, as it allows to use more than ~3 GB of physical RAM, it allows to run 64-Bit processes where needed and 32-Bit processes will still work fine.

Anyway, I've had no problem with 32-Bit x264 and MB Tree, even with 1080p content. So if you run out of memory, you must be enforcing the problem by using "overkill" settings...

[G_M_C]

Switching to 64-Bit OS is always a good thing, as it allows to use more than ~3 GB of physical RAM, it allows to run 64-Bit processes where needed and 32-Bit processes will still work fine.

Anyway, I've had no problem with 32-Bit x264 and MB Tree, even with 1080p content. So if you run out of memory, you must be enforcing the problem by using "overkill" settings...

It was more a theoretical question for now i suppose; But who knows what happens if i have need for a 4k resolution encoding with GOP size of 60 (60fps) for instance.

[LoRd_MuldeR]

It was more a theoretical question for now i suppose; But who knows what happens if i have need for a 4k resolution encoding with GOP size of 60 (60fps) for instance.

Yeah, but at I think at the time when 4k has established as a standatd (if that ever happens), 32-Bit OS will have almost disappeared...

[burfadel]

Well, by the time 4k video comes out h.265 would be the standard format (its just in the planning stage). Then again, depending on the timeframe encoders would probably be wavelet based and not macroblock based. Just a guess, but GPU would probably suit wavelet encoding much better than macroblock :)

[nurbs]

Then again, depending on the timeframe encoders would probably be wavelet based and not macroblock based.
I remember reading in some magazine how wavelet based encoding would in the coming years be used everywhere when I was in school 10 years ago.

Guess it wasn't feasible both from quality and needed cpu power (at least if dirac is any indication).

[G_M_C]

This is all offtopic, but the fact is that more and more apps seem to want more memory. x264 was very modest in that respect, untill mbtree. I asked this question just for myself, just cause i'm still a little nervous to go over to 64 bit.

My machine works like clockwork, very dependend etc. And i dont want to change that if i dont have too. When i do migrate to 64 bit, i get more stuff i have to think about; Avisynth for instance.

Most Avisynth plugins are only 32 bit. I can pipe 32 bit avisynth output to x264-64, i know. But if I dont have to , why should I ?

But going to 64 bit is inevitable, i just want to time my upgrade as good as i can. I'll be going to Win7 then, but i'll wait unill most apps/drivers/tools i use are ported to win7 or commonplace (clsid's tool for preferring non-MS codecs for instance).

And now back to --mbtree testing, the actual topic of this thread :p

[LoRd_MuldeR]

Go WindowsXP x64-Edition today and you'll notice no difference to good old 32-bit WindowsXP, except that 64-Bit applications suddenly start to work and 4 GB of RAM are accessible.

Driver support is good, except for exotic/archaic hardware maybe. 32-Bit apps that used to work under XP x86 will work under XP x64 flawlessly...

[burfadel]

CPU usage would probably be higher for the same quality (although 'same quality' is very had to judge between the two fundamentally different encoding techiniques) but this can be overcome by using the vast parallelism that modern GPU's provide, which I suspect would benefit a wavelet based codec significantly more than that of a macroblock codec.

[LoRd_MuldeR]

CPU usage would probably be higher for the same quality (although 'same quality' is very had to judge between the two fundamentally different encoding techiniques) but this can be overcome by using the vast parallelism that modern GPU's provide, which I suspect would benefit a wavelet based codec significantly more than that of a macroblock codec.

Can you explain why Wavelet-based compression is so much more suitable for massive parallelization than DCT-based compression?

Remember that when GPGPU is involved we are talking about thousands of ultra-lightweight threads...

[G_M_C]

[...]
exotic hardware
[...]


According to most reports my Xonar HDAV13 Slim works best on XP-32 for now. Driver-support for Win7-64 is somewhat lacking or having difficutlies ;)

So, as long as there is no need to upgrade, i'll wait.

And continuing on topic; I've found a good test-clip for testing the fade-in/fade-out artifacting. I'ts a short clip of the BBC documentary "South Pacific", episode "Vulcanoes", the scene where the camara gets lowered into a cave (actually it's an empty lavatube in Hawaii).

Artifacting (loss of shadowdetail) is worst with adaptive AQ, strength 1 and mbtree enabled. Best seems to be AQ=1 (regular AQ), rest same.

[popper]

It was more a theoretical question for now i suppose; But who knows what happens if i have need for a 4k resolution encoding with GOP size of 60 (60fps) for instance.

has anyone actually tryed MBtree encoding a high speed 2K AND 4K 30 second+ sequence for real yet?, iv not seen any posts to that effect here so far, and were would you even find such a Highspeed 2k/4k dataset thats freeware to try online.

[burfadel]

I believe its something to do with the processing stages of the wavelet algorithm. Also a wavelet codec is significantly different to a DCT (macroblock) codec, and would have to be developed from the ground up (at least to be most effective). In this process, it would make sense to make the processing as parallel as possible to make use of the available parallel threads. Remember CPU's are already progressing towards 6 core (12 threads on the i7), and we're talking probably at least a 10 yr timeframe here unless something unforeseen happens, by that stage a CPU may possibly closer resemble what we see a modern GPU today. Its all guess work, even the most intelligble and knowledgeable people in the field can only see a current trends & brick walls (such as the transistor leakage using current technologies as they become smaller) scenario.

In terms of x64 of 32 bit, if you have no need for 64 bit then its not worth the upgrading at all. I would NEVER recommend upgrading to a 32 bit version of Windows 7, as far as I see it there shouldn't even be a 32-bit version of it! If you have to change over operating systems then definitely get 64 bit! The previous statement that there is no different between XP and XP 64 except for the extra RAM available and 64 bit application availability is false :) XP x64 is based off the Server 2003 code which was much more stable and refined for speed than the XP code ever was. Server 2003 & XP x64 carries the Windows version 5.2, whereas XP is 5.1. I ran XP x64 from when it first came out till I got Vista x64, and the difference was noticeable for system useability, especially when I had multiple programmes open. Even back then I could play GTA San Andreas (we're talking about 2005), record a tv show off HDTV, encode in the background at low priority and share out some media stuff to friends all at the same time. Thats something that can be easily done now (even with today's gta 4, Far cry 2 etc), but back then XP just couldn't handle it. The system had 2gb of RAM so it wasn't a RAM issue, XP just had the tendency to crash, freeze up, or run slow whereas XP x64 had no such trouble.

In should point out in terms of gameplay XP is directx 9 only, you can't compare the speed of Crysis from XP to Vista as the Directx mode is different! Some say XP is faster, and although that may be true running a single, non HDD intensive app, when running multiple applications or tasks XP can fall over. It either crashes, freezes, or runs a particular task much slower than it would otherwise. Vista and Windows 7 (particularly x64 versions) can quite happily run multiple tasks (depending on what they are and what priorities they're set to), so you can encode in the background without it making any noticeable difference in performance as long as the priority is set to low. This can be done on x86 etc as well, but try playing a game while encoding, downloading, recording two tv separate digital tv channels (not just 2 different streams), having several people actually watching stuff off your computer (say, a housemate watching something in the loungeroom whilst leaving music on in the bedroom, and two others doing the same thing), and several other tasks and still be able to play poorly written games like GTA 4 without making any noticeable difference in game performance! (hint - on Vista and even on Windows 7, disable application superfetch as per the instructions in another thread.

If you get a new computer or buy a new copy of Windows, in other words just get the x64 version!

This hasn't been entirely off topic, as the now much higher memory use of x264 better suits operation under a x64 operating system even if it doesn't meet the 32 bit RAM limit! considering if you have at least 4gb of RAM. For the most part, you will probably have to remind yourself you're actually encoding if you run it as a background process :)

[burfadel]

Although not directly related to this thread, this also somewhat relates under memory usage for mbtree.

I can't stress the need enough to disable the application superfetch under Vista and Windows 7. The principle behind superfetch is to load applications automatically in to memory based on past application use, so when you need it, the application loads much faster. This is good in principle on a system where the memory useage is relatively stable (such as a workstation that just uses Word and Excel), but it its fundamentally flawed for other system uses such as video encoding. Basically in an environment where memory useage constantly changes, application superfetch causes continuous disk activity as it loads those applications in to the free memory then clears them to make way for normal memory use. This is just plain stupid!

Disabling application superfetch will still mean your memory may show 0 free under task manager, as it will still use your whole memory as cache, but it keeps session only past application use in the cache, not loading and unloading apps you haven't even used. With macroblock enabled, since the memory use of x264 is higher running it can stuff around with superfetch.

DO NOT disable the superfetch service. That will also disable prefetch and other performance enhancing features. Just disable it by doing the following:
- Run regedit (just type 'regedit' in the start menu search area)
- Open 'Hkey_local_machine), --> 'System' --> 'CurrentControlSet' --> 'Control' --> 'Session manager' --> 'Memory Management' --> 'Prefetchparameters'
- Double click on 'Enablesuperfetch' and change it from 3 to 2 (or to 3 if at any other value)
- Make sure 'Enableprefetcher' is set to 3
- Don't change anything else in the registry
- Close regedit and run 'Services.msc' from the start menu search area
- Ensure 'Superfetch' is set to automatic and is started (just to make sure some other app hasn't changed it)
- Close and reboot

Although you may notice some apps taking slightly longer to load, overall it is much kinder on your system and hdd. Application superfetch can load and unload several gigabytes per hour which in my mind is pointless!

[nurbs]

There is some discussion about wavelets in this thread: http://forum.doom9.org/showthread.php?t=147319
Note the first post on the second page.

[burfadel]

Who knows what could be around in 20 years :) I guess everything it just speculation, I suggested wavelets as a possibility, but the maths behind it could be much more complicated than can currently be done on a modern cpu. For this reasoning, consider the top of the line processor 20 years ago for consumers was the 386, and now consider the difference between a 386 and say, Core i7 965, then extrapolate that for the next 20 years. Doesn't quite work like that I know, but from a conceptual view the reasoning is valid. In a more practical sense, the suggested improvement in processing capability will quite possibly break Moore's Law due to parellism (and AMD's suggested reverse hyperthreading which would benefit applications that aren't so capable of parallelism) and other factors . The processing capability could allow for extrinsically complex mathematical processing which may, depending on designers and programmers capabilities, overcome the problems inherent in a wavelet codec design.

[Guest]

Wavelets is OT here. Further posts about it will be silently deleted.

@burfadel

Read and follow the forum rules, and stop taking threads OT.

[Mr. Brown]

Hello i have been testing mbtree last days but my feeling is that it is quite not ready!
positive:
the overall picture quality (psnr,ssim & own eyes + brain 1.0) is better
in my tests (normal movies) i get 20-30 % higher bitrate without mbtree
negative:
sometimes in dark scenes i see very blocky picture where with --no-mbtree the picture was good
and water sometimes also don't look normal

so i prefer encoding with --no-mbtree until the problems are fixed.

[LoRd_MuldeR]

so i prefer encoding with --no-mbtree until the problems are fixed.

Another option would be raising "qcomp" a bit and lower the effect of MBTree RC for now. Worth a try at least, I think...

[Chengbin]

sometimes in dark scenes i see very blocky picture where with --no-mbtree the picture was good
and water sometimes also don't look normal

That's a problem I'm seeing as well.

Is this fixable with weight-p?

[juGGaKNot]

Is this normal ? trying to test mbtree full power

Bit rate mode : Variable
Bit rate : 4 000 Kbps
Maximum bit rate : 12.0 Mbps
Writing library : x264 core 71 r1210 42d6b17
Encoding settings : cabac=1 / ref=5 / deblock=1:-1:-1 / analyse=0x3:0x113 / me=umh / subme=7 / psy=1 / psy_rd=1.0:0.0 / mixed_ref=1 / me_range=16 / chroma_me=1 / trellis=0 / 8x8dct=1 / cqm=0 / deadzone=21,11 / chroma_qp_offset=-2 / threads=3 / nr=0 / decimate=1 / mbaff=0 / bframes=3 / b_pyramid=0 / b_adapt=2 / b_bias=0 / direct=3 / wpredb=1 / keyint=300 / keyint_min=30 / scenecut=40 / rc_lookahead=50 / rc=2pass / mbtree=1 / bitrate=4000 / ratetol=1.0 / qcomp=0.00 / qpmin=10 / qpmax=45 / qpstep=5 / cplxblur=20.0 / qblur=0.5 / ip_ratio=1.60 / aq=2:1.00

set useles2=--level %mylevel% --bitrate %btratex264% --stats "%mypath%\%mpath%\T1\%mymovie%.stats" --log-file "%mypath%\%mpath%\x264_2pass.log" --keyint %kint% --min-keyint %mint% --fullrange on --ref %myrefs%

start "encode" /b /low /wait "%mypath%\bin\x264.exe" --pass 2 --preset slow %useles2% --deblock -1;-1 --trellis 0 --qcomp 0 --subme 7 --qpmax 45 --qpstep 5 --bframes 3 --ipratio 1.6 --pbratio 1.2 --aq-mode 2 --aq-strength 1.0 --sar 1:1 --aud --output "%mypath%\%mpath%\T1\%mymovie%.264" %myxavs%
echo.

also when will the ratefactor be fixed ?

avis [info]: 1184x666 @ 30.00 fps (1800 frames)
x264 [warning]: width or height not divisible by 16 (1184x666), compression will suffer.
x264 [info]: using SAR=1/1
x264 [info]: using cpu capabilities: MMX2 SSE2Slow
x264 [info]: profile High, level 3.2
x264 [info]: slice I:33 Avg QP:16.43 size: 66387
x264 [info]: slice P:761 Avg QP:19.13 size: 27498
x264 [info]: slice B:1006 Avg QP:25.55 size: 6473
x264 [info]: consecutive B-frames: 9.6% 28.6% 44.3% 17.4%
x264 [info]: mb I I16..4: 22.3% 60.0% 17.7%
x264 [info]: mb P I16..4: 6.4% 16.4% 3.2% P16..4: 32.1% 11.6% 8.2% 0.0% 0.0% skip:22.1%
x264 [info]: mb B I16..4: 0.9% 2.3% 0.5% B16..8: 32.8% 1.5% 1.8% direct: 5.3% skip:54.9% L0:46.3% L1:47.5% BI: 6.2%
x264 [info]: final ratefactor: 11.61
x264 [info]: 8x8 transform intra:62.8% inter:65.4%
x264 [info]: direct mvs spatial:99.8% temporal:0.2%
x264 [info]: coded y,uvDC,uvAC intra:56.3% 59.0% 30.1% inter:15.2% 14.8% 3.1%
x264 [info]: ref P L0 72.4% 12.1% 7.7% 4.2% 3.5%
x264 [info]: ref B L0 83.3% 9.1% 4.8% 2.8%
encoded 1800 frames, 3.26 fps, 3950.72 kb/s

[Selur]

Where's the problem with the ratefactor? (you aimed for a specific bitrate not a ratefactor)
I agree that when aiming for 4000 kb/s and getting 3950.72 kb/s it would be nice it rate control would be a bit stricter,...
Have you tried to set --qpmax to 51 and --qpmin to 1 ? (seems to be reasonable with qcomp 0)

[Dark Shikari]

Where's the problem with the ratefactor? (you aimed for a specific bitrate not a ratefactor)
I agree that when aiming for 4000 kb/s and getting 3950.72 kb/s it would be nice it rate control would be a bit stricter,...
Have you tried to set --qpmax to 51 and --qpmin to 1 ? (seems to be reasonable with qcomp 0)More importantly, setting qcomp=0 is probably a really bad idea... :rolleyes:.

Don't touch what you don't understand.

[juGGaKNot]

Don't touch what you don't understand.

0 for maximum mbtree is bad
default might be bad for psy rd as you said
1.0 will disable mbtree

so when will the default be right ?

and if it is 0 why does the bitrate vary so much ? 12 mb ?

Where's the problem with the ratefactor? (you aimed for a specific bitrate not a ratefactor)

it would be nice to have a valid "quality reference"

ratefactor is 11, quality is good but not great and fades suck.

[Sharc]

positive:
... in my tests (normal movies) i get 20-30 % higher bitrate without mbtree
negative:
... sometimes in dark scenes i see very blocky picture where with --no-mbtree the picture was good

It seems that you are contradicting yourself. It's pointless to compare 2 methods at unequal bitrates/filesizes. Is the blockiness still there with mbtree at 20-30% higher bitrate (to compare with the --no-mbtree result)?

[Dark Shikari]

0 for maximum mbtree is bad
default might be bad for psy rd as you said
1.0 will disable mbtree

so when will the default be right ?

and if it is 0 why does the bitrate vary so much ? 12 mb ?



it would be nice to have a valid "quality reference"

ratefactor is 11, quality is good but not great and fades suck.Again, I am not responsible for what happens when you tweak options that you don't understand.

[juGGaKNot]

Again, I am not responsible for what happens when you tweak options that you don't understand.

the fades actually look better with qcomp 0 than with default but still suck.

[Japhsoncross]

I think we just need some patient until weightp is ready, which is particularly useful for fades.

[juGGaKNot]

ofc but in general qcomp 0 looks better now than default

i will try larger lookahead values than the slow profile gives me.

BTW
what values are safe for lookahead

i want to make a script for ram

if 512 than look=20
if 1024than look=40
if 2048 than look=60
if 4096 and windows x64 than look=80
if 8192 and windows x64 than look=keyint/2

would be safe ?

LE :

something like this :

set head=50
GOTO A0
:A0
if %myram% LEQ 128 GOTO set1
if %myram% GEQ 129 GOTO A1
:A1
if %myram% LEQ 256 GOTO set2
if %myram% GEQ 257 GOTO A2
:A2
if %myram% LEQ 512 GOTO set3
if %myram% GEQ 513 GOTO A3
:A3
if %myram% LEQ 1024 GOTO set4
if %myram% GEQ 1025 GOTO A4
:A4
if %myram% LEQ 1536 GOTO set5
if %myram% GEQ 1537 GOTO A5
:A5
if %myram% LEQ 2048 GOTO set6
if %myram% GEQ 2049 GOTO A6
:A6
if %myram% LEQ 3072 GOTO set7
if %myram% GEQ 3073 GOTO A7
:A7
if %myram% LEQ 4096 GOTO set8
if %myram% GEQ 4097 GOTO A8
:A8
if %myram% LEQ 6144 GOTO set9
if %myram% GEQ 6145 GOTO A9
:A9
if %myram% LEQ 8192 GOTO set10
if %myram% GEQ 8193 GOTO A10
:A10
if %myram% LEQ 12288 GOTO set11
if %myram% GEQ 12289 GOTO set12
:SET1
set head=20
GOTO END
:SET2
set head=30
GOTO END
:SET3
set head=40
GOTO END
:SET4
set head=50
GOTO END
:SET5
set head=60
GOTO END
:SET6
set head=70
GOTO END
:SET7
set head=80
GOTO END
:SET8
set head=90
GOTO END
:SET9
set head=100
GOTO END
:SET10
set head=150
GOTO END
:SET11
set head=200
GOTO END
:SET12
set head=250
GOTO END
:END

lookahead 100 uses ~900mb on my sistem so i think its safe, also people with more than 4 gb have a quad so looks k right ?

[Zarxrax]

BTW
what values are safe for lookahead

i want to make a script for ram


The lookahead isnt the only thing on your system that uses ram. It's not even the only thing in the encoding process that uses ram. This is impossible and pointless.

[LoRd_MuldeR]

The lookahead isnt the only thing on your system that uses ram. It's not even the only thing in the encoding process that uses ram. This is impossible and pointless.

It only matters how many memory is used inside the encoder process, because there is a 2 GB limit for that in 32-Bit process (even on a 64-Bit system and OS).

The global memory isn't that critical, because even if the OS runs out of physical RAM, it can still use the page file. It will be slow, but it won't cause processes to fail.

But if Avisynth input is used, this may consume a lot of memory inside the encoder process. Running Avisynth in its own process (avs2yuv) can relax the problem...

[juGGaKNot]

The lookahead isnt the only thing on your system that uses ram. It's not even the only thing in the encoding process that uses ram. This is impossible and pointless.

what if i set lookahead to 100 and someone has 256mb of ram

on my system it goes to 900mb easy, why would it be pointless to have a script based on ram ?

[kemuri-_9]

on my system it goes to 900mb easy, why would it be pointless to have a script based on ram ?

you obviously skipped this part I already mentioned:
frames are relatively complex in their allocation and have a number of factors relating to what parameters were given on the command line (the cpu's cacheline split comes into play here too) in addition to the standard resolution/ mb count factors.
look at the code for x264_frame_new (http://git.videolan.org/gitweb.cgi?p=x264.git;a=blob;f=common/frame.c;hb=HEAD) to get the picture.

the major factor of a frame's size is the video's resolution. then there are other factors based on the numerous options given on the command line and cpu x264 is running on.
because of this, attempting to script the optimal size of the lookahead will only lead to utter failure.
that is unless you actually do take every factor into account and that would generate one entirely nasty script.

Edit:
not to mention that allocation of frames is not the only place where x264 uses ram, it's just generally the largest.

[lych_necross]

Another option would be raising "qcomp" a bit and lower the effect of MBTree RC for now. Worth a try at least, I think...
qcomp option aside, couldn't you also lower the CRF value a little bit to compensate?

[DarkZell666]

I suppose it doesn't matter if the lookahead value isn't optimal, it's just got to be reasonable, and it should increase with the amount of free ram, nevermind if the optimal value is 25% higher than what the script gives. The pitfall here is the script needs to be as pessimistic as possible so as not to overflow the available RAM for systems where pagefile is disabled, since others won't be affected (and if the script could actually detect if the pagefile enabled and act accordinglyt, it's even better :p). A small optimisation is better than none. We've all checked that out many times over here on d9.

x264 itself can't know in advance if the user will launch other processes that will eat up the free ram at the same time, and this with or without mbtree. It doesn't change anything to the situation.

@juGGaKNot: Keep up with your idea, the problem's not trivial but definately solvable :)

[burfadel]

Disabling the pagefile isn't a good idea IMO for anything less than 6GB of RAM, although for 8GB of RAM I'd probably say its better for now. This is relevant because if the RAM overflows due to the mbtree being 'too high', aka set to a predetermined value based on total RAM, including when you have all the programmes you normally use open, then you do not have enough RAM to disable the pagefile in the first place! This would just add complexity to x264. You'd also have to include resolution in this scenario if you wish it to be dynamic, a mb-tree of 100 at 640x352 would still use less RAM than mb-tree of 50 at 1920x1080, yet you'd be limiting mb-tree to 50 for the lower resolution? Mb-tree size would have to be calculated via expected x264 allocation of RAM, such that x264 ram use is kept to say 1gb on 2gb of system RAM etc, with a minimum mb-tree of the current 40 for example. Afterall, setting it too low isn't ideal. Having it dynamic based on RAM alone is not an optimal solution, at least thats what I think :) Maybe a dynamic range of 40-120 (depending on keyint), if keyint is less than that then of course it is limited.

Just a question regarding mb-tree, does it search to say 100 frames every time, or is the scenecut decision taken in to account so the algorithm terminates early where a keyframe is required? I take it the mb-tree data is reused? I see the memory use is fairly stable for x264, if mb-tree does terminate for that time due to scenecut decision does the RAM allocation for x264 stay as a 'frame pool' size? it seems to otherwise the memory would vary significantly which would degrade performance due to Windows memory & cacheing system?...

[IgorC]

I set lookahead at 300 but avinaptic informs it's 250.
Why is that?

Settings of two passes:
x264.exe --slow-firstpass --aq-mode 2 --rc-lookahead 300 --mbtree --threads 2 --pass 1 --stats "cx264_stat.log" --qcomp 0.6 --bframes 16 --b-adapt 2 --weightb --subme 10 --keyint 300 --ref 16 --trellis 2 --mixed-refs --8x8dct --partitions all --direct auto --bitrate 380 --no-fast-pskip --me tesa --merange 16 --deblock 0:0 --psy-rd 0.2:0.0 -o t.mp4 1.avs --psnr --ssim

x264.exe --aq-mode 2 --mbtree --rc-lookahead 300 --threads 2 --pass 3 --stats "cx264_stat.log" --qcomp 0.6 --bframes 16 --b-adapt 2 --weightb --subme 10 --keyint 300 --ref 16 --trellis 2 --mixed-refs --8x8dct --partitions all --direct auto --bitrate 380 --no-fast-pskip --me tesa --merange 16 --deblock 0:0 --psy-rd 0.2:0.0 -o t2.mp4 1.avs --psnr --ssim

[Dark Shikari]

Max is 250 for now.

[akupenguin]

Just a question regarding mb-tree, does it search to say 100 frames every time, or is the scenecut decision taken in to account so the algorithm terminates early where a keyframe is required?
mbtree terminates at a keyframe in psnr-optimal mode, but not in psy mode, because terminating at a keyframe produces unacceptable fluctuations in quality.

does the RAM allocation for x264 stay as a 'frame pool' size? it seems to otherwise the memory would vary significantly which would degrade performance due to Windows memory & cacheing system?
x264 allocates memory the first time it's needed, and then doesn't free anything until the end of the stream.

[IgorC]

I think all those complains about bad quality on fades have good reason to exist. There are strong artefacts on dark areas wich weren't actually present on 1195+mbtree patch.
I don't know if it's regression (1195+mbtree patch <->1210 ) or it's caused by high value of lookahead ( I raised it from 40 to 250) on Elephant's Dream 360p lossless video (interval 4:07-4:17 looks extremely bad)

I don't have time right now but I will do some test around 1195/1210 revisions and different values of lookahead.

Maybe have I borked something around settings?
x264.exe --slow-firstpass --aq-mode 2 --rc-lookahead 250 --mbtree --threads 2 --pass 1 --stats "cx264_stat.log" --qcomp 0.6 --bframes 16 --b-adapt 2 --weightb --subme 10 --keyint 300 --ref 16 --trellis 2 --mixed-refs --8x8dct --partitions all --direct auto --bitrate 328 --no-fast-pskip --me tesa --merange 16 --deblock 0:0 --psy-rd 0.2:0.0 -o t.mp4 1.avs --psnr --ssim

x264.exe --aq-mode 2 --rc-lookahead 250 --mbtree --threads 2 --pass 3 --stats "cx264_stat.log" --qcomp 0.6 --bframes 16 --b-adapt 2 --weightb --subme 10 --keyint 300 --ref 16 --trellis 2 --mixed-refs --8x8dct --partitions all --direct auto --bitrate 328 --no-fast-pskip --me tesa --merange 16 --deblock 0:0 --psy-rd 0.2:0.0 -o t2.mp4 1.avs --psnr --ssim

[Mr. Brown]

It seems that you are contradicting yourself. It's pointless to compare 2 methods at unequal bitrates/filesizes. Is the blockiness still there with mbtree at 20-30% higher bitrate (to compare with the --no-mbtree result)?

I have tested all modes 2pass & 1pass of course i have noticed the higher bitrate at 1pass crf 22 with --no-mbtree
and 2pass with mbtree and with --no-mbtree at same bitrrate is really a difference in quality in dark scenes.

[Forteen88]

..when Trahald finishes MMCO-related stuff.I'm just wondering, when (approximately) is that finished? Are we talking weeks, month(s), year? I'm not sure if I should wait, or not care (and continue encoding) if it takes too long :)

EDIT: Thanks froggy1 and Underground78

[CruNcher]

--me hex --ref 2 --mixed-refs --trellis 1 --subme 5

Does Mbtree need anything of this to actually work (dependent on it) ? (result bellow answered this question ;) )

CRF = 23 (--mbtree)

x264 [info]: SSIM Mean Y:0.9764634
x264 [info]: PSNR Mean Y:43.242 U:47.222 V:46.687 Avg:44.071 Global:42.283 kb/s:
1234.12

encoded 3634 frames, 24.63 fps, 1234.38 kb/s

CRF = 25 (--no-mbtree)

x264 [info]: SSIM Mean Y:0.9749972
x264 [info]: PSNR Mean Y:43.137 U:46.977 V:46.403 Avg:43.930 Global:41.698 kb/s:
1253.87

encoded 3634 frames, 26.91 fps, 1254.10 kb/s

CRF = 24 (--no-mbtree)

x264 [info]: SSIM Mean Y:0.9776573
x264 [info]: PSNR Mean Y:43.724 U:47.395 V:46.867 Avg:44.491 Global:42.291 kb/s:
1440.21

encoded 3634 frames, 26.38 fps, 1440.44 kb/s

CRF = 23 (--no-mbtree)

x264 [info]: SSIM Mean Y:0.9801169
x264 [info]: PSNR Mean Y:44.315 U:47.829 V:47.341 Avg:45.058 Global:42.895 kb/s:
1656.50

encoded 3634 frames, 25.57 fps, 1656.73 kb/s

CRF = 21 (--mbtree)

x264 [info]: SSIM Mean Y:0.9810367
x264 [info]: PSNR Mean Y:44.344 U:48.054 V:47.617 Avg:45.141 Global:43.430 kb/s:
1628.80

encoded 3634 frames, 23.49 fps, 1629.05 kb/s

Nice saving for the Speed cost :)

[nakTT]

I'm just wondering, when (approximately) is that finished? Are we talking weeks, month(s), year? I'm not sure if I should wait, or not care (and continue encoding) if it takes too long :)
Yeah. I also looking for the same clue.

:thanks:

[microchip8]

Yeah. I also looking for the same clue.

:thanks:

IIRC and correct me if I'm wrong, the GSOC ends on 24th this month and that's when those working on stuff for x264 have to deliver.

[nakTT]

IIRC and correct me if I'm wrong, the GSOC ends on 24th this month and that's when those working on stuff for x264 have to deliver.
Thanks for sharing this valuable info. I guess I will hold my encoding and wait until everything sorted out at the end of this month

:thanks:.

[Underground78]

Timeline for weighted prediction is here (http://wiki.videolan.org/SoC_2009/Weighted_P-frame_Prediction#Timeline) but I think it can take some time before the patch is committed.

[Riki]

What would be a reasonably high setting. Right now Im testing it with 100 but wonder if I can go higher. I have a dual xeon mac pro with 16GB of ram, running x264 in 64bit so it should be able to handle it, I just want to stay in a reasonably high range.

[nurbs]

--preset veryslow uses a lookahead of 80 frames, so I guess you are already in high territory, considering only --preset placebo is slower.

[juGGaKNot]

--preset veryslow uses a lookahead of 80 frames, so I guess you are already in high territory, considering only --preset placebo is slower.

it does not

else if( !strcasecmp( optarg, "veryslow" ) )
{
param->analyse.i_me_method = X264_ME_UMH;
param->analyse.i_subpel_refine = 10;
param->analyse.i_me_range = 24;
param->i_frame_reference = 16;
param->i_bframe_adaptive = X264_B_ADAPT_TRELLIS;
param->analyse.i_direct_mv_pred = X264_DIRECT_PRED_AUTO;
param->analyse.inter |= X264_ANALYSE_PSUB8x8;
param->analyse.i_trellis = 2;
param->i_bframe = 8;
param->rc.i_lookahead = 60;

http://mewiki.project357.com/wiki/X264_Settings

[nurbs]

Oh well, my point still stands.

[akupenguin]

High would be 250. mbtree doesn't have a huge speed cost, defaults are limited mostly for memory.

[Boolsheet]

Does mbtree in intra-only mode actually do something?
Because x264 (JEEB's unpatched 1214 x64 build for me) throws a warning but no error and exits if intra-only and mbtree are on:
"x264 [warning]: specified frame type (3) is not compatible with keyframe interval"

Nothing is written in the output file.
Anyone else getting this with mbtree on and --keyint 1?

If I'm understanding it correctly, mbtree only affects the frames in one GOP. There would be no reason to use mbtree with keyint 1.
I don't know... maybe a check for intra-only could fix this if mbtree is not used for anything else.

[Dark Shikari]

Does mbtree in intra-only mode actually do something?
Because x264 (JEEB's unpatched 1214 x64 build for me) throws a warning but no error and exits if intra-only and mbtree are on:
"x264 [warning]: specified frame type (3) is not compatible with keyframe interval"

Nothing is written in the output file.
Anyone else getting this with mbtree on and --keyint 1?

If I'm understanding it correctly, mbtree only affects the frames in one GOP. There would be no reason to use mbtree with keyint 1.
I don't know... maybe a check for intra-only could fix this if mbtree is not used for anything else.Sounds like a bug; I'll look into it.

Update 1: Breaks on keyint < 3.

Update 2: Fixed locally; thanks for the bug report.

[Riki]

High would be 250. mbtree doesn't have a huge speed cost, defaults are limited mostly for memory.

wow, thats interesting to note!

Thanks for the others who responded too, man this new feature is AMAZING for camtasia training videos, you have NO IDEA! :)

Great work Dark

[G_M_C]

Max is 250 for now.

But it is still clamped to --keyint isnt it (which is logical to me) ?

[wyti]

Yes, but why this limit of 250 ?
On SD contents you can easily go beyond 250 with the 2Go of RAM on 32bits, and with 64bits there would be ne reason to hard limit it.
Is there a erason behind this max value or is this only "idiot proof" ?

[G_M_C]

Yes, but why this limit of 250 ?
On SD contents you can easily go beyond 250 with the 2Go of RAM on 32bits, and with 64bits there would be ne reason to hard limit it.
Is there a erason behind this max value or is this only "idiot proof" ?

What value do you use for --keyint ?

[wyti]

depend on content bu usualy with anime I use about 1000 (seeking don't mind cause with my core i7 it's fast enough^^) and with desktop capture i go up to 5000 (yeah pretty insane) so i think higher mbtree than 250 might help me.

[laserfan]

OK, you're just making this u--wait a minute, what's this new statsfile, and why is it using up all my hard disk space?

Oh, that? (http://i26.tinypic.com/2s6odxt.jpg) Kyahahahahahaha. (http://mirror05.x264.nl/Dark/ahaha.wav)As a casual user of x264, I just did my first encoding with it and the other (relatively new) defaults using JEEB's b1214M and it looks absolutely amazing--thanks to DS and all who contributed to its creation!

I do so few of these that I save all my files & logs e.g. I used (and save) the video.stats file cuz I'm able to see where the I frames are for some post-encoding chapter marks for my blu-ray muxed output. But I see nothing in that monster video.stats.mbtree that is readable-thus-useable. So it's fine just to toss it? And if there DOES turn-out to be some problem with the encoding there won't be any "post your mbtree file" requests will there?! :eek: ;)

[Dark Shikari]

As a casual user of x264, I just did my first encoding with it and the other (relatively new) defaults using JEEB's b1214M and it looks absolutely amazing--thanks to DS and all who contributed to its creation!

I do so few of these that I save all my files & logs e.g. I used (and save) the video.stats file cuz I'm able to see where the I frames are for some post-encoding chapter marks for my blu-ray muxed output. But I see nothing in that monster video.stats.mbtree that is readable-thus-useable. So it's fine just to toss it? And if there DOES turn-out to be some problem with the encoding there won't be any "post your mbtree file" requests will there?! :eek: ;)Correct; the MB-Tree file is just a massive dump of quantizers, with nothing human-readable; technically it contains frametypes too, but that's just so that x264 can sanity check the input to make sure it's not completely junk.

[laserfan]

Correct; the MB-Tree file is just a massive dump of quantizers, with nothing human-readable; technically it contains frametypes too, but that's just so that x264 can sanity check the input to make sure it's not completely junk.Thanks for your reply, which is actually quite a relief to me! :D

[benwaggoner]

Thanks for the others who responded too, man this new feature is AMAZING for camtasia training videos, you have NO IDEA! :)
That's probably the content I'd expect to benefit the most, really, since you can have 100% perfect matches without any noise, plus some perfect motion vector matches with perfectly 2D motion.

[LeonLanford]

sorry I just want to ask a little thing,
will it be compatible with bipyramid in the future?

Thanks

[Fr4nz]

sorry I just want to ask a little thing,
will it be compatible with bipyramid in the future?

Thanks

Yes, but there isn't a timeline yet. Should be soon anyway.

[Snowknight26]

http://i28.tinypic.com/20trs4m.png

Only 5500 frames in. Seems like a large amount of RAM. I wonder if all of it is being released properly. What's the theoretical maximum RAM that mbtree alone could take up with the lookahead set to 250 on a 1920x1080 encode?

[nm]

Only 5500 frames in. Seems like a large amount of RAM. I wonder if all of it is being released properly. What's the theoretical maximum RAM that mbtree alone could take up with the lookahead set to 250 on a 1920x1080 encode?
http://forum.doom9.org/showthread.php?p=1314527#post1314527

250 * 12 MB + metadata > 3000 MB, so your observation seems pretty normal.

[Sagittaire]

I have really spectacular result on OPSNR on my old trailer test: more than 0.2 dB progress. This time x264 is on par with historical DivX3 at half bitrate with really recent build for DivX3 (libavcodec).

[yaous]

sorry for the newbie question..

what's the difference between AQ and MB-Tree?

is MB-Tree something like 'detailed AQ'?

[nurbs]

As I understand it AQ lowers the quantizer in less complex parts of a frame in order to avoid artefacts like blocking on uniform backgrounds while MB-Tree lowers quantizer on macroblocks that are referenced more often by other frames to aid prediction.
That may be oversimplified, but I think it basically covers it.

[TheImperial2004]

A quick question : What will happen if you ran out of RAM while encoding ? Does x264 release RAM after it encode the frame ?

I asked this because , while I encode , RAM reached 0 without crashing and the OS stays responsive ???

[Chengbin]

A quick question : What will happen if you ran out of RAM while encoding ? Does x264 release RAM after it encode the frame ?

I asked this because , while I encode , RAM reached 0 without crashing and the OS stays responsive ???

I think if you run out of RAM, x264 closes by itself.

[LoRd_MuldeR]

A quick question : What will happen if you ran out of RAM while encoding ? Does x264 release RAM after it encode the frame ?

AFAIK x264 will allocate the required memory at the beginning of the encode and not release any memory until the encoding is finished.

I asked this because , while I encode , RAM reached 0 without crashing and the OS stays responsive ???

32-Bit processes can allocate a maximum of 2 GB of RAM, even if more physical RAM is available and even on a 64-Bit OS.

So probably x264 ran into the 2 GB limit, while the OS still had enough RAM available for its own stuff...

[TheImperial2004]

32-Bit processes can allocate a maximum of 2 GB of RAM, even if more physical RAM is available and even on a 64-Bit OS.

So probably x264 ran into the 2 GB limit

I'm not sure becuase I use AviDemux , but if encoding a file with Mb-tree needs 3GB for example , and x264 limited at 2GB . Will that affect encoding quality or compression capability ?

[J_Darnley]

x264 cannot run if it cannot allocate all the memory it needs.

[TheImperial2004]

Then that means , Mb-tree will produce better results with bigger RAM ?

[LoRd_MuldeR]

I'm not sure becuase I use AviDemux , but if encoding a file with Mb-tree needs 3GB for example , and x264 limited at 2GB . Will that affect encoding quality or compression capability ?

Avidemux is only provided as a 32-Bit binary at the moment. So there is a 2 GB limit of the entire Avidemux process, including x264 (which is running inside the Avidemux process).

If x264 cannot allocate the required memory, it will simply fail. However this is unlikely to happen, because even with 1080p content there's enough memory available for reasonable RC Lookahead values.

And since Avidemux doesn't offer options the adjust the RC Lookahead yet, you are protected against insane RC Lookahead values :D

Then that means , Mb-tree will produce better results with bigger RAM ?

No, it doesn't! It means that MB-Tree allocates more memory than the "classical" ratecontrol.

Furthermore the memory required by MB-Tree ratecontrol depends highly on the RC Lookahead value. The higher the RC Lookahead is raised, the more memory will be required by x264!

But as said before, Avidemux uses the default RC Lookahead value, which is 40. So memory consumption remains reasonable ;)

A higher RC Lookahead probably gives better results (while consuming more memory), but even the "placebo" preset only uses a RC Lookahead of 60 (the current maximum is 250).

[TheImperial2004]

So , An RC of 250 will still work with 2GB while encoding 1080p ?

Can you provide a regular Libx264 build with RC of 250 so we can test this ?

:thanks:

[J_Darnley]

So , An RC of 250 will still work with 2GB while encoding 1080p ?

No! Loren has said that the buffered frames for 1920x1080 take 12 MB so with a lookahead of 250 x264 requires 12*250 = 3000 MB. This does not account for other memory that x264 will require.

[LoRd_MuldeR]

So , An RC of 250 will still work with 2GB while encoding 1080p ?

No, it probably wouldn't! Such insane RC Lookahead values would try to allocate an extraordinary amount of RAM and probably run into the 2 GB limit.

I already told you that even x264's "veryslow" and "placebo" presets only use a RC Lookahaed of 60. Why do you want to go even higher ???

Can you provide a regular Libx264 build with RC of 250 so we can test this ?

Nope, because I won't encourage people to use insane settings. My libx264 builds are already patches to increase RC Lookahead slightly over the default ;)

(With SubME=8 it will be raised to 50; with SubME=9 it will be raised to 60. And yes, I know this is a hackish workaround ^^)

[TheImperial2004]

So unless you use x64 version of x264 , you can't use 250 with 1080p (which can't be done with AviDemux AFAIK) ... I guess we'll have to upgrade to 64bit sooner or later :)

[LoRd_MuldeR]

So unless you use x64 version of x264 , you can't use 250 with 1080p

Probably not. But why do want to use such insane values? Do you think higher is always better ???

Anything above RC Lookahead 60 will probably give only a very subtle improvement (if at all), but require more memory for sure.

So don't blindly use overkill settings. Instead try to find the sweet spot!

[TheImperial2004]

You got me there Mulder :)

I like your way of explaining things ;)

[MfA]

Since it's tied to qcomp I assume this only changes quantizers after motion estimation right? If so, any plans to use the knowledge of future blocks in the motion refinement step too?

[Dark Shikari]

Since it's tied to qcomp I assume this only changes quantizers after motion estimation right?Huh? Qcomp didn't change quantizers after motion estimation either.

[TheImperial2004]

Probably not. But why do want to use such insane values? Do you think higher is always better ???

Anything above RC Lookahead 60 will probably give only a very subtle improvement (if at all), but require more memory for sure.

So don't blindly use overkill settings. Instead try to find the sweet spot!
I also noticed that when I use 2-Pass mode , I got a .MBTREE file and my free RAM stays at a rasonable value . But with CRF I don't get such a file and RAM reaches Zero in an instant ... Why can't we make Libx264 generate this file regardless of the encoding mode ?

[MfA]

Let me rephrase that, when you do RDO during subpixel refinement does it take into account the errors caused in dependent blocks for different motion vectors? (I guess you'd have to re-encode dependent blocks with the settings from the previous pass but with a changed MC residual.)

[kemuri-_9]

I also noticed that when I use 2-Pass mode , I got a .MBTREE file and my free RAM stays at a rasonable value . But with CRF I don't get such a file and RAM reaches Zero in an instant ... Why can't we make Libx264 generate this file regardless of the encoding mode ?

the file is generated during the 1st pass which is then used in the 2nd pass.

you can also generate it with crf by specifying --pass 1, but unless you do a 2nd pass, you won't see the benefit of this file

[Dark Shikari]

Let me rephrase that, when you do RDO during subpixel refinement does it take into account the errors caused in dependent blocks for different motion vectors? (I guess you'd have to re-encode dependent blocks with the settings from the previous pass but with a changed MC residual.)No, x264 doesn't have a macroblock-level lookahead.

[ajp_anton]

I also noticed that when I use 2-Pass mode , I got a .MBTREE file and my free RAM stays at a rasonable value . But with CRF I don't get such a file and RAM reaches Zero in an instant ... Why can't we make Libx264 generate this file regardless of the encoding mode ?Have you tried --slow-firstpass ?

[TheImperial2004]

the file is generated during the 1st pass which is then used in the 2nd pass.

you can also generate it with crf by specifying --pass 1, but unless you do a 2nd pass, you won't see the benefit of this file
You can't specify any CLI commands in AviDemux ... :)

[J_Darnley]

Have you tried --slow-firstpass ?
This is is not a library option anyway.

[Snowknight26]

It is.
http://git.videolan.org/?p=x264.git;a=blob;f=x264.c

178 H0( " --slow-firstpass Don't use faster settings with --pass 1\n" );

[kemuri-_9]

It is.
http://git.videolan.org/?p=x264.git;a=blob;f=x264.c

no it is not: x264.c is not a part of libx264

[J_Darnley]

It is.
http://git.videolan.org/?p=x264.git;a=blob;f=x264.c
From x264.c

378 static struct option long_options[] =
379 {
...
386 { "slow-firstpass", no_argument, NULL, OPT_SLOWFIRSTPASS },
...
817 case OPT_SLOWFIRSTPASS:
818 b_turbo = 0;
819 break;
...
857 /* Set faster options in case of turbo firstpass. */
858 if( b_turbo && b_pass1 )
859 {
860 param->i_frame_reference = 1;
861 param->analyse.b_transform_8x8 = 0;
862 param->analyse.inter = 0;
863 param->analyse.i_me_method = X264_ME_DIA;
864 param->analyse.i_subpel_refine = X264_MIN( 2, param->analyse.i_subpel_refine );
865 param->analyse.i_trellis = 0;
866 }


[EDIT] Thanks kemuri

[EDIT2]Also:

diff --git a/x264.h b/x264.h
index 26ac421..e61040e 100644 (file)
--- a/x264.h
+++ b/x264.h
@@ -35,7 +35,7 @@

#include <stdarg.h>

-#define X264_BUILD 67
+#define X264_BUILD 68

/* x264_t:
* opaque handler for encoder */
@@ -83,7 +83,6 @@ typedef struct x264_t x264_t;
#define X264_CQM_FLAT 0
#define X264_CQM_JVT 1
#define X264_CQM_CUSTOM 2
-#define X264_RC_NONE -1
#define X264_RC_CQP 0
#define X264_RC_CRF 1
#define X264_RC_ABR 2
@@ -103,8 +102,7 @@ static const char * const x264_transfer_names[] = { "", "bt709", "undef", "", "b
static const char * const x264_colmatrix_names[] = { "GBR", "bt709", "undef", "", "fcc", "bt470bg", "smpte170m", "smpte240m", "YCgCo", 0 };

/* Colorspace type
- * legacy only; nothing other than I420 is really supported.
- */
+ * legacy only; nothing other than I420 is really supported. */
#define X264_CSP_MASK 0x00ff /* */
#define X264_CSP_NONE 0x0000 /* Invalid mode */
#define X264_CSP_I420 0x0001 /* yuv 4:2:0 planar */
@@ -118,8 +116,7 @@ static const char * const x264_colmatrix_names[] = { "GBR", "bt709", "undef", ""
#define X264_CSP_MAX 0x0009 /* end of list */
#define X264_CSP_VFLIP 0x1000 /* */

-/* Slice type
- */
+/* Slice type */
#define X264_TYPE_AUTO 0x0000 /* Let x264 choose the right type */
#define X264_TYPE_IDR 0x0001
#define X264_TYPE_I 0x0002
@@ -129,14 +126,16 @@ static const char * const x264_colmatrix_names[] = { "GBR", "bt709", "undef", ""
#define IS_X264_TYPE_I(x) ((x)==X264_TYPE_I || (x)==X264_TYPE_IDR)
#define IS_X264_TYPE_B(x) ((x)==X264_TYPE_B || (x)==X264_TYPE_BREF)

-/* Log level
- */
+/* Log level */
#define X264_LOG_NONE (-1)
#define X264_LOG_ERROR 0
#define X264_LOG_WARNING 1
#define X264_LOG_INFO 2
#define X264_LOG_DEBUG 3

+/* Threading */
+#define X264_THREADS_AUTO 0 /* Automatically select optimal number of threads */
+
/* Zones: override ratecontrol or other options for specific sections of the video.
* See x264_encoder_reconfig() for which options can be changed.
* If zones overlap, whichever comes later in the list takes precedence. */

No mention of any related change

[Snowknight26]

Whoops, libx264. Shows you how well my selective reading works. :o

[LoRd_MuldeR]

Why can't we make Libx264 generate this file regardless of the encoding mode ?

Because the stats file is for 2-Pass mode and there's no point in having a stats file for Single-Pass modes (CQP, CRF, 1-Pass ABR).

[kemuri-_9]

Because the stats file is for 2-Pass mode and there's no point in having a stats file for Single-Pass modes (CQP, CRF, 1-Pass ABR).

reasons can be left to the person and their whims,
that is why every mode can actually create 2pass .stat files... even lossless mode

it's up to the application using libx264 to allow this in the end or not, so.... idk if avidemux prevents this or not.

[LoRd_MuldeR]

reasons can be left to the person and their whims,
that is why every mode can actually create 2pass .stat files... even lossless mode

it's up to the application using libx264 to allow this in the end or not, so.... idk if avidemux prevents this or not.

I'm aware that not only ABR mode can create a stats file. For example CRF can be used the create a stats file and then that stats file can be used for the second pass.

Still my point stands: If you aren't going re-use the stats file for a second pass, it would be useless to create a stats file (except for debugging purposes, maybe).

[detmek]

Hi. This is my first post here, but I have been frequently visiting this forum for over a year.
About stats file in one pass mode. Leiming's GUI has an option conditional double pass that will run first pass CRF or ABR and create stats file. If output file size is in expected margins, it's OK. If it's not, it will run another pass and will use stats from the first pass.
P.S. Sorry, english is not my primary language.

[J_Darnley]

Yes, you can do that. I believe Dark Shikari has said that is what Facebook does. If you have a question, please end with a question mark.

[Dark Shikari]

Yes, you can do that. I believe Dark Shikari has said that is what Facebook does.Correct.

[TheImperial2004]

D.S. can you tell us what new features are coming to x264 ? I'm following your diary (http://x264dev.multimedia.cx/) but nothing mentioned there ...

[detmek]

Yes, my post is confusing. It's not a question. I just wanted to comment conversation between kemuri-_9 and LoRd_Mulder about mbtree stats in one pass mode. It can be usefull feature and can be easily implemented in MeGUI but I don't know about Avidemux, especially through libx264.
Btw, is mbtree less efficient in 2-pass mode? I tried it today and I got SSIM 0.9885540 without mbtree and 0.9888014 with mbtree. Test clip is Diablo 3 trailer 720p.

[Disabled]

I'm encoding videos in CRF mode but append the --pass 1 option to get a stats file. I do it, because later I delete the encode and always keep the source. But if I need an encode again, I just do a second pass based on the stats file I saved. Now with the mbtree file being quite large, I wonder if it would be possible to delete the mbtree file and only keep the stats file? So that on a second pass with mbtree enabled but no file found, it re-does the mb-tree algorithm to save the space.
I tried it, but it doesn't work. Would it be complicated to implement it, or are there some technical limitations to what I want?

[benwaggoner]

I'm encoding videos in CRF mode but append the --pass 1 option to get a stats file. I do it, because later I delete the encode and always keep the source. But if I need an encode again, I just do a second pass based on the stats file I saved. Now with the mbtree file being quite large, I wonder if it would be possible to delete the mbtree file and only keep the stats file? So that on a second pass with mbtree enabled but no file found, it re-does the mb-tree algorithm to save the space.
I tried it, but it doesn't work. Would it be complicated to implement it, or are there some technical limitations to what I want?
Have you tried applying lossless compression to the mbtree file? Perhaps 7zip et al could shrink it down to something more easily storable. I'd expect (without actually haven't looked at it in the slightest :)) that it'd have a fair amount of redundency and hence compressibility.

[LoRd_MuldeR]

I'd expect (without actually haven't looked at it in the slightest :)) that it'd have a fair amount of redundency and hence compressibility.

Not as much as I would have expected...

[Dark Shikari]

Have you tried applying lossless compression to the mbtree file? Perhaps 7zip et al could shrink it down to something more easily storable. I'd expect (without actually haven't looked at it in the slightest :)) that it'd have a fair amount of redundency and hence compressibility.I explicitly designed it so that it wouldn't. It shouldn't be that compressible.

[juGGaKNot]

1232 stall about 2-3 minutes between passes, it this because it does the first 60 frames fast ? ( lookahead ) ?

also mbtree works in the first pass right ? normal fast 1pass can cause quality problems ?

just asking, i have not seen any.

[gizzin]

If they can beat x264 in visual quality on ordinary test clips without postprocessing, I'll eat my hat.


lol...

[moviefan]

If I experience some very very rare banding in dark scences (most probably because the source is a little weak there), Would lowering qcomp from now 0.75 help? As far as I know, qcomp balances between CBR and CQ and mbtree takes more bits from complex scences for redistribution at lower qcomp values. Except for debanding a little, could this help? Is the default 0.6 for qcomp generally recommended for clean 1080p sources?

[Dark Shikari]

If I experience some very very rare banding in dark scences (most probably because the source is a little weak there), Would lowering qcomp from now 0.75 help? As far as I know, qcomp balances between CBR and CQ and mbtree takes more bits from complex scences for redistribution at lower qcomp values. Except for debanding a little, could this help? Is the default 0.6 for qcomp generally recommended for clean 1080p sources?You can raise adaptive quantization; lowering qcomp probably won't help (at least with MB-tree on).

[xyloy]

Awesome news!

In fact, I did already a test of mb-tree with my DVD of Terminator 2 Extreme Edition, but I did not know how to turn it off (nor what it was) before reading this thread.

So I did a little mb-tree/no mb-tree versus test (x86 x264 rev.1251 from x264.nl) with the intro of Day of the Tentacle (captured with DOSBox ZMBV codec and with PCM for audio), just for the fun of it. :P

Source :
Day of the Tentacle
DOSBox ZMBV (8 bits per pixel) capture
320x200 pixels, 70 FPS

AviSynth script:

DirectShowSource(".\tentacl_000.avi")
ConverttoYV12()


It was a two-pass 55 kbps encode (I blindly tried first with 10 as a nominal bitrate, but x264 said something like "failed 2nd pass curve, minimal bitrate should be around 52 kbps").

first, here are the command line results :
1st pass without mb-tree

D:\>"c:\Program Files (x86)\megui\tools\x264\x264.exe" --pass 1 --bitrate 55 --
stats "D:\DOTT(no-mbtree).stats" --keyint 700 --min-keyint 70 --ref 16 --no-fast
-pskip --b-adapt 2 --b-pyramid --direct auto --subme 9 --trellis 2 --partitions
all --me umh --merange 32 --thread-input --no-dct-decimate --output NUL "D:\DSS.
avs" --fullrange on --no-mbtree
avis [info]: 320x200 @ 70.09 fps (52217 frames)
x264 [warning]: width or height not divisible by 16 (320x200), compression will
suffer.
x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 Cache64
x264 [info]: profile Main, level 2.1
x264 [info]: frame I:96 Avg QP:37.00 size: 2888
x264 [info]: frame P:50094 Avg QP:39.95 size: 100
x264 [info]: frame B:2027 Avg QP:47.36 size: 85
x264 [info]: consecutive B-frames: 93.9% 2.1% 1.7% 2.3%
x264 [info]: mb I I16..4: 71.9% 0.0% 28.1%
x264 [info]: mb P I16..4: 19.7% 0.0% 0.0% P16..4: 2.3% 0.0% 0.0% 0.0% 0
.0% skip:78.0%
x264 [info]: mb B I16..4: 0.2% 0.0% 0.0% B16..8: 4.5% 0.0% 0.0% direct:
7.3% skip:88.0% L0:35.3% L1:64.4% BI: 0.3%
x264 [info]: final ratefactor: 51.90
x264 [info]: direct mvs spatial:99.7% temporal:0.3%
x264 [info]: coded y,uvDC,uvAC intra:1.3% 14.1% 6.6% inter:0.7% 1.5% 0.8%
x264 [info]: kb/s:58.8

encoded 52217 frames, 208.49 fps, 58.89 kb/s

2nd pass without MB-tree

D:\>"c:\Program Files (x86)\megui\tools\x264\x264.exe" --pass 2 --bitrate 55 --
stats "D:\DOTT(no-mbtree).stats" --keyint 700 --min-keyint 70 --ref 16 --no-fast
-pskip --b-adapt 2 --b-pyramid --direct auto --subme 9 --trellis 2 --partitions
all --me umh --merange 32 --thread-input --no-dct-decimate --output "D:\DOTT(no-
mbtree).mp4" "D:\DSS.avs" --fullrange on --no-mbtree
avis [info]: 320x200 @ 70.09 fps (52217 frames)
x264 [warning]: width or height not divisible by 16 (320x200), compression will
suffer.
x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 Cache64
x264 [info]: profile High, level 2.1
mp4 [info]: initial delay 285362 (scale 10000000)
x264 [info]: frame I:96 Avg QP:29.66 size: 5023
x264 [info]: frame P:50094 Avg QP:32.06 size: 78
x264 [info]: frame B:2027 Avg QP:36.53 size: 69
x264 [info]: consecutive B-frames: 93.9% 2.1% 1.7% 2.3%
x264 [info]: mb I I16..4: 31.9% 30.3% 37.8%
x264 [info]: mb P I16..4: 0.6% 0.3% 0.0% P16..4: 2.1% 0.4% 2.2% 0.0% 0
.0% skip:94.3%
x264 [info]: mb B I16..4: 0.1% 0.0% 0.0% B16..8: 9.0% 0.6% 0.2% direct:
0.3% skip:89.7% L0:42.2% L1:57.6% BI: 0.2%
x264 [info]: 8x8 transform intra:34.8% inter:41.4%
x264 [info]: direct mvs spatial:96.5% temporal:3.5%
x264 [info]: coded y,uvDC,uvAC intra:13.0% 27.2% 18.1% inter:0.4% 1.1% 0.7%
x264 [info]: ref P L0 77.0% 3.9% 2.3% 0.8% 0.6% 1.2% 1.9% 2.8% 1.7% 2.
0% 1.1% 0.6% 0.7% 0.8% 1.6% 1.0%
x264 [info]: ref B L0 90.0% 2.0% 0.8% 0.4% 0.2% 1.5% 1.7% 1.2% 0.7% 0.
4% 0.2% 0.2% 0.2% 0.2% 0.2%
x264 [info]: ref B L1 97.3% 2.7%
x264 [info]: kb/s:48.7

encoded 52217 frames, 46.57 fps, 48.74 kb/s

1st pass with MB-tree

D:\>"c:\Program Files (x86)\megui\tools\x264\x264.exe" --pass 1 --bitrate 55 --
stats "D:\DOTT(mbtree).stats" --keyint 700 --min-keyint 70 --ref 16 --no-fast-ps
kip --b-adapt 2 --b-pyramid --direct auto --subme 9 --trellis 2 --partitions all
--me umh --merange 32 --thread-input --no-dct-decimate --output NUL "D:\DSS.avs
" --fullrange on --rc-lookahead 250
avis [info]: 320x200 @ 70.09 fps (52217 frames)
x264 [warning]: width or height not divisible by 16 (320x200), compression will
suffer.
x264 [warning]: b-pyramid + mb-tree is not supported
x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 Cache64
x264 [info]: profile Main, level 2.1
x264 [info]: frame I:96 Avg QP:30.97 size: 4086
x264 [info]: frame P:50087 Avg QP:37.59 size: 96
x264 [info]: frame B:2034 Avg QP:48.12 size: 72
x264 [info]: consecutive B-frames: 93.9% 2.1% 1.7% 2.3%
x264 [info]: mb I I16..4: 54.0% 0.0% 46.0%
x264 [info]: mb P I16..4: 17.4% 0.0% 0.0% P16..4: 2.1% 0.0% 0.0% 0.0% 0
.0% skip:80.5%
x264 [info]: mb B I16..4: 0.2% 0.0% 0.0% B16..8: 5.2% 0.0% 0.0% direct:
5.6% skip:88.9% L0:34.7% L1:65.1% BI: 0.2%
x264 [info]: final ratefactor: 41.96
x264 [info]: direct mvs spatial:98.0% temporal:2.0%
x264 [info]: coded y,uvDC,uvAC intra:1.9% 10.1% 5.3% inter:0.7% 1.3% 0.8%
x264 [info]: kb/s:57.3

encoded 52217 frames, 93.35 fps, 57.44 kb/s

2nd pass with MB-tree

D:\>"c:\Program Files (x86)\megui\tools\x264\x264.exe" --pass 2 --bitrate 55 --
stats "D:\DOTT(mbtree).stats" --keyint 700 --min-keyint 70 --ref 16 --no-fast-ps
kip --b-adapt 2 --b-pyramid --direct auto --subme 9 --trellis 2 --partitions all
--me umh --merange 32 --thread-input --no-dct-decimate --output "D:\DOTT(mbtree
).mp4" "D:\DSS.avs" --fullrange on --rc-lookahead 250
avis [info]: 320x200 @ 70.09 fps (52217 frames)
x264 [warning]: width or height not divisible by 16 (320x200), compression will
suffer.
x264 [warning]: b-pyramid + mb-tree is not supported
x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 Cache64
x264 [info]: profile High, level 2.1
mp4 [info]: initial delay 285362 (scale 10000000)
x264 [info]: frame I:96 Avg QP:26.35 size: 6781
x264 [info]: frame P:50087 Avg QP:34.74 size: 86
x264 [info]: frame B:2034 Avg QP:40.48 size: 53
x264 [info]: consecutive B-frames: 93.9% 2.1% 1.7% 2.3%
x264 [info]: mb I I16..4: 29.3% 24.4% 46.3%
x264 [info]: mb P I16..4: 0.7% 0.5% 0.0% P16..4: 2.8% 0.4% 1.3% 0.0% 0
.0% skip:94.2%
x264 [info]: mb B I16..4: 0.1% 0.0% 0.0% B16..8: 8.9% 0.5% 0.1% direct:
0.1% skip:90.2% L0:46.0% L1:53.9% BI: 0.1%
x264 [info]: 8x8 transform intra:40.6% inter:43.9%
x264 [info]: direct mvs spatial:94.1% temporal:5.9%
x264 [info]: coded y,uvDC,uvAC intra:11.3% 20.5% 15.1% inter:0.4% 1.4% 0.7%
x264 [info]: ref P L0 63.8% 6.3% 4.3% 1.6% 1.5% 2.2% 2.6% 3.5% 2.2% 2.
7% 1.6% 1.2% 1.3% 1.3% 2.4% 1.4%
x264 [info]: ref B L0 90.3% 1.2% 0.5% 1.3% 1.5% 1.6% 0.2% 1.4% 0.8% 0.
5% 0.1% 0.4% 0.1% 0.1% 0.2%
x264 [info]: kb/s:54.3

encoded 52217 frames, 46.03 fps, 54.31 kb/s

Screenshots :

Source :
http://img411.imageshack.us/img411/6307/dottsource.png

two passes encode without mb-tree :
http://img21.imageshack.us/img21/2165/dottnombtree.png

two passes encode with mb-tree and --rc-lookahead 250 :
http://img411.imageshack.us/img411/9574/dottmbtree.png
(I could have posted other screenshots, but you can see the difference right away : there's only need to look at the actions verbs' text, almost entirely unreadable without mb-tree. There's also a huge difference on the action phrase, on the old clock, and on Bernard's arms)

DOSBox' video capture used as source (http://www.megaupload.com/?d=1I92DMSY) (13.92 MB)
No mb-tree video (http://www.megaupload.com/?d=IZR6THTA) (8.83 MB)
mb-tree video (http://www.megaupload.com/?d=72H92WHO) (9.33 MB)
(all with sound)

In short : mb-tree is very impressive! :)

[MatLz]

Hi xyloy,
Your encode with mbtree has more bitrate than the without mbtree encode. Can you do a try for an approximately equal bitrate?

[poisondeathray]

It is impressive... but the mb-tree version has 11.5% more bitrate.
EDIT: MatLz beat me ... :)

[LoRd_MuldeR]

And why is the video encoded at 70fps? The motion in that video isn't anywhere near 70fps, it's more like 15fps ;)

[xyloy]

DOSBox encodes it at 70 FPS. I dunno why, but every capture I do with DOSBox' own capture capabilities is at 70 FPS. That's probably why I can't capture 3D games without terrible lags (and DOSBox can use only one CPU..).

Anyway, in short : I don't know why, and it seems to be hardcoded that way DOSBox' code. :confused:



Edit:
Hi xyloy,
Your encode with mbtree has more bitrate than the without mbtree encode. Can you do a try for an approximately equal bitrate?
I'll try with 200 kbps. May be I'm too close to the "minimum bitrate" that x264 guessed for this source (x264's 2nd pass bitrate curve warning message I reported).

[LoRd_MuldeR]

DOSBox encodes it at 70 FPS. I dunno why, but every capture I do with DOSBox' own capture capabilities is at 70 FPS. That's probably why I can't capture 3D games without terrible lags (and DOSBox can use only one CPU..).

Anyway, in short : I don't know why, and it seems to be hardcoded that way DOSBox' code. :confused:



It's probably related to the screen refresh-rate that DOSBox simulates. However you should down-sample the video before encoding it!
http://avisynth.org/mediawiki/ChangeFPS

I guess that 10 fps should be sufficient to retain all motion in that clip. Which means that currently 6/7 of all frames are useless.
If there's no motion between two consecutive frames, these "dummy" frames usually shouldn't eat too many bits.
But on the other hand at those ultra-low bitrates coding those frames, even if they contain no data, may cause a significant overhead...

[MatLz]

[QUOTE=xyloy;1325025]I'll try with 200 kbps.[QUOTE]

But maybe we will not see the difference with a 'high' bitrate like 200kbps.
Just increase the bitrate to 60 for the nombtree encode, maybe the result will be near as 54kbps

[xyloy]

Ok, here it is.

AviSynth script:

DirectShowSource(".\tentacle_000.avi")
ConvertToYV12()


command line results :

D:\>"c:\Program Files (x86)\megui\tools\x264\x264.exe" --pass 1 --bitrate 60 --
stats "D:\DOTT(60kbps-no-mbtree).stats" --keyint 700 --min-keyint 70 --ref 16 --
no-fast-pskip --b-adapt 2 --b-pyramid --direct auto --subme 9 --trellis 2 --part
itions all --me umh --merange 32 --thread-input --no-dct-decimate --output NUL "
D:\DSS.avs" --fullrange on --no-mbtree
avis [info]: 320x200 @ 70.09 fps (52218 frames)
x264 [warning]: width or height not divisible by 16 (320x200), compression will
suffer.
x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 Cache64
x264 [info]: profile Main, level 2.1
x264 [info]: frame I:96 Avg QP:36.04 size: 3332
x264 [info]: frame P:50095 Avg QP:38.86 size: 106
x264 [info]: frame B:2027 Avg QP:46.39 size: 87
x264 [info]: consecutive B-frames: 93.9% 2.1% 1.7% 2.3%
x264 [info]: mb I I16..4: 69.9% 0.0% 30.1%
x264 [info]: mb P I16..4: 18.5% 0.0% 0.0% P16..4: 2.6% 0.0% 0.0% 0.0% 0
.0% skip:78.9%
x264 [info]: mb B I16..4: 0.2% 0.0% 0.0% B16..8: 4.7% 0.0% 0.0% direct:
6.9% skip:88.2% L0:35.2% L1:64.3% BI: 0.5%
x264 [info]: final ratefactor: 50.43
x264 [info]: direct mvs spatial:99.7% temporal:0.3%
x264 [info]: coded y,uvDC,uvAC intra:1.4% 14.1% 6.7% inter:0.9% 1.6% 0.9%
x264 [info]: kb/s:62.1

encoded 52218 frames, 208.90 fps, 62.25 kb/s

D:\>"c:\Program Files (x86)\megui\tools\x264\x264.exe" --pass 2 --bitrate 60 --
stats "D:\DOTT(60kbps-no-mbtree).stats" --keyint 700 --min-keyint 70 --ref 16 --
no-fast-pskip --b-adapt 2 --b-pyramid --direct auto --subme 9 --trellis 2 --part
itions all --me umh --merange 32 --thread-input --no-dct-decimate --output "D:\D
OTT(60kbps-no-mbtree).mp4" "D:\DSS.avs" --fullrange on --no-mbtree
avis [info]: 320x200 @ 70.09 fps (52218 frames)
x264 [warning]: width or height not divisible by 16 (320x200), compression will
suffer.
x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 Cache64
x264 [info]: profile High, level 2.1
mp4 [info]: initial delay 285362 (scale 10000000)
x264 [info]: frame I:96 Avg QP:27.30 size: 6234
x264 [info]: frame P:50095 Avg QP:29.80 size: 86
x264 [info]: frame B:2027 Avg QP:35.05 size: 75
x264 [info]: consecutive B-frames: 93.9% 2.1% 1.7% 2.3%
x264 [info]: mb I I16..4: 33.9% 21.9% 44.2%
x264 [info]: mb P I16..4: 0.6% 0.3% 0.1% P16..4: 2.5% 0.4% 1.2% 0.0% 0
.0% skip:94.9%
x264 [info]: mb B I16..4: 0.1% 0.1% 0.0% B16..8: 9.1% 0.6% 0.2% direct:
0.3% skip:89.6% L0:40.8% L1:59.0% BI: 0.3%
x264 [info]: 8x8 transform intra:29.3% inter:35.6%
x264 [info]: direct mvs spatial:96.5% temporal:3.5%
x264 [info]: coded y,uvDC,uvAC intra:13.8% 26.8% 19.0% inter:0.4% 1.5% 0.8%
x264 [info]: ref P L0 72.4% 4.8% 2.9% 1.1% 0.7% 1.4% 2.2% 3.4% 2.0% 2.
3% 1.2% 0.7% 0.8% 0.9% 1.9% 1.2%
x264 [info]: ref B L0 89.2% 2.3% 0.7% 0.5% 0.2% 1.6% 2.0% 1.2% 0.8% 0.
4% 0.2% 0.2% 0.3% 0.3% 0.3%
x264 [info]: ref B L1 97.3% 2.7%
x264 [info]: kb/s:54.5

encoded 52218 frames, 45.05 fps, 54.52 kb/s

D:\>"c:\Program Files (x86)\megui\tools\x264\x264.exe" --pass 1 --bitrate 55 --
stats "D:\DOTT(55kbps-mbtree).stats" --keyint 700 --min-keyint 70 --ref 16 --no-
fast-pskip --b-adapt 2 --b-pyramid --direct auto --subme 9 --trellis 2 --partiti
ons all --me umh --merange 32 --thread-input --no-dct-decimate --output NUL "D:\
DSS.avs" --fullrange on --rc-lookahead 250
avis [info]: 320x200 @ 70.09 fps (52218 frames)
x264 [warning]: width or height not divisible by 16 (320x200), compression will
suffer.
x264 [warning]: b-pyramid + mb-tree is not supported
x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 Cache64
x264 [info]: profile Main, level 2.1
x264 [info]: frame I:96 Avg QP:30.97 size: 4086
x264 [info]: frame P:50088 Avg QP:37.59 size: 96
x264 [info]: frame B:2034 Avg QP:48.12 size: 72
x264 [info]: consecutive B-frames: 93.9% 2.1% 1.7% 2.3%
x264 [info]: mb I I16..4: 54.0% 0.0% 46.0%
x264 [info]: mb P I16..4: 17.4% 0.0% 0.0% P16..4: 2.1% 0.0% 0.0% 0.0% 0
.0% skip:80.5%
x264 [info]: mb B I16..4: 0.2% 0.0% 0.0% B16..8: 5.2% 0.0% 0.0% direct:
5.6% skip:88.9% L0:34.7% L1:65.1% BI: 0.2%
x264 [info]: final ratefactor: 41.96
x264 [info]: direct mvs spatial:98.0% temporal:2.0%
x264 [info]: coded y,uvDC,uvAC intra:1.9% 10.1% 5.3% inter:0.7% 1.3% 0.8%
x264 [info]: kb/s:57.3

encoded 52218 frames, 94.71 fps, 57.44 kb/s

D:\>"c:\Program Files (x86)\megui\tools\x264\x264.exe" --pass 2 --bitrate 55 --
stats "D:\DOTT(55kbps-mbtree).stats" --keyint 700 --min-keyint 70 --ref 16 --no-
fast-pskip --b-adapt 2 --b-pyramid --direct auto --subme 9 --trellis 2 --partiti
ons all --me umh --merange 32 --thread-input --no-dct-decimate --output "D:\DOTT
(55kbps-mbtree).mp4" "D:\DSS.avs" --fullrange on --rc-lookahead 250
avis [info]: 320x200 @ 70.09 fps (52218 frames)
x264 [warning]: width or height not divisible by 16 (320x200), compression will
suffer.
x264 [warning]: b-pyramid + mb-tree is not supported
x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 Cache64
x264 [info]: profile High, level 2.1
mp4 [info]: initial delay 285362 (scale 10000000)
x264 [info]: frame I:96 Avg QP:27.02 size: 6432
x264 [info]: frame P:50088 Avg QP:35.69 size: 86
x264 [info]: frame B:2034 Avg QP:40.84 size: 53
x264 [info]: consecutive B-frames: 93.9% 2.1% 1.7% 2.3%
x264 [info]: mb I I16..4: 29.3% 25.5% 45.2%
x264 [info]: mb P I16..4: 0.8% 0.5% 0.0% P16..4: 2.8% 0.4% 1.8% 0.0% 0
.0% skip:93.6%
x264 [info]: mb B I16..4: 0.1% 0.0% 0.0% B16..8: 9.2% 0.5% 0.1% direct:
0.1% skip:90.0% L0:46.5% L1:53.4% BI: 0.1%
x264 [info]: 8x8 transform intra:36.6% inter:44.6%
x264 [info]: direct mvs spatial:94.1% temporal:5.9%
x264 [info]: coded y,uvDC,uvAC intra:10.3% 19.4% 13.7% inter:0.4% 1.4% 0.7%
x264 [info]: ref P L0 66.6% 6.2% 4.3% 1.5% 1.4% 1.9% 2.3% 3.1% 2.0% 2.
5% 1.4% 1.1% 1.2% 1.2% 2.1% 1.2%
x264 [info]: ref B L0 89.5% 1.3% 0.6% 1.1% 1.9% 2.0% 0.2% 1.4% 0.7% 0.
4% 0.1% 0.4% 0.1% 0.1% 0.1%
x264 [info]: kb/s:54.2

encoded 52218 frames, 43.81 fps, 54.25 kb/s

Screenshots :

Source :
http://img36.imageshack.us/img36/7026/dottsource2.png
two passes at 60 kbps without mb-tree :
http://img179.imageshack.us/img179/2165/dottnombtree.png
two passes at 55 kbps with mb-tree and --rc-look-ahead 250 :
http://img245.imageshack.us/img245/9120/dott55kbpsmbtree.png

Files :
No-mbtree version at 60 kbps (http://rapidshare.de/files/48336024/DOTT_60kbps-no-mbtree_.mkv.html)
mb-tree version at 55 kbps (http://rapidshare.de/files/48335993/DOTT_55kbps-mbtree_.mkv.html)

[MatLz]

Well, it's not for annoying you, but when I said you to change the bitrate to 60, it was for exactly the same source (number of frames, fps)
And now the difference is not really noticeable cause divide fps by 7 is near multiply bitrate by 7 in term of quality

[LoRd_MuldeR]

That clip is really small. Maybe you could apply something like NNEDI2_rpow2(rfactor=2) (http://web.missouri.edu/~kes25c/nnedi2.zip) before encoding it.

At the upscaled version differences may become more easy to spot...

[xyloy]

Well, it's not for annoying you, but when I said you to change the bitrate to 60, it was for exactly the same source (number of frames, fps)
:o

I'll edit my previous post, then.

And now the difference is not really noticeable cause divide fps by 7 is near multiply bitrate by 7 in term of quality
Yep, that's what I figured.

That clip is really small. Maybe you could apply something like NNEDI2_rpow2(rfactor=2) before encoding it.
Sounds interesting, I'll give it a try.

[MatLz]

(Out of topic)
Why the encoders add lot lot lot lot of colors if the source is only count 256 colors?

[LoRd_MuldeR]

(Out of topic)
Why the encoders add lot lot lot lot of colors if the source is only count 256 colors?

x264 works on the YV12 colorspace. This means there are 12 bits per pixel: 8-Bit luma (brightness) per pixel and 2x8-Bit chroma (color) per 4-pixel-block.

Even if the source originally only had 256 colors, it will be up-converted to that "higher" color-resolution before it's fed into the encoder...

[Slabrdix]

Hi
I recently did some avinaptic DRF Analysis on MB-Tree Encoded Files
and all my results looked similar to this

[ DRF analysis ]
Average DRF: 26.961957
Standard deviation: 0.336384
Max DRF: 27

DRF<19: 0 ( 0.000 %)
DRF=19: 1 ( 0.000 %)
DRF=20: 0 ( 0.000 %)
DRF=21: 0 ( 0.000 %)
DRF=22: 16 ( 0.006 %)
DRF=23: 16 ( 0.006 %)
DRF=24: 3114 ( 1.232 %)
DRF=25: 53 ( 0.021 %)
DRF=26: 19 ( 0.008 %)
DRF=27: 249630 ( 98.727 %) #########################
DRF>27: 0 ( 0.000 %)

The Deviation was very very small, everything all P/B Frames seemed to have one DRF
ist that normal for MB-tree ?

[LoRd_MuldeR]

It has been explained various times why those frame-quantizers are absolutely useless as a quality measurement :search:

[MatLz]

I understand.
But result is bad...
Just take the source pic in xyloy's first post.
Do two imagesource scripts with one with converttoyv12.....
It's horrible!

[LoRd_MuldeR]

I understand.
But result is bad...
Just take the source pic in xyloy's first post.
Do two imagesource scripts with one with converttoyv12.....
It's horrible!

That's chroma subsampling! You can upscale the source by a factor 2 before converting it to YV12 in order to keep the full chroma resolution of the original...

http://en.wikipedia.org/wiki/Chroma_subsampling#Sampling_systems_and_ratios

[akupenguin]

Why the encoders add lot lot lot lot of colors if the source is only count 256 colors?
Because there aren't any good ways to lossily compress paletted images.

[VFR maniac]

Add missing fclose( rc->p_mbtree_stat_file_in ).


diff --git a/encoder/ratecontrol.c b/encoder/ratecontrol.c
index cb8a9d4..0037b12 100644
--- a/encoder/ratecontrol.c
+++ b/encoder/ratecontrol.c
@@ -841,6 +841,8 @@ void x264_ratecontrol_delete( x264_t *h )
x264_free( rc->psz_mbtree_stat_file_tmpname );
x264_free( rc->psz_mbtree_stat_file_name );
}
+ if( rc->p_mbtree_stat_file_in )
+ fclose( rc->p_mbtree_stat_file_in );
x264_free( rc->pred );
x264_free( rc->pred_b_from_p );
x264_free( rc->entry );

[Forteen88]

Is there a noticeably quality-increase when increasing from default rc-lookahead to 250?

EDIT: thx LoRd_MuldeR, I didn't know that "Placebo"-preset had that rc-lookahead value. Yeah, that surely gives me a hint :)

[LoRd_MuldeR]

Is there a noticeably quality-increase when increasing from default rc-lookahead to 250?

The fact the even x264's Placebo preset uses a value of "only" 60 should give you a hint on what the reasonable range for that option is ;)

Also note that the memory requirement for rc-lookahead 250 will be extraordinary...

[Snowknight26]

Also note that the memory requirement for rc-lookahead 250 will be extraordinary...

Over 4GB @ 1920x816.

[LoRd_MuldeR]

Over 4GB @ 1920x816.

This will be possible only with a 64-Bit build of x264 running on a 64-Bit OS. And of course more than 4 GB of physical RAM should be available ;)

Any 32-Bit build would certainly abort with memory allocation failure...

[burfadel]

Doesn't it also depend on keyframe frequency? if the maximum distrance between a keyframe is 80 frames, and you stipulate 250 as the lookahead, wouldn't it be more disadvantageous rather than advantageous to run such a high lookahead? (or needing to reduce CRF if in CRF mode?)...

[akupenguin]

if the maximum distrance between a keyframe is 80 frames, and you stipulate 250 as the lookahead
x264 doesn't let you do that.

[burfadel]

Sorry I mis-worded my response :) I meant the actual keyframe distance. If numerous I frames are needed for the material such that the maximum keyframe interval will only ever be 80 for that material, setting a too high rc-lookahead could reduce the quality since it would be limited the quality of the more frequent I-frames? (I hope that makes more sense...)

[akupenguin]

MB-tree doesn't look beyond scenecuts, only beyond keyframes forced by max-keyint. If you have I-frames more frequently than required by keyint, then they must have been scenecuts, so excess lookahead isn't detrimental. And if not, then the whole lookahead is useful.

[user822]

wow this one of the amazing downloads i ever had!

i tested it with Megui and Dxva Crf22 with provided and this version, my results with a small animation-clip:

old: 18,5MB
new: 9,97MB

Thanks alot!

[Chengbin]

wow this one of the amazing downloads i ever had!

i tested it with Megui and Dxva Crf22 with provided and this version, my results with a small animation-clip:

old: 18,5MB
new: 9,97MB

Thanks alot!

CRF has changed. With the same CRF, the MBtree encode will be much smaller.

[user822]

CRF has changed. With the same CRF, the MBtree encode will be much smaller.

thats the positive thing here :)

but same crf means same quality doesn't it?

[Chengbin]

thats the positive thing here :)

but same crf means same quality doesn't it?

No. It does not.

The scale for CRF changed, you can't compare CRF.

[LoRd_MuldeR]

but same crf means same quality doesn't it?

No, it doesn't. Same CRF gives roughly similar quality, but only for the same x264 version (CRF was re-defined several times in the past), the same settings (there are settings that effect both, the size and the quality, at a given CRF) and the same kind of source. The usual method is as follows: Try to find the highest possible CRF value that still gives satisfactory quality. And then use it!

[user822]

oke thanks for telling me that

just did some comparisions with Elecard Video Quality Estimator, i saw some little difference!

[me7]

oke thanks for telling me that

just did some comparisions with Elecard Video Quality Estimator, i saw some little difference!

The devs try to keep CRF as similar as possible. The addition of MB-Tree changed x264's behaviour so CRF was readjusted to produce the same results for "The Curse of the Black Pearl" as before if I remember correctly. Therefore it is no surprise that you see little diferrence, but this does not guarantee that the results stay the same for all sources.

[Zelos]

as mbtree drops some frames and so the final file is size reduced, do mbtree affect the overall quality ?

[dj_tjerk]

as mbtree drops some frames and so the final file is size reduced, do mbtree affect the overall quality ?
Where did you hear/read this?

[Dark Shikari]

as mbtree drops some frames and so the final file is size reduced, do mbtree affect the overall quality ?If it didn't affect the quality, what would be the point of it to begin with?

[Zelos]

that will be magic :p

[chipzoller]

Is mb-tree still conflicting with b-pyramids in r1271? If so, and if they're both enabled in the same line, does mb-tree override b-pyramid or the other way around?

[Dark Shikari]

Is mb-tree still conflicting with b-pyramids in r1271? If so, and if they're both enabled in the same line, does mb-tree override b-pyramid or the other way around?MB-tree overrides B-pyramid.

[benwaggoner]

MB-tree overrides B-pyramid.
Something you're working on? I'd think MB-tree would be a very useful tool in making optimum B-pyramid strucure decisions. But that could pretty quickly become a non-Euclidian tree :).

[Fr4nz]

Something you're working on? I'd think MB-tree would be a very useful tool in making optimum B-pyramid strucure decisions. But that could pretty quickly become a non-Euclidian tree :).

Hmm no, I think he means that "b-pyramid" is disabled if "mb-tree" is on :)

[kemuri-_9]

Hmm no, I think he means that "b-pyramid" is disabled if "mb-tree" is on :)

ben was asking if he's working on fixing it to where mb-tree and b-pyramid can be used together.

[benwaggoner]

ben was asking if he's working on fixing it to where mb-tree and b-pyramid can be used together.
...and giving a nod to it being quite tricky, since once you've got adaptive b-pyramid patterns, you don't even know frame is referencing which at first, which would make having mbtree be used in making b-pyramid decisions rather complex.

Hopefully the current mode makes decisions optimally enough that mbtree could be used after those decisions are locked down, and so decode order would already be known.

[yesgrey]

Is mb-tree still conflicting with b-pyramids in r1271? If so, and if they're both enabled in the same line, does mb-tree override b-pyramid or the other way around?
MB-tree overrides B-pyramid.
Even if I enable b-pyramid after the preset (that enables mb-tree by default)?
I did not knew about this and only after finishing the encoding I noticed a warning about it...
Here is the command line used:
x264 --preset slow --tune grain --crf 18.0 --level 4.1 --b-pyramid --ref 4 --aud
and the warning: [warning]: b-pyramid + mb-tree is not supported

So, can I assume that the b-pyramid enabling was ignored?

Thanks.

[Snowknight26]

Right.

[yesgrey]

Great.
I need to look carefully at any warnings in my next encodes... this time I got look.;)

[Chengbin]

One question suddenly hit me today.

DS, on what source did you get a 70% SSIM improvement on?

[Dark Shikari]

One question suddenly hit me today.

DS, on what source did you get a 70% SSIM improvement on?touhou

[benwaggoner]

touhou
After the first time I encoded from Touhou, I did a whiteboard pitch for a similar approach :). That source has such an intense combination of high-motion high-frequency and low-motion low-frequency regions that it utterly begs for some kind of motion-adaptive differential quantization.

It's nice to see that it's more generally applicable. I was able to do something similiar in PEP by setting QP manually per region of the frame, but that's not very scalable for content with edits.

[Chengbin]

I'm just curious, what part of mbtree conflicts with b-pyramid?

[IgorC]

One question suddenly hit me today.

DS, on what source did you get a 70% SSIM improvement on?

The answer is on the 2d page.

I'm just curious, what part of mbtree conflicts with b-pyramid?
The answer is on the 19th page.

[IgorC]

After the first time I encoded from Touhou, I did a whiteboard pitch for a similar approach :). That source has such an intense combination of high-motion high-frequency and low-motion low-frequency regions that it utterly begs for some kind of motion-adaptive differential quantization.

It's nice to see that it's more generally applicable. I was able to do something similiar in PEP by setting QP manually per region of the frame, but that's not very scalable for content with edits.
That's it!
In my humble opinion 'motion-adaptive differential quantization' is more informative,intuitive and easier to understand what actually this algorithm does. If I woulnd't read original post then I wouldn't get even a remote idea what is actually 'Macroblock tree ratecontrol'.

[Dark Shikari]

That's it!
In my humble opinion 'motion-adaptive differential quantization' is more informative,intuitive and easier to understand what actually this algorithm does. If I woulnd't read original post then I wouldn't get even a remote idea what is actually 'Macroblock tree ratecontrol'.But it's not correct either.

Motion-adaptive differential quantization was already tried years ago in x264 and it didn't work, so it wasn't committed.

[benwaggoner]

That's it!
In my humble opinion 'motion-adaptive differential quantization' is more informative,intuitive and easier to understand what actually this algorithm does. If I woulnd't read original post then I wouldn't get even a remote idea what is actually 'Macroblock tree ratecontrol'.
Well, I'll let DS decide if that's a fair description of how it works. My whiteboard meandering was more VC-1 centric, and focused on back-propogation of rate distortion optimization based on how improving quality in earlier reference macroblocks would reduce the number of bits required to get a good match in later motion vectors.

But I never thought it through in nearly the implementable detail that DS obvious has had to. I spent two years as a banking software maintenance engineer, and haven't been able to write code since :).

[IgorC]

But it's not correct either.

Motion-adaptive differential quantization was already tried years ago in x264 and it didn't work, so it wasn't committed.
I'm not an expert of signal theory neither know how actually mbtree is implemented. However can you explain a little bit more?
What would be mbtree in terms of signal processing theory if it's not already in those?

Thus instead of lowering quality in high-complexity scenes (like x264 currently does), it'll only lower quality on the complex part of the scene, while for example a static background will remain high-quality. It also has many other more subtle effects, some potentially negative, most probably not.
It sounds like adaptive quantization based on motion analysis.
More probably give a name of 'differential' quantization (DQ) wasn't appropriate here. Pretty sure I'm wrong but DQ quantize the differences so it has something to do with motion estimation, isn't it?

But seems like it's not the way how mbtree was implemented.

The only conclusion is that it's pretty dangerous to give names as I actually do. XD

[benwaggoner]

Pretty sure I'm wrong but DQ quantize the differences so it has something to do with motion estimation, isn't it?
Differential Quantization ("DQuant") simply means that the quantization parameter varies over the blocks in a frame. The default mode of codecs historically has been to use the same QP for a whole frame, and then do rate control by varying QP between frames. There's plenty of older codecs that don't support DQuant on a bitstream level.

[akupenguin]

"Motion-adaptive differential quantization" sounds like it describes some adaptation based on current motion. MB-tree is based on how well inter prediction works (which is only indirectly related to the amount of motion), and measures it over the next 40 frames (or however much you tell it to analyze).

[IgorC]

Differential Quantization ("DQuant") simply means that the quantization parameter varies over the blocks in a frame.
Isn't it adaptive quantization on macroblock level? http://mewiki.project357.com/wiki/X264_Settings

http://www.engineer.tamuk.edu/SPark/chap6.pdf


The default mode of codecs historically has been to use the same QP for a whole frame, and then do rate control by varying QP between frames. There's plenty of older codecs that don't support DQuant on a bitstream level.
Like divX ASP encoder.

"Motion-adaptive differential quantization" sounds like it describes some adaptation based on current motion. MB-tree is based on how well inter prediction works (which is only indirectly related to the amount of motion), and measures it over the next 40 frames (or however much you tell it to analyze).
thank you for clarifying.

[Redsandro]

What does it actually do?

It tracks the propagation of information from future blocks to past blocks across motion vectors. It could be described as localizing qcomp to act on individual blocks instead of whole scenes. Thus instead of lowering quality in high-complexity scenes (like x264 currently does), it'll only lower quality on the complex part of the scene, while for example a static background will remain high-quality. It also has many other more subtle effects, some potentially negative, most probably not.

I am obviously a bit less technical than most in this topic, but can I ask you to tickle my curiosity by explaining why this is necessary? I always thought that that's what a 2 pass encode is for.

[Astrophizz]

Whereas a 2-pass would alter the quality based on whole frames, macroblock tree ratecontrol allows variable quality within a frame.

[Redsandro]

Ah, thanks for the clarification. Is the result also MP4 specification complient so I can put it on BD and it plays in my HDTV?
Do you know when this patch was committed? Because I can use the improved quality, but I also need to use Jeeb's patched build (1183) for a tweak (I forgot the name) that makes the avc stream Blu-Ray complient, because standard x264 apparently does not. At least back in the day when I was figuring out what to use.

[Astrophizz]

Yes, it's still spec compliant... though I can't remember exactly when the patch was committed.

[laserfan]

Do you know when this patch was committed?I like this page:

http://x264.nl/x264/changelog.txt

[Redsandro]

Thanks.
Let me translate that into something that makes it more easy to read:
commit bb66c482242a0747823661b212114c1a2f015fe3 r1197
Author: Jason Garrett-Glaser <darkshikari@gmail.com>
Date: Tue Aug 4 17:46:33 2009 -0700

Macroblock-tree ratecontrol
On by default; can be turned off with --no-mbtree.
Uses a large lookahead to track temporal propagation of data and weight quality accordingly.
Requires a very large separate statsfile (2 bytes per macroblock) in multi-pass mode.
Doesn't work with b-pyramid yet.
Note that MB-tree inherently measures quality different from the standard qcomp method, so bitrates produced by CRF may change somewhat.
This makes the "medium" preset a bit slower. Accordingly, make "fast" slower as well, and introduce a new preset "faster" between "fast" and "veryfast".
All presets "fast" and above will have MB-tree on.
Add a new option, --rc-lookahead, to control the distance MB tree looks ahead to perform propagation analysis.
Default is 40; larger values will be slower and require more memory but give more accurate results.
This value will be used in the future to control ratecontrol lookahead (VBV).
Add a new option, --no-psy, to disable all psy optimizations that don't improve PSNR or SSIM.
This disables psy-RD/trellis, but also other more subtle internal psy optimizations that can't be controlled directly via external parameters.
Quality improvement from MB-tree is about 2-70% depending on content.
Strength of MB-tree adjustments can be tweaked using qcompress; higher values mean lower MB-tree strength.
Note that MB-tree may perform slightly suboptimally on fades; this will be fixed by weighted prediction, which is coming soon.

Jeeb's build is from september so looks like I can use both! :)
But it looks like it's on by default, so I already have been using it, with default lookahead settings. Don't remember seeing a separate statsfile hanging around, though.

[Dark Shikari]

There are NAL-HRD patches for newer x264s. You should avoid using old versions of MB-tree as there were a lot of early bugs (read: at least 1-2 dozen) that we have fixed since.

[Redsandro]

Ah yes, HRD was it, my television likes that. I will check them out. But honestly I used MeGUI to construct a command line for my scripts and I thought whoever supervises the package updates probably has a compatibility reason for making included x264.exe choises. There were regular updates but x264.exe hasn't been updated since that november version.

I guess I can grab a new patched version with better MBtree implementation. I could at the same time get the 64 bit version to push the lookahead farther.

[Dark Shikari]

Ah yes, HRD was it, my television likes that. I will check them out. But honestly I used MeGUI to construct a command line for my scripts and I thought whoever supervises the package updates probably has a compatibility reason for making included x264.exe choises. There were regular updates but x264.exe hasn't been updated since that november version.What? MeGUI has autoupdates for the latest x264. Let me guess, you're using that broken 1051 version that never gets updated because the autoupdate server is broken? :rolleyes:

[Redsandro]

Uuuuhhhhhh....

Yep. That's me. Looks like you're right..

In my own defense, I hardly ever start MeGUI, only occasionally to see if there's a newer x264 or some new profile settings I can steal. (I use scripts for encoding but I just like the way MeGUI conveniently updates stuff.)
Thanks for pointing it out, :P I will manually update now.

[Digital Corpus]

I'm using x264 through mencoder. I'm currently on v1301. I'm having problems with my two pass encodings failing due to mbtree. Should I post what is going on or upgrade to the latest and try again to see if the same thing happens?

[ajp_anton]

I'm using x264 through mencoder. I'm currently on v1301. I'm having problems with my two pass encodings failing due to mbtree. Should I post what is going on or upgrade to the latest and try again to see if the same thing happens?Since you're not posting what's "going on", you're assuming the devs already know about the problem, which means it's most likely already fixed in the latest version. Why don't you try it?

[LoRd_MuldeR]

Digital Corpus, most important you should try to use x264 directly, not through MEncoder. If it works okay then, there must be a problem on MEncoder's side.

Also you didn't mention what problem with 2-Pass and MB-Tree you get in detail...

[microchip8]

@Digital Corpus

This is a MEncoder problem and one of my h264enc users also contacted me about this. I advise to switch in using crf if you can instead of 2-pass. It does not happen in crf mode (of course)

I set up the script the usual way; using '-2p -p vhq' preset, pretty much default all the way, nowdays I pick 'neroAAC' for audio and Matroska container.
Both movies encode fine until the very last frames, in the second pass, when I see

demux_mpg: 24000/1001fps progressive NTSC content detected, switching framerate.
Pos: 0.9s 52f ( 0%) 6.92fps Trem: 0min 0mb A-V:0.083 [0:1536]
Skipping frame!
Pos:2515.1s 60334f (23%) 47.94fps Trem: 68min 3642mb A-V:0.082 [1318:1536]
Skipping frame!
ODML: Starting new RIFF chunk at 1023MB. 62min 3561mb A-V:0.024 [1290:1536]
ODML: Starting new RIFF chunk at 2047MB. 32min 3364mb A-V:0.058 [1233:1536]]
ODML: Starting new RIFF chunk at 3071MB. 0min 3074mb A-V:0.050 [1200:1536]
x264 [error]: Incomplete MB-tree stats file.min 3073mb A-V:0.035 [1200:1536]
x264_encoder_encode failed
Pos:9414.2s 225749f (99%) 47.17fps Trem: 0min 3073mb A-V:0.038 [1200:1536]
Flushing video frames.

Now the thing just hangs and if I Ctrl-C it, the mux gets corrupt. The second movie also complains on the last frames in the second pass, with a similar
error message from x264.

[Dark Shikari]

Such a failure can occur if the first and second pass don't have identical frames.

[Redsandro]

What? MeGUI has autoupdates for the latest x264. Let me guess, you're using that broken 1051 version that never gets updated because the autoupdate server is broken? :rolleyes:
Why isn't there a fixed version on sourceforge then? Even the latest unstable release doesn't update beyond the tools I already have. The update server is not broken, because it updates back when I revert to an even older version I have backed up.

[Digital Corpus]

Since you're not posting what's "going on", you're assuming the devs already know about the problem, which means it's most likely already fixed in the latest version. Why don't you try it?I know I didn't post detail making my post in itself a bit weak. I developed a corrupt ssh session in the terminal window I was using so I didn't have a chance to grab the output. Once I have a complete non-mbtree file encoded, I'll post the repetitive errors involving something with invalid motion vectors and attempting to fall back to a I frame, iirc.


Digital Corpus, most important you should try to use x264 directly, not through MEncoder. If it works okay then, there must be a problem on MEncoder's side.

Also you didn't mention what problem with 2-Pass and MB-Tree you get in detail...Also, I'm not familiar with using x264 directly when working with raw MPEG-2 transport streams that may have possible corruption (I get 99.9% reception). I know I can pass those through mencoder without an issue, but not x264. I haven't had time to test passing a YUV stream to x264 via a pipe yet either so if you guys could help on that, that would be cool.


@Digital Corpus

This is a MEncoder problem and one of my h264enc users also contacted me about this. I advise to switch in using crf if you can instead of 2-pass. It does not happen in crf mode (of course)Yeah I gathered that so from looking at the errors a little more closely earlier on. I'm concerned with first attempt getting finite bitrates/file sizes within about +/- 5%. I'm developing my server as kinda like a public DVR so I need to know how much space each recording and encoded copy will take up when it come to maintenance and judging capacity.

[McArty]

Why isn't there a fixed version on sourceforge then? Even the latest unstable release doesn't update beyond the tools I already have. The update server is not broken, because it updates back when I revert to an even older version I have backed up.

Take a look here:
http://forum.doom9.org/showthread.php?t=149383
Worked for me.

[LoRd_MuldeR]

Why isn't there a fixed version on sourceforge then? Even the latest unstable release doesn't update beyond the tools I already have. The update server is not broken, because it updates back when I revert to an even older version I have backed up.

Maybe the update server is still alive, but obviously nobody does update the files on the update server. So it's broken - in a way ;)

[Digital Corpus]

Okay, server kernel panicked last night so I had to redo the encoding to duplicate my bug. I have a 20Mbit up connection so if you wish to download any of the files below, my end usually isn't the bottleneck.

Terminal output:
http://pastebin.com/m517d6f30

Command used:
mencoder "~V~_11-03-2009_20-00-00.ts" -o /hdhr/demux/"V"/"~V~11-03-2009~720.avi" -aspect 16:9 -tskeepbroken -really-quiet \
-passlogfile /hdhr/demux/"V"/"V~11-03-2009~720.log" -ofps 30000/1001 -sws 9 -vf harddup,scale=1280:720 -ovc x264 -x264encopts \
bitrate=5500:frameref=5:me_range=16:level_idc=31:subq=6:threads=4:pass=2:nofast_pskip:nodct_decimate:keyint=300:keyint_min=15:\
mixed_refs:deblock=1,1:bframes=0:nob_adapt:cabac:qp_min=1:qp_max=51:qp_step=25:ratetol=25:qcomp=0.75:qblur=0.25:direct_pred=auto:\
partitions=all:8x8dct:me=umh:psy-rd=0.4,0.2:trellis=1:psnr:ssim -oac pcm -channels 2 -srate 48000
(I know that CRF is better than 2-pass, but I need to carefully portion disk space thus I use 2-pass. If any of the above can be optimized to be better, I'm all ears)

The two-pass log file is automatically deleted via my scripts. I'll re-run the encode if someone wants to have a look

mbtree file:
http://atlas.selfip.net/V~11-03-2009~720.log.mbtree (521MiB) (http://atlas.selfip.net/V~11-03-2009~720.log.mbtree)

Source:
http://atlas.selfip.net/~V~_11-03-2009_20-00-00.ts (~4.7GiB) (http://atlas.selfip.net/~V~_11-03-2009_20-00-00.ts)

Encoded file:
http://atlas.selfip.net/~V~11-03-2009~720.mov (863MiB) (http://atlas.selfip.net/~V~11-03-2009~720.mov)
Edit: I doubt this matters but despite .mov extension, this is a MP4Box created .mp4 container. The Xbox 360 is picky about extensions.

I pulled this from the .h264 file just to be on the safe side:
x264 - core 78 r1301M bcba15d

I'm going to go work on using mencoder with -tskeepbroken to pipe to x264 and see if this bug/problem happens again.

[poisondeathray]

As some may recall, there were issues with fades when using mbtree. The original source & post can be found here
post #167
http://forum.doom9.org/showthread.php?p=1312672

So I tested the weightp r1332 generic build from x264.nl

--preset slower --pass 2 --bitrate 1000
http://img338.imageshack.us/img338/8672/069slower.th.png (http://img338.imageshack.us/i/069slower.png/)


--preset slower --pass 2 --bitrate 1000 --no-mbtree --b-pyramid normal
http://img338.imageshack.us/img338/4034/069slowernombtreebpyram.th.png (http://img338.imageshack.us/i/069slowernombtreebpyram.png/)


To my eyes, not much has changed with the addition of weightp on this source. I suppose I should test mbtree with weightp disabled to see the difference that weightp (only) makes - but in regular use, my decision is usually mb-tree vs. no mb-tree with b-pyramid.

Yes, qualitatively, other frames are improved with mb-tree, but the trend seems to be fades are significantly worse. It's almost as if bitrate is redistributed from fades/low luminance frames to other frames

[burfadel]

well, if you look at it objectively the --no-mbtree encoded are larger, in 2-pass mode this translates into higher quality with --no-mbtree disabled! In CRF mode, it means the ability to use a lower CRF (even if you use a decimal CRF), which again leads to a higher qulaity picture. I'm talking about the main part, not the fades!

[poisondeathray]

well, if you look at it objectively the --no-mbtree encoded are larger, in 2-pass mode this translates into higher quality with --no-mbtree disabled! In CRF mode, it means the ability to use a lower CRF (even if you use a decimal CRF), which again leads to a higher qulaity picture. I'm talking about the main part, not the fades!

Are you looking at the .png size? The coded frame size is higher without mbtree, so yes it should be larger. weightp was supposed to detect fades in improve those coded frame sizes... I think... but there was little to no improvement in the couple of sources I've tried so far.

mbtree can definitely be beneficial in some scenarios, but it almost always comes with signicantly worse fades and dark scenes. Perhaps the default --qcomp value isn't optimal, but increasing it will lower the mbtree effect, and basically we are back to square one. What i am seeing is a sort of "tradeoff." Some frames are improved at the expense of others. But the "expense" is quite severe in my opinion, at least with default qcomp. I'm looking for more balance.

Remember this is a bitrate constrained scenario @1000kbps, but "martinfrombern" did tests with varying bitrates, and the trend was still the same.
http://forum.doom9.org/showthread.php?p=1313104#post1313104

Also these are just some observations, so be careful to draw any firm conclusions yet

[Dark Shikari]

What the hell do you want me to do, turn fade detection on all modes and lower the quantizers by 10 in every fade? :rolleyes:

I'm giving up; people won't be satisfied unless their fades look flawless even while the rest of the video looks like utter crap. And even then, they'll complain that 90% of the bitrate of the video is spent on the fade.

[poisondeathray]

What the hell do you want me to do, turn fade detection on all modes and lower the quantizers by 10 in every fade? :rolleyes:

I'm giving up; people won't be satisfied unless their fades look flawless even while the rest of the video looks like utter crap. And even then, they'll complain that 90% of the bitrate of the video is spent on the fade.


Relax DS, I'm not "complaining", I'm just providing some observations and feedback on this new weightp patched build... Is it possible something was wrong with the build?

In my opinion, mbtree with some settings is not suitable for some sources in some scenarios - but people can choose to use what ever settings they want to. :)

[juGGaKNot]

Maybe weightp + new normal b-pyramid :)

[Betsy25]

After the abysmal puzzle MeGUI has become to go searching for a trustable version, now also x264 joins this clan.

Happy encoding days for the end of the year....:rolleyes:

[Inspector.Gadget]

Boo hoo I don't know how to use search.

[Fr4nz]

After the abysmal puzzle MeGUI has become to go searching for a trustable version, now also x264 joins this clan.

Happy encoding days for the end of the year....:rolleyes:

Well, given that x264 is opensource, no one is paid for it and you didn't give any help in making it, I think a ban from this forum is a proper thing for you!

[Betsy25]

Well, given that x264 is opensource, no one is paid for it and you didn't give any help in making it, I think a ban from this forum is a proper thing for you!

Oh My ! A forum can sometimes also be used to vent a little.
There's nothing really wrong with that. Are you a communist ?

What my biggest gripe is, for 6 months this forums gets filled with thousands of posts of people trying to get MeGUI working, now the same thing is going on for some month regarding x264 ?

Are you going to ban all of them also ?

Perhaps a little sticky regarding the state of MeGUI and latest x264 releases would make it much easier for those serving nothing more that "Use the Search!".

A little addition to the toppost of the main MeGUI thread is not that much to be asked ?

[Guest]

Guys, stop the flame wars and stay on topic.

[thewebchat]

What the hell do you want me to do, turn fade detection on all modes and lower the quantizers by 10 in every fade? :rolleyes:

I'm giving up; people won't be satisfied unless their fades look flawless even while the rest of the video looks like utter crap. And even then, they'll complain that 90% of the bitrate of the video is spent on the fade.

How about a mode where every frame is visually indistinguishable from the source, regardless of however many bits this will cost. One of the things that bothers me with x264 is that no matter how high I drive the bitrate, some parts just tend to look Pretty Bad, and fades/dissolves are one of them.

[LoRd_MuldeR]

How about a mode where every frame is visually indistinguishable from the source, regardless of however many bits this will cost.

That mode exists for a long time:

--qp 0 :D

[thewebchat]

Visually indistinguishable is quite a bit different from mathematically, Lord_Mulder. What I'm talking about is some way to get even quality distribution over all frames, without doing something dumb like setting every frame to be IDR. This is probably not psychovisually optimal, but I'm sure someone somewhere cares about getting archival quality across all frames.

[LoRd_MuldeR]

Visually indistinguishable is quite a bit different from mathematically, Lord_Mulder.

Yeah. And as we all know, the perfect objective quality metric that 100% agrees with the HVS doesn't exist! So how do you want to define "visually lossless" in a computable way?

The best thing we have as a "constant quality" mode nowadays is x264's CRF mode. So if you want "visually lossless" quality, go and find the CRF value that looks lossless for your eyes.

And another remark: Take 10 random people and let them subjectively define "visually lossless" quality. You'll get 10 different results. So how should a computer program know ???

[Sharc]

How about a mode where every frame is visually indistinguishable from the source, regardless of however many bits this will cost. One of the things that bothers me with x264 is that no matter how high I drive the bitrate, some parts just tend to look Pretty Bad, and fades/dissolves are one of them.
A plain 1:1 copy will serve the purpose best, I guess ....:rolleyes:

[Dark Shikari]

Yeah. And as we all know, the perfect objective quality metric that 100% agrees with the HVS doesn't exist! So how do you want to define "visually lossless" in a computable way?

The best thing we have as a "constant quality" mode nowadays is x264's CRF mode. So if you want "visually lossless" quality, go and find the CRF value that looks lossless for your eyes.

And another remark: Take 10 random people and let them subjectively define "visually lossless" quality. You'll get 10 different results. So how should a computer program know ???
<pengvado> there's nothing wrong with a high level analysis of which areas are important. it's just that our strong AI isn't quite advanced enough to make it work :)
<Dark_Shikari> the thing is you don't only need an AI, you need a human-like AI, since you need to measure which areas a *human* would judge as important
<pengvado> oh right. we don't need an AI, we can just use the chained grad student strategy

[juGGaKNot]

<pengvado> there's nothing wrong with a high level analysis of which areas are important. it's just that our strong AI isn't quite advanced enough to make it work :)
<Dark_Shikari> the thing is you don't only need an AI, you need a human-like AI, since you need to measure which areas a *human* would judge as important
<pengvado> oh right. we don't need an AI, we can just use the chained grad student strategy

Old, old.

[Disabled]

... turn fade detection on all modes and lower the quantizers by 10 in every fade? :rolleyes:

Well... wouldn't it be possible to give us a switch that with weightp enabled allows us to increase quants for them, or disable/reduce the effect of mbtree for fades or something like that?
Or I remember you didn't want us to give 1000 switches nobody really understands, so wouldn't it be possible to tune the quants for fades per default now that fades are detected with weightp?

[Dark Shikari]

Well... wouldn't it be possible to give us a switch that with weightp enabled allows us to increase quants for them, or disable/reduce the effect of mbtree for fades or something like that?
Or I remember you didn't want us to give 1000 switches nobody really understands, so wouldn't it be possible to tune the quants for fades per default now that fades are detected with weightp?It already does that.

Weightp improves quality in fades by at least a few quantizers "worth" of quality. Then, Weightp is run in lookahead to change the macroblock costs for MB-tree, which then effectively lowers quantizers by 1-3. Further, if weightp isn't on, the former is still done, and in order to compensate for lack of weightp, the quantizer in fades is lowered by another 1-2.

[LoRd_MuldeR]

Well... wouldn't it be possible to give us a switch that with weightp enabled allows us to increase quants for them, or disable/reduce the effect of mbtree for fades or something like that?

Remember that lowering the quantizers in some places (e.g. fades) means raising the quantizers in other places. It's a trade-off!

(Assuming we don't want to increase the overall size. Of course we can always "buy" better quality by spending more bits. But that's not the point here, I think)

[benwaggoner]

Old, old.
If it worked for fractal compression...

[poisondeathray]

Weightp improves quality in fades by at least a few quantizers "worth" of quality. Then, Weightp is run in lookahead to change the macroblock costs for MB-tree, which then effectively lowers quantizers by 1-3. Further, if weightp isn't on, the former is still done, and in order to compensate for lack of weightp, the quantizer in fades is lowered by another 1-2.

In this particular example, weightp alone changed the frame quantizer from 24.80 (--weightp 0) to 24.31. Without mb-tree and using normal b-pyramid , the frame quantizer was 20.78. So the net change was 0.49 for weightp in this sample

I took the frame quantizers from ffdshow's osd - I don't know how accurate the values are - but they seem to correlate with the "quality" of the frame.

--preset slower --pass 2 --bitrate 1000
frame quantizer 24.31
http://i37.tinypic.com/6dra53.png

--preset slower --pass 2 --bitrate 1000 --weightp 0
frame quantizer 24.80
http://i33.tinypic.com/641168.png

--preset slower --pass 2 --bitrate 1000 --no-mbtree --b-pyramid normal
frame quantizer 20.78
http://i36.tinypic.com/a3igwk.png



And even at 3x the bitrate, although the reported frame quantizer is lower than the 1Mbps encode without mbtree, I think the without mbtree looks better on that frame. (For completeness, no mbtree at 3Mbps was included as well)

--preset slower --pass 2 --bitrate 3000
frame quantizer 19.15
http://i35.tinypic.com/vfib05.png

--preset slower --pass 2 --bitrate 3000 --no-mbtree --b-pyramid normal
frame quantizer 15.17
http://i36.tinypic.com/2nbdnxt.png

[me7]

Does weightp have an effect on VBV-limited devices? I just did a few test encodes for my iPod touch 3G and noticed that weightp causes short freezes, can be a decoder bug too of course (Main Profile at Level 3.0, same settings apart from --weightp 0 and --weightp 2).

[poisondeathray]

Does weightp have an effect on VBV-limited devices? I just did a few test encodes for my iPod touch 3G and noticed that weightp causes short freezes, can be a decoder bug too of course (Main Profile at Level 3.0, same settings apart from --weightp 0 and --weightp 2).

this is from the changelog, not sure if your device falls into that category ?

All weightp modes may cause minor breakage in non-compliant decoders that take shortcuts in deblocking reference frame checks.
"Smart" may cause serious breakage in non-compliant decoders that take shortcuts in handling of duplicate reference frames.

[yuvi]

A stab in the dark: try lowering the refs. My iPod touch 1g maxes at 6 at any resolution despite claiming to support baseline level 3.0, and weightp increases number of ref frames.

[thewebchat]

Yeah. And as we all know, the perfect objective quality metric that 100% agrees with the HVS doesn't exist! So how do you want to define "visually lossless" in a computable way?

The best thing we have as a "constant quality" mode nowadays is x264's CRF mode. So if you want "visually lossless" quality, go and find the CRF value that looks lossless for your eyes.

And another remark: Take 10 random people and let them subjectively define "visually lossless" quality. You'll get 10 different results. So how should a computer program know ???

The problem is that no matter how many bits you throw at it, with the current bit distribution, none of them go to the parts that look bad. Instead, you just end up wasting space and CPU making already transparent parts "more" transparent. For example, on a downscaled BD test I did, even at CRF14, there was significant degradation and ringing in dissolves and fades to/from white (with weightp), loss of grain/fine particle details, and banding. Now, I'm sure most people don't look at individual frames and compare them, but I think every encoder should be able to reach transparency within a few multiples of normal rates. Actually, maybe there's magic options that I should be tweaking, but I really am not an expert in x264.

[Sharc]

......even at CRF14, there was significant degradation and ringing in dissolves and fades to/from white (with weightp), loss of grain/fine particle details, and banding. Now, I'm sure most people don't look at individual frames and compare them, ....
For the discovery of banding one does actually not need to look at individual frames because bandig often spans many frames (e.g. flat, dark scenes). I agree that lowering the crf does not always help. Perhaps some 'adaptive' dithering of the source could help here? Just guessing, don't know if it is viable.

[me7]

A stab in the dark: try lowering the refs. My iPod touch 1g maxes at 6 at any resolution despite claiming to support baseline level 3.0, and weightp increases number of ref frames.

Refs were set to 5 anyway, the problem was caused by --weightp 2. Setting weightp to 1 seems to fix it.

[Dark Shikari]

Refs were set to 5 anyway, the problem was caused by --weightp 2. Setting weightp to 1 seems to fix it.Weightp 2 adds two more refs, not one, so set refs to 4.

[shon3i]

Is Weightp 1/2 good choice for standalone devices? and how to calulate max reference frames if Weightp is used? 2 for blu-ray?

[Dark Shikari]

Is Weightp 1/2 good choice for standalone devices? and how to calulate max reference frames if Weightp is used? 2 for blu-ray?The calculation is exactly the same as normal. The only difference is for broken devices that don't obey the spec.

[shon3i]

Ok, thanks.

[me7]

Weightp 2 adds two more refs, not one, so set refs to 4.

Does weightp 1 add 1 more ref frame?

The calculation is exactly the same as normal. The only difference is for broken devices that don't obey the spec.

Does this mean that weightp adds more refs that should be handled different then normal refs, but some broken decoders handle them like normal refs and therefore need a lower number of normal refs to stay within H264 level limitations?

[shon3i]

number of normal refs to stay within VBV limitations?I just have same dillema since we haved b-pyramid violation some time ago. I run encode and i will check if pass verification for level 4.1

[Dark Shikari]

The specification places no limits on reference frames, only on DPB usage. If you replicate reference frames via reordering, that doesn't use extra DPB space.

[me7]

The specification places no limits on reference frames, only on DPB usage. If you replicate reference frames via reordering, that doesn't use extra DPB space.

Can you explain what you meant by this post:
Weightp 2 adds two more refs, not one, so set refs to 4.

[DarkZell666]

Can you explain what you meant by this post:Weightp 2 adds two more refs, not one, so set refs to 4.

Total number of refs used = number of refs you asked for + number of refs added by weightp. (ie: 5+2=7 in your case).

[G_M_C]

Hmmm, does that make encoding for bluray less efficient on 1080p projects ?

The number of refs set in the commandline a big influence on encoding efficiency. On 1080p BD projects, you can only have max refs of 4. If i can only set refs=2 when using weightp=2 project, a lot of efficiency is lost as opposed to refs=4 without weigtp (on a 1080p project).

Or am i wrong in this conclusion ?

[nurbs]

As I read it you only need to reduce refs by two if your decoder isn't spec compliant.

[yuvi]

To make it clear, the commit message referred to CoreAVC 1.x breakage, which was due to something completely different and breaks no matter the number of reference frames. I don't think any other decoders were known to have problems specific to weightp.

This iPhone/iPod breakage may or may not be related to my guess of non-spec limitations with references (could someone confirm whether ref <= 4 works with weightp 2 and ref > 5 breaks with weightp 1?) and is probably unique either to Apple or to the Samsung chips used.

[Dark Shikari]

Hmmm, does that make encoding for bluray less efficient on 1080p projects ?

The number of refs set in the commandline a big influence on encoding efficiency. On 1080p BD projects, you can only have max refs of 4. If i can only set refs=2 when using weightp=2 project, a lot of efficiency is lost as opposed to refs=4 without weigtp (on a 1080p project).

Or am i wrong in this conclusion ?Why is it that even when I say something half a dozen times nobody ever reads my posts?

The limitation is on DPB size, not number of reference frames. Of course you can set refs=4 with Blu-ray encoding. Whoever said you couldn't? Nobody.

[G_M_C]

Why is it that even when I say something half a dozen times nobody ever reads my posts?

The limitation is on DPB size, not number of reference frames. Of course you can set refs=4 with Blu-ray encoding. Whoever said you couldn't? Nobody.

Yes, that can be. But you also said that weightp adds two reference frames;

Weightp 2 adds two more refs, not one, so set refs to 4.

And I reacted to that. So actually DO read your posts, i might not fully understand your thoughtprocess behind your posts, but i DO read them. And i react to them.

And btw; My STD does NOT accept ref=4 and weightp=2 together on 1080p. Nor does ffdshow, MPC or CoreAVC. ArcSoft TMT3 works however.

Expiriments on 720p (max refs on BD beeing 9) gives;
refs=8 plus weightp=2 no go
refs=6 plus weightp=2 works fine.
All other setting in the CMD kept the same, only refs changed. Can give CMD if needed. Used latest Techouse build, posted in the patched thread.

So weightp does influence the settings I have to use for making BD's.

PS: Running a full-length 720p encode now with refs 6/weightp 2. Will mux to AVCHD/BD to see what happens on my STD.

[rack04]

And btw; My STD does NOT accept ref=4 and weightp=2 together on 1080p. Nor does ffdshow, MPC or CoreAVC. ArcSoft TMT3 works however.

Haven't tested on my standalone player but in my tests ffdshow works fine with those settings. Can you upload a sample that doesn't work?

[G_M_C]

Haven't tested on my standalone player but in my tests ffdshow works fine with those settings. Can you upload a sample that doesn't work?

Overwritten the sample with the 720p i'm doing now. I'll see now if ffdshow works with that one, hold on (I've allready seen that CoreAVC doesnt work with this one, but that was allready known to happen).

Nope the raw .264 stream doesnt want to play correctly with my version of ffdshow (using Albains latest beta to test if we can get HD-audio streaming running).

PPS: Is Techouse's build maybe at fault ?

This is my CMD btw
x264.exe --threads auto --thread-input --psnr --ssim "%IN_TITLE%.avs" --stats "%IN_TITLE%.stats" --output "%OUT_TITLE%.264" --pass 2 --bitrate 4900 --deblock -2:-2 --fps 25 --aud --slices 4 --nal-hrd --vbv-bufsize 30000 --vbv-maxrate 24000 --sar 1:1 --weightp 2 --bframes 3 --b-adapt 2 --ref 6 --no-fast-pskip --qpmax 40 --ipratio 1.3 --pbratio 1.2 --direct auto --subme 10 --trellis 2 --psy-rd 1.0:0.2 --partitions all --me umh --merange 24 --mvrange 511 --aq-strength 1.0 --aq-mode 1 --rc-lookahead 75

[rack04]

Here are a couple samples I just encoded for you to try.

http://www.mediafire.com/?mmj5dyqodoz

EDIT: I just saw your post. Why are you trying to play the raw h264?

[G_M_C]

Here are a couple samples I just encoded for you to try.

http://www.mediafire.com/?mmj5dyqodoz

EDIT: I just saw your post. Why are you trying to play the raw h264?

Monitor the progress, and since it's a test; See if the decoder " takes" it. No sence in spending time muxing if the decoder flukes anyway ...

[Sharc]

Test clip 720p / weightp=2 / refs=3 plays fine with mpc-hc H264/AVC (DXVA) and coreAVC, but SAP BDP-S360 rejects the avchd disc.

Added:
Problem was with the disc. I burned new samples and they all play fine with the SAP (1080p, 720p, 576).

[shon3i]

Good to know, anyway Scenarist BD reject the stream at importing stage, with unspecified error message.

x264 cmd i used to create stream:
--tune film --bitrate 8306 --pass 2 --stats .stats --keyint 24 --min-keyint 2 --slices 4 --vbv-maxrate 30000 --vbv-bufsize 30000 --sar 1:1 --level 4.1 --threads 12 --aud --nal-hrd --output fnf2.264 fnf2.avs

[Dark Shikari]

The Blu-ray specification does not have any problem with what x264 is currently doing, so Scenarist is broken (or NAL-HRD patch is, but I doubt that).

http://i33.tinypic.com/2ijkg2g.png

Also, what the hell is this discussion doing in this thread? It's completely off-topic.

[shon3i]

@Dark Shikari, my mistake, i tryed to import some other bad stream, not one which i produced for test. Scenarist passes stream, and muxing is just fine. Resulting BD plays fine on TMT2 and 3 and PowerDVD. Tomorow i will test on some BD Player (LG 370 and 390) and WD HDTV player. And i will post results.

btw i compared fades to other non p-weight, and i see nice improvment.

[madeinlisboa]

Hi,

do you think mbtree is useful with very grainy sources like 300 or it is just a waste of processor time? Thank you

[Dark Shikari]

Hi,

do you think mbtree is useful with very grainy sources like 300 or it is just a waste of processor time? Thank youIt should be useful, just less so than usual.

[asarian]

Switching to 64-Bit OS is always a good thing, as it allows to use more than ~3 GB of physical RAM, it allows to run 64-Bit processes where needed and 32-Bit processes will still work fine.
Interesting. :)

I wonder how that works out in practice, though. Can current (32-bit) AviSynth filters still be used? (provided I install 32 bit AviSynth, of course, on my new Windows 7 64-bit install). And then still use 64-bit x264? I very vaguely remember a post of yours, a good while back, in which you used some neat piping trick to mix 32 and 64 bit processes.

I'm not at the limit yet (x264 only uses ca 1.4 GB yet); but I don't have a lot of room to manoeuvre in, either (like for increasing rc_lookahead and merange).

Thanks.

[Redsandro]

As always, 32 bit filters are not compatible with 64 bit software. Thats's a problem for me, as I used avisynth for all my proxies in Adobe After Effects and Première. It was an awesome workflow while it lasted.

Now, the software has become 64 bit only, and I cannot use 64 bit avisynth because most useful filters are 32 bit only.

I wish there were 2 separate versions at the same time on my system, like .avs for 32 bit avisynth and .a64 for 64 bit avisynth. That would solve some of the problems. 64 bit simple proxies and 32 bit fancy filter stuff.

[asarian]

As always, 32 bit filters are not compatible with 64 bit software. Thats's a problem for me, as I used avisynth for all my proxies in Adobe After Effects and Première. It was an awesome workflow while it lasted.

Now, the software has become 64 bit only, and I cannot use 64 bit avisynth because most useful filters are 32 bit only.
Yeah. Too bad at that. I do remember LoRd_MuldeR having made some sort of program, though, a while back, in which he piped the output of 32-bit AviSynth to 64-bit x264. Can't find it any more, though. :(

[Redsandro]

Yes you can do piping in the command window with different programs. If you want to do right-click .avs or .avi to .mp4 encodes from Windows, using 32 bit avisynth and 64 bit x264, you can use my x264box script (http://www.redsandro.com/?app) where I use avs2yuv.exe. Or if you just want to know how it's done you can look at the source, but basically it's something like avs2yuv.exe [AVSFILE] -o - | x264-64.exe [PARAMETERS] - --demuxer y4m

For audio you can do something similar using bepipe.exe from the behappy package.

[Redsandro]

Since x264 is obviously the bottleneck, I think there is a 1% speed difference at very best.

But if you encode a lot, encode audio too, want to mux and use the same settings for speed/quality in one click (okay two :P) you should try my script. It's VBscript, you can fully customize it.

I use it to create mp4 proxies for video editing, but I change the default settings to be faster for that, and slow for the final encode.

[ImmortAlex]

That program is pipebuf.exe, and command line is something like

pipebuf.exe avs2yuv.exe video.avs -raw -o - : x264.exe ... -o video.mkv - 720x576 : 4

I still use it because of some bug in ffmpeg64 which breaks decoding with ffdshow through DirectShowSource() in AviSynth64. Maybe that bug was fixed, I don't know. That solution works well for me, and I am too lazy to test AviSynth64 again :)

[Redsandro]

Oh yeah, buffered pipe, there was an article on this forum somewhere about the ideal buffer size.

IIRC the best speed improvement was a couple dozen of frames on a big encode, that would be less than 1%.
But with the name of the program known one could surely find back that topic and see the comparisons.

[asarian]

Thanks very much, guys! :) invaluable information, as usual! Time to start some serious work on Windows 7 64-bit!

[asarian]

Or if you just want to know how it's done you can look at the source, but basically it's something like avs2yuv.exe [AVSFILE] -o - | x264-64.exe [PARAMETERS] - --demuxer y4m

That works brilliantly. :) Thanks!

Any chance I can get avs2yuv to show x264's estimated time display? I kinda miss that now.

[rack04]

That works brilliantly. :) Thanks!

Any chance I can get avs2yuv to show x264's estimated time display? I kinda miss that now.

You will have to add the number of frames, i.e. --frames XXXXX

pipebuf.exe avs2yuv.exe [AVSFILE] - | x264.exe [PARAMETERS] - --frames XXXXXX --stdin y4m - : 4

[asarian]

You will have to add the number of frames, i.e. --frames XXXXX

pipebuf.exe avs2yuv.exe [AVSFILE] - | x264.exe [PARAMETERS] - --frames XXXXXX --stdin y4m - : 4
Thanks. Too bad, though; means you have to determine the total amount of frames first, then add it as parameter. Sorta wrecks the batchfile processing I had made for it. :) Who knows, maybe I should just ask the avs2yuv author to simply restore the 'eta' function.

[LoRd_MuldeR]

Thanks. Too bad, though; means you have to determine the total amount of frames first, then add it as parameter. Sorta wrecks the batchfile processing I had made for it.:)

You don't have to. But without "--frames" set properly x264 won't be able to display the progress properly, as the total number of frames is unknown in the case of STDIN input.

So if you want proper progress display with STDIN input, you must pass the number of frames to x264 explicitly. You can use avs2yuv.exe for that purpose...

"avs2yuv.exe" -frames 1 "%INPPATH%" -o NUL

[asarian]

You don't have to. But without "--frames" set properly x264 won't be able to display the progress properly, as the total number of frames is unknown in the case of STDIN input.
Duh on me! LOL. Yes, of course, we're streaming from STDIN here: x264 can't possible know upfront.

So if you want proper progress display with STDIN input, you must pass the number of frames to x264 explicitly. You can use avs2yuv.exe for that purpose...

"avs2yuv.exe" -frames 1 "%INPPATH%" -o NUL
Your knowledge in all areas video never ceases to amaze me. :) Thank you very much!

[asarian]

Oh man, this proves even more useful than I thought!


"c:\x264\avs2yuv.exe" -frames 1 "M:\jobs\2010.avs" -o NUL
M:\jobs\2010.avs: 1920x796, 24000/1001 fps, 166750 frames


This way I can even extract total resolution from it too, which avs2yuv seems to calculate accurately, even with multiple crops:


FFVideoSource("M:\jobs\2010.mkv").ConvertToYV12()
Crop(0, 140, 0, -140)
MCTemporalDenoise(settings="high")
Crop(0, 2, 0, -2)


VERY useful, as I can now create a small function, inside the batch-file, to auto-determine the max allowed ref frames for the source! Yay avs2yuv! :)

[asarian]

Okay, based on LoRd_MuldeR's very useful 1-liner command for avs2yuv, I wrote, like I said, a little batch file script accordingly, which will extract the number of frames, to restore the x264 'eta' functionality with avs2yuv, and which will calculate the appropriate max number of reference frames allowed for the input source. Both will be parsed to the full avs2yuv/x264 command line.

Adapt path-names and such to your own personal needs.

---------------
@echo off
cls

echo Retrieving info for: "%1.avs" ...

"c:\x264\avs2yuv.exe" -frames 1 "M:\jobs\%1.avs" -o NUL 2>"c:\x264\res.txt"
set /P _line=<"c:\x264\res.txt"

set _restline=%_line:*.avs: =%
set _restlinea=%_restline:*x=%
call set _width=%%_restline:x%_restlinea%=%%

set _restlineb=%_restlinea:*, =%
call set _height=%%_restlinea:, %_restlineb%=%%

set _restlinec=%_restlineb:* fps, =%
set _restlined=%_restlinex: frames%

call set _frames=%%_restlinec:%_restlined%=%%

set /a _reframes = 12582912 / (((%_width% * %_height%) * 3) / 2)

echo .

echo Width: %_width%
echo Height: %_height%
echo Frames: %_frames%
echo Reframes: %_reframes%

echo .

"C:\x264\avs2yuv.exe" "M:\jobs\%1.avs" -o - | "C:\x264\x264.exe" - --demuxer y4m --frames %_frames% --crf %2 --sar 1:1 --level 4.1 --aud --nal-hrd none --vbv-bufsize 70000 --vbv-maxrate 60000 --ref %_reframes% --b-pyramid normal --mixed-refs --bframes 8 --b-adapt 2 --weightb --no-fast-pskip --direct auto --subme 10 --trellis 2 --partitions all --8x8dct --me tesa --threads auto --merange 32 --aq-strength %4 --aq-mode 2 --rc-lookahead 60 --qpmin %3 --thread-input --output "C:\video\%1.264"

[nurbs]

Same command line:
C:\x264\avs2yuv.exe" "M:\jobs\%1.avs" -o - | "C:\x264\x264.exe" - --demuxer y4m --frames %_frames% --crf %2 --sar 1:1 --level 4.1 --aud --preset placebo --vbv-bufsize 70000 --vbv-maxrate 60000 --bframes 8 --merange 32 --aq-strength %4 --aq-mode 2 --qpmin %3 --output "C:\video\%1.264"

Half of the stuff in your command line were defaults or included in placebo. As long as you don't specify the number of reference frames manually x264 will automatically reduce them if you use --level.
I don't think you need --aud unless you are doing Blu-Ray/AVCHD, which you aren't. If you don't specify --merange the above command line will use 24 which is still plenty. You should think about using the tunings (film/animation) so that psy-trellis is used and the psy and aq strenght is set according to content.

[asarian]

Same command line:
C:\x264\avs2yuv.exe" "M:\jobs\%1.avs" -o - | "C:\x264\x264.exe" - --demuxer y4m --frames %_frames% --crf %2 --sar 1:1 --level 4.1 --aud --preset placebo --vbv-bufsize 70000 --vbv-maxrate 60000 --bframes 8 --merange 32 --aq-strength %4 --aq-mode 2 --qpmin %3 --output "C:\video\%1.264"

Half of the stuff in your command line were defaults or included in placebo.
Yeah, I should really clean up that old, clunky (adapted) command line to the newer stuff. :)

As long as you don't specify the number of reference frames manually x264 will automatically reduce them if you use --level.
Well, the idea was actually to increase them to as high as is allowed for the current DPB. So, it will be 5 for 1920x800, and 9 for 1280x720, etc.

[nurbs]

--preset placebo uses 16 reference frames. With --level 4.1 they will be reduced. So, it will be 5 for 1920x800, and 9 for 1280x720, etc.

[asarian]

--preset placebo uses 16 reference frames. With --level 4.1 they will be reduced. So, it will be 5 for 1920x800, and 9 for 1280x720, etc.
Ah, I didn't realize that. :) Thanks. I thought '--level 4.1' would always just reduce it to 4 (standard Blu-Ray).

[Redsandro]

@echo off
cls

echo Retrieving info for: "%1.avs" ...

"c:\x264\avs2yuv.exe" -frames 1 "M:\jobs\%1.avs" -o NUL 2>"c:\x264\res.txt"
set /P _line=<"c:\x264\res.txt"

set _restline=%_line:*.avs: =%
set _restlinea=%_restline:*x=%
call set _width=%%_restline:x%_restlinea%=%%

set _restlineb=%_restlinea:*, =%
call set _height=%%_restlinea:, %_restlineb%=%%

set _restlinec=%_restlineb:* fps, =%
set _restlined=%_restlinex: frames%

call set _frames=%%_restlinec:%_restlined%=%%

set /a _reframes = 12582912 / (((%_width% * %_height%) * 3) / 2)

echo .

echo Width: %_width%
echo Height: %_height%
echo Frames: %_frames%
echo Reframes: %_reframes%

echo .

"C:\x264\avs2yuv.exe" "M:\jobs\%1.avs" -o - | "C:\x264\x264.exe" - --demuxer y4m --frames %_frames% --crf %2 --sar 1:1 --level 4.1 --aud --nal-hrd none --vbv-bufsize 70000 --vbv-maxrate 60000 --ref %_reframes% --b-pyramid normal --mixed-refs --bframes 8 --b-adapt 2 --weightb --no-fast-pskip --direct auto --subme 10 --trellis 2 --partitions all --8x8dct --me tesa --threads auto --merange 32 --aq-strength %4 --aq-mode 2 --rc-lookahead 60 --qpmin %3 --thread-input --output "C:\video\%1.264"

Nice batch! I didn't know you could do that line splitting in Command.

[Broski]

Here are a couple samples I just encoded for you to try.

[spam link removed]

EDIT: I just saw your post. Why are you trying to play the raw h264?

It seems very tricky. The version isnt stable at all sometimes and you have to workaround a lot unfortunately. :(

[Guest]

Making vague and irrelevant replies to ancient posts is often a sign of an incipient spammer. You're not going to start spamming us are you?

[Disabled]

Just check where the quoted links are pointing to...

[Guest]

Ha ha. Thanks, Disabled, for pointing that out. He had me fooled.:stupid: