Current Patches, Where to get them, How they affect speed/output

[Dark Shikari]

So problem is generaly i x264 VBV model then.No, it's the 2-pass VBV algorithm. My one-pass is still fine.

I would recommend using CRF + VBV for most ordinary encoding tasks if you're having issues with 2-pass + VBV.

[ACrowley]

In fact we are talking of the encoded movie...


maybe you read a few Popstings above:rolleyes:

Im talking about the reencode in Fact!

As i say, 2pass VBR with VBV Buffer 30000/ MaxBitrate 40000 works fine in all cases last Time
But not with Tuningmode -Grain "and or" Xmen Wolferin Origins BluRay Source.
So i was talking about the Source :)

[Fr4nz]

No, it's the 2-pass VBV algorithm. My one-pass is still fine.

I would recommend using CRF + VBV for most ordinary encoding tasks if you're having issues with 2-pass + VBV.

Do you plan to fix the 2-pass algorithm in the near future?

[nm]

A few posts back:
It seems I'll really have to spend some time investigating this weekend...

[Fr4nz]

A few posts back:

He didn't mention the VBV 2 pass algo.

[nm]

That was implied by "in the meantime, just use 1-pass mode instead", and he clarified it already in the next post that you quoted.

[techouse]

x264_x64_r1280_unpatched (http://techouse.project357.com/builds/revision1280/x264.exe) | MD5 (http://techouse.project357.com/builds/revision1280/x264.md5)
GCC 4.4.1 20090803 (x64.core2.Komisar), unpatched, generic, fprofiled

________________________________________________________________________________

x264_x86_r1280_techouse (http://techouse.project357.com/builds/x264_x86_r1280_techouse.7z) | INFO (http://techouse.project357.com/nfo/x264_x86_r1280_techouse.txt)
GCC 4.4.1 20090803 (x86.core2.Komisar), fprofiled, -march=core2

x264_x64_r1280_techouse (http://techouse.project357.com/builds/x264_x64_r1280_techouse.7z) | INFO (http://techouse.project357.com/nfo/x264_x64_r1280_techouse.txt)
GCC 4.4.1 20090803 (x86_64.core2.Komisar), fprofiled, -march=core2

Patches used:

x264_win_zone_parse_fix_06.diff

[JEEB]

Nice to see that the r16 works for some people :3 Edited and diff -NdurwBE'd the 1271 patch so that it patches (linky (http://jeeb.pastebin.ca/1601582)), let's see how it goes.

x264 r1281 64bit unpatched:
download (http://jeeb.fiveforty.jp/x264/revision1281/x264.exe) ; hash (http://jeeb.fiveforty.jp/x264/revision1281/x264.md5)

built on Oct 7 2009, gcc: 4.3.4 20090220 (prerelease) (x64.generic.Komisar)
fprofiled, otherwise defaults

________________________________________________________________________________

x264 r1281 32bit
download (http://jeeb.fiveforty.jp/x264/1281/x264.exe) ; release notes (http://jeeb.fiveforty.jp/x264/1281/relnotes.txt)

built on Oct 7 2009, gcc: 4.3.4 20090220 (prerelease) (x32.generic.Komisar)
fprofiled, -march=i686


x264 r1281 64bit
download (http://jeeb.fiveforty.jp/x264/1281_x64/x264.exe) ; release notes (http://jeeb.fiveforty.jp/x264/1281_x64/relnotes.txt)

built on Oct 7 2009, gcc: 4.3.4 20090220 (prerelease) (x64.generic.Komisar)
fprofiled, -march=core2


patched with:

x264_win_zone_parse_fix_06.diff (http://kemuri9.net/dev/x264/patches/x264_win_zone_parse_fix_06.diff)
x264_hrd_pd_interlace.16_r1281.diff (http://jeeb.pastebin.ca/1601582) (fixed a stupid error pointed by VFR Maniac)

[VFR maniac]

x264_hrd_pd_interlace.16_r1281.diff (http://jeeb.pastebin.ca/1601373)

The code is obviously wrong.
It should be the following.
@@ -1493,8 +1521,11 @@ static void update_vbv( x264_t *h, int bits )
if( rct->buffer_fill_final < 0 )
x264_log( h, X264_LOG_WARNING, "VBV underflow (frame %d, %.0f bits)\n", h->i_frame, rct->buffer_fill_final );
rct->buffer_fill_final = X264_MAX( rct->buffer_fill_final, 0 );
- rct->buffer_fill_final += rct->buffer_rate;
+ // we will honor vfr (when applicable) for the final buffer while prediction routines
+ // will continue to use the average frame rate
+ rct->buffer_fill_final += vfrlowhigh == 3 ? rcc->buffer_rate_vfr_high : rcc->buffer_rate_vfr_low;
rct->buffer_fill_final = X264_MIN( rct->buffer_fill_final, rct->buffer_size );
+ h->param.rc.i_buffer_fill_final = x264_clip3f( rct->buffer_fill_final, 0, rct->buffer_size - 1000 );
}

// provisionally update VBV according to the planned size of all frames currently in progress

[JEEB]

Ugh, that's one of the parts I manually edited. inb4 FFFFUUUUU.

Thanks and redoing. Now this is why I link my patches.

[EDIT]
I guess this one is fine then with your edit?
Linky (http://jeeb.pastebin.ca/1601582)

[Underground78]

Does this patch (http://mailman.videolan.org/pipermail/x264-devel/2009-October/006412.html) do the same thing than Trahald's one ?

[shon3i]

Does this patch (http://mailman.videolan.org/pipermail/x264-devel/2009-October/006412.html) do the same thing than Trahald's one ?
Yes but is broken

[rack04]

Yes but is broken

What is broken about it?

[shon3i]

What is broken about it?
http://forum.doom9.org/showthread.php?p=1326570#post1326570

I don't know is been updated since then.

[rack04]

http://forum.doom9.org/showthread.php?p=1326570#post1326570

I don't know is been updated since then.

It was updated Wed Oct 7 21:03:07 CEST 2009

Attached is the new version of the HRD patch with several changes due
to the inputs from Dark_Shikari, Manao and Trahald.
The two major fixes cpb_removal_delay for IDR frames and calculation
of initial_cpb_removal_delay.


Here is a build if you want to test:

x264 r1281M x86 (http://www.megaupload.com/?d=7T5VMMGS)

Built by rack04 on October 7, 2009, 3:24:36 PM CST
gcc.exe (GCC) 4.3.4 (x86.core2.Komisar)
--extra-cflags="-march=core2"
make fprofiled

Patched with:

x264-r1281-nal_hrd-pic_struct-v8.diff (http://mailman.videolan.org/pipermail/x264-devel/2009-October/006412.html)
x264_win_zone_parse_fix_06.diff (http://kemuri9.net/dev/x264/patches/x264_win_zone_parse_fix_06.diff)

[shon3i]

Very nice thanks :) I'm going now to test

[techouse]

x264_x64_r1281_unpatched (http://techouse.project357.com/builds/revision1281/x264.exe) | MD5 (http://techouse.project357.com/builds/revision1281/x264.md5)
GCC 4.4.1 20090803 (x64.core2.Komisar), unpatched, generic, fprofiled

________________________________________________________________________________

x264_x86_r1281_techouse (http://techouse.project357.com/builds/x264_x86_r1281_techouse.7z) | INFO (http://techouse.project357.com/nfo/x264_x86_r1281_techouse.txt)
GCC 4.4.1 20090803 (x86.core2.Komisar), fprofiled, -march=core2

x264_x64_r1281_techouse (http://techouse.project357.com/builds/x264_x64_r1281_techouse.7z) | INFO (http://techouse.project357.com/nfo/x264_x64_r1281_techouse.txt)
GCC 4.4.1 20090803 (x86_64.core2.Komisar), fprofiled, -march=core2

Patches used:

x264_win_zone_parse_fix_06.diff (http://kemuri9.net/dev/x264/patches/x264_win_zone_parse_fix_06.diff)
x264-r1281-nal_hrd-pic_struct-v8.diff (http://mailman.videolan.org/pipermail/x264-devel/attachments/20091007/cfbeacbe/attachment-0001.obj)

[shon3i]

Ok i do a deep test bettween x264_hrd_pd_interlace.16_r1281.diff and x264-r1281-nal_hrd-pic_struct-v8.diff (Rack04 vs JEEB build) and x264-r1281-nal_hrd-pic_struct-v8.diff is still broken. Elecard Buffer Analyser now scan whole stream and at the end show message "HRD parameters are not presented", while Scenarist MUI Generator just give ERROR message. 264_hrd_pd_interlace.16_r1281.diff still produce compilant stream :)

btw cmd i used for testing:

-profile high --level 4.1 --pass 2 --bitrate 8000 --stats ".stats" --thread-input --deblock -2:-2 --keyint 24 --min-keyint 2 --b-adapt 2 --direct auto --ref 4 --slices 4 --ipratio 1.1 --pbratio 1.1 --vbv-bufsize 30000 --vbv-maxrate 30000 --rc-lookahead 60 --aq-mode 2 --me umh --partitions all --trellis 2 --psy-rd 1.0:0.2 --no-dct-decimate --no-fast-pskip --sar 1:1 --aud --nal-hrd --output "output" "input"

[ACrowley]

I made a few encodes with CRF 18. Im wondering that the average Bitarte is around 6-8 Mbps with CRF 18 ? Isn it a litte bit low ? Quants are fine so far

[LoRd_MuldeR]

I made a few encodes with CRF 18. Im wondering that the average Bitarte is around 6-8 Mbps with CRF 18 ? Isn it a litte bit low ? Quants are fine so far

Why do you complain about too low bitrate? You can't expect a specific bitrate from a specific CRF value, because CRF highly depends on the source.

So if the quality is okay at CRF 18 and 6-8 Mbps, then be happy :)

And if the quality is not okay for you at CRF 18, simply use an even lower CRF value. The CRF should be chosen by target quality, not by target bitrate.

For target bitrate encodes there is 2-Pass mode ...

[ACrowley]

Why do you complain about too low bitrate? You can't expect a specific bitrate from a specific CRF value, because CRF highly depends on the source.

So if the quality is okay at CRF 18 and 6-8 Mbps, then be happy :)

And if the quality is not okay for you at CRF 18, simply use an even lower CRF value. The CRF should be chosen by target quality, not by target bitrate.

For target bitrate encodes there is 2-Pass mode ...

im not complainig. I just wondering because i was expecting higher Bitrate with CRF 18. I usually encode 1080p with 8-15Mbps.

[LoRd_MuldeR]

im not complainig. I just wondering because i was expecting higher Bitrate with CRF 18. I usually encode 1080p with 8-15Mbps.

1. As said before, CRF highly depends on the source. So your current source may simply need fewer bits to look good.

2. MB-Tree RC redefined the CRF scale and generally reduces the bitrate requirement (that is: it retains better quality at same bitrate, or same quality at lower bitrate).

Maybe you are comparing a MB-Tree RC encode to an old encode that was done without MB-Tree ???

[ACrowley]

1. As said before, CRF highly depends on the source. So your current source may simply need fewer bits to look good.

2. MB-Tree RC redefined the CRF scale and generally reduces the bitrate requirement (that is: it retains better quality at same bitrate, or same quality at lower bitrate).

Maybe you are comparing a MB-Tree RC encode to an old encode that was done without MB-Tree ???

Its latest encode from Xmen Origins Wolferine Bluray.
The Source incldues a lot of Scenes with heavy Grain so i was suprised that the average Bitrate/Filesize is so low with crf 18.

Yes, some old encdes without MBtree have higher Bitrate/Size at CRF 18 so i cant compare it with ne MBtree encodes.

Maybe i should go down to crf 16 or 17

THX for your help

[LoRd_MuldeR]

Maybe i should go down to crf 16 or 17

Nobody can tell you what CRF values give acceptable quality for you, because this is highly subjective!

Simply use the highest possible CRF value that still gives good results for your eyes. If CRF 18 looks good for you, then no need to go any lower.

But if it looks bad for you, don't hesitate to try CRF 17 or 16 ;)

[ACrowley]

Nobody can tell you what CRF values give acceptable quality for you, because this is highly subjective!

Simply use the highest possible CRF value that still gives good results for your eyes. If CRF 18 looks good for you, then no need to go any lower.

But if it looks bad for you, don't hesitate to try CRF 17 or 16 ;)

Im very satisfied with my 2pass VBR encodes which are mostly around 10-12Mbps average Bitrate.

I want the same Quality/Size. CRF was only a thought because it works faster. I will test it .
THX

[LoRd_MuldeR]

Im very satisfied with my 2pass VBR encodes which are mostly around 10-12Mbps average Bitrate.

I want the same Quality/Size. CRF was only a thought because it works faster. I will test it .
THX

Again: If you use CRF mode, then choose your CRF value with a specific level of quality in mind, not with a specific target bitrate in mind!

So you should adjust the CRF value until you get the desired quality instead of modifying it until you hit a specific bitrate. For the latter case we have 2-Pass mode :D

[komisar]

Hey, guys, gcc-miscompiling-saga again...
I have tested unpatched builds again and found the following results:
1281 builds:
x264.nl gcc 3.4.6 -- OK
BugMaster gcc 4.3.3 TDM -- OK
BugMaster gcc 4.3.4 20090220 (prerelease) (x32.generic.Komisar) (not profiled) -- OK
Komisar gcc 4.3.4 (x86_64.core2.Komisar) -- OK
Komisar gcc 4.3.4 (x86.generic.Komisar) -- OK
Komisar gcc 4.4.1 (x86.core2.Komisar) -- FAILURE
JEEB gcc 4.3.4 20090220 (prerelease) (x64.generic.Komisar) -- FAILURE
techouse gcc 4.4.1 (x86_64.core2.Komisar) -- FAILURE
rack04 (patched build) gcc 4.3.4 (x86.core2.Komisar) -- FAILURE


gcc sucks again... :eek:

[JEEB]

Oh nice~

Thank you for your information, I'll get to building a new GCC and a new x264 as soon as I'll get home :/ Kind of hatin' the whole miscompilation stuff.

In short: *censored* (Holy Hanyuu I hate these cases)

[kemuri-_9]

what is the problem that is occurring in x264 with these 'broken' builds?
and what is the issue in gcc that's causing them

[LoRd_MuldeR]

Komisar, how exactly do you test the builds? And does "FAILURE" mean crash, bad output or what? How would I realize? :confused:

BTW: Did anybody else test GCC 4.4.1 TDM-2 yet?

[MythCreator]

My build is compile by TDM GCC。。。And what things are failure on 4.4.X?

[LoRd_MuldeR]

My build is compile by TDM GCC。。。And what things are failure on 4.4.X?

That's what I wonder too. There are basically two things that can happen:

1. Unexpected crash -> easy to detect and easy to fix (re-build until it stops crashing)
2. Bad output -> hard to detect, unless the output is broken so bad that it causes obvious artifacts

Since (1) does happen with my GCC 4.4.1 TDM-2 builds and there also is no obvious indication for (2), I'd assume these builds are okay.
But there may be a more subtle problem. So how do I check properly ???

[techouse]

Yea, I wanna know that too. :/

[MasterNobody]

LoRd_MuldeR, techouse
If you were at x264dev IRC yesterday you would know that it doesn't cause obvious artifacts. It is caused by used addressing in x264 and gcc. The quick workaround is replacing of all dct2x2[0][coeff] with dct2x2[coeff>>1][coeff&1] in macroblock.c

[komisar]

Test made by "%x% --threads 1 --ssim -o %x%.264 TestVideo.y4m"

"FAILURE" mean BIG difference (above 5 kilobytes) (FAILURE) with x264.nl x32 build... :)
Raw stream of "OK"-bulds is identical with x264.nl x32 build...

[kemuri-_9]

we discussed the issue on #x264dev last night and discovered the discrepancy in the output and the underlying cause in the code.

the underlying code was found in
x264_mb_optimize_chroma_dc within encoder/macroblock.c

the problem with the output not being bitwise equivalent is that the chroma macroblocks can be slightly different from what they should be.

the visual difference is generally unnoticeable and there's a slight bitrate difference when encoding in CRF mode.

the method to test if your build has this issue:
1. take a build that was listed as 'OK' by komisar and use it as a reference
2. ./x264_reference --qp 24 --ipratio 1 -o a1.h264 <input>
3. ./x264_reference --qp 25 --ipratio 1 -o a2.h264 <input>
4. ./x264_your_own --qp 24 --ipratio 1 -o b1.h264 <input>
5. ./x264_your_own --qp 25 --ipratio 1 -o b2.h264 <input>

if your build has problems then
a1.h264 & b1.h264 should be different
a2.h264 & b2.h264 should be bitwise equivalent

the problem has only been reported for mingw as of yet, so it's either only affecting mingw or no one's just tested for other platforms.

[tph]

dct2x2[2][2] could also be changed into dct2x2[4] instead (after adjusting all relevant code), at least that seems to do the trick for me and should theoretically be faster than doing some bit manipulation. The same probably holds for all other DCT arrays.

[kemuri-_9]

dct2x2[2][2] could also be changed into dct2x2[4] instead (after adjusting all relevant code), at least that seems to do the trick for me and should theoretically be faster than doing some bit manipulation. The same probably holds for all other DCT arrays.

that's the plan of attack as it already stands:

<kemuri-_9> so what is going to be done to fix the issue?
<pengvado> eliminate multidimensional arrays in any context that needs to iterate over multiple dimensions
<pengvado> if the standard doesn't allow that feature to work, then we can't use it
<pengvado> you know, most of the time it's not actually useful to treat dct as 2d...

[techouse]

LoRd_MuldeR, techouse
If you were at x264dev IRC yesterday you would know that it doesn't cause obvious artifacts. It is caused by used addressing in x264 and gcc. The quick workaround is replacing of all dct2x2[0][coeff] with dct2x2[coeff>>1][coeff&1] in macroblock.c
OK, I'll use this quick workaround and post 3 builds in a few minutes.

EDIT:

x264_x64_r1281_fix_unpatched (http://techouse.project357.com/builds/revision1281fix/x264.exe) | MD5 (http://techouse.project357.com/builds/revision1281fix/x264.md5)
GCC 4.4.1 20090803 (x64.core2.Komisar), unpatched, generic, fprofiled

Patches used:

the quick GCC 4.4.x workaround (http://pastebin.ca/1612400)

________________________________________________________________________________

x264_x86_r1281_fix_techouse (http://techouse.project357.com/builds/x264_x86_r1281_fix_techouse.7z) | INFO (http://techouse.project357.com/nfo/x264_x86_r1281_fix_techouse.txt)
GCC 4.4.1 20090803 (x86.core2.Komisar), fprofiled, -march=core2

x264_x64_r1281_fix_techouse (http://techouse.project357.com/builds/x264_x64_r1281_fix_techouse.7z) | INFO (http://techouse.project357.com/nfo/x264_x64_r1281_fix_techouse.txt)
GCC 4.4.1 20090803 (x86_64.core2.Komisar), fprofiled, -march=core2

Patches used:

x264_win_zone_parse_fix_06.diff (http://kemuri9.net/dev/x264/patches/x264_win_zone_parse_fix_06.diff)
x264_hrd_pd_interlace.16_r1281.diff (http://jeeb.pastebin.ca/1601582)
the quick GCC 4.4.x workaround (http://pastebin.ca/1612400)

[burfadel]

Is this issue still present in the GCC 4.5.0 beta?

Using the latest ffdshow and mpc-hc from xvidvideo.ru both of which are compiled with gcc 4.5.0, they seem to work fine...

[Kurtnoise]

Keep in mind that mpc-hc uses lavc decoder, not x264...

[LoRd_MuldeR]

LoRd_MuldeR, techouse
If you were at x264dev IRC yesterday you would know that it doesn't cause obvious artifacts. It is caused by used addressing in x264 and gcc. The quick workaround is replacing of all dct2x2[0][coeff] with dct2x2[coeff>>1][coeff&1] in macroblock.c

Test made by "%x% --threads 1 --ssim -o %x%.264 TestVideo.y4m"

"FAILURE" mean BIG difference (above 5 kilobytes) (FAILURE) with x264.nl x32 build... :)
Raw stream of "OK"-bulds is identical with x264.nl x32 build...

Thanks for the info!

But since I build x264 as a shared library, I guess testing will be more difficult for me, because there are more places where things can go wrong and I don't have a reference build :eek:

Anyway, could anybody be so kind and post a patch with the "quick workaround" for me :thanks:

[techouse]

Anyway, could anybody be so kind and post a patch with the "quick workaround" for me :thanks:

Here you go :)

http://pastebin.ca/1612400

[LoRd_MuldeR]

Here you go :)

http://pastebin.ca/1612400

That's all? Thanks for the patch :D

[techouse]

That's all? Thanks for the patch :D
Yea, well, altogether 4 entries had to be replaced in a single file, so yea, that's all :D

[shon3i]

OK, I'll use this quick workaround and post 3 builds in a few minutes.

EDIT:

x264_x64_r1281_fix_unpatched (http://techouse.project357.com/builds/revision1281fix/x264.exe) | MD5 (http://techouse.project357.com/builds/revision1281fix/x264.md5)
GCC 4.4.1 20090803 (x64.core2.Komisar), unpatched, generic, fprofiled

Patches used:

the quick GCC 4.4.x workaround (http://pastebin.ca/1612400)

________________________________________________________________________________

x264_x86_r1281_fix_techouse (http://techouse.project357.com/builds/x264_x86_r1281_fix_techouse.7z) | INFO (http://techouse.project357.com/nfo/x264_x86_r1281_fix_techouse.txt)
GCC 4.4.1 20090803 (x86.core2.Komisar), fprofiled, -march=core2

x264_x64_r1281_fix_techouse (http://techouse.project357.com/builds/x264_x64_r1281_fix_techouse.7z) | INFO (http://techouse.project357.com/nfo/x264_x64_r1281_fix_techouse.txt)
GCC 4.4.1 20090803 (x86_64.core2.Komisar), fprofiled, -march=core2

Patches used:

x264_win_zone_parse_fix_06.diff (http://kemuri9.net/dev/x264/patches/x264_win_zone_parse_fix_06.diff)
x264-r1281-nal_hrd-pic_struct-v8.diff (http://mailman.videolan.org/pipermail/x264-devel/attachments/20091007/cfbeacbe/attachment-0001.obj)
the quick GCC 4.4.x workaround (http://pastebin.ca/1612400)
techouse please don't use x264-r1281-nal_hrd-pic_struct-v8.diff because can't produce compilant BD streams. Obviously you don't read this topic cearfully. For now use x264_hrd_pd_interlace.16_r1281.diff or not use any hrd patch.

[techouse]

techouse please don't use x264-r1281-nal_hrd-pic_struct-v8.diff because can't produce compilant BD streams. Obviously you don't read this topic cearfully. For now use x264_hrd_pd_interlace.16_r1281.diff or not use any hrd patch.Sorry, I must have missed it! I'll get on it straight away.

EDIT:
Fixed. The links to the builds remain the same, the binareis have changed, so please RE-download them.

[Dark Shikari]

techouse please don't use x264-r1281-nal_hrd-pic_struct-v8.diff because can't produce compilant BD streams. Obviously you don't read this topic cearfully. For now use x264_hrd_pd_interlace.16_r1281.diff or not use any hrd patch.Isn't the first patch the one sent out by Alex Giladi? I recall an earlier version of that one passed official Sony BD Verifier tests (newer hasn't been tested yet).

[ACrowley]

Is there a special Reason why i get his Warning with the fixed rev1281 Techhouse Build ?
x264 [warning]: NAL HRD parameters require VBV max bitrate and buffer size to be specified

I never got this Warning before. My cmd is without nal hrd Parameter. I dont think nal hrd is enabled by default/autom(maybe through Level 4.1 Settings) ?
--preset veryslow --tune grain --crf 18.0 --level 4.1 --deblock -3:-3 --ref 5 --no-fast-pskip --partitions p8x8,b8x8,i4x4,i8x8 --subme 9 --bframes 3

Questions over Questions the last Days....

[shon3i]

Isn't the first patch the one sent out by Alex Giladi? I recall an earlier version of that one passed official Sony BD Verifier tests (newer hasn't been tested yet).
Yes, i think is from Alex Giladi, but Elecard and Sonic reject streams completly, while Trahald's work fine up to rev16, because in later rev's (19) changed some calculations which crash x264 with GOP 24