x264 development

[LigH]

@ MatLz:

No, your posts are not invisible, at least not to me (but I am no developer).

Maybe try IRC #x264 on FreeNode too.

[kemuri-_9]

But as my posts seem invisible...you won't care, right ?

your posts are not invisible, but i would like if you reported benchmarks comparing this last build against the standard pthread-win32 build.

both normal (frame-based) threading and sliced-threading should be benchmarked.

[mariush]

Can you give some command line to test for slice threading... never tried it so I don't know how it's done.

windows 2003 r2 , 32 bit. 30sec mpeg2 video


--preset medium --tune film --bitrate 1024 -o film.mkv "Video1108-2159(TV35).mpg"
ffms [info]: 768x576p 1:1 @ 25/1 fps (vfr)
x264 [info]: using SAR=1/1
x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 Cache64
x264 [info]: profile High, level 3.1
x264 [info]: frame I:9 Avg QP:24.50 size: 19309
x264 [info]: frame P:636 Avg QP:26.00 size: 5636
x264 [info]: frame B:137 Avg QP:29.16 size: 1556
[...]
x264 [info]: final ratefactor: 24.58
x264 [info]: Weighted P-Frames: Y:3.1%
x264 [info]: ref P L0: 54.3% 12.7% 22.9% 10.0% 0.2%
x264 [info]: ref B L0: 76.8% 22.3% 0.8%
x264 [info]: ref B L1: 96.6% 3.4%
x264 [info]: kb/s:1015.79


1745 default: encoded 782 frames, 51.23 fps, 1015.97 kb/s
1745 patch : encoded 782 frames, 50.76 fps, 1015.97 kb/s


--preset slow --tune film --bitrate 1024 -o film.mkv "Video1108-2159(TV35).mpg"
ffms [info]: 768x576p 1:1 @ 25/1 fps (vfr)
x264 [info]: using SAR=1/1
x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 Cache64
x264 [info]: profile High, level 3.1
x264 [info]: frame I:7 Avg QP:24.57 size: 23178
x264 [info]: frame P:244 Avg QP:25.71 size: 8813
x264 [info]: frame B:531 Avg QP:26.83 size: 3194
[...]
x264 [info]: final ratefactor: 24.31
x264 [info]: Weighted P-Frames: Y:7.0%
x264 [info]: ref P L0: 43.7% 11.8% 21.8% 8.1% 9.0% 5.5% 0.2%
x264 [info]: ref B L0: 66.5% 21.6% 9.3% 2.6%
x264 [info]: ref B L1: 89.3% 10.7%
x264 [info]: kb/s:1025.13


1745 default: encoded 782 frames, 28.71 fps, 1025.31 kb/s
1745 patch : encoded 782 frames, 28.63 fps, 1025.31 kb/s

update sliced threads


--preset slow --sliced-threads --tune film --bitrate 1024 -o film1.mkv "Video1108-2159(TV35).mpg"
ffms [info]: 768x576p 1:1 @ 25/1 fps (vfr)
x264 [info]: using SAR=1/1
x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 Cache64
x264 [info]: profile High, level 3.1
x264 [info]: frame I:7 Avg QP:24.69 size: 23125
x264 [info]: frame P:243 Avg QP:25.87 size: 8809
x264 [info]: frame B:532 Avg QP:26.99 size: 3223
[...]
x264 [info]: final ratefactor: 24.50
x264 [info]: Weighted P-Frames: Y:5.8%
x264 [info]: ref P L0: 43.6% 12.0% 21.3% 8.3% 9.1% 5.6% 0.2%
x264 [info]: ref B L0: 66.9% 21.1% 9.3% 2.7%
x264 [info]: ref B L1: 89.5% 10.5%
x264 [info]: kb/s:1027.34


1745 default: encoded 782 frames, 20.49 fps, 1027.53 kb/s
1745 patch : encoded 782 frames, 20.31 fps, 1027.53 kb/s

the patch is consistently slower than the default one. Not sure the fps is correct on the patched version as fps in the last case was for 90% of time at about 17-8.5% and at the end goes up to 20% and then at the end shows 20.31 fps.

files on the patched version are 6 bytes longer in all cases as far as I see.

[Dark Shikari]

--sliced-threads (surprise!)

[Underground78]

Windows XP SP3 (32 bits)
--no-progress --output NUL test.avs
avs [info]: 688x368p 0:0 @ 25/1 fps (cfr)

default build:
encoded 1000 frames, 30.70 fps, 874.79 kb/s
encoded 1000 frames, 30.68 fps, 874.79 kb/s

w32thread build:
encoded 1000 frames, 30.89 fps, 874.79 kb/s
encoded 1000 frames, 30.78 fps, 874.79 kb/s

--sliced-threads --no-progress --output NUL test.avs
avs [info]: 688x368p 0:0 @ 25/1 fps (cfr)

default build:
encoded 1000 frames, 27.15 fps, 885.06 kb/s
encoded 1000 frames, 27.25 fps, 885.06 kb/s

w32thread build:
encoded 1000 frames, 26.81 fps, 885.06 kb/s
encoded 1000 frames, 26.85 fps, 885.06 kb/s

It looks like your test build is slightly slower with sliced-threads enabled.

[kemuri-_9]

It looks like your test build is slightly slower with sliced-threads enabled.

there does seem to be a growing consensus on this...
tho frame-based threading still seems to be rather inconclusive.

[jpsdr]

I've encoded several files with last version (1766), with QP file like this :

2172 I
16309 I
32532 I
48514 I
62530 I
77288 I
92527 I
108749 I
122511 I
136382 I
152510 I
154670 I
155375 I

What was my surprise when i've tried to make chapters with scenarist, with a lot of warning for each chapters saying that frame wasn't an I frame !
Encoding with previous version (1745) worked fine. It seems that QP file are somehow broken.
I'm using Jeeb's version, with following commande line :

REM Set of max bitrate (ici le bitrate max)
set MAX_BR=15000

REM Set of Buffer (ici le buffer)
set BUF_BR=15000

REM 1ère passe
x264_x64.exe --profile "high" --preset "placebo" --tune %TUNING% --pass 1 --bitrate %2 --stats %STAT_FILE% --level "4.0" --qpmin 0 --vbv-maxrate %MAX_BR% --vbv-bufsize %BUF_BR% --keyint 48 --min-keyint 2 --mvrange 511 --ref 6 --bframe 3 --b-pyramid "strict" --open-gop "bluray" --fade-compensate 0.7 --subme 7 --me "umh" --aq-mode 2 --aud --nal-hrd "vbr" --colorprim "bt709" --transfer "bt709" --colormatrix "bt709" --videoformat "ntsc" --sar 1:1 --qpfile %CHAPTERS% --threads 0 --thread-input --output NUL %E_SRC% 2> %LOG_FILE_1%

REM 2ème passe
x264_x64.exe --profile "high" --preset "placebo" --tune %TUNING% --pass 2 --bitrate %2 --stats %STAT_FILE% --level "4.0" --qpmin 0 --vbv-maxrate %MAX_BR% --vbv-bufsize %BUF_BR% --keyint 48 --min-keyint 2 --mvrange 511 --ref 6 --bframe 3 --b-pyramid "strict" --open-gop "bluray" --fade-compensate 0.7 --aq-mode 2 --aud --nal-hrd "vbr" --colorprim "bt709" --transfer "bt709" --colormatrix "bt709" --videoformat "ntsc" --sar 1:1 --qpfile %CHAPTERS% --threads 0 --thread-input --output %E_DST% %E_SRC% 2> %LOG_FILE_2%

Note : Tune was animation.

[Dark Shikari]

Seems I didn't actually fully fix it to make the qpfile work without the -1s. Use the -1s like before in the meantime, I'll fix it in the next push.

[jpsdr]

Argh... 48h of encoding wasted...:(

[Dark Shikari]

Argh... 48h of encoding wasted...:(Sorry, I fail extremely badly. Not only did I screw it up, but I'm so bad at string parsing I can't actually get it to work now either.

[jpsdr]

Do you mean with -1 it's still not working with 1766 release ?
If it's the case, please tell me, i'll stop the encoding i've restarted, there no point letting my PC run if it's "for nothing"...

[Underground78]

Do you mean with -1 it's still not working with 1766 release ?
If it's the case, please tell me, i'll stop the encoding i've restarted, there no point letting my PC run if it's "for nothing"...

I am quite sure it still does not work in 1766 but it should be fixed in r1768 (http://git.videolan.org/gitweb.cgi?p=x264.git;a=commit;h=39787c80bdd5992935c480a8eedcfb6d50625334).

[jpsdr]

I've just checked with one of my encoded files, with -1 added it was working fine.
But it seems that there is a new version on it's way with some other fixes (at least something on HRD), so i'll wait next Jeeb release.

[Underground78]

I've just checked with one of my encoded files, with -1 added it was working fine.
But it seems that there is a new version on it's way with some other fixes (at least something on HRD), so i'll wait next Jeeb release.

Yes, my bad, I misread your post ... :o

[rack04]

Is anyone able to download the latest git? Maybe the site is having problems because I get an error at http://git.videolan.org/

[aegisofrime]

Is anyone able to download the latest git? Maybe the site is having problems because I get an error at http://git.videolan.org/

It was broken a while back but ok now. Got me worried that the x264 project might be dead :(

Edit: I noticed that the latest revision features improvements regarding high bit depth. For the noobs among us (me), does this refer to 10-bit output and if so, how's the speed relative to 8-bit?

[burfadel]

Just wondering whether the following AMD programmers manual (including AVX, XOP, FMA4 etc) is of any use:
http://support.amd.com/us/Processor_TechDocs/26568.pdf

[thegame]

I am confused on the format for the qp file, is this right

1212 I

or is this right

1212 I -1

Thanks

and is there a way to verify your I-Frame placement once x264 MeGUI encoded?

[Dark Shikari]

The quantizer (rightmost value) is optional.

[thegame]

The quantizer (rightmost value) is optional.

OK so -1 or no -1 it is all good either way, but is there a way to check the I-Frames in the raw AVC x264 encoded file afterwords? is there anything that will let you check and make sure they are correct?

OH and Thank you by the way for the quick response.

[Dark Shikari]

OK so -1 or no -1 it is all good either way, but is there a way to check the I-Frames in the raw AVC x264 encoded file afterwords? is there anything that will let you check and make sure they are correct?

OH and Thank you by the way for the quick response.The statsfile from a multi-pass encode, or a stream analyzer.

[mariush]

AMD’s software optimization guide for their upcoming Bulldozer processor: http://support.amd.com/us/Processor_TechDocs/47414.pdf

It seems that AMD has released almost all of the details of it’s new architecture to the world and after quickly skimming it some of the features are mildly impressive.
Let’s hope AMD matches it’s new chip with a decent marketing campaign, maybe something like this video: http://www.youtube.com/watch?v=usGkq7tAhfc

(copy/pasted from semiaccurate.com, added links)

DS, any interesting stuff there that would encourage one to choose 2/4 socket AMD systems for their next encoding system instead of Intel?

[aegisofrime]

AMD’s software optimization guide for their upcoming Bulldozer processor: http://support.amd.com/us/Processor_TechDocs/47414.pdf

It seems that AMD has released almost all of the details of it’s new architecture to the world and after quickly skimming it some of the features are mildly impressive.
Let’s hope AMD matches it’s new chip with a decent marketing campaign, maybe something like this video: http://www.youtube.com/watch?v=usGkq7tAhfc

(copy/pasted from semiaccurate.com, added links)

DS, any interesting stuff there that would encourage one to choose 2/4 socket AMD systems for their next encoding system instead of Intel?

Thanks for that question, I was thinking of that as well.

[burfadel]

Revision 1936 seems to try to use internal FFMS etc even if specified to use avisynth input. It therefore crashes.

This is using x264 from www.x264.nl (had to use the second mirror).

[burfadel]

Anyone had the same issue? running rev 1924 work perfectly fine, but 1936 returns the input error regardless of the avisynth source filter used (ffvideosource, mpeg2source etc). Is related to the way the version of x264.nl is compiled (and related to why its not listed on mirror01, so mirror02 etc has to be used), or is this an error that also affects other build environments?

[kemuri-_9]

what issue are you trying to report, exactly?

first you say that you can't force avisynth input with --demuxer avs, but that is working.

and then you come back and say you get an error, without posting any useful information about it.

[burfadel]

Sorry, was in a rush earlier!

It was probably why I missed the fact the command line options have changed again, and I was using Staxrip (for easy batch processing). I had --open-gop normal set, which has been changed to --open-gop and with --b-pyramid now having the 'normal' setting which is set as default.

If anyone else is caught out by this, its easy enough to change in staxrip, just set open-gop to none (so it doesn't appear in the command line down the bottom) and then on the 'command-line' tab, enter --open-gop

[Dark Shikari]

Whoops, should have warned the GUI authors I was changing one of the options ;)

[upyzl]

Re #1377

Maybe this is the reason?
commit f9e7f531048df7219a44203031c8f79bae6170d0 r1935
Author: Jason Garrett-Glaser <jason@x264.com>
Date: Wed Apr 6 17:15:50 2011 -0700

Consolidate Blu-ray hacks into --bluray-compat
This option is now required for Blu-ray compatibility.
--open-gop bluray is now gone (using bluray-compat and open-gop implies a Blu-ray compatible open-gop).
This option doesn't automatically enforce every aspect of Blu-ray compatibility (e.g. resolution, framerate, level, etc).

[CruNcher]

Wow at last, i remember when we had this discussion you where against device specific profiles inside the CMD Encoder and said that's the job of the GUI guys and libx264 implementer, seems that thinking changed finally, though looking in what the CMD Encoder has changed into over the years it was just a logic decision :D

[Dark Shikari]

Wow at last, i remember when we had this discussion you where against device specific profiles inside the CMD Encoder and said that's the job of the GUI guys and libx264 implementer, seems that thinking changed finally, though looking in what the CMD Encoder has changed over the years it was just a logic decision :DActually, nothing changed at all. --bluray-compat DOES NOT set all options for Blu-ray compatibility. It simply enables Blu-ray-specific hacks.

It will not set the resolution for you, or your framerate, or your bitrate, or any other things required by Blu-ray.

[Midzuki]

Hummmm, x264 is getting better and better...

and somewhat fatter :devil: as well :D

@burfadel: don't use GUIs for x264.exe :p

[Groucho2004]

Hummmm, x264 is getting better and better...
Yep.

and somewhat fatter :devil: as well :D

Really? Haven't noticed. Mine is still at ~778K.

[shon3i]

Actually, nothing changed at all. --bluray-compat DOES NOT set all options for Blu-ray compatibility. It simply enables Blu-ray-specific hacks.

It will not set the resolution for you, or your framerate, or your bitrate, or any other things required by Blu-ray.
Can you be more specific what hacks? what settings is affected, beside opengop why b-piramid is not in this aslo since strict is de facto name for bluray

[Dark Shikari]

Can you be more specific what hacks? what settings is affected, beside opengop why b-piramid is not in this aslo since strict is de facto name for blurayStrict is useful for temporal scalability in general, not just Blu-ray.

Blu-ray hacks:

1) min-CR + level 4.1 hack
2) Special b-pyramid SEI
3) B-frames cannot reference frames outside their minigop
4) Open-GOP keyframe interval hack

[sneaker_ger]

1) min-CR + level 4.1 hack

What's that?


In general I must say I don't like these "half-assed" options. It's sane to force any Blu-Ray specific hacks with an additional parameter, but overall the x264 options don't follow any straight line:

--profile is forced, overriding any explicit options
--level sets the level flag + reduce number of ref frames, but only if not explicitly forced by the user, meaning that you can falsely flag your files. It also doesn't care for frame sizes or vbv.
--bluray-compat now limits e.g. ref frames to 6, which will work for 720p, but not for 1080p etc.

This inconsistency of the options leads to confusion and the fact that these things are not documented in --fullhelp, but have to be read on doom9 or the git, doesn't really help either.

[Dark Shikari]

What's that?Blu-ray spec specifies a different MinCR than the H.264 spec for level 4.1.

In general I must say I don't like these "half-assed" options. It's sane to force any Blu-Ray specific hacks with an additional parameter, but overall the x264 options don't follow any straight line:

--profile is forced, overriding any explicit options
--level sets the level flag + reduce number of ref frames, but only if not explicitly forced by the user, meaning that you can falsely flag your files. It also doesn't care for frame sizes or vbv.
--bluray-compat now limits e.g. ref frames to 6, which will work for 720p, but not for 1080p etc.

This inconsistency of the options leads to confusion and the fact that these things are not documented in --fullhelp, but have to be read on doom9 or the git, doesn't really help either.In short: Blu-ray compat limits analysis options, not bitrate/resolution/fps. Is that simple enough for you?

If you want to help, you can work on cleaning up and finishing the --device/autolevel patch, which will simplify and clean up all this.

[sneaker_ger]

Blu-ray spec specifies a different MinCR than the H.264 spec for level 4.1.

Ah, I just recalled reading about that before. Thanks.

In short: Blu-ray compat limits analysis options, not bitrate/resolution/fps. Is that simple enough for you?

Don't get me wrong. I was and will be able to put up correct and sane command lines for x264cli. I just feel that inconsistency leads to confusion (not of me, who spends lots of time on doom9) but of others, and that things should be done right or not at all.

If you want to help, you can work on cleaning up and finishing the --device/autolevel patch, which will simplify and clean up all this.

I'm afraid I'm not qualified for that, but it's good to know that some of the devs might feel similar and that x264 gets more refined every day.

[burfadel]

@burfadel: don't use GUIs for x264.exe :p

lol! not a bad point, unfortunately thats easier said than done! Now if there were a x264 specific gui that was easy to use allowed for batch processing, easy settings adjustment, external audio encoders, cutting for personal tv recorded episodes/movies, including outputting the cut info to an external audio encoder, output to mkv, parallel batch encoding (there's a way to do that in Staxrip), 'segmented single file encoding', and other useful things then I'd be using that instead :D

What I meant by 'segmented file encoding', is scanning around halfway in a file to be encoded for a keyframe change, then encoding the first half of the file to that keyframe change and the second half of the file from that keyframe change in parallel, at low priority, which will allow for full cpu use across all CPU's (even 8 logical/physical such as in Intel 'Sandy Bridge' or AMD 'Bulldozer'), whilst not impending on normal computer use. The two halves can then be appended at the end of the encode, much like you can do in mkvtools except operating on the raw streams. The audio stream could even be added whilst doing the append :).

[CruNcher]

burfadel you want a professional solution like Cinevision or CCE-HD, Blu-Code ect that though yet doesn't exists with x264 @ its core the most pro solution is Pegasys Masterworks which in some parts fails in its framework entirely compared to what you can achieve manually with the right tools ;)

[laserfan]

Blu-ray hacks:

1) min-CR + level 4.1 hack
2) Special b-pyramid SEI
3) B-frames cannot reference frames outside their minigop
4) Open-GOP keyframe interval hack
So apart from --open-gop bluray being moved into this, the --bluray-compat option includes 3 other new features that did not heretofore exist with the base builds of x264? :confused:

[J_Darnley]

Except for 2 and 3 they did. Read the changes to see how they were activated.

[CruNcher]

Is it known what the compression loss and or win in encoding speed/decoding of --bluray-compat will be compared to leaving it out and if their could be other interoperability problems occurring with that option for non Blu-Ray scenarios, will it be significant @ all also PSNR/SSIM and PSY ?

[shon3i]

--bluray-compat should has no impact on quality or de/encoding speed. min-CR + level 4.1 hack is need to accomplish bd specs, and is not need for level 4.0 and lover, and others are just pure hacks to meat exact bd specification and pass verifiers.

[Dark Shikari]

--bluray-compat should has no impact on quality or de/encoding speed. min-CR + level 4.1 hack is need to accomplish bd specs, and is not need for level 4.0 and lover, and others are just pure hacks to meat exact bd specification and pass verifiers.The B-frame reference crippling costs a non-trivial amount of compression.

[rallymax]

Strict is useful for temporal scalability in general, not just Blu-ray.

Blu-ray hacks:

1) min-CR + level 4.1 hack
2) Special b-pyramid SEI
3) B-frames cannot reference frames outside their minigop
4) Open-GOP keyframe interval hack

Hi Dark',

does 1) imply that it sets the level at 4.1 too? (ie can you have --bluray-compat and a --level xx setting on the command line?

[sneaker_ger]

Hi Dark',

does 1) imply that it sets the level at 4.1 too? (ie can you have --bluray-compat and a --level xx setting on the command line?

If I'm not mistaken it does not.

Also the code (http://git.videolan.org/gitweb.cgi?p=x264.git;a=blobdiff;f=encoder/ratecontrol.c;h=6726fbd1f6e39dd53df2cc149c54e01de38d5915;hp=ce9a93effee77ca5d666b80cc0ba606fab798fb7;hb=f9e7f531048df7219a44203031c8f79bae6170d0;hpb=f422ec93254ed3f9883acac0bb3f67e3b4ea960c) doesn't seem to be caring about the level at all, which makes me wonder if this is correct for levels other than 4.1. :confused:

/edit:
seems that this has already been discussed:
2011-04-07 00:29:22 < kierank> line 1211 the mincr thing is only for level 4.1
2011-04-07 00:29:32 < Dark_Shikari> yes because all the other levels already have mincr 4
2011-04-07 00:29:33 < Dark_Shikari> right?
2011-04-07 00:29:45 < kierank> can't remember
2011-04-07 00:29:46 < Dark_Shikari> or is it ,in the spec, only for level 4.1?
2011-04-07 00:29:47 < Dark_Shikari> check it
2011-04-07 00:32:23 < kierank> yeah all the other levels that are used in blu-ray have mincr=4

[shon3i]

imply that it sets the level at 4.1 too?No. If you use --level 4.1 and --bluray-compat then x264 will use MinCR that accomplish bd specs, otherwise will use AVC spec

[rallymax]

No. If you use --level 4.1 and --bluray-compat then x264 will use MinCR that accomplish bd specs, otherwise will use AVC spec

For clarity... what happens if the level is 4.0 or lower?

[shon3i]

Nothing since level 4.0 and lower use same as avc specs, only for level 4.1 bd specs describe that different mincr is required.