x264 development

[Dark Shikari]

OpenCL is only used on the lookahead, which is only used on the first pass.

The HD4600 and GT440 are both likely at least one order of magnitude too slow to be useful for OpenCL lookahead, which is very inefficient.

[Stereodude]

OpenCL is only used on the lookahead, which is only used on the first pass.

The HD4600 and GT440 are both likely at least one order of magnitude too slow to be useful for OpenCL lookahead, which is very inefficient.Thanks for the clarification and answers. I guess that explains why my first pass FPS went from 73 to 24 when I enabled it using the GT440. I can't get it to work on the HD4600 (thread here (http://forum.doom9.org/showthread.php?t=168168)), but it sounds like it's not worth messing with.

[CarlPig]

I have a little problem with OpenCL:

why say x264 this: x264 [info]: OpenCL: unable to open clbin file for writex264 [info]: profile Main, level 3.1

see here:
x264 [info]: OpenCL acceleration enabled with NVIDIA Corporation GeForce GTX 560 Ti
x264 [info]: Compiling OpenCL kernels...
x264 [info]: OpenCL: unable to open clbin file for writex264 [info]: profile Main, level 3.1

[Romario]

Excuse me, I have a question for x264 devs.

What's happening with x264 development and optimisations specificaly for Haswell ? Can we expect news, soon ?

[MasterNobody]

Look at x264's changes (http://git.videolan.org/gitweb.cgi?p=x264.git;a=shortlog). All commits that have AVX2 in the name can be categorized as "optimizations specifically for Haswell". So I am not sure about which news your are talking as it is already there (i.e. not news).

[nhakobian]

Excuse me, I have a question for x264 devs.

What's happening with x264 development and optimisations specificaly for Haswell ? Can we expect news, soon ?

Search is your friend. Or in this case, just some browsing in the current thread >.<

http://forum.doom9.org/showthread.php?p=1631310#post1631310

[burfadel]

Intel are talking about AVX3.2 on Skylake (successor to Broadwell, which succeeds Haswell). The question is, what it brings? Also, where does AVX3 and AVX3.1 come into play? It could be that there is no AVX3, just AVX3.1, but still, is AVX3.1 coming with Broadwell? Any theoretical advantages gained from it for encoding (if anyone knows what instructions are involed)?

[mastrboy]

Has anything changed in the recent revisions for the scene change detection code or key frame code?

Just encoded a series and seems like there's almost no key frames at scene-changes, noticed when trying to time some subs with Aegisub, already tried increasing scenecut from default 40 up to 100, but no difference on those particular scenes, also checked the scenes transition and there's really no good reason why those scenes shouldn't be detected as scenechanges, the luma difference on the transitions is quite high.

(I know I can force a keyframe with a qpfile, but x264 is supposed to put a key frame on all scene changes it detects automatically, right?)

Tried both 2334 and 2345 builds (using 64bit, 10bit)

[aldix]

I have a little problem with OpenCL:

why say x264 this: x264 [info]: OpenCL: unable to open clbin file for writex264 [info]: profile Main, level 3.1

see here:
x264 [info]: OpenCL acceleration enabled with NVIDIA Corporation GeForce GTX 560 Ti
x264 [info]: Compiling OpenCL kernels...
x264 [info]: OpenCL: unable to open clbin file for writex264 [info]: profile Main, level 3.1

Having the exact same thing. What's this about?

Nvidia Geforce GT 650M, 2GB VRAM

[LigH]

x264 seems to have no permission to write a file which is related to OpenCL processing. I wonder which file it is, and in which directory it shall be written...

[MasterNobody]

x264 seems to have no permission to write a file which is related to OpenCL processing. I wonder which file it is, and in which directory it shall be written...
The file that controlled by --opencl-clbin option and defaults to "x264_lookahead.clbin" i.e. in the current working directory.

[LigH]

So if that is in "C:\Program Files\MeGUI", then the UAC may not grant writing rights there. It should be relocated to the directory where the video file is.

[aldix]

Okay but does it mean that additional speed via OpenCL works or not?

[LigH]

If this file can't be created, OpenCL lookahead is probably disabled. But it may not be a massive speedup anyway, except your GPGPU is exceptional.

[MasterNobody]

If this file can't be created, OpenCL lookahead is probably disabled. But it may not be a massive speedup anyway, except your GPGPU is exceptional.
Wrong. OpenCL still works (it has info level not warning or error). This file is only used as cache i.e. to not compile OpenCL functions every run i.e. to speed up start up time.

[LigH]

Sorry, then... :o -- But it would be interesting to know your GPU model and your speedup.

[LoRd_MuldeR]

Even if the warning causes no harm, it worries some users.

And since, on Windows, programs will almost always installed under "C:\Program Files\Something" (and since that directory is protected by UAC), most users will need to use "--opencl-clbin" manually in order to avoid the warning.

What about changing the default to "/tmp/x264_lookahead.clbin" on Unix-like systems. And to "%TMP%\x264_lookahead.clbin" on Windows?

[MasterNobody]

Even if the warning causes no harm, it worries some users.

And since, on Windows, programs will almost always installed under "C:\Program Files\Something" (and since that directory is protected by UAC), most users will need to use "--opencl-clbin" manually in order to avoid the warning.

What about changing the default to "/tmp/x264_lookahead.clbin" on Unix-like systems. And to "%TMP%\x264_lookahead.clbin" on Windows?
Where I said the directory "where x264.exe is"? Current working directory don't have to be the same directory (look at any link-file or in the case when x264 is called from cmd by full path current working directory will be current one in cmd). Also same directory is used for 2pass files by default so I don't see the reason while clbin should be special case (if GUIs want they can change this path by --opencl-clbin option).

[LoRd_MuldeR]

Where I said the directory "where x264.exe is"? Current working directory don't have to be the same directory (look at any link-file or in the case when x264 is called from cmd by full path current working directory will be current one in cmd). Also same directory is used for 2pass files by default so I don't see the reason while clbin should be special case (if GUIs want they can change this path by --opencl-clbin option).

You didn't. And I didn't say you did.

However, if you run x264 from the command-line, you will usually navigate to the directory "where x264.exe is" and then type "x264 [parameters]", so this directory will end up to be your working directory most of the time.

GUI's could work around this, by explicitly setting the working directory for the x264 process to %TMP%, indeed. But that's one more thing the caller/application has to know and take care of.

Attached is a simple patch to set the default clbin directory to %TMP%, if available, or to "/tmp" otherwise. It's untested, because I haven't managed to compile x264 with OpenCL yet. But the idea should be clear...

(BTW: For the 2-Pass stats file I would actually prefer, if it did default to the directory where the selected output file is located)

[MasterNobody]

However, if you run x264 from the command-line, you will usually navigate to the directory "where x264.exe is" and then type "x264 [parameters]", so this directory will end up to be your working directory most of the time.
I am not sure that this common scenario usage of x264 cli (it is simpler to add x264.exe to %PATH% and call it from anyway). Also I mostly sure users who use x264 cli do NOT install it to Program Files (as if they normal users they don't have rights to write it there and x264 cli doesn't have installer to install it there). The only one who install it to Program Files are probably GUI's installers. Dunno about Linux users.

P.S. All I have written is only my opinion and I am dunno what other developers think about that. i.e. if you want to change this behaviour you need to convince someone other from developers.

[Marin85]

I think most users who use x264 cli actually use batch files for it. So, the working directory usually ends up being the current directory of the batch file/source files/wherever you put the batch file.

[LigH]

Either a batch file, or a user interface (e.g. MeGUI, StaxRip, ...) with their installation directory being the default, if they don't relocate the "current directory" to the one where the source is located just before calling the encoder.

Like many things, this applies to directories as well: The less you assume, the less you assume wrong... ;) — I hope GUI authors will specify most file locations explicitly, but if they don't, a default location with probable write permission is certainly useful.

[LoRd_MuldeR]

Nice to see that the x264 devs have changed their mind and Unicode support for Windows finally seems to be on the way:
https://github.com/DarkShikari/x264-devel/commit/ae1008d58a895b174e8007896ec98c75f419221d

About 2 years ago, when I suggested adding Unicode support for Windows and provided a very similar patch (https://github.com/lordmulder/x264-utf8/commit/67d3cfef6b3c0266b19ff0bb3d66f13f04d1988c), the answer was "just rename the input file" ;)

[Selur]

" Unicode support for Windows "
NICE!

[Chumbo]

" Unicode support for Windows "
NICE!
+100. :D

Nice to see that the x264 devs have changed their mind and Unicode support for Windows finally seems to be on the way:
https://github.com/DarkShikari/x264-devel/commit/ae1008d58a895b174e8007896ec98c75f419221d

About 2 years ago, when I suggested adding Unicode support for Windows and provided a very similar patch (https://github.com/lordmulder/x264-utf8/commit/67d3cfef6b3c0266b19ff0bb3d66f13f04d1988c), the answer was "just rename the input file" ;)
Yeah, I was a recipient of the "rename the file" answers. LOL

[Chumbo]

I wanted to try the opencl flags to see what kind of performance gains I can get, if any, but when I use the following flags:--opencl --opencl-clbin "c:\temp\SYSTEM\x264_lookahead.clbin"I get a fatal application exist dialog withclGetPlatformIDs is unimplemented. Sorry about that!
Any ideas? Am I missing an opencl SDK or something like that? That was my initial thought. My x264 info:x264 0.135.2345 f0c1c53
(libswscale 2.1.2)
(libavformat 55.1.0)
built on Jul 8 2013, gcc: 4.6.3
configuration: --bit-depth=8 --chroma-format=all
x264 license: GPL version 2 or later
libswscale/libavformat license: LGPL version 2.1 or later

[EDIT] Forgot to add that my video card is a Sapphire (AMD) HD5670.

[MasterNobody]

Chumbo
Try to update drivers for video card to the latest. But I doubt you get any speed up (I guess it would speed down really) with such old video card even if you will be able to run with OpenCL.

[Chumbo]

Chumbo
Try to update drivers for video card to the latest. But I doubt you get any speed up (I guess it would speed down really) with such old video card even if you will be able to run with OpenCL.
Yeah, I was just curious to see if there are any improvements. I do have the latest drivers by the way. I think I may have to install the Intel OpenCL SDK. Probably not worth the hassle in the end.

[Selur]

when I use:
x264 --pass 1 --bitrate 1500 --profile high --level 5.1 --keyint infinite --bframes 16 --open-gop --b-bias 20 --direct auto --b-adapt 2 --lookahead-threads 0 --no-mbtree --rc-lookahead 60 --ipratio 1.5 --pbratio 2 --qpmax 21 --qpstep 8 --aq-mode 2 --aq-strength 0.5 --nr 1000 --non-deterministic --stats "C:\temp\07_25_46_5310_01.stats" --fps 30000/1001 --input-res 1920x1080 --output NUL -

x264 --preset placebo --pass 2 --bitrate 1500 --profile high --level 5.1 --keyint infinite --open-gop --b-bias 20 --lookahead-threads 0 --no-mbtree --ipratio 1.5 --pbratio 2 --qpmax 21 --qpstep 8 --me umh --subme 9 --trellis 0 --psy-rd 1:0.0 --aq-mode 2 --aq-strength 0.5 --vbv-maxrate 300000 --vbv-bufsize 300000 --nr 1000 --non-deterministic --colormatrix bt470bg --stats "C:\temp\07_25_46_5310_01.stats" --fps 30000/1001 --input-res 1920x1080 --output "C:\temp\07_25_46_5310_02.264" -
I end up with:
Bit rate : 6 405 Kbps
Nominal bit rate : 1 500 Kbps
Maximum bit rate : 8 604 Kbps
when I use:
x264-10bit --pass 1 --bitrate 1500 --profile high10 --level 5.1 --keyint infinite --bframes 16 --open-gop --b-bias 20 --direct auto --b-adapt 2 --lookahead-threads 0 --no-mbtree --rc-lookahead 60 --ipratio 1.5 --pbratio 2 --qpmax 81 --qpstep 8 --aq-mode 2 --aq-strength 0.5 --nr 1000 --non-deterministic --stats "C:\temp\07_28_32_5210_01.stats" --fps 30000/1001 --input-res 1920x1080 --output NUL -

x264-10bit --preset placebo --pass 2 --bitrate 1500 --profile high10 --level 5.1 --keyint infinite --open-gop --b-bias 20 --lookahead-threads 0 --no-mbtree --ipratio 1.5 --pbratio 2 --qpmax 81 --qpstep 8 --me umh --subme 9 --trellis 0 --psy-rd 1:0.0 --aq-mode 2 --aq-strength 0.5 --vbv-maxrate 720000 --vbv-bufsize 720000 --nr 1000 --non-deterministic --colormatrix bt470bg --stats "C:\temp\07_28_32_5210_01.stats" --fps 30000/1001 --input-res 1920x1080 --output "C:\temp\07_28_32_5210_02.264" -
(only difference es are: the profile, the vbv and qpmax values and that the second encode is 10bit)
I end up with:
Bit rate : 1 500 Kbps
Maximum bit rate : 2 674 Kbps
which is more of what I expected, when specifying an average bit rate of 1500.
(source was: http://forum.videohelp.com/attachments/19486-1377217794/rws_asa.avi ; 8 and 10bit x264 core 135 r2345 was from komisar's site)

Is there some simple explanation why x264 totally fails to hit the target bit rate (1500kBit/s, aiming for 3MB) when using 8bit (ending up with 12.7MB) , but hits it quite well with 10bit (3.01MB)?
(I redid this a few times to make sure this was not something that 'just' happened.)

for completeness, here's also the decoder call I used for this:
mencoder -lavdopts threads=2 -really-quiet -of rawvideo -o - -ovc raw -vfm ffmpeg -noskip -vf scale,format=i420 -forcedsubsonly -nosub -nosound -mc 0 "C:\Users\Selur\Desktop\rws_asa.avi"

The settings I choose may all be 'strange', but shouldn't these two both hit the target size?

Cu Selur

Ps.: I uploaded both reencodes to: http://www.embedupload.com/?d=8GIKSUGBVO
(would be nice if some one else could also try to reproduce this so that I know if my system is somehow at fault; which I doubt, but you never know)

[vivan]

--qpmax 21, that's why. Actually x264 throwed warning:
x264 [warning]: Error: 2pass curve failed to converge
x264 [warning]: target: 1500.00 kbit/s, expected: 6017.52 kbit/s, avg QP: 21.0000
x264 [warning]: try increasing target bitrate or increasing qp_max (currently 21)

[Selur]

Argh, totally overlooked that!! Thanks, should have been 51. ;)

[sneaker_ger]

--qpmax 51 is an outdated default value, btw.
1. it is not x264's default anymore
2. it never was the maximum qp for 10 bit encoding in the first place
3. even higher values are allowed for the vbv emergency mode, i.e. you turned it off

[Selur]

@sneaker_ger: I know :)

[LoRd_MuldeR]

FFMS2 input seems to be broken with latest x264. I get a deadlock with x264 r2358 and a command-line like this:

"C:\Program Files (x86)\MuldeR\Simple x264 Launcher v2\toolset\x64\x264_8bit_x64.exe" --crf 20.0 --preset medium
--output C:\Temp\Out.mkv --index C:\Temp\foo.ffindex E:\Samples\Lossless\crew.704x576.avi

x264 will hang before even printing out anything to the console, so I suspect something gets locked up in the new Win32 Unicode support code, maybe when processing the CLI argument? :confused:

Avisynth or VapurSynth input works fine though.

http://i.imgur.com/Z7z1L4ds.jpg (http://i.imgur.com/Z7z1L4d.jpg)


[UPDATE]

This problem seems to be related to the "--index" argument! As soon as I exclude that one from the above command-line, it won't deadlock anymore.


[UPDATE 2]

Just noticed that a fix which most likely relates to this problem is on the way:
https://github.com/DarkShikari/x264-devel/commit/dfdb6465dea2990a4531d076ed2644c8ccb0f3a9

Now that I see the fix, I remember that I came across the very same bug in FFMS2 about two years ago, when I implemented Unicode support for x264.

Also, back at that time, I came up with the same workaround (https://github.com/lordmulder/x264-utf8/commit/67d3cfef6b3c0266b19ff0bb3d66f13f04d1988c#diff-6) :)


[UPDATE 3]

Fix for the actual bug in FFMS2 was provided by Komisar here:
http://komisar.gin.by/x.patch/check_idx_file.diff

Hopefully this will finally be committed into FFMS2, after all the time this nasty bug is around...

[Selur]

*gig* I totally missed the "--index <string> Filename for input index file" option to begin with. :)

[mastrboy]

Henrik Gramner [Mon, 8 Jul 2013 21:06:42 +0200 (12:06 -0700)]
Transparent hugepage support

Combine frame and mb data mallocs into a single large malloc.
Additionally, on Linux systems with hugepage support, ask for hugepages on
large mallocs.

This gives a small performance improvement (~0.2-0.9%) on systems without
hugepage support, as well as a small memory footprint reduction.

On recent Linux kernels with hugepage support enabled (set to madvise or
always), it improves performance up to 4% at the cost of about 7-12% more
memory usage on typical settings..

It may help even more on Haswell and other recent CPUs with improved 2MB page
support in hardware.

Will Windows take advantage of this by default? Or do I need to adjust the GPO setting "Lock Pages in Memory" ?

[Selur]

anyone got some overview what '--acvintra-compat' enforces, restricts ?

from a first look it seems to enforce:

--bframes 0 (-> no bpyramid, intra refresh, weigthed bframes,...)
--ref 1 (-> no mixed ref)
--weightp 0
--keyint 1
--aud
some custom quantizer matrices
--stitchable

since this is probably not all it does, I wanted to as if someone can post some sort of 'proper' overview which normally available settings are restricted or 'removed' by '--avcintra-compat' ?

[LoRd_MuldeR]

For a complete list, you probably should look at the code, as it does quite a lot.

This should be the interesting part:
http://pastie.org/private/pmu6fxzte34ihtl0sdzfuw

Note, however, that "param.b_avcintra_compat" is used directly in a few other places too (e.g. analysis.c and ratecontrol.c), not only in the parameter verification function.

So this is not limited to adjusting the existing options, like a simple "preset" would do. It also seems to influence things that are not exposed by any other options.

[Dark Shikari]

avc-intra enables a horrible set of internal hacks that can't be emulated using external parameters.

[Selur]

To be frank I'm not really interested into all the internal hacks but more about what is external accessible parameters are restricted (or forced into a specific value) by it.

[LoRd_MuldeR]

As said before, you best look at the code that verifies/updates the parameters.

I'd say it does (at least) the following:
- Works only with 10-Bit x264
- Force key-frame interval to 1 (not really surprising ^^)
- Force intra refresh to off
- Force weight-p to off
- Force number of reference frames to 1
- Force BluRay compatibility to off
- Force support for VFR to off
- Force access unit delimiters to on
- Force signal HRD information to off
- Force deblocking filter to off
- Force 8x8dct to on
- Force rate-control mode to 1-Pass ABR
- Force VBV to enabled, with specific parameters
- Force sliced threads, if used, to 2 (for 1080p) or to 1/5 (for 720p)
- Force using a specific custom quantization matrix

[Selur]

Works only with 10-Bit x264
I read it that it would also work with normal 8-Bit encoding, but that not many applications support this king of AVC-Intra :D

Force support for VRR to off
what's VRR ?

Cu Selur

[LoRd_MuldeR]

I read it that it would also work with normal 8-Bit encoding, but that not many applications support this king of AVC-Intra :D

Regardless of whether it would work in theory, x264 currently refuses to encode AVC-Intra as anything but 10-Bit.

Note the return statement.

what's VRR ?

That was a typo. Should read "VBR", of course ;)

[MasterNobody]

That was a typo. Should read "VBR", of course ;)
Probably still typo and you really meaned disable VFR (Variable Frame Rate) and not VBR (Variable Bit Rate). But you can say VBR is also disabled because it use CBR.

[LoRd_MuldeR]

Probably still typo and you really meaned disable VFR (Variable Frame Rate) and not VBR (Variable Bit Rate).

Argh, I need some sleep :scared:

[Selur]

Thanks for clearing that up :)

[Thalyn]

I don't mean to derail the conversation here, but just out of curiosity... has there been any additional work on adding MVC support to x264? The last content I can find on the subject is nearing its second birthday, and that's a long time in development.

I do recall reading that it wasn't being seriously looked at for a time due to it compressing roughly the same as frame-interleaved 3D. But I'm hoping that was reconsidered given the lack (absence?) of interleave-aware players and the greater legacy compatibility of MVC (ie Watch it in 2D if the player doesn't recognise MVC, instead of it staggering back and forth).

[Sharc]

I don't mean to derail the conversation here, but just out of curiosity... has there been any additional work on adding MVC support to x264? The last content I can find on the subject is nearing its second birthday, and that's a long time in development.

I do recall reading that it wasn't being seriously looked at for a time due to it compressing roughly the same as frame-interleaved 3D. But I'm hoping that was reconsidered given the lack (absence?) of interleave-aware players and the greater legacy compatibility of MVC (ie Watch it in 2D if the player doesn't recognise MVC, instead of it staggering back and forth).
MVC support would be awesome as tsMuxeR now seems to develop into the same direction.

[jdobbs]

MVC support would be awesome as tsMuxeR now seems to develop into the same direction.Add me to the list of people who would love to see it.

[benwaggoner]

Is much happening with x264 development? The last checkin was three months ago.

Perhaps x265 is getting a lot of attention from the core devs :)?

http://git.videolan.org/gitweb.cgi?p=x264.git;a=shortlog