Getting the latest x264

[Groucho2004]

Ah, and about opencl, i should have said : To build with it, it seems that i need to install in the build environement some others third party things i don't know about, so, too much bother for something from my point of view is not worth it.I use a pretty vanilla MingW from here (https://files.1f0.de/mingw/) and I can build x264 with OpenCL support without any additional "things".

[Dogway]

Updated the benchmark. I got higher resutls on vlan build so I did a second run to average results, tmod was getting lower results so disabled progress and now it's on par with the other two.

[asarian]

Updated the benchmark. I got higher resutls on vlan build so I did a second run to average results, tmod was getting lower results so disabled progress and now it's on par with the other two.

Thanks for the comparison. The FPS drop in 10-bit encoding is pretty significant; ouch.

[LigH]

I build with a pretty generic MSYS2 / MinGW environment which is kept up-to-date by the media-autobuild_suite. If OpenCL is enabled for ffmpeg, I suppose it is enabled for x264 as well.

And my H is capital.

[Dogway]

And my H is capital.

Ok fixed!

[chompy]

Hi, is there any updated build patched with fade-compensate (last was jpsdr's x264 t_mod r2932)? Thanks

[jpsdr]

In a few days, if everything goes well...

[chompy]

In a few days, if everything goes well...

Great :thanks:

[jpsdr]

I'm also working, nice and slow, to implement almost all the t_mod patches to the real r2969 version, not the "special" version i'm using for now.
Almost, because there is two patches i'll not implement : the avi output and the audio support. These patches are big and very difficult, but the most critical issue, is that they are using deprecated ffmpeg functions, needing a too old version of ffmpeg not compatible anymore with actual real r2969 version.
And i have no idea/skill/knowledge for updating these patches which need to be rewritten with actual ffmpeg functions.

[chompy]

I'm also working, nice and slow, to implement almost all the t_mod patches to the real r2969 version, not the "special" version i'm using for now.
Almost, because there is two patches i'll not implement : the avi output and the audio support. These patches are big and very difficult, but the most critical issue, is that they are using deprecated ffmpeg functions, needing a too old version of ffmpeg not compatible anymore with actual real r2969 version.
And i have no idea/skill/knowledge for updating these patches which need to be rewritten with actual ffmpeg functions.

Nice, lots of thanks again for your work!

[jpsdr]

I've finsihed implemening all the patches (even the "audio" and "avi ouput") of the tmod version on the last real r2970 version, avaible on my github on t_mod_New, builds are in progress, release soon.
Quick test seems ok, but i'm not using all the tmod features (the "audio" part for exemple, i'm not using it, so not tested...).

[jpsdr]

Done, i've released builds of the t_mod_New version, it's avaible on my github.

[Gser]

Done, i've released builds of the t_mod_New version, it's avaible on my github.

What exactly is the difference between t_mod and the vanilla x264? I noticed there are at least some new AQ modes, what do they improve upon and what kind of content are they for?

[jpsdr]

I don't know what vanilla has, you'll have to compare the patch applied.
About the new AQ, i think you refers to the TriAQ patch, wich add aq2 and aq3 mode. The original author of this patch made it most for anime purpose. Even if nothing prevent you to use them on something else than anime, for the few i've understood from aq3 mode, its purpose is mostly for anime "kind" video.
After, aq2 mode may be a more generic use.
Unfortunately, there was almost no information on this patch... :(
But my real advise would be to try.

[benwaggoner]

I don't know what vanilla has, you'll have to compare the patch applied.
About the new AQ, i think you refers to the TriAQ patch, wich add aq2 and aq3 mode. The original author of this patch made it most for anime purpose. Even if nothing prevent you to use them on something else than anime, for the few i've understood from aq3 mode, its purpose is mostly for anime "kind" video.
After, aq2 mode may be a more generic use.
Unfortunately, there was almost no information on this patch... :(
But my real advise would be to try.
aq-mode 3 is the same as aq-mode 2 but with a bias to lower QPs in the low luma range. If anything, it's most useful for film sources where quantization or banding in grain cause issues in shadow detail.

[jpsdr]

Don't mix aq-mode (2 or 3) with aq2-mode or aq3-mode ! They are absolutely not the same thing.
aq2-mode and aq3-mode are others aq modes added by the TriAQ patch, as i've said.

[benwaggoner]

Don't mix aq-mode (2 or 3) with aq2-mode or aq3-mode ! They are absolutely not the same thing.
aq2-mode and aq3-mode are others aq modes added by the TriAQ patch, as i've said.
Oh, right, thanks.

I think some renaming is in order!

[StvG]

Some info (https://forum.doom9.org/showthread.php?p=1172141#post1172141). MixAQ = aq2, OreAQ = aq3. After that MixAQ+OreAQ=TriAQ(+ some additional updates).

[sneaker_ger]

FWIW:
https://artifacts.videolan.org/x264/

Hi, all.

Here is URL for new x264 binaries: https://artifacts.videolan.org/x264/
Binaries of old x264 versions still can be downloaded from: http://download.videolan.org/pub/x264/binaries/
https://mailman.videolan.org/pipermail/x264-devel/2019-July/012705.html

[tormento]

I build with a pretty generic MSYS2 / MinGWAnd my H is capital.
Would you please tell me the differences between your builds and the "official" ones?

Would it be possible to have some ICL ones to test too?

[LigH]

I can't compare with the "official" builds, as I don't know how the "official" builds were created. I guess they don't contain lavf and ffms which MABS does include. And MABS uses MSYS2 / MinGW with GNU C/C++ in a current stable release (v10.1 at the moment). My latest build is linked and documented here (https://forum.doom9.org/showthread.php?p=1916663#post1916663).

I can't use ICL, I don't have any experience with it.

[LoRd_MuldeR]

I can't compare with the "official" builds, as I don't know how the "official" builds were created.

It looks like they are using GitLab CI/CD (https://docs.gitlab.com/ee/ci/yaml/README.html) runners to create the "official" builds, and you can see their configuration file here:
https://code.videolan.org/videolan/x264/-/blob/master/.gitlab-ci.yml

Anyway, the "official" builds have libavcodec/libavformat support enabled, but not FFMS. Also, they use GCC 8.2.

[tormento]

I can't use ICL, I don't have any experience with it.
Thanks.

You can download ICL with free subscription too, if interested.

[nakTT]

FWIW:
https://artifacts.videolan.org/x264/


https://mailman.videolan.org/pipermail/x264-devel/2019-July/012705.html
Thanks for the sharing.

[FranceBB]

Can someone with the rights on the x264 GitLab merge this pull request?

https://code.videolan.org/videolan/x264/-/merge_requests/6

[jpsdr]

Prayer answered it seems...
https://code.videolan.org/videolan/x264/-/commit/ae03d92b52bb7581df2e75d571989cb1ecd19cbd

[FranceBB]

Yep, I've got the email the other day: it has been merged. :D
Time to celebrate, then. ;)

annuntio vobis gaudium magnum: habemus Intra Class 300-480 xD

[jpsdr]

New version build.

[chompy]

New version build.

@jpsdr: Will you continue updating your x264 t_mod versions? Thanks!

[jpsdr]

I will when there will be interesting/worthwile commits (bug fixes or encode improvement), as it take a lot of time for me to make the build. Didn't check if there was commits the last 3 weeks, but until there, the commits were not worth spending time.

[chompy]

I will when there will be interesting/worthwile commits (bug fixes or encode improvement), as it take a lot of time for me to make the build. Didn't check if there was commits the last 3 weeks, but until there, the commits were not worth spending time.

Great, lots of thanks for your great work!!

[jpsdr]

New t_mod version for those who are still interested.

[mastrboy]

New t_mod version for those who are still interested.

Thank you.

[kebulek]

Thanks! :)

[FranceBB]

Nice, thank you, Jean Philippe! :D

[chompy]

New t_mod version for those who are still interested.

Lots of thanks!!

[FranceBB]

Hi there guys, I've just noticed that if I use --asm=avx512 with x264 8bit, it works, however if I use it while encoding with a 10bit profile, it doesn't, it only goes up to AVX2.
Is this expected? Is this because no one wrote AVX512 intrinsics for 10bit?

[LoRd_MuldeR]

Hi there guys, I've just noticed that if I use --asm=avx512 with x264 8bit, it works, however if I use it while encoding with a 10bit profile, it doesn't, it only goes up to AVX2.
Is this expected? Is this because no one wrote AVX512 intrinsics for 10bit?
You probably mean "assembly code" functions. An "intrinsic" is a function (well, not really a function, just something that can be called like a function) which is built directly into your compiler.

Anyways, as far as I can tell, the only places where the flag X264_CPU_AVX512 appears is in cpu.c and dct.c.

Obviously the code in cpu.c detects the CPU features. It's in dct.c where x264 actually uses the X264_CPU_AVX512 flags to enable (or not) certain function:
https://raw.githubusercontent.com/mirror/x264/0bb85e8bbc85244d5c8fd300033ca32539b541b7/common/dct.c

But, if I read the pre-processor directives in that file correctly, then the X264_CPU_AVX512 flag does not appear in the HIGH_BIT_DEPTH part at all; it only appears in the !HIGH_BIT_DEPTH part.

In other words, there is nothing in the "high bit-depth" code path that gets enabled/disabled depending on the availability of AVX-512 support.

Probably there simply are no "high bit-depth" functions that would benefit from AVX-512 instructions, or at least nobody nobody cared enough about AVX-512 to figure it out ;)

[FranceBB]

You probably mean "assembly code" functions.

Yep, sorry, I meant manually written assembly optimizations. :)




if I read the pre-processor directives in that file correctly, then the X264_CPU_AVX512 flag does not appear in the HIGH_BIT_DEPTH part at all; it only appears in the !HIGH_BIT_DEPTH part.


I see! So it's normal that I don't see it being enabled there as no one actually wrote anything for it in the high bit depth part, so it only works for 8bit. Gotcha. :)


In other words, there is nothing in the "high bit-depth" code path that gets enabled/disabled depending on the availability of AVX-512 support.


Gotcha, which is why I couldn't see it being enabled. Thanks for taking a look at it. :)


Probably there simply are no "high bit-depth" functions that would benefit from AVX-512 instructions


Nah, that's very unlikely, I just think no one wrote assemblies 'cause 10bit H.264 is much less used compared to the 8bit flavors (and indeed the only two groups of people who use it are those encoding in intra class to target broadcast hardware SDI playout ports in professional settings and those encoding anime, like fansubbers doing reverse upscale of BDs xD)



nobody cared enough about AVX-512 to figure it out ;)

Yeah, that's much more likely.
I wonder if Henrik Gramner will get back on this, given that he's the one behind the AVX-512 commits (among plenty of other things)...

[FranceBB]

Remember the whole discussion about OpenCL?
I actually tested again but this time on Amazon AWS machines.

m6i.4xlarge 16c/16th AVX-512 CPU Only
1h 24m

m6i.4xlarge 16c/16th AVX-512 CPU + OpenCL GPU
1h 03m


The content is 1h 32min 25seconds and 3 frames long.

The video is MPEG-2 50 Mbit/s FULL HD yv16 25i with PCM 24bit lossless 5.1 audio bobbed to 50p and with the chroma downscaled to yv12 in Avisynth.


x264.exe "Z:\AVS Script.avs" --preset medium --profile high --level 4.1 --ref 4 --deblock -1:-1 --crf 25 --keyint 50 --aud --overscan show --range tv --opencl --colormatrix bt709 --transfer bt709 --colorprim bt709 --videoformat component --nal-hrd vbr --vbv-maxrate 25000 --vbv-bufsize 25000 --output "I:\temp\raw_video.h264"

ffmpeg.exe -i "Z:\AVS Script.avs" -vn -sn -af loudnorm=I=-24:LRA=12:tp=-2 -c:a aac -b:a 550k -ar 48000 "I:\temp\audio.aac"

mp4box.exe -add "I:\temp\raw_video.h264" -add "I:\temp\audio.aac" "I:\temp\final_output.mp4"


adding --opencl saved around 20 min compared to not adding it.
So... yeah, it still makes sense to have it on.

[rwill]

m6i.4xlarge 16c/16th AVX-512 CPU + OpenCL GPU
1h 03m


Which GPU is available in an m6i.4xlarge instance ?

[FranceBB]

Which GPU is available in an m6i.4xlarge instance ?

Since everything you run through AWS is virtualized (unless you specifically require bare metal), it's hard to know, but I believe they're running an NVIDIA Tesla M6 8GB GDDR5.

[benwaggoner]

I actually tested again but this time on Amazon AWS machines.

m6i.4xlarge 16c/16th AVX-512 CPU Only
1h 24m

m6i.4xlarge 16c/16th AVX-512 CPU + OpenCL GPU
1h 03m
...
adding --opencl saved around 20 min compared to not adding it.
So... yeah, it still makes sense to have it on.
Yeah, that follows the "about a 20% speedup" rule of thumb.

That said, a c6a.4xlarge (AMD EPYC 7R13) may well offer the same performance software-only at a lower per-hour instance cost.

[DTL]

Probably there simply are no "high bit-depth" functions that would benefit from AVX-512 instructions, or at least nobody nobody cared enough about AVX-512 to figure it out ;)

As from old intel whitepaper from start of x265 at the start of 201x - https://www.intel.com/content/dam/develop/external/us/en/documents/mcw-intel-x265-avx512.pdf

The >8bit MPEGs more benefit from AVX512 because it uses 2x wider computing of source and result and immediate numbers (16bit for src and out and 32bit immediate) in compare with 8bit MPEGs (8bit src and out and 16bit immediate possible). Median performance boost from AVX512 over AVX2 is about 40% at 'processing kernels' of MPEG encoder.
https://i1.imageban.ru/out/2023/02/10/2617fe727c4ae8dcae5bac905fb3f07c.png

But 10bit x264 looks never was interested for major usage and no developers resources were for design either wider AVX512 computing for existed x264 architecture (SIMD 'workunit' size of 512 bytes max for AVX2 register file) or even much more complex design separate brunch of 4x enlarged 'workunit' size for 2048 bytes register file for AVX512 x64 environment. And that AVX512 brunch will only be executed faster on rare exist at endusers AVX512 environment in prevoius years. And no rich investor put some grant to support AVX512 redesign of x264 (to use in some possible commertials).

When some company start project from AVX512 hardware it can invest in pro programmers to design software solution for target platform and use all benefits of AVX512 environment. Splitting problem chunks to 'large workunit' size up to 2048 bytes with best processing performance at AVX512 platform. Also using alignment of workunits addressing to 64bytes cacheline size for fastest transfer to and from AVX512 register file and dispatch ports. And freeware opensource developers for multiplatform development typically making C-reference solutions to rely on compiler vectorization is possible and may limit 'workunit size' of algorithm to smaller wider used by endusers platforms because it works faster on small register file chips (less reloads from cache). So practically when opensource developers design and profile for best performance some computing algorithms at cheap old platforms with small sized register file (128 or 256 bytes for SSE(2) and AVX(2)) they actually optimize algorithm to run only at small register file sized platforms. And resources of high-cost in the past AVX512 platforms left underused. Full optimizing for AVX512 includes both usage of 2x wider execution ports (and some more faster instructions) and 2x sized datawords transfer and usage of 4x larger register file increasing processed 'workunit' size. And changing 'workunit' size may need the significant redesign of software (like processing 4 blocks in single pass instead of 1 block at AVX2 and so on).

AVX512 programs significantly more complex in design and debug. Without AVX512 chip it is possible to design via intel SDE software simulator but it can not provide correct profiling for performance results.
May be with progress of AI like ChatGPT or others we can see some progress in using of new compute platforms for solving old tasks like x264. Because it looks resources of opensource freeware programmers fast dying with ending of current civilization.


m6i.4xlarge 16c/16th AVX-512 CPU Only

x264.exe "Z:\AVS Script.avs" --preset medium --profile high --level 4.1 --ref 4 --deblock -1:-1 --crf 25 --keyint 50 --aud --overscan show --range tv --opencl --colormatrix bt709 --transfer bt709 --colorprim bt709 --videoformat component --nal-hrd vbr --vbv-maxrate 25000 --vbv-bufsize 25000 --output "I:\temp\raw_video.h264"


Do programmers of x264 builds in 2022..2023 already included AVX512 usage in auto-detection of execution platform capabilities ? Or user still need to force it with --asm avx512 (or may be --asm=avx512) command line option ?



I can't use ICL, I don't have any experience with it.

Intel C compiler can use multi-file interprocedural optimization (not everytime works and may need to fix sources). It may visibly helps to performance of complex program with many small processing functions (and many C source files). Though with increasing of complexity of program the probability of successful full program multi-file IPO decreases. But you still can try to enable it for separate projects of solution. Sometime you need to run several compiling runs and one of it finally ends with successful multi-file IPO.

Also it possibly best compiler to make AVX512-targeted builds (and intel chips builds with individual optimization to many intel chips families).

[FranceBB]

But 10bit x264 looks never was interested for major usage and no developers resources were for design either wider AVX512 computing for existed x264 architecture (SIMD 'workunit' size of 512 bytes max for AVX2 register file) or even much more complex design separate brunch of 4x enlarged 'workunit' size for 2048 bytes register file for AVX512 x64 environment.


Yeah, x264 is almost always used in the 8bit flavor for distribution, however professionally AVC Intra and XAVC Intra are two very common use cases for both FULL HD and UHD workflows (like Intra Class 300 and 480) and there AVX512 would definitely speed things up a lot. It's a shame that no one invested into it, though. :(


no rich investor put some grant to support AVX512 redesign of x264 (to use in some possible commercials).


We're already supporting Multicoreware for x265 (and the future x266) by being part of their partner program, but unfortunately they're not the ones behind x264... :(
I'm sure that there are other companies using x264 to encode either AVC Intra files or XAVC files using the Sony and Panasonic profiles for linear playout beside us, so if they could chip in and support the development, that would be greatly appreciated.



Do programmers of x264 builds in 2022..2023 already included AVX512 usage in auto-detection of execution platform capabilities ? Or user still need to force it with --asm avx512 (or may be --asm=avx512) command line option ?


Ironically, x264 works automatically with AVX-512 once it detects a compatible CPU to run, however x265 still needs --asm=avx512 to make use of them.
Why? Dunno.

[DTL]

"professionally AVC Intra and XAVC Intra are two very common use cases for both FULL HD and UHD workflows (like Intra Class 300 and 480) and there AVX512 would definitely speed things up a lot"

Intra may be already very simple and too fast - so the total workflow may be too few benefit if someone make intra more faster. Most slow things may happen at inter frame encoding where motion search required. Also as I see today 10bit x264 builds for AVC-Intra is almost impossible to found. May be it somewhere in special builds of ffmpeg.

8bit 'classic' x264 throws an error at YV12 avisynth input -

at -I 1 --avcintra-class=100
x264 [error]: 8-bit AVC-Intra is not widely compatible
x264 [error]: 10-bit x264 is required to encode AVC-Intra
x264 [error]: x264_encoder_open failed


With simple I-frames only simulation with -I 1 it runs at about 12 fps vs 3 fps for IPB encoding. About 4x faster for 'intra-only' encoding.

"I'm sure that there are other companies using x264 to encode either AVC Intra files or XAVC files using the Sony and Panasonic profiles for linear playout beside us, so if they could chip in and support the development, that would be greatly appreciated."

If some commertials uses freeware for production it may be too poor company to pay even for its existing poor workers. Not have any funds to pay to pro programmers to make opensource software better. If you have some funds you may try to open some offering contract at software developers jobs sites with exact task to pay for - like make x264 build run 2x faster at AVX512 defined chip with your required arguments and provided sample footage. May be such jobs exist and already solved but not offered opensource as a commit to standard project for free.

Also current business models are about making money for business owners and not make job amount of hired workers lower for the same payments. So if worker want to encode faster and spent less work time it is only task for worker to read intel docs for chip and make software run faster. The business owner will not pay for it.
Also as stated in typical jobs contracts: All enchancements made by worker for software are new funds of business owner. So it may be illegal for workers contracts to share any enchancements made to x264 at the payed time by business owner.

[FranceBB]

Also as I see today 10bit x264 builds for AVC-Intra is almost impossible to found.


Just use any of the JPSDR builds and you're good to go.


8bit 'classic' x264 throws an error at YV12 avisynth input -

at -I 1 --avcintra-class=100
x264 [error]: 8-bit AVC-Intra is not widely compatible
x264 [error]: 10-bit x264 is required to encode AVC-Intra
x264 [error]: x264_encoder_open failed



Yeah cause Panasonic and Sony specs are 10bits only, so the preset won't allow you, however you can do it manually if you really want to without using the preset.
Keep in mind that the command line switch is a bit like the Blu-ray switch: it will make life easier for you, but you can also use the options and do it manually. https://forum.doom9.org/showthread.php?t=182715


With simple I-frames only simulation with -I 1 it runs at about 12 fps vs 3 fps for IPB encoding. About 4x faster for 'intra-only' encoding.


Well there are also the "Long GOP" version of AVC as per Sony specifications and those use a closed specific GOP and definitely make use of P and B.

About the last point, it's not true, this forum is the demonstration as we have people contributing to open source projects from all over the world and working on several different companies. Our Ben here works at Amazon for instance and they contributed to x265 with the NEON assembly optimization for ARM among other things, for instance, Steinar works at NRK and contributed to the Sony and Panasonic flavours we talked about in x264, Kieran works at Open Broadcast System and made x262 etc (I could go on and on and on), so companies do contribute to open source encoders.

[DTL]

I'm sure that there are other companies using x264 to encode either AVC Intra files or XAVC files using the Sony and Panasonic profiles for linear playout beside us, so if they could chip in and support the development, that would be greatly appreciated.


For I-frames only encoding you may check this build - https://github.com/DTL2020/x264/releases/tag/Rel_130223 . It have (temporarily for performance check) disabled some _x9_8x8 and _x9_4x4 SAD/SATD compare in intra analysis. At old E7500 CPU it runs about 34% faster but produces about 10% more bitrate (may be with fixed bitrate AVC-Intra modes it will only degrades quality in some degree and depending on frame complexity).
Later I hope to put this function call to multi-block SIMD processing of AVX2/AVX512 to keep full quality.

Updated: fixed build to include processing of 4:2:2 and 10bit format required for AVX-Intra with skipped _x9 macroblocks compare. It looks this functions still not exist at all for single call _x9 processing and for > 4:2:0 and/or > 8bit processing always performed longer loop of checking each predictor separately.

At i5-11600 CPU this build with simulated AVX-Intra 100 for FullHD frame settings (from https://forum.doom9.org/showthread.php?p=1940382#post1940382 post) run at about 22.9 fps (jpsdr build 'winthreads' marked run at about 20 fps).

As more profiling shows for I-frames only high bitrates encoding with CAVLC only compression:
It looks x264 not any optimized for such production. The AVC-Intra over 50 class not allow CABAC (having some asm optimizations) and so x264 only can run with CAVLC compression. CAVLC have only C-implementation and almost no asm optimizations. And it is not about math computing but mostly shuffling small enough and random length byte streams. So SIMD units unlikely can help alot here. It mostly memory-bound task. At least at first look at CAVLC compressor.
Also it much more display x264 feature to have lower performance at the lower compression rate. If you remove hard fixed bitrate from AVC-Intra and allow x264 to run at crf-ratecontrol with VBR it run significantly faster. At IBP encodings it is also visible but much lower. So at I-frames only and high fixed bitrates performance looks like limited by rate control logic to keep required very high bitrate stable. If disable CBR and allow low crf - the analysis run several times faster (about 80 fps vs 20fps).
May be for professional usage you can found other intra-frame encoder MJPEG-like with much better implementation of fixed CBR output.

[blob2500]

Latest 'official' x64 build (r3179): 3MB only? :confused:

https://artifacts.videolan.org/x264/release-win64/

[LigH]

Most probably not containing all the encoders specialized in various internal resolutions (8+10 bit). Especially the higher resolutions require a lot more efforts and code to handle more than 1 byte per video component in Assembler.

New MABS compile: x264 0.164.3179 12426f5 (https://forum.doom9.org/showthread.php?p=1997983#post1997983)