Build x265 with Intel compiler

[jpsdr]

Hello.

I've tried (but failed), to make a build of x265 with Intel Compiler, out of curiosity of benchmark purpose vs my others builds.

I have access to Visual Studio 2019 with Intel Parallel Studio 2020 Update 4 installed, integrated within Visual Studio.

I first tried to do the following : With CMake create a Visual Studio project, and after with VS, in the project switch to Intel Compiler, and build. It failed, it generates errors.

I then tried with CMake to create a visual Studio Project, but with the switch Intel On, it failed creating the solution.

So, my questions are :
- Is it possible in the first place to build x265 with Intel Compiler ?
- If yes, what is the proper CMake command line for creating the solution ?

Edit :
The commande line i've tried but failed is the following :

G:\CMakex64\bin\cmake -G "Visual Studio 16" -A x64 ../../../source -DENABLE_VAPOURSYNTH=ON -DENABLE_AVISYNTH=ON -DHIGH_BIT_DEPTH=ON -DENABLE_HDR10_PLUS=ON -DEXPORT_C_API=OFF -DENABLE_SHARED=OFF -DENABLE_CLI=OFF -DMAIN12=ON -DSTATIC_LINK_CRT=OFF -DCMAKE_CXX_COMPILER_ID="Intel" -DCMAKE_CXX_FLAGS_RELEASE="/sdl- /Oi /Ot /Oy /GT /GL /GF /GS- /Gy /Qpar /arch:AVX2"

[john33]

Using VS 2019 Community Edition and Intel DPC++/C++ Compiler for Windows (downloadable for free from: https://www.intel.com/content/www/us...html#dpcpp-cpp):

**********************************************************************
** Visual Studio 2019 Developer Command Prompt v16.11.25
** Copyright (c) 2021 Microsoft Corporation
**********************************************************************

C:\Program Files (x86)\Microsoft Visual Studio\2019\Community>d:

D:\> cd x265-r3.50.0.94\build\vc16-x86_64

D:\x265-r3.50.0.94\build\vc16-x86_64>md 12bit

D:\x265-r3.50.0.94\build\vc16-x86_64>md 10bit

D:\x265-r3.50.0.94\build\vc16-x86_64>md 8bit

D:\x265-r3.50.0.94\build\vc16-x86_64>cd 12bit

D:\x265-r3.50.0.94\build\vc16-x86_64\12bit>cmake -G "Visual Studio 16 2019" -A x64 ../../../source -DCMAKE_SYSTEM_VERSION=8.1 -DHIGH_BIT_DEPTH=ON -DEXPORT_C_API=OFF -DENABLE_SHARED=OFF -DENABLE_CLI=OFF -DMAIN12=ON
-- cmake version 3.26.0
.
Load the x265.sln into VS2019 IDE and switch to Intel 19.2 compiler, compile and copy the lib to the 8bit dir as per multilib.bat
.
D:\x265-r3.50.0.94\build\vc16-x86_64\12bit>cd..\10bit

D:\x265-r3.50.0.94\build\vc16-x86_64\10bit>cmake -G "Visual Studio 16 2019" -A x64 ../../../source -DCMAKE_SYSTEM_VERSION=8.1 -DHIGH_BIT_DEPTH=ON -DEXPORT_C_API=OFF -DENABLE_SHARED=OFF -DENABLE_CLI=OFF
-- cmake version 3.26.0
.
Load the x265.sln into VS2019 IDE and switch to Intel 19.2 compiler, compile and copy the lib to the 8bit dir as per multilib.bat
.
D:\x265-r3.50.0.94\build\vc16-x86_64\10bit>cd..\8bit

D:\x265-r3.50.0.94\build\vc16-x86_64\8bit>cmake -G "Visual Studio 16 2019" -A x64 ../../../source -DCMAKE_SYSTEM_VERSION=8.1 -DEXTRA_LIB="x265-static-main10.lib;x265-static-main12.lib" -DLINKED_10BIT=ON -DLINKED_12BIT=ON
-- cmake version 3.26.0
.
Load the x265.sln into VS2019 IDE and switch to Intel 19.2 compiler, compile and copy the lib to the 8bit dir as per multilib.bat.
.
D:\x265-r3.50.0.94\build\vc16-x86_64\8bit>"C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30133\bin\Hostx64\x64\LIB.EXE" /ignore:4006 /ignore:4221 /OUT:Release\x265-static.lib x265-static-main.lib x265-static-main10.lib x265-static-main12.lib
Microsoft (R) Library Manager Version 14.29.30148.0
Copyright (C) Microsoft Corporation.  All rights reserved.
.
8bit\Release dir now contains:
libx265.dll
libx265.exp
libx265.lib
x265-static.lib
x265.exe
.
D:\x265-r3.50.0.94\build\vc16-x86_64\8bit>cd release

D:\x265-r3.50.0.94\build\vc16-x86_64\8bit\Release>x265 -h
x265 [INFO]: HEVC encoder version 3.50+94+-c5f3522 [Mod by Patman]
x265 [INFO]: build info [Windows][ICC 2021][64 bit] 8bit+10bit+12bit

No reason why it shouldn't work with Intel 19.0. There may be a much more elegant process but it's not that complicated and it follows the multilib.bat script step-by-step, and it works.

[john33]

If you just want the binary to test, I can provide it. The compile, above, has AVX2 compiler optimisations, but that's just using a modified CMakeLists.txt.

[jpsdr]

Thanks, but opening the sln, switching to Intel Compiler and "build", i've allready tried and it produced errors, but i'll try again and provide more feedback information.

[john33]

I've just done a quick comparison on a 42 minute TV episode (unfortunately I didn't retain the precise stats!) and I think the GCC 12.2.0 compile, with similar compiler optimisations was slightly (by the odd fps) faster than the Intel compile, but the Intel encode was slightly smaller - 292,544KB vs 292,973KB. No discernable difference in quality.

[jpsdr]

I've created the solution with the following command line :

G:\CMakex64\bin\cmake -G "Visual Studio 16" -A x64 ../../../source -DENABLE_VAPOURSYNTH=ON -DENABLE_AVISYNTH=ON -DHIGH_BIT_DEPTH=ON -DENABLE_HDR10_PLUS=ON -DEXPORT_C_API=OFF -DENABLE_SHARED=OFF -DENABLE_CLI=OFF -DMAIN12=ON -DSTATIC_LINK_CRT=OFF -DCMAKE_CXX_FLAGS_RELEASE="/sdl- /Oi /Ot /Oy /GT /GL /GF /GS- /Gy /Qpar /arch:AVX2"

Then open the sln, switch to Intel Compiler and build solution.
The translated result is :

Start generation...
1>------ Begin generation : Project : ZERO_CHECK, Configuration : Release x64 ------
1>Checking Build System
2>------ Begin generation : Project : common, Configuration : Release x64 ------
3>------ Begin generation : Project : dynamicHDR10, Configuration : Release x64 ------
4>------ Begin generation : Project : encoder, Configuration : Release x64 ------
5>------ ignore : Project : uninstall, Configuration : Release x64 ------
5>Project not selected

2>Building Custom Rule C:/PRG/x264/x265_git/source/common/CMakeLists.txt
2>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\MSBuild\Microsoft\VC\v160\Platforms\x64\PlatformToolsets\Intel C++ Compiler 19.1\Toolset.targets(347,5): error MSB4018: Unexpected faillure of task "ICL".
2>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\MSBuild\Microsoft\VC\v160\Platforms\x64\PlatformToolsets\Intel C++ Compiler 19.1\Toolset.targets(347,5): error MSB4018: System.ArgumentNullException: can't be null.
2>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\MSBuild\Microsoft\VC\v160\Platforms\x64\PlatformToolsets\Intel C++ Compiler 19.1\Toolset.targets(347,5): error MSB4018: Parameter name : key
2>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\MSBuild\Microsoft\VC\v160\Platforms\x64\PlatformToolsets\Intel C++ Compiler 19.1\Toolset.targets(347,5): error MSB4018:    à System.Collections.Generic.Dictionary`2.FindEntry(TKey key)
2>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\MSBuild\Microsoft\VC\v160\Platforms\x64\PlatformToolsets\Intel C++ Compiler 19.1\Toolset.targets(347,5): error MSB4018:    à System.Collections.Generic.Dictionary`2.TryGetValue(TKey key, TValue& value)
2>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\MSBuild\Microsoft\VC\v160\Platforms\x64\PlatformToolsets\Intel C++ Compiler 19.1\Toolset.targets(347,5): error MSB4018:    à Intel.Build.ICLTasks.ICL.IsOMPDisabled()
2>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\MSBuild\Microsoft\VC\v160\Platforms\x64\PlatformToolsets\Intel C++ Compiler 19.1\Toolset.targets(347,5): error MSB4018:    à Intel.Build.ICLTasks.ICL.set_OffloadOpenMPTarget(String value)
2>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\MSBuild\Microsoft\VC\v160\Platforms\x64\PlatformToolsets\Intel C++ Compiler 19.1\Toolset.targets(471,7): error MSB4026: The parameter"OffloadOpenMPTarget=%(ClCompile.OffloadOpenMPTarget)" for the task "ICL" is not valid.
2>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\MSBuild\Microsoft\VC\v160\Platforms\x64\PlatformToolsets\Intel C++ Compiler 19.1\Toolset.targets(347,5): error MSB4063: Impossible to initialise task "ICL" with the input parameters.
2>Generate project "common.vcxproj" end -- FAIL.

3>Building Custom Rule C:/PRG/x264/x265_git/source/dynamicHDR10/CMakeLists.txt
3> Same errors as above
3>Generate project "dynamicHDR10.vcxproj" end -- FAIL.

6>------ start generation : Project : hdr10plus-static, Configuration : Release x64 ------
4>Building Custom Rule C:/PRG/x264/x265_git/source/encoder/CMakeLists.txt
4> Same errors as above
4>Generate project "encoder.vcxproj" end -- FAIL.

7>------ Begin generation: Project : x265-static, Configuration : Release x64 ------
6>Building Custom Rule C:/PRG/x264/x265_git/source/CMakeLists.txt
7>Generating ipfilter16.asm.obj
6>LINK : : error LNK1181: unable to open input file 'C:\PRG\x264\x265_git\build\VS2019_Intel\12bit\dynamicHDR10\dynamicHDR10.dir\Release\json11.obj'
6>Generate projet "hdr10plus-static.vcxproj" end -- FAIL.

7>Generating h4-ipfilter16.asm.obj
7>Generating pixel-a.asm.obj
7>Generating const-a.asm.obj
7>Generating cpu-a.asm.obj
7>Generating ssd-a.asm.obj
7>Generating mc-a.asm.obj
7>Generating mc-a2.asm.obj
7>Generating pixel-util8.asm.obj
7>Generating blockcopy8.asm.obj
7>Generating pixeladd8.asm.obj
7>Generating dct8.asm.obj
7>Generating seaintegral.asm.obj
7>Generating sad16-a.asm.obj
7>Generating intrapred16.asm.obj
7>Generating v4-ipfilter16.asm.obj
7>Generating h-ipfilter16.asm.obj
7>Generating loopfilter.asm.obj
7>Building Custom Rule C:/PRG/x264/x265_git/source/CMakeLists.txt
7>LINK : : error LNK1181: unable to open input file 'C:\PRG\x264\x265_git\build\VS2019_Intel\12bit\encoder\encoder.dir\Release\analysis.obj'
7>Generate project "x265-static.vcxproj" end -- FAIL.
8>------ start generation : Project : ALL_BUILD, Configuration : Release x64 ------
8>Building Custom Rule C:/PRG/x264/x265_git/source/CMakeLists.txt
9>------ Generation ignored : Project : INSTALL, Configuration : Release x64 ------
9>Project not selected for this configuration 
========== Generation : 2 success, 5 failed, 0 à jour, 2 ignored ==========

[john33]

I noticed that your build was with "ENABLE_HDR10_PLUS=ON", so I have recompiled using my method and it compiles successfully. The only difference I can see between what you are attempting and what I am doing is that I have edited the "CMakeLists.txt" file to enable the build options whereas you are are specifying them in the command line. It would seem that my method works. Can I suggest that you try editing "CMakeLists.txt"?

[jpsdr]

Editing CMakeLists.txt is a little out off my skills... I don't realy know how to do it properly, i don't realy know how it works. If you can provide me an edited file.
And what files must be edited ? Only the one in "source", or also the others in the subdirectories ?

[john33]

CMakeLists.txt and amended multilib.bat within this zip file:https://www.rarewares.org/files/CMakeLists.zip

I'm no expert on this, but the following lines are the ones that have been amended/added:

options - 26-29 and 216-219.

compiler flags - 171-181(msvc and Intel) and 231(gcc).

Then manually run the command lines in multilib.bat line by line, but don't run the 'MSbuild' lines, load the generated sln file into VS2019, switch solution to whichever Intel compiler you have available and rebuild the solution. After each build, copy/rename the 12bit/10bit libs before you proceed. Whether you choose to run lib merge at the end of the batch file is up to you. The .exe has already been created.

Hope that helps.

[excellentswordfight]

Originally Posted by john33

I've just done a quick comparison on a 42 minute TV episode (unfortunately I didn't retain the precise stats!) and I think the GCC 12.2.0 compile, with similar compiler optimisations was slightly (by the odd fps) faster than the Intel compile

It was a few years ago I played with different compilers, but when I last did it that was my conclusion as well, the GCC build was the fastest. But the difference was still rather negatable, this is also somewhat expected that there isnt any major speed difference.

[john33]

Originally Posted by excellentswordfight

It was a few years ago I played with different compilers, but when I last did it that was my conclusion as well, the GCC build was the fastest. But the difference was still rather negatable, this is also somewhat expected that there isnt any major speed difference.

Indeed. Some years ago the Intel compiles (I was mainly compiling audio apps then) were notably quicker than most everything else but in recent years, that seems to have swung in favour of GCC, with the latest Clang not too far behind.

[jpsdr]

I downloaded your files, checked the differences with Winmerge, didn't see anything i though could fix, but nevertheless...
I replaced the CMakeLists.txt with your's, generate the solution with the exact same command lines in the .bat your provided.
Then open the .sln in VS, switch to Intel Compiler, generate, and... Unfortunately, as i was afraid of, no change, the exact same errors.

[john33]

OK, that's disappointing. Let me give it some thought and I'll get back to you.

[john33]

I just tried to compile this using Intel 19.0, I don't have 19.1, and it fails with a similar set of errors as yours. The problem must, therefore, be with the compiler. If you are able, and you wish to pursue this, you'll need to install the latest Intel compiler from the link above. It is a 'no cost' product. Alternatively, as already mentioned, if you only wish to use it for test purposes, I can provide the Intel 19.2 compiled binary.

[jpsdr]

I'm under Windows 7, so i can't install the new version. There must be some kind of settings in the CMakeLists the compiler don't like... Or some setting missing... But wich one...?
For now, i'm dropping this.

[john33]

Originally Posted by jpsdr

I'm under Windows 7, so i can't install the new version. There must be some kind of settings in the CMakeLists the compiler don't like... Or some setting missing... But wich one...?
For now, i'm dropping this.

Ah, now that's question!! Unfortunately, I don't have an answer at the moment. I will probably revisit this over time as I don't like unanswered questions like this. If I find an answer I'll be sure to let you know.

[jpsdr]

Ok, thanks.

[DTL]

Originally Posted by john33

I've just done a quick comparison on a 42 minute TV episode (unfortunately I didn't retain the precise stats!) and I think the GCC 12.2.0 compile, with similar compiler optimisations was slightly (by the odd fps) faster than the Intel compile, but the Intel encode was slightly smaller - 292,544KB vs 292,973KB.

It is very strange and sad. Looks many 'big' software projects finally end in unstable computing results from different build tools and it may cause either 'simple' errors in output or some critical compute errors and random crashes.

[benwaggoner]

Originally Posted by john33

I've just done a quick comparison on a 42 minute TV episode (unfortunately I didn't retain the precise stats!) and I think the GCC 12.2.0 compile, with similar compiler optimisations was slightly (by the odd fps) faster than the Intel compile, but the Intel encode was slightly smaller - 292,544KB vs 292,973KB. No discernable difference in quality.

I believe that using --frame-threads >1 can cause that sort of indeterminacy. You might want to try again with that option set. Depending on the resolution you're encoding and how many cores you are, setting it to 1 might slow you down a bunch or potentially speed you up a bit. If you wind up with lots of idle cores, you could try using --pmode to get some speed back.

[Balling]

Originally Posted by john33

I've just done a quick comparison on a 42 minute TV episode (unfortunately I didn't retain the precise stats!) and I think the GCC 12.2.0 compile, with similar compiler optimisations was slightly (by the odd fps) faster than the Intel compile, but the Intel encode was slightly smaller - 292,544KB vs 292,973KB. No discernable difference in quality.

First of all new Intel compiler uses clang. Second of all gcc 12.2.0 is like the worst gcc release in years. So many horrific bugs.