Issue with GCC 9.1.0 and ffms2 and x264

[jpsdr]

I don't know exactly where to put the information (here or in h264), so for now i put it here.

I build :
- ffmpeg 4.1.4.
- ffms2 (last commit a9e8f73 of 19/03/2019.)
- lsmash (last git commit 88208b5 of 25/02/2019.)
and after, i build x264.

Under MSYS2, with GCC provided by msys2 (get with "pacman") :
With version 7.4.0 Rev1 (x86) and 8.3.0 Rev2 (x64) ffms2 is included and checked/tested "Ok" by the configure script of x264.

With version 9.1.0 Rev3 (x86/x64), ffms2 is not included, it seems that the check/test by the configure script fails.

The code and the building scripts are of course exactly the same in all cases.

[MasterNobody]

Have you looked into config.log why it fails? Or at least pastebin it somewhere so we can look it also.

[jpsdr]

Pastebin config.log :
Fail with gcc 9.1.0 Rev3 (x86) here (https://pastebin.com/rLhuZNJp).
Success with gcc 7.4.0 Rev1 (x86) here (https://pastebin.com/cDdquBSx).

[MasterNobody]

Looks like problem is with pkg-config. Can you also pastebin generated ffms2.pc files?

[jpsdr]

ffms2.pc created by :
gcc 9.1.0 (fail) here (https://pastebin.com/stNwqHaS).
gcc 8.1.0 (success) here (https://pastebin.com/cAxcGzZU).
Except directory wich is different (but it's normal as i use 8.1.0 for the winthread build, and 9.1.0 for posix build), i don't see any difference.
Didn't keep the 7.4.0 posix, but if realy needed, i can do it (it's just that i'll have to redo all the ffmpeg and ffms2 build).

[MasterNobody]

Hm, looks same. Can you check what pkg-config --libs ffms2 and pkg-config --cflags ffms2 returns for both of them (gcc 9.1.0/gcc 8.1.0)?

[jpsdr]

gcc 9.1.0 :
-LG:/msys32/local-x264_x86/posix/t_mod/lib -lffms2

-IG:/msys32/local-x264_x86/posix/t_mod/include

gcc 8.1.0 :
-L/local-x264_x86/posix/t_mod/lib -LG:/msys32/mingw32/lib -L/local-x264_x86/posix/t_mod/lib -LG:/msys32/mingw32/lib -L/local-x264_x86/posix/t_mod/lib -lz -lffms2 -lavformat -lm -lbz2 -lz -lavcodec -pthread -liconv -lm -llzma -lz -lswscale -lm -lswresample -lm -lavutil -pthread -lm -luser32 -lbcrypt

-I/local-x264_x86/posix/t_mod/include

gcc 7.4.0 :
-L/local-x264_x86/posix/t_mod/lib -LG:/msys32/mingw32/lib -L/local-x264_x86/posix/t_mod/lib -LG:/msys32/mingw32/lib -L/local-x264_x86/posix/t_mod/lib -lz -lffms2 -lavformat -lm -lbz2 -lz -lavcodec -pthread -liconv -lm -llzma -lz -lswscale -lm -lswresample -lm -lavutil -pthread -lm -luser32 -lbcrypt

-I/local-x264_x86/posix/t_mod/include

For the three test cases, i've configured PKG_CONFIG_PATH with the exact same location.

[MasterNobody]

For the three test cases, i've configured PKG_CONFIG_PATH with the exact same location.
If PKG_CONFIG_PATH was the same than what was changed? pkg-config executable? Than it is the culprit.

[jpsdr]

For now, it seems.

[jpsdr]

Odds ! "pkg-config --version" returns 0.29.2 in all cases, when "pkg-config --libs ffms2" returns different results for the exact same PKG_CONFIG_PATH, and so for also the exact same ffms2.pc file. :confused:

But... I've noticed that there is two pkg-config.exe files.
One in "mingw32/bin" and another in "usr/bin".

But, if i go in the directory where there is pkg-config.exe and ask the version :
"usr/bin" returns version 0.29.2.
GCC 9.1.0 "mingw32/bin" returns 0.29.2.
GCC 8.1.0 "mingw32/bin" returns 0.29.2.

Ok, i'm totaly :eek:

[filler56789]

Odds ! "pkg-config --version" returns 0.29.2 in all cases, when "pkg-config --libs ffms2" returns different results for the exact same PKG_CONFIG_PATH, and so for also the exact same ffms2.pc file. :confused:

But... I've noticed that there is two pkg-config.exe files.
One in "mingw32/bin" and another in "usr/bin".

But, if i go in the directory where there is pkg-config.exe and ask the version :
"usr/bin" returns version 0.29.2.
GCC 9.1.0 "mingw32/bin" returns 0.29.2.
GCC 8.1.0 "mingw32/bin" returns 0.29.2.

Ok, i'm totaly :eek:

It seems the MSYS2 devilopers have skrewed royally their latest MSYS2 releases... this is the reason why I returned to their builds from the year 2016 and decided to NEVER AGAIN invoke the command "pacman -Syuu".

[filler56789]

You're really spreading FUD.

The reason why jpsdr has two pkg-config.exe in different directory is that he installed both mingw-w64-i686-pkg-config and pkg-config packages. They don't conflict with each other. Which one will get executed depends on the shell type. msys_shell -mingw32 will execute /mingw32/bin/pkg-config and msys_shell -msys will execute /usr/bin/pkg-config.

Thanks for the detailed info.
Anyway it won't be easy to make me change my mind.
Last week I was playing around with "the latest and greatest" :rolleyes: MSYS2 and found out that the mingw32 and mingw64 directories were full of "strange" and "garbage" stuff — in other words, full of files that were not supposed to be there...

[jpsdr]

Building ffms2 has always been tricky.

To build x264 with ffms2, you need a little modification in ffms.h (thanks Komisar).

About GCC, for posix thread model, i use the one provided with MSYS2 (get with pacman).
For win32 thread model, i use one provided here : https://sourceforge.net/projects/mingw-w64/files/Toolchains%20targetting%20Win32/Personal%20Builds/mingw-builds/

When you build ffms2, there is first a lot of tests done, and if they all pass Ok, the build begins.

If i build ffms2 with the MSYS2 posix GCC version, the build goes to the end properly, and finish Ok. (*)
If i try to build ffms2 with a posix GCC version from the link above, the build fails at the end during the link of the EXE file. With this version, the build never ends properly. But, except the exe file, all the others files (libraries) are build properly.

To build ffms2 with a win32 thread model GCC it's lot tricky. You need a not so trivial patch (at least for me), including putting back numthreads files, again, thanks Komisar.
Now, after patched, if you try to build ffms2 with a win32 GCC version from the link above, the build fails at the end during the link of the EXE file. But, except the exe file, all the others files (libraries) are build properly. You have to install the files manualy.

(*) : Well... That was true until around a year ago. Now, even with the MSYS2 posix GCC, the build fails at the end during the link of the exe. For now, personnaly, i don't know a GCC version which can build ffms2... It's not a big deal, as i don't need the EXE files, and all the other stuff i need are made properly (libraries, .pc file, etc...).

Unlike ffmpeg where the is a --disable-programs option, practical when you just want to build/install the libraries and don't need the EXE files, i've not been able to find such an option for ffms2.
It would be very practical, because it would prevent the build to fail at the end, and allow make install to install...

Out of curiosity, i took the GCC provided by filler56789. Big and good surprise, it was a win32 thread model.
So i build ffmpeg, build ffms2... no... This version unfortunately doesn't even begin to build, it fails during the preliminary tests... :(
Edit :
It fails with the following :

checking whether the i686-w64-mingw32-g++ linker (g:/msys32/mingw32/i686-w64-mingw32/bin/ld.exe) supports shared libraries... yes
checking dynamic linker characteristics... Win32 ld.exe
checking how to hardcode library paths into programs... immediate
checking zlib.h usability... no
checking zlib.h presence... no
checking for zlib.h... no
checking for inflateEnd in -lz... no
configure: error: in `/g/x264/ffms2/Build_x264':
configure: error: cannot locate zlib.h and -lz, specify a valid zlib installation using --with-zlib=DIR
See `config.log' for more details
make: *** Pas de cible spécifiée et aucun makefile n'a été trouvé. Arrêt.
make: *** Aucune règle pour fabriquer la cible «*install*». Arrêt.

And if i run pacman -S --needed zlib it says that zlib is allready installed and up to date...
Another ffms2 mystery of why it fails here with this GCC version, and not with the others GCC version.
Tested with 8.3.0 and 9.2.0, same results, fails here on the zlib check...

[jpsdr]

Just tested the fresh new 9.2.0 MSYS2 GCC version, still no ffms2 when building x264... :(

I begin to wonder if it's not a GCC 9.x issue...:confused:

[Wolfberry]

Your issue sounds like a broken environment.

And if i run pacman -S --needed zlib it says that zlib is allready installed and up to date...

You want mingw-w64-i686-zlib (for 32bit) or mingw-w64-x86_64-zlib (for 64bit), if you are compiling native binaries for windows.

I build ffms2 using the meson build system from HolyWu: https://github.com/FFMS/ffms2/pull/338, and adds -DFFMS_STATIC to Cflags in ffms2.pc

No need for any patches from Komisar.

[jpsdr]

Tested to run pacman -S --needed mingw-w64-i686-zlib but it says allready installed and up to date.
Issue is that swapping gcc version by only renaming directories. Once zlib is installed, it seems it doesn't check in the directory if it exist, but it checks in a database if it's installed.
Finaly i think, when i swap gcc version renaming directories, even if i swap to a directory without zlib, for the environment zlib is installed.
The gcc versions i usualy use are probably provided with zlib allready installed.

[filler56789]

Maybe off-topic but surely necessary:

pacman -S --needed mingw-w64-i686-zlib

Packages named mingw-w64-* should not even exist to begin with, since the (supposed) goal of MSYS2 is emulate a unix(oid) building-environment on Windows as much as possible.

And especially when they are placed inside the /mingw##/bin/ directories.

</END_OF_RANT>

[StvG]

Tested to run pacman -S --needed mingw-w64-i686-zlib but it says allready installed and up to date.
Issue is that swapping gcc version by only renaming directories. Once zlib is installed, it seems it doesn't check in the directory if it exist, but it checks in a database if it's installed.
Finaly i think, when i swap gcc version renaming directories, even if i swap to a directory without zlib, for the environment zlib is installed.
The gcc versions i usualy use are probably provided with zlib allready installed.

Clean Pacman cache (pacman -Scc) and reinstall zlib.

I'm preferring this GCC for MSYS2 - https://gcc-mcf.lhmouse.com/ ; https://github.com/lhmouse/mcfgthread/wiki ; https://github.com/lhmouse/mcfgthread/wiki/Benchmarking

[jpsdr]

Tested with pacman -S --overwrite --needed mingw-w64-i686-zlib (as --force is deprecated). I've seen the zlib stuff appears in the mingw32 directory of filler56789 GCC version. Unfortunately, ffms2 build with it is still failling on the zlib test.
I'll stop here, it's not like if i haven't allready others GCC versions working.
I'll take a look at StvG links, thanks for these.

[jpsdr]

Tested ffms2 integration in x264 with the 8.3.1 from the link provided by StvG. ffms2 builds, but unfortunately, the integration in x264 fails.
The pkg-config --libs ffms2 returns the same value than 9.1.0 (or 9.2.0) MSYS2 GCC (see previous post).
Can't test a 8.1.x because these GCC versions begin with 8.3.1.

As far as i know, it doesn't exist mcf thread build option for ffmpeg, ffms2 or x264, only posix or win32threads. What is the closets ? My guess would be Win32threads, but...

[StvG]

Yes, Win32threads.

About ffms2 issue linking EXE file I'm using this patch (line #1348 + #1349) (https://github.com/jb-alvarado/media-autobuild_suite/blob/master/build/media-suite_compile.sh#L1348)

About ffms2 integration in x264 this works fine --extra-ldflags="$(PKG_CONFIG_PATH=/usr/i686-w64-mingw32/lib/pkgconfig pkg-config --libs --static ffms2)" (https://forum.doom9.org/showthread.php?p=1828372). Just tested with 8.3.1 x86 from that gcc-mcf link.

[jpsdr]

and adds -DFFMS_STATIC to Cflags in ffms2.pc


My building script for ffms2 is (setting part of _cf, etc... skipped) :

PKG_CONFIG_PATH="${_dir}/lib/pkgconfig" \
CFLAGS="${_cf}" \
CXXFLAGS="${_cxx}" \
LDFLAGS="${_ld}" \
../configure --prefix=${_dir} --build="i686-w64-mingw32" --host=${_host} \
--enable-static --disable-shared
make -j1
make install


What should i add in it for the -DFFMS_STATIC be automaticaly include to Cflags in ffms2.pc ?

EDIT :
I think i just can patch ffms2.pc.in... I'll test.

EDIT 2 :
Thanks to StvG and Wolfberry, i've been able to fix the exe error link and the integration, but still needed a little patch for this.

[jpsdr]

@StvG
What do i have to add in the compile (or linker) command line to avoid the message telling me that mcfgthreads-12.dll is missing ?
I build everything in static, but clearly it's not enough... :(

[Wolfberry]

Did you read https://github.com/lhmouse/mcfgthread/wiki#distribution-notes ?

At the moment, mcfgthread can't be linked statically and the file libmcfgthread.a is a copy of libmcfgthread.dll.a. Hence, if you want to distribute a program linked againast mcfgthread, you must always distribute mcfgthread-*.dll with it.

You may try gcc build provided here: http://msystem.waw.pl/x265/

[jpsdr]

Argh... I just found out, i was about to change my post, but too late. Thanks anyway, i'll test.

[jpsdr]

Thanks, these GCC work, but it seems that with win32 thread model GCC, ffms2 needs Komisar's not so trivial patch (at least for me), including putting back numthreads files.

[StvG]

You can use this C++11 implementation https://github.com/Cwc-Test/mingw-std-threads instead that Komisar's patch.
Copy all mingw.*.h files to /mingw32/include/c++/**/
** - GCC version
Then from ffms2 root folder:
sed -i 's/#include <mutex>/#include "mingw.mutex.h"/g' src/core/ffms.cpp
sed -i 's/#include <thread>/#include "mingw.thread.h"/g' src/core/videosource.cpp

[jpsdr]

Thanks, i'll test, it's seem "cleaner" than Komisar's patch.

EDIT
Thanks, this works fine. I prefer this where there is no need to tweak the ffms2 code, just the headers.

[jpsdr]

Thanks to all who have helped me, i have now a working good build process.