FFmpegSource - C-plugin

[qyot27]

Since it's easier, I'll keep this first post updated with newer builds whenever one happens.

Issues tracker (for C plugin issues only): https://github.com/qyot27/ffms2_cplugin/issues

FFMS2 C-plugin r1352+142

Optimized for Pentium III and SSE (32-bit)
Optimized for Core2 (64-bit)

ffmpeg version N-102567-g9b131e8500 Copyright (c) 2000-2021 the FFmpeg developers
built with gcc 11.1.0 (GCC)
libavutil 57. 0.100 / 57. 0.100
libavcodec 59. 1.100 / 59. 1.100
libavformat 59. 2.100 / 59. 2.100
libavfilter 8. 0.101 / 8. 0.101
libswscale 6. 0.100 / 6. 0.100
libswresample 4. 0.100 / 4. 0.100
libpostproc 56. 0.100 / 56. 0.100

32-bit configuration:

    --prefix=/home/qyot27/ffmpeg_build_for_ffms2/32bit
    --cross-prefix=x86_64-w64-mingw32-
    --enable-gpl
    --enable-version3
    --disable-w32threads
    --disable-encoders
    --disable-muxers
    --disable-doc
    --disable-debug
    --disable-devices
    --disable-avdevice
    --enable-libdav1d
    --enable-avisynth
    --disable-decoder=aac_fixed
    --disable-decoder=ac3_fixed
    --disable-decoder=mp1
    --disable-decoder=mp2
    --disable-decoder=mp3
    --disable-decoder=mp3adu
    --disable-decoder=mp3on4
    --cpu=pentium3
    --extra-cflags='-m32 -mfpmath=sse -march=pentium3 -msse -mtune=pentium3'
    --extra-ldflags='-m32 -L/usr/i686-w64-mingw32/lib'
    --target-os=mingw32
    --arch=x86

64-bit configuration:

    --prefix=/home/qyot27/ffmpeg_build_for_ffms2/64bit
    --cross-prefix=x86_64-w64-mingw32-
    --enable-gpl
    --enable-version3
    --disable-w32threads
    --disable-encoders
    --disable-muxers
    --disable-doc
    --disable-debug
    --disable-devices
    --disable-avdevice
    --enable-libdav1d
    --enable-avisynth
    --disable-decoder=aac_fixed
    --disable-decoder=ac3_fixed
    --disable-decoder=mp1
    --disable-decoder=mp2
    --disable-decoder=mp3
    --disable-decoder=mp3adu
    --disable-decoder=mp3on4
    --cpu=core2
    --extra-cflags='-march=core2'
    --target-os=mingw32
    --arch=x86_64

FFMS2 C-plugin r1315+119 (Final LastXP build)

Optimized for Pentium-III and SSE (32-bit)
Optimized for Core2 (64-bit)

ffmpeg version r90798 master-21da248b5f HEAD-a56580b117
contains: lastxp_wincrypt
Copyright (c) 2000-2018 the FFmpeg developers
built with gcc 7.2.0 (GCC)
libavutil 56. 13.100 / 56. 13.100
libavcodec 58. 17.100 / 58. 17.100
libavformat 58. 11.101 / 58. 11.101
libavfilter 7. 15.100 / 7. 15.100
libavresample 4. 0. 0 / 4. 0. 0
libswscale 5. 0.102 / 5. 0.102
libswresample 3. 0.101 / 3. 0.101
libpostproc 55. 0.100 / 55. 0.100

32-bit configuration:

    --prefix=/home/qyot27/ffmpeg_build_for_ffms2/32bit
    --cross-prefix=i686-w64-mingw32-
    --enable-gpl
    --enable-version3
    --disable-w32threads
    --enable-avresample
    --disable-encoders
    --disable-muxers
    --disable-doc
    --disable-debug
    --disable-devices
    --disable-avdevice
    --extra-cflags='-mfpmath=sse -march=pentium3 -msse -mtune=pentium3'
    --target-os=mingw32
    --arch=x86

64-bit configuration:

    --prefix=/home/qyot27/ffmpeg_build_for_ffms2/64bit
    --cross-prefix=x86_64-w64-mingw32-
    --enable-gpl
    --enable-version3
    --disable-w32threads
    --enable-avresample
    --disable-encoders
    --disable-muxers
    --disable-doc
    --disable-debug
    --disable-devices
    --disable-avdevice
    --extra-cflags='-march=core2'
    --target-os=mingw32
    --arch=x86

FFmpeg has now switched from wincrypt to bcrypt, and this is too far-reaching of a change for any selective revert to handle - the system library it links against changed (and bcrypt.dll is not available on XP; it's part of Vista+'s API), and far more importantly, the code that requires the random_seed functions that the *crypt library is called in for is spread throughout the FFmpeg codebase - it's used in encoders, decoders, and [de]muxers, so unlike the winthread/pthread thing last time, this time XP compatibility is probably broken for good. While it could probably hang on for a while doing increasingly labor-intensive merge integrations to revert the bcrypt commits, make sure everything still compiles, and that no unwanted behavior arises because of the use of winthread, I'm not particularly enthused to do so, and the alternatives (Wine, ReactOS, using Wine in Windows if you can get it compiled) are more compelling in the long term.

[Reino]

Originally Posted by qyot27

There was a vote regarding this on the FFmpeg-devel mailing list a couple weeks ago (thread starts here), so as of commit 9b121dfc32810250938021952aab4172a988cb56, support for Windows XP was dropped in FFmpeg-git, which now requires the use of the newer lock and thread APIs introduced in Vista.

I've seen the vote and as far as I know this only concerns w32threads.
https://ffmpeg.org/pipermail/ffmpeg-devel/2017-December/222869.html:

...
Windows XP is hereby not a supported build target anymore. It was
decided in a project vote that this is OK. (Technically, it could still
be built for Windows XP using an external pthread lib as of this
commit.)

I've just released another WinXP compatible FFmpeg build using pthreads. I haven't extensively tested it myself, but so far it's working just fine.

[lvqcl]

Originally Posted by CoRoNe

I've seen the vote and as far as I know this only concerns w32threads.

No, the vote was to drop WinXP support:
"We should drop XP support, and allow unconditional use of Windows Vista APIs"

Originally Posted by CoRoNe

I've just released another WinXP compatible FFmpeg build using pthreads. I haven't extensively tested it myself, but so far it's working just fine.

Looks like adding --disable-w32threads option is currently enough to restore ffmpeg compatibility with WinXP. But it can break with any new commit.

[manolito]

Thanks so much qyot27 for this new version...

As opposed to the previous version this one works under plain vanilla AviSynth v2.6 again. I tested it under WinXP with a non-SSE2 capable CPU, no problems whatsoever.


Thanks again and Cheers
manolito

[qyot27]

Originally Posted by lvqcl

Looks like adding --disable-w32threads option is currently enough to restore ffmpeg compatibility with WinXP. But it can break with any new commit.

I sort of forgot that w32threads were a separate method. And I actually do use --disable-w32threads (IIRC, because FFMS2's C++11 threading requires MinGW-w64's win32-pthreads, and I can't remember if w32threads plays nice with that), so I guess it might actually hang on a while longer.

The uncertainty about it breaking somewhere deeper at any time, though...

Originally Posted by manolito

As opposed to the previous version this one works under plain vanilla AviSynth v2.6 again.

It's been quite some time since I did the update to support AviSynth+'s huge number of new colorspaces in the C plugin parts, so I don't remember whether I actually fixed that differently than the previous build or not. Good to hear that it's working, though.

[StainlessS]

Thank you Q Branch, your continued support has always been appreciated (now where did I leave Moneypenny?).

[FranceBB]

@qyot27... thank you very much indeed for supporting XP

[TheFluff]

y'all XP diehards might want to take note of this new shiny CPU exploit that you will most likely never get a patch for
if you do dangerous things such as "using a web browser with javascript enabled" you are vulnerable

[manolito]

Originally Posted by TheFluff

y'all XP diehards might want to take note of this new shiny CPU exploit that you will most likely never get a patch for
if you do dangerous things such as "using a web browser with javascript enabled" you are vulnerable

Yes, and according to this:
https://www.theregister.co.uk/2018/0...u_design_flaw/
I will also not get a slowdown of my computer for up to 30% which these (future) patches will introduce...

Cheers
manolito

[lvqcl]

ffmpeg-related tasks are usually computationally intensive, so performance penalty for them should be close to 0.

P.S. anyway, even 30% performance hit is better than a new ransomware like WannaCry or (Not)Petya.

[Andouille]

Originally Posted by TheFluff

y'all XP diehards might want to take note of this new shiny CPU exploit that you will most likely never get a patch for
if you do dangerous things such as "using a web browser with javascript enabled" you are vulnerable

What does this have to do with video processing ?



@qyot27... thank you

[manolito]

Originally Posted by qyot27

Good to hear that it's working, though.

After a long testing session I have to report that this new version indeed works nicely, but only under WinXP. Under Win7-64bit it causes problems...

This seems weird, normally it works the other way around, so I was sceptical and tested it on 3 different machines. My ancient desktop PC running WinXP and two newer machines with Intel Core i5 CPUs running under Win7-64bit.

I used AVStoDVD for testing. This software comes with ffms2_r1140+101-avs+vsp.7z from Oct 2016. Replacing ffms2 with this new build causes no problems under WinXP. But on the two Win7 machines the encoders (HCenc and FFmpeg) as well as Wavi crash after they have finished their work. The terrible Windows popup telling the user that the software has stopped working because of some problem. And the software needs to be closed down. No further explanation.

I tried some workarounds like adding the folder from where ffms2 is loaded by AVStoDVD to the environment path variable. I also disabled DEP completely, but nothing helped. I ended up going back to the old ffms2 version from Oct 2016.


Any ideas what is happening here?

Cheers
manolito

[Midzuki]

Originally Posted by manolito

After a long testing session I have to report that this new version indeed works nicely, but only under WinXP. Under Win7-64bit it causes problems...

This seems weird, normally it works the other way around, so I was sceptical and tested it on 3 different machines. My ancient desktop PC running WinXP and two newer machines with Intel Core i5 CPUs running under Win7-64bit.

I used AVStoDVD for testing. This software comes with ffms2_r1140+101-avs+vsp.7z from Oct 2016. Replacing ffms2 with this new build causes no problems under WinXP. But on the two Win7 machines the encoders (HCenc and FFmpeg) as well as Wavi crash after they have finished their work. The terrible Windows popup telling the user that the software has stopped working because of some problem. And the software needs to be closed down. No further explanation.

Just confirming, yes the latest C plugin is problematic. Tested with avs2yuv and x265.exe under 64-bit Windows 7. avs2yuv crashed after the encoding was finished.

[StainlessS]

ancient desktop PC running WinXP

So WXP_32 and W7_64, did anyone try WXP_64 ? [EDIT: or indeed W7_32]

[manolito]

Originally Posted by Midzuki

Just confirming, yes the latest C plugin is problematic. Tested with avs2yuv and x265.exe under 64-bit Windows 7. avs2yuv crashed after the encoding was finished.

Thanks for confirming my findings...

The thing which I do not understand is why it works under WinXP. Of course the old desktop computer has a single core CPU (Celeron Coppermine 1.1 Ghz, made in the year 2000), and it has very low system RAM (576 MB). But it runs the same version of AviSynth (2.61 Alpha VC6) as the other Win7-64 computers.

So I tried a few more things under Win7-64. First I disabled Hyperthreading, then I completely disabled multicore processing. I added the "threads=1" parameter to ffvideosource. And I tried to force AVStoDVD to only use 1 CPU core globally. Nothing helped.

I could not even get AVSMeter to create a report about the AVS script which used ffms2. AVSMeter crashed immediately.


So I have to conclude that this latest version of the qyot27 C-Plugin does not work under a 64-bit OS using the standard AVS versions (I forgot to mention that I also tried using different builds of AVS versions 2.60 and 2.61). I did not test it under AVS+ because I have no intention to upgrade to AVS+ in the forseeable future.


Cheers
manolito

[qyot27]

It's been fixed locally. I'll wait to post a new build until all this recent C-plugin discussion has been split off to a new thread, though.

The 'why does it work on XP?' issue had me stumped (I'd mentioned it in passing with the build before this), but as this MSDN thread notes is more than likely down to XP's heap manager not being as comprehensive about warning users about corruptions that occur. So the bug is still there, but XP doesn't bother telling you about it.

The actual issue itself may have been a longstanding bug from back when kemuri-9 was the one doing C-plugin stuff (since the actual line that fails was virtually unchanged since then) or it was a regression that happened when the allocation method for AviSynth.dll changed recently to get rid of calloc (in which case it'd be my fault for not noticing it was trying to NULL the avs_lib after it had been freed).

[Selur]

It's been fixed locally. I'll wait to post a new build until all this recent C-plugin discussion has been split off to a new thread, though.

Any new on the new build?

[qyot27]

Originally Posted by Selur

Any new on the new build?

Yup.

FFMS2 C-plugin r1293+111

It's built against the same FFmpeg as last time, so I won't bother with that infoblock again.

EDIT 2018-04-23: Newer build available here.

[burfadel]

What's the code separation like between the main branch and the FFMS2000 experimental branch?

[Midzuki]

Originally Posted by qyot27

Yup.

FFMS2 C-plugin r1293+111

It's built against the same FFmpeg as last time, so I won't bother with that infoblock again.

for the new build, now it works without causing crashes.