Windows XP compatible non-SSE2 FFmpeg binaries

[FranceBB]

@Liisachan... After a few tests and wrapping my head around this, it looks like I got it wrong.
Basically you're absolutely right and it's effectively the other way round, namely that the ffms2 C plugin here only supports UTF-8 Script + UTF-8 Avisynth, but it doesn't support ANSI WinLatin.

As per the finding of the post in the AVSPmod mod topic, I updated the list with the new proper tests and it's actually more interesting than I initially thought:



Version: AvsPmod 2.7.6.0 x64
OS: Windows 10 Enterprise x64
Avisynth: 3.7.3 r4013 x64

Test file: àèéìòù.mkv

Script Encoding: ANSI
Script Avisynth encoding: System locale mbcs

FFVideoSource() r1387 ok
LWLibavVideoSource() v1156 ok
DirectShowSource() ok

Script Encoding: UTF-8
Script Avisynth encoding: System locale mbcs

FFVideoSource() r1387 ok
LWLibavVideoSource() v1156 ok
DirectShowSource() ok


Script Encoding: UTF-8
Script Avisynth encoding: UTF-8

FFVideoSource() r1387 ok
LWLibavVideoSource() v1156 ok
DirectShowSource() fail




Version: AvsPmod 2.7.6.0 x86
OS: Windows XP Professional x86
Avisynth: 3.7.3 r4013 x86

Test file: àèéìòù.mkv

Script Encoding: ANSI
Script Avisynth encoding: System locale mbcs

FFVideoSource() 2.40 (C Plugin) 2023-09-04 fail
LWLibavVideoSource() 2015-03-16 ok
DirectShowSource() ok


Script Encoding: UTF-8
Script Avisynth encoding: System locale mbcs

FFVideoSource() 2.40 (C Plugin) 2023-09-04 fail
LWLibavVideoSource() 2015-03-16 ok
DirectShowSource() ok


Script Encoding: UTF-8
Script Avisynth encoding: UTF-8


FFVideoSource() 2.40 (C Plugin) 2023-09-04 ok
LWLibavVideoSource() 2015-03-16 ok
DirectShowSource() fail



So, to recap:


ffms2 r1387
- Supports ANSI Script + mbcs Avisynth
- Supports UTF-8 Script + mbcs Avisynth
- Supports UTF-8 Script + UTF-8 Avisynth

ffms2 c plugin 2023-09-04 (the one from this topic)
- Supports UTF-8 Script + UTF-8 Avisynth
(anything else will only work if the file doesn't have accents)

libav v1156
- Supports ANSI Script + mbcs Avisynth
- Supports UTF-8 Script + mbcs Avisynth
- Supports UTF-8 Script + UTF-8 Avisynth

libav 2015-03-16
- Supports ANSI Script + mbcs Avisynth
- Supports UTF-8 Script + mbcs Avisynth
- Supports UTF-8 Script + UTF-8 Avisynth


DirectShowSource
- Supports ANSI Script + mbcs Avisynth
- Supports UTF-8 Script + mbcs Avisynth



So the question for Reino now actually shifted to whether he's willing to support Enhanced ANSI like WinLatin ANSI via system locale (mbcs) for the ffms2 C plugin by doing what StvG is doing in the normal ffms2 r1387 plugin or not.


Apologies for the whole mess, but it was actually far more complicated than I originally anticipated.

[Liisachan]

Originally Posted by FranceBB

Apologies for the whole mess, but it was actually far more complicated than I originally anticipated.

Thanks for a lot of testing! Maybe UTF-8 is the way to go because some file names can’t be encoded in legacy ANSI. For example, if the file name is “Suzumiya Haruhi no Yūutsu.mkv", the letter ū is not supported by ANSI no matter whether your locale is ja-JP or WinLatin.

Another subtlety: e.g. WinLatin has the letter œ as 0x9C while this character is not U+009C but U+0153 in Unicode (incl. UTF-8 encoding). So, to support various encodings, the program basically needs to have a lot of big conversion tables.

Although there are things like `MultiByteToWideChar` & `WideCharToMultiByte` which do conversations for you, they too have a lot of pitfalls: e.g. é can be written as two characters:
U+0065 [ e ] LATIN SMALL LETTER E: U+0301 [ ́ ] COMBINING ACUTE ACCENT
instead of just U+00E9 [ é ] LATIN SMALL LETTER E WITH ACUTE, meaning the conversation is not one-to-one.

On the other hand, even using UTF-8, older versions of Windows may be bad at handling non-BMP code points (U+10000 and above). One way to test this would be including an emoji (a familiar example of non-BMP) in your file name such as: “I LOVE �� French fries.mkv” Though maybe no one would ever use such a weird file name, Unicode can be a huge headache if not fully supported on the OS level… Practically, just supporting UTF-8 (esp. BMP, below U+FFFF) is simple and sufficient?

Happy holidays!

PS: This forum itself doesn’ṭ support non-BMP ^^; The character between “I LOVE” and “French fries” is a single code point U+1F35F, but it seems that they are treated here as 2 “letters” (a surrogate pair).

[Reino]

Another round of binaries. Yes, I've still managed to compile some!

I'm afraid Gianluigi won't be updating his repo anymore, seeing his latest commit is from a year ago. So unless there's someone who can help me with '0003-load-shared-libfdk-aac-library-dynamically.patch', my builds will be without libfdk-aac, because sadly I still can't code in C/C++.

I've managed to add support for JPEG XL (libjxl). It took me quite some time and a lot of trial and error, but it works.

The cryptography-library MbedTLS can't be updated anymore. I found out the hard way that v3.5.0 (and v2.28.5) now requires Python 3.8. Next time I'll have a look at whether LibTLS (part of LibreSSL) might be substitute.

Originally Posted by FranceBB

Version ffms2-2.18-756-4d45afc-avs-vsp-win32-shared-xpmod-sse from 2023-05-04 throws a "platform returned code 126: the specified module could not be found" error as if it couldn't really load ffms2.dll.

That's because, as I said, 'ffms3.dll' now relies on 'avcodec-60.dll', 'avformat-60.dll', 'avutil-58.dll', 'swresample-4.dll' and 'swscale-7.dll' from the shared build.

Originally Posted by FranceBB

about version ffms2-2.18-756-4d45afc-avs-vsp_ffmpeg-5.2-2131-fcd557a-win32-xpmod-sse from 2023-01-08, it looks like it doesn't fully support UTF-8 characters.

I've seen the discussion in the other threads, but I just compile the source is all I can say. I really don't an answer to this other than that the source would probably be to blame.

Originally Posted by FranceBB

So the question for Reino now actually shifted to whether he's willing to support Enhanced ANSI like WinLatin ANSI via system locale (mbcs) for the ffms2 C plugin by doing what StvG is doing in the normal ffms2 r1387 plugin or not.

Which is? Correct me if I'm wrong, but shouldn't qyot27 be looking is this then? It's his repo.

Originally Posted by j7n

You could increase the sevenzip dictionary size to 96 MB to create a significantly smaller package of around 30 MB for the static build.

I wasn't aware. It took me quite some time to figure out the correct commandline. First of all the (old) 7zip binary from the Cygwin install doesn't work in this case. I'm getting: ERROR: Can't allocate required memory!. So I had to do this manually with the local 7-Zip 23.01 install (outside of the Cygwin environment). Somehow its GUI won't allow me to select a higher dictionary size than 64MB (for LZMA2). After some reading and searching I came up with a working commandline that works. Thank god my 2GB RAM was enough.

Originally Posted by j7n

Do you perhaps have a rough estimate which new codecs take up the majority of the huge code size?

I haven't added support for any new codec for a long time, so I'd say it's the source (the FFmpeg repo) that's slowly getting larger and larger.

[FranceBB]

Originally Posted by Reino

my builds will be without libfdk-aac

Well, the only "positive" thing is that FFMpeg's internal AAC encoder got better in the meantime as it went from "it totally sucks" to "somewhat usable" in 6.2. That, together with the restricting licensing conditions of FDK_AAC means that FDK_AAC won't be missed anyway.

Originally Posted by Reino

I've managed to add support for JPEG XL (libjxl).

Amazing.

Originally Posted by Reino

'ffms3.dll' now relies on 'avcodec-60.dll', 'avformat-60.dll', 'avutil-58.dll', 'swresample-4.dll' and 'swscale-7.dll' from the shared build.

Right, right, yeah, I did move them all to plugins+ but I was surprised that it didn't work.
Turns out it was something stupid I did: although I moved FFMS2.avsi to plugins+ along with ffms3.dll, ffmsindex.exe, avcodec-60.dll, avdevice-60.dll, avfilter-9.dll, avformat-60.dll, avutil-58.dll, postproc-57.dll, swresample-4.dll and swscale-7.dll and I did remove ffms2.dll, I forgot to remove FFFMS2Cplugin.avsi xD

Once I got rid of that, it worked like a charm.
Version 6.2 is working like a charm too.

Originally Posted by Reino

I've seen the discussion in the other threads, but I just compile the source is all I can say.
Shouldn't qyot27 be looking is this then? It's his repo.

Yep yep, Stephen (i.e qyot) should be looking at this on his repo, sorry about the mixup, sometimes I forget about this:

Originally Posted by Reino

sadly I still can't code in C/C++

I'll ask him and perhaps we'll get support in the C plugin too.
For reference, ffms2 C++ version was maintained by Myrsloyk who then left it hanging in 2021, SvtG picked it up and he's the current maintainer of ffms2 C++ version while qyot is the maintainer of ffms2 C version, so yeah, if qyot ports the ANSI WinLatin support from the SvtG C++ version to his C version then we'll also have it in your XP builds.


Anyway, thank you for another round of builds to start off this 2024!
Windows XP Forever!

[FranceBB]

Actually, although using FFVideoSource() works, whenever I load it in Avisynth an error is thrown.

Import("ffms2.avsi")
LoadCPlugin("ffms3.dll")

throws the following error as a separate window in AVSPmod mod:

Exception WindowsError: 'exception: access violation reading 0x00000020' in <bound method AVS_ScriptEnvironment.__del__ of <avisynth.AVS_ScriptEnvironment object at 0x038376D0>> ignored

While AVSMeter returns:

Exception 0xC0000005 [STATUS_ACCESS_VIOLATION]
Module: Cannot determine module
Address: 0x00161DCD

however everything seems to be working and the file is indexed nonetheless and I can seek through it etc.
Weird. Pretty weird.

Out of curiosity, would you also be able to provide a static build for ffms2?
I just wanna rule any issue out.

[Reino]

Originally Posted by FranceBB

Out of curiosity, would you also be able to provide a static build for ffms2?

Well, especially for you then. It's up.

About the error; Same here with both the shared as well as the static build. In AVSMeter 3.0.9.0 I'm getting the same error. In AvsPmod 2.5.1 I'm getting a slightly different message though:

Exception WindowsError: 'exception: access violation reading 0xD0029BD8' in <bound method
PIScriptEnvironment.__del__ of <avisynth.PIScriptEnvironment instance at 0x01F65198>> ignored

I'm clueless as to what could be the cause. If there's one person who could possibly answer this, I think it's, again, qyot27.

[FranceBB]

Originally Posted by Reino

Well, especially for you then. It's up.

Awww.

Originally Posted by Reino

About the error; Same here with both the shared as well as the static build.

Yes I can reproduce.

Originally Posted by Reino

I'm clueless as to what could be the cause. If there's one person who could possibly answer this, I think it's, again, qyot27.

Perhaps.
What I can add to this, though, is that I made the following tests:

2020-05-03 Ok
ffms2-2.18-736-43dc804-avs-vsp_ffmpeg-4.3-3133-1128aa8-win32-xpmod-sse

2020-09-05 Ok
ffms2-2.18-744-7e1e75d-avs-vsp_ffmpeg-4.4-853-276d86a-win32-xpmod-sse

2021-01-02 Ok
ffms2-2.18-748-c48a6db-avs-vsp_ffmpeg-4.4-2460-2c6f532-win32-xpmod-sse

2021-05-21 Ok
ffms2-2.18-756-4d45afc-avs-vsp_ffmpeg-4.5-697-5541cff-win32-xpmod-sse

2023-01-08 Ok
ffms2-2.18-756-4d45afc-avs-vsp_ffmpeg-5.2-2131-fcd557a-win32-xpmod-sse

2023-05-04 Error
ffms2-2.18-756-4d45afc-avs-vsp-win32-shared-xpmod-sse + ffmpeg-6.1-588-4006c71-win32-shared-xpmod-sse

Exception WindowsError: 'exception: access violation reading 0x00000020' in <bound method AVS_ScriptEnvironment.__del__ of <avisynth.AVS_ScriptEnvironment object at 0x0381BB50>> ignored

2023-09-04 Error
ffms2-2.18-841-d42a696-avs-vsp-win32-shared-xpmod-sse + ffmpeg-6.2-609-238f9de-win32-shared-xpmod-sse

Exception WindowsError: 'exception: access violation reading 0x00000010' in <bound method AVS_ScriptEnvironment.__del__ of <avisynth.AVS_ScriptEnvironment object at 0x0381BAF0>> ignored


2024-01-01 Error
ffms2-2.18-842-f6098b6-avs-vsp-win32-shared-xpmod-sse + ffmpeg-6.2-609-238f9de-win32-shared-xpmod-sse

Exception WindowsError: 'exception: access violation reading 0x00000020' in <bound method AVS_ScriptEnvironment.__del__ of <avisynth.AVS_ScriptEnvironment object at 0x038376D0>> ignored


2024-01-02 Error
ffms2-2.18-842-f6098b6-avs-vsp_ffmpeg-6.2-609-238f9de-win32-xpmod-sse

Exception WindowsError: 'exception: access violation reading 0x00000030' in <bound method AVS_ScriptEnvironment.__del__ of <avisynth.AVS_ScriptEnvironment object at 0x0381BAF0>> ignored


All these tests were made using AviSynth+ 3.7.3 (r4013, master, i386) on Windows XP Professional x86, however I also tried 2024-01-02 static on Windows 10 Enterprise x86 too and it errored in the exact same way:

Exception 0xC0000005 [STATUS_ACCESS_VIOLATION]
Module: C:\Users\bucciantinif\Desktop\Utility\AVSMeter\AVSMeter.exe
Address: 0x00821F80

Exception WindowsError: 'exception: access violation reading 0xC0DDAFD1' in <bound method AVS_ScriptEnvironment.__del__ of <avisynth.AVS_ScriptEnvironment object at 0x04C18CF0>> ignored


I don't know whether Stephen (qyot) has some insights / can shed some light on this...
Right now, however, I'll revert to the last working one, namely the 2023-01-08 build ffms2-2.18-756-4d45afc-avs-vsp_ffmpeg-5.2-2131-fcd557a-win32-xpmod-sse.

[Reino]

Originally Posted by FranceBB

2023-01-08 Ok
ffms2-2.18-756-4d45afc-avs-vsp_ffmpeg-5.2-2131-fcd557a-win32-xpmod-sse

2023-05-04 Error
ffms2-2.18-756-4d45afc-avs-vsp-win32-shared-xpmod-sse + ffmpeg-6.1-588-4006c71-win32-shared-xpmod-sse

Exception WindowsError: 'exception: access violation reading 0x00000020' in <bound method AVS_ScriptEnvironment.__del__ of <avisynth.AVS_ScriptEnvironment object at 0x0381BB50>> ignored

At first I thought some FFmpeg commit would probably be the cause, but after having test-compiled different versions of both FFmpeg and FFMS2, it turns out the culprit really is FFMS2 afterall. To be specific; "Fix memory leak" is the latest still working revision. I can't say which of the commits after that is the real culprit, because 16 of them are all part of a pull-request. So yes, Stephen and/or this "Asd-g" person should be able to shed some light on this issue, if you ask me.
About this pull-request; I hardly use AviSynth anymore, so I'm not up to speed with FFMS2 development, but I don't understand why they're removing 'FFMS2-cplugin.avsi'. I remember using the convenient function FFmpegSource2() a lot back then.

Anyway,... new builds uploaded soon.

[qyot27]

Originally Posted by Reino

At first I thought some FFmpeg commit would probably be the cause, but after having test-compiled different versions of both FFmpeg and FFMS2, it turns out the culprit really is FFMS2 afterall. To be specific; "Fix memory leak" is the latest still working revision. I can't say which of the commits after that is the real culprit, because 16 of them are all part of a pull-request.

I can't reproduce it on Win 10, no matter whether I'm using my own build of the C plugin, one of your builds, avsmeter 3.0.0.3 or 3.0.9.0, Avspmod, or AviSynth+ 3.7.3 or the build of r4040 I ran off a couple weeks ago for the static/HEAD versions of ImageSeq and TimeStretch.

It's completely possible to zero in on which commit might have been responsible, I don't do merges unless there's not really any other option (read: the integration commits that occur when pulling in changes from the master branch). Virtually every other time I merge a pull request (on ffms2_cplugin, AviSynth+, etc.) it's as a rebase, so there is no merge commit and the history is still traversible.

About this pull-request; I hardly use AviSynth anymore, so I'm not up to speed with FFMS2 development, but I don't understand why they're removing 'FFMS2-cplugin.avsi'. I remember using the convenient function FFmpegSource2() a lot back then.

Because it was rendered redundant by https://github.com/qyot27/ffms2_cplu...fdc5115eeddab9

Seriously, remove the FFMS2*.avsi file from the autoload folder or the Import line from the script and try to use FFmpegSource2 anyway - it'll still work if you're using the dll with those updates.

The C++ plugin got some refactoring years ago, and one of the things that happened was all those functions were added directly into the plugin, so they were removed from FFMS2.avsi. Since I'd not gotten around to doing the equivalent things in the C plugin, FFMS2-cplugin.avsi was created to preserve those functions specifically so the C plugin could still use them. With the update of the C plugin, that was no longer necessary, so the content of FFMS2.avsi and FFMS2-cplugin.avsi would be exactly the same, making the latter completely unnecessary (and really, if the only thing you ever used the .avsi for was to use the *Source wrapping functions and not any of the other ones¹, you don't even need to keep FFMS2.avsi around, either).

¹ FFFormatTime, FFColorSpace, FFColorRange, FFCropping, FFSampAR, FFPictType, and FFInfo

[Reino]

Stephen, for what it's worth:

b3a4a0d33849df3d6e864c79aa8fb600dd41f99b   No compilation errors, works correctly
8bc3bb3955f8c27c344ccbc29602680f59925128   Compilation error: "undefined reference to `muldivRational'"
b593c8e65df63917c7b3e0464fc382d52d40359d   No compilation errors, but AVSMeter reports: "Script error: there is no function named "FFVideoSource""
[...]
d4680ec8f72cabd9248d179a9a30aa5fd5badc00   No compilation errors, but AVSMeter reports: "Script error: there is no function named "FFVideoSource""
d42a696e24a079cc9eba9bdf084669bb30db4828   No compilation errors, but AVSMeter reports: "Exception 0xC0000005 [STATUS_ACCESS_VIOLATION]
                                                                                         Module:   Cannot determine module
                                                                                         Address:  0x00332758"

So strictly speaking, it all starts with "Port frame properties".

Originally Posted by qyot27

I can't reproduce it on Win 10

You don't have WinXP still laying around? It could very well be an issue that only occurs on WinXP.

[qyot27]

With a couple of the frame properties, it mirrors essentially what the C++ plugin does, which includes a usage of muldivRational in order to reduce the result of the DurNum and DurDen pointers used to calculate the _DurationNum/Den properties.

muldivRational is getting re-used from the VapourSynth headers (VSHelper4.h, specifically). So even if the function itself is self-contained (which it appears to be), the headers being there are making it try to look for an install of VapourSynth. My guess is that - because VS dropped XP support long ago - it isn't installed, or if it is, it may even be checking for an API4 version, which is way too recent for XP and wouldn't be installed for that reason.

Try with the patch I just pushed up to the 'cplugin_muldivtest' branch:
https://github.com/qyot27/ffms2_cplu...in_muldivtest/

Originally Posted by Reino

You don't have WinXP still laying around? It could very well be an issue that only occurs on WinXP.

The only two machines that still have XP on them are effectively in storage. The closest I could come to it would be spinning up a ReactOS VM, but that's a completely different set of issues to navigate.

[Reino]

Thank you. Though it didn't help, because I'm still getting the same "Cannot determine module" error.

[qyot27]

Okay, so it moves from a straightforward failure to load the plugin (and hence the functions provided by it), to the access violation issue when the Plus-specific API calls are made optional at load.

Does it happen with a debug build of FFMS2?

[Reino]

If you mean compiling with ./configure --enable-debug, then yes, same "Cannot determine module" error at load.
Because it's a debug-build I was expecting a more verbose message.

As a side note, which FranceBB also already mentioned; This is what AVSMeter is spitting out. With AvsPmod, despite the "Exception WindowsError" error message in a separate window, 'ffms3.dll' appears to work just fine. Which is... weird.