FFmpegSource - C-plugin

[FranceBB]

Thank you very much indeed! :D
The only thing that was slowly killing XP was the lack of updated indexers, but with ffms2 still supported, XP is gonna live for a very long time! :D

[kedautinh12]

For those interested, I've made a new Windows XP compatible non-SSE2 build. See https://rwijnsma.home.xs4all.nl/files/ffms2/ (https://rwijnsma.home.xs4all.nl/files/ffms2/).

Can you build x64 ver??

[StainlessS]

Can you build x64 ver??
++ 1, thanks.

[Reino]

Can you build x64 ver??No, I can't. Sorry.

[manolito]

@qyot27

Just the other day I made some seek tests with several source filters using Atak's SeekTester utility. See here:
https://forum.doom9.org/showthread.php?p=1927119#post1927119

The puzzling thing for me is that your latest FFMS2 C-plugin 1342+129 from last year is the only ffms2 build which handles VP9 video without any seek errors. Amazing...

What did you do differently than all the other guys like Reino, Myrsloik or StvG? Can you share your secret?


Cheers
manolito

[qyot27]

I simply paid attention to the pull requests opened over on the main FFMS2 repo and merged in PR#323 (https://github.com/FFMS/ffms2/pull/323), which is noted in the PR as not being a final version of the patches (so it's not merged into the default c_plugin branch when cloned), and on the C-plugin repo I added a couple extra patches on top to handle a version check failure and making it aware of AV1. The official builds won't include a WIP patchset, the StvG builds apparently reverted to an even earlier version of FFMS2's VP9 parsing instead of using PR323, and unless Reino used the patches_plusvp9av1 branch (https://github.com/qyot27/ffms2_cplugin/commits/patches_plusvp9av1), those are basically the same as the unpatched upstream version (if that branch was used, then maybe something changing in FFmpeg's libraries might be the cause).

I obliquely mentioned that those patches were included way back here (https://forum.doom9.org/showthread.php?p=1869630#post1869630).

[FranceBB]

I too had issues with ffms2 on VP9 streams, but, as pretty much everyone does, as soon as I find a problem with ffms2, I use LSMASH.
Using LWLibavVideo and Audio Source on the very same VP9 stream solved the issue as it was indexed correctly and I'm talking about the latest XP Compatible version of LSMASH which is quite old now.
Very interesting...

[StvG]

I simply paid attention to the pull requests opened over on the main FFMS2 repo and merged in PR#323 (https://github.com/FFMS/ffms2/pull/323), which is noted in the PR as not being a final version of the patches (so it's not merged into the default c_plugin branch when cloned), and on the C-plugin repo I added a couple extra patches on top to handle a version check failure and making it aware of AV1. The official builds won't include a WIP patchset, the StvG builds apparently reverted to an even earlier version of FFMS2's VP9 parsing instead of using PR323, and unless Reino used the patches_plusvp9av1 branch (https://github.com/qyot27/ffms2_cplugin/commits/patches_plusvp9av1), those are basically the same as the unpatched upstream version (if that branch was used, then maybe something changing in FFmpeg's libraries might be the cause).

I obliquely mentioned that those patches were included way back here (https://forum.doom9.org/showthread.php?p=1869630#post1869630).
The patches from your patches_plusvp9av1 branch are included in my build - https://forum.doom9.org/showthread.php?p=1921581#post1921581 ( https://pastebin.com/0wS5syun - lines from #23 to #39; from #275 to #291; from #347 to #366).

[qyot27]

Along with multiple other pieces that alter PTS/DTS and discontinuity handling (likely what Myrsloik meant when realfinder tried to submit it upstream (https://github.com/FFMS/ffms2/pull/367)), which there is no patch history or topic separation for to show what got changed relative to any other patches.

[StvG]

Just tested...

FFMS2 C-plugin 1342+129 isn't seek accurate with default threads here - 1st image - c-plugin, 2nd image - cpp-plugin.

https://images2.imgbox.com/96/b1/rXEBX4g3_o.png
https://images2.imgbox.com/fe/55/Y58CKw9u_o.png

As you can see cpp-plugin has the same amount of errors as my build ( https://forum.doom9.org/showthread.php?p=1927067#post1927067 ) - 6 errors.

[manolito]

This is weird...

What is the value for your default threads? For my CPU I get 4 default threads, and the test finishes without any errors.

(This applies to the C-Plugin. The CPP-Plugin from this build is not really useable).

[qyot27]

CPU used? Because running it just now with v1.2 of that program didn't flag anything. i5-9400, so 6 cores total.

As manolito hinted at, I'm betting it's something related to # of default cores/threads on a particular setup. There's also this:
https://forum.doom9.org/showthread.php?p=1921643#post1921643

Which seems like it wouldn't be too hard to get it to misbehave on a Threadripper but be completely fine on a typical consumer CPU with a traditionally moderate # of cores. Whether the issue is really in the testing setup or the plugin or in libavcodec/format itself is another matter.

[StvG]

@manolito, with my build you will have default threads 3 (cpu threads * 0.75).

Tested with v1.2 and the results for the all 3 builds are the same.
Tested with i9 7900x on vm with 16 vcores and 16GB RAM.
All 3 builds with threads=10 have the same amount of errors (166 for c-plugin and 6 for cpp-plugins).
All 3 builds with threads=8 are seek accurate.

Edit: @manolito, do you use 32-bit or 64-bit?

Edit2: Tested on vm with 4 vcores.
c-plugin doesn't have seek errors.
my build with default threads (3) has 14 errors.
my build with threads=4 doesn't have seek errors.
@manolito, you can try with manual set threads=4.

[qyot27]

New build (https://forum.doom9.org/showthread.php?p=1829061#post1829061), for the first time in a while.

[manolito]

@qyot27
Just tested your new build, no problems so far, everything works. No seek issues with my default 4 threads. Thanks a lot...

@StvG
with my build you will have default threads 3 (cpu threads * 0.75)
And this seems to cause problems. Uneven # of threads are not handled well by libavcodec. And it also contradicts the documentation which says:
The number of decoding threads to request from libavcodec.
Setting it to less than or equal to zero means it defaults to the number of logical CPU's reported by Windows.

I only use 32-bit AVS+ (latest pinterf 3.6.2 test3) and AVS plugins.

You are right, specifying threads=4 fixes all seek errors for me. The same goes for threads=2 and threads=1. Looks like only uneven thread numbers cause the errors.


Unfortunately the Reino XP compatible ffms2 builds do not react to specifying different # of threads. Too bad...


A little OT for this thread - I also tested your latest LSMASH build which caused corrupted frames upon seek operations. As you already said, these errors disappear when using 2 threads. But using 4 threads does not help at all. Using only 1 thread also gets rid of the seek problems.


So it looks like the number of libavcodec threads is the key. Go figure...


Cheers
manolito

[StvG]

threads=5/7 doesn't give errors. Only threads=3 gives errors when threads are <=8. It's the same for c-plugin.

About LSMASH - I didn't changed anything than applying the pull request for frame properties and changing exlibs (removed mfx, added dav1d and bcrypt).
It's interesting that official build with default threads has ~same speed as my build with 2 threads - ( https://forum.doom9.org/showthread.php?p=1927055#post1927055 ).

[Reino]

Unfortunately the Reino XP compatible ffms2 builds do not react to specifying different # of threads. Too bad...I've compiled ffms2 (and FFmpeg) on an old AMD Athlon XP cpu (single core of course) for use on similar old hardware, so this is no surprise.

[manolito]

I've compiled ffms2 (and FFmpeg) on an old AMD Athlon XP cpu (single core of course) for use on similar old hardware, so this is no surprise.

Yes, I know, but this is not really my point. What I wanted to say is that the StvG ffms2 builds when dealing with a VP9 stream do sometimes have seek errors, but this depends on the number of libavcodec threads. Your builds (and also qyot27's older XP compatible builds) do not react to specifying different numbers of threads. In fact they do not handle VP9 streams at all. This is probably caused by the XP compatible libraries, not much anyone can do about it...

The other thing is that I do not believe that compiling on an old single core Athlon necessarily prevents multithreading. Your XP compatible FFmpeg builds sure do multithreading nicely when they are run on a modern CPU.


Cheers
manolito

[manolito]

Just noticed that Reino has built new XP compatible versions of ffms2 and also FFmpeg... :thanks:

Will start testing right away.

[FranceBB]

Just noticed that Reino has built new XP compatible versions of ffms2 and also FFmpeg... :thanks:

Will start testing right away.

Ah! Thanks for the heads up, mate! I'll test it straight away too! :)

[manolito]

Just finished my usual test routine for these latest ffms2 and FFmpeg builds. Pretty boring, because there were no problems whatsoever, everything JUST WORKED.

Thanks again Reino for taking care of us retro WinXP users...

[FranceBB]

Just finished my usual test routine for these latest ffms2 and FFmpeg builds. Pretty boring, because there were no problems whatsoever, everything JUST WORKED.


Same here, everything is working fine. I didn't have time to test it with VP9, though, to see if there were any problems (you know, the ones raised with parallelism etc).


Thanks again Reino for taking care of us retro WinXP users...

+1
Thank you, Reino, for keeping ffms2 C plugin updated. Without it, we wouldn't have any updated indexer on XP, so this is of vital importance. :)

[Reino]

Always good to hear. Appreciated.

[manolito]

I didn't have time to test it with VP9, though, to see if there were any problems (you know, the ones raised with parallelism etc).

Sorry, I forgot to mention that for VP9 video sources this build behaves the same as the previous build from Sept. 2020. Atak's seek tester reports hundred of errors. So for decoding VP9 under WinXP you need to use a different source filter. FWIW the latest XP compatible version of LSMASH lsmash-r784-150316-XP does handle VP9 sources without any seek errors. This version is included in the current MeGUI-32bit download.

[FranceBB]

Sorry, I forgot to mention that for VP9 video sources this build behaves the same as the previous build from Sept. 2020. Atak's seek tester reports hundred of errors.

Ok, so just like the old one, got it.



So for decoding VP9 under WinXP you need to use a different source filter. FWIW the latest XP compatible version of LSMASH lsmash-r784-150316-XP does handle VP9 sources without any seek errors.

Yeah, that's exactly what I've done last time and it's what I'm gonna do in the future too, then.

[Reino]

Is this a WinXP issue? Or does this only happen with my binaries? If not, then have you considered reporting this issue (bug?) to the libvpx developers?

[manolito]

Is this a WinXP issue? Or does this only happen with my binaries?

I don't think that it is a direct WinXP issue. Have a look here:
https://forum.doom9.org/showthread.php?p=1927119#post1927119
All tests were done under Win7-64.

The StvG versions up to Oct. 2020 as well as the latest official Myrsloik version 2.40 have these problems, too. The newest StvG version from Nov.23.2020 fixed it, though.

The older qyot27 C-Plugin versions do have the issue up to the latest final XP compatible version. All builds after this are fine.

So I tend to think that some of the patches used by qyot27 in the current non-XP versions took care of the problem. Probably these patches cannot be used for XP compatible versions.

[qyot27]

New build (https://forum.doom9.org/showthread.php?p=1829061#post1829061). Mostly boring, just nearly 6 months' newer FFmpeg.

[manolito]

I think I found a small discrepancy between the ffms2 documentation and the actual plugin behavior:

For the "colorspace" parameter in ffvideosource the doc says that "YV12" can be replaced by "YUV420P8". Does not work for me, ffms2 throws an error if I do this.

The same thing happens with Reino's latest XP compatible C-Plugin (I did not test StvG's latest C++ build).

No big deal, but it would be nice if this was fixed, because LWLibavVideoSource has no problem with it.

[qyot27]

I haven't touched the docs whatsoever to describe the functionality of the C plugin. The docs are for the C++ plugin (generally speaking, though, the options and whatnot are the same, save for the C++ plugin having the FFmpegSource2/FFMS2 function inside the plugin now instead of in the .avsi...but the .avsi is more or less self-documenting).

You can see the difference pretty easily here:
https://github.com/ffms/ffms2/blob/master/src/avisynth/avssources.cpp#L39
https://github.com/qyot27/ffms2_cplugin/blob/bc1f95cf9fad2cf5622b49ee8f120c92d169c13d/src/avisynth_c/avs_utils.c#L33

That said, it was an oversight that I didn't alias the newer names for the 2.5/2.6 pix_fmts into the table in avisynth_c/avs_utils.c when I added all the others.

[qyot27]

New build, first post. Not sure what happened with the branch numbering.

[manolito]

:thanks:

[Reino]

@Reino

could you have a look at this post:
https://forum.doom9.org/showthread.php?p=1943204#post1943204Done (https://rwijnsma.home.xs4all.nl/files/ffms2/). Untested though.

[FranceBB]

Done (https://rwijnsma.home.xs4all.nl/files/ffms2/). Untested though.

Thank you!! :D

[qyot27]

New build, first post. Includes both Windows and macOS.

[kedautinh12]

Thanks

[qyot27]

New build, first post.