FrameSeeker - Reliability Source Filter Tester for VapourSynth

[ChaosKing]

You can now easily test the frame accuracy of your favourite source filter.
Based on jackoneills code https://gist.github.com/dubhater/3a2c8a59841cae49ecae25cd47ff78d2

Just place your video and plugin files in the test_chamber folder. Run TEST.bat. After finishing open the csv file in excel.

Download (64bit) (https://www.dropbox.com/s/illkfly7wq72ay7/VapourSynth64PortableTEST.rar?dl=1)

There are 3 setting you can edit in the bat file:
set start=0
set end=300
set tests=ffms2 lsmas lsmasav ffms2seek0

test means:
- ffms2: ffms2.Source()
- lsmas: lsmas.LWLibavSource()
- lsmasav: lsmas.LibavSMASHSource()
- ffms2seek0: ffms2.Source(seekmode=0)
- dgi: dgdecodenv.DGSource()
- d2v: d2v.Source(rff=False)



Depending on the file it is crucial to test at least 200-250 frames. Otherwise it could detect it as frame accurate!
This is a minimal portable VS+python package. No need to install anything.

If you have exotic files, please share them if you can :thanks:

I will try to add D2vSource and DGDecNV in the future (But I don't have access to DGDecNV, can take some time)

Some results:
https://i.imgur.com/oeUttbl.png

Samples:
https://www.mainconcept.com/us/getting-started/samples.html
https://samples.mplayerhq.hu/

[WorBry]

If you have exotic files, please share them if you can :thanks:


Not sure if they qualify as exotic, but the Canon XF-HEVC (10bit 422) sample clips that ifb pointed to are of interest:

https://forum.doom9.org/showthread.php?p=1870119#post1870119

[StainlessS]

Not sure if fair test with some files on LibavLSMASHSource, it is only supposed to deal with ISO file containers, ie

*.mov, *.mp4, *.m4v, *.3gp, *.3g2, *.mj2, *.dvb, *.dcf, *.m21,

Leastwise that is true of Avisynth version, and seems to also be true of Vaporsynth version, given the results.

[ChaosKing]

Not sure if fair test with some files on LibavLSMASHSource, it is only supposed to deal with ISO file containers, ie

*.mov, *.mp4, *.m4v, *.3gp, *.3g2, *.mj2, *.dvb, *.dcf, *.m21,

Leastwise that is true of Avisynth version, and seems to also be true of Vaporsynth version, given the results.

Maybe I could write something like "unsupported". I know what it only supports "ISO Base Media" but I test every file regardless of the file extension. It just takes 1-2 sek to try to open the file :D

@WorBry thx for file, will test and add it to the list.

Anyone want avisynth ffms/lsmash support? :D

[asarian]

Yeah, I'd love to see DGDecNV support in there!

[StainlessS]

"unsupported" sounds good.

I dont often use LSmash, but over the last couple of years has (I think) never failed me on ISO container (unless eg flv file with mp4 extension).

[ChaosKing]

Redownload. Added d2vsource and DGDecodeNV. Not much tested. You need to copy your DGDecodeNV.dll in the dgdecodenv folder and the index program to VapourSynth64\bin.

[asarian]

^^ Wonderful!

[ChaosKing]

Can someone point me to the newest avisynth ffms2/lsmash 32&64 bit versions?

Edit: Updated results. Finally found something where lsmash is not frame accurate.

[WorBry]

Can someone point me to the newest avisynth ffms2/lsmash 32&64 bit versions?


HolyWu posted new 64bit builds today ?

https://forum.doom9.org/showthread.php?p=1870563#post1870563

https://forum.doom9.org/showthread.php?p=1870564#post1870564

[AzraelNewtype]

I have a TV ripped MPEG-2 transport stream that is reporting open errors for every version of lsmasav bundled, as well as the latest version from that thread. This is sort of baffling since I've definitely run a script with this source using LWLibavSource, though it seemed to be playing more poorly than I'd have expected. In fact, the entire reason I bothered with this test in the first place was that it was running, but not it appeared to have errors that I couldn't pin down, and my d2v generator was choking a bit.

Specifically, the output for all the lsmash tests was:


[importer: Error]: failed to find the matched importer.
crash!? vslsmashsource-r935-3edd194-win64.dll
lsmasav: vslsmashsource_2017.dll

though obviously the dll name cited changed for each. They all had the same error state though.

Weirdly, despite being a hard telecined stream, ffms2 passed nearly every version, so my actual usage problem is solved even if my confusion about the test remains.

Also, as a side note, it's sort of frustrating that the script fails if you have a space in any directory in the path above where you're running the thing. Moving the whole package to drive root worked for obvious reasons, once I figured out what the error I was getting actually meant.

[ChaosKing]

Hmm I'm thought I fixed the spaces problem. Maybe only locally, will check it.

lsmasav = lsmas.LibavSMASHSource(). It will only work with mp4, mov and other iso based container. So you should test with "lsmas" = lsmas.LWLibavSource(). This is a simple script, it just tries to open the video file with the specified plugins in Test.bat. I will add a hint if lsmasav is used.
Thx for reporting :)

Btw There is a D2VWitch V3 now (FrameSeeker uses V2 currently) https://forum.doom9.org/showthread.php?p=1870867#post1870867

[AzraelNewtype]

Oh right, both of the filternames have libav in the name in VS, unlike avisynth where the ISO Media Container one is just LSMASH(Audio|Video)Source, while the indexing version is essentially the same. I mostly just copy the line from a script where it's working because remembering the actual name of those source filters is super obnoxious. Was expecting only one to have it and stopped reading closely when I saw it, and sort of assumed lsmas would mean lsmashsource (which isn't a thing so of course it isn't). Whoops.

I have d2vwitch v3, but when I tried putting a d2v it generated into the source video folder it just got nuked so I assumed it was doing something internally to generate. Of course now I see d2vwitch is just in the bin folder, so I may as well update it when I rerun the test for lsmas.

Update: using the correct option like a person who knows how to read reports no issues with either. So it's back to confusion about L-SMASH, tempered by the knowledge that there's no particular need to use it.

[StvG]

FrameSeeker reports seeking issues with this ffms2 (default threads) and sample (https://cloud.owncube.com/s/oEpyKNKQoHXrgKd) but SeekTester v1.2 (https://forum.doom9.org/showthread.php?t=176878) reports no issues.

[ChaosKing]

Same here. But is fine with threads=1. Maybe this is how it's called in SeekTester.


clip = c.ffms2.Source(args.file) :devil:
clip = c.ffms2.Source(args.file, threads=1) :cool:

[StvG]

Yes, threads=1 is fine but SeekTester with default threads is fine too.

[ChaosKing]

There is something strange here. I created a fake avi with avfs.exe with just -> ffms2("snow.MTS"). VS opens with AviSource.
FrameSeeker was still bad (with default thread value). BUT, if you set any thread number, like threads=1, threads=16 it is always frame accurate. ( I tried 1,2,3,4,16,32)

Now back to ffms2.Source(threads=N). ok are N = 1, 2, 3 (higher N failed) hmm


Btw. SeekTester will also fail if you raise the difficulty with: Reverse( FFVideoSource("%FILEPATH%",cachefile="%PROGRAMTEMP%\%FILENAME%.ffindex") )

[Atak_Snajpera]

Btw. SeekTester will also fail if you raise the difficulty with:
Code:

Reverse( FFVideoSource("%FILEPATH%",cachefile="%PROGRAMTEMP%\%FILENAME%.ffindex") )


But Now you are comparing NON-LINEAR ACCESS (via reverse) with NON-LINEAR ACCESS (via reverse + trim)! You may get fake results!

With REVERSE
https://i.postimg.cc/NGZSN5FW/Untitled-1.png

WITHOUT REVERSE
https://i.postimg.cc/vH9jCb6D/Untitled-2.png

[ChaosKing]

Why reverse + trim? I did not use trim at all.
I see reverse() as an additional test. If LINEAR + NON-LINEAR access AND REV()+REV() are both ok, your source filter should be 99.9999% frame accurate in all situations.

[Atak_Snajpera]

Non linear Access is forced by trim function in my tool. It is done under the Hood.

[GeoffreyA]

ChaosKing, I tried modifying it to use BestSource but it wasn't working. Admittedly, very hastily. Also, the latest versions of FFMS2 and LSMASH fail. It's a fantastic tool, by the way.

[ChaosKing]

Try these (mostly untested)

https://www.dropbox.com/scl/fi/u0uijzasszdknr4fd3raa/seek-test_files.zip?rlkey=7sg23c1i2v4br5m7idd5v3x7d&dl=1

[GeoffreyA]

Try these (mostly untested)

https://www.dropbox.com/scl/fi/u0uijzasszdknr4fd3raa/seek-test_files.zip?rlkey=7sg23c1i2v4br5m7idd5v3x7d&dl=1

Thanks! I'll give them a go in the morning.