L-SMASH Source

[Reel.Deel]

There are also https://github.com/HomeOfAviSynthPlusEvolution/L-SMASH-Works

Obviously now outdated due to the recent changes made by HolyWu. But according to MeteorRain, his version is:

The same patch set HolyWu used, which was copied from enccc (https://github.com/enccc/L-SMASH-Works/tree/mod), and one more patch to use swresample instead of avresample, which was copied from l33tmeatwad (https://github.com/l33tmeatwad/L-SMASH-Works).

I've started a section in the LSMASHSource wiki (http://avisynth.nl/index.php/LSMASHSource#Archived_Downloads) noting some of these changes. I hope I did not miss anything, I went back and tried to see if there's anything else but it seems some of HolyWu's earlier posts are gone :(.

[MeteorRain]

Since Holy released all the patches, I'll (after a vacation unfortunately) try to integrate them into the repo.

Lots of works are piled up on me, so please be patient. (Also there's work to do on f3kdb and minideen etc.)

[Atak_Snajpera]

L-SMASH-Works-r935+31-20190818 (https://www.mediafire.com/file/ext0tds407sodbq/L-SMASH-Works-r935+31-20190818.7z/file)


LWLibavVideoSource no longer indexes audio streams. It reduces both the file size and parsing time of the index file. LWLibavAudioSource will re-create the index file for the source file which was already indexed by LWLibavVideoSource so as to index audio streams.
Print indexing progress to stderr.
Tell lavf to discard unwanted packets so they needn't be demuxed.
Remove InputFilePath field from the index file. It's unnecessary and troublesome when users rename or move the source file.
Automatically re-create the index file when the file size or the last modification time of the source file doesn't match.


Could you remove that checking of "Last modification time"?
It is annoying that lsmash now always re-indexes file which was just copied.

original
video=LWLibavVideoSource("original_video.MP4",cachefile="C:\index.lwi")

Copy
video=LWLibavVideoSource("C:\copy_of_original_video.MP4",cachefile="C:\index.lwi")

The same file but with different modification time. For example FFMS2 does not care about "Last modification time". I think it only checks file size.

[MeteorRain]

Atak,

Although I agree there's other ways of doing the check, I don't think Windows file copy actually changes the mtime.

If you use cp, you can use --preserve=timestamps.

I might want to change it to some quick partial checksum if mtime mismatches.

[Atak_Snajpera]

The issue is that i AM copying files using my own custom code (streams and no win api). In this case last modification date is different. I think that Simple check for file size is enough.

[Taurus]

@HolyWu
Tried to redownload your build from the mediafire link.
"Download not available"....
Thanks for your efforts!
Aaah, I just saw it is working again,
maybe a mediafire hiccup ;-)

[Atak_Snajpera]

FFMS2 also checks partial SHA-1 hashing in addition to file size, but I really don't bother to do it. After all video files aren't like text files that normal users would open them in Notepad++ and edit some random bytes. Hence I agree that simply checking file size should be sufficient for most users. Redownload from the usual post.

Once again thank you for this quick change :)

[jpsdr]

As isaid, i used to make builds using the method described here (with VS 2015 at the time) : https://github.com/BrunoReX/build-scripts/blob/master/L-SMASH/readme.md
I've been able to find out the main issue, it's probably a change in msys2 behavior, which doesn't keep the %PATH% by default.
I fixed it, make all the gcc builds, but when compiling the VS project, i have the following error :
LINK : fatal error LNK1104: unable to open file 'lsmash.lib'

I take any idea or advice.

[MeteorRain]

Hence I agree that simply checking file size should be sufficient for most users.

Not exactly correct. A potential issue is that user try to cut a mpeg-ts file at fixed length and overwrite to the same output file. Re-opening such segment the second time will probably crash the code.

I second the idea of checksuming, and if you don't bother doing that, I might.

[MeteorRain]

LSMASHSource-Release_r935+33.zip (https://down.7086.in/AviSynthPlus%20Filters/LSMASHSource-Release_r935%2B33.zip)
Commit Diff (https://github.com/HomeOfAviSynthPlusEvolution/L-SMASH-Works/commit/9613f894ed40e67104c4f3df335c9e322b9c5d1f)

I have changed the logic. If size and timestamp match, assume unchanged. If size matches but not timestamp, hash (first 1MiB + last 1MiB) and compare with recorded hash. If file is too small, only first up to 1MiB counts. Hash method is xxhash which is extremely fast.

A missing hash record will not force regenerating the index file as long as size and timestamp match.

Let me know how that works.

I also include an easy-compiling branch and a cmake build script. Get VS 2017 / 2019 and cmake installed, and use the following batch file to build your copy.

@echo off

mkdir build\x86
pushd build\x86
cmake -DCMAKE_GENERATOR_PLATFORM=Win32 -D_DIR=x86 ..\..\
popd
mkdir build\x64
pushd build\x64
cmake -DCMAKE_GENERATOR_PLATFORM=x64 -D_DIR=x64 ..\..\
popd
cmake --build build\x86 --config Release
cmake --build build\x64 --config Release

[Atak_Snajpera]

Do you guys know why decoding VC-1 using vc1_cuvid is also broken? Unlike software decoder frames are not corrupted but they are decoded in incorrect order.
sample -> https://www.mediafire.com/file/dldzx9smmbxf60m/vc1_sample.mkv/file

script
LoadPlugin("LSMASHSource.dll")
video=LWLibavVideoSource("vc1_sample.mkv",cachefile="vc1_sample.mkv.lwi",decoder="vc1_cuvid")

Decoding works fine in MPC-HC with these settings
https://i.imgsafe.org/79/798bacc148.png

PS. yes I have latest driver installed.

[DJATOM]

@Atak_Snajpera
HW decoder is actually inactive on your screenshot.

[Atak_Snajpera]

@Atak_Snajpera
HW decoder is actually inactive on your screenshot.

But CPU usage is 3 times lower so it works.
https://i.imgsafe.org/7b/7bde32fd60.png

Also there is H/W indicator
https://i.imgsafe.org/7b/7ba4848e67.png

[MeteorRain]

https://down.7086.in/AviSynthPlus%20Filters/LSMASHSource-r935%2B34.zip

Stop indexing progress spamming

-- Now only refresh at every 1%.

[stax76]

@HolyWu

With my 49 PGS subtitle 50 GB file using your last build I'm getting 60 seconds load time, with MeteorRain's build it's only 10 seconds.

edit:

both was tested with 20 PGS subtitles, with 49 PGS subtitles MeteorRain's build took 30 seconds and yours 7 minutes. :)

[MeteorRain]

Yea that was what I immediately noticed when I was encoding some stuff.

[Natty]

:thanks:https://down.7086.in/AviSynthPlus%20Filters/LSMASHSource-r935%2B34.zip

Stop indexing progress spamming

-- Now only refresh at every 1%.

[Atak_Snajpera]

@MeteorRain
Quick question: Why LSMASHSource.dll is 2 times smaller than HolyWu's version?

[DJATOM]

Different compilers or some features disabled at configuring time?

[MeteorRain]

@MeteorRain
Quick question: Why LSMASHSource.dll is 2 times smaller than HolyWu's version?

Size of ffmpeg makes huge differences. I disabled lots of features, and that's probably why.

Last build was on VS 2017 n4.1.3. I'll probably do a refresh VS 2019 n4.2.0 soon for the next build.


"C:\Programs (x86)\Microsoft Visual Studio\2017\Community\VC\Auxiliary\Build\vcvars32.bat"
chcp 65001
./configure --toolchain=msvc --prefix=/ffmpeg-msvc32 --cpu=i686 --arch=i386 --target-os=win32 --disable-pthreads --disable-bzlib --disable-iconv --disable-lzma --disable-xlib --disable-zlib --disable-muxers --disable-encoders --disable-network --enable-gpl --enable-version3 --disable-programs --disable-doc


"C:\Programs (x86)\Microsoft Visual Studio\2017\Community\VC\Auxiliary\Build\vcvars64.bat"
chcp 65001
./configure --toolchain=msvc --prefix=/ffmpeg-msvc64 --arch=x86_64 --target-os=win64 --disable-pthreads --disable-bzlib --disable-iconv --disable-lzma --disable-xlib --disable-zlib --disable-muxers --disable-encoders --disable-network --enable-gpl --enable-version3 --disable-programs --disable-doc

[ChaosKing]

@MeteorRain It would be nice if you could also add vapoursynth support in your next LSmash release on github :)

[MeteorRain]

@ChaosKing, support of what? Current release had no problem running under VapourSynth. Any particular thing I should be aware of?

[ChaosKing]

The current r935+2 release on github only supports avisynth and it would be nice if the next release (r935+X) in your github repo could also include vapoursynth support :)
I ask because I would like to add it to vsrepo.
Currently it is included in avsrepo https://github.com/theChaosCoder/avsrepo/blob/master/local/lsmash.json

[MeteorRain]

Oh you meant github release. Yea definitely.

Also please be aware it's not the original / official release, but rather a fork.

[Atak_Snajpera]

L-SMASH-Works_20190903.7z (https://github.com/HolyWu/L-SMASH-Works/releases/download/20190903/L-SMASH-Works_20190903.7z)


Fix abnormal loading time.
Improve progress printing.
Use both file size and partial checksum (xxhash) for file signature comparison in index file.
LWLibav: Fix inaccurate framerate in some files.
VapourSynth: Replace configure with Meson build system.
VapourSynth: Fix missing setError invocation when there is an error.
Fix frame corruption in some VC-1 files by a workaround. Tested Atak_Snajpera's sample by seek-test.py and confirmed to be frame accurate.


Can you reduce size of .dll like MeteorRain did here https://forum.doom9.org/showthread.php?p=1883899#post1883899

[FranceBB]

L-SMASH-Works_20190903.7z (https://github.com/HolyWu/L-SMASH-Works/releases/download/20190903/L-SMASH-Works_20190903.7z)


Fix abnormal loading time.
Improve progress printing.
Use both file size and partial checksum (xxhash) for file signature comparison in index file.
LWLibav: Fix inaccurate framerate in some files.
VapourSynth: Replace configure with Meson build system.
VapourSynth: Fix missing setError invocation when there is an error.
Fix frame corruption in some VC-1 files by a workaround. Tested Atak_Snajpera's sample by seek-test.py and confirmed to be frame accurate.


Thank you!

[Atak_Snajpera]

I don't understand. Does a size difference of 10 MB really matter on current HDD or internet bandwidth?
Anyway, the culprit is not in FFmpeg as I already knew of disabling unused features when I began providing my builds ages ago.

The main reasons are that different compiler being used and two additional libraries being linked in, one is libaom and the other is libmfx.
For libaom it's used for AV1 decoding. I think I will just ditch it since it's way slower than libdav1d. Just use FFMS2 for AV1 and VP9 decoding as it passed random seek test after some proper patching. Unfortunately for AVC and HEVC decoding it still has random seek issue with some samples even using StvG's latest build, while LWLibav passed.
For libmfx it's used for Intel Quick Sync Video decoding. I need someone with capable Intel GPU to test whether the _qsv decoders also have random seek issue like the _cuvid decoders do, so as to decide whether I should keep it or ditch it.

Yes it matters if you use distributed encoding in ripbot264 where all tools and plugins are being downloaded from host PC. I just want to reduce starting time as much as possible on ~80Mbps (10MiB/s) LAN/Wi-FI. I'm already using compressed (by upx) ffmpeg.exe (20MiB),x264(8MiB) and x265(7MiB) in order to reduce encoding delay. I also decided to compress index file with 7zip in order to reduce index file from ~30MiB to ~1MiB.
https://i.imgsafe.org/5c/5c0a9ddd47.png

example script

#VideoSource
LoadPlugin("\\HOST-PC\RipBot264temp\Tools\AviSynth plugins\lsmash\LSMASHSource.dll")
video=LWLibavVideoSource("\\HOST-PC\RipBot264temp\job1\video.mkv",threads=0,cachefile="\\HOST-PC\RipBot264temp\job1\video.mkv.lwi")

cmd.
"\\HOST-PC\Ripbot264temp\Tools\ffmpeg\bin\ffmpeg.exe" -loglevel panic -i "\\HOST-PC\RipBot264temp\job1\Chunks\1.avs" -strict -1 -f yuv4mpegpipe - | "\\HOST-PC\Ripbot264temp\tools\x264\x264_x64.exe" --stdin y4m --output "\\HOST-PC\RipBot264temp\job1\Chunks\1.264" -

[StvG]

I don't understand. Does a size difference of 10 MB really matter on current HDD or internet bandwidth?
Anyway, the culprit is not in FFmpeg as I already knew of disabling unused features when I began providing my builds ages ago.

The main reasons are that different compiler being used and two additional libraries being linked in, one is libaom and the other is libmfx.
For libaom it's used for AV1 decoding. I think I will just ditch it since it's way slower than libdav1d. Just use FFMS2 for AV1 and VP9 decoding as it passed random seek test after some proper patching. Unfortunately for AVC and HEVC decoding it still has random seek issue with some samples even using StvG's latest build, while LWLibav passed.
For libmfx it's used for Intel Quick Sync Video decoding. I need someone with capable Intel GPU to test whether the _qsv decoders also have random seek issue like the _cuvid decoders do, so as to decide whether I should keep it or ditch it.

Disabling aom AV1 encoder and aom tests will save space if you didn't do it already.

Can you share those AVC and HEVC samples that have seeking issues with ffms2?
Thanks.

[videoh]

Can you share those AVC and HEVC samples that have seeking issues with ffms2? I second that request. I would like to test them with DG tools.

[videoh]

Thank you, HolyWu.

[Reel.Deel]

L-SMASH-Works_20190903.7z (https://github.com/HolyWu/L-SMASH-Works/releases/download/20190903/L-SMASH-Works_20190903.7z)


Fix abnormal loading time.
Improve progress printing.
Use both file size and partial checksum (xxhash) for file signature comparison in index file.
LWLibav: Fix inaccurate framerate in some files.
VapourSynth: Replace configure with Meson build system.
VapourSynth: Fix missing setError invocation when there is an error.
Fix frame corruption in some VC-1 files by a workaround. Tested Atak_Snajpera's sample by seek-test.py and confirmed to be frame accurate.


The list here seem to suggest that MeteorRain's changes are also include. Yet the L-Smash GitHub repo (https://github.com/HolyWu/L-SMASH-Works/commits/master) says otherwise, am I missing something?

MeteorRain's L-Smash GitHub repo (https://github.com/HomeOfAviSynthPlusEvolution/L-SMASH-Works/commits/master) is up-to-date minus the your recent commits.

[Atak_Snajpera]

Two issues with latest version

1) Corrupted (grey frames) frames after seeking
Sample -> https://www.mediafire.com/file/ana6jo26jz4n6sw/Aquaman_sample.mkv/file
https://i.imgsafe.org/54/547128fbf2.png
https://i.imgsafe.org/54/5471b5bdd2.png

2) webm with AV1 returns this error
Sample -> https://www.mediafire.com/file/lq3xmqsdeewme38/Stream2_AV1_HD_6.8mbps.webm/file
https://i.imgsafe.org/54/5471d29141.png

[Atak_Snajpera]

1. I got that sample from an ripbot user. I guess that this avc was originally muxed by makemkv. IT is odd that Simple remuxing (MKV to mkv)with latest mkvtoolnix did not fix that issue.
2.ok

[sneaker_ger]

The PPS changes mid-stream (pic_init_qp_minus26 changes). Mkvmerge seems to detect this when doing a "full" remux (1. demux to ES. 2. mux ES) and prepends the SPS/PPS to every keyframe. In the original sample it is only prepended on an actual change compared to the prior keyframe. Since both frame #97 and #104 use the same pic_init_qp_minus26 frame #104 does not have this prepended PPS. Then decoder tries to use first/global pic_init_qp_minus26 and breaks.

[Atak_Snajpera]

Crash after seeking
Sample -> https://www.mediafire.com/file/xhcyglsbsz4nbgm/camcorder_25i_4-3.avi/file
https://i.imgsafe.org/66/66c8bb586f.png
https://i.imgsafe.org/66/66c920bbbd.png

FFVideoSource works fine.

[Atak_Snajpera]

Cannot reproduce. I'm using seek-test.py (https://gist.github.com/dubhater/3a2c8a59841cae49ecae25cd47ff78d2) for testing.

Because your tool does not check ALL frames.

Just try to open this script in MPC-HC.

LoadPlugin("LSMASHSource.dll")
LWLibavVideoSource("camcorder_25i_4-3.avi",cachefile="camcorder_25i_4-3.avi.lwi")
Trim(233,-1)

https://i.imgsafe.org/68/682fd279e1.png

[poisondeathray]

Atak - can you share your Seek Tester tool ?

[videoh]

One way to test all frames is to put Reverse() at the end of your script and then play the video.

[ChaosKing]

Because your tool does not check ALL frames.


The script does check all frames.

I tested it via lsmas.LWLibavSource() and avisource.AVISource("myscript.avs) in vapoursynth -> no seeking issues.

EDIT
Tried also Reverse() in my avs script, still frame accurate.

EDIT2
If I open the avs (with reverse) in vsedit i get an error

vapoursynth.Error: Avisynth read error:
CAVIStreamSynth: System exception - Access Violation at 0x00007FFB37302EB8

But there is no crash via vspipe...

EDIT3
Argh tested with the wrong script... so I tested it again ... VS is fine, but AVS crashes! But only with reverse.
Reverse in my avsisynth script -> instant crash System exception - Access Violation at 0x00007FFB37302EB8
LWLibavVideoSource("camcorder_25i_4-3.avi")
#Reverse()


std.Reverse() in VS is not frame accurate anymore.
avisource.AVISource(args.file).std.Reverse()

log files
- lsmash in VS https://pastebin.com/anyvB4pf
- avisyource + reverse in VS https://pastebin.com/XFkuT9JT (it aborts after 3 errors)

EDIT4
lsmas.LWLibavSource(args.file).std.Reverse() -> lsmas: failed to get video track. Now I'm confused.

EDIT5
ok ... final results
VS
lsmas.LWLibavSource(args.file) #ok
lsmas.LWLibavSource(args.file).std.Reverse() #ERR failed to get video track.

AVS via VS - script: LWLibavVideoSource("camcorder_25i_4-3.avi")
avisource.AVISource(args.file) # ok
avisource.AVISource(args.file).std.Reverse() # not frame accurate

AVS script with:
LWLibavVideoSource("camcorder_25i_4-3.avi")
reverse()
= instant crash: System exception - Access Violation at 0x00007FFB37302EB8

EDIT 6 - Bonus round
Remuxed it to mkv

VS
lsmas.LWLibavSource(args.file) # ok
lsmas.LWLibavSource(args.file).std.Reverse() # [importer: Error]: failed to find the matched importer.

AVS via VS
avisource.AVISource(args.file) # ok
avisource.AVISource(args.file).std.Reverse() #ok

avs+reverse via vdub still crashes.

[videoh]

Does anybody care?

[ChaosKing]

https://github.com/HolyWu/L-SMASH-Works/releases/tag/20190910


Update to FFmpeg 4.2.1.
VideoSource: Add parameter prefer_hw to conveniently switch between software and hardware decoder. Consult README for details.
Fix seeking issue in CUVID decoding.
Fix hardware decoder outputting YUV420P8 by default on high bit depth clip.
Optimize P010->YUV420P10 conversion for hardware decoder by SSE2.
LWLibav: Fix access violation in avi file when seeking randomly.

Fixed all errors for me :thanks:

[FranceBB]

https://github.com/HolyWu/L-SMASH-Works/releases/tag/20190910


Update to FFmpeg 4.2.1.
VideoSource: Add parameter prefer_hw to conveniently switch between software and hardware decoder. Consult README for details.
Fix seeking issue in CUVID decoding.
Fix hardware decoder outputting YUV420P8 by default on high bit depth clip.
Optimize P010->YUV420P10 conversion for hardware decoder by SSE2.
LWLibav: Fix access violation in avi file when seeking randomly.


Once again, thank you! :D

[poisondeathray]

Fix seeking issue in CUVID decoding.


Thanks,

I can confirm this does fix issues on previous lsmash builds .

Very nice :)

[Reel.Deel]

https://github.com/HolyWu/L-SMASH-Works/releases/tag/20190910


Update to FFmpeg 4.2.1.
VideoSource: Add parameter prefer_hw to conveniently switch between software and hardware decoder. Consult README for details.
Fix seeking issue in CUVID decoding.
Fix hardware decoder outputting YUV420P8 by default on high bit depth clip.
Optimize P010->YUV420P10 conversion for hardware decoder by SSE2.
LWLibav: Fix access violation in avi file when seeking randomly.


I cannot load the following apng image with LWLibavSource(). It gives the error LWLibavVideoSource: failed to get the video track. I only tried with latest avs+ (x64).
I tried with L-SMASH-Works_20190903 / 0906 / 0910 and even MeteorRain's LSMASHSource-r935+34 but all of them give the same error. Last version that works is L-SMASH-Works-r935+26-20190811. Gifs work fine though.

https://imgur.com/download/9qLuzS4/

[poisondeathray]

"Regular" PNG isn't working either (images, or PNG in MOV) . There were a bunch of ffms2 versions like that earlier as well

[qyot27]

PNG support requires zlib.

[poisondeathray]

Is there an AVC in MXF parsing issue with L-SMASH-Works_20190910 ?

1a) 10bit422 AVC can be read in MP4 but not MXF "failed to get video track" . But "L-SMASH-Works-r935+3-win64-20190401" was able to read both
1b) But rewrapping that AVC stream into a TS makes it readable

2) 8bit422 MPEG2 in MXF can be read, so can't be a "global" MXF issue with this version

[poisondeathray]

Sorry false alarm . I can't reproduce the issue now. Maybe some windows weirdness (I rebooted somewhere in between) or something I mixed up

[Atak_Snajpera]

Another possible bug. L-SMASH does not return some frames.

https://i.postimg.cc/L5M5bzMn/Untitled-1.png
Again FFVideoSource works fine.

https://i.postimg.cc/4yP54vZS/Untitled-2.png

LoadPlugin("LSMASHSource.dll")
LWLibavVideoSource("TEST.mkv")
Trim(1,-1)

Sample -> https://www.mediafire.com/file/4f7onn4q2l0jo92/TEST.mkv/file

[Atak_Snajpera]

L-Smash does not fallback to software decoders automatically like written in documentation.

+ prefer_hw (default : 0)
Whether to prefer hardware accelerated decoder to software decoder.
Have no effect if 'decoder' is specified.
- 0 : Use default software decoder.
- 1 : Use NVIDIA CUVID acceleration for supported codec, otherwise use default software decoder.
- 2 : Use Intel Quick Sync Video acceleration for supported codec, otherwise use default software decoder.

For example I have NVIDIA card and it works fine with prefer_hw=1 but my CPU does not support Intel Quick Sync Video so with prefer_hw=2 i get this
https://i.imgsafe.org/a7/a7129ea9ab.png

I would also suggest value -1 as AUTO
+ prefer_hw (default : 0)
Whether to prefer hardware accelerated decoder to software decoder.
Have no effect if 'decoder' is specified.
-1 : Use first available hardware decoder,otherwise use default software decoder.
0 : Use default software decoder.
1 : Use NVIDIA CUVID acceleration for supported codec, otherwise use default software decoder.
2 : Use Intel Quick Sync Video acceleration for supported codec, otherwise use default software decoder.