L-SMASH Source

[MeteorRain]

Originally Posted by Atak_Snajpera

I have another feature request.
Store index file in compressed form. Currently index file can be extremely large in comparison to .ffindex.
video.mkv is remuxed Alien Covenant DVD (2h movie). Just take a look how tiny .ffindex is


Large index file is not a problem if you load that file locally from SSD. Normally It should take less than 1s. However it is a different story if you have to load that file directly via network 100Mbps LAN or wi-fi n.



That large uncompressed index file significantly delays starting of encoding process. It gets even worse if more than one server is downloading that file.

example


Now 6 servers are downloading 200MiB index file at the same time. (1.2GiB total). Assuming that real transfer rate for LAN100Mbps is only ~10MiB/s we can easily predict that encoding process will be delayed by about 2 minutes!

So my suggestion is to always compress index file with some basic zip algorithm. LZMA (normal) can compress that index file to ~5MiB while Zip to ~10MiB.
Time saving will be immediately noticeable. Delay will be measured in seconds instead of minutes.

Removing audio part (I do all the time) will save tones of space. Not just transferring, but loading and parsing the index itself could take up to minutes for a large file, until you remove the audio part from the index.

It's as simple as removing lines containing Type=1 or Channel. You can use sed to manipulate the file.

[Atak_Snajpera]

Originally Posted by MeteorRain

Removing audio part (I do all the time) will save tones of space. Not just transferring, but loading and parsing the index itself could take up to minutes for a large file, until you remove the audio part from the index.

It's as simple as removing lines containing Type=1 or Channel. You can use sed to manipulate the file.

Thanks for your tip! After processing by sed I got indeed much smaller file



Nevertheless It is still too much for me. (4s extra delay on LAN100). Additionally to your 'hack' I decided to keep that index file in compressed form (LZMA-normal) for encoding server app. So instead of using index file directly now server has to first decompress index file to some temp folder.

[Atak_Snajpera]

Originally Posted by HolyWu

L-SMASH-Works-r935+31-20190817

• LWLibavVideoSource no longer indexes audio streams. It reduces both the file size and parsing time of the index file. LWLibavAudioSource will re-create the index file for the source file which was already indexed by LWLibavVideoSource so as to index audio streams.
• Print indexing progress to stdout.
• Tell lavf to discard unwanted packets so they needn't be demuxed.
• Remove InputFilePath field from the index file. It's unnecessary and troublesome when users rename or move the source file.
• Automatically re-create the index file when the file size or the last modification time of the source file doesn't match.

LOL! You are reading in my mind! I had to always change that from local path (c:\something\...) to network path (\\XEON-PC\...)

Now only one thing is missing. Compression of index file like in FFMS2 index file. Thank you.

[stax76]

@HolyWu

Very exciting improvements. I'm sure it's not only GUI authors that have been waiting for this long time.

[ChaosKing]

Originally Posted by HolyWu

L-SMASH-Works-r935+31-20190817

• LWLibavVideoSource no longer indexes audio streams. It reduces both the file size and parsing time of the index file. LWLibavAudioSource will re-create the index file for the source file which was already indexed by LWLibavVideoSource so as to index audio streams.
• Print indexing progress to stdout.
• Tell lavf to discard unwanted packets so they needn't be demuxed.
• Remove InputFilePath field from the index file. It's unnecessary and troublesome when users rename or move the source file.
• Automatically re-create the index file when the file size or the last modification time of the source file doesn't match.

THX
Would it be possible to add a gpu parameter? I saw that there is a supported HW decoders list, so maybe just a cuvid_gpu/qsv_gpu = true/false parameter could be added to avoid nvidia intel gpu detection? It should then set the hevc, h264, mjpeg, etc. decoder accordingly.

[Atak_Snajpera]

Originally Posted by HolyWu

Sorry. It probably won't happen, at least from me.

Ok. No problem.

[FranceBB]

Thank you for the update, once again!

[real.finder]

HolyWu, why not use github?

and the changes you made has no public source code?

[jpsdr]

Thanks for your builds.
A while ago, i used to make builds using the method described here (with VS 2015 at the time) : https://github.com/BrunoReX/build-sc...MASH/readme.md
Unfortunately, one day, the part with VS201x x86 Native Tools Command Prompt suddenly stoped working. Didn't know if it was something in msys2 or VS which has changed.
A little question : Your release said r935+31, and your link showes 6 patches, when, i was expected 31.
The only source i know is this one : https://github.com/VFR-maniac/L-SMASH-Works/branches
Is there another one ?

[DJATOM]

There are also https://github.com/HomeOfAviSynthPlu.../L-SMASH-Works

[Reel.Deel]

Originally Posted by DJATOM

There are also https://github.com/HomeOfAviSynthPlu.../L-SMASH-Works

Obviously now outdated due to the recent changes made by HolyWu. But according to MeteorRain, his version is:

Originally Posted by MeteorRain

The same patch set HolyWu used, which was copied from enccc, and one more patch to use swresample instead of avresample, which was copied from l33tmeatwad.

I've started a section in the LSMASHSource wiki noting some of these changes. I hope I did not miss anything, I went back and tried to see if there's anything else but it seems some of HolyWu's earlier posts are gone .

[MeteorRain]

Since Holy released all the patches, I'll (after a vacation unfortunately) try to integrate them into the repo.

Lots of works are piled up on me, so please be patient. (Also there's work to do on f3kdb and minideen etc.)

[Atak_Snajpera]

Originally Posted by HolyWu

L-SMASH-Works-r935+31-20190818

• LWLibavVideoSource no longer indexes audio streams. It reduces both the file size and parsing time of the index file. LWLibavAudioSource will re-create the index file for the source file which was already indexed by LWLibavVideoSource so as to index audio streams.
• Print indexing progress to stderr.
• Tell lavf to discard unwanted packets so they needn't be demuxed.
• Remove InputFilePath field from the index file. It's unnecessary and troublesome when users rename or move the source file.
• Automatically re-create the index file when the file size or the last modification time of the source file doesn't match.

Could you remove that checking of "Last modification time"?
It is annoying that lsmash now always re-indexes file which was just copied.

original
video=LWLibavVideoSource("original_video.MP4",cachefile="C:\index.lwi")

Copy
video=LWLibavVideoSource("C:\copy_of_original_video.MP4",cachefile="C:\index.lwi")

The same file but with different modification time. For example FFMS2 does not care about "Last modification time". I think it only checks file size.

[MeteorRain]

Atak,

Although I agree there's other ways of doing the check, I don't think Windows file copy actually changes the mtime.

If you use cp, you can use --preserve=timestamps.

I might want to change it to some quick partial checksum if mtime mismatches.

[Atak_Snajpera]

The issue is that i AM copying files using my own custom code (streams and no win api). In this case last modification date is different. I think that Simple check for file size is enough.

[Taurus]

@HolyWu
Tried to redownload your build from the mediafire link.
"Download not available"....
Thanks for your efforts!
Aaah, I just saw it is working again,
maybe a mediafire hiccup ;-)

[Atak_Snajpera]

Originally Posted by HolyWu

FFMS2 also checks partial SHA-1 hashing in addition to file size, but I really don't bother to do it. After all video files aren't like text files that normal users would open them in Notepad++ and edit some random bytes. Hence I agree that simply checking file size should be sufficient for most users. Redownload from the usual post.

Once again thank you for this quick change

[jpsdr]

As isaid, i used to make builds using the method described here (with VS 2015 at the time) : https://github.com/BrunoReX/build-sc...MASH/readme.md
I've been able to find out the main issue, it's probably a change in msys2 behavior, which doesn't keep the %PATH% by default.
I fixed it, make all the gcc builds, but when compiling the VS project, i have the following error :
LINK : fatal error LNK1104: unable to open file 'lsmash.lib'

I take any idea or advice.

[MeteorRain]

Originally Posted by HolyWu

Hence I agree that simply checking file size should be sufficient for most users.

Not exactly correct. A potential issue is that user try to cut a mpeg-ts file at fixed length and overwrite to the same output file. Re-opening such segment the second time will probably crash the code.

I second the idea of checksuming, and if you don't bother doing that, I might.

[MeteorRain]

LSMASHSource-Release_r935+33.zip
Commit Diff

I have changed the logic. If size and timestamp match, assume unchanged. If size matches but not timestamp, hash (first 1MiB + last 1MiB) and compare with recorded hash. If file is too small, only first up to 1MiB counts. Hash method is xxhash which is extremely fast.

A missing hash record will not force regenerating the index file as long as size and timestamp match.

Let me know how that works.

I also include an easy-compiling branch and a cmake build script. Get VS 2017 / 2019 and cmake installed, and use the following batch file to build your copy.

@echo off

mkdir build\x86
pushd build\x86
cmake -DCMAKE_GENERATOR_PLATFORM=Win32 -D_DIR=x86 ..\..\
popd
mkdir build\x64
pushd build\x64
cmake -DCMAKE_GENERATOR_PLATFORM=x64 -D_DIR=x64 ..\..\
popd
cmake --build build\x86 --config Release
cmake --build build\x64 --config Release