Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > Avisynth Development

Reply
 
Thread Tools Search this Thread Display Modes
Old 14th August 2019, 23:47   #841  |  Link
MeteorRain
結城有紀
 
Join Date: Dec 2003
Location: NJ; OR; Shanghai
Posts: 894
Quote:
Originally Posted by Atak_Snajpera View Post
I have another feature request.
Store index file in compressed form. Currently index file can be extremely large in comparison to .ffindex.
video.mkv is remuxed Alien Covenant DVD (2h movie). Just take a look how tiny .ffindex is


Large index file is not a problem if you load that file locally from SSD. Normally It should take less than 1s. However it is a different story if you have to load that file directly via network 100Mbps LAN or wi-fi n.



That large uncompressed index file significantly delays starting of encoding process. It gets even worse if more than one server is downloading that file.

example


Now 6 servers are downloading 200MiB index file at the same time. (1.2GiB total). Assuming that real transfer rate for LAN100Mbps is only ~10MiB/s we can easily predict that encoding process will be delayed by about 2 minutes!

So my suggestion is to always compress index file with some basic zip algorithm. LZMA (normal) can compress that index file to ~5MiB while Zip to ~10MiB.
Time saving will be immediately noticeable. Delay will be measured in seconds instead of minutes.
Removing audio part (I do all the time) will save tones of space. Not just transferring, but loading and parsing the index itself could take up to minutes for a large file, until you remove the audio part from the index.

It's as simple as removing lines containing Type=1 or Channel. You can use sed to manipulate the file.
__________________
Projects
x265 - Yuuki-Asuna-mod Download / GitHub
TS - ADTS AAC Splitter | LATM AAC Splitter | BS4K-ASS
Neo AviSynth+ filters - F3KDB | FFT3D | DFTTest | MiniDeen | Temporal Median

Last edited by MeteorRain; 14th August 2019 at 23:54.
MeteorRain is offline   Reply With Quote
Old 15th August 2019, 12:51   #842  |  Link
Atak_Snajpera
RipBot264 author
 
Atak_Snajpera's Avatar
 
Join Date: May 2006
Location: Poland
Posts: 7,806
Quote:
Originally Posted by MeteorRain View Post
Removing audio part (I do all the time) will save tones of space. Not just transferring, but loading and parsing the index itself could take up to minutes for a large file, until you remove the audio part from the index.

It's as simple as removing lines containing Type=1 or Channel. You can use sed to manipulate the file.
Thanks for your tip! After processing by sed I got indeed much smaller file



Nevertheless It is still too much for me. (4s extra delay on LAN100). Additionally to your 'hack' I decided to keep that index file in compressed form (LZMA-normal) for encoding server app. So instead of using index file directly now server has to first decompress index file to some temp folder.
Atak_Snajpera is offline   Reply With Quote
Old 17th August 2019, 15:52   #843  |  Link
Atak_Snajpera
RipBot264 author
 
Atak_Snajpera's Avatar
 
Join Date: May 2006
Location: Poland
Posts: 7,806
Quote:
Originally Posted by HolyWu View Post
L-SMASH-Works-r935+31-20190817
  • LWLibavVideoSource no longer indexes audio streams. It reduces both the file size and parsing time of the index file. LWLibavAudioSource will re-create the index file for the source file which was already indexed by LWLibavVideoSource so as to index audio streams.
  • Print indexing progress to stdout.
  • Tell lavf to discard unwanted packets so they needn't be demuxed.
  • Remove InputFilePath field from the index file. It's unnecessary and troublesome when users rename or move the source file.
  • Automatically re-create the index file when the file size or the last modification time of the source file doesn't match.
LOL! You are reading in my mind! I had to always change that from local path (c:\something\...) to network path (\\XEON-PC\...)

Now only one thing is missing. Compression of index file like in FFMS2 index file. Thank you.

Last edited by Atak_Snajpera; 17th August 2019 at 16:11.
Atak_Snajpera is offline   Reply With Quote
Old 17th August 2019, 17:40   #844  |  Link
stax76
Registered User
 
stax76's Avatar
 
Join Date: Jun 2002
Location: On thin ice
Posts: 6,837
@HolyWu

Very exciting improvements. I'm sure it's not only GUI authors that have been waiting for this long time.
stax76 is offline   Reply With Quote
Old 17th August 2019, 17:45   #845  |  Link
ChaosKing
Registered User
 
Join Date: Dec 2005
Location: Germany
Posts: 1,795
Quote:
Originally Posted by HolyWu View Post
L-SMASH-Works-r935+31-20190817
  • LWLibavVideoSource no longer indexes audio streams. It reduces both the file size and parsing time of the index file. LWLibavAudioSource will re-create the index file for the source file which was already indexed by LWLibavVideoSource so as to index audio streams.
  • Print indexing progress to stdout.
  • Tell lavf to discard unwanted packets so they needn't be demuxed.
  • Remove InputFilePath field from the index file. It's unnecessary and troublesome when users rename or move the source file.
  • Automatically re-create the index file when the file size or the last modification time of the source file doesn't match.
THX
Would it be possible to add a gpu parameter? I saw that there is a supported HW decoders list, so maybe just a cuvid_gpu/qsv_gpu = true/false parameter could be added to avoid nvidia intel gpu detection? It should then set the hevc, h264, mjpeg, etc. decoder accordingly.
__________________
AVSRepoGUI // VSRepoGUI - Package Manager for AviSynth // VapourSynth
VapourSynth Portable FATPACK || VapourSynth Database

Last edited by ChaosKing; 17th August 2019 at 17:47.
ChaosKing is offline   Reply With Quote
Old 17th August 2019, 18:13   #846  |  Link
Atak_Snajpera
RipBot264 author
 
Atak_Snajpera's Avatar
 
Join Date: May 2006
Location: Poland
Posts: 7,806
Quote:
Originally Posted by HolyWu View Post
Sorry. It probably won't happen, at least from me.
Ok. No problem.
Atak_Snajpera is offline   Reply With Quote
Old 17th August 2019, 21:30   #847  |  Link
FranceBB
Broadcast Encoder
 
FranceBB's Avatar
 
Join Date: Nov 2013
Location: Royal Borough of Kensington & Chelsea, UK
Posts: 2,883
Thank you for the update, once again!
FranceBB is offline   Reply With Quote
Old 18th August 2019, 04:31   #848  |  Link
real.finder
Registered User
 
Join Date: Jan 2012
Location: Mesopotamia
Posts: 2,587
HolyWu, why not use github?

and the changes you made has no public source code?
__________________
See My Avisynth Stuff
real.finder is offline   Reply With Quote
Old 18th August 2019, 10:44   #849  |  Link
jpsdr
Registered User
 
Join Date: Oct 2002
Location: France
Posts: 2,309
Thanks for your builds.
A while ago, i used to make builds using the method described here (with VS 2015 at the time) : https://github.com/BrunoReX/build-sc...MASH/readme.md
Unfortunately, one day, the part with VS201x x86 Native Tools Command Prompt suddenly stoped working. Didn't know if it was something in msys2 or VS which has changed.
A little question : Your release said r935+31, and your link showes 6 patches, when, i was expected 31.
The only source i know is this one : https://github.com/VFR-maniac/L-SMASH-Works/branches
Is there another one ?
__________________
My github.
jpsdr is offline   Reply With Quote
Old 18th August 2019, 13:35   #850  |  Link
DJATOM
Registered User
 
DJATOM's Avatar
 
Join Date: Sep 2010
Location: Ukraine, Bohuslav
Posts: 377
There are also https://github.com/HomeOfAviSynthPlu.../L-SMASH-Works
__________________
Me on GitHub
PC Specs: Ryzen 5950X, 64 GB RAM, RTX 2070
DJATOM is offline   Reply With Quote
Old 18th August 2019, 21:19   #851  |  Link
Reel.Deel
Registered User
 
Join Date: Mar 2012
Location: Texas
Posts: 1,664
Quote:
Originally Posted by DJATOM View Post
Obviously now outdated due to the recent changes made by HolyWu. But according to MeteorRain, his version is:

Quote:
Originally Posted by MeteorRain View Post
The same patch set HolyWu used, which was copied from enccc, and one more patch to use swresample instead of avresample, which was copied from l33tmeatwad.
I've started a section in the LSMASHSource wiki noting some of these changes. I hope I did not miss anything, I went back and tried to see if there's anything else but it seems some of HolyWu's earlier posts are gone .
Reel.Deel is offline   Reply With Quote
Old 19th August 2019, 02:14   #852  |  Link
MeteorRain
結城有紀
 
Join Date: Dec 2003
Location: NJ; OR; Shanghai
Posts: 894
Since Holy released all the patches, I'll (after a vacation unfortunately) try to integrate them into the repo.

Lots of works are piled up on me, so please be patient. (Also there's work to do on f3kdb and minideen etc.)
__________________
Projects
x265 - Yuuki-Asuna-mod Download / GitHub
TS - ADTS AAC Splitter | LATM AAC Splitter | BS4K-ASS
Neo AviSynth+ filters - F3KDB | FFT3D | DFTTest | MiniDeen | Temporal Median
MeteorRain is offline   Reply With Quote
Old 19th August 2019, 19:29   #853  |  Link
Atak_Snajpera
RipBot264 author
 
Atak_Snajpera's Avatar
 
Join Date: May 2006
Location: Poland
Posts: 7,806
Quote:
Originally Posted by HolyWu View Post
L-SMASH-Works-r935+31-20190818
  • LWLibavVideoSource no longer indexes audio streams. It reduces both the file size and parsing time of the index file. LWLibavAudioSource will re-create the index file for the source file which was already indexed by LWLibavVideoSource so as to index audio streams.
  • Print indexing progress to stderr.
  • Tell lavf to discard unwanted packets so they needn't be demuxed.
  • Remove InputFilePath field from the index file. It's unnecessary and troublesome when users rename or move the source file.
  • Automatically re-create the index file when the file size or the last modification time of the source file doesn't match.
Could you remove that checking of "Last modification time"?
It is annoying that lsmash now always re-indexes file which was just copied.

original
video=LWLibavVideoSource("original_video.MP4",cachefile="C:\index.lwi")

Copy
video=LWLibavVideoSource("C:\copy_of_original_video.MP4",cachefile="C:\index.lwi")

The same file but with different modification time. For example FFMS2 does not care about "Last modification time". I think it only checks file size.
Atak_Snajpera is offline   Reply With Quote
Old 19th August 2019, 22:42   #854  |  Link
MeteorRain
結城有紀
 
Join Date: Dec 2003
Location: NJ; OR; Shanghai
Posts: 894
Atak,

Although I agree there's other ways of doing the check, I don't think Windows file copy actually changes the mtime.

If you use cp, you can use --preserve=timestamps.

I might want to change it to some quick partial checksum if mtime mismatches.
__________________
Projects
x265 - Yuuki-Asuna-mod Download / GitHub
TS - ADTS AAC Splitter | LATM AAC Splitter | BS4K-ASS
Neo AviSynth+ filters - F3KDB | FFT3D | DFTTest | MiniDeen | Temporal Median
MeteorRain is offline   Reply With Quote
Old 19th August 2019, 23:18   #855  |  Link
Atak_Snajpera
RipBot264 author
 
Atak_Snajpera's Avatar
 
Join Date: May 2006
Location: Poland
Posts: 7,806
The issue is that i AM copying files using my own custom code (streams and no win api). In this case last modification date is different. I think that Simple check for file size is enough.
Atak_Snajpera is offline   Reply With Quote
Old 20th August 2019, 09:31   #856  |  Link
Taurus
Registered User
 
Taurus's Avatar
 
Join Date: Mar 2002
Location: Krautland
Posts: 903
@HolyWu
Tried to redownload your build from the mediafire link.
"Download not available"....
Thanks for your efforts!
Aaah, I just saw it is working again,
maybe a mediafire hiccup ;-)

Last edited by Taurus; 20th August 2019 at 09:43. Reason: Working again..
Taurus is offline   Reply With Quote
Old 20th August 2019, 11:54   #857  |  Link
Atak_Snajpera
RipBot264 author
 
Atak_Snajpera's Avatar
 
Join Date: May 2006
Location: Poland
Posts: 7,806
Quote:
Originally Posted by HolyWu View Post
FFMS2 also checks partial SHA-1 hashing in addition to file size, but I really don't bother to do it. After all video files aren't like text files that normal users would open them in Notepad++ and edit some random bytes. Hence I agree that simply checking file size should be sufficient for most users. Redownload from the usual post.
Once again thank you for this quick change
Atak_Snajpera is offline   Reply With Quote
Old 20th August 2019, 16:36   #858  |  Link
jpsdr
Registered User
 
Join Date: Oct 2002
Location: France
Posts: 2,309
As isaid, i used to make builds using the method described here (with VS 2015 at the time) : https://github.com/BrunoReX/build-sc...MASH/readme.md
I've been able to find out the main issue, it's probably a change in msys2 behavior, which doesn't keep the %PATH% by default.
I fixed it, make all the gcc builds, but when compiling the VS project, i have the following error :
LINK : fatal error LNK1104: unable to open file 'lsmash.lib'

I take any idea or advice.
__________________
My github.
jpsdr is offline   Reply With Quote
Old 26th August 2019, 02:01   #859  |  Link
MeteorRain
結城有紀
 
Join Date: Dec 2003
Location: NJ; OR; Shanghai
Posts: 894
Quote:
Originally Posted by HolyWu View Post
Hence I agree that simply checking file size should be sufficient for most users.
Not exactly correct. A potential issue is that user try to cut a mpeg-ts file at fixed length and overwrite to the same output file. Re-opening such segment the second time will probably crash the code.

I second the idea of checksuming, and if you don't bother doing that, I might.
__________________
Projects
x265 - Yuuki-Asuna-mod Download / GitHub
TS - ADTS AAC Splitter | LATM AAC Splitter | BS4K-ASS
Neo AviSynth+ filters - F3KDB | FFT3D | DFTTest | MiniDeen | Temporal Median
MeteorRain is offline   Reply With Quote
Old 26th August 2019, 04:46   #860  |  Link
MeteorRain
結城有紀
 
Join Date: Dec 2003
Location: NJ; OR; Shanghai
Posts: 894
LSMASHSource-Release_r935+33.zip
Commit Diff

I have changed the logic. If size and timestamp match, assume unchanged. If size matches but not timestamp, hash (first 1MiB + last 1MiB) and compare with recorded hash. If file is too small, only first up to 1MiB counts. Hash method is xxhash which is extremely fast.

A missing hash record will not force regenerating the index file as long as size and timestamp match.

Let me know how that works.

I also include an easy-compiling branch and a cmake build script. Get VS 2017 / 2019 and cmake installed, and use the following batch file to build your copy.

Code:
@echo off

mkdir build\x86
pushd build\x86
cmake -DCMAKE_GENERATOR_PLATFORM=Win32 -D_DIR=x86 ..\..\
popd
mkdir build\x64
pushd build\x64
cmake -DCMAKE_GENERATOR_PLATFORM=x64 -D_DIR=x64 ..\..\
popd
cmake --build build\x86 --config Release
cmake --build build\x64 --config Release
__________________
Projects
x265 - Yuuki-Asuna-mod Download / GitHub
TS - ADTS AAC Splitter | LATM AAC Splitter | BS4K-ASS
Neo AviSynth+ filters - F3KDB | FFT3D | DFTTest | MiniDeen | Temporal Median
MeteorRain is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 15:47.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.