L-SMASH Source

[videoh]

We're having so much fun I decided to try with GPU encoding for this use case we are discussing:

DGSource() + NVENCC + 2080 Ti + 7700K: 1:05.

I tried to choose settings that produced results perceptually approximating x264 medium.

[LigH]

Isn't the video decoding done in a separate PureVideo decoder chip which may not have developed as much as the 3D rendering and shader operations? ... The closet was the room with the least needs for space in this house.

[FranceBB]

That means the video renderer can't keep up because of the limited PCIe bandwidth? Very interesting.

Also the number of lanes that the CPU has with the GPU is important. For instance, Xeon CPUs like the W-3265M has 64 PCI-E lanes, while its consumer counterpart probably has less lanes. The thing is that it's not only a matter of GPU, but also a matter of which CPU you are using. As a matter of fact, having a very good GPU with a not so good CPU will limit its efficiency.

[redbtn]

Can I safely index MKV (hevc) with audio tracks and subs for now? I read somewhere earlier that i need remux MKV with only video track for better indexing, cuz another tracks can impact on accuracy.

[videoh]

Isn't the video decoding done in a separate PureVideo decoder chip which may not have developed as much as the 3D rendering and shader operations? ... The closet was the room with the least needs for space in this house. Quite right, LigH. Seems to me that the main nVidia development effort goes into gaming-relevant things. We really need super high PCI bandwidths too.

[MeteorRain]

Can I safely index MKV (hevc) with audio tracks and subs for now? I read somewhere earlier that i need remux MKV with only video track for better indexing, cuz another tracks can impact on accuracy.

Fairly safe. It used to index all other tracks and would create huge index file. Now they are all skipped. Let us know if you hit issues again.

[StvG]

@HolyWu, I built LSMASHSource from your repo using this CMake file (https://pastebin.com/raw/cMFdyuye) taken from @MeteorRain repo. When tested for seeking issues with that file (https://www.mediafire.com/file/dldzx9smmbxf60m/vc1_sample.mkv/file) my build has errors but your build is fine. Something wrong with my building process? ffmpeg 4.2.1 / 4.3 used without difference.
There is patch for replacing avresample with swresample in your repo but avisynth exlibs - #pragma comment( lib, "libavresample.a" ).

[StvG]

Thanks for the info.

[FNSCAR]

@HolyWu ,I downloaded your build from release page of your github but there is no vsLSMASHSource.dll in folder.
Does it support vapoursynth?

[ChaosKing]

@HolyWu ,I downloaded your build from release page of your github but there is no vsLSMASHSource.dll in folder.
Does it support vapoursynth?

It works for avisynth and vapoursynth.

[FNSCAR]

It works for avisynth and vapoursynth.

I update vapouraynth to latest R47.2 and it works.Thank you.

[DJATOM]

R47.2 is buggy. Stick with R46 or go to latest R48 test. It contains fixes for all spotted bugs of R47.

[Atak_Snajpera]

Question: Would be possible to use multiple threads to speed up indexing process? For example by using some kind of chunk indexing?
I'm asking because NVMe SSDs with read speed of 3GiB/s+ are getting much more affordable. I have Xeon E5-2690@3.2GHz (8C/16T) and I see that i'm experiencing a bottleneck due to single threaded code.
In this example I'm indexing 4k movie (~50Mbps) from RAM disk...
https://i.imgsafe.org/c7/c7b40e940e.png
1,24GiB/s is a max I can get on single core. ffmsindex is even more cpu intensive (only ~0,5GiB/s).

Currently indexing of ~60 GiB 4k movie takes ~1 minute. With proper multi-threading we could easily reduce indexing time to ~15s!

[kedautinh12]

https://github.com/HolyWu/L-SMASH-Works/releases/latest

Uploaded new binary which ditches libaom and is built by clang-cl. The binary size is smaller than the previous ICL build and decoding speed seems to be marginally faster.




No problem. Here (https://drive.google.com/open?id=1Wja3h-8UmgzzZ7ID2cacoRsoKAmGnsmi).

Your Latest still error with .vp9, add this video and check
https://drive.google.com/file/d/1FAl-hS_jsvUGaHFhpK8sImwc91PUt2xK/view?usp=drivesdk

[stax76]

Question: Would be possible to use multiple threads to speed up indexing process? For example by using some kind of chunk indexing?
I'm asking because NVMe SSDs with read speed of 3GiB/s+ are getting much more affordable. I have Xeon E5-2690@3.2GHz (8C/16T) and I see that i'm experiencing a bottleneck due to single threaded code.
In this example I'm indexing 4k movie (~50Mbps) from RAM disk...
https://i.imgsafe.org/c7/c7b40e940e.png
1,24GiB/s is a max I can get on single core. ffmsindex is even more cpu intensive (only ~0,5GiB/s).

Currently indexing of ~60 GiB 4k movie takes ~1 minute. With proper multi-threading we could easily reduce indexing time to ~15s!

Maybe you are asking the wrong question, how about MakeMKV creating an index? Answer: It's just code, anything is possible.

[kedautinh12]

https://down.7086.in/AviSynthPlus%20Filters/LSMASHSource-r935%2B34.zip

Stop indexing progress spamming

-- Now only refresh at every 1%.

Still error with vp9, only old ver r929 of VFR-maniac still fine

[poisondeathray]

Your Latest still error with .vp9, add this video and check
https://drive.google.com/file/d/1FAl-hS_jsvUGaHFhpK8sImwc91PUt2xK/view?usp=drivesdk

This is AVC. Works ok. Rename the file characters

[kedautinh12]

This is AVC. Works ok. Rename the file characters

I was renamed and load ok but still error with image in video

[poisondeathray]

I was renamed and load ok but still error with image in video

what error? what time ?

[kedautinh12]

what error? what time ?

When i load video with LWLibavVideoSource still error with L-SMASH-Works_20190917 and LSMASHSource-r935+34
https://drive.google.com/open?id=115BKlC1q0kjdEjgsmqYi9jrKPfOO2smu

But i load ok with L-SMASH-Works r929
https://drive.google.com/open?id=16ckCimru9cIALC6wH_G738RIUaCO2MtC

[poisondeathray]

I see no errors with 20190917

Are you sure it's the correct video ? Earlier you said ".vp9" . This is AVC in MP4.

[kedautinh12]

I see no errors with 20190917

Are you sure it's the correct video ? Earlier you said ".vp9" . This is AVC in MP4.

Ah sorry my fault, this video right
https://drive.google.com/open?id=1y0MdC98p4AGr52nddZwXqzCMQwNSrTlx
Error frame with vp9 in .MKV

[Atak_Snajpera]

Creation of audio index file needs some optimalization because file size can be extremely large.
Example: 10 min TrueHD
https://i.postimg.cc/BQLdyWsH/Capture.png

It looks like that every 1 minute of TrueHD audio adds extra 10 MiB to index file. With typical 100+ minute movie index file will be over 1 GiB! Parsing that big ass file on single core is just super ineficient.

Question: Do we really need those repeated information in each line? From what I see only values for POS,PTS,DTS are changing. Furthermore values PTS and DTS are increasing in predictable manner.

<LSMASHWorksIndexVersion=0.0.2.0>
<LibavReaderIndexFile=15>
<FileSize=526166830>
<FileHash=0x7b6795d7>
<LibavReaderIndex=0x00000180,1,truehd>
<ActiveVideoStreamIndex>-0000000001</ActiveVideoStreamIndex>
<ActiveAudioStreamIndex>+0000000000</ActiveAudioStreamIndex>
Index=0,Type=1,Codec=86060,TimeBase=1/90000,POS=0,PTS=0,DTS=0,EDI=0
Channels=8:0x63f,Rate=48000,Format=s32,BPS=24,Length=40
Index=0,Type=1,Codec=86060,TimeBase=1/90000,POS=576,PTS=75,DTS=75,EDI=0
Channels=8:0x63f,Rate=48000,Format=s32,BPS=24,Length=40
Index=0,Type=1,Codec=86060,TimeBase=1/90000,POS=740,PTS=150,DTS=150,EDI=0
Channels=8:0x63f,Rate=48000,Format=s32,BPS=24,Length=40
Index=0,Type=1,Codec=86060,TimeBase=1/90000,POS=902,PTS=225,DTS=225,EDI=0
Channels=8:0x63f,Rate=48000,Format=s32,BPS=24,Length=40
Index=0,Type=1,Codec=86060,TimeBase=1/90000,POS=1050,PTS=300,DTS=300,EDI=0
Channels=8:0x63f,Rate=48000,Format=s32,BPS=24,Length=40
Index=0,Type=1,Codec=86060,TimeBase=1/90000,POS=1194,PTS=375,DTS=375,EDI=0
Channels=8:0x63f,Rate=48000,Format=s32,BPS=24,Length=40
Index=0,Type=1,Codec=86060,TimeBase=1/90000,POS=1340,PTS=450,DTS=450,EDI=0
Channels=8:0x63f,Rate=48000,Format=s32,BPS=24,Length=40
Index=0,Type=1,Codec=86060,TimeBase=1/90000,POS=1476,PTS=525,DTS=525,EDI=0
Channels=8:0x63f,Rate=48000,Format=s32,BPS=24,Length=40
Index=0,Type=1,Codec=86060,TimeBase=1/90000,POS=1624,PTS=600,DTS=600,EDI=0
Channels=8:0x63f,Rate=48000,Format=s32,BPS=24,Length=40
Index=0,Type=1,Codec=86060,TimeBase=1/90000,POS=1762,PTS=675,DTS=675,EDI=0
Channels=8:0x63f,Rate=48000,Format=s32,BPS=24,Length=40
Index=0,Type=1,Codec=86060,TimeBase=1/90000,POS=1912,PTS=750,DTS=750,EDI=0
Channels=8:0x63f,Rate=48000,Format=s32,BPS=24,Length=40
Index=0,Type=1,Codec=86060,TimeBase=1/90000,POS=2058,PTS=825,DTS=825,EDI=0
Channels=8:0x63f,Rate=48000,Format=s32,BPS=24,Length=40
Index=0,Type=1,Codec=86060,TimeBase=1/90000,POS=2196,PTS=900,DTS=900,EDI=0
Channels=8:0x63f,Rate=48000,Format=s32,BPS=24,Length=40
Index=0,Type=1,Codec=86060,TimeBase=1/90000,POS=2346,PTS=975,DTS=975,EDI=0
Channels=8:0x63f,Rate=48000,Format=s32,BPS=24,Length=40
Index=0,Type=1,Codec=86060,TimeBase=1/90000,POS=2490,PTS=1050,DTS=1050,EDI=0
Channels=8:0x63f,Rate=48000,Format=s32,BPS=24,Length=40

[Atak_Snajpera]

https://github.com/HolyWu/L-SMASH-Works/releases/download/20191115/L-SMASH-Works_20191115.7z


Update to FFmpeg-20191114-73ee53f.
VapourSynth: Export mastering display metadata and content light level in frame properties.
VideoSource: Improve capability check in prefer_hw.
LWLibav: Fix VP9 decoding issue with superframes.
VideoSource: Enable AV1 decoding via libdav1d.
Add parameter ff_loglevel.
LWLibav: Adjust the structure of index file to reduce file size.


Thank you for your excellent work!

[fg118942]

https://github.com/HolyWu/L-SMASH-Works/releases/download/20191115/L-SMASH-Works_20191115.7z


Update to FFmpeg-20191114-73ee53f.
VapourSynth: Export mastering display metadata and content light level in frame properties.
VideoSource: Improve capability check in prefer_hw.
LWLibav: Fix VP9 decoding issue with superframes.
VideoSource: Enable AV1 decoding via libdav1d.
Add parameter ff_loglevel.
LWLibav: Adjust the structure of index file to reduce file size.


HolyWu, thank you so much for fixing the issue of VP9.
However, using LSMASHVideoSource in version 20191115 will cause Access Violation.

[2ndR]

Perfect

[hydra3333]

Thank you HolyWu.
I have just visited this thread for the first time in a while.
I have changed my icky x264 build process to use your LSW git accordingly :)
Nice work !!!!


edit:
PS is there any possibility you may turn on the Issues tab in your github fork ?
Apparently,
- Go to the Settings page of your fork.
- Check the box next to Issues
- You can now file issues on your own fork and they will not be placed in the main repo.

[FranceBB]

https://github.com/HolyWu/L-SMASH-Works/releases/download/20191116/L-SMASH-Works_20191116.7z


LibavSMASH: Fix access violation.


Thank you, once again! :)

[Patman]

https://github.com/HolyWu/L-SMASH-Works/releases/download/20191116/L-SMASH-Works_20191116.7z


LibavSMASH: Fix access violation.


Hi HolyWu,

thanks for your build. I think there is a problem when some special characters like ' are included in the file name. Can you please check that? For me the LibavSMASHSource Filter is corrupted. LWLibavSource works.

[StainlessS]

Is the offending character perhaps slanting single quote (`) ?

EDIT: Normal Single quote ('), Grave (`), [B]Acute [forwards slant] (ˊ).

EDIT: Patman, post your actual problem filename.

EDIT: I note these new functions in Avs+:- http://avisynth.nl/index.php/Internal_functions#String_functions

StrToUtf8

StrToUtf8(string) AVS+
Converts string from ANSI to UTF8.

StrFromUtf8

StrFromUtf8(string) AVS+
Converts string from UTF8 to ANSI

[LigH]

Not to forget typographic single quotes in nationally specific styles...

[Patman]

Cannot reproduce.

AVS:
https://i.postimg.cc/LqkvWJYd/test.png (https://postimg.cc/LqkvWJYd)

VS:
https://i.postimg.cc/xJwCb15C/test.png (https://postimg.cc/xJwCb15C)

Thanks for your fast reply. The failure was in call of the file. Thanks again.

EDIT: Patman, post your actual problem filename.

The Redwood's - single quote

[StainlessS]

The Redwood's - single quote
So I'm assuming that its "The Redwood's" with ordinary single quote & without a path, and also without relative path ie not ".\The Redwood's".
SPACE in filename can also be a problem for some tools, HASH(#) used to be problem in MeGUI input filename,
used to interpret it as part of some kind of command, no longer a problem for quite some time.
Assuming that problem exists without using eg MeGUI, and only load into Avsinth (or avs+).
Does it work OK when single quote stripped from filename (and of course the call to src filter).

EDIT: Some commands/calls to functions/filters fail on W7+ where used to work on XP (dont know about Vista),
M$ did something that no longer converts eg "name.xxx" to a proper relative path, used to work fine on XP.

EDIT: IIRC, W7+ can interpret filename without path/relative path, to "C:\"+Filename, which then re-targets to Virtualstore (or something like that).

[Patman]

So I'm assuming that its "The Redwood's" with ordinary single quote & without a path, and also without relative path ie not ".\The Redwood's".
SPACE in filename can also be a problem for some tools, HASH(#) used to be problem in MeGUI input filename,
used to interpret it as part of some kind of command, no longer a problem for quite some time.
Assuming that problem exists without using eg MeGUI, and only load into Avsinth (or avs+).
Does it work OK when single quote stripped from filename (and of course the call to src filter).

EDIT: Some commands/calls to functions/filters fail on W7+ where used to work on XP (dont know about Vista),
M$ did something that no longer converts eg "name.xxx" to a proper relative path, used to work fine on XP.

Hi StainlessS,

there was a failure in the code structure of my command. I used vapoursynth to test some plugins and functions. The full path is E:\x264\test\The Redwood's.mp4. After i corrected the code in the command line, everything works fine. Thanks for the help.

[Rumbah]

I noticed that using the cache file option with version 20191116 and Vapoursynth 64bit results in two lwi files being created.

Using the code line below results in the file creation at Q:\%folder%\video.lwi and the standard file at source + ".lwi" as well.
core.lsmas.LWLibavSource(source=r"%Source%", cachefile=r"Q:\%folder%\video.lwi")
Or do I use the option wrong?

[StainlessS]

Cannot reproduce. :confused::confused::confused:

I'm guessin' that Rumbah did not delete existing files beforehand [just a silly faux pas - suggest Rumbah retries after delete both].

[Rumbah]

I just tested again with fresh files an folders and got the same result. Will try to get a reproducible test case to share.

[kedautinh12]

https://github.com/HolyWu/L-SMASH-Works/releases/download/20191116/L-SMASH-Works_20191116.7z


LibavSMASH: Fix access violation.


Your build very good with many .ts but with some .ts duration of video doesn't match with duration of audio
Example:
https://drive.google.com/a/my.smccd.edu/file/d/1rUauSk-C-QxYxxJDgkFGZDV5CiiGomr1/view?usp=drivesdk

[Rumbah]

Well, sorry for the false alarm, it works now, I had an mistake in my program.

[StainlessS]

Rumbah, can you enlighten us on what the mistake was, might give a clue for some other problem in future [also we like a good giggle :) ].

[Rumbah]

I have a script that converts a video with x264 and splits the source to X parts to encode the parts in parallel.
I found the performance to be better with less threads and more parts than with straight 36 threads per encode (a Ryzen 3900X with 12cores/24threads).

So I create the parts via Vapoursynth and L-Smash to get the cutting frame accurate so I have
video = core.lsmas.LWLibavSource(source=r"%Source%", cachefile=r"Q:\%directory%\video.lwi")in more than one part of the script.

First it was just video = core.lsmas.LWLibavSource(source=r"%Source%")but I wanted all the temp files in one directory for easier cleaning after finishing. And I just forgot one place to change it so the lwi file got created in both places.

[fg118942]

Your build very good with many .ts but with some .ts duration of video doesn't match with duration of audio
Example:
https://drive.google.com/a/my.smccd.edu/file/d/1rUauSk-C-QxYxxJDgkFGZDV5CiiGomr1/view?usp=drivesdk

Not patched FFmpeg, the Repeat First Field flags will not work, so you need to use the following patch.
https://gist.github.com/maki-rxrz/5a7a2c789e4369fa34853b5358fb8a29

[StainlessS]

Rumbah, thanx for the explanation.

[kedautinh12]

Not patched FFmpeg, the Repeat First Field flags will not work, so you need to use the following patch.
https://gist.github.com/maki-rxrz/5a7a2c789e4369fa34853b5358fb8a29

Oh, but how i can use this??

[fg118942]

Oh, but how i can use this??

We'll have to wait for HolyWu to upload a new version with this patch applied.
After that, write the following avs.

LWLibavVideoSource("***.ts",repeat=true)

[kedautinh12]

We'll have to wait for HolyWu to upload a new version with this patch applied.
After that, write the following avs.

LWLibavVideoSource("***.ts",repeat=true)

Nice, waiting for HolyWu

[Forteen88]

Thanks HolyWu.

[kedautinh12]

https://github.com/HolyWu/L-SMASH-Works/releases/download/20191127/L-SMASH-Works_20191127.7z


Fix MPEG-2 decoding issue with RFF flags. (maki-rxrz)
Fix interlaced H.264 decoding issue in some files.


Wow, thanks so much

[kedautinh12]

https://github.com/HolyWu/L-SMASH-Works/releases/download/20191127/L-SMASH-Works_20191127.7z


Fix MPEG-2 decoding issue with RFF flags. (maki-rxrz)
Fix interlaced H.264 decoding issue in some files.


when i decoding LWLibavVideoSource("***.ts",repeat=true) with repeat=true can Fix MPEG-2 decoding issue with RFF flags (maki-rxrz) and Fix interlaced H.264 decoding issue in some files. Very nice

[LigH]

I found an MP4 video provided by the Mediathek of the German broadcaster ZDF. MediaInfo reports a framerate of 50 fps (720p50), AviSynth's Info() reports the same when using FFVideoSource or LwLibavVideoSource. But LSMASHVideoSource reports 0.0246 fps (1798/73073).

I wonder how to send you a sample. If I use any tool to cut out a small scene, it would create completely new headers, possibly fixing this issue. So I fear I would have to upload ~680 MB. And if you tried to download it using MediathekView, it might be geoblocked if you are outside Germany. (direct URL (https://rodlzdf-a.akamaihd.net/none/zdf/19/11/191122_sendung_hsh/4/191122_sendung_hsh_3328k_p36v14.mp4), may or may not work in a web browser)