L-SMASH Source

[FranceBB]

Testing on a brand new content, the Sony UHD HDR PQ, RGB Full Range 12bit 4000 nits Remaster of Big Fish, the movie:

https://i.imgur.com/4DPzFAi.png
https://i.imgur.com/PRjylTP.png

Indexing is in progress, the .lwi file is being created and it's growing:

https://i.imgur.com/lgkAqXj.png

but it's gonna take a while I'm afraid:

https://i.imgur.com/hgHndS6.png
https://i.imgur.com/qDcw9zH.png

anyway, the fact that the indexing is in progress is indeed very promising.
I'll keep you posted, guys, thanks! :)

[kedautinh12]

Yeah 835 gb :D

[FranceBB]

Yeah 835 gb :D

Ladies and gentlemen: it works like a charm! :D

https://i.imgur.com/Ic3WacT.png

I replied to the ticket and closed it. https://github.com/HomeOfAviSynthPlusEvolution/L-SMASH-Works/issues/16
Very very very well done, guys!
I gotta be fair, LWLibav is my favorite Avisynth indexer 'cause you guys always picked my issues up and helped me (unlike *some other devs of another indexer which shall remain nameless*).
You did that with the DNX 185 frame slice issue, you did that with the IPCM audio decoding issue and now you've done it again with IMF support.
Thank you so much, I really mean it, whenever I had an issue, you've been there for me.

[kedautinh12]

Ladies and gentlemen: it works like a charm! :D

https://i.imgur.com/Ic3WacT.png

I replied to the ticket and closed it. https://github.com/HomeOfAviSynthPlusEvolution/L-SMASH-Works/issues/16
Very very very well done, guys!
I gotta be fair, LWLibav is my favorite Avisynth indexer 'cause you guys always picked my issues up and helped me (unlike *some other devs of another indexer which shall remain nameless*).
You did that with the DNX 185 frame slice issue, you did that with the IPCM audio decoding issue and now you've done it again with IMF support.
Thank you so much, I really mean it, whenever I had an issue, you've been there for me.

But i think you need rep quickly next time :D

[FranceBB]

But i think you need rep quickly next time :D

Yeah, definitely, but it's been a really busy week as we're migrating our Loudness Correction workflows from the old unsupported Dolby DP600 to Emotion, so I had to learn how to create workflow in there to create DolbyE (https://i.imgur.com/6LJ18Kl.png) and how to call their APIs (https://i.imgur.com/EqR0xSi.png) and make them work with our system etc etc etc
I wish I could spend all my time at work on Avisynth, but sadly I can't... :(

[mastrboy]

Is it possible to expose the "apply_grain" parameter for the dav1d decoder to avisynth in LSMASHVideoSource/LWLibavVideoSource? (https://videolan.videolan.me/dav1d/structDav1dSettings.html#a900e1cfb50a473fb1640f9af5a8faaa5)

[kedautinh12]

Ask Author's L-Smash works
https://github.com/HomeOfAviSynthPlusEvolution/L-SMASH-Works/issues

[flossy_cake]

Hello, is it possible to use DXVA2-copyback decoding with LWLIbavVideoSource? Or can it be configured to use LAV Video Decoder which supports DXVA2-copyback?

I tried prefer_hw, and I see there is decoder parameter also, but nothing in the wiki saying it supports DXVA2-copyback.

edit: also is it possible for LWLibavVideoSource to "signal downstream" so that the renderer (eg. MadVR) receives all the correct metadata like matrix, primaries, dynamic range etc. These all seem to go missing when I use Avisynth -- normally when I use LAV Splitter & Decoder without Avisynth it signals all those metadata to MadVR which helps with colour accuracy.

[flossy_cake]

Perhaps something like this?

LoadPlugin("LSMASHSource.dll")
LWLibavVideoSource("video.mkv")
Trim(0,-1)



Is it normal for cachefile creation to take 30 seconds for a 1 hour of video loading from SSD with an i7-4790k?

Each cache file is 16MB * 56 episodes = 896mb of cachefiles just for 1 series... :eek:

edit: tried making the cachefile from ffmpeg vs MPC-HC and the time is the same. Tried setting threads=n manually. CPU usage is around 4% during cachefile creation...bottleneck maybe?

[FranceBB]

Ok, since no one is replying, I'll answer a few questions.

1) Frame properties

Whenever an indexer indexes a source in a modern version of Avisynth+ frame properties are passed through and you can see them if you open the script in AVSPmod mod and you right click on the preview. Before those there were only clip properties, but now with frame properties everything an indexer indexes is passed through and can be used by third party programs receiving the uncompressed audio and video stream in RAM from the frameserver and use those, like it happens in ffmpeg. The fact that your final player/decoder opening the AVS doesn't get them means that it doesn't support frame properties, so it's not an LWLibavVideoSource() issue but rather your player, so you should ask them to support frame properties.

P.s beware that frame properties support is still experimental for most Avisynth filters etc


2) Indexing

Indexing can take a very long time and it's almost always a bandwidth problem. Honestly, 30 seconds for a 1h video sounds pretty damn fast to me! It's pretty clear that you're working on small file size videos, probably consumer stuff, so 30 seconds for 1h is pretty good/alright in terms of performance. Keep in mind that when I index UHD masterfiles over a 1 Gbit/s network it takes around 7 hours to index a 1h and 30 min movie (around 700-800 GB), so I'd say your performance is good. Last but not least, indexing files are larger for LWLibavVideoSource() and LWLibavAudioSource() than they are for FFVideoSource() and FFAudioSource(), so it's normal. Oh, almost forgot, one final note, if you happen to have like MP4 with H.264 and AAC etc or any other ISO format, you can use LSMASHVideoSource() and LSMASHAudioSource() as indexers in Avisynth but unlike normal indexers, they're gonna use the index already available inside the container to load the content straight away. Although this is supposed to be safe if the container is fine, it might lead to issues if there's something wrong with the container.

[flossy_cake]

Thanks

[Katie Boundary]

Is there an XP-compatible fork of this?

[FranceBB]

Is there an XP-compatible fork of this?

Unfortunately, no. :(
Not an updated one at least... :(
Last time LWLibav was compiled for XP, 16bit interleaved was still the default for high bit depth sources, so you can imagine how old it is...

[manolito]

The latest XP compatible LSmash Version I am aware of is this one:

lsmash-r784-150316-XP

It is from 2015, let me know if I should upload it...

[StainlessS]

lsmash-r784-150316-XP Here :- http://megui.tmebi.de/stable/

[flossy_cake]

Is the author perhaps interested in releasing a version without mandatory caching, for use by those who just use it to watch video in realtime and don't need frame accurate seek?

Or am I misunderstanding things and is frame accurate seek needed for filters to work properly -- and if so, which ones?

:thanks:

[FranceBB]

Is the author perhaps interested in releasing a version without mandatory caching, for use by those who just use it to watch video in realtime and don't need frame accurate seek?


You can't, unless you're willing to use the index inside the container and take it for good (which might not be).
So, if you take LSMASHVideoSource() and LSMASHAudioSource(), it will NOT index the file but rather use the index inside the container and take it for good. Not only this might not be, but some containers just don't have it like .ts to name one, therefore it's limited to ISO Standard containers only (like mp4), however if this is what you want to preview files etc, then go for it. ;)



Or am I misunderstanding things and is frame accurate seek needed for filters to work properly -- and if so, which ones?


You do need to have frame accurate indexing in any scenario, to be fair. Imagine this:

1) You index a source with let's say DirectoShowSource() using the LAV Filters.

2) You want to trim a commercial, so you use Trim(1231, 1554) and then you go on. Everything looks fine in AVSPmod.

3) You run the encode, only to find out that the commercial didn't really start nor end where you thought it was gonna be and you gotta start over.


This is why having frame accurate indexing is important.
It's also important if you want to hardsub subtitles 'cause when you send out low res proxies to the subtitling team and you get subs back, you want not just timecodes but frames to be accurate (especially if they have been doing typesetting on them). It's also important if you use ANY temporal filter, like MVTools to perform vector calculations or if you wanna perform like temporal denoise etc.

I mean, I can't imagine any reason not to use frame accurate indexing.

There are of course still cases that remain valid to use things like DirectShowSource(), for instance, some proprietary codecs like the Red Raw etc for which you legally installed the proprietary codec and you can leverage on DirectShowSource() in Windows to provide the decoded frames, but you can't use any open source indexer 'cause they wouldn't be able to index those files as they don't have decoders.

[flossy_cake]

Thanks I tried LSMASHVideoSource but like you said it's quite limited and none of my videos open with it.

I'm only using Avisynth for real time playback viewing of my video library. Real time decimation, deint, cropping, aspect, chroma shift etc. so I probably don't need frame accurate seeking.

If anyone knows of a way to disable caching/indexing please advise -- maybe there is some undocumented string for one of the cache params to forcefully disable it?

[FranceBB]

If anyone knows of a way to disable caching/indexing please advise -- maybe there is some undocumented string for one of the cache params to forcefully disable it?


You can't disable indexing unless you use DirectShowSource() + third party codecs like LAV.

About the "I'm using it only to play stuff in my library", why not using PotPlayer which supports Avisynth post-processing via an Avisynth Script?
PotPlayer will pipe to Avisynth automatically under the hood, thus avoiding indexing.

[StainlessS]

Thanks I tried LSMASHVideoSource but like you said it's quite limited and none of my videos open with it.

LSmashVideoSource is for ISO containers only, according to my list M2TS is not type ISO.

Function IsISOFileName(String s) {s=RT_GetFileExtension(s) Return(s==".mov"||s==".mp4"||s==".m4v"||s==".3gp"||s==".3gpp2"||s==".3g2"||s==".mj2"||s==".dvb"||
\ s==".dcf"||s==".m21"||s==".m4a"||s==".m4b"||s==".m4p"||s==".k3g"||s==".jpm"||s==".jpx"||s==".mqv"||s==".ismv"||s==".isma"||s==".f4v")}


https://en.wikipedia.org/wiki/ISO/IEC_base_media_file_format

https://en.wikipedia.org/wiki/Comparison_of_video_container_formats

EDIT: Sort of related, ftyp's :- http://www.ftyps.com/
Also see note 1 [what is an ftyp] (a sort of FourCC).

LSMashVideoSource() is just for ISO [ie MP4 and similar] files.
Otherwise use LWLibavVideoSource(). [An alternative video source in LSmash dll]

Same for the audio, LSMashAudioSource(), and LWLibavAudioSource().

[flossy_cake]

About the "I'm using it only to play stuff in my library", why not using PotPlayer which supports Avisynth post-processing via an Avisynth Script?
PotPlayer will pipe to Avisynth automatically under the hood, thus avoiding indexing.

Thanks for the tip. Currently I'm exploring the same kind of thing with MPC-HC using CrendKing's Avisynth filter (https://github.com/CrendKing/avisynth_filter). As far as I can tell it inserts Avisynth between LAV decoders and the renderer, so no indexing required either, but I'm having a couple of dealbreaking issues with it (see Issues tab in the Git).

[kedautinh12]

L-SMASH-Works 20221109
https://github.com/HomeOfAviSynthPlusEvolution/L-SMASH-Works/releases

[FranceBB]

L-SMASH-Works 20221109
https://github.com/HomeOfAviSynthPlusEvolution/L-SMASH-Works/releases

What an important changelog!


AviSynth: added support for XYZ12LE format (represented as RGB48).

Thank you so so so so much Asd-g! :D
I've never felt so welcomed and cared for, honestly!
Every single time I raise an issue with this indexer, it's taken into consideration and addressed.

I love LWLibav :D



By the way, I tested it and it does indeed work.
While FFVideoSource() converts XYZ to YUV, LWLibavVideoSource() preserves the original XYZ inside the video, thus displaying it correctly and allowing us to work directly with XYZ! :D

https://i.imgur.com/h9gHQ4m.png

[kedautinh12]

@FranceBB, you can comment about XYZ in FFMpeg, i think author will update for it :D

[flossy_cake]

Here is a clip which LWlibavVideoSource cannot seem to play properly: https://drive.google.com/file/d/1lHB_EPFEKAQEiKab5LlYq0EyY9OOObeT/view?usp=share_link

It seems to have something to do with that soft telecine / repeat fields issue causing it to go out of sync with audio because the repeat field flags aren't being obeyed correctly (discussion about that back on page 58 (https://forum.doom9.org/showthread.php?t=167435&page=58)).


file = "480i multiple cadences.mkv"

AudioDub(LWlibavVideoSource(file, repeat=false), LWLibavAudioSource(file, stream_index=-1))

ScriptClip(last, """SubTitle(String(propGetAny(last, "_FieldBased")))""")


If I set repeat=true I get:

https://i.ibb.co/3yd8PbB/1.png

The _FieldBased frame property might offer a clue: the panning sequence starting at 1:25 is showing a _FieldBased value with pattern: 0-1-2, which means "progressive-bff-tff" which doesn't seem right for a 3:2 sequence, so maybe it has something to do with that?

FFMpegSource2 doesn't populate the _FieldBased prop, but can play it correctly with rffmode=1 which means "Honor all pulldown flags":


FFMpegSource2(file, rffmode=1, atrack=-1)


But, something is still not right: if I do simple bob deinterlacing with:


FFMpegSource2(file, rffmode=1, atrack=-1)
Bob()


The result is that "jumping back and forth" artefact which looks like the field order is wrong. So I try to correct that with:


FFMpegSource2(file, rffmode=1, atrack=-1)
AssumeTFF()
Bob()


^ this is better, but there are still some sequences which appear to have the wrong field order -- look at the "Series Story Editor" at 00:14, it still moves in the wrong direction for one frame before jumping forward 2 frames. I would have thought that was impossible with Bob().


edit: it seems these dynamic changes in field order need to be obeyed, in addition to the repeat field flags, in order to get correct playback. Still, it doesn't make sense to me as I thought analogue CRT would just scan field1 followed by field2 at all times. Seems my understanding is completely wrong: https://www.dvmp.co.uk/digital-video.htm

[kedautinh12]

I think you need report here to fix faster
https://github.com/HomeOfAviSynthPlusEvolution/L-SMASH-Works/issues

[flossy_cake]

I think you need report here to fix faster
https://github.com/HomeOfAviSynthPlusEvolution/L-SMASH-Works/issues

Thanks, done.

[isidroco]

I have a problem with small lagarith clips made from pictures with repeated null frames: https://www.mediafire.com/file/hwdf8075reip6ta/07_-_creditos.avi/file
Using it to make a slideshow. DirectShowSource will add an extra frame. And LWLibavVideoSource will only return one frame instead of 150.
This is my test script:

nameS="07 - creditos"
extS=".avi"
# v= DirectShowSource( nameS + extS , ConvertFPS=true)
# v= trim(v, 1, length= framecount(v)-1 )
v= LWLibavVideoSource(nameS+extS)
v= ScriptClip(v, """
Subtitle(String( int(current_frame/30))+" seconds",
\ align=5, size=height(v)/16 )
""")
# v= info(v, size= height(v)*.05)
return v

[Hotte]

Video: AVDH 1280x720 29.97fps 97223kbps [V: Apple Video Media Handler [eng] (dnxhd dnxhr hqx, yuv422p10le, 1280x720, 97223 kb/s)]
Audio: PCM 48000Hz stereo 1536kbps [A: Apple Sound Media Handler [eng] (pcm_s16le, 48000 Hz, stereo, s16, 1536 kb/s)]

Codec ID: sowt

My guess would be that the endianness of the PCM audio was not correctly respected.

Hi, all my Videos containing pcm_s16be or pcm_s16le audio are fully distorted when read with LSmashAudioSource, while LSmashVideoSource is just fine. I've got the latest build from 20221109. It is a pity to wait for indexing of workaround solutions just to get audio right.

Any news about this ?

[kedautinh12]

Are you try LWLibavAudioSource??

[Hotte]

I did and it works. But it needs indexing, which I'd like to avoid and should not be necessary due to the fact that LSmashVideoSource works perfectly on my files for the video-stream without any indexing.

And: I do not understand why these files play instantly without issues and without any indexing if you open them directly with VDub2.

Something seems wrong with LSmashAudioSource and PCM-Audio from everyday-movs from everyday-cams like Nikon D5300 or Panasonic G9.

[kedautinh12]

You can create issue to fix faster or developer will explain about it
https://github.com/HomeOfAviSynthPlusEvolution/L-SMASH-Works/issues

[Hotte]

You can create issue to fix faster or developer will explain about it
https://github.com/HomeOfAviSynthPlusEvolution/L-SMASH-Works/issues

Thanks. I opened issue #26

[StainlessS]

I do not understand why these files play instantly without issues and without any indexing if you open them directly with VDub2.
I think that is because the ISO container files [mp4 etc] that LSmashVideoSource() opens, already have some sort of index, as do AVI files.

EDIT: As posted in #1422,

Function IsISOFileName(String s) {s=RT_GetFileExtension(s) Return(s==".mov"||s==".mp4"||s==".m4v"||s==".3gp"||s==".3gpp2"||s==".3g2"||s==".mj2"||s==".dvb"||
\ s==".dcf"||s==".m21"||s==".m4a"||s==".m4b"||s==".m4p"||s==".k3g"||s==".jpm"||s==".jpx"||s==".mqv"||s==".ismv"||s==".isma"||s==".f4v")}

At least one of above is an audio-only ISO file. [".m4a"]

Only use LWLibavVideoSource() for non ISO video files. [attempt to open non ISO with LSmashVideoSource() will fail].

EDIT:
Here is a reasonably simple function to open files based on extension, works pretty well and could be used as an example to create your own,
https://forum.doom9.org/showthread.php?p=1874068#post1874068
Is implemented as an Avisynthesizer_Mod template.

See also the following posts where may have been improved.

And here:- https://forum.doom9.org/showthread.php?p=1880511#post1880511
Hello_Hello did his own version of a similar script.

Edit: Also, here description of my function (presumably the last mod of mine in that thread), where can change preferred order of source filters tried [EDIT: via Pref arg].
Well sometimes a source filter dont work proper, and is good to have an alternative, and way to force that alternative. [and 5 is better than 2]

Currently lookin' summick like this


Req:- RT_Stats, CallCmd, MIFO_Lib(MediaInfo script lib), MediaInfo CLI v0.7.83+ (may work with later versions)
Also, FFMS, LSmash, RawSourcePlus or RawSource (one of them), DSS2.

Function GetSeq(String "vFn",String "aFn"="",String "Pref"="",Int "VTrack"=-1,Int "ATrack"=-1,Val "fpsNum"=Undefined,Val "fpsDen"=Undefined,
\ Bool "Cache"=True,String "CacheDir"="",Bool "CacheDel"=False,Int "Threads"=0,Int "RFFMode"=0,String "TimeCodes"="",Int "AdjustDelay"=-1,String "VarPrefix"="",

vFn, Default "", default is NO video file. Can also use VTrack=-2 for no video. If "" and Vtrack != -2, then returns a single frame BlankClip, YV12 640x480, subtitle "No Video File".
aFn, Default "". Where "", will use audio from vFn. Can use ATrack=-2 for no audio.

Pref, Default "" {Same as "AIRFLD"). Order in which source filters are tried.
"A" = AviSource() (Only if extension is *.AVI, WavSource if Audio is WAV).
"I" = LSmashVideoSource() (Only if extension is an ISO type file, ie mov, mp4, m4v, 3gp, 3g2, mj2, dvb, dcf, m21.).
"R" = RawSourcePlus() (Only if extension is ".y4m".) RawSourcePlus if Avs+ and RawSource if Not Avs+.
"F" = FFVideoSource().
"L" = LWLibavVideoSource(), from LSMash.
"D" = DSS2()

VTrack, Default: -1 (auto. Video track number. If -2, disable video.
ATrack, Default: -1 (auto). Audio track number. If -2, disable audio.

Cache, Default True. False disables the writing of an index file (LWLibavSource and FFMS2 only).
CacheDir, Default Undefined. The directory location of the index file written by FFMS2 only. NOTE Directory NOT filename, No backslash at end of folder name.
If cache==true and CacheDir = "%TEMP%" then will be set to system environment user TEMP folder.
CacheDel, Default False [forced False if Cache=false] . If true, then auto delete FFMS[.ffindex] & LWLibavSource[.lwi] index files on clip closure.

Threads, Default 0(auto). The number of threads for LSMASH / FFMS4 to use.

RFFMode, Default 0. Honour repeat field flags (LWLibavSource & FFMS2 only).
FFMS2: 0=false, 1=true, 2=force film.
LWLibavSource: 0=false, 1=true, 2=false.


fpsnum,
fpsden, Defaults Undefined. Framerate for VFR->CFR conversion. # *** INGNORED if AVI ***
Specify FrameRate using fpsnum (Numerator) and fpsden (Denominator)
num only, eg fpsnum=25.0, FrameRate = 25.0 FPS
num & den, eg fpsnum=24000 fpsden=1001, FrameRate = 23.976 FPS (Both should be type Int).
neither specified FrameRate = UnDefined, ie whatever source filter thinks it is. (no VFR->CFR conversion)

Timecodes, Default "". FFMS only, write Matroska Timecodes to filename Timecodes.
AdjustDelay, Default -1. FFMS Only.
VarPrefix, Default "". FFMS only, Variable for FFInfo().

[Katie Boundary]

lsmash-r784-150316-XP Here :- http://megui.tmebi.de/stable/

Have I mentioned this year that you're my favorite person on Doom9? :)

[StainlessS]

It's lovely to be somebodies fav, cos everybody else hates me :(

but, seems hello_hello posted about the file first on the D9, [2017]
It appears to be missing for a 32bit operating system setup and dare I say it..... XP.

I think for the older L-Smash it should be msvcr100.dll.
There's a copy in the "lsmash-r784-150316-XP.7z" file but not in the "MeGUI\tools\lsmash" folder. I assume there should be?

I'm not having problems as I have the appropriate runtime installed, but I thought I'd mention it.

[real.finder]

It's lovely to be somebodies fav, cos everybody else hates me :(

but, seems hello_hello posted about the file first on the D9, [2017]

but I don't hate you :eek:

[StainlessS]

Not yet you dont, give it time. :devil:

EDIT: In reply to gispos next post::
Well you should, I'm nasty :devil:

[gispos]

I don't hate you either, my dear. :)

[FranceBB]

It's lovely to be somebodies fav, cos everybody else hates me :(

Nah, who could hate you?
You're like a lovely added grandpa for us ehehehehehe

[StainlessS]

I've had enough of you sychophants, now be quiet. :)

Sick-o-phants
https://www.cosgan.de/images/smilie/ekelig/n0203.gif https://www.cosgan.de/images/smilie/tiere/k015.gif

EDIT To Below, : And MXM 2U2 LigH.

https://www.cosgan.de/images/smilie/xmas/z700.gif

where is HanFrunz, bit late again.

[LigH]

MXM 2U2, Tinman.

[FranceBB]

For those who didn't get the GitHub notification, asd-g posted a new version here:

https://github.com/HomeOfAviSynthPlusEvolution/L-SMASH-Works/releases

Changelog:
LSMASHAudioSource: fixed lsmash wrong endian box. (#26)
FFmpeg updated to e0cb89c.
l-smash updated to 642c4cd.

[LeXXuz]

Anyone else still having problems with special characters in the file path/name like German umlauts?

[nechto]

I have issues with Ru symbols ...

[kedautinh12]

Anyone else still having problems with special characters in the file path/name like German umlauts?

Me too with Vietnamese alphabet :D

[guest]

@ LeXXuz, nechto & kedautinh12...

check this out:-

https://github.com/HomeOfAviSynthPlusEvolution/L-SMASH-Works/issues/31

and

https://superuser.com/questions/1715715/can-i-enable-unicode-utf-8-worldwide-support-in-windows-11-but-set-another-enco

This may "fix" the special characters problems...

But I'm having decoder problems.

UPDATE:- Setting the UTF-8 didn't fix my decoding problem, BUT, either of the "test" builds DID :). So now "we" just need a "fixed" L-Smash release. :)

[LeXXuz]

:thanks: I'll give that a spin. :)

[guest]

There's a new build :)

https://github.com/HomeOfAviSynthPlusEvolution/L-SMASH-Works/releases/tag/20230507

[FranceBB]

Yep, I've got the notification from GitHub yesterday evening but I was watching the Coronation Concert on the BBC (the lights display with the drones was stunning as always) and I didn't feel like booting Windows ehehehehehe
Anyway, as far as Italian characters are concerned, looks like the UTF-8 issue is fixed.

https://i.imgur.com/tKqb9tR.png

I'll leave you guys testing on other esoteric languages :P