L-SMASH Source

[qyot27]

Is ConvertToRGBP10() not an option at present?
IMO, there's way too many* new pixel formats added to AviSynth+ to add explicit bridging functions for all of them to the core. A better place for them - if you really feel like you need direct shortcuts, rather than using the more generic functions added to AviSynth+ to handle this - would be to break them out into an AVSI file like the colors_rgb list is. Otherwise it's a lot of nearly-identical code copy-pasted over and over again with only minor changes, for each destination format.

*you can see them all listed in the AviSynth+ headers (link directs only to the new set, just scroll up a little for the 2.5/2.6 formats):
https://github.com/AviSynth/AviSynthPlus/blob/MT/avs_core/include/avisynth_c.h#L146

That's 52 new pixel formats. And most of the internal bridges would just be invoking a combination of ConvertBits and ConvertTo[YUV***|PlanarRGB(A)], which like I said, is much easier to just break out into an AVSI:
function ConvertToRGBP10(clip c)
{
ConvertToPlanarRGB(c)
ConvertBits(10)
}
That is, of course, just an example and doesn't enable most of the underlying extra params from ConvertBits and ConvertToPlanarRGB, but those could be passed through with minimal effort, right there in the AVSI.

[Richard1485]

Oh, I see. I'm not one for writing procedures/functions, but that looks pretty simple. And if that's how they would be implemented internally, then I don't mind doing it the long way. Thanks!

[qyot27]

Obviously, the optimal solution would be for a generic master ConvertToPixelFormat() (or just plain old ConvertTo(), but I figured the main one I suggest should be more explicit about what it does) function that can convert between YUV(A), Y, Planar/Packed RGB(A) and change bitdepth at the same time, either by the user selecting the bitdepth separately or by taking it from the dest format name. A master function like that wouldn't massively reduplicate code and, IMO, would be fine to include in the core*. For instance:
input()
ConvertToPixelFormat("YUV420P16", extra_opts)
or
input()
ConvertToPixelFormat("YUV420", bit_depth=16, extra_opts)
Or you know, the same function as-written could do either of those, since 'YUV420P16' and 'YUV420' are both valid format names; the difference is just whether you'd have to declare any of the extra_opts explicitly because there's also a dedicated bit_depth= parameter.

*and before anyone hits me with the 'AviSynth+ doesn't do implicit conversions' thing, filters shouldn't silently change the input format to do the processing. A master ConvertTo function is always an explicit conversion, so that criticism doesn't apply.

[Richard1485]

That solution does sound feasible. Even if it's never included in the core, perhaps someone will write an .avsi for it all at some point.

[pinterf]

Then there is the z.lib resizer plugin, see z_ConvertFormat as a swiss-army knife.
Color-space, primaries, bit-depth, size conversion - all-in-one.
https://forum.doom9.org/showthread.php?p=1784316

[Gser]

Yeah the newest L-smash does not work for VC-1 at all,at least not in avs+. Getting super corrupt frames with grey bits everywhere.

[LigH]

Decoding interlaced VC-1 was always a problem, not sure if it was ever completely fixed... can you provide samples? If it fails for "usual" progressive VC-1 as well, which worked with earlier versions, then there may be a regression in libavcodec?

[sneaker_ger]

Two samples:
https://drive.google.com/file/d/0BwxFVkl63-lERnhlY1pHeW5wbjQ/view (interlaced)
https://drive.google.com/file/d/0BwxFVkl63-lETFVpOWRVVGNfeEU/view (progressive)
(via https://kodi.wiki/view/Samples )

No problem with ffmpeg git.

[videoh]

The files look fine as both decode without problems with DGDecNV and Avisynth+.

I don't understand the big problem with interlaced VC-1. There's nothing special about it.

[LigH]

May be in combination with the container, correct demultiplexing and joining the contained video streams from interleaving segments. FFMS2 used to have issues with interlaced AVC in TS (may have been due to its early dependency on Haali). "Nobody is perfect"... and I know nothing about the details, there are only vague memories.

Quick test: Both L-SMASH Works r929 (20170224) Static in MeGUI and r941 (20180530) Shared fail the same way, already with the progressive source (VC-1_23.976_sample.mkv). FFMS 2.23-1 Static in MeGUI and FFMS2 (bonus) Shared decode it correctly. — Known as L-SMASH-Works issue #58 (https://github.com/VFR-maniac/L-SMASH-Works/issues/58).

At least I hope that they got correctly assigned...

ClearAutoloadDirs()
LoadPlugin("E:\Programme\MeGUI\tools\lsmash\LSMASHSource.dll")
LwLibavVideoSource("VC-1_23.976_sample.mkv")
vs.
ClearAutoloadDirs()
LoadPlugin("E:\Programme\AviSynth+\plugins\LSMASHSource.dll")
LwLibavVideoSource("VC-1_23.976_sample.mkv")

Note: The FFMS2_2.23.1_MSVC_SharedLibs.7z contains 64 bit libav DLL's in the x86 subdirectory. The ones in LSMASHSource_r941_MSVC_SharedLibs_hydra3333.7z are correct for 32 bit.

[Gser]

May be in combination with the container, correct demultiplexing and joining the contained video streams from interleaving segments. FFMS2 used to have issues with interlaced AVC in TS (may have been due to its early dependency on Haali). "Nobody is perfect"... and I know nothing about the details, there are only vague memories.

Quick test: Both L-SMASH Works r929 (20170224) Static in MeGUI and r941 (20180530) Shared fail the same way, already with the progressive source (VC-1_23.976_sample.mkv). FFMS 2.23-1 Static in MeGUI and FFMS2 (bonus) Shared decode it correctly. — Known as L-SMASH-Works issue #58 (https://github.com/VFR-maniac/L-SMASH-Works/issues/58).


Just plain progressive. I remuxed it to m2ts and it played properly, then I reremuxed it to mkv and got the same problems again. So yeah their seems to be a problem with decoding MKV. If I remember correctly this happened even before the update to the new ffmpeg.

PS. Don't make the same mistake I did with updating FFMS, the package only includes the 32-bit version of ffmsindex

[l33tmeatwad]

this is a avs 2.5 API. can you update this to avs 2.6? greetings
Low priority on my todo list, but I'll try to get around to it.

[l33tmeatwad]

Patched LSMASHSource to use swresample instead of avresample since ffms2 now requires that and I like the ability to have these share the same libs without excess bloat. Needs to be tested, it can be found on GitHub (https://github.com/l33tmeatwad/L-SMASH-Works/releases). Would appreciate it if someone with more experience working with the FFmpeg libs would look over it...

[magiblot]

As the wiki (http://avisynth.nl/index.php/LSMASHSource/LWLibavVideoSource) suggests, I tried to set decoder="h264_qsv,mpeg2_qsv" hoping that I would experience a speed boost or a reduction in CPU usage (assuming 'qsv' stands for 'Intel QuickSync Video'), but I saw none.

In fact, it doesn't matter what I put into the decoder parameter, LWLibavVideoSource won't complain at all.

Am I not using this feature properly or is it not implemented yet?

Thanks.

[DJATOM]

It need to be compiled with mfx_dispatcher in order to make qsv decoding works. Also It's not frame accurate, or more precise notice "it might return green frames on seeking". But I still got ~400 fps with 4670k, so I'm using qsv for fast source preview.
I also managed to make cuvid decoder works, but it has issues with seeking as well.

[magiblot]

Oh. Since a index file is created, I was hoping accurate seeking would be possible. Is it still any better than DirectShowSource?

How do I compile the plugin with mfx_dispatcher?

Thanks.

[LigH]

I would assume that the index file is used for all frames only when L-SMASH Source decodes using the software decoder in libavcodec. But if you use a hardware decoder provided by your GPU (if that is implemented at all), then it will need to upload the video stream to be decoded into the decoder chip at least GOP-wise, and lose frame-accurate control partially.

[magiblot]

I have never written or even looked into a video decoder.

However, if I understand right the idea of GOP, and asuming that you seek to a frame in the middle of a GOP, couldn't the source plugin get the decoded GOP from the decoder and then discard frames until it gets to the sought frame?

Since the source video was indexed previously, I guess the source plugin knows where each GOP starts and in what position inside the GOP the requested frame is.

However, I also guess that if it was that simple it would have already been done.

[DJATOM]

Smart decoder might not discard frames behind decoded one in the GOP, as it might be requested upon further seeking. But issue with qsv is by Intel, as they broke something (Donald described this issue on his forum). Probably the only workaround is to check for decoded frame and request new on error.

[Selur]

A fresh build to support av1 would be great. :)
Thanks!

[Monamona]

VFR maniac,

Thank you for providing useful filters.

In the future, is there any chance to have an option to create and read the index file (.lwi) in any directory other than at the source file?

I want to read a file located in DVD or BD by LWLibavVideo(Audio)Source, without copying the file to HDD.

Thank you in advance.

[LigH]

@Monamona:

Reading from authored files on optical discs is not recommendable for at least 2 reasons:

a) authored content may be logically spread across several file segments, and L-SMASH Works cannot read from more than one source file, and surely not follow the logical path (e.g. ProGram Chain across VOB segments of a DVD Video, as defined in IFO files)

b) optical devices are stressed by random instead of sequential reading, and multi-threaded decoding may seek forth and back

To convert with AviSynth based tools, you should always extract a logical "movie" from the optical disc to a harddisk into one continuous file (e.g. MakeMKV can handle both DVD and Blu-ray Video media).

[redbtn]

If cache=0 i get vspipe error. This is L-Smash or VapourSynth problem?
[2019-02-15][02:29:44] "C:\Program Files (x86)\VapourSynth\core64\vspipe.exe" --info F:\MKV\input.vpy -
[2019-02-15][02:29:44]
[2019-02-15][02:29:44] [hevc @ 0000020ecf48bc20] missing picture in access unit
[2019-02-15][02:29:44] [hevc @ 0000020ecf48bc20] missing picture in access unit
[2019-02-15][02:29:44] [hevc @ 0000020ecf48bc20] missing picture in access unit
[2019-02-15][02:29:44] [hevc @ 0000020ecf48bc20] missing picture in access unit
[2019-02-15][02:29:44] [hevc @ 0000020ecf48bc20] missing picture in access unit
[2019-02-15][02:29:44] [hevc @ 0000020ecf48bc20] missing picture in access unit
[2019-02-15][02:29:44] [hevc @ 0000020ecf48bc20] missing picture in access unit

and more same strings

[Sparktank]

Thanks for this and FFMPEG updates.

[VS_Fan]

L-SMASH-Works-r935+3-win64-20190401.7z (https://www.mediafire.com/file/x9jh1an91n9bg31/L-SMASH-Works-r935+3-win64-20190401.7z/file)

Thanks, here my problem report:

Avisynth Error: LSMASHVideoSource [Fatal]: Failed to read an input file

vapoursynth.Error: lsmas: failed to open source file.

[VS_Fan]

Sorry, should have tested more before reporting: I thought there was no need for a script as it was just the source filter, I am testing my new batch of 12 recently found DV 199x recordings (AVI).

Your build is fine. I found that instead of LSMASHVideoSource and lsmas.LibavSMASHSource, both LWLibavVideoSource (AviSynth) & lsmas.LWLibavSource (VapourSynth) work just fine on these AVI files.

Thanks again

[LigH]

@VS_Fan: As documented: "LSMASH...Source" functions support only containers compatible to the ISO base media file format (https://en.wikipedia.org/wiki/ISO_base_media_file_format) (MOV, MP4, 3GPP and similar), but not AVI.

@HolyWu: Thank you for the new Win64 build. A pity I can't use it when I need to use legacy AviSynth 2.5 filters which are available for Win32 only.

[fvisagie]

@HolyWu: Thank you for the new Win64 build. A pity I can't use it when I need to use legacy AviSynth 2.5 filters which are available for Win32 only.

And I am on a Win32 *system*, so any update for that would be roundly welcomed! :-)

[stax76]

If I understand it correctly then l-smash for avs will by default output a double width format for 10bit (or more) sources. I can use ConvertFromDoubleWidth to get it into planar but how can I detect in avisynth code if it has double width format and which bit depth it is?

[stax76]

I can do in pseudo code:

if width / height > 3.5 then ConvertFromDoubleWidth

[stax76]

935 from the link above might have a memory leak problem.

[LigH]

If you want to use L-SMASH Works in AviSynth also for audio processing, you may want to prefer LwLibavAudioSource over LSMASHAudioSource even for ISO base media containers. I had an MP4 as source which contains AVC video and AAC audio (surprisingly in 44.1 kHz, possibly accidently resampled). Using LSMASHAudioSource, the audio contained annoying crackles; using LwLibavAudioSource, the sound was clean.

[stax76]

If you want to use L-SMASH Works in AviSynth also for audio processing

Audio processing with avs is not something staxrip does by default.

Which build are you using? Maybe have a look in task manager regarding memory consumption.

[LigH]

My report was not directed specifically to you, stax76, rather to the general audience. :p

I have a file date 2018-05-30. Probably LSMASHSource r941 MSVC hydra3333.

[stax76]

@LigH

staxrip was using 929, I've checked 941 memory usage and it's fine.

L-SMASH Source 941:

http://www.mediafire.com/file/id4dp2zqha08wvx/

[LigH]

I don't think the crackling sound of my issue is related to a memory leak, rather to the L-SMASH demultiplexer handling audio streams incorrectly (the libav demultiplexer instead works correctly).

[Forteen88]

I don't get it, which L-SMASH should I use? Should I use L-SMASH-Works-r935+3-win64-20190401 or L-SMASH Source 941, or l33tmeatwad's "SWResample Test"-version that was released 2018-Jun-20?

[stax76]

Removed parameter stacked. Now supports native high bit depth format.

Sounds awesome, thank you.

[stax76]

What I don't understand is, when I close the avifile API the memory is not instantly reclaimed entirely.

[MeteorRain]

I also have a copy at https://github.com/HomeOfAviSynthPlusEvolution/L-SMASH-Works basically I integrated patches from multiple forks.

Binaries compiled by myself can be found at https://down.7086.in/AviSynthPlus%20Filters/LSMASHSource-Release_r935%2B26.zip .

[Sparktank]

L-SMASH-Works-r935+25-win64-20190711 (https://www.mediafire.com/file/tjtjrwc63vjxi9e/L-SMASH-Works-r935+25-win64-20190711.7z/file)


Thanks for this update!

basically I integrated patches from multiple forks.

Can you tell us a little more about these patches?

[MeteorRain]

The same patch set HolyWu used, which was copied from enccc, and one more patch to use swresample instead of avresample, which was copied from l33tmeatwad.

[stax76]

:thanks:

[Atak_Snajpera]

Feature request:
Add option cachefile to LWLibavVideoSource so user can specify where index file will be created.
Example
LWLibavVideoSource("..\video.mkv",cachefile="C:\temp\MyIndex.lwi")

Currently LWLibavVideoSource is creating index file in the same folder as source video and thus creating an annoying mess.

BTW. Standalone indexer similar to ffmsindex.exe would be also useful.

[MeteorRain]

I recalled I found one on GitHub before, but it was not maintained for a few years.

[StainlessS]

To maybe avoid possible misunderstanding and unnecessary work,



ffms2_ORDERED_Function_List


There follows a list of all function names together with CPP style argument specifiers that inform
Avisynth the argument types and optional names. Optional arguments have square brackets surrounding
their name as in [name] and are followed by a type specifier character that gives the type.
Unnamed arguments are not optional. eg "cc[arg1]b[arg2]i" would be two compulsory unnamed clip args,
followed by optional 'arg1' of type bool and optional 'arg2' of type int.

# Argument type specifier strings.
c - Video Clip
i - Integer number
f - Float number
s - String
b - boolean
. - Any type (dot)
# Array Specifiers
i* - Integer Array, zero or more
i+ - Integer Array, one or more
.* - Any type Array, zero or more
.+ - Any type Array, one or more
# Etc
###################################


FFAudioSource "[source]s[track]i[cache]b[cachefile]s[adjustdelay]i[utf8]b[varprefix]s"
FFGetLogLevel ""
FFGetVersion ""
FFIndex "[source]s[cachefile]s[indexmask]i[dumpmask]i[audiofile]s[errorhandling]i[overwrite]b[utf8]b"
FFSetLogLevel "i"
FFVideoSource "[source]s[track]i[cache]b[cachefile]s[fpsnum]i[fpsden]i[threads]i[timecodes]s[seekmode]i[rffmode]i[width]i[height]i[resizer]s[colorspace]s[utf8]b[varprefix]s"

Perhaps it is not the standalone, but plugin function that is required. [Ffindex.Exe is not required to call ffindex() in script].

and just for good luck.


LSMASHSource_ORDERED_Function_List

LSMASHAudioSource "[source]s[track]i[skip_priming]b[layout]s[rate]i[decoder]s"
LSMASHVideoSource "[source]s[track]i[threads]i[seek_mode]i[seek_threshold]i[dr]b[fpsnum]i[fpsden]i[stacked]b[format]s[decoder]s"
LWLibavAudioSource "[source]s[stream_index]i[cache]b[av_sync]b[layout]s[rate]i[decoder]s"
LWLibavVideoSource "[source]s[stream_index]i[threads]i[cache]b[seek_mode]i[seek_threshold]i[dr]b[fpsnum]i[fpsden]i[repeat]b[dominance]i[stacked]b[format]s[decoder]s"

[MeteorRain]

I meant I found a standalone version of lsmash indexing tool on GitHub -- if that was causing misunderstanding.

Although I'm just using symbolic links to link the source into encoding directory, so no pollution to the sources.

[StainlessS]

Was intended just incase Atak_Snajpera thought ffmsindex.exe was required for indexing in script via ffmsindex(clip).
[I at one time thought that the executable was necessary for indexing].

[stax76]

cachefile feature would be very helpful and it would be awesome if the filter would write progress like 'Progress: 1%' to stdout.

Please note that any code can write to stdout, console tools can but GUIs and libraries like LSMASHSource.dll can do that as well.

An indexing app would be nice but using ffmpeg or vspipe will work as well:

Using proc As New Proc
proc.Header = "Index LWLibav"
proc.Encoding = Encoding.UTF8

If p.Script.Engine = ScriptEngine.AviSynth Then
proc.File = Package.ffmpeg.Path
proc.Arguments = "-i " + p.Script.Path.Escape + " -hide_banner"
Else
proc.File = Package.vspipe.Path
proc.Arguments = p.Script.Path.Escape + " NUL -i"
End If

proc.AllowedExitCodes = {0, 1}
proc.Start()
End Using

[manolito]

Another question about the different LSMASH versions...

A while ago Ligh recommended to use the LSMASH versions which come with the current MeGUI distribution. Maybe not the latest and greatest, but proven and well tested. I did that (pleasantly surprised about the included latest XP-compatible version), and the non-XP version from the MeGui download is r929.

I used this version without any problems, but now I see that there are some newer builds which also incorporate a couple of patches, and I am curious again... :)

There is r941 by hydra3333, and also interesting is r935 by MeteorRain. I only need the 32-bit versions, and they should work under classic AVS 2.60 as well as under the latest AVS+ version. And stability is my major concern.

I did some test conversions with r935 by MeteorRain under classic AVS 2.61 Alpha without any problems. What confused me is that the readme says that this is a plugin for AVS+ and VapourSynth, no mention of classic AVS. Can I safely ignore this for the 32-bit version, or am I playing with fire?

Cheers
manolito