L-SMASH Source

[VFR maniac]

The for-loop does nothing if no AVX2 support since width32 is set to 0 i.e. _width32 is set to 0.
And, if the address of the source buffer is non-mod32, _width32 is also set to 0 and the loop also does nothing in this case.

Edit: the issue has nothing to do with rev715.
I also replicate the issue by using his build.
Probably it is a bug of linked ffmpeg.

Edit2: He rebuild and reupload. Try again.

[Zathor]

Thanks for your work!

There seems to be a memory problem with all builds which I have tried (702 to 715). I am using it from within AviSynth+ (r1576) - but have tried also other AviSynth versions. The script will be called from within MeGUI several times (auto crop, deinterlace detection, encoding, ...) and after a while a script cannot be opened anymore with the error "[Fatal]: Failed to avformat_open_input." It can then be noticed that the memory consumption of the MeGUI process is high (up to 2GB) and it will not be freed when the script is closed. I have to close MeGUI and then the last operation can be continued. Also other users have the same problem: http://forum.doom9.org/showthread.php?p=1676500#post1676500

I have to try it with other programs as well (e.g. VirtualDub), but with other source filters I do not have the problem (ffms, dgindex, dgindexnv, ...) - only exception is the c-plugin of ffms which also seems to have a memory leak.
EDIT: Just tried it with VirtualDub. I opened and closed a simple script
LWLibavVideoSource("C:\test.mkv.lwi")
pointing to test.mkv - lwi size is ~50 MB. During every open/close cycle the memory increase and will not be freed completly.
I made those tests with r715 v2.

[Music Fan]

Edit: the updated r715 works. Thanks.
Is it still this link (given above) or another ;
https://docs.google.com/file/d/0BwV03nn6LPd9Y21BSHJUblVQRHc/edit?usp\u003ddrive_web&pli=1

[wOxxOm]

Is it still this link (given above) or anotherNo, the new one has _v2 in its name, just use the folder link and you'll see the updated file there, click it, then download.

[Music Fan]

Thanks, it's there but I can't download it, nothing happens when I click on the link :confused: ;
https://drive.google.com/folderview?id=0BwV03nn6LPd9OXJUWVVMMXZmNUU&usp=sharing#list
same problem when displayed like this (logical) ;
https://drive.google.com/folderview?id=0BwV03nn6LPd9OXJUWVVMMXZmNUU&usp=sharing

[Reel.Deel]

This link should work - https://docs.google.com/file/d/0BwV03nn6LPd9M1YycUVka29xVEU/edit

[filler56789]

Thanks, it's there but I can't download it, nothing happens when I click on the link :confused:

It seems that, since "some time ago", you must be logged in to your GMail or Google account, in order to to be allowed to download the files :mad: :(

[Music Fan]

This link should work - https://docs.google.com/file/d/0BwV03nn6LPd9M1YycUVka29xVEU/edit
Thanks, it does.
How do you find this link ?

[Music Fan]

It seems that, since "some time ago", you must be logged in to your GMail or Google account, in order to to be allowed to download the files :mad: :(
Mmmh, I have neither the one nor the other.:o

[yipeskop]

I think I have a bug to report. Upon trying to inverse telecine some ads, I realized I was getting weird artifacting. Upon inspection with a script of just LWLibavVideoSource("video").SeparateFields(), I found that the field order became completely messed up part way through the video. It was fine in the first half or so, but then started messing up some point after. Trying the script with both AssumeTFF() and AssumeBFF() didn't work, and trying LWLibavVideoSource with a dominance of both 1 and 2 didn't work. I've tried it on some other telecined material and it has the same issue. I'm using r715, by the way. I've also had this issue confirmed by a friend of mine who was using r714.

Here is one of the files in question (https://dl.dropboxusercontent.com/u/42171557/00011.m2ts) that it was having trouble displaying correctly. This video is a mix of progressive and telecined material. It starts having the field ordering issues once it reaches the Arata Kangatari ad around 1:20. Although do note that this issue also happened with a video that was a mix of pure interlace and telecine and also a video that was just purely telecine. The one purely interlaced video I tried it with worked perfectly fine (aside from the script causing mpc to crash near the end, but the video encoded fine with x264 at least)

edit: a correction about the sample video I gave in this post. it's a mix of pure interlaced and telecined. Not sure why I said it was progressive.

[wOxxOm]

you must be logged in to your GMail or Google account, in order to to be allowed to download the filesIt's fixed now, you can download anonymously from Drive.

the field order became completely messed up part way through the videoWell, there are *many* different videos actually, so different field order is completely expected.

[yipeskop]

Well, there are *many* different videos actually, so different field order is completely expected.

No, the whole file is TFF, as a simple avisynth script of FFVideoSource().AssumeTFF().SeparateFields() will show. Both FFVideoSource and DirectShowSource handle the videos I mentioned just fine. Only LWLibavVideoSource is giving wrong output, which is why I said it's probably a bug.

And in any case, I also said it was having issues with a purely telecined source anyway. So "*many* different videos" has nothing to do with it. I'll put the 750MB purely telecined ad in this comment as soon as it's done uploading if you or anyone wants to check it.

edit: And here's the telecine sample video (https://dl.dropboxusercontent.com/u/42171557/00010.m2ts)

[wOxxOm]

yipeskop, I think LWLibavVideoSource gets it wrong from the very beginning by dropping one field. Here's what appears to be a fix: LWLibavVideoSource("00011.m2ts").SeparateFields().DuplicateFrame(0).Weave()And the same goes for 0010.m2ts
So yeah there's a bug obviously.

[VFR maniac]

I think I have a bug to report. Upon trying to inverse telecine some ads, I realized I was getting weird artifacting. Upon inspection with a script of just LWLibavVideoSource("video").SeparateFields(), I found that the field order became completely messed up part way through the video. It was fine in the first half or so, but then started messing up some point after. Trying the script with both AssumeTFF() and AssumeBFF() didn't work, and trying LWLibavVideoSource with a dominance of both 1 and 2 didn't work. I've tried it on some other telecined material and it has the same issue. I'm using r715, by the way. I've also had this issue confirmed by a friend of mine who was using r714.

Here is one of the files in question (https://dl.dropboxusercontent.com/u/42171557/00011.m2ts) that it was having trouble displaying correctly. This video is a mix of progressive and telecined material. It starts having the field ordering issues once it reaches the Arata Kangatari ad around 1:20. Although do note that this issue also happened with a video that was a mix of pure interlace and telecine and also a video that was just purely telecine. The one purely interlaced video I tried it with worked perfectly fine (aside from the script causing mpc to crash near the end, but the video encoded fine with x264 at least)

edit: a correction about the sample video I gave in this post. it's a mix of pure interlaced and telecined. Not sure why I said it was progressive.

The file changes its field order in material level on the way of the stream but indicates TFF on the whole of the stream.
The stream is encoded simply as TFF MBAFF 60i.
So, the proper behaviour of the decoder is just returning a frame by frame.
I think L-SMASH Works is working properly.

No, the whole file is TFF, as a simple avisynth script of FFVideoSource().AssumeTFF().SeparateFields() will show. Both FFVideoSource and DirectShowSource handle the videos I mentioned just fine. Only LWLibavVideoSource is giving wrong output, which is why I said it's probably a bug.


I can't get why FFVideoSource and DirectShowSource handle it correctly(?).

Edit: note that L-SMASH Works doesn't seek by PTS but frame or picture number unlike FFVideoSource and DirectShowSource.

[VFR maniac]

I think I have a bug to report. Upon trying to inverse telecine some ads, I realized I was getting weird artifacting. Upon inspection with a script of just LWLibavVideoSource("video").SeparateFields(), I found that the field order became completely messed up part way through the video. It was fine in the first half or so, but then started messing up some point after. Trying the script with both AssumeTFF() and AssumeBFF() didn't work, and trying LWLibavVideoSource with a dominance of both 1 and 2 didn't work. I've tried it on some other telecined material and it has the same issue. I'm using r715, by the way. I've also had this issue confirmed by a friend of mine who was using r714.

Here is one of the files in question (https://dl.dropboxusercontent.com/u/42171557/00011.m2ts) that it was having trouble displaying correctly. This video is a mix of progressive and telecined material. It starts having the field ordering issues once it reaches the Arata Kangatari ad around 1:20. Although do note that this issue also happened with a video that was a mix of pure interlace and telecine and also a video that was just purely telecine. The one purely interlaced video I tried it with worked perfectly fine (aside from the script causing mpc to crash near the end, but the video encoded fine with x264 at least)

edit: a correction about the sample video I gave in this post. it's a mix of pure interlaced and telecined. Not sure why I said it was progressive.

Sorry, this is a bug of workaround for auto construction of a frame from a pair of field coded picture.
The cause comes from that the first some pictures have repeat=0 i.e. are signaled as field coded since the stream starts at non-IDR picture and libavcodec parser misdetects picture parameters.

Edit: Fixed at rev717: https://github.com/VFR-maniac/L-SMASH-Works/commit/0c4d9dfec0363096fc4ca5db09a9c62f9ad55499

[yipeskop]

The repository in the OP that hosts the binary builds of lsmash-works hasn't seemed to have updated since r715, and I was rather looking forward to trying out the newest version to see if it actually fixed the bug I was experiencing.

Could someone upload a build of the latest version please? I'd compile it myself, but I don't really know what I'm doing.

[the_weirdo]

Could someone upload a build of the latest version please?

You can try this build (https://dl.dropboxusercontent.com/u/18695757/L-SMASH-Works-r717-32bit.7z). 32-bit only and you need to install MSVC++ 2013 Redist (x86) (http://go.microsoft.com/?linkid=9832156) if you don't have it installed already.

[yipeskop]

Hey thanks, both of you!

The bug seems to have been mostly fixed. All the test files that were giving me trouble before are now working just fine.

There's just one small issue left. Using LWLibavVideoSource on these files without specifying the field dominance results in the field dominance becoming incorrectly reversed part way through the video. You can see this at 1:22 in the first example video I posted earlier and at 1:14 in the second example video I posted earlier. For reference, this issue does not happen with FFVideoSource.

The script I used to verify this issue was again just simply
LWLibavVideoSource("video").SeparateFields()

This is very easily worked around, though, by simply using AssumeTFF() or AssumeBFF() as needed, forcing the whole input to be read as TFF or BFF. Changing the values for the dominance argument for LWLibavVideoSource didn't seem to effect those two videos one way or another, however.

Again, thank you though. With LWLibavVideoSource, I now actually have something that can handle m2ts files without any annoying issues, like FFVideoSource has.

[burfadel]

You can try this build (https://dl.dropboxusercontent.com/u/18695757/L-SMASH-Works-r717-32bit.7z). 32-bit only and you need to install MSVC++ 2013 Redist (x86) (http://go.microsoft.com/?linkid=9832156) if you don't have it installed already.

Did you build that yourself?

Maybe a run through on how to compile it would help some people out :).

Even better would be someone new to upload optimised compiles to somewhere. That r717 build seems a little quicker than those that were on the google repository.

[the_weirdo]

Did you build that yourself?

Maybe a run through on how to compile it would help some people out :).

Even better would be someone new to upload optimised compiles to somewhere. That r717 build seems a little quicker than those that were on the google repository.

Yes, I built that myself. However, I didn't use any special config for that build. Don't know why there's difference in performance (cann't reproduce on my computer, though), as I believe his builds and mine might be using the (almost) same build environment.

[burfadel]

Ah ok, that's interesting then!

I would build it myself and make some optimised builds and upload it somewhere (and keep them updated), but I tried doing that a couple of months ago and couldn't get it to work. Can you explain how you compiled it so I can work out where I am going wrong?

Thanks!

[Reel.Deel]

Downloads from https://drive.google.com/folderview?id=0BwV03nn6LPd9OXJUWVVMMXZmNUU&usp=sharing seem to be gone. :(

-----

@the_weirdo

Thanks for r717.

[Taurus]

Downloads from https://drive.google.com/folderview?id=0BwV03nn6LPd9OXJUWVVMMXZmNUU&usp=sharing seem to be gone. :(


It is working here...
Maybe a server hickup? :p

[filler56789]

A couple of hours ago, there was no L-Smash Works 717 archive in that folder :confused:
Anyway, Google keeps sucking too much at file sharing :mad: :p

[LigH]

So far I never had problems with MediaFire yet...

[CarlPig]

r720 is available on Google Drive, check link from post #177

[burfadel]

r720 is available on Google Drive, check link from post #177

It is now. It wasn't updated for a little while, and R717 was missed, and there were no downloads there for a few days.

It's good that it's up though! Ready to use, properly done binaries are important for projects such as this, as not everyone wants to build. They would more likely go to an alternative, even if it isn't as good if they had to.

[siella]

LWLibavAudioSource
This function uses libavcodec as audio decoder and libavformat as demuxer

LSMASHAudioSource
This function uses libavcodec as video decoder and L-SMASH as demuxer

is the main difference only demuxer way?

[LigH]

As far as I know, yes; L-SMASH demuxer is specifically for the MP4 container. In general, the libavformat demultiplexer supports MP4 too, L-SMASH is probably only required for very special issues.

P.S.: Both use audio decoders.

[LigH]

But the disadvantage that it supports only MP4 containers, correct?

[AMED]

Thanks for your work!

There seems to be a memory problem with all builds which I have tried (702 to 715). I am using it from within AviSynth+ (r1576) - but have tried also other AviSynth versions. The script will be called from within MeGUI several times (auto crop, deinterlace detection, encoding, ...) and after a while a script cannot be opened anymore with the error "[Fatal]: Failed to avformat_open_input." It can then be noticed that the memory consumption of the MeGUI process is high (up to 2GB) and it will not be freed when the script is closed. I have to close MeGUI and then the last operation can be continued. Also other users have the same problem: http://forum.doom9.org/showthread.php?p=1676500#post1676500

I have to try it with other programs as well (e.g. VirtualDub), but with other source filters I do not have the problem (ffms, dgindex, dgindexnv, ...) - only exception is the c-plugin of ffms which also seems to have a memory leak.
EDIT: Just tried it with VirtualDub. I opened and closed a simple script
LWLibavVideoSource("C:\test.mkv.lwi")
pointing to test.mkv - lwi size is ~50 MB. During every open/close cycle the memory increase and will not be freed completly.
I made those tests with r715 v2.
Has there been any luck with this problem reported by Zathor?

I have tested R720 and can see that it's still not releasing memory.

[Reel.Deel]

Has there been any luck with this problem reported by Zathor?

I have tested R720 and can see that it's still not releasing memory.

Not yet, the issue is is open. (https://github.com/VFR-maniac/L-SMASH-Works/issues/28)

[Groucho2004]

I have tested R720 and can see that it's still not releasing memory.
I have tried to reproduce this issue but no matter how many times I re-load in VDub or Avsp, it releases the memory as it should.

AVS+ r1576 (32 Bit)
LSMASH r720v2
WinXP 32

[Zathor]

I use the Process Explorer to check the memory usage. And with every closing/opening cycle I see a steady increase in the memory usage (e.g. the peak values). Currently I am testing with a 1080p movie and with every cycle I get ~5MB more.

VirtualDub 1.9.11
LSMASH r720v2 (32bit)
AVS+ r1576 (32bit) and AVS 2.5.8 (plain, 32bit)
XP (32bit)

[Groucho2004]

I use the Process Explorer to check the memory usage. And with every closing/opening cycle I see a steady increase in the memory usage (e.g. the peak values). Currently I am testing with a 1080p movie and with every cycle I get ~5MB more.

VirtualDub 1.9.11
LSMASH r720v2 (32bit)
AVS+ r1576 (32bit) and AVS 2.5.8 (plain, 32bit)
XP (32bit)
Tried exactly that, still releases the memory properly. Maybe it's the splitter you're using. Try loading the raw stream into LWLibavVideoSource.

[Zathor]

Correct. With a raw avc stream there is no leak, but when muxed into MKV there is one.

[Zathor]

After a few limited tests it seems that r725 fixes the memory leak when using a container. I have to do more tests yet, but looks promising.

[AMED]

I'm still seeing the memory leak with r725 when using MKV. It is much better than previous versions though. I'm seeing about a 50MB increase with every bluray movie job when using One Click Mode in MeGUI build 2500.

[VFR maniac]

I fixed a memory leak of files with audio stream @ rev724 (https://github.com/VFR-maniac/L-SMASH-Works/commit/7c1dddbe3e3114e4b2582a303aa8c3fc13457930).
I cannot replicate any memory leak now, and it is difficult to debug unless any sample is supplied.

[AMED]

Here is a sample (https://www.mediafire.com/?dgaa17v2cztlzek).

[VFR maniac]

Here is a sample (https://www.mediafire.com/?dgaa17v2cztlzek).

I can't replicate the issue.
I think your memory leak is not derived from L-SMASH Works.

[AMED]

Thanks for taking a look. I shall look elsewhere for the cause of this problem.

[fvisagie]

Hi All,

When I open many files as in

...
AudioDub(LWLibavVideoSource("20140510154111.mts"), LWLibavAudioSource("20140510154111.mts")) ++ \
AudioDub(LWLibavVideoSource("20140510154135.mts"), LWLibavAudioSource("20140510154135.mts")) ++ \
...

either of LWLibavVideo/AudioSource() would at some point fail as follows

LWLibavAudioSource: failed to get the audio track.
(C:\Users\fvisagie\Videos\Home Videos\20140510 Collene & Kallie Pre-shoot\1. Footage.avs, line 38)


I'm using Avisynth 2.6.0 Alpha 4 with L-SMASH WORKS r725 on Windows 7 Pro SP1 32-bit. The machine has 4GB RAM and a quad-core i5-2540M.

Setting higher-than-default values with SetMemoryMax() makes no difference I could notice.

The failure happens anywhere from the 38th to the 41st input, but it always occurs when the Avisynth parent process gets to about 1666MB RAM usage as files are opened. At that point, total physical memory utilisation across all processes is only ~75%, meaning that not even physical memory has been exhausted, let alone virtual memory.

Another interesting observation: when I reduce the number of plugin threads, more files can be opened. The lower the number of threads, the less the amount of memory used per file and the more files that can be opened.

In that vein, keeping threads at e.g. 3 and adding more and more files to increase memory usage, once 1666MB is reached Avisynth still fails. In this case no LWLibavVideo/AudioSource() message is generated, but instead the host application would fail with some strange error. For example, Windows Media Player complains about the video card and AvsPmod lists a string of Python errors in its Error Window.

The bottom line is that whichever of threads or files I increase, once the usage of ~1666MB has been reached Avisynth fails.

Because not even physical memory has been exhausted by that point, Avisynth and/or the plugin hasn't run into a platform resource limitation; the problem must be something else.

Where does this problem lie, and what are the prospects for fixing it?

Many thanks,
Francois

[Groucho2004]

What do you use to open the script?

Also, the max. user process memory on a 32 Bit OS is 2Gb. Your 1666 MB are pretty close to that limit, no surprise there.

Reducing the number of threads to 1 for each call should enable you to open more files.

[fvisagie]

What do you use to open the script?

AvsPmod for editing, and Windows Media Player, VirtualDub, MPC-HC etc. for comparison.

Also, the max. user process memory on a 32 Bit OS is 2Gb. Your 1666 MB are pretty close to that limit, no surprise there.

Reducing the number of threads to 1 for each call should enable you to open more files.

I've tried that, thanks, but then it's ssslllooowww!!!!!


PS. Would moving to 64-bit Windows help (my machine has a 64-bit CPU), or would 32-bit AVS still be a limiting factor?

[Groucho2004]

I've tried that, thanks, but then it's ssslllooowww!!!!!Could you load more clips?

BTW, if you have access to a 64 Bit OS you could try Avisynth+ 64 Bit. It *might*(not sure) allow you to load more clips. But then you would need to use VirtualDub 64 Bit to open the script.

[SamKook]

Last time I reached 75% RAM utilisation in windows, it started to try and use the pagefile(which I had disabled since 75% = 12GB for me) and everything started to get unstable and crashing. Not sure why exactly it doesn't want to use the rest of the RAM whitout acting weird though.

If the software you use to load the avisynth script is large adress aware, you can get to close to 4GB before it will crash on a 64 bit OS instead of 2GB on a 32bit OS. If the software isn't(like virtualdub), there are application to switch the flag in the exe.

[qyot27]

/LARGEADDRESSAWARE might mitigate the issue a little on 32-bit, but good luck trying to get a hold of large address aware builds of everything in the toolchain.

Also, the /3GB flag would need to be set on the 32-bit OS to let it know it can go higher.

[LigH]

It is probably more useful to concatenate the MTS segments somehow before processing them (e.g. convert to a contiguous MKV); furthermore, L-SMASH Source supporting playlists may be useful, especially for Blu-ray sources.

But I am not certain if that would work reliably in your case, the segment filenames seem to be based on DVB recordings, not Blu-ray disks. If they contain different programmes (e.g. a different set of audio streams), concatenating may fail.

[fvisagie]

Thanks for the suggestions.

It is probably more useful to concatenate the MTS segments somehow before processing them

Barring one drawback, this works well for me, concatenating multiple (AVCHD) M2TS to M2TS with ffmpeg. The drawback is that while the camera records one scene per file for convenient editing, those scene transitions are lost in concatenation. Do you know of a way to record scene transitions when concatenating with ffmpeg, perhaps with the aid of shell scripting? I've tried all sorts of things but nothing so far looks even remotely promising.