FFmpegSource

[TheFluff]

fpsnum and fpsden are broken in 2.15. They have been fixed in SVN, and I'll probably do a 2.15.1 release to fix them, but until then you can either a) not use them, b) downgrade to 2.14, or c) find someone to compile a SVN version for you.

[lintran]

fpsnum and fpsden are broken in 2.15. They have been fixed in SVN, and I'll probably do a 2.15.1 release to fix them, but until then you can either a) not use them, b) downgrade to 2.14, or c) find someone to compile a SVN version for you.
Thank you. But do you mean fpsnum and fpsden are always broken in 2.15 or sometime or in some special cases?
Because it only broken with that file that i posted. I tried to test with many other h264 in.mov files (like Apple Trailers), everything are OK. So is it still safe to use?

[TheFluff]

Thank you. But do you mean fpsnum and fpsden are always broken in 2.15 or sometime or in some special cases?
Because it only broken with that file that i posted. I tried to test with many other h264 in.mov files (like Apple Trailers), everything are OK. So is it still safe to use?

By broken I mean "it may crash near the end of the video". If it doesn't crash, the output should be fine. See http://code.google.com/p/ffmpegsource/issues/detail?id=41

[henryho_hk]

Can I .....
1) enable "Always skip in-loop H.264 deblocking" as in ffdshow?
2) control the number of threads in the ffmpeg-mt version"?

And the 2.15 version seems to parse/decode VC-1 m2ts files incorrectly (BBC Life Bluray Disc 1's 00026.m2ts which is a clip about color tuning). When I jump to frame 1917, the ffmpeg-mt version (both 32 and 64 bit) shows a grey frame followed by a few corrupted frames... and the vanilla ffmpeg version shows a non-corrupted frame, but of a wrong frame number. If I demux the m2ts file to VC1, both mt and non-mt versions seem to work fine.

[TheFluff]

Can I .....
1) enable "Always skip in-loop H.264 deblocking" as in ffdshow?
2) control the number of threads in the ffmpeg-mt version"?

And the 2.15 version seems to parse/decode VC-1 m2ts files incorrectly (BBC Life Bluray Disc 1's 00026.m2ts which is a clip about color tuning). When I jump to frame 1917, the ffmpeg-mt version (both 32 and 64 bit) shows a grey frame followed by a few corrupted frames... and the vanilla ffmpeg version shows a non-corrupted frame, but of a wrong frame number. If I demux the m2ts file to VC1, both mt and non-mt versions seem to work fine.

1) Not at the moment. Noted as a feature request, though. (edit: why do you want to do this anyway?)
2) Yes, use the "threads" parameter to ffvideosource().

As for the m2ts stuff: do you have Haali's media splitter installed? The issue you describe sounds suspiciously like you're trying to seek in a ts with libavformat's demuxer (don't do that unless you've set seekmode=-1 first, or it'll end in tears). In general, ffms2's support for ts/m2ts is pretty poor; you'll get better results if you remux the file to mkv with eac3to or gdsmux first.

[henryho_hk]

1) coz some BD bitrates are so high that there is little need to do any deblock.... and I want to save the CPU for other stuffs.
2) thanks.

m2ts .... I have Haali's splitter installed and so dss2() works (too bad that they dun make any x64 version). I do expect issues from libavf's demuxer but not MT and non-MT versions behaving differently.

[TheFluff]

1) coz some BD bitrates are so high that there is little need to do any deblock.... and I want to save the CPU for other stuffs.
2) thanks.

m2ts .... I have Haali's splitter installed and so dss2() works (too bad that they dun make any x64 version). I do expect issues from libavf's demuxer but not MT and non-MT versions behaving differently.

If the bitrate is so high that there's no need to deblock the deblocking filter will actually not get used, so there's not really any significant amount of CPU power to save here.

The MT issues may be ffmpeg or it may be something else, it is a mystery. I don't build it myself so I won't debug it for you. If you upload a small sample where seeking isn't frameaccurate with the vanilla ffmpeg I'll take a look, though.

[TheRyuu]

ffms2-mt-r452.7z (http://ffmpegsource.googlecode.com/files/ffms2-mt-r452.7z)
ffms2-mt-r452-avs64.7z (http://ffmpegsource.googlecode.com/files/ffms2-mt-r452-avs64.7z)

There was a rather large merge a few days ago that brings ffmpeg-mt up to date (first one since 12/2010 I believe). And I guess ffmpegsource got a fix or two since 2.15.

Edit: Added avs64 build.

[zys4416]

Could anyone answer the #987 (http://forum.doom9.org/showpost.php?p=1462468&postcount=987) question? Because I get the same problem now. Thanks a lot!

[kemuri-_9]

FFMS2 does output I420 and not YV12.
megui sucks at handling this, use something else.

I could spit out techno babble here about how the difference between the two doesn't matter (which it doesn't for everything except the VFW interface),
but that won't change the issue that megui doesn't handle it well.

[zys4416]

I see. Would there by any problem if I ignore megui's warning, avoid adding CoverToYV12() to the end of the script and start encoding after that?
Thx again!

[kemuri-_9]

I see. Would there by any problem if I ignore megui's warning, avoid adding CoverToYV12() to the end of the script and start encoding after that?
Thx again!

It's been a long while since i looked at ConvertToYV12(), but last i remember it did nothing if the input was I420 or YV12.
this might've changed in 2.6, i'll have to check later...

x264cli will work for sure (As it uses avisynth directly rather than the vfw interface)

not sure about xvid's cli as i am not aware of how it reads .avs files.

and i don't recall what other video encoders megui uses these days, so can't say anything about any other possible ones.

[zys4416]

The encoder megui usually uses is Mencoder, at least for the version 0.3.4 i'm using.

[kemuri-_9]

The encoder megui usually uses is Mencoder, at least for the version 0.3.4 i'm using.

i don't recall megui ever using mencoder, so i grabbed a package off of sf.net (which ended up being newer than 0.3.5.30 as that's what's last listed before svn revisions are used instead) and mencoder is not there either.
i found x264cli and a fairly outdated xvid_encraw though.

there's notes in the changelog about
* 1895 [Snow] replaced mencoder with ffmpeg
* 1894 [pre-rendering job] replaced mencoder with ffmpeg for MeGUI x86
so that it seems at one point mencoder was actually used, but ffmpeg is now used instead of mencoder for pre-rendering and snow output.

anyway,
as i said before, x264cli uses avisynth directly and has no problems with I420 input.
xvid_encraw uses vfw, but had no issues with encoding from a ffms2 loaded file (I420) it seems, though i have avs 2.6 and i'm not sure if that makes a difference...
ffmpeg also uses vfw.
and I don't know about mencoder, but a hunch says it also uses vfw.

[Zathor]

Could anyone answer the #987 (http://forum.doom9.org/showpost.php?p=1462468&postcount=987) question? Because I get the same problem now. Thanks a lot!
Please update to a recent MeGUI build as this problem has been fixed a while ago. If even with a recent build (1989 or above) the problem appears please post you complete log in order to fix it. But please post it here: http://forum.doom9.org/showthread.php?t=151159

[zys4416]

Thank you all for your help, the problem is solved after i updated megui.
x264cli is good stuff, I usually use that directly on linux. However I do prefer gui on windows.

[TheFluff]

This same issue was discussed 30 pages and two years back, see http://forum.doom9.org/showthread.php?p=1283724#post1283724 for the dramatic conclusion. (tl;dr: what everyone calls YV12 is actually I420; MeGUI authors didn't know this and tried to ConvertToYV12() when this will in fact be a no-op since the video already is YV12; ancient versions of MeGUI would happily go into an infinite ConvertToYV12() loop. Use something that isn't MeGUI if you have to use a GUI. Or you could just update it, I guess, but I personally try to discourage people from using MeGUI.)

[burfadel]

Any chance of a non MT r459, utilising the latest ffmpeg?

The r459-MT works for me, but ONLY when doing a single encode. If I try to encode 2 different things at once the second one fails trying to write to protected memory (so Staxrip complains). At least I can have one instance running, which is better than r452-MT which didn't work at all.

The non MT version for me isn't really any slower, and in fact since I can't do two different encodes simulatenously with the MT version its actually noticeably slower overall!

To encode two things at once, I add them all the the queue list in staxrip, select every second item in the queue in one instance, and in another instance of staxrip select the opoosite every second item. The two lists don't interfere with each other which is good, however if you stop both of them and reload, only the last list will show so you have to resort them.

Anways, can you look into this? obviously the MT version is calling for the same address space in the second instance, which is not an issue with the non-MT one. Believe me, running two encodes simultaneously is the perfect choice for multithreading, regardless of house much you try and mutlithread the source filters and avisynth filters, its the ideal way to get 100 percent CPU usage. It probably gets both encodes done 15 percent quicker than doing each one individually?... (even with MT).

[burfadel]

On a separate note, I do appreciate FFmpegsource, its my favourite source filter :)

I was wondering whether mpeg2 support could be improved. It works fine, however it can be very slow when cutting adverts out of a 2 hr movie. Some parts of the mpeg2 support is really quick, but other parts have this slowdown. I really want to get off the mpeg2source filter. Not that I don't like the mpeg2source filter, but it is not updated anymore and the actual decoder it uses is ancient.

[TheFluff]

On a separate note, I do appreciate FFmpegsource, its my favourite source filter :)

I was wondering whether mpeg2 support could be improved. It works fine, however it can be very slow when cutting adverts out of a 2 hr movie. Some parts of the mpeg2 support is really quick, but other parts have this slowdown. I really want to get off the mpeg2source filter. Not that I don't like the mpeg2source filter, but it is not updated anymore and the actual decoder it uses is ancient.

What container are we looking at?

[TheFluff]

http://mod16.org/ffms2/ffms2-r460.7z

Compiled with libav from just now. Not tested so I have no idea if it actually works. There have been no significant user-visible changes since 2.15.

[burfadel]

What container are we looking at?

.m2v, as output from ProjectX 0.90.04.00.b32 from 31.10.2009 (from oozoon.de)

Thanks for the r460 as well, it seems to be working perfectly! The above slugginess issue is still there though when dealing with .m2v files. I tried a 720x576 30 min file and it was really quick (as expected), quicker than using mpeg2source (dginex). A 1440x1080 30 min recording was very fast, you can trigger it to be very sluggish or even seeming by it by moving the preview back and forth a little! It usually pauses when you first try and navigate through the file. At a completely uneducated guess, is it because its dealing with a completely unindex file? - the same reason why dgindex was used in the first place on these files, unlike sourcing from say, an mkv or .avi file?

The problem is frequent enough for me not to use it for .m2v files, but they're the only ones affected.

This is on Windows 7 SP1 kernel 7601.21701.amd64fre.win7sp1_ldr.110408-1634 (had it before with Windows 7 RTM though), Q9400 processor. This occurs on any opening of the .m2v files, regardless of whether there are other encodes running or not. I also tried it on another friends computer (similar specs though) and it had the same results.

[burfadel]

Just curious, any news on this in regards to multiple instances of ffms-mt?

I should point out I am using Avisynth 2.60 non-MT AviSynth_260_110418_MSVC2010.exe from http://xhmikosr.1f0.de/

Of course, I could always just use the Avisynth-MT and ffms-MT, but I can safely say that doing 2 encodes in parallel is much faster! (this is relative to filters used and current per-encode cpu use), and having it set to low priority means you don't even notice it running computer performance wise, even if the CPU is running at 100 percent. With more cores I can only imagine this being even more noticeable.

[TheRyuu]

Just curious, any news on this in regards to multiple instances of ffms-mt?

What happens when you manually set the threads=1 or half your cores for each script?

[burfadel]

It appears to be working now, I've encoded a fwe things (in parallel) and left it running. Strange!

I have also been torrenting, recording stuff off tv and watching some stuff, and everything is working like it should (like it usually does). I don't know why it wasn't working before and is now, its probably a case of 'in the scenario of having...', in other words something which only shows when certain conditions are met.

[burfadel]

Ok, been encoding all this time, and started it before my last post. Its only now that it caused a crash that it was trying to write to write protected memory.

It occurs when one instance of encoding (using x264) is being undertaken, and the second instance starts to encode. It worked quite well for several instances changes, so its a bit hard to track down...

[sumawo13]

Can someone explain to me why indexing only uses 1-2% CPU? It takes 10-15 minutes to index a Blu-ray video stream remuxed from M2TS to MKV, and then when I load my script in VirtualDub, it takes another 6 minutes for the script to open, all the while VirtualDub using about 1-2% CPU, it does this everytime I load the script, even if I specifiy the index file directly in the script.

[LoRd_MuldeR]

Can someone explain to me why indexing only uses 1-2% CPU? It takes 10-15 minutes to index a Blu-ray video stream remuxed from M2TS to MKV, and then when I load my script in VirtualDub, it takes another 6 minutes for the script to open, all the while VirtualDub using about 1-2% CPU, it does this everytime I load the script, even if I specifiy the index file directly in the script.

Because the indexing process obviously is I/O-limited and not CPU-limited. A faster CPU doesn't help, if the program is idle, waiting for the HDD, most of the time ;)

[TheRyuu]

Can someone explain to me why indexing only uses 1-2% CPU? It takes 10-15 minutes to index a Blu-ray video stream remuxed from M2TS to MKV, and then when I load my script in VirtualDub, it takes another 6 minutes for the script to open, all the while VirtualDub using about 1-2% CPU, it does this everytime I load the script, even if I specifiy the index file directly in the script.

It should not have to index the same file twice, make sure your ffmsindex is using the same ffms2.dll file (they come together in the 7z for a reason) that you're loading with your LoadPlugin() line, otherwise the versions won't match in the ffindex file and it'll remake the index anyway.

Some of the older versions had a bug that would have them reindex files even if the index file already existed (although I'm pretty sure it only affected m2ts or something like that) so you should be fine but update to the latest anyway.

[sneaker_ger]

Is there any reason to index a file manually beforehand in the first place? Despite buggy programs hanging up on loading the script due to the long wait.

[TheRyuu]

Is there any reason to index a file manually beforehand in the first place? Despite buggy programs hanging up on loading the script due to the long wait.

If you want to see the progress I guess. I find it more useful on bigger files like lossless h264 or BD material where I don't want to blindly wait 5-10 minutes.

[sneaker_ger]

I see. Just wondered as to why people were doing it manually.

[Rumbah]

If I'm converting a Bluray to PSP or DivX avi I have to write an avs file and look through the subtitles. While I'm busy there I'm running the index process.

[henryho_hk]

It seems ffindex() from r259-x64 is re-indexing MKV files unnecessarily. I mux a H264 MKV from a M2TS using eac3to. The AVS script is simply:

ffindex("abc.mkv")
ffvideosource("abc.mkv")

If I open the AVS script in VirtualDub x86, the index file is created once and re-used when opened subsequently. But when opened in VirtualDub x64, the index file is overwritten every time. Does anyone have similar experience?

[kypec]

I have never ever used ffindex() in AVS script, what is the point of doing so?
Though I don't have deep knowledge how ffindex() inside Avisynth works I'd assume that what you are experiencing in VD x64 is probably correct behaviour in this case.
My assumption is based on the following:
ffvideosource() is clever enough to create (when used for non-indexed material yet) the index automatically and will re-use that index as long as the source didn't change.
Calling ffindex() explicitly OTOH just does what you are asking it to do = to index the source file over and over again.

If my understanding is wrong then feel free to correct me, thanks.

[TheFluff]

It seems ffindex() from r259-x64 is re-indexing MKV files unnecessarily. I mux a H264 MKV from a M2TS using eac3to. The AVS script is simply:

ffindex("abc.mkv")
ffvideosource("abc.mkv")

If I open the AVS script in VirtualDub x86, the index file is created once and re-used when opened subsequently. But when opened in VirtualDub x64, the index file is overwritten every time. Does anyone have similar experience?

r259? That's like a year and a half old, bro. Also, you can't load avs64 in x86 virtualdub, so that's a completely different FFMS2 compile you're using there. Note that index files are not compatible between x64 and x86 compiles, even if they're the exact same revision using the exact same ffmpeg.

I have never ever used ffindex() in AVS script, what is the point of doing so?
Though I don't have deep knowledge how ffindex() inside Avisynth works I'd assume that what you are experiencing in VD x64 is probably correct behaviour in this case.
My assumption is based on the following:
ffvideosource() is clever enough to create (when used for non-indexed material yet) the index automatically and will re-use that index as long as the source didn't change.
Calling ffindex() explicitly OTOH just does what you are asking it to do = to index the source file over and over again.

If my understanding is wrong then feel free to correct me, thanks.

From the documentation:

FFIndex(string source, string cachefile = source + ".ffindex", int indexmask = -1,
int dumpmask = 0, string audiofile = "%sourcefile%.%trackzn%.w64", int errorhandling = 3,
bool overwrite = false, bool utf8 = false, string demuxer = "default")
Indexes a number of tracks in a given source file and writes the index file to disk, where it can be picked up and used by FFVideoSource or FFAudioSource. Normally you do not need to call this function manually; it's invoked automatically if necessary by FFVideoSource/FFAudioSource. It does, however, give you more control over how indexing is done and it can also dump audio tracks to WAVE64 files while indexing is in progress.


So yes, unless you have some particular reasons for overriding certain default indexing behaviors, there's generally no need to use FFIndex().

FFIndex() will not overwrite an existing index file if it is considered valid, unless you explicitly pass overwrite=true as a parameter. On the other hand, if it thinks the index file isn't valid it'll be overwritten.

FFVideoSource()'s automatic indexing behavior is subtly different. From the manual again:
string cachefile = source + ".ffindex"
The filename of the index file (where the indexing data is saved). Defaults to sourcefilename.ffindex. Note that if you didn't change this parameter from its default value and FFVideoSource encounters an index file that doesn't seem to match the file it's trying to open, it will automatically reindex and then overwrite the old index file. On the other hand, if you do change it, FFVideoSource will assume you have your reasons and throw an error instead if the index doesn't match the file.

[henryho_hk]

Sorry, it's r459, not r259. Silly typo :D I've done another test.... now it's r462 and the script is simply

ffvideosource("abc.avi") # a normal 500MB AVI

I keep opening the same AVS in VirtualDub 1.9.11. When I re-open (again and again) the AVS in VD x86, it loads instantaneously and the index file (file date) is not updated. Then when I try it in VD x64, it pauses for quite a while and the index file (file date) is updated every time. If I open the AVS script in VD x86 and x64 alternately, the index file is always updated. What I observed is:

1) x64 never re-uses the index file it produces.
1) Even both are r462, x86 and x64 do not share their index files.

[kemuri-_9]

Sorry, it's r459, not r259. Silly typo :D I've done another test.... now it's r462 and the script is simply

ffvideosource("abc.avi") # a normal 500MB AVI

I keep opening the same AVS in VirtualDub 1.9.11. When I re-open (again and again) the AVS in VD x86, it loads instantaneously and the index file (file date) is not updated. Then when I try it in VD x64, it pauses for quite a while and the index file (file date) is updated every time. If I open the AVS script in VD x86 and x64 alternately, the index file is always updated. What I observed is:

1) x64 never re-uses the index file it produces.

I am unable to reproduce this issue, the 64bit plugin is using the index cache and is not reindexing every time, as you state it is.
This was tested with both ffmpeg-mt and ffmpeg vanilla.

[TheRyuu]

I wasn't able to reproduce it either.

[henryho_hk]

Would it be related to the version of AVISynth x64, OS, codecs, or media splitters installed?

[burfadel]

The issues I had regarding running two encodes simultaneously with r459 and previous builds now seems to be resolved :)

Thanks!

[TheFluff]

1) Even both are r462, x86 and x64 do not share their index files.
I just posted this in the post before yours but since you apparently didn't read it I'll just say it again:

Index files are NOT COMPATIBLE between different FFMS binaries. Generally a given index file can ONLY be used with the exact binary it was generated with.

Would it be related to the version of AVISynth x64, OS, codecs, or media splitters installed?

No.

[TheRyuu]

Index files are NOT COMPATIBLE between different FFMS binaries.

Additional checks now take place to better ensure this.
And it will recreate the index if such an incompatibility is found.

ffms2-mt-r464.7z (http://ffmpegsource.googlecode.com/files/ffms2-mt-r464.7z)
ffms2-mt-r464-avs64.7z (http://ffmpegsource.googlecode.com/files/ffms2-mt-r464-avs64.7z)

[burfadel]

The issues I had regarding running two encodes simultaneously with r459 and previous builds now seems to be resolved :)

Thanks!

The issue is still there. It depends on the source being encoded. For example, I just tried an old xvid encode with r462-mt and r464-mt, no luck! back to r460 non-MT for now...

[henryho_hk]

Installed r464 and cleaned up my plugin directory and the AVIS script. Now it works as expected. Thank you very much for your help. But it is a bit disappointed that x32 and x64 of the same revision write different index files.

BTW, I am also doing multi-process encodes. 4 parallel single/minimal-threaded pass-0 encodes are really faster (at the time being), especially on a SSD.

[burfadel]

Yeah doing multiple encodes is the way to go :) although it seems the MT version doesn't like some source files doing this (like xvid...). h.264 etc is fine.

[TheFluff]

Doing multiple encodes of different files in different script environments (i.e. different instances of the encoding program) at the same time cannot possibly affect anything. You're either getting an unrelated crash or you're just doing it wrong. You're not using that silly MT() filter or something, are you? (post your entire script)

[IanB]

Doing multiple encodes of different files in different script environments (i.e. different instances of the encoding program) at the same time ...All the processes running on the machine although independent from each other still share the operating resource of the machine, principally RAM and swap space.

Unfortunately both the Avisynth core and many plugins tend to handle memory allocation failures ungracefully, i.e. they crash and burn.

[TheFluff]

All the processes running on the machine although independent from each other still share the operating resource of the machine, principally RAM and swap space.

Unfortunately both the Avisynth core and many plugins tend to handle memory allocation failures ungracefully, i.e. they crash and burn.

If he's managed to completely run out of both physical and virtual memory that in itself is sort of impressive (since he's using avs64 I'm assuming he has at least 4GB of physical RAM, plus at least as much swap), but it's also completely unrelated to FFMS2. Furthermore, it should be easy to detect that sort of problem just by staring at the task manager.

[LoRd_MuldeR]

Unfortunately both the Avisynth core and many plugins tend to handle memory allocation failures ungracefully, i.e. they crash and burn.

If memory allocation fails, it either means the process has exceeded its maximum virtual memory size (which can easily happen with 32-Bit processes, but is completely unrelated from other processed running on the system) or the operating system really cannot allocate any additional memory pages (i.e. all the physical memory and all the swap-space on the HDD is completely full). The latter case indeed can be influenced by other processes, but its unlikely to happen under normal circumstances. And, if it does happen, it will cause serious trouble for all programs running on the computer. Windows would also pop up a fat warning message in that case...