FFmpegSource

[TheRyuu]

Originally Posted by sneaker_ger

Is there any reason to index a file manually beforehand in the first place? Despite buggy programs hanging up on loading the script due to the long wait.

If you want to see the progress I guess. I find it more useful on bigger files like lossless h264 or BD material where I don't want to blindly wait 5-10 minutes.

[sneaker_ger]

I see. Just wondered as to why people were doing it manually.

[Rumbah]

If I'm converting a Bluray to PSP or DivX avi I have to write an avs file and look through the subtitles. While I'm busy there I'm running the index process.

[henryho_hk]

It seems ffindex() from r259-x64 is re-indexing MKV files unnecessarily. I mux a H264 MKV from a M2TS using eac3to. The AVS script is simply:

ffindex("abc.mkv")
ffvideosource("abc.mkv")

If I open the AVS script in VirtualDub x86, the index file is created once and re-used when opened subsequently. But when opened in VirtualDub x64, the index file is overwritten every time. Does anyone have similar experience?

[kypec]

I have never ever used ffindex() in AVS script, what is the point of doing so?
Though I don't have deep knowledge how ffindex() inside Avisynth works I'd assume that what you are experiencing in VD x64 is probably correct behaviour in this case.
My assumption is based on the following:
ffvideosource() is clever enough to create (when used for non-indexed material yet) the index automatically and will re-use that index as long as the source didn't change.
Calling ffindex() explicitly OTOH just does what you are asking it to do = to index the source file over and over again.

If my understanding is wrong then feel free to correct me, thanks.

[TheFluff]

Originally Posted by henryho_hk

It seems ffindex() from r259-x64 is re-indexing MKV files unnecessarily. I mux a H264 MKV from a M2TS using eac3to. The AVS script is simply:

ffindex("abc.mkv")
ffvideosource("abc.mkv")

If I open the AVS script in VirtualDub x86, the index file is created once and re-used when opened subsequently. But when opened in VirtualDub x64, the index file is overwritten every time. Does anyone have similar experience?

r259? That's like a year and a half old, bro. Also, you can't load avs64 in x86 virtualdub, so that's a completely different FFMS2 compile you're using there. Note that index files are not compatible between x64 and x86 compiles, even if they're the exact same revision using the exact same ffmpeg.

Originally Posted by kypec

I have never ever used ffindex() in AVS script, what is the point of doing so?
Though I don't have deep knowledge how ffindex() inside Avisynth works I'd assume that what you are experiencing in VD x64 is probably correct behaviour in this case.
My assumption is based on the following:
ffvideosource() is clever enough to create (when used for non-indexed material yet) the index automatically and will re-use that index as long as the source didn't change.
Calling ffindex() explicitly OTOH just does what you are asking it to do = to index the source file over and over again.

If my understanding is wrong then feel free to correct me, thanks.

From the documentation:

FFIndex(string source, string cachefile = source + ".ffindex", int indexmask = -1,
    int dumpmask = 0, string audiofile = "%sourcefile%.%trackzn%.w64", int errorhandling = 3,
    bool overwrite = false, bool utf8 = false, string demuxer = "default")

Indexes a number of tracks in a given source file and writes the index file to disk, where it can be picked up and used by FFVideoSource or FFAudioSource. Normally you do not need to call this function manually; it's invoked automatically if necessary by FFVideoSource/FFAudioSource. It does, however, give you more control over how indexing is done and it can also dump audio tracks to WAVE64 files while indexing is in progress.

So yes, unless you have some particular reasons for overriding certain default indexing behaviors, there's generally no need to use FFIndex().

FFIndex() will not overwrite an existing index file if it is considered valid, unless you explicitly pass overwrite=true as a parameter. On the other hand, if it thinks the index file isn't valid it'll be overwritten.

FFVideoSource()'s automatic indexing behavior is subtly different. From the manual again:

string cachefile = source + ".ffindex"

The filename of the index file (where the indexing data is saved). Defaults to sourcefilename.ffindex. Note that if you didn't change this parameter from its default value and FFVideoSource encounters an index file that doesn't seem to match the file it's trying to open, it will automatically reindex and then overwrite the old index file. On the other hand, if you do change it, FFVideoSource will assume you have your reasons and throw an error instead if the index doesn't match the file.

[henryho_hk]

Sorry, it's r459, not r259. Silly typo I've done another test.... now it's r462 and the script is simply

ffvideosource("abc.avi") # a normal 500MB AVI

I keep opening the same AVS in VirtualDub 1.9.11. When I re-open (again and again) the AVS in VD x86, it loads instantaneously and the index file (file date) is not updated. Then when I try it in VD x64, it pauses for quite a while and the index file (file date) is updated every time. If I open the AVS script in VD x86 and x64 alternately, the index file is always updated. What I observed is:

1) x64 never re-uses the index file it produces.
1) Even both are r462, x86 and x64 do not share their index files.

[kemuri-_9]

Originally Posted by henryho_hk

Sorry, it's r459, not r259. Silly typo I've done another test.... now it's r462 and the script is simply

ffvideosource("abc.avi") # a normal 500MB AVI

I keep opening the same AVS in VirtualDub 1.9.11. When I re-open (again and again) the AVS in VD x86, it loads instantaneously and the index file (file date) is not updated. Then when I try it in VD x64, it pauses for quite a while and the index file (file date) is updated every time. If I open the AVS script in VD x86 and x64 alternately, the index file is always updated. What I observed is:

1) x64 never re-uses the index file it produces.

I am unable to reproduce this issue, the 64bit plugin is using the index cache and is not reindexing every time, as you state it is.
This was tested with both ffmpeg-mt and ffmpeg vanilla.

[TheRyuu]

I wasn't able to reproduce it either.

[henryho_hk]

Would it be related to the version of AVISynth x64, OS, codecs, or media splitters installed?

[burfadel]

The issues I had regarding running two encodes simultaneously with r459 and previous builds now seems to be resolved

Thanks!

[TheFluff]

Originally Posted by henryho_hk

1) Even both are r462, x86 and x64 do not share their index files.

I just posted this in the post before yours but since you apparently didn't read it I'll just say it again:

Index files are NOT COMPATIBLE between different FFMS binaries. Generally a given index file can ONLY be used with the exact binary it was generated with.

Originally Posted by henryho_hk

Would it be related to the version of AVISynth x64, OS, codecs, or media splitters installed?

No.

[TheRyuu]

Originally Posted by TheFluff

Index files are NOT COMPATIBLE between different FFMS binaries.

Additional checks now take place to better ensure this.
And it will recreate the index if such an incompatibility is found.

ffms2-mt-r464.7z
ffms2-mt-r464-avs64.7z

[burfadel]

Originally Posted by burfadel

The issues I had regarding running two encodes simultaneously with r459 and previous builds now seems to be resolved

Thanks!

The issue is still there. It depends on the source being encoded. For example, I just tried an old xvid encode with r462-mt and r464-mt, no luck! back to r460 non-MT for now...

[henryho_hk]

Installed r464 and cleaned up my plugin directory and the AVIS script. Now it works as expected. Thank you very much for your help. But it is a bit disappointed that x32 and x64 of the same revision write different index files.

BTW, I am also doing multi-process encodes. 4 parallel single/minimal-threaded pass-0 encodes are really faster (at the time being), especially on a SSD.

[burfadel]

Yeah doing multiple encodes is the way to go although it seems the MT version doesn't like some source files doing this (like xvid...). h.264 etc is fine.

[TheFluff]

Doing multiple encodes of different files in different script environments (i.e. different instances of the encoding program) at the same time cannot possibly affect anything. You're either getting an unrelated crash or you're just doing it wrong. You're not using that silly MT() filter or something, are you? (post your entire script)

[IanB]

Originally Posted by TheFluff

Doing multiple encodes of different files in different script environments (i.e. different instances of the encoding program) at the same time ...

All the processes running on the machine although independent from each other still share the operating resource of the machine, principally RAM and swap space.

Unfortunately both the Avisynth core and many plugins tend to handle memory allocation failures ungracefully, i.e. they crash and burn.

[TheFluff]

Originally Posted by IanB

All the processes running on the machine although independent from each other still share the operating resource of the machine, principally RAM and swap space.

Unfortunately both the Avisynth core and many plugins tend to handle memory allocation failures ungracefully, i.e. they crash and burn.

If he's managed to completely run out of both physical and virtual memory that in itself is sort of impressive (since he's using avs64 I'm assuming he has at least 4GB of physical RAM, plus at least as much swap), but it's also completely unrelated to FFMS2. Furthermore, it should be easy to detect that sort of problem just by staring at the task manager.

[LoRd_MuldeR]

Originally Posted by IanB

Unfortunately both the Avisynth core and many plugins tend to handle memory allocation failures ungracefully, i.e. they crash and burn.

If memory allocation fails, it either means the process has exceeded its maximum virtual memory size (which can easily happen with 32-Bit processes, but is completely unrelated from other processed running on the system) or the operating system really cannot allocate any additional memory pages (i.e. all the physical memory and all the swap-space on the HDD is completely full). The latter case indeed can be influenced by other processes, but its unlikely to happen under normal circumstances. And, if it does happen, it will cause serious trouble for all programs running on the computer. Windows would also pop up a fat warning message in that case...