Any way to cache indexing in x264?

[benwaggoner]

So, I'm doing a bunch of multibitrate encoding tests.

x264 can spend a remarkable amount of time indexing sources files (say, ProRes HQ of a feature-length title). For low bitrates from HD sources, the indexing can take nearly as much time as the encoding itself!

Is there any way to cache the index from one x264 run to another? For example, does x264 support opening index files creating in another tool?

[Guest]

Can't you do the indexing once with an appropriate source filter and then feed the frames via an Avisynth script?

[mandarinka]

If you use x264 CLI and ffms2 input/demuxer, then yes, using --index "path\file" option, IIRC.

[nhakobian]

ffms2 also has a standalone indexing tool. From what I remember, if it (ffms2) finds output from this program, it uses it by default. I think the index file needs to have the same name as the file with the extention .ffindex.

If you're using avisynth, the ffms2 library also exposes the FFIndex() function which triggers an indexing.

I found some more information here: http://ffmpegsource.googlecode.com/svn/trunk/doc/ffms2-avisynth.html

[Guest]

I was referring for example to using DGDecNV: Make a project with DGIndexNV (does the indexing), then serve this simple Avisynth script into x264:

DGSource("file.dgi")

The indexing is done only once (when the project is saved in DGIndexNV) no matter how many times the script is served into x264.

[kabelbrand]

If you use x264 CLI and ffms2 input/demuxer, then yes, using --index "path\file" option, IIRC.

Either this or you might want to use ffmpeg/libx264 instead which does not require indexing for ProRes, MPEG2, ...

[benwaggoner]

Either this or you might want to use ffmpeg/libx264 instead which does not require indexing for ProRes, MPEG2, ...
Yeah, in general. But when I'm tweaking lots of tweaky little one-off things for tuning, it's easier to use straight-up x264.exe rather than having to do the weird ffmpeg parameter passthrough thing.

[sneaker_ger]

If it works for your files you could use --demuxer lavf which does not index the files. Experiences may vary.

[Blue_MiSfit]

Yeah, either use --demuxer lavf or use --index foo.idx to keep the index between encodes.

Other than that, you're looking at AviSynth input, which is either going to require QTInput or a DirectShow <-> QuickTime wrapper. I use MediaLooks for precisely this use case. I handle MPEG-2 with MainConcept's splitter and decoder. Indexing is a pain in the butt, I totally agree :)

[benwaggoner]

Yeah, either use --demuxer lavf or use --index foo.idx to keep the index between encodes.
I'll try the --demuxer lavf thing. Are there formats it is known to work with and/or not work with?

Other than that, you're looking at AviSynth input, which is either going to require QTInput or a DirectShow <-> QuickTime wrapper. I use MediaLooks for precisely this use case. I handle MPEG-2 with MainConcept's splitter and decoder. Indexing is a pain in the butt, I totally agree :)
Ugh, using the ancient, slow 32-bit QuickTime decoder for Windows would probably slow things down more than indexing does. At least I get a nice 64-bit ProRes decoder from the command line.

The ideal way to handle this would be to have command line parameters that would let me save the generated index and then reload it for later encodes from the same source. All that lovely index data is already in RAM somewhere, just waiting to get packed into a nice little file somewhere.

Feature request!

[the_weirdo]

Feature request!

x264 already has this feature. Seem like you missed this:
If you use x264 CLI and ffms2 input/demuxer, then yes, using --index "path\file" option, IIRC.

[sneaker_ger]

Ugh, using the ancient, slow 32-bit QuickTime decoder for Windows would probably slow things down more than indexing does. At least I get a nice 64-bit ProRes decoder from the command line.

There's also L-SMASH-Source for AviSynth and VapourSynth which does not need to index mp4/mov/ProRes files because it uses the container's index.

[benwaggoner]

I'll try the --demuxer lavf thing.
In my testing with LAGS AVI files, --demuxer lavf worked perfectly on my ProRes .mov, LAGS .avi, and MPEG-2 .mpg source files. Thanks for that tip!

Are there any downsides to using this? If so, why isn't it the default?

[Blue_MiSfit]

I recall that ffms2 was the first compressed demuxer in x264, and lavf was introduced later.

ffms2 is of course based in libavcodec, but heavily modified. I think it is considered to be more reliable, particularly with regard to frame accuracy when seeking (given the indexing process), but lavf might be just as good these days. I really don't know. I also recall there being a lot of changes made for handling of MPEG-TS.

Maybe an expert will comment :)

Generally, I try lavf first. If that doesn't work, I'll either use ffms2 or AviSynth depending on requirements.

[MasterNobody]

I recall that ffms2 was the first compressed demuxer in x264, and lavf was introduced later.
Wrong. They where added at the same time and in one patch (http://git.videolan.org/gitweb.cgi?p=x264.git;a=commit;h=8b9dcd8d50be201f1bedc9b19331432969f37d98).
Also a preference of ffms2 over lavf was commented in patch itself. At least at the time of adding input support ffms2 was more reliable than lavf for input (at least for some formats like m2ts iirc).

[Blue_MiSfit]

I stand corrected! I think I could benefit from some ECC in my brain :)