MpegAutoIndex | utility & function for loading MPEG2/MPEG4/VC1 + audio [2012/01/05]

[vampiredom]

# MPEGAutoIndex
# 2012/01/05 -> Fixed issue with DGIndex and paths with / instead of \
# 2012/01/04 -> Documentation correction
#               New version of DGIndexNV.exe and DGDecodeNV.dll
# 2011/12/30 -> Added CALL_25_Helper.exe to allow ' in filenames (and the possibility of other escape sequences)
#               Various cleanups
# 2011/11/02 -> AAC->WAV and FLAC->WAV is disabled by default, prefers BassAudioSource (see below)
#               Added global MpegAutoIndex_disable_WAV_conversion (default: true)
#               Included FLAC for optional WAV conversion
#               Added FLAC support and fallback for other demuxed audio formats with BassAudio
#               Defer loading of NicAudio and BassAudio
#               Included BassAudio
#               Checks for existence of license.txt files when NV=true or DI=true
# 2011/11/01 -> Updated to latest mp4box.exe
#               Added support for h264 .mkv files [NV=true is required]
#               Allows FFMS2 or FFMS as fallback for audio (with higher priority than DSS)
# 2011/10/31 -> Updated to latest NicAudio.dll, now prefers RaWavSource() to DSS for all .WAV files
#               Changed AAC downmix to 32-bit float
#               Removed unnecessary files from distribution
#               Changed DiAVC project extension from .dgi to .dgv to allow easier switching between NV and DI
#               Added support for VC1 streams in .wmv containers by demuxing via ASF2VC1.exe
#               Added Tebasuna51's downmixing scripts as fallbacks for when Downmix=true
# 2011/10/30 -> First public release

Download Latest Version Here

All requirements are included. It can also use FFMS or FFMS2 as fallbacks for audio (this can be useful for .mkv containers and some .mov files) but these are neither required nor included.

MpegAutoIndexSource()

This function loads MPEG2, MPEG4, and VC1 sources - and automatically indexes them (creating .dga or .d2v or .dgi 'project' files as needed). Indexing is skipped (by default) when the project file exists. [NOTE: VC1 functionality and interlaced MPEG4 sources require a license for Donald Graft's decoders -> http://neuron2.net]

The goal is to be a sort of "Swiss Army Knife" for MPEG2 and MPEG4 sources (with audio) – allowing users to get their video (with audio) into AviSynth quickly, reliably with the least amount of hassle. The distribution includes all required .exe and .dll files.

Audio tracks are demuxed and automatically dubbed via a generated and imported script snippet (e.g: myproject.d2v -> myproject.DemuxedAudioSource). The .DemuxedAudioSource file will reference all demuxed tracks. It will attempt to choose the "best" option when multiple tracks are found, but this can be changed by modifying the .DemuxedAudioSource file after it is generated.

You can also use Donald Graft's NV CUDA-enhanced indexer/decoder (set NV=true). If you have registered DG's tools, copy and paste your "license.txt" file into the "/plugins/MPEGAutoIndex/DGIndexNV/" folder. It also has the ability to use Donald Graft's DiAVC indexer/decoder (set DI=true]). Likewise, copy your "license.txt" into the "/plugins/MPEGAutoIndex/DGAVCIndexDi/" folder.

Note that although AVC/MPEG-4 video can be handled without an NV license (via DGAVCIndex), it is fairly limited and does not support interlaced video (among other formats). Therefore, using the NV option is recommended for AVCHD and other MPEG-4 formats.

MPEGAutoIndexSource also has options for downmixing, normalizing, and outputting audio separately to AC3 or WAV format, which can be useful for DVD authoring or when further audio processing is needed.

MPEGAutoIndexSource uses faad.exe for decoding AAC audio. It also incorporates mp4box for demuxing .mp4 and H.264 Quicktime .mov files as needed, allowing them to be indexed properly in most cases.

Installation:

1. Unzip the files into your AviSynth 2.5/plugins folder, so that MpegAutoIndex_loader.avsi is loaded automatically.
2. Copy your "license.txt" file for Donald Graft's tools into the "DGIndexNV" and "DGAvcIndexDi" subfolders

Updating to latest version from an existing installation:

1. Delete the plugins/MpegAutoIndex folder entirely and replace with it with the MpegAutoIndex folder in the .zip file
2. Copy your "license.txt" file for Donald Graft's tools into the "DGIndexNV" and "DGAvcIndexDi" subfolders

Usage:

# MPEGAutoIndexSource(filename [, options])

# Open an MPEG4 file
MpegAutoIndexSource("foo.mp4")

# Open an AVC stream using DGIndexNV and hardware double-rate deinterlacing
MpegAutoIndexSource("foo.mts", NV=true, args="deinterlacing=2")

# Open an AVC stream using DiAVC and hardware double-rate deinterlacing
MpegAutoIndexSource("foo.mts", DI=true, args="deinterlacing=2")

# Open an NTSC MPEG2 DVD stream using DGIndex, force it to 24P (film)
# Downmix the multichannel audio to stereo, Normalize the audio to 99%
MpegAutoIndexSource("VTS_01_1.VOB", FieldOperation=1, Downmix=true, PeakAudioLevel=0.99)

# Open an AVC stream from BluRay using DGIndexNV, forcing 24p
# Downmix the multichannel audio to stereo, normalize it to 100%
# Output the audio as a 256kbps AC3 file
MpegAutoIndexSource(
\     "12345.m2ts",
\     NV=true,
\     args="fieldop=1",
\     Downmix=true,
\     PeakAudioLevel=1.00,
\     AC3Out=true, AC3Bitrate=256
\ )

# Open an VC-1 .wmv file
MpegAutoIndexSource("vc1.wmv", NV=true)

# Open a .mkv container with h264 video
MpegAutoIndexSource("h264.mkv", NV=true)

Detailed information and parameters are listed at the top of the MpegAutoIndex/MpegAutoIndex.avsi file.

Note that this function tends to work rather well with MT() - at least in the "normal" and NV=true modes. There seem to be some issues and crashes with MT + DiAVC, however, but this has not been tested extensively.

# So, you should be able to:
SetMTMode(2)
MPEGAutoIndexSource("VTS_01_1.VOB")

# ... or:
SetMTMode(2)
MPEGAutoIndexSource("12345.m2ts", NV=true)

Supported extensions and input formats:

MPEG2 ... (.vob, .m2v, .mpv, .mpeg, .mpg, .m2t)
MPEG4 ... (.avc, .264, .h264, .m4v, .m2ts, .mts, .mov, .mkv)
VC1 ..... (.vc1, .wmv, .m2ts)

Audio Formats: PCM, AC3, AAC, DTS, MP3, MP2, MPA, FLAC [+ FFMS & DirectShow support as fallback]

NOTE: .mkv containers and VC1 video require setting NV=true (and a license for neuron2's tools)



Enjoy!

[vampiredom]

New version uploaded (that was fast, eh?). The main differences are support for VC1 .wmv files and some audio improvements. Refer to the Installation or Updating instructions above.

[looney]

Thanks. It looks alike great all-in-one SourceImport tool. But what about ultimately apexing mkv files support? Is there any chance that one of most popular containers (matroska) nowadays could be supported in this aio? I think it would save us enormous amount of space and time because demuxing some 15GB file still takes more than 4min on two different drives, just to chop out some scene for testing.

[vampiredom]

Thanks, looney! I really hope that people find it useful. A little background before I answer your question: I am a video editor and I wrote this because I get a bunch of source footage in different formats; some of it HDV (MPEG2), .mts (h264) ... other stuff, like Canon 5D footage, is h264 in an .mov container... sometimes the original footage is lost and now it is only on DVD or BluRay or .mp4 files. I wrote this function primarily for "professional" applications such as this (because I wrote it for myself).

That being said, I love for this to be as good as useful as possible for everybody – and I much appreciate feedback on how I can improve it, which is why I posted it here.

** EDIT **
I just posted a new version that allows .mkv files to be indexed. See my post below and follow the updating instructions in the top post.

[vampiredom]

Added support for h264 .mkv files. This requires the NV=true parameter, which requires a license from Donald Graft for his NV indexing tools [neuron2.net].

Follow the instructions for installing or updating in the top post of this thread.

Thanks for the suggestion, looney!

[vampiredom]

OK, I am now working on implementing FLAC audio capability as this seems to be fairly popular in Matroska files. Perhaps y'all could help on this.

Which would you prefer?

1. Using BassAudioSource() to load the FLAC audio
2. Pre-converting FLAC->WAV

Obviously, each would have its advantages: BassAudioSource() would load the clip faster, as there is no need to wait for the WAV conversion. Using WAV files, however, seems a bit more reliable and will quite possibly speed things up in the long run (using less CPU when decoding, etc.)

Thoughts?

[vampiredom]

In answer to my own post above, FLAC functionality has now been included via BassAudioSource(). Similarly, AAC decoding is now also handled by BassAudioSource() instead of converting to WAV via faad.exe.

FLAC->WAV and AAC->WAV is now disabled via the global parameter, MpegAutoIndex_disable_WAV_conversion, which is true by default.

To enable intermediary WAV file rendering for AAC and FLAC (and bypassing BassAudioSource) add this line to the top of your script:

global MpegAutoIndex_disable_WAV_conversion = false

The default setting should speed up loading of long h264 files with AAC and FLAC audio. Performance seems to be quite good and still runs fine with SetMTMode(2).

[vampiredom]

It is my goal to continue development of this as a way of providing the convenience of FFMS2 with the reliability of MPEG2Source() and DGSource().

There have been a lot of views on this thread and it looks like over 100 people have downloaded it. Not to nag or be impatient, but have any of you tried it out? Does it work as advertised? Are there and features you'd like to see added or modified? Are my instructions and documentation correct? Any issues?

This tool has already been enormously useful to me, for my need. I am very much looking forward to feedback on this so I can make it be all it can be for everybody else.

Thanks.

[smok3]

Looks very interesting, will try as soon a proper job (dvd input) is required.

[vampiredom]

Thanks! Remember, it is not only for DVD sources: I commonly use it for input from HDV and AVCHD camcorders. It also handles h264 Quicktime files rather well.

I have recently added support for Matroska and Windows Media containers (but these require a CUDA accelerated video card and a license for DGIndexNV).

[StainlessS]

Originally Posted by vampiredom

Not to nag or be impatient, but have any of you tried it out? Does it work as advertised?
Are there and features you'd like to see added or modified?
Are my instructions and documentation correct? Any issues?
Thanks.

Hi vampiredom,

Have not as yet tried your new plug, although I will at some stage.
The guys and gals (including me) that frequent D9 are a fickle bunch,
they might read your new fangled all singin all dancin lil ol treat,
but they ain quite so bothered bout tryin em out. Bit of a case of,
"That sounds nice, next".
Been there myself several times, dont worry, if your product is really good,
it will be business as usual this time next year, and you'll be just as miffed.
The advise is, give it some time, you coming out with updates everyday makes it
seem like it might be a good idea to wait until it's near stable, hence the
hesitancy.
Also, it dont look good to get yourself a strike for 'Spamming', we know you are eager
to have recognition, but best not as a spammer.
Perhaps some time your offering will be recognised as a valuable addition, and if not,
you can always console yourself with the fact that you have something better than most
people. (swines!).

Chill bro, have a good one.

StainlessS.

[Yellow_]

Fickle, yeah count me in. :-) Using FFMS2, so with regard to HDV and H264 AVC could you suggest the benefits over FFMS2 for decoding afore mentioned, not ungreatful, need enlightenment. :-)

[vampiredom]

Chill bro, have a good one.

Right on. I will, and thanks for the advice!

with regard to HDV and H264 AVC could you suggest the benefits over FFMS2 for decoding

stability, quality, frame-accuracy, speed...

FFMS2 may do a great job with some sources but, when it comes to MPEG2 and MPEG4 TS, it is kind of a potshot in my experience.

[JeffBDVS]

vampiredom,

Thanks for a great tool! It's nice to be able to point to my files with a single command, run the script in VirtualDub, and have MAIS do for me all those separate steps I used to have to do for myself. So far it's been very stable with SetMTMode(), even when using NV=true. Well done!

-Jeff

[vampiredom]

Thanks for all your help testing this stuff, Jeff!

[redfordxx]

Originally Posted by vampiredom

stability, quality, frame-accuracy, speed...

What speed? Creating index or general playback?
Do you have any comparison on postprocessing FFMS versus DG? Moslty I use the slightest deblocking possible with all sources, which is 30/60 on d2v and 10/60 on ff.


How about colormatrix? Is included or should I take care?


If I have two files with following names and OverwriteIndex=false, does it recognize that the index is outdated and will create new one?
File.mpg
File.vob

[vampiredom]

What speed? Creating index or general playback?

General playback... though this does depend on on the source and options and whether you are using NV=true or DI=true. It is also very good at random seeking once it has been indexed.

Do you have any comparison on postprocessing FFMS versus DG?

No, but if you have any feedback out that, I would love to know.

OverwriteIndex=false, does it recognize that the index is outdated and will create new one?

No. You'd have to manually delete the outdated index files or set OverwriteIndex=true... But that is a good idea. I'll think about a way to implement that in a future release.

How about colormatrix? Is included or should I take care?

No color conversion is performed, so you'd have to take care of that. I should include a "d2v finder" function that allows users to quickly retrieve the path to the .d2v created for a particular file, so they can use the hinting with colormatrix. Hmmm.

[redfordxx]

So now there are no hint available? I cannot put colormatrix after the MPEGAutoIndexSource?

[vampiredom]

So now there are no hint available? I cannot put colormatrix after the MPEGAutoIndexSource?

Oh, sorry ... my bad. Yes, it should work with hints. Try ...

MpegAutoIndexSource("clip.ts", args="info=3")
ColorMatrix(hints=true, dest=0)

.. for example.

[redfordxx]

I am using it as follows:

strSF=((RightStr(srcfile,4)!=".mpg")&&(RightStr(srcfile,5)!=".mpeg")&&(RightStr(srcfile,4)!=".vob")) ? strSF : """MpegAutoIndexSource(filename=srcfile,IndexDir=srcfile+".index")"""
Eval(strSF)

There seem to be several issues I have to investigate, however one seems to be that it has problem with ' as part of srcfile. Maybe you might wanna look into that.