methods for opening MP4 and MKV in avisynth

[halsboss]

Hi, I searched but didn't manage to find anything concise... I have coreAVC and the haali that comes with it.

Can someone please clarify whether these are the latest methods of opening MP4 or MKV in AVIsynth ?

i) demux using MKVToolnix's MKVextract to extract to elementary streams and then run DGAVCDec over the elementary video and then open it with DGAVCDecode.dll in avisynth

ii) use DirectShowSource("InputMP4orMKV",fps=theFPS,convertfps=true,audio=false) directly on the MP4 or MKV and haali and coreAVC will take care of it (MKVinfo should yield the framerate,.. you should know it anyway).

Do both methods work OK with temporal processing ?

There must be a reason neuron2 has created the DGAVCDec tool http://forum.doom9.org/showthread.php?t=122598

Anyone care to indicate their preference for getting the video into avisynth and why ?

[buzzqw]

other way could be FFmpegSource

BHH

[actionman133]

DGAVC seems to have better memory management and access than DirectShowSource, as (I imagine) the majority of its source stems from his excellent DGDecode for MPEG-2.

DirectShowSource works okay, but I have had problems in the past where it can't reliably access the right frame and using seek=false is just cumbersome, memory hungry and time consuming...

That's my experience...

[Guest]

My NV tools can open MKV directly, e.g., DGAVCDecNV.

[mavinashbabu]

Hi,

Not sure if it is related to my same question http://forum.doom9.org/showthread.php?p=1285419#post1285419 here

is this the same that is being discussed in this thread.

EDIT: My Question has been answered not sure if it applies here http://forum.doom9.org/showthread.php?p=1285424#post1285424

Thanks
Avinash

[halsboss]

My NV tools can open MKV directly, e.g., DGAVCDecNV.
Thanks, am browsing the thread DGAVCDec to look for it and came across this Use DirectShowSource() or DSS2(). It seems there are minor "paff" (?) issues with DGAVCDec. DGAVCDecNV is here http://forum.doom9.org/showthread.php?t=141104

Just wondering, what are the relative pros/cons of DirectShowSource and DGAVCDec(NV) to help people choose amongst the options ?

Given I have coreavc and haali installed, does directshowsource have drawbacks ? Like this ... not sure if was related to the codec (eg coreavc or something else) DGAVC seems to have better memory management and access than DirectShowSource... DirectShowSource works okay, but I have had problems in the past where it can't reliably access the right frame and using seek=false is just cumbersome, memory hungry and time consuming...

edit: ah, came across this. I assume that means DSS isn't frame-accurate ? Pity, I like coreavc.

the main purpose of using CUDA is to get correct decoding for the streams that are not handled by libavcodec, and to access some of the post-processing capabilities of CUDA. Also, to add support for VC-1 in the future.Does CoreAVC correctly handle the sources that libavcodec doesn't? If so then the advantage DGAVCdecNV has over it with directshowsource is frame accuracy? Yes, that is the point of DGAVCDecNV, to be a frame accurate Avisynth source filter. Many people also find DGAVCIndexNV to be a useful little app in its own right for quickly inspecting AVC streams, getting PIDs, seeing what audio is present, etc. The NV support also brings you the PureVideo deinterlacing. And it will bring you VC-1 support in the future. For some it also brings a performance inprovement when transcoding. For fast systems, that will be modest.

[Blue_MiSfit]

DSS2 works very well - it's an improved DirectShowSource.

I use it for my BluRay encodes all the time :)

neuron2's tools are excellent as well, provided you have a half decent nVidia card. They're particularly handy for me when it comes to handling true 1080i60 content. PureVideo's bob deinterlacer is fairly solid, and runs real-time on the GPU. Compare that to glacial speeds (and better quality, but not by night and day) from TempGaussMC or MCBob.

~MiSfit

[halsboss]

DSS2 works very well - it's an improved DirectShowSource. I use it for my BluRay encodes all the time :) Excellent ! I used to have a link to it and tried it out for another matter which turned out to be a packed-bitstream related issue, so I'll have to find it again :) I wonder if there's a reason it isn't included "with" an avisynth install like DirectShowSource is ?

edit: found this ... then using DSS2(...) to load the MKV. DSS2 is part of avss.dll, which can be found in your installation directory for haali media splitter.
and this thread discussing DSS2 http://forum.doom9.org/showthread.php?p=1093911 and http://forums.animesuki.com/showthread.php?t=48608 DSS2 is Haali's implementation of Directshowsource. It has the advantage of being frame-accurate, but only supports VFR by converting it to CFR; at least that was the case when I tried it last time, it might have changed.
and
Well if this is going to be turned into a general DSS2 thread... the syntax is: dss2(string infile, float fps)
i.e.
loadplugin("avss.dll")
dss2("X:/test.mkv",fps=23.976)
Unlike the original DSS, it does not support loading audio. It also always converts VFR to CFR; i.e. it acts like the original DSS with convertfps=true does.
also
Avisource uses VfW codecs and the VfW framework only. Think of it like VirtualDub but in Avisynth. As long as the AVI in question doesn't have any particularly funky hacks, it will always have frame-accurate seeking.
Directshowsource uses Directshow codecs and is hence the Avisynth equivalent of Windows Media Player. It's usually NOT frame-accurate, but it does support VFR, kind of; either through converting to CFR (convertfps=true) or by just reading all frames and ignoring timecodes.
DSS2 is Haali's implementation of Directshowsource. It has the advantage of being frame-accurate, but only supports VFR by converting it to CFR; at least that was the case when I tried it last time, it might have changed.
BUT
There are some problems with it, however (M2TS files from Blu-ray, using Haali's splitter and ffdshow in the graph):
- artifacts when seeking in AVC video. They disappear after a few frames.
- When seeking in VC-1 video, sometimes it shows two or three duplicate frames before continuing in the stream.
- It can't load graph files (This isn't a bug, but it'd be a nice feature).
So you can't rely on perfect picture quality in the first few frames after seeking. But at least you know that it is the right frame unlike with DSS, where sometimes the whole stream gets shifted by a few frames.
It just means you have seeked to an open GOP. That's the price you pay for frame accuracy. If you suppress the orphaned frames, then you haven't really seeked to the frame number requested. This is why the DG tools back off by one GOP and decode all the frames necessary to correctly decode the seeked-to frame.(DSS2)... relies on the stream having completely accurate and continuous timestamps throughout the file, and of course raw streams don't have timestamps. Just clarifying the different niches for DSS2() versus DG. I think the SPS/PPS injection heuristics are a bit better in the DG tools also, which can be relevant for some streams.

[halsboss]

And buzzqw and LoRd_MuldeR kindly pointed out ffmpegsource http://forum.doom9.org/showthread.php?t=127037 in the dev thread.

Not sure if this is relevant.. they both start with FF :) http://forum.doom9.org/showthread.php?t=120317 As poisondeathray suggested try to move away from ffdshow for the time being, until it handles this content correctly.

I'll try to collate/summarise in a post at the end, with links, after people in the know have opportunity to put their 2c. No doub't that'll need some correction too as I try to read between the lines of the threads and the posts here.

edit: Ps does this forum have a facility for creating a table ?

[halsboss]

So far, given the above, for frame-accuracy in avisynth temporal processing on an MKV or MP4 source then seems like the options are

- DGAVCDecNV can do elementary streams and MKV, but not MP4. It's for NVIDIA card owners only.
- DGAVCDec has more limitations than the NV version (dependency on libav library?)
- possibly ffmpegsource however I'm not sure (neuron2 has some things to say in his DG threads on the libav library and is using an older version, can someone to interpret that and it's effect, if any, on ffmpegsource ?)
- not directshowsource as it's not frame accurate
- not DSS2 as there are artifacts when seeking in AVC video (or there were, when the post was made)

[TheRyuu]

- DGAVCDecNV can do elementary streams and MKV, but not MP4. DGAVCDec has more limitations than the NV version. It's for NVIDIA card owners only.

Because it uses CUDA for decoding (and thus only runs on Nvidia cards, but is faster for decoding). It also costs money.

- possibly ffmpegsource however I'm not sure (neuron2 has some things to say in his DG threads on the libav library and is using an older version, can someone to interpret that and it's effect, if any, on ffmpegsource ?)

I don't really follow (understand the question?). ffmpegsource is built using the current ffmpeg version at the time of compilation.

- not DSS2 as there are artifacts when seeking in AVC video (or there were, when the post was made)

Never had a problem with this, even with m2ts BD streams... You shouldn't rule out dss2 as a viable option, although with mkv/mp4 I'd use fffmpegsource over dss2 probably.

The options for opening a mkv or mp4 and be frame accurate are:
-ffmpegsource
-dss2

I don't see an issue with either of those and I'm not really sure what exactly the problem is here...

[halsboss]

I don't really follow (understand the question?). ffmpegsource is built using the current ffmpeg version at the time of compilation. Well, pending clarification from someone who knows, as I understand it ffmpeg is built using the libav library. Neuron2 says he can't use new versions for a good reason :-I cannot use the newer versions because they are broken for accurate frame random access. which in turn implies that ffmpegsource might also be broken too for frame accurate random access (eg possibly with temporal processing). I don't know positively one way or another about ffmpegsource because of that. Probably it's fine, but how to know for sure and not heresay? What are its Pro's and Con's compared to DGAVCDec(NV) ... eg can it handle all the same things just as well and avoid the reported DSS2 issues too?

Never had a problem with this, even with m2ts BD streams... You shouldn't rule out dss2 as a viable option But how should I rule it in or out ? Some people say there's problems with DSS2 and frame seeking in terms of some individual frames being returned as gibberish if Seek'd in the middle of a GOP, another says maybe there isn't a problem, everyone positive sounding in their statements.

The options for opening a mkv or mp4 and be frame accurate are:
-ffmpegsource
-dss2I see you omitted DGAVCDec(NV) probably because it wasn't free, and included one with reported errors and another with a possible questionmark over it. I'm very nearly decided to fork out a couple of peanuts to donate to acquire the NV since it just about seems like the best of the options without conflicting reports. I bought coreavc codec for a couple of peanuts for the same reason. There seems no "agreed, reported by all as reliable" alternative.

Hopefully you see the problem now, with conflicting reported errors and "no there isn't"s for some of the s/w on the list... what IS the situation when information apparently conflicts ? If I had the skills and samples I'd test them all, but I don't.

[Blue_MiSfit]

If you have an nVidia card, don't mind spending just a few measly bucks, and don't mind spending the time to index any source you want to process, neuron2's DG***IndexNV tools are by far the best option. They are extremely reliable, give you excellent free deinterlacing, and are perfectly frame accurate, even when seeking inside a GOP.

If on the other hand you just need to process BluRay or other H.264 MKV / MP4 sources straight through without any trims or seek operations, and don't want to spend a dime, then there is absolutely nothing wrong with DSS2 at all. It works extremely well, and I've done literally hundreds upon hundreds of BluRay backups using this method :)

~MiSfit

[TheFluff]

just fyi ffmpegsource has never used libavformat to open mkv's, hth

e: afaik there has never been any particular frame accuracy issues with mp4's and libavformat

[halsboss]

from the dev forum (slightly abbreviated) ...
Libavformat is one big random access issue just waiting to happen so that sounds about right. The difference is most likely how we treat it. FFMS2 seeks, looks at the returned timestamp, goes back further if it wasn't the intended frame to seek to and so on until it finally succeeds or throws an error (default behavior). Sometimes lavf is so inaccurate it can't match up. That's life. However FFMS2 always goes back to the previous GOP (if it succeeds at all) so that is not an issue. What DSS2 does I'm not really sure of.
I think they've proposed a new seeking API as a SoC project this year though...

[halsboss]

EDIT Dec 2009, this summary is not accurate and is out of date... refer the the following 15 or so posts for clarification.

In summary, as at May 2009,


DirectShowSource has problem returning "good" frames when seeking. May be OK for "straight through without any trims or seek operations" and I guess that means without multi-threaded processing either. Want to process the video in some way... fine if you don't mind some glitches. Opens MKV, MP4 provided you have the right splitter and codec installed. Free.
DSS2 (DirectShowSource2) has a problem returning "good" frames when seeking. Appears OK for "straight through without any trims or seek operations" and I guess that means without multi-threaded processing either. Want to process the video in some way... fine if you don't mind some glitches. Opens MKV, MP4 provided you have the right splitter and codec installed. Free.
ffmpegsource tries its heart out but is not guaranteed to seek/return the frame you think you need. Appears OK for "straight through without any trims or seek operations" and I guess that means without multi-threaded processing either. Want to process the video in some way... seems like a very handy tool and fine if you don't mind the possibility of an occasional glitch. Opens MKV, MP4. Free.
DGAVCDec works. Is video-card independent. Is based on an older version of the libav library where seeking issues are not apparent, but as a consequence it handles less "types" of AVC streams than the "NV" version, or ffmpegsource, or a new libav library version could if it seeked properly. Can't have it both ways I guess. Have to demux MKV and MP4 to elementary streams before using it, so team it with a free demuxer. Free.
DGAVCDecNV works. Offloads processing to the GPU. Works only with Nvidia 8xxx series and above video cards with VP2/3. ATI seems to have made things challenging and hence left their customers "in the cold" a bit. Is currently the most reliable solution if you want to process video with trims or seek operations and multi-threaded processing or a range of AVC "stream types". Opens MKV. Have to demux MP4 to elementary streams before using it, so team it with a free demuxer. Costs a trivial few peanuts of donation to the author; I forked out... and now I don't have to wonder "what glitches are in my output video because I was a cheapskate".


Happy to see updates... opinions welcome; it seems there are a few "either way" on some things :)

[buzzqw]

@halsboss

please update the first post too... this thread could be a faq

BHH

[halsboss]

Sure, not sure what you'd like though. PM me and I'll do it.

[TheFluff]

In summary, as at May 2009,


DirectShowSource has problem returning "good" frames when seeking. May be OK for "straight through without any trims or seek operations" and I guess that means without multi-threaded processing either. Want to process the video in some way... fine if you don't mind some glitches. Opens MKV, MP4 provided you have the right splitter and codec installed. Free.
DSS2 (DirectShowSource2) has a problem returning "good" frames when seeking. Appears OK for "straight through without any trims or seek operations" and I guess that means without multi-threaded processing either. Want to process the video in some way... fine if you don't mind some glitches. Opens MKV, MP4 provided you have the right splitter and codec installed. Free.
ffmpegsource tries its heart out but is not guaranteed to seek/return the frame you think you need. Appears OK for "straight through without any trims or seek operations" and I guess that means without multi-threaded processing either. Want to process the video in some way... seems like a very handy tool and fine if you don't mind the possibility of an occasional glitch. Opens MKV, MP4. Free.
DGAVCDec works. Is video-card independent. Is based on an older version of the libav library where seeking issues are not apparent, but as a consequence it handles less "types" of AVC streams than the "NV" version, or ffmpegsource, or a new libav library version could if it seeked properly. Can't have it both ways I guess. Have to demux MKV and MP4 to elementary streams before using it, so team it with a free demuxer. Free.
DGAVCDecNV works. Offloads processing to the GPU. Works only with Nvidia 8xxx series and above video cards with VP2/3. ATI seems to have made things challenging and hence left their customers "in the cold" a bit. Is currently the most reliable solution if you want to process video with trims or seek operations and multi-threaded processing or a range of AVC "stream types". Opens MKV. Have to demux MP4 to elementary streams before using it, so team it with a free demuxer. Costs a trivial few peanuts of donation to the author; I forked out... and now I don't have to wonder "what glitches are in my output video because I was a cheapskate".


Happy to see updates... opinions welcome; it seems there are a few "either way" on some things :)

you got about half of them wrong and turned this into a neuron2 advertising thread, congratulations


DSS2 should always be frameaccurate (depending of course on your system's directshow splitter/decoder chain for the format in question) but always converts to CFR. If this is a problem don't use it.
ffms2 is usually frameaccurate; with mkv it's probably always frameaccurate (after a bug that made it drop the first or second frame was fixed last week); with mp4 lavf is probably good enough to be frameaccurate for all sane cases.


it's funny how you mention the open gop issue that can happen with m2ts as if it was relevant for mkv/mp4; it's also funny how you think you can determine the truth only by reading doom9 posts

[Guest]

DSS2 should always be frameaccurate False. If the timestamps are not fully correct and continuous it fails. It also fails for elementary streams.

[TheFluff]

False. If the timestamps are not fully correct and continuous it fails. It also fails for elementary streams.

mkv's and mp4's should always have correct timestamps, if not it is a corrupt file and all bets are off regardless of source filter

and this thread is not about elementary streams now is it

[Guest]

I responded to your statement, which did not contain any qualifiers, and which explicitly referred to other formats.

[poisondeathray]

Maybe you can add that another potential pitfall of DSS/DSS2 is that it predisposes you to all the filters in your directshow chain. e.g. if you forget to turn off processing in ffdshow (you might have used for playback) , etc... those effects get passed through when you encode. So if you are not very diligent you can get burned.

Also when using DSS/DSS2, the choice of decoder becomes more important, and each may behave differently or have quirks. e.g. there are differences between using ffdshow vs ffdshow-mt vs. coreavc vs. divx h.264 etc...

[TheFluff]

I responded to your statement, which did not contain any qualifiers, and which explicitly referred to other formats.
hmm, yes, a semantic argument, how productive

Does DSS2 even open elementary streams at all? What directshow source filter could be used to open them? I already stated that DSS2's capabilities are obviously dependent on your system's dshow splitters/decoders.

I obviously didn't qualify "this filter may not work correctly with broken files herp derp" because I thought it was self-evident, but apparently not!

[halsboss]

you got about half of them wrongWell, help fix them with some facts. and turned this into a neuron2 advertising thread, congratulationsmy, my, a little pointed aren't we. That isn't my intent, why would you want to suggest so ? Don't answer that.

it's funny how you mention the open gop issue that can happen with m2ts as if it was relevant for mkv/mp4;No it isn't funny, you SAW I am asking for clarification. It isn't relevant for mkv/mp4 ?
it's also funny how you think you can determine the truth only by reading doom9 postsIt's a good a means as any, reading contributions of people who know things, including you. Why else have a forum. Why are you so emotionally involved ? If you think you can fix it up then please by all means clarify. Some info appears to conflict with others and that's been raised above too, for knowledgeable people like you to collaborate and clarify for the betterment of others who use the forum.

PS I see I omitted a googled link I came across re the possibility DSS2 and "glitchy" (not non-frame-accurate) frames after a seek, will see if I can find it again. Nope, lost.

PPS please post a better table if you like. Just food for thought, should, probably good enough aren't entirely comforting. that may be life, but does it change the table ? If so please feel free to change it.

[TheFluff]

No it isn't funny, you SAW I am asking for clarification. It isn't relevant for mkv/mp4 ?
Not really, no.

It's a good a means as any, reading contributions of people who know things, including you. Why else have a forum. Why are you so emotionally involved ? If you think you can fix it up then please by all means clarify. Some info appears to conflict with others and that's been raised above too, for knowledgeable people like you to collaborate and clarify for the betterment of others who use the forum.
Unlike peer reviewed scientific papers, doom9 posts aren't really all that reliable (just the likelihood of them being outdated is a significant factor) and secondary-source research based on them cannot be considered as relevant to reality. If you want to find something out, test it yourself.

[halsboss]

Unlike peer reviewed scientific papers, doom9 posts aren't really all that reliable (just the likelihood of them being outdated is a significant factor) and secondary-source research based on them cannot be considered as relevant to reality. If you want to find something out, test it yourself.Good thoughts. Mostly people pragmatically go with what others, whom they choose to consider knowledgeable, say rather than reinvent the wheel; and this stuff isn't going to end the world. The objective of the thread was to tap into others knowledge as it is today, raising and putting to rest furfies and obsolete info.

So, DSS2 is frame accurate, then and it relies on a well behaved and understood splitter and filter chain (haali, coreavc?). And seeking to the middle of a GOP (?) won't return dud looking frames because it's not relevant to mkv/mp4. And ffmpegsource is similar (albeit the words above re libav are very much less than confidence-building). There was a little recent discussion here http://forum.doom9.org/showthread.php?t=139092 but I'm not sure of a conclusion.

[poisondeathray]

I use DSS2 frequently, and it works fine. Another quirk is that the fps is often not perfect, but off by a bit. (it's advisable to check with info() in avisynth), and one method is to use AssumeFPS() to correct it. e.g AssumeFPS(24000,1001) for film

[TheFluff]

So, DSS2 is frame accurate, then and it relies on a well behaved and understood splitter and filter chain (haali, coreavc?). And seeking to the middle of a GOP (?) won't return dud looking frames because it's not relevant to mkv/mp4. And ffmpegsource is similar (albeit the words above re libav are very much less than confidence-building). There was a little recent discussion here http://forum.doom9.org/showthread.php?t=139092 but I'm not sure of a conclusion.
I reiterate:
ffmpegsource has never used libavformat to open mkv's
It has always used Haali's matroska parser, which doesn't have any particular random access issues. I have never seen any random access issues of note with mp4's and lavf either; what Myrsloik mentioned in the ffms2 thread mostly applies to less well supported/specified containers.

As for the open GOP issue, the entire thread seems focused on h264, and h264 doesn't even have open GOP's as such, AFAIK (IDR-frames and multiple references kind of ruin the idea). You're also unlikely to ever run into an MPEG4 ASP stream with open GOP in the wild (you weren't thinking of downloading stuff, were you, you naughty creature?) unless you encode one that way yourself (so don't do that). I wouldn't ever put MPEG2 into MP4 or MKV either so that is out too. In either case, if you do somehow run into an open GOP with DSS2 the splitter (if it is a good splitter) is likely to back up to nearest reliable decoding start point for you anyway. I imagine the reason it doesn't do this with M2TS is that Haali's TS splitter isn't all that perfect (yet).

[halsboss]

Excellent info, thankyou people. Will post an updated summary soon, pending any more clarifications.

edit: points are well made, although I admit phrases like "(if it is a good splitter) is likely to" don't inspire confidence it'll actually do the right thing; albeit no reports of bad things it seems.

[vie2233hil]

What's the difference between these options quality-wise? directshow ones could use coreavc which is thougth of having less quality output than libavcodec based options? Does libavcodec provide good quality?

[7ekno]

What's the difference between these options quality-wise? directshow ones could use coreavc which is thougth of having less quality output than libavcodec based options? Does libavcodec provide good quality?

Quality doesn't apply to AVC decoding, every decoder is based on the same specifications, and therefor, barring any bugs, should have identical output ...

Just to further the discussion also, DirectShowSource(), the original one, is pretty bad for HD (720p/1080p) MKVs, the last time I tried it (even with Haali/CoreAVC, Haali/FFmpeg and Haali/QTdecode) as a "straight through" encode, there was the occasional (like one every 10-15secs) frame jump / misplacement ... at the time I rectified the problem with the original FFmpegsource() ... haven't completely tested version 2 of either ...

Since I don't own a Nvidia 88xx series or better card, am waiting to see how this thread concludes ;)

7ek

[Mr VacBob]

As for the open GOP issue, the entire thread seems focused on h264, and h264 doesn't even have open GOP's as such, AFAIK (IDR-frames and multiple references kind of ruin the idea).

h264 can emulate this pretty well; it has recovery point SEIs, which streams can use to provide seek points without IDR. I can't remember if ffmpeg supports them (or if it's even conceptually possible to support them).

And of course, if there's some PAFF stream lavc doesn't decode, or some problem preventing DGAVCDec from upgrading lavc, I hope it's in ffmpeg roundup...

[medp7060]

Excellent info, thankyou people. Will post an updated summary soon, pending any more clarifications.

edit: points are well made, although I admit phrases like "(if it is a good splitter) is likely to" don't inspire confidence it'll actually do the right thing; albeit no reports of bad things it seems.

have you updated the summary?