LAV Filters - DirectShow Media Splitter and Decoders

[turnip]

I'm using LAVSplitter.ax 0.63 and ffdshow video decoder on Windows. I can play an AVI from the first frame, but if I start playback from anywhere else I get no video.

I can see that when played from the start the splitter video stream includes the SPS & PPS NAL units, but after a set position these units are absent from the start of the video stream. I believe this is the reason the decoder fails to work.

Is there a way to make the LAV splitter insert the SPS and PPS NAL units into the video stream following a set position?

[LigH]

Your AVI container is probably incomplete. Try to re-multiplex it. VirtualDub would probably offer to re-index it upon opening, I guess?

[turnip]

I have no control over the AVI files. I must try to play the AVI files I am given from the point the user specifies, which may not be the start.

[LigH]

If you try to jump into the middle of an un-indexed AVI, the splitter may not be able to detect where any frame starts; not to mention discovering the timecode of the frame, if it finds one at all. And correct decoding can even only start at specific frames, not any (usually I frames, possibly even IDR for AVC).

It is not the job of a decoder to fix an incomplete file. If it doesn't have any keyframe index, then the best a decoder can do is to slowly scan the file around the given position until it finds a valid decoding start position, which will take its time (up to the previous playing time), or start to decode whatever it finds, which will cause video errors (maybe even decoder crashes). But only if the splitter is able to discover where the multiplexed frames of video and audio streams start at all. That may require to linearly decode from the beginning to the seek position. Which may take a lot of time... VirtualDub's re-indexing will also start from the beginning, scanning the rest sequentially, to rederive a keyframe index.

If you must be able to play from an arbitrary seek position, first fix the media file.

By the way, which content formats do such AVIs contain? Possibly "online recorder" AVIs with AVC video? I'm curious about a MediaInfo analysis.

[foxyshadis]

I have no control over the AVI files. I must try to play the AVI files I am given from the point the user specifies, which may not be the start.

Remux them as they come in then. Use ffmpeg or mkvmerge to immediately create a new file that allows arbitrary seeking, and you won't have to worry about that.

[nevcairiel]

Is there a way to make the LAV splitter insert the SPS and PPS NAL units into the video stream following a set position?

If LAV Splitter would know the SPS/PPS, then it wouldn't be a problem in the first place. :)
However, for it to fail, it must be lacking the SPS/PPS information, in this case they need to be repeated in the stream .. if the stream only contains them once right in the beginning, then you are screwed and the stream is not seekable.

[turnip]

1. Thank you all for taking time to respond to my question.

2. As I say, my circumstances are that the user presents an AVI and I either play it successfully or I fail to play it. It is not an option for me to tell the user to re-encode his video.

3. This may or may not be relevant: I notice in LAVFilters\qsdecoder\frame_constructors.cpp in CAVCFrameConstructor::ConstructHeaders() the comment "// Keep a copy of the SPS/PPS to be placed into the decode stream (after each new segment)." This code is not used when splitting my AVI, but it suggests to me there are circumstances when it is necessary to insert these NAL units into a video stream. (I have only a very limited understanding of the LAV splitter code. But I have built the splitter from scratch which, contrary to the comment in the README "Compiling is pretty straight forward using VC++2013", took me an embarrassingly long time to overcome one problem after another.)

4. Nev, your comment has made my heart sink! The stream does indeed have the SPS/PPS just once at the beginning. I hoped there might be a simple way I could get the splitter to read the stream start to obtain the SPS/PPS units and then insert them at the start of the stream following a set position.

5. The Mediainfo output for a sample AVI file is below.

General
Complete name : C:\test.AVI
Format : AVI
Format/Info : Audio Video Interleave
File size : 26.7 MiB
Duration : 33s 170ms
Overall bit rate : 6 756 Kbps

Video
ID : 0
Format : AVC
Format/Info : Advanced Video Codec
Format profile : Main@L4.0
Format settings, CABAC : Yes
Format settings, ReFrames : 2 frames
Format settings, GOP : M=1, N=8
Codec ID : H264
Duration : 33s 167ms
Bit rate : 6 550 Kbps
Width : 1 280 pixels
Height : 720 pixels
Display aspect ratio : 16:9
Frame rate : 30.000 fps
Color space : YUV
Chroma subsampling : 4:2:0
Bit depth : 8 bits
Scan type : Progressive
Bits/(Pixel*Frame) : 0.237
Stream size : 25.9 MiB (97%)

Audio
ID : 1
Format : ADPCM
Codec ID : 2
Duration : 33s 170ms
Bit rate mode : Constant
Bit rate : 192 Kbps
Channel(s) : 1 channel
Sampling rate : 48.0 KHz
Bit depth : 4 bits
Stream size : 782 KiB (3%)
Interleave, duration : 42 ms (1.27 video frame)


If I grep for "nal_unit_type" shows SPS and PPS appear only at start of BWV.avi
nal_unit_type : 7 ( Sequence parameter set )
nal_unit_type : 8 ( Picture parameter set )
nal_unit_type : 5 ( Coded slice of an IDR picture )
nal_unit_type : 1 ( Coded slice of a non-IDR picture )
nal_unit_type : 1 ( Coded slice of a non-IDR picture )
nal_unit_type : 1 ( Coded slice of a non-IDR picture )
nal_unit_type : 1 ( Coded slice of a non-IDR picture )
nal_unit_type : 1 ( Coded slice of a non-IDR picture )
nal_unit_type : 1 ( Coded slice of a non-IDR picture )
nal_unit_type : 1 ( Coded slice of a non-IDR picture )
nal_unit_type : 1 ( Coded slice of a non-IDR picture )
nal_unit_type : 1 ( Coded slice of a non-IDR picture )
nal_unit_type : 1 ( Coded slice of a non-IDR picture )
nal_unit_type : 1 ( Coded slice of a non-IDR picture )
nal_unit_type : 1 ( Coded slice of a non-IDR picture )
nal_unit_type : 1 ( Coded slice of a non-IDR picture )
nal_unit_type : 1 ( Coded slice of a non-IDR picture )
nal_unit_type : 5 ( Coded slice of an IDR picture )
nal_unit_type : 1 ( Coded slice of a non-IDR picture )
nal_unit_type : 1 ( Coded slice of a non-IDR picture )
nal_unit_type : 1 ( Coded slice of a non-IDR picture )
nal_unit_type : 1 ( Coded slice of a non-IDR picture )
nal_unit_type : 1 ( Coded slice of a non-IDR picture )
...neither SPS nor PPS occur ever again

[nevcairiel]

It might help if you can share one of those files.

[turnip]

Hi Nev. Yes I will upload a sample this evening. (I just tried but it fails with some "security token failure", so I'll have to try again when I get home tonight.)

[turnip]

Hi Nev, The only sample I have is 27 MB. I've attached it truncated to 200KB. I've put the original here: https://www.dropbox.com/s/ecpj1iabog8e8bo/turnip.avi?dl=0

The sample is produced by a portable video camera. On Monday I'll be able to get a smaller complete sample.

[nevcairiel]

27MB isnt exactly huge, so thats ok.

[nevcairiel]

Looking for Samples!

I'm interested in adding support for AAC 960, and for that I need a few samples using this format.
Its a AAC format primarily used in DAB+ broadcasts, and if anyone is interested in this feature, and has a sample at hand that he can share with me, it might expedite its availability!

Thank you for listening. :)

[rd1979]

There is no VP9 DXVA specification (to my knowledge), so there is nothing to support.

Who would be responsible for creating the spec? Microsoft? If so, I don't expect to see it. Ever.

[Aleksoid1978]

Hi Nev. I need your help :) How can i check DXVA compatibility without connect to the renderer ? Can you help ?

[LigH]

I'm interested in adding support for AAC 960, and for that I need a few samples using this format.

How about http://www.dabmon.de:8002/? (via (https://trac.ffmpeg.org/ticket/1407)) – tested with VLC 2.1.5 and foobar2000 1.3.6, so chances are good that you can grab a stream

Noticed a flaw trying Energy München: Plays well in foobar2000 during music but breaks when switching to moderation

[Reino]

I'm interested in adding support for AAC 960, and for that I need a few samples using this format.

See post 1643189.

[SeeMoreDigital]

Looking for Samples!

I'm interested in adding support for AAC 960, and for that I need a few samples using this format.
Its a AAC format primarily used in DAB+ broadcasts, and if anyone is interested in this feature, and has a sample at hand that he can share with me, it might expedite its availability!
See post 1643189.The HE-AAC v2 (Arrow Classic Rock) sample might present a challenge...

[mindbomb]

I'm having an issue with recordings made with the amd gaming app. When played with dxva copyback or native, there is a large audio delay. The delay still exists with software decoding, but it is reduced a lot. When muxed to an mkv, the problem goes away, but mkvmerge gives the warning "The AVC video track is missing the 'CTTS' atom for frame timecode offsets. However, AVC/h.264 allows frames to have more than the traditional one (for P frames) or two (for B frames) references to other frames. The timecodes for such frames will be out-of-order, and the 'CTTS' atom is needed for getting the timecodes right. As it is missing the timecodes for this track might be wrong. You should watch the resulting file and make sure that it looks like you expected it to"

sample: https://www.mediafire.com/?1osasr6dr80c6ls

[nevcairiel]

Try with the latest nightly it should work better. Even better would be yelling at AMD to write better files.

[mindbomb]

Try with the latest nightly it should work better. Even better would be yelling at AMD to write better files.

ok, that helped a lot, the delay seems gone with the nightly. So does this type of thing not happen when using nvidia's shadowplay?

[romulous]

So does this type of thing not happen when using nvidia's shadowplay?

I've not ever had any problems running any of my Shadow Play videos with LAV. I can't guarantee that Shadow Play would never produce a broken file, but I can say that it hasn't done that on my system. The only real issue I've ever had with it is the way that it doesn't properly capture 16:10 at that exact native resolution (my 1920x1200 is captured at 1188xsomething) - but that is by design apparently. Stupid NVIDIA.

[nevcairiel]

Hi Nev, The only sample I have is 27 MB. I've attached it truncated to 200KB. I've put the original here: https://www.dropbox.com/s/ecpj1iabog8e8bo/turnip.avi?dl=0

The sample is produced by a portable video camera. On Monday I'll be able to get a smaller complete sample.

This sample seems to work fine, and while there are no SPS/PPS in the stream at a later point, LAV Splitter manages to find the SPS/PPS and puts them into the media type for the decoder to use even after seeking.

Are you actually using LAV Splitter for this, and not the Microsoft AVI Splitter or something else? I can reproduce the problem with other AVI Splitters, but not with LAV Splitter.

Edit:
I just read you use ffdshow video decoder, maybe try with LAV Video instead? It works fine for me with LAV Splitter + LAV Video in any case.

[mastan]

ok, that helped a lot, the delay seems gone with the nightly. So does this type of thing not happen when using nvidia's shadowplay?

I've recordered several dozens of shadowplay 1920x1080 60fps videos and didn't encounter any problems converting, playing it with MPC HC/LAV or directly uploading to youtube.

[LigH]

^ Such success will depend heavily on the used hardware. Don't even try this on a laptop with energy-saving harddisk (I doubt it will even be offered for on-board GPUs); hardware dedicated for both gaming and video editing will be recommendable.

[turnip]

Nev, You're a busy guy, but can you give me a quick pointer to help me understand this?

I've found that the AVI I uploaded here will play from any point if I use the LAV splitter and LAV video decoder, but not if I use the LAV splitter and ffdshow video decoder. How is the LAV video decoder able to function correctly without SPS/PPS? Is it able to deduce whatever it needs from just the IDR and non-IDR NAL units? Could you point me at the source file where this might be done?

Edit: sorry, I just realised you had asked me to try this above. Note that the LAV splitter does not send SPS/PPS to the LAV decoder after a seek, yet the video plays correctly. (Also, for various reasons, I cannot simply switch to the LAV decoder.)

[nevcairiel]

LAV Splitter actually discovers the SPS/PPS from the beginning of the stream and sets it in the media type (a global struct with stream information for the decoder), from where the decoder can read it. Apparently, ffdshow just doesn't read this info.

[turnip]

Thank you. I'll investigate. If ffdshow requires the information in the SPS/PPS, and the LAV splitter has the information, perhaps I could fabricate SPS/PPS for ffdshow...?

[nevcairiel]

ffdshow also has the information, it just needs to make use of it. But ffdshow is old, out of date and no longer under development, so unless you somehow manage to fix it yourself, there is probably not much thats going to happen.

[red5goahead]

ffdshow also has the information, it just needs to make use of it. But ffdshow is old

Yes, it's true and I'm stuck since a long time with Lav filter, the best. So do you never consider to add some kind of post-processing as ffdshow does? Imho most of these ones are useless but some not , as sharpen

[foxyshadis]

Yes, it's true and I'm stuck since a long time with Lav filter, the best. So do you never consider to add some kind of post-processing as ffdshow does? Imho most of these ones are useless but some not , as sharpen

Use the ffdshow Raw video processor. You can just uninstall or reduce the merit of the main decoder, but still use it for post-processing while LAV does the decoding.

[red5goahead]

Use the ffdshow Raw video processor. You can just uninstall or reduce the merit of the main decoder, but still use it for post-processing while LAV does the decoding.

Of course I did and I use it with Media Portal but imho would be fine if was embedded in Lav codec to simplify besides the ffdshow is no longer under development

[Hikari]

Thanks for the hard work bro. You're tha man!

[salam2009]

Hey guys,
I'm so frustrating with MPC-BE crashes everytime I seek in any video ONLY if (EVR -custom-) was selected which I prefer to use among the others! (like MadVR, Haali, etc.)
Also, I use LAV Filters as "External" and I can't get them to work unless I change the video renderer!
Your help would definitely make my day!
Thank you!

Cheers,
Salaam

[Nintendo Maniac 64]

Hey guys,
I'm so frustrating with MPC-BE crashes everytime I seek in any video ONLY if (EVR -custom-) was selected which I prefer to use among the others! (like MadVR, Haali, etc.)
Also, I use LAV Filters as "External" and I can't get them to work unless I change the video renderer!

Doesn't MPC-BE have internal LAVfilters like MPC-HC does? In that case you should be able to follow the process I spoke of before:


There are nightly builds (http://files.1f0.de/lavf/nightly/) now, at least anytime there's a change. (Horray!) I've had no problems with the most recent one so far.
Fun fact: the DLLs and stuff in that installer seem to be perfectly compatible with MPC-HC.

In other words, if you run the installer in something like sandboxie, you can copy the contents from the x86 and/or x64 folder into the LAVFilters and/or LAVFilters64 folders in the MPC-HC directory and everything will "just work".

[06_taro]

The installers are compiled by Inno Setup. If you need DLLs only, you can use an unpacker like innounp (http://innounp.sourceforge.net/) or innoextract (http://constexpr.org/innoextract/) instead of using sandboxie:
innounp -x -dDIR INSTALLER.EXE
innoextract -e -dDIR INSTALLER.EXE

[sneaker_ger]

Doesn't MPC-BE have internal LAVfilters like MPC-HC does?
I think it doesn't. It still comes with filters based on the original ones by Gabest.

[salam2009]

Doesn't MPC-BE have internal LAVfilters like MPC-HC does? In that case you should be able to follow the process I spoke of before:

I think it doesn't. It still comes with filters based on the original ones by Gabest.

Exactly! I had to add it as external but couldn't figure out why MPC-BE crashes during seeking whereas MPC-HC doesn't while the last one uses LAV as internal!
The problem only occurs with EVR (custom) which I need to use.

[clsid]

There may be differences in the EVR-CP implementations of MPC-BE and MPC-HC.

[bilditup1]

I'm having some trouble playing back a vfr mp4 with MPC-HC, using the LAV filters it ships with - the audio keeps going out out of sync, slowly, over time, until it becomes noticeable/awful.

I made the file like so:

Generated timecodes v1 file with 2-pass TFM()/TDecimate() (http://avisynth.nl/index.php/VFR#encoding_to_vfr_.28mkv.29).
Inputted that file into x264 when encoding, using --tc-filein, and specifying an output timecodes v2 file using --tc-fileout. For some reason, x264's automatic time base generation failed multiple times, so I manually specified "1001/360000", which is a total shot in the dark that I just copied based on old logs.
Took the 264 output and muxed it to mp4 (without anything else).
Used mp4fpsmod (http://github.com/nu774/mp4fpsmod) to add the timecodes outputted by x264 to the mp4, using -c switch to enable dts compression.
Muxed that file into a new mp4 with audio and subs, with -ipod switch to ensure iDevice compatibility.


On two different machines running MPC-HC 32-bit, one on Win8.1 Pro x64 and one running Win7 Pro x64, and shutting down all CPU- and GPU-eating processes that could possibly affect playback, the file slowly drifts out of sync until things become seriously unwatchable, about two and a half minutes in. The same file plays fine on an iPad Mini 2 running iOS 8.1.2.

I'm pretty sure the same thing happens if I mux audio and subs first, and then add timecodes, but need to double-check.

I can provide a section of it here, but need to test to make sure the same issue happens again with the shorter file. Noteworthy is that this does not seem to happen with every vfr mp4 I've made (this is only the second one). This one though is about 30% video, whereas the other one had a few hundred frames at most.

Any ideas?

[foxyshadis]

What happens if you feed the original timecode file into mp4fpsmod, instead of x264's output? They should be the same, since x264 doesn't add or drop frames, but the TDecimate version is definitely correct -- or the cause of the other problems. Make sure you use tcfv1=False in TDecimate though, to generate v2 timecodes.

I'm so out of touch I didn't know that program existed, so I guess I can update the wiki once I test it out.

[nautilus7]

@Nev

Hi, I have a video file (https://mega.co.nz/#!ixhkBaTb!h15OTDcw90FaE0d1ThORuopxO9xHCve9PQWYg9-4BUY) that doesn't playback using mpc-hc with internal LAV filters. It is a satellite capture (DVB-S2) and VLC can playback it fine. Could you please have a look what is wrong? Thanks.

[hello_hello]

@Nev

Hi, I have a video file (https://mega.co.nz/#!ixhkBaTb!h15OTDcw90FaE0d1ThORuopxO9xHCve9PQWYg9-4BUY) that doesn't playback using mpc-hc with internal LAV filters. It is a satellite capture (DVB-S2) and VLC can playback it fine. Could you please have a look what is wrong? Thanks.

I can't tell you much regarding the "why" but I had a quick play and MPC-HC can display it if ffdshow is decoding and the mpeg2 codec is set to libavcodec. Switch to libmpeg2 and it displays the same way as when LAV is decoding. Just a black screen. MPC-BE 1.4.3 with it's internal decoder = black screen. PotPlayer default settings = black screen.

Muxing programs don't seem to have a good time with it. MKVMergeGUI won't acknowledge the video stream exists, and TSMuxer only sees a single audio stream, although GDSMux sees the video and two audio streams. Even after remuxing as an MKV, the problem of the video not displaying remains.

[nautilus7]

Yes, I know mkvmerge can't deal with this video stream. Thanks for the advice about ffdshow.

[bilditup1]

What happens if you feed the original timecode file into mp4fpsmod, instead of x264's output? They should be the same, since x264 doesn't add or drop frames, but the TDecimate version is definitely correct -- or the cause of the other problems. Make sure you use tcfv1=False in TDecimate though, to generate v2 timecodes.

I'm so out of touch I didn't know that program existed, so I guess I can update the wiki once I test it out.

The original timecodes I've been generating are v1, had no idea that you could generate v2. When I used the v1 generated by TDecimate directly, I think the sync issues were worse, but I never checked them on both MPC-HC and my iPad, and anyway by now I'm confused enough with all my attempts that I'll just do it over.

Ha, I only discovered mp4fpsmod after copious searching (which natch led me to a couple of threads back here).

Thanks for chiming in, will report back...

[-TiLT-]

First of all, if this idea came up on the 929 other pages of this thread, please excuse me. I did not read them all.

I am focussing on audio streams with "stretch by" switch (altering the default duration) in mkv.

Situation:

Video track @ 25fps
1st audio sync to video @ 25fps
2nd audio sync to video @ 23.976fps

Ways to deal with this:

1. Speedup or Slowdown one of the audio streams (plus remux at a different video framerate if needed)
2. Timestretch one of the audio streams
3. Use the "stretch by" switch of mkvmerge

Problems:

1. Despite the fact of a quality loss due to reencoding, speedup and slowdown cause human voices and music to sound irritating to some audiences. And irritating means chimpunkish when speed up or dozy when slowed down.
2. Again, reencoding would reduce the quality and timestretching artifacts can be even more disturbing.
3. The "stretch by" feature is beeing dealt with different approaches, all with their own disadvantages

I. Ignoring the setting and causing Audio/Video desynchronization
II. Dropping or looping portions of the audio. (The most primitive timestretching)
III. Timestretching on the fly with the above inherent problems and a high cpu usage
IV. Playing back the audio untouched and dropping or duping video frames on a regular and short interval to prevent notably desynchronisation (this is what LAV Filters do, I think) but with the drawback of juddery motion.

Approaches I. and II. fail miserably, approach III. does what the name of the setting intends but fails on slow machines and produces hearable artifacts.
Approach IV. almost spot lands, as the audio-quality is preserved and the stream can even be passed through, but tumbles on the finishing straight by adding judder to the video whereas a simple adjustment of the videos framerate would suffice.

I tried to spot the reason for the frame-duping/dropping and the only thing I came up with was that a rate-change during playback might possibly violate the rules of a playback processing-chain. So this could be solved by re-initialising the chain with a different framerate.

My research brought up a deprecated mkv option TrackTimecodeScale http://matroska.org/technical/specs/index.html#TrackTimeCodeScale
So we cannot use this.
But as LAV Filters are able to calculate the amount of video frames to be duped or dropped when using a switch-stretched audio track, the same calculation could be used to determine a new video framerate, I assume.
Or at least, the calculation shoud be close enough to reduce drops/dupes to a minimum (E.g. one adjustment every 10 minutes).

I kindly ask to change the way LAV deals with switch-stretched audio to a complete framerate change instead of dropping/duping frames or at least adding a switch to let the user choose the behavior.

Cheers

[nevcairiel]

LAV just outputs the audio/video as they are in the file. I could explain more, but thats what it comes down to, and thats where it already ends. You cannot have two audio streams with different timings in the same file and magically expect it to work, it just won't, and it never will.
"stretch by" doesn't work with audio, it'll result in endless glitching. stretching only works with video or subtitles, however if you have two different audio timings, you can of course not sync to both at the same time, so what you are trying to do is just not going to work.

The only way to solve it is to timestretch one of the audio tracks and reencode it. There are high-quality time stretchers around that do not alter the pitch of the audio, so the quality loss is minimal.

[NikosD]

I did a quick test on DXVA copy-back between the last two versions 0.67 vs 0.64.

I used Core i7-4790 and iGPU HD 4600 in Playback Performance benchmark mode at 1280x720 on two HEVC files (1080p & 2160p)

The results are impressive.
Huge drop of CPU usage about 40% for 1080p with a slight increase in performance ~10%
Big drop of CPU usage about 30% for 4K with a significant 40% increase in performance!

DXVA native in Playback Performance uses more CPU than DXVA copy-back!

My question is if DXVA native could benefit from these optimizations, too.

[nevcairiel]

All I optimized is exclusively in use for CB (ie. the code that copies the video to system memory), Native doesn't use any of it.
I would be somewhat surprised if Native used more CPU, but maybe that CPU usage is in the renderer somewhere?

[detmek]

My CPU (Pentium G3220) does not support HEVC decoding but I tested H.264/720p file. DXVA Copy-back in 0.67 nightly LAV works much better compared to intel QuickSync decoding.
http://www.dodaj.rs/t/2s/pD/3AUxZ4Pc/lav-qs.jpg (http://www.dodaj.rs/?2s/pD/3AUxZ4Pc/lav-qs.png)http://www.dodaj.rs/t/31/mw/aIth3oa/lav-cb.jpg (http://www.dodaj.rs/?31/mw/aIth3oa/lav-cb.png)http://www.dodaj.rs/t/2X/MT/3LInFGNn/dec-lav-qs.jpg (http://www.dodaj.rs/?2X/MT/3LInFGNn/dec-lav-qs.png)http://www.dodaj.rs/t/30/8r/3yYjQB8Y/dec-lav-cb.jpg (http://www.dodaj.rs/?30/8r/3yYjQB8Y/dec-lav-cb.png)
Half the CPU usage with pure decoding and higher speed with lower CPU usage with rendering! Very nice. Thank you nev!

[stax76]

@nevcairiel

Showing the config dialog from the windows start menu result in a blurry mess on a High DPI system, it can be fixed with calling SetProcessDPIAware.