LAV Filters - DirectShow Media Splitter and Decoders

[Siso]

Thanks for the sample, I can reproduce the issue. Will take a look.

Edit:
Fixed

Thank you for your quick response.

[nevcairiel]

LAV Filters 0.80

LAV Splitter
- NEW: Introduced the IURLSourceFilterLAV interface to allow opening URLs with custom user agent and referrer
- NEW: Added support for WebP images
- Changed: Increased the length of the advanced subtitle selection field, so its no longer cut off after 255 characters
- Changed: Improved buffering behavior on badly interleaved video files
- Fixed: Audio streams with an unknown/unsupported codec are no longer selected for playback, as long as others are present
- Fixed: Improved accuracy of reported FPS from AviSynth scripts

LAV Video
- NEW: D3D11 support for HEVC 4:2:2 and 4:4:4 hardware decoding
- NEW: Dolby Vision extension metadata is exported for renderers to use
- Changed: Added additional media types to support more video streams
- Changed: Updated dav1d for significant AV1 decoding improvements
- Fixed: Improved handling of H.264 4:4:4 files encoded by certain versions of x264
- Fixed: VP9 DXVA2/D3D11 decoding could result in artifacts on some clips
- Fixed: Decoding ProRes reports more accurate color details

LAV Audio
- Changed: Added support for additional ADPCM audio codecs


Download: Installer (both x86/x64) (https://files.1f0.de/lavf/LAVFilters-0.80.exe) -- Zips: 32-bit (https://files.1f0.de/lavf/LAVFilters-0.80.zip) & 64-bit (https://files.1f0.de/lavf/LAVFilters-0.80-x64.zip)

After a long wait, finally new update.
Headline change in this update, for me, D3D11 HEVC 4:2:2/4:4:4 support. Both in copy-back, and in theory in native, although I'm not sure if any renderer supports native (yet).

I do not currently plan to bring that to DXVA2 as well due to certain limitations, but maybe that changes in the future.

As always, report issues and regressions, in as much detail as possible.

[lvqcl]

Thanks!
LAV Filters 0.80
Headline change in this update, for me, D3D11 HEVC 4:2:2/4:4:4 support. Both in copy-back, and in theory in native, although I'm not sure if any renderer supports native (yet).


It seems that MPC VR supports 4:4:4 native (maybe it also supports 4:2:2, but I cannot test it right now).

But: I can get HEVC 4:4:4 8-bit native decode only if I manually enable AYUV format in LAV Video settings. If it's disabled then I get copyback with YV24.

No problems with 10-bit and 12-bit HEVC.

[nevcairiel]

That is expected, because AYUV is the 4:4:4 8-bit hardware surface format for the decoder. AYUV caused some issues in the past with some renderers, but maybe it should be enabled by now.

Note that 4:2:2 >8-bit native is not currently supported because LAV does not support Y210/Y216 output formats natively. I should probably add it at some point.

[Blight]

nev:
Is there a way to control the timeout on network connections? (.ts / .m3u8 streams)

I noticed that with some bad connections, the timeout can be ~20seconds, it would be nice to be able to control the timeout value.

[tv-viewer]

The librempeg (https://github.com/librempeg/librempeg) fork of ffmpeg has added support for a number of codecs not supported in mainline ffmpeg, most notable to me being AC-4 due to its use in ATSC 3.0 broadcasts in the USA, Korea, and elsewhere. Can AC-4 support be added to LAV Filters?

[v0lt]

most notable to me being AC-4 due to its use in ATSC 3.0 broadcasts in the USA, Korea, and elsewhere.
Please provide recording files of real broadcasts with AC-4 audio (not tests, not demos).

[tv-viewer]

Please provide recording files of real broadcasts with AC-4 audio (not tests, not demos).
Happy to. Is there a preferred place to upload them?

[v0lt]

Happy to. Is there a preferred place to upload them?
drive.google.com, mega.io, www.dropbox.com, www.mediafire.com and other services.

[ipanema]

If I connect a custom DirectShow filter that only accepts samples in RGB format to the output of the LAV video decoder when D3D11 hardware acceleration is selected in the settings, will it output the RGB frames in GPU memory?

I know that the custom filter's input pin must also implement ID3D11DecoderConfiguration to receive samples in GPU memory.

But reading the source code suggests that only NV12 and some other YUV pixel formats can be output in GPU memory - for example in d3d11va.cpp we have:

CMediaType mt = m_pCallback->GetOutputMediaType();
if ((m_SurfaceFormat == DXGI_FORMAT_NV12 && mt.subtype != MEDIASUBTYPE_NV12) ||
(m_SurfaceFormat == DXGI_FORMAT_P010 && mt.subtype != MEDIASUBTYPE_P010) ||
(m_SurfaceFormat == DXGI_FORMAT_P016 && mt.subtype != MEDIASUBTYPE_P016) ||
(m_SurfaceFormat == DXGI_FORMAT_AYUV && mt.subtype != MEDIASUBTYPE_AYUV) ||
(m_SurfaceFormat == DXGI_FORMAT_Y410 && mt.subtype != MEDIASUBTYPE_Y410) ||
(m_SurfaceFormat == DXGI_FORMAT_Y416 && mt.subtype != MEDIASUBTYPE_Y416) ||
(m_SurfaceFormat == DXGI_FORMAT_YUY2 && mt.subtype != MEDIASUBTYPE_YUY2) ||
(m_SurfaceFormat == DXGI_FORMAT_Y210 && mt.subtype != MEDIASUBTYPE_Y210) ||
(m_SurfaceFormat == DXGI_FORMAT_Y216 && mt.subtype != MEDIASUBTYPE_Y216))
{
DbgLog((LOG_ERROR, 10, L"-> Connection is not the appropriate pixel format for D3D11 Native"));

SafeRelease(&pD3D11DecoderConfiguration);
}


So is LAV video decoder only capable of outputting frames in RGB format in CPU (not GPU) memory?

[nevcairiel]

So is LAV video decoder only capable of outputting frames in RGB format in CPU (not GPU) memory?

LAV Video can only output hardware frames in the same format as the decoder outputs them. First it would depend on the video you are decoding, as that would dictate the format, and secondly, RGB is not among those. Even if a video was RGB (eg. HEVC 4:4:4 in RGB), it would come out as AYUV/Y410 with a RGB color matrix instead.

LAV can certainly not convert any video to RGB as a hardware surface. If you require that, its on you to make happen.

[ipanema]

Thanks for confirming that. Sounds like the custom filter needs to accept nv12 to receive it in GPU memory, then use a shader to convert it to RGB, still in GPU memory.

[ipanema]

Just one more quick confirmation...

If the LAV decoder is configured to use D3D11, and it is outputting RGB (either because it is the only output format selected in the config window, or the downstream filter will only accept RGB), it will be outputting in CPU memory BUT the actual decoding will still be done in hardware (not using the software decoder)?

(So LAV will convert the decoded frames from NV12 in GPU memory to RGB in CPU memory for output.)

[clsid]

Yes, copyback mode can output/convert in all colorspace in cpu mem.

What does your custom filter do? Ideally it should support nv12/p010/etc.

[tv-viewer]

Please provide recording files of real broadcasts with AC-4 audio (not tests, not demos).
Couple of real ATSC 3.0 broadcasts from HDHomeRun:
WJCT-DT https://www.dropbox.com/scl/fi/8hcfd5hscr8cmd73gdpeq/WJCT.ts?rlkey=3fz6xh5n9cnnu57ql7ldwmgye&st=gdq04678&dl=0
KDAF-DT https://www.dropbox.com/scl/fi/63knsuhsdocns60oo82q0/KDAF.ts?rlkey=spa9qtat1n0dw1mdi8wa6klac&st=g9xho4xg&dl=0

[v0lt]

Thanks.
Checked in MPC-BE - plays.
Sent to ticket #8349 (FFmpeg) (https://trac.ffmpeg.org/ticket/8349#comment:96).

[ipanema]

Thanks - OK, think I've got it now.

Eventually hoping the filter will write text or other graphics directly onto the surface/texture in GPU memory - but it's early days so far! Still quite a bit to learn. Thanks again.

[mclingo]

I haven't got much DTSX material So I'm not sure how long I've had this issue when I play Dolby Atmos it correctly plays all the sounds on the correct speakers I have 5.1 .2 which is a 5.1 standard setup with two front height speakers. when I play DTSX it correctly plays the rear height out of the front heights But plays the front heights out of left and right where it should be summing both heights to the front heights, This works correctly in atmos so it can't be an issue with lav setup?

i have a yamaha V685 anyone any ideas

[tebasuna51]

LAV filters can't decode properly 3D audio from DTSX or Atmos. You need a proprietary decoder to do so (only Cavernize can decode EAC3 Atmos).
To obtain the 3D audio you need to tell the player to pass the Atmos or DTSX directly to your Yamaha for decoding.
Therefore, it can never be a problem with the LAV filters.

What is your player? Verify if have passtrough (bitstreaming HDMI) for audio TrueHD, Eac3, DTS...

[mclingo]

you can't get any height audio without using pass through, as atmos is perfectly fine it should tell you I'm already using bit streaming, this is just dts-x issue as far as I can tell

[mclingo]

I've actually had some progress with this using chat GPT Of all things, it told me to check what my front height present speakers were set to on my AVR which I just assumed was set correctly because atmos was working, however they were set to "overhead" and not "height" When they were set to it "height" it sent my front height audio out of my height speakers however it sent my rear height information out of my rear speakers, but as I don't have any rear heights having it come out of the might be better than come out of the front heights anyway.

I'm still not sure why it's not summing all high audio information to my front heights as my AVR is definitely set up as 5.1.2

[tebasuna51]

If you use always passtrough the problem concern always to your AVR YAMAHA RX-V685, maybe you can open other thread out off this LAV filter thread.

I have a Denon AVR and I can't help you with your Yamaha, sorry.

[mclingo]

out of interest, in LAV audio options, what should be ticked their for my setup, 5.1.2 using front heights to sum all height audio - I currently have expand 6.1 to 7.1 ticked, use legacy 5.1 layout ticked and convert to standard channels is unticked. Thoughts?

[huhn]

only matters for PCM if you are bit streaming and rare formats that may not be supported.

you have to know if you have the legacy 5.1 or "modern" 5.1 as a base layout.

[mclingo]

thanks for the info, not sure what you mean by modern or base, my AVR is set to BASIC which is 5.1.2 front height presence, maybe its normal behaviour, maybe ATMOS sums all height audio to whatever height speakers you have setup and DTS sends front height audio to front and if you dont have back heights it sends them to back surround, kind of makes sense, just thought someone might know if that was the case.

[tebasuna51]

out of interest, in LAV audio options, what should be ticked their for my setup, 5.1.2 using front heights to sum all height audio - I currently have expand 6.1 to 7.1 ticked, use legacy 5.1 layout ticked and convert to standard channels is unticked. Thoughts?
As I mentioned before, this is excluded from LAV filters when the source is Atmos or DTSX audio, as they cannot be properly decoded.

However, if these formats have been previously decoded to PCM 5.1.2, as I explained in Surround 3D (https://forum.doom9.org/showthread.php?t=186285) (I don't know how for DTSX), it would be interesting to investigate how a player that uses LAV filters can send that PCM to an AVR.

In my specific case, a Denon AVR, I was only able to achieve this with a trick (https://forum.doom9.org/showthread.php?p=2017560#post2017560), as any configuration I used failed to recognize the high front channels.

[mclingo]

right got you, I think I have it setup now with something I can live with, not much DTS in my collection anyway so great, thanks.

[clsid]

MPC-HC audio switcher has a channel mapping feature that you could use to redirect those two channels.

[mclingo]

that would only work with PCM surely though?

[clsid]

Of course.

When bitstreaming, the audio stream is send in its original compressed form to the receiver, and the receiver is completely and totally responsible for doing ALL decoding / processing / channel mixing / volume control. No exceptions. On the PC side the audio can not be manipulated at all (unless it is transcoded).

[tebasuna51]

that would only work with PCM surely though?
Like I say in my thread:

"This way, whether mpc-hc decodes a correct EAC3/m4a/flac 5.1.2 or an AAC/Opus 7.1 with inverted 5.1.2 channels, the result will be the desired one: hearing the top channels through the fake back channels."

LAV filters can decode previously (if EAC3 you need unmark the bitstreaming for it) and after the AudioSwitcher do the job.
I recommend m4a generated by ffmpeg with libfdk

[Kurt.noise]

Not sure if this is the right place to ask new request but is it possible to have the APV codec support since its officially available through FFmpeg 8.0 release ?

:thanks:

[v0lt]

Not sure if this is the right place to ask new request but is it possible to have the APV codec support since its officially available through FFmpeg 8.0 release ?
1. First, tell us what you need this codec for? :rolleyes:
2. Where is this codec used?
3. Please provide samples with this codec. Files should not be made in FFmpeg.

[Kurt.noise]

1. First, tell us what you need this codec for? :rolleyes:
why not ? why we have RealVideo, FLV1, VP3 in there ? Noone use them nowadays :rolleyes:
2. Where is this codec used?
All Android 16 devices (https://www.androidauthority.com/android-16-beta-1-3518608/) + Samsung stuff, mpv and related players, not enough for you ?
3. Please provide samples with this codec. Files should not be made in FFmpeg.
https://forum.doom9.org/showthread.php?t=185868

[v0lt]

why not ? why we have RealVideo, FLV1, VP3 in there ? People have old video files. :rolleyes:
People have old video files.
All Android 16 devices + Samsung stuff, mpv and related players, not enough for you ?
This does not answer the question of what this codec is used for.
https://forum.doom9.org/showthread.php?t=185868
There are no samples except one test file encoded with FFmpeg.

[LigH]

The media-autobuild suite added support for this high quality high resolution intermediate codec, similar to Apple ProRes.

The APV codec is a professional video codec, which was developed in response to the need for professional level high quality video recording and post production. The primary purpose of the APV codec is for use in professional video recording and editing workflows for various types of content.

https://github.com/AcademySoftwareFoundation/openapv (raw samples in test subdir)

[Blight]

Looks like ffmpeg v8 supports whisper AI's voice to subtitle.

It would be very cool if there was an setting to enable/disable support for it and have the subs exposed as an embedded sub pin.

[Kurt.noise]

Not sure if this is the right place to ask new request but is it possible to have the APV codec support since its officially available through FFmpeg 8.0 release ?

:thanks:
Looks like there is only the encoder at the moment. So, forget what I said.

[LigH]

No, OpenAPV is available as both encoder and decoder.

ffmpeg -codecs | find "apv"
DEVIL. apv Advanced Professional Video (encoders: liboapv)

in my build of today, made by M-AB-S.

[v0lt]

What professional software (except FFmpeg) uses the "Advanced Professional Video" codec? :D

[LigH]

Hey ... how long did it take for "professional software" to support the Matroska container after it was just even introduced in the freeware video scene?

AVP is free and it is useful. Shall it be ignored only because Apple ProRes is more widely used right now? How much trouble is adding it, even?

[v0lt]

AVP is free and it is useful. Shall it be ignored only because Apple ProRes is more widely used right now? How much trouble is adding it, even?
I think the first steps for a professional codec should be from the content creation software side (Kdenlive, OpenShot, ...). Then there will be more videos and support in players.

APV is only introduced in the new version of FFmpeg 8.0. There is simply no other information available right now.

This is a bit reminiscent of the situation with the Essential Video Coding (EVC) codec. There was a lot of noise, but video files with this codec never became widely available. I have never come across it at all.

[LigH]

As most OpenSource video editors use the ffmpeg core, it is quite probable that they will support AVP as soon as they update their core to version 8.0 of ffmpeg. I will not install them to look into them for you, though. Just reporting some information from their websites (which may not be up to date until someone edits them):

Kdenlive:
Support for high (HD) and ultra-high (UHD/4K) resolution camcorder files, including: MPEG-2, MP4, Matroska, WebM, Apple ProRes 422 HDV; H264 AAC HDV ...
Export to any format supported by ffmpeg, including DVD (PAL and NTSC), MPEG-2 and MP4 (PAL, NTSC and HD, UHD/4K), Matroska-H264/H265, AAC, AC3, MP3, Apple ProRes 422

OpenShot:
Based on the powerful FFmpeg library, OpenShot can read and write most video and image formats. For a full list of supported formats, see the FFmpeg project. OpenShot's export dialog defaults to some of the more common formats, but with the advanced tab, you can use any FFmpeg format.

If anyone here actively uses any of them, please look into codec lists of the most recent versions and report here.

[Sunspark]

Sure, this second APV isn't needed, but it should be allowed later.

This is from the Android 16 release notes:

"In Android 16, we're launching platform support for the Advanced Professional Video (APV) codec. The APV codec is a high-bitrate intra-frame codec designed to give content creators top-quality captures and edits.

Also, Google plans to transition all users from VP8, VP9, and AVC (H.264) to AV1. App developers prefer AV1, the next generation of codecs, to avoid transcodes in the backend and reduce latency. Hardware codecs are still recommended, especially for encoding, despite the fact that AV1 software codec support is improving.

You can adopt AV1 for a higher quality, reliability, and concurrency and consider APV support in the Camera and Gallery app."

[nevcairiel]

Since APV is supported in FFmpeg without the need of an external library, its not even any real effort to support and will be added once I update FFmpeg the next time soon.

[Orf]

nevcairiel,
In a custom renderer I'm trying to implement support for getting D3D11 texture samples from LAV video decoder directly. I'm utilizing your ID3D11DecoderConfiguration and IMediaSampleD3D11 interfaces for that purpose.

Connection media subtype NV12 is used and LAV Video decoder correctly calls GetD3D11AdapterIndex (I return 0) and
ActivateD3D11Decoding so I have d3d11 interfaces. At the point the DS graph is built LAV video decoder reports 'd3d11 native' is used.

But at point when first sample arrived LAV video decoder get switched to software decoding and first sample is never IMediaSampleD3D11 but IMediaSample instead. At this time it also reports 'avcodec' is used.

I try to investigate why it happens and found that it is not hardware related because:
1) if I return error from ActivateD3D11Decoding LAV video decoder switched to 'd3d11 cb native' and it works fine
2) if I build graph with madVR as a renderer at first and then disconnect LAV Video decoder from madVR and connect it to my renderer 'd3d11 native' starts work with my renderer just fine too

I seems to me that LAV video decoder expects from renderer to do something else, which madVR does and my renderer doesn't.
I try to find what it might be examining LAV source code but didn't find anything that gives me a clue.

So I need your favour with that, can you please shed some light on this?

[nevcairiel]

One thing to check might be that in HW native mode, LAV needs to provide the sample allocator - so if such a mode is enabled, you should not provide an allocator from the renderer, and let LAV provide it.

[lvqcl]

No latest nightly build?..

[Orf]

One thing to check might be that in HW native mode, LAV needs to provide the sample allocator - so if such a mode is enabled, you should not provide an allocator from the renderer, and let LAV provide it.
You are right, thank you for your help. Manage to get it work after returning E_FAIL from input pin GetAllocator method, but now I'm wonder is it better to always let LAV provide an allocator if currently I'm using nothing more then the default CBaseInputPin GetAllocator implementation ?

[nevcairiel]

For software input most renderers will try to provide a certain stride thats ideal for their purposes, for which you need to control the allocator - if there is no need to do so, you can just let LAV provide one.