LAV Filters - DirectShow Media Splitter and Decoders

[nevcairiel]

Hi nevcairiel.
With the new Matroska spec (though it's not implemented in FFmpeg, the code from Plex's patches is available), Dolby Vision metadata can be inserted and decoded from MKV headers.
Would it be possible to parse the HEVC RPU NAL units and make that information available to the renderer (for example, madVR) through the LAV decoder?
HW acceleration might cause issues, though.

I have working code to parse this RPU data (at least for profile 7 files), but I'm not sure if it's feasible for that metadata to be useful.
I think it would be nice if the renderer could have access to pre defined per-frame information about min, max, average luminance and etc.

Just wondering if it's something that would be possible, because I'd be interested in helping to add support.

I wouldn't mind supporting that, but getting a renderer to support it is the bigger problem here, as madVR seems to be stuck.

[SamuriHL]

Which is a shame because everything else is in place to play them. It would be really nice to have that support on a PC.

[JarrettH]

Will there be further updates?

[VBB]

See here for the most up-to-date builds: https://files.1f0.de/lavf/nightly/?C=M;O=A

Here for commit/change log: https://github.com/Nevcairiel/LAVFilters/commits/master

[quietvoid]

Yea I think getting madshi to add support would be complicated.
Though NVIDIA seems to have APIs to enable the Dolby Vision playback chain on a display, no idea how the data is actually sent through.
It might just be the whole RPU, since the decoder is expected to verify the integrity of it.

I didn't really think this through.. maybe once people start sending patches to ffmpeg, as it appears Plex's code is incomplete.

[vosya]

In version LAVFilters-0.74.1-91, LAV Splitter has stopped detecting the presence of an audio stream with the following characteristics in m2ts, mkv video containers (possibly in others):
Audio
ID : 4352 (0x1100)
Menu ID : 1 (0x1)
Format : PCM
Format settings : Big / Signed
Muxing mode : Blu-ray
Codec ID : 128
Duration : 29 min 9 s
Bit rate mode : Constant
Bit rate : 2 304 kb/s
Channel(s) : 2 channels
Channel layout : L R
Sampling rate : 48.0 kHz
Bit depth : 24 bits
In version LAVFilters-0.74.1-90, such streams are recognized normally.

[nevcairiel]

In version LAVFilters-0.74.1-91, LAV Splitter has stopped detecting the presence of an audio stream with the following characteristics in m2ts, mkv video containers (possibly in others):
Audio
ID : 4352 (0x1100)
Menu ID : 1 (0x1)
Format : PCM
Format settings : Big / Signed
Muxing mode : Blu-ray
Codec ID : 128
Duration : 29 min 9 s
Bit rate mode : Constant
Bit rate : 2 304 kb/s
Channel(s) : 2 channels
Channel layout : L R
Sampling rate : 48.0 kHz
Bit depth : 24 bits
In version LAVFilters-0.74.1-90, such streams are recognized normally.

Will be fixed soon, thanks!

[Brazil2]

LAV Filters 0.74.1 (released 2019/03/19)
Please, make a new release.
Many people don't like to use beta versions and many sites won't spread beta versions as well.

[ryrynz]

Please, make a new release.


Just a wild guess here but I expect Nev would've released something this month if not early next anyway..
Nothing is stopping anyone using nightlies until then as these releases are essentially ongoing stable builds, a release build for LAV is no different from the nightlies other than in name.

[manolito]

nevcairiel thinks otherwise... :D
https://forum.doom9.org/showthread.php?p=1833908#post1833908

[ryrynz]

Bruh, that was 2018, things have changed :P

[el Filou]

There have been quite a lot of changes recently so I would assume you have to wait a bit for a release just to make sure everytrhing is OK. Wasn't there issues with hardware AV1 decode for instance?
Not useful to make a release if you've got to provide a hotfix quickly afterwards.

[clsid]

This 1440p VP9 Profile2 stream has massive frames drops with CUVID decoding. DXVA2 works fine.
https://www.youtube.com/watch?v=n3Dru5y3ROc
Played in MPC-HC with youtube-dl

[nevcairiel]

As i've stated recently, I plan to make a release when AV1 is done and some additional fixes are in.

This 1440p VP9 Profile2 stream has massive frames drops with CUVID decoding. DXVA2 works fine.
https://www.youtube.com/watch?v=n3Dru5y3ROc
Played in MPC-HC with youtube-dl
CUVID is not currently being developed further.

[clsid]

May I suggest that you finally remove the obsolete and outdated QuickSync in upcoming release? And migrate existing setting to DXVA2 copyback.
And move CUVID to bottom of the dropdown list and maybe rename it to "CUVID (deprecated)". By not having the vendor name in front of the option should make some people less tempted to select the inferior choices.

[huhn]

yeah for some reason people like to say i selected cuvid because i have a nvidia card.

[VBB]

@clsid - Since you're here, did you recently increase the default built-in LAV stream analysis duration from 1000 to 2100? I only found this, because MPC-HC did a full reset for some reason, and that particular setting seems to have fixed my lossless audio dropouts on resume :)

[clsid]

Yes, that fixes several (audio) issues. So maybe nev can change the default in the official builds as well.

[nevcairiel]

The only thing that option is supposed to affect is detecting of streams in header-less containers. If it has any impact on actual playback, please give me a case to reproduce.

[clsid]

Yes, I changed it for solving issues with certain online streams. I don't have a working example for you.
I can't recall it having effect on local files. VBB's issue probably got solved by something else.

[VBB]

Well, it seems to make a difference when pausing files with lossless audio. Perhaps clsid can explain it some more. After all, he made the change ;)

Or not LOL

I'm just happy it works now, regardless of what fixed it. If it makes any difference, none of my files are local, so in essence everything is streamed.

[Snowknight26]

Well, it seems to make a difference when pausing files with lossless audio. Perhaps clsid can explain it some more. After all, he made the change ;)

Is this the issue you mean?

https://forum.doom9.org/showthread.php?p=1914474#post1914474

[VBB]

Yes, that's the one.

[chros]

@nevcairiel, does LAVVideo decoder support HLG HDR? Sample (https://4kmedia.org/travelxp-4k-hdr-hlg-sample)
The output PIN info states: "VideoTransferFunction : 5 (BT.709)"
I'm asking because of madvr and this (https://forum.doom9.org/showthread.php?p=1930606#post1930606).
Cheers

[huhn]

try with mpcVR is does it doesn't send the metadata to madVR that's the current situation. can't handle it anyway.

[nevcairiel]

Indeed, madVR does not support HLG, and the HLG VideoTransferFunction function was used for a special madVR value in the past, hence with madVR it maps it to another format. Other renderers should receive the HLG flag (16), and they can then deal with it.
Sending the HLG flag to madVR would not give you anything good until madVR learns to deal with HLG. HLG is very different from HDR10, so no just masquerading as the same and somehow work.

[chros]

Thanks for the confirmation, I missed the part somehow that it's working with e.g. mpcVR (otherwise I wouldn't have asked it), sorry :o
Do you know any directshow renderer that can output HLG in HDR10 properly? Just noticed that e.g. VLC does (https://forum.doom9.org/showthread.php?p=1930642#post1930642) ...

[Grimsdyke]

@Nev, sorry to bother you but it would be nice if you could look into something.

As you may have heard the latest test build of MadVR caused playback issues for few users but an OS and/or driver update fixed it for most of them.
Unfortunately I still have the issue and I blamed my hardware and waited for a fix from Madshi.
However, the devs of MPC-BE are currently improving their own renderer and I wanted to look at it and was surprised that there is the same problem !

Issue : A file plays with sound but there is no image just a green (madvr) or black (mpcvr) screen when using LAV video decoder in D3D11 Native mode !

It works with every other setting but then I have performance problems, especially on 4K HDR. Is there maybe anything broken in native ?

[chros]

Which GPU is that and which driver? Take a look at my signature, those work fine.

[jkauff]

I'm running the latest test build (114) and have had no problems on either of my computers. One has a GTX 1660 Super, the other a GTX 1080. Using the most recent Nvidia driver, as I always do.

[Yups]

Tigerlake AV1: https://imgur.com/a/SR95o3C

D3D11 AV1 hardware decoding works, DXVA2 is corrupted. Tested on newest LAV and MPC HC.

[VictorLS]

Tigerlake AV1
D3D11 AV1 hardware decoding works, DXVA2 is corrupted.
Thanks - can you show DXVA Checker of Tigerlake like
https://i.postimg.cc/fV49HXZb/AMD-RX6800-DXVAchecker.png (https://postimg.cc/fV49HXZb)

[Yups]

Thanks - can you show DXVA Checker of Tigerlake like
https://i.postimg.cc/fV49HXZb/AMD-RX6800-DXVAchecker.png (https://postimg.cc/fV49HXZb)


I already did here: https://forum.doom9.org/showthread.php?p=1929258#post1929258

For some reason I cannot check D3D11 decoding performance from LAV in DXVA Checker: https://imgur.com/a/T1uJqZf

On a further note the inhouse Windows 10 player with the Microsoft AV1 extension works better, there is basically zero CPU and GPU load while playback (with Firefox running background): https://imgur.com/a/nKGVknN


And this is from MPC Home Cinema and LAV D3D11: https://imgur.com/a/zh0ELEC

1W more IA cores usage and it uses the GPU in addition to Video Decode and Processing.

I'm using the AV1 4K Summer Nature sample from here: https://www.elecard.com/videos

[nevcairiel]

DirectShow does not natively support D3D11, so we invented our own interface to do it with reduced overhead. DXVA Checker does not understand that. Neither do most video renderers. If you want to test full native D3D11 playback, you need to use madVR, or maybe mpcVR, which I believe supports that as well (none of which come natively with MPC-HC)
Of course you would have to make sure that the renderer then doesn't use additional resources, for a fair comparison.

DXVA2 not working sounds like a driver issue to me, the data being provided to the hardware is the exact same, even generated by the same code, for both DXVA2 and D3D11. Of course I cannot tell either way without access to hardware, and until cheap-ish NUCs come out, thats out of the question.

[Yups]

DXVA2 playback corruption is indeed a driver issue. I switched from 9077 back to 8853 and the issue is gone.

[Yups]

Unfortunately yes, 9030/9039/9077 same issue. The last working is 8935 which I will use for now.

[VictorLS]

back to 8853 and the issue is gone...last working is 8935 which I will use for now
Great job. Many thanks for DXVA Checker screenshot.
Can you confirm Intel QuickSync, DXVA2 and D3D11 hardware acceleration with TigerLake and LAV Video Decoder 0.74.1.92 while playing recorded from SAT feed 4:2:2 h265 4K 50fps file 20180428-175510_4K ENC 3 RMAD VS LEG.ts (563 MB) https://yadi.sk/d/m0s5ztpj3XnUyw and youtube 8K AV1 60fps file you can download I've described here https://forum.doom9.org/showthread.php?p=1903584#post1903584
Please provide CPU and VideoEngine load while playing both files.

As I understand there isn't Win7x64 driver for Intel Iris Xe graphics - am I wrong?

[nevcairiel]

There is no 4:2:2 hardware acceleration in DXVA2 or D3D11, and QuickSync in LAV is dead and likely to be removed.

[VictorLS]

There is no 4:2:2 hardware acceleration in DXVA2 or D3D11
Even if it's supported by Intel Xe? If so add it in LAV Video Decoder, please.
https://i.postimg.cc/2V9MwFjv/Intel-Iris-Xe-DXVAchecker-422.png (https://postimg.cc/2V9MwFjv)

[Yups]

Great job. Many thanks for DXVA Checker screenshot.
Can you confirm Intel QuickSync, DXVA2 and D3D11 hardware acceleration with TigerLake and LAV Video Decoder 0.74.1.92 while playing recorded from SAT feed 4:2:2 h265 4K 50fps file 20180428-175510_4K ENC 3 RMAD VS LEG.ts (563 MB) https://yadi.sk/d/m0s5ztpj3XnUyw and youtube 8K AV1 60fps file you can download I've described here https://forum.doom9.org/showthread.php?p=1903584#post1903584
Please provide CPU and VideoEngine load while playing both files.

As I understand there isn't Win7x64 driver for Intel Iris Xe graphics - am I wrong?


LAV doesn not support 4:2:2 by the looks of it and your AV1 sample is 4K only, not sure this is what you ask for. Intel does not support Windows 7 anymore.

[el Filou]

Yeah those are Intel-specific modes and not standard, just like NVIDIA has an MJPEG DXVA mode (even up to 4K) that only their own decoder uses. Unfortunately now most decoders provided by manufacturers are Media Foundation only.

[VictorLS]

el Filou
Thanks for explanation but I'm sure in case of h265 4:4:4 hardware acceleration on Turing and Ampere videocards works with LAV Video Decoder - it's a pity nVIDIA don't listen me and haven't implemented h265 4:2:2 support yet so I'll don't going to buy any new videocard in nearest future ;)
your AV1 sample is 4K only, not sure this is what you ask for
youtube-dl -F gives
571 mp4 7680x4320 4320p60 HDR 36464k , av01.0.17M.10.0.110.09.16.09.0, 60fps, video only, 831.39MiB
and MediaInfo confirm that:
General
Complete name : C:\Fireworks 2019 Hibikiya Omagari Enka [8KHDR]-d1VF4DfBPOY.mp4
Format : dash
Codec ID : dash (iso6/av01/mp41)
File size : 831 MiB
Duration : 3 min 30 s
Overall bit rate : 33.1 Mb/s
Encoded date : UTC 2019-12-24 16:03:06
Tagged date : UTC 2019-12-24 16:03:06

Video
ID : 1
Format : AV1
Format/Info : AOMedia Video 1
Format profile : Main@L6.1
HDR format : SMPTE ST 2086, HDR10 compatible
Codec ID : av01
Duration : 3 min 30 s
Bit rate : 33.1 Mb/s
Width : 7 680 pixels
Height : 4 320 pixels
Display aspect ratio : 16:9
Frame rate mode : Constant
Frame rate : 59.940 (60000/1001) FPS
Color space : YUV
Chroma subsampling : 4:2:0
Bit depth : 10 bits
Bits/(Pixel*Frame) : 0.017
Stream size : 831 MiB (100%)
Title : ISO Media file produced by Google Inc.
Encoded date : UTC 2019-12-24 16:03:06
Tagged date : UTC 2019-12-24 16:03:06
Color range : Limited
Color primaries : BT.2020
Transfer characteristics : PQ
Matrix coefficients : BT.2020 non-constant
Mastering display color primaries : Display P3
Mastering display luminance : min: 0.0050 cd/m2, max: 1000 cd/m2
Codec configuration box : av1C

[nevcairiel]

Even if it's supported by Intel Xe? If so add it in LAV Video Decoder, please.

This is not part of the DXVA specification, and Intel does not provide any documentation or works with Microsoft to update the proper one.

[Yups]

@VictorLS

I have tried this 8K60 sample. Unfortunately I have a black screen when I try this video in MPC-HC which seems unrelated to LAV or decoding in general, even without hardware decoding I have a black screen. Could it be HDR related?

The Microsoft mediaplayer app works and it runs without any frame drops from what I can see and keep in mind I'm running this 8K video on a 1080p screen, not sure if this makes a difference. The decoding engine runs at about 75% utilization and clocks with 1100 Mhz. Here the screen: https://imgur.com/a/BSFThv5

[Yups]

I have tried a non HDR 8K60 AV1, the problem is that in MPC HC with LAV it runs in slow motion and stutters. Oddly enough there is no visible hardware limitation, Video Decode about 35-40%, GPU usage 10%, CPU usage 10%. In Chrome directly or with the Microsoft player app no problem. GPU decode usage goes higher on them, it's like they can better utilize the GPU.

[el Filou]

Is the DXVA decoder using native mode?

[Yups]

Yes it was running DXVA2 native. I just tried D3D11 and it runs quite a lot better. The slow motion/severe stuttering is gone, however there are still visible frame drops. The load differs, there is a 50% GPU load. GPU decoding is the same at 35-40%. CPU usage is down from 10% to 3%.

[nevcairiel]

The DXVA2 interface and DirectShow are not well suited for modern stuff, unfortunately. For example, DXVA2 does not even specify any way to apply Film Grain for AV1. It works on NVIDIA because it applies it magically on its own, but other vendors might not do that. This is specified for D3D11 and D3D12 video decoding.

I have been thinking about Media Foundation for a while, but I'm not going to create a player, filters, and renderer on my own, and there arent really many other people left.
Someone got a chicken to my egg?

[VictorLS]

This is not part of the DXVA specification, and Intel does not provide any documentation or works with Microsoft to update the proper one...The DXVA2 interface and DirectShow are not well suited for modern stuff, unfortunately...I have been thinking about Media Foundation for a while...
Thanks - it seems Intel don't want I'll buy their Xe videocard ;) due to unsupported 4:2:2 by excellent software like LAV Video Decoder.
I'm not going to create a player, filters, and renderer on my own
BTW have you tried MPC Video Renderer by MPC-BE developers team? If yes what do you think? I'm impressed with HLG support and fluent playing of MPCVideoRenderer-v0.4.10.1542_git2020.12.15-45a2b1c (630 KB) https://yadi.sk/d/X0EVMKP4TcmnHQ but while playing http://forum.doom9.org/showthread.php?p=1919134#post1919134 4:2:2 h264 1080i50 BT.709 file from SAT feed Formula1.2019.Round06.Monaco.Race.Sat.Feed.1080i.H264.Multi.Language_fromMKVtsMuxerNotHDMVcompatible.ts (501 MB) https://yadi.sk/i/X1nPqEAqIAZXoQ I've got non-deinterlaced video until I uncheck all 4:2:2 Output Formats but YUY2 see MPC-VR_screenshots.7z (10 MB) https://yadi.sk/d/M23aZqcB66YGSw


I have tried this 8K60 sample. Unfortunately I have a black screen when I try this video in MPC-HC which seems unrelated to LAV or decoding in general, even without hardware decoding I have a black screen.
Many thanks for your great experiments and screenshots (just add Win10's 3D and Video Decode graphs in a future, please).
Could it be HDR related?
I don't know any AV1 8K60 without HDR(HLG) but I believe on 8K24 PQ Samsung QLED 8K HDR NASA 1minutes49seconds
youtube-dl -f 402 https://youtu.be/N8q7iTw14O8 mp4 7680x4320 4320p HDR 22607k , av01.0.16M.10.0.110.09.16.09.0, 24fps, video only, 151.81MiB
youtube-dl -f 571 https://youtu.be/N8q7iTw14O8 mp4 7680x4320 4320p HDR 32602k , av01.0.16M.10.0.110.09.16.09.0, 24fps, video only, 235.29MiB
and 8K30 HLG Go Central Japan 8K 3minutes55seconds
youtube-dl -f 402 https://youtu.be/XsVGYfsb3sI mp4 7680x4320 4320p HDR 19517k , av01.0.16M.10.0.110.09.18.09.0, 30fps, video only, 454.81MiB
youtube-dl -f 571 https://youtu.be/XsVGYfsb3sI mp4 7680x4320 4320p HDR 30943k , av01.0.16M.10.0.110.09.18.09.0, 30fps, video only, 708.09MiB
files (402 and 571 differ by bandwidth) will palying fluent with D3D11 and even DXVA2 (native) on your TigerLake.

The Microsoft mediaplayer app works and it runs without any frame drops
Despite of 'Intel does not provide any documentation or works with Microsoft' try 4:2:2 h265 4K 50fps SAT feed file 20180428-175510_4K ENC 3 RMAD VS LEG.ts (563 MB) https://yadi.sk/d/m0s5ztpj3XnUyw in Microsoft mediaplayer, please and look at Video Decode - may be it'll be not zero.

[Yups]

Many thanks for your great experiments and screenshots (just add Win10's 3D and Video Decode graphs in a future, please).


You can see these infos from HWinfo and actually HWinfo is more detailed.


Despite of 'Intel does not provide any documentation or works with Microsoft' try 4:2:2 h265 4K 50fps SAT feed file 20180428-175510_4K ENC 3 RMAD VS LEG.ts (563 MB) https://yadi.sk/d/m0s5ztpj3XnUyw in Microsoft mediaplayer, please and look at Video Decode - may be it'll be not zero.


Isn't it the same video from yesterday? I already did test it with the Windows 10 movie app player in #24187 and yes it runs, although it is hard on the limit I would say. Considering the GPU runs at only 1100 Mhz it's a really good result.
As for local AV1 playback on Iris Xe this player is the fastest especially for a notebook, the system power is lower which is really important. This player is super fast and efficient.