BestSource (BS)

[Myrsloik]

The time has finally come for a universal source filter that doesn't suck. FFMS2 was good back when only B-frames roamed the earth but now we have "super totally keyframes (IDR)", "referenced but hidden frames (VPx)" and "just doesn't work because FFmpeg internals changed frames". Insanity!

BestSource solves all of these issues by only performing linear decoding. Plus some tricks with multiple decoder instances and a large cache to hide some of the potentially slow seeking. Throwing more hardware at a problem does help. Especially if you have 16 cores.

Indexing files to get exact sample and frame counts is enabled by default but can be turned off by settings exact=False to speed up file opening. All indexing information is also stored in a single json file to not gunk up your pristine filesystems.

Indexless mode generally seems to works fairly well for avi and mp4 files. Do however note that nvops are dropped so some avi files will turn into VFR.

64 bit and VapourSynth only. BestAudioSource is now discontinued.

Usage:
core.bs.VideoSource(source="meh.mkv")
core.bs.AudioSource(source="meh.mkv")

[kedautinh12]

Can it's support avs too??

[mastrboy]

Very interesting, not very often we get new source filters to play with.

Have you done any scale testing with a large json file? If you plan to store every file's index in the same file it will grow pretty quick. (And json parsing is more CPU expensive than one would expect)
How many movies can be indexed before the json file gets so big it starts to have adverse effects on opening the file?

[Myrsloik]

Very interesting, not very often we get new source filters to play with.

Have you done any scale testing with a large json file? If you plan to store every file's index in the same file it will grow pretty quick. (And json parsing is more CPU expensive than one would expect)
How many movies can be indexed before the json file gets so big it starts to have adverse effects on opening the file?

Have you done any scale testing with a large json file? <- nope

If you plan to store every file's index in the same file it will grow pretty quick. <- the "index" data is only ~300 bytes for a file where most of it is the filename

How many movies can be indexed before the json file gets so big it starts to have adverse effects on opening the file? <- 100k+?

You can also set a custom cache path to split it into smaller chunks if you ever encounter this problem

[mastrboy]

Have you done any scale testing with a large json file? <- nope

If you plan to store every file's index in the same file it will grow pretty quick. <- the "index" data is only ~300 bytes for a file where most of it is the filename

How many movies can be indexed before the json file gets so big it starts to have adverse effects on opening the file? <- 100k+?

You can also set a custom cache path to split it into smaller chunks if you ever encounter this problem

Compared to the index created by LWLibavVideoSource, BestSource sounds a lot more efficient, I just indexed a 1:45hour long movie to check and .LWI index file ended up using 13,2 megabytes.

And by having the functionality to specificy custom paths should indeed be a good solution if the file ends up too big.

I guess my concerns about the filesize where wrong...

[mastrboy]

Did some quick benchmarks on decoding speed and indexing speed.
Seems to be on par with core.ffms2.Source for decoding speed, but a lot slower than both ffms2 and lsmas for indexing

Decode speed test:
output = core.ffms2.Source(source=rf"00014.m2ts")
output.set_output(0)
Script evaluation done in 0.16 seconds
Output 10001 frames in 25.63 seconds (390.17 fps)


output = core.lsmas.LWLibavSource(source=rf"00014.m2ts")
output.set_output(0)
Script evaluation done in 0.19 seconds
Output 10001 frames in 19.46 seconds (513.80 fps)


output = core.bs.VideoSource(source=rf"00014.m2ts")
output.set_output(0)
[matroska,webm @ 000002D34F33BF40] Unsupported encoding typeUnsupported encoding typeCould not find codec parameters for stream 5 (Subtitle: hdmv_pgs_subtitle (pgssub)): unspecified size
Consider increasing the value for the 'analyzeduration' (0) and 'probesize' (5000000) options
[matroska,webm @ 000002D34F33BF40] Could not find codec parameters for stream 6 (Subtitle: hdmv_pgs_subtitle (pgssub)): unspecified size
Consider increasing the value for the 'analyzeduration' (0) and 'probesize' (5000000) options
Script evaluation done in 0.09 seconds
Output 10001 frames in 25.49 seconds (392.37 fps)

Index speed test (Powershell to measure time):
core.ffms2.Source(source=rf"00014.m2ts")
Measure-Command { vspipe -e 1 -p .\test.vpy . }
TotalSeconds : 3,2294295

core.lsmas.LWLibavSource(source=rf"00014.m2ts")
Measure-Command { vspipe -e 1 -p .\test.vpy . }
TotalSeconds : 3,809994

core.bs.VideoSource(source=rf"00014.m2ts")
Measure-Command { vspipe -e 1 -p .\test.vpy . }
TotalSeconds : 86,2428886


Hardware: AMD 3900x, 32GB 3200 Ram, Samsung EVO970 NVME. (Windows10)

The "cachepath" parameter doesn't write a file, it seems to go trough the process of creating a index, but I never get a file written.
I tried with and without full path. (cachepath="c:/test/indexfile.json" | cachepath="indexfile.json")

[lansing]

Tested with my blu-ray m2ts file, 24 minutes video, my machine is a 5900x.
BestSource took 1:12 to index with all threads running at about 50%, seeking forward or backward took over 10+ seconds.

ffms2 took 5 seconds to index and seeking is instant.

[Myrsloik]

...
The "cachepath" parameter doesn't write a file, it seems to go trough the process of creating a index, but I never get a file written.
I tried with and without full path. (cachepath="c:/test/indexfile.json" | cachepath="indexfile.json")

It's a path. So you should specify something like c:\test\ and then bsindex.json will be read/written from there.

Getting an exact frame count is slower than FFMS2 because every frame is decoded and not just parsed (possibly incorrectly).

Obviously seeking is slower with linear decoding only. What did you expect? It's more or less BestAudioSource's idea applied to video as well.

I wasn't kidding when I said that this is the solution is based on throwing lots of hardware at a problem. This would have been complete insanity without the cheap 16 core CPUs we now have.

[ChaosKing]

I guess you work with ffms2/lsmas but switch to BS™ for the encode :devil:

[mastrboy]

It's a path. So you should specify something like c:\test\ and then bsindex.json will be read/written from there.

Works fine when just specifying directory, I assumed I could specify full filename as most Path options in Windows usually is the full path and usually Directory or Folder is used as terms instead...
Maybe throw an error if it fails to write a index file, rather than silently continue could further prevent this confusion.

[Yomiko]

I guess you work with ffms2/lsmas but switch to BS™ for the encode :devil:

What's going to be the benefit?

[ChaosKing]

Faster seeking times while you work on your script and accurate seeking while encoding (temporal filters will thank you!)

Does it have av1 support? ( I have a RTX 3070Ti + Ryzen 3600. As far as I know the GPU does support hardware accelerated AV1 decoding)
Video: rav1e_b_4600.ivf
-----------------------------------
[av1 @ 000001931E36A8C0] Your platform doesn't suppport hardware accelerated AV1 decoding.
[av1 @ 000001931E36A8C0] Failed to get pixel format.
[av1 @ 000001931E36A8C0] Missing Sequence Header.
...
[av1 @ 000001931EAED980] get_buffer() failed
[av1 @ 000001931EAED980] thread_get_buffer() failed
[av1 @ 000001931EAED980] Failed to allocate space for current frame.
[av1 @ 000001931EAED980] Get current frame error
[av1 @ 000001931EAED980] Failed to get reference frame.
[av1 @ 000001931EAED980] video_get_buffer: image parameters invalid
[av1 @ 000001931EAED980] get_buffer() failed
[av1 @ 000001931EAED980] thread_get_buffer() failed
[av1 @ 000001931EAED980] Failed to allocate space for current frame.
[av1 @ 000001931EAED980] Get current frame error
[av1 @ 000001931EAED980] Failed to get reference frame.
[av1 @ 000001931EAED980] video_get_buffer: image parameters invalid
[av1 @ 000001931EAED980] get_buffer() failed
[av1 @ 000001931EAED980] thread_get_buffer() failed
[av1 @ 000001931EAED980] Failed to allocate space for current frame.
[av1 @ 000001931EAED980] Get current frame error
[av1 @ 000001931EAED980] Failed to get reference frame.
[av1 @ 000001931EAED980] video_get_buffer: image parameters invalid
[av1 @ 000001931EAED980] get_buffer() failed
[av1 @ 000001931EAED980] thread_get_buffer() failed
[av1 @ 000001931EAED980] Failed to allocate space for current frame.




It could be the new king of accurate frame seeking.
Seeking test chart:
https://i.imgur.com/VSp4gl5.png

[Myrsloik]

Faster seeking times while you work on your script and accurate seeking while encoding (temporal filters will thank you!)

Does it have av1 support? ( I have a RTX 3070Ti + Ryzen 3600. As far as I know the GPU does support hardware accelerated AV1 decoding)
...


It could be the new king of accurate frame seeking.
Seeking test chart:
https://i.imgur.com/VSp4gl5.png

I didn't compile it with any av1 decoder so probably not. Will add it in the next test build.

Maybe I should take a look at GPU decoding too...

[Myrsloik]

Test 6: Now compiled with libdav1d and the cache path now is actually the full path to the file to cache things in. Link in first post.

[jlw_4049]

Is there anyway to keep indexing time to move a little quicker?

[Myrsloik]

Is there anyway to keep indexing time to move a little quicker?

That would be FFMS2 you're talking about.

[jlw_4049]

That would be FFMS2 you're talking about.

What benefits would you get using BS over FFMS2?

[Rumbah]

For me it works like intended.

Simple 1080p H264 decoding is about 8% slower than LsmashSource with indexing prior to the decoding test (1080fps vs 990fps).

For my use case I'll stay with Lsmash as I use Vapoursynth to split videos in 8 equal parts and encode them in parallel. This is much slower with BS as it has to decode much more. For my use it was about a factor of 7-8.

[Myrsloik]

GPU decoding added. Simply add hwdevice="cuda" or "d3d11va" to activate. Updated link in the first post.

[Rumbah]

I did some testing. File was a 10 Mbit/s h264 file, 1920x960 , 45mins.
Hardware is a 12 core/24 threads Ryzen 3900X, Nvidia 3060 Ti, Windows 10.

Indexing speed:
LSmash : 2s
BS_CPU : 69s (about 50% CPU usage)
BS_GPU_DX11 : 252s (about 0% CPU usage, 75% GPU usage)
BS_GPU_Cuda : Error ([h264 @ 000001E1CF0A8200] No decoder surfaces left) the numbers change till you close the process

Full decoding of the file without indexing:

LSmash : 64s
BS_CPU : 72s (about 50% CPU usage)
BS_GPU_DX11 : 117s (about 5% CPU usage, 75% GPU usage)
BS_GPU_Cuda : Error ([h264 @ 000001E1CF0A8200] No decoder surfaces left)


So for me Cuda does not work at all, it just throws the error above.

Full CPU decoding is still a little bit slower than LSmash Libav.
GPU indexing takes pretty long but does not use CPU ressources.
GPU decoding only is much faster than GPU indexing.
GPU decoding uses one full CPU core so the decoding there might be CPU bound.

[Myrsloik]

The CPU decoding speed is determined by how FFmpeg was compiled and isn't that interesting. Make a -O99 gcc compile if you care. I tend to prefer easy to develop for workflows...

Now for the interesting observations:
Somehow your linear decoding with DX11 is faster than indexing. Indexing should be faster or the same since it's doing the same thing more or less (just without reading back decoded frames).

Cuda doesn't seem to work properly for anyone. Some people get as far as to get decoded frames with mild corruption. Probably some kind of FFmpeg bug. Maybe it works better outside windows.

[ChaosKing]

Cuda decoding for vob files seems to work
My quick test (version 7)

Ryzen 3600 OC, GPU 3070 TI
cpu: 1200fps
cuda: 780 fps
dx11: 710 fps

EDIT
CUDA h264 BD works, but sometimes a green frame is shown
h264 10 works too

[Rumbah]

The CPU decoding speed is determined by how FFmpeg was compiled and isn't that interesting. Make a -O99 gcc compile if you care. I tend to prefer easy to develop for workflows...
No worries, I was just curious.

Now for the interesting observations:
Somehow your linear decoding with DX11 is faster than indexing. Indexing should be faster or the same since it's doing the same thing more or less (just without reading back decoded frames).
Ok, tested some more with different files and got the following results:

33 min 5 Mbit/s h264 1080p

Indexing

Software : 42s (50% CPU)
DX11 : 187s (0% CPU, 75% GPU) Video Decode
CUDA : crash

Decoding

Software : 45s (50% CPU)
DX11 : 97s (0% CPU, 75% GPU) Video Decode
CUDA : crash


33 min 15 Mbit/s h265 2160p

Indexing

Software : 269s (50% CPU)
DX11 : 111s (1.8% CPU, 98% GPU) Video Decode
CUDA : 105s (1.1% CPU, 100% GPU) Video Decode

Decoding

Software : 340s (50% CPU)
DX11 : 977s (4.8% CPU, 42% GPU) 3D
CUDA : 770s (4.3% CPU, 50% GPU) Video Decode

So h264 Cuda still crashes for me and that indexing is slower than decoding seems reproducible for me.
For the 4k hevc decoding Cuda works and is faster than DX11.
But hardware decoding is much slower than indexing. For DX11 Decoding the Windows task manager shows 3D instead of video decode. For Cuda it shows video decode but it still only uses 50% GPU and is much slower than indexing.

The indexing tests were done with "vspipe --info file.vpy", the decoding tests with "vspipe file.vpy ." and a pre existing index.

[Myrsloik]

Reading back frames is really slow according to your tests. I'm not surprised. Don't expect big performance improvements until someone makes things more threaded and have prefetch for GPU decoding. Probably won't be done by me anytime soon.

A speed comparison with DGDecNV would be very interesting if you have time to do it.

[Rumbah]

Sure thing. Same as before, just added the DGDecodeNV results.

One thing I noted is that LSmash and BSSource return a 10bit video for the HEVC file (it is a 10bit file) while DGDecode returns 16bit.

33 min 5 Mbit/s h264 1080p

Decoding

Software : 45s (50% CPU)
DX11 : 97s (0% CPU, 75% GPU) Video Decode
CUDA : crash
DGDecodeNV : 60s (5% CPU, 100% GPU) Video Decode


33 min 15 Mbit/s h265 2160p

Decoding

Software : 340s (50% CPU)
DX11 : 977s (4.8% CPU, 42% GPU) 3D
CUDA : 770s (4.3% CPU, 50% GPU) Video Decode
DGDecodeNV : 307s (4.9% CPU, 67% GPU) Video Decode

[_Al_]

I guess you work with ffms2/lsmas but switch to BS™ for the encode :devil:
to not forget to actually switch those source plugins, this could be used:

import vapoursynth as vs
from vapoursynth import core
source = r"meh.mkv"
if __name__ == '__vapoursynth__':
my_vnode = core.bs.VideoSource(source)
else:
my_vnode = core.ffms2.Source(source)
.
.
.

[Yomiko]

unfortunately you will have __vapoursynth__ in previewers as well

[ChaosKing]

I would simply pass an arg parameter to vspipe and check for that instead of __vapoursynth__
http://www.vapoursynth.com/doc/output.html#options

[_Al_]

Does vsedit also changes __main__ to "__vapoursynth__"? Why? I thought this is done by vspipe only and the way preview is done, is, to get a video node using by vs.get_output() or vs.get_outputs() then when requesting a frame, changing it to RGB an putting on screen.

ok, just tested it with some very old version of vsedit but __main__ is not even defined when trying to preview script, perhaps because of using .vpy, not .py, to evaluate script, not sure. If using a python previewer with .py script, that __main__ checking for "__vapoursynth__" is fine.

[Yomiko]

It's a feature of vsscript, not vsedit or vspipe.

[tebasuna51]

Please support AviSynth+ r4001 (https://forum.doom9.org/showthread.php?p=1989323#post1989323) including the audio channel mask.

[Myrsloik]

R1 RC2 (https://github.com/vapoursynth/bestsource/releases/tag/R1-RC2) is out. Help with the testing so there can finally be a proper release.

[Guest]

Is it possible to change this script to use BS instead of LSmash / LWLibavAudioSource :-

#AudioSource
LoadPlugin("C:\RipBot264\Tools\AviSynth plugins\lsmash\LSMASHSource.dll")
audio=LWLibavAudioSource("H:DV edits\Season 3\faded audio & subs\S03 E04 vol 3.0 fade.eac3",cachefile="E:\Temp\RipBot264temp\job3\S03 E04 vol 3.0 fade.eac3.lwi").ConvertAudioToFloat.ResampleAudio(48000)

my test, didn't work :-

#AudioSource
LoadPlugin("C:\RipBot264\Tools\AviSynth plugins\BestSource-R4-x64\BestSource.dll")
audio=BSAudioSource("H:DV edits\Season 3\faded audio & subs\S03 E04 vol 3.0 fade.eac3",cachefile="E:\Temp\RipBot264temp\job3\S03 E04 vol 3.0 fade.eac3.lwi").ConvertAudioToFloat.ResampleAudio(48000)

[Emulgator]

Drop the cachefile.
.lwi belongs to LWLibav.

[Guest]

Drop the cachefile.
.lwi belongs to LWLibav.

Interesting to get a reply from you, as you commented on my questions about this on the RipBot264 thread :)

So something like this :-

#AudioSource
LoadPlugin("C:\RipBot264\Tools\AviSynth plugins\BestSource-R4-x64\BestSource.dll")
audio=BSAudioSource("H:DV edits\Season 3\faded audio & subs\S03 E04 vol 3.0 fade.eac3").ConvertAudioToFloat.ResampleAudio(48000)

[tebasuna51]

So something like this :-

#AudioSource
LoadPlugin("C:\RipBot264\Tools\AviSynth plugins\BestSource-R4-x64\BestSource.dll")
audio=BSAudioSource("H:DV edits\Season 3\faded audio & subs\S03 E04 vol 3.0 fade.eac3").ConvertAudioToFloat.ResampleAudio(48000)

Yes.
BTW ConvertAudioToFloat do nothing, because the decoded audio from lossy formats is always float.

[Guest]

Yes.
BTW ConvertAudioToFloat do nothing, because the decoded audio from lossy formats is always float.

Thanks again, teb...

But leaving the ResampleAudio(48000) would be recommended, just in case the audio track is lower.

[Guest]

Drop the cachefile.
.lwi belongs to LWLibav.

:thanks:, that worked, it was easier than I thought it was going to be, just a quick edit to the generated scripts, and you're away. :)

[Myrsloik]

Time for a bump. R6 is out now and fixes a lot of problems with CFR mode.

[tebasuna51]

Thanks Myrsloik, I found it at https://github.com/vapoursynth/bestsource/releases

In first post I read: "Full release is on github now so remove test versions"
Maybe you can add the link to found the last version.

Also you say: "64 bit and VapourSynth only."
But work fine in AviSynth+ (thanks)

[Kurt.noise]

Hi,

Time for a bump. R6 is out now and fixes a lot of problems with CFR mode.
Tested R6 within Avisynth+ 3.7.3 on Windows 11 , using this avs script :
LoadPlugin("E:\tools\bestsource\BestSource.dll")
BSVideoSource("E:\Video\HEVC.mp4",HWDevice=d3d11va)
I get an error playing the script through mpc-hc :

https://iili.io/dGNKzcx.md.png (https://freeimage.host/i/dGNKzcx)

Same issue replacing d3d11va by dxva2 or vulkan hardware device type.

And also same error playing the script through ffmpeg executable...

Without hwdevice argument, that works, of course, as expected.

And using such hardware decoding directly through FFmpeg, that works also correctly :
https://iili.io/dGN5O8P.md.png (https://freeimage.host/i/dGN5O8P)

Any idea ?

[Kurt.noise]

Erff, I missed the quotation mark since its a string :D

LoadPlugin("E:\tools\bestsource\BestSource.dll")
BSVideoSource("E:\Video\HEVC.mp4",HWDevice="d3d11va")

Sorry for the noise...vulkan is still not supported yet, right ?

[Myrsloik]

Erff, I missed the quotation mark since its a string :D

LoadPlugin("E:\tools\bestsource\BestSource.dll")
BSVideoSource("E:\Video\HEVC.mp4",HWDevice="d3d11va")

Sorry for the noise...vulkan is still not supported yet, right ?

I actually haven't tested it. It's also possible the release wasn't compiled with support enabled. Will investigate before next release

[amayra]

out of curiosity why is there no GStreamer plugin for VapourSynth ?

I had a horrible FFmpeg experience back when I used to use Windows 7 before I moved to Linux

[isidroco]

Does BestSource work on Avisynth 64 non plus version?

[Myrsloik]

Does BestSource work on Avisynth 64 non plus version?

Probably not? Not tested at least.

[Myrsloik]

R11 is out. It fixes the framerate guessing problems reported earlier in the thread.

[StainlessS]

BestSource R14
https://github.com/vapoursynth/bestsource/releases

Avisynth script

LoadPlugin(".\BestSource.dll") # ALSO in plugins

#LSmashVideoSource(".\v.mp4") # OK

BSVideoSource(".\v.mp4") # Avisynth Open Failure: Script Error: There is no function named "BSVideoSource". (c:\Z\v.avs, line 5)
#BSSource(".\v.mp4") # Avisynth Open Failure: Script Error: There is no function named "BSSource". (c:\Z\v.avs, line 6)

return last

Am I doin' summick daft, or dont it work no more. :(

EDIT:

32 Bit: Avisynth (x86) is not installed
64 Bit: AviSynth+ 3.7.5 (r4289, 3.7, x86_64) (3.7.5.0)

[Myrsloik]

Right, can't use ci builds made with gcc for releases because then avisynth doesn't work. Redownload the binaires and it should be fixed now.

[StainlessS]

I had downed it a number of times before, but was the first time I'de tried it, <my template scripts tend to use LSmash or other, based upon file extension>
Works a treat now, thanx, you're such a sweetie :)

[Chikuzen]

Hi.
When opening 16-bit PNG with BSVideoSource, the R and B channels seem to be swapped.


BSVideoSource("https://media.xiph.org/sintel/sintel-4k-png16/00006661.png")
Spline36Resize(1024, 436)
G=last.showGreen()
B=last.showBlue()
R=last.showRed()
StackVertical(last, mergeRGB(B, G, R))

result:
https://i.imgur.com/yuCMay9.png

[Myrsloik]

Will be fixed in the next release

[Deim0s]

HI,
AviSynth+ 3.7.5
drc_scale=0 doesn't work on ac3 files.
BSAudioSource("file.ac3", drc_scale=0)
Dynamic range compression still applied.

[Myrsloik]

HI,
AviSynth+ 3.7.5
drc_scale=0 doesn't work on ac3 files.
BSAudioSource("file.ac3", drc_scale=0)
Dynamic range compression still applied.

How can you tell it's not applied?

[Deim0s]

ffmpeg -drc_scale 0 -request_sample_fmt flt -i input.ac3 -acodec pcm_f32le output.wav
screen
(https://s1.directupload.eu/images/260408/3osntje5.png)

FFAudioSource("file.ac3", drc_scale=0)
screen (https://s1.directupload.eu/images/260408/datenu74.png)

BSAudioSource("file.ac3", drc_scale=0)
screen (https://s1.directupload.eu/images/260408/yxwfkt4i.png)
Also if ffmpeg and FFAudioSource with drc_scale=1.

[Myrsloik]

ffmpeg -drc_scale 0 -request_sample_fmt flt -i input.ac3 -acodec pcm_f32le output.wav
screen
(https://s1.directupload.eu/images/260408/3osntje5.png)

FFAudioSource("file.ac3", drc_scale=0)
screen (https://s1.directupload.eu/images/260408/datenu74.png)

BSAudioSource("file.ac3", drc_scale=0)
screen (https://s1.directupload.eu/images/260408/yxwfkt4i.png)
Also if ffmpeg and FFAudioSource with drc_scale=1.

Fixed, R18 released with the fix.

[Selur]

Is there some experience how long it takes that a release ends up available through pip? (assuming it's done automated)
and right after posting new bestsource is available through pipt *gig*

[Myrsloik]

Is there some experience how long it takes that a release ends up available through pip? (assuming it's done automated)
and right after posting new bestsource is available through pipt *gig*

About 10 minutes unless a job randomly gets stuck. It got randomly stuck.

[Myrsloik]

R20 is out (https://github.com/vapoursynth/bestsource/releases/tag/R20)

Now good avs+ binaries are also automatically built

[Emulgator]

Thanks for R20 AVS+ !
(Was about to report R19 AVS+ which failed to open while R17 still could.)

[Myrsloik]

R22 is out. It reworks hardware decoding to be vulkan only. Along the way some FFmpeg performance bugs got fixed as well. Many arguments are removed (extra hw frames) and other bullshit. Simply set gpu=True and get decoding. Requires VapourSynth R80 with its native GPU support.


Note that for Avisynth it effectively kills any gpu decoding since it doesn't have the concept of native gpu frames.

[wonkey_monkey]

Note that for Avisynth it effectively kills any gpu decoding since it doesn't have the concept of native gpu frames.

It couldn't just download the decoded planes from the GPU, as it presumably did previously?

Edit: never mind, I see the note that it's assumed to be faster to decode on CPU and avoid the download.

[Columbo]

"it's assumed to be faster to decode on CPU and avoid the download"

ASS-U-ME ;)

[Myrsloik]

If it greatly upsets you you're free to keep using R21. Or fork it. Or add GPU support to avs+ so we can all gpu together.

[Columbo]

Hi Myrsloik,

I am seeing something with BestSource that I do not understand, and I wonder if I have set something up wrong.

On one clip, hardware decoding with hwdevice="cuda" runs slower than the default software decoding:

BestSource, software (default): 29.82 fps
BestSource, hwdevice="cuda": 15.27 fps
DGSource, for reference: 106.8 fps

I include the DGSource figure only to show what NVDEC can deliver on this clip and machine, which is why the cuda figure surprised me.

Details:

BestSource: R20 (BestSource-R20-win64-msvc.dll)
Clip: HEVC elementary stream, 7680x4320, 10-bit 4:2:0, 60000/1001 fps, about 1570 frames
Machine: i7-11700K, 32 GB DDR4-3200, RTX 4090, Windows 10
Command: vspipe.exe -p script.vpy .

Script:

core.std.LoadPlugin("BestSource-R20-win64-msvc.dll")
video = core.bs.VideoSource(source=r'clip.hevc')
video.set_output()

For the hardware run the only change was adding hwdevice="cuda". All other settings were left at their defaults.

All figures are from runs made after the index had been built.

During the cuda run, nvidia-smi dmon shows the decoder is in use, but only lightly: the dec column sits at about 25% on average, ranging from 12% to 45%. So NVDEC seems to be engaged but waiting most of the time.

My questions:

1. Is this expected for a clip of this size, or does it point to something wrong in my setup?
2. Are there settings I should change for hardware decoding, such as threads or extrahwframes?
3. Is there anything I can measure or log that would help show where the time is going?

I can provide the clip if that would help.

Thanks for any pointers.

EDIT: I ran one more test that may help narrow it down.

I put the same clip through FFmpeg alone with hardware decoding and a null output:

ffmpeg -benchmark -hwaccel cuda -i clip.hevc -f null -

That gave 46 fps. As I understand it, in this test FFmpeg downloads the frames as p010le and does no format conversion.

So on this clip and machine:

FFmpeg, hwaccel cuda, null output: 46 fps
BestSource, hwdevice="cuda": 15.27 fps
BestSource, software (default): 29.82 fps

That leaves about 44 ms per frame between the FFmpeg figure and the BestSource cuda figure.

One thought, and I may well be wrong: in cuda mode the frames arrive as P010, with interleaved chroma and the 10-bit values in the high bits, while BestSource delivers planar P10 with the values in the low bits. At 8K that is about 50 million samples per frame to shift and rearrange. Could that conversion be where the time goes? In software mode the decoder produces planar P10 directly, which might explain why software is faster here.

If that is the cause, would it make sense to offer the output as P16 in hardware mode, so that no shift is needed?

[Myrsloik]

Hi Myrsloik,

I am seeing something with BestSource that I do not understand, and I wonder if I have set something up wrong.

On one clip, hardware decoding with hwdevice="cuda" runs slower than the default software decoding:

BestSource, software (default): 29.82 fps
BestSource, hwdevice="cuda": 15.27 fps
DGSource, for reference: 106.8 fps

...

A few things to note here. I deprecated everything except the vulkan decoder in the latest R22 release in favor of outputting GPU resident frames. I even helped test FFmpeg patches to make the vulkan performance not suck on my discrete amd gpu and prodded them to fix all reported vulkan verification layer problems. These fixes willl probably be in the upcoming ffmpeg 9.1 release.

Now this doesn't have much to do with cuda itself but GPU support in ffmpeg seems to overall have very few developers combined with few users to test it. As a parallel to your problem I reported just as abysmal decode numbers for h264 in vulkan.

The general hardware decoding support is also implemented in a very lazy way that reads back gpu frames synchronously using ffmpeg's convenience functions. They are convenient but things like hashing gpu frames is a thing best done on the gpu. So the design before R22 is a bit shit. After implementing the software decoding I realized that hardware decoding only needed like 20 lines of code to poorly enable and I did it.

My guess is that it's an ffmpeg bug since even null output gives you less than half the performance. Maybe file a bug report and see what they say. The similar vulkan path suffered from excessive synchronization until recently that killed performance.

My prediction for the future is that CUDA will lose its relevance now that vulkan has proper extensions to expose the same functionality in a more portable way. At least for h264/h265/vp9/av1 which it can handle.

[Columbo]

Thank you for your interesting and candid answer, that clears up a lot.

What you describe fits what I measured. For this clip a frame is about 100 MB once it is in system memory (7680x4320, 4:2:0, two bytes per sample). At that size the decode itself is not the expensive part. Moving the frame is.

Some numbers from my own work on the same clip and machine (RTX 4090, PCIe 4.0, DDR4-3200), in case they are useful to you:

NVDEC decode: about 7.5 ms per frame, so the hardware tops out near 130 fps.
Download to pinned memory, running alone: about 3.8 ms per frame, roughly 26 GB/s.
The same download while another thread copies the previous frame: about 7.7 ms per frame.

That last one surprised me. The download and the copy each move 100 MB, and when they overlap they compete for memory bandwidth and both slow down. So on a DDR4 machine the wall for 8K is the RAM, not the GPU and not the bus. I did not expect that.

It also makes me think your move to GPU resident frames is the right direction. The readback is where the time goes, so not doing it is the only real cure. The catch is that it only pays off while everything downstream stays on the GPU. As soon as one filter or an encoder needs the frame in system memory, somebody has to bring 100 MB down per frame (and possibly back up, depending on the filter chain), and the same wall is waiting. We'll see if you can get better traction in getting people to code GPU-aware filters than I did with my CUDASynth idea.

On the hashing, I agree it belongs on the GPU if it has to be done. On the CPU it is one more full pass over every frame, and at this size every pass costs milliseconds.

That leads to a question about the design, asked out of curiosity. As I understand it, you identify frames by hashing them because a seek in FFmpeg cannot be trusted to land where it was asked to. My own indexer takes a different route. It records the stream position of each GOP and the order of the frames, then starts decoding at a known position and counts, so it never needs to look at the pixels to know which frame it has. It also keeps indexing quick, because the indexer only has to parse the stream and not decode it. With hashes, every frame has to be fully decoded at index time, so indexing takes about as long as decoding the whole file, and I noticed that changing hwdevice made it index the clip again. Was the position based approach not workable across everything FFmpeg can open, or was there another reason to go with hashes? I can see that parsing every format yourself is a lot of work, so I am interested in how you weighed it.

One thing I am curious about. On an Nvidia card, Vulkan video decoding ends up driving the same decoder hardware as CUDA does. So I would expect the decode ceiling to be the same either way, and the difference to come down to how frames are handled afterwards. Is that what you see, or does the Vulkan path behave differently in practice?

As it stands, none of the FFmpeg based routes can play this clip in real time on my machine. It is a 59.94 fps stream, and I get 46 fps from FFmpeg with a null output, about 30 in software, and about 15 through the cuda path. I take your point that the 46 looks like an FFmpeg problem, and I will consider filing a report with them. However, FFmpeg has studiously ignored all my input over the years so I'm inclined not to bother.

It's always a pleasure to discuss things with you!

[Myrsloik]

You all know about FFMS2. Basically it does the best it can to seek to a specific location. However many FFmpeg demuxers don't implement byte position seeking, have odd destination quirks (looking at you y4m) or can't seek to the start of the stream again reliably. There are many fallback strategies like trying to feed it PTS or DTS depending on the container but in the end you generally end up with discontinuous timestamps (m2ts, mpeg and such) or simply badly behaved demuxers/decoders for more rare formats.

If you only support your 3 favorite containers it could probably be done, when supporting everything it just results in silent corruption and frustration too often. Not saying that FFMS2 is bad at what it does, many people have used it to great success, you are however quite limited if you don't start to modify FFmpeg itself and silent API changes will get you.

I also really don't want to maintain a separate FFmpeg fork just with demuxer seek modifications and internal RNG reset hooks. With the "no ffmpeg patching" you end up with BestSource or FFMS2 more or less.

Hashes are then the only logical choice because it's the only way to verify the output at all. There are large numbers of seek frame reordering bugs as well, for example linear decoding and seeking can for many formats get you different output. Add the hardware decoders that are even more likely to interject unrelated frames or random corrupt gunk. Sadly even if you think you do everything right the output will sooner or later not be right anyway.

Back to the hwdevice reindexing, that's done because it's a different decoder. It may not drop the same corrupted frames or deliver the same timestamps. Therefore the index wouldn't match. It's that bad even on formats that are supposed to have bitexact decoding.

You're right that the speed is very similar since it's the same hardware block being used and then only the most simply image copies. I think I saw some FFmpeg test where it's within 10% (bugs excluded) for all the different apis on the same hardware.

CUDA is however still meaningfully faster for certain types of filters, for example nnedi3vk-apimod runs about 20% slower than the best CUDA implementation of the same filter due to vulkan not having extensions that expose all the matrix math units in an efficient way.

For uploads and downloads if you use a transfer queue (dedicated hardware) one transfer in each direction concurrently maxes things out. At least in all my tests.

[Columbo]

Thank you, that answers my questions. I appreciate your detailed explanation and hope it's useful for others too.

[PatchWorKs]

My prediction for the future is that CUDA will lose its relevance now that vulkan has proper extensions to expose the same functionality in a more portable way. At least for h264/h265/vp9/av1 which it can handle.
Absolutely, even where's unexpected: https://hydrogenaudio.org/index.php/topic,129865.msg1086189.html#msg1086189

[Myrsloik]

Absolutely, even where's unexpected: https://hydrogenaudio.org/index.php/topic,129865.msg1086189.html#msg1086189

Very unexpected. Didn't think it would be actually worth the butt pains for fairly simple audio calculations.

Another place where it works surprisingly well is in my vibe coded mvutensils on gpu. Around 3-4x faster than a 9800x3d and the algorithm (apart from the the block motion vector predictors) runs more or less 1:1. Didn't expect it to go so well when x264 declared gpu motion search dead many years ago. All in compute shaders. I'll probably release it soon after some testing.