LAV CUVID Decoder - High Quality Hardware decoding for NVIDIA

[nevcairiel]

Originally Posted by pankov

sadly, no
For the moment (and it's not even certain that this is fixable) LAV CUVID decoder will not output the doubled framerate which results in jerky playback.
You get nice deinterlaced picture but with judder which is not acceptable for sports ... or anything else if you are serious about your quality experience.

Someone that knows more about actual technical details of deinterlacing .. how does the doubled framerate work?
I assume it doesn't simply output the same image twice, the two will be minimally different, right?

If i have 60 fields, how do i create 60 frames out of that?
the CUVID API has some parameters that i can set to get different images out of the post-processor .. but i'm not sure what i really want to get.

I haven't really bothered to look into the technical details of deinterlacing up until now - it also seems to be a well guarded secret.

[madshi]

Originally Posted by nevcairiel

Someone that knows more about actual technical details of deinterlacing .. how does the doubled framerate work?
I assume it doesn't simply output the same image twice, the two will be minimally different, right?

If i have 60 fields, how do i create 60 frames out of that?
the CUVID API has some parameters that i can set to get different images out of the post-processor .. but i'm not sure what i really want to get.

I haven't really bothered to look into the technical details of deinterlacing up until now - it also seems to be a well guarded secret.

You probably know this already, but let me explain, just to be safe:

First of all you must strictly separate between movie and video content.

movie content was originally shot in 24p (or in Europe eventually in 25p). This is then converted to 60i (or 50i) by first interlacing the progressive frames and then repeating some of the interlaced fields, if necessary (not with 25p -> 50i). Meaning that for perfect deinterlacing you need to throw away the repeated fields (if any) and then fuse the correct original fields together to get back to the original 24p/25p. This is called IVTC. Important to note is that all the interlaced fields originate from only 24 (or 25) different points in time.

video content was originally shot in 60i or 50i. It's important to note here that every interlaced field comes from a different point in time (!!). There are different kinds of cameras. Some shoot directly interlaced. Others shoot progressive and then throw away (or filter away) half the scanlines to produce interlaced output. Deinterlacing video content to 30p or 25p is VERY wrong. The reason for that is that you cannot simply put two fields together which come from a different point in time. Basically the only proper way of deinterlacing video content is to create 60 or 50 progressive frames. The most simple way to do that is to simply take one field at a time and scale it 2x up vertically. This will give you 60p or 50p with fluid motion. But of course this will not be very sharp. This is called BOB deinterlacing. There are better methods. But one thing is clear: The output *must* be 50p or 60p.

I don't suppose either DXVA or CUVID can automatically handle all this for you, but I don't really know. My best guess is that you need to offer users a switch between movie / video content. Then for video content you should make CUVID output 50/60 progressive frames. For movie content you should make it output 24/25 progressive frames. I don't know if the latter is possible. 25p output should be no problem. But for NTSC/ATSC movie content maybe CUVID can only output 30p, I don't know.

Of course things can get even more complicated than this. E.g. broadcasters like to blend *video* fonts / advertisements over movie content. Then you have a funny mixture of video and movie content in one stream. This is why better hardware deinterlacers decide per pixel (not per frame) whether the content is video or movie based. For this kind of content the best output is probably 60p / 50p.

[tiny]

Regarding the case of deinterlacing and CUVID it might be interesting to look at the (rather lenghty) dialog between Doom9's forum user neuron2 and Nvidia that took place during the development of DGAVCDecNV.

That dialog is available here: http://neuron2.net/dgdecnv/cuda/cuda.html

[yesgrey]

Originally Posted by nevcairiel

I assume it doesn't simply output the same image twice, the two will be minimally different, right?

Right.

Originally Posted by nevcairiel

If i have 60 fields, how do i create 60 frames out of that?

Originally Posted by tiny

Regarding the case of deinterlacing and CUVID it might be interesting to look at the (rather lenghty) dialog between Doom9's forum user neuron2 and Nvidia

Here is a quote from it that might answer your question...
"I'm assuming you mean 60Hz (or double frame rate) deinterlacing. This can be implemented by calling cuvidMapVideoFrame() twice: the first time with second_field=0, and the second time with second_field=1. The actual deinterlacing mode is still currently going to be the driver default (may be different on different GPUs and OSes), but it would allow you for example to convert 30Hz interlaced to 60Hz progressive (better than dropping one of the fields)."

[nevcairiel]

Originally Posted by yesgrey

Here is a quote from it that might answer your question...
"I'm assuming you mean 60Hz (or double frame rate) deinterlacing. This can be implemented by calling cuvidMapVideoFrame() twice: the first time with second_field=0, and the second time with second_field=1. The actual deinterlacing mode is still currently going to be the driver default (may be different on different GPUs and OSes), but it would allow you for example to convert 30Hz interlaced to 60Hz progressive (better than dropping one of the fields)."

That was actually one of the things i found in the API that i was going to test, having some confirmation that it might even do what i want it to do, great!

[yesgrey]

Here is another quote that might be useful. It seems in some situations one problem can arise, and the NVidia guy suggests a way of solving that problem.

I've noticed that one frame remains salted away somewhere when the deinterlacer is enabled and I am unable to flush it. When I do my reset and start pushing NALUs again, the hidden frame from the previous decoding comes out and I cannot find a way to flush it.

Any ideas?

The unflushable frame that I mentioned in my last mail appears to be in the D3D instance somehow, because I can get rid of it by killing the server and re-starting it.

Would you know if there is a way to flush it out with a D3D call? If I kill and remake the D3D instance, I'm going to get hit by the crash on second D3D instantiation problem again.

I've seen this problem as well: what is happening is that when this particular high-end deinterlacing method is used, it unfortunately introduces one field delay, which does suck (not a problem for normal playback, but a big issue for a generic API like NVCUVID).

So far, I haven't been able to get rid of this, but if your only concern is being able to flush it, one way to do so is to start at second_field=1 after a discontinuity.

Ultimately, the best way to get rid of this is to perform the deinterlacing within NVCUVID using a cuda kernel, rather than relying on VMR9, with sometimes some rather obscure side effects, but this requires fairly significant amount of changes in nvcuvid, so it's going to take a while.

Aha, I confirm what you say. It has nothing to do with unsuccessful flushing.

If my field sequence shown in [] delimited frames is:

[a b] [c d] [e f] [g h] [i j] ...

Then when I deinterlace double rate, I get this sequence of bobbed frames:

X a b c d e f g ...

Where X is a leftover (field now a frame) from the end of the last decode sequence. The first field is just garbage.

And yes, your suggested strategy looks good. It's a complication for my random access code so I'll have to implement it carefully.

@neuron2,
I hope you don't mind for me to put this quote here. In case you do, let me know and I'll remove it.

[yesgrey]

Another thing that might be worth it to try is if the double rate is only achievable when using BOB, or if it can also be achieved when using adaptive.

[madshi]

Adaptive is for video, not for movies. It would be a major design flaw if it couldn't do double rate. The only deinterlacing modes that shouldn't do double rate are IVTC and Weave.

[yesgrey]

Originally Posted by madshi

Adaptive is for video, not for movies.

Right.

Originally Posted by madshi

It would be a major design flaw if it couldn't do double rate.

Not necessarily, it would depend on how the adaptive mode works.
Let's consider the first 6 frames of a 60fps interlaced video:
a b c d e f
I think that deinterlacers typically work outputting half the frame rate by combining the frames in pairs, like this: ab cd ef
However, this is not the best solution. One better solution would be outputting the original frame rate but with frames created using the adjacent frames, like this: ab abc bcd cde def ef

According to what I've quoted in my previous posts, it seems the CUVID deinterlacing works like the first one, grouping the frames in pairs, and if that's the case, I don't think it would be possible to get double rate using other than BOB, because the second frame will not have any info about the next frame. However, without knowing the internals of it I can't say for sure.

Furthermore, neuron2's tool only allows double frame rate using bob. If it would work I don't see why neuron2 didn't do it...

[madshi]

Originally Posted by yesgrey

I think that deinterlacers typically work outputting half the frame rate by combining the frames in pairs

No, they don't. Doing that would simply be broken design, nothing else. No single CE (consumer electronics) hardware deinterlacer I've ever heard of does that.

"Adaptive" probably means "motion adaptive deinterlacing" which means that for moving image parts BOB is used while for static image content WEAVE is used. BOB always outputs 60p/50p, unless you throw away every other field, which would again be broken design.

[yesgrey]

Originally Posted by madshi

No single CE (consumer electronics) hardware deinterlacer I've ever heard of does that.

You're right, but I was talking about software deinterlacers, not hardware ones.

Originally Posted by madshi

BOB always outputs 60p/50p, unless you throw away every other field, which would again be broken design.

I agree. From what I've read on the discussion between neuron2 and nvidia, it seems it's not a broken design, because internally the deinterlacer uses the original frame rate, but then it only outputs half of it unless told otherwise...

[madshi]

Originally Posted by yesgrey

You're right, but I was talking about software deinterlacers, not hardware ones.

Well, I don't know much about HTPC software deinterlacers. If some/many of them do 60i -> 30p for video content then IMHO the deinterlacer devs didn't know what they were doing.

[CruNcher]

Eh it's an option not that you have todo it 60i->60p takes alot of resources and bandwith (depending on resolution) so some want to save that nothing wrong with it, since VPx and UVD on the Desktop tough it can be done with Yadiff matching quality (currently) @ 1080p with fairly low system requirements and shaders in Realtime @ relatively low power consumption, so only bandwith is an issue
Sure doing that on a soccer game to save some bandwith would be crazy unless you want to create a more film style look for the Game and leave what in real was captured by the camera
Might be currently one of the worlds most advanced Software Deinterlacer (http://forum.doom9.org/showthread.php?t=156028)
And here it might be one of the Worlds most advanced Hardware counterpart developed from US Military Research (http://www.hqv.com/index.cfm?page=tech.de-interlacing)

[Dogway]

Is LAVCUVID suited for bluray playback? Im doing some tests and it performs laggish, ffdshow is OK, it's apocalypto bluray.

[VincAlastor]

thank you for LAV CUVID. LAV CUVID doesn't run judder-free on my notebook with nvidia 8400m gt gpu and latest mpc hc, madvr 0.53 and LAV splitter, but thats not so important, to play around with your great tools is fascinating enough .
is it possible to fuse LAV CUVID, LAV splitter and madVR into one amazing new high quality avisynth source filter with cuda acceleration and maybe first-time avisynth dxva acceleration if madshi integrates dxva in madVR?

thank you nevcairiel and thank you madshi my videos look fantastic!

[madshi]

Originally Posted by CruNcher

Eh it's an option not that you have todo it 60i->60p takes alot of resources and bandwith (depending on resolution) so some want to save that nothing wrong with it

Well, if you want to save resources and bandwidth you can also decode only key frames and throw away all other frames. Nothing wrong with that, either...

[spartan711]

I have consistent stuttering every 10 seconds. ~ 8 frames dropped every time. Here is the MediaInfo profile of the media.

Video 
ID : 1 
Format : AVC 
Format/Info : Advanced Video Codec 
Format profile : High@L4.1 
Format settings, CABAC : Yes 
Format settings, ReFrames : 5 frames 
Codec ID : V_MPEG4/ISO/AVC 
Duration : 1h 47mn 
Bit rate : 8 062 Kbps 
Width : 1 920 pixels 
Height : 800 pixels 
Display aspect ratio : 2.40:1 
Frame rate : 23.976 fps 
Color space : YUV 
Chroma subsampling : 4:2:0 
Bit depth : 8 bits 
Scan type : Progressive 
Bits/(Pixel*Frame) : 0.219 
Stream size : 5.95 GiB (91%) 
Writing library : x264 core 79 r1342 e8501ef 
Encoding settings : cabac=1 / ref=5 / deblock=1:0:0 / analyse=0x3:0x113 / me=umh / subme=7 / psy=1 / psy_rd=1.0:0.2 / mixed_ref=1 / me_range=16 / chroma_me=1 / trellis=1 / 8x8dct=1 / cqm=0 / deadzone=21,11 / chroma_qp_offset=-3 / threads=12 / nr=0 / decimate=1 / 

mbaff=0 / constrained_intra=0 / bframes=3 / b_pyramid=0 / b_adapt=1 / b_bias=0 / direct=1 / wpredb=1 / wpredp=0 / keyint=250 / keyint_min=25 / scenecut=40 / rc_lookahead=40 / rc=2pass / mbtree=1 / bitrate=8062 / ratetol=1.0 / qcomp=0.60 / qpmin=10 / qpmax=51 / qpstep=4 / cplxblur=20.0 / qblur=0.5 / ip_ratio=1.40 / aq=1:1.00 
Language : English 

Audio 
ID : 2 
Format : AC-3 
Format/Info : Audio Coding 3 
Mode extension : CM (complete main) 
Codec ID : A_AC3 
Duration : 1h 47mn 
Bit rate mode : Constant 
Bit rate : 640 Kbps 
Channel(s) : 6 channels 
Channel positions : Front: L C R, Side: L R, LFE 
Sampling rate : 48.0 KHz 
Bit depth : 16 bits 
Compression mode : Lossy 
Stream size : 494 MiB (7%) 
Language : English 

Text 
ID : 3 
Format : UTF-8 
Codec ID : S_TEXT/UTF8 
Codec ID/Info : UTF-8 Plain Text 
Language : English

Hardware is
Processor: C2D T5800 2.0 Ghz
RAM: 4GB
Video Card: 9600M GT
OS: 64 bit Windows 7 Pro

Process priority is on high, and I have nothing else running.

[Tom Keller]

Originally Posted by VincAlastor

is it possible to fuse LAV CUVID, LAV splitter and madVR into one amazing new high quality avisynth source filter with cuda acceleration and maybe first-time avisynth dxva acceleration if madshi integrates dxva in madVR?

MadVR is great (not just cause it's the only video renderer that works perfect with the LAV CUVID Decoder for me )! But since madVR simply IS a video renderer, it has no place in AviSynth, since AviSynth doesn't need any video renderer at all. It "only" outputs uncompressed video to any appropriate application - so AviSynth doesn't need to do video rendering.

However... i guess it should be possible to use the same (or at least: similar) scaling and upsampling algorithms inside AviSynth (for HQ resizing and color space conversion), which are used by madVR.

As to the rest: with DGDecNV one source filter with CUDA acceleration still exists. It's not free though - but it works damn good.

[roozhou]

Originally Posted by Tom Keller

However... i guess it should be possible to use the same (or at least: similar) scaling and upsampling algorithms inside AviSynth (for HQ resizing and color space conversion), which are used by madVR.

Avisynth's built-in resizers is CPU only. And more importantly, unlike libswscale and most resizers used in video renderers, Avisynth's built-in resizers do not apply any colorspace conversion.

e.g. You want to watch a 720x480 DVD video fullscreen on a 1920x1080 monitor. MadVR will convert 720x480 YV12 -> 1920x1080 RGB(8bit or 10bit) in one step. The intermediate results are 16bits or higher so rounding errors are minimized. Unfortunately avisynth has to use two filters.

LanczosResize(1920,1080).ConvertToRGB32()

The chroma is upscaled two times, so it is slower and the quality is lower.

[VincAlastor]

Originally Posted by Tom Keller

MadVR is great (not just cause it's the only video renderer that works perfect with the LAV CUVID Decoder for me )! But since madVR simply IS a video renderer, it has no place in AviSynth, since AviSynth doesn't need any video renderer at all. It "only" outputs uncompressed video to any appropriate application - so AviSynth doesn't need to do video rendering.

However... i guess it should be possible to use the same (or at least: similar) scaling and upsampling algorithms inside AviSynth (for HQ resizing and color space conversion), which are used by madVR.

As to the rest: with DGDecNV one source filter with CUDA acceleration still exists. It's not free though - but it works damn good.

thank you, i thought for dxva avisynth need a videorenderer. and madvr's gpu scaler and gpu upsampler are missing in avisynth. i use DGDecNV!, but avi's can not be decoded, indexing is nerving and an adaptive deinterlacer is missing too. and maybe LAV CUVID is faster....

a new all in one high quality hardware accelerated source filter were good for avisynth, and a strong and fast bundle of LAV splitter, LAV CUVID, ffdshow/ffmpeg and (parts of) madVR could make it possible (i think, i wish) ...if i had studying informatics and experience, damn...

do you thing my nvidia 8400m gt isn't to slow for LAV CUVID and full hd material???