CoreCodec/H.264 Codec "CoreAVC"

[madshi]

What that thread was about was really conversion from 8bit to 10bit (for encoding), not the other way around.

For 10 to 8, there really is only one proper way to do it, and that is dithering, with saturation/clipping.
Agreed.

(Of course ideally no downconversion should be done at all, passing the full bitdepth downstream, but only very few DirectShow filters accept high bitdepth content.)

[mandarinka]

However, doesn't x264's upconversion match swscale's downconversion? I was under the impression that when x264 converts 128 to 514, swscale will convert the 514 back to 128, which is why nobody noticed this shift in 10bit, initially.
While the proper process would be 128 -> (x264) -> 512 -> (swscale) -> 128. Dithering only makes the conversion higher quality, but doesn't help when the conversion uses wrong relation, does it?

[nevcairiel]

The downconversion in swscale uses dithering and shifting, there is nothing special there.

[cyberbeing]

However, doesn't x264's upconversion match swscale's downconversion? I was under the impression that when x264 converts 128 to 514, swscale will convert the 514 back to 128, which is why nobody noticed this shift in 10bit, initially.
For 10-bit -> 8-bit conversion it seems to be a swscale implementation problem, rather than a swscale problem.

Implentation #1: madVR, MPlayer (swscale), and FFMS2 (lav?) overall match the color/hue of each other.

Implentation #2: FFDShow (swscale) and LAV Video (swscale) match the color/hue of each other.

Implemation #1 is assumed to be correct way, but nobody on the FFDShow/LAV-Video side has taken the time to figure out why their swscale results differ from MPlayer swscale results and others...


The downconversion in swscale uses dithering and shifting, there is nothing special there.

The nothing-special 10-bit -> 8-bit near-matching downconversion of swscale implementation in FFDShow/LAV-Video, does come much closer to reversing the skewed encoded levels of x264 swscale 8-bit -> 10-bit than the 10-bit -> 8-bit near-matching downconversion of madVR, ffms2, and the swscaled implementation in MPlayer.

Since nobody seems to have any desire to fix swscale in x264 anytime soon, I think this is a valid question for CoreCodec.

A) For the time being does CoreAVC assume all 10-bit x264 encodes have skewed levels, and actively attempt to reverse the problem in 10-bit -> 8-bit conversion? When x264 is fixed, a quick 3.0.x release is pushed out with proper 10-bit -> 8-bit conversion?

B) Does CoreAVC support both a skewed x264 10-bit levels correction as well as proper conversion from the start, and add a toggle in the settings dialog?

C) Does CoreAVC only do proper 10-bit -> 8-bit conversion and show the skewed levels in the source as madVR, ffms2, and MPlayer do?

D) Does CoreAVC do none of the above and just mimic FFDShow/LAV-Video 10-bit -> 8-bit?

[nevcairiel]

I looked at the conversion in swscale that was used before, and it does use some scaling logic that trys to avoid overflows, maybe that logic was tuned against a broken encode - on the first glance it didn't look all that special, but i didn't double check the math, as its not needed for my case.

The next LAV Video will do proper 10bit -> 8bit conversion, because i implemented my own algorithm for dithering (which is much faster then swscale, because i wrote it using SSE2, and avoids overflows automatically).
If the source is broken, then the source is broken. For the best quality, you shouldn't convert it to 8-bit anyway (which means you'll see the encoder error with madVR, too)

You should instead pressure x264 to fix it, rather then trying to get every 10-bit capable decoder out there to implement hacks to work around bugs in the encoder.

[cyberbeing]

You should instead pressure x264 to fix it

I just did a Google search, and it looks as if today, a libav dev finally took interest in fixing the problem. So maybe this actually will get resolved in the near-future:

http://thread.gmane.org/gmane.comp.video.libav.devel/9285

[JEEB]

Implentation #1: libav swscale (by BBB)

Implentation #2: ffmpeg swscale (by michaeln)
And they are both broken at the moment, as you have found out (at least when comparing to BT.709 and MS's limited YUV range implementations) :)

Also, it would be possible to use the borked algorithm based on the SEI encoder version information in case it exists in the stream, but the possibility of this should only be thought about after x264 gets fixed in this sense :) .

[mandarinka]

That would however try to "fix" the skew in all encodes from affected x264 revisions, which could be wrong - for example, if you feed x264 with video that already is 10bit, it won't run its conversion (I assume?), video will have the proper levels and thus the workaround will mess it up.

[nevcairiel]

Indeed, the conversion to 10bit needs to be fixed to match native 10bit material as closely as possible, so that one playback processing path can be devised that works perfectly, no matter what was the source.

[madshi]

@BetaBoy, allow me to follow up on the discussion about RGB output levels we had in the other thread. Please have a look at this sample file:

http://www.mediafire.com/?wbmyif3yf2mijti

It's a smooth 8bit gray ramp. Now please playback that ramp with CoreAVC, by outputting NV12 to madVR and switch madVR to PC levels. Afterwards switch CoreAVC to RGB output and use any renderer you like. Compare image quality. Here are screenshots for a quick comparison:

screenshot 1: (http://madshi.net/levelsChange1.png) CoreAVC outputs NV12 (video), madVR converts to RGB (PC)
screenshot 2: (http://madshi.net/levelsChange2.png) ffdshow outputs RGB (video), madVR converts to RGB (PC)
screenshot 3: (http://madshi.net/levelsChange3.png) CoreAVC outputs RGB (PC), madVR displays the result untouched
screenshot 4: (http://madshi.net/levelsChange4.png) ffdshow outputs RGB (PC), madVR displays the result untouched

You see the problem?

Changing levels in 8bit is a very bad idea, and that's what CoreAVC is doing (and ffdshow, too). The same banding artifacts will occur with YUV output, too, of course, if you modify the input/output levels. IMHO the default configuration of CoreAVC should be to leave levels untouched. If the source is encoded in limited range (16-235) then CoreAVC should output both YUV and RGB as limited range by default. If the source is encoded in full range (0-255) then CoreAVC should output both YUV and RGB as full range by default. Only this way gradients will stay smooth. madVR performs any necessary levels conversions in floating point math and dithers the result down to 8bit, which is of course much higher quality. So levels changes should best be left to the renderer, at least by default.

Of course this is just my personal opinion. If you want to implement a better quality algorithm for changing levels (using higher internal bitdepth, with a final dithering pass), that'd be fine with me, too.

Ideally, whatever you do with the levels, it would be great if you could inform the downstream filters about it. That way the renderer would be able to automatically adjust. The easiest way to pass this information along would be to use the DXVA_ExtendedFormat structure (which is an optional part of the VIDEOINFOHEADER2 structure), as mentioned in the other thread. For the output levels information you'd just have to fill the "NominalRange" part of the structure. Of course if you want to fill the other parts, too, that'd be nice. You'll find more information about how to convert the h264 header elements into DXVA_ExtendedFormat values in the following post:

http://forum.doom9.org/showthread.php?p=1519740#post1519740

[BetaBoy]

Madshi.... Thx again. We cannot do much for CoreAVC 3.0 considering we are about to go 'gold' and release it but we are looking at the suggestions for followup releases.

[madshi]

That's just fine, BetaBoy. After all CoreAVC works well with its default settings. So my suggestions are not urgent.

[TheFluff]

IMHO the default configuration of CoreAVC should be to leave levels untouched. If the source is encoded in limited range (16-235) then CoreAVC should output both YUV and RGB as limited range by default. If the source is encoded in full range (0-255) then CoreAVC should output both YUV and RGB as full range by default.

Excuse me for a second here, but how is it that the phrase "limited range RGB" makes any sense whatsoever to you? I'd very much like to be enlightened, because you seem to be in the possession of some truly Lucy-in-the-Sky-with-Diamonds-level shit and I'd like to try it.

[mandarinka]

I was under the impression that projectors (and maybe some TVs) commonly expect limited range in rgb too, meaning that if you feed them full range, you'll get overcontrasted image with darks and brights clipped to black/white.
In any case ffdshow (and other decoders) do have a setting to output 16-235 range RGB, apparently for this reason...

(please correct me if that's not true...)

[madshi]

Excuse me for a second here, but how is it that the phrase "limited range RGB" makes any sense whatsoever to you? I'd very much like to be enlightened, because you seem to be in the possession of some truly Lucy-in-the-Sky-with-Diamonds-level shit and I'd like to try it.
There are 2 types of displays:

(1) Computer monitors, mostly LCD.
(2) CE (Consumer Electronics) displays, like LCD TVs, plasma TVs, projectors etc.

In the computer world, black has historically always been RGB(0,0,0). So if you send RGB to a (1) display, the display expects black to be at RGB(0,0,0). However, in the CE world, things are quite different. There we have things like BTB (blacker than black) and WTW and black is usually expected to be at RGB(16,16,16). Standalone hardware DVD players and Blu-Ray players usually output black at RGB(16,16,16), if you switch them to RGB output. Some DVD/Blu-Ray players and some TVs/projectors can be switched between limited range (black = 16) and full range (black = 0) mode, but not all. The PS3 and Xbox360 can be switched between limited and full range, too, AFAIK. There are different names for that switch, e.g. Limited vs. Full, or Studio vs. Computer, or TV vs. PC, or Normal vs. Extended etc...

99.9% of all h264 consumer content is encoded in YCbCr 4:2:0 with black at Y = 16 and white at Y = 235. If you switch the h264 decoder to output RGB, the decoder can either output black at RGB(0,0,0) or at RGB(16, 16, 16). The latter requires less processing. Outputting black at RGB(0,0,0) means that after YCbCr -> RGB conversion you need to perform a stretch operation to get black from 16 to 0 and white from 235 to 255. This stretch operation introduces visible banding if you perform it in 8bit. That's why I'm asking Core to skip this stretch operation and to inform the downstream filters about whether the CoreAVC RGB output has black at 0 or 16.

All cleared up now?

[fastplayer]

All cleared up now?
By the way, you explain technical "stuff" very comprehensible. :)

[madshi]

Thanks. :)

[Stephen R. Savage]

Outputting black at RGB(0,0,0) means that after YCbCr -> RGB conversion you need to perform a stretch operation to get black from 16 to 0 and white from 235 to 255. This stretch operation introduces visible banding if you perform it in 8bit.

This is a completely illogical claim. Conversion between YUV and RGB and from limited to full-range are both linear processes, so there is no reason that they must be done in two steps.

[madshi]

This is a completely illogical claim.
You could have left this sentence away, you know. No need to be impolite.

Conversion between YUV and RGB and from limited to full-range are both linear processes, so there is no reason that they must be done in two steps.
Doing it in two steps is how it's usually done, and probably done by CoreAVC. You're right, though, in that it could be done in one step. But even if you do it in one step, you'll still get banding, if you use simple rounding. The only way to avoid banding would be to use dithering or to output a higher bitdepth than 8 bit.

[Stephen R. Savage]

Doing it in two steps is how it's usually done, and probably done by CoreAVC. You're right, though, in that it could be done in one step. But even if you do it in one step, you'll still get banding, if you use simple rounding. The only way to avoid banding would be to use dithering or to output a higher bitdepth than 8 bit.

Everything that you say here is correct, and yet it still doesn't logically follow that "limited-range" RGB is a preferable output format. Even if the default output range in CoreAVC were changed, it would still be rounded (probably truncated, actually), etc. Further, a direct conversion from limited YUV to full-range RGB does not add any more rounding error than a conversion from limited YUV to limited RGB, since both involve the same number of 8-bit multiply/divide operations.

The only thing that will change is that suddenly everyone will have broken levels in something that used to work just fine.

[madshi]

Everything that you say here is correct, and yet it still doesn't logically follow that "limited-range" RGB is a preferable output format. Even if the default output range in CoreAVC were changed, it would still be rounded (probably truncated, actually), etc. Further, a direct conversion from limited YUV to full-range RGB does not add any more rounding error than a conversion from limited YUV to limited RGB, since both involve the same number of 8-bit multiply/divide operations.
I'm sorry, but I disagree. Let's look at a simple gray ramp:

Y: 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, ...
Cb: 128, 128, 128, ...
Cr: 128, 128, 128, ...

Try to convert that to full range RGB and to limited range RGB with rounding. You'll get smooth output with no banding with limited range RGB, but you'll get banding with full range output. The simple reason for that is that with a gray ramp the Y values map directly to the limited range RGB output. E.g. YCbCr 16,128,128 converts to exactly RGB 16.0,16.0,16.0. Of course it would be preferred to perform dithering after the YCbCr -> RGB conversion. When dithering is used, both limited and full range output should be banding free.

If you don't believe me, just look at the screenshots I posted. Compare ffdshow limited vs. full range RGB output. These screenshots undeniably proof what I just said. Unless you believe that I faked the screenshots or made a mistake? If so, it should be easy for you to double check yourself.

The only thing that will change is that suddenly everyone will have broken levels in something that used to work just fine.
All renderers (other than madVR) passthrough RGB untouched. If CoreAVC always outputs PC levels, users which have a projector/TV which needs TV levels will get broken levels, while users with a computer type monitor will get correct levels. IMHO this behaviour is already broken right now. CoreAVC has three switches for output levels, "PC", "TV" and "Auto detect". All I'm asking for is to honor these switches for RGB output, too. If users need CoreAVC to output RGB with PC levels, like it does now, it should be no problem for them to manually change the CoreAVC settings accordingly.

[Stephen R. Savage]

All renderers (other than madVR) passthrough RGB untouched. If CoreAVC always outputs PC levels, users which have a projector/TV which needs TV levels will get broken levels, while users with a computer type monitor will get correct levels. IMHO this behaviour is already broken right now. CoreAVC has three switches for output levels, "PC", "TV" and "Auto detect". All I'm asking for is to honor these switches for RGB output, too. If users need CoreAVC to output RGB with PC levels, like it does now, it should be no problem for them to manually change the CoreAVC settings accordingly.

Shocking! Computer software is written to specifications that match typical computer hardware!

[Audionut]

Shocking! Computer software is written to specifications that match typical computer hardware!

That might have made sense 10 years ago. Now it just goes to show that your a typical user who thinks his setup is default and anything else shouldn't be accounted for.

[robpdotcom]

Isn't the fact that madVR does things differently from "all other renderers" one of the things that makes it better than all other renderers?

[Stephen R. Savage]

That might have made sense 10 years ago. Now it just goes to show that your a typical user who thinks his setup is default and anything else shouldn't be accounted for.

What about my typical user?

As for this meaningless ad hominem attack, what madVR and CoreAVC do make no difference to me, seeing as I use neither. I merely wanted to point out the delusion required to claim that "limited-range RGB" as a default made any sense.

[Nevilne]

Honestly adding a menu choice of how to handle RGB input seems to be the best option for madvr, you can see some valid points from both camps.

But back to coreavc, are there any benchmarks from 3.0 beta or is it under NDA?

[madshi]

Shocking! Computer software is written to specifications that match typical computer hardware!
In case you didn't know, a key market for both CoreAVC and madVR is HTPCs. And if you look up "HTPC" at wikipedia, it says:

> Standard PC units are usually connected to
> a CRT or LCD display, while HTPCs are designed
> to be connected to a television

And "televisions" typically expect limited range RGB, not full range RGB. So designing CoreAVC to forcefully always output full range RGB doesn't make much sense.

I merely wanted to point out the delusion required to claim that "limited-range RGB" made any sense.
Delusion? You want to ignore the fact that (rounded or truncated) full range RGB output results in visible banding, while limited range RGB output does not (see screenshots)? You want to ignore the fact that TVs and projectors need limited-range RGB? Ok, I give up. If you choose to ignore hard proof and facts then there's really no point in discussing this any further.

[Stephen R. Savage]

Delusion? You want to ignore the fact that (rounded or truncated) full range RGB output results in visible banding, while limited range RGB output does not (see screenshots)? You want to ignore the fact that TVs and projectors need limited-range RGB? Ok, I give up. If you choose to ignore hard proof and facts then there's really no point in discussing this any further.

If some esoteric hardware requires a different, obscure RGB format, the answer is not to change the default behavior and break compatibility with the majority of end-users. As for your claim regarding banding, first of all, essentially no end-user cares about quality; this is a verifiable fact. More importantly, if you really cared about quality, you would only use the YV12/NV12 format anyway, since any RGB, regardless of range, will already be rounded or truncated to 8-bit.

When someone chooses RGB output, especially in something like CoreAVC which doesn't even pretend to have high-quality conversion, chances are they aren't concerned about banding or anything like that, but merely want to quickly display something in some application that requires RGB. And chances are, such legacy applications will require full-range RGB, the standard format.

Besides, even if the default behavior for RGB conversions were changed, you would still be on here telling people to use YV12 and trust in the Magic Power (TM) of madVR color conversions. So, really, I don't see why it's such a big deal to you.

[ranpha]

If some esoteric hardware requires a different, obscure RGB format, the answer is not to change the default behavior and break compatibility with the majority of end-users. As for your claim regarding banding, first of all, essentially no end-user cares about quality; this is a verifiable fact. More importantly, if you really cared about quality, you would only use the YV12/NV12 format anyway, since any RGB, regardless of range, will already be rounded or truncated to 8-bit.


Since when TVs and projectors has been categorized as 'esoteric hardware'? I bet that TV/projectors that supports limited range will greatly outnumber those that support full range RGB. And since when limited range RGB has become a 'different,obscure RGB format'? Engineers from AMD (the champion of limited range RGB) will want to have a word with you about that.

BTW, 'quality' and 'RGB' are not mutually exclusive options even in the context of CoreAVC. All CoreCodec needs to do is to fix the bug mentioned in this thread before. You can have your cake and eat it too you know.

When someone chooses RGB output, especially in something like CoreAVC which doesn't even pretend to have high-quality conversion, chances are they aren't concerned about banding or anything like that, but merely want to quickly display something in some application that requires RGB. And chances are, such legacy applications will require full-range RGB, the standard format.

If one of the 'applications that requires RGB' (which definitely is not mainstream video players like MPC-HC or WMP or KMPlayer et. al.) can actually use CoreAVC, you can definitely subvert its requirements and make it output YV12 or NV12.

Oh BTW, can you give an example of an 'application that requires RGB' that can also use CoreAVC? I cannot think of one.

[ryrynz]

I believe this argument has gone beyond logical debate.
I think your time could be much better spent doing something other than arguing the merits of implementing a feature for which the both of you have successfully proven the benefits of implementing. ;)

[BetaBoy]

We just went 'gold' with CoreAVC 3.0. We will release CoreAVC 2.6 later today and have set a release date of Tuesday Sept 6th for CoreAVC 3.0.

[BetaBoy]

The release is being sent now...


CoreAVC H.264 Video Codec - Version 2.6.0.0 (20110830)
- ADD: DXVA fallback to software
- ADD: Improved DXVA handling for interlaced streams
- ADD: DXVA 2 Long slice support
- CHG: Use container AR when there is no stream AR
- FIX: Improved Frame order handling
- FIX: Hardware deinterlacing field order

Haali Media Splitter - Version 1.11.233.7 (20110830)
- FIX: Various DTS Audio bugs
- ADD: Improved DTS support
- ADD: Support for MVC 3D videos

[BetaBoy]

As you can see we also pushed out a new Haali splitter that addresses some (not all) of the leading issues as well as adding support for MVC. This is ahead of of releasing CoreMVC 3D which we expect to do after the CoreAVC 3.0 dust settles.

[TheShadowRunner]

If/Since CoreAVC 2.x won't support decoding of Hi10P/10-bit video, would it be possible for you to please do a maintenance release for 2.x so that it automatically won't be used when a 10-bit video is played?

Guess not :/

[BetaBoy]

Try it and see.

[nevcairiel]

Try it and see.

I did, and 2.6 does infact not refuse, it trys to decode, and ends up with a black screen. It doesn't fallback to a decoder that is 10-bit capable.

[TheShadowRunner]

Try it and see.

Oh an unadvertized feature? This sounds promising ;)
A somewhat related question, I'm stuck with nvcuvid.dll version 266.58 because of a nVidia bug that has Windows XP BSOD when playing flash videos with certain cards (8x00 / 9x00) and any drivers above 266.58.
Does the nvcuvid.dll version make a difference when it comes to decoding with CUDA? (I guess I could replace my current nvcuvid.dll 266.58 by one from the latest 280.26 driver package if it does)

[BetaBoy]

You beat me too it (on the feature) ;-)

nevcairiel... correct 2.6 will not fallback to another codec, but show a black screen for hi10 content.

TheShadowRunner... From our discussions internally, no it should not matter what DLL you are using.

betaking... Sorry about that... the accounts are a 'rolling' update and sometimes emails get ahead of the update to your account. Check back in a lil bit, it should be there.

[betaking]

Hi.betaboy I received e-mail prompted me to go to the website to update to 2.6, but I found that after landing, or you can still download version 2.5.5!
and i can not found Haali Media Splitter - Version 1.11.233.7 (20110830) in haali's website!

[TOM_SK]

[Off-topic]
@Marlin: Hi, is there any affiliate/reseller program for CoreAVC Professional Edition ? I looked everywhere on CoreCodec website but can't find any.

Thanks.

[BetaBoy]

TOM_SK... sure.... once you login to your account... Under 'Corporate' you'll see 'Affiliates', just hi the 'activate' button to enable it for your account. It will then give you the info you need.

RE: https://customers.corecodec.com/affiliates.php

[CiNcH]

2.5.5 broke DXVA with DVBViewer DVBSource filter

2.6 breaks compatibility with DVBViewer DVBSource filter "completely". Tray icon is blue (indicating no DXVA) and picture inside the DVBViewer stays black. It seems as if no data arrives at the renderer.

You promised us to look into the thing!?

[BetaBoy]

and i can not found Haali Media Splitter - Version 1.11.233.7 (20110830) in haali's website!
You will not find this version on Haali's site yet as its only in CoreAVC 2.6 and 3.0 because of all the work we have done to it.

I'll sync with him on updating his site.

[BetaBoy]

CiNcH.... the testers we had for DVBViewer did not indicate anything was broken. We are looking into it (again).

[BetaBoy]

A note... we are rolling out the 2.6 update to current customers first. We will list it's availability to the public next.

[nussman]

2.5.5 broke DXVA with DVBViewer DVBSource filter

2.6 breaks compatibility with DVBViewer DVBSource filter "completely". Tray icon is blue (indicating no DXVA) and picture inside the DVBViewer stays black. It seems as if no data arrives at the renderer.

You promised us to look into the thing!?

Same here ...
Maybe you need some new testers for dvbviewer? :rolleyes:

[BetaBoy]

We also found that some people are getting the error: Error loading C:\Program Files\CoreCodec\CoreAVC Professional Edition/CoreAVCDecoder64.ax
The specified module could not be found.

Added to the todo.

[TOM_SK]

We also found that some people are getting the error: Error loading C:\Program Files\CoreCodec\CoreAVC Professional Edition/CoreAVCDecoder64.ax
The specified module could not be found.

Added to the todo.

Got the same error when installing CoreAVC on 32-bit Windows 7 HP. Btw, pls. check your pm.

[BetaBoy]

yeah.... everything installs fine. It just looks like for some 32bit users they get the error.

[nussman]

I got the same message @Win7 32bit, but thats not the main problem with coreavc 2.6! ;)