madVR - high quality video renderer (GPU assisted)

[madshi]

because if you have a 1080p source, a 720p display and do all kind of PP(sharpening etc) you wanna do it in the native res of your display IMHO

same for SD, it'll look far better if you upscale to 720p then PP at this resolution than upscale PP'ed SD to 720p in mVR
Generally noise reduction should be done on the unscaled image while sharpening etc should be done on the scaled image. So you could do noise reduction and then let madVR upscale. But if you want to do sharpening, probably doing that before madVR is better.

[leeperry]

if you want to do sharpening, probably doing that before madVR is better.
indeed. but because upscaled SD is a no-no in YV12, you'd need to use ColorMatrix() to convert from 601 to 709 in the ffdshow Avisynth filter. Problem solved.

[yesgrey]

It's probably a matter of preference though, do u want perfect audio or video playback..
You can have both, but you would need an RME or Lynx soundcard.:D
Yes, I know they are not cheap, I am only saying that it's possible.
There is no magic way to change the playback speed, even slightly and leave the audio untouched and at the right pitch.
Yes, you are correct, it's not magic, it's for real. See the answer above.
I think Yesgrey believes it is effectively lossless (yes I know that is oxymoron).
No, I don't believe that. That's why I've searched for the solution I beieved and wanted, and created it...;)

[Jong]

You can have both, but you would need an RME or Lynx soundcard.:D
Yes, I know they are not cheap, I am only saying that it's possible.

Yes, you are correct, it's not magic, it's for real. See the answer above.

No, I don't believe that. That's why I've searched for the solution I beieved and wanted, and created it...;)OK. fair enough! I was not accusing you. But I'm sure on Reclock's forums you felt it was a very very close call and inaudible to most. But if I have misrepresented I apologies unreservedly!

[yesgrey]

- does this loss in quality even with a LOSSLESS original track get lost in the IMPERFECT equipment and acoustic environment most, if not all of us, live with?

IMHO this is what I think about it. I've searched a lot for my "perfect" solution, not because I thought I could hear any difference (with my current audio gear), but because I needed the CPU power that was used for performing the resampling in reclock.;)

good point. I assumed it would simply change the sampling freq once and for all...if it keeps changing the audio pitch constantly, then I'd definitely prefer resampling...yesgrey can you confirm?
What madshi said only applys to reclock when resampling. When using my method the audio is always at the correct pitch. Currently the drivers do not support changing the soundcard's clock in real time, so I set it manually. This means that could exist any desync between audio and video when using my method, but the desync will be the difference between both clocks, which should be negligible...;)

[flanger216]

Quite true. But then the question is: Why scaling in ffdshow with 8bit precision when madVR scales with 16bit precision?

Generally noise reduction should be done on the unscaled image while sharpening etc should be done on the scaled image. So you could do noise reduction and then let madVR upscale. But if you want to do sharpening, probably doing that before madVR is better.

I was going this route for two reasons: I have an underpowered GPU and a much more powerful CPU, and I can use spline resizing in ffdshow no problems; with madVR doing the scaling, I start to get stutters with anything past bicubic. I also presumed that it'd be more effective to do NR after scaling, so I was performing denoise3D after the resizer in ffdshow.

But I didn't realize that, of course, the scaler in madVR would operate at a higher bitdepth. And if NR should in fact be done before any scaling, then that leaves zero need or reason to scale with ffdshow; I'll just stick with the bicubic resizer in madVR.

[yesgrey]

OK. fair enough! I was not accusing you. But I'm sure on Reclock's forums you felt it was a very very close call and inaudible to most.
Yes, it's true, but inaudible is not lossless.;)
For example, a lot of receptors have a digital volume. If you feed them an analog signal, it will convert the signal to digital and then back to analog, because it's the only way it could change the volume. I have a receiver like this, that's why I said that currently I cannot hear any difference...:o I'm planning on building my own amps full analog with analog volume control, but have not found the time yet...

This is my last post about audio. Let's stop hi-jacking madVR's thread. If you want to keep discussing this, please create a new thread about it...;)

[flanger216]

well the dialogs sound too slow if you A/B of course, and on my favorite movie(Revolver) I can hear when it doesn't play at 24fps :scared:

0.1% pitch difference is vey much audible I think, for music at least..

Definitely - the differential threshold for human hearing is roughly 0.4 Hz (averaged across the spectrum), and a 0.1% pitch decrease is going to cause a lot more variance than that.

[Egh]

Your renderer uses the video resolution that comes from the previous filter. If he has resized SD video to HD in ffdshow leaving it in YV12 it'll be converted to RGB using BT.709 instead of BT.601.
Quite true. But then the question is: Why scaling in ffdshow with 8bit precision when madVR scales with 16bit precision?

Still, I guess the problem could be worked around. But things like this are really ultra low priority for me right now.


I second that this feature needs to be implemented. a) Haali Renderer has it b) quite a lot of times you may wish to use different decoder from ffdshow and thus no option to apply AVS conversion. c) Encoders may have or have not applied ColorMatrix() for the release. Therefore resolution is not reliable factor to identify 601/709. In my opinion, best feature to implement would be a choice of LUT to load, so that a user could switch a LUT with reinitialization on the the fly.

[Mark_A_W]

Just FYI guys, I've been playing with Beliyaal's MPC-HC build, with Reclock in "Slave" + "Original Speed" mode (therefore "off"), using WASAPI.


Playback is 99% perfect. After about an hour I get a small audio/video glitch, but otherwise smooth - smooth enough that I can't detect any anomolies on my projector.

It seems to me, that after all these years, Reclock resampling may not be needed - and I'm Reclock's number 1 fan. I'm not insisting on the resampling bit! If playback can be smooth without it, all the better (perfect audio!).

I'm perfectly happy with 23.976fps, I can't tell the difference between that and 24.000fps. As long as Madshi's solution isn't incompatible with Reclock, you can turn on PalSpeedDown or change 23.976 to 24.000, but if MadVR works without it, that means we can ditch the resampling for "normal" use, because it is a dirty hack.

(I use Room Correction too :) Although, not for testing.)

[racerxnet]

Madshi,

When is the next update coming out??????

Are we on track now????:confused:

Mak

[Thunderbolt8]

I guess at the weekend.

[Bitmonster]

It seems to me, that after all these years, Reclock resampling may not be needed - and I'm Reclock's number 1 fan. I'm not insisting on the resampling bit!
ReClock's resampling was never needed for AC3/DTS S/PDIF audio. And the audio frame drop/repeat problem can be worked around on most displays, if you take the time to optimize them correctly. Since PowerStrip has the option to get the theoretical pixel clock in high resolution this is actually much easier to achieve than it was some years ago.

I have documented this approach some time ago, but it is currently only written in german:
http://www.avpedia.org/wiki/HTPC_Bildfrequenz_Optimierung

With some translation tools you might be able to get the idea (and the possibility to actually use it). What I havn't tested till now is if this also works for externally decoded HDMI audio (because of missing equipment). But the chance is there, that it will work the same way.

[ericgur]

My occupation for the last 6 years has been development of video processing algorithms for profesional video processors (ASIC) and later GPU/SW. I'm very familiar with high quality scaling, denoising, deinterlacing, film cadence detection and color manipulation.

Using FFDShow to scale YCbCr is not ideal for both quality and performance issues:
Quality - the CbCr components are compressed by a factor of ~2 compared to RGB, meaning a difference of 1 Cb or Cr can mean 2 values of RGB. That's the main reason for keeping 16 bits after the scaler. Also the (MadVR) scaler produces 4:4:4 video which doesn't need UV/CbCr upsampling before conversion to RGB.

Performance - you'll spend time sending large buffers to the GPU. In the future MadVR might perform other PP algorithm like automatic brightness/contrast enhancements, color enhancement (smart saturation, skin tone correction), noise reduction, etc. These would work faster if done before upscaling.

Sharpening can be done in 2 places, before and after the scaler and always after noise reduction, never before.
Pre-scaling sharpening must be limited to a weak high pass emphasis filter (all negative coefs except the center) and the post scaling filter should be a larger (at least 5x5) adaptive symmetric filter to avoid over-sharpening.

Here's CSC with brightness, contrast, saturation without hue (who needs hue these days?).

Y' = Brt + (Cnt * 1.164) * (Y-16)
Cb = Cb - 128
Cr = Cr - 128
R = Y' + (Sat * Cnt * 1.5957) * Cr
G = Y' + (Sat * Cnt * -0.3916) * Cb + (Sat * Cnt * -0.8135) * Cr
B = Y' + (Sat * Cnt * 2.0175) * Cb

R = max(min(R, 255), 0)
G = max(min(G, 255), 0)
B = max(min(B, 255), 0)

Note - magic numbers corespond to BT601 - PC scale.
Most of the multipliers used are calcualted offline - the shader only multiplies Cb twice (for G and B) and Cr twice (for R and G). To make this work for 16bit multiply the round numbers by 256.

About noise reduction - it's never done after scaling, always on native resolution, but after deinterlacing. A good noise reduction algorithm performs smooting (filtering) in the temporal axis for static pixels (IIR style filter is best) and spatial filtering (adaptive median filter works nice) for motion pixels. You'll need a decent motion detector to implement this.

[Mark_A_W]

ReClock's resampling was never needed for AC3/DTS S/PDIF audio. And the audio frame drop/repeat problem can be worked around on most displays, if you take the time to optimize them correctly. Since PowerStrip has the option to get the theoretical pixel clock in high resolution this is actually much easier to achieve than it was some years ago.

I have documented this approach some time ago, but it is currently only written in german:
http://www.avpedia.org/wiki/HTPC_Bildfrequenz_Optimierung

With some translation tools you might be able to get the idea (and the possibility to actually use it). What I havn't tested till now is if this also works for externally decoded HDMI audio (because of missing equipment). But the chance is there, that it will work the same way.

I used to do this, back in the days of AC3/DTS, but that ceased a few years back now.

That method is no good for DD+/TrueHD/DTS-MA/LPCM. For me now, it's analogue out, as I don't have HDMI (and have no plans for it, in fact I'm planning to ditch the Rotel Processor all together and go direct from the analogue out to the poweramps).


Edit: In my earlier post I should have said "Without Reclock messing with the audio", rather than "Resampling".

[Bitmonster]

That method is no good for DD+/TrueHD/DTS-MA/LPCM.
Why?

Even if you do analogue out, this method also ensures that any correction will only be done in the most minimal way that is achievable.

[Mark_A_W]

Why?

Even if you do analogue out, this method also ensures that any correction will only be done in the most minimal way that is achievable.

Because even if the clock is perfect, Reclock is still resampling.

This is close to where I am now, with Reclock - my clock is not quite fully optimised, but it's very, very close.


If there is another way to get smooth playback, and from testing Beliyaal's MPC HC builds there is, then I'm all for it.

[yesgrey]

leeperry is right. The 3D LUT is giving different results compared with the PS script.
I have already found the cause, and changed cr3dlut to give the same results as the PS script, but I'm still investigating it to see which is the more accurate...
If you want to see the results of my investigation go here (http://forum.doom9.org/showthread.php?p=1274884#post1274884).
I've also just released cr3dlut v2.1. You can also see it there...

[yesgrey]

Just FYI guys, I've been playing with Beliyaal's MPC-HC build, with Reclock in "Slave" + "Original Speed" mode (therefore "off"), using WASAPI.
Remember that this will only work correctly if you have exact 24.0/1.001 (or an integer multiple of it) as the display refresh rate.

Here's CSC with brightness, contrast, saturation without hue (who needs hue these days?).

With these formulas, I can add brightness, contrast, saturation control to the 3D LUTs, but maybe it would be rather pointless, because the user would have to create a new 3DLUT each time he changes any of the controls... what do you think?

[TinTime]

With these formulas, I can add brightness, contrast, saturation control to the 3D LUTs, but maybe it would be rather pointless, because the user would have to create a new 3DLUT each time he changes any of the controls... what do you think?

Having extra options never hurts. Well, almost never :)

I guess it makes more sense to have these controls in the renderer. However, there could be occasions where generating and storing a custom LUT for specific movies might be nice (e.g. the odd times when the brightness is off or something), although this would then involve manually overwriting the standard madVR LUTs and then replacing them after watching the movie. A bit of a pain.

[leeperry]

If you want to see the results of my investigation go here (http://forum.doom9.org/showthread.php?p=1274884#post1274884).
I've also just released cr3dlut v2.1. You can also see it there...
but ddcc works fine in realtime, and doesn't in LUT mode :confused:

anyway, to prove that the RGB32 10bit stuff matters, instead of using gray ramps you may wanna use Tim Burton's animes.

they're basically constant gray ramps, and they look most impressive :eek:

I was dubious about 10bit dithering in regular movies, but I'm sold...it feels as if there were more gradients than regular a RGB32 conversion(it also seems to offer a sharper picture) http://forum-images.hardware.fr/images/perso/screetch.gif

it gives the feeling of increased contrast ratio/deeper picture depth and simply MORE simultaneous shades on screen, just a quick example :

http://thumbnails9.imagebam.com/3295/8116b432946370.gif (http://www.imagebam.com/image/8116b432946370)

[Egh]

Performance - you'll spend time sending large buffers to the GPU. In the future MadVR might perform other PP algorithm like automatic brightness/contrast enhancements, color enhancement (smart saturation, skin tone correction), noise reduction, etc. These would work faster if done before upscaling.

Sharpening can be done in 2 places, before and after the scaler and always after noise reduction, never before.
Pre-scaling sharpening must be limited to a weak high pass emphasis filter (all negative coefs except the center) and the post scaling filter should be a larger (at least 5x5) adaptive symmetric filter to avoid over-sharpening.


Best way to make picture sharper -- tweak scaling settings ;)

Also, you are wrong with your assumption, generally speaking, it is not certain if madVR actually does upscale all the time. In case your native LCD resolution is only fit for 720p then 1080p will be downscaled by the renderer.

[Mark_A_W]

Remember that this will only work correctly if you have exact 24.0/1.001 (or an integer multiple of it) as the display refresh rate.




Of course!! I have a true multisync CRT projector :)

[madshi]

My occupation for the last 6 years has been development of video processing algorithms for profesional video processors (ASIC) and later GPU/SW. I'm very familiar with high quality scaling, denoising, deinterlacing, film cadence detection and color manipulation.
Cool! :) Do you still work in that area? Do you have signed an NDA or something that would stop you from sharing some of your knowledge with the HTPC world?

Sharpening can be done in 2 places, before and after the scaler and always after noise reduction, never before.
Pre-scaling sharpening must be limited to a weak high pass emphasis filter (all negative coefs except the center) and the post scaling filter should be a larger (at least 5x5) adaptive symmetric filter to avoid over-sharpening.
I've been told by a Gennum employer that although you can do sharpening/detail enhancement before scaling, it's better for image quality to do it after (up)scaling.

Here's CSC with brightness, contrast, saturation without hue (who needs hue these days?).
Thanks much - that looks simple enough!!

Do you happen to have the CSC coefficients at hand for the other 3 cases (BT601 video levels, BT709 PC/video levels)?

With these formulas, I can add brightness, contrast, saturation control to the 3D LUTs, but maybe it would be rather pointless, because the user would have to create a new 3DLUT each time he changes any of the controls... what do you think?
Wouldn't it make more sense to do these things via shader math and "only" use the 3dlut for gamut/gamma correction? Of course that would mean that the 3dlut would do Y'CbCr -> Y'CbCr. Or maybe Y'CbCr -> YCbCr, if we implement linear light processing. Brightness, contrast, saturation and CSC would then be done via shader math. Should be no problem performance wise, since my current shaders are mostly memory bandwidth limited, anyway. Adding in some math should not slow down things much, if at all. What do you think?

anyway, to prove that the RGB32 10bit stuff matters, instead of using gray ramps you may wanna use Tim Burton's animes.

they're basically constant gray ramps, and they look most impressive :eek:

I was dubious about 10bit dithering in regular movies, but I'm sold...it feels as if there were more gradients than regular a RGB32 conversion(it also seems to offer a sharper picture)

it gives the feeling of increased contrast ratio/deeper picture depth and simply MORE simultaneous shades on screen
Could you create a comparison screenshot so we can directly compare 16bit + dithering vs. 8bit? Thanks!

Also, you are wrong with your assumption, generally speaking, it is not certain if madVR actually does upscale all the time.
Has he assumed that somewhere? I don't read his post like that. Furthermore, it doesn't matter much if you upscale or downscale. Noise/artifact removal should still be done before scaling (because scaling makes the artifacts more difficult to remove). However, sharpening should probably be done on the higher resolution picture. So that would be before downscaling and after upscaling. ericgur?

[tetsuo55]

it gives the feeling of increased contrast ratio/deeper picture depth and simply MORE simultaneous shades on screen, just a quick example :That image looks like a real life photo

[cyberbeing]

That image looks like a real life photo
That's because it is (stop motion photography/animation with dolls). :p
It's not an anime, cartoon, or 3d rendered.

[FoLLgoTT]

@madshi
There is an idea in my head which may be realized with a 3D LUT.

The problem: If you calibrate your projector to a gamma factor other than 2.2 (e.g. a CRT in a dark room looks great with 2.5) the saturation doesn't stay constant over the luminance anymore. In HCFR you can measure and visualize this fact easily. Saturation will be too high in the middle IRE's, looking like a hump.

Would it be possible with a 3D LUT to only change the saturation over luminance? Maybe an editor similar to my VideoEqualizer (http://www.avsforum.com/avs-vb/showthread.php?t=1042160) would be a good idea. I don't know any hardware or software which can do this. If this works it could be unique. :)

[tetsuo55]

That's because it is (stop motion photography/animation with dolls). :p
It's not an anime, cartoon, or 3d rendered.

Yeah but mr burton does a lot of post-processing

[madshi]

@madshi
There is an idea in my head which may be realized with a 3D LUT.

The problem: If you calibrate your projector to a gamma factor other than 2.2 (e.g. a CRT in a dark room looks great with 2.5) the saturation doesn't stay constant over the luminance anymore. In HCFR you can measure and visualize this fact easily. Saturation will be too high in the middle IRE's, looking like a hump.

Would it be possible with a 3D LUT to only change the saturation over luminance? Maybe an editor similar to my VideoEqualizer (http://www.avsforum.com/avs-vb/showthread.php?t=1042160) would be a good idea. I don't know any hardware or software which can do this. If this works it could be unique. :)
That's a question for yesgrey3, not for me... ;)

[leeperry]

Could you create a comparison screenshot so we can directly compare 16bit + dithering vs. 8bit?
sure, I'll read the first page again and try!
In HCFR you can measure and visualize this fact easily. Saturation will be too high in the middle IRE's, looking like a hump.
not sure this is related, but I asked tritical if we could import the measured saturations to counter-balance the gamut conversion...but he said that the data Color.HCFR kept wouldn't help.

[Casshern]

Why not offer both:
1) apply settings after LUT conversion in RGB space - this is a little lossy but enough for a preview
2) if user likes what he sees, he can then use the second option to generate a corresponding lut - to have best quality without two uncessary operations

Having extra options never hurts. Well, almost never :)

I guess it makes more sense to have these controls in the renderer. However, there could be occasions where generating and storing a custom LUT for specific movies might be nice (e.g. the odd times when the brightness is off or something), although this would then involve manually overwriting the standard madVR LUTs and then replacing them after watching the movie. A bit of a pain.

[Mark_A_W]

...but he said that the data Color.HCFR kept wouldn't help.


I'd like to use HCFR data somehow. I have one, and I'd like to close the loop. Digital screen correction, to match my digital room correction ;)

[leeperry]

I'd like to use HCFR data somehow. I have one, and I'd like to close the loop. Digital screen correction, to match my digital room correction ;)
well, we discussed it in the ddcc() thread, have a look!
proper gamut conversions w/ mismatched saturation % is indeed pointless.
but I've discussed it w/ several peeps on the HCFR forum, and they agree w/ yesgrey....colors w/ a saturation >75% are hardly ever used(if ever).

saturations on my HC3100 look like this, so I'm quite safe :
http://www.image-load.eu/out.php/t158063_satu.png (http://www.image-load.eu/out.php/i158063_satu.png)

[FoLLgoTT]

@yesgrey3
Obviously you are the right man for my question. :)
Would a saturation editor be possible?

Here is an example for the influence of gamma of the saturation measured from my calibrated Sony G90:

Gamma 2.2:
http://img352.imageshack.us/img352/3046/saettigung22.th.png (http://img352.imageshack.us/my.php?image=saettigung22.png)

Gamma 2.5:
http://img165.imageshack.us/img165/6196/saettigung25.th.png (http://img165.imageshack.us/my.php?image=saettigung25.png)

@leeperry
sure, I'll read the first page again and not sure this is related, but I asked tritical if we could import the measured saturations to counter-balance the gamut conversion...but he said that the data Color.HCFR kept wouldn't help.

I would be satisfied with a manual input solution. I don't like to depend on a specific program. I like general and flexible solutions and have no problems with running through a few measure-setup-iterations. But importing HCFR data as a additional feature would surely be nice.

[yesgrey]

Wouldn't it make more sense to do these things via shader math and "only" use the 3dlut for gamut/gamma correction? ... Of course that would mean that the 3dlut would do Y'CbCr -> Y'CbCr. Or maybe Y'CbCr -> YCbCr, if we implement linear light processing... What do you think?
Yes, but if a user sets the values only once and never changes them again it would be nice to create a 3DLUT including those settings... Let's wait to see how it ends up. For now we could stick just with doing it in the shaders and latelly, if it would be really necessary, I could add it, it would be simple.
The Y'CbCr -> Y'CbCr is already supported by cr3dlut, though I don't know if it's working correctly, because I haven't tested it yet. The Y'CbCr -> YCbCr would be possible to add, but then we would have to perform the final gamma encoding via 1D LUTs to have higher precision and a full control of it.
But remember that several other things that we might want to do via the 3DLUT should be done at the end of all image processing... we can always use 2 3DLUTs in the processing chain... 512MB is almost the basic reference in the current graphics cards.:D

Would it be possible with a 3D LUT to only change the saturation over luminance?
I think it should be possible, I'll have to look into it to see how...

I'd like to use HCFR data somehow. I have one, and I'd like to close the loop.
It's just adapting HCFR's output to cr3dlut's input, or cr3dlut's input to HCFR's output...

Digital screen correction, to match my digital room correction ;)
That's our final goal...;)

[leeperry]

I think it should be possible, I'll have to look into it to see how...
the ability to input the measured saturations for primaries/secondaries in your cr3dlut .ini would be pure awesomeness....but it'd take someone really brave to check it in Color.HCFR again because there's a mininum of 48 manual test patterns from the original test DVD to be done in Color.HCFR :o

or a script would need to be made to send a "next chapter" hotkey to MPC and then click on "next" in Color.HCFR :cool:

I already asked the Color.HCFR coders a while back if they could sync their manual DVD patterns w/ MPC, but they didn't care too much and seemed very busy already.

and I can prolly live w/ +5% of saturation at max, considering tints >75% are hardly ever used....right? :D

[yesgrey]

the ability to input the measured saturations for primaries/secondaries in your cr3dlut .ini would be pure awesomeness...
It's also in my ToDo list... but no ETA.

[ericgur]

Best way to make picture sharper -- tweak scaling settings ;)

Also, you are wrong with your assumption, generally speaking, it is not certain if madVR actually does upscale all the time. In case your native LCD resolution is only fit for 720p then 1080p will be downscaled by the renderer.

Well, when downscaling in one or both axises, there shouldn't be any sharpening before downsampling in that axis. You'll need a low pass filter. Lanczos does it for you if you recalculate the coefficients taking the scaling factor into account. One way to shapen the image using Lanczos is to convolve a mild high pass (e.g. [-0.07, 1.14, -0.07]) with each of the Lanczos filters (for all phases).
Another option is to modify the Lanczos formula itself.

In the following code 'x' is the sampling point relative to the center-left pixel in the sampling window. winSize is the size of the sampling window (8 for Lanczos4, 6 for Lanczos3, etc.)

In standard Lanczos, 't' equals halfWinSize.
If 't' is raised (must never be smaller than halfWinSize) the Lanczos function will more and more resemble a cropped sinc function and produce a sharper image with more ringing.

static const double pi = 3.14159265359;
static const double eps = 1e-9;
double Sinc(double x)
{
return (fabs(x) < eps) ? 1.0 : sin(x) / x;
}
double Lanczos(double x, int winSize, double t = 0)
{
int halfWinSize = winSize >> 1;
if (t < halfWinSize)
t = halfWinSize;

if(fabs(x) >= halfWinSize)
return 0.0;

x *= pi;

return Sinc(x) * Sinc(x/t);
}

[ericgur]

Cool! :) Do you still work in that area? Do you have signed an NDA or something that would stop you from sharing some of your knowledge with the HTPC world?

Sadly I can't and won't share propriatery algorithms. At this level, my text book knowledge is helpful without the getting the lawyers involved. All the information I provide can be found in printed text books and articles and is not a trade secret.

I've been told by a Gennum employer that although you can do sharpening/detail enhancement before scaling, it's better for image quality to do it after (up)scaling.

You can try for yourself, the trick is to do a mild sharpening before upscaling, preferably after noise reduction. And the filter must be a high pass.

Do you happen to have the CSC coefficients at hand for the other 3 cases (BT601 video levels, BT709 PC/video levels)?

The CSC coeeficients are taken from Video Demystified - a very useful book for video. I don't have a copy of it at home, so I'll post them later on. Send me a PM with your mail, I have a PDF of this book and I can mail it to you (when I return to work after the weekend).

Wouldn't it make more sense to do these things via shader math and "only" use the 3dlut for gamut/gamma correction? Of course that would mean that the 3dlut would do Y'CbCr -> Y'CbCr. Or maybe Y'CbCr -> YCbCr, if we implement linear light processing. Brightness, contrast, saturation and CSC would then be done via shader math. Should be no problem performance wise, since my current shaders are mostly memory bandwidth limited, anyway. Adding in some math should not slow down things much, if at all. What do you think?

I think the big LUTs are not user friendly due to their load time. What you're currently doing is no match for the modern GPUs. Even the low end integrated GPUs can do it easily.


Has he assumed that somewhere? I don't read his post like that. Furthermore, it doesn't matter much if you upscale or downscale. Noise/artifact removal should still be done before scaling (because scaling makes the artifacts more difficult to remove). However, sharpening should probably be done on the higher resolution picture. So that would be before downscaling and after upscaling. ericgur?

See my previous post on how to do sharpening. The mild pre upscaling sharpening gives little sharpness with little artifacts. Sharpening after scaling is more tricky as you'll need an adaptive algorithm - preferebly an edge directed filter. These are hard to do in realtime SW.

[madshi]

I think the big LUTs are not user friendly due to their load time. What you're currently doing is no match for the modern GPUs. Even the low end integrated GPUs can do it easily.
They can do CSC via shader math, but not full gamut correction. I have some ideas on how to mask the LUT loading times... ;)

See my previous post on how to do sharpening. The mild pre upscaling sharpening gives little sharpness with little artifacts. Sharpening after scaling is more tricky as you'll need an adaptive algorithm - preferebly an edge directed filter. These are hard to do in realtime SW.
Have you seen e.g. the "LimitedSharpenFaster" AviSynth sharpening filter? What is your opinion about its quality?

Thanks!

[honai]

@madshi

Now that you've mentioned it I think it's safe to pull up a request for including LSF or, better yet, LSFmod in madVR. ;)

[flanger216]

@ericgur

Is it consistently useful to apply a mild, post-scaling sharpener for video presentations? Or is it something that's typically only done on softer source material?

I'm sure this is a "try it and see if you like it" sort of thing; I was just wondering if you had any theoretical guidelines on the subject.

[yesgrey]

I think the big LUTs are not user friendly due to their load time. What you're currently doing is no match for the modern GPUs. Even the low end integrated GPUs can do it easily.
I'm not so sure about that... maybe it's true.
Here are some performance numbers:
cr3dlut
Total number of RGB combinations processed:
256*256*256 = 16777216
processing time: 4656ms (Chromatic_Adaptation 2)
processing time: 2968ms (Chromatic_Adaptation 1)
Tested in a C2Duo E2160@2.7GHz

cr3dlut using shaders???
Let's extrapolate considering the number of RGB combinations in a HD movie:
1920*1080 = 2073600/frame
expected processing time: ~575ms (CA 2)
expected processing time: ~367ms (CA 1)
Since this is a very parallelizable task, if we consider that instead of two it can process 100 at a time
expected processing time: ~11.5ms (CA 2)
expected processing time: ~7.34ms (CA 1)
This is using 64bit FP, if we use 32bit FP the times could be cut almost in half...

madshi, if the extrapolation depicted above is reasonable, maybe you would want to consider making all cr3dlut work in the shaders. There would be another big advantage... with 3DLUTs we are limited to 8bit per component input, due to the LUT size; using shaders, you could have full 16bit input/output or even more...;)

[leeperry]

Is it consistently useful to apply a mild, post-scaling sharpener for video presentations? Or is it something that's typically only done on softer source material?
that's the beauty of LSF, it won't sharpen up if there's no need :)
I've tried a lot of sharpening filters, this one is fast and the most impressive I've seen. And there's a version that uses a separate DLL(coded in assembly) to accelerate the processing slightly.

[Thunderbolt8]

would it be possible under a reasonable amount of work to have a switch which then uses the multi core cpu rather than gpu for the work? for example when watching AVC with a quadcore the cpu load is mostly ~30-50%, so theres still some power left which could be used. but I suffer from a terrible graphic card which is just too slow, so perhaps the cpu would be able to do that work as well. dunno how whether it might be too much work to add though.

[ericgur]

I'm not so sure about that... maybe it's true.
Here are some performance numbers:
cr3dlut
Total number of RGB combinations processed:
256*256*256 = 16777216
processing time: 4656ms (Chromatic_Adaptation 2)
processing time: 2968ms (Chromatic_Adaptation 1)
Tested in a C2Duo E2160@2.7GHz

cr3dlut using shaders???
Let's extrapolate considering the number of RGB combinations in a HD movie:
1920*1080 = 2073600/frame
expected processing time: ~575ms (CA 2)
expected processing time: ~367ms (CA 1)
Since this is a very parallelizable task, if we consider that instead of two it can process 100 at a time
expected processing time: ~11.5ms (CA 2)
expected processing time: ~7.34ms (CA 1)
This is using 64bit FP, if we use 32bit FP the times could be cut almost in half...

madshi, if the extrapolation depicted above is reasonable, maybe you would want to consider making all cr3dlut work in the shaders. There would be another big advantage... with 3DLUTs we are limited to 8bit per component input, due to the LUT size; using shaders, you could have full 16bit input/output or even more...;)

Running CSC on an octa core Intel (2x5450 Xeons) workstation takes a few (2-3) ms on 1080p using integer math (fixed point) on 16bit per color component - without even using SSE. GPUs have more crunching power and the time should be under 1ms. It doesn't matter how many different pixels exist as each pixel is calculated on its own. Runtime of random pixels and a black image is identical, only the resolution matters. I highly recommend working in 12 or 16 bit (12 bit is excellent and allows illegal values that can cropped later). 16 bit is overkill but in the SW world, it costs about the same as 10/12 bit. In HW using more bits is horribly expensive...

[ericgur]

would it be possible under a reasonable amount of work to have a switch which then uses the multi core cpu rather than gpu for the work? for example when watching AVC with a quadcore the cpu load is mostly ~30-50%, so theres still some power left which could be used. but I suffer from a terrible graphic card which is just too slow, so perhaps the cpu would be able to do that work as well. dunno how whether it might be too much work to add though.

Some algorithms are not GPU friendly and should be done in CPU. A good example is statistics gathering like histograms, average brightness, etc. These types of algorithms are usually very simple to implement in a multi core CPU.

[madshi]

madshi, if the extrapolation depicted above is reasonable, maybe you would want to consider making all cr3dlut work in the shaders. There would be another big advantage... with 3DLUTs we are limited to 8bit per component input, due to the LUT size; using shaders, you could have full 16bit input/output or even more...;)
I'm not sure. You planned to add some new features to cr3dlut which should make calculations a lot more complex, I guess. I don't really like the idea of doing this via shaders if we can achieve the same result via a simple texture lookup. Furthermore, shaders are usually only 32bit. Being limited to 8bit input is not as bad as it sounds, thanks to trilinear interpolation.

Running CSC on an octa core Intel (2x5450 Xeons) workstation takes a few (2-3) ms on 1080p using integer math (fixed point) on 16bit per color component - without even using SSE. GPUs have more crunching power and the time should be under 1ms. It doesn't matter how many different pixels exist as each pixel is calculated on its own. Runtime of random pixels and a black image is identical, only the resolution matters.
You are misunderstanding yesgrey3. Once again, the purpose of the 3dlut is not to do CSC, only. The main purpose of the 3dlut is complex gamut correction, of course in linear light. I expect that doing this via shader math would cost too much performance to be reasonable.

would it be possible under a reasonable amount of work to have a switch which then uses the multi core cpu rather than gpu for the work?
I have already thought about offering an option to do chroma upsampling in the CPU (with full quality = 16bit). Not sure whether I will really implement this, though.

@madshi

Now that you've mentioned it I think it's safe to pull up a request for including LSF or, better yet, LSFmod in madVR. ;)
That's 3 steps too far. Let me get basic playback working fine before even thinking about adding funny features like that...

[leeperry]

ok, so that's the french BD of "Nightmare Before Christmas"

left is downscale to 768p in ffdshow(spline36)>LSF 1.0@40>PC conversion>ConvertToYUY2()>t3dlut HD.3dlut>HR in RGB32

right is downscale to 768p in ffdshow(spline36)>LSF 1.0@40>PC conversion>mVR w/ HD.3dlut in YV12

the gamma is slightly darker in mVR(which increases the contrast), and mVR's PNG's are also 13% bigger than HR.

http://thumbnails5.imagebam.com/3298/0bce6132975822.gif (http://www.imagebam.com/image/0bce6132975822) http://thumbnails14.imagebam.com/3298/57410b32975823.gif (http://www.imagebam.com/image/57410b32975823)
http://thumbnails18.imagebam.com/3298/bbe05932975824.gif (http://www.imagebam.com/image/bbe05932975824) http://thumbnails18.imagebam.com/3298/c08bac32975828.gif (http://www.imagebam.com/image/c08bac32975828)
http://thumbnails16.imagebam.com/3298/c43b8832975831.gif (http://www.imagebam.com/image/c43b8832975831) http://thumbnails17.imagebam.com/3298/89c64f32975835.gif (http://www.imagebam.com/image/89c64f32975835)
http://thumbnails16.imagebam.com/3298/56719532975838.gif (http://www.imagebam.com/image/56719532975838) http://thumbnails17.imagebam.com/3298/62feee32975840.gif (http://www.imagebam.com/image/62feee32975840)
http://thumbnails5.imagebam.com/3298/4761a132975843.gif (http://www.imagebam.com/image/4761a132975843) http://thumbnails9.imagebam.com/3298/60574232975845.gif (http://www.imagebam.com/image/60574232975845)
http://thumbnails18.imagebam.com/3298/6823bb32975847.gif (http://www.imagebam.com/image/6823bb32975847) http://thumbnails18.imagebam.com/3298/0d5b8332975851.gif (http://www.imagebam.com/image/0d5b8332975851)
http://thumbnails.imagebam.com/3298/c2aa3b32975853.gif (http://www.imagebam.com/image/c2aa3b32975853) http://thumbnails18.imagebam.com/3298/c262da32975855.gif (http://www.imagebam.com/image/c262da32975855)
http://thumbnails18.imagebam.com/3298/443e3032975858.gif (http://www.imagebam.com/image/443e3032975858) http://thumbnails17.imagebam.com/3298/71e3ea32975860.gif (http://www.imagebam.com/image/71e3ea32975860)
http://thumbnails12.imagebam.com/3298/178a5932975862.gif (http://www.imagebam.com/image/178a5932975862) http://thumbnails18.imagebam.com/3298/d6482b32975863.gif (http://www.imagebam.com/image/d6482b32975863)

[yesgrey]

Running CSC on an octa core Intel (2x5450 Xeons) workstation takes a few (2-3)
You are misunderstanding yesgrey3. Once again, the purpose of the 3dlut is not to do CSC, only. The main purpose of the 3dlut is complex gamut correction, of course in linear light.
yes, ericgur, the 3DLUT is for performing color gamut correction, so we could get accurate colors within our display's color gamut, which usually is different (in some cases much more wider) than the source's color gamuts.

The main purpose of the 3dlut is complex gamut correction, of course in linear light. I expect that doing this via shader math would cost too much performance to be reasonable.
In fact, the first working version of my color gamut conversion worked with PS scripts inside mpc-hc, and it worked pretty fast. Of course it does not have the current 64bitFP precision, but the difference is not very high...

I'm not sure. You planned to add some new features to cr3dlut which should make calculations a lot more complex, I guess. I don't really like the idea of doing this via shaders if we can achieve the same result via a simple texture lookup. Furthermore, shaders are usually only 32bit. Being limited to 8bit input is not as bad as it sounds, thanks to trilinear interpolation.
Yes, the calculations should be more complex, but I don't know if it will be a lot slower... I presume it would not be much more slower than it is now. The biggest penalty comes from the chromatic adaptation using the full Bradford transform model.
32 bit is all that is needed, we only need 64bit because the gamma decoding/encoding.
The biggest advantage of the 3DLUT is that you know that it will never use more GPU resources that it currently uses.:)
Its biggest disadvantage, is the size.
The decision is yours. At any time, cr3dlut's code could be transformed in shader code, it would not be a very hard task...;)