FineSharp converted to shaders, for MPC-HC + madVR, or MPDN - V1.11

[madshi]

Wow, this is great - I've always been a fan of FineSharp and planned to look into making it work for madVR at some point in the future. Now -Vit- has done all the work, thank you very much!



@-Vit-, have you found some madVR settings which damage the alpha channel? I think madVR's shader processing should preserve the alpha channel between passes. The alpha channel will probably get lost during scaling (I skip the alpha channel there to save performance). But since all the FineSharp scripts should probably be executed right after each other, there should be no problem, correct?

FWIW, I plan to add a more complex shader editor to madVR in the near future, which should make it possible to put something like your HLSL FineSharp scripts into one "shader plugin" (or however it will be called). I plan to support multiple rendering passes, and also exportable options/parameters which the user can then modify through the madVR GUI or maybe assignable keyboard shortcuts etc. Of course the shaders will also be profilable, so users can use different shaders/settings depending on e.g. source resolution or framerate etc. Haven't thought this all fully through yet, though. Do you have any wishes that you'd like to see implemented in madVR to make porting of AviSynth scripts more easy/nice for you?

One thing I've been wondering about is whether it might make sense to handle grain outside of the shaders. E.g. maybe it would make sense to remove/reduce the grain before running all the shaders and scaling etc, and then after all shaders and scaling is done, maybe the grain could be put back in somehow? As opposed to every script doing this on its own separately. To be honest, although I'm a fan of several AviSynth scripts, I haven't really looked into how most of them work inside yet. So maybe what I wrote above is an old hat for AviSynth experts. Just wondering if there's anything I could do in madVR to make things easier/better...

> The shaders are not compatible with the MadVR
> setting: "run custom pixel shaders in video levels
> instead of PC levels".

I expect several shaders might have problems with that setting. I still added it as an option to make some shaders perform faster for users of very slow GPUs. In the madVR shader editor I plan to offer an option like "this shader/plugin needs to run in PC levels", which would then effectively force that "run custom pixel shaders in video levels" option off, when such a shader is activated by the user.

[Zachs]

Originally Posted by Zachs

Scratch that. I can no longer replicate the problem. It may have been a problem with my own development scripts that caused the problem which probably explains why no one else reported the problem.

Just a quick update - FineSharp scripts now run perfectly well under MPDN. It was indeed a bug in the render script system. This is now fixed.

Even better, you can now tell MPDN how / when you want to enable FineSharp via render script - e.g. you can choose to enable them for 720p videos only, or use different sets of FineSharp shader files for 1080p etc.

[-Vit-]

@madshi
No I haven't found any problem with the alpha channel. I only mentioned it because it's kinda undocumented to do that. If it's officially supported, then I'll remove that note. There is no need for the alpha channel after the shader chain completes so it's fine to drop it for the scaling afterwards.

--

The single most powerful feature that could be added to shader chaining would be the ability to store intermediate results and then use multiple inputs for a later shader
E.g.
Source->Shader1->Shader2->A
Source->Shader3->B
Source and A and B->Shader4->Output

That would make most non-temporal scripts achievable in theory I suppose (memory and performance notwithstanding). Technically it shouldn't be difficult. But what user interface to provide? The MPDN guys have some kind of C# meta-scripting language, I haven't looked at in in detail, but I think it can do something like that. Or some kind of GUI can work for simpler cases.

--

There are a couple of scripts that remove grain then add it back later, QTGMC for example. It's not especially common though. You mention adding grain back in after scaling. QTGMC has an option to scale (well, deinterlace) the grain using a different process than the degrained image - it injects extra noise with similar local properties. Would people want something like that in the madVR scaling, "resampled grain"? I don't know...

It would be ideal to do temporal (motion compensated) denoising for sharpening. With spatial-only denoising you might (temporarily) lose the fine detail that you want to sharpen. However, temporal work in the shaders will be a much greater challenge I guess.

--

I ought to note that I'm mostly retired now, lol. (edit: from video-related development that is)

[madshi]

Originally Posted by -Vit-

No I haven't found any problem with the alpha channel. I only mentioned it because it's kinda undocumented to do that. If it's officially supported, then I'll remove that note. There is no need for the alpha channel after the shader chain completes so it's fine to drop it for the scaling afterwards.

I've just rechecked my rendering chain and I see no way that the alpha channel could be lost/hurt in between of multiple shaders / shader passes. So you can remove the alpha channel note.

Originally Posted by -Vit-

The single most powerful feature that could be added to shader chaining would be the ability to store intermediate results and then use multiple inputs for a later shader
E.g.
Source->Shader1->Shader2->A
Source->Shader3->B
Source and A and B->Shader4->Output

That would make most non-temporal scripts achievable in theory I suppose (memory and performance notwithstanding). Technically it shouldn't be difficult. But what user interface to provide? The MPDN guys have some kind of C# meta-scripting language, I haven't looked at in in detail, but I think it can do something like that. Or some kind of GUI can work for simpler cases.

Haven't looked at the MPDN solution yet in detail, either. But I think the ability to store intermediate results in an extra texture and reuse it later should be doable in any case.

Originally Posted by -Vit-

There are a couple of scripts that remove grain then add it back later, QTGMC for example. It's not especially common though. You mention adding grain back in after scaling. QTGMC has an option to scale (well, deinterlace) the grain using a different process than the degrained image - it injects extra noise with similar local properties.

Wow. That's *exactly* what I've been thinking about myself. Reducing the grain, then scaling it seperately or just replacing it with grain which has similar local properties. That sounds quite awesome to me. At least in theory.

Personally I like grain and I prefer not to remove it. But I do notice that some algorithms work much better if the grain is removed/reduced before. E.g. the madVR debanding algorithm (based on f3kdb) doesn't work well if there's lots of grain in the image.

Originally Posted by -Vit-

Would people want something that in the madVR scaling, "resampled grain"? I don't know...

From what I've read in the madVR thread, some people hate grain while others love it. So everybody may have a different opinion. Personally, I like grain (at least if it's in the original source), so I personally would like a concept of "resampled grain" or "restaurated grain" or whatever a good term for that would be. I'm not sure if I'd prefer to try to scale the grain or to replace it with "new" noise which just has similar local properties, though. Whatever looks better in the end, I guess...

In any case, I think it might be good for performance if grain is removed/reduced only once before all the custom shaders are run, and then eventually (if the user wants it) added back in at the very end. This way all the custom shaders could skip the de-graining.

Originally Posted by -Vit-

It would be ideal to do temporal (motion compensated) denoising for sharpening. With spatial-only denoising you might (temporarily) lose the fine detail that you want to sharpen. However, temporal work in the shaders will be a much greater challenge I guess.

From my side, I don't think it would be a big problem. madVR already has a relatively large frame queue size, so I could make the past and future frames available to a shader. A big question would be how to make such temporal work perform well. It might make sense for a shader to only look at the current and next frame and store motion information in a separate texture. This might perform better than to reevaluate motion over multiple past and future frames again and again for every video frame. But that's a topic that the shader writer would have to worry about. Technically it would be possible for madVR to make past and future frames available. In certain situations (e.g. after a seek) past frames might have to be set to be identical to the current frame, of course.

Originally Posted by -Vit-

I ought to note that I'm mostly retired now, lol.

From your daily job? Or from home cinema related development? If you're retired from your daily job, that should give you more time for home cinema related development, no?

In any case, thanks a lot for your work, I much appreciate it!

Would you be ok if I shipped your FinalSharp shaders with a future madVR version? I'm not sure yet in which form I would do that (e.g. preconfigured plugin or totally built into the settings GUI etc), but in any case I'd of course note somewhere, at least in the readme, that the original script is from Didée and that you did the HLSL conversion. And of course I'll ask Didée's permission, too, if you're ok with it.

[Shiandow]

Originally Posted by madshi

FWIW, I plan to add a more complex shader editor to madVR in the near future, which should make it possible to put something like your HLSL FineSharp scripts into one "shader plugin" (or however it will be called). I plan to support multiple rendering passes, and also exportable options/parameters which the user can then modify through the madVR GUI or maybe assignable keyboard shortcuts etc. Of course the shaders will also be profilable, so users can use different shaders/settings depending on e.g. source resolution or framerate etc. Haven't thought this all fully through yet, though

That sounds somewhat similar to what Zachs and I are doing with MPDN. Anyway, consider me interested!

Originally Posted by -Vit-

The single most powerful feature that could be added to shader chaining would be the ability to store intermediate results and then use multiple inputs for a later shader
E.g.
Source->Shader1->Shader2->A
Source->Shader3->B
Source and A and B->Shader4->Output

That would make most non-temporal scripts achievable in theory I suppose (memory and performance notwithstanding). Technically it shouldn't be difficult. But what user interface to provide? The MPDN guys have some kind of C# meta-scripting language, I haven't looked at in in detail, but I think it can do something like that. Or some kind of GUI can work for simpler cases.

That's pretty much what we've currently achieved with the so called "Render Scripts". Although a render script also needs to handle settings and determine how it is displayed in the GUI, if you want something that more closely resembles a shader then you should look at the Filter class. For example with the current render script API you could build a Filter which performs that chain of shaders with the code:

PHP Code:

   var A = CreateFilter(Shader1, SourceFilter);
       A = CreateFilter(Shader2, A);
   var B = CreateFilter(Shader3, SourceFilter);

   return CreateFilter(Shader4, SourceFilter, A, B); 


For a good example you could look at the implementation of NEDI (the relevant part is in the "CreateNediScaler" method, the rest just deals with preparing shaders, settings, GUI, etc.)

Originally Posted by madshi

Haven't looked at the MPDN solution yet in detail, either. But I think the ability to store intermediate results in an extra texture and reuse it later should be doable in any case.

I think that MPDN's render scripts are currently the closest thing to the more advanced shader editor you had in mind, although we're still working on making it easier to write them yourself. It would be nice if MadVR could have something similar, or even better, if render scripts could be made compatible with MadVR. Anyway I think it's worth a look and if you have any questions, feel free to ask.

[-Vit-]

Originally Posted by madshi

Wow. That's *exactly* what I've been thinking about myself. Reducing the grain, then scaling it seperately or just replacing it with grain which has similar local properties. That sounds quite awesome to me. At least in theory.

Well, I didn't find it quite so awesome in QTGMC. It works well, and makes a nice fine result... but it's not much different from just rescaling the grain (although it does help to use a different rescaler for the grain). Having said that QTGMC only rescales vertically 2x, so it's not adding much grain - perhaps I shouldn't be surprised that it is underwhelming. However for a big resize, making a lot more grain - I wonder how well it will work? Will you be able to retain the "flavor" of the grain?

Originally Posted by madshi

In any case, I think it might be good for performance if grain is removed/reduced only once before all the custom shaders are run, and then eventually (if the user wants it) added back in at the very end. This way all the custom shaders could skip the de-graining.

Since not many filters degrain and readd, then surely this doesn't represent much of a performance gain? Having said that, maybe more filters should work with the grain separately...?

Originally Posted by madshi

I could make the past and future frames available to a shader. A big question would be how to make such temporal work perform well.

This discussion is heading towards replacing all of avisynth with shaders and a shader chaining language... Except that's a huge amount of work.

Originally Posted by madshi

From your daily job? Or from home cinema related development?

Retired from video related development. Too much other work. I can help out a little, but only a little...

Originally Posted by madshi

Would you be ok if I shipped your FinalSharp shaders with a future madVR version?

Of course. I probably should optimize them though. I wrote the shaders a year ago just as a convenience for myself. Didn't make any effort to make them fast.

Originally Posted by Shiandow

That's pretty much what we've currently achieved with the so called "Render Scripts".

I had a look at that NEDI script and saw it had the power needed to make complex chaining scripts. It's cool... but, and I mean this as constructive criticism, all that C# baggage decreases readability considerably. You mention that you want to make them easier to write, what do you have in mind?

[madshi]

Originally Posted by -Vit-

Well, I didn't find it quite so awesome in QTGMC. It works well, and makes a nice fine result... but it's not much different from just rescaling the grain (although it does help to use a different rescaler for the grain). Having said that QTGMC only rescales vertically 2x, so it's not adding much grain - perhaps I shouldn't be surprised that it is underwhelming. However for a big resize, making a lot more grain - I wonder how well it will work? Will you be able to retain the "flavor" of the grain?

Good questions. I'll make sure to look into the QTGMC grain algorithms when I find some time.

Originally Posted by -Vit-

Since not many filters degrain and readd, then surely this doesn't represent much of a performance gain? Having said that, maybe more filters should work with the grain separately...?

Well, it's useful at least for debanding and sharpening. Maybe also for dehaloing and deblocking? I'm not sure (yet).

Originally Posted by -Vit-

This discussion is heading towards replacing all of avisynth with shaders and a shader chaining language... Except that's a huge amount of work.

Well, "all of AviSynth" aims a bit high, I would think. But I sure would love to have at least one good HLSL implementation available for all key algorithms which are most useful for real time playback. Coming to mind is mostly good quality denoising, dehaloing, deblocking and maybe de-mosquito-ing. I'd also love to have the core functionality of QTGMC available. I can dream, can't I?

I'm not a big fan of using an additional scripting (or other) language to do shader chaining etc. Personally, I'd prefer to simply use whatever means HLSL offers, and to do the chaining/configuration with a nice and easy to use GUI.

Originally Posted by -Vit-

Retired from video related development. Too much other work. I can help out a little, but only a little...

Ah, that's too bad. But I can understand you very well. Development can eat time like crazy.

Originally Posted by -Vit-

Of course.

Thank you!

Originally Posted by -Vit-

I probably should optimize them though. I wrote the shaders a year ago just as a convenience for myself. Didn't make any effort to make them fast.

If you do have some development time available for video related stuff, and if I could voice my preferences for how to spend your limited resources I'd much prefer if you could convert maybe some other algorithm to HLSL without any optimization. With your AviSynth experience you could do that a hundred times faster than I ever could. Optimizing the HLSL shaders is something a lot of people can do (including myself). Understanding and converting AviSynth scripts is much harder - at least for me. But of course it's all up to you! I'm already very happy with getting FineSharp. It was a very nice surprise when I stumbled over this thread today.

[Shiandow]

Originally Posted by -Vit-

I had a look at that NEDI script and saw it had the power needed to make complex chaining scripts. It's cool... but, and I mean this as constructive criticism, all that C# baggage decreases readability considerably. You mention that you want to make them easier to write, what do you have in mind?

Yeah using c# is both a curse and a blessing, a lot of changes have been efforts to remove extra boilerplate, simplify common patterns etc.

Anyway did you look at the "MyRenderScript.cs" class? That is currently the simplest way to write a render script. Hopefully it can be made even simpler, although I'm not sure how much more can be done without inventing some kind of script language.

[madshi]

Originally Posted by Shiandow

I think that MPDN's render scripts are currently the closest thing to the more advanced shader editor you had in mind, although we're still working on making it easier to write them yourself. It would be nice if MadVR could have something similar, or even better, if render scripts could be made compatible with MadVR. Anyway I think it's worth a look and if you have any questions, feel free to ask.

From what I can see, your render script solution looks very powerful and flexible. But I hate DotNet, and I'm not a big fan of C#, either. I'd prefer to not introduce just another language into it all. I think not all of the flexibility and power you're offering is really necessary. I'd prefer a simpler solution and so I'll probably try to get along with just HLSL and a good GUI editor. But these are just my current thoughts. I haven't fully made up my mind yet.

[Shiandow]

Originally Posted by madshi

From what I can see, your render script solution looks very powerful and flexible. But I hate DotNet, and I'm not a big fan of C#, either. I'd prefer to not introduce just another language into it all. I think not all of the flexibility and power you're offering is really necessary. I'd prefer a simpler solution and so I'll probably try to get along with just HLSL and a good GUI editor. But these are just my current thoughts. I haven't fully made up my mind yet.

I suspected that allowing arbitrary scripts in c# was probably not entirely what you were thinking off. And I agree that you could probably achieve a lot of it using only HLSL and an easy way of chaining shaders. It would be nice if it could be made compatible with MPDN though, although that shouldn't be a problem if the way you save these 'shader chains' is easy to read.

[madshi]

I can make the file format readable, or at least explain how it works, that's not a problem.

[Zachs]

The ultimate goal for MPDN's render script is to have AviSynth like simplicity with none of the C# baggage on the top level (AviSynth scripting level), yet allowing you to go deeper and deeper if you need to implement more complicated filters, which is exactly the model AviSynth offers via its 'plugin' system. For the plugin system, you could use C++/CLI, VB.NET, or C# if you compile them into DLLs for the former two languages, or either C# script or compiled DLL if you opt for the latter.

If anyone feels like joining MPDN's render script development, feel free to contact either Shiandow or myself.

[-Vit-]

Just updated the first post with v1.1. It's faster now.

I just had to get rid of those badly implemented sorts and medians, replaced them now with more efficient network-sort-style sequences in RemoveGrain4 and FineSharpB. Not theoretically optimal, but close enough - I wanted to avoid conditionals and excessively long shaders.

I tweaked the second sharpening stage (xstr and xrep), which were not operating correctly. You'll probably not notice, that stage has a small impact in this script.

I noticed how well the sharpener works with NNEDI3 luma doubling (the sharpener doesn't affect chroma). Really helps when using pre-resize and post-resize shaders together. Although NNEDI3 does use quite a bit of GPU time.

[madshi]

Thank you for the updated shaders! Yes, NNEDI3 + FineSharp works very well. When playing with doubling the resolution of test images I also like a combination of NNEDI3, FineSharp, LimitedSharpen and aWarpSharp2, though:

nnedi3x(dw=1, nns=2, qual=1)
LimitedSharpenFaster(strength=100)
aWarpSharp2(depth=5)
FineSharp()

At least this looks nice to me with this test image: castle. Probably a touch oversharpened, but at least it's crisp.

[-Vit-]

Originally Posted by madshi

I'd much prefer if you could convert maybe some other algorithm to HLSL without any optimization. With your AviSynth experience you could do that a hundred times faster than I ever could. Optimizing the HLSL shaders is something a lot of people can do (including myself). Understanding and converting AviSynth scripts is much harder - at least for me.

I guess you should give me a list of scripts that need HLSL versions. No promises, but on the rare occasions that I get some time I can have a look.

[kasper93]

@-Vit-: There is typo in FineSharpC.hlsl:L20 missing one ")".

[-Vit-]

Originally Posted by kasper93

@-Vit-: There is typo in FineSharpC.hlsl:L20 missing one ")".

Yes, thanks. Updated the first post, please update everyone - I double checked this time, lol

This happens because the shader editor in MPC-HC isn't connected to the files, so it's easy to get out of sync. It's better in the new versions of MPC-HC, but that's doesn't have shortcut keys for shaders, which makes using them annoying.

[fairchild]

I can't seem to get any sharpening to work/be noticeable to my eyes with latest 1.7.7 MPC-HC using the CTRL+Y toggle which basically just toggles through all the different shader presets. I have the default set to no shaders, then I have one setup/labeled as FineSharp which has the order as detailed in first post for pre-size (figured this one would be the best to use to not over sharpen)

ToYUV
RemoveGrain11
RemoveGrain4
FineSharpA
FineSharpB
FineSharpC
ToRGB

Then I have one which has the above shaders in both pre-resize and post-resize, and when I have a 720p clip on a 1080p screen and I toggle I don't see a difference at all in the image. I must be doing something wrong? (yes I'm using latest MadVR, tried with NNEDI3 luma doubling on and off and notice no difference)

[Neet009]

Mediafire's link still finesharp_shaders_1.1.zip, not finesharp_shaders_1.11.zip.And it seems doesn't work here too.(Both MPC-BE and MPC-HC)

[-Vit-]

Originally Posted by Neet009

Mediafire's link still finesharp_shaders_1.1.zip, not finesharp_shaders_1.11.zip.And it seems doesn't work here too.(Both MPC-BE and MPC-HC)

One of those days... Link in the OP fixed.