Video player Shaders guide (MPC-HC, mpc-BE, mpv)

[butterw2]

Video player Shaders (unofficial guide)

Video Adjustment shaders Pack (dx9, dx11 hlsl and mpv glsl.hook): [A-Pack] v1.50 Released (2023-11) (https://forum.doom9.org/showthread.php?t=184996)
Contrast adaptive Sharpening (AMD CAS) shader (dx9/dx11). (https://forum.doom9.org/showthread.php?p=1993947#post1993947)
Filmgrain Shaders (https://forum.doom9.org/showthread.php?p=1994369#post1994369) grain.hlsl (dx9/dx11), mpv


In addition to video player shaders, this topic covers configuration, user interface and performance of open-source video players available on Windows.
MPC-HC MPC-BE, mpv, mpv.net are feature packed video players that offer great features and performance, together with extensive configuration options.

Post index:
# Comparison of shader features in video players (https://forum.doom9.org/showthread.php?p=1919502#post1919502)
# Shaders Folder, Paths & extensions (https://forum.doom9.org/showthread.php?p=1929231#post1929231)
# New Shader features in Mpc-hc (v1.9.8.65) (https://forum.doom9.org/showthread.php?p=1930240#post1930240)
# Custom hotkeys, Seeking, Chapters, Playlist (https://forum.doom9.org/showthread.php?p=1931840#post1931840)
# Screenshots (https://forum.doom9.org/showthread.php?p=1931990#post1931990)
# Comparison of video player features (Win10) (https://forum.doom9.org/showthread.php?p=1932940#post1932940)
# Mpc Command Line Switches (https://forum.doom9.org/showthread.php?p=1927315#post1927315)
# Video Player installation https://forum.doom9.org/showthread.php?p=1934473#post1934473
# Color mapping with Shaders https://forum.doom9.org/showthread.php?p=1935482#post1935482
# (custom) windowed zoom: 150% (https://forum.doom9.org/showthread.php?p=1987810#post1987810)
# Time display formats (https://forum.doom9.org/showthread.php?p=1990821#post1990821), ex: 01:25 / 53:15

MPC video-renderer, Mpc-vr
# Shader support in mpc video renderer (mpc-vr) (https://forum.doom9.org/showthread.php?p=1927466#post1927466)
# Surface/texture format (https://forum.doom9.org/showthread.php?p=1992643#post1992643)

MPV
# mpv (windows): my tips (https://forum.doom9.org/showthread.php?p=1996300#post1996300) New
# Mpv shader basics (https://forum.doom9.org/showthread.php?p=1926639#post1926639)
# mpv hotkeys (https://forum.doom9.org/showthread.php?p=1929051#post1929051)
# mpv built-in scalers (https://forum.doom9.org/showthread.php?p=1936377#post1936377) (2023-10) new Defaults !
# mpv Auto-Profiles (https://forum.doom9.org/showthread.php?p=1929848#post1929848)
# How to change settings/properties at runtime with mpv (https://forum.doom9.org/showthread.php?p=1936710#post1936710)
# default input.conf. --no-input-default-bindings (https://forum.doom9.org/showthread.php?p=1990542#post1990542)
# custom OSD in mpv (https://forum.doom9.org/showthread.php?p=1990436#post1990436)
# mpv script: avg-bitrate.js (https://forum.doom9.org/showthread.php?p=1990487#post1990487)
# more mpv custom OSD scripts (https://forum.doom9.org/showthread.php?p=1990763#post1990763)
# mpv-android (https://forum.doom9.org/showthread.php?p=1990911#post1990911)
# restart-mpv (https://forum.doom9.org/showthread.php?p=1994854#post1994854)
# custom context menu (https://forum.doom9.org/showthread.php?p=1995031#post1995031)
# custom overlay: modern progress bar (https://forum.doom9.org/showthread.php?p=1995689#post1995689)
# mpv scripts: lua or js ? (https://forum.doom9.org/showthread.php?p=1995935#post1995935)
# mpv script: short time format OSD, btime.js (https://forum.doom9.org/showthread.php?p=1996141#post1996141)

# Cartoon Shaders https://forum.doom9.org/showthread.php?p=1928755#post1928755

Videohelp:
# What shaders should I use in potplayer? (https://forum.videohelp.com/threads/411833-What-shaders-should-I-use-in-potplayer)
# Pixel-shaders for video playback (.hlsl, .glsl) (https://forum.videohelp.com/threads/397797-Pixel-shaders-for-video-playback-%28-hlsl-glsl%29)
---

Fix Mpc-hc issues with evr-cp (dx9) on intel Xe driver (used by 11th gen and later intel igpu, ex: intel uhd730)
Issue: Broken seeking after reaching end of file, repeat doesn't work. Affects all versions of mpc-hc.
Fix: rename mpc-hc64.exe to mpc.exe.

Mpc-hc + Shaders install (with default dx9 EVR-CP renderer) unofficial guide
32 vs 64 bit program version doesn't matter for shaders.

Mpc-hc installs the default shaders in the player directory Shaders subfolder. They are not required to run mpc-hc (they can be modified/deleted as required). It is also possible to download shader files available on the internet or write your own. Files with .hlsl extension in Shaders subfolder are listed in `Option->Shaders` submenu.

Install directory does matter.
To be able to modify shader parameters you need to have write access to the directory where the shader.hlsl file is located (by default the Shaders subfolder).
Also to have the option to use mpc-hc.ini, mpc-hc must have write access to the Install directory (if not the option is greyed out). This will not be the case for a standard user on Win10 by default with a `c:\Program Files\MPC-HC` install. To get around this, you need to either re-install to a user directory or change folder permissions. For testing, the simplest solution is to copy a portable version + mpc.ini in its own directory (no-install required).


How to get corresponding Shaders Menu when you install Shaders files ?
To be able to apply Effects quickly, they need to be saved as Shader Presets. You can then change Effects through the Shaders right-click Menu
Shader presets can be created/saved by using the `Select Shaders...` menu option, but by editing mpc-hc.ini we can get a ready to use Shaders Menu directly.

https://raw.githubusercontent.com/butterw/bShaders/master/MPC-HC_Shaders-Menu.jpg

Mpc-hc Shader Menu mod guide:
Mpc-hc settings are stored either in the registry or in an .ini file, based on setting `Mpc-hc->Options->Player->Store settings in .ini file.`. This option will be greyed out, if mpc-hc doesn't have write access to the install folder.

Editing mpc-hc.ini (Close Mpc-hc before you start. It is advisable to make a backup copy before making any changes)

With a text editor, search for the Shaders\Presets section. Here is an example of what you might have:
[Shaders\Presets]
0= Disable
1=Sepia
2=Luma+Grain
3=Pixelate
PostResize0=
PostResize1=
PostResize2=.\LumaSharpen.hlsl;.\PS_Noise.hlsl
PostResize3=
PreResize0=
PreResize1=.\Sepia.hlsl
PreResize2=
PreResize3=.\bPixelate.hlsl

You can directly edit preset names, Pre and PostResize shader chain filepaths. Be careful that all numbers must remain available: deleting 1=Sepia would cause issue here !
Shaders Menu display order: Preset names are sorted alphabetically. To control the display order, preset names could be numbered (00, 01, etc.).
For more advanced users, Shader Presets can be used to setup Multi-Pass shaders.


Modifying shader files
.hlsl files contain the Shader function and parameter values. In most cases, the effect has parameter values that can be modified. This can be done with any text editor (I use Notepad++, a multitabbed editor which offers c-syntaxic coloring). After saving the changes, they are applied instantly to the video playing.
In the case of LumaSharpen.hlsl which was developed for SweetFX game shaders pack, the default parameter values are not those recommended in this forum for video content playback.
If you want to switch between different parameter values for the same shader without editing, you can create a new shader with the desired parameter values, ex: LumaSharpen_3-1.20-0.015.hlsl. With this approach the Shaders subfolder (and the shader list) can quickly become overcrowded, so it is important to know which shaders you want to use, together with the associated parameter values.

[v0lt]

I think there isn’t enough shader compilation checking.

For DirectX 9
fxc.exe /T ps_3_0 /Fc "Shader9.txt" "Shader9.hlsl"
For DirectX 11
fxc.exe /T ps_4_0 /Fc "Shader11.txt" "Shader11.hlsl"

[butterw2]

I haven't tried the external dx11 renderer so far, but I understand it requires dx11 shaders, which have a slightly different syntax than dx9 shaders.

Doesn't the compiler have a backwards compatibility mode (fxc /Gec ?) to enable use of dx9 shader files in dx11 ?

All the currently available third party .hlsl shaders available online were written for dx9.

[butterw2]

Optimized 2-pass Gaussian Blur (9-tap) shader:
bShaders (https://github.com/butterw/bShaders)\blurGauss.hlsl >> blurGauss_Y.hlsl
the kernel is hardcoded so you can't adjust parameters. Run it multiple times to achieve a stronger effect (ex: 5x).

below are Blur Gaussian and Blur Gaussian (5x) shader presets for mpc-hc.

mpc-hc.ini:
[Shaders\Presets]
0= Disable
1=Blur Gaussian
2=Blur Gaussian (5x)
PostResize0=
PostResize1=
PostResize2=
PreResize0=
PreResize1=.\blurGauss.hlsl;.\blurGauss_Y.hlsl
PreResize2=.\blurGauss.hlsl;.\blurGauss_Y.hlsl;.\blurGauss.hlsl;.\blurGauss_Y.hlsl;.\blurGauss.hlsl;.\blurGauss_Y.hlsl;.\blurGauss.hlsl;.\blurGauss_Y.hlsl;.\blurGauss.hlsl;.\blurGauss_Y.hlsl

For a near gaussian blur with an adjustable parameter, use boxBlur (3x) based on 2pass blurMean.
A multipass Kawase blur, also can approximate a gaussian depending on parameters.
For best performance with Heavy bluring, Dual-Kawase downsizes the video, blurs, and resizes-up.

[butterw2]

Mpc-hc and mpc-be have a "View>Pan&Scan" zoom-crop built-in feature. This transformation happens before user shaders.

Editing PnS Presets: specify center (x,y) and zoom (x, y).
- center (0.5, 0.5): middle of the screen
- zoomX and zoomY should be equal to keep the source Aspect Ratio. 1: no magnification.

The most useful preset is probably "Zoom to widescreen" which will crop horizontal and vertical borders on the corresponding 4/3 source material when displayed on a 16/9 screen.

- `Numpad 5` hotkey: resets PnS zoom (also displays source AR). PnS zoom is also reset when you close the player.
- You can manually adjust zoom using Numpad 9(+) and 1(-) hotkeys.


Observations:
- Center of Screen PnS fixed zoom works well, whatever the source Aspect Ratio.
- I initially tried to display only part of the source image with Pan&Scan (ex: fill the screen with first quarter of frame), but found pre-resize shaders more capable for this use case.
The issue with PnS (beside the confusing x, y values) is black bars on resize, when the source doesn't have screen 16/9 AR (ex: 21/9 film content, 9/16 vertical video).

Quarter zoom PnS presets: x, y values (zoom: 2, 2), works well for 16/9 input.
- top left Quarter: 1, 1, - top right: 0, 1.
- bottom left Quarter: 1, 0 - bottom right: 0, 0.

Fs2xRightSide PnS preset could be used for right side offset 2x zoom for non 16/9 landscape.

in mpc-hc.ini: [Settings\PnSPresets]
Preset0=Widescreen zoom,0.500,0.500,1.333,1.333
Preset1=Quarter Top-L,1,1,2,2
Preset2=Fs2xRightSide-L,0,0.5,2,2

You can generally combine user pixel shaders with any mpc-hc zoom feature (including Right-Click: Video Frame > xxx and Zoom > xxx).

"Video Frame > Touch window from outside" is a centered fullscreen crop mode based on the input video Aspect Ratio. Fills the window/screen by zooming and cropping the video.

[butterw2]

Linear sampling is a performance optimization technique for gpu pixel shader. The idea is to reduce the number of pixel (texture) fetches required by an algorithm by sampling between pixels and exploiting hardware linear interpolation.

a 2-pass pre-resize shader test was performed:
https://github.com/butterw/bShaders/tree/master/test_linearSampling
- pass-1 generate a black and white stripe test motif,
- pass-2 use linear sampling

Expected: interpolation, colored Red
Result: no interpolation (Nearest Neighbor). If the offset is set to 0.5 pixel: produces some rounding artefacts !

https://raw.githubusercontent.com/butterw/bShaders/master/test_linearSampling/Offset%200.5%20Sampling%20artefact_zoomed.png

Software and hardware setup
- mpc-hc v1.9.6 + EVR-CP. Also tested in mpc-be
- old intel igpu (hd4400), not the latest Win10 + drivers


## Linear Sampling (in hlsl): tex2D(s0, tex + k*p1) where k is non integer
the effect of Nearest Neighbor sampling is to round k.x, k.y to the nearest integer. If k.x or k.y is a half, you then get some rounding artefacts.
While it certainly is possible to get rid of the rounding artefacts, they are a symptom of the larger problem: Linear Sampling not being supported, which has major implications for the math analysis.

Linear sampling is exploited by perf optimized convolution kernel shaders (such as: Luma Sharpen, gaussian blur, etc.). If it isn't supported, the shaders still produce an output, but a suboptimal one. The provided test allows for the user to test whether linear sampling is actually supported.

Edit (linear sampling works in mpv): the same test using rgb32 png input was performed on the same configuration with a glsl shader (mpv video player, vo=gpu), and I'm getting the expected result there !
https://github.com/butterw/bShaders/tree/master/test_linearSampling

Edit2 (hw linear sampling isn't supported in mpc-hc/be): hw linear sampling also not working as expected in mpc-hc/be with mpc-vr as video renderer in dx9 and dx11 modes.
https://github.com/clsid2/mpc-hc/issues/748
This impacts shaders such as LumaSharpen.hlsl

[VictorLS]

butterw2
Can you help us (SAT amateurs) in writing HLGtoSDR(BT.709) hlsl shader https://forum.doom9.org/showthread.php?t=176909 i.e. adopt libplacebo (https://code.videolan.org/videolan/libplacebo/-/blob/master/src/shaders/colorspace.c#L819) to MPC-HC's shader we can use in SmartDVB - quality of present shaders is bad - but VLC and MPV players give good enough quality while playing HLG files on SDR monitor?

[butterw2]

If glsl fragment shader code exists for the desired color transformation that would mostly autotranslate to hlsl usable in mpc-hc/be.

[VictorLS]

I know but result of that translation is bad enough or some 'know how' must be known to get it working well.
Besides - I don't really sure libplacebo (only?) do desirable color transformation (HLGtoSDR) or VLC uses something else - I'm not programmer at all and moreover I'm not shader programmer (
Here's somehow working HLGtoSDR shaders https://transfiles.ru/svler - Convert HDR to SDR.hlsl in HDR folder work well enough but for ordinary HDR files like 23_Samsung_HDR_Wonderland.mkv (97 MB) https://yadi.sk/i/ZgkOWjcEwWQNkQ - not for HLG files like 20190831-160457_RTL UHD.mkv (135 MB) https://yadi.sk/i/SRloXWrOP-L-xA
I even don't know that (not in HDR folder) shaders was taken from libplacebo or somewhere else ;)
PS. Here's long time (more than 4 minutes) test files https://forum.doom9.org/showthread.php?p=1919134#post1919134

[butterw2]

I'm afraid, my experience on the subject of HDR to SDR tonemapping is rather limited.
mpc-be does some tonemapping by default (using hlsl shaders I believe). If I understand you correctly, you claim that the method used by vlc and mpv gives more accurate results ?

If there was a mpv glsl .hook file, that might be a better starting point for a port than libplacebo (c file with inlined GLSL).
I've tried the provided "Convert HDR to SDR.hlsl" shader in mpc-hc. The colors seem oversaturated for my taste (while the colors are washed out without it). The processing is not that complex (~100 lines of actual code), so it should be achievable to get the shader part working if a reference glsl implementation exists.

Regarding the provided sample files:
- 20190831-160457_RTL UHD.mkv is UHD-50 hevc 10bit HDR 41Mbps
- 23_Samsung_HDR_Wonderland.mkv UHD hevc 10bit HDR 22.5Mbps
Without considering the issue of tonemapping, these files do not play smoothly on an old pc/integrated graphics + 1080p SDR screen like mine.

For testing purposes, short samples are probably fine, I would suggest downconverting the most relevant ones to 1080p, reducing the bitrate 10/20x, maybe remove the audio, provide screenshots (with frame timestamp/frame number) of what you regard as desirable (from mpv, avisynth) vs what was achieved and host this with the code on a server that will not delete your files.

[VictorLS]

Thanks for reply.
I'm afraid, my experience on the subject of HDR to SDR tonemapping is rather limited
It's a pity but I believe in you ;)
Btw I have SDR (BT.709) monitor too - that's why I want to find good HLGtoSDR shader but couldn't yet (
20190831-160457_RTL UHD.mkv is UHD-50 hevc 10bit HDR 41Mbps
Not only ordinary HDR, but HLG same time - that's why working with ordinary HDR video like
23_Samsung_HDR_Wonderland.mkv UHD hevc 10bit HDR 22.5Mbps
Convert HDR to SDR.hlsl shader gives good result but with HLG 20190831-160457_RTL UHD.mkv oversaturated.
mpc-be does some tonemapping by default (using hlsl shaders I believe)
Yes, you're quiet right (btw MPC-HC must output to madVR or use Convert HDR to SDR.hlsl shader with any other renderer to achieve such results - am I right?)
If there was a mpv glsl .hook file, that might be a better starting point for a port than libplacebo (c file with inlined GLSL)
From MPC-BE developers I've heard mpv does it on a fly so they are couldn't extract using by mpv shader while playing HLG file (btw they are looking at my researching for HLGtoSDR shader and ready to implement it in MPC-BE like now HDR if result will be positive). Besides quality of tonemapping depends of version of mpv - older than August 2019 are better (newer gives some rose instead of red) on my own so I'd prefer shader extracted from VLC ;)
For testing purposes, short samples are probably fine
I agree in your case - you can find them in the bottom of first post https://forum.doom9.org/showthread.php?t=176909
I would suggest downconverting the most relevant ones to 1080p
I guess it's wrong way because HLG metadata can be lost during any conversion so see first post (https://forum.doom9.org/showthread.php?t=176909) for your other wishes ;)

[butterw2]

Tone mapping is required to correctly display any HDR encoding (typ 1080p Hevc 10bit HDR encode from UHD bluray) displayed on a Standard Dynamic Range computer screen (almost all current monitors are non-HDR).

mpc-hc doesn't do any tone mapping by default: so colors look washed out. If you enable the provided Convert HDR to SDR.hlsl shader, the results seem quite watchable for HDR content. It's also a reasonably lightweight shader (1 texture, 44 arithmetic).
Edit: the code likely comes from mpc-be and has 2 tuning parameters (LuminancePeak_Gain, Gamma).
Such a shader can improve the viewing of a washed-out color video (ex: encode of an HDR source to SDR with no tonemapping applied, or undersatured SDR video). It shouldn't be used otherwise.

mpc-be and vlc come with built-in SDR Tone mapping, which is applied automatically (there doesn't seem to be a setting to deactivate it !).

The samples (RTL title screen) in your first post here https://forum.doom9.org/showthread.php?t=176909
seem like a good starting point regarding the specific case of Hybrid Log-Gamma (HLG) HDR used in UHD satellite broadcasts .

- 23_Samsung_HDR_Wonderland.mkv (UHD hevc 10bit HDR 22.5Mbps)
-20190831-160457_RTL UHD.mkv (UHD-50 hevc 10bit HLG HDR 41Mbps): quite specific sample because it contains a lot of white smoke. I actually prefer no tone mapping on this sample vs oversaturated result.

Also, on my PC performance is awful with VLC with these 2 samples.

[VictorLS]

Tone mapping is required to correctly display any HDR encoding (typ 1080p Hevc 10bit HDR encode from UHD bluray) displayed on a Standard Dynamic Range computer screen (almost all current monitors are non-HDR)
mpc-hc doesn't do any tone mapping by default: so colors look washed out. If you enable the provided Convert HDR to SDR.hlsl shader, the results seem quite watchable for HDR content. It's also a reasonably lightweight shader (1 texture, 44 arithmetic).
So, I was right when wrote "MPC-HC must output to madVR or use Convert HDR to SDR.hlsl shader with any other renderer to achieve such results"
mpc-be and vlc come with built-in SDR Tone mapping, which is applied automatically (there doesn't seem to be a setting to deactivate it).
You're right.
The samples in your first post here https://forum.doom9.org/showthread.php?t=176909
seem like a good starting point regarding the specific case of Hybrid Log-Gamma (HLG) HDR used in some satellite broadcasts(RTL title screen).
Some people said this is bad point so I hard work to write red-flagged F1 to someone do right shader with colorimeter ;)
SDR FullHD H.264 20190511-130113_ARENA HD_flag.ts (38 MB) https://yadi.sk/i/HiuZ5Rb2RKkc0g
HLG 4K H.265 20190511-152944_RTL UHD_flag.ts (67 MB) https://yadi.sk/i/985DVw34OHQrrA
-20190831-160457_RTL UHD.mkv (UHD-50 hevc 10bit HLG HDR 41Mbps): quite specific sample because it contains a lot of white smoke.
I've uploaded that to see how new HLGtoSDR shader shows grass - it's looking ugly now with or without tonemapping in ordinary DirectShow players.
I actually prefer no tone mapping on this sample vs oversaturated result.
You're right but I'd want get VLC (OpengGL output in Win7 or D3D11 output in Win8-10) colors with HLGtoSDR shader
Also, on my PC performance is awful with VLC with these 2 samples.
and weak PC can't stop experiments with colors because of watching mostly in pause mode ;)

[butterw2]

# Comparison of user shader features in the main video players for Windows

To Everyone: Please point out any mistakes/changes, so I can update this guide as needed. The reason for this thread is that the available documentation is sparse and mostly buried in very old/long threads.
You don't need to be a C++ dev to customize or write pixel shaders.

## directX hlsl shaders:
VLC v3.10 in dx9 video output mode
- only one user shader (post resize) !
built-in effects/filters only work in dx11 mode !
shader is set in config options > advanced, but can also be set via command-line
! code: the output alpha channel must be != zero, otherwise black screen output.
shader modification: stop/start video.

MPC-HC v1.9.6 (EVR-CP) -- Recommended for shader dev, New features added in dev build --
- Configurable Shader Presets: a named/saved user defined shader chain.
- in menu Select shaders (uses Options shortcut "O")
- switch shader presets via contextual Menu in fullscreen mode
- pre and post resize pixel shader chains
- multipass shaders possible
- Shader modification: new parameters are applied immediatedly (! or next frame in the case where you empty the shader chain).
- Compiler output with Debug Shaders (+ shortcut)
- time in seconds since application start available and updated every frame.

- KMPlayer 4.0.7.1 can fully operate with shaders when paused


MPC-BE v1.5.4 (EVR-CP)
- ! No Shader Presets: single pre and post resize shader chains
- Shortcuts to enable/disable pre and post resize shader chains (CTRL-P, CTRL-ALT-P)
- "Save output image" Screenshot includes shader output (Shift+F5)
- ! Shader modification: close/restart player.
- ! unlike mpc-hc, Alpha Channel is initially set to 1 (because of this, some shaders that assume initial alpha is 0 will not work properly without correction).


---
## glsl fragment shaders intro here (https://forum.videohelp.com/threads/397797-Pixel-shaders-for-video-playback-(-hlsl-glsl)#post2597322)

Mpv .hook (vo=gpu, gpu-api=d3d11)
set via glsl-shader in config file mpv.conf.
(Shift-I, Shift-2) to display the processing chain. Hotkeys can be added in input.conf (https://forum.doom9.org/showthread.php?p=1929051#post1929051) to switch individual shaders, disable/enable shaders or list active shaders.
- Output is a blue-screen if a shader has a syntax error. You need to restart player to modify shaders.
- user shaders with custom hook points: source shaders (Luma, Chroma), chroma upsampling, custom resizers, pre and post resize RGB.
- multipass shaders in a single file, can downsample textures, can re-use the output of previous passes.
- complex shader chain possible by saving textures.
- uniform variable "random" available (PRNG) and updated each frame
- compute shaders
! few .hook effect shaders currently available (vs hlsl, glsl), but easy enough to adapt .frag glsl to .hook

[butterw2]

# Video playback pipeline on Windows in Standard Dynamic Range (SDR, Video+Display)

Mpc-hc works in 8-bit by default.

0) Graphics Driver Settings (mostly disabled >> set to "Application Settings")
- features supported by gpu hardware.
- Output should be set to the same as your display (typ for PC: RGB full range)

1) Video File (container: .mp4, .mkv)
- Video (Codec: x264, x265)
- typ. YUV420 (chroma channels are subsampled), Gamma-Encoded 8/10bits,
- Resolution+Progressive+Framerate, ex: 720p30,
- color primaries and matrix: bt709 (HDTV)
- color range: Limited (Low: 16-235), (vs Full/PC: 0-255)
...streaming/bluray movies and TV shows are mastered in Limited range.
- Audio channels
- Subtitle channels
Mpc-hc/be Menu: File > Properties: MediaInfo.dll file information.

2) DirectShow player x86 or x64 (ex: mpc-hc, etc.)
- Features: window, play, seeking, configuration/Menu/Hotkeys, shaders, screenshots

Modern video players come with the decoders required to play any standard video without the need to install codecs/codec packs.
- DShow graph: Splitter > Decoder* > Renderer*
2a) LAV Filters - ffmpeg based DirectShow Splitter and Decoders (LAV Splitter >> LAV Video + LAV Audio)
MPC-HC/BE Menu: Play > Filters > Video Decoder
The decoder defines the color space at the renderer input (typ. same as source: YUV or RGB, 8, 10, 16bits)

2b) Renderer: DirectX 9 EVR-CP (Enhanced Video Renderer-Custom Presenter >Win8) + DXVA2 gpu hw Acceleration*
- default in MPC-HC

MPC-HC/BE Menu: View > Renderer Settings > Display Statistics (Ctrl+J) overlay
Input, ex: NV12, YV12
- Save Image... (Alt+I) input Screenshot
- YUV >> RGB Conversion
2c) RGB Processing Surface: int8, int10, float16
- Color Controls (Brightness, Contrast, Hue, Saturation)
- Zoom, Pan&Scan
- Rotation
- Pixel Shaders (gpu)
- RGBA float4(32, 32, 32, 32)
- Pre-resize user Pixel Shader chain
- Resizer (ex: bicubic A=-0.75, to 1080p)
- Post-resize user Pixel Shader chain (in Screen Space)
BackBuffer/Display ?

2d) Overlay (Alpha blended)
- Subtitles
- OSD, On Screen Display

MPC-BE (Shift+F5): Save Displayed Image (auto) - output Screenshot
Mixer Output ?

3) Output interface/cable: typ. hdmi
> RGB Monitor/HDTV (Resolution/Framerate, reported bit depth: 8 or 10bit), ex: 1920x1080, 60Hz, 8bit
- Settings on Monitor. (Brightness/Contrast/Gamma, RGB Range, etc.)
- SDR (sRGB), color: bt709.
- Gamma-decode

[v0lt]

Mixer Output ?
EVR Mixer usually requires a specific output format. See "DXVA Checker -> Processor device -> DXVA1/2 -> OutputFormat" for details. This is essential for AMD and Intel graphics cards to function properly. For Nvidia video cards, we may use some "unsupported" formats as output.

[butterw2]

# Playing a video file from .avs (Avisynth)
- Video players such as mpc-hc/be can play .avs files.
- Frameservers such as Avisynth can be used to process video (they are widely used pre-encoding).
So, can you process video files for realtime display using .avs and is there any benefit to do so ?

Avisynth processing is done primarily on the cpu vs pixel shaders, which use gpu processing.
- For many basic video related operations this is less efficient, even with multi-threading, but it also has fewer limitations.
- Avisynth scripts rely mainly on high quality existing internal/external functions/libraries (compiled .dll C++ plugins).


.avs Display Pipeline:
Video file (.mp4, ex: x264/x265 1080p, YUV420) >> .avs: source Filter(video_filename): Avisynth cpu processing: Output (ex: YV12 or RGB32)
>> Input of Video Player: Video Player processing (RGB), ex: gpu pixel shaders >> Monitor (RGB)

Performance:
- as expected, significant cpu overhead with an .avs script vs normal playing. Depending on the Avisynth filters used, realtime playback may however be achieveable.
- Avisynth filters can be used in complement to gpu pixel shaders.

Software configuration: Win10 x64
- DirectShow Video player: Mpc-be x64, EVR-CP renderer
- Avisynth+ v3.6.1 x64
- external Source plugin: LSMASHSource.dll x64, based on libavcodec, with gpu hw decoding support.
My tests were performed with Avisynth, but the more recent python script alternative Vapoursynth looks promising. It offers direct access to the python software stack (ex: numpy vector math, matplotlib plots, etc.).

## Limitations:
- LSMASHVideoSource / LSMASHAudioSource only support mp4/mov input ! Wider input file format support is available (ex: mkv) with LWLibavVideoSource, but input file must initially be indexed (ex: 10s) !
- video_filename is typically changed by editing the .avs script ! To avoid this, video_filename.avs can be coded so as to load video_filename.mp4 by default.
- On script syntax error, you only get "Cannot render the File" message in video player ! To get full error msg you need to use either AvsPmod, Virtualdub2 or AvsMeter.

Conclusion:
It is currently possible, but not very practical to play an avs script in a video player. This would probably only make sense if the same effect was not possible as a pixel shader and an equivalent player feature was not available.
Using a per-file script does mean file-specific processing settings are saved. Undesirable lossy re-encoding of a compressed source may be avoided.

Avisynth input can however be useful as a test tool for video (https://github.com/butterw/bShaders/tree/master/test_LimitedRange):
- YUV-to-RGB colorspace conversion can be performed in Avisynth (with rec709 range expansion by default). In this case the YUV-to-RGB conversion at the video player input is bypassed.
- Resize frame
- Generate test-patterns (ex: ColorBarsHD).
- Go to a specific frame (vs Directshow video player: seeking is not guaranteed to be frame accurate !).
- Supports multiple video inputs. You could also use multiple clips from the same source (ex: could be used for a split-screen effect or to highlight differences)
- Load/process/save images (ex: lossless .png screenshots) with the same code.
- Output data files (ex: .csv)

[butterw2]

# Typical PC Display (Color Range) Pipeline :
Compressed video file (yuv420p8, Limited Range, Color range: Limited or untagged) >> mpc-hc Video player (Range expanded to Full Range at yuv-rgb32 conversion at renderer input)
>> graphics driver (rgb32 Output, Full Range) >> Hdmi >> Monitor (Full Range)

- Historically TV/HDTV have used limited range input vs PC monitor full RGB range.
- Most current displays should allow you to set the desired input range (if you change this value, you probably also need to adjust brightness, contrast, gamma for good results).
For accurate representation of grey levels and colors your monitor/HDTV and graphic card settings should match: Full-PC or Limited-Low.
Display range can be checked on screen with a pixel picker and a generated SMPTE ColorBarsHD test pattern input. (https://github.com/butterw/bShaders/blob/master/test_LimitedRange/Readme.md)

If the input video file is Full Range but lacks the correct video stream tag (Color range: Full) in the Mediainfo properties, mpc-hc/be will treat it as Limited range !
To avoid a clipped color levels output, you should correct the tagging, which can be done losslessly for x264/x265 using ffmpeg:
ffmpeg -i "input.mp4" -c:v copy -bsf:v h264_metadata=video_full_range_flag=1 "output_full.mp4"
(for x265): -bsf:v hevc_metadata=video_full_range_flag=1

More info about ffmpeg video bitstream filters here: https://ffmpeg.org/ffmpeg-bitstream-filters.html#h264_005fmetadata
This can be used to crop a displayed video file without having to re-encode it for instance.

An out-of-range highlighting pixel shader such as https://github.com/butterw/bShaders/blob/master/test_LimitedRange/bHighL.hlsl can be used to highlight pixels that are out of the defined range.
Pixel shaders can of course be used to alter levels at playback, but this is not typically necessary.

[chros]

If the input video file is Full Range but lacks the correct video stream tag (Color range: Full) in the Mediainfo properties, mpc-hc/be will treat it as Limited range !
To avoid a clipped color levels output, you should correct the tagging, which can be done losslessly for x264/x265 using ffmpeg:
ffmpeg -i "input.mp4" -c:v copy -bsf:v h264_metadata=video_full_range_flag=1 "output_full.mp4"
(for x265): -bsf:v hevc_metadata=video_full_range_flag=1
I think this is excatly what I need for my "issue" (https://forum.doom9.org/showthread.php?t=181702).
Do you know how to list all the video properties of a file with ffmpeg?
You can reply there as well, thanks!

[SirMaster]

Hi @butterw2

I see you have some experience in writing HLSL.

I am interested in writing some of my own shaders but I find myself pretty lost with the syntax.

I am a developer, so I have a lot of programming experience, but maybe you help me get off to a start.

I found this shader which works, but I would like to start by simplifying it.

https://pastebin.com/VMfh8tNL

This shader allows performing RGB color convergence adjustment (useful for 3 chip projectors).

However I notice that when I enable it, it seems to always be blending pixels which screws up 1:1 pixel mapping.

I think this is due to the fact that this shader allows for sorts of scaling and geometric adjustments of the color which I don't need at all.

I would like to modify the shader to only do simple whole pixel adjustments of the red and blue colors.

If you have time, do you think you can help me get started?

[butterw2]

hlsl pixel shader Intro (dx9)

If you have time, do you think you can help me get started?
It's actually incredibly simple if you are familiar with c syntax, provided you understand what you are trying to do. Some knowledge of vector math will be useful for more advanced image processing.

I wrote a basic intro here (also details limitations):
https://forum.videohelp.com/threads/397797-Pixel-shaders-for-video-playback-(-hlsl)#post2587323

Checking out/modifying existing shaders is a good way to get started.
You can have multiple return statements: use the screen to display your intermediate results and comment them out as required.
define your user parameters with #define

For dev, I would recommend mpc-hc/ notepad++ with an extended c syntaxic coloring for .hlsl.
Notepad++ > Settings > Style Configurator...
C .hlsl user-defined
- instructions: main tex2D saturate dot pow max min lerp mul frac sign step sqrt clamp
- types: sampler float4 float3 float2 float3x3

User Defined Language hlsl (https://raw.githubusercontent.com/butterw/bShaders/master/img/scr_syntaxic_coloring_npp.png).xml is also possible (but it is a bit flaky)

mpc-hc allows you to display the debug output / perf of your code.

----
// This shader can perform software alignment by Catmull-Rom spline6 interpolation for a 3LCD projector's red and blue panels
// This file is part of Video pixel shader pack. // (C) 2011 Jan-Willem Krans (janwillem32 <at> hotmail.com)
I must say I have no idea what this pixel shader is used for (some type of projector calibration ?).

// This shader should be run as a screen space pixel shader.
- screen space pixel shader: use as post-resize shader.

// This shader is meant to work with linear RGB input and output. Regular R'G'B' with a video gamma encoding will have to be converted with the linear gamma shaders to work properly.
see https://forum.doom9.org/showthread.php?t=181651
There is an overhead to converting to linear gamma, so if the effect is negligible in practice you might want to just ignore it, but it could affect the accuracy of the result.
If this is the only shader you intend to use, you could integrate the linear gamma conversion into the shader. Otherwise, I typically use a 10bit integer surface for processing in mpc-hc to avoid too much precision loss with the shader chain.

[SirMaster]

Thanks, I actually just e-mailed Jan-Willem Krans this afternoon asking about it and he whipped up a simple shader to do what I wanted in this case.

#define RedControls 0
#define BlueControls 0
// RedShiftLeftToRight and BlueShiftLeftToRight, a value of 3. will shift three pixels to the right, 0 is disabled
#define RedShiftLeftToRight 0.
#define BlueShiftLeftToRight 0.
// RedShiftTopToBottom and BlueShiftTopToBottom, a value of 3. will shift three pixels to the bottom, 0 is disabled
#define RedShiftTopToBottom 0.
#define BlueShiftTopToBottom 0.

sampler s0;
float2 c1 : register(c1);

float4 main(float2 tex : TEXCOORD0) : COLOR
{
float4 s1 = tex2D(s0, tex);// base pixel
#if RedControls == 1
s1.r = tex2D(s0, tex+c1*float2(RedShiftLeftToRight, RedShiftTopToBottom)).r;// base red pixel
#endif
#if BlueControls == 1
s1.b = tex2D(s0, tex+c1*float2(BlueShiftLeftToRight, BlueShiftTopToBottom)).b;// base blue pixel
#endif
return s1;
}

And yes, this sort of shader is useful for fixing the RGB panel convergence in 3-chip projectors where the 3 color producing panels are not in perfect alignment.

Basically a projector can look like this:

https://lowtek.ca/roo/wp-content/uploads/2008/03/avsbad3.jpg

Some higher end projectors have internal software corrections for this sort of issue that you can adjust, but others do not, and that's where this shader comes in.

I used his shader that you posted, but that one has complex pixel interpolation and breaks the perfect 1:1 pixel mapping that I wanted. this new simple shader he wrote or me does exactly what I wanted now.


I do have some other ideas for some simple shaders that I want to try making and I may post here if I have questions about how to do something as I start tinkering around with them.

[butterw2]

You just wanted a configurable (x, y) pixel offset on red and blue channels. Very simple indeed using a pixel shader. This could also be used as an effect of sorts, I suppose.

if I can add anything at all, p1 would be a more common name for the pixel offset:
float2 p1: register(c1);

[SirMaster]

Here is a shader I made for simple 4 way masking.

#define width 1920.
#define height 1080.

#define left 0
#define right 0
#define top 0
#define bottom 0

sampler s0;

float4 main(float2 tex : TEXCOORD0) : COLOR
{
if(tex.x >= (left/width) && tex.x <= 1-(right/width) && tex.y >= (top/height) && tex.y <= 1-(bottom/height)) return tex2D(s0, tex);
return float4(0, 0, 0, 1);
}

This sort of video masking is a feature found in most higher-end projectors as there can be reasons why you may want to mask parts of the image for certain projection screen sizes and shapes and aspect ratios and such.

This is useful for projectors that don't have a built in 4 way masking control.

[butterw2]

Here's my own optimized implementation from barMask.hlsl (Mode==112)
"if" conditionals are quite inefficient with pixel shaders, so the code uses a boolean insideBox function instead.
All functions in HLSL are inline. An inline function generates a copy of the function body (when compiling) for each function call. #define macro functions are also commonly used.

/* Custom Border Mask defined in pixels
(1 texture, 10 arithmetic)
barMask.hlsl (Mode==112)
*/


#define BorderColor 0
#define Top 0
#define Left 0
#define Bottom 0
#define Right 0

sampler s0: register(s0);
float2 p1: register(c1);

bool insideBox(float2 tex, float2 topLeft, float2 bottomRight){
/* returns true if tex coordinates inside the box, returns false otherwise. !insideBox: outsideBox */
float2 s = step(topLeft, tex) - step(bottomRight, tex);
return s.x * s.y;
}

/* --- Main --- */
float4 main(float2 tex: TEXCOORD0): COLOR {
if (insideBox(tex, p1*float2(Left, Top), 1-p1*float2(Right, Bottom))) return tex2D(s0, tex);
return BorderColor;
}

[SirMaster]

Here's my own optimized implementation from barMask.hlsl (Mode==112)
if conditionals are quite inefficient with pixel shaders.

Oh, awesome!

[SirMaster]

hlsl pixel shader Intro (dx9)


// This shader is meant to work with linear RGB input and output. Regular R'G'B' with a video gamma encoding will have to be converted with the linear gamma shaders to work properly.
see https://forum.doom9.org/showthread.php?t=181651
There is an overhead to converting to linear gamma, so if the effect is negligible in practice you might want to just ignore it, but it could affect the accuracy of the result.
If this is the only shader you intend to use, you could integrate the linear gamma conversion into the shader. Otherwise, I typically use a 10bit integer surface for processing in mpc-hc to avoid too much precision loss with the shader chain.

I am confused by this part.

I normally have the whole chain set to RGB full range (lav filters / madVR, GPU output, and display).

And everything is set to and calibrated to 2.2 power law gamma.

So do I need to be concerned about doing some sort of linear gamma conversion?

I am not sure why simply shifting the colored pixel channels around and adding black border masking would need to worry about anything like this.

[butterw2]

# Linear Gamma Conversion https://forum.doom9.org/showthread.php?t=181651 was mentioned in the original LCD calibration source code you linked and yes it can be confusing.

If you are just moving pixels around or masking the frame: it is not necessary.

Then there are many cases where it might be theoretically necessary (any processing of pixel values), but it makes little or no difference to the end result: just ignore.

In a few cases however it is necessary to convert to linear gamma before processing and then gamma encode when you are done.

Incorrect processing is so common however, that the result will sometimes look "strange/wrong" when you go through the trouble/overhead of doing it right.

[Alexkral]

Hi @butterw2

Maybe you can help me with this.

I'm using a Lanczos 3 shader that I found on the internet to do the VDSR scaling in my app. The results are very good, quite similar to AviSynth and Matlab, and using Lanczos instead of Bicubic as is done on the VDSR paper, allows the final result to be much sharper. This is the code:

/*
Copyright (C) 2010 Team XBMC
http://www.xbmc.org
Copyright (C) 2011 Stefanos A.
http://www.opentk.com

This Program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation; either version 2, or (at your option)
any later version.

This Program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with this library.

*/

sampler s0 : register(s0);
float4 p2 : register(c2);

#define get(x, y) tex2D(s0, float2(x, y)).rgb
#define FIX(c) max(abs(c), 1e-5);
static const float PI = 3.141592653;

float3 weight3(float x)
{
const float radius = 3.0;
float s1 = FIX(2.0 * PI * (x - 1.5));
float s2 = FIX(2.0 * PI * (x - 0.5));
float s3 = FIX(2.0 * PI * (x + 0.5));
float3 sample = float3(s1, s2, s3);
return sin(sample) * sin(sample / radius) / (sample * sample);
}

float3 line_run(float ypos, float3 xpos1, float3 xpos2, float3 linetaps1, float3 linetaps2)
{
return
get(xpos1.r, ypos) * linetaps1.r +
get(xpos1.g, ypos) * linetaps2.r +
get(xpos1.b, ypos) * linetaps1.g +
get(xpos2.r, ypos) * linetaps2.g +
get(xpos2.g, ypos) * linetaps1.b +
get(xpos2.b, ypos) * linetaps2.b;
}

float4 main(float2 tex : TEXCOORD0) : COLOR
{
float2 pos = tex + p2.zw * 0.5;
float2 f = frac(pos / p2.zw);

float3 linetaps1 = weight3(0.5 - f.x * 0.5);
float3 linetaps2 = weight3(1.0 - f.x * 0.5);
float3 columntaps1 = weight3(0.5 - f.y * 0.5);
float3 columntaps2 = weight3(1.0 - f.y * 0.5);

float suml = dot(linetaps1, float3(1, 1, 1)) + dot(linetaps2, float3(1, 1, 1));
float sumc = dot(columntaps1, float3(1, 1, 1)) + dot(columntaps2, float3(1, 1, 1));
linetaps1 /= suml;
linetaps2 /= suml;
columntaps1 /= sumc;
columntaps2 /= sumc;

float2 xystart = (-2.5 - f) * p2.zw + pos;
float3 xpos1 = float3(xystart.x, xystart.x + p2.z, xystart.x + p2.z * 2.0);
float3 xpos2 = float3(xystart.x + p2.z * 3.0, xystart.x + p2.z * 4.0, xystart.x + p2.z * 5.0);

return float4(
line_run(xystart.y , xpos1, xpos2, linetaps1, linetaps2) * columntaps1.r +
line_run(xystart.y + p2.w , xpos1, xpos2, linetaps1, linetaps2) * columntaps2.r +
line_run(xystart.y + p2.w * 2.0 , xpos1, xpos2, linetaps1, linetaps2) * columntaps1.g +
line_run(xystart.y + p2.w * 3.0 , xpos1, xpos2, linetaps1, linetaps2) * columntaps2.g +
line_run(xystart.y + p2.w * 4.0 , xpos1, xpos2, linetaps1, linetaps2) * columntaps1.b +
line_run(xystart.y + p2.w * 5.0 , xpos1, xpos2, linetaps1, linetaps2) * columntaps2.b, 1.0);
}


Since Sinc resampling is sharper than Lanczos, what I would like now is to try a Sinc shader, but honestly, I don't understand all the math involved. So far I have only tried to change the weight3 function like this:

float3 weight3(float x)
{
const float radius = 3.0;
float s1 = FIX(2.0 * PI * (x - 1.5));
float s2 = FIX(2.0 * PI * (x - 0.5));
float s3 = FIX(2.0 * PI * (x + 0.5));
float3 sample = float3(s1, s2, s3);
return sin(sample) / sample;
}

This works, but the result is more blurry, so I'm assuming the shader needs a more in-depth change, or even a rewrite.

I'm not even sure anyway that it will serve to improve the results, since the VDSR network is more limited in what it can improve when using good scaling, and the models that I'm using are quite reduced, but I think that it may be worth the try.

Once again thank you in advance for you help.

EDIT: p2 are the dimensions: (width, height, 1/width, 1/height)

[butterw2]

Your AviSynthAiUpscale project seems interesting, VDSR (Very Deep Super Resolution) is a deep learning approach for enlarging an image, but I haven't done anything with pixel shader resizers beyond bicubic so far. The reason being that Lanczos-3 requires a large kernel and is theoretically non separable (2D sinc-windowed sinc: k(r) with r polar coordinate).

Bicubic methods use a (4x4) convolution kernel req 16 texture taps. A simple improvement is to separate the horizontal and vertical passes, bringing it down to 8 taps.

Lanczos-3 (kernel: 2a*2a, a=3) req 36 taps, as demonstrated by your code (36 texture, 199 arithmetic).
Your code comes from xbmc, Kodi. Do you have an url ? Seems similar to https://github.com/xbmc/xbmc/blob/master/system/shaders/convolution-6x6_d3d.fx

However mpc-be has a 2-pass "compensated Lanczos3" implementation resizer_lanczos3_x.hlsl which seems to be 6+6 ? https://sourceforge.net/p/mpcbe/code/HEAD/tree/trunk/src/Shaders/Resizers/resizer_lanczos3_x.hlsl
I'm assuming you care more about quality than performance, but it may still be worth checking out.

by Sinc method do you mean SincResize from Avisynth ?
"uses the truncated sinc function. It is very sharp, but prone to ringing artifacts. "

Once you have the correct kernel, implementation shouldn't be too challenging.
If you are trying to implement the avisynth method, validation of the shader would be straightforward enough as you just need to compare to the avisynth reference.


# Pixel shader resizers in video players
Video player can use pixel shaders to perform scaling/resize (vs cpu sw scaling or fixed hw gpu scaling). Mpv allows user-defined scaling shaders, enabling the use of high-quality upscalers (a discrete gpu would be recommended for these shaders as they are not lightweight).
Mpc-hc/be does not allow user-defined resize shaders with the EVR-CP renderer: you must use one of the pre-defined resize methods. A pixel shader cannot change the frame resolution, but you can still test a scaling shader, by zooming in to the top left quarter screen for instance, ex: 1080p screen resolution and 1080p input video in fullscreen (no scaling by video player).
A resize shader will map the output coordinates (tex) to the corresponding input pixels to interpolate output pixel values. This is typically implemented using a convolution kernel (a more efficient 2-pass horizontal then vertical resizing approach is possible if the kernel is separable).
Note: resize operations should theoretically be performed in linear gamma.

[Alexkral]

Ok, thanks for your answer anyway.

Your code comes from xbmc, that's from Kodi. Do you have an url ?

Yes, I adapted it from this: https://github.com/libretro/common-shaders/blob/master/windowed/shaders/lanczos6.cg

However mpc-be seems to have a 2-pass "compensated Lanczos3" implementation which seems to be 6+6, look for Resizers\resizer_lanczos3_x.hlsl ?

I didn't know about this, I'll take a look at it.

by Sinc method do you mean SincResize from Avisynth ?
"uses the truncated sinc function. It is very sharp, but prone to ringing artifacts. "

Yes, everything is explained here (http://avisynth.nl/index.php/Resampling) as well as the other resampling methods, but I find it too complicated, and as I told you I'm not even sure that it can be useful, maybe when I have more time I will try again. I was hoping that since the Sinc kernel is simpler than the Lanczos kernel, maybe it could be done by slightly modifying the Lanczos code, but without really knowing how it works it seems that it will not be possible.

[butterw2]

I was hoping that since the Sinc kernel is simpler than the Lanczos kernel, maybe it could be done by slightly modifying the Lanczos code, but without really knowing how it works it seems that it will not be possible.

It's not fundamentally very complicated, but this code implementation is maybe not the simplest. They use .r, .b, .g instead of [0], [1], [2] for instance.
Switching kernel functions should just work, but performance will not be dramatically improved.
For reference, a 2-pass Catmull-rom resizer only requires (8 texture, 44 arithmetic).


# Convolution Kernels
A convolution kernel k(x, y) is a table of weights of size X*Y (ex: 3x3, 5x5 or 7x7). In a pixel shader, a convolution kernel might be calculated from parameters using the kernel function (if symmetry is present, it can be used to reduce the number of calculations required) or if constant the kernel can be hardcoded. The kernel is typically normalized so that the sum of weights is equal to one.
To calculate each output pixel value, the corresponding X*Y input pixels values are sampled and multiplied by their corresponding kernel weight to finally be summed.
A 2D separable kernel can be written as kh(x)*kv(y)=k(x, y) with kh and kv 1D-kernels. This allows 2D processing to be performed as 2 successive 1D-shader passes resulting in much more efficient operation in the case of large kernels.

[butterw2]

# Split screen display with pixel/fragment shaders:
This simple code modification can be applied to any pixel shader, and is useful to adjust parameter values and visualize what a shader actually does...
ex: Edge detection from Edge Sharpen https://raw.githubusercontent.com/butterw/bShaders/master/img/hsplit_scr_Edge_Detect_1080p.jpg

float4 main(float2 tex: TEXCOORD0): COLOR {
color=tex2D(s0, tex); //source pixel
/* horizontal split screen, left-half: no effect, right-half: with effect */
if (tex.x<0.5) return color;
...
}

It's of course possible to modify the code to display the same half-frame (with and without effect):
if (tex.x<0.5) return tex2D(s0, tex+float2(0.5, 0)); //Comparison Splitscreen: no effect

Note: This technique can be used pre or post-resize in mpc-hc/be (but post-resize, it only fully works in fullscreen).


it can also be applied to a mpv glsl .hook: https://github.com/butterw/bShaders/blob/master/mpv/bSide.hook.glsl
vec4 hook() {
vec4 color=HOOKED_tex(HOOKED_pos);
if (HOOKED_pos.x<0.5) return color;
...
}

Vertical split screen: if (tex.y<0.5) return color; //top-half: no Effect


For a sharpening filter, enabling/disabling the shader might be more effective to grasp the visual difference.


It's also of course possible to take screenshots (https://forum.doom9.org/showthread.php?p=1931990#post1931990) for comparisons.

[butterw2]

# Edge Detection

I've posted a couple of edge detection shaders based on convolution kernels at https://gist.github.com/butterw

- Edge_Sharpen.hlsl: mpc-hc Edge Sharpen optimization and luma edge detection mod (GPL v3 licensed)
Screenshot linked in preceding post.
Edge sharpening can help a soft source which is a bit soft without generating artifacts, it is one of the applications of edge detection.

On a clean high-res source, the following luma edge detection methods work very well on text, objects, hair, but maybe not so great on teeth.
- Frei-Chen edge detection in luma (mpv .hook shader)
can be used in combination with NoChroma.hook
Directly processing source luma seems to give better result (vs obtaining luma from rgb).



- Sobel/Canny Edge detection:
https://github.com/butterw/bShaders/bSobel_Edge.hlsl,
I've also made available a mpv .hook port of a glsl version of Sobel (in rgb).
Sobel is the most common algorithm for edge detection. It can be used stand-alone or as part of the Canny algorithm. The final stages of Canny are better implemented on cpu or compute shader however.

Canny edge detection video filter in ffmpeg (cpu): https://ffmpeg.org/ffmpeg-filters.html#edgedetect
mpv --vf=lavfi="[edgedetect=low=0.1:high=0.4]" input.mp4

These filters are also available (apply a grayscale filter on output):
mpv --vf=sobel input.mp4
mpv --vf=prewitt input.mp4
mpv --vf=roberts input.mp4

[Alexkral]

Switching kernel functions should just work

So I gave this another try (not knowing how it works yet). As I see it, the kernel is in the weight3 function, so I have changed it again like this:

float3 weight3(float x)
{
const float radius = 3.0;
float s1 = 2.0 * PI * (x - 1.5);
float s2 = 2.0 * PI * (x - 0.5);
float s3 = 2.0 * PI * (x + 0.5);
float ret1 = (abs(s1) < 2.0 * radius) ? sin(s1) / s1 : 0;
float ret2 = (abs(s2) < 2.0 * radius) ? sin(s2) / s2 : 0;
float ret3 = (abs(s3) < 2.0 * radius) ? sin(s3) / s3 : 0;
return float3(ret1, ret2, ret3);
}


I don't know why the samples are multiplied by two, but since it seems that it's needed, I also multiply the taps by two. This produces a VERY sharp result, even sharper than AviSynth's SincResize, so I think I'm on the right track. But if I debug the shader, I see 16 texture, 150 arithmetic, so I don't know what I should call it :p. Maybe it could be Sinc 2 taps? If I change 2.0 * radius to 7.0, the result is very similar and the code compiles to 36 texture, 209 arithmetic.

The metrics are not good but that doesn't matter too much to me, I'll use it to train some networks and then we'll see what the result is.

[butterw2]

# Optimizing gpu arithmetic operations

Whether variable x is float or float4, mad (multiply addition) counts as a single operation on gpu, ex: x*x + y, 2*x +3*4.2 (1 arithmetic op)

Weight3 is the kernel function, yes, but the code uses vectorized float3 operations, which can make it more difficult to understand.
I don't know how poorly a naive (brute force) implementation performs. There doesn't seem to be other code available for a single pass lanczos-3 pixel shader.
If you want to try it out, I can upload the 2-pass lanczos-3 shader code from mpc-be (which I've adapted to run in mpc-hc). Texture fetches hit the gpu harder than arithmetic ops, but it is still useful to reduce arithmetic ops when possible.

You need to help out the compiler in some cases, check the compiler output to see if this saves one operation for instance:
float s1 = 2.0 * PI * (x - 1.5);
can be written as:
float s1 = 2 * PI * x - 2 * PI *1.5; //1 arithmetic

Instead of floats s1, s2, s3
you could use one vector operation to calculate float3 s
and then use s.x, s.y, s.z

# Cost of hlsl functions (in arithmetic operations)
Single op: A*B+C (mad), float dot(A, B), frac(A), saturate(A)
2: lerp(A, B, 0.5 ), floor(A), step(A, 0.5), clamp(0.2, 0.8, A)
3: float length(A)
4: fmod(A, 2)
4,5: smoothstep(0, 1.0, A)
trunc(A)
---
ops>8: If multiple operations on float variables are needed, look at packing a vector.
sqrt(A)
sin(A)

[Alexkral]

If you want to try it out, I can upload the 2-pass lanczos-3 shader code from mpc-be (which I've adapted to run in mpc-hc).

Yes please, that could be helpful, most of what I've learned has been just looking at the code.

Instead of floats s1, s2, s3
you could use one vector operation to calculate float3 s
and then use s.x, s.y, s.z

I needed to change that from the original code (the link I posted) because the max function gave wrong results, but you are right, now it can be done this way.

[butterw2]

Compensated Lanczos3 pass X (from mpc-be, adapted for mpc-hc user shader with 2x magnification):
https://gist.github.com/butterw/8190fe5f9c24687ee96bb6f188b9222b
You'll also need the second pass (pass-Y), which is the same as pass-X, but applied to y-axis.

In hlsl, texel centers (tex) are situated at half-pixel coordinates. If you don't need 2x magnification, don't start from my adaptation, start from the original code (just define dxdy as required, I prefer using p (or p1) for this, the pixel widths are then p.x, p.y).

float4 p2: register(c2); //in this application, c2 contains: (width, height, 1/width, 1/height)
#define p p2.zw //pixel widths (p.x, p.y)




# Testing a shader with a png image input
A good way to test a video player shader is to use a png image as input. Output will update automatically when you modify/save the shader and an output screenshot can be saved for comparison if required.
Input: RGB(A) png (preferably of same resolution as the output screen), shader (preferably pre-resize shader), output (fullscreen preferred for screenshot in windows)
The idea is to bypass compression artifacts (png is lossless), yuv2rgb conversion/range expansion, video player scaling, so as to only test the pixel shader.

[Alexkral]

Thanks, I'll have some time in the next few days to take a look at all this, hopefully soon I will understand a little bit more.

[Alexkral]

So I didn't have much luck trying to change this code either, but it's ok, I don't want to keep trying or filling this thread with posts related to this. I just wanted to point out that the mpc-be code is wrong, the changes you made fixed it, but it still produces a half pixel shift to the left (same as the xbmc code but in the opposite direction), you can fix it like this:

float coord = (tex.x - p1.x)*0.5*p0.x;

Other than this, the performance of the mpc-be shader is only slightly worse than the other implementations.

[butterw2]

The mpc-be resizer uses pixel scale for the tex.x coordinate.
ex tex.x: 421.5 for a 1000x1000 output frame, whereas the typical hlsl scale, used in mpc-hc/be user shaders is [0, 1.]

For 2x magnification, the corresponding input texel center would be at: 210.75 -0.75 +0.5=210.5 in output frame coordinates, given the half-pixel offset of texel centers in hlsl.

The perf of my port is (6 texture, 79 arithmetic) for X-pass (I've removed the t==0 case because it's not required).
I would expect a significant perf improvement in 2-pass (12 texture, 158 arithmetic) vs xbmc(36, 199) because the number of texture fetches is much lower.
That said I haven't measured frame times, the bottleneck may be different on igpu (vs dgpu) and going 2-pass (vs single pass) may introduce additional overhead with avisynthshader.

I am not clear how you use the lanczos shader in your app, is it fixed 2x upscaling, arbitrary ratio resize or is it 1x interpolation ?

float4 p2: register(c2); //(W, H, px, py)
Does the avisynth script define the content of the registers ? Why does avisynthshader need the c2 register here ? Is there a difference between c2 and the values in c0 and c1 (as used by mpc-hc) ?

While it isn't off-topic here (an avisynthshader script could potentially be used for testing and maybe even in realtime ?), you are welcome to reply in your own thread if you wish.


# Mpc-hc/be Registers

player(cpu).>> registers
.................>> frame (ex: RGBA) >> pixel Shader1(gpu) >> processing surface >> pixel Shader2 >> Output

Shaders must be compiled, before they are executed (ex .cso). They can be cached on disk.
The application executing the shader can pass parameters to it through registers.

Registers contains read-only data set by the application. They can be updated every frame and are readable by user pixel shaders.
These parameters are made available by mpc-hc/be: all individual values are floats
sampler s0: register(s0); //the input frame texture
float4 p0: register(c0);
#define W p0.x //(p0[0]) //image Width
#define H p0.y //p0[1]) //image Height
#define Counter (p0[2]) //frame counter, ! it doesn't get reset when you stop video.
#define Clock (p0[3]) //time counter in seconds
float2 p1: register(c1); //p1.xy
#define px p1.x //1/W
#define py p1.y //1/H

Testing parameter/variable values
A crude method to test the content of H would be as follows:
if (abs(H-1080)<0.001) return float4(1, 0, 0, 1); //return a red pixel/screen if H is close to 1080
rounding point error means testing for equality doesn't always work with floats !
In this case (and with int variables) it is safe to write: if (H==1080) return float4(1, 0, 0, 1);

[Alexkral]

I was referring to the metrics (psnr and ssim), I didn't test the speed.

The VDSR network learns to estimate a residual image, which is the difference between a high-resolution reference image and a low-resolution image that has been upscaled to match the size of the reference image. The low-resolution image in usually upscaled using bicubic interpolation, but I have obtained better results using Lanczos. I trained several models with different scaling factors, which are used depending on the upscaling ratio. VDSR allows for multiscale training, but it hasn't worked well for me. So it is arbitrary ratio.

In AviSynth Shader, when using the Width and Height parameters to resize the texture, the c0 and c1 registers are set to the output values, so I need to use another parameter to set another register with the input values.

[butterw2]

What is the benefit of training with a pixel shader resizer vs the built-in avisynth cpu implementation ?

The code for a 2-pass separable resizer (an approximation in this case) is maybe easier to understand because you only deal with 1 dimension at a time. I'm not sure how far you want to go with this or if it is even desirable, but I would assume it is possible to modify the code to achieve a larger kernel (lanczos4, etc.) without degrading performance (meaning speed) vs xbmc lanczos3.

Edit: Since you are an AvisynthShader user, are there any adaptation steps needed to made shaders compatible or does any hlsl effects shader for mpc-hc work by default ?

[Alexkral]

There is no benefit when training, the results of the shader and the AviSynth and Matlab implementations are very similar, in fact I use Matlab's Lanczos for training and then the shader for testing in the app. The benefit comes when testing, because the bottleneck in AviSynth Shader is in the memory transfers between the CPU and the GPU, so it is slower to send the upscaled texture, than to upscale it in the GPU. This is what prevents me from using AviSynth's SincResize.

Anyway, I'm doing some tests now with the modified xbmc code and the difference in the results doesn't seem to be too significant, so I don't think using Lanczos4 would make any difference.

As for AviSynth Shader, other than what I said about the registers, I think there are only a few issues with some color formats. Using RGB as input/output, the order of the channels in the shader is changed to BGRA and they must be returned in that order.

[butterw2]

# Effects available as video pixel shaders (gpu) Grayscale and bi-color Effects

The effects listed here are meant to be used in video players (.hlsl: mpc-hc/be, etc., .hook/glsl: mpv). It may also be possible to use them for processing/encoding of video or images. Typical video player input is compressed yuv420 (color video with chroma subsampling). Mpc-hc shaders operate on pre or post resize RGB, whereas mpv shaders can additionnaly operate on source YUV (or RGB).
Many effects have parameters that can be adjusted in the source file, and some guidance will be provided as to further customizations.

Effects apply a visual style to a video. They modify the aspect of the displayed content to change the viewing experience. They are not meant to be used permanently.
Some effects are deliberately designed to degrade the displayed video quality or imitate the elements of past video/film technology.
Some are intended for use with specific types content (Internet Videos, Music Videos, Home Captures) rather than Movies.


## Grayscale Effects (in 8bit output is 256 shades of gray)

Monochrome
- In mpv, you can zero the chroma channels from the source YUV (NoChroma.hook). This is the recommended approach if the content is in fact grayscale.
- The more general approach is to calculate luma = dot(color_RGB, LumaCoeff_RGB). The value of LumaCoeff will change how the picture looks (SweetFx.Monochrome
provides values corresponding to different film types, ex: Agfa 200X). The sum of the R, G And B components of LumaCoeff should be equal to 1.

Edge Detection
Most edge detection methods work on and output Luma. The algorithms use an adjustable detection Threshold.
- Sobel_Edge.hlsl, Sobel.hook
- Frei-Chen.hook
- Edge_Sharpen.hlsl (mod/perf optimization of mpc-hc version).
One application of Edge Detection is edge sharpening. Colored Edges can also be overlayed on the original image for stylistic effect.

Halftone dots.hlsl art effect, for video rather than movies
greyscale (mostly black and white) dot pattern, can be used twice. The mpv version (dots.hook) keeps chroma by default.



A color tinted image can be obtained from grayscale, by using the grayscale values as one of the color channels, ex: red channel.
To obtain a tinted image of any tint (ex: Sepia): color_RGB = luma * TintCoeff_RGB;
A sepia effect should ideally not change the original brightness. The white color from the original should not be changed too much. https://github.com/butterw/bShaders/blob/master/Effects/Sepia.md

In mpv, the source chroma NoChroma shader could be modified to apply small offset to the grayscale value (0.5, 0.5) to get tint.


## Bi-Color Effects (Output palette only contains 2 colors)
These effects typically output Black + White output by default, but this can be changed to any color combination.

bi-Threshold.
A Binary Threshold applied to a greyscale image (typ. luma)
To avoid a full-range black and white, which can be glaring, you can clamp the output, for instance to 16-235 RGB levels.

Stipple Effect
Grayscale dithering: A grayscale image can be transformed into a bw (2-color) image with an error-diffusion algorithm (ex: atkinson, floyd-steinberg). These algorithms cannot be efficiently implemented as pixel shaders as they are not parallel, but it may be possible to use compute shaders.


# Color desaturation
The intermediate image between the color original (C) and the grayscale version (G) is a desaturated color image. It is obtained by performing a linear interpolation between C and G with s the desired strength of the effect (s is a weight between [0, 1.0] 0: C, 1: G).
desaturated_color = lerp(C, G, s); // in glsl: mix(C, G, s)

lerp(x, y, s) = x*(1-s) + y*s = x + s(y-x), with x, y: RGB vectors.

# Alpha Blending (Transparency)
The same function is used for RGB alpha blending between the color image (C) and the background (B):
alpha between [0, 1.0] 0: B (transparent image), 1.0:C (opaque)
alpha_blended_RGB = lerp(B, C, alpha)

Note: the alpha channel is present but unused in mpc-hc and mpv RGBA colors.

[butterw2]

# Lightweight Noise/FilmGrain Shaders

On a low intensity setting, Noise/FilmGrain improves perceptual quality of most compressed videos. In particular content with low or no noise, (ex: the source noise may have been removed at encoding to improve compressibility). It may also help mask artifacts.

The main reason not to use Filmgrain in mpc-hc/be was the performance hit on lower end systems. These new shaders offer a major speed improvement over previously available shaders without creating a visible noise pattern.

https://github.com/butterw/bShaders/tree/master/FilmGrain_Noise
- bNoise.hlsl (1 texture, 12 arithmetic)
- FilmGrain1.hlsl (1 texture, 30 arithmetic) mpv FilmGrain.glsl port.

FilmGrain/Noise refers to dynamic additive grayscale noise with a gaussian distribution for FilmGrain and a band-limited uniform distribution for Noise.

Some limited quality testing was done based on the output noise histogram obtained through screenshots, but more feedback would be welcome.

[butterw2]

# Unfocused Effect (ex: Large Kernel Gaussian Blur)

Implementation in mpc-hc of a pixel shader preset for fast multipass blurring was previously discussed page1 (separable kernel, blurring a downsized image, hw linear sampling, approximate gaussian algorithms: Boxblur, Kawase Blur). It's certainly possible, but not that simple to achieve high quality, high performance blur.

When upscaling, Blur should always be performed pre-resize (less pixels to handle, stronger blur effect on a lower resolution image for a fixed Kernel size). When downscaling (ex: 4K content to 1080p) Blur should be performed post-resize.
For larger kernel sizes (k), a 2-pass separable kernel approach (horizontal pass, followed by vertical pass) performs much better than a single pass 2D-convolution:
O(2k) vs O(k^2) texture fetches required, with k the kernel size.

In Mpv
mpv can apply ffmpeg libavfilter video filters to its input (uses mainly cpu). However the boxblur filter doesn't seem to work for me. It does work in ffmpeg/ffplay (a simple SDL cross-platform media player using ffmpeg): ffplay -i "video.mp4" -vf boxblur=10:10

mpv supports extra features with regard to shaders vs default mpc-hc (hw linear sampling, multipass in a single shader, multiple textures with custom resolutions, compute shaders), which means it should be possible to implement fast high quality blur shaders (https://github.com/butterw/bShaders/tree/master/blur).

Downsizing the frame (W, H)/2 at the input of the blur shader enables stronger blurs for the same gpu usage. However decreasing the resolution too low (y<240 ?) will impact the quality of the effect.

EDIT: Blurred sidebars, which are popular on mp4 internet videos (ex: 1:1 AR displayed fullscreen on a 16/9 monitor) can be achieved using a realtime avisynth script (https://github.com/butterw/bShaders/blob/master/blur/fill_bb.avs) (cpu) playable in in mpc-hc/be or mpv.

[butterw2]

# Mpv shader basics

Tests performed with a portable version of mpv-x64 on Win10:
https://mpv.io/installation/
this default mpv version doesn't come with a GUI like mpc-hc/be: it's exe (+command-line) + On-Screen-Controller + Hotkeys.

Install directory (ensure it has write permission).
create ./portable_config for configuration files:
- mpv.conf (config file, not provided by default, but you do really need one !!!)
- input.conf (custom hotkeys will override defaults)

you can put shaders directly in the install dir or you can use a subfolder. No user shaders are provided by default.

There are 3 ways to select/control shaders in mpv:
- default configuration (mpv.conf)
- launch command-line (can add shaders or override default configuration)
- hotkeys


Example ./portable_config/mpv.conf (vo=gpu d3d11 with dxva2-copy on Win10):
#-- Shaders --------
glsl-shader="noChroma.hook"
glsl-shader="./shaders/CrossHatch.hk"
# glsl-shader="./shaders/freichen.hk"
# glsl-shader="./shaders/cs_wkgroups.hk.glsl"
--gpu-shader-cache-dir="C:\Temp" # ! set the shader cache to a directory with write access

#-- Video ------
hwdec=dxva2-copy #hwdec=d3d11va-copy
hwdec-codecs=all
gpu-api=d3d11
gpu-context=d3d11
vo=gpu
#profile=fast
audio-device=auto

#-- Scalers -- Luma/Rgb scale (dscale), Chroma (cscale)
dscale = mitchell #cscale= bilinear
scale = catmull_rom

##-- Screenshots --------
screenshot-format=png
screenshot-jpeg-quality=96

## --Window settings--------
# Start in fullscreen mode by default. #fs=yes
# force starting with centered window #geometry=50%:50%
# geometry=720x720 #fixed window size
# don't allow a new window to have a size larger than 90% of the screen size # autofit-larger=90%x90%
# mpv doesn't have an enforceable option to start at 100% scale:
autofit-larger=1280

keep-open=yes # Do not close the window on exit.

[butterw2]

Mpv shaders are written in glsl using a specific .hook syntax, they can contain a number of independent passes each with associated HOOK points.
Intro here: https://forum.videohelp.com/threads/397797-Pixel-shaders-for-video-playback-(-hlsl-glsl)#post2597322
The main hook points in the processing pipeline are: LUMA, CHROMA (source yuv), NATIVE(yuv), MAIN (pre-resize RGB) and OUTPUT (post-resize RGB).
A single .hook file can contain multiple passes. To disable a pass, you can specify a non-existing HOOK point (ex: None).

A Pre-scaler (custom Luma or Chroma Upscaling Shader) works as follows:
Source Resolution, ex:1080p >> user Shader (ex: HOOK LUMA) >> texture resolution (ex: 2x W, H) >> (built-in scaler) >> Screen Resolution 2160p

## Using custom shaders in mpv
default shaders are configured in mpv.conf (with multiple glsl-shader= lines or a single glsl-shaders line).
Shader configuration can additionnally be configured with command-line launch parameters.

to append a shader for playback (ex: chromaOff):
mpv --glsl-shader="./shaders/NoChroma.hook" "video.mp4"

to desactivate or override config file shaders:
mpv --glsl-shaders="" "video.mp4"

If an enabled shader has a syntax error, you will get a blue screen display and the compiler error output in the launch terminal.

[butterw2]

mpc-hc > Help > Command Line Switches lists available command line options.
They can also used by editing the properties of the Windows launch icon.

The following launches mpc-hc
sets the shader preset named "Blur",
opens video1,
followed by video2 in the playlist:

mpc-hc /shaderpreset "Blur" "video1.mp4" "video2.mp4"