Emulating MadVR's HDR to SDR tonemapping using FFMPEG

[EmKayVe]

remove the space before zscale=s=1920x1080

Awesome! Thanks that worked.

I'd love to learn more about ffmpeg configuration. Can anyone point me in the right direction for a thread that can help me with this - like audio tracks and such?

So I feel like the HDR to SDR conversion taking place with the configuration in the ffmpeg encode is acceptable. Is there a way to improve upon it or is that pretty much as good as it gets.

[tebasuna51]

https://www.ffmpeg.org/documentation.html

All about command line decoders/encoders/filters...

[EmKayVe]

Ok thank you. I am now only looking for a line to pass through the audio. Then I'll take the resulting file and finish up in Handbrake. I'll just use ffmpeg to tonemap HDR to SDR.

[Beam]

What am i doing wrong?

ffmpeg -ss 00:01:00.12 -i Sony_4K_HDR_Camp.mp4 -frames 1 -vf zscale=t=linear,format=gbrpf32le,zscale=p=bt709,tonemap=tonemap=hable,zscale=t=bt709:m=bt709:r=tv Sony_4K_HDR_Camp_HABLE.png


ffmpeg version 4.1.3 Copyright (c) 2000-2019 the FFmpeg developers
built with FreeBSD clang version 6.0.1 (tags/RELEASE_601/final 335540) (based on LLVM 6.0.1)
configuration: --prefix=/usr/local --mandir=/usr/local/man --datadir=/usr/local/share/ffmpeg --pkgconfigdir=/usr/local/libdata/pkgconfig --enable-shared --enable-pic --enable-gpl --enable-postproc --enable-avfilter --enable-avresample --enable-pthreads --cc=cc --disable-alsa --disable-libopencore-amrnb --disable-libopencore-amrwb --disable-libaom --disable-libass --disable-libbs2b --disable-libcaca --disable-libcdio --disable-libcelt --disable-libcodec2 --disable-libdav1d --disable-libdavs2 --disable-libdc1394 --disable-debug --disable-htmlpages --disable-libdrm --enable-libfdk-aac --disable-libflite --disable-fontconfig --disable-libfreetype --disable-frei0r --disable-libfribidi --disable-gcrypt --disable-libgme --enable-gmp --enable-gnutls --enable-version3 --disable-libgsm --enable-iconv --disable-libilbc --disable-libjack --disable-libklvanc --disable-libkvazaar --disable-ladspa --enable-libmp3lame --disable-liblensfun --enable-libbluray --disable-librsvg --disable-librtmp --disable-libxml2 --disable-lv2 --disable-mbedtls --enable-mmx --disable-libmodplug --disable-libmysofa --enable-nonfree --disable-openal --disable-opencl --disable-libopencv --disable-opengl --disable-libopenh264 --disable-libopenjpeg --disable-libopenmpt --disable-openssl --enable-optimizations --disable-libopus --disable-libpulse --enable-runtime-cpudetect --disable-librubberband --disable-sdl2 --disable-libsmbclient --disable-libsnappy --disable-sndio --disable-libsoxr --disable-libspeex --disable-libsrt --enable-sse --disable-libssh --disable-libtensorflow --disable-libtesseract --disable-libtheora --disable-libtwolame --disable-libv4l2 --disable-indev=v4l2 --disable-outdev=v4l2 --disable-vaapi --disable-vapoursynth --disable-vdpau --disable-libvidstab --disable-libvorbis --disable-libvo-amrwbenc --disable-libvpx --disable-libwavpack --disable-libwebp --enable-libx264 --enable-libx265 --disable-libxavs2 --disable-libxcb --enable-libxvid --disable-outdev=xv --enable-libzimg --disable-libzmq --disable-libzvbi
libavutil 56. 22.100 / 56. 22.100
libavcodec 58. 35.100 / 58. 35.100
libavformat 58. 20.100 / 58. 20.100
libavdevice 58. 5.100 / 58. 5.100
libavfilter 7. 40.101 / 7. 40.101
libavresample 4. 0. 0 / 4. 0. 0
libswscale 5. 3.100 / 5. 3.100
libswresample 3. 3.100 / 3. 3.100
libpostproc 55. 3.100 / 55. 3.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'Sony_4K_HDR_Camp.mp4':
Metadata:
major_brand : isom
minor_version : 1
compatible_brands: isom
creation_time : 2016-02-03T08:01:30.000000Z
Duration: 00:02:07.15, start: 0.000000, bitrate: 75806 kb/s
Stream #0:0(und): Video: hevc (Main 10) (hvc1 / 0x31637668), yuv420p10le(tv, bt2020nc/bt2020/smpte2084), 3840x2160 [SAR 1:1 DAR 16:9], 75620 kb/s, 59.94 fps, 59.94 tbr, 60k tbn, 59.94 tbc (default)
Metadata:
creation_time : 2016-02-03T07:59:49.000000Z
handler_name : Video Media Handler
encoder : HEVC Coding
Stream #0:1(eng): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 192 kb/s (default)
Metadata:
creation_time : 2016-02-03T07:59:49.000000Z
handler_name : Sound Media Handler
File 'Sony_4K_HDR_Camp_HABLE.png' already exists. Overwrite ? [y/N] y
Stream mapping:
Stream #0:0 -> #0:0 (hevc (native) -> png (native))
Press [q] to stop, [?] for help
[hevc @ 0x805186600] First slice in a frame missing.
Last message repeated 6 times
[hevc @ 0x8059b6000] First slice in a frame missing.
Last message repeated 6 times
Output #0, image2, to 'Sony_4K_HDR_Camp_HABLE.png':22.77 bitrate=N/A speed=N/A
Metadata:
major_brand : isom
minor_version : 1
compatible_brands: isom
encoder : Lavf58.20.100
Stream #0:0(und): Video: png, rgb48be, 3840x2160 [SAR 1:1 DAR 16:9], q=2-31, 200 kb/s, 59.94 fps, 59.94 tbn, 59.94 tbc (default)
Metadata:
creation_time : 2016-02-03T07:59:49.000000Z
handler_name : Video Media Handler
encoder : Lavc58.35.100 png
frame= 1 fps=0.3 q=-0.0 Lsize=N/A time=00:00:00.01 bitrate=N/A speed=0.00537x
video:808kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown


And this is result: :confused::confused:

https://i.imgur.com/1dptKUt.png

[Gravitator]

What am i doing wrong?
Tried it?

ffmpeg.exe -ss 00:01:00.12 -i Sony_4K_HDR_Camp.mp4 -frames 1 -vf zscale=transfer=linear,tonemap=clip,zscale=transfer=bt709,format=yuv420p c:\users\dave\desktop\ffmpeg.png

[manolito]

What am i doing wrong?

I believe I figured it out... :devil:

Have a look here:
https://forum.doom9.org/showthread.php?p=1898420#post1898420

My HDR to SDR readme for using FFmpeg:
Using FFmpeg to convert HDR sources to SDR:
======================================================================

This video filter call will do it (thanks to dipje and Atak_Snajpera):

zscale=s=704x396,zscale=tin=smpte2084:min=bt2020nc:pin=bt2020:rin=tv:t=smpte2084:m=bt2020nc:p=bt2020:r=tv,zscale=t=linear:npl=100,format=gbrpf32le,zscale=p=bt709,tonemap=tonemap=hable:desat=0,zscale=t=bt709:m=bt709:r=tv,format=yuv420p,colormatrix=bt709:bt601

These parameters are easy to modify for HD output. Edit the "zscale"
parameter for the desired size and remove the "colormatrix" parameter.

This FFmpeg parameter line is meant to be inserted in the "Picture
Settings -> More" field of dmMediaConverter. If you want to use them
in an existing FFmpeg commandline you need to add them as a video
filter and enclose the parameters in double quotes.
Like:
-vf "parameter list"


Note:
If you use the source file directly as the FFmpeg input you may get
borked colors in the output depending on the source format. The safest
way to correct this is to use an AVS script with just the source
filter as the input for FFmpeg (of course only if AviSynth is installed).
Otherwise you have to pre-convert the source to a lossless intermediate
file keeping all the source properties. Then use this lossless file as
the input for FFmpeg and apply the HDR to SDR filter.


Cheers
manolito

[PatchWorKs]

Hi everyone, I honestly don't know if these are overlapping color correction (or equalization ?) problems, but here's another interesting 3ad - I'm partecipating in - about them:

https://forum.videohelp.com/threads/395939-ffmpeg-Color-Range

[damian101]

Think I got the desaturation a bit figured out. You / we never change the color primaries and/or the colormatrix. That means the primaries are still in bt2020. We tonemap the brightness but not much else.

There might be a simpler method, but what I do:
zscale=t=linear,format=gbrpf32le,zscale=p=bt709,tonemap=tonemap=hable,zscale=t=bt709:m=bt709:r=tv

(after the whole 'setting the correct input format' thing).

The first zscale=t=linear turns the YUV format into linear-space-RGB. (Linear space is only a thing in RGB). But the colorprimaries are still in bt2020. (Note, not the colormatrix. The colormatrix is a YUV only thing. But the colorprimarie is basically your colorspace).
Then comes the format=gbrpf32le to turn it all into floating-point RGB. _Then_ do we tell zscale to change the primaries to bt709. According to the zimg documentation changing primaries need to be done in linear space anyway, and this way we make sure the data is linear first. Because we're changing from bt2020 to bt709, we actually start to clip the data at very saturated colors. This is why we first convert to floating point, so this data isn't lost, it's just out-of-gamut.
Then we do the tonemapping, and because it now has data that falls outside of the 0.0 - 1.0 range, it can do it's thing and start desaturating the colors where it thinks it's needed.


Don't do this. Primarily because tonemap desat will clip out-of-gamut pixels to black, which produces ugly artifacts. I also can't quite follow your reasoning, even without desat I can only think of negative possible side effects of this. If you want to prevent desaturation, just set desat to 0. Which is something I don't recommend btw. You always want at least some desaturation for clipped pixels, otherwise some pixels will clip to white way too late, or even never, when one of the three primary colors is completely missing, regardless of brightness This can create unrealistic brightness differences, which is way more distracting than desaturation, especially without reference. Smaller desat value means more desaturation. Don't go below 1 though, at least at 0.2 and lower desat I encountered very visible desaturation on unclipped pixels as well. The default of 2.0 is fine, I normally go lower to catch some edge cases (did some encodes with 1.3 and 1.2, with higher npl lower desat values tend to be a better idea), I rarely mind the desaturation in movies, which might be because I tend to use very high npl (400-700) with mobius, so only extremely bright pixels are desaturated at all. If you want to prevent desaturation at the cost of less accurate brightness differences I recommend a higher desat value, 4 for example. Wouldn't really recommend going higher than that, way too many cases where brighter light sources clip later than less bright light sources because of color differences.

[damian101]

desat destroys image. Desat must be disabled. Period!
http://i.cubeupload.com/34c2yw.png

Well, that's probably because you applied BT.709 primaries before tonemapping.

[damian101]

Single threaded tonemapping is creating huge bottleneck on my 8C/16T CPU. Is there any way to speed up zscale by using multi-threading?
http://i.cubeupload.com/RLMy7N.png

http://i.cubeupload.com/JR9KMb.png

zscale colorspace conversion seems indeed to be the bottleneck. However, it's not much slower than tonemapping in my testing, so not a huge bottleneck.

Some benchmarks:

-vf zscale=t=linear:npl=300,format=gbrpf32le -> 19 fps

-vf zscale=t=linear:npl=300,format=gbrpf32le,zscale=t=bt709:p=bt709:m=bt709 -> 10 fps

-vf zscale=t=linear:npl=300,format=gbrpf32le,tonemap=tonemap=mobius -> 13 fps

-vf zscale=t=linear:npl=300,format=gbrpf32le,tonemap=tonemap=mobius,zscale=t=bt709 -> 9.5 fps

-vf zscale=t=linear:npl=300,format=gbrpf32le,tonemap=tonemap=mobius,zscale=t=bt709:p=bt709:m=bt709 -> 8.3 fps

-vf zscale=t=linear:npl=300,format=gbrpf32le,tonemap=tonemap=mobius,zscale=t=bt709:p=bt709:m=bt709,format=420p -> 7.2 fps

-vf zscale=t=linear:npl=300,format=gbrpf32le,tonemap=tonemap=mobius,zscale=t=bt709:p=bt709:m=bt709:d=ordered,format=yuv420p -> 7.2 fps

-vf zscale=t=linear:npl=300,format=gbrpf32le,tonemap=tonemap=mobius,zscale=t=bt709:p=bt709:m=bt709:d=error_diffusion,format=yuv420p -> 6.3 fps

-vf zscale=t=linear:npl=300,format=gbrpf32le,tonemap=tonemap=mobius,zscale=t=bt709:p=bt709:m=bt709 -pix_fmt yuv420p -> 4.3 fps

-vf zscale=t=linear:npl=300,format=gbrpf32le,zscale,format=yuv420p -> 12 fps

-vf zscale=t=linear:npl=300,format=gbrpf32le -pix_fmt yuv420p -> 6.3 fps

You can also see that swscale format conversion is the slowest operation of them all, so make sure you do it with zscale instead.

mobius and hable are only insignificantly slower than clip

[damian101]

Thank you for the post.
The CL generates 3840x2160 rawvideo. If 1920x1080 is needed instead, add one more resizing filter:

format=yuv420p,zscale=s=1920x1080


Resize before tonemapping to speed tonemapping up a lot.

[Anuskuss]

Is there an alternative for when zscale isn't available?

[huhn]

try libplacebo

[Anuskuss]

try libplacebo

And what if that's not available either (this is not a hypothecial question)?

[huhn]

i don't know a 3rd thing that act as a scaler in ffmpeg.

[manolito]

If you are not sold on FFmpeg and AviSynth+ is an alternative for you, then you can try DGHDRtoSDR by Donald Graft.

Good luck
manolito

[Anuskuss]

Thanks for the tips but neither AviSynth+ nor DGHDRtoSDR are suitable alternatives - I need to use ffmpeg. In the end I decided (https://github.com/intel/media-driver/issues/1833#issuecomment-2227584676) to go for OpenCL:
ffmpeg -i hdr.mp4 -vf format=p010,hwupload,tonemap_opencl=tonemap=mobius:format=nv12:m=bt709:p=bt709:r=tv,hwdownload,format=nv12 thumb-%06d.jpg

The colors are not perfect but better than doing no color conversion at all. If someone knows an ffmpeg filter which improves upon this please let me know.

[GeoffreyA]

The colors are not perfect but better than doing no color conversion at all. If someone knows an ffmpeg filter which improves upon this please let me know.

I gave it a go and managed to get improved results:

ffmpeg -init_hw_device opencl -i HDR.mkv -vf hwupload,tonemap_opencl=tonemap=reinhard:param=0.1:desat=0:format=nv12:t=bt709:p=bt709:m=bt709:r=tv,hwdownload,format=nv12 SDR.mp4

While not perfect, it is not too far from the zscale/tonemap version that I use:

ffmpeg -i HDR.mkv -vf zscale=t=linear:npl=291,format=gbrpf32le,zscale=1920:-1:f=spline36,tonemap=reinhard:param=0.6:desat=0,zscale=t=709:p=709:m=709:r=limited:dither=error_diffusion,format=yuv420p,sidedata=delete OUTPUT

If it's too dark, raise "param" a little at a time. Another problem is that the brightness fluctuates per scene: there is a "threshold" parameter but I didn't test it.

[manolito]

Is there an alternative for when zscale isn't available?

I still do not understand why zscale is not available when you use a plain vanilla current version of FFmpeg. Could you upload a short sample of your source file and specify the desired target size so forum members could do some experimentation?


Cheers
manolito

[Anuskuss]

when you use a plain vanilla current version of FFmpeg

I'm using a fork of ffmpeg called Plex Transcoder which is used by Plex (https://www.plex.tv) (among other things) for thumbnail generation. In it's default state it will create washed out thumbnails for HDR content so I hijacked the command and inserted the OpenCL tonemap filter.

On my desktop I can just do zscale+tonemap which looks way better of course. Maybe someday I'll investigate how the thumbnails are stored on disk and replace them myself, although that sounds tedious and like something I'll eventually forget to do.

[GeoffreyA]

ffmpeg -init_hw_device opencl -i HDR.mkv -vf hwupload,tonemap_opencl=tonemap=reinhard:param=0.1:desat=0:format=nv12:t=bt709:p=bt709:m=bt709:r=tv,hwdownload,format=nv12 OUTPUT


Not perfect but an improvement. If too dark, raise "param" a little at a time.

[Anuskuss]

Not perfect but an improvement. If too dark, raise "param" a little at a time.

While I do appreciate the effort I don't think I'll ever touch the tuning parameters, as I find them too subjective. Sure I can create some beautiful colors with some fine tuning but I have no guarantee that it will look good in the next scene, let alone in a different movie. I'd rather have some "sane" defaults that some color theorist came up with or something that is based on some sort of defined standard. I mean Plex is basically doing the same thing anyway (just with an additional scale_vaapi to get the data into the right format) (https://github.com/intel/media-driver/issues/1833#issuecomment-2226959167) and while I don't think they do the best thing possible, they must've put some thought into it.

[GeoffreyA]

I agree and am of the encoding philosophy to keep the picture "as is." (Except in cases where work is needed, such as debanding anime or fixing problematic material.) What nags me is that, while there are standard ways of tone mapping, for example the ITU-R recommendations, it seems that in practice, one always has to tune the parameters a bit. I wish this were a "solved problem" but we're not there yet and subjectivity is involved.

At present, I have a project where I'm tone mapping the Lord of the Rings 4K remastering down to SDR 1080p, trying to match the remastered 1080p Extended Edition I have access to, and while I'm getting excellent results, it is challenging to match the reference exactly, and getting one scene right may fail another.

[Argaricolm]

I delete my question. To do same as MadVR does we need to do "linear" tonemapping (libplacebo has it).