piping HDR Rec.2020 from Vapoursynth to ffmpeg via vspipe

[hydra3333]

I'm green (so to speak) around Rec.2020 so feel free to refer me off to another link.

Just wondering about HDR Rec.2020 output from Vapoursynth and piping it to ffmpeg via vspipe.

In Vapoursynth Getting Started "Output with VSPipe" examples in http://www.vapoursynth.com/doc/gettingstarted.html are there any changes to the commandlines in relation to piping HDR ? Would it be possible to add an example or two in here ? Or is it already documented somewhere else in lay terms ?

Thanks

[dipje]

(Most) of the HDR content is no different than most other YUV videodata, just YUV420P10.

If your script outputs something like YUV420 P10, with 2020 non-constant YUV matrix, limited range / tv range, 2020 color primaries, smpte2084 transfer characteristics, I would go with something like this:

vspipe -y <input.vpy> - | ffmpeg -i - -vf zscale=min=bt2020nc:m=bt2020nc:rin=tv:r=tv,pin=bt2020:p=bt2020:tin=smpte2084:t=smpte2084

Followed by any other filters and other ffmpeg parameters you might need.
Piping the data from Vapoursynth to ffmpeg isn't the problem,but you need to tell ffmpeg what the characteristics of the videostream are, that's where the big zscale filter comes in.
It's just basically a no-op, converting from input to output without change, but in doing that the videostream gets tagged with the correct parameters so ffmpeg (and other filters / codecs) know the characteristics of the stream.

Now, what you want to do with that HDR stream in ffmepg is up to you and I don't know much use-cases besides trying to tonemap it to SDR... but good luck.

[hydra3333]

Thank you !

Yes, the use case is to encode into avc/sdr for playback on older media player.

At this point I'm unsure whether it is easier/quicker/optimal to convert to sdr in vapoursynth using the (new ish) tonemap plugin, or just pass it on via vspipe into ffmpeg's filters.
Encoding will be by nvenc in ffmpeg, and I'm not sure what conversion (if any) that offers.
I also need to check whether Donald Graft's DGSource offers some sort of conversion.

[dipje]

HDR to SDR requires tonemapping, which is a separate filter and (never) the task of input readers.

For a somewhat working HDR to SDR pipeline in ffmpeg, read https://forum.doom9.org/showthread.php?t=175125

or as a shortcut

-vf zscale=tin=smpte2084:min=bt2020nc:pin=bt2020:rin=tv:t=smpte2084:m=bt2020nc:p=bt2020:r=tv,zscale=t=linear:npl=100,format=gbrpf32le,zscale=p=bt709,tonemap=tonemap=hable:desat=0,zscale=t=bt709:m=bt709:r=tv,format=yuv420p

(but to make sure in the first part you set all the correct characteristics, they can't be autodetected when piping. A 'ffprobe' on your input file might give you all you need).

I have the sneaky suspicion that you don't really understand what HDR video is if you think it's just another input format.

[hydra3333]

Originally Posted by dipje

HDR to SDR requires tonemapping, which is a separate filter and (never) the task of input readers.

For a somewhat working HDR to SDR pipeline in ffmpeg, read https://forum.doom9.org/showthread.php?t=175125

or as a shortcut

-vf zscale=tin=smpte2084:min=bt2020nc:pin=bt2020:rin=tv:t=smpte2084:m=bt2020nc:p=bt2020:r=tv,
zscale=t=linear:npl=100,format=gbrpf32le,zscale=p=bt709,tonemap=tonemap=hable:desat=0,
zscale=t=bt709:m=bt709:r=tv,format=yuv420p

(but to make sure in the first part you set all the correct characteristics, they can't be autodetected when piping. A 'ffprobe' on your input file might give you all you need).

Thank you ! Great link, too. OK ffprobe. I wonder if mediainfo does similar and gives enough. I'll have a look.

Originally Posted by dipje

I have the sneaky suspicion that you don't really understand what HDR video is if you think it's just another input format.

Yes, I am definitely caught short about HDR (and now HDR+ by the looks).
With a little reading was I vaguely aware that (unless I'm wrong about this too) HDR required more bits, eg 10 or even 12, and therefore a video file (eg hevc) did not have "standard 8bit" content like almost everything else, and that HDR->SDR was not a straightforward conversion and needed something called "tone mapping" eg per this vapoursynth plugin https://github.com/ifb/vapoursynth-tonemap which it seems had code ported from ffmpeg.
For other newbies like me, this may help a bit: https://en.wikipedia.org/wiki/High-dynamic-range_video

About DGSource() the manual says:

fulldepth: true/false (default: false)
When fulldepth=true and the encoded video is HEVC 10-bit or 12-bit, then DGSource() delivers 16-bit data to Avisynth with the unused lower bits zeroed.
The reported pixel format is CS_YUV420P16.
If either of the two conditions are not met, then DGSource() delivers 8-bit YV12 or I420 data, as determined by the i420 parameter.
When fulldepth=false and the video is HEVC 10-bit or 12-bit, then the video is dithered down to 8-bit for delivery.
If you need a reduced color space (less than 16 bits) for your high-bit-depth processing, you can use ConvertBits() as needed after your DGSource() call.

I'm not sure what this bit means "When fulldepth=false and the video is HEVC 10-bit or 12-bit, then the video is dithered down to 8-bit for delivery", so I'l need to trot on over to DG's site and do some reading I guess.
From what I have read so far, mpeg4 AVC seems to do 10 and 12 bit too (HDR?), so unless I have misunderstood that as well, then I'll enquire about that as well.

Hmm, I wonder, is there much h.264 10/12 bit around or is it all h.265 (or soon to be av1) ?

[dipje]

There is 10bit h264 for sure, and in the video-editing world it's quite normal actually to use more than 8bit per plane.
But you won't find (much) HDR content in h264. It's possible in theory, but I guess "it is not normal" so its rare.

That DGSource line is basically meant to be read like this: "If the file has more than 8bit per plane, return the real 10bit or 12bit per plane data, or - for compatibility - should I convert it to 8bit per plane".

The extra bits per plane is a separate thing compared to HDR. In the HDR world, it's just normal to put the data in 10bits or more per plane. Some HDR standards use 10bit, some more.

More bits per plane results in more precision in describing the colors. Technically speaking, 8bit goes from 0 to 255, so you have 256 numbers to describe a value. In 10bit, numbers go from 0 to 1023, so you have 1024 numers to describe a value.
So every bit added, doubles the precision.

HDR contains much more information (much more color range, much more light range) so the extra precision is needed to be able to describe all that extra data without loosing details. That's the noobish way to explain it .
Normal video can look OK in 8bit, but HDR has way more information crammed into the data. If you still use 8bits for all the (extra) data, you're loosing precision somewhere. That's why 10bit / 12bit is 'normal' in HDR world.

HDR also contains much more color information like I said. This means the numbers you get in the pixels have a different meaning compared to SDR videos. The information is in a different colorspace is how that's called. That's why if you play back HDR content on a screen that doesn't "understand" HDR, it looks flat and dull. It doesn't interpret the color data in the correct way.

Then the last part, which is actually the most important part . All through time, images and videos in the computer world contained pixels. Talking 8bit (for simplicity), that means you would get (eventually) values between 0 and 255. And every monitor would just use those values as they are. That means '255' is "as bright as the monitor can go", and '0' is "as dark as the monitor can go".
HDR-video tries to decouple the monitor from the video. That means values are not described "as high as you can go", or "80% of your brightness" anymore, but they are described in a 'natural' way. The values describe how bright something should be in absolute luminance values, 'nits'. So even if the output monitor cannot go as bright, the HDR file just describes 'this pixel with the flashlight should be 1400 nits bright'.

It's up to the monitor (or software) to display that pixel "as good as they can". Not a lot of screens can go as bright as 1400 nits for example. So the monitor needs to transform the scene in something that represents the brightness data as good as possible. This is the tonemapping and a crucial part of HDR playback (so this is true, even if you playback HDR content on HDR screens.. since every screen is different, every screen needs to map the HDR data to its possibilities.. With a SDR screen, this is just an extreme transformation).

So basically, a HDR file needs to be converted from its wide colorspace (Rec 2020 for example) to a normal SDR colorspace (Rec 709 for example). In doing this, there needs to be decisions about "what to do with colors that Rec 709 cannot contain". For example, what to do with a pixel that is soooo intens red that Rec709 does not contain that color.

And the same for the brightness. You can assume most SDR screens go to +/- 100 nits bright. So what do you do with information that is 500 nits bright? 1000 nits bright? 5000 nits bright? Decisions, decisions, decisions.
That is tonemapping.

This is also why people say that the 'correct' way to convert HDR to SDR, is to load the HDR content in a HDR-capable video editor, and scene by scene change the colors/brightness so that they look good on SDR.
If that's too much hassle (like it normally is :P) there are tonemapping filters, or recommended tone-curves to try to get the SDR result close).

[hydra3333]

Thank you so much ! That clarifies a lot of things, in simple easy to read terms, for this lay user.