[DoVi_Scripts] Multi-Function Scripts for Dolby Vision processing and a lot more...

[Kuler087]

Technically, the only way to achieve what you want to do would be:

1- generate a DV XML in resolve or doviscripts
2- encode the HDR10 to prores 422 (recreate the master)
3- encode FEL or MEL + XML in DEE (dolby encoding engine)
4- mux new BL + EL in tsmuxer


Desired use-case. A combination of content mapping type and enhancement-layer type (fel - full
enhancement-layer, mel - minimal enhancement-layer). Values: no_mapping_with_mel|no_mapping_with_fel
|map_to_1000_nits_with_fel|map_to_1000_nits_with_mel|map_to_600_nits_with_fel.

Dolby Vision to profile 7 encoding workflow.

-h [ --help ] Show this help.
--toolset-dir arg (=D:\DoVi_Scripts\tools\DEE)
Toolset installation directory.
-l [ --license ] arg (=D:\DoVi_Scripts\tools\DEE\license.lic)
Toolset license file.
--print-all arg (=0) Enable log printing. Values: 0|1.
--progress arg (=0) Show progress in percentage. Values: 0|1.
--overwrite arg (=0) Allow overwriting existing files. Values: 0|1.
--temp-dir arg (=D:\DoVi_Scripts\tools\DEE)
Directory to store temporary files.
--keep-temp arg (=0) Keep temporary files after execution. Values: 0|1.
--ffmpeg arg (=ffmpeg) FFmpeg executable.
--dvesverifier arg Dolby Vision ES Verifier executable. If not specified, the script will skip the verification step.
--input-format arg Input format followed by format-specific options. Use "--morehelp input-format" for more details.
Values: jpeg2000_list|jpeg2000_mxf|prores_list|prores_mov|tiff_list.
-i [ --input ] arg Input file with mezzanine video. Input directory for list-based inputs.
-m [ --input-metadata ] arg Optional input metadata file. If not specified, the application attempts to extract metadata from
the input.
-b [ --output-bl ] arg Base-layer output file in HEVC format.
-e [ --output-el ] arg Enhancement-layer output file in HEVC format.
--metadata-offset arg (=0) Offset added to each frame index, when accessing frame metadata from the source.
--start arg (=0s) Start position in 'xs' format, where 'x' represents seconds or 'xf' format, where 'x' is a frame
number.
--duration arg (=-1) Duration in 'xs' format, where 'x' represents seconds or 'xf' format, where 'x' is a frame number.
'-1' means "end of file".
--end arg (=-1) End position in 'xs' format, where 'x' represents seconds or 'xf' format, where 'x' is a frame
number. '-1' means "end of file".
--encoder arg HEVC encoder. Values: x265|beamr|impact.
--preset arg Encoder preset. If not specified, selected encoder's default preset is used. Use "--morehelp preset"
for more details.
--data-stream arg (=0) Use data streaming instead of temporary files, wherever possible. Values: 0|1.
--yuv-buffer-size arg (=-1) YUV buffer size in bytes. '-1' means "assign automatically". Relevant only if 'data-stream' is
enabled.
-p [ --encoder-pass-num ] arg (=2) Number of encoder passes. Values: 1|2.
--use-case arg Desired use-case. A combination of content mapping type and enhancement-layer type (fel - full
enhancement-layer, mel - minimal enhancement-layer). Values: no_mapping_with_mel|no_mapping_with_fel
|map_to_1000_nits_with_fel|map_to_1000_nits_with_mel|map_to_600_nits_with_fel.
--morehelp arg Show more help about selected topic. Values: input-format|preset.

[SeeMoreDigital]

Oh wow...

Thanks for such a wonderfully detailed response. I wish I had the knowledge to understand it. Indeed I have no idea how to see/open-up the files inside the .bin file you uploaded. :o

When people here are adding MEL information to their non Dolby Vision video streams are they all re-encoding the 4K HDR HEVC video stream Apple ProRes 422?

[Kuler087]

When people here are adding MEL information to their non Dolby Vision video streams are they all re-encoding the 4K HDR HEVC video stream Apple ProRes 422?

I don't know anyone that does that but only official dolby encoding engine (DEE) can create profile 7 stream properly. So whatever they are doing, it's wrong.
My guess is they generate DV in resolve or CM_analyze and then transfer all the levels and scene cut to the P7 rpu.

if you want to try:

1- demux your generic MEL P7 rpu with workflow 1 (skip second input and select ''e'' when asked) MEL framecount must be larger than the BL you want to add DV
2- take the generated P8 RPU I gave you and transfer the scene cuts with workflow 2-3 (input the P7 rpu first then the P8 rpu then select ''s'')
3- do the same as step 2 but this time transfer Level 1-2-3-5-8-9 (input the P7 rpu that has new scene cuts first then the P8 rpu then select ''123589'')
4- inject the new P7 rpu that contains the generated levels and scene cuts into the MEL hevc
5- mux the BL hevc with the new MEL hevc from step 4 to ST-DL with workflow 4-1, I think dovi_tool will trim the EL to match the BL but I'm not sure
6- demux the layers with workflow 2
7- mux the layers from step 6 in tsmuxer.

but again, even if this may (or not)works, the player layer's decoding will be wrong, similar to these (https://docs.google.com/spreadsheets/d/15i0a84uiBtWiHZ5CXZZ7wygLFXwYOd84/edit?gid=671480574#gid=671480574) and all of this is useless and can be done properly by simply injecting the generated P8 rpu into your HDR10 rip with workflow 1

[SeeMoreDigital]

I don't know anyone that does that but only official dolby encoding engine (DEE) can create profile 7 stream properly. So whatever they are doing, it's wrong.

My apologies I meant to say is: When people here are adding MEL information their non Dolby Vision video streams in order to create 'profile 8' .m2ts, .mp4 or .mkv files, are they all re-encoding the 4K HDR HEVC video stream Apple ProRes 422?

[Kuler087]

The P7 MEL to P8 conversion is lossless and doesn't require any re-encoding (workflow 4-1).
Works with FEL too, but you have to make sure the EL doesn't change the BL brightness. It's not rare to see a 1000nits HDR10 base layer expanded to 4000nits with FEL.

[Ts9001]

I've tried to go through this thread and tried to understand some of the details in relation to DV and what is possible and what is not. However I'm still struggling to understand if it's possible to have a P7 FEL REMUX mkv file that is then encoded to a smaller file size and then somehow injecting the P7 FEL DV into this file (not converting and baking to P8)?

So in nutshell this would happen:

Start: Movie_remux.mkv (70GB, P7 FEL)
End: Movie_encoded.mkv (15GB, P7 FEL)

Then if the only way to "preserve" P7 FEL is to convert and bake to P8, does it really require nvidia GPU as referred to in the YT tutorial or can it be done also using CPU encoding?

[Kuler087]

Yes, the only way is to bake FEL to P8/HDR10.
Nvidia GPU is no longer required, but there's a bug in the latest stable for non-nvidia PCs, so get the beta version I posted (https://forum.doom9.org/showpost.php?p=2015467&postcount=1011) a few days ago.


can it be done also using CPU encoding?
Even with an Nvidia GPU, the encoding is done by the CPU with x265.exe unless you opt to use NVenc.

[Ts9001]

Yes, the only way is to bake FEL to P8/HDR10.
Nvidia GPU is no longer required, but there's a bug in the latest stable for non-nvidia PCs, so get the beta version I posted (https://forum.doom9.org/showpost.php?p=2015467&postcount=1011) a few days ago.

Even with an Nvidia GPU, the encoding is done by the CPU with x265.exe unless you opt to use NVenc.

Thanks for your quick answer.

I actually have a 3060ti. Does it qualify for "Nvidia GPU" in this case? I somehow mixed Nvidia and NVenc in my head and thought that my GPU doesn't qualify for Nvenc encoding. So am I good to go with the latest stable version if I want to do some baking?

If may ask what is the GPU needed for in this process?

[Kuler087]

The GPU is only used for decoding the BL + FEL (dgindex). The script will detect you have an Nvidia GPU, and everything will work fine with the latest stable.
If you prefer not to use your GPU for decoding, you can force ffms2 at line 249: set force_ffms2=YES
Either way, the quality will be the same and x265.exe will be used unless you change line 148 to NVENC.

:: In 8-2, choose the video encoder you want to use: x265.exe or x264.exe or NVenc gpu (choices are: NVENC or X265 or X264) (default= X265) NVenc is faster but much worse quality. X264 can't do HDR.
set Encoder=X265

[Kontrarian]

@Kuler087
Any thoughts on the differences with the ms.weight that I showed in post #1046 (https://forum.doom9.org/showpost.php?p=2015703&postcount=1046)?

[Kuler087]

Sorry, somehow I didn't see your post until now.

It is interesting and strange to see that some streaming services don't have the same metadata. I'm guessing the one missing ms.weight is the original/older grade, and maybe they didn't like the look so the colorist/studio made some changes later to the mezzanine file.
ms.weight affects highlight details. Values under 2048 soften and values over sharpen, but in all the tests I did on many devices, this metadata is ignored so it doesn't matter for DV playback. (https://drive.google.com/file/d/1-UWofUytes79a9TxMVVXu_efEsf_Ta8o/view?usp=drive_link). If you delivered the 100nits trim, it probably would make a difference.

ps you should remove pictures that show pirated content filename. This forum has rules about that...

https://i.ibb.co/wNF0Psdn/Resolve-olp-WEMaf4-E.gif

[Gatorman3385]

I have a BAKED P7 to P8 encode where I used the original FEL RPU. Can I replace the RPU with a transferred cmv4.0 RPU, or do I need to to re-encode with DoVi_Baker and the new RPU?

[Kuler087]

Yes, you can restore cmv4.0 without having to re-encode.

[Kontrarian]

Sorry about the pictures, I fixed them.

So the RPU with the ms.weight just sitting flat at 0 would in theory make the entire movie look softer if the metadata wasn't being ignored.
Highly unlikely that was their intent.
There's probably something wrong with that metadata and your RPU is probably correct.

Why do you think devices are ignoring this metadata?

[Kuler087]

make the entire movie look softer if the metadata wasn't being ignored.
It would soften only the highlights and only when they are compressed to fit the TV target brightness.
When the colorist uses ms.weight, it's almost always a negative value, and low values like that (0-512) are common. Sometimes Dolby keeps too many details when it tone maps, and it can look a bit unnatural, similar to madVR's highlight recovery algo.

I have no idea why all the devices (TV internal media player included) ignore this metadata. It might be only for the 100, 600, 1000nits trims delivery.

[Ts9001]

The GPU is only used for decoding the BL + FEL (dgindex). The script will detect you have an Nvidia GPU, and everything will work fine with the latest stable.
If you prefer not to use your GPU for decoding, you can force ffms2 at line 249: set force_ffms2=YES
Either way, the quality will be the same and x265.exe will be used unless you change line 148 to NVENC.

:: In 8-2, choose the video encoder you want to use: x265.exe or x264.exe or NVenc gpu (choices are: NVENC or X265 or X264) (default= X265) NVenc is faster but much worse quality. X264 can't do HDR.
set Encoder=X265

I tried to run 8-2-1 and unofortunately in the muxing process there was an error as seen below. I only changed the speed of the encoding and CRF in the .bat file. Any idea what could be the reason for this error?

The file 'D:\DoVi.Scripts\Movie_DV.mkv' has been opened for writing.
'D:\DoVi.Scripts\Movie_DV-FEL_to_DVP8-HDR10.hevc' track 0: Extracted the aspect ratio information from the video bitstream and set the display dimensions to 3840/2160.
Error: Could not write to the destination file: 112 (L)
Deleting TEMP folder...

[Kuler087]

mkvtoolnix error 112 means ''not enough space on HDD'' I think.
The encoded hevc file should still be in the output path?

[Ts9001]

mkvtoolnix error 112 means ''not enough space on HDD'' I think.
The encoded hevc file should still be in the output path?

Damn, I didn’t pay attention to available space on my
drive. Need to do some clean up.

I couldn’t find any other files than Movie_DV.mkv whose size was only 400MiB. But need to recheck once I’m back at my PC.

Before starting a new process I would have questions regarding audio tracks and subs:

Usually I want to preserve Dolby Atmos tracks if available and convert e.g. DTS-HD MA 5.1 to DDP+ 5.1. And for subs I want to preserve only one language. I noticed in the .bat file something about these topics (e.g. for audio to preserve only the main track or all) but these didn’t answer to my case.

Am I able to re-encode my encoded file with video pass through and only touching audio and subs without having an effect to DV?

What would be the easiest way to handle audio tracks and subs in relation to 8-2-1?

[Kuler087]

What would be the easiest way to handle audio tracks and subs in relation to 8-2-1?

::choose if you want to mux all the audio from the input or just the main audio track. This is valid only if "set MUX=YES" (YES or NO default = YES )
set mux_all_audio=NO
::choose if you want to mux all the subtitles(up to 30) from the input or just forced/full. This is valid only if "set MUX=YES" (YES or NO default = YES )
set mux_all_sub=NO
:: you can disable ddp encoding for the all the workflows (except 8-1) (YES or NO default = NO). (except 8-1)
set encode_DDP=YES

At the end of all the settings, you can select the language you want.

Am I able to re-encode my encoded file with video pass through and only touching audio and subs without having an effect to DV?
8-1-1 can encode the audio only but it doesn't mux. So you just have to mux the tracks you want manually with mkvtoolnix

[ac777103]

interested in opinions on a PC spec I'm working on, I mainly run 6-2 and 3-1. Considering 8GB RTX 4060 (should I go Ti?) and something like an Intel Ultra 5 10-core, with two M2 drives and perhaps 16GB of memory. You may not believe it, but I am currently using an Intel NUC, so I'm assuming there is huge potential for performance improvement with a new build. I'm not into gaming, wondered if I might drop to RTX 3050 or would that be false economy? Welcome suggestions on anything I should be looking at to get good performance for these workflows, thanks.

[Kuler087]

The biggest performance improvement with dovi_scripts will come from the NVME pci-e drives. On my PC I have 3 m2 drives: 1TB for Windows ( I leave it empty with only apps on it), 2TB for work/output path and another 1TB dedicated only for the TEMP folder. This allows me to inject or process a movie RPU + muxing in less than 10min. I recommend WD black gen4 pci-e (https://www.amazon.com/WD_BLACK-SN850X-Internal-Gaming-Heatsink/dp/B0B7CKZGN6?pd_rd_w=EsZ8i&content-id=amzn1.sym.cd152278-debd-42b9-91b9-6f271389fda7&pf_rd_p=cd152278-debd-42b9-91b9-6f271389fda7&pf_rd_r=3EV3CE65N3MRC671P5NB&pd_rd_wg=Iv6yd&pd_rd_r=b1f7101e-4246-4414-8190-d10c02830a75&pd_rd_i=B0D9WTKV1B&th=1) I've had overheating and throttling speed issues with Samsung nvme drives

RTX4060 is good and 6-2 will be faster for sure(probably the same speed as my 3070). It takes about 10 minutes for a 2h movie on mine.
For 3-1, I don't know because of the CM_analyze prores slow performance bug on Windows system. In Resolve, the 4060 will perform well.
Also, I'm not sure of the cause, but CM_analyze(3-1) always crashes on my Windows 11 PC. No issue with my old Win10 PC.

If you multi-task while encoding or using resolve, you might want to increase RAM to 32gb

https://i.ibb.co/fV5g2j3m/explorer-s-ZCue0-Lksv.png

[ac777103]

thanks, lots of good advice there.

[ac777103]

forgot to ask, can Resolve be integrated into 3-1 automatically, or would I need to learn how to use Resolve (assuming I pay for a licence).

[Kuler087]

Yes, I think Resolve can be scripted, and a while ago, someone told me(PM) that he wrote a script for that. I forgot who he is. If you read this, can you share your script?

Resolve is easy to use with my tutorial, but indeed, it is not free.

[coopzr]

https://i.ibb.co/Dgrrr5R3/2025-03-0711-47-34-ezgif-com-optimize.gif
What causes these artifacts outside of the active frame & does it have anything to do with DV? I see this often when measuring letterbox.

[Kuler087]

In 6-1, the contrast and black level are altered on purpose for easier cropping.
remove ''Levels(0, 2.5, 255, 0, 255, coring=false)'' and you will notice how harder it is to see the letterbox in darker movies.

[Kontrarian]

What are the "<file_name>.mkv.measurements" and "<file_name>_Generated.bin" we get after doing 6-2 on an HDR file?
Is there anything we can do with them?
Should we keep them?

[Ts9001]

mkvtoolnix error 112 means ''not enough space on HDD'' I think.
The encoded hevc file should still be in the output path?

I created some more space on my drive and managed to complete 8-2-1. I ended up having Movie_DV.mkv file. Is there any way to verify that it's now actually "P8_baked" instead of just P7 FEL converted into P8? Or do I just trust that this is how 8-2-1 works for P7 FEL that it automatically does the baking?

Btw, what is the workflow in the current version of Dovi_script to verify the DoVi in general since I cannot find similar workflow as shown in the YT tutorial (2-2)?

[Kuler087]

What are the "<file_name>.mkv.measurements" and "<file_name>_Generated.bin" we get after doing 6-2 on an HDR file?
Is there anything we can do with them?
Should we keep them?

measurements file can be opened in tools/madMeasurementAnalyzer.exe for even more info about the input brightness.

The bin file is just that measurement file converted to DV rpu. you can delete.

[Kuler087]

I created some more space on my drive and managed to complete 8-2-1. I ended up having Movie_DV.mkv file. Is there any way to verify that it's now actually "P8_baked" instead of just P7 FEL converted into P8? Or do I just trust that this is how 8-2-1 works for P7 FEL that it automatically does the baking?



If the movie FEL changes the BL brightness, you can measure the encode in 6-2 and compare it to the original, example (https://slow.pics/c/3VTRACWK). You can also do screenshot comparisons in 7-2. (https://docs.google.com/spreadsheets/d/15i0a84uiBtWiHZ5CXZZ7wygLFXwYOd84/edit?gid=1226038728#gid=1226038728)

If the filename says P7FEL to P8HDR10, you can be sure that fel was baked.

Btw, what is the workflow in the current version of Dovi_script to verify the DoVi in general since I cannot find similar workflow as shown in the YT tutorial (2-2)?
are you referring to the sync verification? the workflow is 2-1

[ac777103]

measurements file can be opened in tools/madMeasurementAnalyzer.exe for even more info about the input brightness.

The bin file is just that measurement file converted to DV rpu. you can delete.

I've found that if I take the Generated bin file from 6-2 and copy to the directory with my source mkv, and match the filename, then 3-1 skips the madvr step, which otherwise would be repeated in a 6-2 followed by 3-1 workflow. Are there any downsides, eg if I don't exclude the letterbox in 6-2 measurements, does that mess up the output from 3-1 in some way, eg corrupting the MaxFALL in the final dv8 file?

[Kuler087]

yep, this is exactly why 6-2 keeps the madvr generated RPU. Excluding the letterbox doesn't matter because only the scene cuts are used.
Keep in mind that madvr scene cut detection is not as good as Resolve. Madvr is lot more sensitive to flashes or fast movement scenes and sometimes it will miss obvious scene cuts

[ac777103]

thanks. If I get myself a new PC, sounds like getting Resolve would be a very useful add-on.

[Kuler087]

yeah, Resolve is a wonderful software. I don't know what I would do without it.

[ac777103]

I see Resolve is available for macOS. If I can go from HDR10 mkv to dv8 mkv using macOS + Resolve I'd rather use a compact mac desktop than get a tower case PC. Any advice on what a mac-based workflow would look like, as alternative to 6-2 then 3-1 ?

[Kuler087]

I think Apple products are garbage, so no, I can't help you with that sorry.

[ac777103]

setting aside the merits or lack thereof of Apple hardware, perhaps we could just consider what additional software is needed apart from Resolve to achieve the same end product as a 6-2/3-1 workflow. Welcome comments from anyone who may have looked into this.

EDIT: I see CM Analyze is also available for macOS, so a 3-1 for macOS feels tantalisingly close if the madVR dependency can be solved.

[Kontrarian]

I believe @Kuler087 mentioned that someone once made it so the script uses Resolve instead of madVR.
If you could track down who that was then that might be the answer you need.

[Kuler087]

I believe @Kuler087 mentioned that someone once made it so the script uses Resolve instead of madVR.
If you could track down who that was then that might be the answer you need.

I emptied my mailbox recently but If I remember correctly the user that wrote a script for Resolve was: @Black Ops (https://forum.doom9.org/member.php?u=225092)

[Kontrarian]

Deleted

[ac777103]

just trying to install on a different PC and I cannot open the tools pack 7z file no matter how carefully I type the password. Has it changed?

[Kuler087]

The password is case sensitive: DoVi.Scripts
If you still can't extract it, download it from the MEGA link. IIRC, there's only a password for the GoogleDrive link because, for some reason, it flags one of the tools as a virus (false positive of course).

[Kontrarian]

I just downloaded the package from github and tested it with 7zip using the password "DoVi.Scripts" and everything was fine.

[ac777103]

apologies, I had a "senior moment" there, missed the upper case V. doh!

[Kontrarian]

@Kuler087
If I want to do 3-1 but I don't want to use madVR's shot detection can I get the shots from Resolve before hand and then some how have 3-1 use that by matching the file name or something like ac777103 says he kind of does with 6-2 first in this post (https://forum.doom9.org/showpost.php?p=2015877&postcount=1081)?
But instead of getting the shots from 6-2 we get them from Resolve.

[Kuler087]

Yes, but there's no point doing that. Just do the DV analysis/generation in Resolve; it will be much faster than 3-1.

[ac777103]

in the 3-1 workflow, can the step that creates the prores file be set to use the nvidia gpu? Just trying this out on a laptop I forgot I had (to my surprise it had an unused nvme slot which is now my new D: drive). The laptop has a GTX 1060 GPU, and 6-2 was much faster than I'm used to, but the prores conversion is still slow.

[Kuler087]

The GPU doesnt make the prores encoding faster when the input is a frames indexed avisynth script (3-1).

What you want to do is possible with DS but requires more manual actions:

1- Disable indexing (line 251 set disable_indexing=YES)
2- Encode to prores with workflow 8-2-3. By default, it uses qscale 1 (lossless), to speed things up, you may want to compress more (line 168 -qscale:v 6 )
3- Process the prores file in 3-1 ( you may have to add this in the filename if madvr fails to measure it: transfer=HDR.matrix=2020)

[ac777103]

thanks, so an alternative to my current 6-2 then 3-1 would be

8-2-3 to encode to prores with GPU support (once settings updated per your advice)
rename the prores file per your suggestion
6-2 on the prores (would this be faster / slower than 6-2 on the mkv, or have different accuracy?)
rename the _generated.bin file from 6-2 to match the prores file
3-1 on the prores file which should jump straight to CM Analyze as madVR measured in 6-2 and the prores file has already been created.

I might have read somewhere that 3-1 uses qscale=4? Just want to make the comparison vs current workflow fair.

[Kuler087]

Well you dont have to use 6-2. You can input the prores file in 3-1 and madvr will generate the scene cuts the same way 6-2 does.

If you absolutely want to use 6-2 before 3-1, you can do this:

1- Enable prores for 6-2: line 239 set crop.prores62=YES
2- Set the script to keep prores: line 144 set keep_prores=YES
3- Run 6-2 and opt to crop but don't enter any value: press enter to skip (default=0)

This will convert to prores with GPU acceleration(qscale 8 though but should be fine), measure HDR, and create the RPU for the scene cut
Rename the rpu the same as the prores file (with transfer=HDR.matrix=2020) and run 3-1

I might have read somewhere that 3-1 uses qscale=4?
yes, but it can be changed at line 224: set qscale=4