[DoVi_Scripts] Multi-Function Scripts for Dolby Vision processing and a lot more... [Archive] - Page 8

subnrg

31st May 2024, 22:57

Hello @Kuler087.
I want to sincerely thank you for what must have been an immense amount of experience and work to put this together. Although I'm technically not a noob since I've been dabbling with encoding for over a decade, but I've never had the need to get too into the weeds on this stuff. I've been using staxrip forever, but never was able to figure out how to get their DV support to work. I think I've got most everything down now... thanks to you. I've read through this entire DS forum and watched a chunk of your very helpful videos, but I just have a few questions:

1. My goal is to take a remux file (normally P7.6) and make 2 files for each of these. One that is a 1080p 6M VBR DV8.1 EAC3 .mp4 file (that is roku compatible), and one that is 4k 18M VBR DV8.1 original lossless audio in .mkv (mainly used on Shield) for my Plex servers.
So far, I've been able to translate my staxrip encode settings into your .bat files, and created one for HD and one for 4K .mkv. The workflow that seems to work best and most consistently for me is:
- 8.2.1 & 8.1.1
- Mux .hevc .ec3 and subs from original file through mkvtoolnix
- Mux in staxrip to .mp4 (for some reason I haven't had luck w your batch muxer)
Do you think this is the best approach for this, and/or might there be a way to automate some of this?

2. For my HD encodes, I'm always downconverting to 5.1. Does the DDP encoding always convert from the original lossless, or does this default to the core for THD or DTS? Does it employ any dynamic compression or normalization?

3. For subs, one of the best features that I like in handbrake, is to burn in forced subs. Any advice how to make that work with this without tons of extra work?

Kuler087

1st June 2024, 00:01

- Mux in staxrip to .mp4 (for some reason I haven't had luck w your batch muxer)

with the mp4 version, if you rename your .hevc .ec3 .srt files the same(and same path), you can input just the .hevc file and the script workflow 5 will pick the audio and sub. Otherwise, you can just mux every track you want into an MKV and then run it in Workflow 5 which would get you proper track names and forced flags
For the mp4 version, you'll have to test which muxer version works for you (at the line 60 set mp4_version=NEW or OLD)

Do you think this is the best approach for this, and/or might there be a way to automate some of this?
Yes, one day, I should add muxing and audio encoding in the video encoder workflows eventually ... For now, they just output raw hevc files.

Does the DDP encoding always convert from the original lossless, or does this default to the core for THD or DTS?
it always encodes from the original lossless audio and you can set the script to downmix to 5.1 at line 76. The bitrate is controlled at line 78
7.1 ec3 requires the EAE folder (not included in the tools pack) and the MP4 ''NEW'' version has 7.1 encoding disabled.

Does it employ any dynamic compression or normalization?

No normalization/compression.

For subs, one of the best features that I like in handbrake, is to burn in forced subs. Any advice how to make that work with this without tons of extra work?

Nop can't help you with that, and DS will never do that. I hate burned-in subs.

subnrg

1st June 2024, 18:00

Thanks brotha! Appreciate the info. I totally agree with the burned-in subs. The only time I want those is if they are speaking Klingon or something like Handbrake does with "Foreign Audio Scan". Because I'm trying to create Roku/Firestick friendly DV .mp4 files, I can't use the PGS subs in that container and the IDX conversions are a bit wonky. Although I've found with some specific instructions, GPT4o does an okay job of fixing the srt files.

One other question/suggestion... you have an option of "choose if you want to also mux the lossless audio when you convert to DD+." If this option (or separate option) could apply to 8-1-1, it could save me from having to mux the original back into my 4k rips.

wyup

3rd June 2024, 13:08

I'm getting significantly different plot results from different workflows: 6-2 (madVR) and 6-6 (json with ProRes conversion and Resolve detecting scene cuts and hdr10+):

MaxCLL 6-2: 1378 6-6: 915
MaxFALL 6-2: 607 6-6: 238
Scenes: 6-2: 2765 6-6: 2000

I guess their algorythms are different, but which is the most accurate?

Kuler087

3rd June 2024, 13:48

Resolve scopes/waveforms are wrong if you dont downscale your timeline with ''custom cubic'' algo.
also, 6-2 madvr includes the black bars if you don't crop so this will directly affect your maxfall.

https://i.ibb.co/G7zR9HH/resolve-scaling.png

wyup

3rd June 2024, 17:23

Okay thanks, and for the Timeline Output in Master Settings, what should I choose, the video native resolution or also 1920x1080 as the output scaling?
The video file I'm evaluating is already cropped at 3840x 2076 pixels.

Kuler087

3rd June 2024, 17:42

master settings resolution is your input native resolution so 3840x2076

wyup

4th June 2024, 12:41

I re-generated the HDR10+ json in Resolve from Prores cropped video and obtained by 6-6 the new B plot. It is slightly different than the unscaled one:

6-6 unscaled: MaxCLL: 915 MaxFALL: 238
6-6 downscaled: MaxCLL: 901 MaxFALL: 228

Still far from 6-2: MaxCLL: 1378 MaxFALL: 607

What can I do with the 6-2 .measurement file? I played it alongside the original file with madvr filter and MPC-BE and I got random blanking during playback. Is the .measurement video automatically loaded? I can't find a option within madvr filter to import external .measurement file.

Kuler087

4th June 2024, 13:59

HDR10plus or DV will create metadata intended for playback(restrictions/rules in the way they are calculated) while madvr just measures the pixels as they are so differences are expected. So the 1378nits pixels spike could be ignored by the algo(should be visible in the scopes though) but overall, shot-by-shot maxcll is almost identical and the point is that for some reason, resolve does some resampling if you don't downscale. Anyway, static/single maxcll/maxfall values are pretty much meaningless metadata.

I can't find a option within madvr filter to import external .measurement file.
I don't use it and don't know if it works or is useful but the option for external files is there.

https://i.ibb.co/WktqZmv/Capture-d-cran-2024-06-04-081623.png

subnrg

4th June 2024, 20:25

Hey K.
I had everything working well, then all of a sudden I'm getting errors. I deleted and reinstalled everything and started with a fresh .bat file (only changing the output/temp directory and nvenc), but I'm still getting errors that seem like they may be PTS or nvenc issues? The .mkv i mux via mkvtoolnix seems to play fine, but when I try to mux back to MP4 in staxrip (whether lsmash or ffvideosource) i'm getting:

Script Error
FFVideoSource: The index does not match the source file
(P:\Muxes\Awaken (2018) UHD Disc Sample_temp\Awaken (2018) UHD Disc Sample_source.avs, line 5)

MODE.B shows the script was completed, but no file is rendered. I'm thinking something was updated or I messed with something I shouldn't have. Any ideas?

guest

4th June 2024, 20:34

subnrg

4th June 2024, 20:54

Why aren't you using the MP4 Muxer in DS ??

It isn't working for me. It shows that it's completed, but no output. There has to be some driver or setting that's not compatible. Racking my brain trying to figure it out.

Kuler087

4th June 2024, 20:54

when I try to mux back to MP4 in staxrip (whether lsmash or ffvideosource) i'm getting:

Script Error
FFVideoSource: The index does not match the source file
(P:\Muxes\Awaken (2018) UHD Disc Sample_temp\Awaken (2018) UHD Disc Sample_source.avs, line 5)

This usually happens when you try to import a file with the same filename as another file you already imported in staxrip. Solution: Delete temp folder or rename input

MODE.B shows the script was completed, but no file is rendered. I'm thinking something was updated or I messed with something I shouldn't have. Any ideas?

not enough info to help you but it sounds like you're not using the MP4 version of the script. You have to show me your log (with 'echo on' line 2 preferably) like this:
https://justpaste.it/fbvgg

subnrg

4th June 2024, 22:36

This usually happens when you try to import a file with the same filename as another file you already imported in staxrip. Solution: Delete temp folder or rename input

not enough info to help you but it sounds like you're not using the MP4 version of the script. You have to show me your log (with 'echo on' line 2 preferably) like this:
https://justpaste.it/fbvgg

I do always delete the temp folder unless I'm doing 2 different encodes off the same source file to save time. There's something else going on. If I remux in XMedia Recode, it work just fine. Go figure.

And you were correct about using the mp4.bat being the reason I got no output. I was using the .mp4 version before I deleted everything and started over. But I'm still getting either remuxes with no audio or messed up audio like this one:
https://justpaste.it/3ecjd

Kuler087

4th June 2024, 22:50

I dont know, everything looks normal... Try the old mp4 muxer. line 60 set mp4_version=OLD

guest

7th June 2024, 05:27

Hi Kuler,

I'm using 2.0.7 beta, and I am randomly getting this annoying error:-

Workflow 3-1

Parsing XML metadata...ast frame: 0.045 seconds. ETA: 2:15:08.091
Error: The system cannot find the file specified. (os error 2)
File not found - S01E05 4K HDR vol 3.0 (1)_DV.xml
The system cannot find the file specified.
Error: The system cannot find the file specified. (os error 2)
Error: expected value at line 21 column 35
Error: The system cannot find the file specified. (os error 2)
Parsing RPU file...
Error: The system cannot find the file specified. (os error 2)

I just noticed that 2.0.7 (proper) is out, so I will grab that and see what happens.

==========================================

I ran a Profile 5 DV mkv thru MP4 Muxer, and just at the end got this what I think is an irrelevant error :-

====== PROCESSING LOG ======================
Number of written frames = 168973
Number of Errors= 0
Processing finished successfully.
muxing into a DoVi MP4 file. Be patient, it can take a while. 40min+ on slow HDD
[1/2] Parsing track "E:\Dovi_Temp_MP4\tempstemp.folder66\HDR.hevc"...
hevc: ******************** 100%
[2/2] Saving output file(s) "E:\Dovi_Temp_MP4\tempstemp.folder66\temp.mp4"...
written: ******************** 100%
Broken argument specifier or file name missing - check usage with -h
'.Jerry.Zucker' is not recognized as an internal or external command,
operable program or batch file.

Was has Jerry Zucker got to do with anything ??

The movie is Top Secret.

von Suppé

7th June 2024, 08:42

Was has Jerry Zucker got to do with anything ??

The movie is Top Secret.

At its time a commemorably funny movie, created by Jim Abrahams and the Zucker brothers. Dunno how "Jerry.Zucker" is conjured up here though.

guest

7th June 2024, 08:56

At its time a commemorably funny movie, created by Jim Abrahams and the Zucker brothers. Dunno how "Jerry.Zucker" is conjured up here though.

I think it might be the dodgey way it was compiled, only being Profile 5 is not preferrable...hopefully I can re-encode it something better.

Cheers, thanks for the info :)

Kuler087

7th June 2024, 12:01

Nothing changed in 3-1 and the mp4muxer in a while but please remember to always post the complete log and not just parts of it.
A small sample would be nice too.

Was has Jerry Zucker got to do with anything ??

Maybe some of the track names have strange characters (EG Russian text for the tracknames cause this kind of error)

guest

7th June 2024, 14:09

Nothing changed in 3-1 and the mp4muxer in a while but please remember to always post the complete log and not just parts of it.
A small sample would be nice too.

Yeah, not sure why this is happening, I processed the same file again, and there was no error...just one of those things.

Maybe some of the track names have strange characters (EG Russian text for the tracknames cause this kind of error)

This movie hasn't got many tracks, tho.

And this so called DV, is some shitty thing that the colours all messed up, similar to a True Lies "DV" I got some time back, and I ended up downloading a "non DV" version, and this Top Secret has the same issue, not worth "playing" with, so I'm getting it again, "non DV".

Kuler087

7th June 2024, 15:58

the issue with that file is as i suspected, there are special characters in the track name of the commentary track
''Commentary by co-directors Jim Abrahams and David & Jerry Zucker, with co-producers Jon Davison and Hunt Lowry''

the DTSHD track has a delay btw and delays for lossless audio is not supported in the OLD mp4muxer workflow. so you might want to set keep_lossless=NO for this one

https://justpaste.it/g5pvd

guest

8th June 2024, 02:57

I generally run most videos thru MKVToolNix first to remove the stuff I don't want, but I just threw this in to see what happens to the Profile 5, but I acquired a much better conversion, it's actually Profile 8.1, so I will go with that one.

guest

8th June 2024, 04:22

I think I may have asked you this before...

I recall that you're running an Intel 13900KS, but I've forgotten what GPU you use, I'm guessing maybe a 4080, or even a 4090, just for all the CUDA cores !!! :)

You do use CUDA in preference to CPU, even the 13900 ?!?!

I've only got a 3050 & 3060, so even a 4070 Ti Super should process things a LOT faster, correct ??

Would save a lot of time.

Kuler087

8th June 2024, 13:46

3070

I always use my CPU for encoding. GPU encoding is not great.

guest

8th June 2024, 14:04

3070

I always use my CPU for encoding. GPU encoding is not great.

That IS interesting, thanks for that..

I'll have to change mine from CUDA to CPU, then!

It will be interesting to find out which CPU is faster, my 13900KF, or the 7950X.

You've just saved me a lot of money :) or not.

UPDATE:-

So I am doing another 3-1 with CUDA disabled, on the 7950X, and at the start of that CUDA process it shows approx ETA is 4:10:00, but with CUDA enabled (RTX 3060), it would be approx 3:10:00 ??? :(

Then I tried the same scenario on the 13900KF, and at the start of the CUDA process it shows approx ETA is 4:35:00, but with CUDA enabled (RTX 3050), it would be approx 3:30:00 ??? :(

Your RTX 3070 (with 5,888 CUDA cores) would be a little faster again, so my way of thinking is, an RTX 4080 Super (with 10,240 CUDA cores) would be close to 3 times faster than the RTX 3060, (with 3,584 CUDA cores).

In CPU "mode", it's only using 1-2% on both CPU's

So to me, that appears that the 7950X is faster than the 13900KF, maybe similar to your 13900KS

Doing a DS 3-1 on my 5950X with dual GTX 770's, (1532 CUDA cores) unfortunately it only uses 1 GPU, and its S-L-O-W, (ETA 6:00:00) so I kind of have to disagree that CPU is faster than CUDA...

Yosho

10th June 2024, 04:07

Can someone help with this MP4 playback problem on an LG C9 tv? I tried this mp4 script to make a p7 st-dl with GxK uhd but it comes out stuttering. P8 has issues too for some reason, but P7 DT-DL made with it works well though, no problems with playing back. I looked through script, saw OLD and NEW Muxer, tried both, but ST-DL stutters no matter what.. Am I doing something wrong? Modes 4 and 5 is what I used. No luck for st-dl. Made a mkv with makemkv, dropping in bdmv, dropping in mpls, and tried just the m2ts with no luck.
I tried dvvt tool as well but it also came out stuttering the same way unfortunately.

Kuler087

10th June 2024, 11:59

yeah, unfortunately, STDL MP4 does not work very well on 2017-2019 LG TVs. You should get the Ugoos AM6b+ (https://discourse.coreelec.org/t/dolby-vision-for-minix-u22x-j-max-and-ugoos-am6/24273) or you could try the mkv container with Kodi (https://forum.kodi.tv/showthread.php?tid=368381) but I don't know if the EL is used or ignored.

Kuler087

10th June 2024, 12:13

So I am doing another 3-1 with CUDA disabled, on the 7950X, and at the start of that CUDA process it shows approx ETA is 4:10:00, but with CUDA enabled (RTX 3060), it would be approx 3:10:00 ???

Then I tried the same scenario on the 13900KF, and at the start of the CUDA process it shows approx ETA is 4:35:00, but with CUDA enabled (RTX 3050), it would be approx 3:30:00 ???

Your RTX 3070 (with 5,888 CUDA cores) would be a little faster again, so my way of thinking is, an RTX 4080 Super (with 10,240 CUDA cores) would be close to 3 times faster than the RTX 3060, (with 3,584 CUDA cores).

In CPU "mode", it's only using 1-2% on both CPU's

So to me, that appears that the 7950X is faster than the 13900KF, maybe similar to your 13900KS

Doing a DS 3-1 on my 5950X with dual GTX 770's, (1532 CUDA cores) unfortunately it only uses 1 GPU, and its S-L-O-W, (ETA 6:00:00) so I kind of have to disagree that CPU is faster than CUDA...

I was talking about x265/x264 encoding and not about speed but quality.
For 3-1, there's no encoding, just decoding and the GPU is faster for that of course.

guest

10th June 2024, 12:26

I was talking about x265/x264 encoding and not about speed but quality.
For 3-1, there's no encoding, just decoding and the GPU is faster for that of course.

OK, well the majority of my use of DS is 3-1..so the faster I can get that done, the better :)

Each "job" I do takes well over an hour, and that's for a video thats less than a hour long..so long movies will take half the day :(

So I think I'll buy an RTX 4080 Super...

:thanks:

Kuler087

10th June 2024, 13:05

guest

11th June 2024, 11:27

there's a lot of variation in 3-1 because of the prores decoding bug in CM with Windows. Sometimes I can do a 2h movie in just 1h-1h30 and some other times, it can take well over 4h. A 4080 will be a lot faster than a 3060 for sure but some movies will still take forever to process regardless of the GPU.
In Resolve it's always fast.

EDIT: workaround would be to use jpeg-2000 mxf but I'm not able to encode a cm_analyze compatible file with ffmpeg. It only works with the mxf file from Resolve.

Just read your edit...

Have you made a tutorial of this process, or is there one somewhere on how to do it ??

I ordered my Gigabyte RTX 4080 Super today :), 2nd most expensive single PC part I have ever bought :(

Kuler087

11th June 2024, 11:54

Just read your edit...

Have you made a tutorial of this process, or is there one somewhere on how to do it ??

Well I tried a bunch of different settings and CM_analyze just isn't able to decode the ffmpeg produced files.

E:\DoVi_Scripts\tools\ffmpeg.exe -i E:\input.mkv -c:v jpeg2000 -an -profile:v 3 -q 1 -compression_level 30 -pix_fmt yuv422p10le -y E:\output.mxf

E:\DoVi_Scripts\tools\cm_analyze.exe -m 20 --source-format "pq bt2020" --analysis-tuning 1 E:\output.mxf E:\DV.xml

EG, this a Resolve mxf file that works (https://drive.google.com/file/d/1wprNlmCbKX1GCGYfz-YNFfiLSZP-Q9OS/view?usp=sharing)

I ordered my Gigabyte RTX 4080 Super today :), 2nd most expensive single PC part I have ever bought
nice!!

guest

11th June 2024, 12:16

Well I tried a bunch of different settings and CM_analyze just isn't able to decode the ffmpeg produced files.

E:\DoVi_Scripts\tools\ffmpeg.exe -i E:\input.mkv -c:v jpeg2000 -an -profile:v 3 -q 1 -compression_level 30 -pix_fmt yuv422p10le -y E:\output.mxf

E:\DoVi_Scripts\tools\cm_analyze.exe -m 20 --source-format "pq bt2020" --analysis-tuning 1 E:\output.mxf E:\DV.xml

EG, this a Resolve mxf file that works (https://drive.google.com/file/d/1wprNlmCbKX1GCGYfz-YNFfiLSZP-Q9OS/view?usp=sharing)

nice!!

Not really my favourite thing, using command lines, so were these examples of what you've tried, do these work ??

And what do I do with that .mxf file ??

Kuler087

11th June 2024, 12:21

nop, anything I tried in ffmpeg failed to be decoded by cm.
Will try again this weekend...

guest

11th June 2024, 13:01

nop, anything I tried in ffmpeg failed to be decoded by cm.
Will try again this weekend...

But what about that .mxf file ??

Can I do anything with it ??

Kuler087

11th June 2024, 13:14

No, it's just a reference.

Boulder

11th June 2024, 16:49

I tried to have a quick peek to see if this workflow I'm thinking of is there, but not sure so I'll ask. Sorry for being a DoVi n00b, I'm just about to get a proper TV that supports TV Led DoVi so I'm going to create some samples to try it out.

I have a profile 7 STDL MKV file. I was thinking of extracting the BL, FEL and RPU, downscaling and re-encoding the BL video and recombining these three back to MKV. Is the extracting and recombining possible to do with the script? And is it possible to downscale the BL or does the frame size need to match the FEL?

Kuler087

11th June 2024, 17:26

Boulder

11th June 2024, 18:34

It's possible but I wouldn't do that without DEE.exe. Re-encoding the BL and muxing back the original EL can be destructive and cause issues.
Instead, you should bake FEL into the BL and convert the RPU to Profile 8 which will be the same quality as the original FEL P7 playback.

Bake and then re-encode like any normal 4K? (Of course, with proper DoVi related settings.)

I tried to find a way to extract the two layers with the script but didn't see the workflow there? Did I just miss it or is it not there?

Kuler087

11th June 2024, 18:50

Yes, you bake/encode with x265.exe at the same time, it's a one-step process. In dovi_scripts you can create the avs script in 8-2-1 or 7-1

Set echo to ''on'' (line 2) to expose the cmd lines I use and then run 8-2-1. Cancel the encode and go into the temp folder, you will have the demuxed layers + RPU and the AVS script needed to bake/encode.
you can downscale to 1080p at line 120.

::choose if you want 8-2 to downscale to 1080p (default = NO)
set downscale=NO

Then you have workflow 7-1 in manual mode. This will produce an AVS script ready to be encoded (remove the subtitles osd though)

In both ways, my scripts require an nvidia gpu.

Boulder

11th June 2024, 19:42

Thank you for the comprehensive answer, I'll take a deeper look at the process using those workflows :)

Gatorman3385

11th June 2024, 20:08

I'm getting this error when attempting to use the works.mxf. Am I doing something wrong?

I:\Working\Movies\2160>I:\Working\Dolby-Vision-Tools\tools\cm_analyze.exe -m 20 --source-format "pq bt2020" --analysis-tuning 1 I:\Working\Movies\2160\output.mxf I:\Working\Movies\2160\DV.xml
Copyright (c) 2013-2024 Dolby Laboratories, Inc. All Rights Reserved
06/11/2024/15:06:53.060000000 CPU: AMD Ryzen 7 2700X Eight-Core Processor (AuthenticAMD) [ mmx mmxext sse sse2 sse3 ssse3 sse41 sse42 sse4a avx avx2 fma ]
06/11/2024/15:06:53.062000000 cm_analyze: ERROR Unable to open MXF 'I:\Working\Movies\2160\output.mxf' for read: File open failure.

Kuler087

11th June 2024, 20:27

well you have to change the command to:

I:\Working\Dolby-Vision-Tools\tools\cm_analyze.exe -m 20 --source-format "pq bt2020" --analysis-tuning 1 I:\Working\Movies\2160\works.mxf I:\Working\Movies\2160\DV.xml

Gatorman3385

11th June 2024, 21:07

Yup! That would help. :p

Boulder

12th June 2024, 06:03

Yes, you bake/encode with x265.exe at the same time, it's a one-step process. In dovi_scripts you can create the avs script in 8-2-1 or 7-1

Set echo to ''on'' (line 2) to expose the cmd lines I use and then run 8-2-1. Cancel the encode and go into the temp folder, you will have the demuxed layers + RPU and the AVS script needed to bake/encode.
you can downscale to 1080p at line 120.

Then you have workflow 7-1 in manual mode. This will produce an AVS script ready to be encoded (remove the subtitles osd though)

In both ways, my scripts require an nvidia gpu.

One question regarding the process came up. If DoViBaker already uses the RPU, won't the information be processed again during playback as it is input to x265 as well?
EDIT: or is it that if you go DoVi and not static HDR10, you just bake BL+FEL and then input the RPU while encoding to get a proper stream with dynamic metadata.

Kuler087

12th June 2024, 12:05

The original P7 RPU is required when you bake/decode EL+BL to a 16bit RGB PQ stream. Then, that 16bit PQ HDR stream is fed to the x265.exe with the P8 RPU.

Boulder

12th June 2024, 12:14

The original P7 RPU is required when you bake/decode EL+BL to a 16bit RGB PQ stream. Then, that 16bit PQ HDR stream is fed to the x265.exe with the P8 RPU.

Yeah, I was just a little confused when I consulted the DoViBaker examples in GitHub.. in the script there were all three items incorporated in the DoViBaker call, but in this case I would use only BL and EL there and RPU with x265.