Bad quality when hardcode pgs with core.sub.ImageFile

[Br4twurscht]

Hi,

because of issues with displaying some pgs and ass subtitles on my tv (with jellyfin), I want to change my encoding workflow and burn them directly into my encodes. I'm working very much years with StaxRip there it seems, that burn in pgs subs only works with VapourSynth. I also tried SupTitle by myself, but the result was very similar.

The crazy thing is, when I use the preview in StaxRip it look good, but after the encode, the text is not so smooth anymore, which is not nice on my screen, but definitely bad on my OLED TV.

The first picture is saved from inside the preview of StaxRip:
https://forum.doom9.org/attachment.php?attachmentid=18819&stc=1&d=1737669985

The second picture is saved from mpc-hc:
https://forum.doom9.org/attachment.php?attachmentid=18820&stc=1&d=1737670079

This is the resulting vs script:
1: import os, sys
2: import vapoursynth as vs
3: core = vs.core
4:
5: sys.path.append(r"E:\encoding\StaxRip-v2.44.2-x64\Apps\Plugins\VS\Scripts")
6: core.std.LoadPlugin(r"E:\encoding\StaxRip-v2.44.2-x64\Apps\Plugins\VS\SubText\SubText.dll", altsearchpath=True)
7: core.std.LoadPlugin(r"E:\encoding\StaxRip-v2.44.2-x64\Apps\Plugins\Dual\ffms2\ffms2.dll", altsearchpath=True)
8:
9: tcFile = r"E:\encoding\test-movie_HardSubtest_neu\source_[No DV].mkv_temp\source_[No DV]_timestamps.txt" # timestamps file path
10: clip = core.ffms2.Source(r"E:\encoding\test-movie_HardSubtest_neu\source_[No DV].mkv", cachefile=r"E:\encoding\test-movie_HardSubtest_neu\source_[No DV].mkv_temp\temp.ffindex", timecodes=tcFile) if os.path.exists(tcFile) else core.ffms2.Source(r"E:\encoding\test-movie_HardSubtest_neu\source_[No DV].mkv", cachefile=r"E:\encoding\test-movie_HardSubtest_neu\source_[No DV].mkv_temp\temp.ffindex")
11: clip = core.sub.ImageFile(clip, file = r"E:\encoding\test-movie_HardSubtest_neu\source_[No DV].mkv_temp\ID1_[de] {FORCED [DIY]}_default_forced.sup")
12: clip = core.std.Crop(clip, 0, 0, 276, 276)
13: clip = core.resize.Bicubic(clip, 1920, 804)
14: clip = clip[0:801]
15:
16: clip.set_output()


And these are my x265 options (it's an hdr10):
--crf 18 --preset slow --output-depth 10 --ctu 32 --master-display "G(13250,34500)B(7500,3000)R(34000,16000)WP(15635,16450)L(10000000,1)" --hdr10 --colorprim bt2020 --colormatrix bt2020nc --transfer smpte2084 --range limited --max-cll "763,218" --chromaloc 2 --repeat-headers --aud --hrd --selective-sao 2

Do you have an advice for me how to solve this?
If it's in avisynth, I would be happy, too. Currently, I'm still working with avisynth.

Edit: Some additional infos. If I play the original file and let the player show the pgs, then it looks pretty smooth regardless if on the pc with mpc-hc or on the tv with jellyfin.
I tried very much things to solve it by myself, but my knowledge gerading avs and vs is limited. Nevertheless, I tried some filters and also tried to blend it by myself regarding "Example with manual blending" on https://github.com/vapoursynth/subtext/blob/master/docs/subtext.rst

Maybe it's something with the color stuff, because of the 10bit HDR. But if there's a problem, then I'm lost. I'm very happy, that StaxRip does these configurations e.g. in x265 for me.

br
Br4twurscht

[Selur]

Can't really help, since I never tried this, but you might want to upload the images somewhere else (i.e. imgbb.com, mediafire.com, google drive,..) and like to them, since attachments will require approval by a moderator, which can take ages,...)

My guess is the problem is that normally the subtitles are overlayed on the image after HDR is applied and not changed my the HDR data.
In the file you create, you integrate the subtitles into the desaturated version of your source (without HDR information) and then on playback the HDR information is applied to the whole frames which will also alter the colors of the subtitles.
My guess is one would need to add the subtitles to a blank clip, somehow (maybe some LUT) create a desaturated version of the subtitles and then combine them with the desaturated video, so that at playback the HDR info will adjust the color back to what they should look like.
=> best of luck

Cu Selur

[Br4twurscht]

The pictures

Preview in Stax-Rip:
https://i.ibb.co/rQKGFht/Source-No-DV-new-00200-from-preview-of-Stax-Rip.png (https://ibb.co/yNmndCR)

After Encoding:
https://i.ibb.co/zXf6LX8/Source-No-DV-new-vapour-hardpgs-image-File-mkv-snapshot-00-08-118-after-encoding-made-in-mpc-hc.png (https://ibb.co/c6TYp6b)

Thx for your guesses, but can I do this? When you see the images, do you still think it's the way you described?

[Selur]

but can I do this?
No clue what you can and can not do,..... ¯\_(ツ)_/¯
(like I said, I never tried that so no clue whether there is a known method to change this)

hen you see the images, do you still think it's the way you described?
Hard to tell with white subtitles on black background. It seems like a difference in luma which could be caused by the above-mentioned (then the rest of the image would look correct) or just a difference in tv vs pc scale or using some wrong color matrix signaling (in these two cases the difference would be visible on the whole image). If the rest of the image looks correct, then what I described might be the issue.

Cu Selur

[Emulgator]

Aliasing due to resizing after rendering ?
11: clip = core.sub.ImageFile(clip, file = r"E:\encoding\test-movie_HardSubtest_neu\source_[No DV].mkv_temp\ID1_[de] {FORCED [DIY]}_default_forced.sup")
12: clip = core.std.Crop(clip, 0, 0, 276, 276)
13: clip = core.resize.Bicubic(clip, 1920, 804)

[Br4twurscht]

Aliasing due to resizing after rendering ?

Thx, but I think, that will not really help, because see next.


My guess is the problem is that normally the subtitles are overlayed on the image after HDR is applied and not changed my the HDR data.
In the file you create, you integrate the subtitles into the desaturated version of your source (without HDR information) and then on playback the HDR information is applied to the whole frames which will also alter the colors of the subtitles.

I played a little bit with the display options of the mpc-hc regarding hdr an now I'm sure you're right, it's because of the hdr data applied afterwards on the subtitle.

So, firstly I want to understand better your proposal.

My guess is one would need to add the subtitles to a blank clip, somehow (maybe some LUT) create a desaturated version of the subtitles and then combine them with the desaturated video, so that at playback the HDR info will adjust the color back to what they should look like.

I thought, that applying the desaturated version of the subtitles on the desaturated video is exactle what currently happens. And because of the hdr data, the subs are manipulated in the wrong way.

Or did you mean adding the (desaturaded) subs to the hdr saturated video? Is that even possible? And if yes, how?

Thx for your guesses, but can I do this?
That was a typo ;). I wanted to ask (and still want to know :)) how can I do this? Do you have a suggestion for a script?

Br and thx
Br4twurscht

[Selur]

I thought, that applying the desaturated version of the subtitles on the desaturated video is exactle what currently happens. And because of the hdr data, the subs are manipulated in the wrong way.

Or did you mean adding the (desaturaded) subs to the hdr saturated video? Is that even possible? And if yes, how?
atm. the subtitles that get to the desaturead video are the subtitle that normally get layed over the hdr video
My idea, no clue how to do it, would be to use some LUT on the subtitles clip which would do some destaturation which on playback would be nullified by the HDR metadata.

Cu Selur

[Z2697]

Use the matrix, primaries and transfer parameters of the subtitle filter?