Color space conversion slowing down encode-times

[Thaddäus]

Hey everyone,

I've been doing some restoration of old film scans and I've always used Premiere for color-correction etc. and then AviSynth for the final cropping, resizing and a bit of dirt-cleanup in the final encode with x264

For the Premiere-export I've been using Lagarith for quite some time, but ultimately moved to ProRes4444XQ instead lately.

The ProRes has YUV color space and the H.264-encode is supposed to be Blu-ray-compliant, so I need YV12 420.

I've always just used the simple

ConvertBits(8)
ConvertToYV12(matrix="Rec709")

in AviSynth, but I recognized there are some weird artifacts in popping red areas. So I've been doing some research here and stumbled across the Dither-method which gives me much better results.

So instead of the lines above I'm now using:

ConvertBits(8, dither=-1)
ConvertToRGB()
Dither_convert_rgb_to_yuv(lsb=true,output="yv12",chromak="spline64")
ditherpost(mode=6)

This is before crop and resize (just like I had the simple color space conversion before), so it's applied to the full resolution of the ProRes which is about 4K. It looks much better and there are no noticable artifacts left.

Now I wanted to do an encode and even the first pass which had around 5-6 FPS before is now crashing down to around 3,7 FPS. I assume the second pass would be even slower since that's my experience from all the previous encodes which were running at a little above 4 FPS for the second pass.

So I cancelled the encode again and swapped the conversion-part after Crop/Resize, so it only has to be applied to the 1080p-resolution instead of the entire source. This already helped a lot, the first pass is now running at around 5-6 FPS again. But I was wondering if there was any way to “streamline” the command for the Dither-color space conversion in order to reduce encoding times.

I'm looking forward to hearing your thoughts about it!

[hello_hello]

If you're using Dithertools, try this.

ConvertBits(8)
ConvertToYV12()
Gradfun3()

The idea is to convert to 16 bit, apply debanding, and then dither back to 8 bit. GradFun3 does all that, and from memory it has much the same options as DitherPost. I'm not sure converting to RGB and back first would improve the quality (and I assume your method only dithers without applying de-banding anyway).

DitherTools converts to stacked 16 bit and back, which is probably slower than Avisynth+'s native 16 bit.

Dogway has two versions of Gradfun3 here (https://github.com/Dogway/Avisynth-Scripts/tree/master). One function under EX Mods and another version under MIX Mods. They have different dependencies (listed at the top of the scripts).
https://forum.doom9.org/showthread.php?t=182881

Gradfun3Plus only does the debanding though, so you need to convert to 16 bit and dither back to 8 bit like this:

ConvertBits(16)
ConvertToYUV420()
GradFun3Plus()
ConvertBits(8, Dither=0)

Or use Dither=1 if you prefer.
http://avisynth.nl/index.php/ConvertBits

I assume you're using Avisynth+ and adding Prefetch to the end of the script to enable multi-threading?
https://avisynthplus.readthedocs.io/en/latest/avisynthdoc/syntax/syntax_internal_functions_multithreading_new.html

Edit: De-banding and dithering should generally be the final filtering in the script.

[Thaddäus]

Thanks for your suggestions!
Unfortunately GradFun also seems to create those artifacts the same way a simple ConvertToYV12 does. Here are some examples where you can see it pretty obviously in the red apple:

Original:
Just added Crop and Resize to get the same framing

https://i.postimg.cc/d7Cn50q8/Original.jpg (https://postimg.cc/d7Cn50q8)


ConvertToYV12:
ConvertBits(8)
ConvertToYV12(matrix="Rec709")

https://i.postimg.cc/nsmT3LHY/Convert-To-YV12.jpg (https://postimg.cc/nsmT3LHY)


GradFun3:
ConvertBits(8)
ConvertToYV12()
Gradfun3()

https://i.postimg.cc/DWG5xz2c/Grad-Fun3.jpg (https://postimg.cc/DWG5xz2c)


GradFun3Plus:
ConvertBits(16)
ConvertToYUV420()
GradFun3Plus()
ConvertBits(8, Dither=0)

https://i.postimg.cc/jWfvMjsX/Grad-Fun3Plus.jpg (https://postimg.cc/jWfvMjsX)


Dither:
ConvertBits(8, dither=-1)
ConvertToRGB()
Dither_convert_rgb_to_yuv(lsb=true,output="yv12",chromak="spline64")
ditherpost(mode=6)

https://i.postimg.cc/NyXbd0gk/Dither.jpg (https://postimg.cc/NyXbd0gk)

It's still not 100% gone with the dither-method, but I assume you won't be able to find a solution that does it perfectly without any artifacts. However, they're definitely way less than in the other three methods.

[tormento]

Try to use, just at the end of the video:

z_ConvertFormat(resample_filter="spline64",dither_type="error_diffusion",pixel_type="yuv420p16")
libplacebo_Deband(radius=12, iterations=4, temporal=false, planes=[3,3,3])
fmtc_bitdepth (bits=8,dmode=8)

and, perhaps, use hevc in 10 bit to encode.

[coolgit]

Try using neo_f3kdb(preset="medium", output_depth=16) or neo_f3kdb(preset="low", output_depth=16). I do preset low first to see if it is sufficient before trying medium.
crop and resize here or before neo_f3kdb
ConvertBits(10, dither=1)

encode using hevc in 10 bit. Don't do 8 bit x264 as all that debanding work will be thrown away and banding comes back after encode.
Medium preset, --crf 18 --output-depth 10 --limit-refs 3 --no-strong-intra-smoothing --bframes 8 --aq-mode 3 --aq-bias-strength 1.1 --range limited --colorprim bt709 --transfer bt709 --colormatrix bt709 --pools 12 --frame-threads 2 gets the job done for me almost every time.

[wonkey_monkey]

ConvertToYV12:
ConvertBits(8)
ConvertToYV12(matrix="Rec709")



Shouldn't those be the other way round?

What bit depth is the original?

How about:

ConvertToYUV420(matrix = "rec709", chromaresample = "bilinear")
ConvertBits(8) # you can try adding dither = 0 or dither = 1 here too

[Thaddäus]

Try to use, just at the end of the video:

z_ConvertFormat(resample_filter="spline64",dither_type="error_diffusion",pixel_type="yuv420p16")
libplacebo_Deband(radius=12, iterations=4, temporal=false, planes=[3,3,3])
fmtc_bitdepth (bits=8,dmode=8)

and, perhaps, use hevc in 10 bit to encode.

This freezes my computer. :D
I have to add it's not the newest setup (Core i7-6700K CPU, 32 GB RAM, GeForce GTX970), but still did the job for me so far.

Try using neo_f3kdb(preset="medium", output_depth=16) or neo_f3kdb(preset="low", output_depth=16). I do preset low first to see if it is sufficient before trying medium.
crop and resize here or before neo_f3kdb
ConvertBits(10, dither=1)

encode using hevc in 10 bit. Don't do 8 bit x264 as all that debanding work will be thrown away and banding comes back after encode.
Medium preset, --crf 18 --output-depth 10 --limit-refs 3 --no-strong-intra-smoothing --bframes 8 --aq-mode 3 --aq-bias-strength 1.1 --range limited --colorprim bt709 --transfer bt709 --colormatrix bt709 --pools 12 --frame-threads 2 gets the job done for me almost every time.

But in this case I'm not getting YV12 as I want. I also want to stick to a Blu-ray-compliant output for compatibility-reasons.

Shouldn't those be the other way round?

How about:

ConvertToYUV420(matrix = "rec709", chromaresample = "bilinear")
ConvertBits(8) # you can try adding dither = 0 or dither = 1 here too


Doesn't make any difference when it comes to those artifacts in the reds.

Is there any way to get around the RGB-conversion in the dither-method to get a YV12-output? Maybe this helps to cut down the encode-times too.

Prefetch(4) definitely helped to make it faster - the first pass is going through at around 7,5 FPS, but the second crashes down to around 3,5 FPS again.

[wonkey_monkey]

Well I don't get it. I downloaded your Original jpeg, opened it with ImageSource, then plain ConvertToYV12, and the result is pretty much indistinguishable. Your ConvertToYV12 image looks like it's been sharpened.

Can you share an un-reencoded snippet of your source file?

[Thaddäus]

It doesn't just look like that, it has been sharpened - there's also a LSFmod() at the very end. ;)

[wonkey_monkey]

It doesn't just look like that, it has been sharpened - there's also a LSFmod() at the very end. ;)

Well then that makes it difficult to know what artefacts you're trying to fix.

[Thaddäus]

The sharpening is not the problem - the artifacts in the reds are also there if I leave the sharpening and everything else out. I narrowed the problem down to the YV12-conversion, so I just need to find a way to properly convert it (best results so far seem to come from the Dither-method), but hopefully without the encoder-times going up a lot.

Here's the shot I've been using for the screenshots above: https://www.swisstransfer.com/d/893fb380-8984-4aa1-aae8-2ce5ccf78bce
To get the framing I have you can use Crop(592,292,2232,1632) and resize it to 1480x1080. The frame from my screenshot is #74 in that sample.

[wonkey_monkey]

It's the conversion to 4:2:0 that's introducing the artefacts, even without converting to 8-bit. They similarly appear if you just blur chroma, rather than downsampling it.

So I think it's just that chroma and luma are so tightly correlated in producing the correct colours, that blurring/downsampling chroma removes the correlation and leads to the output colours straying.

Or to put it another way, the particular chroma values are hiding variations in luma that would otherwise be visible (try extracty to see luma by itself; it's a lost noisier)

[Thaddäus]

OK, so the problem is coming from the ProRes4444XQ-source?
I just tried to use Lagarith again and the simple

ConvertBits(8)
ConvertToYV12(matrix="Rec709")

again and also seems to help against those artifacts to make them less obvious:

https://i.postimg.cc/S26Y3sNK/Lagarith.jpg (https://postimg.cc/S26Y3sNK)

So the solution is probably to go away from the intermediate ProRes4444XQ that comes out of Premiere, back to a Lagarith AVI.

[wonkey_monkey]

I would guess then that your Lagarith AVI is RGB. If the original scans were RGB, and it was processed in Premiere in RGB, then that's for the best. I hadn't realised 4:4:4 to 4:2:0 could be so destructive.

There's possibly also some RGB clipping going on somewhere. If you convert the ProRes to 32-bit float RGB, some of the G and B values in the apple are negative.

[wonkey_monkey]

I think the "artefacts" are how the film actually looks. They're being suppressed in the conversion from 4:4:4 to RGB for display because of the negative values being clipped. Downsampling chroma just makes them visible in RGB again.

[hello_hello]

At first I thought there was a problem with Avisynth's conversion to YV12, so I tried downscaling the chroma myself. After opening the sample in Avisynth I played around with the following script and the same "artifacts" were present when looking at the script output (I bumped up the saturation a tad to make it easier to see).

CropResize(0,1080, 592,292,2232,1632)
RemoveSpots()
Tweak(Sat=1.1)
LSFMod()
A = last
B = A.BilinearResize(width()/2, height()/2)
CombinePlanes(A.ConvertToY(), B, planes="YUV", pixel_type="YUV420P12")
ConvertBits(8)
Prefetch(8)

As Thaddäus experienced, converting to RGB first seemed to fix it. I'm not sure I understand why though, given the source has full chroma. Maybe the conversion from YUV to RGB and back is altering the colors enough to hide the problem, so to speak.

CropResize(0,1080, 592,292,2232,1632)
RemoveSpots()
Tweak(Sat=1.1)
LSFMod()
ConvertToPlanarRGB()
ConvertToYUV420()
ConvertBits(8)
Prefetch(8)

Thaddäus,
based on the cropping you posted, the resize dimensions should be 1476x1080 rather than 1440x1080, unless you're wanting to squish the picture a little.

[Thaddäus]

Sorry, of course I meant 1480x1080, it's 1.37:1 ratio. 1440 would be 1.33:1.

I also tried converting the ProRes to RGB in AviSynth first before converting to YV12, like it also had to be done in the Dither-method. However, it seems to give me slightly more of those artifacts than the RGB-Lagarith-AVI converted straight to YV12. But you really have to look close to recognize the difference, I guess nobody would be able to notice this at 24 FPS. ;)

BTW: Does it make any difference whether you're using ConvertToYUV420() or ConvertToYV12()? To be honest, I couldn't see any difference in the results.

[hello_hello]

BTW: Does it make any difference whether you're using ConvertToYUV420() or ConvertToYV12()? To be honest, I couldn't see any difference in the results.

ConvertToYUV420() will convert a clip of any bit depth (the bit depth isn't changed).
ConvertToYV12() will throw an error if the source isn't 8 bit.
For an 8 bit source the result should be the same either way.

[wonkey_monkey]

As Thaddäus experienced, converting to RGB first seemed to fix it. I'm not sure I understand why though, given the source has full chroma.

It's not fixing anything. It's breaking it, but in a way that coincidentally looks nicer.

[Z2697]

Don't do debanding uless you have to, like in their examples libplacebo_Deband and neo_f3kdb
If the problem is dithering then this will do


z_ConvertFormat(resample_filter="spline64",dither_type="ordered",pixel_type="YV12")


Stick with ordered dithering when you are going to do lossy compression

[Thaddäus]

Just tried it out and it also gives me those artifacts in the reds with the ProRes-file:

https://i.postimg.cc/3kKhdfXg/z-Convertformat.jpg (https://postimg.cc/3kKhdfXg)

[wonkey_monkey]

Not artefacts. That's just how the video is. There's nothing to fix, but your operations are making it coincidentally look subjectively better.

You can see the difference most clearly by reducing the script to just:

BSSource("SW-sample.mov") # or your source filter
ConvertToRGB(matrix = "...") # switch between rec709 and pc.709

By using the pc.709 matrix, you can (perhaps entirely) eliminate the negative RGB values, and see that the apple is, in reality, noisier than it appears when converted to RGB with the rec709 matrix, which results in more clamping to 0.

[Z2697]

As I said, if the problem is dithering.
Haven't you tought about the problem with sharpening? (plus the fact wonkey_monkey has pointed out)
Sharpening under YUV don't know about the value gonna be clamped after conversion, and something something...