Encoding with 4:4:4

[DeathAngelBR]

I have been testing encodes with yuv444 colorspace, but I'm not sure wether
1) I screwed up the encode and have no idea what I'm doing;
2) High quality, bluray anime is as good as it can get nowadays.

I had to zoom in 400% to see very tiny differences. In some areas, the 4:4:4 image seems to be slightly sharpened or perhaps it's a direct result of higher resolution chroma, or maybe 1).

Screenshots from final 4:4:4 encode
Image 1 (https://drive.google.com/file/d/1zlrnuqXECz0S_6DifrltkJzYRNHQOI6D/view?usp=sharing)
Image 2 (https://drive.google.com/file/d/1fT4Ga0kSCy8tYd2C7k4_M6xGs83tKjiw/view?usp=sharing)
Image 3 (https://drive.google.com/file/d/1-MoIS_yu2u-ZmCZpGr44HGTJk6QqD15g/view?usp=sharing)

Avisynth script (omitting plugin load lines for the sake of simplicity)

LWLibavVideoSource("U:\[BDRIP] Harukana Receive\HARUKANA_RECEIVE_02\BDMV\STREAM\HARUKANA_RECEIVE_02_00001.m2ts.lwi")

mydenoise()
Dither_convert_8_to_16()
GradFun3(lsb_in=true,lsb=true)
Dither_convert_yuv_to_rgb(lsb_in=true) # not sure if
Dither_convert_rgb_to_yuv(lsb=true,output="YV24") # these lines are correct
Dither_resize16(1280,720) # should I have used csp="YV24" here or the function recognizes it?
Dither_out()

function mydenoise(clip c) {

c

super = MSuper(pel=2,sharp=1)
backward_vec2 = MAnalyse(super,isb=true,delta=2,blksize=16,overlap=8)
backward_vec1 = MAnalyse(super,isb=true,delta=1,blksize=16,overlap=8)
forward_vec1 = MAnalyse(super,isb=false,delta=1,blksize=16,overlap=8)
forward_vec2 = MAnalyse(super,isb=false,delta=2,blksize=16,overlap=8)
MDegrain2(super,backward_vec1,forward_vec1,backward_vec2,forward_vec2,thSAD=128)

return last

}

Command line:


"avs4x26x-x64.exe" --seek-mode safe --x264-binary "x264.2851kMod.10bit.x86_64.exe" --preset veryslow --tune animation --crf 15.0 --deblock -1:-1 --keyint 250 --qcomp 0.75
--fade-compensate 0.8 --aq-mode 3 --aq-strength 0.6 --psy-rd 0.40:0.10 --chroma-qp-offset -6 --no-dct-decimate --no-fast-pskip
--colorprim bt709 --transfer bt709 --colormatrix bt709 --input-csp i444 and --output-csp i444 --input-depth 16
--output "harukana receive ep03 video 720p hi10p444.264" "HARUKANA_RECEIVE_02_00000.m2ts.avs"


Screenshots from final 4:2:0 encode
4:2:0
Image 4 (https://drive.google.com/file/d/1U74FjnNS5RaKD8ktaO1NUCSz7N3pvrqE/view?usp=sharing)
Image 5 (https://drive.google.com/file/d/12oVHaBhNrlLzWUdgal3YuWfv5edRk6Ua/view?usp=sharing)
Image 6 (https://drive.google.com/file/d/12iBF0VGXW2-nkGkhuxXn-YaX8QwesQXd/view?usp=sharing)

Script is the same except the dither_convert_xxx_to_xxx lines.

Command line:


"avs4x26x-x64.exe" --seek-mode safe --x264-binary "x264.2851kMod.10bit.x86_64.exe" --preset veryslow --tune animation --crf 15.0 --deblock -1:-1 --keyint 250 --qcomp 0.75
--fade-compensate 0.8 --aq-mode 3 --aq-strength 0.60 --psy-rd 0.40:0.10 --no-dct-decimate --no-fast-pskip
--colorprim bt709 --transfer bt709 --colormatrix bt709 --input-depth 16
--output "harukana receive ep03 video 720p hi10p420.264" "HARUKANA_RECEIVE_02_00000.m2ts.avs"

[FranceBB]

You didn't upload a sample, but you left the name there, so I took a look at Harukana Receive on Crunchyroll and it looks upscaled from a lower resolution master.
Since you are using Dither Tools, I assume you are using the legacy Avisynth 2.6.1.

A few things:

1) I noticed that you are doing Denoise using MVTools at 8bit, but you should apply denoise with 16bit precision, especially for anime, 'cause it's less prone to produce banding. MVTools supports 16bit stacked on normal Avisynth 2.6.1 and normal 16bit on Avisynth+.

2) I noticed that you are applying debanding with 16bit precision before resizing, but I suggest you to apply it after resizing.

Try with this to downscale:


ly = debicubicy(1280,720,lsb_inout=true)
lu = utoy().dither_resize16(1280,720,kernel="bicubic",invks=true,invkstaps=3,src_left=0.25,u=1,v=1)
lv = vtoy().dither_resize16(1280,720,kernel="bicubic",invks=true,invkstaps=3,src_left=0.25,u=1,v=1)
ytouv(lu,lv,ly)


Generally, anime are made in 1280x720 or slightly higher resolution like 810p and animation studios work in RGB 'till the very last moment, when they export to their target (Generally Apple ProRes 10bit 4:2:2 planar yv16). The reason why people do reverse upscale and encode Blu-ray (1080p 4:2:0 8bit) in 1280x720 10bit 4:4:4 is because they try to preserve chroma, which makes sense. But, if they use a 1080p 4:2:0 8bit source and encode in 1080p 4:4:4, then they are just upscaling chroma, which doesn't make sense.
Anyway, many people won't notice any difference between 4:2:0 and 4:4:4 as it's subtle, so it's not a surprise that you might notice relatively little differences.

[DeathAngelBR]

Since you are using Dither Tools, I assume you are using the legacy Avisynth 2.6.1.

Avisynth+ r2420

1) I noticed that you are doing Denoise using MVTools at 8bit, but you should apply denoise with 16bit precision, especially for anime, 'cause it's less prone to produce banding. MVTools supports 16bit stacked on normal Avisynth 2.6.1 and normal 16bit on Avisynth+.

I assume I should use lsb=true in MDegrain after calling dither_convert_8_to_16. Is that correct?

------
edit 2: apparently not.
------

2) I noticed that you are applying debanding with 16bit precision before resizing, but I suggest you to apply it after resizing.

Forgot to change that since I just comment out the convert lines when encoding 1080p before 720p.

Try with this to downscale:


ly = debicubicy(1280,720,lsb_inout=true)
lu = utoy().dither_resize16(1280,720,kernel="bicubic",invks=true,invkstaps=3,src_left=0.25,u=1,v=1)
lv = vtoy().dither_resize16(1280,720,kernel="bicubic",invks=true,invkstaps=3,src_left=0.25,u=1,v=1)
ytouv(lu,lv,ly)


Will do.

------
edit: there seems to be no 64-bit version of debicubic, guess I'm back to my previous resizing.
------

Generally, anime are made in 1280x720 or slightly higher resolution like 810p and animation studios work in RGB 'till the very last moment, when they export to their target (Generally Apple ProRes 10bit 4:2:2 planar yv16). The reason why people do reverse upscale and encode Blu-ray (1080p 4:2:0 8bit) in 1280x720 10bit 4:4:4 is because they try to preserve chroma, which makes sense. But, if they use a 1080p 4:2:0 8bit source and encode in 1080p 4:4:4, then they are just upscaling chroma, which doesn't make sense.
Anyway, many people won't notice any difference between 4:2:0 and 4:4:4 as it's subtle, so it's not a surprise that you might notice relatively little differences.

I figured as much. Besides, encoding 1080p 4:4:4 in 10-bit adds an extra hour or two, not really worth it when letting it run while sleeping.

[FranceBB]

Well, since you are using Avisynth+, there's no need to use Dither_Tools and you can use regular 16bit instead of 16bit stacked (except for GradFun3).
For MDegrain, there's the legacy lsb=true for 16bit stacked, but you should use out16=true - which pinterf added - to use normal 16bit.
For Debicubic, you can use the resizing filter made by JPSDR (ResampleMT) that are able to work with normal 16bit.
In other words, I would change your script to something like:

#Indexing 8bit source
LWLibavVideoSource("U:\[BDRIP] Harukana Receive\HARUKANA_RECEIVE_02\BDMV\STREAM\HARUKANA_RECEIVE_02_00001.m2ts.lwi", format="YUV420P8")

#Feeding MDegrain with 8bit and outputting regular 16bit
super = MSuper(pel=2, sharp=1)
backward_vec2 = MAnalyse(super, isb=true, delta=2, blksize=16, overlap=8)
backward_vec1 = MAnalyse(super, isb=true, delta=1, blksize=16, overlap=8)
forward_vec1 = MAnalyse(super, isb=false, delta=1, blksize=16, overlap=8)
forward_vec2 = MAnalyse(super, isb=false, delta=2, blksize=16, overlap=8)
MDegrain2(super, backward_vec1, forward_vec1, backward_vec2, forward_vec2, thSAD=128, out16=true)

#Downscale inverting the kernel probably used for the upscale
DebicubicResizeMT(1280, 720)

#Converting from normal 16bit to 16bit stacked
ConverttoStacked()

#Debanding with 16bit precision
GradFun3(lsb_in=true,lsb=true)

#Dithering down to 10bit with the Floyd-Steinberg error diffusion
ConvertFromStacked()
ConvertBits(bits=10, dither=1)

In this case, you can stick with 4:2:0 10bit output, which would look fine.
Sadly, GradFun3 does support yv12 16bit stacked, but doesn't support yv24 16bit stacked. You can, however, use f3kdb which supports yv24 16bit stacked.

#Indexing 8bit source
LWLibavVideoSource("U:\[BDRIP] Harukana Receive\HARUKANA_RECEIVE_02\BDMV\STREAM\HARUKANA_RECEIVE_02_00001.m2ts.lwi", format="YUV444P8")

#Feeding MDegrain with 8bit and outputting regular 16bit
super = MSuper(pel=2, sharp=1)
backward_vec2 = MAnalyse(super, isb=true, delta=2, blksize=16, overlap=8)
backward_vec1 = MAnalyse(super, isb=true, delta=1, blksize=16, overlap=8)
forward_vec1 = MAnalyse(super, isb=false, delta=1, blksize=16, overlap=8)
forward_vec2 = MAnalyse(super, isb=false, delta=2, blksize=16, overlap=8)
MDegrain2(super, backward_vec1, forward_vec1, backward_vec2, forward_vec2, thSAD=128, out16=true)

#Downscale inverting the kernel probably used for the upscale
DebicubicResizeMT(1280, 720)

#Converting from normal 16bit to 16bit stacked
ConverttoStacked()

#Debanding with 16bit precision
f3kdb(range=15, Y=45, Cb=30, Cr=30, grainY=0, grainC=0, sample_mode=2, blur_first=true, dynamic_grain=false, opt=3, mt=true, keep_tv_range=true, input_mode=1, input_depth=16, output_mode=1, output_depth=16)

#Dithering down to 10bit with the Floyd-Steinberg error diffusion
ConvertFromStacked()
ConvertBits(bits=10, dither=1)

In this case the output would be 10bit 4:4:4 planar.

Cheers,
Frank.

[DeathAngelBR]

Well, since you are using Avisynth+, there's no need to use Dither_Tools and you can use regular 16bit instead of 16bit stacked (except for GradFun3).
For MDegrain, there's the legacy lsb=true for 16bit stacked, but you should use out16=true - which pinterf added - to use normal 16bit.
For Debicubic, you can use the resizing filter made by JPSDR (ResampleMT) that are able to work with normal 16bit.
In other words, I would change your script to something like:

#Indexing 8bit source
LWLibavVideoSource("U:\[BDRIP] Harukana Receive\HARUKANA_RECEIVE_02\BDMV\STREAM\HARUKANA_RECEIVE_02_00001.m2ts.lwi", format="YUV420P8")

#Feeding MDegrain with 8bit and outputting regular 16bit
super = MSuper(pel=2, sharp=1)
backward_vec2 = MAnalyse(super, isb=true, delta=2, blksize=16, overlap=8)
backward_vec1 = MAnalyse(super, isb=true, delta=1, blksize=16, overlap=8)
forward_vec1 = MAnalyse(super, isb=false, delta=1, blksize=16, overlap=8)
forward_vec2 = MAnalyse(super, isb=false, delta=2, blksize=16, overlap=8)
MDegrain2(super, backward_vec1, forward_vec1, backward_vec2, forward_vec2, thSAD=128, out16=true)

#Downscale inverting the kernel probably used for the upscale
DebicubicResizeMT(1280, 720)

#Converting from normal 16bit to 16bit stacked
ConverttoStacked()

#Debanding with 16bit precision
GradFun3(lsb_in=true,lsb=true)

#Dithering down to 10bit with the Floyd-Steinberg error diffusion
ConvertFromStacked()
ConvertBits(bits=10, dither=1)

In this case, you can stick with 4:2:0 10bit output, which would look fine.
Sadly, GradFun3 does support yv12 16bit stacked, but doesn't support yv24 16bit stacked. You can, however, use f3kdb which supports yv24 16bit stacked.

#Indexing 8bit source
LWLibavVideoSource("U:\[BDRIP] Harukana Receive\HARUKANA_RECEIVE_02\BDMV\STREAM\HARUKANA_RECEIVE_02_00001.m2ts.lwi", format="YUV444P8")

#Feeding MDegrain with 8bit and outputting regular 16bit
super = MSuper(pel=2, sharp=1)
backward_vec2 = MAnalyse(super, isb=true, delta=2, blksize=16, overlap=8)
backward_vec1 = MAnalyse(super, isb=true, delta=1, blksize=16, overlap=8)
forward_vec1 = MAnalyse(super, isb=false, delta=1, blksize=16, overlap=8)
forward_vec2 = MAnalyse(super, isb=false, delta=2, blksize=16, overlap=8)
MDegrain2(super, backward_vec1, forward_vec1, backward_vec2, forward_vec2, thSAD=128, out16=true)

#Downscale inverting the kernel probably used for the upscale
DebicubicResizeMT(1280, 720)

#Converting from normal 16bit to 16bit stacked
ConverttoStacked()

#Debanding with 16bit precision
f3kdb(range=15, Y=45, Cb=30, Cr=30, grainY=0, grainC=0, sample_mode=2, blur_first=true, dynamic_grain=false, opt=3, mt=true, keep_tv_range=true, input_mode=1, input_depth=16, output_mode=1, output_depth=16)

#Dithering down to 10bit with the Floyd-Steinberg error diffusion
ConvertFromStacked()
ConvertBits(bits=10, dither=1)

In this case the output would be 10bit 4:4:4 planar.

Cheers,
Frank.

I'll give that a try! Found a few other threads with similar solutions, and googling around found this (https://pastebin.com/RNCN62VW) and this one (https://github.com/Zeght/Morefun), the later having a more complicated debicubic for 16-bit which I figured I should stay away from.

[DeathAngelBR]

Something went wrong. First it gave out a warning that only dither = -1 or 0 can be used. So I changed to 0, then it said can only be used on 8-bit target. Figured I might update avs+, which I did, but then it insisted on converting to YV12 and crashed. Width is also getting cut by half (640x720p).

edit: added ConvertToDoubleWidth() after ConvertFromStacked() and it seems to work. I wonder if using format="YUV444P8" and then downscaling isn't "breaking the rules"?

[FranceBB]

edit: added ConvertToDoubleWidth() after ConvertFromStacked() and it seems to work. I wonder if using format="YUV444P8" and then downscaling isn't "breaking the rules"?

Well, the input is 4:2:0 8bit anyway, so it's just LWLibav that is outputting a 4:4:4 8bit and then MDegrain2 that is bringing it to 16bit with out=16 for better precision on your filter chain.
I know that this is a topic about 4:4:4, but I would personally stick with 4:2:0 10bit output which is gonna be fine.

By the way: ConvertBits actually allows you to choose Dither=1 http://avisynth.nl/index.php/ConvertBits

[DeathAngelBR]

Well, the input is 4:2:0 8bit anyway, so it's just LWLibav that is outputting a 4:4:4 8bit and then MDegrain2 that is bringing it to 16bit with out=16 for better precision on your filter chain.
I know that this is a topic about 4:4:4, but I would personally stick with 4:2:0 10bit output which is gonna be fine.

By the way: ConvertBits actually allows you to choose Dither=1 http://avisynth.nl/index.php/ConvertBits

But it threw that warning saying it can only be -1 or 0. Just tried it again and says


avs [error]: ConvertBits: dithering is allowed only for scale down


edit: final (video) file size also increased from 350MB to 698MB. Does it mean that


Dither_resize16nr(1280,720,invks=true,csp="YV24")


wasn't really outputting YV24, or f3kdb is adding a ton of grain compared to gradfun3?

[FranceBB]

It can't be grain, 'cause I specified (grainY=0, grainC=0, dynamic_grain=false) which basically means that f3kdb tries to deband without adding grain to neither Luma (grainY) nor Chroma (grainC) and the bare minimum that it adds has to be static (dynamic_grain=false) so that it doesn't mess up with the x264 motion prediction.
The file-size increase might be due to the fact that Debicubic is way sharper than a normal downscaler.
As to ConvertBits, if it outputs that warning, it means that what you are feeding it it's not 16bit.
I just tested my code with ColorBars and it works.

https://i.imgur.com/CK335GT.png

https://i.imgur.com/NM0oWRW.png

[DeathAngelBR]

Your code (crash):

https://i.imgur.com/TKw2BPem.png (https://i.imgur.com/TKw2BPe.png)

Commenting out convertbits (crash):

https://i.imgur.com/KemwLHQm.png (https://i.imgur.com/KemwLHQ.png)

Adding ConvertToDoubleWidth and keeping convertbits (error):

https://i.imgur.com/NdHEYPfm.png (https://i.imgur.com/NdHEYPf.png)

Keeping converttodoublewidth and removing convertbits works, but that file size...

AvsPmod also does show the correct preview, so it's either the piping that's messed up somehow or something else that I can't figure out.

edit: switched to spline36 and size went back to 400MB+. But yea, it's the piping and convertbits combination that's not working.

[Sparktank]

avs4x26x might be outdated.
I'm not sure which piping software is stable enough for current state of AVS+.

I've been using using Zeranoe builds of FFMPEG for encoding with x264.
https://ffmpeg.zeranoe.com/builds/

No piping required, even for 10bit encodes.

It just won't have the patches that kMod uses (or anything from the tMod branch).

[FranceBB]

Yep, it seems that your piping software expects the interleaved format, which is why it ends up with the wrong aspect ratio (640x720, 'cause it thinks that it's double width 1280x720, with 640 MSB and 640 LSB). Just output 16bit interleaved with the command "ConvertToDoubleWidth()", specify the input 16bit and let x264 encode it to 10bit.

Alternatively you can use libx264 of ffmpeg as SparkTank said.

[Natty]

ConvertFromStacked()
ConvertBits(bits=10, dither=1)

which method is better for 16 to 10 conversion? the above or letting f3kdb dithering it down to 10 bit?:thanks:

[FranceBB]

which method is better for 16 to 10 conversion? the above or letting f3kdb dithering it down to 10 bit?:thanks:

Well, f3kdb won't output a real 10bit as it was made back in the stacked/interleaved era, besides, when it outputs to, let's say, 8bit planar, I don't know which dithering algorithm it uses internally.
For practical use, I would let it output 16bit stacked or interleaved, then convert to 16bit planar and finally dither it down with the Floyd Steinberg error diffusion.
The Floyd Steinberg error diffusion is my favourite algorithm 'cause it's a well balanced one and it gives pretty good results. Another dithering algorithm would be the Stucki error diffusion and the Atkinson error diffusion (depending on whether you want your image sharper or not).
Another thing that it's pretty common is to output 16bit to your favourite encoder and let it dither it down in the best possible way (the most compression friendly way). I do it all the time in x265.
Please note though that not every encoder supports 16bit stacked or interleaved, so you would still probably have to convert it to 16bit planar 95% of the time.

[real.finder]

Well, f3kdb won't output a real 10bit as it was made back in the stacked/interleaved era

Well, even if f3kdb won't output a native 10bit, with output_depth=input_depth=10 and input_mode=2 and with ConvertToDoubleWidth/ConvertFromDoubleWidth(10) (http://avisynth.nl/index.php/ConvertStacked) interleaved 10 bit will be like real 10 bit (it's way better than other plugins that use lsb)

and I think anyone can easily port f3kdb to native HBD in avs+ since it already has 10-16 bit interleaved support, the only missed thing is the float (32 bit)