sephirotic
23rd February 2016, 00:43
So, I never bothered fiddling with Dither and full 16 bit filter chains before because I didn't like the 10 bit compatibility limitations of x264, I thought giving enough bitrate to 8 bit encodes, specially for grainy sources, produced great quality with more than acceptable banding.
However, with news about HEVC format accepting 10 bits standardized on all future hardware players, things changed and I started trying to fiddle with 16 bit filter chains for doing 10 bit encodings.
Now, I'm having issues with the following filter chain:
maa2 (ss=2.0, aa=60)
Dither_convert_8_to_16()
SMDegrain(lsb_in=true, lsb=true, lsb_out=true, tr=2, thsad=120, prefilter=3, refinemotion=true)
ConverttoYV24(matrix="rec709")
f3kdb (range=20, input_depth=16, output_depth=16)
Dither_resize16 (1600,900)
Dither_out()
Yes, I want to do a 1600x900 10 bit 4:4:4 HEVC encode using avs4x265.
With the ConverttoYV24 in the middle of the 16 bit processing, I get a couple of garbled green and pink lines in the bottom of the encode such as:
http://i.imgur.com/daZoIAa.png
I realized I can't use most filters without built in lsb feature on its commands inside the 16 bit filter chain, so the possible solutions I found were:
1) Put ConvertTOYV24 BEFORE dithering to 16 bits. However, this is not viable since SMDegrain doesn't work in YV24.
2): Putting smdegrain outside the filter chain and after running it, converting to YV24.
I have the feeling this is not optimal, so I thought about an alternative:
I thought about using the built in Dither_convert feature, but it seems to not be able to convert from YV12 to YV24 directly, so I thought of doing something like this:
maa2 (ss=2.0, aa=60)
Dither_convert_8_to_16()
SMDegrain(lsb_in=true, lsb=true, lsb_out=true, tr=2, thsad=120, prefilter=3, refinemotion=true)
f3kdb (range=20, input_depth=16, output_depth=16)
Dither_convert_yuv_to_rgb(matrix="709")
Dither_convert_rgb_to_yuv(matrix="709", output="yv24")
Dither_resize16 (1600,900)
Dither_out()
This is much slower tough.
Now, several doubts about which of the two process come to my mind:
1) Is it worth to put the SMDegrain inside the 16bit dithered filter chain, o just using in regularly in 8bits with lsb=true would give pretty much the same quality?
2) Will I be losing too much precision by doing two colorspace conversions?
2) Should I specify the range to TV in both the Dither_convert passes?
3) Any other optimization useful to consider in this filter chain for a modern digital animation with fairly amount of banding and fine texture? (GATE) I'm using SMDegrain with prefilter=3 to remove residual blocking from the h264 encode. Any better way to do that?
4) fiddling with pel and subpixel settings on SMdegrain can result in better quality?
Thanks a lot.
However, with news about HEVC format accepting 10 bits standardized on all future hardware players, things changed and I started trying to fiddle with 16 bit filter chains for doing 10 bit encodings.
Now, I'm having issues with the following filter chain:
maa2 (ss=2.0, aa=60)
Dither_convert_8_to_16()
SMDegrain(lsb_in=true, lsb=true, lsb_out=true, tr=2, thsad=120, prefilter=3, refinemotion=true)
ConverttoYV24(matrix="rec709")
f3kdb (range=20, input_depth=16, output_depth=16)
Dither_resize16 (1600,900)
Dither_out()
Yes, I want to do a 1600x900 10 bit 4:4:4 HEVC encode using avs4x265.
With the ConverttoYV24 in the middle of the 16 bit processing, I get a couple of garbled green and pink lines in the bottom of the encode such as:
http://i.imgur.com/daZoIAa.png
I realized I can't use most filters without built in lsb feature on its commands inside the 16 bit filter chain, so the possible solutions I found were:
1) Put ConvertTOYV24 BEFORE dithering to 16 bits. However, this is not viable since SMDegrain doesn't work in YV24.
2): Putting smdegrain outside the filter chain and after running it, converting to YV24.
I have the feeling this is not optimal, so I thought about an alternative:
I thought about using the built in Dither_convert feature, but it seems to not be able to convert from YV12 to YV24 directly, so I thought of doing something like this:
maa2 (ss=2.0, aa=60)
Dither_convert_8_to_16()
SMDegrain(lsb_in=true, lsb=true, lsb_out=true, tr=2, thsad=120, prefilter=3, refinemotion=true)
f3kdb (range=20, input_depth=16, output_depth=16)
Dither_convert_yuv_to_rgb(matrix="709")
Dither_convert_rgb_to_yuv(matrix="709", output="yv24")
Dither_resize16 (1600,900)
Dither_out()
This is much slower tough.
Now, several doubts about which of the two process come to my mind:
1) Is it worth to put the SMDegrain inside the 16bit dithered filter chain, o just using in regularly in 8bits with lsb=true would give pretty much the same quality?
2) Will I be losing too much precision by doing two colorspace conversions?
2) Should I specify the range to TV in both the Dither_convert passes?
3) Any other optimization useful to consider in this filter chain for a modern digital animation with fairly amount of banding and fine texture? (GATE) I'm using SMDegrain with prefilter=3 to remove residual blocking from the h264 encode. Any better way to do that?
4) fiddling with pel and subpixel settings on SMdegrain can result in better quality?
Thanks a lot.