Log in

View Full Version : Debanding anime with VapourSynth and neo_f3kdb


GeoffreyA
5th July 2024, 10:48
Greetings. I hope you're all well.

This is my first post, though I've been reading Doom9 for years and first came across the site in 2004, when I was a teenager embarking on the wonderful world of video encoding in the DVD and MPEG-4 ASP era. Discovering FFmpeg in 2018 reignited my fondness for this field, and I've had nothing but praise for FFmpeg and its maintainers: it was startling how a single tool could do so much, in contrast to the old days when everything seemed a fragmented mess. Recently, tackling the encoding of anime with AV1, I've had trouble with banding, and FFmpeg's deband filter wasn't helping. Long story short, I ended up with VapourSynth rather than AviSynth because I found the former somewhat more intuitive and, more importantly, got it working.

The neo_f3kdb filter did an excellent job, and I didn't need to denoise, degrain, or mask. Before I encode, I thought I'd post my script here and check with the pros whether everything was all right, according to VS conventions, and that nothing was missing. My process is to use a batch file that iterates over the MKV files, passing the script and videos to VSPipe; the script runs, debanding; and the raw data is piped to FFmpeg, where it is encoded with AV1 in two passes. Here is the script:

from vapoursynth import core

clip = core.lsmas.LWLibavSource(filename)

clip = core.fmtc.bitdepth(clip, bits=16)
clip = core.neo_f3kdb.Deband(clip, keep_tv_range=True, output_depth=16)
clip = core.fmtc.bitdepth(clip, bits=10)

clip.set_output()

And the Windows batch file:

mkdir "out"

setlocal
set vscript=Script.vpy
set vspath=D:\Stuff\Movie Tools\Conversion\VapourSynth
set param1=-c:v libaom-av1 -cpu-used 3 -arnr-strength 0 -crf 16 -b:v 0 -g 240

for %%i in (*.mkv) do (

"%vspath%\vspipe" -c y4m "%~dp0%vscript%" - -a filename="%~dp0%%i" | ffmpeg -i - %param1% -pass 1 -f null -
"%vspath%\vspipe" -c y4m "%~dp0%vscript%" - -a filename="%~dp0%%i" | ffmpeg -i - -i "%%i" -map 0:0 -map 1:a? -map 1:s? %param1% -pass 2 -c:a libopus -b:a 160k -c:s copy "out\%%i"

)

endlocal

In short, as a newcomer to VapourSynth and frame-servers, in the VS script, does everything look all right? Thanks in advance for any thoughts or suggestions.

Q3CPMA
24th August 2024, 09:25
Any reason for using `output_depth=16` then fmtconv instead of `output_depth=10` from the get-go? Otherwise, I'd maybe massage those AOM parameters a bit more, https://old.reddit.com/r/AV1/comments/t59j32/encoder_tuning_part_4_a_2nd_generation_guide_to/ has some useful stuff on the subject.

GeoffreyA
24th August 2024, 13:49
Any reason for using `output_depth=16` then fmtconv instead of `output_depth=10` from the get-go? Otherwise, I'd maybe massage those AOM parameters a bit more, https://old.reddit.com/r/AV1/comments/t59j32/encoder_tuning_part_4_a_2nd_generation_guide_to/ has some useful stuff on the subject.

Hi. I used output_depth=16 because, according to the documentation, it doesn't dither when set to 16. I would think that fmtconv has the better dithering implementation (Sierra-24A by default), so I let it handle that part. Am I correct in my interpretation? Or would it be fewer operations, from a quality point of view, to do it straight in the debanding function?

Since then, I have revised my libaom parameters:

-cpu-used 4 -crf 16 -arnr-strength 0 -lag-in-frames 48 -aom-params enable-qm=1

Q3CPMA
24th August 2024, 17:14
Makes sense, though neo_f3kdb seems to have the "pretty decent" Floyd-Steinberg algo available.

GeoffreyA
24th August 2024, 19:49
This made me think, so I think I'll have to test the two versions.

GeoffreyA
25th August 2024, 10:41
If it's of help to others, I thought I'd add my final batch file, which was simplified and refined. The VPY script is the same as the first post.

mkdir "out"

setlocal
set vscript=Script.vpy
set vspipe=D:\Stuff\Movie Tools\Conversion\VapourSynth\VSPipe
set param=-vf setsar=1 -c:v libaom-av1 -cpu-used 4 -crf 16 -arnr-strength 0 -lag-in-frames 48 -g 240 -aom-params enable-qm=1
set param_colour=-colorspace bt709 -color_primaries bt709 -color_trc bt709 -color_range tv

for %%i in (*.mkv *.m2ts) do (

"%vspipe%" -a filename="%~dp0%%i" -c y4m "%~dp0%vscript%" - | ffmpeg %param_colour% -i - %param% -pass 1 -f null -
"%vspipe%" -a filename="%~dp0%%i" -c y4m "%~dp0%vscript%" - | ffmpeg %param_colour% -i - %param% -pass 2 "out\%%i.mkv"

)

endlocal

Here, I also removed the audio encoding, which was first in libopus 160 kbps. Instead, I handled that in a separate script that, using FFmpeg, piped the 24-bit PCM to QAAC, encoding it with TVBR 91. On average, the files were below 160 kbps. Going back to AAC also meant that I could use the same audio for the legacy 8-bit H.264 encode I made. In the end, the video, audio, subtitles, and chapters were muxed together using a batch script and mkvmerge. Indeed, I could just change the video folder, AV1 or H.264, and mux without changing a thing. As I write this, an x265 encoding of the series is underway (crf 18 slow aq-mode=3 psy-rd=1 limit-sao bframes=8).

Touching on libaom, I found that cpu-used 4 is a better balance of speed and quality than 3. The latter slows down encoding a lot more for small, perhaps placebo gains. As for temporal filtering, one has got to set arnr-strength < 2, or enable-keyframe-filtering=0, because of occasional defects on keyframes. The popular forks, aom-av1-psy, aom-av1-lavish, and aom-psy101, do this automatically. Also, enable-qm=1 gains efficiency, and setting lag-in-frames=48 is probably beneficial. I tend to leave out other settings that testing showed made little difference or seemed to have been more of a placebo, cargo-culting nature. At the end of the day, I think, the master controls are cq-level, or crf in FFmpeg, cpu-used, and arnr-strength.

Recently, I gave the SVT-AV1-PSY fork a go. It produces excellent results, particularly with grainy, live-action material. They've done a lot of work, some of which has been merged into mainline, and it seems to be a hair's-breadth behind libaom. Doubtless, it will overtake the latter in the near future and become the x264 of AV1. One can add it to FFmpeg with the Media Autobuild Suite by editing "build/media-suite_deps.sh" and changing the Git path under "SOURCE_REPO_SVTAV1=https://gitlab.com/AOMediaCodec/SVT-AV1.git" to "SOURCE_REPO_SVTAV1=https://github.com/gianni-rosato/svt-av1-psy.git".