First of all, Dither tool is a hack, 'cause 16bit is not officially supported by Avisynth.
Anyway, it works and I've been using it for a while.
You can use f3kdb to dither down to 10bit output, so Dither_quantize and Dither_Out are not needed.
Also, try to avoid using MT Mode, 'cause afaik it doesn't work reliably with Dither Tool.
Code:
FFVideoSource("C:\RMZ\RAW\avatar.mkv")
Trim(18000, 20000)
TFM(order=-1, field=-1,mode=0,cthresh=8)
TDecimate(mode=1, cycleR=1, cycle=25, rate=24.000, hybrid=0)
AssumeFPS(24)
QTGMC(Preset="Medium", Sharpness=0, InputType=3, SourceMatch=3, FPSDivisor=1, ShowSettings=False, TR0=0, TR1=1, TR2=1, Rep0=0, Rep1=0, Rep2=0, RepChroma=false)
Dither_convert_8_to_16()
dfttest(sigma=64, tbsize=1, lsb_in=true, lsb=true, Y=true, U=true, V=true, opt=3, dither=0)
f3kdb(range=15, Y=80, Cb=60, Cr=60, grainY=0, grainC=0, keep_tv_range=True, input_depth=16, output_depth=10)
There's another way to use 16bit in avisynth, which is using HDRCore (16bit interleave), while Dither Tool uses 16bit stack.
HDR Core is slightly faster but less supported.
Also please note that if you are using AVS Studio or VirtualDub for preview, you can't view 10/12/16bit footages, but you gotta encode them first.
In order to view if everything is working fine, just use output_depth=8 in f3kdb, check it in VirtualDub, then, if everything is working fine, change it back to 10 and encode with your favorite encoder (x264, x265 etc).
By the way, may I ask you what are you trying to achieve with TFM, Tdecimate, AssumeFPS and QTGMC? Is your source interlaced? Telecined? Both?