Log in

View Full Version : Overlay 'subtract' not working as expected


Selur
29th March 2025, 00:22
I got the following Avisynth script:
ClearAutoloadDirs()
SetFilterMTMode("DEFAULT_MT_MODE", MT_MULTI_INSTANCE)
LoadPlugin("F:\Hybrid\64bit\Avisynth\avisynthPlugins\LSMASHSource.dll")
LoadPlugin("F:\Hybrid\64bit\Avisynth\avisynthPlugins\MedianBlur2.dll")
LoadPlugin("F:\Hybrid\64bit\Avisynth\avisynthPlugins\grunt-x64.dll")
LoadPlugin("F:\Hybrid\64bit\Avisynth\avisynthPlugins\Average.dll")
LoadPlugin("F:\Hybrid\64bit\Avisynth\avisynthPlugins\TIVTC.dll")
LoadPlugin("F:\Hybrid\64bit\Avisynth\avisynthPlugins\masktools2.dll")
LoadPlugin("F:\Hybrid\64bit\Avisynth\avisynthPlugins\mvtools2.dll")
LoadPlugin("F:\Hybrid\64bit\Avisynth\avisynthPlugins\RgTools.dll")
LoadCPlugin("F:\Hybrid\64bit\Avisynth\avisynthPlugins\BWDIF.dll")
Import("F:\Hybrid\64bit\Avisynth\avisynthPlugins\mtmodes.avsi")
Import("F:\Hybrid\64bit\Avisynth\avisynthPlugins\Srestore.avsi")
Import("F:\Hybrid\64bit\Avisynth\avisynthPlugins\Zs_RF_Shared.avsi")
# loading source: C:\Users\Selur\Desktop\Newhart - S02E03 - clip.mkv
# color sampling YV12@8, matrix: bt601, scantyp: bottom field first, luminance scale: limited
LWLibavVideoSource("C:\Users\Selur\Desktop\Newhart - S02E03 - clip.mkv",cache=false,dr=true,format="YUV420P8", prefer_hw=0)
# current resolution: 720x480
# deinterlacing
BWDIF(field=2)
# removing ghosting
srestore(frate=23.976,omode=6)
Preroll(Int(last.FrameRate()))

## DEBLEND START
o=last
t=o.trim(1,0)
overlay(o, t, mode="subtract", opacity=0.15)
#levels(0,1,218,0,255,false,true)
ov=last
interleave(o.selecteven, ov.selectodd)
## DEBLEND END

# Converting from 8 to 10bit for encoder
ConvertBits(10)
# setting output fps to 23.976fps
AssumeFPS(24000,1001)
# output: color sampling YV12@10, matrix: bt601, scantyp: progressive, luminance scale: limited
return last
which works fine.
I then wanted to do the same in Vapoursynth:
# Imports
import vapoursynth as vs
# getting Vapoursynth core
import sys
import os
core = vs.core
# Import scripts folder
scriptPath = 'F:/Hybrid/64bit/vsscripts'
sys.path.insert(0, os.path.abspath(scriptPath))
# loading plugins
core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/DeinterlaceFilter/Bwdif/Bwdif.dll")
core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/SourceFilter/LSmashSource/LSMASHSource.dll")
# Import scripts
import srestore
import validate
# Source: 'C:\Users\Selur\Desktop\Newhart - S02E03 - clip.mkv'
# Current color space: YUV420P8, bit depth: 8, resolution: 720x480, frame rate: 29.97fps, scanorder: bottom field first, yuv luminance scale: limited, matrix: 470bg, transfer: bt.601, primaries: bt.601 ntsc, format: mpeg-2
# Loading C:\Users\Selur\Desktop\Newhart - S02E03 - clip.mkv using LWLibavSource
clip = core.lsmas.LWLibavSource(source="C:/Users/Selur/Desktop/Newhart - S02E03 - clip.mkv", format="YUV420P8", stream_index=0, cache=0, fpsnum=30000, fpsden=1001, prefer_hw=0)
frame = clip.get_frame(0)
# setting color matrix to 470bg.
clip = core.std.SetFrameProps(clip, _Matrix=vs.MATRIX_BT470_BG)
# setting color transfer (vs.TRANSFER_BT601), if it is not set.
if validate.transferIsInvalid(clip):
clip = core.std.SetFrameProps(clip=clip, _Transfer=vs.TRANSFER_BT601)
# setting color primaries info (to vs.PRIMARIES_BT470_BG), if it is not set.
if validate.primariesIsInvalid(clip):
clip = core.std.SetFrameProps(clip=clip, _Primaries=vs.PRIMARIES_BT470_BG)
# setting color range to TV (limited) range.
clip = core.std.SetFrameProps(clip=clip, _ColorRange=vs.RANGE_LIMITED)
# making sure frame rate is set to 29.97fps
clip = core.std.AssumeFPS(clip=clip, fpsnum=30000, fpsden=1001)
# making sure the detected scan type is set (detected: bottom field first)
clip = core.std.SetFrameProps(clip=clip, _FieldBased=vs.FIELD_BOTTOM) # bff
# Deinterlace using Bwdif
clip = core.bwdif.Bwdif(clip=clip, field=2) # new fps: 59.94
# Making sure content is preceived as frame based
clip = core.std.SetFrameProps(clip=clip, _FieldBased=vs.FIELD_PROGRESSIVE) # progressive
# adjusting frame count and rate with sRestore
clip = srestore.sRestoreMUVs(source=clip, frate=23.9760)

## DEBLEND START
# Create 't' by trimming the first frame
t = clip.std.Trim(first=1)
# Subtract odd image
import misc
ov = misc.Overlay(clip,t,mode="subtract", opacity=0.15) # https://github.com/Selur/VapoursynthScriptsInHybrid/blob/master/misc.py, same as in havsfunc.
# Adjust the levels
ov = ov.std.Levels(min_in=1, max_in=218, min_out=0, max_out=255, planes=0)
# Combine
clip = core.std.Interleave([clip[::2], ov[1::2]])
## DEBLEND END

# adjusting color space from YUV420P8 to RGBH for vsBasicVSRPPFilter
#clip = core.resize.Bicubic(clip=clip, format=vs.RGBH, matrix_in_s="470bg", range_s="limited")
# Quality enhancement using BasicVSR++
#from vsbasicvsrpp import basicvsrpp as BasicVSRPP
#clip = BasicVSRPP(clip=clip, model=7)

# adjusting output color from: RGBH to YUV420P10 for NVEncModel
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P10, matrix_s="470bg", range_s="limited", dither_type="error_diffusion")
# set output frame rate to 23.976fps (progressive)
clip = core.std.AssumeFPS(clip=clip, fpsnum=24000, fpsden=1001)
# output
clip.set_output()


If I don't use the DEBLEND part in both scripts, their output is basically identical, but when I use the DEBLEND part in both, it seems like the Overlay does not work as it should.
Frame 0: Avisynth | Vapoursynth
https://i.ibb.co/bgGBfSvC/Frames-0-Avisynth.png (https://ibb.co/RpJ05x6s)https://i.ibb.co/4gJTtrgh/Frames-0-Vapoursynth.png (https://ibb.co/G3F2pS3z)
Frame 1: Avisynth | Vapoursynth
https://i.ibb.co/cc3fPwmH/Frames-1-Avisynth.png (https://ibb.co/PZjXRMSn)https://i.ibb.co/KxTjMdCd/Frames-1-Vapoursynth.png (https://ibb.co/nMxsH9h9)
see: https://forum.videohelp.com/threads/417939 for source clip


I suspect, that either I'm making a stupid (probably obvious) error, I simply don't see or there is a bug in the Overlay function.

Does anyone spot the problem?
Can anyone reproduce this?

Thanks. :)

Cu Selur

poisondeathray
29th March 2025, 02:51
It looks like subtract formula is different; I didn't look at the code just some quick tests

For a YV12 clip, in avs if you use overlay and subtract itself, you get 0,128,128 or "black"

In vpy using haf.overlay and subtract mode, you get 0,0,0 or something mathematically correct, but probably not expected for YUV color model

Selur
29th March 2025, 07:28
https://github.com/Selur/VapoursynthScriptsInHybrid/blob/1ada95ade06e17032e3e95602e1a236b7464bb0e/misc.py#L156 uses 'x y -' which intuitively seems what I would have expected.
You are correct in http://avisynth.nl/index.php/Subtract it sounds like Avisynth is doing something else (https://github.com/AviSynth/AviSynthPlus/blob/424707f297d4a7ecae3d5b84b35b1a1297ec8782/avs_core/filters/overlay/OF_add.cpp#L197C14-L197C25).