RIFE scene change detection hack via AviSynth border signaling

[shurik_pronkin]

RIFE scene change detection hack via AviSynth border signaling

The problem

RIFE's built-in sc_threshold is based on raw frame-to-frame SAD without motion compensation. It works okay for hard cuts with obvious brightness changes, but fails badly on:


Shot/reverse shot dialogues — same lighting, same background, only the face changes. RIFE doesn't see a scene change and produces ugly morph frames.
If you lower sc_threshold to catch these, RIFE starts duplicating frames on fast motion instead of interpolating them.


One threshold can't solve both problems.

Meanwhile, AviSynth has MVTools with MSCDetection which does motion-compensated SAD — it counts blocks where SAD exceeds thSCD1 after motion compensation and signals scene change if that count exceeds thSCD2. Not magic, still fails on fades and non-translational transforms, but the motion compensation step handles the "fast movement vs scene change" distinction noticeably better than raw frame comparison.

The proper solution would be adding a signal clip input to the RIFE plugin (e.g. an sc_clip parameter) so you could feed any external SCD decision directly. Until that exists, here's a workaround that uses the only channel RIFE currently understands — the pixels themselves.

The idea

Encode scene change metadata directly into the image using a border signal.


SCDetect adds a border strip on top of the frame — black normally, white on scene change
RIFE sees a massive brightness change in the border region → its sc_threshold fires reliably
Crop removes the border after RIFE — clean output


The single-pulse problem

Naive approach: white border on SC frame only.

...black, black, WHITE, black, black...

RIFE sees two transitions (black→white and white→black) and duplicates frames around both — you get ~3 duplicate frames instead of a clean cut.

The solution: toggle

Instead of a single pulse, alternate the border color between scenes:

Scene 1: black black black [SC] Scene 2: white white white [SC] Scene 3: black black black

Now there's exactly one border transition per scene change, coinciding with the actual content change. Result: ~2 duplicate frames at 50fps (40ms, invisible) instead of morph artifacts.

Improved SCDetect function

The standard SCDetect was improved with several features:


"Previous frame calm" check: if frame N-1 also had high YDifferenceFromPrevious, it's continuous motion, not a scene change. Real scene changes have a calm frame before the cut.
border parameter — adds borders and handles toggle internally
blksize — larger MAnalyse blocks average out noise, reducing false SCD on noisy footage (tip from DTL2020)
preblur_strength — three levels of pre-analysis denoise to suppress noise influence on SAD
downscale — BicubicResize before MSuper/MAnalyse, speeds up analysis and further suppresses noise (full resolution not needed for SCD)
Toggle state via global variable — works correctly with linear frame access (encoding)


scdetect.avsi:

Function SCDetect(clip c, string "scFile", int "thSCD1", int "thSCD2", float "mDiff", int "pel", int "search", int "searchparam", bool "preblur", int "preblur_strength", bool "info", int "border", int "blksize", int "downscale"){

scFile = Default(scFile, "scFile.log")
thSCD1 = Default(thSCD1, 360)
thSCD2 = Default(thSCD2, 120)
mDiff = Default(mDiff, 2.5)
pel = Default(pel, 1)
search = Default(search, 4)
searchparam = Default(searchparam, 2)
preblur = Default(preblur, true)
preblur_strength = Default(preblur_strength, 1)
info = Default(info, false)
border = Default(border, 0)
blksize = Default(blksize, 8)
downscale = Default(downscale, 1)

last = preblur ? (
\ preblur_strength == 3 ? c.RemoveGrain(20, 0).RemoveGrain(20, 0).Blur(1.0) :
\ preblur_strength == 2 ? c.RemoveGrain(20, 0).RemoveGrain(20, 0) :
\ c.RemoveGrain(20, 0)
\ ) : c

Assert( IsYV12 , "SCDetect needs YV12 input!" )

mv_clip = (downscale > 1) ? last.BicubicResize(width/downscale, height/downscale) : last

super = mv_clip.MSuper(pel=pel, chroma=false)
vector = super.MAnalyse(pelsearch=pel, search=search, searchparam=searchparam, chroma=false, blksize=blksize)
global scMask = MSCDetection(vector, thSCD1=thSCD1, thSCD2=thSCD2).Crop(0,0,16,16)

global SC_mD = mDiff
global mean = Histogram(mode="levels").Crop(width+64, 64, 128, 128)
global SC_state = 0

borderWhite = c.AddBorders(0, border, 0, 0, color=$FFFFFF)
borderBlack = c.AddBorders(0, border, 0, 0, color=$000000)

(border > 0) ? ConditionalFilter( last, borderWhite, borderBlack, "SC_state", "==", "1" )
\ : nop

FrameEvaluate( """
\ global boolSC =
\ (
\ ( scMask.YPlaneMax == 255 ) &&
\ ( mean.loop(0,0,-1).YDifferenceFromPrevious < SC_mD*2 ) &&
\ (
\ (mean.YDifferenceFromPrevious > SC_mD*3) ||
\ (
\ (mean.YDifferenceFromPrevious > SC_mD) &&
\ (mean.YDifferenceFromPrevious > (mean.YDifferenceToNext+mean.loop(0, 0, -1).YDifferenceToNext+mean.loop(0, 0, 1).YDifferenceToNext)/1.2)
\ )
\ )
\ )
\ global SC_state = boolSC ? (1 - SC_state) : SC_state""" )

( scFile != "nul" ) ? WriteFileStart( scFile, """ "# Scene change frame list" """ ).WriteFileIf( scFile, "boolSC==true", "current_frame" ) : nop

return last
}

Usage example (24p → 50fps):

Clean source:
SCDetect(scFile="nul", search=4, searchparam=2, pel=2, thSCD1=200, thSCD2=100, mDiff=0.5, border=180)
z_ConvertFormat(pixel_type="RGBPS", colorspace_op="709:709:709:l=>rgb:709:709:f")
Rife(gpu_thread=1, model=73, sc_threshold=0.15, sc1=false, sc=true, fps_num=50, fps_den=1, uhd=true, skip=true)
z_ConvertFormat(pixel_type="YUV420P10", colorspace_op="rgb:709:709:f=>709:709:709:l")
Prefetch(2)
Crop(0, 180, -0, -0)

Noisy source:
SCDetect(scFile="nul", search=4, searchparam=2, pel=2, thSCD1=200, thSCD2=100, mDiff=0.5, border=180, blksize=16, preblur_strength=2, downscale=2)
z_ConvertFormat(pixel_type="RGBPS", colorspace_op="709:709:709:l=>rgb:709:709:f")
Rife(gpu_thread=1, model=73, sc_threshold=0.15, sc1=false, sc=true, fps_num=50, fps_den=1, uhd=true, skip=true)
z_ConvertFormat(pixel_type="YUV420P10", colorspace_op="rgb:709:709:f=>709:709:709:l")
Prefetch(2)
Crop(0, 180, -0, -0)

Parameters to tune

thSCD1 MVTools SC sensitivity. Lower = more sensitive. Start: 200
thSCD2 MVTools secondary threshold. Start: 100
mDiff Brightness difference threshold. Start: 0.5
pel Subpixel precision (2 = better for subtle changes). Start: 2
border Border height in pixels. Start: 180
blksize MAnalyse block size. Larger = less noise sensitivity. Start: 8 (clean), 16-32 (noisy)
preblur_strength Denoise before analysis. 1/2/3. Start: 1 (clean), 2-3 (noisy)
downscale Resolution divider for MAnalyse. 1/2/4. Start: 1 (clean), 2 (noisy/speed)
sc_threshold RIFE's own threshold (don't go above 0.15). Start: 0.15

If missing soft scene changes (dialogue shot/reverse shot): lower thSCD1 by 25.
If false positives on fast motion: raise mDiff by 0.25.
If false positives on noisy footage: increase blksize, preblur_strength, or downscale.

Caveats


Toggle relies on linear frame access — works for encoding, may glitch in preview (AvsPmod/VirtualDub seeking). For preview/debugging use info=true without border to see where SC fires.
Crop must come after RIFE and Prefetch.
~2 duplicate frames per scene change at 50fps (40ms) — invisible in playback.
Border height should be large enough for RIFE to react reliably. 180px works well for 1080p.


How it works (TL;DR)

MVTools does motion-compensated SAD which separates fast motion from scene changes better than RIFE's raw frame comparison → the result is encoded as border color (toggle between scenes) → RIFE reads the signal through its own sc_threshold → border is cropped away. An out-of-band workaround until RIFE gets a proper signal clip input.

Credits

Thanks to DTL2020 for the technical breakdown of MSCDetection internals and suggestions on noise handling (blksize, downscale, preblur).

[DTL]

"MVTools detects scene changes accurately "

It is not really perfect but may be a bit better in comparison with simple SAD without motion compensation. The mvtools scenechange detect is simply return of the internal scd function (clip usability bool value) as AVS+ filter return value (colored frame). But internally it is based also on the mean SAD -
https://github.com/pinterf/mvtools/blob/a488b095c4bdc8d81abfd952ab534c015a9d45b7/Sources/MVSCDetection.cpp#L62
if ( mvClip.IsUsable() )
and it is https://github.com/pinterf/mvtools/blob/a488b095c4bdc8d81abfd952ab534c015a9d45b7/Sources/MVClip.cpp#L265-L268
bool MVClip::IsUsable(sad_t nSCD1_, int nSCD2_) const
{
return (!FakeGroupOfPlanes::IsSceneChange(nSCD1_, nSCD2_)) && FakeGroupOfPlanes::IsValid();
}

and it is https://github.com/pinterf/mvtools/blob/a488b095c4bdc8d81abfd952ab534c015a9d45b7/Sources/FakeGroupOfPlanes.cpp#L127-L130
bool FakeGroupOfPlanes::IsSceneChange(sad_t nThSCD1, int nThSCD2) const
{
return planes[0]->IsSceneChange(nThSCD1, nThSCD2);
}

and it is https://github.com/pinterf/mvtools/blob/a488b095c4bdc8d81abfd952ab534c015a9d45b7/Sources/FakePlaneOfBlocks.cpp#L72-L79
bool FakePlaneOfBlocks::IsSceneChange(sad_t nTh1, int nTh2) const
{
int sum = 0;
for ( int i = 0; i < nBlkCount; i++ )
sum += ( blocks[i].GetSAD() > nTh1 ) ? 1 : 0;

return ( sum > nTh2 );
}

Not really very advanced - it is simply 'if ((sum of number of blocks with SAD > thSCD1) > thSCD2) - then this is ref frame from other scene (not use it, signal SCD active). The only difference with currently implemented in RIFE difference is SAD computed for motion compensated blocks. Not for simply direct frames (samlpe-based without any motion compensation) SAD. It looks the algorithm with computing of motion compensated SAD makes better selection. But it still not perfect (may give false positive SCD signal on full frame fades and other _not_ compensated in current MVtools transforms).

Also if we can get some fast motion compensation engine - the same SCD algorithm may be added to RIFE plugin. Free for Windows 10 and higher exist via DX12-ME (but it only outputs MVs and require compatible hardware accelerator of home enduser level from hardware MPEG encoder - need motion compensation with these vectors by user but it is fast enough task). Complete solution with DX12 with SAD computing at the compute shader exist at github (freeware). But it requires windows-only (?) DX12 dependency and may need additional branch of RIFE with SCD via hardware accelerated ME+MC + DX12 Compute.

If RIFE uses Vulkan - may be some simple implementations of the motion compensation already avaialble for Vulkan API ?

Computing of MAnalyse at the CPU is slow enough. To make CPU usage lower you can downscale frame for MSuper to 1/2 or even lower size without significant quality loss I think.

"The only channel RIFE understands is the pixels themselves. No shared frame properties, no API, nothing. "

There are several ways to feed aux data into RIFE filter:
1. You can add your AVS script as a sequence of filters (mvtools) in the SCD part of the RIFE. The output will be Frame object and you can get binary signal from it (like 0,0 coordinate sample colour from frame buffer). Not nice in this way is all the filters params for SCD will be hardcoded in the plugin (or you need to add some way to control like from new filter params).
2. You can add new clip input in the RIFE like 'mscdclip' and new method of the SCD from old algorithm or from new input signal. This is more universal and you can prepare signal clip in AVS script in any ways possible and feed to RIFE SCD input.
Processing of the SCD from 2 frames SAD is in the area of https://github.com/Asd-g/AviSynthPlus-RIFE/blob/72a70f0015ab962b040a72a785390c399d0e79d3/src/plugin.cpp#L415

New processing is about:
sceneChange = false;
if (internal_scd)
sceneChange = get_sad_c(abs_diff.get(), abs_diff1.get()) > d->sc_threshold;
else // use mvscdclip for external signal
{
if (IsFrameNonZero(mscdclip->GetFrame(n)) ) sceneChange = true;
}


You can open issue at the github https://github.com/Asd-g/AviSynthPlus-RIFE with a feature request to support mscd clip input (as output of MSCDetection or any other script) in the RIFE and switch between old non-MC SAD method and SCD signal from this clip. It is very simple and short feature to add.

[shurik_pronkin]

Thanks DTL, really appreciate the detailed breakdown of how MSCDetection works internally — I had a simplified picture of it in my head. Updated the post to properly describe it as motion-compensated SAD rather than claiming it's "accurate", and added a note that this whole border trick is a workaround until a proper solution exists.

The mscdclip idea is exactly right — a signal clip input in RIFE would make this clean and universal. Hopefully Asd-g picks it up at some point, and then this hack can retire gracefully. Until then, the border toggle does the job.

[DTL]

The SCD part of the mvtools (or other ME engine) may be supplemented with MVs analysis too. This may helps to decrease false positive signals on no-motion frame sequences. At a real scene change the MVs are still generated but may be very random in direction. At the no-scene change big average SAD frame like full frame fade the MVs expected to be much more coherent. So either existed MSCDetect filter may be supplemented with more MVs analysis or new filter like MSCDetect2 may be added.
To check the input for the MVs analysis algorithm user may disable SCD in MShow and look at the MVs fields at the scene change.

The SAD also greatly depends on the general noise level. Even with perfect motion compensation SAD will be high at high noise footage. But for SCD only its influence may be decreased by both increasing of block size (partially) and downsizing of the input frame (more significantly) or in other way denoise the input frame.

[shurik_pronkin]

Good points, implemented what's doable from the script side:

blksize — larger blocks average out noise, reducing false SCD on noisy footage. Default 8, set 16 or 32 for noisy sources.

preblur_strength — three levels of pre-analysis denoise (1 = RemoveGrain, 2 = double RemoveGrain, 3 = double RemoveGrain + Blur). Addresses the SAD/noise dependency you mentioned.

downscale — BicubicResize before MSuper/MAnalyse. Speeds up analysis and further suppresses noise at the cost of spatial precision (which we don't need for SCD).

MV coherence analysis would be the real next step but that needs C++ — either extending MSCDetection or a new MSCDetect2 filter. Can't do that from .avsi.

Updated code and usage:

SCDetect(scFile="nul", search=4, searchparam=2, pel=2, thSCD1=200, thSCD2=100, mDiff=0.5, border=180, blksize=16, preblur_strength=2, downscale=2)
z_ConvertFormat(pixel_type="RGBPS", colorspace_op="709:709:709:l=>rgb:709:709:f")
Rife(gpu_thread=1, model=73, sc_threshold=0.15, sc1=false, sc=true, fps_num=50, fps_den=1, uhd=true, skip=true)
z_ConvertFormat(pixel_type="YUV420P10", colorspace_op="rgb:709:709:f=>709:709:709:l")
Prefetch(2)
Crop(0, 180, -0, -0)