Scene change detection

[Chikuzen]

scenechange-0.2.0-2.7z

This is a scene change detection plugin for VapourSynth.
This plugin detects the scene changes of a clip and attaches _SceneChangePrev/_SceneChangeNext property (0 or 1) to all the frames.

I think that this will make it a bit easy to devellop temporal filters.

usage:

>>> import vapoursynth as vs
>>> core = vs.Core()
>>> core.std.LoadPlugin('/path/to/scenechange.dll')
>>> clip = something

>>> clip = core.scd.Detect(clip clip[, int thresh, int interval_h, int interval_v, data log])

Detect scene changes and attach _SceneChange properies to the clip.

thresh: The threshold of average of differences of the luma value between previous/next frames.
When the average of luma differences exceeds this, the frame is judged with a scene change.
(range is 1 to 254*2^(bitdepth - 8), default(or out of range) is assumed 15*2^(bitdepth-8))

interval_h: The horizontal interval of the pixel used for measurement of difference.
(1 to width, default(or out of range) is auto adjust)

interval_v: The vertical interval of the pixel used for measurement of difference.
(1 to height, default(or out of range) is auto adjust)

log: The name of the log file to output.
If this is set, the frame numbers applicable to a scene change are outputted as a text.
Describing full path is recommended.
(default is unset)

>>> clip = core.scd.ApplyLog(clip clip, data log)

Apply _SceneChangePrev/_SceneChangeNext properties to the clip based on the log outputted by scd.Detect.

note:
suppouted color families are GRAY(8/16bits) and YUV(8/9/10/16bits).

[Didée]

Still no better idea than the simple old threshold thing? - Looking at only the difference the difference between N-1 and N1 is moot (high motion etc.). Go second order. Build diff between N-1 and N, then look at how that diff is changing. The method has been there since years in SCSelect(), part of RemoveDirt.dll. It's not perfect either, of course, but much more robust than simple thresholding.

[Chikuzen]

My main purpose is to consider practical use of the frame properties of VapourSynth instead of a better scene change detection algorithm.
yes, the algorithm which this plugin is using is old and simple. If someone writes a better thing, I will use it.

BTW, I think that it is very difficult to port kassandro's plugins to VapourSynth.
His softwares are exactry Open Binary.

[IanB]

_SceneChange property (0 or 1)

Possibly a better model might be to tag the frames either side of a scene change, i.e. 0 frame is normal and in the middle of a scene, 1 frame is the end of a scene, 2 frame is the beginning of a scene, of course the sequence would always be ...,0,0,1,2,0,0,...

This would allow a random access to a temporal filter to avoid fetching a next/prior frame from the information of the current frame alone.


P.S. And Merry Christmas to all

[Myrsloik]

Originally Posted by IanB

Possibly a better model might be to tag the frames either side of a scene change, i.e. 0 frame is normal and in the middle of a scene, 1 frame is the end of a scene, 2 frame is the beginning of a scene, of course the sequence would always be ...,0,0,1,2,0,0,... if you go for the 0 1 2 0 system you also have to define 3 to mean scene change on both sides. Debate on. Im not sure which one will make script writing easier.

This would allow a random access to a temporal filter to avoid fetching a next/prior frame from the information of the current frame alone.


P.S. And Merry Christmas to all

The way I scribbled it down was to mark both the first and the last frame of a scene with 1. So it'd be 0 0 0 1 1 0 0 ... caching should make the multiple frame access mostly free anyway.

Why not simply show this concept with a script? There aready is a built in function to calculate the average difference for a frame.

[Chikuzen]

Originally Posted by Myrsloik

Why not simply show this concept with a script? There aready is a built in function to calculate the average difference for a frame.

Like this?

#!/bin/env python3

import vapoursynth as vs
import sys

class SceneChangeDetect(object):
    def __init__(self, core):
        self.std = core.std
        self.resize = core.resize.Point
        self.th = 0

    def set_scene_change(self, n, f):
        fout = f[0].copy()
        fout.props._SceneChange = ((f[1].props.DiffPrev[0] > self.th) +
                                   (f[1].props.DiffNext[0] > self.th))
        return fout

    def detect(self, clip, th=0.0588, interval_h=None, interval_v=None):
        self.th = th
        if interval_h is None:
            interval_h = int(clip.width / 320)
        if interval_v is None:
            interval_v = int(clip.width / 320)
        curr = self.resize(clip, int(clip.width / interval_h),
                           int(clip.width / interval_v), vs.GRAY8)
        prev = curr[0] + curr
        next = curr[1:]
        curr = self.std.PlaneDifference([curr, prev], 0, 'DiffPrev')
        curr = self.std.PlaneDifference([curr, next], 0, 'DiffNext')
        return self.std.ModifyFrame([clip, curr], self.set_scene_change)

if __name__ == '__main__':
    core = vs.Core()
    core.std.LoadPlugin('G:/vsplugins/d2vsource_beta4_v2.dll')

    clip = core.d2v.Source('D:\HD_Source\sample_1080i_00.d2v", nocrop=True)
    clip = SceneChangeDetect(core).detect(clip)

    log = open('scenecange.log', 'w')
    for n in range(clip.num_frames):
        scene_change = clip.get_frame(n).props._SceneChange[0]
        if scene_change:
            print(n, file=log)
    log.close()

Because I’m used to thinking on C rather than Python

EDIT:
The true reason is for removing scene change detection from the code of temporalsoften(VS version).
I assumed that the biggest bottleneck of that is the part , and considered how to remove them.
As the results, I proposed adding _SceneChange to you, and wrote this plugin.
Since origin was written in C, I've never noticed the way written in Python.

[ajp_anton]

Originally Posted by IanB

Possibly a better model might be to tag the frames either side of a scene change, i.e. 0 frame is normal and in the middle of a scene, 1 frame is the end of a scene, 2 frame is the beginning of a scene, of course the sequence would always be ...,0,0,1,2,0,0,...

This would allow a random access to a temporal filter to avoid fetching a next/prior frame from the information of the current frame alone.


P.S. And Merry Christmas to all

What if there's a 1-frame "scene"?

[IanB]

For a 1-frame scene I might expect something like ...,0,0,1,3,2,0,0,... or based on Myrsloik's comments ...,0,0,1,1,1,0,0,...

I was more or less thinking about the processes and decisions inside a get a frame call. Assuming all the frames get accessed, the cache makes discarding a frame fetch from an adjacent scene pretty costless to the VS system. So I guess for the standard windowed kernel model :-

...
  for (int i = n-k; i < n+k; i++) {
    accumulate frame i processing
  }

just having the left to right scene change status can be sufficient. I was trying to think a little out of the box that maybe some right to left scene change status might be useful to someone at sometime. And yes just having a single bit flag is sufficient, you just have to get all the frames involved first then track the transitions.

Debate on further required.

[Chikuzen]

updated to 0.1.0-2

* add temporalsoften2

seems about +45% faster than temporalsoften(VS version).

[Chikuzen]

updated to 0.1.0-3

* add SSE2 intrinsic code to temporalsoften.c.

about x5.2 faster than 0.1.0-2

[Reel.Deel]

Hi Chikuzen, I was wondering if TemporalSoften2 only supports 8 bit for now? I tried a YUV420P16 source and it resulted in vertical stripes that alternate color with each frame.
Also, how about adding a "planes=[]" parameter? Seems like a good idea to me but you guys might think otherwise.

[Chikuzen]

Hmm, I'm investigating the cause now.

[Myrsloik]

I decided how to flag next/previous scenechanges. See the documentation. Basically there's one flag for next frame transition is a scenechange and one marking the previous transition as a scene change. (unset/0 of course means no scenechange)

[Chikuzen]

updated to 0.2.0

* change property's name from _SceneChange to _SceneChangePrev/_SceneChangeNext.
* fix temporalsoften2

now, tenporalsoften2 use SSE2 on 16bit process.

[Chikuzen]

Originally Posted by HolyWu

In the example of temporalsoften2's readme, it's still using props._SceneChange. Is that correct?

oops.
seems there are other typos, too.
This is sane version.
http://pastebin.com/H7Jfq184

[Chikuzen]

updated to 0.2.0-2
fixed in case rasius=1

[Chikuzen]

Originally Posted by HolyWu

Um...the first frame's _SceneChangePrev and the last frame's _SceneChangeNext is indicated as a scenechange. Is it intended or a mistake, since it has no previus/next frame to compare with?

intended.
_SceneChangePrev is always True at the first frame.
and _SceneChangeNext is always True at the last frame.

[lansing]

is there going to be a 64 bit version for this plugin?

[jackoneill]

Originally Posted by lansing

is there going to be a 64 bit version for this plugin?

There is now.
http://uloz.to/x6gvxpbB/scenechange-win64-7z

[lansing]

Originally Posted by jackoneill

There is now.
http://uloz.to/x6gvxpbB/scenechange-win64-7z

Thanks for the quick update