Scene change detection

[Chikuzen]

scenechange-0.2.0-2.7z (http://www.mediafire.com/download.php?dnld4p98i333idp)

This is a scene change detection plugin for VapourSynth.
This plugin detects the scene changes of a clip and attaches _SceneChangePrev/_SceneChangeNext property (0 or 1) to all the frames.

I think that this will make it a bit easy to devellop temporal filters.

usage:
>>> import vapoursynth as vs
>>> core = vs.Core()
>>> core.std.LoadPlugin('/path/to/scenechange.dll')
>>> clip = something

>>> clip = core.scd.Detect(clip clip[, int thresh, int interval_h, int interval_v, data log])

Detect scene changes and attach _SceneChange properies to the clip.

thresh: The threshold of average of differences of the luma value between previous/next frames.
When the average of luma differences exceeds this, the frame is judged with a scene change.
(range is 1 to 254*2^(bitdepth - 8), default(or out of range) is assumed 15*2^(bitdepth-8))

interval_h: The horizontal interval of the pixel used for measurement of difference.
(1 to width, default(or out of range) is auto adjust)

interval_v: The vertical interval of the pixel used for measurement of difference.
(1 to height, default(or out of range) is auto adjust)

log: The name of the log file to output.
If this is set, the frame numbers applicable to a scene change are outputted as a text.
Describing full path is recommended.
(default is unset)

>>> clip = core.scd.ApplyLog(clip clip, data log)

Apply _SceneChangePrev/_SceneChangeNext properties to the clip based on the log outputted by scd.Detect.

note:
suppouted color families are GRAY(8/16bits) and YUV(8/9/10/16bits).

[Didée]

Still no better idea than the simple old threshold thing? - Looking at only the difference the difference between N-1 and N1 is moot (high motion etc.). Go second order. Build diff between N-1 and N, then look at how that diff is changing. The method has been there since years in SCSelect(), part of RemoveDirt.dll. It's not perfect either, of course, but much more robust than simple thresholding.

[Chikuzen]

My main purpose is to consider practical use of the frame properties of VapourSynth instead of a better scene change detection algorithm.
yes, the algorithm which this plugin is using is old and simple. If someone writes a better thing, I will use it.

BTW, I think that it is very difficult to port kassandro's plugins to VapourSynth.
His softwares are exactry Open Binary (http://www.vapoursynth.com/2012/10/open-binary-introducing-a-practical-alternative-to-open-source/).

[IanB]

_SceneChange property (0 or 1)
Possibly a better model might be to tag the frames either side of a scene change, i.e. 0 frame is normal and in the middle of a scene, 1 frame is the end of a scene, 2 frame is the beginning of a scene, of course the sequence would always be ...,0,0,1,2,0,0,...

This would allow a random access to a temporal filter to avoid fetching a next/prior frame from the information of the current frame alone.


P.S. And Merry Christmas to all

[Myrsloik]

Possibly a better model might be to tag the frames either side of a scene change, i.e. 0 frame is normal and in the middle of a scene, 1 frame is the end of a scene, 2 frame is the beginning of a scene, of course the sequence would always be ...,0,0,1,2,0,0,... if you go for the 0 1 2 0 system you also have to define 3 to mean scene change on both sides. Debate on. Im not sure which one will make script writing easier.

This would allow a random access to a temporal filter to avoid fetching a next/prior frame from the information of the current frame alone.


P.S. And Merry Christmas to all

The way I scribbled it down was to mark both the first and the last frame of a scene with 1. So it'd be 0 0 0 1 1 0 0 ... caching should make the multiple frame access mostly free anyway.

Why not simply show this concept with a script? There aready is a built in function to calculate the average difference for a frame.

[Chikuzen]

Why not simply show this concept with a script? There aready is a built in function to calculate the average difference for a frame.

Like this?

#!/bin/env python3

import vapoursynth as vs
import sys

class SceneChangeDetect(object):
def __init__(self, core):
self.std = core.std
self.resize = core.resize.Point
self.th = 0

def set_scene_change(self, n, f):
fout = f[0].copy()
fout.props._SceneChange = ((f[1].props.DiffPrev[0] > self.th) +
(f[1].props.DiffNext[0] > self.th))
return fout

def detect(self, clip, th=0.0588, interval_h=None, interval_v=None):
self.th = th
if interval_h is None:
interval_h = int(clip.width / 320)
if interval_v is None:
interval_v = int(clip.width / 320)
curr = self.resize(clip, int(clip.width / interval_h),
int(clip.width / interval_v), vs.GRAY8)
prev = curr[0] + curr
next = curr[1:]
curr = self.std.PlaneDifference([curr, prev], 0, 'DiffPrev')
curr = self.std.PlaneDifference([curr, next], 0, 'DiffNext')
return self.std.ModifyFrame([clip, curr], self.set_scene_change)

if __name__ == '__main__':
core = vs.Core()
core.std.LoadPlugin('G:/vsplugins/d2vsource_beta4_v2.dll')

clip = core.d2v.Source('D:\HD_Source\sample_1080i_00.d2v", nocrop=True)
clip = SceneChangeDetect(core).detect(clip)

log = open('scenecange.log', 'w')
for n in range(clip.num_frames):
scene_change = clip.get_frame(n).props._SceneChange[0]
if scene_change:
print(n, file=log)
log.close()


Because I’m used to thinking on C rather than Python :p

EDIT:
The true reason is for removing scene change detection from the code of temporalsoften(VS version).
I assumed that the biggest bottleneck of that is the part , and considered how to remove them.
As the results, I proposed adding _SceneChange to you, and wrote this plugin.
Since origin was written in C, I've never noticed the way written in Python.

[ajp_anton]

Possibly a better model might be to tag the frames either side of a scene change, i.e. 0 frame is normal and in the middle of a scene, 1 frame is the end of a scene, 2 frame is the beginning of a scene, of course the sequence would always be ...,0,0,1,2,0,0,...

This would allow a random access to a temporal filter to avoid fetching a next/prior frame from the information of the current frame alone.


P.S. And Merry Christmas to allWhat if there's a 1-frame "scene"?

[IanB]

For a 1-frame scene I might expect something like ...,0,0,1,3,2,0,0,... or based on Myrsloik's comments ...,0,0,1,1,1,0,0,...

I was more or less thinking about the processes and decisions inside a get a frame call. Assuming all the frames get accessed, the cache makes discarding a frame fetch from an adjacent scene pretty costless to the VS system. So I guess for the standard windowed kernel model :-...
for (int i = n-k; i < n+k; i++) {
accumulate frame i processing
}just having the left to right scene change status can be sufficient. I was trying to think a little out of the box that maybe some right to left scene change status might be useful to someone at sometime. And yes just having a single bit flag is sufficient, you just have to get all the frames involved first then track the transitions.

Debate on further required.

[Chikuzen]

updated to 0.1.0-2

* add temporalsoften2

seems about +45% faster than temporalsoften(VS version).

[Chikuzen]

updated to 0.1.0-3

* add SSE2 intrinsic code to temporalsoften.c.

about x5.2 faster than 0.1.0-2

[Reel.Deel]

Hi Chikuzen, I was wondering if TemporalSoften2 only supports 8 bit for now? I tried a YUV420P16 source and it resulted in vertical stripes that alternate color with each frame.
Also, how about adding a "planes=[]" parameter? Seems like a good idea to me but you guys might think otherwise. :p

[Chikuzen]

Hmm, I'm investigating the cause now.

[Myrsloik]

I decided how to flag next/previous scenechanges. See the documentation (http://www.vapoursynth.com/doc/apireference.html). Basically there's one flag for next frame transition is a scenechange and one marking the previous transition as a scene change. (unset/0 of course means no scenechange)

[Chikuzen]

updated to 0.2.0

* change property's name from _SceneChange to _SceneChangePrev/_SceneChangeNext.
* fix temporalsoften2

now, tenporalsoften2 use SSE2 on 16bit process.

[Chikuzen]

In the example of temporalsoften2's readme, it's still using props._SceneChange. Is that correct?

oops.
seems there are other typos, too.
This is sane version.
http://pastebin.com/H7Jfq184

[Chikuzen]

updated to 0.2.0-2
fixed in case rasius=1

[Chikuzen]

Um...the first frame's _SceneChangePrev and the last frame's _SceneChangeNext is indicated as a scenechange. Is it intended or a mistake, since it has no previus/next frame to compare with?

intended.
_SceneChangePrev is always True at the first frame.
and _SceneChangeNext is always True at the last frame.

[lansing]

is there going to be a 64 bit version for this plugin?

[jackoneill]

is there going to be a 64 bit version for this plugin?

There is now.
http://uloz.to/x6gvxpbB/scenechange-win64-7z

[lansing]

There is now.
http://uloz.to/x6gvxpbB/scenechange-win64-7z

Thanks for the quick update:)

[jose1711]

hi, is this the correct place to report build issues for this plugin?

[jackoneill]

hi, is this the correct place to report build issues for this plugin?

It is.

[an3k]

What's the difference between the included libtemporalsoften2 and jackoneills https://github.com/dubhater/vapoursynth-temporalsoften ? Do I get any interference when both are installed? If so which one should be used?

[Myrsloik]

What's the difference between the included libtemporalsoften2 and jackoneills https://github.com/dubhater/vapoursynth-temporalsoften ? Do I get any interference when both are installed? If so which one should be used?

It's exactly the same

[jackoneill]

What's the difference between the included libtemporalsoften2 and jackoneills https://github.com/dubhater/vapoursynth-temporalsoften ? Do I get any interference when both are installed? If so which one should be used?

There is no interference. Chikuzen's version should be faster. Maybe it has additional functionality as well. I don't remember.

[lansing]

How do I get all the frame numbers of the start of the scene change?

[VS_Fan]

How do I get all the frame numbers of the start of the scene change?
This should do the trick:
import vapoursynth as vs
core = vs.get_core()

ret = core.ffms2.Source(source="video file name")

#ret = core.scd.Detect(clip=ret, thresh=40, log="SC_SCD.TXT")#, interval_h, interval_v)
ret = core.misc.SCDetect(clip=ret, threshold=0.15)

fstats = open ('SC.txt', 'w', encoding='utf-8', errors='strict')
fstats.write('frame_no\n')
for i in range (ret.num_frames):
if ret.get_frame(i).props._SceneChangePrev == 1 :
fstats.write('{}\n'.format(i))
fstats.close()

ret=core.text.FrameProps(clip=ret, alignment=3)#, props
ret.set_output()

It will scan the whole video, create the 'SC.txt' text file, write to it the list of frames detected by the misc.SCDetect filter as start of a scene change, then it will close the file when finished and only after that it will show any video if run from vsedit.

So give it a while, and when it finishes, rename or copy your new 'SC.txt' text file with the frame numbers to avoid rewritting it.

[lansing]

What is the difference between this scene change detection filter and the one in the misc? Right now I'm using the loop from the documentation to loop through the video, which way is faster?


clip = core.scd.Detect(clip, thresh=25)

for frame in clip.frames():
if frame.props._SceneChangePrev == 1 and frame.props._SceneChangeNext == 0:
frame_num = round(frame.props._AbsoluteTime * clip.fps)
#do stuff

[Myrsloik]

What is the difference between this scene change detection filter and the one in the misc? Right now I'm using the loop from the documentation to loop through the video, which way is faster?


clip = core.scd.Detect(clip, thresh=25)

for frame in clip.frames():
if frame.props._SceneChangePrev == 1 and frame.props._SceneChangeNext == 0:
frame_num = round(frame.props._AbsoluteTime * clip.fps)
#do stuff


The misc version is basically a faster replacement of the scd version. It should behave identically.

That's a really awkward way of writing things:
clip = ...

for n in range(length(clip)):
frame = clip.get_frame(n)
if frame.props['_SceneChangePrev'] == 1 and frame.props['_SceneChangeNext'] == 0:
frame_num = n
#do stuff

And another note about speed. Any script that can be run through vspipe will be faster due to multiple frame requests being possible at once.

[lansing]

um both filters didn't do well on scene change in darker areas, I have to set threshold to 0.02 in order to see them.

[Myrsloik]

um both filters didn't do well on scene change in darker areas, I have to set threshold to 0.02 in order to see them.

Of course not, they're based on a simple absolute difference. You need something fancier to get good results everywhere. Try scxvid?

[lansing]

Of course not, they're based on a simple absolute difference. You need something fancier to get good results everywhere. Try scxvid?

I just tried it, I clicked on preview in the editor, and after a minute or two of loading, the preview window popped up and the whole program frozed.