Lining up scenes, cut detection (e.g. for dubbing)

[mg262]

LineUpScenes (http://people.pwf.cam.ac.uk/mg262/posts/LineUpScenes_280505.dll)

Brief description:
Take a list of scenes from two different clips and match them up, detecting & displaying cuts. AFAIK this is only useful for dubbing.

Search keywords: match, matching, dub, dubbing, cut, scene

Full description/Example:

First take the two files (which must have same frame rate) and generate text lists like this:
length of clip
framenumber of 1st frame of scene 1 (always 0)
framenumber of 2nd frame of scene 2
framenumber of 3rd frame of scene 2
...

(Screenshot (http://people.pwf.cam.ac.uk/mg262/posts/scenelist.jpg). See note below on generating files like this.)

Now run this script:

function scaleandnumber(clip c)
{
c
crop(0,0,floor(c.width/32)*32,0) #to make width a multiple of 32
reduceby2.reduceby2
overlay(crop(0,0,-0,26).showframenumber)
return reduceby2
}

function strip(clip i)
{
i
a=selectevery(8,0)
b=selectevery(8,1)
c=selectevery(8,2)
d=selectevery(8,3)
e=selectevery(8,4)
f=selectevery(8,5)
g=selectevery(8,6)
h=selectevery(8,7)

stackhorizontal(a,b,c,d,e,f,g,h)
stackvertical(last, trim(1,0), trim(2,0), trim(3,0), trim(4,0), trim(5,0))
}
#Load sources 'vhs', 'dvd' (720x576 in my example)
LineUpScenes("vhs scene.txt", "dvd scene.txt", vhs.scaleandnumber, dvd.scaleandnumber)
strip

(Clips must be YV12. N.B. the clips are not used for matching, just to provide a visual aid.)

The results will look something like this:
http://people.pwf.cam.ac.uk/mg262/posts/linedup.jpg
The left-hand half of the screen corresponds to the VHS source, and the right hand half to the DVD source. A tab-separated text version of this has been put in the file 'lineup.txt', like this:

...
11085...11135 11804...11854 719
11136...11196 11855...11915 719
11197...11253 11916...11972 719
11254...11301 11973...12020 719
11302...11371 12021...12090 719
11372...11429 12091...12149 720
...

The last number ('719') gives the displacement between the two scenes - so e.g. 11854 = 11135 + 719. As long as it stays the same, we know there hasn't been a cut. Where there is a cut you see something like this (the tab-separated file opens directly in a spreadsheet):

http://people.pwf.cam.ac.uk/mg262/posts/linedup cut spreadsheet.jpg

By looking at the relevant frames in the 'graphical' view (i.e. the output clip), we can see exactly where the cut happened:
http://people.pwf.cam.ac.uk/mg262/posts/linedup cut.jpg

Notes:

- It's fairly straightforward to use this for dubbing -- you just need to create a list of Trims like this:
Trim(3,2132+3) \
+Trim(2133+81, 2922+81) \
+Trim(2923+174, 4096+174) \
+Trim(4097+295, 9359+295) \
...

There will be one trim for each run of numbers (i.e. 719, 1007, 927 above each give a trim.) I can describe how to calculate the numbers if anyone wants.

- I'm not going to discuss scene detection here. IMO, what method you should use depends on what source you have. But if you really have no other way of doing it, you can use this iSSE filter (http://people.pwf.cam.ac.uk/mg262/posts/RoughScene_280505.dll), which I excerpted/hacked/special-cased from the main filterset that I'm writing. Just write clip.RoughScene("output_filename.txt", 0.75). (Lowering the 0.75 will detect more scenes, etc.) I have just put this here to support LineUpScenes; it's not a proper tool + shouldn't be treated as such.

- The reason the 'displacement' fluctuates (e.g. 1007<-->1008) in my example is because the VHS stream has been fps-converted and then converted back (using restore24).

- LineUpScenes method and (messy! messy!) source on request.

[AVIL]

Hi

Your filter do the job (nothing new to you, i supose). I give it a try with two captures of the same video. Different number of dummy frames before the true begin and different number and position of the dropped fames. Alas, your filter has arrived after i have done the alignement manually, but i can confirm that it do correct frame matching ( same matching as mine with subtract() and patience ). In my next project with two captures, your filter will do the job. I give you puntually notice of the result.

Best regards.

[mg262]

@AVIL -- it's really great to know someone else is finding it useful! Thank you very much for letting me know.