Detecting Scene Changes in Sports Broadcasts

[shindigo]

Hey All --

I'm looking for a good method to detect scene changes in a sports broadcast.

I've been playing around with MaskTools mt_motion to detect scene changes and would like some advice.

Here is the sample MOTD clip I have been working with: http://shindigo.net/sceneTest

Here is my attempt at a script to find the scene changes and show the scene number as the file plays:

LoadPlugin("C:\Program Files\AviSynth 2.5\MaskTools2.0a29\mt_masktools.dll")

global SceneNumber = 0
global lastSceneChangeFrame = 0
global Threshold3 = .001
global Diff3 = 0
global minNumFramesPerScene = 10

source = AVISource("MOTDClip.avi", false)
c = source.mt_motion(thT=35).Greyscale.mt_binarize()

return findScenes(c)

function findScenes(clip c)
{
last = c
c = c.ScriptClip(" Diff3 = AverageLuma
SceneNumber = SceneNumber + (((Diff3 <= Threshold3) && (Current_Frame > lastSceneChangeFrame + minNumFramesPerScene)) ? 1 : 0)
lastSceneChangeFrame = (((Diff3 <= Threshold3) && (Current_Frame > lastSceneChangeFrame + minNumFramesPerScene)) ? Current_Frame : lastSceneChangeFrame)
Subtitle(string(Diff3), align=8)
Subtitle(string(SceneNumber), align=2)
Subtitle(string(lastSceneChangeFrame), align=7)
")
return c
}


function FramesElapsed(int currentFrame)
{
return ((currentFrame > lastSceneChangeFrame + minNumFramesPerScene) ? true : false)
}

I'm looking at cases where the average luma of the frame is near zero because mt_motion will return a low-luma image when a scene change is detected. btw, my understanding of mt_motion is very limited. NOTE: the function FramesElapsed is not used.

Here are some things that I'm having trouble with:

The averageLuma of mt_motion(thT=35).Greyscale.mt_binarize() is the same for a scene change and when the image is static for a number of frames. What is a better way to isolate those frames that are scene changes?
Why is the luma (before mt_binarize) of the scene change frame always equal to the thT value that I gave?
This technique does not work when there is a dissolve from one scene to the next. Is there any way to improve this technique (or use another technique) to detect scene changes with dissolve?
Why can't I use the call to FramesElapsed inside of the ScriptClip?


Any help anyone could provide would be greatly appreciated.

Thanks,

sh-

[Eshkhmed]

...with MVTools by Fizick u can try implementation of something like that script^

# using MVTools
Stream=Last
global prv_sc=0
stream=stream.ReduceBy2.ReduceBy2
fw_v1= MVAnalyse(stream,blksize=8, search=2, isb = false, lambda=128, delta = 1, pel=2,searchparam=2, overlap=4, truemotion=true, sharp=2, idx = 1, dct=4)
mask1=MVSCDetection(stream,vectors=fw_v1)
WriteFileIF("Transitions.log", "YDifferenceFromPrevious(mask1) == 255 && YDifferenceFromPrevious(mask1) - YDifferenceToNext(mask1) == 0 && Current_Frame - prv_sc > 25 || Current_Frame == Framecount(mask1)", """string(prv_sc,"%06.0f")""" , """ "," """ , """string(Current_Frame-1,"%06.0f")""", "prv_sc=Current_Frame")

i think this is most precise method to detect scenechanges

[shindigo]

Eshkhmed,

Thanks for the excellent code. I have it working within a ScriptClip call to show the scene number on the video as the scenes change.

One problem: I can't get it to work inside the WriteFileIf.

In fact, I can't get WriteFileIf to work at all:

WriteFileStart is working fine, but WriteFile, WriteFileIf and WriteFileEnd do not write anything to the file. I thought it might be a problem with the file buffer not getting flushed, so I tried using flush=true:

stream.WriteFile("Transitions.log", "current_frame", flush=true)

To write every frame number and flush between every frame, but I still only get the header in the file.

Any idea what I'm doing wrong? btw, I'm using AVISyth 2.5.7

Thanks
sh-

[squid_80]

Try using the full path name when passing the filename to WriteFile() e.g. "stream.WriteFile("c:\blah\Transitions.log", "current_frame", flush=true)

[shindigo]

squid_80 -

Thanks for the advice. I still cannot get it to work. The example I am using is very simple:

fileName = "c:\frames.log"
source = AVISource("MOTDClip.avi", false)
source.WriteFileStart(fileName, """ "Frame List" """)
source.WriteFile(fileName, "current_frame", flush=true)
source.WriteFileEnd(filename, """ "Now the script is closed" """)
return source

This gives:
Frame List
Now the script is closed
in c:\frames.log

I'd be very grateful if you could give it a try and let me know what you get.

Thanks,

sh-

[squid_80]

fileName = "c:\frames.log"
source = AVISource("MOTDClip.avi", false)
source.WriteFileStart(fileName, """ "Frame List" """)
source.WriteFile(fileName, "current_frame", flush=true)
source.WriteFileEnd(filename, """ "Now the script is closed" """)
return source

The script isn't quite right. Try this:
fileName = "c:\frames.log"
source = AVISource("MOTDClip.avi", false)
source = WriteFileStart(source, fileName, """ "Frame List" """)
source = WriteFile(source, fileName, "current_frame", flush=true)
source = WriteFileEnd(source, filename, """ "Now the script is closed" """)
return source

[Wilbert]

I think it should be

fileName = "c:\frames.log"
source = AVISource("MOTDClip.avi", false)
source = WriteFileEnd(source, filename, """ "Now the script is closed" """)
source = WriteFile(source, fileName, "current_frame", flush=true)
source = WriteFileStart(source, fileName, """ "Frame List" """)
return source

because the conditional stuff is done backwards.

[shindigo]

Ah, 'tis a beautiful thing!

Both methods worked. I guess this example is so simple it doesn't demonstrate which order is the correct order if the Writes have variables that they share.

I thought some more about an example that might show the difference between the two suggestions. This is what I came up with:
fileName = "c:\frames.log"
source = BlankClip(length=9)
global i = 0
source = source.WriteFileStart(fileName, """ "Frame List, i=" + string(i) """, "i = i + 1")
source = source.WriteFile(fileName, """string(current_frame) + ", i=" + string(i)""", "i = i + 100", flush=true)
source = source.WriteFileEnd(filename, """ "Now the script is closed, i=" + string(i) """, "i = i + 10")
return source

The code with WriteFileStart first and WriteFileEnd last gives:
Frame List, i=0
0, i=11
1, i=111
2, i=211
3, i=311
4, i=411
5, i=511
6, i=611
7, i=711
8, i=811
Now the script is closed, i=1

The code with WriteFileEnd first and WriteFileStart last gives:
Frame List, i=10
0, i=11
1, i=111
2, i=211
3, i=311
4, i=411
5, i=511
6, i=611
7, i=711
8, i=811
Now the script is closed, i=0

Weird that either way, the code in WriteFileEnd gets executed before any of the WriteFileIf processing.

So if you wanted to use WriteFileEnd to put a footer showing the totals from the WriteFileIf statements, how would you do it? Not that I really need to know, but I am curious.

Thanks again guys - now I can move on with scene detection.

sh-

[squid_80]

because the conditional stuff is done backwards.
I don't understand; WriteFileStart does writing when it's added to the filter chain, WriteFileEnd does writing when the chain is deconstructed (script closed). The order isn't relevant.
So if you wanted to use WriteFileEnd to put a footer showing the totals from the WriteFileIf statements, how would you do it? Not that I really need to know, but I am curious.
I'm not sure you can do this. The expressions for writefilestart and writefileend are evaluated when the script is opened and parsed, not when the writing is done. ScriptClip and FrameEvaluate don't seem to support current_frame == -2 (for writefilexxx, current_frame == -1 occurs when the script is opened and current_frame == -2 occurs when the script is closed).

[Wilbert]

I don't understand; WriteFileStart does writing when it's added to the filter chain, WriteFileEnd does writing when the chain is deconstructed (script closed). The order isn't relevant.
You were right.

So if you wanted to use WriteFileEnd to put a footer showing the totals from the WriteFileIf statements, how would you do it? Not that I really need to know, but I am curious.
Don't know.

[shindigo]

I don't understand; WriteFileStart does writing when it's added to the filter chain, WriteFileEnd does writing when the chain is deconstructed (script closed). The order isn't relevant.

Doesn't my example show that the order is relevant? :confused:

With Start in the script before End the order of expression evaluation is:

Start
End
WriteFile

With End in the script before Start, the order of expression evaluation is:

End
Start
WriteFile

In both cases things get written to the file in Start->WriteFile->End order, but the order the expressions are evaluated in changes with the order of the statements.

Or am I missing out something?

[squid_80]

I was just replying to Wilbert's concern that the start/end filters were the wrong way around. The order isn't irrelevant in all cases (as you've shown) but it is in the scripts that Wilbert and I posted. Writefilestart and writefileend don't work like conditional filters, which is why it's hard to come up with an answer to your last question. (Maybe writefileend should have an after_frame parameter like scriptclip/frameevaluate to allow the expression to be evaluated at writing time rather than parsing time?)

[shindigo]

squid_80 -

Now that I've re-read your post I see that you were only talking about the "Write" part of WriteFileEnd.

Thanks for the clarification.

[scharfis_brain]

please provide an unaltered video sample that has not been recompressed. This AVI Sample is not a good base to start creating algotrithms from.

[shindigo]

scharfis_brain,

Thanks for your interest. Pardon my noobishness, but I'm not exactly sure how to create the sequence of edits I want without recompressing. :shame:

If I create an avs file with the clips joined as I want them should I save from vDub using Video>Direct Stream Copy to avoid recompression? Or do I need to do something else to create a sample of scenes from a larger file?

I thought that Direct Stream Copy would create an AVI with all I-frames and that wouldn't be what I wanted either.

TIA for any direction you can give.

sh-

[foxyshadis]

If you use avisynth, you must recompress. In fact, almost all NLEs will require that; only virtualdub won't (in which case you'll probably want to familiarize yourself with sylia script; the alternative of doing it manually is quite painful) or avidemux (which has no scripting). Recompression doesn't always have to kill quality, though, as long as you record carefully and compress well.

[scharfis_brain]

no just provide an unaltered video sample.

this means to

- either load your source AVI in VDub and use direct stream copy of ONE linear block of time. (don't stitich serveral scenes from throughout the wohle thing together)

- or to load your sample MPG in DGindex, mark the preferred scene and select "save project and demux video"

then upload the resulting file via rapdishare.com

[Eshkhmed]

let's go another way

@echo off
setlocal enabledelayedexpansion
set PluginsPath=c:\Videoedit\avisynth\
for /R %%x in (*.avi) do (
echo LoadPlugin^("%PluginsPath%MVTools.dll"^)
echo LoadPlugin^("%PluginsPath%mt_masktools.dll"^)
echo LoadPlugin^("%PluginsPath%DirectShowSource.dll"^)
echo DirectShowSource^("%%x"^)
echo ConvertToYV12^(Interlaced=true^)
echo Stream=Last
echo global sc_start=0
echo global sc_end=0
echo global prv_sc=0
echo stream_scd=stream.ReduceBy2.ReduceBy2
echo fw_v1= MVAnalyse^(stream_scd,blksize=8, search=1, isb = false, lambda=400, delta = 1, pel=2,searchparam=1, overlap=4, truemotion=true, sharp=1, idx = 1, dct=4^)
echo motion_mask=MVSCDetection^(stream_scd,vectors=fw_v1^)
echo Motion_Mask=Subtract^(motion_mask.trim^(1,0^), motion_mask^)
echo sc_detect^(motion_mask^)
echo #return FrameEvaluate^(FrameEvaluate^(ScriptClip^(motion_mask, "subtitle(difference)"^), "difference = String(text)"^), "text = AverageChromaU(motion_mask.trim(1,0))"^)
echo function sc_detect^(clip c^) {
echo global clip=c
echo c=WriteFileIF^(c, "%%~nx.log", "sc_start < sc_end && sc_end ^!!= prv_sc", """string(sc_start,"%%06.0f")""" , """ "," """ , """string(sc_end,"%%06.0f")""", "prv_sc=sc_end"^)
echo c=FrameEvaluate^(c,"global sc_end= (AverageChromaU(clip.trim(1,0)) >= 254 || (Current_Frame == FrameCount(clip)-1) ) ? Current_Frame+1 : sc_end"^)
echo c=FrameEvaluate^(c,"global sc_start = ( AverageChromaU(clip.trim(1,0)) <= 1.0 ) ? Current_Frame+1 : sc_start"^)
echo return c
echo }
) > "%%~nx.avs"
for /R %%x in (*.avs) do ( "c:\videoedit\tools\avs2avi.exe" "%%x" -p 0 -c "null" -o n )

don't forget change paths...

[shindigo]

scharfis_brain -

I put up an unrecompressed sample at http://shindigo.net/SceneTest Look for the link to MOTDClip2.avi. This file was compressed originally using DivX by gazzar at fbtz, so I don't have the uncompressed original.

This is easier for me than RapidShare, but perhaps you are not getting good throughput on the download. I would be interested to know what kind of speeds you get if you download this file.

The script that Eshkhmed originally gave worked well for me in detecting hard cut scene changes even on recompressed files.

I'm moving on to working on detecting scene changes that have dissolves. If you have any suggestions on how to tackle these I'd be interested to hear them.

My idea is to use the fact that most of the dissolves go either from a shot with the football pitch in it to a shot without the pitch in it or vice versa.

So I designed an algorithm that can look ahead n frames and detect whether the scene has moved dramatically toward or away from an ideal (u,v) chroma pair that is the average chroma when a scene shows the players on the pitch.

I'm still working on it, but I will post when it is done.

Eshkhmed -

Looks like your script will generate a series of avs files for finding scene changes in a set of avi files. This might also be very useful to me.

Thanks again,

sh-

[Eshkhmed]

...at last with this batch getting the best result, but it still very sensitive to high speed camera movement.. (sorry for my bad english) :

@echo off
setlocal enabledelayedexpansion
set PluginsPath=c:\Videoedit\avisynth\
for /R %%x in (*.avi) do (
echo LoadPlugin^("%PluginsPath%MVTools.dll"^)
echo LoadPlugin^("%PluginsPath%mt_masktools.dll"^)
echo LoadPlugin^("%PluginsPath%DirectShowSource.dll"^)
echo DirectShowSource^("%%x"^)
echo ConvertToYV12^(Interlaced=true^)
echo global sc_start=0
echo global sc_end=0
echo global prv_sc=0
echo Stream_scd=Last.Bob^(0,0.5^).ReduceBy2.crop^(8,8,-8,-8^)
echo bwc= MVAnalyse^(stream_scd,blksize=16, search=1, isb=true, lambda=0, delta = 1, pel=2,searchparam=1, overlap=8, truemotion=false, sharp=2, pnew=0, idx = 1, chroma=false, dct=4^)
echo fwc= MVAnalyse^(stream_scd,blksize=16, search=1, isb=false, lambda=0, delta = 1, pel=2,searchparam=1, overlap=8, truemotion=false, sharp=2, pnew=0, idx = 1, chroma=false, dct=4^)
echo inter = MVFlowInter^(stream_scd, bwc, fwc, time=50, ml=100, idx=1^)
echo stream_scd=Interleave^(stream_scd, inter^)
echo fw_v1= MVAnalyse^(stream_scd,blksize=8, search=3, isb=false, lambda=0, delta = 1, pel=2,searchparam=1, overlap=4, truemotion=false, sharp=2, pnew=128, idx = 2, dct=4^)
echo motion_mask=MVSCDetection^(stream_scd,vectors=fw_v1, thSCD2=156^)
echo Motion_Mask=Subtract^(motion_mask.trim^(1,0^), motion_mask^)
echo sc_detect^(motion_mask, stream_scd^)
echo #return FrameEvaluate^(FrameEvaluate^(ScriptClip^(overlay(stream_scd,motion_mask.crop^(0,0,0,-250^)^), "subtitle(chr(89)+String(YDifferenceFromPrevious(stream_scd)) + chr(32) + chr(32) + chr(32) + chr(76) + chr(83) + difference + chr(32) + chr(32) + chr(32) + chr(76) + chr(69) + string(log(YDifferenceFromPrevious(stream_scd)+0.1)+0.05) )"^), "difference = String(text)"^), "text = log(YDifferenceToNext(stream_scd)+0.1)+0.05"^)
echo function sc_detect^(clip c, clip d^) {
echo global clip=c
echo global src=d
echo c=WriteFileIF^(c, "%%~nx.log", "sc_start < sc_end && sc_end ^!!= prv_sc && sc_end - sc_start ^> 1", """string(sc_start,"%%06.0f")""" , """ "," """ , """string(sc_end,"%%06.0f")""", "prv_sc=sc_end"^)
echo c=FrameEvaluate^(c,"global sc_end= (AverageChromaU(clip.trim(-1,0)) >= 254 && YDifferenceFromPrevious(src) == 0 && ydtn_end_par >= 3.55 || (Current_Frame == FrameCount(clip)-1) ) ? Round((Current_Frame-2)/4) : sc_end"^)
echo c=FrameEvaluate^(c,"global sc_start = ( AverageChromaU(clip.trim(-1,0)) <= 1.0 && YDifferenceFromPrevious(src) > 32.0 && ydtn_start_par >= 3.55 ) ? Round((Current_Frame+3)/4) : sc_start"^)
echo c=FrameEvaluate^(c,"global ydtn_end_par = log(YDifferenceToNext(src)+0.001)+0.05"^)
echo c=FrameEvaluate^(c,"global ydtn_start_par = log(YDifferenceFromPrevious(src)+0.001)+0.05"^)
echo return c
echo }
) > "%%~nx.avs"
for /R %%x in (*.avs) do ( "c:\videoedit\tools\avs2avi.exe" "%%x" -p 0 -c "null" -o n )
@rem for /R %%x in (*.avs) do ( del /f /q "%%x")

[Starduster]

This is almost exactly what I'd like to do. However, I'm still very new at AVIsynth and am not sure where to put the write line. I'd like to only write the scene changes to the log file. The following code writes a line for every frame. I'd also like to write the time into the clip where the scene changes.

So, I have two questions:
1. How can I write only one line per scene change
2. How can I determine the time into the clip at scene change

Thanks so much for any help you can provide.

Steve

LoadPlugin("d:\AviSynth\mt_masktools.dll")

global SceneNumber = 0
global lastSceneChangeFrame = 0
global Threshold3 = .001
global Diff3 = 0
global minNumFramesPerScene = 10
global fileName = "d:\avisynth\frames.log"
global myLast=0

source = directshowsource("d:\avisynth\Encode72710AM.wmv")
source = ConvertToYV12(source, Interlaced = true)
c = source.mt_motion(thT=35).Greyscale.mt_binarize()

return findScenes(c)

function findScenes(clip c)
{
last = c
c = c.ScriptClip(" Diff3 = AverageLuma
SceneNumber = SceneNumber + (((Diff3 <= Threshold3) && (Current_Frame > lastSceneChangeFrame + minNumFramesPerScene)) ? 1 : 0)
lastSceneChangeFrame = (((Diff3 <= Threshold3) && (Current_Frame > lastSceneChangeFrame + minNumFramesPerScene)) ? Current_Frame : lastSceneChangeFrame)
Subtitle(string(Diff3), align=8)
Subtitle(string(SceneNumber), align=2)
Subtitle(string(lastSceneChangeFrame), align=7)
")
c = c.WriteFile(fileName, """string(Diff3) + "," + string(lastSceneChangeFrame) + "," + string(SceneNumber) + "," + string(Current_Frame)""", flush=true)
return c
}


function FramesElapsed(int currentFrame)
{
return ((currentFrame > lastSceneChangeFrame + minNumFramesPerScene) ? true : false)
}

[Starduster]

Well, I figured out the time issue, Current_Frame/ clip.Framerate.

c = c.WriteFile(fileName, """string(Diff3) + "," + string(lastSceneChangeFrame) + "," + string(SceneNumber) + "," + string(Current_Frame/ c.Framerate)""", flush=true)

But the writing only when the scene changes is still giving me a problem. Obviously, it would be easy to just check a variable and write only when the scene number changes but the correct syntax has escaped me so far.

I know there's no If/Then/Else but if there were, I'd write something like:
if (myLast != SceneNumber) {
myLast = SceneNumber
c = c.WriteFile(fileName, """string(Diff3) + "," + string(lastSceneChangeFrame) + "," + string(SceneNumber) + "," + string(Current_Frame/ c.Framerate)""", flush=true)
}

I just can't get the var = expression ? true : fase syntax right for doing the above. Any suggestions on this would be very much appreciated.

Thanks!

[IanB]

WriteFileIf(fileName, "myLast != SceneNumber", ...

[Starduster]

I'm feeling pretty stupid here... that looks so easy!! I know I'm doing something lame.

Unfortunately, same results. It writes a line for every frame. I've tried changing the != to <> to = and they all give the same results.

c = c.WriteFileIf(fileName, "myLast != SceneNumber", """string(Diff3) + "," + string(lastSceneChangeFrame) + "," + string(SceneNumber) + "," + string(Current_Frame/ c.Framerate)""", flush=true)
myLast = SceneNumber
return c

myLast and SceneNumber are both defined a global equal to 0. So what have I done?

Thanks for the reply... maybe one more? ;-)

Steve