Log in

View Full Version : Clean up dirty scene changes


Bexley
6th May 2011, 01:23
This may be a really dumb question, but.....

I'm trying to restore an old movie ( 24FPS film) where the vast majority of scene changes are just nasty. I'm not sure if it was a fault in the original editing where it was only expected to be seen once on a projection screen, or if it's a problem in the telecine process for DVD. Either way, the long and short of it is that the frame immediately before and immediately after every scene change needs to be replaced.

I started out trying to just delete the frames, but I quickly realized that in an 80-minute movie that's a LOT of frames and there are audio consequences in some places from deleting just 2 frames. So my next thought was to replace the frames with the adjacent frames. I started out with FreezeFrame, but that's WAY too time consuming so I'm currently using this script that I found in another thread to do that. It's pretty good.

Function RestoreBadFrames(clip objVideo)
{

Global objClip = objVideo

objVideo = objVideo.ScriptClip( "AnalyzeBadFrames()")
objVideo = objVideo.FrameEvaluate( "global intFrameNumber = current_frame")
objVideo = objVideo.FrameEvaluate( "global fltDiffFromNext = 0.50*YDifferenceToNext(objClip) + /
0.25*UDifferenceToNext(objClip) + 0.25*VDifferenceToNext(objClip)")
objVideo = objVideo.FrameEvaluate( "global fltDiffFromPrevious = 0.50*YDifferenceFromPrevious(objClip) /
+ 0.25*UDifferenceFromPrevious(objClip) + 0.25*VDifferenceFromPrevious(objClip)")

Return objVideo
}

Function AnalyzeBadFrames(clip objVideo)
{
objVideo = ( fltDiffFromPrevious < 0.25 && fltDiffFromNext < 0.25 ) ? /
objVideo.FreezeFrame(intFrameNumber,intFrameNumber,intFrameNumber-1) : objVideo

objVideo = ( fltDiffFromPrevious > 24.0 ) ? objVideo.FreezeFrame(intFrameNumber, /
intFrameNumber, intFrameNumber + 1) : objVideo
objVideo = ( fltDiffFromNext > 24.0 ) ? objVideo.FreezeFrame(intFrameNumber, /
intFrameNumber, intFrameNumber - 1) : objVideo

Return objVideo
}

This is a huge improvement, but I'm still not quite happy because while on low-motion scenes it's fine, it creates a slight stutter because of the duplication on higher action scenes.

So here's the (probably) dumb question. Is it possible to actually interpolate new frames at a scene change where you don't have both a previous and next frame for reference? I mean something like this:

ABCD | EFGH

Where D would look at the motion over frames AB&C and make a calculated guess of what D would be, and then apply the same process in reverse for E? (i.e., examine the motion in reverse for HG&F and then calculate a likely E)

Possible? Or am I just dreaming?

mastrboy
6th May 2011, 16:55
Seems we are currently dealing with the exact same problem on different sources. I wrote i similear script to replace those exact same scenechange frames for a anime: http://forum.doom9.org/showthread.php?t=161046

i think you should be able to create a "fake" frame with mflowinter and some tricks, but this is beyond my skills.
From the manual of mvtools:
MFlowinter:
Motion interpolation function. It is not the same (but similar) as MVInterpolate function of older MVTools version. It uses backward "mvbw" and forward "mvfw" motion vectors to create picture at some intermediate time moment between current and next (by delta) frame. It uses pixel-based (by MFlow method) motion compensation from both frames. Internal forward and backward occlusion masks (MMask kind=2 method) and time weighted factors are used to produce the output image with minimal artefactes. True motion estimation is strongly recommended for this function.

http://forum.doom9.org/showthread.php?t=161046 seems to have been deleted, pasting in the script here if someone comes looking into this ancient thread:


#################
# Version: 0.2 by mastrboy
# Purpose: Removing or trying to repair "unclean/broken" frames in beetween a scenechange
# How it works:
# replacing frame before and after a scene change with
# previous/next neigbhoor frame which hopefully is clean, but only with a frame similear enough (defined by SameDiff)
#
# (Only been tested on anime, for live content it might not be suitable)
##################
function SCReplaceFrame(clip input,int "SCDiff", int "SameDiff", int "dfactor",bool "chroma", bool "show", bool "replace", bool "inter_repair", bool "use_scselect",bool "logging") {
SCDiff = default(SCDiff,50)
SameDiff = default(SameDiff,8)
dfactor = default(dfactor,2)
chroma = default(chroma,true)
show = default(show,false)
replace = default(replace,true)
inter_repair = default(inter_repair,false)
use_scselect = default(use_scselect,true)
logging = default(logging,false)
filename = "SCReplaceFrame.log"

sclip = GScriptclip(input,args="SCDiff, SameDiff, chroma, show, replace, inter_repair, logging, filename","""
# Create diff variables
DiffToNext = (chroma==true) ? YDifferenceToNext() + UDifferenceToNext() + VDifferenceToNext() : YDifferenceToNext()
DiffFromPrevious = (chroma==true) ? YDifferenceFromPrevious() + UDifferenceFromPrevious() + VDifferenceFromPrevious() : YDifferenceFromPrevious()

# Check if frame is to be replaced or not
replace_frame_with_prev = (DiffToNext > SCDiff && DiffFromPrevious < SameDiff) ? true : false
replace_frame_with_next = (DiffToNext < SameDiff && DiffFromPrevious > SCDiff) ? true : false

# Show difference on frame if true
(show==true) ? subtitle("Current Frame: "+string(current_frame) + "\nDiffToNext: " + string(DiffToNext)+"\nDiffFromPrevious: " + string(DiffFromPrevious)+"\n\nScenechange treshold: >"+string(SCDiff)+"\nSimilear frame treshold: <"+string(Samediff),align=7,lsp=10) : last
(show==true && replace_frame_with_prev==true) ? subtitle("Replace/Repair with prev frame: " + string(current_frame-1),align=9,lsp=2) : last
(show==true && replace_frame_with_next==true) ? subtitle("Replace/Repair with next frame: " + string(current_frame+1),align=9,lsp=2) : last

# write to log file
(replace_frame_with_prev==true && logging==true) ? WriteFile(filename, "current_frame") : last

# replace frames only if replace = true and inter_repair = false
(replace_frame_with_prev==true && replace==true && inter_repair==false) ? trim(0,- current_frame) : last
(replace_frame_with_next==true && replace==true && inter_repair==false) ? trim(1,0) : last

# An attempt to repair scene only if replace = false and inter_repair = true
super = MSuper()
backward_vectors = MAnalyse(super, isb = true, delta=1)
forward_vectors = MAnalyse(super, isb = false, delta=1)

inter_backward = MFlowInter(super, backward_vectors, forward_vectors, time=50, ml=70,blend=false)
inter_forward = MFlowInter(super, backward_vectors, forward_vectors, time=50, ml=70,blend=false)

repaired_backward = repair(inter_backward,last)#.subtitle("Repaired/Interpolated back",align=1)
repaired_forward = repair(inter_forward,last)#.subtitle("Repaired/Interpolated for",align=1)

(replace_frame_with_prev==true && replace==false && inter_repair==true) ? trim(repaired_backward,current_frame-1,-1) : last
(replace_frame_with_next==true && replace==false && inter_repair==true) ? trim(repaired_forward,current_frame,-1) : last

""")

select_prev = sclip.selectevery(1,0)
select_next = sclip.selectevery(1,0)
output = (use_scselect==true) ? input.SCSelect(select_next,select_prev,input,dfactor=dfactor) : sclip

return output
}

function repair_scene(clip input, int "frame_number") {
frame_number = default(frame_number,1)

super = MSuper(input)
backward_vectors = MAnalyse(super, isb = true, delta=1)
forward_vectors = MAnalyse(super, isb = false, delta=1)

inter_backward = MFlowInter(input, super, backward_vectors, forward_vectors, time=50, ml=70,blend=false)
inter_forward = MFlowInter(input, super, backward_vectors, forward_vectors, time=50, ml=70,blend=false)

repaired_backward = repair(inter_backward,input)#.subtitle("Repaired/Interpolated back",align=1)
repaired_forward = repair(inter_forward,input)#.subtitle("Repaired/Interpolated for",align=1)

output = trim(input,0,frame_number - 1) + trim(repaired_backward,frame_number - 1,-1) + trim(repaired_forward,frame_number + 1,-1) + trim(input,frame_number+2,0)

return output
}

function repair_scene2(clip input, int "frame_number") {
frame_number = default(frame_number,1)

sclip = GScriptclip(input,args="input,frame_number","""
super = MSuper(input)
backward_vectors = MAnalyse(super, isb = true, delta=1)
forward_vectors = MAnalyse(super, isb = false, delta=1)

inter = MFlowInter(input, super, backward_vectors, forward_vectors, time=50, ml=70,blend=false)
repaired = repair(inter,input)

back_repaired = repaired.selectevery(1,-1).subtitle("Repaired back",align=1)
forward_repaired = repaired.subtitle("Repaired forward",align=1)
output = ConditionalFilter(input, back_repaired, input, "current_frame", "==", "frame_number")
output = ConditionalFilter(input, forward_repaired, output, "current_frame", "==", "frame_number+1")
output

""")

return sclip
}

function repair_scene3(clip input, int "direction") {
direction = default(direction,1)
# 1 = back , 2 = forward

sclip = GScriptclip(input,args="input,direction","""
super = MSuper(input)
backward_vectors = MAnalyse(super, isb = true, delta=1)
forward_vectors = MAnalyse(super, isb = false, delta=1)

inter = MFlowInter(input, super, backward_vectors, forward_vectors, time=50, ml=70,blend=false)
repaired = repair(inter,input)

back_repaired = repaired.selectevery(1,-1)#.subtitle("Repaired back",align=1)
forward_repaired = repaired#.subtitle("Repaired forward",align=1)
output = ConditionalFilter(input, back_repaired, forward_repaired, "direction", "==", "1")
output
""")

return sclip
}

# Alternative version
function SCRepair(clip input,bool "chroma",string "outputdir") {
chroma = default(chroma,true)
BlockChangeThresh = 500 # Increase to reduce number of scenes detected
Num_blocks_changed = 130 # Increase to reduce number of scenes detected
crop_value = input.height - input.height/100*10 # Crop to % of image

super = MSuper(input,pel=2)
backward_vectors = MAnalyse(super, isb = true, delta = 1, blksize=16)
scenechange = MSCDetection(input, backward_vectors, thSCD1=BlockChangeThresh, thSCD2=Num_blocks_changed)

#msc_eval = ConditionalFilter(image, input, "AverageLuma(SceneChange)", "greaterthan", "250",show=false)
#image = ImageWriter(ConvertToRGB24(input),"d:\SC\", type="jpeg").ConvertToYV12()
#diff_eval = ConditionalFilter(msc_eval, input, "YDifferenceFromPrevious() + UDifferenceFromPrevious + VDifferenceFromPrevious", "lessthan", "14",show=false)
#cropped = crop(0,crop_value,0,0)
#crop_eval = ConditionalFilter(cropped,diff_eval, input, "YDifferenceFromPrevious() + UDifferenceFromPrevious + VDifferenceFromPrevious", "greaterthan", "12",show=false)
sclip = GScriptclip(input,args="input,crop_value,chroma,super,backward_vectors,scenechange,outputdir","""
DiffToNext = (chroma==true) ? YDifferenceToNext() + UDifferenceToNext() + VDifferenceToNext() : YDifferenceToNext()
DiffFromPrevious = (chroma==true) ? YDifferenceFromPrevious() + UDifferenceFromPrevious() + VDifferenceFromPrevious() : YDifferenceFromPrevious()

forward_vectors = MAnalyse(super, isb = false, delta=1,blksize=16)
inter = MFlowInter(super, backward_vectors, forward_vectors, time=50, ml=70,blend=false)
repaired = repair(inter,input)
back_repaired = repaired.selectevery(1,-1).subtitle("Repaired back",align=1)
forward_repaired = repaired.subtitle("Repaired forward",align=1)

cropped = crop(input,0,crop_value,0,0)
crop_eval = (YDifferenceFromPrevious(cropped) + UDifferenceFromPrevious(cropped) + VDifferenceFromPrevious(cropped) > 12) ? true : false
diff_eval = (DiffToNext > 28 && DiffFromPrevious < 14) ? true : false
msc_eval = (AverageLuma(SceneChange) > 250) ? true : false

#eval3 = ConditionalFilter(input, back_repaired, input, "crop_eval", "==", "true")
image = ImageWriter(ConvertToRGB(),string(outputdir)+"\", current_frame, current_frame, "jpeg",true).ConvertToYV12
eval3 = ConditionalFilter(input, image, input, "crop_eval", "==", "true")
eval2 = ConditionalFilter(input, eval3, input, "diff_eval", "==", "true")
eval1 = ConditionalFilter(input, eval2, input, "msc_eval", "==", "true")

output = eval1
output
""")

output = input.SCSelect(sclip,sclip,input,dfactor=1.8)
return output
}

Didée
6th May 2011, 20:28
For the basic job of finding scenechanges & replacing their last/first frames, you could also use SCSelect (http://forum.doom9.org/showthread.php?p=1140719#post1140719).

About using motion compensated replacements - that's possible, but the strategy depends on the actual case. If the dirty frames are not all too dirty, then a direct motionsearch+compensation could be done. But if those frames are severely damaged, then it's more problematic. One could try to play tricks, like searching motion between (N-2)->(N-1), and apply the vectors to (N-1). Would work if the motion is mostly "global", but will probably produce funky results if there's "local" motion of smaller objects.

A proper method of motion extrapolation is not offered by MVTools, AFAIK.

mastrboy
7th May 2011, 22:35
Edit: Nevermind, figured it out:

source = last
prev = source.selectevery(1,-1).WriteFile("test.txt", "current_frame")
next = source.selectevery(1,1)
source.SCSelect(next,prev,source,dfactor=2.0)


BTW, why can WriteFile call 'current_frame' outside of scriptclip when no other functions can?

is there a way to call WriteFile() combined with Scriptclip and the variable "current_frame" with SCSelect to write a log file of detected scenes?

Gavino
7th May 2011, 23:52
why can WriteFile call 'current_frame' outside of scriptclip when no other functions can?
Because WriteFile is itself a run-time (http://avisynth.org/mediawiki/Runtime_environment) filter (like ScriptClip).

is there a way to call WriteFile() combined with Scriptclip and the variable "current_frame" with SCSelect to write a log file of detected scenes?
Something like this should work (ScriptClip not needed):
source = last
scbegin = source.WriteFile("log.txt", """ "Start: " """, "current_frame")
scend = source.WriteFile("log.txt", """ "End: " """, "current_frame")
source.SCSelect(scbegin,scend,source,dfactor=2.0)

Bexley
8th May 2011, 18:25
mastrboy, I like your script a lot better than the one I've been using, if for no other reason than the "show" argument makes it incredibly easy to tweak the settings. Because the frames in my source are so funky at the change, I had to bump the SameDiff up to over 30, and in a few cases over 50. It just feels instinctively wrong to put it that high, but it seems to be necessary in my case.

One problem I have run into with your script is that I need to apply different thresholds to different parts of the film but your function seems to want to apply whatever the last value is to the entire source.

I have a jogging sequence at the beginning of the film that is very high motion, and it needs completely different thresholds than the rest of the film. I tried to set it up like this:

a=Trim(0,1396)
b=Trim(1397,12643).SCReplaceFrame(SCDiff=58,SameDiff=54,show=true)
c=Trim(12644,0).SCReplaceFrame(SCDiff=44,SameDiff=33,show=true)
AlignedSplice(a,b,c)


But it applies the c values to the entire movie, causing my jogging scene to not work at all. If I remove the SCReplaceFrame function on clip c, it works again like it should on b. Am I missing something simple?

Didee - in my case these frames are pretty funky, but the defects tend to be in brightness/contrast and chroma areas. The bottom half of the frame will have blown out whites or suddenly drop in brightness, or will have some reddish tinting in some cases. The actual image is OK, although there are a few with really bad scratches and what appears to be glue or something on the film at the splice.

I'm really not very experienced at all with MVTools. What might the sort of function you are talking about look like?

Gavino
8th May 2011, 18:58
mastrboy, ...
One problem I have run into with your script is that I need to apply different thresholds to different parts of the film but your function seems to want to apply whatever the last value is to the entire source.
This is because of the global variables.
See my post here.

mastrboy
8th May 2011, 20:31
And I'm currently working on an updated version that utilize Gscript instead. Might post it later tonight if I'm done.

Edit: updated to v0.2 with some minor added features.

pbristow
9th May 2011, 22:20
Didee: Regarding extrapolation (rather than interpolation), have you seen this thread? http://forum.doom9.org/showthread.php?p=1498155#post1498155