[solved] syncing clips with speed variation

[raffriff42]

I had to sync two clips recently where one of the clips sometimes ran way ahead, other times way behind. To get it to work I needed to sync the two clips at many intermediate points. After much @#$%& I arrived at this solution. It works well enough for me! Hope you like it too! Thanks to Gavino for GScript, which made this code possible, or at least much, much easier!

> > > Not a script for beginners! < < <

Usage:

• Frame rates of the 2 clips must match! (use ConvertFPS etc as required)
• Intra-frame-only sources are a plus! Without them, watch out for possible out-of-order frames.

1. Set bypass=true (in blue below) to disable any speed changes.
   (note - I use "1>0" for "true" and "0>0" for "false" - it saves typing)
2. Create two comma-delimited lists of frame numbers (sync points), one for each video;
   (clips must be approximately in sync @ frame 0)
   (first sync point must be > 0 for both clips)
3. Get sync at any useful point which is common to the 2 clips;
   (how you get these sync points depends on the source material - the classic method is to use a clapperboard)
4. Once you have found a your sync point, add the frame numbers to each list.
5. Add more sync points until clip synchronization is "good enough"
   (note - since we are doing linear interpolation, correcting for nonlinear speed variations can never be exact)
   (note - sync points must be monotonic - in increasing order)
   (any video remaining after the last sync point simply plays at normal speed)
6. Set bypass=false to enable the speed changes and see the results.
7. Review, adjust and maybe add sync points as needed.
8. There will be some unavoidable audio artifacts in the sync-corrected clip; try to use audio from the reference clip only.

#Import("uusync.avsi")

clpRef=XXSource("...")
clpSyn=XXSource("...") ## we want to match sync to clpRef

## two lists of frame numbers, one for each video
sSyn="207,1182,2650,3251,4021,4577,5039,6416"
sRef="212,1218,2715,3312,4082,4640,5100,6451"

clpSyn_2 = UUSyncSegments(clpSyn, clpRef, sSyn, sRef, debug=1>0, bypass=0>0)
return StackHorizontal(clpRef, clpSyn_2)

Debug output is useful for finding what the nearest sync points are and what speed adjustment is being applied.

The clips in the samples below are still out by a frame or two; that's because I rushed getting my sync points.
(this was a test video with random speed changes intentionally added)





uusync.avsi

# http://forum.doom9.org/showthread.php?t=147846
#LoadPlugin("GScript\GScript.dll")

GScript("""
#######################################
### sync 'syn' clip to 'ref' based on multiple common markers
## requires GScript http://forum.doom9.org/showthread.php?t=147846
##
## NOTE: 
## - frame rates must match (use AssumeFPS/ChangeFPS etc)
## - clips must be approximately in sync @ frame 0
## - first sync point must be > 0 for both clips
##
## @ syn     - clip to be synced
## @ ref     - reference clip
## @ ssyn    - 'syn' sync points (comma-delimited list)
## @ sref    - 'ref' sync points
## @ pitchfix - if true, maintain audio pitch with speed 
##                (default false; allow pitch to rise or fall)
## @ noblend - if true, do not blend frames (default true)
## @ debug   - if true, show debugging info (default false)
## @ bypass  - if true, pass 'syn' unchanged to output
##
## @ return clip 'syn' with speed adjusted to sync to clip 'ref'
##
## @version 1.0 22-Oct-2015 raffriff42
## @version 1.1  2-Dec-2015 head & tail handling; debug output; pitchfix 
## @version 1.2  3-Dec-2015 handling of first sync point
##
function UUSyncSegments(clip syn, clip ref, string sSyn, string sref, 
\           bool "pitchfix", bool "noblend", bool "debug", bool "bypass")
{
    Assert(ref.FrameRateNumerator==syn.FrameRateNumerator
    \   && ref.FrameRateDenominator==syn.FrameRateDenominator,
    \     "UUSyncSegments: clips 'syn' and 'ref' must have same framerate")

    pitchfix = Default(pitchfix, false)
    noblend  = Default(noblend, true)
    debug    = Default(debug, false)
    bypass   = Default(bypass, false)

    ## chop list into "_L" (left) and "_R" (right) parts
    ps     = FindStr(ssyn, ",")
    pr     = FindStr(sref, ",")
    ssyn_L = (ps==0) ? ssyn : LeftStr(ssyn, ps-1)
    ssyn_R = (ps==0) ?  ""  : MidStr(ssyn, ps+1)
    sref_L = (pr==0) ? sref : LeftStr(sref, pr-1)
    sref_R = sref
    
    ## get first sync point
    fsyn_L = Round(Value(ssyn_L))
    fref_L = Round(Value(sref_L))

    Assert(fsyn_L > 0 && fref_L > 0, 
    \   "UUSyncSegments: first sync point must be > 0 for both clips")
    Assert(fsyn_L < syn.FrameCount, 
    \   "UUSyncSegments: 'syn' frame number >= FrameCount")
    Assert(fref_L < ref.FrameCount, 
    \   "UUSyncSegments: 'ref' frame number >= FrameCount")

    syn = (!debug) ? syn 
    \   : syn.ShowFrameNumber(x=16, y=syn.Height-16)

    factor = Float(fsyn_L) / Float(fref_L) 
    R = syn.Trim(0, fsyn_L-1)
    \      .UUChangeSpeed2(factor, pitchfix, noblend)

    R = (!debug) ? R : R.Subtitle(
    \        "syn in, out: 0, " + String(fsyn_L)
    \    + "\nref in, out: 0, " + String(fref_L)
    \    + "\nspeed mult.: " + String(factor, "%0.3f")
    \    , font="courier", size=R.Height/16, lsp=0)

    count = 256 ## infinite loop prevention
    while (count>0 && StrLen(ssyn_R)>0) {

        fsyn_L = Round(Value(ssyn_L))
        fsyn_R = Round(Value(ssyn_R))

        pr     = FindStr(sref_R, ",")
        sref_L = (pr==0) ? sref_R : LeftStr(sref_R, pr-1)
        sref_R = (pr==0) ?   ""   : MidStr(sref_R, pr+1)

        fref_L = Round(Value(sref_L))
        fref_R = Round(Value(sref_R))

        Assert(fsyn_L < syn.FrameCount && fsyn_R < syn.FrameCount, 
        \   "UUSyncSegments: 'syn' frame number >= FrameCount")
        Assert(fref_L < ref.FrameCount && fref_R < ref.FrameCount, 
        \   "UUSyncSegments: 'ref' frame number >= FrameCount")
        Assert(fsyn_R > fsyn_L, 
        \   "UUSyncSegments: 'syn' sync point " + String(count) + " <= previous")
        Assert(fref_R > fref_L, 
        \   "UUSyncSegments: 'ref' sync point " + String(count) + " <= previous")

        factor = Float(fsyn_R - fsyn_L) / Float(fref_R - fref_L)
        S = syn.Trim(fsyn_L, fsyn_R-1)
        \      .UUChangeSpeed2(factor, pitchfix, noblend)

        S = (!debug) ? S : S.Subtitle(
        \        "syn in, out: " + String(fsyn_L) + "," + String(fsyn_R)
        \    + "\nref in, out: " + String(fref_L) + "," + String(fref_R)
        \    + "\nspeed mult.: " + String(factor, "%0.3f")
        \    , font="courier", size=S.Height/16, lsp=0)

        R = R ++ S

        ps     = FindStr(ssyn_R, ",")
        ssyn_L = (ps==0) ? ssyn_R : LeftStr(ssyn_R, ps-1)
        ssyn_R = (ps==0) ?   ""   : MidStr(ssyn_R, ps+1)

        count = count - 1
    }
    R = R ++ syn.Trim(fsyn_R, 0)
    return (bypass) ? syn : R
}
""")

#######################################
### change speed over a wide range 
### with frame blending in fast forward by default.
##
## @ factor - 0.33 for 1/3 speed, etc; 
##    min = 0.001; max = min(FrameCount, 1000) 
##    (if > FrameCount, duration would be 0) ;
##    if argument is out of range, an error occurs.
##
## @ pitchfix - if true, maintain audio pitch with speed 
##    (default false; allow pitch to rise or fall)
##
## @ noblend - if true, never blend frames (default false)
##
## @ version 1.0 24-Feb-2014 raffriff42
##
function UUChangeSpeed2(
\           clip C, float factor, bool "pitchfix", bool "noblend")
{
    Assert(factor>=0.001, 
    \  "UUChangeSpeed2 bad argument: factor<0.001")
    Assert(factor<=1000.0, 
    \  "UUChangeSpeed2 bad argument: factor>1000")
    Assert(factor<=C.FrameCount, 
    \  "UUChangeSpeed2 bad argument: factor>FrameCount")

    pitchfix = Default(pitchfix, false)
    noblend  = Default(noblend, false)

    AS  = C.KillAudio.AssumeFPS(C.FrameRate * factor)
    ASI = C.KillAudio.AssumeFPS(C.FrameRate / factor)

    temprad = Round(0.5 * factor) 
    TS = (factor<1.0 || temprad==0 || noblend) 
    \  ? AS 
    \  : Overlay(
    \       AS, 
    \       AS.TemporalSoften(
    \           temprad, 255, 255, 48, 2), 
    \       opacity=0.7)

    ffx = Min(Max(0.0, 25.0/factor), 100.0)

    (factor<1.5 && noblend==false)  
    \  ? AS.ConvertFPS(C) 
    \  : TS.ChangeFPS(C.Framerate, linear=(factor<10.0))

    A = (pitchfix)
    \  ? C.KillVideo.ConvertAudioToFloat.TimeStretch(tempo=factor*100.0)
    \  : C.KillVideo.ConvertAudioToFloat.TimeStretch(rate=factor*100.0)

    A = (C.AudioBits==32) ? A.ConvertAudioTo32bit
    \ : (C.AudioBits==24) ? A.ConvertAudioTo24bit
    \ : (C.AudioBits==16) ? A.ConvertAudioTo16bit
    \ : (C.AudioBits==8)  ? A.ConvertAudioTo8bit
    \ : A

    return (Abs(factor-1.0) < (0.5/C.FrameCount)) ? C
    \ : (C.HasAudio==false) ? Last 
    \   : Last.AudioDub(A)
}

[manolito]

That was a great show, back in 1994. And Paula Cole was so sweet, wasn't she...

[StainlessS]

Perfect, if you could automate creation of frame number lists (not an insurmountable problem)

[raffriff42]

> if you could automate creation of frame number lists
Thanks StainlessS, but I had to match two visually unrelated clips*; I don't see how to really automate that process. Only a human such as myself, who knew how the 2nd clip (think alternate camera angle) related to the reference, could know when it was in sync or not. The video was only about twelve minutes anyway, so finding a dozen sync points took only a few minutes. (I'll release it to YouTube when & if I get around to finishing it) Now, if the task had been really huge and repetitive, I might have tried to work up some automation. Maybe next time...

* (and audibly unrelated - the 2nd clip was MOS)

[johnmeyer]

If you want a one-button solution to this problem, I recommend you look at:

PuralEyes

You used to be able to get it by itself, and for not much money. Now it looks like it is only available as part of an expensive suite. It's too bad that so many companies are following the Adobe Creative Suite model.