Syncing Two Videos

[magikarp99]

I have two almost identical videos. One has a complete uninterrupted luma signal, but no chroma signal. The other video has both chroma and luma signals but has dropped frames. The dropped frames are just missing from the stream entirely, there are no blank frames in place.

By overlaying the two signals in difference mode and computing the average luma I can detect where the first dropped frame is. But because this first dropped frame puts the two videos out of sync, this will then incorrectly identify all proceeding frames as dropped.

How can I get Avisynth to identify every dropped frame in the video with a chroma signal, and insert the missing luma frame at that point?

I'm very grateful for any help.

[poisondeathray]

This findcuts script by gavino might help

http://forum.videohelp.com/threads/310044-video-censor-detection?p=1912566#post1912566

[raffriff42]

Another possibility is "Glitch Analyzer (detect dropped frames)" by jmac698
http://forum.doom9.org/showthread.php?p=1462931

[jmac698]

I'm afraid Glitch Analyzer is for finding dropped frames, but it works by preparing a special video with a timecode.
If you could post some short samples where there's frame drops, I could take a look at it. I need the samples to tune the frame-matching parameters. I have an idea for this, and it's a good excuse to work on some frame sync code I've always been meaning to work on.

p.s. I mean samples from both videos in the same area. You can use virtual dub to cut them.

[StainlessS]

I think that the DBase funcs of RT_stats could (maybe) be of great assistance in these type of problems,.

EDIT: alignment might best be done from End to Beginning, after detectiion.

EDIT: GScript is a given. (We love Big G {EDIT: Despite his reluctance to introduce a 'break' statement, for which, he will never be forgiven })

EDIT: Never, Ever !

[magikarp99]

Originally Posted by poisondeathray

This findcuts script by gavino might help

http://forum.videohelp.com/threads/310044-video-censor-detection?p=1912566#post1912566

Thank you, this is a good example of GScript. I have used this as a basis.

Originally Posted by raffriff42

Another possibility is "Glitch Analyzer (detect dropped frames)" by jmac698
http://forum.doom9.org/showthread.php?p=1462931

Thanks for the suggestion.

Originally Posted by StainlessS

EDIT: GScript is a given. (We love Big G {EDIT: Despite his reluctance to introduce a 'break' statement, for which, he will never be forgiven })
Another possibility is "Glitch Analyzer (detect dropped frames)" by jmac698
http://forum.doom9.org/showthread.php?p=1462931
EDIT: Never, Ever !

Indeed, I had not come across GScript before and it appears to bring sorely need functionality to AVISynth! A break statement would be great.

Originally Posted by jmac698

I'm afraid Glitch Analyzer is for finding dropped frames, but it works by preparing a special video with a timecode.
If you could post some short samples where there's frame drops, I could take a look at it. I need the samples to tune the frame-matching parameters. I have an idea for this, and it's a good excuse to work on some frame sync code I've always been meaning to work on.

p.s. I mean samples from both videos in the same area. You can use virtual dub to cut them.

I would really appreciate that.

I have got a short script, which works fairly well, going already; I based it upon the examples supplied here. There a few kinks to sort out though. I will upload a sample clip and my script soon.

Any suggestions on where to upload the video clips?

[poisondeathray]

Originally Posted by magikarp99

Any suggestions on where to upload the video clips?

There are plenty of free file hosts

mediafire.com or sendspace.com seem to be popular for hosting samples around here

[StainlessS]

Any suggestions on where to upload the video clips?

Groucho2004 recommends DropBox.com rather than Mediafire, I've recently registered but have yet to copy my stuff from MediaFire.com.
Recommended as being lightweight and unobtrusive.

[magikarp99]

Just uploading some video clips. I have removed the audio to save some file size.

I simplified the issue a bit before. Essentially the chroma video has 3 types of drops:

1. A frame is just missing, there is no visual indication unless compared to another video.
2. Two consecutive frames are missing, but a black frame is present in their place.
3. Three consecutive frames are missing, but two black frames are in their place.

The script I currently have must start with the first frame of each video correctly in sync. The luma levels must also be matched very closely beforehand. It then proceeds to compute two values iteratively:

1. The luma difference between the current frames in the luma and chroma videos.
2. The luma difference between the current frame in the chroma video and the next frame in the luma video.

If the difference is lower for the second value, then it is assumed that a dropped frame has been reached. Another indication of a dropped frame is if average luma value for the frame is 16.

This proves to be very effective, but there are two issues:

1. The gain/offset of the luma signal alters abruptly at points in the video. This can lead to a false positive detection of a dropped frame. Which then causes most succeeding frames to be false positives. e.g. at frame 1517 in the examples.
2. Currently I am using overlays to add a chroma signal to the returned luma frames. This quickly eats up memory, so this isn't an ideal solution. I may have to split overlays into multiple scripts and concatenate the resulting videos unless someone can suggest a better solution.

It works well in this example all the way up to frame 1517 where it makes a false positive identifitication.

Here is the script, put this in a file named resync.avs:

# ReSync
# Based on FindCuts (Gavino, 13th Sept 2009)
# http://forum.videohelp.com/threads/310044-video-censor-detection?p=1912566#post1912566 
# 
# Resyncs two videos when luma channels are almost identical.
# Intended to sync video with chroma channel, but dropped frames
# with video without chroma, but an intact, uninterrupted luma signal
#
# Chroma video has following varieties of drops
# 1. Frame is dropped silently
# 2. Two frames are dropped, and one blank frame takes their place
# 3. Three frames are dropped, and two blank frames take their place
#
# Requires GScript (http://forum.doom9.org/showthread.php?t=147846)

function ReSync(clip full, clip cut, string "logFile") {
  log = Defined(logFile)
  txt = "Finding cuts ..."
  dropped = "Dropped"
  sep = ", "
  sep2 = ": "
  if (log) { WriteFileStart(full, logFile, "txt") }

  # Videos from to return frames
  show = full
  show_cut = cut
  # Crops noise from top and bottom of frames
  full = full.Crop(0, 14, 0, -8)
  cut = cut.Crop(0, 14, 0, -8)
  # Video shifted forward one frame
  next_full = full.Trim(1, 0)
  result = BlankClip(show, length=0)
  current_frame = 0
  limit = full.FrameCount

  # loop over sequence of cuts:
  while (current_frame < limit) {
    start = current_frame
    # find next dropped frame:
	# compares luma difference between current frames, and next frame
	# also checks frame is not blank
	 while (current_frame < limit && !(LumaDifference(full, cut) > (LumaDifference(next_full, cut))) && AverageLuma(cut) > 16) {
         end = current_frame
	     # Values for logging
	     if (log) {
	         diff_current = LumaDifference(full, cut)
             diff_next = LumaDifference(next_full, cut)
             WriteFileStart(full, logFile, "end", "sep2", "diff_current", "sep", "diff_next", append = true)
	         current_frame = end
	     }
	     current_frame = current_frame + 1
     }
	 missing = current_frame
	 if (log) {
	     diff_current = LumaDifference(full, cut)
         diff_next = LumaDifference(next_full, cut)
         WriteFileStart(full, logFile, "missing", "sep2", "diff_current", "sep", "diff_next", append = true)
	     current_frame = missing
	 }
	 avg_luma1 = AverageLuma(cut)
	 current_frame = current_frame + 1
	 avg_luma2 = AverageLuma(cut)
	 #current_frame = current_frame - 1
	 # if reached a blank frame
	 if (avg_luma1 == 16) { 
	     # if two successive blank frames
	     if (avg_luma2 == 16) {
		     if (log) {
		         WriteFileStart(full, logFile, "dropped", "sep2", "missing", append = true)
			     WriteFileStart(full, logFile, "dropped", "sep2", "missing + 1", append = true)
		     }
			 # Append uninterrupted section of chroma video to result
			 # Overlay previous and succeeding chroma frames with luma frames and append
			 result = result + show_cut.Trim(start, end) + Overlay(show.Trim(missing, missing + 2), show_cut.Trim(missing - 1, missing - 1) + show_cut.Trim(missing - 1, missing - 1) + show_cut.Trim(missing + 2, missing + 2), mode="chroma", opacity=1)
			 # Pad videos
			 cut = cut.Loop(2, 0, 0)
		     show_cut = show_cut.Loop(2, 0, 0)
			 current_frame = missing + 3
		 } else {
	         if (log) {
			     WriteFileStart(full, logFile, "dropped", "sep2", "missing", append = true)
		         WriteFileStart(full, logFile, "missing", "sep", "avg_luma1", append = true)
			 }
			 # Append uninterrupted section of chroma video to result
			 # Overlay previous and succeeding chroma frames with luma frames and append
		     result = result + show_cut.Trim(start, end) + Overlay(show.Trim(missing, missing + 1), show_cut.Trim(missing - 1, missing - 1) + show_cut.Trim(missing + 1, missing + 1), mode="chroma", opacity=1)
			 # Pad videos
		     cut = cut.Loop(2, 0, 0)
		     show_cut = show_cut.Loop(2, 0, 0)
		     current_frame = missing + 2
	     }
     # else single frame dropped silently
	 } else {
	     # Append uninterrupted section of chroma video to result
	     # Overlay previous chroma frame with luma frame and append
		 result = result + show_cut.Trim(start, end) + Overlay(show.Trim(missing, missing), show_cut.Trim(missing - 1, missing - 1), mode="chroma", opacity=1)
		 # Pad videos
	     cut = cut.Loop(2, 0, 0)
		 show_cut = show_cut.Loop(2, 0, 0)
	     if (log) {
             WriteFileStart(full, logFile, "missing", append = true)
             current_frame = missing + 1 # WriteFileStart sets current_frame to -1 !!!
         } else {
	      current_frame = current_frame + 2
	     }
     }
   }
   #return AudioDub(result, full)
   return result
 }

Use it like this:

chroma = AviSource("chroma_example.avi")
luma = AviSource("luma_example.avi")

#Match the luma signals
luma = ColorYUV(luma, off_y=-16, gain_y=-1, cont_y=0)
chroma = ColorYUV(chroma, off_y=-18, gain_y=0)
luma = Limiter(luma, 16, 235, 16, 240)
chroma = Limiter(chroma, 16, 218, 16, 240)

GImport("resync.avs")

ReSync(luma.ConvertToYV12(), chroma.ConvertToYV12(), "cutlog.txt")

#Uncomment this to see difference between frames and luma signals
#StackHorizontal(ScriptClip(Overlay(luma.Crop(0, 14, 0, -8), chroma.Crop(0, 14, 0, -8).Greyscale(), mode="Difference", opacity=1).ConvertToYV12(), "Subtitle(String(AverageLuma))"), Overlay(VideoScope(Greyscale(luma.Crop(0, 14, 0, -8)), "side", true, "Y").ConvertToYV12(), VideoScope(Greyscale(chroma.Crop(0, 14, 0, -8)), "side", true, "Y").ConvertToYV12(), mode="Difference", opacity=1).Crop(720, 0, 0, 0))

Will post sample videos when they have uploaded.

[magikarp99]

http://www.mediafire.com/watch/2a7ndbzak5n8vaa/chroma_example.avi
http://www.mediafire.com/watch/2txf022cvx3l41l/luma_example.avi

[magikarp99]

Originally Posted by magikarp99

The gain/offset of the luma signal alters abruptly at points in the video. This can lead to a false positive detection of a dropped frame. Which then causes most succeeding frames to be false positives. e.g. at frame 1517 in the examples.

I think my problem may be solved if I can normalise the luma of one video to the other on a frame to frame basis. Any suggestions?

This popped up in a search: http://forum.doom9.org/showthread.php?t=138662
I haven't had a chance to try any of the suggestions there yet.

[jmac698]

I'm sorry I don't think I'll have time to look at it, so it's up for grabs.

[magikarp99]

Originally Posted by jmac698

I'm sorry I don't think I'll have time to look at it, so it's up for grabs.

Thanks anyway.

I think I have got it almost perfect now. Sorted out both my memory and levels issues I believe. Just doing a final test. If it all works I will clean up my script and put it up here

MergeChroma is much more efficient than using Overlay. It has a small memory footprint and is much faster.

To sort out the levels issues I am using ColorLike on every frame.

[magikarp99]

This script works almost perfectly. It is a mess, and I don't really have the time to clean it up too much. I'm sure it could also be made more efficient.

File attached.