Repair frames, AVSLib array, crop or animate to save forgotten quick release disaster

[MrJ]

This is my first post so be gentle please.

I turned up to shoot a video and couldn't find the quick release plate which screws the camera to the tripod. I didn't have time to get another so I put a lump of plasticine between camera and tripod and taped it up as best I could. When I got home and checked the footage I found the camera had slowly leaned back with the lens heading toward the roof. I think the heat from the camera must have gradually softened the plasticine. Thankfully I just pressed record so there is no tilting, panning or zooming to complicate matters. It is helped by that fact that there is always a fixed microphone in shot so I can be fairly sure how the camera moved. I think it's a fairly unusual set of circumstances which is probably why I can't find anyone else writing about it until I read the previous repair frames post here today.

I went through the video and now have a mathematical formula which tells me how the angle changed with time. For every 1920 x 1080 frame I have, I want to cut out a 1280 x 720 one. I can crop in avisynth as long as I want the crop to be in the same place in each frame. However with this crop I want the top left hand corner (or centre of the 1280 x 720 frame) to change with time (or framenumber). As my camera moved by accident in a mathematically controlled way, I know exactly where I want the cuts to go and can easily make a spreadsheet or text file with the information for the thousands of necessary cuts. At the moment I could only export stills and cut them by hand and then put them back together like Magic Lantern users do with dng files. I want a program to read in the x and y co-ordinates of the cut along with the framenumber and want the program to then save the 1280 x 720 picture in some format, either a video format or some other image one which I can convert back to video.

I think the AVSLib software can do it but I'm finding their tutorials hard to follow as I haven't done much programming in years. Also avisynth is different to the other programming languages I've used so it's a whole new way of thinking. I think someone who is familiar with the code could made something which worked in 5 or 10 minutes whereas I would probably take me 5 or 10 hours.

I don't have After Effects or Nuke or any software like that installed although I'm looking in buying something which can handle canon dng raw files to shoot some video on my 50D. Maybe I could download an evaluation copy, solve this one-off problem and see if I want to buy for the Raw video stuff.

I'd be very grateful for any suggestions.

(The equation for how many pixels to crop is in the form p = ax^2 + bx + c where x is frame number and I know constants a, b, c. The R squared value for the curve is something like 0.9996 from memory so I'm confident that if I can get the thing cut well, the video will at least be watchable).

[Gavino]

Originally Posted by MrJ

For every 1920 x 1080 frame I have, I want to cut out a 1280 x 720 one. I can crop in avisynth as long as I want the crop to be in the same place in each frame. However with this crop I want the top left hand corner (or centre of the 1280 x 720 frame) to change with time (or framenumber).
...
The equation for how many pixels to crop is in the form p = ax^2 + bx + c where x is frame number and I know constants a, b, c.

Use ScriptClip() to crop at a per-frame varying position.
Outline:

... #source filter
ScriptClip("""
  x = current_frame
  Crop(...) # parameters based on x
""")

[MrJ]

Originally Posted by Gavino

Use ScriptClip() to crop at a per-frame varying position.

Thanks Gavino.
I was trying to avoid sounding like a total avisynth noob but, gulp here goes..

LoadPlugin("C:\...DGDecode.dll")
vid=mpeg2source("F:\....d2v")
#As is no doubt obvious I'm not sure of the syntax for this. It needs a vid and more
ScriptClip("""
    x = current_frame   #variable
    a = 2.9017E-08      #constants
    b = 2.41882E-03
    c = -2.47594795
    
    pdown = a*x^2+b*x+c    #number of pixels to crop from the top of the 1080 frame
    pd=Round(pdown)        #whole number of pixels to crop from the top of the 1080 frame
    
    pup = 360-pd           #whole number of pixels to crop from the bottom of the 1080 frame
    
Crop(320, pd, -320, -pup)
""")

I'm posting this not because I think it's anywhere near right but to try and clarify what I'm trying to do. Any pointers gratefully received.

[Gavino]

That's the basic idea, but the script is not quite right.

The simplest way to set up the source clip is not to assign it to a variable (eg 'vid'), but leave it unassigned (implicitly assigned to the special variable last) and it will then be used as the input to ScriptClip().
Scientific floating-point notation is not supported so you need to write the numbers out with leading zeros.
There is no ^ operator.

Because of restrictions on Crop() positions for different video types, it may be preferable instead to use the cropping facility of a resizer, which supports subpixel cropping positions.

LoadPlugin("C:\...DGDecode.dll")
mpeg2source("F:\....d2v")
ScriptClip("""
    x = current_frame   #variable
    a = 0.0000000029017      #constants
    b = 0.00241882
    c = -2.47594795
    
    pdown = a*x*x+b*x+c    #number of pixels to crop from the top of the 1080 frame
    # no need to round
    
    pup = 360-pd           # number of pixels to crop from the bottom of the 1080 frame
    
Spline36Resize(1920, 1080, 320, pd, -320, -pup)
""")

[Gavino]

Previously, I said:

Originally Posted by Gavino

Because of restrictions on Crop() positions for different video types, it may be preferable instead to use the cropping facility of a resizer, which supports subpixel cropping positions.

It will also give a smoother continuous pan than jumping by discrete pixels via Crop().

Thanks to StainlessS, I have noticed a few errors in my script above. It should be:

LoadPlugin("C:\...DGDecode.dll")
mpeg2source("F:\....d2v")
orig = last
ScriptClip(PointResize(1280,720), """
    x = current_frame   #variable
    a = 0.0000000029017      #constants
    b = 0.00241882
    c = -2.47594795
    
    pdown = a*x*x+b*x+c    #number of pixels to crop from the top of the 1080 frame
    # no need to round, and can be negative
       
    orig.Spline36Resize(1280, 720, 320, pdown, -320, 720)
""")

[StainlessS]

Try something like this:

#LoadPlugin("C:\...DGDecode.dll")
#mpeg2source("F:\....d2v")
#As is no doubt obvious I'm not sure of the syntax for this. It needs a vid and more

VideoFile = "D:\TESTVOBS\PAL_I_Top_169_P-JurassicPark.d2v"
Mpeg2Source(VideoFile)
ConvertToYUY2()                  # Cant crop YV12 on odd vertical boundaries
BilinearResize(1920,1080)        # I dont have HD clip handy

ORG=Last
DUMMY=PointResize(1280,720)      # ScriptClip MUST return same dimensions clip
DUMMY.ScriptClip("""
    x = current_frame            # variable
    #a = 2.9017E-08              # constants
    a = 0.000000029017           # Avisynth Syntax - Script Variables: "Note that exponent-style notation is not supported."
    #b = 2.41882E-03
    b = 0.00241882
    c = -2.47594795              # ERROR, Causes -ve crop at eg current_frame==0

    #pdown = a*x^2+b*x+c         # number of pixels to crop from the top of the 1080 frame
    pdown = a*(x*x)+b*x+c        # Pow(x,2) Slower

    pd=Round(pdown)              # whole number of pixels to crop from the top of the 1080 frame

    pd=(pd<0) ? 0 : pd           # Limit to +ve values ONLY (crop would fail, due to -ve c)

#   pup = 360-pd                 # whole number of pixels to crop from the bottom of the 1080 frame. USE 720 for Height
#    RT_Debug(String(x),String(pd))
    Return ORG.Crop(320, pd, -320, 720)   # return cropped original, NOT DUMMY
""")
ConvertToYV12()   # EDIT: Back to original CS

EDIT: Darn that Gavino fella, he's just too quick.

PS, Welcome to the forum, MrJ.

EDIT: In Gavino script, you need to edit both instances of 'pd' to 'pdown'.

[IanB]

A word of note :- the ScriptClip of the Resize gives a far superior visual result for frame transition smoothness, but ScriptClip builds and tears down the graph every frame and the resizers are a relatively complex filter to build. Crop on the other hand is about as simple as it gets.

The resizers are implemented as separable horizontal and vertical functions. As the horizontal part is invariant for this application it may be worthwhile taking that element out of the ScriptClip.

LoadPlugin("C:\...DGDecode.dll")
mpeg2source("F:\....d2v")
Spline36Resize(1280, Height(), 320, 0, -320, 0)
orig = last
ScriptClip(PointResize(1280,720), """
    x = current_frame   #variable
    a = 0.0000000029017      #constants
    b = 0.00241882
    c = -2.47594795
    
    pdown = a*x*x+b*x+c    #number of pixels to crop from the top of the 1080 frame
    # no need to round, and can be negative
       
    orig.Spline36Resize(1280, 720, 0, pdown, 0, 720)
""")

[Gavino]

Originally Posted by IanB

As the horizontal part is invariant for this application it may be worthwhile taking that element out of the ScriptClip.

Good point, Ian.

It's also worth saying for Mr J's benefit that the script assumes a progressive source.
If the source is interlaced, it will have to be deinterlaced before the main processing.

[raffriff42]

This is great to see. I am learning a lot from you guys. In the words of Yogi Berra, "You can observe a lot by just watching."

One thing, if Spline36Resize is in use anyway for sub-pixel cropping purposes, shouldn't you enlarge a bit at the same time - just enough to keep the project resolution of 1920x1080 after processing?

[StainlessS]

My first instinct was to go with a resize back to 1920x1080, but as OP seemed happy with crop, did not.
Mr G and Mr B, gave an excellent masterclass here, and RaffRiff42 aint the only one who learnt a lot.
One additional mod, (hardly worth mentioning) is that first two resizes in IanB's script could become crops
thus saving a few microseconds (2nd being only dummy).

Anyway, I think the below should suffice if OP would like as RaffRiff42 suggests

LoadPlugin("C:\...DGDecode.dll")
mpeg2source("F:\....d2v")
Spline36Resize(Width(), Height(), 320, 0, -320, 0)
ScriptClip("""
    x = current_frame        #variable
    a = 0.0000000029017      #constants
    b = 0.00241882
    c = -2.47594795
    
    pdown = a*x*x+b*x+c    #number of pixels to crop from the top of the 1080 frame
    # no need to round, and can be negative
       
    Spline36Resize(Width(),Height(), 0, pdown, 0, 720)
""")

[MrJ]

Originally Posted by StainlessS

OP seemed happy with crop

Thanks a lot guys. This is a masterclass for me and I'm learning a lot.

Sorry I haven't posted replies yet. I do intend to. I was so delighted with the vertical motion fix that I thought I might torture myself by correcting the horizontal drift as well. It's not as pronounced, about 4 times less than the vertical but now the vertical is fixed I'm noticing it slightly. It's not really necessary since it just looks like the camera is smoothly panned rather than falling down which looked terrible.

I corrected the levels, which were shocking and added the audio. I was using a friend's camera and he's got some hot pixels which are now doing an interesting upward drift with the original script and executing parabolic motion with my horizontal corrections entered. I don't know whether it's best to try and do something about them immediately after deinterlacing or to try and include something in the script clip or something else.

I wanted the 1280 x 720 in case I put it on vimeo or youtube but I might also put it on a 720 x 576 PAL DVD despite the camera being NTSC. I'm trying to fix the horizontal motion at the moment. I've done something daft with my numbers but I'll try and figure it out before I amuse you with it.

[StainlessS]

I dont usually touch HD, but I would think it might be better to deal with eg Hot Pixels before performing
any resizing (or resize used to subpixel crop), the resize would I think tend to 'spread' the hot pixels
and make them harder to remove later. Have not ever used it but UnDot() may assist, but dont know
if best to deinterlace 1st or not (or maybe there is an interlaced Undot like filter).
StickBoy's scripts might be useful using progressive only filters on interlaced content.
here: (jdl-interlace.avsi, on the old avisynth site)
EDIT: Link Removed

EDIT: @ Wilbert: It seems that even the new Avisynth.nl site points back at the old one for Stickboy's scripts.
All the other links @ http://avisynth.nl/index.php/Shared_functions
are OK (only the "Old Wiki" link points back at old site).

EDIT: Here it is on new site: http://www.avisynth.nl/users/stickboy/
Found it using Google search string "Stickboy site:avisynth.nl"

EDIT: Wilbert, have updated link to Stickboy scripts on Avisynth.nl.
EDIT: Just went back to Shared_functions, my edit does not seem to have taken, perhaps it needs authorization or I did
something wrong.

[MrJ]

Originally Posted by StainlessS

UnDot() may assist, but dont know
if best to deinterlace 1st or not (or maybe there is an interlaced Undot like filter).
StickBoy's scripts might be useful using progressive only filters on interlaced content.

I've never tried StickBoy's scripts. I did try using removedeadpixels from:

http://forum.doom9.org/showthread.php?t=98624

but the results of me trying to use it didn't look good. I'm interested in this because my Canon 50D shows no dead pixels when shooting stills or raw but has a couple when I shoot h264. My friend's HD camera doesn't seem to have any remapping software so I think he can only send it back or try and deal with them post shooting.

EDIT:fixed link. Thanks StainlessS, Andy

[StainlessS]

Pertinent Stickboy functions are JDL_UnfoldFieldsVertical & JDL_FoldFieldsVertical, with flip=true, allows (MOST) spatial/temporal
progressive filters on interlaced.

EDIT: you can insert proper links in post by clicking on the little globe (auto link insertion is currently broken).

[MrJ]

Originally Posted by StainlessS

# ERROR, Causes -ve crop at eg current_frame==0

Thanks Stainless Steel.

Well spotted with the zero frame error. I did my number crunching from the start of the clip. I had turned on the camera early while I fiddled around with the microphones. I'm glad you both posted different solutions. I always learn things that way. I've added a new parameter dv which should avoid this error even with crop and also reframes the video.

Originally Posted by Gavino

It will also give a smoother continuous pan than jumping by discrete pixels via Crop().

I like the results from Spline36Resize. I would never have thought of using it before posting here so thanks a lot.

Originally Posted by IanB

The resizers are implemented as separable horizontal and vertical functions. As the horizontal part is invariant for this application it may be worthwhile taking that element out of the ScriptClip.

This was an education. I only included data for the vertical motion in my earlier post because the the vertical motion was more pronounced and very displeasing to the eye. I've now included a horizontal correction too although I probably didn't need to as the panning is very slow and not particularly displeasing to the eye. All the same your post was really helpful in showing how the resizer works.

Originally Posted by Gavino

Good point, Ian.

It's also worth saying for Mr J's benefit that the script assumes a progressive source.
If the source is interlaced, it will have to be deinterlaced before the main processing.

The camera is a JVC GZ-HD7. It's an early Everio HDD model and heats up considerably when in use which is what caused the problem in the first place. I've used plasticine before with a DV tape camera and had no trouble whatsoever. I must remember to leave the quick release plate with the tripod. If I want to take stills it's attached to the video camera and vice versa. The camera is NTSC and I usually work with PAL so am not clear on deinterlacing to covert between formats. It's also complicated by the camera having about 3 dead pixels which I think the h264 is smearing around making it look like a whole cluster is bad. Unfortunately there's no way to remap the pixels on the camera without sending it back to the manufacturer. Any advice on deinterlacing and fixing dead pixels would be welcome although even with them the video is a million times better than it was a couple of days ago.

Originally Posted by raffriff42

shouldn't you enlarge a bit at the same time - just enough to keep the project resolution of 1920x1080 after processing?

The video is probably heading for Vimeo or YouTube so 1920 x 1080 is a bit big for them. Having said that I might well resize for a 720 x 576 PAL DVD but that might give me problems deinterlacing. I might also make an AVCHD DVD if I feel like some extra effort. I've still not mastered the art of deinterlacing. Again any advice of where to read etc would be welcome.

Originally Posted by StainlessS

you can insert proper links in post by clicking on the little globe (auto link insertion is currently broken).

Fixed now thanks.

#LoadPlugins
mpeg2source("f:\....d2v")
ConvertToYUY2()
RemoveDeadPixels(869, 544, 15) # try to fix dead pixel YUY2 only
#separatefields?
#Bob?

orig = last
ScriptClip(PointResize(1280,720), """
    x = current_frame    #time dependant variable
    
    av = 0.0000000364  #constants for fixing vertical motion
    bv = 0.00176
    cv = -10.6
    dv = 110             #number of pixels to crop off the top of the 1080 frame to reframe shot.
    
    ah = 0.00000000952  #constants for fixing horizontal motion
    bh = 0.000477
    ch = -0.3
    dh = 320             #number of pixels to crop off the left edge of the 1920 frame to recentre frame

    
    pd = av*x*x+bv*x+cv    #number of pixels to crop off the top of the 1080 frame to remove accidental tilt (increasing quadratic)
    pdown = pd + dv         #number of pixels to crop off the top of the 1080 frame to remove accidental tilt and reframe.(value = 110 at frame zero then increasing quadratic)
    
    pl = ah*x*x+bh*x+ch    #number of pixels to ADD to left edge of the 1920 frame to remove accidental pan (increasing quadratic)
    pleft = dh - pl        #number of pixels to SUBTRACT from left edge of the 1920 frame to remove accidental pan and recentre. (value = 320 at frame zero then decreasing)
  
    orig.Spline36Resize(1280, 720, pleft, pdown, 1280, 720)
""")

converttoYV12.SmoothLevels(9,1,170,0,255,show=false,Lmode=3).SmoothTweak(0,1,0.9,0,3)
vid=last
#aCam=FFaudiosource("F:....mp2")                                                #audio from camera
aMic=wavsource("F:....wav")                              #audio from microphone
audiodub(vid,aMic).trim(4000,52903).FadeIn(30).FadeOut(60)

It's still not finished but I still haven't read all the deinterlacing stuff yet. I thought you might like to comment on the script so far.
Andy

[creaothceann]

Originally Posted by MrJ

I've still not mastered the art of deinterlacing. Again any advice of where to read etc would be welcome.

Try QTGMC(FPSDivisor=2).

[StainlessS]

Are you sure you need PC levels ?

  SmoothLevels(9,1,170,0,255,show=false,Lmode=3)

[johnmeyer]

As I've stated in previous posts, AVISynth is not the best tool for every job. A MUCH easier way to deal with this is to use a video editor, such as Vegas, which permits keyframing motion. All you have to do is insert a few keyframe points, and at each point line up some fixed object so that it is in the same location as before the camera started to move. The keyframing will generate linear motion between each keyframe anchor.

But that's not linear, you might say. True, but as anyone who has taken calculus knows, any arbitrary curve can be approximated -- quite accurately -- by a series of straight lines. My guess is that fewer than half a dozen keyframe points would be needed. The whole thing could be fixed in less time than it took me to write this, and you'd have the added bonus of being able to interact with what you are doing, rather than deal with the motion indirectly through script variables.

[StainlessS]

I have to say that John Meyer is totally correct, however as someone who enjoys banging their head against the wall, maybe I'm not the best person to comment. (I would still do as MrJ) using eg Avisynth. Just because it is easy, dont make it more enjoyable.

For me, I ask the question, what would be easier, not attempting to assist on D9, or attempting to assist on D9.
The answer is always gonna be not attempting to assist on the D9, and yet, some people are misguided enough to still do it.

John is obviously correct, horses for courses, but I still dont use the fancy stuff myself. (I guess if you make your living
doing it, then fancy stuff can well be more convenient).

To: John Meyer, Jmac698, Martin53, Forensic; et al, you are all are very gifted people, with just a little effort you could
amaze the world with your inventiveness, C, CPP, aint anything prohibitive, I'm not gonna tell you that it is easy, it aint,
but it is not that much more difficult the eg Avisynth script, its where and when you have to insert them there damn
semicolons (you gotta do that a lot), give it a whirl and Amaze Me!

Oops, sorry, Jmac already does that, still waiting to be amazed!

EDIT: Some problems can be infinitely easier in C than script, you can do something in an hour that would take weeks in script
(and well faster).

[MrJ]

Originally Posted by StainlessS

Are you sure you need PC levels?

No I'm not sure. I haven't read the documentation for on SmoothLevels for a long time so anything I did know about it is half forgotten. I was finding that shooting light subjects like snow with the camera on automatic gave really underexposed shots. I use virtualdub's levels to get numbers for the first and third settings but can't even remember what the others do. If I don't know I tend to leave it at default. Were you going to suggest something else?

Originally Posted by creaothceann

Try QTGMC(FPSDivisor=2).

I've downloaded it. I'll have a look and see if it's easy enough get running.

Originally Posted by johnmeyer

A MUCH easier way to deal with this is to use a video editor, such as Vegas, which permits keyframing motion.

I just downloaded Lightworks to see if I can do it there out of interest. I'm delighted with the results avisynth produced and I've not figured out deinterlacing yet. I haven't seen any guides on how to solve this exact problem. It probably fairly unique in that I'm trying to do the inverse of a not too straightforward function. Do the NLEs let you input co-efficients? [/QUOTE]

Originally Posted by johnmeyer

you'd have the added bonus of being able to interact with what you are doing, rather than deal with the motion indirectly through script variables.

I didn't find the visual feedback too bad. If you use AvspMod to look at the files you can zoom in and take readings off the fixed object every 1000 frames or so. The mouse pointer gives you the x and y co-ordinates in AvspMod so I could note the xs and ys in a spreadsheet and use LINEST to get the co-efficients needed without an immense amount of jiggery-pokery. I'm sure there's other tracking software that could do it faster. From there AvspMod gave quite good visual feedback on where to reframe the shots without it looking too bad. I'm not sure which NLE would suit my needs. I thought I'd look at ivsEdits LE and Lightworks and see. I'd be surprised if I could get it rock steady with a NLE in under 5 minutes especially since I've never used one before. Having said that it would have taken a lot longer than that in avisynth even if I knew the code.