Tracking and Stabilising

[mustardman]

Hi,
I've recently returned to the world of video editing (repairing) after a long time away, and I have a clip to repair.

I have a clip (about 5 seconds) where the subject of interest moves quite a bit within the frame, and I would like to track and stabilise the subject (a persons head) so I can further process it and eventually superresolution the result. Captured from VHS, taken inside under incandescent lights (this was well before LEDs and Compact Fluros existed!). Camera quality was OK for the time, but woeful by todays standards. As a result the video is very noisy, but actually cleans up very well with a temporal filter... except when the subject moves!!

The problem:
I've only found two tracking filters (one for AVIsynth and one for VirtualDub), but both simply track the object so it can be pushed to the center of a new frame. Neither correct for zoom or rotation, only X/Y movement.

The Deshaker plugin for VirtualDub is awesome, but the problem is the area for stabilisation is fixed into the frame (eg: I could chose a 100x100 block in the center, but it stabilises the block as a whole, ie: the average of the object plus the background... I only want the object stabilised... Good thing is it will correct for zoom & rotation and I can disable any X/Y correction if required.

My solution to this problem is:
1. Track the object and move it into a new frame so it is always in the center (the AVIsysnth tracker seems a better bet than the VirtualDub tracker - which has some undesirable limitations).
2. Set up Deshaker to process the new frame with the subject now constrained to a known area. Deshaker can then do the zoom & rotation corrections (and final X/Y corrections).
3. Further processing (de-noise/temporal).
4. Superresolution the result.

I guess I came to this forum seeking clarification on 1 & 2 (track & stabilise), but I've pretty much convinced myself that there is no option other than the two-step process I've described.

However, I am open to any suggestions or comments.

Cheers,
MM

[poisondeathray]

You can try it, but deshaker and depan/depanstabilize in avisynth are probably not ideal tools for this. They are for intended smoothing out camera motion, not for tracking objects or stabilization based on specific objects /features / textures . You can't tell them exactly what to track or what not to track (or at least not very easily or accurately); nor are they very accurate for "rock stable" tracking

Usually you would use something dedicated for object tracking - visual effects trackers - such as after effects' point tracker, or mocha. There are other specialized tracking software as well that handle specific types of situations, but it depends on what the exact tracking requirements are. Those types of trackers are extremely accurate, but often have problems with low quality clips & noise ; often techniques like degrain/denoise preprocessing then regrain are used

Can you describe the clip - What type of motion for the head? e.g. does the person turn their head in the clip, or is it mostly camera motion only? When you say "rotation", what type of rotation is it ? Are there any perspective changes ? Or if you post the clip, a video is worth 10,000 words.

For free software , blender and natron have tracking and stabilzation now, there are tutorials on youtube


Just curious, what are you using for "superresolution" the result ?

[mustardman]

Hi,

Cheers for the reply. I've heard of blender, but only in relation to creating CGI type clips, not for dealing with 'real world' video. The other one, natron, I've not heard of at all. I've not heard of mocha either - I must be really out of date!

Yes, I agree that doing a stabilise straight-up is not a solution, hence thinking about object tracking first.

I have a very old copy of 'after effects' for premiere v6. I had not considered it for tracking/stabilising.

The clip I have is about 4 to 5 seconds long, and I've snipped what is probably the most indicitative part out of it and attached it rather than uploading to a file hosting site. It is not actually zipped, you just need to take the zip extension off and call it mp4 to play it (although you probably figured that out already). You'll have to pause & step through it as it would be lucky to be a seconds worth. The compression artefacts are not there in the original (DV capture direct from VHS). The video looks OK when played, but a frame grab shows up how bad the noise in the picture really is. It is also really dark and level-adjusting makes the noise stand out even more.

Unfortunately adjusting the levels starts to show the compression introduced in the DV encoding process (an ADVC100 I think). I have the original tape (if it still plays!), but don't know how to brighten (gamma) the image between the VCR and the capture card, hence am doing it after capture.

There is not much rotation admittedly, but there is zoom. My intent is to do the best I can with most of the 4-5 second clip and then pick out 3-5 frames to superresolution.

About the same time I got Premiere, I also got a premiere plug-in called "VideoPics". I did some playing with it yesterday on a different clip, and it is actually remarkably good. It seems to deal quite well with zooming, as well what you'd expect it to do with X/Y movement. It is quite old, but I don't think the maths behind the superresolution technique have changed at all, and I can't for the life of me find any decent superresolution implementations that are readily available (ie: free). I don't expect CSI type results from it either... more looking to reduce stair-stepping and hopefully smooth some of that noise out.

I have not tried 'VideoPics' on the clip in question yet... that's later today (hopefully).

Thanks,
MM.

[poisondeathray]

It can take a while for attachments to be approved ; sometimes posting to a 3rd party file hosting site is a good idea

I haven't seen the video yet, but if it's mainly just stair-stepping / aliasing and noise issues - there are other approaches/filters that you can use in avisynth as well . If you want someone to look at that and make suggestions, it would be a better idea to post the original (or at least the DV version) of the clip

[mustardman]

Hi,

The video file as DV compressed PAL is available at https://www.sendspace.com/file/vqludq

I had a play with just the SuperResolution plugin (VideoPics) last night to see how it would go. My previous clip had only minimal X/Y movement (and some zoom) which probably accounted for a maximum movement of 10 pixels per frame (PAL 720 x 576). The plugin handled this very well (surprisingly well in fact). The whole image was static and the movement was caused by camera operations (zoom) and camera operator (shakiness).

As expected my biggest problem with the "new" clip is the significant image movement coupled with the background movement (from panning the camera). The plugin is 'dumb' and tries to match up the whole frame. Hence if the background moves the opposite direction to the subject, it does an average and motion blurs both! The high contrast date/time stamp does not help in this regard either.

I'm considering picking a few frames that might be OK, exporting them as images, editing them to erase the background (make it all black or all white) and then putting it back to video to do the final superresolution. I don't think any photostitching apps would "improve" on the base resolution like superres would.

Of course the backing music track will have to be the theme from CSI...

Cheers,
MM

[poisondeathray]

I had a play with just the SuperResolution plugin (VideoPics)

I'm not familiar with that VideoPics plugin and I can't find a website - do you have more info ? There are slightly different variations on SR algorithms, I'm just curious what it uses and how it performs

My intent is to do the best I can with most of the 4-5 second clip and then pick out 3-5 frames to superresolution.

So you want to take a few frames and combine / SR them into a single frame composite ? ie. not for "video" ?




In this example, I chose bobbed frame numbers 45-55 as a demo of what you might expect using a feature tracking stabilizer (mocha in this case)
http://www.mediafire.com/file/udr80x..._0045-0055.zip

The "couch" was stabilized in this sequence. Because it's a known stationary object it can be used as a reference. Basically you're removing the camera motion, as if it were locked off on a fixed tripod. What's left over is the subject's motion , moving head , facial movement, etc... Since it's stabilized, you can see how much easier it would be to rotoscope or mask out the background if your plugin gets "confused" by the background. That should be a piece of cake even in the old AE version you have. Stabilization and rotoscoping go hand in hand. You can pick and choose frames, but if you choose frames that have too much head movement, in my experience all SR algorithms will tend to blur more than you want, and work better when you have better alignment. Even slight movement can contaminate the result.

A technical issue is interlaced video. Each even/odd field is vertically offset, so you get an up/down motion. Smart bobbers compensating for this . QTGMC attempts to interpolate the missing information from each field and temporally smooths things over, so you reduce stair stepping / aliasing from the deinterlacing artifacts. But you reduce noise and detail as well. So I used QTGMC in "sourcematch" mode to keep as much of the original noise and signal because you're attempting to apply an external SR algorithm.

I use a full range matrix to convert to RGB, so the contrast will look different. Most of the FX type programs use RGB. Deshaker uses RGB. The reason for full range is there is a chance you can recover some more shadow detail if it dipped below Y=16 . A "standard" Rec conversion to RGB will clip those values. Or if you adjust values in YUV before converting to RGB, that's another approach

Instead of stabilizing camera motion, you can track and stabilize features or targets on moving objects. If you target, say eyes, or some other feature, and try to stabilize around that. But if the head turns or eyes blink, it's easy to get "slippage" or skew the track, and you have to make adjustments. Again you have the same issue with contamination with SR if the object/feature is not the same - e.g. if mouth is open vs. mouth closed it's just going to blur and blend yielding undesireable results

Blender has a "planar tracker" now and is basically an open source variation of what mocha does. There should be tutorials on youtube

[mustardman]

Quote 1:
This plugin I talk about is quite old (very old actually). The only information I've been able to find is https://www.macworld.com/article/1021882/videopics.html Check the date... 2001...
I suppose when they talk about Priemiere v5, it gives away how old it is. Neither of the links in the article go anywhere sensible.
SR was pretty big back then with lots of hype. The company in question was the only one that had a product that used it, and almost 20 years ago, $700 was a lot of cash (it still is!).

Quote 2:
Yes, the intent is to get one reasonable "photo" out of the video clip. You have obviously seen it, and I think you agree that it is going to be a task. Many years before I got into video (it was about '95) I did a pause somewhere within the clip using an analog VCR and photographed the screen (a curved glass CRT no less), and it looked surprisingly good for a single field. I should try and find that photo (on genuine film no less) to see if it might actually be useable. Anyway...


I will have to have a look at the pieces of software you mention. I've always used Donald Grafts smart bob deinterlacer, and although I'd heard of QTGMC in the periphery, I've never ever looked at it.
Video processing has certainly moved on from people dealing with SD resoulution (like me), and it took a crushed camera that used tape to make me upgrade (about 4 years back). Why did I wait so long! HD is so superior!

When I get back to my normal PC I'll download your clip and read through your response in greater detail. I thank you for taking the time to look at it.

Cheers,
MM

[StainlessS]

Maybe not useful for this thread purpose, but so as to include it for anyone finding this thread in future,
here AvisynthTracking (actually given the name of 'AvisynthTrackin' in zip, minus the final 'g'):- https://forum.doom9.org/showthread.php?t=154010

Requires some version of Python (not the Vapoursynth one, so I think maybe v2.0).
(As I recall, I had v3.0 Python installed, and had to figure out that 'print' had been changed from statement to function 'print()' [for v3.0], or vice-versa,
so I just used v3.0 myself [after editing]).

I tried it once for a short time, tracking C3PO coming over some sand dunes, tracked him from some distance (quite small) up
to reasonably close, so is able to track with changes in size.
Dont know if anything could be done to eg create masks and deshake non tracked object, then re-layer the tracked object,
compensating for the deshaking.