Preprocessing deshaker (script under development)

[redfordxx]

Hi everyone

I want to find a way to improve VDub's DeShaker for videos with handheld camera with lot of rotation around the x and y axis.

It is kind of discovery...but I hope will end up with useful workflow.

EDIT: what is wobbling

I did quite time consuming tests and created maps of camera rotations which I show below. I noticed that after right rotation the number of OK blocks increases. But, although it increases in total it might (and probably will) increase only in certain part of image. And I mean background of course. I think that's the wobbling. That different parts of image have different motion estimation.

So rotation does not help enough.

What might be helpful? Probably
1) spliting the frame to smaller parts
2) analyze the motion there by Deshaker
3) from these smaller vectors derive some transformation
4) apply the transformation to the frame
5) analyze whole transformed frame by Deshaker
Suitable transformation? No idea.
Morover, before that should be probably masked moving objects...
And that would be only one prev frame relation... now put it into series!

Example of the research:
https://i.postimg.cc/mrMSkVwz/frame-3167-lvls-8-step-45-max-80.png (https://postimages.org/)
What I see here is OK blocks value of the rotated image (up down, left right...)
- when I leave the camera where it is, I have 11,96 OK blocks. pretty bad
- when I move camera up (more exactly 5 degrees in 342 degrees direction) I get improvement ... 19.7 OK blocks
- but, I have improvements also in other directions (6 deg in 195 direction I get 19.57 OK blocks) and (7 deg in 167 direction I get 18.58)

If you are lucky, there is only one spot with improvement. But I was mostly unlucky...

So I am stopping the rotation theory until I have better idea...:-(



OLD:
Basic idea ... steps:

1 - create multiple clip versions simulating rotation using quad with added borders (3x3 matrix or even more clips)

2 - process them with deshaker (so we have 3x3 estimations for each frame)

3 - find the best estimation (according to OK blocks) for each frame (I don't see better tool for it than Excel right now)

4 - create input log with coordinates

5 - process it again this time using log file from (4) --- I think there are some scripting possibilities for calling functions based on logfile

6 - go to (3) and I can iterate theoretically as much as I want

7 - now I look in the exported data for (short) series of rotations potentially to be corrected. I have rules:

- the values of OK blocks are significantly higher than the center

- the rotation goes one way and then back to zero position

8 - create input log with coordinates

9 - quad the original clip according to (8). That would be the base of all the next processing

10 - deshaker 1st pass

11 - deshaker 2nd pass

[johnmeyer]

I don't entirely follow your idea, but it sounds like you are planning to modify the smoothing results by editing the log file. There is no reason that couldn't work. You could even do it in Excel, something I do all the time with AVISynth log files from other video functions.

Of course you have to be able to come up with better algorithms than what are built into Deshaker, and you won't be able to actually look at the pixel data directly, something that may be required because the transforms for things like rolling shutter, and even for rotation, are actually done in multiple dimensions.

When I wrote my guide and then my Deshaker automation scripts for Vegas, I had some brief correspondence with Gunnar Thalin, Deshaker's author, and got a little bit of a sense about how much more difficult the process is than the simple-minded description I provided at the beginning of my seven-page guide.

[edit]My old guide from 2004 has now disappeared from all the sites that used to host it. Here is the original Word file I used to created it. This version doesn't have the dialog shots, but the dialogs have all changed in the newer Deshaker versions, so their absence in this guide shouldn't matter.

https://drive.google.com/file/d/1hgJGIyxH1Sdbm8Cl19dyD_rUxm09ylnD

[redfordxx]

Not really.

The point is, that the camera shaking is defined by 6 parameters (x,y,z move and x,y,z rotation)

Deshaker estimates 4 of them. Omitting x and y rotation.

There is no way I can improve deshaker changing these 4 parameters.

I want to use DeShaker fully but on fake video(with a little improved shape using QUAD).

[johnmeyer]

OK. Good luck.

[redfordxx]

But if you have some deeper knowledge maybe you can help:
I am going to add borders to prevent losing data during warping. Then the border will be masked out.
Is there some rule about how much to add? I added double of search block and the estimation was slightly different.

[StainlessS]

Not sure what you are doing (and I aint no Deshaker expert), but maybe take a look here:- https://forum.doom9.org/showthread.php?t=173158

Using Sawbones/FrameSurgeon, can set up an eg 3x4 stack window image, including input and output, and say 9 variations of your
deshaker whotsit's, can (using Sawbones) insert commands into Cmd text file [EDIT: or use SCMD string in FrameSurgeon instead of using sawbones] to
select Frames/Ranges to use and from which variation clip.
Sawbones/FrameSurgeon supports up to 9 FXn replacement clips (ie your deshaker variation clips).

EDIT: 9 FXn clips, for use with both input and output[result] clips, then would need 1 dummy[blank] clip to make up 3x4 = 12 clips.

I would suggest that you pre-prepare as lossless, all 9 variation clips before using SawBones/FrameSurgeon.

EDIT: Demo script included with Sawbones/FrameSurgeon v1.11

Import("../FrameSurgeon.avs")

### Config #############################################
FINAL=False # Change to TRUE to do real DELETES (FALSE only shows where deletes would occur, ALWAYS use FALSE with SawBones)
### One of below strings
SCMD=""" # Demo Test
CP 100 # CP, Single frame args ONLY. Replace 100 with frame 99.
CN 101 # CN, Single frame args ONLY. REplace 101 with frame 102.
I9 110 # Id, Single frame args ONLY. Interpolate 9 frames starting at frame 110.
FX1 200,299 # Replace 200->299 with same range from FX1 clip. (FXd clips default source if not supplied, Subtitled if Show)
FX2 300,399
FX3 400,499
FX4 500,-50 # Replace 50 frames starting at 500.
FX5 550,-50 # Replace 50 frames starting at 550. NOTE, as not defined in this script FX5 defaults Subtitled source clip.
FX6 600,-100 # Replace 100 frames starting at 600. NOTE, as not defined in this script FX6 defaults Subtitled source clip.
FX7 700,-100 # Replace 100 frames starting at 700. NOTE, as not defined in this script FX7 defaults Subtitled source clip.
DEL 800,899
DEL 900,-100 # Del 100 frames starting from 900
DEL 2000,0 # Del from 2000 to end of clip.
"""
#SCMD=""

### One of below strings (CMD file is processed before SCMD, where multiple replacements on same frame)
CMD="FrameSurgeon_DEMO_Cmd.TXT" # Command file, perhaps created via SawBones.
#CMD = ""

SHOW=!FINAL # Show ClipClop OR Prune Metrics
SHOWMODE=0 # 0=ClipClop metrics, 1=Prune Metrics (Show remaining after deletions rather than replacements)
DV=4 # 0=ERRORS ONLY, 4=FULL Debug to DebugView(Google)
NOERR=False # Set True if wanting to IGNORE FXd commands for undefined FXd clips.

# MvTools, Interpolate args (see mvTools)
PEL=2
SHARP=2
RFILTER=2
ML=100.0
#
STACK=(!FINAL || SHOW) # Auto stack window OFF when FINAL==True and SHOW==FALSE
########################################################

### Source clip ###
src=Colorbars.killaudio.ConvertToYV12
src=src.ShowFrameNumber(Size=src.Height/6,x=src.Width/2,y=src.Height/3)

### Simulated replacement clips ###
FX1 = src.FlipVertical
FX2 = src.FlipHorizontal
FX3 = src.Blur(1.58).Blur(1.58).Blur(1.58).Blur(1.58).Blur(1.58).Blur(1.58).Blur(1.58).Blur(1.58).Blur(1.58).Blur(1.58).Blur(1.58)
FX4 = src.Sharpen(1.0).Sharpen(1.0).Sharpen(1.0)

### Edit or No Edit ###
CMD=(CMD!="") ? RT_GetFullPathName(CMD) : ""
PROCESS = (SCMD!="" || (CMD!="" && Exist(CMD))) # Skip FrameSurgeon if no String nor file commands (yet).

Result = (PROCESS)
\ ? FrameSurgeon(src,scmd=SCMD,cmd=CMD,Show=Show,ShowMode=SHOWMODE,dv=DV,
\ fx1=FX1,fx2=FX2,fx3=FX3,fx4=FX4,
\ pel=PEL,sharp=SHARP,rfilter=RFILTER,ml=ML,
\ Final=FINAL,NoErr=NOERR)
\ : src

### Stack For View ###
BSZ=4 BCOL=$FF00FF
Ssrc=src.RtSub("Source",True) SFX1=FX1.RtSub("FX1",True) SFX2=FX2.RtSub("FX2",True)
SFX3=FX3.RtSub("FX3",True) SFX4=FX4.RtSub("FX4",True) SResult=Result.RtSub("Result",True)
TOP=StackHorizontal(Ssrc,Ssrc.BlankClip(width=BSZ,Color=BCOL),SFX1,Ssrc.BlankClip(width=BSZ,Color=BCOL),SFX2)
BOT=StackHorizontal(SFX3,Ssrc.BlankClip(width=BSZ,Color=BCOL),SFX4,Ssrc.BlankClip(width=BSZ,Color=BCOL),SResult)
(STACK) ? StackVertical(TOP,TOP.BlankClip(height=BSZ,Color=BCOL),BOT) : Result
Return Last

# Stack Overhead RT_Subtitle Text, with optional FrameNumber shown.
Function RtSub(clip c,string Tit, Bool "ShowFrameNo") {
c.BlankClip(height=20)
(Default(ShowFrameNo,False)) ? ScriptClip("""RT_Subtitle("%d] %s",current_frame,""""+Tit+"""")""") : RT_Subtitle("%s",Tit)
Return StackVertical(c).AudioDubEx(c)
}

https://i.postimg.cc/rz7xnkjk/Frame-Surgeon-DEMO-00.jpg (https://postimg.cc/rz7xnkjk)

EDIT: I probably did not pick the best result frame[for you] for above image, is frame command as below, and not an FXn frame. [It does Interpolations as well as replacements]

I9 110 # Id, Single frame args ONLY. Interpolate 9 frames starting at frame 110.


EDIT: this a better pick using command FX3 400,499 ie replace with FX3 clip range 400-499, @ source frame 450

FX3 400,499


https://i.postimg.cc/WdhszhxR/Frame-Surgeon-DEMO-01.jpg (https://postimg.cc/WdhszhxR)

[redfordxx]

I already have this.function GetQuadString ( int w, int h, int "l", int "t", int "r", int "b", int "borders") {
l = default(l,0)
t = default(t,0)
r = default(r,0)
b = default(b,0)
borders= default(borders,0)

String(borders) +","+String(borders) +" "+String(-t)+","+String(-l)+" "
\ +String(borders+w)+","+String(borders) +" "+String(t)+","+String(-r)+" "
\ +String(borders) +","+String(borders+h) +" "+String(-b)+","+String(l)+" "
\ +String(borders+w)+","+String(borders+h) +" "+String(b)+","+String(r)

}


########## prepare quad ##########
function DoQuad (clip input, int "l", int "t", int "r", int "b", int "borders") {
l = default(l,0)
t = default(t,0)
r = default(r,0)
b = default(b,0)
borders= default(borders,0)

w=input.width-2*borders
h=input.height-2*borders

strx=GetQuadString(w,h,l,0,r,0,borders)
stry=GetQuadString(w,h,0,t,0,b,borders)

o = (l==0 && r==0) ? input : input.quad(stry, interleaved = true, relative = true)
(t==0 && b==0) ? o : o.quad(strx, interleaved = true, relative = true)
}


######### get multiple quads to analyse #######
function AnalyseMatrix (clip i0, borders, string "des_log_out", string "quad_log_in", string "des_set1", string "des_set2", int "radius", int "step") {
radius = default(radius,1)
step = default(step,8)
quad_log_in = default(quad_log_in,"")
des_log_out = default(des_log_out,"")
#des_set1 = default(des_set1,"19|1|20|4|1|0|1|0|640|480|1|1|1000|1000|1000|1000|4|1|0|2|8|30|400|4|")
des_set1 = default(des_set1,"19|1|20|4|1|0|1|0|640|480|1|1|1000|1000|1000|1000|4|1|0|2|0|30|400|4|")
#des_set2 = default(des_set2,"|1|0|0|0|0|0|0|0|0|0|0|0|0|1|15|15|5|15|0|0|30|30|0|0|0|0|1|1|0|10|1000|1|88|1|1|20|5000|100|20|1|1|ff00ff")
des_set2 = default(des_set2,"|1|0|0|0|0|0|0|0|0|0|0|0|0|1|15|15|5|15|0|0|30|30|0|0|0|0|1|1|0|10|1000|0|88|1|1|20|5000|100|20|1|1|ff00ff")

#inplement scripting
#quad_log_in == "" ? blankclip : i0
i1=i0.Trim(1,0)
#here will be for cycle later (for higher radius)
int_clip = (radius==1) ? Interleave ( i0, i1 ,
\ i0.DoQuad( step, step, 0 , 0 , borders) , i1
\ , i0.DoQuad( step, 0 , 0 , 0 , borders) , i1
\ , i0.DoQuad( step, 0 , 0 , step, borders) , i1
\ , i0.DoQuad( 0 , step, 0 , 0 , borders) , i1
\ , i0.DoQuad( 0 , 0 , 0 , 0 , borders) , i1
\ , i0.DoQuad( 0 , 0 , 0 , step, borders) , i1
\ , i0.DoQuad( 0 , step, step, 0 , borders) , i1
\ , i0.DoQuad( 0 , 0 , step, 0 , borders) , i1
\ , i0.DoQuad( 0 , 0 , step, step, borders)
\ ) : i0

des_clip = (des_log_out=="") ? int_clip
\ : int_clip.deshaker(des_set1+des_log_out+des_set2)

des_clip
}Do you think Sawbones/FrameSurgeon would be any useful around the part where creating the matrix (the big Interleave)?

[StainlessS]

No idea what that lot is doing (I aint ever used quad and rarely have ever used Deshaker).

SawBones, uses VirtualDub2 as GUI for creating command text file for FrameSurgeon, pressing key sequences [in VD2]
inserts commands into text file, FrameSurgeon then sustitutes source frames/ranges with FXn frames/ranges.
You would need the final output versions [lossless AVI] of your Deshaker variations as FXn inputs for Sawbones/FrameSurgeon.

EDIT: Its a total mystery to me what your script does :)

[redfordxx]

I have prepared Rotation script and am going back to work here.
Can I ask for some help to rewrite that kind of script using for loops in AVS+?
i1=i0.Trim(1,0)
#here will be for cycle later (for higher radius)
int_clip = (step==8) ? Interleave ( i0, i1 ,
\ i0.RotateCamera( FoV, 360/step * 0.0, radius, has_borders=add_bord, info=info, lock_central=true) , i1
\ , i0.RotateCamera( FoV, 360/step * 1.0, radius, has_borders=add_bord, info=info, lock_central=true) , i1
\ , i0.RotateCamera( FoV, 360/step * 2.0, radius, has_borders=add_bord, info=info, lock_central=true) , i1
\ , i0.RotateCamera( FoV, 360/step * 3.0, radius, has_borders=add_bord, info=info, lock_central=true) , i1
\ , i0.RotateCamera( FoV, 360/step * 4.0, radius, has_borders=add_bord, info=info, lock_central=true) , i1
\ , i0.RotateCamera( FoV, 360/step * 5.0, radius, has_borders=add_bord, info=info, lock_central=true) , i1
\ , i0.RotateCamera( FoV, 360/step * 6.0, radius, has_borders=add_bord, info=info, lock_central=true) , i1
\ , i0.RotateCamera( FoV, 360/step * 7.0, radius, has_borders=add_bord, info=info, lock_central=true)
\ ) : i0.RotateCamera( FoV, 360/step * 0.0, radius, has_borders=add_bord, info=info, lock_central=true)

I need to go through 360 / step * float

[Hotte]

Very nice to hear that sbd is trying to pick up the challenge to optimize this excellent filter.

Let me just through in sth from the boardwalk.

I found so far that rotation issues are not really bad with Deshaker. It is the wobbling side effect that distracts when using wide-angle lenses. This is sad since wide angle is used a lot these days, think of drones etc. The wobbling issue is sth which Deshaker shares with all the other - even commercial - stabilizers out there. In some thread sbd told, that wobbling is a very complicated issue, because some whatever "3D" approach is to be considered to compensate for the perspective correction. Sounds threatening...

With the most recent version 5 of the Mercalli suite I can see really significant improvement with the wobbling compensation with the new RT-Filter (not the old Pro-Filter). But the temporal border compensation of Deshaker is still so much more powerful than with Mercalli so that I tend to not go for the (expensive) commercial solution.

Just talking about a dream....Deshaker....no wobbling

[redfordxx]

Just talking about a dream....Deshaker....no wobbling
You might check the 1st post update...

[poisondeathray]

Did your test camera have rolling shutter ? and did you correct for lens distortions before the runs?

I can see this possibly helping in some limited other cases - x,y rotations with global shutter cameras, corrected lens distortion, and there when there no perspective or parallax issues and image is essentially on 1 focal plane. Distant objects will appear to move at different rates compared to closer objects. Similar to uncorrected lens distortions with periperhal and central

How well did %OK blocks correlate with human eyes? You mentioned you got improvements in other directions simultaneously... Anther metric or method might be PSNR sort of like avs optimizer (zopti) uses. You might be able to combine or refine the process

Regardless of the end results, I appreciate the testing and effort put into this. And you came up with a nice camera rotation emulation script for avs - so kudos for that

[redfordxx]

Smartphone ... I believe has no rolling shutter.

Distorion is small but tested, not much influence.

[redfordxx]

I knew there could be issue with the distance ... but hoped the rotation would help more than it does...

[poisondeathray]

Smartphone ... I believe has no rolling shutter.


100% of camera phones / smartphones have rolling shutter . This definitely can pollute your results

RS is the worst "evil" by far; it throws all types of tracking attempts off, even 3d camera solving

Generally, you'd have to correct for RS (and lens distortion) before you do anything else . But there are no complete RS solutions, not even paid, not even higher end professional tracking software that cost many times more than mercalli

[redfordxx]

How about assumption, that there are no multiple objects in multiple distances, but only one plane....not 90 degrees to the viewer but at certain angle.
Everything outside this plane will be considered moving objects or bad vectors.

That would leave us with 8 variables in total: 4 deshaker's, 2 camera rotation, 2 plane angle
Could that be close to reality enough?


But it is loooot of calculation I don't feel like.

[Hotte]

You might check the 1st post update...

Thank you. I found out that tele or normal lenses will produce very little to no wobbling when being deshaked. Wides are prone to it.
I believe this has little or nothing to do with optical barrel distortion but with the different "motion speed" in the center compared to the edges. It is the character of the lens and cannot just simply be compensated.

So "that different parts of image have different motion estimation" sounds right to me.

[redfordxx]

100% of camera phones / smartphones have rolling shutter . This definitely can pollute your results
damn!

[redfordxx]

I give it some effor later maybe, but i probably cannot put any transformation before the deshaker first pass because I would destroy the chance for the deshaker to deal witth RS.
Only debarrel goes before deshaker? That would be correct?

I have the a,b,c parameters but I don't know whether I should set pin true or false?