Avisynth "morph" function, way to automate it and stability concerns

[abolibibelot]

I'm trying to correct footage (important and unique) which was taken with a camera having a dysfunctional stabilization system, generating jerkiness all by itself (mostly on the vertical axis), even when it was held with a relatively steady hand (steady enough so that the stabilization system would normally have corrected the slight amount of jerkiness instead of doing the opposite – anyway, what's done is done).
I used the stabilization module coming with the editor I'm using (Magix Video Deluxe), which made quite an improvement, but couldn't correct individual frames which are blurred because of that jerkiness.
Then I discovered Avisynth's “morph” function found on this forum, which can effectively interpolate those bad frames from the surrounding good ones. That thing is quite impressive, if there isn't too much motion the result is virtually perfect, the interpolated frames aren't even noticeable ; there are very few cases where it doesn't produce a satisfying result. But that's an almost overwhelming task to go through about 35 minutes of footage looking for blurred frames and correcting them one by one...

So, is there a way to automate the process, to detect the blurred frames with some script and correct them on-the-fly, or at least generate a list of frame numbers which I could then convert to a list of morph(x, y) calls ?

And also, I'm having concerns about the possibility of running such a heavy-duty Avisynth script. So far I have about 700 "morph" calls, it proved unstable within AVSPmod when more than a few dozens are active, I have to disable them regularly to be able to continue, and it makes it crash quite often if I forget. I have a quite powerful machine, i7 6700K and 16GB of RAM, but won't even such a system choke if I try to run an x264 encode with that kind of script as input ? Or is Avisynth designed in such a way that commands are treated in their sequential order without putting an excessive strain on the system ? Is AVSPmod known to have stability issues with long scripts ?

Thanks.

[wonkey_monkey]

How many bad frames in a row do you get, at most?

[abolibibelot]

How many bad frames in a row do you get, at most?

Most are unique (but there can be several in close succession, with good frames in between), in some cases there are two in a row, there are very few areas with more than two in a row.

Here's an example (in one area where there happens to be three in a row) :


(native frame)


(interpolated frame using morph function)

(Is there a way to resize pictures so as to not affect the page's layout ? O_o)

[StainlessS]

Perhaps of use for manual fix:- https://forum.doom9.org/showthread.p...light=Sawbones

EDIT: Play clip in VirtualDubFilterMod whilst using Sawbones, stop on problem frame, press a SawBones key sequence
to insert a command into command file, repeat until clip end.

Process in Avisynth using above created Command file and FrameSurgeon script function.

Perhaps quicker and less effort than attempting to write successful detection function.
(Max interpolation of 9 frames, EDIT: If greater than 9 frames, could leave until totally manual fix later)

EDIT: For later manual fix, Clipclop's DoctorFrames Script handles up to 20 interpolated frames.
Clipclop:- https://forum.doom9.org/showthread.php?p=1520076

[abolibibelot]

I'm currently trying this method. I installed all the required plugins (took some time to know what was required and how to get it), created an empty Command.txt file, made an AVS script with :

AVISource("I:\20131224\20151224.avi")
ConvertToYV12(matrix="Rec709")
FrameSurgeon(Cmd="G:\20131224 M2TS\Command.txt")

which loads into VirtualDub FilterMod, I ran SawBones.exe with both windows visible as instructed, but the keyboard shortcuts don't work so far. Did I forget something or do something wrong ?

Do you know how FrameSurgeon performs compared with Morph ?
Is DoctorFrames() another frame interpolation function, or a part of the tools used by FrameSurgeon ?

Perhaps quicker and less effort than attempting to write successful detection function.

If I have to do it myself that's for sure, as I've never written any kind of program yet... But I guess that this would be a tough task even for someone with good programming skills and knowledge and experience in the particular field of digital video.





EDIT :

(Max interpolation of 9 frames, EDIT: If greater than 9 frames, could leave until totally manual fix later)
EDIT: For later manual fix, Clipclop's DoctorFrames Script handles up to 20 interpolated frames.

Do you mean 9 frames in total, or 9 consecutive frames ? If 9 consecutive, that's plenty enough, if it's 9 total, I need much, much more than that (as I said, I already made more than 700 morph() calls).

[StainlessS]

Meant 9 consecutive frames for interpolate.

Have found bug in FrameSurgeon, can you make this change to line 201
was

        BSZ     = c.Height/40  FSZ=c.Height/5
        LBOX    = c.LetterBox(BSZ,BSZ,BSZ,BSZ,$0000FF)

Mod

        BSZ     = c.Height/(40*2)*2  FSZ=c.Height/(5*2)*2
        LBOX    = c.LetterBox(BSZ,BSZ,BSZ,BSZ,$0000FF)

What OS, and keyboard are you using ?

EDIT: Script that I'm gonna play with

Import("FrameSurgeon.avs")

AVISource("1941 Flint Michigan Parade [Low, 360p].mp4.AVI")
Crop(0,0,Width/4*4,Height/4*4)  # fix odd dimensions
#Return Info

ConvertToYV12(matrix="Rec709")
FrameSurgeon(Cmd="Command.txt",Show=True,Final=False)

Return Last

EDIT: Added Show=True

EDIT: You may need to run as administrator, and maybe any Antimalware might block keyboard control stuff in SawBones,
might think some kind of keyboard logger or something.

EDIT: When click from some other active window to VDub window, should flash up a "SawBones is watching" message for a few secs.
(NEXT TO MOUSE)

EDIT: Sawbones icon should be visible on task bar (RHS) when active.

EDIT:
You have script loaded into VirtualDubFilterMod,
Existing "Command.txt",
Command.txt loaded into running and visible edit window of "NotePad.exe",
Sawbones visible on TaskBar.
And Open VirtualDubFilterMod script editing window, on taskbar.

EDIT: Mine aint working either, maybe some change to current VirtualDubFilterMod or something, give us a little while to investigate further.

[abolibibelot]

What OS, and keyboard are you using ?

Windows 7 Pro, AZERTY (french) keyboard. Don't have any antimalware stuff installed currently (I know it's bad... it's a relatively new install, I was using Avira Antivirus before but it became really annoying, not sure what to use now).

When click from some other active window to VDub window, should flash up a "SawBones is watching" message for a few secs.
Sawbones icon should be visible on task bar (RHS) when active (NEXT TO MOUSE).

I get a very quick flash but no message, even though the process does appear in Task manager. What is RHS ? What do you mean by “next to mouse” ?
Also, I have a gazillion Notepad instances opened... And I'm using Notepad2 as a replacement for the default Notepad (but normally it should behave just the same for anything that uses Notepad, shouldn't it ?).

EDIT : There's a SawBones icon on task bar, when I hover over it there's “(Paused)” beside the name, and if I click on it (left or right click), a contextual menu appears with “Script paused” checked and “Exit”, but I can't uncheck it.
EDIT : OK, now I can, it's no longer paused, but still doesn't work.

Have found bug in FrameSurgeon, can you make this change to line 201

What kind of issue was it provoking, and how did this change solve it ? (Just curious...)

[poisondeathray]

Are you using avisynth/avspmod x64 ? or x86 ?

To rule out if it's an avspmod issue, try vdub/vdubfiltermod

Are you using a recent mvtools2 ? There are older versions that are prone to crashes. Try the pinterf branch

But it's probably a trim() issue . Each call has multiple trims and avisynth doesn't like thousands of trims

As a workaround for now, split up the script, save out the script to an intermediate for each section






For detection, I wouldn't trust most approaches on important footage. Too much room for error. Even if it's 95% accurate you risk missing bad sections; but even worse - false positives mean you risk losing good original frames.

Moreover, doing yourself and previewing gives you the option to evaluate the results. There are several different interpolation function variants and various settings that might get better results if your "default" result was poor. Example for one section a larger blocksize might help, for another maybe dct=1, etc..

Or sometimes, the interpolated result is worse than the original, then you can *see* that and use another method or adjust accordingly






But brainstorming some ideas - maybe frame differences . The problem with avisynth runtime functions such as YDifferenceFromPrevious is that it will fail with 2 sequential blurry frames . Or something normal like turning on a light would be flagged inappropriately. Lots of scenarios for misdetection

Maybe motion vectors. Or maybe something like deshaker's log file run through some excel script to filter the results . Large deltas maybe in rotation should be areas where there is large motion

I don't know of anyway to automatically/accurately detect unwanted "what is a blurred frame"

[StainlessS]

I get a very quick flash but no message, and no icon visible on task bar, even though the process does appear in Task manager. What is RHS ? What do you mean by “next to mouse” ?

RHS means Right Hand Side.

If icon not visible then is not running properly, dont know why (especially as no anti-malware running), it should work ok on W7 (I think).
Anybody confirm work/not on W7 ?

Can ONLY use standard NotePad, it looks for the system NotePad, and nothing else.
Also, NOT good idea to have multiple instances open, it will not have a clue which one it should insert commands into.

What kind of issue was it provoking, and how did this change solve it ?

The Mod code block contains the fix (In previous post).

Dont know why it previously did not work for me, suspected some kind of Graphics card shortcuts were interfering, but "Ati hotkey Poller" is disabled in my
services and so should not be problem. I did a reboot, and now its working fine, no idea what the problem was.

Suggest try standard NotePad, maybe icon not visible is not a restricting problem.

Azerty Keyboard might need some mods to your Sawbones.ini file, if you can figure out which keys (if any) dont work as expected.
(just need to try the command key sequences in docs and see that they all insert correct commands into Command.txt).

EDIT: The text "(NEXT TO MOUSE)", I had pasted at end of wrong line, fixed in original post in blue, sorry for confusion.

EDIT: Just seen your edit that icon is visible when mouse hovers over it, that is probably down to your TaskBar config settings
and is therefore ok, it should work fine if you use the system notepad. (it has to use system notepad to access its controls,
some other notepad eg NotePadPlus or whatever, has different executable name and different controls, so will not work).
Azerty may though still be a small problem for you to fix in ini file.

EDIT: You can do a simple Right Click on Command.txt and "Open With" NotePad.exe in context menu (its probably available there as an option without you having to look for it in its directory [on XP32 NotePad.exe is in system32]).

[abolibibelot]

I made a test with AVSPmod, running FrameSurgeon on the same frame shown above :


The result seems to be more precise than Morph, it deals with moving objects better (that man's legs for instance), with no ghosting effect. I'll try it with problematic frames which I couldn't improve satisfyingly with Morph.

And so, can I safely run a script with hundreds of frame interpolations, with either one of those functions ?

[StainlessS]

You also need Show=True in FrameSurgeon script to view edits within Sawbones when you press the "CTRL/ALT," [thats a comma at the end] sequence,
ie as so (for my test)

Import("FrameSurgeon.avs")

AVISource("1941 Flint Michigan Parade [Low, 360p].mp4.AVI")
Crop(0,0,Width/4*4,Height/4*4)  # fix odd dimensions
#Return Info

ConvertToYV12(matrix="Rec709")
FrameSurgeon(Cmd="Command.txt",Show=True,Final=False)

Return Last

EDIT:

And so, can I safely run a script with hundreds of frame interpolations, with either one of those functions ?

Thousands [if you find a limit, let me know]

EDIT: If you find that you need to change ini for AZERTY (French I think) keyboard, then please post mods in the Sawbones/FrameSurgeon thread for other users of Azerty, thanx.

EDIT:
Here doc for the "CTRL/ALT," (control / alt /COMMA) command, SAVE & REFRESH, ie view edited clip in SawBones,
with deletes shown marked for deletion only (otherwise would screw up further editing (thats why FINAL=FALSE is set))

    CTRL/ALT/,          Save & Refresh.
                        Ie, Save NotePad Command.txt, refresh/reload Avisynth Script, jumping back to original position in VDubFilterMod.
                        NOTE, v1.04, We have Added this "CTRL/ALT/," key combo.
                        You can change this key combination used for "Save & Refresh" by changing the ini file item
                        "KEY_Save=^!," to something else, in sequence they are CTRL(^) SHIFT(+), ALT(!) and ',' keys
                        pressed together. The Sawbones.ini file is created upon first use of the executable.
                        NOTE, it is quite difficult to find key combo's that are not used system wide, and also not used by VirtualDubFitlerMod.
                        It does also not improve matters where eg Graphics cards install a whole bunch of additional hotkey combos, reducing what
                        is available to the user. (Who actually thinks they need permanent system wide hotkeys to access graphic card settings,
                        almost useless and just an annoyance [particularly if they supplant your own hotkeys], disable if at all possible).
                        The key combos for "Interpolate Range" and "Save & Refresh", may not work on some international keyboards and so you can
                        change key combo in the ini file, a URL link to further info will be embedded in the ini file.
                        Hopefully all of the other Sawbones key combos are available to you, they were made deliberately a little awkard
                        in the hope that they are not used elsewhere on your system, if they already in-use then I'm afraid you may need to mod the
                        Sawbones source and rebuild it (after ascertaining what keys ARE available to you).
                            NOTE, Save & Refresh requires that VirtualDubFilterMod Script Editor window is also open to refresh the avisynth script
                        (can be left open on task bar, you do not need to see the contents of the script window). If Script Editor is NOT open,
                        then only a NotePad save of the Command text is done, we cannot refresh/reload the avs script into VDub if script editor
                        is not open (or not avialable, so will not work in Standard VirtualDub).
                            IMPORTANT:- See also above DELETE commands.

EDIT:
Here, result of sawbones DEL 0 (delete frame 0) command, after Save & Refresh key sequence executes,
as seen in VirtualDubFilteRMod with SawBones active.



EDIT: When final = TRUE, above frame deleted for real and not just marked for deletion (Must use Final=False if using SawBones).

[abolibibelot]

If icon not visible then is not running properly, dont know why (especially as no anti-malware running), it should work ok on W7 (I think).

I found it afterwards (I may have terminated the process and forgot to re-run it). But I still don't get a message saying "SawBones is watching".

Can ONLY use standard NotePad, it looks for the system NotePad, and nothing else.

Just seen your edit that icon is visible when mouse hovers over it, that is probably down to your TaskBar config settings
and is therefore ok, it should work fine if you use the system notepad. (it has to use system notepad to access its controls,
some other notepad eg NotePadPlus or whatever, has different executable name and different controls, so will not work).

Oh, right... Couldn't it be implemented in a future revision ? Even if each text editing tool has a different layout, CTRL+V works in all of them.

You can do a simple Right Click on Command.txt and "Open With" NotePad.exe in context menu (its probably available there as an option without you having to look for it in its directory [on XP32 NotePad.exe is in system32]).

Doing that just opens Notepad2. I tried to open directly the notepad.exe inside system32, it does the same (even though it seems to be the native one, it says it's from Microsoft).
Copying it to another directory : same. Changing its name to notepad3333.exe : it doesn't run at all ! O_o
Anything else I can try, short of uninstalling Notepad2 ?

[StainlessS]

I found it afterwards (I may have terminated the process and forgot to re-run it). But I still don't get a message saying "SawBones is watching".

Not sure, think will only flash that message if both VirtualDubFilterMod (which identifies itself as VirtualDub and has same controls), and NotePad.exe are both running.

Oh, right... Couldn't it be implemented in a future revision ? Even if each text editing tool has a different layout, CTRL+V works in all of them.

NO, needs more than Paste from clipboard, and controls in some other version of a notepad exe are not likely to be named exactly the same and of same type.

You surely must be able to load notepad via Accessories menu, if not then they are some bad habits that NotePad2 has got, and I would get rid of it.

Sorry, have no intention of writing multiple versions of Sawbones for all variations of notepad replacements, that was why system NotePad was chosen.

AutoiIt source is supplied for anyone willing to spend (what seems like a daft waste of time) to get past NotePad2 peculiarities.

[StainlessS]

From AutoIt source

If you can find names for below, in NotePad2, then might work

Global Const $EdTitle     = "[CLASS:Notepad]"                        ; NotePad
Global Const $EditControl = "[CLASS:Edit; INSTANCE:1]"

Below should work, so long as names above ok and NotePad2 access CTRL/S as Save As.

Func NotepadSave()
	ControlSend($EdTitle, "", $EditControl, "^s" , 0) ; Send CTRL/S to NotePad, ie Save (FileSelector pop-up if not named)
	If NOT WinExists($ScriptTitle) Then
		ToolTip("Sending NotePad Save ONLY")
	Else
		ControlSend($ScriptTitle, "", $ScriptControl, "{F5}" , 0)   ; Send F5, save File and  Refresh to VirtualDubFilterMod Script Editor
		ToolTip("Sending ScriptEditor Save & Refresh")
	EndIf
	$ShowCnt = $ShowCntLimit * 4
Endfunc

Then Re-Build, F5 key I think in AutoIt editor.

Can use the window tool (forget what its called in autoIt) to identify the Class name of NotePad2 and also the edit control (which may already be called Edit).

EDIT: The window tool mentioned above is AU3Info on the tools menu of the ScITe editor for AutoIt (available on the AutoIt website when download). It identifies internal names of controls and stuff of windows including program windows and controls.

EDIT: By the way, I use PsPad and NotePad+, neither of which prevents me using system NotePad, if they did, I would ditch them.

EDIT: Is NotePad2 that you use, this one ? :- http://notepad2.com/

EDIT: If it is above notepad2, then below looks like it should work, change and F5 to rebuild.

Global Const $EdTitle     = "[CLASS:Notepad2]"                        ; NotePad2
Global Const $EditControl = "[CLASS:SysListView32; INSTANCE:1]"

OK, compiled SawBones for above NotePad2 from http://notepad2.com/
Available from SendSpace below this post, until link expires.

EDIT: The build key in AutiIt editor is actually F7 not F5.

[abolibibelot]

You surely must be able to load notepad via Accessories menu, if not then they are some bad habits that NotePad2 has got, and I would get rid of it.

Well, it was designed to replace the native Notepad in every way, so it does it as it's supposed to. But it's optional, and reversible. Since it's far richer in features and also runs much faster with large files, I think it's very useful and welcome tweak. It uses some Registry trick :
“The Notepad2 Setup Tool can be used to replace Windows Notepad with Notepad2, using a clean, unintrusive registry-based method (read about the technical details).”
I would say that it's the native Notepad's behaviour which is problematic, making it necessary to use such a trick to open text files with anything else by default.

AutoiIt source is supplied for anyone willing to spend (what seems like a daft waste of time) to get past NotePad2 peculiarities.

Well, I'll try that tomorrow... (or later today should I say – it's 8 AM here and I haven't slept yet !)
Thanks for the explanations, and of course many thanks for designing those tools...

In any case, if I decide to use FrameSurgeon from now on (I'll probably do as it seems to be an improvement over an already excellent little function), how can I translate all the commands I already wrote for Morph without having to type them again ? I was thinking of using OpenOffice Calc for that purpose.


Another test with a problematic frame : in this case FrameSurgeon doesn't perform better than Morph, it's actually worse :

[StainlessS]

Have supplied NotePad2 version in above post.

Is it correct NotePad2 ?

EDIT:

In any case, if I decide to use FrameSurgeon from now on (I'll probably do as it seems to be an improvement over an already excellent little function), how can I translate all the commands I already wrote for Morph without having to type them again ? I was thinking of using OpenOffice Calc for that purpose.

Post a zip of morph commands, I'll convert for you.

When working OK, you will be able to see result when using Sawbones/FrameSurgeon and chose to use Morph if bad (later).

EDIT: OK, had a quick look at your "technical details" link, looks like same NotePad2 (same original author) but a modified version (much later version mod, by someone else) but guess that there is still a fair chance of it working ok, give new Sawbones a bash.
By the way, I dont have much intention of making future multiple versions of sawbones, you will have to make the mods
yourself, copy and paste of two strings, and pressing F7 is not so very hard.

[abolibibelot]

After deleting the Notepad2 Registry key (see link in previous post) I can run the native Notepad (I saved it so I could re-import it afterwards).
Now I have the “SawBones is WATCHING” message when clicking on the VDFM window, and CTRL+DEL works as intended. But CTRL+SHIFT+ALT+1 doesn't work. I'll try to edit the .ini file later today.

[StainlessS]

Great stuff.

To assist in CTRL+SHIFT+ALT+1, suggest test first if being used by something else.

eg, copy some exe to desktop.
create a shortcut to it.
right click "Properties", and click on "Shortcut Key", and enter the shortcut.
If it prevents you from entering the shortcut, then looks like something else is already using it (maybe Graphics card).

EDIT: Anyway, I've gotta get some sleep too, Good Night.

EDIT: If doing above and successful setting of shortcut, then go back and remove the shortcut, or it will be in-use.

[StainlessS]

NOTE:
For CTR/ALT/SHIFT/1,
it requires a range, ie you need to mark VirtualDub range with HOME end END keys, or it will not work (flashes a 'no range' type message I think).

EDIT: Maybe would be more sensible to presume current frame interpolation, but then that would work different to the other
(multiple frame) interpolation commands.

EDIT: Nope, was talking rubbish above, when I press CTRL/ALT/SHIFT/1, it does indeed insert current frame for interpolation.
It's the "CTRL SHIFT ALT /" (last one is slash) Interpolate Range command sequence that needs a vdub marked range.

[abolibibelot]

By the way, I use PsPad and NotePad+, neither of which prevents me using system NotePad, if they did, I would ditch them.

Again, that's optional ; on the page I linked the author explains that this feature was first proposed as an unofficial trick, then implemented in the default install (most likely because many users requested it). He also explains that the same trick is used by ProcessExplorer (from SysInternals which has been acquired by Microsoft) to replace the native Task manager, so it's nothing shady or malicious.

It works after temporarily deleting the registry key, but I'll still try to re-build the program to use with Notepad2, following your instructions above. (I recently installed AutoIt, hoping that it could help me with a totally different task, but didn't get very far.)

Post a zip of morph commands, I'll convert for you.

That's really nice of you, but I'd rather learn the method, and it may be useful for someone reading this thread later on and having a similar issue.

When working OK, you will be able to see result when using Sawbones/FrameSurgeon and chose to use Morph if bad (later).

Yes, anyway I will review each processed frame before proceeding with the encoding. For this particular frame I have deemed the result with Morph unsatisfying and disabled the corresponding command, so neither of them can improve over the native frame (unless the function can be tweaked) ; there are only a few like that so it's not so bad (and it happens when there is a lot of motion so it's barely noticeable anyway). I tested FrameSurgeon with other frames which were problematic with Morph and FrameSurgeon generally produced a decent result.