warp - geometry correction with control points (also includes new version of quad)

[wonkey_monkey]

https://i.imgur.com/5Aerv2g.png

warp v0.1 (http://horman.net/avisynth/) <- click here, it's a link!

warp is a geometry correction filter which uses control points and a thin plate spline (as used in morphing) to correct or distort a video's geometry. I have no idea if anyone is going to find this useful.

Documentation (such as it is), examples, and source are included. The same DLL also provides a new version of quad (https://forum.doom9.org/showthread.php?t=165978) which supports more colour spaces (all the "classic" ones, plus 8-bit planar RGB).

x86 and x64 DLLs, both static and dynamically linked, are provided. SSE2 is definitely required, possibly SSE3. Sorry, luddites!

It's so fancy it even has its own demo/tutorial video: https://www.youtube.com/watch?v=SG1eex7HmLM

Recommended companion software: Hugin (http://hugin.sourceforge.net/)

warp (v0.1)
===========
A collection of geometry correction filters using control points, specified
either as a list of numbers, a string, or with a Hugin project file
(see http://hugin.sourceforge.net/)

(c) David Horman, 2019

Description
===========
warp takes a list of control points - pairs of image coordinates - and
warps the input video using a thin plate spline (as used in morphing)
such that the coordinates of the first point in a pair are moved to the
position of the second point in the pair, with the rest of the image
distorted as smoothly as possible.

Control points are specified either directly as filter parameters, or as a
whitespace- and/or comma-separated string, or by passing the filename of a
Hugin project file (.pto). A separate guide to creating a warp-compatible
Hugin project is included.

quad is a reimplentation of my old quad filter. It remaps an image based on
one or two specified quadrilaterals.

warp_series takes map frames generated by warp and uses them as animation
keyframes, blending them to provide smooth transitions between warps.

warp_grid is a helper filter which draws a grid on a video clip.

Please refer to the individual documentation files for usage and parameters.

[poisondeathray]

Nice, thanks! I will test drive it

I was just going to ask about alignment options in avisynth; something similar to hugin or photoshop autoalign with perspective and warping corrections instead of just global frame translation/rotations

[StainlessS]

Lovely WonkyWilly, :)

Maybe point out availability [EDIT: Where to get the plug] via your sig (took me bout 40 -> 60 seconds to figure it out).

[wonkey_monkey]

Lovely WonkyWilly, :)

Maybe point out availability via your sig (took me bout 40 -> 60 seconds to figure it out).

Huh? Figure what out?

And my availability is none of your business, cheeky ;)

[StainlessS]

Aha, the link is the enourmous warp v0.1 below the top image, me is a dick https://www.cosgan.de/images/smilie/froehlich/c030.gif

[poisondeathray]

Works great. Nice video tutorial

When combined with gmic clut method , it's a power color matching technique . The benefit is you can use the BD chroma (or other src image) resolution . It's a bit colourlike on steroids. Results are better than some commercial color matching plugins and workflows. The negatives are the workflow is a bit clunky and requires per scene matching
https://forum.doom9.org/showthread.php?p=1862294#post1862294

The haldclut => cube conversion can be done with python script instead of the one suggested in that thread. It's faster/easier , batchable and the crc check is the same.
LUT-Convert-master
https://github.com/mikeboers/LUT-Convert
Since it's python based, it makes me wonder if it could be made adaptable to vapoursynth


Perhaps out of the scope of this plugin discussion -

While manual control point is great for accuracy and fine tuning - I'd like to see some way of auto matching possibilities - are you aware of any ? The hugin bundled ones aren't that great . eg. Are there any FOSS engines that hugin can use externally ? Even if it's a rough control point auto layout matching generation, it would reduce the amount of manual fine tweaking later

I'm wondering if this can somehow be used/incorporated into the autooverlay plugin . It "feels" like a powerful pairing in the making
https://forum.doom9.org/showthread.php?t=175247

[wonkey_monkey]

Yeah, I was disappointed with Hugin's automatic control point thing as well. Firstly, it works better if the videos are the same resolution - and it's probably not a terrible idea to NNEDI the DVD footage first, before trying to match it to Bluray footage. Secondly, I think all such control point finders (Autopano is another) are optimised for matching photos from the same set, rather than different quality/colour footage like this. And thirdly, I expect it's ultimately an intractable problem anyway, in that you can never guarantee you won't get bad control points.

Hugin also has a fine tune for control points, but I also had problems with that (I mention it at the same start of the video, but in the end I didn't bother expanding on it).

Maybe something can be done with mvtools, using large block sizes and filtering for good/consistent vectors.

[StainlessS]

Welcome to the forum Wonkey, I'm sure that you will find it a non stressful place where you can expend lots of decades.

EDIT: You got Wonkey Monkey . com, be quick else you got squatters to eradicate. [hitmen are expensive, or so I've been told]

[trondmm]

Thank you. This is great.

The biggest problem I have at the moment, is that it sometimes fails silently. i.e. Nothing happens, there's no error message, and debug=WARP_DEBUG_ISOMAP returns a completely black image

This happens when the pto-file doesn't exist, but also when the file actually exists. I'm not sure if warp for some reason is unable to open the file, or if there's something in the file that causes warp() to croak.

If I copy the points from the file manually, warp() works as expected.

[trondmm]

Ah. Found it!

If I change the control point lines from
c n1 N0 x1001 y747 X952 Y784 t0

to

c n0 N1 x1001 y747 X952 Y784 t0

it works.

[wonkey_monkey]

Hmm. I assumed Hugin would always put them in ascending order. For now, just make sure image #0 is in the left pane and image #1 is in the right pane.

[StainlessS]

Wonkey,
A few probs in docs for warp and quad.

from quad,.txt

warp(
(clip)
(floats) (optional)
(string) string (optional)
(bool) swap = false
(bool) relative = false
(bool) pto_resize = true
(int) resample = WARP_BICUBIC = 2
(int) quality = 6
(float) scale = 1
(float) scalex = 1
(float) scaley = 1
(float) scale1 = 1
(float) scalex1 = 1
(float) scaley1 = 1
(float) scale2 = 1
(float) scalex2 = 1
(float) scaley2 = 1
(clip) overlay = NULL
(int) overlay_threshold = 0
(int) threads = 0 (automatic)
(bool) interleaved = false
(int) debug = 0
(int) map = [undefined]
)


Should I think be as so

quad( # Wrong name, was warp
(clip)
(floats) (optional)
(string) string (optional)
(float) strength = 1.0 # Missing from description (also no documentation)
(bool) swap = false
(bool) normal = true # Missing from description (also no documentation)
(bool) relative = false
(bool) pto_resize = true
(int) resample = WARP_BICUBIC = 2 # EDIT: Maybe should be QUAD_BICUBIC
(int) quality = 6
(float) scale = 1
(float) scalex = 1
(float) scaley = 1
(float) scale1 = 1
(float) scalex1 = 1
(float) scaley1 = 1
(float) scale2 = 1
(float) scalex2 = 1
(float) scaley2 = 1
(clip) overlay = NULL
(int) overlay_threshold = 0
(int) threads = 0 (automatic)
(int) map = [undefined] # Wrong order
(bool) interleaved = false # Wrong order
(int) debug = 0 # Wrong order
)


Need same changes for warp.txt (except for the warp->quad name change).

Extracted prototypes


warp_ORDERED_Function_List


There follows a list of all function names together with CPP style argument specifiers that inform
Avisynth the argument types and optional names. Optional arguments have square brackets surrounding
their name as in [name] and are followed by a type specifier character that gives the type.
Unnamed arguments are not optional. eg "cc[arg1]b[arg2]i" would be two compulsory unnamed clip args,
followed by optional 'arg1' of type bool and optional 'arg2' of type int.

# Argument type specifier strings.
c - Video Clip
i - Integer number
f - Float number
s - String
b - boolean
. - Any type (dot)
# Array Specifiers
i* - Integer Array, zero or more
i+ - Integer Array, one or more
.* - Any type Array, zero or more
.+ - Any type Array, one or more
# Etc
###################################


BICUBIC ""
BILINEAR ""
DEBUG_GRID ""
DEBUG_ISOMAP ""
DEBUG_MARKERS ""
DEBUG_OVERLAY ""
DRAFT ""
grid "c[size]i[darken]f[colour]i"
LINEAR ""
quad "cf*[string]s[strength]f[swap]b[normal]b[relative]b[pto_resize]b[resample]i[quality]i[scale]f[scalex]f[scaley]f"
"[scale1]f[scalex1]f[scaley1]f[scale2]f[scalex2]f[scaley2]f [overlay]c[overlay_threshold]i[threads]i[map]i[interleaved]b[debug]i"
QUAD_BICUBIC ""
QUAD_BILINEAR ""
QUAD_DEBUG_GRID ""
QUAD_DEBUG_ISOMAP ""
QUAD_DEBUG_MARKERS ""
QUAD_DEBUG_OVERLAY ""
QUAD_DRAFT ""
rekey "ci"
series "cc+i*[resample]i[quality]i[transition]i[overlay]c[overlay_threshold]i[threads]i[debug]i"
SINE ""
warp "cf*[string]s[strength]f[swap]b[normal]b[relative]b[pto_resize]b[resample]i[quality]i[scale]f[scalex]f[scaley]f"
"[scale1]f[scalex1]f[scaley1]f[scale2]f[scalex2]f[scaley2]f[overlay]c[overlay_threshold]i[threads]i[map]i[interleaved]b[debug]i"
WARP_BICUBIC ""
WARP_BILINEAR ""
WARP_DEBUG_GRID ""
WARP_DEBUG_ISOMAP ""
WARP_DEBUG_MARKERS ""
WARP_DEBUG_OVERLAY ""
WARP_DRAFT ""
warp_grid "c[size]i[darken]f[colour]i"
WARP_LINEAR ""
warp_rekey "ci"
warp_series "cc+i*[resample]i[quality]i[transition]i[overlay]c[overlay_threshold]i[threads]i[debug]i"
WARP_SINE ""


EDIT: This could stand some attention too

warp(
(clip)
(clips)
(floats) (optional) # Is Int (providing floats would produce error)
(string) string (optional) # Dont exist
(int) resample = WARP_BICUBIC = 2
(int) quality = 6
(int) transition = WARP_LINEAR = 0
(clip) overlay = NULL
(int) overlay_threshold = 0
(int) threads = 0 (automatic)
(int) debug = 0
)


warp_series "cc+i*[resample]i[quality]i[transition]i[overlay]c[overlay_threshold]i[threads]i[debug]i"


and this

warp(
(clip),
(int)
)

warp_rekey "ci"


EDIT: No docs for Grid() and Warp_Grid(), Series(), or Rekey().
EDIT: Might want to note possible conflict with V.C Mohan Grid() plugin.

[wonkey_monkey]

In my defence... I hate writing documentation.

EDIT: No docs for Grid() and Warp_Grid(), Series(), or Rekey().

There is no Grid, Series, or Rekey, just warp_grid, warp_series, and warp_rekey.

[StainlessS]

In my defence... I hate writing documentation.



There is no Grid, Series, or Rekey, just warp_grid, warp_series, and warp_rekey.


I beg to differ, see the prototypes extracted from the dll. (2 posts previous)


Edit: doin' the docs is the best part of coding.

[wonkey_monkey]

Well that wasn't me. Does Avisynth automatically split names at underscores and create alternative filter names?

[lansing]

This is a very interesting method of color matching (more like chroma swapping) that relies solely on the quality of the warping. In perfect warped results, this method can achieved 100% accuracy of color mapping. However, the amount of work is way too much right now. Do you have ideas how to turn this thing into a gui?

[wonkey_monkey]

Ideas yes, plans no. Hugin gets you about 90% of the way there already, although it's rather awkward having to save and load pairs of frames. Copy/paste into Hugin would make it a lot easier, but I doubt it's something they'd implement. I was going to incorporate a control point GUI in something else I'm working on (which may or may not ever see the light of day), but I ended up steering from a two-pane GUI and it's a bit complicated to reimplement now.

StainlessS - I think it was him, he's usually the one with the crazy ideas - has previously posted scripts for using AutoHotKey to record clicks on a VirtualDub window, I believe. That could work with a stackhorizontal clip of the two videos, although I have no idea how you'd extract which frame you're on, nor would it give any visual feedback like Hugin does.

[StainlessS]

Does Avisynth automatically split names at underscores and create alternative filter names?

Arh, yes. Avs allows for using DllName_Filtername, something like DeBlock_DeBlock() to avoid using an alternative plugin (eg in DGDecode dll DeBlock),
It, (the prototype extractor) is probably getting a bit confused as dll is warp.dll and filters pre-pended with "warp_".
The Prototype extractor usually tries to avoid outputting the dll prepended name[it obviously is failing to do that here, and drops the 'warp' bit].

The crazy idea in question is SawBones, using AutoIt (AutoHotKey was once AutoIt, and broke away due to developer differences, and since has diverged quite a bit).
The AutoIt source is supplied in SawBones/FrameSurgeon zip.
[AutoIt, not that much different to Basic. Has a massive library of apps and source, initially developed to perform system admin tasks. Compiled to self contained native executable, no dll's required].

EDIT: Yep, below fails [there is no function named 'Grid'].

Colorbars
Grid()

[redfordxx]

Here (https://forum.doom9.org/showthread.php?t=177930) I almost finished my script for camera rotation. I have question about quad:
Normally, in real life, when I rotate camera 180 degrees, there should be nothing. Using quad, there is again the same picture. See...
ColorBars().Trim(0,1000).Killaudio()
AddBorders(400,400,400,400)

quad( 720,520,720,1478, 1040,520,1549,1478, 1040,400,1144,1066, 720,400,720,1066,
\ interleaved=true, relative=false, debug=4)

Is there any option to remove the picture of the other side?

[wonkey_monkey]

Quad is a 2D filter, and only knows where you've placed the corners in 2D space. You'll need to add an extra check to see if the corners are in the same order (clockwise or anti-clockwise) as before they were transformed - if not, move all the points to one side before calling Quad so you get a black video.

http://mathworld.wolfram.com/PolygonArea.html

[redfordxx]

Two more questions, thank you...
1. debug = 4 is changing colorspace ... is there something to do about it?

2. In the script above, I have added borders to not to crop the moved image. However, quad calculates the transformation of the black borders also! Which is useless and time consuming. Can I avoid it somehow?

[wonkey_monkey]

1. You can convert back again, that's all. It's only meant for debugging purposes so colourspace support wasn't a priority.

2. Quad has no way of knowing that you've added borders that don't need to be looked at. You could scale your input image up instead, to the desired size, and adjust your corner coordinates appropriately, but that might introduce a bit of generational loss as you'll be resampling twice.

[redfordxx]

Quad has no way of knowing that you've added borders that don't need to be looked at.
Might be nice feature, to add borders so that quad does not go out of the picture...

[redfordxx]

Another feature request:
Choose the color of the borders....

[introspected]

Thanks for the great plugin.
I try to use it for warp transformations during auto-align process in the AutoOverlay plugin.
It works fine but only first few thousand frames. After that the process freezes because Warp plugin creates too many handles.
It reproduces with avisynthplus 3.5.1/3.6.1 and any sample script like this: colorbars.quad(0.1,0.1,0.8,0.2,0.9,0.8,0.2,0.9, normal=true)

[wonkey_monkey]

Possibly something I fixed during innumerable rewrites since the last uploaded version. Please try:

http://horman.net/avisynth/download/warp0.1b.zip

[introspected]

Quad now works fine but Warp throws floating "Access Violation". Script: ColorBars.warp(0,0, 0,0, 640,0, 0,0, 640,480, 0,0, 0,480, 0,0, relative = true)

[wonkey_monkey]

Strange, it works fine for me. x86 or x64?

[introspected]

The error frequency depends on how many control points specified. It reproduces very rare for me with 3 points and very often with 5 both x86/x64 and AvsPmod/x264:
ColorBars.Warp(0,0, 0,0, 640,0, 0,0, 640,480, 0,0, 0,480, 0,0, 320,240, 0,0, relative = true)
It throws only on script loading.

Thanks for support.

[StainlessS]

It throws only on script loading.
Constructor or frame 0 GetFrame().
[probably stating the obvious]

[wonkey_monkey]

I think I found the problem. Please download again.

http://horman.net/avisynth/download/warp0.1b.zip

[introspected]

I think I found the problem. Please download again.
Now it works perfectly.

[StainlessS]

Wonkey,
Which is the most current version of Warp, the multiple 0.1b's are now missing, the 0.1 (as in first post) cant be right,
I have 2 different version 0.2 (including one from your home site)
as from your sig, version as in AvisynthPluginInit3

return "warp/quad v0.2";


Maybe you could like include a puzzle or something, for the user to solve to figure out which version is most recent.

Come on man, get it together and give some indication of version, at least on the zip.
[then everybody could re-version the dll in some way, maybe even re-compile with a proper version resource].

[Reel.Deel]

Wonkey,
Which is the most current version of Warp, the multiple 0.1b's are now missing, the 0.1 (as in first post) cant be right....


I believe v0.1b (2019/09/13) is the latest version: warp0.1b.zip (https://web.archive.org/web/20200913202653if_/http://horman.net/avisynth/download/warp0.1b.zip) - archive.org

... give some indication of version, at least on the zip.

I second that, archive.org sometimes refuses to archive files if it has the same name even if the current file is newer than the old one.

[StainlessS]

Thanks RD :)

My ISP dont let me access achive.org, dont know why [blocked for a couple of years maybe].

It dont really matter, I was just making a current archive together with recent-ish added plugs,
I dont dare look in my big in-box to try and order about 10 years worth of plugs, and stuff.

EDIT: Incidentally, here is the real Wonkey download directory [that the wonkey web page points at]:-
http://horman.net/avisynth/download/

EDIT: Strangely, I got a version of warp from 12 Jan 2019 [from my big in-box, dll date in zip]
that also professes in source to be version 0.2 [ie supposedly more recent than the latest posted one].

[SaurusX]

I finally made the jump and tried this technique matching a Blu-ray to DVD. After watching that YouTube video the whole process just seemed intimidating, but I guess dealing with python over the last two years made it less so. The goal is to create datasets for machine learning that require very close alignment of images for super resolution upscaling. After making a few tweaks to the process for this different use case, things are looking really good. I seem to be able to get away with doing one warp for an entire video and then being choosey as to which images to use. This opens up a lot of possibilities.

[rgr]

Thanks RD :)

My ISP dont let me access achive.org, dont know why [blocked for a couple of years maybe].

It dont really matter, I was just making a current archive together with recent-ish added plugs,
I dont dare look in my big in-box to try and order about 10 years worth of plugs, and stuff.

EDIT: Incidentally, here is the real Wonkey download directory [that the wonkey web page points at]:-
http://horman.net/avisynth/download/

EDIT: Strangely, I got a version of warp from 12 Jan 2019 [from my big in-box, dll date in zip]
that also professes in source to be version 0.2 [ie supposedly more recent than the latest posted one].

Do you still have the dll? I can't find anywhere to download it.

[wonkey_monkey]

I think the one on the download page is still the latest version. I'm just bad at keeping track of versions.

I have a new/replacement version in the works which performs warping on the GPU. It's not much quicker - the CPU version was heavily optimised in ways not really possible on a GPU - but it's better quality, has more interpolation options, and has a completely different and much more sensible animation system.

But I was going to release it along with a GUI I'm working on, because it has a helper mode for warping. And because I've decided to redo that with Vulkan, it might take a while. Let me think about an interim version, if I can cobble together some documentation.

[StainlessS]

I think the one on the download page is still the latest version.
If thats the one that rgr quoted me as providing link, then dat one is 403, Forbidden !

[rgr]

If thats the one that rgr quoted me as providing link, then dat one is 403, Forbidden !

Remove download from link.

https://horman.net/avisynth/