AutoChromaFix - fix detected chroma shift

[zee944]

I wrote a function which automatically fixes the chroma shift errors. It can happen after colorspace conversions/pixel cropping/adding was done on the video with not enough far-seeing. Or at least, that was my target.

The main idea is to select the edges both on the chroma and luma plane, compare them, and find out in which position they would match the best. If the necessary shift values has been figured out, the shift can be executed (or just you could just display the values). Since the method is quite slow and there's no averaging involved, it is more practical to use it only as guide, checking how much chroma shifting the video roughly need and decide on the exact values on your own.

The script needs GScript and mt_masktools. I've tried to use the For loop of Gavino's GScript, but it didn't seem to work within GScriptClip environment (I'm not sure if it should or shouldn't), so I made my own For loop.

It's still a beta, and not a finished function. It works, however I'm not sure it works perfectly right. I've tested it on bad material and on good material too which I ruined with this line on purpose:


MergeChroma(Lanczos4Resize(width, height, 1.5, -1.0)) # generates chroma shift error on the video


AutoChromaFix() does it back roughly, and by looking at it, the result is fine, but I'm not sure it's as accurate as it should be. I'm quite unsure about this part of the script:


tempU=MergeChroma(Lanczos4Resize(width, height, Xtotry, Ytotry)).UToY.PointResize(width, height).mt_edge("sobel", U=2, V=2)
tempV=MergeChroma(Lanczos4Resize(width, height, Xtotry, Ytotry)).VToY.PointResize(width, height).mt_edge("sobel", U=2, V=2)


The PointResize part, to be exact. Is this the right way to resize the U and V plane? Will the U and V values at their right place, where they should be to be compared with the Y plane? Don't they need a half-pixel or one pixel shift after resizing? Can someone tell me?


The script itself:


LoadPlugin("mt_masktools[2.0a30].dll")
LoadPlugin("GRunT[1.0.1].dll")

MPEG2Source("whatever.d2v")
AssumeFPS(23.976)

MergeChroma(Lanczos4Resize(width, height, 1.5, -1.0)) # generates chroma shift error on the video

#AutoChromaFix(referenceframe=15000)
#AutoChromaFix(accuracy=0.1)
AutoChromaFix()

function AutoChromaFix(clip clp, float "referenceframe", bool "nodisplay", bool "nofix", float "accuracy", int "maxshift")
{
# 1.0beta, by z. ndmn, 2013.I.1.
referenceframe = default(referenceframe, -1) # if no reference frame is given then the function will do different chroma correction on each frame
# if reference frame is given then it'll do chroma correction on the full video based on one (reference) frame
accuracy = default(accuracy, 0.25) # 0 < precistiy <= 0.5. '0.5' mean half-pixel accuracy. The lower the better but slower. Default is quarter-pixel accuracy.
maxshift = default(maxshift, 2) # maximum assumed chroma shift error to fix. A two pixel shift (default value) is quite a robust chroma error, but if your source
# possibly have bigger shift than that, raise it, but the script will be slower. Has to be even number. (Note: Actually the function
# goes a little beyond the given limit (between +0.25 and +0.5, depending on the accuracy). For instance if 2 is given, then it'll
# look for shifts up to somewhere between 2.25..2.5.)
nodisplay = default(nodisplay, False) # displays detected shift values by default, it can be turned off
nofix = default(nofix, False) # fixes chroma by default, it can be turned off (minimal speed gain only, the time consuming part is finding the right values)
clp

StartFrame = referenceframe<0 ? 0 : referenceframe
EndFrame = referenceframe<0 ? FrameCount : referenceframe

GScriptClip("""
global BestChromaMatchHorizontal = 0
global BestChromaMatchVertical = 0
ForLoop(last, -1*(maxshift), maxshift, "CheckChromaHorizontal",
\step=1.0 , INT1=StartFrame, INT2=EndFrame, FLO1=-1*(maxshift), FLO2=maxshift, FLO3=1.0)
ForLoop(last, BestChromaMatchHorizontal-0.5, BestChromaMatchHorizontal+0.5, "CheckChromaHorizontal",
\step=accuracy , INT1=StartFrame, INT2=EndFrame, FLO1=BestChromaMatchVertical-0.5, FLO2=BestChromaMatchVertical+0.5, FLO3=accuracy)
function CheckChromaHorizontal(clip clp, float n, float loopvariable, string "STR1", string "STR2", string "STR3", int "INT1", int "INT2", int "INT3",
\float "FLO1", float "FLO2", float "FLO3", bool "BOO1", bool "BOO2", bool "BOO3") { clp
ForLoop(last, FLO1, FLO2, "CheckChromaVertical", step=FLO3, INT1=INT1, INT2=INT2, FLO1=loopvariable)
function CheckChromaVertical(clip clp, float n, float loopvariable, string "STR1", string "STR2", string "STR3", int "INT1", int "INT2", int "INT3",
\float "FLO1", float "FLO2", float "FLO3", bool "BOO1", bool "BOO2", bool "BOO3") { clp
StartFrame=INT1
EndFrame=INT2
Xtotry=FLO1
Ytotry=loopvariable
Trim(StartFrame, EndFrame)
Tweak(sat=2.0)
LumaEdges=GreyScale().mt_edge("sobel", U=2, V=2)
tempU=MergeChroma(Lanczos4Resize(width, height, BestChromaMatchHorizontal, BestChromaMatchVertical)).
\UToY.PointResize(width, height).mt_edge("sobel", U=2, V=2)
tempV=MergeChroma(Lanczos4Resize(width, height, BestChromaMatchHorizontal, BestChromaMatchVertical)).
\VToY.PointResize(width, height).mt_edge("sobel", U=2, V=2)
ChromaEdgesBestSoFar=Overlay(tempU, tempV, mode="Add")
#ChromaEdgesBestSoFar=tempU

tempU=MergeChroma(Lanczos4Resize(width, height, Xtotry, Ytotry)).UToY.PointResize(width, height).mt_edge("sobel", U=2, V=2)
tempV=MergeChroma(Lanczos4Resize(width, height, Xtotry, Ytotry)).VToY.PointResize(width, height).mt_edge("sobel", U=2, V=2)
ChromaEdgesToTry=Overlay(tempU, tempV, mode="Add")
#ChromaEdgesToTry=tempU

IsItBetter = AverageLuma(Overlay(LumaEdges, ChromaEdgesToTry, mode="subtract")) <
\AverageLuma(Overlay(LumaEdges, ChromaEdgesBestSoFar, mode="subtract")) ? \
True : False
global BestChromaMatchHorizontal = (IsItBetter==True) ? Xtotry : BestChromaMatchHorizontal
global BestChromaMatchVertical = (IsItBetter==True) ? Ytotry : BestChromaMatchVertical
}
}

(nodisplay==True) ? last : subtitle("chroma correction h shift="+string(BestChromaMatchHorizontal)+" "+"v shift="+string(BestChromaMatchVertical))
(nofix==True) ? last : MergeChroma(Lanczos4Resize(width, height, BestChromaMatchHorizontal, BestChromaMatchVertical)) # we're fixing the chroma eventually
""", args="clp, StartFrame, EndFrame, accuracy, nodisplay, nofix, maxshift", local=true)

function ForLoop(clip clp, float from, float to, string commtoex, float "step", string "STR1", string "STR2", string "STR3", int "INT1", int "INT2", int "INT3",
\float "FLO1", float "FLO2", float "FLO3", bool "BOO1", bool "BOO2", bool "BOO3")
{
# 1.0, by z. ndmn, 2012.XII.23.
step = default(step, 1) # if no step is given, it'll be 1
NumberOfTotalRuns = floor (((to - from) / step) + 1) # number of times the loop body will run
ForLoopEngine(clp, NumberOfTotalRuns, NumberOfTotalRuns, from, step, commtoex, STR1=STR1, STR2=STR2, STR3=STR3, INT1=INT1, INT2=INT2, INT3=INT3,
\FLO1=FLO1, FLO2=FLO2, FLO3=FLO3, BOO1=BOO1, BOO2=BOO2, BOO3=BOO3)

function ForLoopEngine(clip clp, int NumberOfTotalRuns, float NumberOfRemainingRuns, float loopvariable, float step, string commtoex,
\string "STR1", string "STR2", string "STR3", int "INT1", int "INT2", int "INT3", float "FLO1", float "FLO2", float "FLO3", bool "BOO1", bool "BOO2", bool "BOO3")
{
clp
Eval(commtoex+"(last, (NumberOfTotalRuns - NumberOfRemainingRuns) + 1, loopvariable, STR1=STR1, STR2=STR2, STR3=STR3,
\INT1=INT1, INT2=INT2, INT3=INT3, FLO1=FLO1, FLO2=FLO2, FLO3=FLO3, BOO1=BOO1, BOO2=BOO2, BOO3=BOO3)")
clp=last
Assert(NumberOfRemainingRuns > 0)
return (NumberOfRemainingRuns == 1)
\ ? clp
\ : ForLoopEngine(clp, NumberOfTotalRuns, NumberOfRemainingRuns - 1, loopvariable + step, step, commtoex, STR1=STR1, STR2=STR2, STR3=STR3,
\INT1=INT1, INT2=INT2, INT3=INT3, FLO1=FLO1, FLO2=FLO2, FLO3=FLO3, BOO1=BOO1, BOO2=BOO2, BOO3=BOO3)
}
}

}

[Gavino]

I've tried to use the For loop of Gavino's GScript, but it didn't seem to work within GScriptClip environment
It should work, as long as you call GScript() itself inside the (G)ScriptClip() run-time script, ie something like:
GScriptClip("""
...
GScript(" # have to use single quotes here since already inside triple quotes
...
for (...) { ... }
...
"
...
""", ...)
However, you will run out of quoting options if you need to use a string literal inside the GScript part, as you will be already inside both the GScriptClip triple quotes and the GScript single quotes.
Better would be to put the GScript code into a separate function called from the run-time script.

More generally, you can speed up your code by taking the function definitions outside the run-time script, since currently the function code has to be parsed again for every frame.
Also, nested function definitions are not really useful in Avisynth, since all functions are global in scope.

I haven't had time to study how your function works, so can't answer your other questions at the moment.

[Mounir]

Interesting script and right on time because i'm working with a video currently which start with a -2-2 (X,Y) and toward to middle shift 3 -3 (X,Y) without apparent reason (no drop out).
I'll try your script and report back

[Mounir]

Well i have tested it, overall it's not bad but there are many misses. My source is interlaced i've used qtgmc before applying the autochromafix indeed (no ivtc).

The problem is whatever chroma shift fix you do should be done prior the deinterlacing stage imo because the deinterlacer blur (accuracy lost on chroma planes)

results:
autochromafix:
http://imageupload.org/en/file/245327/autochromafix.jpg.html

Me (manual):
http://imageupload.org/en/file/245328/me.jpg.html

Original:
http://imageupload.org/en/file/245329/original.jpg.html

[zee944]

However, you will run out of quoting options if you need to use a string literal inside the GScript part, as you will be already inside both the GScriptClip triple quotes and the GScript single quotes. Better would be to put the GScript code into a separate function called from the run-time script.
Yeah, I suspected that it was because of the quotes, but I wasn't sure. I'll separate the GScript code into a function when I rewrite the function, great idea, didn't think of it.

The nested functions was because I wanted the For loop to look like a conventional loop (in a programming language) as much as possible. It was just closer to my thinking.

I haven't had time to study how your function works, so can't answer your other questions at the moment.
When you have time to look into it please tell me what do you think, especially about the chroma plane resizing part. :)

Well i have tested it, overall it's not bad but there are many misses.

Thanks for the tryout. As I suggested in the first post it's more of an experimental script, first draft of an idea which needs further refining (and rewriting).