Linear interpolation of color correction with key frames

[Fizick]

I try create a new plugin (beta version) ColorKeyFrame - Linear interpolation of color correction with key frames.

(It is useful for clips with varying clopors, for example for DV camcorder clips in complex light condition)

The plugin is similar and based on ColorYUV AviSynth filter.
In Avisynth it is a hard to set different color correction for various part of clip, especially smoothly varying correction.
(the Animate function is not very convinient).
There are two functions in new plugin: ColorKeyFrame and ColorInterpolate.
Several ColorKeyFrame commands can be used to define color correction data for some given frames (key frames).
ColorInterpolate function make color correction by given values at key frames
and smooth transition of color correction between key frames by linear interpolation of the color correction data.
If you define exactly one key frame, the color correction by ColorInterpolation
will be uniform for whole clip, just as ColorYUV.

First version of the plugin uses some hack of Avisynth interface to transmit its setting
(the similar method is used by DGDecode (hints), IVTC, DePan, MVTools (motion vectors), etc).
Output of ColorKeyFrame function is special clip with color correction data
(for given key frame and previous key frames) in framebuffer.
This output can be used as a source for next call of ColoKeyFrame
for defining of correction of some other key frame, and so on.
Finally the output of last ColorKeyFrame is used as second input clip (keys)
for ColorInterpolation function to interpolate the color correction.

Now all frames store the same data (it is not per-frame property).
May be more apppropriate method is exist for storing and transmitting of per-clip data?
Firstly I supposed that user_data in plugininit2 function may be used.
Or may be strings?
(Text file may be used too)

[gzarkadas]

Just a thought (maybe wrong):

Since both filters are inside one dll, you can put the ColocKeyFrame data in an STL container (a map) and use child's address as the key. Because many ColorKeyFrame calls may be chained to a single clip, the value of the map should be a container also (a stack, a queue, a vector, a list,..., depending on how ColorInterpolation accesses the data).

ColocKeyFrame pseudocode:

your_type data = CalcData(child, keyframe);
size_t key = (size_t) child->p;
map<size_t, list<your_type>>::iterator it = global_map.find(key);
if (it == global_map.end()) {
// first time; init value container
global_map[key] = new list<your_type>(1, data);
}
else {
// not first time; add an element
it->second.push_back(data);
}


ColocInterpolation pseudocode:

size_t key = (size_t) second_child->p;
map<size_t, list<your_type>>::iterator it = global_map.find(key);
if (it == global_map.end()) {
// no values; handle error
...
}
else {
// it->second contains the list with data; do your stuff
...
}

[Fizick]

I do not familiar with C++ :)
what if we have several clips in one script to tune the colors?


Probably we must add some option to avisynth 2.6 (2.7) for per-clip metadata?

[gzarkadas]

Damn, this forces me to think more :). Can you perhaps supply the argument lists of ColorKeyFrame and ColorInterpolation? A bit more of information would certainly help. Up to this point, as I re-read more carefully your 1st post's description, I can only conclude (hopefully correct) that either:

you seek a linearly changing ColorYUV on any of its argument per frame, based on values on frames f0, f1, f2, ..., or
something different, which however needs only a single-frame clip to hold the info needed by ColorInterpolation.


In the first case you could (since STL is not a choice) setup

a structure to hold child->p, f0, f1, f2, ..., arg10, arg11, arg12, ..., ..., argn0, argn1, argn2, ..., (except for the first all others should be lists)
a linked list of those structs to identify the struct associated with its first argument (the source clip) with a linear (in size) search

Then you could (in each step a scan of the linked list for a match on child->p is implied)

In each ColorKeyFrame, insert a new "column" to the struct that corresponds to clip passed in (ie add an item in each list)
In ColorInterpolation::GetFrame construct for each k=1,...,n an f0, argk0, f1, argk1, ... array to pass to Spline (if up to ~25 keyframes are enough) or your linear interpolation function (if > ~25 keyframes are wanted), gather all calculated values and make the final env->Invoke to ColorYUV.


In the second case ColorKeyFrame could simply return this clip as its value and the user would need to supply it as an optional arg in each succesive ColorKeyFrame call. If the above is however considered complicated for the user (probably is) then the following recipy could be used (ColorKeyFrame returns the source clip):

In the constructor of ColorKeyFrame, env->GetVar for a global var that is based on child->p value as a hex string (plus a long prefix to reduce the posibility of conflicting with user script variables). Example: __color_key_frame_0x2E48AB0C
If var is found and is a clip, update it (we are inside a succesive call to ColorKeyFrame); if not create it (we are inside the 1st call for this IClip*)
In the constructor of ColorInterpolation, env->GetVar for the above global var; if found proceed else exit with error


Another approach that came to me while writing those lines - but which did *not* devoted much time to think of - would be to:

make a base filter class that holds a private frame buffer (initialised to child->GetFrame(0) ?) and returns it for every value of n in its GetFrame.
derive ColorKeyFrame from it to modify appropriately the frame buffer based on the other arguments

That way we modify a single frame buffer only at the constructor, irrespectively of number of frames.

I will stop here because my head started complaining for scratching it too hard ; hope something of the above is of use.

[Fizick]

gzarkadas,
I wonder what information do you ask?
You can read a source code! :)

Thabks for discussion.
Firsly I wanted to use NewVideoFrame and GetFrame in constructor.
I also try to use only frame 0. But it does not give any speed increasing.
Some more discuss later.

Anyway, I released version 0.2 (21.04.2007)
Fixed correction before first and after last key frames.

[gzarkadas]

I wonder what information do you ask?
You can read a source code! :)

Well, I missed that one; I thought it was still unpublished. Downloaded and will start reading soon :) .

[gzarkadas]

I have read it and from what I see it is a linear ColorYUV interpolator (case 1 from my 2nd post).

IMHO there are two options, to improve it:

Option 1

Make ColorKeyFrame a simple function that returns a formatted string by applying sprintf to its arguments. No need to pass a clip in, since it is only used as a data container. Say (AviSynth syntax):

ColorKeyFrame(int frame, float gain_y, float off_y, float gamma_y, \
float cont_y, float gain_u, float off_u, float gamma_u, float cont_u, \
float gain_v, float off_v, float gamma_v, float cont_v)

Modify ColorInterpolation to accept an arbitrary number of ColorKeyFrame-formatted strings as additional arguments ("cs+" as string to env->AddFunction). Unpack then the strings into the filter's constructor, construct an array of KeyData structs and store it as a class member.

That way in GetFrame you only need to make the interpolation from internal-to-the-filter data; everything else has been done inside the constructor.


Option 2

Make your own data management as I have described in my previous posts (with STL containers preferably, else you will have to low-level manipulate data structures yourself). It is more elaborate but it does not being limited to 59 keyframes as Option 1 above (due to 60 args limit of AviSynth).


IMO the <= 59 keyframes limitation of Option 1 is not that serious (one can always Trim the source clip and apply ColorInterpolation part by part), since most of the time the number of keyframes will be limited; I would prefer it since it is a lot simpler. It is easy also to call the filter from the script either by assigning to intermediate string vars or by using the following example construct:


ColorInterpolation(clp,
\ ColorKeyFrame(38, gain_y=295, gamma_v=1.1),
\ ColorKeyFrame(235, gamma_y=0.95, gamma_v=1.1),
...
\ ColorKeyFrame(12120, gain_y=427, gamma_u=0.89) )


If however you prefer the 2nd option, I could try (in about a month from now) build an STL container-based implementation of keydata.

[Fizick]

I understand your suggestion.
It may be done this way(s).
But I am not sure what is better (faster) and simpler - too many agruments.
More comments from other users (and developers) is needed.

Right now I encode my worst DV video I ever shoot,
in dark cafe and outdoor, with varying color balance, gain, and huge noise in dark.
I create this filter specifilally for this video :) and I am quite happy with current syntax.
Here is a script (encoding speed to MPEG2 DVD is 0.725 fps).
I use AvsP program for editing (copy-paste), frames number insert, and preview. We also can use sliders, but it is not very confinient (they all must be renamed after copying of line)

avisource("F:\capture\september\jubiley.06-09-03_19-59.00.avi") ++ \
avisource("F:\capture\september\jubiley.06-09-03_20-00.00.avi")
keys1=colorkeyframe(0,off_y=16,gain_y=-36,off_u=0,off_v=0)
keys1=colorkeyframe(keys1,749,off_y=16,gain_y=-36,off_u=0,off_v=0)
keys1=colorkeyframe(keys1,812,off_y=6,gain_y=0,off_u=0,off_v=0)
keys1=colorkeyframe(keys1,840,off_y=0,gain_y=150,off_u=8,off_v=-16)
keys1=colorkeyframe(keys1,946,off_y=0,gain_y=150,off_u=8,off_v=-16)
keys1=colorkeyframe(keys1,988,off_y=0,gain_y=150,off_u=16,off_v=-22)
keys1=colorkeyframe(keys1,1115,off_y=0,gain_y=150,off_u=16,off_v=-22)
keys1=colorkeyframe(keys1,1125,off_y=0,gain_y=170,off_u=16,off_v=-22)
keys1=colorkeyframe(keys1,1247,off_y=0,gain_y=170,off_u=16,off_v=-22)
keys1=colorkeyframe(keys1,1277,off_y=0,gain_y=170,off_u=16,off_v=-26)
street=ColorInterpolate(last,keys1)
#return street.coloryuv(analyze=true)
#cafe normal light
avisource("F:\capture\september\jubiley.06-09-03_20-02.00.avi") ++ \
avisource("F:\capture\september\jubiley.06-09-03_20-03.00.avi") ++ \
avisource("F:\capture\september\jubiley.06-09-03_20-03.01.avi") ++ \
avisource("F:\capture\september\jubiley.06-09-03_20-06.00.avi") ++ \
avisource("F:\capture\september\jubiley.06-09-03_20-07.00.avi")
keys2=colorkeyframe(0,off_y=0,gain_y=40,off_u=10,off_v=-15)
keys2=colorkeyframe(keys2,1802,off_y=10,gain_y=130,off_u=4,off_v=-8)
keys2=colorkeyframe(keys2,2059,off_y=0,gain_y=130,off_u=4,off_v=-8)
keys2=colorkeyframe(keys2,2579,off_y=0,gain_y=130,off_u=4,off_v=-8)
keys2=colorkeyframe(keys2,2683,off_y=16,gain_y=180,off_u=4,off_v=-8)
keys2=colorkeyframe(keys2,2736,off_y=10,gain_y=180,off_u=4,off_v=-8)
keys2=colorkeyframe(keys2,2736,off_y=16,gain_y=220,off_u=4,off_v=-8)
keys2=colorkeyframe(keys2,3737,off_y=16,gain_y=220,off_u=4,off_v=-8)
keys2=colorkeyframe(keys2,3775,off_y=0,gain_y=100,off_u=4,off_v=-8)
keys2=colorkeyframe(keys2,4044,off_y=0,gain_y=100,off_u=4,off_v=-8)
keys2=colorkeyframe(keys2,4053,off_y=16,gain_y=140,off_u=4,off_v=-8)
keys2=colorkeyframe(keys2,4071,off_y=16,gain_y=140,off_u=4,off_v=-8)
keys2=colorkeyframe(keys2,4097,off_y=0,gain_y=120,off_u=6,off_v=-12)
keys2=colorkeyframe(keys2,4627,off_y=0,gain_y=120,off_u=6,off_v=-12)
keys2=colorkeyframe(keys2,4645,off_y=10,gain_y=150,off_u=6,off_v=-12)
keys2=colorkeyframe(keys2,4965,off_y=10,gain_y=150,off_u=6,off_v=-12)
keys2=colorkeyframe(keys2,4980,off_y=0,gain_y=130,off_u=6,off_v=-13)
keys2=colorkeyframe(keys2,5489,off_y=0,gain_y=130,off_u=6,off_v=-13)
keys2=colorkeyframe(keys2,5513,off_y=16,gain_y=190,off_u=6,off_v=-12)
keys2=colorkeyframe(keys2,5650,off_y=16,gain_y=190,off_u=6,off_v=-12)
keys2=colorkeyframe(keys2,5664,off_y=16,gain_y=270,off_u=6,off_v=-12)
keys2=colorkeyframe(keys2,5738,off_y=16,gain_y=270,off_u=6,off_v=-12)
keys2=colorkeyframe(keys2,5754,off_y=0,gain_y=200,off_u=6,off_v=-12)
keys2=colorkeyframe(keys2,6070,off_y=0,gain_y=250,off_u=6,off_v=-12)
keys2=colorkeyframe(keys2,6529,off_y=0,gain_y=250,off_u=6,off_v=-12)
cafenormal=ColorInterpolate(last,keys2)
#return last.coloryuv(analyze=true)
#cafe dark
avisource("F:\capture\september\jubiley.06-09-03_20-27.00.avi") ++ \
avisource("F:\capture\september\jubiley.06-09-03_20-30.00.avi").trim(78,0) ++ \
avisource("F:\capture\september\jubiley.06-09-03_20-43.00.avi") ++ \
avisource("F:\capture\september\jubiley.06-09-03_20-44.00.avi") ++ \
avisource("F:\capture\september\jubiley.06-09-03_20-55.00.avi") ++ \
avisource("F:\capture\september\jubiley.06-09-03_21-08.00.avi") ++ \
avisource("F:\capture\september\jubiley.06-09-03_21-09.00.avi").trim(690,0) ++ \
avisource("F:\capture\september\jubiley.06-09-03_21-10.00.avi")
keys3=colorkeyframe(0,off_y=0,gain_y=180,off_u=5,off_v=-6)
cafedark=ColorInterpolate(last,keys3)
#return cafedark.coloryuv(analyze=true)

street ++ cafenormal ++ cafedark
#return last
converttoyv12(true)

separatefields()
i=last
d=degrainmedian(i,interlaced=true,mode=0)
vf1=mvanalyse(d,pel=2,blksize=8,isb=false,idx=1,overlap=4,sharp=2,delta=1)
vf=mvanalyse(d,pel=2,blksize=8,isb=false,idx=1,overlap=4,sharp=2,delta=2)
vb=mvanalyse(d,pel=2,blksize=8,isb=true,idx=1,overlap=4,sharp=2,delta=2)
vf2=mvanalyse(d,pel=2,blksize=8,isb=false,idx=1,overlap=4,sharp=2,delta=4)
vb2=mvanalyse(d,pel=2,blksize=8,isb=true,idx=1,overlap=4,sharp=2,delta=4)
mvdegrain2(i,vb2,vb,vf,vf2,thsad=600,idx=2,thSCD1=600)
#return last.weave
g=mvdepan(last,vf1)
depanstabilize(data=g,cutoff=3,mirror=15,blur=30)
#return last.weave
weave
fft3dgpu(sigma=1.5,interlaced=true,plane=4,bt=3,ow=16,oh=16)
#return last
#stackvertical(i,last)

limiter()
converttoyuy2(true)


I also believe, that key frames approach may be useful for denoiser too. I can add "sigma" parameter to ColorKeyFrames and use it in new (modified) version of fft3dfilter (or other denoiser).

[cobo]

ColorKeyFrame only works if I use whole numbers in the variables, not with decimal numbers.

This gives the expected result -> ColorKeyFrame(217,off_u=-0, off_v=8)
This doesn't change the clip at all -> ColorKeyFrame(217,off_u=-0.25, off_v=8.25)

Is it possible to make it work with decimals?

[Fizick]

Probably yes for input. But correction will be integer anyway.
I wonder, that clip is not changed. Will look to code.

EDIT: the bug is found. Will release fixed version today.

[cobo]

Thanks. I could have sworn I was seeing a difference in the output of ColorYUV between using the values 0.5 and 1, but after checking carefully I see that it's correction is integer too.