Color correction using reference clip

[thetoof]

Hi all,

here's the thing: I have four episodes I'd like to encode. Only one of them looks fine and the others are either greenish, bluish or reddish.
Only the first few frames of the opening are exactly the same for all four, so how can I use avisynth to correct the whole clips with the values calculated from comparing at most the first 370 frames?

I read about colourlike and masktools, but couldn't figure out a way to use only a section to apply the result on the whole clip.

Please, no suggestions regarding manual color correction.

[canuckerfan]

you could use something like conditionalreader to apply your colour correction to all the frames except for the opening scene.

[thetoof]

Nope, misunderstanding.

I want to correct ALL the clip using only the opening scene (since it's the only thing similar) as a reference.

[canuckerfan]

oh i see what you mean. yea, that's way beyond me :o

[thewebchat]

How are the colors related (linear, exponential, rgb space, yuv space, etc) in the clips? A sample is needed.

[thetoof]

http://www.mediafire.com/download.php?mnnmzxlonyg

For rule 6 checkers, this is an intermediate Lagarith rendering pass after removal of field blending.

[thewebchat]

"Could not locate decompressor for FOURCC 'lags'"

How about you use H.264 so we can read your files. Also, your LZMA actually increased the size of the archive by 1MB.

[thetoof]

Don't you know Lagarith as in http://lags.leetcode.net/codec.html

and that the en/decoding effort is close to none compared to H.264?

Yes, that's not important for this case, but remember that it is an intermediate file between filtering steps.

And yes, 7z's not good for compressing compressed stuff, but at least it's only one file to upload/link to. I'm that lazy. In our day and age... I don't think 1mb is going to change much.

If you still can't read the files after installing the codec, I'll use H.264. Never had a problem before with Lagarith on this forum though.

Thanks for the time your taking to do this.

[thewebchat]

Just by using a naive comparison with ColorYUV(analyze=true), I can see a trend, but it's hard to confirm it with only one scene since the levels are the same throughout it. Are there any other scenes in the opening sequence?

Edit: For example, by looking at the ColorYUV stats, I saw that the levels between a and c were shifted by -5.5 in Y and +4.4 in V. After using a mt_lut to shift the levels back, the clips are of roughly the same colors. Also, the histogram shows a lot of information in Y below the 16 mark and above the 235 mark. However, it looks washed out when converted to RGB with PC range. What's going on with that?

Edit2: Also, sorry, this is a little outside my range of experience.

[thetoof]

There's one. Samples coming.

[Sagekilla]

If you want, you could do something along the lines of:


RefClip = Source("...").Trim(0,X) # Trim up to the last good frame
BadClip = Source("...").Trim(X,-1)

count = BadClip.Framecount()
repeats = int(count / X) + 1
excess = (X * repeats) - count

RefClip.Loop(repeats).Trim(excess,-1) # doesn't really matter where you trim it

WriteHistogram(RefClip, ...)
WriteHistogram(Badclip, ...)


Or if you want to base it off one frame:

RefClip = Source("...").Trim(0,1)
BadClip = Source("...")

count = BadClip.Framecount()

RefClip.Loop(count)

WriteHistogram(RefClip, ...)
WriteHistogram(Badclip, ...)


Of course, there'd be issues with using one (or even a handful of frames) to color correct, but if it works, it works.

[Gavino]

Quite a few errors in there.
First one should be:
RefClip = Source("...").Trim(0,X) # Trim up to the last good frame
BadClip = Source("...").Trim(X+1, 0)

count = BadClip.Framecount()
repeats = count / (X + 1)
remainder = count % (X+1)

remainder == 0 ? RefClip.Loop(repeats) : RefClip.Loop(repeats) + RefClip.Trim(0, remainder)
...


and the second one:
RefClip = Source("...").Trim(0,-1)
...

[poisondeathray]

You might have problems color matching/correcting that unless you mask out the logo.

It appears as 255,255,255 (pure white) in all your color cast samples

[Sagekilla]

Quite a few errors in there.


Yeah, I'm a bit rusty with Avisynth at the moment ;) Been away for it for nearly the whole summer.

Thanks for the corrections Gavino.

[thetoof]

Delay = power outage

have fun with those everybody http://www.mediafire.com/download.php?tw1gnmhzhiw

[lansing]

toof, your sample video are all broken when extracting

[thetoof]

Well that's weird... they open fine with vdub on my computer. Can anyone else confirm?? (extracted from the archive in a separate folder and then opened) - never mind, redling the archive shows the issue. Happened during the file transfer, which I'm redoing now

-edit
On a side note, using only a portion of a clip and applying the resulting histogram on the whole clip fails for me. Sagekilla, can you tell me if you can do so on another movie or something (use only 100 consecutive frames for the ref clip and apply it on another clip)?. Gives me a b0rked output.

poisondeathray, you have point. After masking out the pure white, could one use mt_makediff to find a fixed value to correct the colors of the rest of the clip? Or am I totally mistaken on its usage?

[thetoof]

Sorry for double post, but heh...

New link: http://www.mediafire.com/download.php?gjmu5zntdgj

thewebchat, I used smoothlevels(preset="tv2pc",protect=16) for levels conversion... maybe that has something to do with the YV12-RGB conversion problem. Instead of mt_lut, we could try coloryuv(off_v=x,off_y=x) to fix this... will try later.

[Sagekilla]

On a side note, using only a portion of a clip and applying the resulting histogram on the whole clip fails for me. Sagekilla, can you tell me if you can do so on another movie or something (use only 100 consecutive frames for the ref clip and apply it on another clip)?. Gives me a b0rked output.


It was merely a thought ;) I wasn't 100% sure if this would work or not.

Was the issue with colorization, as in some frames came out with some absurd colors? Like a frame turning purple or something strange like that. Or was it more along the lines of severe distortion where the content of the frame is no longer recognizable?

If the latter (or either really) I'd say I'm not completely surprised. While reading the documentation, it seems like it relies on having a 1:1 relationship between frames to properly work. I'd imagine that analyzing a black frame and applying some color transform based on that to another frame would have odd results.

If I could, I would do some more testing and try to help you find a solution. Unfortunately, due to my luck (or lack thereof) my desktop -and- laptop are both broken at the moment. (laptop fan broke -- it overheats after 10 minutes)

With that said, I don't know if you'll be able to properly color correct your video with only a partial reference clip. Unless you can get a better video (In which case, I'd imagine you'd just use that one in the first place since) you may not be able to restore the video.


Side note: It would be interesting if a new plugin could be created that calculated a bunch of color weights based on a reference clip with correct colors, and apply it to a longer, and damaged, clip without requiring 1:1 processing.

[thetoof]

Was the issue with colorization, as in some frames came out with some absurd colors? Like a frame turning purple or something strange like that.
This one
Side note: It would be interesting if a new plugin could be created that calculated a bunch of color weights based on a reference clip with correct colors, and apply it to a longer, and damaged, clip without requiring 1:1 processing.
Wondering about the math here.
Would it be taking the average difference between two ref frames/sections to apply it on the whole clip or some fancy way with a curve and all that? (Hmmm, curves, luts.... LaTo, Didée, others I'm forgetting, you around?)
To all, please advise... or even post a script and make people happy :p

edit: About the pure white logo poisondeathray mentionned... since it seems like a constant in all videos... I guess it could be simply ignored

[poisondeathray]

edit: About the pure white logo poisondeathray mentionned... since it seems like a constant in all videos... I guess it could be simply ignored

Doing this in avisynth is way out of my league (I would usually do this sort of thing in after effects) , but that's kind of my point:

The logo doesn't seem affected by the color cast; ie. it seems it was overlayed after the color screwups

So if you use section that has the logo as a reference it will skew the results when analyzing; and if you apply corrective values to any section that has a logo on a frame basis, then you will colorize the logo and each video will look different...

EDIT: ok I just had a look at your clips5-8, there is no logo there, and the "white" letters do have the color cast on affected clips. So correction could theoretically work there. I guess the question is how prevalent is the pure white logo? (what sections does it occur & how frequent)

[thetoof]

You're right... never mind what I said.
Wondering: should the required processing ignore all constants for the calculations and then reapply the original pixels of those portions after correction? (or correct everything except them... you get the point)

[poisondeathray]

Wondering: should the required processing ignore all constants for the calculations and then reapply the original pixels of those portions after correction? (or correct everything except them... you get the point)

Sorry toof, I'm not sure I understand or that I can help very much (this stuff is way over my head...) :confused:

Couldn't you do some sort of difference calculation for RGB values (good clip - bad clip), then use those values to apply (subtract from) to the bad clip? This is basically how it's done in other software to selectively remove hue or color casting. This is assuming you mask out the logo or use sections where there is no logo...

[thetoof]

Yes, hence my question about averaging the calculations (goodref-badref) or using some sort of weights or curves to do it.

[poisondeathray]

If the discoloration is constant (which it appears to be in the samples) , wouldn't a fixed corrective number be more appropriate?

[thetoof]

Well, it seems to, though that's hard to judge based on a few hundred frames only. Sure a fixed value would do the job (especially if an adaptive method is hard to come by), but heh, maybe it'd be worth a shot to go beyond... if not for this case, maybe others!

[thetoof]

For the record, I ended up using thewebchat's suggestion: comparing values obtained with coloryuv(analyze=true) in different scenes.

Something nice would've been a tool to ajust even saturation, contrast and whatnot based on the ref clip. Anyone up for the challenge?

[EuropeanMan]

You might have problems color matching/correcting that unless you mask out the logo.

It appears as 255,255,255 (pure white) in all your color cast samples

When you say mask out the logo...what EXACTLY does that mean? From a n00b standpoint, I "read" - logo can be removed from the frames, leaving you with the frames... BUT if the logo is removed, does the UNDERLYING material have the rest of the frame, or JUST the SHADOW of the logo?

[thewebchat]

Neither. In this case, he only means that we shouldn't consider the pixels in the logo when calculating the average hue/saturation/brightness/whatever of the frame.

For example, consider this sample: 5, 6, 4, 9001
What he means is akin to pretending that the "9001" doesn't exist when calculating the average of that sample. E.g. (5+6+4)/3 = 5, not (5+6+4+9001)/4 = 2253.

[EuropeanMan]

^ Sweet, thanks :)

[zee944]

Oh, my favourite topic. Making a common part of two video sources look the same, then applying the difference to one of the sources. Unfortunately, no plugin made for specifically that. It's way too complicated.

But in this case, the workaround is quite simple. Forget contrast and gamma. Draw a specific luma curve in VirtualDub (with Gradation Curves filter by BugsBunny), load it to your clip with GiCoCu(), then use a simple manual color correction to make the colours look nearly the same. Only one manual correction. That's it.

It's useless to achieve a (near) perfect match between the shared parts of the two video, because the more complicated the tweaking is, the rest of the scenes will look unnatural and off. Just use an as simple color correction as ColorYUV or RGBAdjust, and the result (the rest of the scenes) will probably be reasonably good.

[Revgen]

I read about colourlike and masktools, but couldn't figure out a way to use only a section to apply the result on the whole clip.

That's why when I really want to colourlike a whole clip I write histograms for each scene, colourlike each scene independently and join the scenes together afterward.

[g-force]

thetoof,

here's a little something I've been working on that maybe you can tweak for your purposes:

global ideal = goodclip.FreezeFrame(0,1000000000,100)
global input = badclip .FreezeFrame(0,1000000000,100)
global filtd = badclip
global t = .4

scriptclip(filtd,"""

y1l = Yplanemin(ideal,t)
y1u = Uplanemin(ideal,t)
y1v = Vplanemin(ideal,t)
y2l = AverageLuma(ideal)
y2u = AverageChromaU(ideal)
y2v = AverageChromaV(ideal)
y3l = Yplanemax(ideal,t)
y3u = Uplanemax(ideal,t)
y3v = Vplanemax(ideal,t)

x1l = Yplanemin(input,t)
x1u = Uplanemin(input,t)
x1v = Vplanemin(input,t)
x2l = AverageLuma(input)
x2u = AverageChromaU(input)
x2v = AverageChromaV(input)
x3l = Yplanemax(input,t)
x3u = Uplanemax(input,t)
x3v = Vplanemax(input,t)

al = x3l>x1l?(y3l-y1l+(x1l-x3l)*(y1l-y2l)/(x1l-x2l))/((x1l-x3l)*(x2l-x3l)):0
bl = x2l>x1l?((y2l-y1l)/(x2l-x1l))-al*(x1l+x2l):0
cl = y1l-al*x1l*x1l-bl*x1l

au = x3u>x1u?(y3u-y1u+(x1u-x3u)*(y1u-y2u)/(x1u-x2u))/((x1u-x3u)*(x2u-x3u)):0
bu = x2u>x1u?((y2u-y1u)/(x2u-x1u))-au*(x1u+x2u):0
cu = y1u-au*x1u*x1u-bu*x1u

av = x3v>x1v?(y3v-y1v+(x1v-x3v)*(y1v-y2v)/(x1v-x2v))/((x1v-x3v)*(x2v-x3v)):0
bv = x2v>x1v?((y2v-y1v)/(x2v-x1v))-av*(x1v+x2v):0
cv = y1v-av*x1v*x1v-bv*x1v

filtd.MT_lut("x x * "+string(al)+" * x "+string(bl)+" * + "+string(cl)+" +",y=3,u=2,v=2)
last. MT_lut("x x * "+string(au)+" * x "+string(bu)+" * + "+string(cu)+" +",y=2,u=3,v=2)
last. MT_lut("x x * "+string(av)+" * x "+string(bv)+" * + "+string(cv)+" +",y=2,u=2,v=3)

""")

[g-force]

Now productized, try:

YcUrVeMatch(goodclip.FreezeFrame(0,1000000,100),badclip.FreezeFrame(0,1000000,100),badclip)

Function YcUrVeMatch(clip gref, clip bref, clip "input", float "thresh", string "planes")

{# Make YUV min, max and avg of second clip match first clip - v1.1 - by G-force
# or, make third clip match difference between first two

input = Default(input,bref) #default clip to process is second clip, or specify input
thresh = Default(thresh,.4) #how much (in %) to throw away in min/max calc
planes = Default(planes,"all") #default is "all". Alternates: "luma only" or "chroma only"

global gref_g = gref
global bref_g = bref
global input_g = input
global t_g = thresh
global planes_g = planes

scriptclip(input_g,"""

y1l = planes_g=="chroma only"? 0 : AverageLuma(gref_g)
y1u = planes_g=="luma only"? 0 : AverageChromaU(gref_g)
y1v = planes_g=="luma only"? 0 : AverageChromaV(gref_g)

y2l = planes_g=="chroma only"? 0 : Yplanemin(gref_g,t_g)
y2u = planes_g=="luma only"? 0 : Uplanemin(gref_g,t_g)
y2v = planes_g=="luma only"? 0 : Vplanemin(gref_g,t_g)

y3l = planes_g=="chroma only"? 0 : Yplanemax(gref_g,t_g)
y3u = planes_g=="luma only"? 0 : Uplanemax(gref_g,t_g)
y3v = planes_g=="luma only"? 0 : Vplanemax(gref_g,t_g)

x1l = planes_g=="chroma only"? 0 : AverageLuma(bref_g)
x1u = planes_g=="luma only"? 0 : AverageChromaU(bref_g)
x1v = planes_g=="luma only"? 0 : AverageChromaV(bref_g)

x2l = planes_g=="chroma only"? 0 : Yplanemin(bref_g,t_g)
x2u = planes_g=="luma only"? 0 : Uplanemin(bref_g,t_g)
x2v = planes_g=="luma only"? 0 : Vplanemin(bref_g,t_g)

x3l = planes_g=="chroma only"? 0 : Yplanemax(bref_g,t_g)
x3u = planes_g=="luma only"? 0 : Uplanemax(bref_g,t_g)
x3v = planes_g=="luma only"? 0 : Vplanemax(bref_g,t_g)

#compute coefficients for y=ax^2+bx+c using min, avg, max for luma, U, V

al = x1l==x2l? 0: x1l==x3l? 0: x2l==x3l? 0:
\ (y3l-y1l+(x1l-x3l)*(y1l-y2l)/(x1l-x2l))/((x1l-x3l)*(x2l-x3l))
bl = x1l==x2l? 0:
\ ((y2l-y1l)/(x2l-x1l))-al*(x1l+x2l)
cl = y1l-al*x1l*x1l-bl*x1l

au = x1u==x2u? 0: x1u==x3u? 0: x2u==x3u? 0:
\ (y3u-y1u+(x1u-x3u)*(y1u-y2u)/(x1u-x2u))/((x1u-x3u)*(x2u-x3u))
bu = x1u==x2u? 0:
\ ((y2u-y1u)/(x2u-x1u))-au*(x1u+x2u)
cu = y1u-au*x1u*x1u-bu*x1u

av = x1v==x2v? 0: x1v==x3v? 0: x2v==x3v? 0:
\ (y3v-y1v+(x1v-x3v)*(y1v-y2v)/(x1v-x2v))/((x1v-x3v)*(x2v-x3v))
bv = x1v==x2v? 0:
\ ((y2v-y1v)/(x2v-x1v))-av*(x1v+x2v)
cv = y1v-av*x1v*x1v-bv*x1v

planes_g=="chroma only"? input_g :
\ input_g.MT_lut("x x * "+string(al)+" * x "+string(bl)+" * + "+string(cl)+" +",y=3,u=2,v=2)
planes_g=="luma only"? last :
\ last. MT_lut("x x * "+string(au)+" * x "+string(bu)+" * + "+string(cu)+" +",y=2,u=3,v=2)
planes_g=="luma only"? last :
\ last. MT_lut("x x * "+string(av)+" * x "+string(bv)+" * + "+string(cv)+" +",y=2,u=2,v=3)

""")

Return(last)}

[Gavino]

Good stuff, g-force.
However, your function illustrates one of the classic problems with ScriptClip.
global gref = gref
global bref = bref
global input = input
global t = thresh
global planes = planes

scriptclip(input,"""
...
Using global variables to pass values from a function into a run-time script (eg ScriptClip) is not a good idea, since
1) The globals can be hidden by variables of the same name in the main script. Here for example your function doesn't work in a script that happens to use a variable called gref, bref, input, t, or planes.
2) The globals are shared by all invocations of the function, so the function can only be called once in a script.

Here the global 'input' is unnecessary since, as the clip passed to ScriptClip, it is available as 'last' in the inner script.
The other globals can be eliminated by using GRunT, which provides a way of passing values directly into a run-time script. Thus, you could write
GScriptClip(input,"""
input = last # or just use last instead of input later
...
""", args="gref, bref, t=thresh, planes")

[g-force]

[QUOTE=Gavino;1332571]
1) The globals can be hidden by variables of the same name in the main script. Here for example your function doesn't work in a script that happens to use a variable called gref, bref, input, t, or planes.
2) The globals are shared by all invocations of the function, so the function can only be called once in a script.
QUOTE]

Thanks Gavino,

Your insight is very much appreciated. So what if I just made the globals really weird names that are unlikely to be used in any other part of the script? Still doesn't address being able to call the function again, so I'll consider doing a version with GrunT.

[g-force]

Updated script based on (some of) Gavino's comments. ;)

-G

[Gavino]

As you say, this is not really solving the problems, just putting a stronger bandage on the wound.
OK, for a highly specialised function like yours, it's unlikely anyone would want to call it twice in a script.
But really, using globals with ScriptClip like this is a hostage to fortune and should be consigned to the history books (IMHO, of course :)).

"I really do not like this global variable business with ScriptClip ..." - stickboy

[g-force]

As you say, this is not really solving the problems, just putting a stronger bandage on the wound.
OK, for a highly specialised function like yours, it's unlikely anyone would want to call it twice in a script.
But really, using globals with ScriptClip like this is a hostage to fortune and should be consigned to the history books (IMHO, of course :)).

"I really do not like this global variable business with ScriptClip ..." - stickboy

Gavino,

I couldn't agree more. I actually have a plan to use this someday with multiple calls, so I will eventually have to have a GrunT'd version. Just haven't had a chance to download and try it out yet.

-G

[thetoof]

Oh, sweet! I hadn't looked at this thread after september. I'm just about to put this in practical use again. Thanks.

[thetoof]

Doesn't work well for me. Here's my line:
YcUrVeMatch(o.trim(0,142)+o.trim(359,544)+o.trim(2076,2278),i.trim(11,153)+i.trim(359,545)+i.trim(2077,2278),i)
What I'm trying to do:
- 2 episodes, 1 reference and 1 to correct
- A few similar frames in the opening (thus the multiple trim+splice)
- luma and chroma fluctuations throughout the clip when applying on all of it, even on the g/bref sections

Do you need samples or you can duplicate the issue on your computer with some of your files?

[thetoof]

Triple post... anyways.

In the end, the problem was that I used multiple sections of the clip to gather as much data as I could. However, the output looks the same (and only has decimal differences when checking with coloryuv(analyze=true)) when I only use the first section.

The function seems to do this (not very clear, but I think you'll see what I mean):
Frame 200 gref = frame 200bref, but = frame1052 input, but the correction applied on frame 1052 is the same as it would've been on 200, hence the b0rked output.

[g-force]

thetoof,

sorry I've been a away so long. Anyway, the way I wrote this requires that if you're using the alternate input to apply the correction to, the good reference (gref) and the bad reference (bref) have to be EXACTLY the same frame, except that one has desired color, the other has bad color. Then the differences between those 2 frames are applied to the clip "input".

so, do this:

YcUrVeMatch(o.trim(2278,2278),i.trim(2278,2278),i)

and make sure that frame 2278 of both clips are time aligned. Hope that helps.

-G