mt_lutxy() on chroma and luma

[mathmax]

Hello

I would like to remove some bad colours in my video.. and, as a first step, to locate them with a mask. First, I make the difference of each frame with the preceding one and with the following one.

prev = clp.SelectEvery(1,-1)
clip1 = mt_makediff(clp, prev, chroma="process")

next = clp.SelectEvery(1, 1)
clip2 = mt_makediff(clp, next, chroma="process")

This gives me two clips, basically grey 128, but with some spots... the challenge is now to build a new clip that take the pixel, from either clip1 or clip2, that is the closest to grey 128.

I thought about using mt_lutxy(), somehow this way:

mt_lutxy(clip1, clip2, x 128 - abs y 128 - abs < x y ?)
but it seems that it only process on the luma... whereas I would like to take both luma and chroma into account. How should I do?

Thank you in advance for any advice.

[cretindesalpes]

mt_lutxy(clip1, clip2, "x 128 - abs y 128 - abs < x y ?", chroma="process")
processes both luma and chroma.

[mathmax]

That seems perfect, thank you :)

I wonder what is exactly inside the variable x. That is strange to compare x to a unique value 128, whereas the "distance" to the grey is in fact determined with 3 values : U, Y and V.
Does chroma="process" translate abs(x-128) to ((Xu-128)²+(Xy-128)²+(Xv-128)²)¹/²?

The next step is to convert the resulting clip into a hard mask.. In the case a pixel is close enough to grey 128, it must be converted to black.. else it must be converted to white. How can I do this ?
And btw, it would be even better to process on chroma only.. is it possible?

[Didée]

That's a misunderstanding. With lutxy, there is no connection at all "between" the different channels. Of the spatially-same pixel in clip1 and clip2, lutxy computes Y with Y, and U with U, and V with V components. That's all.
If you need to make relations between the different channels, you have to transfer those channels into the same channels of different clips, then process.

Example: to build a relation between U and V values (of one single clip), you would go

Lutxy(clip,clip.swapuv(),"the formula",U=3,V=3)


BTW ... every had a look into MaskTools' documentation? There are good explanations in there. ;)

[mathmax]

Ah ok.. but how can I do in the case I want to compare the colours of two different clips? I would need a function with 6 parameters (Xy, Xu, Xv, Yy, Yu, Yv).. I can't find such function in MaskTools' documentation.. does it even exist?

What does "U=3,V=3" stand for?

[cretindesalpes]

"U=3" means: process the U plane. Default is 1 for U and V planes, meaning they are ignored and contain garbage. But go and read the masktools2 documentation!

There is no masktools function taking 6 arguments but you can do it in multiple steps:

# Compute the square distances between clp and prev for each plane. Ignores Y
# +64 because we use adddiff after
mt_lutxy (clp, prev, "x y - 2 ^ 128 / 64 +", y=1, u=3, v=3)

# Add the U and V planes, result in U plane.
# (dU² + 64) + (dV² + 64) - 128 = dU² + dV²
mt_adddiff (SwapUV (), y=1, u=3)

# Square root, result on the U plane (here starting at 0, not 128).
clip1 = mt_lut ("x 128 * 0.5 ^", y=1, u=3)


# Repeat the same thing with clp and next
…


# Smallest difference
smallest = mt_logic (clip1, clip2, "min", y=1, u=3)

# Mask with hard threshold set to 5
mask = smallest.mt_binarize (5, y=1, u=3)

# Transfer the U plane to the Y plane (if you need to)
# If chroma is subsampled (YV12), resize it to match the luma size. We have now dU² + dV² in Y
mask = mask.UToY ().PointResize (mask.Width (), mask.Height ())

# Do something with the mask
mt_merge (c1, c2, mask, luma=true)

Code not tested, it must contain some errors but you’ll get the idea.

[mathmax]

Thank you very much.

Probably I don't have the last version of the documentation because the Y,U,V parameters were not mentioned:
http://manao4.free.fr/mt_masktools.html

Some questions:


Why do you divide by 128 and then multiply by 128? Why not :


mt_lutxy (clp, prev, "x y - 2 ^ 64 +", y=1, u=3, v=3)
mt_adddiff (SwapUV (), y=1, u=3)
clip1 = mt_lut ("x 0.5 ^", y=1, u=3)

[cretindesalpes]

They are documented, in the "II) Common parameters" section. BTW I'm referring to masktools 2 (functions prefixed with mt_).

I divide and multiply by 128 to preserve the range as possible but it was more or less a reflex. It may not be a good idea if you are mostly interested in low values… You could choose a different scale, or just use the L1 norm (sum of absolute values)

[jmac698]

Does http://avisynth.org/mediawiki/MaskHS help?
I've also made a script to convert from yuv to hsv.

The documentation on the website of masktools is out of date; you have to use the file in the download. This bites people many, many times.

[mathmax]

Does http://avisynth.org/mediawiki/MaskHS help?
ah yes.. that could also do the trick :)

They are documented, in the "II) Common parameters" section. BTW I'm referring to masktools 2 (functions prefixed with mt_).
Indeed.. I just didn't see them in the function's signature.


I divide and multiply by 128 to preserve the range as possible but it was more or less a reflex.
what do you mean by preserve the range? Since you divide and then multiply by 128.. isn't it like you have done nothing finally?

Then, I would like to merge the chroma of the following frame according the mask.. but it doesn't work.. :-/
mt_merge(clp, next, mask, y=2, u=3, v=3)

[jmac698]

By range he may mean to keep the maximum numerical accuracy in an integer calculation. This would be his habit. In this case it didn't do anything; however he still had the habit.

lol but I suppose he should answer :)

[cretindesalpes]

If you square 100 it will become 10000 which cannot be stored in a byte (0–255) for temporary storage. Actually the operation could have been done differently, using a simple mt_makediff for the basic planar difference and mt_lutxy to do the sqrt(U²+V²) operation, so squares, addition and square roots are done in a single step. Before reading that you would need only the chroma plane processed, I tried to write the same kind of script without using mt_lutxyz (because this one is long to initialise and is memory-heavy) then modified it afterwards. Sorry if I confused you, I am a bit confused myself…
mt_makediff (clp, prev, y=1, u=3, v=3)
clip1 = mt_lutxy (SwapUV (), "x 128 - 2 ^ y 128 - 2 ^ + 0.5 ^", y=1, u=3)

For mt_merge, I used luma=true, which is a special parameter indicating that the mask contained in the Y plane should apply to U and V too after appropriate scaling, and that U and V will be processed without needing to set u=3 and v=3 explicitly.

[mathmax]

But I want to merge only the chroma.. As I can't figure out how mt_merge() works, I used this line instead:

overlay(clp, mergechroma(clp, ref), mask=mymask)

Thank you very much for showing how to break the problem down in several steps.. that's amazing!