Double-blend IVTC Removal

[foxyshadis]

MOmonster's FixBlendIVTC 0.9 (http://forum.doom9.org/showthread.php?t=95924), part of R_pack, has definitely surpassed my function on speed, reliability, quality, and tweakability, so I advise you all to grab a copy of it and let my old version rot. ;) And it now includes a readme!

The same caveat applies: Any form of pre-processing (other than a simple blur or resize) will disrupt the blend detection.

Requires 2.0a22+ (http://forum.doom9.org/showthread.php?t=98985), Average (http://forum.doom9.org/showthread.php?t=100626), Removegrain (http://home.arcor.de/kassandro/RemoveGrain/RemoveGrain.rar), and Decomb (http://neuron2.net/decomb/decombnew.html) for decimating.


-------------------

Old News


v0.4, now with automatic pattern-matching! MOmonster and mg262 contributed greatly to helping me achieve that.

Uses difference masks and edge masks to improve unghosting. The less filtering you use beforehand, the better detection and reversal works. Unblend/removeblend is faster and about as accurate with single blends, especially those without a pattern.

Requires masktools and Average (http://forum.doom9.org/showthread.php?t=100626). Note that Momonster is actively developing a higher quality and faster version, I'll probably post that here when he finishes.


function fx_unblendTC(clip input, bool "nopp") {
nopp = default(nopp,false)
## Non-volatile variables ##
global source = input
global greymain = source.levels( 16, 1, 200, 128, 128, coring = false)
global test = Average(greymain, 1, source,0.25, source.trim(1,0),-0.5, source.trim(2,0),0.5, \
source.trim(3,0),-0.25).reduceby2().YV12Lut("x 128 - 3 ^ 4 / 128 +")
global grey = greymain.reduceby2
global framez5 = blankclip(test,length=5) + test
global framez4 = blankclip(test,length=4) + test
global framez3 = blankclip(test,length=3) + test
global framez2 = blankclip(test,length=2) + test
global framez1 = blankclip(test,length=1) + test
global frame0 = test
global frame1 = test.trim(1,0) + blankclip(test,length=1)
global frame2 = test.trim(2,0) + blankclip(test,length=2)
global frame3 = test.trim(3,0) + blankclip(test,length=3)
global frame4 = test.trim(4,0) + blankclip(test,length=4)
global frame5 = test.trim(5,0) + blankclip(test,length=5)

# frames to use for output
sframe3 = input.duplicateframe(0)
sframe4 = input
sframe5 = input.trim(1,0)
sframe6 = input.trim(2,0)

# unblend output and various post-processing measures
unblend1 = Average(sframe3, -1.0, sframe4, 2.0)
unblend2 = Average(sframe5, 2.0, sframe6, -1.0)
edgemerge = fx_unblend_edges(unblend1,unblend2)
unblend1 = edgemerge.selectevery(2,0)
unblend2 = edgemerge.selectevery(2,1)
diffframe1 = overlay(sframe4, sframe3, mode="difference") \
.levels(120,1,192,0,255,coring=false).binarize(threshold=30,upper=true).blur(.5)
diffframe2 = overlay(sframe5, sframe6, mode="difference") \
.levels(120,1,192,0,255,coring=false).binarize(threshold=30,upper=true).blur(.5)
unblend1 = MaskedMerge(unblend1,sframe6,diffframe2)
unblend2 = MaskedMerge(unblend2,sframe3,diffframe1)

# one-step deblend, use for speed
global output = Average(sframe3, -0.5, sframe4, 1.0, sframe5, 1.0, sframe6, -0.5)

global unblended = nopp ? output : Average(unblend1, .5, unblend2, .5)

## Volatile variables ##
global isblend = false
global isblend1 = false
global isblend2 = false
global isblend3 = false

## Conditional Chain ##

d99=scriptclip(source, """(isblend1 == true ? unblended : \
(isblend2 == true ? unblended.duplicateframe(0) : source))""")

# new pattern must start minimum of 4 from last.
d5=FrameEvaluate(d99, "isblend = (!isblend1 && !isblend2 && !isblend3) && \
(residue / div < .2)")
# closer the pattern match (max 8) the higher the allowed threshold
d4=FrameEvaluate(d5, "global div = frametesta + frametestz >= 7.5 ? 500 : \
(frametesta + frametestz >= 7 ? 5 : (frametesta + frametestz >= 6 ? 3 : \
(frametesta + frametestz >= 4 ? 1 : .01)))")
# assign weights to pattern positions
d2=FrameEvaluate(d4, " global frametestz = (( min(diffz1,diffz2) > residue ?2:-4 ) + \
( diffz3 > residue ?.5:-.5 ) + ( diffz4 > residue ?.5:0 ) + \
( diffz5 < min(diffz1,diffz2,diffz3,diffz4) ?1:0 ))
global frametesta = (( min(diff1,diff2) > residue ?2:-4 ) + \
( diff3 > residue ?.5:-.5 ) + ( diff4 > residue ?.5:0 ) + \
( diff5 < min(diff1,diff2,diff3,diff4) ?1:0 ))")
# set up differences
d1=FrameEvaluate(d2, "global residue = LumaDifference(frame0,grey)
global diffz1 = LumaDifference(framez1,grey)
global diffz2 = LumaDifference(framez2,grey)
global diffz3 = LumaDifference(framez3,grey)
global diffz4 = LumaDifference(framez4,grey)
global diffz5 = LumaDifference(framez5,grey)
global diff1 = LumaDifference(frame1,grey)
global diff2 = LumaDifference(frame2,grey)
global diff3 = LumaDifference(frame3,grey)
global diff4 = LumaDifference(frame4,grey)
global diff5 = LumaDifference(frame5,grey)
isblend3 = isblend2
isblend2 = isblend1
isblend1 = isblend")
return(d1)
}

function fx_unblend_edges(clip sup1, clip sup2) {
edgemask1 = sup1.edgemask(thY1=2,thy2=2,thC1=4,thC2=4,type="special").levels(96,1,160,0,255,coring=false)
edgemask2 = sup2.edgemask(thY1=2,thy2=2,thC1=4,thC2=4,type="special").levels(96,1,160,0,255,coring=false)
diffedgemask1 = overlay(edgemask1, edgemask2, mode="subtract").inflate().blur(.7)
diffedgemask2 = overlay(edgemask2, edgemask1, mode="subtract").inflate().blur(.7)
edgemerge1 = MaskedMerge(sup1,sup2,diffedgemask1)
edgemerge2 = MaskedMerge(sup2,sup1,diffedgemask2)
return interleave(edgemerge1,edgemerge2)
}


Because it improves the deblending with a difference mask, the stronger the luma difference the less artifacts will show in the output.

A few clips for your perusal below: (Xvid q4)
2: blend (http://foxyshadis.slightlydark.com/random/cb_blendclip2.avi) unblend (http://foxyshadis.slightlydark.com/random/cb_unblendclip2.avi)
3: blend (http://foxyshadis.slightlydark.com/random/cb_blendclip3.avi) unblend (http://foxyshadis.slightlydark.com/random/cb_unblendclip3.avi)
4: blend (http://foxyshadis.slightlydark.com/random/cb_blendclip4.avi) unblend (http://foxyshadis.slightlydark.com/random/cb_unblendclip4.avi)
5: blend (http://foxyshadis.slightlydark.com/random/cb_blendclip5.avi) unblend (http://foxyshadis.slightlydark.com/random/cb_unblendclip5.avi)
6: blend (http://foxyshadis.slightlydark.com/random/cb_blendclip6.avi) unblend (http://foxyshadis.slightlydark.com/random/cb_unblendclip6.avi)

Usage is just fx_unblendTC(). If there are long stretches of no blends, it shouldn't hurt to leave it on but it will slow it down. It typically won't get the very first and last blends of a clip, so a few still show in the samples even though they were caught fine when I used the full video.

After reversal use TDecimate or FDecimate as required. Random access is unreliable so make sure that every frame is touched in order (tdec mode 5 won't). It is recommended for now to write the output to a lossless temp file and working from there. (Inline Fdec worked fine for 4-6 but messed up the patterns for 2-3.)

v0.2 - I realized I was using subtract and it was throwing off some calulations. Also, added static area detection, much better on static parts (both forward and backward). Removed unnecessary conditionalfilters.
v0.3 - yv12subtract for full reversals, a few additional options.
v0.4 - rewritten as conditional, automated pattern detection.

[Didée]

With all respect for the work you put into this ... when I tried it on one source with "blend-IVTC" I've lying around, your function acted as a perfect NOP. Not one single pixel was changed. Input==Output, bit-identical.

Are you sure everything's okay with that function? Perhaps some hardcoded values should be parametrized?

[foxyshadis]

I'm not actually sure. The first frame number does have to be the start of a 5-cycle of 3 clear and 2 blends - even if it's across a scenechange, it won't hurt it. Can I see a piece of your clip? Having more than the three or four I have (which are mostly very similar in their blending patterns) would be really nice to have, although the effect is definitely there on the ones I tried.

I'll probably go back in with a much wider test body for the next iteration, I've had a few ideas percolating.

And don't worry about my feelings, it's just a personal quest for perfection that I hope might be of use to others someday as well. =p

[MOmonster]

I couldn´t test the function till now. I think it is only for NTSC footage (right?). But also if Didee tested a pal-clip, I think there should happen something. It looks a bit komplex for me, but therefore it avoids the evaluate enviroment. How good get it clear with Edits.

EDIT:
I´m a bit confused :confused: . I have seen pal to ntcs blend-conversions (cycle=6) and of course ntsc to pal blend-conversions (cycle=25), but you work on the cycle 5?
Can anybody explain me, what is mean with blend-IVTC (or better blend-TC)?

[foxyshadis]

Telecined footage, but instead of being IVTC'd like any sane method, frame-blended to remain 30p without being wildly interlaced every few frames. That's my sources. If pal->ntsc clips are much different, I'd have to see a few. (Just changing the cycle to 6 would be simple though.)

I'm afraid it wouldn't be any good with all blends through a whole film, it's meant to use clean frames around the blends to wipe away blend-artifacts. I'm still working on different algorithms on the hunt for something better. Maybe I should use some of your guys' really complex scripts for cribbing. :p

Edit: Oh geez I'm such a huge idiot, I need to redo a few things. Give me a bit to sort things out.

Also, I hate it when inspiration strikes at 4 in the morning.

Edit2: Okay, I modified it to use difference rather than subtract. Also got my static area inspiration working.

I suppose my examples aren't great. What I do is:
x=mpeg2source("file")
x=x.fx_unblend(316,343)
return x

This returns it in 30fps with the unblended frame duplicated. Once I have the ranges I want, I run tdecimate on it. But I included 24 in case someone wants it. Once I get it more thorough and more efficient, I'm going to add parameters for noise thresholds too; right now it's made for really noisy sources, but it shouldn't hurt clean ones much. Cycle shouldn't be hard, I just need to think about how to apply it with selectevery/interleaveevery.

Edit3: Changed the thread title to be clearer as to its purpose.

[MOmonster]

Yes, this is what I also thought yesterday evening. Here I can only play this file with vlc (slow motion), because it´s not my pc, but of course that doesn´t give me so much informations.
If I understand it right, this is the pattern:
example1: A AB B BC C D DE E EF G
And we know, the blendweights are all 50:50, because it was a result of deinterlacing. The Blends doesn´t seem to be bad catchable (good wight and clear structures because it´s a Anime), so Cdeblend or restore24 should also are able to kill the most blends.
But if I´m right with the blendweight, why not just using a simple differencing? If we had the pattern example (example1), then the differences between B and BC (B->BC) should be really similar to the difference BC->C and:
D->DE ~ DE->E
E->EF ~ EF->F
...
And in general the difference C->D would be significantly bigger than the differences around. So if:
C->D * thresh > BC->C (and then of course also B->BC) &&
C->D * thresh > D->DE (and then of course also DE->E)
with a thresh around 0.7 or 0.75, we know the pattern. Its the only Difference, that can be significantly bigger than the differences around, because all other differences has a really similar difference.
If C->D isn´t the biggest Difference for example because of a scenechange, then we just keep the last pattern till the next clear decission.

That´s the way, I would solve the problem, but of course this can fail and is maybe worse than your way, but I like to make it a bit more easier and faster. ;)
If I was wrong with the blendweight or the pattern, please tell me. I never has sawn such sources before.

[foxyshadis]

If it was that easy, I'd have never started this. Sorry I'm not very good at explaining though, I guess I'd hoped it was more common but it doesn't seem that it is.

The pattern is A B C CD DE | E F G GH HI | I... where my filter returns A B C D D | E F.... So for every group of five frames to be returned to 4 requires extracting the last one from the frames around it somehow. I've seen premiere barf up something like this before, and it came up again while working on a set of videos someone would like me to restore from low-quality mpegs, so I made this up.

I know one other person posted recently (http://forum.doom9.org/showthread.php?t=100952) with a problem like this, so I put it out in case it was a larger problem to see if there was any demand for developing it beyond my immediate need. It is rather a special case, I know.

I did spend some time before I started experimenting with crestore, restore24, and restorefps, and I got the best results out of the latter, but not great. I didn't want to clutter up other threads with my bizarre blending patterns. :p

Edit: Wow, I feel tremendously stupid having done all that just to figure out it could be boiled down to several add/subtract statements. ;_; It'd be even simpler if overlay would return negative differences as well as positive.

# experimental
subframe1 = overlay(frame4, sframe3, mode="subtract")
subframe1swap = overlay(sframe3, frame4, mode="subtract")
subframe2 = overlay(frame5, sframe6, mode="subtract")
subframe2swap = overlay(sframe6, frame5, mode="subtract")
frame1add = overlay(frame4, subframe1, mode="add").overlay(subframe1swap, mode="subtract")
frame2add = overlay(frame5, subframe2, mode="add").overlay(subframe2swap, mode="subtract")

# finally, merge. naive will be 50% orig, 25% each blend, mine hopefully rather better.
deblended = merge(frame1add,frame2add)

The output could still be quite improved, maybe with some of the other stuff I wrote, but... sigh. What a way to waste a weekend. I'll chalk it up to a learning experience and rusty math.

Edit2: (I'm editing all over this thread.) Actually, a straight subtract brings out some horrific mpeg artifacting, since it doesn't play nicely with mpeg's psychovisual model. Heavy pre-filtering helps but not really enough. Wow. I'll have to ponder this new wrench.

[MOmonster]

Ok, I finally get it. Now, your function is also more clear for me. Maybe in a week, I can have a closer look on the source and your script, but now I have no suggestions for you.
Keep up the good work. ;)

Edit: Is it right that the blendweight is for all blends 50:50?

[mg262]

It'd be even simpler if overlay would return negative differences as well as positive.I haven't tested it in this incarnation, but the Average plug-in I recently released should be able to cope with negative weights, if it is any use...

http://forum.doom9.org/showthread.php?t=100626&highlight=Average

(In fact the code was just pulled straight out of the experimental fieldblending reversal plugin, which computes plenty of weighted averages to reverse blending. What all the maths in that is for is to figure out the correct add/subtract weights for arbitrary patterns... and to detect/cope with pattern breaks. RestoreFPS was hardwired to reverse a specific blending pattern, which is why it couldn't deal with your case that well... )

[MOmonster]

@mg262
So your Average Filter take also negative Weights. This of course would make it much more effective. :D
Is this possible? (I can´t test it here)
Average(alpha, -1.0, beta, 1.0, gamma, 1.0)

[mg262]

Yes. If you use weights which are (IIRC) < -2 or >= 2, it will use C instead of assembly, with a corresponding slowdown -- but I don't think there are any other limitations. It does exactly what it says, namely "average" = sum with given weights, so if you use it to subtract a clip from itself you will get 0 in both luma and chroma (not 128)... obviously you can get around this by adding an appropriate BlankClip as an extra argument. (In the case of fieldblend reversal, I don't think this should ever be an issue.)

I've just run a brief test and everything seemed to be fine, but if you have any problems, poke me...

foxyshadis, you can have your thread back now ;)

[MOmonster]

Ok, because I want to learn, too, I created also a small function against this problem:
find last version on page 3
The function is fast and easy, but there are three problems, two because of Average (stability and a green right). The other is the scenechanging. The function don´t work on the first five frames and can have problems on scenechanges.
If it is not stable enough on your pc, I can also use Lutxy instead of average (but that wouldn´t be that amasingly fast). ;)

Edit: mg262 has fixed the bug. Use the newest Average version and everything is running fine. ;)

[foxyshadis]

I actually started on a rewrite using yv12subtract, it was much simpler than the overlay mess, but I think I hit a bug in avisynth's caching. (It's weird, if I returned side-by-side the difference behind and in front of the blend frames, it worked like it should, but if I showed the output in the left the right pane changed! It became a compare to two frames forward instead of the next one, and it seems to cause the output to mess up. And it happens so rarely but consistently that I can't make a testcase without going "here's my function and here's the frames it screws up on", even minor function tweaks changes where it shows up.)

I'm going to give yours a shot, it looks promising. I really need to get more comfortable in conditional environments, I mostly get a lot of "I don't know what 'blah' means" even when I work from the doc's examples.

[MOmonster]

I changed the code a little bit for you.
Because your sample is an anime source, maybe there are 12fps scenes. For this I changed the pattern conditions a bit and added a alternative code. Because for 12fps parts there is only one blend of course we can´t use the same method. I´m not sure if "Cowboy Bebop" has 12fps sequences, because your sample is not typical for that, but there are many other sources, that has still many 12fps sequences.

[foxyshadis]

It has 12 and 24 fps sequences, so thanks for considering that. (Although with a single blend the recovered frame is obviously the same as the frame where the other blend would've been, you just have to figure out which is which. In testing mine I noticed that if you chose the wrong offset it would be very obvious even though it should have been roughly equivelent, in a single pattern, whichever way you went.)

[MOmonster]

So the pattern of your function is really constant? My function try to find the pattern with the LumaDifference, if it is not clear the function decimate the some frame like in the last pattern but it doesn´t try to restore a clear frame, because he could use the wrong frames and that will look worse then the source.

[foxyshadis]

I should add some more samples, that way you'll have more to test against if you want. The pattern changes every scene change and occasionally within a scene (I manually shifted it in those cases).

2 (http://foxyshadis.slightlydark.com/random/cb_rawclip2.avi), 3 (http://foxyshadis.slightlydark.com/random/cb_rawclip3.avi), 4 (http://foxyshadis.slightlydark.com/random/cb_rawclip4.avi), 5 (http://foxyshadis.slightlydark.com/random/cb_rawclip5.avi), 6 (http://foxyshadis.slightlydark.com/random/cb_rawclip6.avi)

(They should be uploaded within 10 minutes, all 3-5 megs and about a hundred frames.)

I'm thinking that with less clean sources difference squared would be better than plain lumadifference, but I'm not sure how to do that in pure avs script. (mg, since you're so awesome at whipping up quick filters, would it be possible to make a ssd/satd/whatever filter with your restorefps code? Unless one already exists that I don't know about.)

[mg262]

More than happy to! (But I'm out all day today, so you will have to wait a little -- sorry about that.) I exposed my blend detection method for use yesterday (check the RestoreFPS thread)... that might be what you need? Otherwise I can easily expose the SSD as well.

[foxyshadis]

Wow, that's hot! I hadn't seen your edits in the other thread. I'll test it out right now.

(I think Y/U/VDifferenceToNextSquared functions might be useful to other authors, but I don't know how much. I have no idea how useful satd would be, but that's mvtools land.)

[mg262]

So, for anyone else reading this thread, foxyshadis and I sent a couple of PM's back and forth... the upshot being that although the BlendAngle is not meant for consecutive blends, although it seems not to be as bad as I thought. I also promised to give the error measure I would use for this particular problem (if you don't want to wade through the maths, skip to Conclusion)...

We start with this model for how our blended frames A,B,C,D are produced:
A=x
B=0.5x+0.5y
C=0.5y+0.5z
D=z

And we are trying to recover the original unblended pictures x,y,z. There are two methods you can use, one simple one where you just use x=A and z=D, and a more complicated one where you try and use the information in B to give a better guess for x. I'm going to stick with the simple method.

The least squares method I use in RestoreFPS gives the following estimate y' for y:
y'=-0.5A + B + C -0.5D
also as noted
x'=A
z'=D
(And MoMonster has independently come up with the same thing.)

Now, we can plug these guesses for the original frames back in to our model to see what they would have actually given when blended.

B' = 0.5x'+0.5y' = 0.5A + 0.5(-0.5A + B + C -0.5D) = 0.25A + 0.5B + 0.5C - 0.25D
C' = 0.5y'+0.5z' = 0.5(-0.5A + B + C -0.5D) + 0.5D = -0.25A + 0.5B + 0.5C + 0.25D

Now, if the original model was sensible, B' and C' should be very similar to B and C respectively. So the sum squared differences between B and B', and between C and C', should be small. I.e., the following two quantities should be small:

(B'-B)^2 = (0.25A - 0.5B + 0.5C - 0.25D)^2
and
(C'-C)^2 = (-0.25A + 0.5B - 0.5C + 0.25D)^2

These are the same quantity... so to determine whether in the model is a good one (i.e. blends lie in this particular pattern), we compute (0.25A - 0.5B + 0.5C - 0.25D)^2 and see if it is small -- i.e. we compute (0.25A - 0.5B + 0.5C - 0.25D) for each pixel, square the results and add.

Note that we can multiply this quantity, (0.25A - 0.5B + 0.5C - 0.25D), by any constant we choose (it just affects the blend-detection threshold). So we could use A-2B+2C-D... but the original (or a version with even smaller coefficients) may be preferable to prevent overflow.

Probably the easiest way to implement this is to use Average or some other plug-in to compute (0.25A - 0.5B + 0.5C - 0.25D) (offset at 128), and I will throw together a plug-in that subtracts 128 from each pixel, squares, and adds together the results to provide a quantity you can threshold on.

I try to stay away from the actual equations where possible... I hope that wasn't too unreadable. In any case,

Conclusion:
To check whether you have this pattern:
Frame n =A=x
Frame n+1=B=0.5x+0.5y
Frame n+2=C=0.5y+0.5z
Frame n+3=D=z
Compute (0.25A - 0.5B + 0.5C - 0.25D), and see if the average squared value of this is small. [You can try using other averages, like LumaDifference.]

Edit: the "complicated" estimate of x is 0.9A+0.2B-0.2C+0.1D (and z similarly). But calculating something like this may magnify the errors due to MPEG encoding... so be careful if you try it.

[MOmonster]

I have allready the idea for a blenddetection that is easy and not to slow and makes no difference between 12 or 24fps, so maybe it will work better. But I can´t test it now, tomorrow I think I can tell more.

@Clouded
There is allready a possibility to get the SSD for low differences. With Lutxy we can create a squared diffmask and then get the value with AverageLuma. But this way is slow and strong limited (only for low differences), so that a function would be really useful for many of my functions.
I´ll test your blenddetection also this evening ;) .
For what is the complicated way useful? We can change the influence of every frame on the result, but for this example we don´t know what has the better quality. Or is the only for the explaination? If I give A a higher weight, then we can´t compare the result so good.

[mg262]

Assuming that noise is introduced after blending...
another thing - i couldn't find a pertinent bit to quote, but you mention you're assuming a "noiseless" blend between fields. the converter is digital, so no matter how noisy the source is, the actual blending will not have any noise (except rounding error, but that should be very minor). however, after mpeg-2 compression a fair bit of noise could be introduced....and also making sensible assumptions about the noise (such as that you get the same amount on every frame), then the maths predicts that

0.9A+0.2B-0.2C+0.1D

will give you a better (more accurate/less noisy) guess at what the original frame was than simply using

A

. On the other hand, the reduction in noise is only predicted to be 10%, so I'm not sure it's worth trying. (Until I did the calculation, I expected a slightly larger effect...)


foxyshadis+MoMonster,

Something I forgot to mention in the other thread (sorry about this... yesterday was very crazy) is that you can dump the number-streams produced by BlendWeight, BlendAngle to a text file by writing this kind of thing:bob
source = last
angle = blendangle(source)
WriteNumber(angle, "angle.txt")
-- but be warned that that will calculate and dump the numbers of all the frames -- so either use it on short clips or be prepared to wait.

I was thinking of putting together a function called AverageSquare which subtracts 128 from each pixel value, squares them all, averages the values for each frame, and returns the result like blendangle or blendweight so you can threshold on it. Average-Square has all the nice properties of Sum-Square, plus the numbers are smaller and comparable between frames of different sizes. (Taking the square root of this gives another number, RMS or root-mean-square -- in some ways nicer because it actually corresponds to a legal luma value, and because it directly corresponds to the distancess on the triangles I drew.) Probably just C++ to begin with, with assembly added when I have a little more time.

So you would write something like
A = source
B = source.trim(1, 0)
C = source.trim(2, 0)
D = source.trim(3, 0)

Average(D, -0.25, A,0.25,B, -0.5,C,0.5)
#D first because length of the result is taken from the first argument... perhaps I should change this to the shortest length?
blendprobability= AverageSquare()
If(blendprobability.isbelow(...),...,...)

Would that be satisfactory?

Edit: bug in that script... I should have added an extra BlankClip argument to Average in order to add 128 to all the pixel values...

[foxyshadis]

Clouded, Momonster, one of the things I found after I switched to yv12subtract (which is essentially the same as average but in two steps), is that a straight difference will do an admirable job of giving back the original frame but still rather artifacty, and each has artifacts from the clear frame next to it, so a further difference of the two and swapping areas with edges that don't appear in the other. (Edges are easiest, since I don't know how to decide which frame to pick parts from in an overall difference.) It significantly improves the recovery from a compressed source.

ie, this:

edgemask1 = reversedframe1.edgemask(thY1=2,thy2=2,thC1=4,thC2=4,type="special") \
.levels(96,1,160,0,255,coring=false)
edgemask2 = reversedframe2.edgemask(thY1=2,thy2=2,thC1=4,thC2=4,type="special") \
.levels(96,1,160,0,255,coring=false)
diffedgemask1 = overlay(edgemask1, edgemask2, mode="subtract").inflate().blur(.7)
diffedgemask2 = overlay(edgemask2, edgemask1, mode="subtract").inflate().blur(.7)
frame1 = MaskedMerge(frame1, frame2, diffedgemask1)
frame2 = MaskedMerge(frame2, frame1, diffedgemask2)


The specific edgemask, inflate, and blur values are just what gave better results on a few segments of this source after testing, I made no real effort to find the 'best' possible values but I might be able to do something if it'd help.

I also have something that I realized is a sort of clipped median, parts of a clean and blended frame that are essentially equal should be propogated across where the average would move the value, but this I haven't tested as much so I don't know how useful it really is.

diffframe1 = overlay(blendframe4, clearframe3, mode="difference")
diffframe2 = overlay(blendframe5, clearframe6, mode="difference")
# if identical areas are found between subsequent frames, use them, forget the algorithm
samemaskframe1 = diffframe1.levels(120,1,192,0,255,coring=false) \
.binarize(threshold=30,upper=true).blur(.5)
samemaskframe2 = diffframe2.levels(120,1,192,0,255,coring=false) \
.binarize(threshold=30,upper=true).blur(.5)
frame1 = MaskedMerge(frame1,clearframe6,samemaskframe2)
frame2 = MaskedMerge(frame2,clearframe3,samemaskframe1)

Again the precise values are more 'what worked' than calculated anyhow.

I don't know if you want to build the post-processing into it, but at least the first step would probably be essential for encoded sources and couldn't really be done post-merge. Unless yours just leaves the two separate frames in the output, though a merge and copy is also good for artifact reduction.

Oh yeah, question. Your filter scans ahead as well as back, right? One of the things that annoys me about removeblend (ack, all these unblenders, I'm getting confused) is that it'll take a frame that's just marginally within the threshold and skip a one that's way under it since it can't have two in a row, rather than looking ahead and picking the best. (Which also changes its forward/backward behavior.) I'm probably going to try modifying the source for that one.

[mg262]

AverageSquare, 24 October 2005 (http://people.pwf.cam.ac.uk/mg262/posts/blend/AverageSquare_24Oct05.dll)

Just use source.AverageSquare(), and you will get a number stream like that produced by BlendAngle()... you can use the same IsAbove+If/WriteNumber functions on it. It works as noted above, i.e. subtract 128 from pixel values before taking the square.

N.B. these number streams are just like clips... you can use Trim(1,0) to get the value for the "next" frame, and more generally cut/splice/interleave them just like clips.

Proper reply tomorrow... but the filter will definitely look ahead. Don't worry too much about feedback on the blending methods... for the general case it's quite hard to squeeze things into the maths. I'm pretty much limited to just pushing it through and seeing what comes out... although I envisage postprocessing in scripts which will depend on the source type -- for example my blended material is largely VHS, so macroblocks aren't a problem!

[foxyshadis]

It isn't so much macroblocks as residuals of the previous/next frame that were supposed to be in the blend but were compressed away. Hmm. I thought VHS tended to similarly compress/smear areas with minor differences, but maybe that's only older tapes.

[mg262]

Just for fun, try this...
avisource("...")
source = last
A = source
B = source.trim(1, 0)
C = source.trim(2, 0)
D = source.trim(3, 0)
grey =d.levels( 16, 1, 200, 128, 128, coring = false)

residueframe=Average(Grey, 1, A,0.25,B, -0.5,C,0.5,D, -0.25)
residue= AverageSquare(residueframe)
WriteNumber(residue)
ispatternstart =residue.isbelow(0.7)
singleframe = ispatternstart.Trim(0, -1)
isfirstblend = singleframe +ispatternstart
issecondblend = singleframe+singleframe+ispatternstart

if (isfirstblend,A.Trim(1,1)+Average(A,-0.5,B, 1,C,1,D, -0.5), \
if (issecondblend,A.Trim(1,2)+Average(A,-0.5,B, 1,C,1,D, -0.5), \
source.greyscale))
stackvertical(source, last)

You may need to adjust the 0.7 threshold. Frames that have been passed through unchanged are grayscaled to make it easier to see the reconstruction. The blend detection method here is very primitive and uses no pattern guidance -- the filter should do a lot better (like I said, this is just for fun!). I don't actually think AverageSquare() is the best thing to use in a reversal script in these circumstances... we maywant something that penalises large values even more heavily. Try looking at stackvertical(source, residueframe), noting that the residue is low on the frame immediately before the blended pair.

NB decimation should occur at the same time as reversal, i.e. all the secondblends should be removed... but that's hard to achieve in a script.* So I just duplicated frames.

*Just as I wrote that I had an idea... I will elaborate on it tomorrow.

Edit: it should be possible to improve the results by borrowing one of MoMonster's ideas and blurring the residueframe. This needs a blur with fixed weights -- Blur() or one of the gaussian blurs or box blurs floating round this forum. Or... ReduceBy2() or even ReduceBy2().ReduceBy2(). Just change

residue= AverageSquare(residueframe)

To

residue= AverageSquare(residueframe.ReduceBy2())

[foxyshadis]

I would tend to prefer Tdecimate; simply removing the frame would work, but I'm afraid all the scene changes would add up and throw off audio sync. Maybe a random distribusion would keep the sync to within +- 2 frames? Static scenes might do the same, but the filter is probably sensitive enough to ignore that.

Watching you is like sitting in on a good professor, very clear and always bubbling with new ideas. ;) I apologize for taking your time off your own projects though. I'm off to test this.

Okay, I understand how it works now, I'm going to look for ways to tweak it for better accuracy. There's some obvious ways (you're right, it needs a stronger differencing, but levels helps there) and some that'll be a bit tougher (pattern matching with realignment). I should be able to massage momonster's script into that. Is it bad that I like working on the method more than the video? >.> Must be my lack of time-consuming projects lately.

Just curious, but is there a way to get at the actual numeric value, rather than a boolean lessthan/morethan? I'd like to compare with nearby frames; if it has the lowest residue with a 4 frame radius, assume it is the correct and then put in sanity checks for pattern changes, and vice versa for others. Unless a localminimum would be better implemented another way; to be honest I don't know enough about your celbackground tools and I should probably dig deeper into them.

[MOmonster]

@foxyshadis
I had a closer look on your examples and created this function:
newer version on the next page I hope there are no strange bugs in it, because I tested it on another Pc and I couldn´t copy it, so I written what I know :D .
Your second example is really a nightmare. Also this function don´t detect the blends at the beginning of a scenechange (edit), but the result seems to be accurate enough. The quality could be a little bit better, maybe you can add your ideas, also it seems to be better just using two frames for restoring.
Instead of tweak you should set a deblocker or another spatial filter, but there has to be a filter, else you can´t use decimate or tdecimate after. Maybe I can use the decimation method of Crestore to avoid the use of a slow decimater. We´ll see.

@Clouded
Sorry I couldn´t test your filter, there was a burning problem. I´ll try it once more. :D

Edit: Ok, this version wasn´t free of bugs, hoply this one is it now. I add YPlaneMinMaxDifference to get the frames with the better quality, also deblock() from DGdecode.dll is used. The blenddetection is the same as before, but quality and speed (because tdecimate isn´t necessary anymore, won´t give better results!) is better. :D
I won´t do much more against this problem, because the function works good and I still never had a similar problem source.

Edit2: I was able to copy the code, it should now really be bugfree.

[mg262]

I'm really sorry I have taken so long to reply -- I woke up in a coding mood on Tuesday and this thread somehow slipped my attention.

@foxyshadis,
I'm really embarrassed now! Don't underestimate how much I learn from you guys -- my approach to things is very much to throw maths at it; sometimes it works, sometimes it doesn't, but it's not a proper substitute for the intuition for video that you scripters have -- I marvel at some of the things that are accomplished with scripts. For that reason, I particularly enjoy writing things to be used in complex scripts... and I learn a lot from seeing how they are used. So, please don't worry about distracting me from my 'own projects'!

Is it bad that I like working on the method more than the video?Believe it or not, I have yet to back up a DVD per se :p

I'll add functions called Minimum (taking any number of arguments), Maximum, and IsEqualTo -- that should let you do what you need? (The idea behind these number-per-frame and boolean-per-frame pseudo-clips is that it is very easy to code new functions to process them -- so do list anything that might conceivably possibly be useful and I will throw it in. Incidentally, don't worry about the delving into Cel Background -- it set up the framework for these pseudo-clips, but it's not finished or properly documented yet.) If that isn't enough to deal with the issue, I can try and look up how to expose things to/read information back from ConditionalFilter/ScriptClip -- but I find the weird back-to-front inside-out syntax of these very unwieldy, so I prefer to use things that behave like clips where possible.

@MoMonster,

At the time I started writing assembly, I had no decent references in front of me so I didn't know what was needed to allow things to run on non-SSE2 boxes (turns out that the answer is very easy -- just don't use 128-bit registers), and also because I was used to being on the low-end of the technology scale I underestimated how many people couldn't use SSE2. I am now writing iSSE as standard for new plug-ins, and I will rewrite the blend reversal to deal with that. Sorry for the inconvenience. I'll reply to your PM shortly.

Edit:
ReverseBlend, 29 October 2005 (http://people.pwf.cam.ac.uk/mg262/posts/blend/ReverseBlend_29Oct05.dll)

Contain functions Minimum, Maximum, IsEqualTo as promised. Usage example:

next = residue.trim(1, 0)
WriteNumber(maximum(residue, next, next.Trim(1, 0)), "maximum.txt")

[I just added the file name argument to WriteNumber as well -- thought it was already there, but obviously I missed it... .]

Min and Max can be used as synonyms for Minimum and Maximum.

[foxyshadis]

Don't worry about time, I was actually pretty busy at work myself and then working on testing whether my minimum pattern would really help in a real function. Thank you for taking the time to implement my suggestion though. =D

Avisynth supports overloading functions based on argument types, right, so jdl's min/max for int/float types won't be affected?

I actually switched to yv12lut ("x 128 - 3 ^ 2 / 128 +", ie, x^3) and lumadifference with momonster's framework for the time being, and it really made a difference on accuracy once I added minimum-finding. My method is to take the 5-clip average to find the blend, then combine the two two-frame averages in the same way I did the others to clean them up a bit more than a direct 4-frame average. Definitely faster with average, less intermediate operations.

I'll rewrite for your min/max. The best thing about it is that there are no arbitrary thresholds (aside from a maximum sanity check), it should be able to work on segments with high or low noise without much problem. I'll post the current one once I put some finishing touches on it and then the next one later when I can get around to it.

[foxyshadis]

Okay, I've replaced the first post with the updated function, using momonster's framework, clouded's residuals, and my weighted pattern-matching. It all comes together with extremely high accuracy and only one noticed false positive (in a static scene). It could be trivially modified to support other patterns.

Maybe a small speedup would be a dup detection for "blends" across duplicates, at least save the effort of testing and unblending; I avoided keeping all frame lumadiffs so random access is sort of reliable. Rewriting for clouded's numeric-clips will give a much greater speedup, without all the lumadiffs. Random access is very slow now since it has to compute all ten frames around one, and it and moving backwards will use bad patterns because of the volatile variables, but going forward corrects things.

(Any way of avoiding these (isblendx) in non-conditional functions? Idempotence is important. I'm thinking a numeric clip that could be set to true/false (ie, isabove(1) maybe) and shifted around to look for a pattern start. I need it because it's a hard rule that no pattern can start less than 4 frames after the last, otherwise the last one wouldn't be detectable.)

I want to think about whether anything else could be included, maybe logged output, and generate a few more clips as examples before I move on to the next version.

Usage is just fx_unblendTC().

If anyone wants to hammer at the detection to see if you can tweak the thresholds for better results, you should replace d99 with:

d99=scriptclip(source, """(isblend1 == true ? unblended.subtitle("blend",align=9) : (isblend2 == true ? unblended.duplicateframe(0).subtitle("blend",align=9) : source)).subtitle(string(residue)).subtitle(string(frametesta + frametestz),align=4,y=-1)""")

Basically it's just a debug that shows whether the fame is blended, the absolute residual difference, and the final result of the tests.

(I considered a per-test debug mode but the last thing I need is even more variables.)

I'm so happy to get this done, I learned so much but I want to get back to actual encoding now. ;p

[MOmonster]

Nice function foxyshadis. Not so easy to understand, but I´ll have a closer look on it this evening (I can´t understand all the LumaDifference calls for now). Have you testet it allready with Fdecimate? If not, please don´t give the advice to use. In general it works not good together with the coditional enviroment.

@mg262
Thanks for the filter modification and your helpful pm. I´ll draw on your offer if its time, thanks.

[foxyshadis]

Thanks, I hadn't actually tested! If it accesses the frames slightly out of order then it will cause a few problems. I'll test it out and report back.

All the lumadifference calls just estimate the diff. between a frame and flat grey (which is theoretical perfect blending pattern match), for the 4 frames on either side of it, as well as an extra check for those 4 compared to the fifth. framezx/diffzx are frames before the current, and framex/diffx are after. (Sorry I couldn't come up with better names!)

[FredThompson]

Where are the samples?

[mg262]

Post 17: http://forum.doom9.org/showthread.php?p=727409#post727409

[FredThompson]

Ah, I was thinking before/after but these are a nice addition to my bag o' test samples.

[foxyshadis]

Momonster: Thanks for alerting me; 4, 5, and 6 work just fine with fdecimate while 2 and 3 failed pretty badly. All default settings. I think that partly has to do with short samples, but I made a recommendation to write a temp file anyway. (I have yet to test tdec.)

Fred: Samples are up now, apologies for being lazy. ;)

[FredThompson]

Thanks for the samples, very impressive.

[MOmonster]

I can´t load your function. This is the problem:
# unblend output and various post-processing measures
unblend1 = Average(sframe3, -1.0, sframe4, 2.0)
unblend2 = Average(sframe5, 2.0, sframe6, -1.0)
edgemerge = fx_unblend_edges(unblend1,unblend2)
Avisynth doesn´t know the function fx_unblend_edges, of course there isn´t a function like this. Can you also give this function?
They all only give the difference to a grey flat, but than you don´t have to use 10 times LumaDifference every frame. Only one time, that´s the advantage of frameevaluate, also if this mean that the first 10 frames aren´t treated correct. In general the first frames are black. The other thing is that PlaneMinMaxDifference is similar, not far so accurat, but similar and it seems to work. I just say that because it is faster and can be used outside the enviroment (just faster). Bart S give us this possibility.

@mg262
The filter still doesn´t work on my Pc (only these two, all your other filters work). I don´t know why, but it isn´t that important, I´ll test it on another Pc in two weeks or so.

EDIT:
@foxyshadis
The deblended xvid files, are they made from the blended xvid files or from the huffyuv files you linked earlier. It´s important for the quality comparison.

[foxyshadis]

You're kidding, did I really forget to post that? Eek. I totally realized I needed to earlier , I wonder how it slipped my mind. Posted in the main post too, it's just a little helper function.


function fx_unblend_edges(clip sup1, clip sup2) {
edgemask1 = sup1.edgemask(thY1=2,thy2=2,thC1=4,thC2=4,type="special").levels(96,1,160,0,255,coring=false)
edgemask2 = sup2.edgemask(thY1=2,thy2=2,thC1=4,thC2=4,type="special").levels(96,1,160,0,255,coring=false)
diffedgemask1 = overlay(edgemask1, edgemask2, mode="subtract").inflate().blur(.7)
diffedgemask2 = overlay(edgemask2, edgemask1, mode="subtract").inflate().blur(.7)
edgemerge1 = MaskedMerge(sup1,sup2,diffedgemask1)
edgemerge2 = MaskedMerge(sup2,sup1,diffedgemask2)
return interleave(edgemerge1,edgemerge2)
}


They're made from the huffyuvs. Any filtering, compression, or changes to the source will weaken the detection (and the reversal), except crop-type and levels (since it's a linear transform). I hate to think what mpeg->xvid would do to it. (Interesting stress test possibility though.)

I used a lot of lumadiffs so I could skip around without having to wait for new values to reload, and to leave an extra step out of development, but there's probably an intelligent way I didn't think of to shift around or dump and refresh all data on random access. (If I had done it one per loop, it's almost guaranteed the chained fdecimate results, for one, would look a lot worse.) If I could find that way there'd be no need for temp files or cautious random access. I'll look into planeminmaxdiff too.

I had a think about this while I was napping, and the best solution would be to use current_frame and a last_frame. Then you do something like:
frame5 = (cf - lf > 0 && cf - lf < 10) ? select(cf - lf,frame5,frame4,frame3....) : lumadifference(...)
You would change the bounds as necessary for each frame ([0,-10] and [10,0]), so that any and only frames that are required are calculated, enabling random access while being as efficient as the standard "shuffle" that most conditional chains have.

On the other hand, it's a mess, isn't it?

(For instances like this I've always wished programming languages would have a ( 5 < cf - lf < 15 ) condition syntax which would be cleaner than (cf - lf > 5 && cf - lf < 15) and avoid explicit temp vars for functions with side effects. If wishes were fishes...)

[mg262]

foxyshadis,

It would have to be a function, but it would be easy enough to write something that took three numbers (or any number of numbers) and told you whether or not they were arranged in increasing order. That's either for built in AVISynth types or the things I gave you to play with. Do you want one?

MOmonster,

I'm really sorry the filter isn't working -- it's definitely something I want to look at. (Though I'm a bit buried in Motion at the moment.) I think that it might be that it makes use of SSE2... . In any case, if you can give me a minimal crashing script I will try and sort it out (but let's not hijack this thread).

[foxyshadis]

That would be kind of neat as a curiosity, but I mostly run into it in other languages. Thanks though!

By the way, have you decided what you are going to call those clips? I call them numeric, but I'm sure there's a pithier name.

[MOmonster]

@foxyshadis
I tested your script and yes it gives a better output than my FixBlendIVTC function. But because I´m ambitious, I build a high quality version of my function. FixBlendIVTC_HQ detects some more blends and has a counter variable to avoid wrong blend detections, but much more important for the most reversed Blends the quality is far better than with my last version and also better than with your function. So, it´s worth a try. I put the both scripts (little tweaked FixBlendIVTC + HQ version) in the attachment. You need Average and Masktools to run it (for the simple version only Average). I recommed to use it that way.FixBlendIVTC_HQ()
decimate(cycle=5, quality=0)
The decimate filter from the decomb package is fast, tested a lot and does absolutly what it should do, so there is no reason to use another decimater or another qualitymode.

Edit:
Sorry, I gave you an older tryout and not the last version. I changed the important things I remember and uploaded the updated one (the last will come tomorrow). With better quality I don´t mean that all pictures looks better. My function has no artifact protection, so especially on the first and your last clip the output of your function looks better. But the blend reversing itself seems to be better (especially on other higher quality samples, I created for tests). Just have a look on your second sample. The crows show good what I mean. For your special clip it would be nice to combine your artifact protection with my reversing.
Maybe if I have some more time :D .

Edit2: Attachments removed to a later post.

[MOmonster]

Ok, I uploaded the last version. After a closer look to your function I recognize that the HQ version works really similar. If you take a look at the luma channel my function gives a nice output, but the simple chroma reversing is worse than your output. It shouldn´t be hard to change this :) .

[foxyshadis]

I downloaded them and I'll test them out, but it'll probably be a few days before I really can. Thanks though! More fullproof blenddetection is more important anyway, and it's really the harder problem, any improvements in that area are welcome.

[MOmonster]

I allready find a more accurate blenddetection. It´s slower, but works really nice. I won´t have time to complete the function this weekend, but the next week I think I´ll upload it. So, don´t waste your time with testing these versions :) .

[foxyshadis]

Will do. Aha, you've been bitten by the bug, haven't you? ;)

[MOmonster]

I couldnt test it with longer clips and recognized a small problem with the counter, but nothing much important. Is this the bug you mean or is there another I don´t know about. :confused:

[foxyshadis]

Sorry, the phrase must not have translated, I meant you've become a bit addicted to finding the perfect solution too. I hadn't actually tested yet. ^^;

[MOmonster]

Ok, yes I know what you mean. Sorry for my bad English.
I finished the blenddetection and it does what it should do. The big problem left is the chromamasking. It seems that MaskedMerge just take the chroma of the first clip and doesn´t mask it. Also with chroma linking of the mask (FitY2UV()) I have no luck. If I change U2Y and create a second Masked clip and use this chroma channel, it looks better. But this is not really effective and gave me only one nice chromaplane. Have you any ideas how to link the chroma to the luma? Make I something wrong with MaskedMerge?