Analysis Filter: BlendWeight

[mg262]

Here's a filter I wrote while trying to analyse something... it's not polished, not optimised and probably of fairly marginal use but just in case, here goes:
_______________________________________


BlendWeight (http://people.pwf.cam.ac.uk/mg262/BlendWeight.dll)

Brief description:
Takes a frame which is a weighted average (i.e. blend) of previous and next frames, and calculates the weights used.

Full description:
BlendWeight(clip, int) - takes a YV12 clip and a frame number n, and finds alpha such that the difference of [n] and

(1-alpha) * [n-1] + alpha * [n+1]

is minimised in the 2-norm. (Here [k] is frame k, treated as a vector of pixels).

(The size of the difference in the 2-norm is the sum of the squares of the differences for each pixel -- I think this may be the same as the SAD = square average difference? Anyway, it's a reasonably sensible measure.)

Only the Y plane is used; chroma information is ignored.

Alpha is written to weight.txt and a descriptive line is appended to BlendWeight log.txt.

The output is identical to the luma of the original filter except for frame n; the output for frame n is re-blended from prev. and next frames using the alpha computed. (Chroma is deleted.)


Example:
_______________________________________
Loadplugin("D:\cp\BlendWeight\Release\BlendWeight.dll")

avisource("fr3.avi")
#ditch the bottom field
separatefields()
selecteven( )

converttoyv12()
greyscale()

o=last

BlendWeight(4310)
BlendWeight(4312)
BlendWeight(4314)
BlendWeight(4316)
BlendWeight(4318)

BlendWeight(4335)
BlendWeight(4337)
BlendWeight(4339)

subtract(o,last)
stackvertical(o,last)
_______________________________________

This gives the following log file:

Frame 4310 = 0.2033 x prev. Frame + 0.7967 x next Frame
Frame 4312 = 0.3909 x prev. Frame + 0.6091 x next Frame
Frame 4314 = 0.6176 x prev. Frame + 0.3824 x next Frame
Frame 4316 = 0.7962 x prev. Frame + 0.2038 x next Frame
Frame 4318 = 1.0149 x prev. Frame + -0.0149 x next Frame
Frame 4335 = 0.2140 x prev. Frame + 0.7860 x next Frame
Frame 4337 = 0.3912 x prev. Frame + 0.6088 x next Frame
Frame 4339 = 0.6028 x prev. Frame + 0.3972 x next Frame


Notes:

I wrote this while trying to analyse a DVD with blended fields. The results in the case I'm looking at are thrown off slightly by a slight horizontal wobble; I might write a version to compensate for this. But the filter seems to be accurate enough to bring out the pattern in the case above (i.e. very close to multiples of 0.2). If anyone else has seen this pattern anywhere, I would be grateful if you would tell me!

[scharfis_brain]

IMO those weights occur on PAL -> NTSC conversions.

[mg262]

>IMO those weights occur on PAL -> NTSC conversions.

It's a PAL source. Do you mean NTSC->PAL as well? I wrote this as a prerequisite to a plugin to reverse 24->25fps conversion, making the assumption that the resampling was regular - i.e. that it looked like this:

http://people.pwf.cam.ac.uk/mg262/fpsconvert.jpg

(example for 4fps -> 5 fps; so

new 0 = old 0
new 1 = 1/4 old 0 + 3/4 old 1
new 2 = 2/4 old 1 + 2/4 old 2
new 3 = 3/4 old 2 + 1/4 old 3
new 4 = old 3 )

But when I tried to start the reverse conversion it became obvious that something more complex was happening. (BTW I was considering top field only.)

I trimmed the example above to make it more readable, but here's a wider pattern - as you can see, the weights seem to step up by 0.2. each time:

Frame 4096 = 0.2024 x prev. Frame + 0.7976 x next Frame
Frame 4098 = 0.3927 x prev. Frame + 0.6073 x next Frame
Frame 4100 = 0.6036 x prev. Frame + 0.3964 x next Frame

Frame 4237 = 0.2158 x prev. Frame + 0.7842 x next Frame
Frame 4239 = 0.4203 x prev. Frame + 0.5797 x next Frame
Frame 4241 = 0.6024 x prev. Frame + 0.3976 x next Frame
Frame 4243 = 0.7997 x prev. Frame + 0.2003 x next Frame
Frame 4245 = 1.0060 x prev. Frame + -0.0060 x next Frame

Frame 4310 = 0.2033 x prev. Frame + 0.7967 x next Frame
Frame 4312 = 0.3909 x prev. Frame + 0.6091 x next Frame
Frame 4314 = 0.6176 x prev. Frame + 0.3824 x next Frame
Frame 4316 = 0.7962 x prev. Frame + 0.2038 x next Frame
Frame 4318 = 1.0149 x prev. Frame + -0.0149 x next Frame

Frame 4335 = 0.2140 x prev. Frame + 0.7860 x next Frame
Frame 4337 = 0.3912 x prev. Frame + 0.6088 x next Frame
Frame 4339 = 0.6028 x prev. Frame + 0.3972 x next Frame

Frame 4344 = 0.6090 x prev. Frame + 0.3910 x next Frame
Frame 4346 = 0.8051 x prev. Frame + 0.1949 x next Frame
Frame 4348 = 0.9963 x prev. Frame + 0.0037 x next Frame

Those frames are selected by hand to be ones where the blend is visible and the camera is stationary. Animation seems to change every other 'underlying' frame (field? anyway, I'm only looking at top field), whereas the camera moves every frame.

Most runs of blended frames like these are followed by sequences which have no blended frame, but where each frame is duplicated, which is consistent with animation changing every other frame.

There doesn't seem to be any periodicity in the occurrence of the runs of blended frames.

Anyway, thx for the thoughts...


Edit: I think sometimes animation stays for 3 underlying frames, causing quick changes like:

Frame 4385 = 0.4200 x prev. Frame + 0.5800 x next Frame
Frame 4387 = 0.6087 x prev. Frame + 0.3913 x next Frame
Frame 4389 = 0.8085 x prev. Frame + 0.1915 x next Frame

Frame 4392 = 0.5987 x prev. Frame + 0.4013 x next Frame

[scharfis_brain]

Some time ago, I wrote a article about weird types of interlacing
(basically, how weird FILM to PAL conversions in real life are).

including the FILM->NTSC->PAL conversion:
http://home.arcor.de/scharfis_brain/ExotischesInterlacing/#2.3

but it is in german, and I do NOT intend to translate it.
(also it will need a major re-write, but I do have some lacks in time and ambition to do so)

majbe the sketches on that page will help you to understand the NTSC->PAL conversions

btw.: you need to analyse ALL fields. not only the even or odd ones alone.
restore24 is able to restore fieldblended video.
It uses a special edge-dection-mask for identifying the blends.
Your approach of weight-detection is similar to the approach of removeblend.dll
it is even capable to resote 29.97fps progressive content, that has been blended into PAL.
but 59.94fps into PAL are - who wonders - not recoverable.

In real life, there is NO constant pattern to work with.
every scene cut will adjust the pattern offset. THis is why deblending video is THAT hard to realize.

But I am VERY interested into your ideas.

[mg262]

Thank you for the link. (Babel Fish copes with it nicely.) There is absolutely *no* Way I would have figured out that some of those weird patterns happen without your explanations.

I've now analysed some of the bottom field too and the whole thing looks a bit like your pattern 2.2.2, except with four blended frames on each field rather than six. The basic pattern does repeat every 50 fields, but gets interrupted at weird places - some that are not by any stretch of the imagination scene cuts.


Since you're interested, original idea was like this (sticking with the 4->5 fps example throughout):

1. I assumed conversion was as in the previous post, but not that everything lined up; this picture shows what I mean:

http://people.pwf.cam.ac.uk/mg262/fpsconvert2.jpg

So I was going to use the BlendWeight filter to figure out how shifted the bottom part was relative to the top part. With a little work it is still possible to break the situation into blocks of 4 pixels each of which is converted to 5 pixels, so it still basically looks like the situation described above.

2. The conversion described above could be written as

y=Ax,

Where

y is the vector of new pixels (size 5), x is the vector of old pixels (size 4), and A is the 5x4 matrix:

http://people.pwf.cam.ac.uk/mg262/fpsmatrix.jpg

3. Now to reverse this, we can again use least squares regression. We have A and y and want to find x minimising the size of (y-Ax) in the 2-norm. A bit of algebra gives:

x = (A'A)^{-1}A'y where A' is the transpose of A

4. A particular coefficient of x corresponds to one frame in the output; so this could have been implemented nicely in AviSynth as follows:

For a given frame,
- find the appropriate blocks,
- compute matrix A (which doesn't have to be as nice as in the example above because of the shifts),
- compute (A'A)^{-1}A' (which is very quick compared to processing the pixels),
- take the row (vector) corresponding to the frame in question
- for each pixel, take the appropriate vector of corresponding pixels in prev/following frames, and multiply (dot product) with the row of (A'A)^{-1}A'.

So e.g. you might find that for a particular new frame each pixel ends up as
(-.3) * frame 5 + .8 * frame 6 + .4 * frame 7 + (-.2) * frame 8
[totally made up example with silly coefficients].


This method would work fine on your case 2.2.2, and I think on all your other examples (though I haven't checked carefully) were it not for the scene cuts. It might still be possible to make it work but it depends on knowing/figuring out more about the scene cuts.

As I said, in my case, they don't always seem to be scene cuts. If you have any idea when/where/how they are placed I would really like to know! In the meantime, I think I'm just going to have to analyse the thing (perhaps automating part of the process) until I get a better idea of where they are placed.


What else... restore24 I found out about, but only after I had this idea, so I decided to implement it anyway. But I'm going to look at restore24 whether or not this works -- anything by Didée is not to be missed!

[Mug Funky]

it's not just scenechanges - there's also the possibility of further editing after conversion. for example the region-4 '80s Astroboy boxset (PAL)*. footage was converted to PAL from a japanese master, then re-edited to match up with the american version (which had been heavily edited in parts). a lot of the time these edits happened on scenechanges, but sometimes it's right in the middle of a scene. worse still, premiere pro has b-frame issues and would occasionally shift frames around at scenecuts (previewing and editing would be fine, but on export the frames are in the wrong order. if i ever meet adobe's programming team, there'll be chunks to scrape off the wall).

i'm not sure how much material like that is out there, but i'd say there's quite a bit.

* actually, it looks much, much better than the american version which uses abominably bad film as a source, so if you want good astroboy, either get the r4 one or the japanese "tetsuwan atomu".

[edit]

if you need samples of field-blends, i've got shelves of the stuff :) i hope you like anime.

[kassandro]

Originally posted by mg262

(The size of the difference in the 2-norm is the sum of the squares of the differences for each pixel -- I think this may be the same as the SAD = square average difference? Anyway, it's a reasonably sensible measure.)

SAD is the 1-norm (sum of the absolute values of the differences) and it has the advantage that Intel has hard wired it into SSE instruction unit of their cpus (the psadbw instruction). However, in your problem the 2-norm is the right norm (with the 1-norm it couldn't be solved in reasonable time). In fact, your problem is a very special and very simple least square problem and already Gauss (the guy on the 10 DM bill), who was the first to study such problems, observed that the square norm (= 2-norm) is the way to go.
Intel should have also hard wired the square norm into the cpu, because it is also the natural norm for DCT and the Fourier transform in general and if the search for motion vectors would be done with the square norm instead of SAD the motion vectors would be of considerably higher quality, because the 1-norm doesn't bode well for the Fourier transform.

[mg262]

@Mug Funky

For the clip I'm working on (Jayce + Wheeled Warriors, French DVD 1st ep), I also happen to have a US DVD version, and the ratio works out to be about 25:29.974, so I'm hoping there's no manual cuts. Though from what you say, it's always possible that both are separately edited - but less likely here than with anime.

OTOH, if I rule out manual cuts, it's really hard to explain cases like this:

http://people.pwf.cam.ac.uk/mg262/posts/break.jpg

(Top field in top half; "0.6" means 0.6 * prev top field + 0.4 * next top field.) The normal pattern is like this:

http://people.pwf.cam.ac.uk/mg262/posts/pattern.jpg


I could try and restore 24fps from the NTSC version and use a least squares type approach to figure out how the PAL version derives from it, but there are the obvious difficulties like vertical resolution, colour matching... plus the fact that the US source seems to have a changing telecide pattern.

>if you need samples of field-blends, i've got shelves of the stuff
Thank you - I may well take you up on that... if I ever get something that works on this 'easy' case!

@kassandro

I haven't messed with ASM since pre-MMX days, although I might have to if I want to use this filter to analyse each frame. A quick search suggests that MMX PMADDWD and/or SSE2 PMULUDQ might be useful in computing the 2-norm -- although it will still be much slower than the 1-norm. Just found an elementary link on MMX dot product...

http://www.tommesani.com/MMXExamples.html#VectorDotProduct


Edit:

FWIW, It's just occurred to me that the least squares approach can also be used to reverse vertical resizing, e.g. reversing 720 x 240 -> 720 x 288; it works just as above, with

y = Ax

x = column of pixels in original source (240)
y = column of pixels in converted source (288)
A = 288 x 240 matrix used in resizing.

Computing ((A'A)^-1)A' is slower now, but once it's been done it can be used on every frame (field). Actually, the problem probably breaks nicely into lots of fast 6x5 matrices, but I have to think that through... .

[Mug Funky]

excellent. i'm watching this thread eagerly now. field-blends are the enemy of good compression. though since i got a DVD burner it's not been such a problem (re-encoding to reasonably high bitrates isn't a big problem with field-blends. QuEnc is teh shizzy for interlace).

de-blending has implications for a lot of other stuff too, though. un-converting a converted source can be quite beneficial in production situations as well.

oh, and in case i gave the wrong impression above - not all anime is messed up like the example i gave. it's the exception, not the rule. just something to keep in mind that pattern changes can and probably will happen. telecide-like overrides could probably help this - hardcore encoder people will have no qualms about manually stepping through their sources to get a good encode out of them.

[edit]

this situation may or may not be reversible - interlaced material that's been resized as progressive. these sources are rare and usually total crap, but i wonder if it's possible to pull the original fields out of it? if the material has been lanczos-downsized, i'd say no, because all frequencies above fs are filtered out by it (hence you don't actually see the familiar wavy pattern with lanczos pal to ntsc), but other resizers may retain enough info on the original. maybe i'm dreaming.

[kassandro]

Mug Funky,
the method of least squares is a method to solve a system of overdtermined linear equation in some sense. In the case of BlendWeight you have only one unknown, namely alpha, and you have 720x576 equations for full D1 clip. Thus in the case of BlendWeight you have an extremeley overdetermined system of linear equations. In the case of reversing an upsized clip you have still an overdetermined linear system but now with many unknown variables. Thus the method of least squares can be applied to reverse upsizing, i.e. you can downsize with it. I haven't really though about it, but probably the least square method results in the usual bilinear downsizing. Now, if you want to reverse downsizing, you unfortunately get an underdetermined linear system and the method of least squares cannot be applied. Thus the method of least squares can only be used for downsizing (and probably doesn't give anything new) but not for upsizing.


@mg262:
you are right, SSE can be nicely applied to BlendWeight. Actually you can compute the square norm and the inner product with another vector simultaneously. In this way with SSE 4 pixels can be processed simultaneously and with SSE2 even 8. Thus BlendWeight can be made pretty fast.

[scharfis_brain]

@mg262:
OTOH, if I rule out manual cuts, it's really hard to explain cases like this:

not hard to explain:
the underlayed video is not animated at 24 fps.
it is animated at much lower fps, like 12, 8 or 6 fps.
this means that blends can only occur if there is a movement change between two fields.

[mg262]

@Mug Funky

>un-converting a converted source can be quite beneficial in production situations as well.

In the case I'm working on, I'm hoping it will remove a fair amount of slow horizontal jitter (maybe better described as wobble) which looks like it was introduced during/after framerate conversion. (It will need a slight tweak to BlendWeight to do this.) I don't know how common jitter is though.

@kassandro

Bilinear downsizing and reupsizing would blur slightly at each stage, whereas reupsizing with least squares should hopefully remove the blur from downsizing. I think. (Even if I'm right, the effect might be too small to be worth bothering with.)

I'm not sure how useful the method would be in practice -- it would require you to know the exact original size (to the pixel) and the exact resizing mechanism. The only use I can think of is to reverse NTSC->PAL, and even then it could be messed up by crop/letterboxing. It was a random thought which might not be worth much!

@scharfis_brain
The animation is at 12fps (although the camera moves at 24fps) - i.e. each 'picture' is filmed twice. So during still scenes, the effect you're talking about means a blend is only visible every other frame, which gives the gaps between numbers in the normal pattern:

http://people.pwf.cam.ac.uk/mg262/posts/pattern.jpg

But I don't understand how that explains the picture-pattern I posted above, which had something like this:

TF: 0.2 --- 0.4 --- 0.6 --- --- 0.4 --- 0.6 --- 0.8
BF - no blurs

Maybe what happening is that occasional pictures are filmed once or three times rather than twice? I originally didn't think that was likely because I thought it will bring out the 'missing' numbers in the normal pattern, which I guessed were probably 0.1, 0.3, 0.5, 0.7, 0.9 (which don't turn up). They could be something else though.

I've been doing some more analysis; I modified the filter to scan the whole film and produce output like this:

2: 1002.5: (0.1988) 1.92
4: 1004.5: (0.3971) 2.29
6: 1006.5: (0.6109) 0.72
8: 1008.5: (0.8111) 0.66
2: 1015.0: (0.1964) 0.54
4: 1017.0: (0.3987) 1.01
6: 1019.0: (0.6049) 2.19
8: 1021.0: (0.8041) 2.36
4: 1027.5: (0.4001) 3.07
6: 1029.5: (0.6126) 2.79
8: 1031.5: (0.8060) 1.08

(2: is just for convenience; then frame number; then exact blend parameter, then closeness of match.)

- the 2-4-6-8 runs show up pretty clearly, so I have been able to break them up by hand and measure the periods between them. This kind of thing appears (measured in fields):

25, 25, 25, 25, 21, 25, 50, 25, 25, 25, 41, 25, 25, 25, 46, 25, 25, 60, 25, 25, 21, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 40, 25, 25, 25, 25, 25, 25, 31, 25, 25, 50, 15, 25, 21, 25, 25, 25, 10, 25, 25, 25, 25, 25, 25, 25, 21, 35, 50,

So the 'normal' pattern does seem to be the one above, with 25 fields between runs of blended fields. I don't know what causes the anomalies (if not cuts), although one possibility is that occasionally the picture is filmed one or three times rather than twice? I think this would involve assuming that the in between numbers are really 0.2, 0.4, etc. (rather than 0.1, 0.3, ...) so the full blending pattern looked like this:

http://people.pwf.cam.ac.uk/mg262/posts/pattern2.jpg

I'm currently analysing these frames, which may help clear things up...

http://people.pwf.cam.ac.uk/mg262/posts/loss.jpg

(it's a little hard to see at this scale, but the blurred 'disc/saw' in the third top field is distinct from the one in the second.)

[sh0dan]

I haven't had any material to properly test the filter on, but I thought it would be a good idea to make a conditional filter out of this.

That way it could return blend weights on the fly, so a filter could make intelligent decisions based on this information (duplicate frames, etc). It should be rather simple to code, but since I don't have the source it's hard to help. Basicly you should implement two functions. One returning the blendweight to the previous and one to the next frame.

The AviSynth code has examples (mentioned in another thread), and decomb has the IsCombed() function.

I don't know how useful it would be in scripts, but it's just an idea.

[AVIL]

Hi all,

¿Anybody knows how to use the last version of this plugin?. Dll is dated 18/11/2006. It don't accepts parameters and the output is a clip with a frame of 256x32.

Thanks in advance.

[Fizick]

try read a source.
Wilbert published some mg265's source here

[Wilbert]

http://forum.doom9.org/showthread.php?t=118430

[AVIL]

Hi,

From blendweight.cpp :

public:

BlendWeight(PClip _child, int startFrame, int endFrame, IScriptEnvironment* env);


....

double weight = num/denom;

......

return weight;

However the script :

AviSource("p1_40000_49999.avi").blendweight(100,200)
returns error:

Invalid arguments to function "blendweight"

If I try the script:

AviSource("p1_40000_49999.avi").blendweight()

It returns a fake clip of 256x32

And the script:

s=AviSource("p1_40000_49999.avi")


scriptclip(s,"subtitle(string(blendweight()))")

returns clip untouched, so blendweight don't returns any value that string can process.

IMHO the binary not belongs to source.

Regards.

[FlimsyFeet]

Simple question: if this filter is telling you that a particular frame is made up of 0.6 x prev frame ('A') + 0.4 x next frame ('B'), how would you restore it to a non blended frame?

I.e would you invert frame B and merge it onto your blended frame, or use overlay in 'subtract' mode, or some other method?

[wonkey_monkey]

I've been considering this exact problem recently.

Flimsy, if a particular frame is a blend of a previous frame and the next frame, wouldn't you just replace it with one of those?

You'd only need some other method such as you suggest if you only had one "good" frame and one blended frame, and you wanted to restore the missing good frame. As it happens I've had to do exactly that for chroma only in my project, so I wrote a quick filter which did (A-B/2)*2 where A's frame has a blend of "correct" chroma and B chroma, and B is a clean frame. This was good enough, although I wonder if gamma should ever be taken account when dealing with luma blends? I also tried to write a filter like BlendWeight, but never got good answers from it.

If I was doing a frame de-blending strictly in plain AviSynth, I would probably do a mix of levels adjustment and subtract overlay, which seems to work more-or-less correctly in Photoshop, at least.

David

[FlimsyFeet]

Flimsy, if a particular frame is a blend of a previous frame and the next frame, wouldn't you just replace it with one of those?
Well, normally yes. But in this case I'm working with a bobbed clip which I'm hoping to weave back to get a full resolution frame.

So field 1 is from frame A, field 2 is a blend of frame A+B, field 3 is from frame B. I want to unblend field 2 so I can restore frame A.

The pattern is not actually that simple and becomes a nightmare later on, but I'll save that for a new thread.

[Didée]

That sounds remotely like fieldblended normconversion. And no, it's impossible to achieve what you want to. In that case, you have clean top-fields of A and B, and the blended one is a blend of the bottom-fields of A and B. See the problem? No clean bottom-fields to calculate the unblending. The difference that you want to restore, i.e. the difference between top and bottom, is the difference that makes it impossible to sucessfully perform the un-blending.

That's the reason why Srestore (or Restore24) never even tried to use active un-blending, but ever only resorted to simple, passive substituting with a neighbor.

[pbristow]

What Didée says is correct, but don't give up all hope: There are some tricks you can try get *closer* to the original bottom fields, although you'll never get a completely clean version.

Let's call our original sequence of uncorrupted frames F1, F2, F3, etc.
The original top fields are T1, T2, T3... ; The original bottom fields are B1, B2, B3...

We'll call the munged bottom fields M1, M2, M3, where M1 = B1|B2, M2 = B2|B3, etc.
(I'm using a vertical bar, "|" to indicate "blended with")

So the video data we're trying to unscramble, at a given point, runs as follows:

T1, M1, T2, M2, T3, M3...
which equates to:
T1, B1|B2, T2, B2|B3, T3, B3|B4 ...

For the moment, let's simplify and assume there was no rate conversion, and the all the blend weights are equal (we can adapt what follows to deal with the changing blend weights later).
While we can't recover, say, B2 cleanly from this, what we can do is create a new and better approximation to B2, by changing the proportion of what else is mixed with B2. For a start, try blending all the M fields with their neighbours. Remembering that I said all blends weights are equal, we'd now have:

X1 = M0 | M1 = ( B0 | B1 ) | ( B1 | B2 )
X2 = M1 | M2 = ( B1 | B2 ) | ( B2 | B3 )
...etc.

So our new field X2, say, instead of having half of one filed and half of the following now has a *quarter* of the following field, and a quarter of the preceding one, mixed with half of the desired field. The desired image is much more clearly visible, although it is now accompanied by two weak ghost images instead of a single strong one.

What can we do to reduce the strength of those ghost images even further? Well, we do some more blending, but now using a negative rather than positive blend weight. Let's use "+" to indicate a postivie blend, and "-" for a negative one. Our new strem of fields is going to be created like this:

Y2 = X2 - 0.5 * ( X1 + X3 )
= M1 + M2 - 0.5 * ( ( M0 + M1 ) + ( M2 + M3 ) )
= ( B1 + B2 ) + ( B2 + B3 ) - 0.5 * ( (B0 + B1) + (B1 + B2) + (B2 + B3) + (B3 + B4) )
...and with various things reinforcing each other or cancelling each other out, at least to some degree, you end up with:
= 0.5* B1 + B2 + 0.5* B3 - 0.5 * ( (0.5*B0) + B1 + B2 + B3 + (0.5*B4) )
= -0.25*B0 + 0.5*B2 -0.25*B4

So now instead of positive ghosts of B1 and B3, we have negative ghosts of B0 and B4. Great. How is *that* any better?

Well, we can cancel out the ghosts of B0 and B4, but then we'd have positive ghosts of B(-1) and B5, and so on. BUT, if instead of using the "correct" weighting of 0.5 each time we cancel out a pair of ghosts, we use a slightly lower weighting, then eventually we get to a state where there very many *very faint* ghosts, all low enough to be virtually noise. If we slot our now re-constituted lower fields in between the original, clean top fields, and re-weave them into frames, we can apply spatial noise filtering to clear away nearly all the residue of these ghost images (especially if we custom design the denoiser to favour data that matches the corresponding top field over anything that looks suspiciously similar to one of the neighbouring fields).

Complex? Yes. But I ran a proof of concept once a long time ago, and the end result was definitely much better than the starting point. (In that case, I didn't even have the benefit of any clean fields! It was a clip that somhow had every frame blended 50/50 with it's neighbour. Probably it had been a progressive source that got treated as interlaced with the wrong field cadence, then resized downwards, so that the mis-matched fields became firmly mixed together. I was able to almost completely eliminate ghost images on a fairly long clip, albeit *VERY SLOWLY*, and with a lot of banding and noise creeping in from the rounding errors in all those repeated blend operations. It would be much better to do the whole thing as a plug-in in C, using 16 bit integers, rather than crudely hacking something together in an avisynth script using only 8 bit clip data.

...and that's before we even get to the problem you've got, of the blend weights varying from one frame to the next! Your clean-up tool would need to do a lot of algebra on-the-fly using the detected blend weights in the source clip to calculate the required "re-blend" weights. :\
However, as a first approximation I would recommend trying the first step, converting 2-in-one blends into 3-in-one blends (i.e. replacing one strong ghost image with two weaker ones), using your detected blend data to decide the correct mix on each field/frame, and see if that looks better when woven back with the top fields than what you started with. You should see that each frame more firmly establishes a particular moment of time, and then normal de-combing techniques might be all you need to get a decent result.

Have fun! :)

[jmac698]

@pbristow
Good job, I have exactly the same problem of 50/50 frames, with no clean frames at all, except in low motion areas. I've followed the same reasoning of combining frames. You got one step further in the many ghosts idea and I'm encouraged that it works. I explored another approach. Forget about exact solutions to linear equations, there's constraint based solutions. And in advanced matrix decomposition research, there was a demo of a projector on a wall with some pictures, and it blindly separated the wall from the film.
I think either of these methods combined with motion tracking could work well. I mean watch the edges and when they split apart you can tell which edge belongs to which frame.
I worked out a program that could perfectly reconstruct 50/50 blends but it depended on some known spatial areas, such as black or white, where the pixels of both frames could be perfectly known, and worked it's way outwards spatially. (I'd have to look it up again).

[pbristow]

As luck would have it, I just found the online copy of my sample still from that test:
http://ic.pics.livejournal.com/pbristow/777175/2282/original.jpg

Top left is the original blended frame, bottom left is the result after 2 stages of "de-blending", bottom right is the difference between the two; The other two frames shown (top middle and top right) are the intermediate blends. I chose this sample frame because the telegraph poles in centre frame make the difference so clear, but not also there are still strange "halo" artefacts around the faces at either side. When actually playing the video, though, these were barely noticable, and the overall effect was much easier to watch than the original video.

You can also see the effects of rounding and arithmetic overflow in the strange "painted" look of the sky. (N.B. trying to subtract darkness from nearly full white to get something even brighter doesn't work, if you've got no way of representing "more than full white" / "less than total black" in your intermediate video data! You can work around this by initially reducing the brightness scale of all video to, say, 64-191, but that loses you a full bit of precision. Much better to work in signed 16 bit integers, and start by scaling the video levels *up* to use 12 or more bits (i.e. multiply all the values by 4, 8 or 16. Dn't go any higher than that, though: You want to keep at least one full bit of headroom.)

Be aware that if the contrast or gamma have been tweaked after the original blending occured, then it will not be possible to get *exact* cancellations no matter how precise the arithmetic is (unless you can reverse those changes first), because bright regions would then effectively have different blend weights to dark ones.

[FlimsyFeet]

I wondered if anyone would have an answer to the question I asked back in post #18, forgetting the discussions that followed, whcih are fascinating but a little over my head.

Or is it not possible to reverse a blend using simple methods?

I am intending to eventually start a thread in the "usage" forum to discuss specific problems with my source.

[FlimsyFeet]

If I was doing a frame de-blending strictly in plain AviSynth, I would probably do a mix of levels adjustment and subtract overlay, which seems to work more-or-less correctly in Photoshop, at least.

Sorry, I missed this bit. Will try Subtract().

Edit: do you mean something like this, if blended_frame = 0.6 x prev_frame + 0.4 x next_frame

blended_frame=blended_frame.overlay(subtract(blended_frame, next_frame), opacity=0.4, mode="hardlight")

[StainlessS]

Here an entry on wiki that might help:
http://avisynth.org/mediawiki/Removing_blended_film_frames_on_vcds

Not exactly what you need but show how subtract and levels can be used.

EDIT: FYI, the linked type of blending is exactly what the Exblend filter was written for.

EDIT: Link update on Avisynth.nl Wiki:-
http://avisynth.nl/index.php/Removing_blended_film_frames_on_vcds

[staticmotion]

My first post, and it's resurrecting a 10 year old thread! Sorry, but after lots of searching this still seemed the most appropriate.

I too have a source that has 100% blended frames (50% weighting between the original frame before and current). Looking at the wiki example above (thanks StainlessS) and the concept from pbristow I knocked together this script which results in a marked improvement:
base=AVIsource[file]
selected=base.trim(1,0)
after=base.trim(2,0)
prior=base

blend1=subtract(selected,prior.Levels (0, 1, 255, 128, 255)).levels (0,1,127,0,255)
blend2=subtract(selected,after.Levels (0, 1, 255, 128, 255)).levels (0,1,127,0,255)

result=overlay(blend1,blend2, mode="blend", opacity=0.5)
return (result)

However, the possibility of further improvement bites at me... and although I get the concept of the further ghost reductions in pbristow's post above I'm afraid the exact coding for that is beyond my grasp...
Is anyone kind enough to take the time to help me out please? Would be really grateful, thanks!

[wonkey_monkey]

I didn't get good results with your script. It doesn't do anything to emphasise one frame over any other, so the original (lost) two frames still have the same weight in the output. The trick, I think, is to first blend it again so that one of the original frames has a greater weight than the other two - then you can start messing around.

A proper mathematical solution escapes me, but I got a reasonable result with:

function prev(a) {
return a.trim(0,-1)+a
}

function next(a) {
return a.trim(1,0)+a.reverse.trim(0,-1)
}

function sharpen1(clip a) { return expr(a.prev, a, a.next, "2 y * x z + 0.5 * -") }
function blur2(clip a) { return expr(a.prev.prev, a, a.next.next, "0.75 y * x z + 0.125 * +") }
function blur4(clip a) { return expr(a.prev.prev.prev.prev, a, a.next.next.next.next, "0.75 y * x z + 0.125 * +") }
function sharpen3(clip a) { return expr(a.prev.prev.prev, a, a.next.next.next, "1.25 y * x z + 0.125 * -") }

function unblend(clip a) {
a.convertbits(32)
expr(last, last.next, "x y + 0.5 * ") # a+b => a+2b+c
sharpen1
sharpen1
blur2
sharpen1
blur2
blur4
sharpen3
blur2
convertbits(8)
}

base = AVISource(file)
unblend(base)


It just depends how much residual ghosting you're willing to put up with.

[StainlessS]

Post #27 Link updated to current Avisynth.nl Wiki:-
Removing_blended_film_frames_on_vcds
http://avisynth.nl/index.php/Removing_blended_film_frames_on_vcds

[staticmotion]

Thanks wonkey_monkey for taking the time to look at this. I fear we may we talking about different types of source material though? The video I'm looking at has every frame blended, not a recurring pattern of "clean" and "dirty".

I've attached a clip of the source here: https://we.tl/t-uPEs9OyCMD

FYI: it's an early episode of Doctor Who. If you aren't familiar with how the frames became blended, outdoor sequences were recorded onto 16mm film at 25fps, then telecined into the 25i electronic studio recording (interior scenes). The BBC destroyed the video tapes after transmission, but not before film recording them back onto 16mm film for overseas sales. Unfortunately the vision mixing during the studio recording sessions didn't strictly cut on frames only, but on fields. So some of the telecine sequences were one field out on the original interlaced tapes. Not a problem on playback of those tapes, but when film recording the tapes onto the final 16mm film (with spot wobble on the monitor to hide interlace lines on the 25i studion interior sections) those "one field out" telecined sections became blended frames on the film.

I'd assumed 50% weighting across each two frames, but thinking about it - the "stored-field technique" of the 1960s film recording process was created by making the first field brighter on the phosphur monitor so it left an "afterprint glow". Maybe in the source (which is the progressive telecine of the 25fps 16mm negative for overseas) one of the original film frames is more prominent than the other?

[wonkey_monkey]

That isn't all 50/50 blends. It varies quite a bit. Where it is 50/50, the results can be:

https://i.imgur.com/EsA5wgK.png

[original, yours, mine]

And a simulated one:

https://i.imgur.com/iy5oz8D.png

[staticmotion]

Nice to see a bit more Who in the thread! (and laughed out loud at your EastEnders Directors Cut on YT)

Yeah, I see how your script can give better results on some frames. If it isn't a 50/50 blend each time then is it possible to calculate the weighting and adjust the unblend on a per frame basis? This whole thread started with a script to calculate the ratio of blends but unfortunately the link to that is now dead.

These out of phase film inserts are the last major problem to overcome in the DVD and now Bluray Doctor Who restorations... it would be incredible to come up with a method to really improve them!

[wonkey_monkey]

Might be something AI's finally capable of - albeit slowly - though I have zero idea where to start with that.

[staticmotion]

I've been thinking the same about AI...if a system could be trained on simulated blends...

[staticmotion]

Had a play around and came up with v2 of my script:

base=[file source]
selected=base.trim(1,0)
after=base.trim(2,0)
prior=base
blend1=subtract(selected,prior.Levels (0, 1, 255, 128, 255)).levels (0,1,127,0,255).convertbits(16)
blend2=subtract(selected,after.Levels (0, 1, 255, 128, 255)).levels (0,1,127,0,255).convertbits(16)
result1=overlayplus(blend1,blend2, mode="blend")
result2=overlayplus(blend1,blend2, mode="average")
result3=overlayplus(result1,result2, mode="blend")
result4=overlayplus(result1,result3, mode="darken")
return (result4)

An improvement on before... I've tried your script wonkey_monkey on a few different clips but find the repeated ghosts too distrcting in motion. But as shown above, fantastic on some frames.