Averaging *and improving* multiple captures

[Karyudo]

I have three copies (of the same pressing...) of a movie on LD. Each has its own various video flaws -- like rot and other dropouts -- and of course noise.

I'd like to try capping all three copies and layering them with AviSynth. Layering is good for NR (I've read all the threads I can on the topic) and should be even better in my case than some other people's experiments, because I'd be using caps from different discs. So I expect the noise should go down, even when using pretty good equipment and decent discs.

Ideally, I'd imagine you'd have a filter that averages three (or more) caps for noise, but eliminates rot and other problems. I have seen all sorts of workarounds for averaging more than one source -- but isn't it about time to have a true Average(source1, source2, source3,...,sourceX) function in AviSynth?? That would be cool...

However, I'm afraid that by averaging over three sources, I'll actually end up *summing* the total number of video flaws (even though each flaw will be minimized)! I'd get a picture that has less obvious flaws (that's good), but a copy of every flaw from all three original caps (that's bad).

I've never seen a discussion of this particular case. People have either been working from one copy of a VHS tape or some other physical media, or from a number of repeated vidcaps of broadcast sources -- which of course don't generally have many media-induced flaws such as rot or dropouts.

This is where I think this becomes an interesting problem for AviSynth gurus: I want (and think it's possible) to start with three rotting copies of something, and output a smooth, artifact-free image! To reduce rot artifacts, even working with two caps and the existing averaging routines just isn't the same, since from two frames, you still can't easily tell which frame is "right" and which one has a flaw. It's like two witnesses arguing in court: you can't be sure who's right until a third party steps in and sides with one or the other!

Although I know next to nothing about coding, I can imagine the strategy I'd take: I'll bet you could work from the assumption that for any given set of three identical frames, small variations in all three are noise (and should be averaged across all three, as in my "Average()" example above), but that any large variation in one of the three frames is a video artifact/defect, and should be ignored, averaging only the two frames closest to each other in the region where the third deviates. Maybe allow manual tuning of the deviation threshold with a user-entered value, otherwise use some default threshold parameter. That *should* give an optimally-averaged picture (reducing noise), while at the same time killing rot dead!

Shouldn't it?

Any ideas? Suggestions? Someone who can whip this up overnight?? :P

[E-Male]

so the idea is that filter takes 3 clips as input and outputs the average except when one of the 3 clips differs from the other 2 much more than the 2 from each other, in this case the more different clip is ignored and only the 2 more alike clips are averaged.
this is done for every pixel

right so far?


shouldn't be hard to code (as long as it hasn't to be optimized)

i might look at it when my audio projects are done


2 more ideas about it:
-maybe take the color of the drop-outs as parameter
-simpler version: for every pixel only average the 2 clips which are most similar

[Karyudo]

Originally posted by E-Male
right so far?

Yeah, I think so!

shouldn't be hard to code (as long as it hasn't to be optimized)

Considering it would be a rare case, and it would already take extra time to make three caps instead of one, I don't think I (or any other user) would be too worried about speed optimizations...

i might look at it when my audio projects are done

Mighty kind of you!

-maybe take the color of the drop-outs as parameter

I don't get what this means. Like you'd tell it the drop-out is white? Or what?

-simpler version: for every pixel only average the 2 clips which are most similar

I wonder: does this actually do a better job, too, as well as eliminate drop-outs? I'd imagine two sources being close as being the two sources that are most accurate at that pixel position. You'd be treating the third source as an outlier, and discarding it. Maybe that would give even better results!

[E-Male]

Originally posted by Karyudo
I don't get what this means. Like you'd tell it the drop-out is white? Or what?
laser-rot drop-out are white, while film-dropouts are black, and of course there are other more or less similar video-errors
so if you're dealing with LDs you set it to white, and when a pixel is white in only one of the captures, it's likely a drop out and could be eliminated independendly from thresholds


I wonder: does this actually do a better job, too, as well as eliminate drop-outs? I'd imagine two sources being close as being the two sources that are most accurate at that pixel position. You'd be treating the third source as an outlier, and discarding it. Maybe that would give even better results!
it might do well on the drop-outs
but it might be worse on noise reduction as it always(!) uses only 2 sources, although 3 are avaible

[Karyudo]

Originally posted by E-Male
laser rot dropouts are white, while film dropouts are black (and of course there are other more or less similar video errors)
so if you're dealing with LDs, you set it to white, and when a pixel is white in only one of the captures, it's likely a dropout and could be eliminated independently from thresholds

Ah. I see. That sounds clever!

it might do well on the dropouts
but it might be worse on noise reduction as it always(!) uses only 2 sources, although 3 are available

But it would choose the two closest sources to average for each pixel, right? So it's averaging the two sources least likely to be plagued by noise? You see what I'm getting at: if A has a pixel value of 10, B has 12, and C has 20, then C probably is further away from the "true" value than A or B, so it is dropped and A and B are averaged to 11. Wouldn't that do better than (or at least as good as) a true 3-source average (which would give you 14)?

If C in the above example is 87, then it's a dropout, and it's definitely ignored.

I wonder: could the function take n samples, and average the x (= n-n') user-selected samples? So you could, in theory, do 3 caps, and do a true average of all three with:

"Average(source1, source2, source3, use=3 thresh=128)"

...or do dropout elimination with:

"Average(source1, source2, source3, use=2, thresh=128)"

Or, if you're really picky and have a lot of time and drive space to make caps, you could use 7 caps, and average the closest 5 with:

"Average(source1, source2, source3, source 4, source 5, source 6, source 7, use=5 thresh=128)"

Something like that?? This one function would solve almost all averaging and dropout-killing cases, wouldn't it?!?

Now that my idea is being taken seriously (rather than just me daydreaming), I'm psyched to try some sample filters out! That's not pressure for you, E-Male, just that this is some fun stuff. I wish I'd taken CompSci, rather than MechEng...

[E-Male]

i implemented my simplified version
http://e-rels.dyndns.org/twooutofthree.dll
use with 3 clips as parameters
example:
r=twooutofthree(a,b,c)
notes:
for best results use best clip as a and worst clip as c
a, b & c must have same length, resolution and colorspace
only tested it very very roughly
it's based on simplesample v1.1 (and v1.6)
i'll make the sourcecode avaible if anyone likes it (no need to hit me over the head with gpl, i know the rules)
please post how well it works

[scharfis_brain]

cool!

this means: if the three videos are full of framedrops, we are automatically getting a drop-free video :)
(if the drops are at differnet places all over the videos)

[E-Male]

if all goes well, yes
i got no sources of this kind, so i'm waiting for test-results

[Karyudo]

Originally posted by scharfis_brain
cool!

this means: if the three videos are full of framedrops, we are automatically getting a drop-free video :)
(if the drops are at differnet places all over the videos)

That's what I'm hoping, anyway! The end result should be better than the sum of the parts...

@E-Male:

Your link seems to be broken somehow. Or maybe your dynamic DNS hasn't propagated yet? (I use DynDNS, too). In any case, I'll have to wait to try your filter. I'm looking forward to running some tests, and posting some samples!

[E-Male]

my connectio might be maxxed out, please keep retrying
if it doesn't work PM me your e-mail address and i'll send it to you

[Karyudo]

Originally posted by E-Male
please keep retrying

I did, and I've got it now. Thanks! I'll prepare some sources and do some testing. Today is my wedding anniversary, though, so don't expect anything immediately...

[Karyudo]

OK, I've got it working! I spent some frustrating time with a dialog box that said the output of my script was not a video clip. Just my luck: this is the week the script kiddies have hacked the Avisynth.org site. Grr...

And then I remembered that:

2oo3=TwoOutOfThree(A,B,C)

won't work because the name starts with a number. Changed it to "TooT", and everything's fine! (As an aside, maybe "TooT" is a good name for the filter??)

My clips are almost too clean for testing purposes. After a bunch of searching, I found a dropout (which is *black*, by the way) in one of the input streams. The output stream shows nothing! So far, so good...

I'm going to make a few very short clips, and artificially add some noise and random-like flaws. Give your filter a big-time workout...

[Malcolm]

There is a thread about averaging multiple (VHS) captures: Filter debate: Averaging multiple captures and blur filters (http://forum.doom9.org/showthread.php?s=&threadid=42733)

Greetings,
Malcolm

[SoonUDie]

Sounds awesome! Thanks for the filter.

[E-Male]

i think i read that thread (or a similar one)

but this filter's advantage is not the averaging, it's that he ignores the source that's most different (checked seperatle for every component of every pixel)

[Karyudo]

I definitely read (well, skimmed...) that thread. It's pretty cool stuff for averaging multiple caps of the same source, but what we're up to here is a bit different, I think -- and in a profound way.

Now, for the first time, you'll be able to take three somewhat-flawed tapes or discs or TV caps and use "TooT" (that's "Two out of Three") to actually *repair* the video using the best two sources on a pixel-by-pixel basis! So any dropouts that appear on any of the three sources are (almost certainly) ignored, and the "good" pixels from the remaining two sources are averaged, to remove both the offending video glitch *and* reduce noise.

I've tried a very quick sample from a set of pretty-clean sources, and this baby works like a charm!

I'll do more work today and tonight, and I hope to post a little kit of sources and scripts to be able to demonstrate just how well this works...

[E-Male]

i nice way to show examples would be to use stackvertical and stackhorizontal to get a clip looking like this:
+-+-+
|a|b|
+-+-+
|c|r|
+-+-+
then you can post some example frames (pick ones with bad sources of course) as lossless png files
i think some pictures tell more then thousends of words about what that filter can/can't do

[Karyudo]

I also think a do-it-yourself sample set is worth a thousand pictures! I'll definitely come up with something that shows the results clearly. Even if the results aren't perfect.

Not much chance of that, though: I think this is a winner!

[Karyudo]

OK! Here we go. I've put together some samples to show how the TwoOutOfThree filter works. I took one clip of 16 frames, and applied some AviSynth subtitles to it to make three distinct clips, and then used the filter to fix the video. Pretty hot, I'd say. You can download the 1.4MB zip file and try it for yourself:

TooT_Test_ Kit.zip (http://www3.telus.net/11811/doom9/TooT_Test_ Kit.zip)

Now to start capping three copies from LD to HuffYUV -- ouch!

[E-Male]

nice sample, can't wait for some more practical (from your LDs)

new version can be found here
http://forum.doom9.org/showthread.php?s=&threadid=78338
new rounding (not visible i think)
new name
correct GPL release

[arnie.d]

Can someone explain how to use this filter. I can't get it to work. I have three clips called 1.avi, 2.avi, 3.avi. The twooutofthree.dll is in the same folder. I used this as script:

a=avisource("1.avi")
b=avisource("2.avi")
c=avisource("3.avi")

toot=TwoOutOfThree(A,B,C)

but I get the message: there is no such function: twooutofthree

[LocalH]

LoadPlugin("twooutofthree.dll")

[arnie.d]

LoadPlugin("twooutofthree.dll")
Yes I did that but should I use "toot" or "r" since both don't work for me.

EDIT: whoops... I typed twoutofthree instead of twooutofthree, sorry.

But the next problem is virtualdub says: "the script's return value was not a video clip." What does that mean?

[Wilbert]

It means your script is wrong (ie is doesn't return a videoclip). Try

LoadPlugin(...)
a=avisource("1.avi")
b=avisource("2.avi")
c=avisource("3.avi")

TwoOutOfThree(A,B,C)

[arnie.d]

Thanks, it's working!

[kingmob]

Disclaimer: this is not meant negatively

You could have solved this yourself quite easily. I think it's quite nice of this forum that you can ask every question here and noone really puts you down for it, but it would not only be better for us, but for you also, if you had read the error message.
Now i know most people don't actually do that, but most of the times the error message is actually very descriptive. It's not only there so that you can easily ask for the solution, but also so that you can find the solution yourself. Besides that, a search here or on avisynth.org would probably have solved your last problem in an instant, now you had to wait for the answer. While the error message basicly exactly said what you had to do.
And before i get flamed for it, i don't really mind him asking it, i just think it's good advice.

[arnie.d]

Disclaimer: this is not meant negatively

You could have solved this yourself quite easily. I think it's quite nice of this forum that you can ask every question here and noone really puts you down for it, but it would not only be better for us, but for you also, if you had read the error message.
Now i know most people don't actually do that, but most of the times the error message is actually very descriptive. It's not only there so that you can easily ask for the solution, but also so that you can find the solution yourself. Besides that, a search here or on avisynth.org would probably have solved your last problem in an instant, now you had to wait for the answer. While the error message basicly exactly said what you had to do.
And before i get flamed for it, i don't really mind him asking it, i just think it's good advice.

I don't use avisynth often and I don't know much about it. I f I have a problem I use "search" first but most of the times I can't find what I'm looking for. And I really didn't know what "the scripts return value" was. Besides there is no description of how to write an avisynth script. I need examples.

[kingmob]

I don't use avisynth often and I don't know much about it. I f I have a problem I use "search" first but most of the times I can't find what I'm looking for. And I really didn't know what "the scripts return value" was. Besides there is no description of how to write an avisynth script. I need examples.
They are on avisynth.org.

[E-Male]

if you get some good results, please post some screenies

[Karyudo]

Here's a sample I should have posted ages ago. Well, maybe not: I didn't annotate it until today. I think it's pretty boring, because it's sort of difficult to find samples where there are two sources with problems (of which this sample isn't one), and where the dropout is visible (in case you missed it, I've circled it in red).

I think the value of TooT, though, is more where there are dropouts and you can't easily pick them out by eye. The fact that TooT eliminates even those errors that you might not find by yourself (and be able to correct by hand) is where its strength lies.

Anyhoo, the sample:
http://img469.imageshack.us/img469/9738/tootsampleannotated3bg.th.jpg (http://img469.imageshack.us/my.php?image=tootsampleannotated3bg.jpg)