toot: new "advanced-averaging" filter (ignores worst source)

[E-Male]

http://e-rels.dyndns.org/downloads/toot.rar

This plugin takes 3 clips and for each component of each pixel
the 2 of the 3 sources that are most similar are averaged

example:
r=toot(a,b,c)

notes:
-a, b and c must have equal length, resolution and colorspace
-for best results a should be the best and c the worst clip

this plug-in is ideal if you have three copies of the same LD
pressing of a movies with different rot-based drop-outs
but i should also do well on "advanced-averaging" 3 caps
of the same source

old thread:
http://forum.doom9.org/showthread.php?s=&threadid=78187

[Soulhunter]

Hmm, what result would I get when...

- Feeding 3 differently denoised versions of a captured source?
- Feeding 3 differently denoised versions of the same DVD source?


Bye

[E-Male]

well, try it and tell us

as i said it will output the average of the 2 most similar sources
for every component of every pixel

i don't have any practical results, yet
but i think it'll do bet woth 3 caps

not 3 filtered versions of 1
but you can try it, maybe it'll give good results

you could also try using the original source plus 2 heavily filtered versions (maybe one temporal and one spatial)
hm, i think i'll look into that

[Soulhunter]

Played around with "Layer" to blend 2 differently denoised sources...
My idea was that this procedure should smooth out the boundaries*
Results where pleasant... :)

* The small annoying area between denoised and non-denoised areas


Bye

[E-Male]

i quickly tried this:

--------------------------------------
c=source.temporalsoften(1,255,255)
c=c.temporalsoften(1,255,255)

b=source.blur(1.58)
b=b.blur(1.58)

a=source.undot

r=toot(a,b,c)
--------------------------------------

may be something to look into

[Mug Funky]

not a bad idea that - you could combine it with something like MVtools for denoising, that way when it finds uglies (which i understand will still show up when bad vectors are used) it will ignore them, and maybe replace it with a traditional temporal or spatial soften.

[E-Male]

sounds interesting
can you go in detail some more what you got in mind?

[Mug Funky]

well for example, if you were to do a motion compensated denoise (with MVdenoise or something similar), then running toot with the original, or a traditionally denoised clip would be a good insurance policy against false motion vectors causing smears or other artefacts. please note that i'm not sure whether MVtools already takes bad matches into account (i've been away from my computer for over a month now, and wont be able to do avs scripting for another 3 weeks or so).

it seems to me that toot could be used as an extra (automated) layer of protection against ugly artefacts produced by otherwise excellent filters (RemoveDirt performs disastrously on scenes with shaking objects, or fast back-and-forth global motion - it can detect half of the frame as "dirt", and replace it with a blend of previous and next frames, causing a frighteningly obvious "shear" in the frame)

[E-Male]

so
a = original
b = mvdenoise
c = simple noise filter
right?

you gotta take into account that this will reduce the effetc of mvdenoise and the other filter by 50% since it averages their output with the original (or with each other)

i think about writing another similar filter that just output b or c depending on what's closer to a, maybe that would be better for working with other noise reduction filters

[Mug Funky]

ah yeah, i forgot about the 3 clip thing.

maybe:

a=denoise
b=denoise
c=mvdenoise

it probably wouldn't be worth the slowdown in that case.

i really must get back to my computer so i can play with these things. i'm working blind ATM.

[E-Male]

so you mean
a = b
?

in this case you'll only get a (or b, which is a) as output, as 2 equal sources will always be least different

[Mug Funky]

ah, of course, where's my brain.

it'd have to be original, denoised, and MVdenoised then. and that would reduce the effect somewhat. hmmph.

i guess there's plenty of other ways to insure against erroneous vectors (masking is what i usually do for this stuff)

[E-Male]

the effect would be reduced by teh averaging

if you wanna go in detail about mvdenoise, please don't do it here
i don't want this thread to go off-topic, thx
anything done using toot can be discussed here of course, no maatter what you combine it with

[CraigBjorn]

I recently tried another aplication for this plugin, trying to remove CCD noise. I scaned a picture 3 times, assuming the noise would be at different pixels each time, and then ran it through this script

---------------------------------
src1 =
src2 =
src3 =
#Turns images into three movies

r=toot(src1,src2,src3)
#Process to remove CCD noise

ImageWriter(r,"c:\",4,4,"BMP")
#Write the new image
---------------------------------

The result was ok, only some minor improvements though. either there is something wrong with my thought process or I should first run some filters on the images, try it yourself and tell me if you get ne better results.

[E-Male]

what is ccd noise?
could you make your 3 scans avaible?

[ronnylov]

The download link does not work.
Where can I find this filter?

[E-Male]

link fixed

[ronnylov]

Thanks!

[tedkunich]

E-male:

Thanks for the cool filter. I was wondering how difficult it would be to create a variant of your filter that on a pixel basis, compares two clips and if any of the Y, U or V components are some user supplied threshold different between the two clips, choose the lower value?

Here is the situation - I'm trying to capture an LD to DVD, but there is a significant amount of "laser rot" in my disks. On my player, the spots show up as white, blue, red, or green spots that are much much brighter than the surrounding pixels. I borrowed a friends LD player, and much to my surprise, his player shows hardly any of these "spots" however his player has significantly more noise in the picture than mine. What I would like to do is somehow detect the "spots" and then replace those offending pixels with the "clean" ones from the other player. I looked at your source code, but I'm lost as I know nothing about C++: if it were in C (and I had a compatible compiler) I would give it a go. ;)

Any chance that this could be easily implemented?


Thanks

Ted

[E-Male]

http://e-rels.dyndns.org/downloads/abdespot.dll

abdespot(a, b, c)

a = spotty clip
b = noisy clip
c = threshold

uses a, except when a > b+threshold, then b i used
all 3 components of the pixel are checked independently but with the same threshold

that should be the "filter that on a pixel basis, compares two clips and if any of the Y, U or V components are some user supplied threshold different between the two clips, choose the lower value"

yv12 only


doesn't crash for me, but is non-tested for correct (or even usefull) results

waiting for feedback

CU
E-Male

[tedkunich]

Wow!!! that was fast!!!!

I'll give it a try tonight when I get home.


Thanks

Ted

[tedkunich]

E-male: stupid question, but was is the range for the threshold parameter? I set a threshold of 255 (making a big assumtion) and noticed that the filter is swapping pixels.


Thanks

T

Originally posted by E-Male
http://e-rels.dyndns.org/downloads/abdespot.dll

abdespot(a, b, c)

a = spotty clip
b = noisy clip
c = threshold

uses a, except when a > b+threshold, then b i used
all 3 components of the pixel are checked independently but with the same threshold

that should be the "filter that on a pixel basis, compares two clips and if any of the Y, U or V components are some user supplied threshold different between the two clips, choose the lower value"

yv12 only


doesn't crash for me, but is non-tested for correct (or even usefull) results

waiting for feedback

CU
E-Male

[E-Male]

thearetically the range is the full int range
practically it's 0-255

"uses a, except when a > b+threshold, then b i used" should explain the effect

[tedkunich]

well, something is not working - if I set the threshold to the max, in theory clip a should be returned in virtually all instances, correct? I can move the luma of clip b prior to your filter and the results are the luma of clip b.

It does seem to work on one bright dot in clip a - it is definitely replaced with something, but the whole clip in general is altered - hue, saturation, and luma... would it be helpfull to send you a couple of frame sample of the two clips?

Thanks,

T

PS - just went through the video essentials calibration for the two players and now have the appropriate settings for both players. I'll try again, but the two clips were fairly close to each other.

[E-Male]

i'll dauble-check my code
if i can't find anything some shots would be good, i need a, b, and the filter result with different thresholds incl 0, 255, 10, 50, 100 and maybe some others

[E-Male]

the filter picks the lower value ignoring the threshold, no idea why

[tedkunich]

E-Male, could you post the code, I'm kind of intrigued as to see how you implemented it. What compiler did you use? Is it something that is downloadable???

Thanks

Ted

[E-Male]

source (http://e-rels.dyndns.org/downloads/abdespot.cpp)
free compiler (http://forum.doom9.org/showthread.php?s=&threadid=74608)
needs avisynth.h from avisynth source

[E-Male]

fixed the plug-in
please test
(fixed source up as well)

[Karyudo]

Thanks for your recent work in this area! It's cool to have some new tools that assume you have access to more than one copy of a source.

TooT works great where you have three copies of the same source: two will be nearly identical, and it's clear the oddball to be ignored is the third. However, it's more common (not to mention cheaper) to have just two copies of a given source. For example, I have two copies of the "faces" THX LD of The Empire Strikes Back, which is already more than most people want to pay good money for!

Clearly, there is almost certainly enough *picture* data to create a clean copy from two flawed sources, but without a way to "inform" the filter, there's not enough *decision-making* data (with three sources, it's the redundancy that provides the decision-making data). Without any decision-making data, a filter can't choose which of the two copies is correct when it finds differences between them.

I've thought of some possible solutions:

1) a "control" clip that is only used to decide on differences between two clips; and/or
2) some temporal analysis to see which of two clips is more likely to be flawed; and/or
3) some spatial analysis to see which of two clips is more likely to be flawed.

In case (1), the control clip could be a lesser-quality capture (for example, a lower-quality capture from VHS used to compare two higher-quality LD captures), or a capture from a different release of the same material (for example, I have a copy of The Empire Strikes Back Special Edition that could help a filter decide on which of two "faces" captures is right at any given point).

In case (2), deriving some sort of numerical value for frames 1 and 3 (that is, nearly-identical frame 1's and frame 3's one frame before and after dissimilar frame 2's) might indicate whether frame 2a is more likely correct than frame 2b. For example, in a low-motion scene, if the values for frame 1 and 3 are 115 and 117 respectively, and the values calculated for frames 2a and 2b are 12 and 118, then it's more likely that picture data from frame 2b is a better choice than data from frame 2a. Some user-selectable threshold would be desirable, to reduce artifacts (like motion blur from temporal smoothers).

In case (3), comparing a pixel to the surrounding pixels in both sources may indicate which source is more likely correct. For example, if a pixel differs from its 8 nearest (spatial) neighbours by no more than 5 levels in one source, but differs by 100 levels in the other source, then the first source is likely the one to choose. Again, some user-selectable threshold would be desirable, to reduce artifacts.

In fact, just using the temporal and spatial techniques developed for other filters should probably be enough, because a lot of those filters are both trying to identify problems *and create data to fix them*. In the case of two flawed sources, the problem is merely to decide which of the two is more likely correct; no picture information needs to be created. As long as even a quick-and-dirty analysis is reasonably sound, the resulting picture should be perfect.

Two further details/abstractions might be possible:

- using two different sources (for example, from two different video transfers), eliminate dust, etc., included as part of the video on pre-recorded sources. Note that none of the analysis above can correct dust, etc., that's actually been recorded (as opposed to dropouts or rot) on a tape or LD!

- The Decomb package by Donald Graft (aka neuron2) allows dumping the field-matching results to a text file, and allows a user-created text file to override the field-matching process. In a similar way, a list of data from frames that differ between two versions (including the filter's selection) could be exported to the proposed comparison filter, and then this (hopefully short) list of frames could be reviewed by the user, to semi-automatically clean the video. An
imported text file could include rows with [frame_number, source] included (for example, "2354, a") that would tell the filter which of the input sources should be used. This would get a lot more complicated in the case of both frames being flawed, but in different parts of the picture.

Anyway, that's a lot of rambling on my part. It's what I do best; I have no coding ability...

[tedkunich]

@E-Male

Thanks, looks like that did the trick.... only got to play with it for a few mimutes last night - late night at the office... :( I'll play with it some more over the next few evenings.

So what was the bug?


T



Originally posted by E-Male
fixed the plug-in
please test
(fixed source up as well)

[tedkunich]

Originally posted by Karyudo
I have a copy of The Empire Strikes Back Special Edition that could help a filter decide on which of two "faces" captures is right at any given point

FYI, the SE edition is quite different than the "faces" set or the definitive set - frame cropping is different, hue and brightness tweeks... etc.... for all practical purposes they are two different movies....


T

[E-Male]

@tedkunich:
i missed a line where the parameter is passed on to the plug-in

"two different movies" is exegerated, with some work (and it will be work) the se could be turned into some kind of control clip
i also thought of the idea of using DVD versions to fix the colors of caps, but that's a bit of anotehr story


@Karyudo:
1) is possible, i thought about something similar
using a variation of toot, that uses either b or c, depending on what is closer to a, without averaging with a
2) and 3) are more complicated

i think something like that will work (rough idea)
-cut and crop the SE to match the originals
-use the histogram (and/or other tools) to fix the colors/sat/hue/... of the caps according to the SE
-run the filter "1)"
-resize, denoise, .... whatever you want
-encode the result

especially the recutting seems a lot of work
plus differnt versions could be scenewise reframed, which would make them useless to each other

so, i'm off to bed, hope i didn't write to much nonsense with my tired mind

[tedkunich]

Thanks!

RE the difference between the SE and the Definitive is quite dramatic when you overlay (actually subtract) one from the other. Each scene has a different framing, ie zooming and cropping. In general, the SE has a wider visual frame than the definitive - I have both sets and have compared them side by side, frame by frame. I was going to originally intercut scenes from my SE version to fix the scenes that are deterioration from "laser rot" on my definitive set, the work needed was going to be WAY more that it was worth. Not saying it cannot be done, but who has a couple of months to go through every frame to fix sizing AND color/hue?!?!? ;)

Thanks again,

T




[QUOTE]Originally posted by E-Male
[B]@tedkunich:
i missed a line where the parameter is passed on to the plug-in

"two different movies" is exegerated, with some work (and it will be work) the se could be turned into some kind of control clip
i also thought of the idea of using DVD versions to fix the colors of caps, but that's a bit of anotehr story

[Karyudo]

Originally posted by tedkunich
the difference between the SE and the Definitive is quite dramatic when you overlay (actually subtract) one from the other.

Oh, I know! (I have multiple copies of each.) However, if you do the calculations (whatever calculations make sense - maybe a histogram?) on any given series of frames in either version (where the scenes are unchanged between SE & DC, of course), then the *trend* should be the same. I'm hoping you wouldn't need the SE to be identical; you just need it to change in a similar way. That should (?) be enough to let the filter decide which of two candidates is better.

[E-Male]

it would be a LOT of work making the 2 versions match, that is for sure

[Karyudo]

I'm trying to simulate (theoretically -- I'm no coder) a decision process that a person would use to decide between two frame candidates.

If you were to step through a pair of sources, knowing that it was equally likely that each was flawed at any given frame, then there would be some clues available to tell you which was the better choice:

1. If you had another copy -- even though it didn't match exactly -- you'd be able to see which of the two candidates looked more like the third copy. The more the flawed copy deviated, the easier it would be. And the less it deviated, the less it would matter!

2. If the frames before and after the candidate frames didn't have a flaw, you'd probably be able to see the flaw pop on for the one frame in question, and you'd know to keep the other frame candidate.

3. If there's junk flying around the frame anyway (like an explosion), then the size, shape, hardness and intensity of the flaw would give you information about whether it's a video problem, a film problem, or supposed to be there! For example, a black or white line of a dozen pixels or so, confined to a single horizontal line, is probably a dropout, and should be fixed.

I don't know how good just a Type 1 comparison of something like a whole-frame histogram would be in determining which source is right. A user-selected threshold value would probably be good. Because you're only ever replacing from a frame that *is* the same (except for the flaw), the control source shouldn't need to be colour-corrected or super-close on the alignment, etc.

At least, I hope not!

In fact, since most (all??) video flaws and most film flaws are primarily black or white, how about comparing monochrome histograms, with a heavy bias to the 0-54 and 200-255 range? I suspect that it would be fairly informative to look at these highly-specific histograms for decision-making. I mean, that's about what you're doing when you look at the frame yourself and note, "There's a black spot here," isn't it?

[E-Male]

"you'd be able to see" <-- that's the HUGH error in your thinking process

yes, a human being would see which is closer, because they are intelligent (mostly :devil: )

but computers are dumb
they can only do math, nothing more

please take that into account every time



about the (b&w)histogram idea:

you might be on the right track with that

BUT "control source shouldn't need to be colour-corrected or super-close on the alignment, etc. - At least, I hope not!" might be exactly the point where stuff gets really difficult



oh and BTW, discussing that on the thread of another filter might be the reason why nobody else noticeses it

[Karyudo]

I've re-posted a bunch of my initial crazy ravings to another thread ("Dust/Flaw Elimination using two copies of a source?"). See if there isn't more interest generated...

E-male, it's super-cool that you're interested enough in these ideas to very quickly put together filters for first me and now tedkunich! I appreciate all your efforts and comments. The project I'm working on (the Definitive Collection, naturally) is one that I don't mind will take a while, knowing that the producers aren't going to get around to releasing it on DVD any time soon!

[scharfis_brain]

my AVS 2.55 does not load the toot.dll

AVISynth open failure
loadplugin: unable to load "C:\x\toot.dll")

(I am NOT (NEVER) using autoloading)

[E-Male]

i tried the newest avisynth 2.55, but it crashes vdubmod on start
with 2.54 toot.dll works fine

[scharfis_brain]

why does it crash AVS 2.55 ?

I can/want not downgrade to 2.54, cause several scripts will not work with 2.54

Could you fix that?

[E-Male]

i can't even start virtualdubmod whith avisynth 2.55 installed so there is no chance i can work on fix
also i got no idea what change in avisynth causes the problems
maybe post an exact description of your error in an
avisynthdevelopment thread

[scharfis_brain]

e-male, I think you have to disable you plugin autoloading...

also, I've posted the exact error MSG some post earlier

[E-Male]

i'll see if i can find the problem

but i mostly just added my algorithm to sample-plug-ins
so i don't know so much about avisynth-programming, most likely not enough to solve this

[BruceBanna]

Can anyone post the TooT dll, please? I think I've searched all the interweb, but I can't find any links which are not dead.

Thanks in advance!

[mustardman]

I too would be very interested in a link (or links) to both TooT and ABdespot. Judging from the date of the last post, these have been dead for quite some time.

Surely somone out there has a copy safely stored in some directory somewhere...?

Cheers,
MM

[TheFluff]

Even if someone happened to have it sitting around the last few posts from seven years ago seem to indicate that it crashes in 2.5.5+.
That said, reimplementing what this filter seems to have done (averaging the values of N most similar pixels out of a selection of M possible) should be absolutely trivial, even if I'm highly doubtful of its practical usefulness.

[mustardman]

@TheFluff
Absolutely true - TooT crashing in 2.5.5 seems to be a pretty serious bug! As to its' usefulness, perhaps it would be good, and perhaps not. My interest is more along the lines of one more screwdriver in the toolkit.

ABdespot seems to be more interesting than TooT, but again, how useful would it really be?

In looking at filters to clean video, FFT3Dfilter seems like a pretty capable piece of software, although reports are that it is slow... mind you it came out in 2004 and is still active(?). Computing power has come quite a way since '04!

Analog home video has lots of weird problems (not all of course). Sometimes I get a vertical field shift on only one field, sometimes the entire frame jumps (extra blank lines at the top of frame), sometimes the horizontal sync gets wobbly... Most of my problems are "grain/noise" from low light situations, camera shake (a big one) and one tape I had has only every odd (or even, depends on where you start counting) field blank. One head on the drum gave out during recording, and it wasn't noticed until the tape was played back! That was a nasty problem because Hsync relies on the entire picture, not just half of it.

Audio filtering is one big problem that very few people are willing to tackle. Cooledit (v1.2a) did a very nice job, and (IMO) worked better than any of the following versions.

Unfortunately I am not a coder, so I have to rely on the work of others to help me do my stuff.

The best filter I have ever come across : Deshaker by Guth!! That is one cool piece of work!

Cheers,
MM.

[ChiDragon]

This filter was designed for problematic LaserDiscs, not tapes. I've compared multiple captures of the same VHS tape in the same VCR and the only difference for a good tape is in the high frequency noise generated by the capture setup (VCR output, cables, the TBC in-line, EMI, whatever). This noise is much finer than the noise/grain present on the tape itself (which remains the same through multiple captures) and could be filtered out just by resizing to 352x480.

I am curious about using multiple copies of the same commercial-duplicated VHS for movies that haven't made it to DVD or LD, though.