New Filters: Conditional Temporal Median Filter and DeSpot filter

[kevina]

I have created a Conditional Temporal Median Filter. The filter is designed to remove temporal noise in the form of small dots and streaks found in some videos. A common cause of this is dirty VHS heads but I have also seen small black or white streaks in broadcast material.

You can find it at http://kevin.atkinson.dhs.org/temporal_median/.

It requires my C API Plugin available at http://kevin.atkinson.dhs.org/avisynth_c/.

Please test it out and let me know what you think. It may occasionally cause AviSynth to crash, however it is definitely usable. It will generally only crash when previewing the video (ie lots of back and forth motion in the timeline). To be safe just reopen the script file before saving your final copy. I will release another version once I track down this bug, of course. Update: This bug is now fixed as of version 0.92

It works as follows:

1) Find pixes that are different from its neighbors by at least P1.

1b) Enlarge outliers based on P2.

2) Determine the size of the specks and reject all those larger than PWIDTHxPHEIGHT

3) Find moving areas of an image by simply comparing each pixel to the previous frame and considering all those which are greater than MTHRES.

3b) Remove pixels determined as noise from the motion map.

4) Denoise the motion map by diffusing and then diluting. This is probably the most important step.

5) Only remove the specks in which there was no motion in the current or the next frame.

[Mug Funky]

that's some good thinking there.

would this work for cigarette burns on bad film transfers?

or dust and scratch on film?

i haven't got a chance to test this yet, but it certainly sounds good.

[kevina]

It might, you need to try it and see. It has only really been tested well on one particular type of noise so the paramaters will most likely need adjusting.

[cablemonkey]

I tried the ConditionalDenoise filter, and have the following to report:

1) Its quite good at detecting and removing spotting in poor film source. So far this is the only filter (other than SpotRemover for vdub) that has been effective in removing the "black dots" noise in my current project.

2) It still has some stability problems. While working with this, AVISynth many times would throw out the following error:

Avisynth read error: Avisynth read error: Avisynth: caught an access violation at 0x6c2016eb, attempting to read from 0x0b96400.

The first hex address would always be the same, but the second would vary. This was using avisynth 2.5.2. Even with it was he only filter in the script (except ConvertToYV12), it would kick out this error, even after closing vdub and reopening.

Looks like it has a good bit of potential, but definitely needs some more testing.

[kevina]

Thanks for the report.

It should be OK to encode a video as it should't crash when not jumping around the video. I am not trying to make up an excuse for the bugs, just a work around.

[kevina]

Just to make sure. Are you using version 0.11 of the C API Plugin?

[morsa]

I have the same crash and I'm using c api version 11.
@Cablemonkey:
Could you post your setting for removing spots?

[kevina]

A new version is now available. This version fixes the nasty crash. It should work OK for everyone.

So please try it out again.

http://kevin.atkinson.dhs.org/temporal_median/

[cablemonkey]

i've been swamped at work (gotta love post-acquisition integrations)and have had to put this project on hold for a bit. I'll let you know when i have have an opportunity to pick it up again.

[unplugged]

Crashes not resolved yet, at least for ConditionalDenoise, couldn't use that with (YV12) 768x576, 720x576 and 720x288; generally it never worked to me.
I haven't tried other res and avisynth is the lastest version.
:(

[kevina]

What processor are you using?

Make sure you are using the avisynth_c.dll included.

[unplugged]

Athlon XP, the DLL is latest 0.14.
Must say that I have tried denoise only with defaults as follows: ConditionalDenoise(interlaced=false)
Maybe am I missing anything... ?

P.S.: there is not much doc about mwidth, mheight and mp parameters. Please add something, thanks :).

[slk001]

I have the latest version and I cannot get ConditionalDenoise to even run. ConditionalMedian looks to be very useful for me, and I want to see if the Denoiser will also.

Whenever the Denoiser runs, Avisynth always catches an access violation.

I run on dual P4s. Your C plugin is used.

Here's the script I am using:

LoadPlugin("d:\program files\HOLDER\mpeg2dec3.dll")
LoadPlugin("d:\program files\AviSynth 2.5\plugins\avisynth_c.dll")
LoadCPlugin("d:\program files\AviSynth 2.5\plugins\CondTemporalMedian.dll")

#mpeg2source("d:\program files\HOLDER\TEST.D2V")
mpeg2source("d:\program files\HOLDER\TEST.D2V").bilinearresize(720,480)

ConvertToYV12

#ConditionalMedianMap()
ConditionalDenoise()

#fadein2(48)

#ResampleAudio(44100)

#version()


This filter has amazing potential - don't give up on it yet!

[slk001]

One more thing... ConditionalDenoise causes Avisynth to give the error when it tries to access the second frame. The first frame holds steady.

[kevina]

OK Everyone. I THINK I finally got it working right. I tested it on a Windows XP system (I normally use Windows 98) and was able to reproduce the problem and fix it.

So give it another go. http://kevin.atkinson.dhs.org/temporal_median/. A non iSSE version is now included.

I also expanded the readme a bit to give a quick guide to tuning the parameters.

[slk001]

Yes, it works quite well now. Here's a script that I have been testing with:

LoadPlugin("d:\program files\HOLDER\mpeg2dec3.dll")
LoadPlugin("d:\program files\AviSynth 2.5\plugins\avisynth_c.dll")
LoadCPlugin("d:\program files\AviSynth 2.5\plugins\CondTemporalMedian-isse.dll")

mpeg2source("d:\Experiment\CHHTEST.D2V").trim(195,0)

conditionalmedian(mthres=30,mwidth=6,mheight=4,mp=11)

ConditionalDenoise(mthres=16,mwidth=5,mheight=3,mp=11,p1=2,p2=2,interlaced=false)

converttoYUY2()

v1=declick(lthresh=35,cthresh=10,map=true)

#stackhorizontal(v1,v2)

V2=blankclip()

audiodub(v1,v2)


Notice that it also uses DG's DECLICK routine. I find that this .DLL helps remove clicks in the frames at scene changes, where there is no prior and post matches.

The CONDITIONALMEDIAN filter cleans up the overall video nicely (this test video is an old film with lots of dust, hairs, etc on the encoding - also, this video also has the chroma shifts due to the age of the film).

The CONDITIONALDENOISE filter removes horizontal noise (the "meteors") on my video (it is a laserdisk capture with scratches on the disk surface).

Many kudos to you, Kevin (and Donald). Thanks for your efforts.

[morsa]

May be some source code from Declick should be included into Kevin Atkinson's filter, cause Graft has stop development and even download availability of Declick.

[Fizick]

Recently I found "Conditional Temporal Median Filter".
Greate works! Thanks to Kevin Atkinson.
I tred to use it for my archive of old 8 mm kino-films (telecined with camcorder and tv-tuner).
These movies have many spots and scratches.
Before I tried similar filters for VirtualDub: "SpotRemover", "Pops", and "Random Noise Remover".
But CTmedian is the most advanced, it makes less artefactes (false detections) on moving parts. Also it removes spots more purely, completely (with outliers).
However, I have one question (proposition).
In many my (and similar) movies, almost all spots are black (from dust).
Sometimes, the CTmedian filter make false detections at transition from dark to light pixels.
Rarely, it transfers a dark spot from previous frame to current!
In contrast, there are another movies, captured from analog TV, with only white spots (snow).
I think, this filter would produce still less artefacts on similar movies, if it makes comparing of pixels not by absolute difference, but simple difference (as option).
For example, some new option may control filter's work as follows:
- detect and clear all transitions (default),
- detect and clear only black spots,
- detect and clear only white spots.
Is it possible in next version of the filter?
By the way, the POPS filter has similar control (tuned black and white spot detection levels).
P.S. I am not a real C-programmer.
P.P.S. Sorry my English.

[Fizick]

Sorry, the my previous post have been prepared 5 day ago (forum rules for first newibe post), but I decide to send it now.
During these days I try some c-programming.
In Kevin's source code (GNU) I found (in file med-f.cpp) function "find_outliers", and change some code (for non-CCE version).
( o - output, p - priveous frame, c - currrent, n - next)

o[x] = 0;
byte mn = min(p[x], n[x]);
byte mx = max(p[x], n[x]);
//Original code by Kevin Atkinson:
// if (c[x] < mn) o[x] = mn - c[x];
// if (c[x] > mx) o[x] = c[x] - mx;
// Modes added by Fizick:
// Set some black-white mode of spots search, from SIGN parameter:
switch (parms.sign){
case 1:
// sign=1 - only black (dark) spots and outliers
if (c[x] < mn) o[x] = mn - c[x];
break;
case -1:
// sign=-1 - only white (light) spots and outliers
if (c[x] > mx) o[x] = c[x] - mx;
break;
case 2:
// sign=2 - only black (dark) spots, any outliers
if (c[x] < mn) o[x] = mn - c[x];
if (c[x] > mx) o[x] = min( (c[x] - mx), parms.p1 );
break;
case -2:
// sign=-2 - only white (light) spots, any outliers
if (c[x] > mx) o[x] = c[x] - mx;
if (c[x] < mn) o[x] = min( (mn - c[x]), parms.p1 );
break;
case 0:
default:
// sign=0 - any spots and outliers (default)
if (c[x] < mn) o[x] = mn - c[x];
if (c[x] > mx) o[x] = c[x] - mx;

I add one new parameter "sign", and of course change some code for it also.
It works!
May be, Kevin Atkinson will put similar code in original source (I not understand some code in parameters parsing).

[Fizick]

As Kevin does not respond, I modify the filter myself (GNU GPL v.2).
Changes:
1. "sign" parameter for removing only black or white specks.
(as described in my previous post.)

2. "per" parameter for more smoothed specks perimeter.
Set percent of smoothing for pixels near perimeter (border) of spot.
per=0 - all pixel at spot outliers are replaced to same pixels
from adjacent frames.
per=100 - pixels near border of spot outliers are not changed,
internal pixels at spot outliers are changed partially,
(with quadratic interpolation),
according to pixel differences from adjacent frames,
smoothing=0 when difference>p1
per=i - intermediate degree of perimeter smoothing.

default per=0, recomended value per=50.

3. More short filter name ctmedian.dll

The filter may be found at:
http://bag.hotmail.ru/ctmedian/ctmedian.dhtml

I am still waiting for Kevin answer.
And testers too.

[morsa]

I know Ia sk for this things all the time, please forgive.It´s just that I find all these amazing filters and I hope we could add them to Avisynth collection.

It seems that this is what we´ve been looking for since a long time.
For your (our) pleasure.

There are two filters for The Gimp and now for the Cinepaint project that are really interesting.They just need an experienced coder to port them to Vdub or Avisynth.
One is a deconvolution one and the other...Wow is an scratch removal filter (at least it says that).

http://sourceforge.net/project/showfiles.php?group_id=75029

There is a port of a Deconv filter from Gimp to Vdub here

http://fcchandler.home.comcast.net/Refocus_YUV_Test.zip

Hoping to find someone kind enough to port them,
Morsa.

[Fizick]

to Morsa:
Thanks, it is interesting.
Cinepaint is free similar Ulead Videopaint?
However "sratch removal" is not filter, but mostly various information. The only source code included is for spatial method, not temporal.

[morsa]

Well the idea now would be to have a detector for the scratchs.Isn't it?.I believe also that this Spatial interpolation would be very good for Logo removal and Subtitles removal.

[Fizick]

I release a new version 0.934 of the filter -
with luma correction and local blur of deleted spots borders.
The filter may be found at:
http://bag.hotmail.ru

COMBINED USAGE

Very good results are may be obtained with combined this filter
with motion estimation and compensation, even if only general motion,
produced my (Fizick) GENMOTION plugin.

[Fizick]

I released a new version of the filter.
Changes 1.0 from 0.934 (Dec 30, 2003)
Added "tsmooth" parameter for temporal smoothing in static areas.
Changed filter name to DeSpot as more appropriate, filter file name to despot.dll,
and function names:
ConditionalDenoise to DeSpot,
ConditionalDenoiseMark to DeSpotMark,
ConditionalDenoiseMap to DeSpotMap,
ConditionalMedian to DeSpotMedian,
ConditionalMedianMap to DeSpotMedianMap.

Corrected info.

The filter may be found at:
http://bag.hotmail.ru

[brett]

So, is there going to be any more work done on this filter? It seems very nice for VHS captures, but it's pretty unusable in its current state.

For example, my heaviest filtering script is:


Convolution3d(c,0,32,128,16,64,10,0)
ConvertToYV12()
Cnr2("xxx",4,5,255)
UnDot()
STMedianFilter(8,32,0,0)
QMF()
BilinearResize(352,208)
UnDot()
Temporalsoften(2,3,3,mode=2,scenechange=6)
Limiter(min_luma=16)


I use it for noisy VHS and cable TV captures. This encodes at 7.33 FPS on an Athlon XP 1600, so that's 3 hours to encode your agerage 1-hour show with commercials taken out.

If I simply add DeSpot to the above script, it's 13 times slower. It goes 0.57 FPS, and takes 37 hours to encode a 45-min show. It would take 3 days to encode your average VHS movie, so it's really not practical.

Maybe there is a similar filter that does well removing the white horizontal lines of noise that show up in old VHS tapes?

[Fizick]

What is your script for DeSpot?
Its speed is depended from parameters setting.

[morsa]

@Fizick

In case this is useful for anything,

http://www.mpi-sb.mpg.de/~hitoshi/research/image_restoration/index.shtml

If someone is interested and if this could be usefull here is another link with some examples and a description of the algorithm in pseudo code.


http://www.cs.berkeley.edu/~efros/research/EfrosLeung.html

[krieger2005]

@Fizick

Thanks for your Filter. It do a good job when you tune them enogh.... But this is the problem: Tuning this filter for a movie is a horrible thing. Not at least because some desciptions of your filter-usage seems to be false

1. you wrote "p1 > mthres > p2". But when im using "p1 > mthres < p2 AND p1>p2" then DeSpotMark show me that some areas were spots.

2.
"sign (default 0)
Set mode for removing of only black or white spots or both:
sign=0 - any spots and outliers (default)
sign=1 - only black (dark) spots and outliers
sign=2 - only white (light) spots and outliers
sign=3 - only black (dark) spots, any outliers
sign=4 - only white (light) spots, any outliers "
But you can only select -2, -1, 0, 1, 2.
When you select 1 or 2 for searching for white/black spots then the filter find dark/light spots too. Why?

3. The Variables
mwidth (default 7)
mheight (default 5)
mp (default mwidth*mheight/3)
seems for some peaple unclear (for me too). Can You explain this a little bit more?

and then technical detail: Avisynth 2.54 does not like your Filter (DeScratch also). When you set it for loading (LoadPlugin) it sait: Could not load ...

I try always to tune the filter for one pic (first). For that i set p2=0 and p1="something" (mthres=p1-1). When the spot where found i tune p2 (+1) or mthres (-1), since the spot were still found by the filter. This method is a "need many time" method and does not give me the results which i estimate -> Artifacts.
For now i do not understand clearly what the filter do so it is difficult to tune the vaiables.
As long, as one can not see which areas were removed by which step of your algorithm it is difficult to tune the values. Something that you could do is to show different steps in different colours: mark_v_forP1, mark_v_forP2...
Then: Can you do that the Mark is transparent (highlighted in the selected colour) so one does not need StackVertical/Horizontal, which does not help, when you have a picturesize of 720/576 (because 720*2=1440, 576*2=1052 an i have the maximum resolution of 1024/768).

But at all the filter does helped me to restorate some movie which were very old and for that :thanks:

[Fizick]

1. no comment yet.
2. Yes, its my mistake in info.
Correct:
// sign=1 - only black (dark) spots and outliers
// sign=-1 - only white (light) spots and outliers
// sign=2 - only black (dark) spots, any outliers
// sign=-2 - only white (light) spots, any outliers
// sign=0 - any spots and outliers (default)

3. It is mostly Kevin code.
MP is not very clear for me too. It is some limit number of moving pixels.

4. I use v.2.53. Problem was discussed here. Wait.

5. Try get pixel values (and differences) with some Photo-editor.
It is not very robust filter, yes.

6. I have not time now. Plese, wait.

[Fizick]

O.K. I return to it.
New experimental version 1.2 of DeSpot plugin (June 01, 2004):
Added parameters Ranked, weak, maxpts, p1percent,
change mark mode to color spot with weak motion map.

If "ranked" parameter is true, the 3 neighbors in previous frame and 3 neighbors in next frame are ranked (ordered by value), and those min and max are used for current difference calculation

If "weak" parameter is used, the some small weak spots are rejeñted also.
If "numpts" parameter is set, the big spots are rejected also.
If "p1percent" parameter is set, then not strong (by criterion P1) spots(which have many outliers) are rejected also.

The color (pink, green or grey) of noise spot now is depend from parity of mark_v.
Weak motion map is also shown now.

Source now is compatible with MS VC6, VC7 (free tools). But I can not port it from avithynth_c to pure avisynth plugin (needed in some help):)

To moderator:
May you add to this thread subject some words:
... and DeSpot filter.

[Fizick]

Now I have a more new version.
Changes 1.3 (June 7, 2004) from 1.2:
Remove parameter "weak" ( "p1percent" is seems more useful).
Add parameter "Dilate" to enlarge spots sizes.
Change parameters order to more functional.
Change default values of some parameters to more optimal.
Update doc.

If "Dilate" mode, spots are enlarged (with range value as the parameter is set) to cover its non-contrast edges and close small gaps between its, by applying a morphological dilate operation to noise (spots) map.

[krieger2005]

Hi,

thanks for development of this filter...
1. Can you put the Link to Despot in the first Post? I must search it always.
- EDIT - I Note, that you does not open the thread... forget this point...
2. I tried DeSpot 1.2 with Avisynth 2.55 alpha using "avisynth_c.dll". It was all okey...
3. Your Site is not updated?

thx
greets krieger2005

[Leak]

Originally posted by krieger2005
1. Can you put the Link to Despot in the first Post? I must search it always.

A small hint: click on the "www" button below Fizick's posts to go straight to his homepage... ;)

np: Gescom - Keynell (Mix 1) (Autechre - Keynell Mixes)

[Fizick]

Originally posted by krieger2005
3. Your Site is not updated?


I have v1.3, but I did not release it :D
I found a small bug.
And I want to do once more update.
Please wait.

I also wait some responce from users about possible bugs and suggestions.

If nobody responded, why I will release new version? :(

[Backwoods]

Originally posted by Fizick
If nobody responded, why I will release new version? :(

Doesn't mean there aren't people using it :)

[Fizick]

O.K. my site is updated with 1.3.

I wait some respond how useful are new options for various sources.

Now I also think about porting DeSpot to native Avisynth plugin (not C).
Anybody will be interested?

[Fizick]

New DeSpot version 2.0.
It is major release update (probably alpha with bugs):

Main interface code is rewrited, and now filter is native Avisynth plugin (not C-plugin).

Add parameter "median" instead of DeSpotMedian function,
Add parameter "show" instead of DeSpotMark, DeSpotMap, DeSpotMedianMap functions.
Replace parameter "mp" to parameter "merode" (relative)
Change "p1percent" to integer.
Update doc.

[iradic]

hi...

www button is ok... but you could add url in your signature (direct link )

bye

[Fizick]

You are seems too lazy...:D

http://bag.hotmail.ru

[krieger2005]

I have problems with this Filter on "higher" motion in movies. I used

Despot(p1=25,p2=10,pwidth=60,pheight=60,mthres=12,ranked=true,dilate=1, sign=2)

and when there is high motion there the filter produce ugly big artefacts. When i increase mthres (e.g. mthres=17) it seems, that the artifact stay the same, when i lowering the threshold the effect is the same.

Summerized for me is it difficult to decide which factor p1,p2 or mthres should be tuned to get right results. As you can imagine i does not understand the algorithm of this filter so good, so maybe you can get give some points, how to tune it.

-- EDIT
I think the filter should get better motionestimation (not best but better) because i think one must set p1 high und p2 low to reduce artefacts. Because i think a simple difference betweet two pixels is too bad to decide, if it is a spot or motion.

greets krieger2005

[Fizick]

For high motions, you may increase Mwidth, Mheight parameters, and try tune Merode.

But of course, motion compensation is the better. Did you try Depan or MVtools ?

New version Despot 2.1 (June 14, 2004) is released:
Add parameter "mratio" parameter (it was internal =3 in all previous versions).
Change default of "merode" to dependent from "mratio".
Update doc.

At last I begin to understand how the filter works at motion denoising stage :D

3) Find moving areas of an image by simply comparing each
pixel to the previous frame and considering all those which
are greater than MTHRES.
3b) Mark motion pixels without noise with weight "mratio" in the motion map.
Mark pixels determined both noise and motion as weight 1 in the motion map.
4) Denoise the motion map by erosion and then dilating
(as a whole it is morphological opening operation).
During erode phase, the motion map is eroded with range mwidth/2 and mheight/2,and zones with small summary weight (less than "mratio*merode/100") are decreased or completely removed from motion map. Such zones correspond to small relative (in percent) numbers of motion neighbors within this range or mostly noisy pixels (spots).

During dilate phase, remained motion zones are enlarged with same range.

[krieger2005]

I try first MVTools. But because the source is very noisy MVTools generate a "block-theater", where block were shift down, up ...
But in some scenes does the MVTool "something", so that DeSpot does less artifacts.

I tried also DePan:

i=last
d=depanestimate(range=5, trust=100)
x=depan(data=d)
return StackHorizontal(x,
\ subtract(x,i).ColorYUV(autogain=true))

The Result was, that i was the same as x. So that DePan has done nothing...

For higher motion-scenes u use:
norm=ConditionalFilter(motion,des,"abs((YDifferenceFromPrevious+YDifferenceToNext)/AverageLuma*100)",">","19")
ConditionalFilter(highmotion,norm,"abs((YDifferenceFromPrevious+YDifferenceToNext)/AverageLuma*100)",">","35")

where des the Result of DeSpot is, motion a blured movie

btw, thanks for the tip with Mwidth, Mheight. I forgot these parameters....

[Fizick]

With Depan, you must use or two Depan(), or DepanInterleave, to compensate global motion for Despot.
Simplest script:

avisource("qwerty.avi")
i=ConverttoYV12()
mdata=DePanEstimate(i)
DePanInterleave(i,data=mdata)
Despot(some parameters)
selectevery(3,1)



BTW, now I try implement some new method of motion detect in DeSpot.

[Fizick]

Changes new DeSpot v.3.0 (June 22, 2004) from 2.1:
Version 3.0 is major release update (probably alpha with bugs):
Add another motion detection method "motpn" (previous to next frame).
Add whole segments removing methods "seg"=1,2. - more safe.
Change luma correction to local in segments (spots) removing mode. Change some defaults.
Code reorganization.

http://bag.hotmail.ru

If new modes will be good in tests, I will remove some old modes in next version.

[tedkunich]

Fizick,

Have been trying out your filter in restoring some captures off of a laserdisc that is experiencing "laser rot". Your filter seems to knock out the white and black spots nicely, but it does not seem to detect/eliminate any chroma information from the removed spots. I have white, red, blue, and green dots - the white ones are gone, but the others have the intensity reduced, but the chroma flair is still visible. Any thoughts, tricks???

I'm using version 3.0 or your filter.


Thanks

Ted

[Fizick]

I think, indedendent color spots detecting may produce false detection with ugly color artefactes.
Probably removing color spot only at places of detected luma spots may be better.
But it is not very simple task becourse of different sampling of color planes in YV12, espessially for interlaced clips.
And it also may produce some artefactes.
Current version of DeSpot can fit luma locally, but how fit color in places of removed spots?
May you propose any algorithm?

[tedkunich]

Fizick,

Probably removing color spot only at places of detected luma spots may be better.


That is exactly what I was thinking of. In my particular case (laser rot) the spots are only visible for one frame - would it be possible to replace the luma and chroma information of the detected spot with a blend or interpolation of frames from either side of the detected defect? I am processing pure progressive frames. Would it be helpfull if I sent you a few frame grabs or a small avi?


Thanks,

Ted

[Fizick]

New version of DesPot.
Changes 3.1 (June 27, 2004) from 3.0:
Add color correction at place of removed spots.

Parameter "color":
true - change color of pixels at places of removed spots to mean value previous, current and next frames,
false - not change color of pixels at places of detected spots.

Try!

[tedkunich]

Fizick,

You are the Man!!!! Looks like the filter works as advertized - I need to play around with the parameters, as I get some smearing around the spot removal, but the results are better than the previous verisons and certainly much, much better than nothing at all!!! Thank You.

If you ever get to the USA, look me up - I owe you a beer!!!!!


T


Originally posted by Fizick
New version of DesPot.
Changes 3.1 (June 27, 2004) from 3.0:
Add color correction at place of removed spots.

Parameter "color":
true - change color of pixels at places of removed spots to mean value previous, current and next frames,
false - not change color of pixels at places of detected spots.

Try!