Filter idea: SANWICH FILTER

[SILICON]

Sorry by my bad english!!!

The idea is for a simple temporal filter for bitrate and noisy reduction.

If one pixel in the previous frame and in the next frame is the same, but in the current frame is unsame, then the pixel in the current frame is noise.

The filter need test the pixel in the previous and in the next frame and charge it in the actual frame by the same value.

In C code:
Pp = pixel in the previus frame
Pc = pixel in the current frame
Pn = pixel in the next frame
T = threshold

If (ABS(Pp-Pn) < T)
{
Pc = (Pp+Pn)/2
}

ARTIFACTS
None artifact espected, because a change in one frame (1/30seg of time) is invisible to human eye.

But a change in one frame is very bad for compress MPEG code and make a bitrate increase

This filter to be very eficient for NTCS 30fps films and for anime films.

I can´t make filters for avisynth (I don´t know), ¿can somebody?

[bill_baroud]

have you _ever_ try to know more about filter like "TemporalSmoother" or "TemporalSoften2" ?

as far as i know, they do exactly what you suggest, no ?

(or i'm totally wrong and must hide myself deep in the forest? )

[slk001]

No, neither TemporalSmoother nor TemporalSoften2 will not do what SILICON wants. He wants a filter that only checks the value of a pixel between two frames (ie, A233 - B233 - C233, where we are evaluating B233) and compares it to the previous and the next frame's values. If it is outside the threshhold, then we are assuming that it is noise, and we are changing its value to the average of A233 + C233. The two temporal filters mentioned above would average all three. This would effectively remove most random noise (TV static, etc). However, this filter will also remove fast moving objects in the field (such as a sword slashing through a scene).

However, I believe that the noise reduction benefits far outweight the loss of this high speed info. It would mean that the filter would have to be applied selectively to video clips without the fast movements.

I think that this is a good idea for a filter.

bill_b don't hide yourself too deep in the forest - we may have to go find you someday!

[E-Male]

sounds like a simple but effective thing to me
someone whó is able should try to code it

[Si]

I've tried coding this (as an exercise in dealing with past and future frames) but it gives bad artifacts.

This probably due to some bug but here is a trial version.

Syntax
Sandwich(threshold,show)

threshold = any number
show = true or false (to show which pixels are being averaged)

I'm off to bed now so I'll get back tomorrow for any comments.
regards
Simon

[High Speed Dubb]

@SILICON,

You&rsquo;re suggesting something related to a temporal median &mdash; i.e., limiting the pixel value to the range of its neighbors. (Actually, I think a temporal median would look somewhat better than averaging between the previous and next pixel values.)

This should work well in stationary regions. But it will have two bad artifacts. It will turn dot crawl into a weird alternating stationary pattern. And (most imporantly) it will completely erase any motion which passes through a pixel for less than two full frames. Unlike slk001, I think the motion artifacts would look pretty bad.

[High Speed Dubb]

...and now that I think about it, this wouldn&rsquo;t fix NTSC chroma rainbows, either. It might work okay for PAL, though.

The reason is that the chroma carrier for NTSC is off by 180 degrees per frame. So the error at frame+1 and frame-1 will be about the same, and averaging them won&rsquo;t help.

With PAL, the carrier is off by 90 degrees per frame, so frame+1 and frame-1 will be off by 180 degrees, allowing for cancellation. On the down side, because the carrier is off by 180 degrees, it is likely that the two pixel values will differ enough that they will exceed the threshold for averaging.

[Acaila]

I could be wrong, but didn't trbarry do exactly this with his motion adaptive temporal smoother?

(Tom: sorry if I got the name wrong..)

EDIT: What I meant was the STMedian filter, but I think the thread is long gone by now as I can't find it anymore.

[SILICON]

Originally posted by slk001
This would effectively remove most random noise (TV static, etc). However, this filter will also remove fast moving objects in the field (such as a sword slashing through a scene).

However, I believe that the noise reduction benefits far outweight the loss of this high speed info. It would mean that the filter would have to be applied selectively to video clips without the fast movements.

I don´t expect artifacts whit this filter.

The fast moving odjects don´t lost. All change in one frame is 1/30 sec (1/25sec in PAL) in the screen. The human eye can´t see changes with less 1/24 sec. If the sword slading very fast, you cant see it.

The sandwich filter are good for:

In NTCS (no telecined) the frames are AABBBCCDDD. Whit sandwich filter al error in the midle B and in the midle D are detected and erase. I sugest use this filter before telecined film.

In animation films, where a lot of screen unchaged in the consecutive frames.

In static films, where the background change only a bit.

In static noise, whit random pixel changes.

Note: posible arifacts expect in interlaced film, if the filter test consecutive frames. In interlaced must test the even frames whith the even frames...

[sh0dan]

@Silicon: Changes at 1/30th second are visible, if they are big enough. If a moving object flickers, it will be visible.
Even telecined material can have three different frames in a row (you're thinking in fields, not frames - see http://www.lukesvideo.com/telecining2.html). Since there are there can be three consecutive frames (1T 2T 3T on the page above), you risk loosing information from 2T, if 1T and 3T are below threshold, but 2T has other information.


However! Your suggested filter is very useable when doing de-bobbing, to avoid interlace flicker (see 100fps.com).

Another option could be to maintain the first pixel, instead of averaging - if would definately improve compression, but artifacts will become more visible - but IMO it's better so SEE the artifacts and remove them, than having blended result.

@siwalters: Yes - the filter seems broken Would you mind also posting the source? I have some ideas I would like to try out.

[Guest]

The fast moving odjects don´t lost. All change in one frame is 1/30 sec (1/25sec in PAL) in the screen. The human eye can´t see changes with less 1/24 sec. If the sword slading very fast, you cant see it.This is simply not true.

The easiest way to prove it is to take a clip and change one frame to total white. Then play the clip at normal speed. The flash is easily seen. (DirecTV makes their lip sync test streams with these single frame white flashes.)

Think about your argument. Noise causes pixels to change every frame. If your argument were true, we wouldn't see the noise and your filtering would not be needed! Let's go to the extreme, we can't see clips at all when the content changes every frame. Not.

Just for fun, please cite your source for this silly idea.

[Si]

@Shodan
Just back in from work - give me a little time to spot the bug myself - I'd hate to have a stupid error published all over this forum :o :D

regards
Simon

[Si]

Tried modifing code - same result - so here it is with source for someone to decide whether its a bug or an algorithim problem.

(You can get it via my site until a moderator approves it)

regards
Simon

[High Speed Dubb]

Wow &mdash; I must have been reading two topics at once. This thread isn&rsquo;t at all about color crosstalk, but I went off on an unrelated tangent about it. My apologies to anyone who got befuddled. :)

@Acaila,

Tom has a temporal median as an extra in his Greedy High Motion (or something like that) deinterlacing filter. I think it has a sanity check which makes sure the current pixel isn&rsquo;t too far from the t-1 and(?) t+1 pixels.

[slk001]

MOTION ARTIFACTS?

High Speed thinks that this filter will give weird motion artifacts. Can you explain how these will occur?

Here's my understanding of how the filter would work. To illustrate, here is a "slice" from a video "stack" that shows the life of 10 pixels as they progress temporally through frames A through K.

__1234567890
A_xxxooooxxx
B_xxoooooxxx Flicker
C_xxxooooxxx
D_xxoooooxxx Flicker
E_xxoooooxxx Flicker
F_xxxooooxxx
G_xxooooooxx Changing scene
H_xoooooooox
I_oooooooooo
J_oooxoooooo Noise pixel
K_oooooooooo


As the code moves through this video slice, it will find the flicker in pixel B3 and average it to the "x" value of pixel A3 and C3. The flicker in pixel D3 and E3 will not be touched, since the values in C3 and E3 are outside the threshhold, as are the values in D3 and F3.
The pixels in the changing scene will also not be touched, for the same reasons.

Noise at J4 will be replaced with the average of I4 and K4.

Maybe my example is too simplistic, but I don't see where motion artifacts can appear. When any motion appears, the threshhold value is exceeded, so the filter does nothing.

[High Speed Dubb]

Problems will show up with motion when a feature is at a pixel for just a single frame.

So in your example, replace line J with
xxxxxxxxxx

All of the motion in that frame would incorrectly be removed as noise. (Also, I1 and I10 would have somewhat odd values if H1 and H10 are similar enough to J1 and J10.)

[slk001]

I agree that high-speed motion will get averaged out and that is why I said earlier that this filter would have to be selectively applied to "low-motion" scenes only, but I still don't see the generation of any artifacts - just maybe "no motion" (where some should be).

The odd values that you mention would have been odd in the first place. Anyway, setting the threshhold properly would prevent this from happening.

[vlad59]

@siwalters

As Dividee already told me (;) ) : you can't assume that all your frames use the same pitch. So you should use the GetPitch function on all your frame.

But IMHO, it shouldn't explain the artefacts (the algo cause it) since I imagine you have tested your filter on an uncropped source.

Hope this helps.

[Si]

@Vlad59
Thanks for the tip.

I'll make sure I do that in future.

regards
Simon

[Si]

I've added a Limit parameter so you can ignore situations like
High Speed Dubb mentioned
So in your example, replace line J with
xxxxxxxxxx

Sandwich(Threshold,Show,Limit)

pseudo code shown below (fp=prev frame,c for current, n for next)

for (int x = 0; x < frame_row_size; x+=1)
....int fpfndiff = abs(fpsrcp[x]-fnsrcp[x]);
....if (fpfndiff < Threshold)
.......int fpfnavg = (fpsrcp[x]+fnsrcp[x])/2;
........int fpfndifffc = abs(fpfnavg-fcsrcp[x]);
........if (fpfndifffc < Limit)
............dstp[x] = fpfnavg;
............if (Show)
................dstp[x] = 128;
........else
............dstp[x] = fcsrcp[x];
.....else
........dstp[x] = fcsrcp[x];
next x

available via my site

regards
Simon

[slk001]

Hey, siwalters,
How about giving us the range of valid values for your filter parameters?

Thanks

[Si]

Its not my filter - I'm just coding it - its SILCON's :)
You can have any integer values (-2,000,000,000 to + 2,000,000,000 I think) you want to try :p

regards
Simon

[High Speed Dubb]

@slk001,

I think we&rsquo;re using different definitions of &ldquo;artifact.&rdquo; To me, it means any errors which are caused by the algorithm.

Changing the threshold won&rsquo;t have any effect on the fast motion problem. It only checks whether t-1 and t+1 are similar &mdash; not whether t=0 is similar to either of those. Simon&rsquo;s limit parameter should mostly avoid the problem, though.

[trbarry]

EDIT: What I meant was the STMedian filter, but I think the thread is long gone by now as I can't find it anymore.

Yeah, the STMedianFilter does a similar thing, both spacially and temporally, but it clips a pixel value to be within the range of it's prev and next neighbors instead of taking the average as suggested above. It was discussed (and spawned) in that "Smooth in Space and Time" thread.

It's at:

www.trbarry.com/Readme_STMedianFilter.txt and
www.trbarry.com/STMedianFilter.zip

It has an additional sanity check that allows a limit to how much to change a pixel. If a pixel would have to change by more than a small amount it assumes it should be left alone. This avoids artifacts such as disappearing flagpoles and such.

Lindsey - I disabled the median filter from the Avisynth version of GreedyHMA since it required a delay of 2 fields that was not compatible with adding the IVTC logic. I always meant to put it back but haven't, and under Avisynth it is probably just better to add STMedianFilter, or others.

- Tom

edit: And, yes, beware that filters like this can indeed cause artifacts if you crank them up too high.

[Wilbert]

in that "Smooth in Space and Time" thread
In that same thread (one of the last posts) I tested your plugin and compared it with other filters. Your plugin gave me a purple hue in my clip. Pictures and more info can be found in that thread.

[SILICON]

[QUOTEJust for fun, please cite your source for this silly idea. [/QUOTE]

Ok, Ok. It is not true (every times). I´m lose :-(

It is true only when the lost pixel are more black the the neibourgs
For example.

T-1 = 255
T = 20
T+1 = 255

if I make t=255 I don´t see the change.

But if

T-1 = 25
T = 255
T+1 = 30

and make T= 255 by T= 20 them you see the change.


I think one solution for this problem:

The noise thar I want clear is erroneus bits, static, etc. This noise are velry little points.
- Cut the frame in boxes of 16x16.
- Make the actual algoritm in the first box
- If in the box changes more of 32 pixels then don't change anything
in this box.
- Repeat whit the next box.

This clear the random and little noise, but don't lost the fast sword (by example).

This clear the noise whith high levels but not blur the image.
The another filters don´t clear the noise pixels idf they are very diferente of the neiborg pixels.

I test the actual sandwich.dll v1.1 and work well with the threshold set to 1 or 2. With high threshold make artifacts in fast action. Whith the change of boxes i think that remove this artifacts.



Thanks for coded

[Si]

Updated version inc SILCON's 16x16 pixel count idea available via my site

Parameters now are
Sandwich(int Threshold,bool Show, int Limit, int Mode, int ChangedPixels)

Mode = 0: Original operation
Mode = 1: If number of bytes (4 bytes = 2 pixels) in a 16x16pixel block > ChangedPixels then the filter just passes through the current pixels for that block - i.e. no Sandwiching.

Mode = 2: If number of bytes < ChangedPixels then block passed through unchanged i.e no Sandwiching (opposite of Mode = 1)

If Show = true then the blocks that will be passed through unchanged will be coloured Purple in Mode 1, Green in Mode 2.

If the frame dimensions are not a multiple of 16, then the bottom and right borders are just passed through unfiltered.
regards
Simon

[E-Male]

maybe i´m doing something totally wrong but the filter makes no difference on my video, i tested with subtract and a very hight threshold, nothing :(

[Si]

Try
Sandwich(64)

You should see some artifacts - if not -you've got the perfect clip that this filter has been looking for :D

regards
Simon

[E-Male]

strange
i used more parameters
with only 1 it works
could you please give some examples and tell what the filter does with each?

[Si]

This filter is just an experiment for Silicon's ideas and an exercise in myself learning Avisynth plugin writing techniques - I don't think it works very well as any sort of useful filter :) (And it may have bugs as well :o )

Parameters of V1.2

Sandwich(int Threshold,bool Show, int Limit, int Mode, int ChangedPixels)


Mode = 0: Original operation

Threshold - If difference between a byte in previous frame and the next frame is less than Threshold, then the current frame byte is replaced by the average of the previous and next frame i.e Sandwiched :)

Show - shows which bytes/pixels are being affected by the filter for testing purposes.

Limit - If the difference between the average (of the previous and next frame byte) > Limit then the byte isn't Sandwiched - it is left alone.

Mode = 1
As per previous post - 21st September 2002 20:59 but you still have to set a Threshold :D

[E-Male]

got it working, but won´t use it for my current project, as the movie hardly had steady cams in use

also there are still are vissible errors sometimes, but i haven´t tried the new modes yet

[SILICON]

I feel it, but you have not understood my idea. Surely because of my badly English

I want clear the little noisy (i.e. the stactic noise(snow), fly noise, etc..)

For this, check the same pixel in the previous and following frame. If the pixels are
equal we will suppose that the pixel of frame present is noise.

Then Sauron said that to this algorithm will clear the fast movements, like a sword.

In this case I suggested of dividing the image in pictures of 16x16. An sword will change a
great number of pixels in a little pictures (in the present frame), although next frame and the previous one was equal.

I think that you must count the pixels that change in present frame, but:
- If the pixel in frame previous are diferent of the pixel in frame following, then the present pixel don´t change. (We will not change it)
- If a pixel is very similar to the one of frame previous or frame following, then it does not change. (We will change it, but we will not count it)
If the pixels that change in the present frame (don´t count the pixels taht changed to similar value) are high, then don´t change any pixels. We have a sword :-)

For don´t mistake anymore, I send you one example.



Example of code:

They are the imput frames. The frames in the example are 4*4 for simplicity
Previous Frame
a(0,0), a(1,0), a(2,0), a(3,0)
a(1,1), a(1,1), a(2,1), a(3,1)
a(1,2), a(1,2), a(2,2), a(3,2)
a(1,3), a(1,3), a(2,3), a(3,3)
a(1,4), a(1,4), a(2,4), a(3,4)

Actual Frame
b(0,0), b(1,0), b(2,0), b(3,0)
b(1,1), b(1,1), b(2,1), b(3,1)
b(1,2), b(1,2), b(2,2), b(3,2)
b(1,3), b(1,3), b(2,3), b(3,3)
b(1,4), b(1,4), b(2,4), b(3,4)

Next Frame
c(0,0), c(1,0), c(2,0), c(3,0)
c(1,1), c(1,1), c(2,1), c(3,1)
c(1,2), c(1,2), c(2,2), c(3,2)
c(1,3), c(1,3), c(2,3), c(3,3)
c(1,4), c(1,4), c(2,4), c(3,4)

Limit = 8 // the max chaged pixels
Threshold = 2 // the threshold for say that two pixels are equals.

// count the pixels I want changed.
int DifferentPixels=0;
for(int row=0; row<4; row++)
for(int col=0; col<4; col++)
if(ABS(a[row,col]-c[row,col]) < Threshold) // is the previous and following pixel equals ?
if(ABS(((a[row,col]+c[row,col])/2) - b[row,col]) < Threshold) // the present pixel is equal to previous and the following one
DifferentPixels ++;
next
next


if(DifferentPixels < Limit)
for(int row=0; row<4; row++)
for(int col=0; col<4; col++)
if(ABS(a[row,col]+c[row,col]) < Threshold)
b[row,col]=(a[row,col]+c[row,col])/2 // changing the pixel
next
next

[High Speed Dubb]

Using big blocks to identify areas of fast motion is a nice idea. You could have problems on the edges of the blocks, but for the most part it should correctly figure out where there is fast motion.

[Si]

Here is v1.3 based on SILICONs code.

I am having real problems with showing the processed pixels (Show=1) and showing the non-processed pixels (Show = -1) so I've given up for them moment and just published where I'm at :(

The current parameters are

Sandwich(Clip,int Threshold,int Show,int Limit,int Mode)

Show is now int not boolean.

Limit is used when Mode=1 instead of a ChangedPixels parameter as per SILICONs pseudo-code.

There is no Mode = 2 in this version.

Show = -1 should show the pixels not processed.

Download available here and via my site.

regards
Simon

[medp7060]

Any download link for Sandwich 1.3? I searched but no luck.

[Guest]

http://www.geocities.com/siwalters_uk/sandwich13.zip

You'll need to rebuild it for Avisynth 2.xx.

[medp7060]

Thanks. I visited the Geocities but could not find a link for it.

However, I could not get it work to test it since I got "no function named Sandwich" error. It may be due to its imcompatibility with Avisynth 2.5X, even I tired LoadPluginEx.dll.

In my clip, the noises could be removed by simply replacing the pixes from previous or next frame, I thought Sandwich might do that.

[Guest]

Stand by...I'll rebuild it for you.

[medp7060]

many thanks for that.

[Guest]

http://neuron2.net/misc/sandwich13.zip

I didn't check what color space it accepts, so read the thread, I suppose.

[medp7060]

Got it, it works with 2.5x. I am playing with it. Thanks again.

[Lyris]

Sorry I can't add much to the filter discussion, but:
The human eye can´t see changes with less 1/24 sec
I would love to know where this comes from, because I keep hearing it over and over again, and it's total nonsense - at least for my eyes!

[um3k]

This thread is almost eight years old...