Alternative to temporal smoothing

[redfordxx]

Hello,
I'd like to ask for help with folowing thing (with anime --- not sure whether applicable on regular movies to the same extent):

Sometimes there is a part of movie, where is motion on totally fixed background. But in the encoded video the background is not totally fixed: there is noise, jumping blocks etc caused by the original source, or low quality encoding.

It is annoying and moreover it eats bits in following compression. Temporal smoothing is one solution for this, but how about completely replace the should-be-same area with the first occurence of that area after scenechange? (based on some threshold and mask)

The problem is with creating the mask. I see two possibilities: the easy one and the correct one (per pixel formula):
easy=Abs(current_frame - previous_frame)>threshold ? 255 : 0
correct=Abs(current_frame - repaired_previous_frame)>threshold ? 255 : 0

Example --- let's observe one pixel in time and see the mask values for easy and correct formulas (threshold=1). The repaired value is the original value or the first-occurece value, depending on the mask.
The values can be e.g.:

original 130 120 10 11 10 9 10 9 11 12 13 14 15 16
easy_mask 255 255 0 0 0 0 0 255 0 0 0 0 0
correct_mask 255 255 0 0 0 0 0 0 255 0 255 0 255
easy_repaired 130 120 10 10 10 10 10 10 11 11 11 11 11 11
correct_repaired 130 120 10 10 10 10 10 10 10 12 12 14 14 16



The question is, how to implement the correct way of processing?
Or is there any filter which makes it already?

Thanks for any suggestions.

R.

[foxyshadis]

I hope clouded sees this, because his CelBackground filter has very similar goals. I think it uses the information in all the frames of a scene to regenerate the background in highest quality. It had quite ambitious goals.

As it is, you got me thinking about what would be a good way to denoise better than dup's blend. and standard spatio-temporal smoothing.

[neuron2]

Jim Casabura's VirtualDub filter 2DCleaner implemented an idea he called pixel locking, such that a pixel's value is clamped until it changes more than a certain amount. It might be worth looking into.

[Mug Funky]

i think i tried that once. i can't remember if it worked...?

just for fun i'll try re-implement it.

btw, straight after scenechanges is probably not the best idea. often the most encoding junk is to be found there.

[edit]

it also occurs to me that it can't just be "pixel locking" because that could in fact hurt compressibility. if it worked on overlapped 8x8 blocks it might just be worth doing.

btw, this would be highly difficult to script... if a frame could be held in memory and manipulated several times (framebuffer? or is that a different thing?), it would be perfect. would also help with motion-compensated denoising.

[redfordxx]

OK, I will try to find what's from clouded.

btw, straight after scenechanges is probably not the best idea. often the most encoding junk is to be found there.yes, I noticed... probably when the scenechange is not keyframe, there is not enough bitrate...[/QUOTE]

it also occurs to me that it can't just be "pixel locking" because that could in fact hurt compressibility. if it worked on overlapped 8x8 blocks it might just be worth doing.or averaging few frames on pixel level and that average lock, that could be ok...?

btw, this would be highly difficult to script... if a frame could be held in memory and manipulated several times (framebuffer? or is that a different thing?)exactly that is where I stopped... some variables as pointers to frames...

Other crazy idea... make it multi pass:
First make small clip with background pattern
- reduced number of frames, for each scenechange only one frame
- save to some text file info, how many times should be specific frame repeated (I hope some conditional filtering enables that)
- encode it high quality / losslessly (that is why highly reduced number of frames needed)

Next pass maskmerge original and pattern clip (reconstructed using the lossless encode and text file...)



Whatever implementation --- lot of memory and time....

[mg262]

Looks like I should clarify things... this is going to be reasonably long (hope no one minds).

Cel Background is working. I don't have shots from the latest version, but here is something to give you an idea:

Example source frame (http://people.pwf.cam.ac.uk/mg262/posts/Bridge%20source%20frame.png)
Reconstructed background (http://people.pwf.cam.ac.uk/mg262/posts/bridge%20pre-sharpen.png)
Sharpened reconstructed background (http://people.pwf.cam.ac.uk/mg262/posts/bridge%20post-sharpen.png)

The background is constructed from all 136 frames in the scene, presently by averaging although much better methods are forthcoming, and in my biased opinion it kills all the noise at the cost of some blurring... although @TBL has convinced me that there was less detail lost than I thought (he added noise to the reconstructed background and it looked considerably sharper). I discovered last week that Didée's beautiful LimitedSharpen, while not too effective on the original (too much noise), worked remarkably on the reconstructed background, as shown in the third screenshot. There's much more to say about the shots... but not here.

So -- practicalities: This filter hasn't really been released yet. This is why:btw, this would be highly difficult to script... if a frame could be held in memory and manipulated several times (framebuffer? or is that a different thing?), it would be perfectIt's not easy to code in AVISynth either -- the backgrounds of different scenes can't easily be returned by a filter, because a) different scenes have different sized backgrounds and b) the size of the background is not known at filter-initialisation time.* AVISynth 2.5 can be pushed into doing it... but it is messy.

Cel also needs awareness of scene breaks, classification of scenes, storage of booleans/integer/floats per frame, storage of a motion vector for each pair of frames in a scene, storage of confidence estimates for most of the above, 15 bit colour depth, transparency in a YUV colour space, and compact storage of foreground regions of uniform colour. AVISynth has been pushed into doing all of these... but it's becoming increasingly awkward to code and causing additional script complexity which puts people off. Added to the fact that the only nontrivial things used from the AVISynth world are MaskTools, LimitedSharpen (which I'm recoding for speed anyway) and Resizers, it looks increasingly like it's time to make a move.

Unlike e.g. Motion, Cel is primarily for personal use. I've tried many solutions that fit the AVISynth model more naturally and are easier to use. None worked (well enough). I have many more dead filters on my HDD than released filters. Someone else might build filters of this kind inside AVISynth -- but I won't, and the same probably goes for @TBL. [What I build may not be of interest to others, as a) Like IsoChroma but noone else, I work in spf territory rather than fps if it squeezes out even a little quality, b) it will be complicated and c) I'm unlikely to have time to document it thoroughly.] Since Cel is my main video project, I'm less active here than before (though I'm certainly not disappearing completely!) -- but do always feel free to PM me if I can help with anything.

[morsa]

Well, my main interest is not always speed but quality.
And BTW, my main stuff is live action not anime :(

PS: If Cel works good enough with real life stuff/live action I guess it could easily become the basis for a re-coloring/ color repair/damaged film repair set of filters...
I say this because if we can isolate moving objects from the background much of the work will be piece of cake..

[Isochroma]

Hahaha! Yes, it just happens that I am salivating over this new filter of yours!

You idea strikes me as excellent, though the messiness of implementing it probably means waiting for AviSynth 2.6 or even 3.0. Until then, thanks for all the hard groundwork which will undoubtedly become part of the framework of the future...

[Mug Funky]

again, cel background/foreground is impressing me. i've seen PhD theses that cover less brain-bending territory...

and considering the large back catalogue of anime out there that would do well to be restored (properly... i've done a little but it certainly didn't go far enough due to time, budget and available tools. bosses want to release stuff like that just in time for christmas), your tool would be well received in whatever form it takes. think of all those early anime/cartoons out there that only exist as a mass of bootlegs and a few scraps of fungus-covered, filthy and scratched film. and most of it has cultural significance too.

when i think of what cel foreground/background could have done for the somewhat degraded astroboy sources i've had to work with, i kinda wish i had a time machine :)

[edit]

just whatever you do, don't let FUNimation get hold of Cel. that would mean yet another full re-release of all the Dragonball series. the world could certainly do without THAT...