high motion blurring to save bits - how to generate?

[kilg0r3]

hi,

i am not a avisynth filter pro so i need your help to realize an idea. i noticed that divx like nandub saves a considerable amount of bits by blurring high motion scenes. since i like the detail xvid maintains, i would like to reproduce the same effect with a filter. does any body know of a filter that starts to smooth when a certain threshold of movement has been reached? of course it would be even bettere if the smooting would increase incrementally instead of just swiching on and off.

thanks in advance for any suggestion

[-h]

I'm working such a filter at the moment, though it'll probably start out by using a higher threshold for the internal temporal smoother, rather than using a spatial smoother for large motion vectors.

I guess that'll find its way into the code eventually.

-h

[Leuf]

Originally posted by -h
I'm working such a filter at the moment, though it'll probably start out by using a higher threshold for the internal temporal smoother, rather than using a spatial smoother for large motion vectors.


Wouldn't you want to do the opposite? A high threshold for a temporal smoother in motion areas will leave a lot of artifacts, whereas a spatial won't.

[-h]

You may be onto something old bean..

-h

[OUTPinged_]

-h, you mentioned some time ago you are going to build a temporal softener with a ME engine. Will it be released as avisynth plugin or as a codec feature?

[-h]

It will be a plugin, not part of the codec (unless I go mad).

-h

[trbarry]

How about also having a little positional blurring? Directors put most all the action and interesting detail in the center portion of the screen. Maybe it would be usefull to blur more toward the edges. In moderation maybe nobody would notice.

- Tom

[kilg0r3]

i am quite sure that we need both temporal and spatial blurring. a limited amount of ghosting is in vital here. spatial softening only will not save too many bits. of course, one has to be very careful with the threshold.

softening on the chroma plane should be very strong. for, even in low motion scenes, one looses not too much detail when applying extreme thresholds on the chromaplane only. i once experimented with settings as high as 255 in convolution3d. the result was not pretty but in fast motion there was nearly not any visible loss in quality. in a situation were there are quick changes in the picture the eye will not have too much time to deal with chroma distinctions.

another good thing would be to include an edge detection algorithm to exclude them from being blurred. I assume that in high motion situations the eye (rather the visual cognitive system) attaches itself to elements which it can recognize most easily. edges are such elements.

@ -h
probabaly marcfd's MAM could be of use since it also usesmotion detection

[iago]

@kilg0r3

I agree with you that both temporal and spatial smoothing being applied to motion parts will be really useful in terms of both saving bits and preventing blockiness (with some vital ghosting effect as you put it). Currently I'm doing some tests on this issue with MAM.

I guess, at the moment, (at least until -h releases the filter he's working on ;)) MAM can be a good solution to achieve this aim using, just for example:

for low motion parts -> TemporalSoften (or some other temporal smoother)
for motion parts -> TemporalSoften (or some other temporal smoother) + SmoothHiq (or some other spatial smoother)

MAM(temporal_smoother,temporal_smoother.spatial_smoother)

The parameters and filters to be used need a lot of testing of course, and I'd be glad to hear from those who have used the above method with MAM to get good results.

best regards,
iago

[Smiff]

[insert random idea]

has anyone thought of looking for skin colours (errmm.. faces?!) and giving them more bits?

It's _faces_ that humans are looking at, maybe 90% of the time, if there are faces on screen.... hmm :)

[/insert random idea]


EDIT: uh oh, this could be racialist, if it weren't looking for black faces, lol :D

[MaTTeR]

Originally posted by Smiff
It's _faces_ that humans are looking at, maybe 90% of the time, if there are faces on screen.... hmm :)

Not exactly true in my case, unfortunately the blocks that float around on walls or other stable objects in the background distract me from the real foreground objects:D I've been fighting this issue for months, especially now with Qpel.

[iago]

@MaTTeR

I know well from the XviD forum that you're a QPel sort of guy, not a B-frames one (just like me) ;). (However, my low system specs still prevent me from using QPel in my encodes.)

BTW, maybe you can give some fast tries (with your high/omnipotent processor power :D) for a couple of hi/lo motion mixed sources using some agressive settings (especially in terms of chroma) for the motion parts such as:

mam(TemporalSoften(1,5,0),TemporalSoften(1,10,100).SmoothHiQ(7,25,100,200,10))

? ;)

best regards,
iago

[MaTTeR]

Originally posted by iago
mam(TemporalSoften(1,5,0),TemporalSoften(1,10,100).SmoothHiQ(7,25,100,200,10))


OMG! That has to be the slowest script I've ever used, averaging 4FPS on a dual Athlon XP 1600 agressively tweaked. lol

Testing now and will report results later...at this speed it will be MUCH later. Were you wanting me to have B-Frames enabled as well? If so, what settings do you like for them?

[iago]

@MaTTeR

LOL! :D

I know such a script is not much usable, but it would be really nice to see its effect on motion parts especially, since a setting of TemporalSoften(1,5,0) can be considered almost secure to not hurt details in low motion parts. The important part is, as the title of the thread suggests, the hi-motion parts ;). And no, please don't use B-frames ;).

BTW, currently I'm testing

---------------------------------------------------------------------
LoadPlugin("C:\FILTERS\MPEG2DEC3.dll")
LoadPlugin("C:\FILTERS\SmoothHiq.dll")
LoadPlugin("C:\FILTERS\mam.dll")
LoadPlugin("C:\FILTERS\UnFilter.dll")
mpeg2source("D:\MATRIX-TEST\MATRIX.d2v",lumoff=-2)
crop(4,84,712,408)
mam(TemporalSoften(1,5,0),TemporalSoften(2,10,150).SmoothHiq(7,30,150,200,20))
LanczosResize(640,256)
UnFilter(3,3)
---------------------------------------------------------------------

with Matrix Chapter 2, which I find very suitable for such a purpose. LOL! It averages ~ 2fps on my Celeron 900!.. Not much usable, but it's worth to see the results I guess ;).

And, really many thanks for taking my words seriously and starting a test encode.

best regards,
iago

edit: Come on Matt, of course it doesn't need to be a full-movie to be tested! ;) Just one chapter with both motion and low-motion scenes would suffice I guess to see the results.

[MaTTeR]

@iago,

I'm still laughing when I look up at the Vdub window and notice the speed. Amazing what one little script can due to CPUs':)

Of course I'm not testing a full movie, its a 12min clip from Pitch Black which contains alot of high motion and some slow motion scenes. I'm anxious to see the results of this myself.

LoadPlugin("C:\Program Files\AviSynth2\plugins\MPEG2DEC3.dll")
mpeg2source("D:\DIvX RIPs\Test Clips\pitch black rip\pitch.d2v")
crop(4,58,714,362)
mam(TemporalSoften(1,5,0),TemporalSoften(2,10,150).SmoothHiq(7,30,150,200,20))
BicubicResize(624,256,0,0.5)

[iago]

@MaTTeR

In my test encode with Matrix Chapter 2 (constant quantizer 2 / h263 quantization / Koepi's 04102002-1 build), such aggressive settings for high-motion parts didn't cause any visible problems.

Just another idea:

TemporalSoften -> for low motion
Convolution3d -> for high motion (since C3D can do both temporal and spatial smoothing on both luma and chroma planes)

I guess that would be faster and more usable ;). Btw, I started testing:

LoadPlugin("C:\FILTERS\MPEG2DEC3.dll")
LoadPlugin("C:\FILTERS\Convolution3d.dll")
LoadPlugin("C:\FILTERS\mam.dll")
LoadPlugin("C:\FILTERS\UnFilter.dll")
mpeg2source("D:\MATRIX-TEST\MATRIX.d2v",lumoff=-2)
crop(4,84,712,408)
mam(TemporalSoften(1,5,0),Convolution3d(1,16,32,16,32,3,0))
LanczosResize(640,256)
UnFilter(3,3)

which doubled the previous encoding speed here. Let's wait, test more, and see the results! ;)

regards,
iago

[Acaila]

mam(TemporalSoften(1,5,0),TemporalSoften(2,10,150).SmoothHiq(7,30,150,200,20))Seriously, the radius of 7 on SmoothHiQ is only useful for slowing everything down a lot. Spatial filtering hardly affects compressibility so 5 would be more than enough for your purposes. SmoothHiQ's speed decreases greatly when increasing the radius.

Convolution3D does seem the better choice in this combo since it reduces two filters down to one filter. Although it will still be quite slow.

Might wanna test with SimpleResize instead. You're determining the effect of blurring filters, the resizing method is irrelevant for these tests. Running tests with the extra slow LanczosResize is just making it harder on yourself :).

[iago]

Acaila,

Yes, theoretically you're right regarding resize filters, but strangely, here on my weird system ;), SimpleResize, LanczosResize, and BicubicResize(0,0.5/0.6) (which are the only ones to be used imho, not to introduce an extra softening effect on the resizing side) average almost the same fps without much difference with the above mentioned filter chains ;).

And well, the combo has already changed to:
-------------------------------------------
mam(TemporalSoften(...),convolution3d(...))
-------------------------------------------
which seems really more reasonable.

regards,
iago

[MaTTeR]

Just got my test results and visually I must say the mam filter chain was slightly too soft for my eyes though I might be able to tweak it slightly for more sharpness. It appeared as though the low motion scenes looked best while the high motion was either too soft or blocky.

MAM script-
Average Quantizer Used for Movie : 3.921
Compressibility : 39.47%
Relative Quality of XviD avi : 51.00%
Absolute Quality of XviD avi : 94.24%

Non-MAM_TemporalSoften(1,5,0) script-
Average Quantizer Used for Movie : 3.856
Compressibility : 38.68%
Relative Quality of XviD avi : 51.87%
Absolute Quality of XviD avi : 94.43%

Both of the encodes were done with Motion Search 6, modulated @ 624 resolution. I didn't notice any strange artifacts either except for the normal Qpel smearing noise problem. Back to the drawing board for now:)

[iago]

It appeared as though the low motion scenes looked best while the high motion was either too soft or blocky. (MaTTeR)

MaTTeR,

I agree with your above observation. However, softness only wouldn't annoy me actually for high-motion scenes, if it wasn't mostly accompanied by some sort of disturbing blockiness. That's what I noticed in my tries with convolution3d for the high-motion parts as well.

regards,
iago

[trbarry]

has anyone thought of looking for skin colours (errmm.. faces?!) and giving them more bits?

On TV it is actually the exact opposite. There is a blur filter that will optionally soften skin tones to make the actresses look younger and healthier. An appearence of smooth skin is greatly desired. This has been an issue in the move to HDTV. ;)

- Tom

[Marc FD]

lol :D
you guy are using the slower filter on earth ^^

is convolution3d really the faster spatialsmoother you've founded ??

[iago]

Marc,

Yes, since convolution3d does both temporal and spatial smoothing in one, it's really faster than the previous (TemporalSoften.SmoothHiQ) combo for the high motion parts in mam ;).

LOL :D

iago

[cjv]

Digging up a semi-old thread...I, unlike iago and others have had great success using:
MAM(unfilter(0,0),Convolution3d(1,128,255,0,0,32,0))
(on SpyGame R1). I am saving approx 2+megs/minute...which translates to approx. 240 megs/movie. I don't notice any blocks in high motion...but I'm using no temporal smoothing at all.

note: I am using c3d with no temporal due to the fact that SmoothHiQ() results in _nasty_ artifacting..almost cartoon-like on my box. :) Unfilter(5,5) is MUCH sharper and much better in lo-mo if enough bits are allocated but for this instance, I'm using unfilter(0,0) which is a void() effect.

cjv

[Marc FD]

i'm happy to see MAM helps, even if it's crap ^^

i'm working on things who could help to blur high motion, to clean slow motion and to have sharp mid motion like RV9 does. i hope it'ld work the way i want ^^

[Rrrough]

@Marc

hi, any plans of converting MAM to YV12 ? or did u give up on it ?
would it be possible of modifying MAM to make it possible using user-specified motion thresholds (like ">low motion value< = use filter a; >mid-motion value< = use filter b; >high motion value< = use filter c" and so on). this is just a hypothetical question, not a filter request (yet ;) ).
sorry if these questions have been answered already, didn't find with search.

cheers

[Marc FD]

> hi, any plans of converting MAM to YV12 ? or did u give up on it ?

hmm. the MAM concept is too basic.

> would it be possible of modifying MAM to make it possible using
> user-specified motion thresholds (like ">low motion value< = use filter
> a; >mid-motion value< = use filter b; >high motion value< = use filter
> c" and so on). this is just a hypothetical question, not a filter
> request (yet ).

it's impossible, because to know the _speed_ of motion, you need to compensate it. and if you compensate it, it's better to use temporal compensed filtering, no ?

in fact, i've wrote a little filter who was doing simple motion compensation to temporally clean still scenes, pan&zooms, and was using spatial cleaning/blur on high-motion. thinking of it, it worked quite good ^^.

[Rrrough]

it's impossible, because to know the _speed_ of motion, you need to compensate it.so MAM isn't motion compensated ? caution - n00b question about - what's the difference between motion compensation and motion adaption (like in MAM) ?
would it be possible to put motion compensation into it ? (=> MCM, not MGM ;))
Actually, I like the idea behind MAM, that you can apply any filter (resp. filter chains) you want for low resp. high motion, but it would be nice having a larger field to chose from (e.g. low, mid, high motion...)
in fact, i've wrote a little filter who was doing simple motion compensation to temporally clean still scenes, pan&zooms, and was using spatial cleaning/blur on high-motion. thinking of it, it worked quite good ^^. sounds great :)

cheers

[Marc FD]

> so MAM isn't motion compensated ? caution - n00b question about - what's
> the difference between motion compensation and motion adaption (like in
> MAM) ?

MAM adapts to motion like a temporal smoother :
if pixels are close between 2 frames : no movement
if there are different : movement.

> would it be possible to put motion compensation into it ? (=> MCM, not > MGM )

like i said, motion compensation is very hard to do.
- first, motion estimation techniques are complex and consuming a lot of CPU power (that's why XviD isn't as fast as MJPEG)
- second, it's far form perfect, and very hard to use (and to code !!)

BUT it's _VERY_ powerfull, and can help in many, many cases...

if MCM was able to estimate mid-motion, what would you do ?? temporal smoothing would create ghosting, and spatial filtering will be very noticable.

it's better to use compensated temporal filtering.

you see why MCM would never exist now ?

[SansGrip]

Time for my shameless plug of the day: did you see the NoMoSmooth (http://forum.doom9.org/showthread.php?s=&threadid=37471) thread? It's my stab at a spatio-temporal motion adaptive smoother.

:D

[Marc FD]

seems to be very close to STMedianFilter ^^

is it able to filter panning and zooms ? (temporally of course)

[SansGrip]

seems to be very close to STMedianFilter ^^

Really? I'll have to take a look at that before continuing :).

Edit: STMedianFilter uses a significantly different algorithm. Also with NoMoSmooth there's no possibility of ghosting. On I go ;).

is it able to filter panning and zooms ? (temporally of course)

Well, during pans and zooms moving edges are spatially filtered while the rest is temporally filtered.

[Marc FD]

> Well, during pans and zooms moving edges are spatially filtered while
> the rest is temporally filtered.

mmhh.. okay i see, it's like MAM with map denoising.

[Rrrough]

if MCM was able to estimate mid-motion, what would you do ?? temporal smoothing would create ghosting, and spatial filtering will be very noticable. I was thinking of a more adaptive way to use denoise filters (raising denoising values with increasing detected motion). so that's probably a bad idea...?
also I was reading the thread about using different resizing methods used with different motion, but didn't try out yet. I think it was possible with MAM. I was thinking of different bicubic resize with different motion (although I love lanczos3 for low-mo scenes, but it gives huge filesizes in hi-mo scenes).
but well, if it's a bad idea after all...

@Sans Grip

did you see the NoMoSmooth thread? yep, will give it a try asap, as you have this lovely YV12 version now !!!

cheers

[SansGrip]

it's like MAM with map denoising.

I knew there'd be a technical term for it. Do you happen to know the pros/cons of this technique?

[SansGrip]

I was thinking of a more adaptive way to use denoise filters (raising denoising values with increasing detected motion).

There's some scope for that in NoMoSmooth. Right now it uses a simple cut-off threshold between "motion" and "no motion", but it could almost certainly be modified to weight the strength of the filtering based on how much motion it sees.

Edit: If you're interested, we should probably take this to the NoMoSmooth thread ;).

[Rrrough]

it could almost certainly be modified to weight the strength of the filtering based on how much motion it sees. would it make sense then, after all ? isn't that the way RV9 and WMV9 are trying to save bits ?

cheers

[SansGrip]

would it make sense then, after all ? isn't that the way RV9 and WMV9 are trying to save bits ?

To me it makes sense to smooth areas in motion, since they're already blurred. But I know nothing about RV and WMV.