Some questions regarding to scene detection

[prokhozhijj]

Is it possible and how better to do it:
1. How to determine if current frame is static or dynamic? I need some number for this. For example 0 will mean static frame and 100 - for the frame with a lot of movement in it.
2. I need some number for contrast measuring. Not just max(luma) - min(luma). But something like statistical dispersion.

Thank you for any advice.

[yup]

prokhozhijj!
Try MVTools2 at http://www.avisynth.org.ru/mvtools/mvtools2.html
MSCDetection make this.
yup.

[prokhozhijj]

prokhozhijj!
Try MVTools2 at http://www.avisynth.org.ru/mvtools/mvtools2.html
MSCDetection make this.
yup.

I know about these tools. This is not exactly what I need. Not scene detection but some metrics about current frame are needed.

[Didée]

I reckon your request is too vague. What exactly do you want to measure? Moreover, the question "what for?" is directly coupled to this.

It seems you generally want to measure the amount of motion. This question keeps coming up since I'm around on this board, and still there is no final answer. And why is that? It is because the question is not specific enough.

a) One can measure the frame-global pixel difference to the neighbor frames. This can work out, or it can not. E.g. on a plain camera panning, the pixel differences are substantial. However they also depend on the contrast of the scenery. A scene with high contrast will show big global difference. A dark scene with low contrast will show rather small global difference in comparison. In particular, a high-contrast scene with only moderate motion may show bigger global differences than a low-contrast scene with lots of motion.

b) Using MVTools and evaluating the length of vectors as a representation of amount of motion. This is much more sophisticated, sounds much "better" at first glance, but brings up different kinds of problems. Quite often the situation is that some parts of a frame are static, other parts contain motion. Working with per-frame metrics will be problematic. Average birth rate in Germany is 0.6 children per family. But you cannot send a 0.6-child into school. You see what I mean.


There are many, many different metrics you can gather. The Q is which kind of metric will fit to what you're going to do.

On the other hand, if you're *sure* about what you want to do, then it should be obvious which metrics are suited and which are not ....

[Mug Funky]

i think currently the state of scenechange detection is a 2 pass thing.

the problem is every video is different, so you almost need to sample the entire clip and analyse the results to be able to reliably detect what is a scenechange, what is just jerky motion or flashbulbs, find the cuts in dark sequences where there's little difference, etc.

MVtools is probably a good way, but the catch is the processing time starts to approach the time it takes to go through and chop up the clip yourself, and is possibly less reliable.

this is one of the problems i'm thinking about a fair amount though. i'll make a post if there's a breakthrough.

[Didée]

When it's about detecting scenechanges, then the speed/reliability ratio of SCSelect() is hard to beat.

Point is, the OP did not ask about scene change detection. :p ;)

[prokhozhijj]

I am asking about these metrics for my improved (as I think :) ) deflicker algorithm. Not for scene change detection. The more motion in current frame - the less strength of deflicker. The more contrast in frame - the more strenght of deflicker. Something like that, in general.

Sometimes flicker occurs which lasts more than one frame. And I need to know if I can to apply in this case strong deflicker or not. Under term "strong" I mean long frames sequence for the low frequency filter.

[ganymede]

Average birth rate in Germany is 0.6 children per family. But you cannot send a 0.6-child into school. You see what I meanA little bit off-topic...
Birth rate can be defined as the nativity or childbirths per 1,000 people per year. According to the UN, the birth rate of Germany was 8.2 for 2005-2010. But I guess you meant fertility rate, which is the average number of children per woman. Fortunately for Germany, this country's fertility rate was 1.36 for the same period, not 0.6 - but this is still under replacement rate. BTW you cannot send 1.36 child to school.

[prokhozhijj]

As I understand it is impossible by means of current Avisynth's plugins to solve my problem. Am I right?

[Didée]

Impossible? No, not at all.

See how the pieces fit together:

http://img12.imageshack.us/img12/4800/notimpossible.th.jpg (http://img12.imageshack.us/i/notimpossible.jpg/)

[Ghitulescu]

Why do people generally need scene detection? Scene detection is something important for codecs and encoders ...

[prokhozhijj]

Why do people generally need scene detection? Scene detection is something important for codecs and encoders ...

Because we are talking about movie (picture in motion). There can be a lot of artefacts if we omit the motion.

For example, please imagine how deflicker will work in static. How it will know if current frame contains flicker or not. Without comparison with others frames it is impossible.

[Didée]

Think a bit further down the road.

For example 0 will mean static frame and 100 - for the frame with a lot of movement in it.

That's one "global" value for the whole frame. Acc. to this, a frame has either low, or medium, or high motion.


Now imagine a VHS home move of the 6-years-old son trying to imitate Bruce Lee in the living room. The room is completely static, the boy is in chaotic motion.

A frame-global metric leads you to pretty much nowhere. +50% miss in the one half, -50% miss in the other half. The average might be correct, but the average doesn't actually exist anywhere in the frame.

Same for contrast.

[Ghitulescu]

Because we are talking about movie (picture in motion). There can be a lot of artefacts if we omit the motion.

That's not scene detection ;) that's motion detection, yes it's useful for all that need to create extensions/plugins/scripts/etc.

Scene detection is used somehow different in video processing (at least in the old school).

What Didée said I think it was implemented in the H.264.

[prokhozhijj]

That's not scene detection ;) that's motion detection


I think that's misunderstanding each other or my tongue-tie.


Scene detection is used somehow different in video processing (at least in the old school).


I am using it in my Local Deflicker and it helps me to avoid some artifacts. ;) And now I am talking exactly about scene change detection.

[prokhozhijj]

Think a bit further down the road.

That's one "global" value for the whole frame. Acc. to this, a frame has either low, or medium, or high motion.

Now imagine a VHS home move of the 6-years-old son trying to imitate Bruce Lee in the living room. The room is completely static, the boy is in chaotic motion.

A frame-global metric leads you to pretty much nowhere. +50% miss in the one half, -50% miss in the other half. The average might be correct, but the average doesn't actually exist anywhere in the frame.

Same for contrast.

Well. Probably it is possible to split frame on pieces, filter them and after combine together.

[yup]

prokhozhijj!
You need create mask, separate frame for static and moving part (using length moving vectors, kind=0, MMask) and apply different filters for this masks.
yup.

[prokhozhijj]

prokhozhijj!
You need create mask, separate frame for static and moving part (using length moving vectors, kind=0, MMask) and apply different filters for this masks.
yup.

Thanks. But there is small problem. The matter is that MVTools sometimes (very often indeed) threats flicker as a motion. I've found solution but it is not perfect at the moment.

[Didée]

If the filter shall be able to deal universally with as little mis-interpretation as possible, then you need to take care of all the major cases that can happen.

A possible route to take:

0) Input

1) create a first rough global deflicker

2) depanestimate on (1)

3) create pan-compensated deflicker on (0) via depaninterleave, using motion data of (2)

4) once more: depaninterleave of (3) using motion data of (2)

5) Do MAnalyse-vectorsearch on (4).


After (4), all sections with (no motion) or (global motion) should be sufficiently deflickered. If areas with local motion are suffering, use step (5) to find areas where local motion is present. These areas then can either be excluded, or be treated individually.

[yup]

Thanks. But there is small problem. The matter is that MVTools sometimes (very often indeed) threats flicker as a motion. I've found solution but it is not perfect at the moment.
Try prefilter before motion estimation and dct=5 at MAnalyse.
yup.

[prokhozhijj]

Didée, yup. Thanks for advices. I will try to implement them