An MVTools() alternative to DeDup() for anime?

[Velocity 7]

Seems that in some anime content there'd be characters off in the distance that'll be talking and the motion involved with their mouths would be small. DeDup only detects this as say, a 0.15% difference, and if threshold isn't set to this it gets dropped as if it were a duplicate. Of course, setting threshold low enough solves this, but then a whole bunch of other stuff could be 0.15%, such as compression artifacts and the like.

Right now my current working theory involves using MVAnalyse() to detect anime motion, which works very well but unfortunately falls short of fading logos or mouth movements from a far distance. DeDup() will very likely be able to handle scene changes and fading logos via a higher threshold than normal (say, 2.0% or more).

The current code for detecting what's considered "motion" through vectors is as follows thus far:


x = "insert video file here"
LoadPlugin("mvtools.dll")
AVISource(x)
BilinearResize(704,396) #?
vec_clip = MVAnalyse(blksize=4, pel=2, search=3, truemotion=true, sharp=1)
MVShow(vectors=vec_clip, scale=2)


The final steps should be something along the lines of:

First pass:
- writing to output file
- DeDup writes difference %s
- MVTools() writes if a vector appeared
- ??? writes if there were small mouth differences

Second pass:
- All frames are considered "dropped"
- any frame with a vector is considered to be "kept"
- any frame with a difference threshold > specified is "kept"
- any frame with a small mouth difference is "kept"
- Write matroska timecode v2 file on the fly


If anyone has any suggestions to improve this theory, please post. :)

[wonkey_monkey]

What about breaking the picture up into blocks, and if the difference in any block in subsequent frames is over a threshold, it's not a dup?

David

[Velocity 7]

Isn't that the original implementation of DeDup? The problem with that method is that compression artifacts are counted as a "difference" as well, especially the stuff that can't be seen by the human eye (if you do a difference between two still frames compressed in MPEG-2, you'll know what I mean).

Would a DeDup() difference of 2.0+% account for things like fade-outs or scene changes (while ignoring everything else)? If so, the DeDup() method may still work but only for things such as that. I'm at work at the moment so I can't test this right now. As a result it'd be something like a combination of vector masking + DeDup() dropping at threshold=2.0...

The last thing on the list is small mouth movements that should be visible to the naked eye...

[Velocity 7]

No one has any suggestions? :(

[Fizick]

Here is my small comments.
1. for mouth and mouse :) movement detecting you need in small block size (about 8 or 16). Currently Dup and DeDup use blocksize=32. That is why MVtools is more detecting program.
DeDup may be easy recompliled for another blocksize.
By the way, in my GetDups plugin (also based on Dup) I changed bloksize to 8 to more reable detecting.
2. For global fading stability some fading compensation must be added to DeDup. (I added global luma compensation to GetDups).
3. For local (logo) fading you can use some mask (on detection clip). Create it by hand and merge.
4. Mpeg compession artifactes is hard task. Probably you must use some pre-filter for detection clip.
So, use first clip (filtered, deflickerd) for detection pass, and second clip for final processing.
5. As for MVTools, now it contains "dct" parameter for stability to fading. I can consider your suggestions about new fuctionality (for example, writing some log).
6. May be I complitely do not understand you. I never used DeDup, and probably did not filter any anime (what is it - Japan cartoon ?) too :)

[foxyshadis]

I saw this a few days back and I've been pondering, but I can't help thinking it might be the wrong approach.

If you went all the way toward "vectoring" the show, extracting the elements and recompositing them into flash, as sprites or vectors, then you'd need lots of really intelligent adaptive compensation that mvtools can help provide.

But the reason dedup only tests for identity is that repeating a frame is the only thing you can do with mkv timecodes. You don't get the complex instructions for move certain areas (or the whole screen) x pixels, or to fade between two scenes by x% per frame. That's what the codec is for, it usually does those things very well. (Especially AVC w/ wbpred, for the fading.)

If you are doing something more complicated than chopping frames, such as sideband instructions for the decoder, I'm interested.

A reasonable modification of Dup() would be to allow most areas of mostly static frames to be blended, while others are passed through, in order to achieve more of the size savings without overly aggressive thresholds. Pixel-based filters are too fine, and frame-based filters too coarse. Finding just the right algorithm is the tricky part.

And yeah, anime is just japanese cartoons, usually lots of 12 or 8 fps sections.

[Velocity 7]

Here is my small comments.
1. for mouth and mouse :) movement detecting you need in small block size (about 8 or 16). Currently Dup and DeDup use blocksize=32. That is why MVtools is more detecting program.
DeDup may be easy recompliled for another blocksize.
By the way, in my GetDups plugin (also based on Dup) I changed bloksize to 8 to more reable detecting.
2. For global fading stability some fading compensation must be added to DeDup. (I added global luma compensation to GetDups).
3. For local (logo) fading you can use some mask (on detection clip). Create it by hand and merge.
4. Mpeg compession artifactes is hard task. Probably you must use some pre-filter for detection clip.
So, use first clip (filtered, deflickerd) for detection pass, and second clip for final processing.
5. As for MVTools, now it contains "dct" parameter for stability to fading. I can consider your suggestions about new fuctionality (for example, writing some log).
6. May be I complitely do not understand you. I never used DeDup, and probably did not filter any anime (what is it - Japan cartoon ?) too :)

Could DeDup() be recompiled with a blocksize parameter then?

[Fizick]

yes, it can.
But I have no time for it. (it is constant in code).
I can recompiler plugin for different block size.