New duplicate detection filter

[Guest]

As a result of my work to develop a new duplicate detection engine, I have created a filter as a testbed for it. Duplicate detection can be useful in many applications: decimation, frame copying for bitrate reduction, field matching, etc.

I want to get some wider exposure on test clips from community members. But I had to give it some useful functionality. :) Therefore, I have implemented basic frame copying for bitrate reduction. Of course, Marc FD pioneered this with his CopySame(), and I hasten to add that this is not intended to compete as his filter is much faster. Perhaps this filter has better thresholding and performance in dark and/or noisy clips. If so, it would be in our interests for Marc and I to combine efforts.

Those interested in a theoretical analysis of the issue should refer to the thread "New smart sharpening filter". I'd like to thank High Speed Dubb in particular for his help and ideas.

As always, I'll appreciate feedback to make it better. Remember, it's a first beta and I don't recommend random access when in copy mode.

http://shelob.mordor.net/dgraft/dup/dup10b1.zip

[Marc FD]

I will test a bit. (not too much ;) )

i _hope_ it's better than CopySame, because improvements are awaiting.
let's tests everything :cool:

[Guest]

Marc, non! Let's keep the best features of each in a combined filter. I'll post the source as soon as I feel it is not too bug-ridden. Smart sharpener code is being released tonight, as I remember you were interested in the edge detection code. It uses separated kernels and you can make it really fast if you low-level code it.

[Marc FD]

too late i edited it BEFORE your post :D (and :p too)
New technique : Now i read my posts and i try to think how an american guy would understand it ;)

Let's keep the best features of each in a combined filter. Great. It was EXACTLY what i was trying to say. But i saw a little bit too late that my words were not clear.

[kyousuke]

our animes will become perfect of you continue you two :p

i'll try tomorrow if i've the time.
hmm combine your filters would be great ^^


@marc, rien à avoir le sujet.
ta un gui de prevu pr avs ?
pq j'ai une idée d'interface mais comm je suis une merde en VB, je crois cé pas demain j'arriverai à coder mon idée d'interface. j'ai en tete un truc simple.
1. deux fenetres (input/output) la vdub pr voir les modif en tps réèl, une barre de navigation, en dessous un textbox avec le script avs en cours. ça cé plutot simple et clair (bien que je sais pas coder ça)

2. quelque part (popup, menu, fenetre sur coté?) les fonctions de base et les plugs de base avec tout les fonctions éditables (bouton radio, cochable ou autres) bon cé vrai que cette partie là est assez titanesque, enfin pr moi ça le serait ^^

bref cé surtout la partie 1 qui me semble interessante pr le coté output tps réèl et édition du script simultanée (pq à part bloc note + le reload de ma video ss vdub, connais pas d'autre sol pr voir le résultat d'un script et cé un peu lourd à la longue...)

sorry for french ^^;;;;;;
sumimasen

+

[sherpya]

neuron2 you are a plugin machine :D

[Dark-Cracker]

hi,

this post have not a direct link with the thread but.

@all the avisynth's filters coders
i think u could perhaps create a sourceforge project with all the avisynth plugin. or perhaps release (when your plugin will be stable) a new BIG plugin for avisynth (with all the filter) in order a lot of people could use your filters.

Ps: very nice work with your filters, and keeping avisynth alive.

@neuron2
PS: do u think create an warpsharp in YUV ?

[Guest]

>PS: do u think create an warpsharp in YUV ?

Of course, I could. But I have no plans to do it because I do not find it to be a useful filter. It's a cute toy, IMHO.

Tell you what, show me a clip that is indubitably improved by warpsharp in a way that you can't get otherwise and I might consider it. Ball's in your court. :)

[ErMaC]

Neuron - I have one of those. Sadly I do not have it here with me in Japan.

There's a scene in the Japanese Region 2 release of Evagelion: Death & Rebirth. In one scene in Rebirth (first shot of the scene with Misato hacking into the MAGI) there's a panel of crosscrossed red lines that exibit this "dot crawl" effect where it looks like speckles have occured (similar to the one in the screenshot I showed you a while ago but much worse). This is present in many other scenes on the disc, but since it's is bright red against a black background it is most apparent here.

The only way I managed to fix that problem was with WarpSharp. Nowadays I might be able to fix it with waveletNoiseReduction, but that is very iffy and it haven't been able to test it on the current clip I have because WNR requires SSE and I'm stuck on a P2 366 in Japan.
People have commented that my encode of Rebirth looked BETTER than the DVD, and I will say that 95% of that is due to the tweaking I did with WarpSharp and SmartSmoother (which at the time were the best tools available), with more of the emphasis on the WarpSharp.

It is a very, very useful filter. When I was doing fansub encode for AnimeInc (I went by the nick DrasticMouse in fansub circles) WarpSharp was an invaluable tool. A very light warpsharp can do wonders for a DivX/MP42 -> DivX transcode, especially when the source is somewhat iffy. One episode of Sugar: the Little Snow Fairy (episode 7) had no high-quality raw to work from because the broadcaster changed their schedule lineup without telling anyone so all the good raw encoders missed it, leaving us with a badly IVTC'd 480x360 resolution raw with cross-coloration artifacts as the best we could use. WarpSharp single-handedly allowed us to make it at least semi watchable, I spent so much time tweaking settings on that encode that most people never noticed the fact that we used a different raw encoder. It was all thanks to WarpSharp.

Now in an era when I'm doing everything I can in AVS to maintain YUV colorspace, a YUY2 WarpSharp would be invaluable to me, and I'm sure to many others as well.

It is a great filter whose effects I've never seen duplicated the same way by anything else.

EDIT: BTW people don't go bother looking for it in the US release of D&R, because the US release isn't really Rebirth - it's just a chunk of End of Eva spliced together with the Rebirth titles. The actual Rebirth was more of a workprint of the first chunk of EoE, and had a few errors in it and a different music score. For EoE they added new music, fixed, some of the problems, reshot everything on new film and rerecorded many of the voices (most obvious when Misato says "Korosareru wa, hayaku isoi de" when they're looking for Rei - if she says it with a great sense of urgency and the last part in a whisper, it's the EoE voice track, if it's playful sounding it's the original Rebirth recording). Yes, I know too much about Eva

[Schlumpf]

@kyosuke

Si je t´ai compris correcte, tu cherches quelque sort de GUI pour Avisynth qui resemble Virtual Dub?
Alors, regarde ce thread ici: Avisynth-Monitor (http://forum.doom9.org/showthread.php?s=&threadid=32125)

Ca paraît d´être un "tool" (mot français qui correspond?) comfortable, bien que j´aie manqué du temps pour le tester jusque à aujourd´hui.


Hope I could help and please excuse my poor french. I know it sucks...

[Didée]

Originally posted by neuron2
>PS: do u think create an warpsharp in YUV ?

Of course, I could. But I have no plans to do it because I do not find it to be a useful filter. It's a cute toy, IMHO.

Tell you what, show me a clip that is indubitably improved by warpsharp in a way that you can't get otherwise and I might consider it. Ball's in your court. :)
(I´m not getting tired of repeating myself ...)

Shall I show you every single TV-cap of mine?

My ceterum censo:
WarpSharp sharpens, but doesn´t make a clip less compressable.

I gave more detailed reasons here (http://forum.doom9.org/showthread.php?s=&postid=176394)

You never will make WarpSharp useless with a sharpener that enhances contrast on a per-pixel-base.

Excuse me, I don´t want to get annoying.
But WarpSharp is a little pearl, and sometimes I fear there will never be a version that operates in YUY2. That would be a shame somehow.

[Guest]

I ask for a clip that proves it and all I get is blah-blah-blah. :)

[ErMaC]

Check your private messages.

[Guest]

Thanks, I will.

Umm, can we move WarpSharp to a separate thread. I'd really like to get some feedback on the Dup filter. Thank you.

[kyousuke]

c'est vrai.
i know this DLL, it's already a beginning :)
perhaps the gui will comme soon

[Marc FD]

i tested the dup.dll some hours...
and it gives me exactly the same result than copysame.
at least in my user point of view.
i definitively can't tell which is better.
i compared both on many scenes, and they worked both or failed both.
the whole trick is in the threshold selection.
i'd say : par

(off topic : is warpsharp that hard to adapt to YUVY2??? )

[Dark-Cracker]

Hi,

i have tested your dup plugin and it was a bit slower than copysame

Parameters used :

Dup(5,80,false,true,10,false)
VS
CopySame(1,4,8,4,0.1,987654321,50,50,3)

(seems the result was the same but my eyes are not perfects.)

but the output filesize was less big with dup.

i must do more test but i have no time.

@all
PS: i am sorry to have speak about warpsharp plugin in this thread, i try to correct my error :) so plz stop to speak about this filter in this thread , respect the donald wish.

bye

[vidiot]

Originally posted by kyousuke
our animes will become perfect of you continue you two :p

i'll try tomorrow if i've the time.
hmm combine your filters would be great ^^


@marc, rien à avoir le sujet.
ta un gui de prevu pr avs ?
pq j'ai une idée d'interface mais comm je suis une merde en VB, je crois cé pas demain j'arriverai à coder mon idée d'interface. j'ai en tete un truc simple.
1. deux fenetres (input/output) la vdub pr voir les modif en tps réèl, une barre de navigation, en dessous un textbox avec le script avs en cours. ça cé plutot simple et clair (bien que je sais pas coder ça)

2. quelque part (popup, menu, fenetre sur coté?) les fonctions de base et les plugs de base avec tout les fonctions éditables (bouton radio, cochable ou autres) bon cé vrai que cette partie là est assez titanesque, enfin pr moi ça le serait ^^

bref cé surtout la partie 1 qui me semble interessante pr le coté output tps réèl et édition du script simultanée (pq à part bloc note + le reload de ma video ss vdub, connais pas d'autre sol pr voir le résultat d'un script et cé un peu lourd à la longue...)

sorry for french ^^;;;;;;
sumimasen

+

:confused:
(because the only thing I liked in french class was the women who teaches...)

But what you are askin for: A gui like VD for tweaking AVISYNTH is already done by belgabor (and it is really fun to tweak filters with that.).
Look at the thread:

http://forum.doom9.org/showthread.php?threadid=32336&perpage=10&pagenumber=3
...and you´ll find virtualdubavs Version 0.2a.

...did I get you right or should I take some more classes french?

Harald

[ACClarke]

Originally posted by vidiot


:confused:
(because the only thing I liked in french class was the women who teaches...)

But what you are askin for: A gui like VD for tweaking AVISYNTH is already done by belgabor (and it is really fun to tweak filters with that.).
Look at the thread:

http://forum.doom9.org/showthread.php?threadid=32336&perpage=10&pagenumber=3
...and you´ll find virtualdubavs Version 0.2a.

...did I get you right or should I take some more classes french?

Harald
you're right
unfortunetly for me my english teacher wasn't pretty as yours :D

[Guest]

Guys, please stay on topic. See that "New Thread" button? :) Thank you.

My tests show Dup() slightly faster than CopySame() 1.1b. CopySame() crashes on several of my clips. The smallest one is 74MB. I can make it available if wanted.

When I can run CopySame() without crashes I can test it properly. I've asked Marc for all source needed to build it (the distribution is not buildable due to missing files), so I can fix the crashing bug.

[Dark-Cracker]

My computer was a P3 it perhaps for that copysame was a bit more speed. doest your dup was SSE optimized (silly question but it perhaps for that) ?
think to post a fixed version of copysame if u could fix the crashbug.

Ps: i hope u and marc could make a good plugin buy muxing your two sources.

PS2 : sorry for my poor english :) Frenchy Powaaaa !!!! :) :) :)

[Guest]

Attached is a new version that is much more usable. When show=true it now prints the metrics (and other information when copy=true) right on the frame itself. This allows you to more easily set your desired difference percentage for declaring a duplicate. Nice big easy-to-read type in sensible English. :) I got the code for the text overlay from that Japanese fellow's IT code, which he just released.

@Dark-Cracker

Mine is not yet at all optimized! It uses floats. :) Patience, my son.

I have an AMD Palamino XP1900+. AMD does floats much better than Intel.

Marc should fix the bug. I can give him a clip that crashes it. If he wants me to look at it, he'll have to publish complete source code. I will be releasing my source very soon.

[Guest]

I forgot to mention... In this new version, the threshold is a float so you can specify fractional percentages, e.g., Dup(threshold=2.5).

[Marc FD]

Originally posted by neuron2
Guys, please stay on topic. See that "New Thread" button? :) Thank you.

My tests show Dup() slightly faster than CopySame() 1.1b. CopySame() crashes on several of my clips. The smallest one is 74MB. I can make it available if wanted.

When I can run CopySame() without crashes I can test it properly. I've asked Marc for all source needed to build it (the distribution is not buildable due to missing files), so I can fix the crashing bug.

Dup is faster on my XP 1600+ too.
BTW AMD fp is _worse_ than Intel's.

i hope the bug you are speaking of is fixed in v1.2
if not, i would check the files for the dumb errors i do too often :rolleyes:
for the build files, see your PMs.

[Guest]

OK, where do I get version 1.2? Thank you.

[Marc FD]

it's a beta. i attached it in the CopySame thread.

[Guest]

Thank you. That seems to fix the crashing. So why have you left the broken one as the only downloadable one at your web site???

[Marc FD]

i thinked it was more stable.:rolleyes:

[Guest]

Please find attached Dup() version 1.1. The changes are as follows:

* Fixed problems finding changes near the edges of the frame.
* Increased speed.
* Shortened text overlay strings so they won't wrap on half-sized frames.
* Improved detection algorithm.

The algorithm in this version will be going into Decimate() unless I receive negative feedback. So please have a good look at it!

http://shelob.mordor.net/dgraft/dup/dup11.zip

[HarryM]

Originally posted by neuron2
Please find attached Dup() version 1.1. The changes are as follows:

* Fixed problems finding changes near the edges of the frame.
* Increased speed.
* Shortened text overlay strings so they won't wrap on half-sized frames.
* Improved detection algorithm.

The algorithm in this version will be going into Decimate() unless I receive negative feedback. So please have a good look at it!

http://shelob.mordor.net/dgraft/dup/dup11.zip

Can I user-define, which frame from identical sequence I want to use for duplication?
First or last?

[Guest]

@HarryM

Justify your request, please. The more convincing you are, the more likely it will be done. :)

[HarryM]

Originally posted by neuron2
@HarryM

Justify your request, please. The more convincing you are, the more likely it will be done. :)

For example, I have this sequence of frames:

a b c d e f g h i j k...

Frames 'c', 'd', 'e' and 'f' are near-identical.


Finally I have

a b c c c c g h i j k... (alternative A)

or

a b f f f f g h i j k...? (alternative B)


I think, alternative B is better. If I use temporal filter, then the 'c' frame is depreciated by ghosting effect from previous (different) frame...

[Belgabor]

Wouldn't then be a b d d d d g h i j k (or with e) better? f might suffer ghosting from g (but if you have ghosting in this case, you'll have it elsewhere as well where it cant be avoided with duplication). But theres another pro for taking a middle frame. Those should have benefited the most from temporal cleaning. (Temporal filtering in AviSynth is bidirectional, isnt it?)

Cheers
Belgabor

[HarryM]

Originally posted by Belgabor
Wouldn't then be a b d d d d g h i j k (or with e) better? f might suffer ghosting from g (but if you have ghosting in this case, you'll have it elsewhere as well where it cant be avoided with duplication). But theres another pro for taking a middle frame. Those should have benefited the most from temporal cleaning. (Temporal filtering in AviSynth is bidirectional, isnt it?)

Cheers
Belgabor

Right! The middle frame is the best...

[Dark-Cracker]

hi,

i have test your filter on an another computer (win98,P3 , 256 MO) and i have an error "Avisynth : stack overflow" i have tested with the avisynth version 2.05 / 2.04 / 1.0b7d and i have the same error msg , i have also tested with the dup version 1.0b1/b2 / and 1.1

have u an idear about this problem ?

Ps: i have also tested with the "SetMemoryMax (40)" at the start of the .avs file.

bye

[Defiler]

Originally posted by neuron2
@HarryM

Justify your request, please. The more convincing you are, the more likely it will be done. :) I also believe this is a good idea. I could provide a short clip to illustrate this, if you are interested. DivX 3.11 encoding artifacts often take the form of evil blocks that appear in the first frame, and then vanish in subsequent frames. When you are taking such clips as input, it's helpful to copy the last frame of a duplicate sequence onto the first.
Before CopySame added this functionality, I just put a "reverse()" statement on either side of it.

[Guest]

@Dark-Cracker

What is your script and does it fail only when Dup() is present? Does it fail on all clips or just one?

[Guest]

Here's my take on it: the first frame c would already have benefited from temporal smoothing. Given this and the fact that to take the last one I would have to read ahead by the maxcopies number, makes me conclude that it is not worth it for this reason alone. Defiler's reason is more persuasive, but he has a workaround. :-)

Dup() was really just a quick testbed for my new duplicate detection code targeted at Decomb(). When I complete the next release of Decomb() I will revisit this issue for Dup().

[Dark-Cracker]

hi,

i have tested with different script but even with a basic script i have this error.


LoadPlugin("D:\AVSFILE\MPEG2DEC.DLL")
LoadPlugin("D:\AVSFILE\Dup.DLL")

Mpeg2Source("c:\TEST\PROJECT0.d2v")
Dup(5,false,true,10,false)

i have tested to with dup() and with AVIsource instead of Mpeg2source but i have still this error.

[Guest]

@Dark-Cracker

Try the attached version 1.11 and let me know.

[Guest]

@dividee

How big is Avisynth's stack?

[Dark-Cracker]

@neuron2
:) :) :) :)
with the version 1.11 the problem was solved :) i have tested with avisynth v2.04 and v2.05 and it work like a charm, thank u very much.

[Guest]

I declared data on the stack. I need to move it into a malloc. Temporarily, I just decreased the maximum array size. The Avisynth stack appears to be very small. :(

[Dark-Cracker]

hi,
strange it was the first time a have a problem with a filter and the avisynth stack.

>Temporarily, I just decreased the maximum array size.

it was a temporaly manipulation because this slowdown the encoding process ?

[Guest]

It was temporary because I could do it in 5 minutes during my lunch hour. :)

To convert to malloc takes a little time for a 2-dimensional array. I'll fix it properly later.

[Marc FD]

Hi :)
I worked a lot on new algos the past week, and i'm finally running out of ideas. I couldn't see the difference between dup and copysame 1.2, and after many tries i only get small improvements with my last algo.
I'm not really sure if i should implement it seriously.
in a hand, it's not bad, on the other hand, the dup filter seems enough for all users...

and i've got too much homework :(

but maybe it would be interessing to compare my cheap engine against dup. to know wich one is better.

I really don't know what to do (and i'm very lazy since i'm back too shool....)

[Suikun]

@neuron2
In the readme of Dup you state that it's working with 32*32 windows.
Sometimes in animes it happens to happen that only a very small part of the video is moving or changing, like a mouth or eyes. So would it be possible to copy all other 32*32 blocks except the ones with the actual movement in it?

[Edit]
Justification for this (as you always ask for them :P):
When I let ffdshow show the motion vectors I often notice motion in apparently still areas. Local copys of previous frames might help there.

[cult]

hope that u forgive my ignorance but is this filter supposed to work with normal movie?I tried in a movie with dup()and dup(copy=true),and I could see some jerkiness in the movement.For example in the start credits the letters wouldnt fade out properly but rather disseaper if u know what I mean...Same goes for some other frames too when the camera was moving slowly.I tried the same movie wihout the filter and the effect was gone.Any ideas?Maybe wrong settings?

[Belgabor]

Originally posted by cult
hope that u forgive my ignorance but is this filter supposed to work with normal movie?I tried in a movie with dup()and dup(copy=true),and I could see some jerkiness in the movement.For example in the start credits the letters wouldnt fade out properly but rather disseaper if u know what I mean...Same goes for some other frames too when the camera was moving slowly.I tried the same movie wihout the filter and the effect was gone.Any ideas?Maybe wrong settings?

No, in a normal case its not. It's made for animation and the detection of 'true' duplicates, aka frames which should be the same but aren't because of noise. This ususally doesn't occur in natural movies. But if you set the thresholds right, normal movies should work too, just you wont have any benefit because of the lacking duplicates.

Cheers
Belgabor

[Dark-Cracker]

hi,

dup it's an filter for the anime in anime there is a lot of same frame and this filter detect these duplicate frame.
it not really destined to be used with the movies (i think i am right). it for that u obtain a crap result.

++