Dup 2.0 for Avisynth 2.5

[Guest]

Here is Dup 2.0 beta 1 for Avisynth 2.5.

It supports both YV12 and YUY2.

The algorithm was changed so that when there is a string of duplicates the last frame of the string is used as the copy frame (previously the first frame was used).

In deference to the great sh0dan, the copy parameter was defaulted to true. :)

Feedback will be appreciated.

[Guest]

Please use this one:

[Gant]

Thanks a lot... was waiting for this one to arrive to 2.5x platform :D

I'll test it as soon as possible... prolly tonight. :)

[SILICON]

One sugestion for speed your filter.

Test the difference with the previous frame only in luma plane.

[Guest]

Originally posted by SILICON
One sugestion for speed your filter.

Test the difference with the previous frame only in luma plane. Been there, done that. See the 'chroma' option.

[gamr]

Downloading to toy with it now.

Just a suggestion, make luma=false too perhaps so as under yv12 it can sample a 1/4 res chroma only (insanely fast, 20% of the original data compared to default settings in yv12 if im not mistaken)

[FuPP]

Hi Neuron !

I've tried your new version; very pleased to see it working with avisynth 2.5 :)

Two little problems at this time :

1/ get weird frames (one on two) when using show=true

2/ still using show=true, displayed information (when good frames) seems strange to me :

ex : frm 123 : using frm 124 and so on...

Is it normal ?


Regards,

FuPP

[Guest]

Originally posted by FuPP
1/ get weird frames (one on two) when using show=trueWhat exactly is a "weird frame"?

2/ still using show=true, displayed information (when good frames) seems strange to me :

ex : frm 123 : using frm 124 and so on...

Is it normal ? That is saying that frame 123 was asked for but the filter determined that 123 and 124 were duplicates, so it used the last duplicate in the string, in this case 124.

[FuPP]

1/

Originally posted by neuron2
What exactly is a "weird frame"?


It means that (sorry for bad english):) (see attachment) :


2/ Not sure to totally get you (because I don't think there were duplicated frames a the place this information was displayed), but will make more tests when I will have solved 1/.


Thx for your quick answer,

FuPP.

[Guest]

Originally posted by FuPP
2/ Not sure to totally get you (because I don't think there were duplicated frames a the place this information was displayed Realize that the filter is looking ahead at frames you haven't reached yet. It doesn't compare backwards any more, only forwards.

[Guest]

Originally posted by FuPP
It means that (sorry for bad english):) (see attachment) :
Ooh, that's the MakeWritable() bug I've been struggling with! sh0dan tells me it is fine, but... I need to look at the Avisynth source code but it is not easy as I have to get a CVS client, blah, blah.

In the short run I'll hack in my personally written MakeWritable2() that I used in Decomb.

[Bulletproof]

That's what happens to me with some filters that I use after a crop, maybe if you put it before a crop it won't do that?

[Guest]

Here's the fixed version with local MakeWritable().

Please let me know if this fixes your weird frames. Thank you for pointing out this problem, FuPP.

[Guest]

Originally posted by Bulletproof
That's what happens to me with some filters that I use after a crop, maybe if you put it before a crop it won't do that? It happens without any cropping. When I replace calls to MakeWritable() (code is in avisynth.dll) with my own routine that performs the same function, the problem goes away. I am still investigating, but it sure looks like a bug in Avisynth 2.5.

[Gant]

Now that i was about to say that i found thta "weird" frame problem someone was ahead of me... lol

Well anyway have to try this new version...

And from what i've seen last night its fast... havent made a table with real values, but hope to do it tonight. :)

Thanks for the update.

[sh0dan]

Seems like Pitch isn't being respected properly - is this something only happening in newer versions, or has all 2.5 alphas had this bug?

Are your CPU's ISSE capable? - otherwise that could be the root of the problem (a faulty MMX bitblit).
Are you using the newest avisynth.h (from CVS).

[sh0dan]

I cannot believe it... I think I found some remains of the days, when YV12 was forced to be mod 16. Anyway - put up a new binary - try that one out, it _may_ solve the problems.

@neuron: Thanks for being so persistent on convincing me there was a bug - I cannot believe it hasn't been found yet - something must have triggered it lately.

[HarryM]

Originally posted by sh0dan
I cannot believe it... I think I found some remains of the days, when YV12 was forced to be mod 16. Anyway - put up a new binary - try that one out, it _may_ solve the problems.

@neuron: Thanks for being so persistent on convincing me there was a bug - I cannot believe it hasn't been found yet - something must have triggered it lately.


Repair the link for download, please. It's related to build 17122002 still.

[UPollaehne]

@HarryM
in the meantime you can try this: AVISynth 2.5 Alpha (http://cultact-server.novi.dk/kpo/avisynth/avisynth_a_030103.zip)

@shodan
Did you update CVS?

[sillKotscha]

Originally posted by sh0dan
Anyway - put up a new binary - try that one out, it _may_ solve the problems.

Guys - you're unbelievable!! Your efforts and reached goals can't be described with my poor english. You work so hard just for your/ our pleasure - I ask myself why the économic situation just sux nowadays with so many genial heads out there?!!

------------------------

just tried to download the 'debugged' new Binary [03-01-2003] but it's still avisynth_a_17.12.02.zip (my explorer cache is empty)

regards Sill

Edit: too late with my post :D

[sh0dan]

Link corrected (the experienced hax0r would still try to download avisynth_a_0301003.zip :)

[FuPP]

@Donald+Shodan

hem, hem...

Well, thanks a lot for your efforts, but I still have the problem both with avs 2.5 03/01 + dupyv12b2 or with Dupyv12b3.

Here's my script (I could have start with that last time... :rolleyes: )

LoadPlugin("C:\video\avsfilters\MPEG2DEC3yv12.dll")
LoadPlugin("C:\video\avsfilters\dupyv12.dll")
mpeg2source("f:\tests\vts_01.d2v",CPU=6,IDCT=2)
Crop(8,4,704,568)
Dup(threshold=5,maxcopies=10,chroma=false,show=true)
BilinearResize(448,542)
AddBorders(16,17,16,17)


:confused:

FuPP

XP1800+, W2K

[FuPP]

Originally posted by Gant
Now that i was about to say that i found thta "weird" frame problem someone was ahead of me... lol

Sorry for that ! ;)

[FuPP]

Oh !

I've discovered that putting another filter (ex : convolution3d or... warpsharp ;)) after Dup make things works : it seems very similar to the problem I've reported to Sansgrip about fluxsmooth...

Hope it can help.;)

[Guest]

@FuPP

Thank you for your results. I was indeed able to duplicate the problem using your script.

Apparently the vi->IsWritable() is not reliable. I've temporarily removed the test of it in MakeWritable2(). The problem seems to go away. Please test with the attached beta 4 and let me know.

@sh0dan

Any comments about vi->IsWritable()?

[Guest]

Originally posted by sh0dan
Seems like Pitch isn't being respected properly - is this something only happening in newer versions, or has all 2.5 alphas had this bug?Don't know because I've been upgrading faithfully and only run the newest versions.

Are your CPU's ISSE capable? - otherwise that could be the root of the problem (a faulty MMX bitblit). Athlon XP.

Are you using the newest avisynth.h (from CVS). No. Should I be? Why? Can you attach it here, please? :)

[FuPP]

@neuron

yes yes yes yes yes !

Nice shot ! It works fine now (and is fast !) Thx a lot for your time.

Best regards,
FuPP.

[Guest]

Originally posted by FuPP
yes yes yes yes yes !

Nice shot ! It works fine now (and is fast !) Thx a lot for your time.
Thank you again, FuPP, for your assistance. I'd better have a look at Decomb. It has the same code. :(

[sh0dan]

Originally posted by neuron2
No. Should I be? Why? Can you attach it here, please? :)
The latest is on the alpha site :)

Another possible problem could be, that you read pitches _before_ making the image writable. Pitch might change when you make an image writeable. Could that be the cause?

[Guest]

Thank you, sh0dan. I will get the new header file and check my pitch usage, as you suggest.

[Guest]

Here is Dup 2.00 beta 6.

It adds a requested feature: blend. When blend=true, the copy frame will be generated from an averaging of all the duplicate frames. This should give some noise reduction. Please read the help file for details. Of course, with blend=true, the filter will be slower.

Now even MarcFD can't argue that Dup is useless. :)

Feedback will be appreciated.

[sh0dan]

You might want to check out the Fluxsmooth thread - this could be the reason for your crashes.

[Guest]

You say that it should be fixed BOTH in the resizer and the application! How would we fix it in the application?

Also, I made the problem go away by removing the test for IsWritable() and its return in MakeWritable(), so that it always copies to a new frame. That doesn't seem to fit with the mechanism you are suggesting?

[sh0dan]

What happends is that your filter returns frames with different pitches.
If you don't make all frames writable, but only some of them, it returns the source frame.

If the source has been cropped right before your filter, it results in a pitch much larger than the image. When you make an image writable, a new frame is created, with a smaller pitch, and the image is copied to this frame. If you make all images writable the new pitch is consistent. But if you on some occations returns the same frame with a old pitch it causes an error in resize.

I've corrected the error in resize.

The easiest way for you to fix it is to create a new frame, and bitblit to it, if there are no changes - otherwise, just make _all_ frames writeable, no matter which frames are requested.

[Guest]

Thank you for the explanation. I did notice it didn't happen if I removed the resize after Dup.

Can I get the corrected Avisynth to test the fix?

[sh0dan]

I'll upload a new binary tomorrow - I don't have access to the server right now. Hope you can wait that long!

[Guest]

No problem, Sir. Thank you for your excellent analysis and debugging on this issue. :D

[FuPP]

I have tested b6, but only on a short clip (about 1700 frames). I will try to make more serious tests later in the week.

b6 is stable ;)

b6 without blend : gain in compressibility (mpeg2) : 4 %
b6 with blend : gain = 5.5 %

command : Dup(threshold=5,maxcopies=20,chroma=false,blend=true)

Cheers,
FuPP.

[MaTTeR]

It's a free lunch so I couldn't resist:D

Has anyone been successful using Dup on clean progressive movies? Just curious more than anything.

[FuPP]

I've made some attempts a few weeks ago (with avs 2.0x compatible version); the filter did not detect subtle movements in still scenes (like a far character walking in the background), the movie got then jumpy.

But seems really efficient on animes !

Regards
FuPP.

[MaTTeR]

@FuPP

Thx for the reply. My experience is also exactly as you describe, just wanted to verify it.

[Ewi]

@neuron2

is it possible to let Dup write a log file which frames are replaced so you can easily check with vdub if there are frames that should not be replaced (for example the described situation with a guy walking far away)...

Thank you...

[Guest]

Originally posted by Ewi
is it possible to let Dup write a log file which frames are replaced so you can easily check with vdub if there are frames that should not be replaced (for example the described situation with a guy walking far away)Just set debug=true and the needed information will be visible through the DebugView utility (available at my web site). Then use the logging capabilities of the DebugView utility if you want to capture the output to a file.

[Ewi]

@neuron2

Ohhh, Ok thank you.

What about this idea:

I don't know exactly how Dup is working but I as far as I know it's working on whole frame and if two (or more) frames have a difference less than a threshold the second frame is replaced by frame 1 (except for blend...). But in anime there are often parts of the picture that don't move (for example a person speaking and only its mouth is changing).

So what is about partly "dupping" the frame. A simple way would be to divide the picture in submatrices, f.e. 3x5 or more, and use dup on each of them. A more complex method could require to detect motion, chroma and lumi changes and to mask the parts of the frame that are under a given threshold of a weight function f(motion[atPixel],chromachange[atPixel],lumichange[atPixel]), and then dup or blend-dup only these parts.

Both of this would allow to go lower with the threshold values that have to be actually used and so perhaps our man on the horizon is walking again with a gain in compression in his pocket....

Thank you...

P.S.: If Dup is working in this or a better way: don't hate me for asking silly questions....

[Guest]

@Ewi

Thank you for your suggestion. The idea was already suggested on the main Dup thread. Search for "New duplicate detection filter". I am considering it but fear that it may fragment motion in an ugly way.

@all

I have released Dup 2.00, the first release version. It has several robustness fixes versus the last beta, and fixes a bug such that the blended frame wasn't used when show=true. It's a good idea for all Dup users to get this version. Please get it from my web site (the Avisynth 2.5 version). The source code is also available there.

Thank you all for your valuable feedback during development.

[Marc FD]

lol, it'ld be a temporalsmoother ^^

>Now even MarcFD can't argue that Dup is useless.

i still argue it's useless.
your work is great, as always, don, it's all the fault of the dumb guy who had the idea ^_^.

are you still using the same algo to detect movement, by searching the biggest mean difference in 32x32 windows ?

because i've a new filter, with a similar approach, and i'd like to know the drawbacks ^^

[Guest]

Originally posted by Marc FD
are you still using the same algo to detect movement, by searching the biggest mean difference in 32x32 windows?Yes.
because i've a new filter, with a similar approach, and i'd like to know the drawbacks ^^Can't say without knowing the purpose of your filter.

[Marc FD]

>Can't say without knowing the purpose of your filter.
just a temporal smoother more ^^.

[Guest]

Originally posted by Marc FD
just a temporal smoother more ^^. Your statement is too brief and cryptic to understand.

Do you mean block-based conditional copying/blending as suggested by Ewi and others?

[Marc FD]

sorry don.

okay, let's go technical

pixel-based 5x5 (or more) interleaved diff with inloop affine (based on the accumulation of 5x5 diffs) blending.