RV9 Animation DropDupe Pre-filter

[bill_baroud]

i agree here with midiguy, there is _always_ a scene that wont match with the treshold you choose for your video ... and then you're forced to lower treshold and miss some other scenes. A way to exclude some frame from duplicate remove would be great :)
(and hardcore anime encoder are used to ... who never did a manual decimation ?? ;) )

[karl_lillevold]

Audio dropout problem fixed along with extra frame drops. Thanks to Sirber for noticing the audio dropouts. Please get new DLL, see 1st post.

I have also experimented with Donald Graft's Dup filter added to the AviSynth script used as input to Producer, as a front-end for DropDupe, and then set the DropDupe thresholds really low, like midiguy suggested. This works better than DropDupe's simplistic approach, and is the recommended method until the built-in detection is improved. I did a couple of test encodes that came out really well with this combination, with 5.1ch sound and everything.

I included this in the 1st post:
Use Dup (AviSynth filter) as front-end
Donald Graft's Dup Avisynth filter (http://shelob.mordor.net/dgraft/dup/dupnew.html) does a better job with noisy clips, and has more advanced options for look-ahead, blend, use-last frame in sequence. If you add this in your Avisynth script before feeding it to Producer with DropDupe set to low thresholds (since Dup creates true duplicated frames), you will get very good results:
In your AviSynth script, include
LoadPlugin("<path_to_plugin>\dup.dll")
Video=Dup(Video,show=false)
Then set show=true, and load avs in VirtualDub to make sure you tune the Dup thresholds to your clip. See Dup documentation. Defaults have worked well, or maybe a little too aggressive, for my limited testing.

Then set the all thresholds for DropDupe to l, before using the .avs script with Dup as your source to Producer. Remember to set show=false, otherwise DropDupe will detect changed frames, and not drop them. Besides you don't want to encode the debug info from the Dup filter.


@midiguy and bill_baroud: Adding a manual method via some kind of user interface would be great for the ultimate anime encode expert, but might take more time than I have right now. I can imagine a VirtualDub plugin with UI that writes out a log file that DropDupe reads.

@Pamel: another improvement that would have been really nice to have, either in an Avisynth front-end filter or directly in this filter.

[iwod]

So i am back... after leaving for a while.......

Really, i think if anybody wants more improvement, they should suggest it to Donald, where he can improve his Dup2.2,

I don't see why we should we invent the wheel if we alreay have a EXtremely good AVISyth filter. May be all this work about manual choosing which frame not to drop could goto avisyth filter, while the dropdupe filter will keep as a simple if it is the same then drop it approach. Which will save some time for karl to improve other things :D

As far as i can see Dup2.2 with Dropdupe works very well. I just don't understand what is the point of improving current dropdupe filter........

Edit...... Donald is currently running out of idea on how to improve on his dup filter. So you may want to suggest it on the Avisyth board.

Or you could post it here and i will merge all your idea and submit to him.

[HomiE FR]

Hi all,

This DropDupe along with Dup 2.20 is really a good team for anime encodes! But I have a little problem with DropDup which doesn't want to drop more than 3 frames in a row, no matter what I choose for maxDroppedFrames (in the example below it's 20, which equals to the parameter maxcopies choosed for Dup).

Below I put a few lines from producer.log which shows the problem:
DD: keep (25245) max: 561 216 328, avg: 27 33 34
DD: DROP (25246) max: 0 0 0, avg: 0 0 0
DD: DROP (25247) max: 0 0 0, avg: 0 0 0
DD: DROP (25248) max: 0 0 0, avg: 0 0 0
DD: keep (25249) max: 0 0 0, avg: 0 0 0
DD: DROP (25250) max: 0 0 0, avg: 0 0 0
DD: DROP (25251) max: 0 0 0, avg: 0 0 0
DD: DROP (25252) max: 0 0 0, avg: 0 0 0
DD: keep (25253) max: 0 0 0, avg: 0 0 0
DD: DROP (25254) max: 0 0 0, avg: 0 0 0
DD: DROP (25255) max: 0 0 0, avg: 0 0 0
DD: DROP (25256) max: 0 0 0, avg: 0 0 0
DD: keep (25257) max: 0 0 0, avg: 0 0 0
DD: DROP (25258) max: 0 0 0, avg: 0 0 0
DD: DROP (25259) max: 0 0 0, avg: 0 0 0
DD: DROP (25260) max: 0 0 0, avg: 0 0 0
DD: keep (25261) max: 0 0 0, avg: 0 0 0
DD: DROP (25262) max: 0 0 0, avg: 0 0 0
DD: DROP (25263) max: 0 0 0, avg: 0 0 0
DD: DROP (25264) max: 0 0 0, avg: 0 0 0
DD: keep (25265) max: 0 0 0, avg: 0 0 0
DD: keep (25266) max: 655 265 341, avg: 36 47 52

And here is the lines of the job used concerning DropDupe:
<prefilter xsi:type="videoDupFrameDropper">
<enabled type="bool">true</enabled>
<maxDroppedFrames type="uint">20</maxDroppedFrames>
<maxAvgSSD type="uint">1</maxAvgSSD>
<maxAreaSSD type="uint">1</maxAreaSSD>
<maxAvgChromaSSD type="uint">1</maxAvgChromaSSD>
<maxAreaChromaSSD type="uint">1</maxAreaChromaSSD>
<earlyExit type="bool">false</earlyExit>
<enableDetailedLogInfo type="bool">true</enableDetailedLogInfo>
<pluginName type="string">rn-prefilter-dupframedropper</pluginName>
</prefilter>

I'd really like to see this small problem solved (I believe this is just a hard limit that Karl put for some testings when there was a bug with timestamps, but this bug has been solved I think).

Thanks in advance.

[karl_lillevold]

this is true, I put in a limit of 3 frames, because I noticed that there were some problems with larger numbers, due to too large distances in the timestamp gaps between certain frames. I still have on my list of things to do, to figure out what the max number really is. Sorry about that, but most of the compression efficiency improvement is already there with 3, even though some scenes and videos have longer periods of completely duplicated frames.

[HomiE FR]

Ok thanks Karl, I think you're right when you say that most of the duplicates are already gone with maxDroppedFrames = 3. I only have to put maxcopies = 3 in Dup so that I don't do too much duplicates that DropDupe can't handle as duplicates. :)
And like you said, the compression efficiency improvement won't get much bigger with maxDroppedFrames = 20, it's just that I like when things are pushed to their limits. :p

[karl_lillevold]

Originally posted by iwod
Really, i think if anybody wants more improvement, they should suggest it to Donald, where he can improve his Dup2.2,

I don't see why we should we invent the wheel if we alreay have a EXtremely good AVISyth filter. May be all this work about manual choosing which frame not to drop could goto avisyth filter, while the dropdupe filter will keep as a simple if it is the same then drop it approach. Which will save some time for karl to improve other things :D

As far as i can see Dup2.2 with Dropdupe works very well. I just don't understand what is the point of improving current dropdupe filter..
Thanks, I have also realized this, and put any further fine-tuning of the detection algorithm in DropDupe on hold for now. Donald's Dup works quite well, and tuning the thresholds is very easy in VirtualDub.

P.S. There is an additional use for DropDupe, that did not even occur to me until someone said they were using it for this purpose, and this is encoding of screen capture training videos. Except for the mouse pointer, things should then be pretty duplicated for long periods of time. Maybe the SSD is hard to adjust for the mouse pointer though, I guess depending on whether or not you want to include it.

[midiguy]

Would it be possible to use to tkae the code from dup and implement it into dropdupe (both are opensource)? That way less processing would be involved (you wouldn't need to process it through two different duplication drop filters).

PS: I PMed you Karl. My email's outgoing server is down, although I can receive mail (and did receive your email).

[iwod]

Originally posted by midiguy
Would it be possible to use to tkae the code from dup and implement it into dropdupe (both are opensource)? That way less processing would be involved (you wouldn't need to process it through two different duplication drop filters).

PS: I PMed you Karl. My email's outgoing server is down, although I can receive mail (and did receive your email).

I don't think this is nessercery as you won't gain much in terms of resources if you combine both scripts.

Even through they are open source, in reality only karl and donald work on their script. By keeping them seperate if you want new features you could always request donald to make it since he is running out of idea how to improve dub. And left Karl to do the simple job of putting the final touch on Dopedrop. This way karl get more free time to improve other part of RV9 :devil:

I don't mean the idea was to shift the work load over to donald. Another reason for that is only because donald is "THE" master in these areas if you visit enough of avisyth forums. Which means he proberly does better coding than karl in this field. You could always try reading his highly technical homepage and his script such as decomb before worshipping him :D

[karl_lillevold]

Originally posted by midiguy
Would it be possible to use to tkae the code from dup and implement it into dropdupe (both are opensource)? That way less processing would be involved (you wouldn't need to process it through two different duplication drop filters).
Initially this may sound like a good idea, and I have considered it. However, with early exit and low thresholds in DropDupe, there is virtually no overhead in running the two back to back. Also, it's an advantage to preview your settings in VirtualDub, and this is only possible with Dup. Integrating the two would add extra work for me, extra maintenance when Dup changes, etc. If your source file is not already an Avisynth script, the only extra work is to write one to include Dup, like I showed above.

Another problem with integrating the two, is how Dup is released under the GPL. I am not an attorney, but have been told this is not compatible with the open source Helix RPSL. RPSL compatible licenses include the Lesser GPL and BSDI Open Source License.

Originally posted by iwod
donald is "THE" master in these areas if you visit enough of avisyth forums. Which means he proberly does better coding than karl in this field. You could always try reading his highly technical homepage and his script such as decomb before worshipping him
I have no doubt about that, and I am sorry to see him leave this forum (http://forum.doom9.org/showthread.php?s=&threadid=57337).

[iwod]

Originally posted by karl_lillevold
Initially this may sound like a good idea, and I have considered it. However, with early exit and low thresholds in DropDupe, there is virtually no overhead in running the two back to back. Also, it's an advantage to preview your settings in VirtualDub, and this is only possible with Dup. Integrating the two would add extra work for me, extra maintenance when Dup changes, etc. If your source file is not already an Avisynth script, the only extra work is to write one to include Dup, like I showed above.

Another problem with integrating the two, is how Dup is released under the GPL. I am not an attorney, but have been told this is not compatible with the open source Helix RPSL. RPSL compatible licenses include the Lesser GPL and BSDI Open Source License.


I have no doubt about that, and I am sorry to see him leave this forum (http://forum.doom9.org/showthread.php?s=&threadid=57337).

What the hell!!!!!!!!!!!!!!!! he is leaving!!!!!!!!!!!!!!!!!!!!!!!!!! :eek: :eek: :eek: :eek: :eek:

[31 Flavas]

hey uhm... i'm want to do a drop dupe encode using Donald Graft's Dup avs filter, but the link doesn't work... Would someone point me to a working link? Thanks.

[Wilbert]

http://neuron2.net/mine.html

[Sirber]

The videodupframedropper.dll included with producer 10 is 3 times larger than the one that we can find on Karl's site. Is it the same?

[karl_lillevold]

UPX

[Sirber]

:confused:

[karl_lillevold]

Sirber! : Google :) http://upx.sourceforge.net/

[Sirber]

ha! that UPX was lost somewhere in my legendary brain :D sorry :)

[Gannjunior]

Hi,
i'd like to test RV codec,'cause I've seen very interesting results in anime encoding at low and very low bitrate.I've downloaded RV10 producer.I'd like to test the setting "...RV9+EHQ+DropDupe, a new 250 kbps RV9 version with this new filter looks virtually identical to the 350 kbps version...",i've seen at the beginning of this discussion.I've downloaded easy real media producer 1.51 e autoRV10 1.0 beta2.My problem is that I need Audio stream with a bitrate of about 20Kbps,but autorv10 doesn't permit me to go under 44kbps,while media producer 1.51 yes.On the other hand,in autorv10 i can edit the 'audience file' to have the "RV9+EHQ+DropDupe@250kbps setting", introducing the code I've in the first page of discussion,that should permit me to use the videodupframedropper.dll in 'prefilter'...But in this case I have another problem,'cause I don't know the 'language' of audience file: in which point of audience file, produced by autorv10,i have to put the code of prefilter for 'dropdupe' etc...?
Any suggestion?
My objective is compress anime(source dvd,vcd,vhs...) with 250kbps in video stream and about a 20kbps music stereo high respons RA8 in audio stream,with the better result between encoding time and quality.And,if i've understood well,videodupframedropper could be very useful...
T.I.A.

ciao!:)

[Sirber]

yes, it would. Check RealAnime at in my signature :).

[Dark-Cracker]

u need add the dropdupe settings like this :

...
<parInputs>
<input xsi:type="avFileInput">
<filename type="string">e:\avis\t03.avi</filename>
<prefilters>
<prefilter xsi:type="videoDupFrameDropper">
<enabled type="bool">true</enabled>
<maxDroppedFrames type="uint">3</maxDroppedFrames>
<maxAvgSSD type="uint">700</maxAvgSSD>
<maxAreaSSD type="uint">6000</maxAreaSSD>
<maxAvgChromaSSD type="uint">200</maxAvgChromaSSD>
<maxAreaChromaSSD type="uint">1500</maxAreaChromaSSD>
<earlyExit type="bool">true</earlyExit>
<enableDetailedLogInfo type="bool">false</enableDetailedLogInfo>
<pluginName type="string">rn-prefilter-dupframedropper</pluginName>
</prefilter>
<!-- -->
</prefilters>
</input>
</parInputs>
...

i think audio at 20 kbps will not give u a correct sound even if u use dual mono but it's only my personnal opinion :)

Ps: out of topic and even if i don't like made personal publicity , next release of autorv10 will add support for dropdupe.

++

[Gannjunior]

Thanks 2 both you, 4 your answers!now i come in your respective encoders' discussion to make you some specific question.

Dark-Cracker,I've been succeed to do work the dropedrupe.the quality obtained,relationated to the size,is really 'embarassing' !! i'm an estimator of xvid,divx...but at bitrate under 350 kbps and 352x or 320x,can't do anything against r.v. ...


i think audio at 20 kbps will not give u a correct sound even if u use dual mono but it's only my personnal opinion
I obviously agree with you,but I've many animes' series in vcd format that "are made with feet" ('fatte con in piedi'),as we would say here in Italy.It should be enough to know a bit of avisinth language to do decent work even with mpeg1 format.These series are taken from very old tv registration on vhs and you can imagine the audio problems...so I can do not much...for this reason,in these chance (220mb of bad quality for 20 minutes!), i choose audio bitrate so low... ;)
ciao!

[Sirber]

I did some Futurama once, here's the settings:

300kbps total
32kbps Sterel Music HR
DropDupe @ 1, default
no filtering

Quality was like the original MPEG, with estimated average 17.71% difference ;).

[rezard]

this prefilter cause sync problem.

in my last encoding, with 65535 'maxDroppedFrames' value, this filter dropped 16870 frames out of original 30833 frames.
when playbacking the resulting file, the first part of source video is skipped. about 1 second.
so, the video is faster than the audio by about 1 second.

I compared the sync in media player classic, becaues seeking in vdubMod via directshowsource() is very unstable.
by the way, by vdubMod I can notice that the resulting file has 30812 frames which is less than 30833 frames of source.

with 3 'maxDroppedFrames' value, the video is faster than source by about 5 frames.
the total number of frames is 30827.

without this filter, the video is still faster than source by about few frames!
the 'encodeAllFrames' tag in job file seem not to work.
the total frames number is 30830.

[E-Male]

ist the maximum value accepted 3 or 5, with any higher value being ignored?

[rezard]

higher values than 3 are working.

[rezard]

by the way, I used Helix DNA producer 10.0.0.545 for last encoding, without changing the 'videodupframedropper.dll'.

[Sirber]

Originally posted by rezard
in my last encoding, with 65535 'maxDroppedFrames' value, this filter dropped 16870 frames out of original 30833 frames.Common sense say that the filter wasn't designed to skip frames up to 2730,625 seconds...

[rezard]

consecutive 2730 seconds(about 45minutes:confused: )?

because I set all thresholds of this filter to 1 and max consecutive copies number of dup() is 20, the number of max consecutive drops can hardly exceed 19.

[Sirber]

65535 frames / 24 FPS = 2730 seconds ;)

Why not just setting 20? Anyway... you know what you are doing :)

[rezard]

I think I will adjust values of thresholds someday to make this filter drop more consecutive frames than 19.

[Dark-Cracker]

@karl

does this filter is still usefull with the params "encodeallframe" or the new RC who doesn't produce VFR output.

Bye.

[Sirber]

encodeallframe is for old RC, which prevent it to drop frames at lower bitrate. The new RC should not drop frames, just get very ugly :)

I think the filter will ask to drop frames even if encodeallframe is set, since encodeallframe is just used in case of low bitrate.

[Dark-Cracker]

>I think the filter will ask to drop frames even if encodeallframe is set, since encodeallframe is just used in case of low bitrate.

i am not sure.

[Sirber]

Karl? :cool:

[karl_lillevold]

encodeAllFrames makes the codec encode all frames that are passed to it without skipping any due to bitrate considerations. However, pre-filters like dropDupe might still drop frames. So encodeAllFrames does not affect the operation of the dropDupe filter.

[Dark-Cracker]

ok, so in fact dropdup is useless for the new RC.

[Sirber]

not recommanded :)

[karl_lillevold]

Originally posted by Dark-Cracker
ok, so in fact dropdup is useless for the new RC.
It should work just fine. What the new RC needs is that the same number of frames are encoded in each pass, so as long as you don't change any settings that could affect how frames are dropped with the dropdup filter in between the two passes, it should function just fine.

[Sirber]

yes, but at same bitrate, better tweak the RC than drop frames...

[karl_lillevold]

with duplicated frames in animation, better drop the duplicated duplicated frames, for better coding efficiency.

[Sirber]

In my tests, for animations, got better results using RC with 10% low boost than using dropdupe with old RC, similar filesize.

[karl_lillevold]

yes, it is true it is hard to get DropDupe to work really well. In many cases, animation looks better when you do code all the frames, just due to the grain/noise differences from frame to frame.

[Sirber]

no noise and no grain, MPEG4 wash it all ;)

[rezard]

but I worry that the 'lowBitrate boost' is likely to make fast motions blurry.
this filter is a must for me because it make output of more quality or less size.
the 'grain/noise differences from frame to frame' is not important at all to me.:)

by the way,
isn't there anyone who noticed that the codec skip (or drop?) the first few or some frames of original video, especially with dropdupe filter?(as I said before)
or is this a known problem?

[Sirber]

"lowBitrateBoost" prevent the curve to go too low. I had problems with bad motion artefacts in low motion scenes. Also, reducing high by 30% is also good. The best way is always to test :) I'm not using those anymore, I'm on Constant Quality @65% :D

[lunaticmoon]

Hi. Qustion! (Though I have not tried yet ^^; )

Could this technique neatly display the frame at equal intervals?

Anime's 24fps-ize requires blow:

A---A---A---A---O---O.....
|
v
A---A---A---A---O Drop ...
|
v
A----A----A----A----A..... display the frame at equal intervals

Behavior fo original RV10/9:

A---A---A---A---O---O.....
|
v
A---A---A---A---O Drop ...
|
v
A---A---A---A-------A..... It doesn't look smooth.

Is this problem solved?

Thanks.

[Sirber]

Displaying 2 times the same frame or only 1 and keep it will produce the same. Or maybe I get it wrong :(

[lunaticmoon]

Sorry, I did not accurately understand English of Sirber.

Anyway, when IVTC was accurately done, the display interval is different though it is 24 frames per second on data.
Therefore, you are sure to have the sense of incompatibility at the same level as the display of 24fps part with 30fps. :(

[Rumbah]

I think what Sirber was trying to say is that it makes no difference if there are 25 equal frames shown in one second or 1 of the equal frames for the whole second.
There should be no difference in smoothness except the frames are not equal but similar.