RV9 Animation DropDupe Pre-filter

[karl_lillevold]

If you would like to try a new pre-filter for RV9 that drops duplicated frames before encoding, please download videodupframedropper.dll and place this DLL in the tools folder in the producer directory. Helix Producer 9.2 M4 is recommended, but it should work in earlier versions as well. If you would rather wait until it is more stable, please do. Parameters etc are likely to change, to make it work for noisier material. This pre-filter is open source, and contributions are welcome, both in the form of suggestions, parameter tuning, and code modifications. See below for more info.

DropDupe filter works by detecting duplicated frames and drop them before the encoder sees them. Since the RM file format supports variable framerate, playback will still appear just like if the frames were encoded.

The DropDupe pre-filter improves compression efficiency with RV9 for two reasons:
1) There is no need to encode the duplicated frames. These frames normally require few bits though, perhaps 5-10% overall.
2) more importantly, the adaptivity in the RV9 codec can do a better job when the duplicated frames are removed, so the overall improvement may in many cases be higher than these 5-10%.

This filter is very useful with RV9-EHQ, because encoding without the duplicated frames is much faster. If every 2nd frame is removed, it will be 2X faster. The DropDupe filter takes very little time by itself, being MMX optimized and it exits early whenever thresholds are exceeded. EHQ at level 80 is strongly recommended in addition to this pre-filter, for the best possible animation encodes.

Recommended (idea from midiguy): Use Donald's Graft's Dup as front-end for DropDupe. See section below.

How to enable the Animation DropDupe filter

GUI Front-ends:
Sirber's RealAnime

Producer Job file:
Insert this code in the prefilters section of the Producer job file (after making sure videodupframedropper.dll is in the tools folder). see post below for complete job file example.

 <prefilter xsi:type="videoDupFrameDropper">
   <enabled type="bool">true</enabled>
   <maxDroppedFrames type="uint">3</maxDroppedFrames>
   <maxAvgSSD type="uint">700</maxAvgSSD>
   <maxAreaSSD type="uint">6000</maxAreaSSD>
   <maxAvgChromaSSD type="uint">200</maxAvgChromaSSD>
   <maxAreaChromaSSD type="uint">1500</maxAreaChromaSSD>
   <earlyExit type="bool">true</earlyExit>
   <enableDetailedLogInfo type="bool">false</enableDetailedLogInfo>
   <pluginName type="string">rn-prefilter-dupframedropper</pluginName>
 </prefilter>

Parameters
maxDroppedFrames : max number of frames to drop before letting a frame pass through.
maxAvgSSD : max SSD per 16x16 area averaged for the luma plane of the image, when comparing current with previous image, before considering the frame to be a new frame.
maxAreaSSD : max SSD in any 16x16 area, before considering the frame to be a new frame.
maxAvgChromaSSD : same as maxAvgSSD, but for the chroma planes (Cb and Cr).
maxAreaChromaSSD : same as maxAreaSSD, but for the chroma planes (Cb and Cr).
earlyExit : exit early from detection function when one of the thresholds is exceeded. Disable for accurate numbers in log.
enableDetailedLogInfo: log information for every frame : DROP/keep and all the measured SSD numbers. Format: (framenum) max: maxSSDluma maxSSDCr maxSSDcb avg: avgSSDluma avgSSDCr avgSSDCb. When SSD is 999999 or 2*threshold for avgluma it means early exit prevented exact calculation. Disable earlyExit to obtain exact numbers.

Note: SSD = Sum of Squared Differences
If a paramater is not included in the job file, the default value will be set as given in the example above. The minimal amount of code in the job file to enable the filter with all parameters set to their default value is

 <prefilter xsi:type="videoDupFrameDropper">
   <enabled type="bool">true</enabled>
   <pluginName type="string">rn-prefilter-dupframedropper</pluginName>
 </prefilter>

Source code
Along with the other Producer pre-filters, this is open source and contributions are welcome! The source code is available through the Helix Community via CVS checkout from producersdk/plugins/transform/videodupframedropper, or simply browse the CVS repository online. Please follow instructions on the HC website for CVS access and build setup info. Maybe you have an idea for your own pre-filter? This is a nice and simple example to start with.

How do I know it's working?
Set enableDetailedLogInfo to 'true' and look in producer.log, or when playing back the RM files after encode, go to Tools->Playback Statistics. Click the Streams tab, and select the RealVideo stream in the dropdown box. The Frame rate will show the resulting frame rate after the duplicated frames were dropped. For instance, for a 23.976 source, with every 2nd frame dropped (on average) it will show around 12 fps. If it does not appear like any frames have been dropped, and the clip does indeed have duplicated frames, you have to enableDetailedLogInfo and look in producer.log for the SSD values found, and adjust thresholds.

Use Dup (AviSynth filter) as front-end
Donald Graft's Dup Avisynth filter does a better job with noisy clips, and has more advanced options for look-ahead, blend, use-last frame in sequence. If you add this in your Avisynth script before feeding it to Producer with DropDupe set to low thresholds (since Dup creates true duplicated frames), you will get very good results:
In your AviSynth script, include
LoadPlugin("<path_to_plugin>\dup.dll")
Video=Dup(Video,show=false)
Then set show=true, and load avs in VirtualDub to make sure you tune the Dup thresholds to your clip. See Dup documentation. Defaults have worked well, or maybe a little too aggressive, for my limited testing.

Then set the all thresholds for DropDupe to l (one), before using the .avs script with Dup as your input to producer. Remember to set show=false, otherwise DropDupe will detect changed frames, and not drop them. Besides you don't want to encode the debug info from the Dup filter.


Thanks
Thanks to Kaiousama for the original idea, testing and feedback, iwod for bringing it back up and experimenting with Avisynth filters, Sirber for example source material and testing, and to Yusaku for posting this little challenge, that I could not resist: "I think that this would be a great feature for anime encodes, but you should file this as a request to XviD (or other video format) developers, as it cannot be done on the container level. (and there is not much of a chance of it getting into encoding applications for RM, IMO)"

todo (will be edited when done)
- take history into account for more reliable and stable detection with noisy clips.
- Assumes MMX is available. This will be fixed shortly as well, but who has a PC without MMX these days?
- You tell me, after trying it out...

notes
this filter was not used for Sirber and Ramirez' anime codec comparison. Even though RV9+EHQ is possibly considered the best result at 350 kbps, pretty close to XviD, but much better than the other codecs, with RV9+EHQ+DropDupe, a new 250 kbps RV9 version with this new filter looks virtually identical to the 350 kbps version (keep in mind, the compression artifacts you may see, are already in the high bitrate DivX source. See this thread).

The next post contains a complete job file example for Helix Producer 9.2. This will not work in the GUI Helix Producer, but if anyone is interested, I will post the code to use for GUI Helix Producer job files later.

[Sirber]

Great!

I'll add it to my software and test it. Thanks Karl!!! Is it me or your team is working quite fast? 1 week for EHQ and 1 week for DD

[karl_lillevold]

Here is a complete job file to use with Helix Producer 9.2 that enables the Animation drop duplicate frames pre-filter. Edit this with your favorite text or XML editor, save to myjobfile.rpjf (or any extension). Let me know if you have an XML editor that works well

Run with producer -j myjobfile.rpjf -lc w,i,d,e

The -lc (logging category) options are w=warning, i=information, d=diagnostics, and e=error messages. Producer's output is also placed in the producer.log file. Sometimes very handy. Even more information (too much) can be logged by having an empty file called debug.txt in the same folder as producer.

<?xml version="1.0" encoding="UTF-8"?>
<job xmlns="http://ns.real.com/tools/job.2.0"
   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
   xsi:schemaLocation="http://ns.real.com/tools/job.2.0 
   http://ns.real.com/tools/job.2.0.xsd">
   <enableTwoPass type="bool">true</enableTwoPass>
   <parInputs>
     <input xsi:type="avFileInput">
       <filename type="string">e:\avis\t03.avi</filename>
       <prefilters>
         <!--                                        -->
         <!-- Animation DropDupe pre-filter settings -->
         <!--                                        -->
         <prefilter xsi:type="videoDupFrameDropper">
           <enabled type="bool">true</enabled>
           <maxDroppedFrames type="uint">3</maxDroppedFrames>
           <maxAvgSSD type="uint">700</maxAvgSSD>
           <maxAreaSSD type="uint">6000</maxAreaSSD>
           <maxAvgChromaSSD type="uint">200</maxAvgChromaSSD>
           <maxAreaChromaSSD type="uint">1500</maxAreaChromaSSD>
           <earlyExit type="bool">true</earlyExit>
           <enableDetailedLogInfo type="bool">false</enableDetailedLogInfo>
           <pluginName type="string">rn-prefilter-dupframedropper</pluginName>
         </prefilter>
         <!--                                        -->
       </prefilters>
     </input>
   </parInputs>

   <parOutputs>
     <output>
       <destinations>
         <destination xsi:type="fileDestination">
           <filename type="string">d:\anime.rmvb</filename>
         </destination>
       </destinations>
       <mediaProfile>
         <audioMode type="string">music</audioMode>
         <disableAudio type="bool">false</disableAudio>
         <disableVideo type="bool">false</disableVideo>
         <resizeQuality type="string">high</resizeQuality>
         <videoMode type="string">normal</videoMode>
         <audienceRefs>
           <audienceRef>My Audience</audienceRef>
         </audienceRefs>
       </mediaProfile>
     </output>
   </parOutputs>
   <audiences>
  <audience>
    <avgBitrate type="uint">600000</avgBitrate>
    <maxBitrate type="uint">1300000</maxBitrate>
         <name type="string">My Audience</name>
    
    <streams>
      <stream xsi:type="videoStream">
      <pluginName type="string">rn-videocodec-realvideo</pluginName>
      <codecName type="string">rv9</codecName>
        <!-- EHQ settings for RV9 -->
        <codecProperties type="bag">
          <encoderComplexity type="uint">80</encoderComplexity>
          <customPacketSize type="uint">16000</customPacketSize>
        </codecProperties>
      <encodingType type="string">vbrBitrate</encodingType>
      <quality type="uint">1</quality>
      <maxStartupLatency type="double">25</maxStartupLatency>
      <maxFrameRate type="double">30</maxFrameRate>
      <maxKeyFrameInterval type="double">10</maxKeyFrameInterval>
      <enableLossProtection type="bool">false</enableLossProtection>
      </stream>
        <stream xsi:type="audioStream">
          <!-- Audio codec flavor determines bitrate and
               stereo vs mono.  See 
               docs/AudienceFile.htm (Audio_Codec_Tables) -->
          <codecFlavor type="uint">28</codecFlavor>
          <codecName type="string">cook</codecName>
          <pluginName type="string">rn-audiocodec-realaudio</pluginName>
          <streamContext type="bag">
          <audioMode type="string">music</audioMode>
          <presentationType type="string">audio-video</presentationType>
          </streamContext>
        </stream>
      
    </streams>
  </audience>
   
  </audiences>
</job>

[Sirber]

Hum...

What if I don't use job files? Can I use it from the Audience file? Could I get the Codec version?

[karl_lillevold]

Originally posted by Sirber
Great!

I'll add it to my software and test it. Thanks Karl!!! Is it me or your team is working quite fast? 1 week for EHQ and 1 week for DD

Thanks! The team in this case has been me by myself. EHQ took more than 1 week though, more like 3-4 I have been so lucky to have enough time to work on RV9 improvements lately. My video codec co-worker, has not been so lucky, having had to struggle with ports to various devices, but he is eager to get going too now, with various potential 2-pass VBR improvements.

What if I don't use job files? Can I use it from the Audience file?

Unfortunately, no. Job files are so much more flexible, I would strongly recommend moving to using those instead. It should be possible to write out a job file with the extra code necessary for job files, and call producer with -j myjobfile.rpjf. Everything (and more) which you can specify on the command line, can relatively easily be added to the job file. Let me know if you have any specific problems.

This filter is one reason. Another reason is that for multi-channel encoding, you have to use separate input files for audio (ac3) and video (avs), and the only way to specify this is via job files.

[Sirber]

I'm a little noob in jobfiles, is there a way to add the job file + the audience, with the jobfile with only prefilter settings? I'm a little lost...

Or I'll move all to job file.

[karl_lillevold]

i was a noob with job files too, still am, but I am learning.

I am not quite sure I understand your question, but if you look at the example above, you see how the audio "My audience" is specified in the job file. Then in the <mediaProfile> section of the <parOutputs> section I specify to use "My Audience" as the audience for this output.

I just tried to use an audience that's defined in its own file, and then run producer with -j jobfile -ad audiencefile, but was not able to. So as you can see, I am still learning job files too. Check the jobfile.htm documentation in the docs sub-folder, or this link:
https://helix-producer.helixcommunit...File/index.htm
from the main Producer 9.2 specifications at
https://helix-producer.helixcommunit...ecs/index.html

[Sirber]

Thanks Karl!

Your help is greatly appreciated

I'll move all to job file then.

[Dark-Cracker]

why not add this filter in the command line like "black level","deinterlace" or "video noise reduction" filter made by realnetwork ? somethink like :

-dup 3,2500,10000

could be more usable IMHO.

>Another reason is that for multi-channel encoding, you have to use separate input files for audio (ac3) and video (avs), and the only way to specify this is via job files.

if u use wavsource() in avisynth script using a 5.1 .wav file real producer will not support it ? using directely .ac3 file need use the directshow filter (and this not always work if u not have the right dshow filter).

Bye.

[karl_lillevold]

Originally posted by Dark-Cracker
why not add this filter in the command line like "black level","deinterlace" or "video noise reduction" filter made by realnetwork ? somethink like :

-dup 3,2500,10000

could be more usable IMHO.

Maybe so, but then I would have to have it added to the spec. Talk to the right Producer people, convince them it's a good idea, etc. etc. Not to mention recompile all of Producer, and not only this pre-filter DLL. More importantly, this is way more flexible, everyone can make changes, add/change parameters, re-compile, plug right in. Everyone has access to the source code for the filter itself. It is fully backwards compatible with previous producer versions. Everyone can add their own pre-filters, independent of Producer versions. You can do none of these things with cmd line parameters.

I know it's a lot of work for the GUI front-ends to move to using job files instead of cmd line paremeters, and I am sorry this is the case, but once you do, you will soon realize it is more flexible.

if u use wavsource() in avisynth script using a 5.1 .wav file real producer will not support it ? using directely .ac3 file need use the directshow filter (and this not always work if u not have the right dshow filter).

This may or may not be possible. I just have not had time to experiment enough (have been busy with video work), so please try it out and see what happens. It has to be a WAVEFORMAT EXT file, and it does not seem like Besweet produces such a file. Avisynth also needs to properly support 5.1 channels, and pass them on to Producer. This may or may not work, as well, just have not tried. Let's move this discussion to the thread I will start soon on RealAudio Multi-Channel encoding

[Dark-Cracker]

yes u are right add it to the command line it's less flexible i have only ask this because all other filter are in cli mode but for sure if this filter should evolute it's not a good idear to recompile a new producer version each time

u are right moving gui on job file request a lot of modifications.

for the audio u are right it's not the good thread.

[Sirber]

In your job exemple, there is this:

<disableAudio type="bool">true</disableAudio>
<disableVideo type="bool">true</disableVideo>

I guess I have to set false there?

[edit]

Producer is scaring me:

Diagnostic: keep max_ssd: 25927, frame_ssd: 5000
Diagnostic: keep max_ssd: 12928, frame_ssd: 5000
Diagnostic: keep max_ssd: 14845, frame_ssd: 5000
Diagnostic: keep max_ssd: 17365, frame_ssd: 5000
Diagnostic: keep max_ssd: 21805, frame_ssd: 5000
Diagnostic: keep max_ssd: 149094, frame_ssd: 5000
Diagnostic: keep max_ssd: 10352, frame_ssd: 5000
Diagnostic: keep max_ssd: 16384, frame_ssd: 5000
Diagnostic: keep max_ssd: 13038, frame_ssd: 5000
Diagnostic: keep max_ssd: 16600, frame_ssd: 5000
Diagnostic: keep max_ssd: 42320, frame_ssd: 5000
Diagnostic: keep max_ssd: 16174, frame_ssd: 5000

what's that?

[karl_lillevold]

ah, yes, my mistake, both should be 'false'. I was using it to compress T03 without audio, thus having one 'true' and one 'false' and edited both to 'true' after posting. Should have been 'false' Thanks! Fixed now.

[Sgt_Strider]

Can this filter be use for anything other than anime or animation type of DVD's? Is this filter effective for "real life" movies like black hawk down, matrix, Band of Brothers, and etc.?

[Sirber]

The filter removes frames with no motion, and in real movies, there is always some actions

[karl_lillevold]

@Sirber: I think I need to reduce the pre-filter logging messages. 'keep' means that the SSD is so high it has decided to keep the frame. When it says 'DROP' it has been dropped. when frame_ssd says 5000, it means it has exited early after finding that maxAvsSSD has been exceeded. max_ssd is the max areaSSD found before the function exited.

@Sgt_Strider: I don't think this filter will provide much benefit unless the frames are truly duplicated, but maybe someone can improve it or write another version that drops frames under other circumstances, like extreme high action, spatial complexity, but how this would look is another matter.

[Sirber]

DupeDrop is dropping about 3/4 of total frames. I may have to reduce the bitrate...

[Kaiousama]

I've tested this new filter....... very very good results:

It does exactly the needed work and the side effects are little and probably easily resolvable.

I've tested it on one of the first akira vobs, the one where testsuo is admiring kaneda's bike and all the next bike racer scene; in this PAL clip no one of the frames had to be dropped because it has Zero duplicated frames --> the filter drops only 20 frames (it is a fraction of a percentage point, so a little error amount)

The Second tested clip is the Juuni Kokki 6th episode, it has 24,16,8,4 fps scenes. The filter remove in the main of cases exaclty the needed frames with no fluidity degradation during the playback. There are only 2 macroscopic errors:
- The first is in the first 5 seconds during the opening, there is an almost white scene where the point of view is inside a fog that is flowing toward the watcher; in this case the filter wrongly drops many frames because the movement is very little and the changing is almost only chromatic
- The second is on the first 10 second in the ending, the screen is black and still and slowly appears a yellow mon, the filter didn't recognise even this scene (probably the same chroma missing feature)

Conclusions : in my opinion the most important next feature is chroma informations handling during the duplicates' recognition.

On another side, this test make me find even a new and (in my opinion) better default values for the filter, here they are:

<prefilters>
       <!--                                        -->
       <!-- Animation DropDupe pre-filter settings -->
       <!--                                        -->
       <prefilter xsi:type="videoDupFrameDropper">
         <enabled type="bool">true</enabled>
         <maxDroppedFrames type="uint">5</maxDroppedFrames>
         <maxAvgSSD type="uint">700</maxAvgSSD>
         <maxAreaSSD type="uint">6000</maxAreaSSD>
         <pluginName type="string">rn-prefilter-dupframedropper</pluginName>
      </prefilter>
   <!--                                        -->
</prefilters>

I've increased the maxDroppedFrames to 5 in order to properly handle 4fps scenes (24fps / 4 fps = 1 good frame every 6, so 5 consequential drop frames)
I've decreased the other 2 values because both in akira (clean source) and Juuni Kokki (noisy source), duplicates' maxAvgSSD was around 300-500 and maxAvgSSD was near 2000-5000.
Try them yourself to check if my defaults are validable as generally usable default values.

@ Karl:

It would be useful to have written into the log file even if EHQ was enabled or not during the encoding session (better with its strenght values).

a little question: when the Dupe filter checks for duplicate, is needed the frame maxAvgSSD AND maxAreaSSD to be verified for recognising the frame as duplicate or it's sufficient one of the 2 conditions to be true to drop the frame?

my opinions : The filter's log information are good as they are, and i think it's a good solution even to have the filter separated from the producer main application (jobs are all but difficult to use).
One thing, it would be good to add to log information the frames' number of each processed frame in order to visually check later on the source clip which are the dropped frames.

[netterpaladin]

Hi,
(a little of topic)

sounds like a very nice improvement for anime encoding. My question now is:

Real seems to be a very good choise as the video codec for anime

BUT matroska seems to be the container of choise for anime because of its awesome subtitle support.

Now if i remember right, real does not work with matroska? Is that true? It seems that the combination of real video and matroska subtitles is the PERFECT Combo. Is there any chance to see that in the future?


Daniel

[Dark-Cracker]

use the search button and u will find topic speaking about the possibility to add support of real video in matroska.