Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.


Go Back   Doom9's Forum > Capturing and Editing Video > Avisynth Development

Thread Tools Search this Thread Display Modes
Old 23rd March 2018, 10:45   #1  |  Link
Registered User
Join Date: Sep 2005
Posts: 135
DupStep, a duplicate frame detector & decimator (x64 only; alpha)

DupStep is a duplicate frame detector/decimator, currently in an "alpha" state. It was written from scratch by me, and is not based on any other similar filter, though it can emulate some. Although I normally don't release projects this early, I would like the community's help with regard to feedback, testing, and feature suggestions, so I'm making it available now.

For those that are wondering how it compares with other, similar filters, I've put up a spreadsheet comparison that I think is somewhat accurate. Please let me know if you spot any incorrect information or if you can fill in some of the blanks (??? cells) and I'll update it appropriately.

The readme is too long to fit within this post, so instead you can read it here.

Keep in mind I'm actively developing it and as it's still in alpha, thus many things are subject to big changes. Right now my priorities for upcoming releases are:
High priority:
- SSIM based metric support
- Reduction kernel instead of atomics
- Greatly improving the displayed information in the overlay (the 'show' parameter)
- Improve the dsstats.pl script to allow parameter input and computed stat output (currently only displays intermediate stats)

Medium priority:
- Adding range-limiting support (Note: This is not the same as duplicate limiting!)
- RGB support
- VFR input

Low priority:
- 32-bit floating point color space support
- Dynamic threshold support
- Adding frame type support (see 'ptp' parameter) to other source filters (e.g. LSMASH, DSS2?)
Last but not least, the relevant links:
My filters: DupStep | PointSize

Last edited by `Orum; Today at 12:05.
`Orum is offline   Reply With Quote
Old 23rd March 2018, 15:25   #2  |  Link
Registered User
Join Date: Sep 2010
Location: Ukraine, Bohuslav
Posts: 87
Interesting. I wonder if it can be used to make VFR from QTGMC 60p output, since Tdecimate doesn't support 60p sections at mode 5. Anyway thanks for your efforts, will check it later.
Me on GitHub | My Telegram
DJATOM is offline   Reply With Quote
Old 2nd April 2018, 21:33   #3  |  Link
Registered User
Join Date: Sep 2005
Posts: 135

New release, with numerous bug fixes. Full changelog:
Speed: OpenCL is now used for metric calculation, allowing for multi-threading on a CPU or using
  a GPU instead
Speed: All frames read during cache generation are now cached on the OpenCL device so they are
  only requested once from AviSynth
Added: New function "DS_dumpocl()" to list OpenCL platforms and devices with index
Param: Added parameters 'oclpi' and 'ocldi'
Param: Default 'cdepth' changed from 2 to 5
Param: Maximum value of 'cdepth' changed from 10 to 250
Param: dsstats.pl now features additional parameters; run "dsstats.pl -h" for information
Fixed: SSD metrics could be incorrect (lower than they should be) for 16-bit video
Fixed: Cache depth promotion was not working in some cases (affected ifmcm modes 1 & 2)
Fixed: Frame type cache was leaking a small amount of memory
Fixed: Cache was delivering incorrect frame deltas when within a radius of 'cdepth' of the final
  video frame, sometimes reading outside of buffer memory (undefined values)
Fixed: Cache had incorrect frame types when cdepth > 1
Other: Documentation updated regarding when 'ptp' should be disabled, and detailed new concerns
  regarding device memory usage during cache generation
Other: dsstats.pl now validates cache version
Other: Minor refactoring
It's also a little bit faster than the previous release when it comes to generating metrics:

Finally, if you're willing to test, please let me know if you get the same cache file data between the previous release and the current one, keeping the following in mind:
  • Set 'cdepth' to the same value on each release to make the comparison easier. Note that the default value of this parameter has changed.
  • Comparisons should be done with the dsstats.pl from v0.02 using metric-only dumping (-m). Redirect the output from each release to a csv file (e.g. "dsstats.pl -m -s v0.01.dsd > v0.01.csv") and see if they are identical for both releases.
  • Alternatively, if you don't want to install perl, either use a source filter other than FFVideoSource() or set cdepth=1 and compare the files directly (e.g. via a hashing algorithm or a bit-for-bit comparison)
  • 16-bit comparisons between v0.01 and v0.02 aren't valid, as v0.01 would generate incorrect metric results for 16-bit video in some circumstances. All other supported depths shouldn't have the problem.
I'm particularly interested to know if the results match between the two if you generate the cache on v0.02 using an AMD GPU. I have not tested DupStep with an AMD GPU as I don't have access to one at the moment.
My filters: DupStep | PointSize

Last edited by `Orum; 2nd April 2018 at 23:35.
`Orum is offline   Reply With Quote
Old Yesterday, 13:03   #4  |  Link
Registered User
Join Date: Sep 2005
Posts: 135

Version 0.03 is released, the big changes being:
  • Completely dynamic cache depth, eliminating the 'cdepth' parameter as well as the hybrid 'ifmcm' modes. Now 'ifmcm' should almost always be set to '0' (the new default) which should give the best quality while being slower than the other modes only when changing parameters (and probably only slightly so at that, as metrics are now reused whenever possible).
  • The addition of block-based metrics, which are very useful in detecting slow fades of static images. SSBD (a block metric) is the new default metric.
  • New 'pco' parameter gives control of how planes are combined when using both luma and chroma.
  • Better default value for 'Cweight' parameter.
The full changelog for this release:
Speed: Cache depth is now dynamic and generates/caches only what is needed for the current
Speed: Cache data from file is always used when available and only recalculated when necessary
  (exception: changing block size recalculates all metrics)
Speed: OpenCL kernel should now be able to take advantage of SIMD instructions for platforms
  that support them
Speed: Avoids unnecessary memory copying when executing OpenCL kernels on a CPU
Added: New function "DS_cachefp()" to cache only frame priorities, allowing 'ptp' to be used
  even when (spatio)temporal filtering is used prior to metric generation
Param: Removed the 'cdepth' parameter as cache depth is dynamic on a per-frame basis now
Param: 'ifmcm' modes 1 and 2 removed and replaced by former modes 3 & 4
Param: Default mode for 'ifmcm' set to 0 (direct)
Param: Removed "raw" metrics (RSAD/RSSD)
Param: New 'metric' settings for block-based metrics
Param: Reordered the 'metric' settings to put squared metrics at lower values than their
  absolute counterparts
Param: Default 'metric' now set to SSBD (2)
Param: Automatic (negative) 'Cweight' uses new values based on empirical data, and no longer
  considers chroma subsampling factor as a factor
Param: New parameter 'pco' to set the plane combination operation
Param: Added 'blksize' parameter to set the block size for block metrics
Param: Added new option to prefer CPU to 'oclpi'; subtract 1 from old options if < -1 to get the
  equivalent new option index for them
Param: Default for 'oclpi' is now set to prefer CPU (-2)
Param: Reordered the parameters
Fixed: Potential overflow when calculating chroma metrics has been fixed
Fixed: A potential race condition that could lead to incorrect results was resolved
Fixed: Corrected a typo in dsstats.pl help (-h)
Fixed: Multiple known issues from v0.02 resolved
Other: Metrics are now all stored as floating point values, pre-normalized to resolution where
  appropriate (previously they were post-normalized, which could cause problems when resolution
  changed between cache generation and usage)
Other: Pixel metrics (SSPD/PAD) are no longer clamped after being scaled by chroma weighting,
  permitting more aggressive chroma weighting
Other: Version bump for cache file; there is no compatibility with caches from older versions
Other: OpenCL device memory usage reduced as it only ever needs to hold the two compared frames
Other: DS_dumpocl() now displays OpenCL version supported (both for platforms and devices)
Other: Significant refactoring
My filters: DupStep | PointSize

Last edited by `Orum; Yesterday at 13:32.
`Orum is offline   Reply With Quote

dedup, deduplicate, duplicate, vfr

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +1. The time now is 16:43.

Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2018, vBulletin Solutions Inc.