RemoveDirt - pfmod

[pinterf]

RemoveDirt v1.0 (20250108)
https://github.com/pinterf/RemoveDirt/releases/tag/v1.0
Milestone: Vapoursynth (API v4) support

RemoveDirt v0.9.2 (20190324)
https://github.com/pinterf/RemoveDirt/releases/tag/v0.9.2

Link to the previous topic:
https://forum.doom9.org/showthread.php?s=&threadid=70856


# RemoveDirt - pfmod
A plugin for removing dirt from film clips.

Port of classic RemoveDirt 0.9 to Avisynth v2.6 interface (x86/x64), adding new color spaces, bugfixes.

Functions in plugin: RestoreMotionBlocks and SCSelect.
This mod does not support other filters appearing in 1.0beta.

Note: Previous v0.9 DLL versions named differently (RemoveDirtT.DLL, RemoveDirtSSE2.DLL) should be deleted from your plugin folder.

- (20190324 v0.9.2)
- RestoreMotionBlocks: 10-16 bit support. Relevant threshold and noise parameters are bit depth independent.
- minor speedup
- SCSelect: add support for 10-16 bits and 32 bit float clips
- SCSelect: add support for planar RGB
- FIX: SCSelect: make it work properly for large frames (>8MPixel)
- FIX: SCSelect: Makes use the whole frame: now counts the rightmost non-mod32 pixels as well.
- Clang 7.0.1 support (LLVM) with Visual Studio 2017 (15.9.9) (LLVM 8.0.0: not tried yet)
- Install LLVM 7.0.1 (http://releases.llvm.org/download.html, Windows pre-built libraries)
- Install Clang Power Tools & LLVM Compiler Toolchain
- https://marketplace.visualstudio.com/items?itemName=caphyon.ClangPowerTools
- https://marketplace.visualstudio.com/items?itemName=LLVMExtensions.llvm-toolchain
- update html docs
- add clang-built DLLs to the released version

- (20190314 v0.9.1)
- project moved to github: https://github.com/pinterf/RemoveDirt
- built using Visual Studio 2017
- x64 build for Avisynth+
- Added version resource to DLL
- Changed to AVS 2.6 plugin interface
- Fix: RestoreMotionBlocks: grey=false: it was copying 8x4 chroma pixels for YV12 and 8x8 for YUY2 instead of 4x4 and 8x4 blocks
- Fix: SCSelect: Old v0.9 SSE2 code omitted difference checking for every second 8 columns
- Removed MMX code, now requires SSE2
- Reports MT Modes for Avisynth+: MT_SERIALIZED for SCSelect
- Reports MT Modes for Avisynth+: MT_MULTI_INSTANCE for RestoreMotionBlocks (may not be any faster)
- Added Y, YV16 and YV24 support besides existing YV12 and planar-hacked-YUY2

- (20190312)
Initial source v0.9 (2005/05/07) moved to VS2017 project

Links
=====
Project: https://github.com/pinterf/RemoveDirt
Forum: https://forum.doom9.org/showthread.php?t=176199
Additional info: http://avisynth.nl/index.php/RemoveDirt

[pinterf]

Finally I used the original Avisynth source, the VapourSynth port had bugs, I'll point to those places on my fork later next week.

[poisondeathray]

Thanks

[SaurusX]

This is much appreciated.

[filler56789]

Thanks!

[Reel.Deel]

Thank you pinterf, I will update the wiki in the next couple of days.

[ChaosKing]

I think the VS port wasn't finished yet. The one dll I found crashes often. Thx for the release.

I always wondered how hard it is to make "hybrid plugins" now which supports vs and avs+, since avs+ now supports 16bit and the same color spaces? Some plugins like this exists: knlmeans, f3kdb, ffms2
Or is it just too cumbersome because of the different threading models?

p.s. You can also install RemoveDirt via avsrepo now

[real.finder]

been waiting for this, thanks

[Myrsloik]

I think the VS port wasn't finished yet. The one dll I found crashes often. Thx for the release.

I always wondered how hard it is to make "hybrid plugins" now which supports vs and avs+, since avs+ now supports 16bit and the same color spaces? Some plugins like this exists: knlmeans, f3kdb, ffms2
Or is it just too cumbersome because of the different threading models?

p.s. You can also install RemoveDirt via avsrepo now

It's generally very simple. The threading models are extremely similar.

[pinterf]

New build.
Download RemoveDirt v0.9.2 (https://github.com/pinterf/RemoveDirt/releases/tag/v0.9.2)
Just now: experimental (and for me quicker) clang build inside.
Original code is almost completely rewritten by now.

- (20190324 v0.9.2)
- RestoreMotionBlocks: 10-16 bit support. Relevant threshold and noise parameters are bit depth independent.
- minor speedup
- SCSelect: add support for 10-16 bits and 32 bit float clips
- SCSelect: add support for planar RGB
- FIX: SCSelect: make it work properly for large frames (>8MPixel)
- FIX: SCSelect: Makes use the whole frame: now counts the rightmost non-mod32 pixels as well.
- Clang 7.0.1 support (LLVM) with Visual Studio 2017 (15.9.9) (LLVM 8.0.0: not tried yet)
- Install LLVM 7.0.1 (http://releases.llvm.org/download.html, Windows pre-built libraries)
- Install Clang Power Tools & LLVM Compiler Toolchain
- https://marketplace.visualstudio.com/items?itemName=caphyon.ClangPowerTools
- https://marketplace.visualstudio.com/items?itemName=LLVMExtensions.llvm-toolchain
- update html docs
- add clang-built DLLs to the released version

[real.finder]

New build.
Download RemoveDirt v0.9.2 (https://github.com/pinterf/RemoveDirt/releases/tag/v0.9.2)
Just now: experimental (and for me quicker) clang build inside.
Original code is almost completely rewritten by now.


:thanks:

any plan to port RemoveGrainT to x64? it only one missed for run RemoveDirtMC_SE in x64

[pinterf]

:thanks:

any plan to port RemoveGrainT to x64? it only one missed for run RemoveDirtMC_SE in x64

Which function is used?

[real.finder]

Which function is used?

TemporalRepair in TemporalDeGrain_SE

[FranceBB]

New build.
Download RemoveDirt v0.9.2 (https://github.com/pinterf/RemoveDirt/releases/tag/v0.9.2)
Just now: experimental (and for me quicker) clang build inside.
Original code is almost completely rewritten by now.

- (20190324 v0.9.2)
- RestoreMotionBlocks: 10-16 bit support. Relevant threshold and noise parameters are bit depth independent.
- minor speedup
- SCSelect: add support for 10-16 bits and 32 bit float clips
- SCSelect: add support for planar RGB
- FIX: SCSelect: make it work properly for large frames (>8MPixel)
- FIX: SCSelect: Makes use the whole frame: now counts the rightmost non-mod32 pixels as well.
- Clang 7.0.1 support (LLVM) with Visual Studio 2017 (15.9.9) (LLVM 8.0.0: not tried yet)
- Install LLVM 7.0.1 (http://releases.llvm.org/download.html, Windows pre-built libraries)
- Install Clang Power Tools & LLVM Compiler Toolchain
- https://marketplace.visualstudio.com/items?itemName=caphyon.ClangPowerTools
- https://marketplace.visualstudio.com/items?itemName=LLVMExtensions.llvm-toolchain
- update html docs
- add clang-built DLLs to the released version

Thank you! :D
Out of curiosity, what are the benefits of the Clang compiled version over the normal x86 version?

Thank you in advance.

[pinterf]

Clang sometimes generates quicker code, it seems that it features with smarter optimizer than MS. For example the above mentioned TemporalRepair has modes that run 10-15% faster in clang version than with MS built DLL.
I was also looking at the generated assembly code based on pure C source. clang was able to vectorize (parallel computing of pixel data using mm registers) situations where MS simply stuck with linear one-by-one processing.

[tormento]

Clang sometimes generates quicker code, it seems that it features with smarter optimizer than MS.
It would be nice to try your MVTools, MaskTools and RgTools with Clang...

[real.finder]

I just tried clang in xp in vm, it's can't load the dll with error 0x36b1, normal one work fine, clang 64 bit work fine in windows server 2016

[tormento]

I just tried clang in xp in vm, it's can't load the dll with error 0x36b1, normal one work fine, clang 64 bit work fine in windows server 2016
XP is 18 years old. Time for it to go out from parents' home :p

[pinterf]

I just tried clang in xp in vm, it's can't load the dll with error 0x36b1, normal one work fine, clang 64 bit work fine in windows server 2016
LLVM (clang) dropped XP support three years ago, since version 3.8. Now I'm using latest version (8.0)

[StainlessS]

It would be nice to try your MVTools, MaskTools and RgTools with Clang...

Not if it totally breaks XP compatibility for only small speed increase.
EDIT: Removedirt v0.9.1 looks to be XP compatible.

[tormento]

Not if it totally breaks XP compatibility for only small speed increase.
Again this XP debate. And again and again.

Who cares about XP anymore? 2%? 3%?

XP can't even correctly manage modern hardware and amount of memory needed to make some video conversion.

Let's be serious.

Anyway, nobody wants to prevent Pinterf to release dual versions...

[Groucho2004]

Again this XP debate. And again and again.You were the first taking the bait. :D

XP can't even correctly manage modern hardware and amount of memory needed to make some video conversion.
You're correct about managing modern hardware.

However, XP64/WinServer2003-64 can certainly manage large amounts of memory, even more than some consumer versions of newer Windows OS's.

Let's be serious.Why? I prefer to have some fun.

[pinterf]

Not if it totally breaks XP compatibility for only small speed increase.
EDIT: Removedirt v0.9.1 looks to be XP compatible.
0.9.2 should be compatible as well, my clang builds are extras.

[tormento]

However, XP64/WinServer2003-64 can certainly manage large amounts of memory, even more than some consumer versions of newer Windows OS's.
I really can't understand this attachment to ancient tech, like if we were in a steampunk world. There are no drivers for modern video cards or chipsets.

Nobody prevents people to drive a Ford model T car but they can't keep on asking and pretending forever to have spare parts and correct octane fuel from gas stations when they are a tiny minority.

Sometimes the price (i.e. time) to supply ancient technology the support they need becomes simply too much, if not even harmful and detrimental for the majority.

[tormento]

0.9.2 should be compatible as well, my clang builds are extras.
I strongly appreciate your trials to give us faster and modern products.

[tormento]

However, XP64/WinServer2003-64 can certainly manage large amounts of memory, even more than some consumer versions of newer Windows OS's.
Not true (https://docs.microsoft.com/en-us/windows/desktop/memory/memory-limits-for-windows-releases).

[Groucho2004]

I really can't understand this attachment to ancient tech, like if we were in a steampunk world. There are no drivers for modern video cards or chipsets.

Nobody prevents people to drive a Ford model T car but they can't keep on asking and pretending forever to have spare parts and correct octane fuel from gas stations when they are a tiny minority.

Sometimes the price (i.e. time) to supply ancient technology the support they need becomes simply too much, if not even harmful and detrimental for the majority.
I simply pointed out your false statement and you keep rambling on with metaphors.
Did someone hit you with XP product boxes when you were a boy?

[tormento]

I simply pointed out your false statement
Which one? Memory? I replied you with the link.

[Groucho2004]

However, XP64/WinServer2003-64 can certainly manage large amounts of memory, even more than some consumer versions of newer Windows OS's.
Not true (https://docs.microsoft.com/en-us/windows/desktop/memory/memory-limits-for-windows-releases).
Limits:
Windows Server 2003 SP2 x64: 1 TB
Windows XP x64: 128 GB

Windows Vista Home Premium x64: 16 GB
Windows Vista Home Basic x64: 8 GB

Windows 7 Home Premium x64: 16 GB
Windows 7 Home Basic x64: 8 GB

[Groucho2004]

I simply pointed out your false statement
Which one? Memory? I replied you with the link.
I only pointed out one false statement. Maybe you read the posts to which you reply?

[tormento]

I only pointed out one false statement. Maybe you read the posts to which you reply?

Windows 10 Home x64 128 GB

Nobody would be so dumb to upgrade to a modern os and go to Vista.

Upgrade from 7 to 10 is free. And a license for 10 is 7/10€ on Amazon. Have I to say more? You defense of indefensible is boring and illogical.

We went far OT. I will stop here.

[Groucho2004]

Windows 10 Home x64 128 GB

Nobody would be so dumb to upgrade to a modern os and go to Vista.

Upgrade from 7 to 10 is free. And a license for 10 is 7/10€ on Amazon. Have I to say more? You defense of indefensible is boring and illogical.
Everything in your post is beside the point. You made a false statement, I corrected you. After that you just posted random ramblings and nonsense such as the price of Windows 10 on Amazon and that people would be dumb to use an OS that you don't like.

You argue like a 5 year old and logical thinking does not appear to be your forte so you should not accuse others of being illogical.

[StainlessS]

0.9.2 should be compatible as well, my clang builds are extras.


Oops, re-downloading ... :)

I like Windows 2000 best, whats the chance of a New version AVS+ RemoveDirt etc, supporting this ol' classic ? https://www.cosgan.de/images/midi/ekelig/e010.gif

[pinterf]

Oops, re-downloading ... :)

I like Windows 2000 best, whats the chance of a New version AVS+ RemoveDirt etc, supporting this ol' classic ? https://www.cosgan.de/images/midi/ekelig/e010.gif
Yeah, win2k. Good idea. One would dig out further archaic things from memories, pre-SSE2, XP, I propose all of you visiting The 25th Hour (see the books of Abarat) for ideas like supporting ZX Spectrum :) Ahh, it means only some extra day, perhaps weeks or years (the worst case) for programmers who are forced to maintain compatibility and do releases but it definitely worth to see the smile on the faces of the lucky ones owning a vintage hardware.

(yep, it's really a good idea to re-read Abarat - Clive Barker is a genius, I recommend you all, much better than encoding stupid series)

[hello_hello]

Upgrade from 7 to 10 is free.

How does that help XP users?

And a license for 10 is 7/10€ on Amazon. Have I to say more? You defense of indefensible is boring and illogical.

Where on Amazon? I saw a few Windows guides for that price, but the cheapest Win10 I found on Amazon.com was just under $100. My local computer shop (Australia) have it at $150 for Win10 Home and $200 for Win10 Pro, and that's for OEM versions, not retail.

[tormento]

Clive Barker is a genius
Agree.

[tormento]

My local computer shop (Australia) have it at $150 for Win10 Home and $200 for Win10 Pro, and that's for OEM versions, not retail.
If you don't want to "break" the OEM license agreement, you can find legitimate Windows 10 Pro ESD retail keys for 59€ here in Italy

Anyway, the smartest thing is to buy a Windows 7 Pro ESD for about 25€ and then upgrade for free by manually downloading Microsoft Upgrade Assistant, still available from Microsoft.

[jpsdr]

...for ideas like supporting ZX Spectrum :)...

ZX Spectrum... My youth (middle school) and my beginig in the computer world...:p

[tormento]

ZX Spectrum... My youth (middle school) and my beginig in the computer world...:p

Mine too. I suppose we are same age :o

[jpsdr]

Probably globaly the same generation. Made a mistake/confusion... ZX spectrum i was more in high school. It was the ZX81 i was middle school...

Yes pinterf, can you make a ZX81 compatible version...? (And it's the good day for this kin of things) :D

[pinterf]

Probably globaly the same generation. Made a mistake/confusion... ZX spectrum i was more in high school. It was the ZX81 i was middle school...

Yes pinterf, can you make a ZX81 compatible version...? (And it's the good day for this kin of things) :D
O.K. but it has to work with the 1 kbyte version as well :).
(I was only window-shopping when I was 13 or so. Unfortunately I possessed none of the popular machines (C64 and Speccy), they costed us too much, I had only a DIY kit based Z80 machine with 6 kbytes RAM during my 15-18 years old era. Optimization, yes, was always welcome)

[StainlessS]

Arh, my ol' ZX81 with 32KB ram pack (complete with lump of blue-tac or optional RAMPACK wobble feature).
After buildin' the ZX81 (with soldering iron), was a hairy moment when nothing happened on switch on,
was heat sink not quite square with board and slightly overlapped the board edge causing short with
earth strip around edge of board, squared up with board, all okey-dokey.
Boxed as new ZX Spectrum now bout £1500, or so I heard.
Wish I had had the power house ZX80, what a machine !

[Groucho2004]

Boxed as new ZX Spectrum now bout £1500, or so I heard.What? That's insane.

[StainlessS]

Qwik look on ebay,most expensive found was £599.00 (for the 128 KB m/c).
https://www.ebay.co.uk/i/293003953054

I think the £1,500 price tag was mentioned in some review.

EDIT: The £599.00 Spectrum above aint quite MINT, the machine might well be, but the box looks well knocked about.
£1,500.00 tag was for mint boxed.

Mind you, a year or two ago,someone put up a Samsung Galaxy Beam GT xxx (dont remember number), a projector phone,
somebody had one on e-bay for $2,500, they were only about $300 new and many others on e-bay were about $80.00.
Guess some people just put em up at rediculous price in the hope that someone stupid comes along.
EDIT: Maybe Galaxy Beam GT i8530.

EDIT: In same link above, Sinclair ZX80 and at same price £599.00.

EDIT To below: 32KB RAMPACK was the biggest possible ram upgrade, came initially with 1KB ram (my 1st rampack was 16KB).

[FranceBB]

my ol' ZX81 with 32KB ram pack

Considering that my first computer shipped with 64MB of RAM and Windows 98, I feel like I'm much younger than you, Jean-Philippe, Tormento and Ferenc are. :P

[pinterf]

Considering that my first computer shipped with 64MB of RAM and Windows 98, I feel like I'm much younger than you, Jean-Philippe, Tormento and Ferenc are. :P
Yep, you are lucky, unfortunately we'll not live out when gravitational-wave detectors would be easily embedded in an iPhone XLIII. :)

[jpsdr]

More serious question : Is there some kind of redistribuable to install to run the clang/llvm version ?
I've tried the releases.llvm link to see, but (at least right now) didn't work (time out).

[pinterf]

I built with /MT so it does not need any redistributables.

[pinterf]

TemporalRepair in TemporalDeGrain_SE

I'm near the end of porting (including to make the source human readable, re-creating plain C language versions, these works take a huge amount of time as well). Right now I put it in the RemoveDirt DLL, I woudn't create a new DLL just for the sake of this single function. Opinion?

[real.finder]

I'm near the end of porting (including to make the source human readable, re-creating plain C language versions, these works take a huge amount of time as well). Right now I put it in the RemoveDirt DLL, I woudn't create a new DLL just for the sake of this single function. Opinion?

isn't Rgtools better for it? they was part of RemoveGrain in first place