RemoveDirt - pfmod

[pinterf]

isn't Rgtools better for it? they was part of RemoveGrain in first place
That was my second thought.

[Reel.Deel]

isn't Rgtools better for it? they was part of RemoveGrain in first place

I second that.

Edit:

pinterf, I started working on the wiki documentation for RemoveDirt (http://avisynth.nl/index.php/RemoveDirt) (still not done), do you know what the cache and gcache parameters do in SCSelect? They are undocumented in Kassandro's wall of text, umm I mean documentation :).

[StainlessS]

RD heres the source,


public:
SCSelect(PClip clip, PClip _scene_begin, PClip _scene_end, PClip _global_motion, double dfactor, bool _debug, bool planar, int cache, int gcache) // the constructor
: GenericVideoFilter(clip), AccessFrame(vi.width, vi.IsYUY2()), scene_begin(_scene_begin), scene_end(_scene_end), global_motion(_global_motion), dirmult(dfactor), debug(_debug), lnr(-2)
{
if( vi.IsYV12() + planar == 0 ) AVSenvironment->ThrowError("SCSelect: only YV12 and planar YUY2 clips are supported");
CompareVideoInfo(vi, scene_begin->GetVideoInfo(), "SCSelect");
CompareVideoInfo(vi, scene_end->GetVideoInfo(), "SCSelect");
CompareVideoInfo(vi, global_motion->GetVideoInfo(), "SCSelect");
hblocks = vi.width / (2*SSE_INCREMENT);
incpitch = hblocks * (-2*SSE_INCREMENT);
scene_begin->SetCacheHints(CACHE_RANGE, 0);
scene_end->SetCacheHints(CACHE_RANGE, 0);
if(gcache >= 0) global_motion->SetCacheHints(CACHE_RANGE, 0); // used nowhere else
if(cache >= 0) child->SetCacheHints(CACHE_RANGE, cache); // child = source to ScSelect (input, or DClip in below script function)
}
};

AVSValue __cdecl CreateSCSelect(AVSValue args, void* user_data, IScriptEnvironment* env) // get user args and calls constructor returning filtered clip
{
enum ARGS { CLIP, SBEGIN, SEND, GMOTION, DFACTOR, DEBUG, PLANAR, CACHE, GCACHE };
return new SCSelect(args[CLIP].AsClip(), args[SBEGIN].AsClip(), args[SEND].AsClip(), args[GMOTION].AsClip(), args[DFACTOR].AsFloat(4.0)
, args[DEBUG].AsBool(false), args[PLANAR].AsBool(false), args[CACHE].AsInt(2), args[GCACHE].AsInt(0)); // cache default 2, gcache default 0
};


No idea how that cash stuff works, I prefer plastic :) [ EDIT: Actually, many UK banknotes are now plastic, but its just not the same. ]

EDIT: So far as I remember, the global_motion clip is the clip delivered when decided that it is NOT a scene change, and may or may not be subtitled by the client/user.

EDIT:
This is almost the same as a script func [ dont think I ever got different results comparing to SCSelect ]
EDIT: SCSelect(input) and SCSelect_Like(dclip) are synonymous except that dclip can be dissimilar colorspace and dimensions to the other script clips.

# RemoveDirt's SCSelect(clip input, clip scene_begin, clip scene_end, clip global_motion, float dfactor, bool debug, bool planar)

Function SCSelect_Like(clip dclip,clip Start,clip End,clip Motion, float "dfactor",bool "debug") {
# Start, End, Motion MUST all be same, dclip can be other colorspace/size (unlike SCSelect).
dfactor=Float(Default(dfactor,4.0))
debug=Default(debug,false)
Global SCM_A=0.0 Global SCM_B=0.0 Global SCM_SC=0 Global SCM_Prev=-1
Motion.ScriptClip("""
n = current_frame
NotNext = (n!=SCM_Prev+1)
Global SCM_A=(NotNext)? RT_LumaDifference(dclip,dclip,n=n-1,n2=n) : SCM_B
Global SCM_B= RT_LumaDifference(dclip,dclip,n=n,n2=n+1)
# 0 = Start of scene, 1 = End of scene, 2 = Global motion
Global SCM_SC=(n==FrameCount-1)?1:(SCM_A>dfactor*SCM_B || n==0)?0:(SCM_B>dfactor*SCM_A)?1:2
(SCM_SC==0) ? Start : (SCM_SC==1) ? End : Last # Choose Start, End or Motion(ie Last)
(debug)?RT_Subtitle("%d ] %6.2f %6.2f SC=%d",n,SCM_A,SCM_B,SCM_SC):NOP
Global SCM_Prev=n
Return Last
""",args="dfactor,Start,End,Dclip,debug") # Needs Grunt for args
return Last
}

[pinterf]

I second that.

Edit:

pinterf, I started working on the wiki documentation for RemoveDirt (http://avisynth.nl/index.php/RemoveDirt) (still not done), do you know what the cache and gcache parameters do in SCSelect? They are undocumented in Kassandro's wall of text, umm I mean documentation :).
They were converted internally to AVS 2.5 cache hints, telling the frame caching system to prefer caching at least the given number of frames. In Avisynth+ these hints do nothing. For avs 2.6, dunno. It probably served fine tuning purposes. Now the parameters are kept for compatibility reasons in order not to break old scripts.

[Reel.Deel]

They were converted internally to AVS 2.5 cache hints, telling the frame caching system to prefer caching at least the given number of frames. In Avisynth+ these hints do nothing. For avs 2.6, dunno. It probably served fine tuning purposes. Now the parameters are kept for compatibility reasons in order not to break old scripts.

Thanks for the info, I will add it to the wiki doc. next time I have a bit of time.

[thescrapyard]

Arh, my ol' ZX81 with 32KB ram pack (complete with lump of blue-tac or optional RAMPACK wobble feature).
After buildin' the ZX81 (with soldering iron), was a hairy moment when nothing happened on switch on,
was heat sink not quite square with board and slightly overlapped the board edge causing short with
earth strip around edge of board, squared up with board, all okey-dokey.
Boxed as new ZX Spectrum now bout £1500, or so I heard.
Wish I had had the power house ZX80, what a machine !

I have a ZX80 I acquired, comes with some paperwork and a letter from Sinclair Research plus rare white ZX80 16K RAM pack. Never been tested it works and I've had over 10 years. Keep meaning to get a proper ZX80 power supply to add to it and at least try a universal power supply to see if it actually works. Not boxed

Wish I had a Jupiter ACE though, worth over £400 even without a box and nothing else. They are really rare and did know somebody who bought one when they first came out many years ago

[StainlessS]

I have a ZX80


EDIT: In same link above, Sinclair ZX80 and at same price £599.00.


You be almost a millionaire then.

Very apt name and avatar Mr Steptoe.

[F1nkster]

How much effort would it take to get a Linux build of RemoveDirt to go with AVS+ on Linux? I'd like to request it, but I respect it may be easier said than done.

[GMJCZP]

Hi, I am having trouble with my DeCelluloid (https://forum.doom9.org/showthread.php?p=1567932#post1567932) script , some greenish frames appear, here is a sample:

Sample2 (http://www.filedropper.com/sample2)

The culprit is RemoveDirt 0.9.3 and SCSelect, I had to reuse the old RemoveDirt SSE2 1.0

[StainlessS]

You using the Clang version again with no SSE 4.1 ?
If you dont have SSE4.1, SSE2 is probably best anyway, I remember there were often problems
with SSE3 and I never used it if SSE2 were available. [speed difference, negligible].

Post your script, and versions of stuff, AVS+, RemoveDirt RGtootls.

Also, whatever problem you have, post image using this.


###
# Function DeCelluloid (adapted by GMJCZP).
###

# Eliminate garbage of top and bottom image of old videos.
# Low Gfactor values => larger detection, more probability appear artifacts
# High Gfactor values => minor detection, more safe.
# Thanks a Didée for the idea.
# https://forum.doom9.org/showthread.php?p=1567932#post1567932
# Requirements: RGTools, RemoveDirt.
# Version 1.0, first attempt.

function DeCelluloid(clip source, float "Gfactor", bool "_grey" ) {
Gfactor=default(Gfactor, 5.0) # Garbage factor
_grey = default(_grey, false) # Source in Black and White
prev = BackwardClense(source, grey=_grey)
next = ForwardClense(source, grey=_grey)
h = height(source)
h_top = int(h*0.20)
h_top = (h_top % 2) == 0 ? h_top : h_top + 1
h_bottom = int(h*0.375)
h_bottom = (h_bottom % 2) == 0 ? h_bottom : h_bottom + 1
source.LetterBox(h_top,h_bottom).SCSelect(next,prev,source,dfactor=Gfactor)
}

FN = "Sample2.mp4".RT_GetFullPathName

LWLibavVideoSource(FN)
O=Last
#return info
DeCelluloid()

TOP=StackHorizontal(O,Last)
D1=ClipDelta(Last,O)
D2=ClipDelta(Last,O,true)
BOT=StackHorizontal(D1,D2)
StackVertical(Top,Bot)
return last

# Return Clip Difference of input clips (amp==true = Amplified, show==true = show background)
Function ClipDelta(clip clip1,clip clip2,bool "amp",bool "show") {
amp=Default(amp,false)
show=Default(show,false)
c2=clip1.levels(128-32,1.0,128+32,128-32,128+32).greyscale()
c1=clip1.subtract(clip2)
c1=(amp)?c1.levels(127,1.0,129,0,255):c1
return (show)?c1.Merge(c2):c1
}

Off to hopsital Appt, now.
EDIT: I dont see any green stuff.

[pinterf]

Please check RgTools (bug in Forward/BackwardClense 10+ bits for non SSE4.1 processors)
https://github.com/pinterf/RgTools/releases/tag/1.2

[kedautinh12]

Thanks

[GMJCZP]

Please check RgTools (bug in Forward/BackwardClense 10+ bits for non SSE4.1 processors)
https://github.com/pinterf/RgTools/releases/tag/1.2

Thank you very much, now everything works correctly.

However, I still have the doubt, why was RGTools the one that caused the problem if SCSelect when canceling it in my tests everything worked fine? That was why I turned to the old RemoveDirt which did work.

My script is:

LwLibavVideoSource ("Sample2.MP4")
DeCelluloid()

PD: I am preemptively now using RemoveDirt 0.9.3 x86 (not clang).

[GMJCZP]

You using the Clang version again with no SSE 4.1 ?
If you dont have SSE4.1, SSE2 is probably best anyway, I remember there were often problems
with SSE3 and I never used it if SSE2 were available. [speed difference, negligible].

Post your script, and versions of stuff, AVS+, RemoveDirt RGtootls.

Also, whatever problem you have, post image using this.


###
# Function DeCelluloid (adapted by GMJCZP).
###

# Eliminate garbage of top and bottom image of old videos.
# Low Gfactor values => larger detection, more probability appear artifacts
# High Gfactor values => minor detection, more safe.
# Thanks a Didée for the idea.
# https://forum.doom9.org/showthread.php?p=1567932#post1567932
# Requirements: RGTools, RemoveDirt.
# Version 1.0, first attempt.

function DeCelluloid(clip source, float "Gfactor", bool "_grey" ) {
Gfactor=default(Gfactor, 5.0) # Garbage factor
_grey = default(_grey, false) # Source in Black and White
prev = BackwardClense(source, grey=_grey)
next = ForwardClense(source, grey=_grey)
h = height(source)
h_top = int(h*0.20)
h_top = (h_top % 2) == 0 ? h_top : h_top + 1
h_bottom = int(h*0.375)
h_bottom = (h_bottom % 2) == 0 ? h_bottom : h_bottom + 1
source.LetterBox(h_top,h_bottom).SCSelect(next,prev,source,dfactor=Gfactor)
}

FN = "Sample2.mp4".RT_GetFullPathName

LWLibavVideoSource(FN)
O=Last
#return info
DeCelluloid()

TOP=StackHorizontal(O,Last)
D1=ClipDelta(Last,O)
D2=ClipDelta(Last,O,true)
BOT=StackHorizontal(D1,D2)
StackVertical(Top,Bot)
return last

# Return Clip Difference of input clips (amp==true = Amplified, show==true = show background)
Function ClipDelta(clip clip1,clip clip2,bool "amp",bool "show") {
amp=Default(amp,false)
show=Default(show,false)
c2=clip1.levels(128-32,1.0,128+32,128-32,128+32).greyscale()
c1=clip1.subtract(clip2)
c1=(amp)?c1.levels(127,1.0,129,0,255):c1
return (show)?c1.Merge(c2):c1
}

Off to hopsital Appt, now.
EDIT: I dont see any green stuff.

TinMan, the madness continues.

But little by little all the problems I had, and did not know because I was away from Doom9, have been solved, The Market is flourishing again thanks to your help.

Hopefully pinterf can clarify my question about RGTools and SCSelect

[pinterf]

RemoveDirt v1.1 (20250108)
https://github.com/pinterf/RemoveDirt/releases/tag/v1.1
Milestone: Vapoursynth (API v4) support

This is my first real Vapoursynth port, so it can probably be fine tuned and may contain bugs.
Nevertheless I tried to separate the internal algorithm to have as much common part as it can be, so parts I was using in Avisynth should work in Vapoursynth as well unless I messed up obvious vsapi calls.
I was not surprised that 98% of the time went into de-Avisynthing and restructuring the existing code.

As for the Avisynth part, there is no change in it except that YUY2 (ancient planar hack) support is removed.

The DLL contains Windows builds x86 and x86, Microsoft and LLVM compiled versions.

For usage check documentation in release pack.

Thanks to Selur for the never ending motivation reminders :). By seeing the number of working hours, no wonder why I postponed this conversion for years :)

EDIT: SCSelect fix in v1.1, thx to Selur for the quick test.

[Selur]

Thanks to Selur for the never ending motivation reminders :).
Thanks to you for keeping up with my nagging and looking into it. :)
:thanks:

Cu Selur

[LightArrowsEXE]

As an avid VS user, also thank you! I hope we can see more ports for both frameservers in the future!

[Jamaika]

It's hard to say if this works correctly. The show option specifies which motion blocks are being changed.
Avisynth function: Clense [function, false, 4]
Avisynth function: ForwardClense [function, false, -1]
Avisynth function: BackwardClense [function, false, -1]
Avisynth function: SCSelect [function, function, function, function, true]
Avisynth function: Repair [function, function, 16, 16]
Avisynth function: Repair [function, function, 16, 16]
Avisynth function: RestoreMotionBlocks [function, function, function, function, false, 70, 1, 2, false, 10, 12, false]
Avisynth function: RemoveGrain [function, 17, 17]

[pinterf]

Thanks for the report, but please provide a meaningful script instead of this debug output.
Also clarify the bug report a bit, what's wrong, the documentation or "show" does not work as intended?

[Jamaika]

I`m amateur. I'm looking at the wiki page. Test script. It seems that the RestoreMotionBlocks can't work without RgTools.
function RemoveDirt(clip input, bool "_grey", int "repmode")
{
_grey=default(_grey, false)
repmode=default(repmode, 16)
clmode=17
clensed=Clense(input, grey=_grey, cache=4)
sbegin = ForwardClense(input, grey=_grey, cache=-1)
send = BackwardClense(input, grey=_grey, cache=-1)
select = SCSelect(input, sbegin, send, clensed, debug=true)
alt=Repair(select, input, mode=repmode, modeU = _grey ? -1 : repmode )
restore=Repair(clensed, input, mode=repmode, modeU = _grey ? -1 : repmode)
corrected=RestoreMotionBlocks(clensed, restore, neighbour=input, alternative=alt, gmthreshold=70, dist=1, dmode=2, debug=false, noise=10, noisy=12, grey=_grey)
return RemoveGrain(corrected, mode=clmode, modeU = _grey ? -1 : clmode )
}

Testing:
AviSource("input_v210.avi", audio=false, pixel_type="V210")
RemoveDirt(false, 16)

If debug=true, then SCSelect sends output of the following type to the DebugView utility.
Hmm,... If I only have ffmpeg then this function is unnecessary. I don't have any information displayed.

The boolean variable debug and show are used for debugging. If show=true, then the blocks, which are marked as motion blocks in the first phase are colored red, those found in the second phase are colored green and finally the motion blocks marked by postprocessing are colored blue. In this way, one can easily check whether the above variables were selected appropriately. if debug=true, then RestoreMotionBlocks sends output of the following kind to the debugview utility:
I only have one color red and blue. Which parameter increases, decreases the motion block area.
When I use RemoveDirt(true, 16) the video won't play and everything is pink.

[pinterf]

When I use RemoveDirt(true, 16) the video won't play and everything is pink.
You intentionally passed grey=true to the script function which passes it over to its various filters. I suppose in this case filters simply do not process and ignore the chroma planes thus they contain garbage. In your case you see pink colors, I can see diagonal lines.

[Jamaika]

You intentionally passed grey=true to the script function which passes it over to its various filters. I suppose in this case filters simply do not process and ignore the chroma planes thus they contain garbage. In your case you see pink colors, I can see diagonal lines.
Maybe but video doesn't play.
For amateurs ffmpeg with new RestoreMotionBlocks.
https://www.sendspace.com/file/8yufh8

[FranceBB]

RemoveDirt v1.1 (20250108)
https://github.com/pinterf/RemoveDirt/releases/tag/v1.1



Thank you! :D
And as always I'm very happy to report that... it works on XP! :)

[pinterf]

Thank you! :D
And as always I'm very happy to report that... it works on XP! :)
You're welcome! Feel free to send me a text anytime if something doesn't work for you. :)