Anything better than DeScratch ?

[StainlessS]

Yo dudes,
Any thing better than DeScratch for crappy src like this,
[EDIT: I did not even bother to try descratch]

[clickme twice]
https://i.postimg.cc/HVJzRb3Q/MetBad.jpg (https://postimg.cc/HVJzRb3Q)

Spotless does some improvment [not much], but was wantin' to maybe try something prior to spotless filter, maybe spatial only.

BD Metropolis (M.O.C. 2010).

Suggestions ?

EDIT: All the scratching is black.

[Selur]

Suggestions ?
Share a short clip.

[StainlessS]

About 50 frames I think:- https://www.mediafire.com/file/pca9cmpmvz24mgw/Met2010.7z/file
34MB in 7z, 45 Extracted Avi.
(could not rapidly figure out how to cut it, so extract to AVI UT_VIdeo).

The later 16mm stuff discovered in argentina [150 min 2010 restore] is a bit worse for wear, earlier stuff quite good.
(cropped left and top {black} from old edit)

[johnmeyer]

Getting rid of the vertical scratches from worn film is tough to do digitally for two reasons. First, the scratches persist from frame to frame so you can't play the RemoveDirt trick of looking at motion-estimated versions of adjacent frames and subtracting differences. Second, while I said that the scratches persist, they also tend to wander a bit side-to-side. This makes it tough to do the other obvious thing which is to construct some sort of mask, like Delogo does, and fuzz out the scratch by using adjacent pixels.

The one good way to deal with film scratches of this type is during the transfer. It is called a "wet gate" transfer and it involves applying some liquid to the film which fills in the scratches and optically causes them to disappear. It is quite effective. However, I assume you don't have access to the actual film.

You should at least try Descratch. I've used it off and on over the years and, in some circumstances, it can make an improvement.

[Selur]

I agree sadly, there seems to be no way to get rid of the scratches with Avisynth&Vapoursynth. (Only hope might be some ML based stuff in Vapoursynth, but I know of none that would really help with the scratches.)

Cu Selur

[StainlessS]

Yip, I guess I was expecting similar answers, thanx anyways.

They probably already tried some scratch reduction, what we got is what is left afterwards.
Not sure why, but the sort of 'fat worm' scratch [midway between her forehead and RHS edge],
gives me the impression that de-scratch processing already been applied.
Unadulterated source musta been real bad.
[EDIT: 'Fat Worm' during clip in motion, not 1st post image (too fat for de-scratching to have much effect)]

So the Masters Of Cinema, 2010 restore [1927 movie] has something like 15% extra crap just like that scratching.
(previously lost due to massive cuts in original USA release, the USA cut is source of a lot of the better condition film)

EDIT: The argentina find of missing sections was a 16mm backup in case of nitrate film destruction due to fire.

EDIT: Frame from better part of restore source BD.
https://i.postimg.cc/GHvmWbWM/Met-00.jpg (https://postimg.cc/GHvmWbWM)
EDIT: Some day all women will be made this way, cold & hard [EDIT: well colder & harder :) ].

[Dogway]

Very interesting clip, to my samples box!

I gave it a try mainly creating the stripes mask.

http://i.imgur.com/CzdN3lVm.png (https://i.imgur.com/CzdN3lV.png) http://i.imgur.com/Hdu4UxTm.png (https://i.imgur.com/Hdu4UxT.png)


src=last

ex_edge("kroon")
ex_boxblur(0,5)
ex_binarize(80,true,hi=-1)
ex_lut("x[-1,0] x[1,0] - abs 5 *").ex_inpand().ex_expand()
ex_boxblur(0,10)
ex_lut("x 40 - 0 max 20 *")
ex_deflate().ex_deflate()


InpaintDelogo(src, mask=last,
\ Automask=0, Loc="0,0,0,0",
\ Mode="inpaint",interp=0,
\ GrainPP=0, turbo=0)
ex_lutxy(src,"y 5 + x > y x ?")


Also tried with a top hat
ex_edge("kroon")
ex_boxblur(0,5)
ex_binarize(44)
ex_makediff(last, ex_luts(mode="min", pixels=ex_shape(2,4)).ex_luts(mode="max", pixels=ex_shape(2,4)), dif=false)
ex_expand().ex_inpand()


Then a brute force denoising, it's very far from good, but well just trying:

smdegrain(tr=2,mode="temporalsoften",blksize=32,prefilter=ex_boxblur(5,0),thSAD=900,contrasharp=false,refinemotion=true)
smdegrain(tr=1,mode="MDegrain",blksize=32,thSAD=300,LFR=false,contrasharp=false,refinemotion=false)

[StainlessS]

Cheers Big Dog, I'll givem a whirl.

When finished, I'll post a collection of FrameSel frames files and ClipClop frames/Range file,
so can apply some correction targetted at various range types for the BD.
[eg Good quality, Crap scratched, Intertitles, Credits etc].

EDIT: The good quality frames are a little noisy for my taste, me dont like too much nasty film grain.

EDIT: I'm wondering if Local Contrast Enhancement type stuff might bring back a little more sharpness
without accentuating calmed scratches (too much) [on your corrected sample].

[VoodooFX]

I gave it a try mainly creating the stripes mask.

What that "ex_lutxy(src,"y 5 + x > y x ?")" at the end does?


Btw, InpaintDelogo params there are defaults or doesn't apply to inpainting, so it can be just as:
InpaintDelogo(src, mask=last, Loc="0,0,0,0")

Probably these params you would want to adjust:
InpaintDelogo(src, mask=last, Loc="0,0,0,0", oPP=5, oPPm=3)
oPP is blur strength after inpainting.
oPPm is mask feathering for PP.

EDIT:

ex_deflate().ex_deflate()
Feathered masks doesn't work for inpainting.

[Dogway]

Thanks VoodooFX, it got me a bit to find some values, specially 'Loc'.
So inpainting wants binary masks?

My lut line is just a max(x,y) since stripes are mostly darker than the content, but I added a small bias to 'y' to recover a bit more.
Maybe I would have better use original inpaint plugin call but was too lazy to research that up. This was me playing a bit with the clip, nothing to call home about.

StainlessS, the first pass before SMDegrain removes the most offending stripes, but for the fainter ones it's harder because it shares many common properties of the clip content. What I would do probably blur horizontally a certain frequency, I was a bit slow at that moment but now it sparked, I can use ex_MFR() to put blurred frequencies in place. ex_MFR() is a new filter so it's not perfect, I need to study the transients contribution, maybe make a LR filter (https://en.wikipedia.org/wiki/Linkwitz%E2%80%93Riley_filter)

[StainlessS]

Thanks Dog, its all Gobble-De-Gook to me:)
(Linkwitz–Riley filter etc)

[Dogway]

It's just another blurring function, but I remembered it from when I was doing some audio.

I tried another thing to clean up low frequencies:

src=last
RatioResize(0.25,"%",kernel="bicubic")
DeStripeV(3,1,255)
ex_boxblur(1,0,mode="weighted")
bicubicresize(1338,974)
ex_LFR(src,last, LFR=150)

And here's the result using also my previous post code after this snippet.

http://i.imgur.com/6A4z28Rm.png (https://i.imgur.com/6A4z28R.png)

From here you can use an unsharpmask to sharpen low frequencies. Although maybe some higher frequencies should be further denoised yet.

EDIT: This is with a final horizontal bilateral
http://i.imgur.com/pXPoPZim.png (https://i.imgur.com/pXPoPZi.png)

The whole thing as I'm also a bit lost already:
src=last
RatioResize(0.25,"%",kernel="bicubic")
DeStripeV(3,1,255)
ex_boxblur(1,0,mode="weighted")
bicubicresize(1338,974)
ex_LFR(src,last, LFR=150)
src=last

ex_edge("kroon")
ex_boxblur(0,5)
ex_binarize(80,true,hi=-1)
ex_lut("x[-1,0] x[1,0] - abs 5 *").ex_inpand().ex_expand()
ex_boxblur(0,10)
ex_lut("x 40 - 0 max 20 *") # Assumes 8-bit
# ex_deflate().ex_deflate()

InpaintDelogo(src, mask=last, Loc="0,0,0,0")
ex_lutxy(src,"y 5 + x > y x ?") # Assumes 8-bit


smdegrain(tr=2,mode="temporalsoften",blksize=32,thSAD=900,LFR=false,DCTFlicker=true,contrasharp=false,refinemotion=true)
smdegrain(tr=1,mode="MDegrain",blksize=32,thSAD=300,LFR=false,contrasharp=false,refinemotion=false)

ex_bilateral(5,0, ithres=10.0, sthres=10.0, dejaggie=false)

# ex_unsharp(0.2,0.4,width()/5, 1)

Sharpened:
http://i.imgur.com/lhUkAxUm.png (https://i.imgur.com/lhUkAxU.png)

[StainlessS]

Not too bad at all, thanx doggy.

[VoodooFX]

Looks good, but are there temporal fluctuations in the result, or video is quite steady?

So inpainting wants binary masks?
Yeap.

[johnmeyer]

Well, if you are now considering simply blurring, please take a quick look at this thread I started many years ago:

How to remove Polavision (instant film) vertical lines (forum.doom9.org/showthread.php?t=163574)

In addition to the suggestions given to me in that thread, I also tried a convolution kernel. It worked and, if I was going to try any generalized blur function on this video, I'd try convolution first.

[Zarxrax]

This looks like an ideal candidate for something like VSGAN, but you would have to train your own custom model which is a fairly difficult and time consuming process.

[Dogway]

Looks good, but are there temporal fluctuations in the result, or video is quite steady?

Should be pretty stable temporally since I use strong temporal settings in SMDegrain but it's not going to be perfect, probably (low frequency) scratches will slowly fade in and out. Given the disaster of the source might be a good compromise. I would add some grain on top to mask these things and add some pshychovisual sharpness.

[StainlessS]

Thanks Zarxrax but I aint gonna spend that much time on it.

Doggy, is smdegrain the only temporal filtering there, [I'd rather only use Spotless, mainly cos I know how to use it :) ]

[Dogway]

Yes, only temporal is SMDegrain. InpaintDelogo only could remove the highest contrast scratches but left the others. I updated DeStripeV() since it was a bit outdated and still have to upload ex_unsharp() with separate strength for vertical and horizontal. Maybe the script could be simplified but I was only doodling with the filters.

[poisondeathray]

I agree sadly, there seems to be no way to get rid of the scratches with Avisynth&Vapoursynth. (Only hope might be some ML based stuff in Vapoursynth, but I know of none that would really help with the scratches.)




I mentioned this in the hinet thread - you can "abuse" "hinet derain" as scratch remover. Hinet was trained with "white" streaks representing "rain". So for "dark" scratches you have to adjust the levels to resemble "white" scratches - e.g. apply levels or similar , then invert the levels (then invert back) .

The majority of the scratches on this sample are slightly too wide to be "raindrops", so a horizontal downscale can help hinet detect more "rain" drops, but of course the tradeoff is softness. Tiles can help (dividing horizontal and vertical segments) - because long scratches are less like the shorter discrete raindrops that hinet was trained on (it was trained on both; some longer streaks as well). You can run different settings and tiles (gives slightly different results, different iterations), then run a median or mean. You can see tile breaks even with padding on single runs, but they tend average out with median and temporal filtering (I bet zopti can help figure out a few optimal runs of downscaling and tiles)

Hinet is spatial only , so you have to apply other filters such as temporal filtering to smooth over.

median of 5 hinet runs (different tiles, basically random), some ttempsmooth, smdegrain and sharpening
https://i.postimg.cc/KRc8VrXQ/hinet-median-temporalfilter.png (https://postimg.cc/KRc8VrXQ)

example of a "base" iteration at 470 width with varying tiles


clip = core.avisource.AVISource(r'Met2010.avi', pixel_type="YV12")
clip = core.std.Crop(clip, left=102, top=106, bottom=2)
#1338x972

clip = core.resize.Bicubic(clip, width=470, height=720, format=vs.RGBS, matrix_in_s="709")
clip = core.std.Levels(clip, min_in=0.11, max_in=0.82, gamma=0.8, min_out=0, max_out=1, planes=[0,1,2])
clip = core.std.Invert(clip)
hin = HINet(clip, tile_x=300, tile_y=330, tile_pad=16, mode=3)
hin = core.std.Invert(hin)
hin = core.std.Levels(hin, min_in=-0.11, max_in=1, gamma=1/0.8, min_out=0, max_out=0.82, planes=[0,1,2])
hin = core.resize.Spline36(hin, format=vs.YUV420P8, matrix_s="709", width=1338, height=972, dither_type="error_diffusion")
hin.set_output()

[Selur]

Nice! :)
The downsizing of the source:
clip = core.resize.Bicubic(clip, width=470, height=720, format=vs.RGBS, matrix_in_s="709")
was something I missed (I did try the HINet+derain part, but without the downscaling beforehand it does not work).
How did you come up with width=470 and height=720 ?

[Frank62]

Simply brilliant!
What do I need to run vapoursynth and this filter?

[SeeMoreDigital]

Jeez... That's astonishing!

[StainlessS]

Well not too bad PDR, but how does it fare with a real pig,
https://www.mediafire.com/file/isohe23g9hd4dhw/MetBits.avi/file
~98MB

[Maybe I try Vapoursynth again :eek: ]

[poisondeathray]

Nice! :)
The downsizing of the source:
clip = core.resize.Bicubic(clip, width=470, height=720, format=vs.RGBS, matrix_in_s="709")
was something I missed (I did try the HINet+derain part, but without the downscaling beforehand it does not work).
How did you come up with width=470 and height=720 ?


It's just one of the random trials / previews (at least hinet isn't as slow as some other ML's algos...). eg. I tried 640x720 , it worked ok, but missed some wider ones. But it also depends on tile width - given a certain frame width, a given tile boundary might miss some defects, that's where the vs-average (median) later helps. The height is also important, too long, and it will miss detection,so the "tiles" and/or vertical resizing are important. You can experiment... a lot.... zopti would probably help - it would probably find a combinations at a wider width for better detail retention


I' m finding the key to any of these ML algos, is "what pre filters/transforms can I apply - to make it similar to the training set conditions?" .

This is from hinet paper, you can see the pattern of "rain" is somewhat similar to some types of "scratches" (except "white"), so it' s just "abusing/misuing" it for another purpose .

https://i.postimg.cc/MM4qTTzL/hinet-derain.png (https://postimg.cc/MM4qTTzL)




What do I need to run vapoursynth and this filter?

Most people would start with the portable fatback from chaosking , and vsedit mod to preview
https://forum.doom9.org/showthread.php?t=175529

And Hinet vpy port from HolyWu (too bad they deleted their D9 account :( :( ) Please come back
https://github.com/HolyWu/vs-hinet

[Frank62]

Thanks a lot, this little hint saves me lots of time for sure! :thanks:

[poisondeathray]

Well not too bad PDR, but how does it fare with a real pig,
https://www.mediafire.com/file/isohe23g9hd4dhw/MetBits.avi/file
~98MB

[Maybe I try Vapoursynth again :eek: ]


I think it should work similar.

One finicky thing I can see - slightly different exposures on different scenes . The levels adjustment will need to be slightly different for different scenes for optimal results. Maybe there is a way it could be auto calculated , not sure. The "goal" is to make them "look" like those reference images from the paper

And to get "thick" defects, you have to scale width smaller (trade off is softer image), so you have to dial it to your taste. But you can combine multiple runs, just make sure you have more runs that do no have the defect if using median (odd number)


eg.
orig
https://i.postimg.cc/LnBCH8B6/080crop-orig.png (https://postimg.cc/LnBCH8B6)

1 pass hinet only , (no multiple runs, no averaging, no denoising, no sharpening or post filters)
https://i.postimg.cc/XrssC1zR/080hinet-480x720-0x720tile-unfiltered.png (https://postimg.cc/XrssC1zR)

images if postimg down
https://www.mediafire.com/file/b8485k3c3k34u5r/hinet_scratches.zip/file

I "randomly" picked 450x720 width , with 240x0 tiles. The levels are different than the 1st example.

import vapoursynth as vs
core = vs.core
from vshinet import HINet

clip = core.avisource.AVISource(r'MetBits.avi', pixel_type="YV12")
clip = core.std.Crop(clip, left=80, top=94, bottom=0)
#undercrop @ 1360x986 , different fluctuating borders

clip = core.resize.Bicubic(clip, width=450, height=720, format=vs.RGBS, matrix_in_s="709")
clip = core.std.Levels(clip, min_in=0.08, max_in=0.85, gamma=0.8, min_out=0, max_out=1, planes=[0,1,2])
clip = core.std.Invert(clip)
hin = HINet(clip, tile_x=240, tile_y=0, tile_pad=16, mode=3)
hin = core.std.Invert(hin)
hin = core.std.Levels(hin, min_in=-0.08, max_in=1, gamma=1/0.8, min_out=0, max_out=0.85, planes=[0,1,2])
hin = core.resize.Spline36(hin, format=vs.RGB24, width=1360, height=986, dither_type="error_diffusion")
hin.set_output()



When you median multiple runs with different settings, the small defects around the tile edges tend to get averaged out; and dirt/noise can still be filtered with "traditional" filters

[StainlessS]

Yip quite astounding PDR (such a showoff).
I got me Python 3.9 and VS installed, (pray for me :) ).

EDIT: OK, going to backtrack and install ChaosKing FatPack.

[Selur]

@Frank62: send you a link to my current Hybrid dev version and my torchAddon to get you started easier. :)

[johnmeyer]

poison, that result deserves an award. Monty, what do we have for the contestant?

[Selur]

@StainlessS: Also send you the links too. The torchpy-Addon includes a portable Vapoursynth version + VSGAN (+ model collection), vsDPIR, vsRIFE (torch), vsBasicVSR++, vsRealESRGAN, vsSwinIR, vsHINet, which should be helpful to get started. :)

[ChaosKing]

Maybe it's time for a cuda fatpack :devil: cudpulent?

[StainlessS]

Yes thanks Selur, me got your PM, but not gonna Download just yet,
I just keep getting swamped with more and more "stuff",
I aint ever gonna get away and have a play with Android app development,
years now I've been trying to escape. :(

[StainlessS]

ChaosK, I'm getting this from "_vs_plugin_check.bat",

#######################################
VapourSynth Video Processing Library
Copyright (c) 2012-2021 Fredrik Mellbin
Core R57
API R4.0
API R3.6
Options: -

Architecture 64bit - Windows-10-10.0.18363-SP0
Python build: ('tags/v3.9.7:1016ef3', 'Aug 30 2021 20:19:38')

#######################################
checking dlls in C:\VideoTools\VapourSynth64Portable\VapourSynth64\vapoursynth64\plugins
#######################################


Errors: Not a VS-Plugin
-------
No entry point found in C:\VideoTools\VapourSynth64Portable\VapourSynth64\vapoursynth64\plugins\concrt140.dll
No entry point found in C:\VideoTools\VapourSynth64Portable\VapourSynth64\vapoursynth64\plugins\opencv_world450.dll

Notices:
-------
libfftw3-3.dll is a dependency by fft3dfilter or mvtools-sf
libfftw3f-3.dll is a dependency by fft3dfilter or mvtools-sf
#######################################
Found 125 dlls. Errors: 2 Notices: 2

Normal ?

[ChaosKing]

looks ok. concrt140 and opencv_world450 are runtimes/helper dlls

[StainlessS]

OK, tanks :)

[zorr]

You can experiment... a lot.... zopti would probably help - it would probably find a combinations at a wider width for better detail retention

The thing with Zopti though is that you need an automatic way to evaluate the quality. Usually the result is compared against a good reference frame using similarity metrics. Here we don't have one, or do we? But in this case it might be possible to cook up a little algorithm which can identify the streaks and then the goal would be to minimize the area they occupy.

@Selur can I have a link too, I'd love to do some experiments with the latest neural algorithms.

[StainlessS]

Selur can I have a link too
12GB,
I'm on mobile broadband - pre pay, robbin gits charge a whole lot more for me not to
give them my bank acc details, I dont wants no contract nor other personal leaks.
way too much for me, 12GB. (maybe later).

EDIT: 12GB would cost me bout £15.00 on my tariff, shit but there you go.
[EDIT: Actually I cheat a bit, If I really pay'ed mobile broadband prices you could more than double my estimates
[EDIT: My previous Mobile Broadband SIM, £10 for 1GB, £15.00 gets you 3GB, £20 gets you 5GB]
PAYG mobile broadband charges are quite ridiculous - but giving access to my bank acc is out of the question.].

[poisondeathray]

You can reduce the bandwidth if you choose to install specific items

You probably don't need all that other stuff like ESRGAN models - I think selur bundles a bunch of other projects and models.


vapoursynth itself (portable or installed) is only ~10MB . A few source filters like AVISource, LSMash, FFMS2, etc.. are a few kb, same with some useful helper scripts like havsfunc (they weren't used in this example)

vsedit mod is <10MB

Hinet is only a few kb, but each model is 338MB (If you're only interested in the "derain" model, you only need that specific one)

! But pytorch /cuda is a quite heavy install - 4-5 GB

[Selur]

@Selur can I have a link too, I'd love to do some experiments with the latest neural algorithms.
Send you a link.

Cu Selur

[StainlessS]

Thanks PDR, I'll D/L in pub (for free) one day.
[I sometimes forget to turn off my 4G Mobile Broadband in pub, and it ends up costing me anyways]
EDIT: Think 5G device is on the cards soon.