big green line removal with overlay mask possible?

[Metal-HTPC]

hey there,

I'm dealing with a HD source which has 2 very annoying big green lines in it for about 10 minutes.

Is there any way to get rid of that? Various versions of RemoveDirtMC were of course not able to filter that away and I doubt that anything is able to anyways so an overlay mask would be the only way. So I digitised and upscaled a VHS and made the frames match to the bluray.

The lines move so they need to be used as the overlay mask but the problem is they're not always a lot brighter than the rest of the picture so the rest of the picture becomes the mask too. There's also the problem of the pictures not lining up exactly and them having different colors. If you could find a way to just use the green lines as the overlay mask it could maybe work.


https://drive.google.com/file/d/1U4_BT-N9d1mnLcWLJUK6_5AHoEqP62ME/view?usp=share_link


https://drive.google.com/file/d/1oIBNpxKB8jjUqwg9hJ402qL_e6RrirZj/view?usp=share_link

Thanks in advance for looking into it
regards

[Reel.Deel]

You should probably upload smaller samples (just a few scenes). Each of those files is 4.7gb.

[johnmeyer]

I didn't see any green line in the first sample.

The second sample has a green-tinted film scratch which wanders back and forth. I do lots of film restoration and scratches are notoriously difficult to remove digitally without a tremendous amount of frame-by-frame work or without some sort of object tracking that is beyond anything yet developed for AVISynth.

You can always try the AVISynth Descratch filter, but I don't hold out much hope that it will work.

[edit]I re-read the OP and realize you are talking about doing an overlay. Don't know how I missed that. So the real problem is auto-generating a mask based on the color and intensity of the scratches, and substituting your lower-res version into that portion of each frame.

Others are better at writing masks than I am and I'm sure you'll get some help (you can also just search this forum and probably get the script you need). The script will also definitely need some way to feather the edges or else the substitution will call attention to itself.

However, the much bigger problem is going to be alignment. I have done almost exactly what you are trying to do where I wanted to add or remove (can't remember which now) subtitles by substituting a lower-res version. The problem was that both the cropping and alignment between the two different sources changed, and also the editing on the two versions was subtly different so all of a sudden the two captures were several frames offset from each other. They also sometimes changed x,y, or rotational alignment with respect to each other. It required a LOT of manual work to finish the project.

[StainlessS]

I'm not sure if this can help
https://forum.doom9.org/showthread.php?p=1850330#post1850330


I think its the script I was thinking of, but the images aren't available now [PostImages.org says server not available, however other images from there are available,
so maybe just a single server (out of a cluster of servers) have problem.]
I'm not gonna down 4.7GB, so cant test.

You may have to turn clip sideways.

EDIT: Maybe not that script. Seems that one is for temporal damage/repair. (temporal frame interpolation)

[VoodooFX]

Can you share less than 200MB samples?

[Metal-HTPC]

thanks for the posts so far guys. I tried to setup the latest dirtbox version of stainless but have no idea how to get it to work in avs+. It seems like RT-stats are the problem. Never used RT stats before would have to wrap my head around it

if you want to take a look at the actual videos here are some samples shrinked and kept short
https://workupload.com/archive/6AqM9AmD

[StainlessS]

EDIT: Maybe not that script. Seems that one is for temporal damage/repair. (temporal frame interpolation)

Sorry, script does not do what I thought it did.

[Metal-HTPC]

yea I got the old john meyer version of it to run but it didnt work out :-/

[johnmeyer]

My "replace bad frames" script is totally the wrong thing to be using here. That script was designed to detect, and optionally replace, a single bad frame, like a photographer's flash, or a dropout, or blank frame. Your green line persists and that is exactly what my script doesn't detect.

[VoodooFX]

Dunno what you want to overlay there because your both videos don't fit by geometry nor by colorimetry.

Anyway, here is mask and inpainted result from InpaintDelogo [frame 108]:

https://thumbs2.imgbox.com/8f/4e/BgclF7b4_t.jpg (https://images2.imgbox.com/8f/4e/BgclF7b4_o.jpg) https://thumbs2.imgbox.com/df/2c/AKgU60Xw_t.png (https://images2.imgbox.com/df/2c/AKgU60Xw_o.png)

https://thumbs2.imgbox.com/a7/7d/1FN1c4hK_t.jpg (https://images2.imgbox.com/a7/7d/1FN1c4hK_o.jpg)


LWLibAvVideoSource("D:\Enforcer-Dust-Linescene.avs.mov.mov.mkv")

# Show=0 will do inpainting
InpaintDelogo(Loc="0,0,-0,-0", Show=4, DynMask=2,
\ DynTune=140, DynTuneInf=4,
\ DynTEdge=1, DynTEdgeThY=25, DynTEdgeInf=4,
\ DynColor=$4EBC80, DynColorTol=50, DynColorInf=4,
\ DynInflate=3)


It's not the proper tool for your problem, nor I tried to get a good mask, just an example...

[kedautinh12]

But his face distorted :D

[VoodooFX]

But his face distorted :D
So? It's to check how well green stripes are covered.

[Selur]

The problem with inpainting is that it is only working on a single frame and thus can't really find fitting content to place inside the removed content and 'guessing' a face isn't easy. ;)
(sadly afaik none of the video inpainting techniques have are available in Vapoursynth or Avisynth)

Vapoursynth script(https://pastebin.com/MzMvMAfC), using vsgan with an inpainting model (1x_NMKD-YandereInpaint_375000_G):
https://i.ibb.co/Jrss0Pp/grafik.png (https://ibb.co/kqSSkFD)

Using cv_inpaint.InpaintShiftmap:
https://i.ibb.co/wRbVpNg/grafik.png (https://ibb.co/7XTPrJC)
Using cv_inpaint.InpaintFSR:
https://i.ibb.co/c2rBmSr/grafik.png (https://ibb.co/HGHSJfH)
Using cv_inpaint.InpaintTelea(clip, mask, 1):
https://i.ibb.co/N3fwfb5/grafik.png (https://ibb.co/qnhbhVK)
Using cv_inpaint.InpaintNS(clip, mask, 10):
https://i.ibb.co/dQ2rX2Q/grafik.png (https://ibb.co/2jtyRtj)
(script for the cv inpaint tests: https://pastebin.com/tLLjLg2c)
=> single image based inpainting fails ;)


Cu Selur

[coolgit]

I had that last year with 3 lines. Fixed it through photoshop.

[Metal-HTPC]

thanks so far, yea I fixed single frames with cloning in ulead photo impact too but that was for a maximum of 120 frames. In this case I'm having 10 minutes which would be too much find to go through.
So I guess the line will remain then

[poisondeathray]

The problem with inpainting is that it is only working on a single frame and thus can't really find fitting content to place inside the removed content and 'guessing' a face isn't easy. ;)
(sadly afaik none of the video inpainting techniques have are available in Vapoursynth or Avisynth)

Yes I'm hoping someone ports ProPainter to VPY. It's really a big improvement over older temporal/video inpainters

Prefiltered with Spotless. Mask from InpaintDelogo above, and roughly painted in holes for coverage (I probably missed a few spots), fed into ProPainter

https://i.postimg.cc/ZK2wz0SH/orig-spotless-mask-propainter.png (https://postimg.cc/XpKfwjXB)

cropped comparison video @ half speed
https://www.mediafire.com/file/hi0pv4bpytoju62/orig,spotless+mask,propainter_12fps.mp4/file

Overall pretty good but some minor issues
1) window shutters a bit wobbly. I think algorithm is confused by the slightly shaky picture. I'm guessing prestabilizing would help. Not show heren, but FGT has stable shutters (but worse problems with other parts)

2) Left eye stays more open on frame 20/21. You would expect both eyes to be similar level of eyelid closing. But it's a fast head turn with the scratch obscuring those frames.


There are dozens of machine learning temporal inpainting projects on github . The newest one, Propainter, is probably the best overall IMO, although FGT++ is supposed to be coming out soon with significant improvements. To list a few:


https://github.com/sczhou/ProPainter
https://github.com/hitachinsk/FGT
https://github.com/nbei/Deep-Flow-Guided-Video-Inpainting
https://github.com/MCG-NKU/E2FGVI
https://github.com/seoungwugoh/opn-demo


Various autosegmenting/automasking/autotracking machine learning algorithms exist - but the ones I've tried tend to work better on objects, not wandering, shape changing scratches like this. Or I haven't found any yet for this type of scratch pattern

Some of the temporal inpainting algorithms have options like adjustable stride length, neighbor references length, iterations, lower precision (float16), internal mask and flow dilation mask parameters... you're often limited by GPU memory depending on the scene and settings . There are tradeoffs eg. If you reduce the references length to reduce memory consumption, you might miss out on usable data from nearby frames . Or if you use too long references , you might get worse results with contamination with slightly less similar data

They are far from perfect, but the newest ones have significantly improved, especially Propainter, FGT. Useful tools in the pipeline for object/logo removal, clean plates, extrapolation - and you can fine tune the results in other programs

[kedautinh12]

Wow, result very well, hope someone port to avs+ too :D

[johnmeyer]

Wow, I didn't think wandering vertical scratches could be removed algorithmically. Your demonstration is absolutely amazing.

I guess I need to get out of the dark ages and figure out how to use these newer tools. If I am understanding you, except for some prefiltering using AVISynth with a few plugins, this was all done in ProPainter, yes?

[Selur]

Nice! What did you use to create the mask?

Cu Selur

[poisondeathray]

If I am understanding you, except for some prefiltering using AVISynth with a few plugins, this was all done in ProPainter, yes?

As mentioned above, I had to manually touch up spots on the mask generated by InpaintDelogo, because the mask was incomplete (holes)

Another big problem for those types scratches is still identification and masking / propogation. I'm still looking for a suitable algorithm for wandering scratches that change shape like that. I'm guessing you need to train something specific for that case.

For the fill part - it's actually better if the scratch moves / or camera moves / or BG elements move - because you have clean data in other frames. If nothing moved, you'd be basically spatial inpainting

The way I see this: it's a timesaver over the old way of manually cloning and tracking textures from other frames to fix defects


What did you use to create the mask?


You probably missed:
Mask from InpaintDelogo above, and roughly painted in holes for coverage (I probably missed a few spots), fed into ProPainter

[Selur]

yeah, I missed that :)

[Frank62]

WOW! Magic becomes reality. One of the good sides of A. I. Will try this! :thanks:

[Metal-HTPC]

that looks super amazing. Even if it would have to be adjusted for each scene it would be worth it. Will definitely give it a go as soon as it is protable in AVS+

[kedautinh12]

If it's port to avs+, it's will help delogo very much :D

[Dogway]

As mentioned above, I had to manually touch up spots on the mask generated by InpaintDelogo, because the mask was incomplete (holes)

Another big problem for those types scratches is still identification and masking / propogation. I'm still looking for a suitable algorithm for wandering scratches that change shape like that. I'm guessing you need to train something specific for that case.

For the fill part - it's actually better if the scratch moves / or camera moves / or BG elements move - because you have clean data in other frames. If nothing moved, you'd be basically spatial inpainting

Nice, looking forward for a solid delogo algorithm. Maybe mask propagation can be dealt with mt_hysteresis or an algorithm of the like like greedy expansion.

As for static masks, in my view spatial inpainting is pretty good actually for photos and such, but as video has more information from past+future frames you can use spatio-temporal inference for more accurate spatial filling. Running a temporal denoiser on top can iron out temporal incosistencies.