OpenCV Inpainting

[Selur]

I read:
https://github.com/WolframRhodium/muvsfunc/wiki/OpenCV-Python-for-VapourSynth

But couldn't get it working.
I posted in the issue tracker over at https://github.com/WolframRhodium/muvsfunc/issues/50
but since I didn't get a reply for 3 weeks I'm wondering if anybody here got inpainting through OpenCV working in Vapoursynth and might be able to help me get it working.


Cu Selur

[Selur]

Got a reply in the issue tracker now, sadly with the new changes the code simply doesn't seem to do anything.

[Selur]

In case someone is interested, thanks to WolframRhodium it is working now using:
##### INPAINTING CODE START

# convert clip to RGB
clip = core.resize.Bicubic(clip, format=vs.RGB24, matrix_in_s="470bg")
input = clip
# build mask
mask = core.imwri.Read(["C:/Users/Selur/Desktop/logo.png"])
mask = core.std.BinarizeMask(mask,threshold=16)
mask = core.std.ShufflePlanes(mask, [0], vs.GRAY)

# apply inpainting
import cv2
import muvsfunc_numpy as mufnp
import numpy as np

def inpaint_core(img, mask, radius=1, flags=cv2.INPAINT_NS):
return cv2.inpaint(img, mask, inpaintRadius=radius, flags=flags)

clip = mufnp.numpy_process([clip, mask], inpaint_core, radius=1, flags=cv2.INPAINT_NS, input_per_plane=False, output_per_plane=False)

##### INPAINTING CODE END
The result isn't that great, but it seems to be working:
https://user-images.githubusercontent.com/843640/196023789-69822cca-18b1-4692-9eee-68954ef5539a.png
https://user-images.githubusercontent.com/843640/196024543-d0198a60-39f2-4636-b6a4-458e0c37959e.png

Cu Selur

[Selur]

I also noticed one could use xphoto.inpaint:
using xphoto inpaint:
(requires additionally the contrib package: 'pip install opencv-contrib-python')

##### INPAINTING CODE START

# convert clip to RGB
clip = core.resize.Bicubic(clip, format=vs.RGB24, matrix_in_s="470bg")

# build mask
mask = core.imwri.Read(["C:/Users/Selur/Desktop/logo.png"])
mask = core.std.BinarizeMask(mask,threshold=16)
mask = core.std.ShufflePlanes(mask, [0], vs.GRAY)

# apply inpainting
import cv2
import muvsfunc_numpy as mufnp
import numpy as np

def xphoto_inpaint(img, img_mask , algorithmType=cv2.xphoto.INPAINT_SHIFTMAP):
img_lab = cv2.cvtColor(src=img, code=cv2.COLOR_RGB2Lab)
img_mask_inv = cv2.bitwise_not(img_mask )
img_dst = np.zeros(img.shape, np.uint8)
cv2.xphoto.inpaint(src=img_lab, mask=img_mask_inv, dst=img_dst, algorithmType=algorithmType)
img_dst = cv2.cvtColor(src=img_dst, code=cv2.COLOR_Lab2RGB)
return img_dst

#clip = mufnp.numpy_process([clip, mask], xphoto_inpaint, algorithmType=cv2.xphoto.INPAINT_FSR_FAST, input_per_plane=False, output_per_plane=False)
clip = mufnp.numpy_process([clip, mask], xphoto_inpaint, algorithmType=cv2.xphoto.INPAINT_FSR_BEST, input_per_plane=False, output_per_plane=False)
#clip = mufnp.numpy_process([clip, mask], xphoto_inpaint, algorithmType=cv2.xphoto.INPAINT_SHIFTMAP, input_per_plane=False, output_per_plane=False)

##### INPAINTING CODE END

https://i.ibb.co/0DHJW0c/xphoto.png (https://ibb.co/1vF2pCL)

Here's a small comparison of the methods:
https://i.ibb.co/HzBMLPb/Frame-0000.png (https://imgbb.com/)
https://i.ibb.co/0YrTrxR/Frame-0350.png (https://imgbb.com/)
https://i.ibb.co/Tkk9ZsY/Frame-0700.png (https://imgbb.com/)
https://i.ibb.co/Fb5gqfJ/Frame1400.png (https://imgbb.com/)

Cu Selur

Ps.: I also tried https://scikit-image.org, but that atm. is way too slow to be usable (minutes per frame on SD).

[cretindesalpes]

If your goal is removing a semi-transparent logo, what about EraseLOGO? The tool is old, but it gave me excellent results with a bit of post-processing. It’s also incredibly fast given modern standards.

EDIT: ah… I realize the VS eraselogo function is probably not the same as the old avisynth EraseLOGO plugin ported from AviUtl.

[Selur]

Got no real goal, I just was wondering why there was no alternative to InpaintDelogo for Vapoursynth, and then I realized there was no port of AvsInpainting for Vapoursynth.
Then I stumbled over https://github.com/WolframRhodium/muvsfunc/wiki/OpenCV-Python-for-VapourSynth and since it didn't work for me, I opened an issue entry over at github.
3 weeks later I still got no reply, so I thought that posting here might help and it kind of did since WolframRhodium when replied to my issue-entry.
-> atm. I'm just playing around and mainly created the later posts so others might have an easier way in case they want to use inpainting in Vapoursynth. :)

Cu Selur

Ps.: looking at https://bleedai.com/designing-advanced-image-filters-in-opencv-creating-instagram-filters-pt-3%E2%81%843/ the 'applyPencilSketch' looks like a gimmicky feature that would be nice to use in Vapoursynth.

[Selur]

Got the pencilSketch working through:
import cv2
import muvsfunc_numpy as mufnp
import numpy as np

def applyPencilSketch(image, sigmaS: int = 20, sigmaR: float = 0.5, shadeFactor: float = 0.02, outputGray: bool = False):
'''
This function will create instagram Pencil Sketch filter like effect on an image.
Args:
image: The image on which the filter is to be applied.
sigma_s: The smoothing sigma value controls the area of the neighbourhood (Has Range between 0-200)
sigma_r: This param controls the how dissimilar colors within the neighborhood will be averaged. For example a larger value will restrcit color variation and it will enforce that constant color stays throughout. (Has Range between 0-1)
shade_factor: This has range 0-0.1 and controls how bright the final output will be by scaling the intensity.
outputGray: This specified whether a color or gray scale sketch should be returned.
Returns:
output_image: A copy of the input image with the Pencil Sketch filter applied.
'''

# Apply Pencil Sketch effect on the image.
gray_sketch, color_sketch = cv2.pencilSketch(image, sigma_s=sigmaS, sigma_r=sigmaR, shade_factor=shadeFactor)

# Return the output image.
if (outputGray):
return gray_sketch

return color_sketch

clip = core.resize.Bicubic(clip, format=vs.RGB24, matrix_in_s="470bg")
clip = mufnp.numpy_process([clip], applyPencilSketch, sigmaS=20, sigmaR=0.5, shadeFactor=0.01, outputGray=False, input_per_plane=False, output_per_plane=False)

Only thing that isn't working about it, is that 'outputGray=True' fails with:
Error on frame 0 request:

too many indices for array: array is 2-dimensional, but 3 were indexed
and I have no clue why. :)

Cu Selur

[poisondeathray]

Has anyone tested any of the machine learning inpainting projects ? There are video/temporal and single image versions

[VoodooFX]

What if you use radius bigger than 1 with cv2.inpaint? What are speeds of those inpainters?

[Selur]

With and "VSPipe.exe c:\Users\Selur\Desktop\test.vpy NUL" using:
# Imports
import vapoursynth as vs
import os
import ctypes
# Loading Support Files
Dllref = ctypes.windll.LoadLibrary("i:/Hybrid/64bit/vsfilters/Support/libfftw3f-3.dll")
import sys
# getting Vapoursynth core
core = vs.core
# Import scripts folder
scriptPath = 'i:/Hybrid/64bit/vsscripts'
sys.path.insert(0, os.path.abspath(scriptPath))
# Loading Plugins
core.std.LoadPlugin(path="i:/Hybrid/64bit/vsfilters/GrainFilter/RemoveGrain/RemoveGrainVS.dll")
core.std.LoadPlugin(path="i:/Hybrid/64bit/vsfilters/GrainFilter/AddGrain/AddGrain.dll")
core.std.LoadPlugin(path="i:/Hybrid/64bit/vsfilters/DenoiseFilter/FFT3DFilter/fft3dfilter.dll")
core.std.LoadPlugin(path="i:/Hybrid/64bit/vsfilters/DenoiseFilter/DFTTest/DFTTest.dll")
core.std.LoadPlugin(path="i:/Hybrid/64bit/vsfilters/Support/EEDI3m.dll")
core.std.LoadPlugin(path="i:/Hybrid/64bit/vsfilters/ResizeFilter/nnedi3/vsznedi3.dll")
core.std.LoadPlugin(path="i:/Hybrid/64bit/vsfilters/Support/libmvtools.dll")
core.std.LoadPlugin(path="i:/Hybrid/64bit/vsfilters/Support/scenechange.dll")
core.std.LoadPlugin(path="i:/Hybrid/64bit/vsfilters/Support/fmtconv.dll")
core.std.LoadPlugin(path="i:/Hybrid/64bit/vsfilters/MiscFilter/MiscFilters/MiscFilters.dll")
core.std.LoadPlugin(path="i:/Hybrid/64bit/vsfilters/SourceFilter/LSmashSource/vslsmashsource.dll")
core.std.LoadPlugin(path="i:/Hybrid/64bit/vsfilters/Support/libimwri.dll")
# Import scripts
import havsfunc
# source: 'G:\TestClips&Co\files\MPEG-2\ZDF_Logoremoval_Disco 1977-04 077 576p Digi-TVRip - Stormjoe .mkv'
# current color space: YUV420P8, bit depth: 8, resolution: 720x576, fps: 25, color matrix: 470bg, yuv luminance scale: limited, scanorder: top field first
# Loading G:\TestClips&Co\files\MPEG-2\ZDF_Logoremoval_Disco 1977-04 077 576p Digi-TVRip - Stormjoe .mkv using LWLibavSource
clip = core.lsmas.LWLibavSource(source="G:/TestClips&Co/files/MPEG-2/ZDF_Logoremoval_Disco 1977-04 077 576p Digi-TVRip - Stormjoe .mkv", format="YUV420P8", stream_index=0, cache=0, prefer_hw=0)
# Setting color matrix to 470bg.
clip = core.std.SetFrameProps(clip, _Matrix=5)
clip = clip if not core.text.FrameProps(clip,'_Transfer') else core.std.SetFrameProps(clip, _Transfer=5)
clip = clip if not core.text.FrameProps(clip,'_Primaries') else core.std.SetFrameProps(clip, _Primaries=5)
# Setting color range to TV (limited) range.
clip = core.std.SetFrameProp(clip=clip, prop="_ColorRange", intval=1)
# making sure frame rate is set to 25
clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)
# setting field order to what QTGMC should assume (top field first)
clip = core.std.SetFrameProp(clip=clip, prop="_FieldBased", intval=2)
# Deinterlacing using QTGMC
clip = havsfunc.QTGMC(Input=clip, Preset="Fast", TFF=True) # new fps: 25
# make sure content is preceived as frame based
clip = core.std.SetFieldBased(clip, 0)
clip = clip[::2]

# resize to square pixel
clip = core.resize.Bicubic(clip=clip, width=768, height=576)

##### INPAINTING CODE START

# convert clip to RGB
clip = core.resize.Bicubic(clip, format=vs.RGB24, matrix_in_s="470bg")
input = clip
# build mask
mask = core.imwri.Read(["C:/Users/Selur/Desktop/logo.png"])
mask = core.std.BinarizeMask(mask,threshold=16)
mask = core.std.ShufflePlanes(mask, [0], vs.GRAY)

# apply inpainting
import cv2
import muvsfunc_numpy as mufnp
import numpy as np

def inpaint_core(img, mask, radius=1, flags=cv2.INPAINT_NS):
return cv2.inpaint(img, mask, inpaintRadius=radius, flags=flags)

clip = mufnp.numpy_process([clip, mask], inpaint_core, radius=1, flags=cv2.INPAINT_NS, input_per_plane=False, output_per_plane=False)

##### INPAINTING CODE END

# set output frame rate to 25fps
clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)

# Output
clip.set_output()
I get:
without the inpainting block: (no clue why this so slow, but cpu usage is the highest in this run)
Output 1498 frames in 15.66 seconds (95.66 fps
with radius=1
Output 1498 frames in 15.18 seconds (98.71 fps)
with radius=2
Output 1498 frames in 15.06 seconds (99.50 fps)
with radius=3
Output 1498 frames in 15.64 seconds (95.81 fps)
with radius=4
Output 1498 frames in 16.78 seconds (89.27 fps)
with radius=5
Output 1498 frames in 18.72 seconds (80.00 fps)
with radius=10
Output 1498 frames in 32.87 seconds (45.58 fps)
with radius=20
Output 1498 frames in 106.13 seconds (14.11 fps)
with radius=100 I get 0.85fps.

With INPAINT_TELEA
with radius=1
Output 1498 frames in 16.31 seconds (91.86 fps)
with radius=2
Output 1498 frames in 16.47 seconds (90.96 fps)
with radius=3
Output 1498 frames in 18.90 seconds (79.26 fps)
with radius=4
Output 1498 frames in 19.79 seconds (75.68 fps)
with radius=5
Output 1498 frames in 20.22 seconds (74.08 fps)
with radius=10
Output 1498 frames in 30.02 seconds (49.90 fps)
with radius=20
Output 1498 frames in 82.80 seconds (18.09 fps)

with radius=100 I get 0.89fps

Cu Selur

[VoodooFX]

That is fast, but it doesn't look good in examples.

xphoto FAST/BEST looks promising, how is speed with it? Isn't there other settings to control it?

Try xphoto on these images:

https://i.imgur.com/ALwbEUR.png https://i.imgur.com/KYfwTZY.png

mask: https://i.imgur.com/K70wHF4.png

[Selur]

No there is no radius or similar for xphoto.inpaint:
Parameters
src source image

INPAINT_SHIFTMAP: it could be of any type and any number of channels from 1 to 4. In case of 3- and 4-channels images the function expect them in CIELab colorspace or similar one, where first color component shows intensity, while second and third shows colors. Nonetheless you can try any colorspaces.
INPAINT_FSR_BEST or INPAINT_FSR_FAST: 1-channel grayscale or 3-channel BGR image.

mask mask (CV_8UC1), where non-zero pixels indicate valid image area, while zero pixels indicate area to be inpainted
dst destination image
algorithmType see xphoto::InpaintTypes
Source: https://docs.opencv.org/4.x/d7/d72/xphoto_2include_2opencv2_2xphoto_2inpainting_8hpp.html

Without inpaint:
VSPipe.exe c:\Users\Selur\Desktop\test4.vpy NUL
Output 100 frames in 0.01 seconds (10002.70 fps)
# Imports
import vapoursynth as vs
import os

import sys
# getting Vapoursynth core
core = vs.core

# Import scripts folder
scriptPath = 'i:/Hybrid/64bit/vsscripts'
sys.path.insert(0, os.path.abspath(scriptPath))

# Loading Plugins
core.std.LoadPlugin(path="i:/Hybrid/64bit/vsfilters/Support/libimwri.dll")
# source: 'C:/Users/Selur/Desktop/photo_1.png'
# current color space: RGB24, bit depth: 8, resolution: 140x140, fps: 25, color matrix: 470bg, yuv luminance scale: limited, scanorder: progressive
# Loading C:\Users\Selur\Desktop\photo_1.png using vsImageReader
clip = core.imwri.Read(["C:/Users/Selur/Desktop/photo_1.png"])
clip = core.std.Loop(clip=clip, times=100)
# Input color space is assumed to be RGB24
clip = clip if not core.text.FrameProps(clip,'_Transfer') else core.std.SetFrameProps(clip, _Transfer=5)
clip = clip if not core.text.FrameProps(clip,'_Primaries') else core.std.SetFrameProps(clip, _Primaries=5)
# Setting color range to TV (limited) range.
clip = core.std.SetFrameProp(clip=clip, prop="_ColorRange", intval=1)
# making sure frame rate is set to 25
clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)
# adjusting output color from: RGB24 to YUV420P8 for x264Model
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P8, matrix_s="470bg", range_s="limited")
# set output frame rate to 25fps
clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)
# Output
clip.set_output()
With xphoto.inpainting:
# Imports
import vapoursynth as vs
import os

import sys
# getting Vapoursynth core
core = vs.core

# Import scripts folder
scriptPath = 'i:/Hybrid/64bit/vsscripts'
sys.path.insert(0, os.path.abspath(scriptPath))

# Loading Plugins
core.std.LoadPlugin(path="i:/Hybrid/64bit/vsfilters/Support/libimwri.dll")
# source: 'C:/Users/Selur/Desktop/photo_1.png'
# current color space: RGB24, bit depth: 8, resolution: 140x140, fps: 25, color matrix: 470bg, yuv luminance scale: limited, scanorder: progressive
# Loading C:\Users\Selur\Desktop\photo_1.png using vsImageReader
clip = core.imwri.Read(["C:/Users/Selur/Desktop/photo_1.png"])
clip = core.std.Loop(clip=clip, times=100)
# Input color space is assumed to be RGB24
clip = clip if not core.text.FrameProps(clip,'_Transfer') else core.std.SetFrameProps(clip, _Transfer=5)
clip = clip if not core.text.FrameProps(clip,'_Primaries') else core.std.SetFrameProps(clip, _Primaries=5)
# Setting color range to TV (limited) range.
clip = core.std.SetFrameProp(clip=clip, prop="_ColorRange", intval=1)


##### INPAINTING CODE START

# convert clip to RGB
#clip = core.resize.Bicubic(clip, format=vs.RGB24, matrix_in_s="470bg")

input = clip
# build mask
mask = core.imwri.Read(["C:/Users/Selur/Desktop/photo_mask.png"])
mask = core.std.BinarizeMask(mask,threshold=16)
mask = core.std.Maximum(mask)
mask = core.std.ShufflePlanes(mask, [0], vs.GRAY)

# apply inpainting
import cv2
import muvsfunc_numpy as mufnp
import numpy as np

def xphoto_inpaint(img, img_mask , algorithmType=cv2.xphoto.INPAINT_SHIFTMAP):
img_lab = cv2.cvtColor(src=img, code=cv2.COLOR_RGB2Lab)
img_mask_inv = cv2.bitwise_not(img_mask )
img_dst = np.zeros(img.shape, np.uint8)
cv2.xphoto.inpaint(src=img_lab, mask=img_mask_inv, dst=img_dst, algorithmType=algorithmType)
img_dst = cv2.cvtColor(src=img_dst, code=cv2.COLOR_Lab2RGB)
return img_dst

clip = mufnp.numpy_process([clip, mask], xphoto_inpaint, algorithmType=cv2.xphoto.INPAINT_FSR_FAST, input_per_plane=False, output_per_plane=False)
#clip = mufnp.numpy_process([clip, mask], xphoto_inpaint, algorithmType=cv2.xphoto.INPAINT_FSR_BEST, input_per_plane=False, output_per_plane=False)
#clip = mufnp.numpy_process([clip, mask], xphoto_inpaint, algorithmType=cv2.xphoto.INPAINT_SHIFTMAP, input_per_plane=False, output_per_plane=False)

##### INPAINTING CODE END

# making sure frame rate is set to 25
clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)
# adjusting output color from: RGB24 to YUV420P8 for x264Model
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P8, matrix_s="470bg", range_s="limited")
# set output frame rate to 25fps
clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)
# Output
clip.set_output()
using INPAINT_FSR_FAST on 1st image:
Output 100 frames in 41.45 seconds (2.41 fps)
https://i.ibb.co/fkThqQB/INPAINT-FSR-FAST.png (https://imgbb.com/)
using INPAINT_FSR_BEST on 1st image:
VSPipe.exe -p c:\Users\Selur\Desktop\test4.vpy NUL (added the -p to see progress)
Script evaluation done in 0.26 seconds
Output 100 frames in 887.56 seconds (0.11 fps)
https://i.ibb.co/Rzzc7kw/INPAINT-FSR-BEST.png (https://imgbb.com/)
using INPAINT_SHIFTMAP on 1st image:
Output 100 frames in 33.97 seconds (2.94 fps)
https://i.ibb.co/K6j03TT/INPAINT-SHIFTMAP.png (https://imgbb.com/)

As a general side note: Doesn't seem to be that any of these are really optimized/parallelized, as the CPU usage is ~3.3% on my system for vspipe.

Off to work now, will run the 2nd image once I'm back from work.

Cu Selur

[VoodooFX]

As a general side note: Doesn't seem to be that any of these are really optimized/parallelized, as the CPU usage is ~3.3%

Can't you parallelize in VS like with Prefetch in AvS?
Forgot to write that you need to expand that mask by one pixel.

AVSInpaint results for comparison:
https://i.imgur.com/CXE1OWd.png https://i.imgur.com/O3NiNc8.png

I don't remember exact FPS, approximately 10 FPS, 5 FPS, 2 FPS. [Single thread. 10 years old CPU]

[kedautinh12]

Can't you parallelize in VS like with Prefetch in AvS?
Forgot to write that you need to expand that mask by one pixel.

AVSInpaint results for comparison:
https://i.imgur.com/CXE1OWd.png https://i.imgur.com/O3NiNc8.png

I don't remember exact FPS, approximately 10 FPS, 5 FPS, 2 FPS. [Single thread. 10 years old CPU]

AvsInpant is my proud cause this is script development than vs in delogo :D

[VoodooFX]

AvsInpant is my proud cause this is script development than vs in delogo :D
I just wonder, would you get comprehensible translation if you google translate (https://translate.google.com) that sentence back to your language? :)

[kedautinh12]

No just use my noob English knowledge :D

[Selur]

Can't you parallelize in VS like with Prefetch in AvS?
I don't think so, at least I don't know of a way to do it,...
Forgot to write that you need to expand that mask by one pixel.
I used core.std.Maximum (http://www.vapoursynth.com/doc/functions/video/minimum_maximum.html) so that should be fine.

Cu Selur

[Selur]

Here's output for the second image.

using INPAINT_FSR_FAST on 2nd image:
Output 100 frames in 35.84 seconds (2.79 fps)
https://i.ibb.co/r0x8KdR/INPAINT-FSR-FAST-2.png (https://imgbb.com/)
using INPAINT_FSR_BEST on 2nd image:

https://i.ibb.co/WyWbSR9/INPAINT-FSR-BEST-2.png (https://imgbb.com/)
using INPAINT_SHIFTMAP on 2nd image:
Output 100 frames in 24.47 seconds (4.09 fps)
https://i.ibb.co/wQ2X4Yz/INPAINT-SHIFTMAP-2.png (https://imgbb.com/)

=> if someone knows a ways to get this faster, let me know.
There are also two inpainting models that could be used through vsgan (https://upscale.wiki/wiki/Model_Database#Inpainting)

Cu Selur

[Selur]

Okay, here's what VSGAN with '1x_NMKD-YandereInpaint_375000_G' produces, using:
(I'm running this on my old Geforce 1070 ti)
# Imports
import vapoursynth as vs
import os
import sys
# getting Vapoursynth core
core = vs.core
# Import scripts folder
scriptPath = 'i:/Hybrid/64bit/vsscripts'
sys.path.insert(0, os.path.abspath(scriptPath))
# Loading Plugins
core.std.LoadPlugin(path="i:/Hybrid/64bit/vsfilters/Support/fmtconv.dll")
core.std.LoadPlugin(path="i:/Hybrid/64bit/vsfilters/Support/libimwri.dll")
# Import scripts
import mvsfunc
# source: 'C:/Users/Selur/Desktop/photo_1.png'
# current color space: RGB24, bit depth: 8, resolution: 140x140, fps: 25, color matrix: 470bg, yuv luminance scale: limited, scanorder: progressive
# Loading C:\Users\Selur\Desktop\photo_1.png using vsImageReader
clip = core.imwri.Read(["C:/Users/Selur/Desktop/photo_1.png"])
clip = core.std.Loop(clip=clip, times=100)
# Input color space is assumed to be RGB24

# resizing using VSGAN
from vsgan import ESRGAN
mask = core.imwri.Read(["C:/Users/Selur/Desktop/photo_mask.png"])
mask = core.std.BinarizeMask(mask,threshold=16)
mask = core.std.Maximum(mask)
green = core.std.BlankClip(clip, color=[0,255,0])
clip = core.std.MaskedMerge(clip, green , mask) # masked clip


vsgan = ESRGAN(clip=clip,device="cuda")
model = "I:/Hybrid/64bit/vsgan_models/1x_NMKD-YandereInpaint_375000_G.pth"
vsgan.load(model)
vsgan.apply() # 140x140
clip = vsgan.clip

# Output
clip.set_output()
on the first image:
Output 100 frames in 54.75 seconds (1.83 fps)
https://i.ibb.co/0cyGs87/yandre-1.png (https://imgbb.com/)
on the second image:
Output 100 frames in 52.94 seconds (1.89 fps)
https://i.ibb.co/tQw64KX/yandre-2.png (https://imgbb.com/)

Applying 2xMaximum:
on the first image:
Output 100 frames in 51.05 seconds (1.96 fps)
https://i.ibb.co/0YQ9mBD/yandre-3.png (https://imgbb.com/)
on the second image:
Output 100 frames in 52.69 seconds (1.90 fps)
https://i.ibb.co/6ySQbLt/yandre-4.png (https://imgbb.com/)

Cu Selur

[Julek]

Can't you parallelize in VS like with Prefetch in AvS?
Only if you use a native C plugin, OpenCV in VS copies the frame via numpy, and Python is always single core.
#edit: I did some tests now and it seems that the algorithm is quite slow too, python speed is not the main problem here.

[Selur]

Okay, so using opencv in Vapoursynth doesn't really make sense and we are hoping for a avsInpaint port to Vapoursynth?

[Julek]

I made a plugin in C++ (which still uses OpenCV internally) to see how the speed is without python, I got ~10 FPS in a 1080p RGB24 (INPAINT_FSR_FAST).

[Selur]

Okay, how was the cpu usage?
Like I wrote before, I barely got 3% here when using it through Python.

[Julek]

Yes, the plugin uses 100% of the CPU, you can test it here (https://github.com/dnjulek/VapourSynth-cv_inpaint).


src = core.imwri.Read('src.png')
mask = core.imwri.Read('mask.png').resize.Point(format=vs.GRAY8, matrix=1)
mask = mask.std.BinarizeMask(50).std.Maximum().std.Maximum()

src = src.std.Loop(10000)
mask = mask.std.Loop(10000)
core.cv_inpaint.InpaintTelea(src, mask, 3).set_output()

InpaintTelea: Output 10000 frames in 4.95 seconds (2019.05 fps)


src = core.imwri.Read('src.png')
mask = core.imwri.Read('mask.png').resize.Point(format=vs.GRAY8, matrix=1)
mask = mask.std.BinarizeMask(50).std.Maximum().std.Maximum()

src = src.std.Loop(10000)
mask = mask.std.Loop(10000)
core.cv_inpaint.InpaintNS(src, mask, 3).set_output()

InpaintNS: Output 10000 frames in 5.66 seconds (1766.30 fps)


src = core.imwri.Read('src.png')
mask = core.imwri.Read('mask.png').resize.Point(format=vs.GRAY8, matrix=1)
mask = mask.std.BinarizeMask(50).std.Maximum().std.Maximum()

src = src.std.Loop(1000)
mask = mask.std.Loop(1000)
core.cv_inpaint.InpaintFSR(src, mask).set_output()

InpaintFSR: Output 1000 frames in 66.64 seconds (15.01 fps)


src = core.imwri.Read('src.png')
mask = core.imwri.Read('mask.png').resize.Point(format=vs.GRAY8, matrix=1)
mask = mask.std.BinarizeMask(50).std.Maximum().std.Maximum()

src = src.std.Loop(200)
mask = mask.std.Loop(200)
core.cv_inpaint.InpaintShiftmap(src, mask).set_output()

InpaintShiftmap: Output 200 frames in 74.99 seconds (2.67 fps)

INPAINT_FSR_BEST and INPAINT_SHIFTMAP are as slow as via python, so INPAINT_FSR_BEST was not implemented.

[ChaosKing]

vsrepo install cv_inpaint

Thank you for creating the plugin.

[Selur]

@Julek: Nice! Thanks. Will do some testing over the weekend.

Cu Selur

[Selur]

Nice!
with:
# Imports
import vapoursynth as vs
import os

import sys
# getting Vapoursynth core
core = vs.core

# Import scripts folder
scriptPath = 'i:/Hybrid/64bit/vsscripts'
sys.path.insert(0, os.path.abspath(scriptPath))

# Loading Plugins
core.std.LoadPlugin(path="i:/Hybrid/64bit/vsfilters/Support/libimwri.dll")
# source: 'C:/Users/Selur/Desktop/photo_2.png'
# current color space: RGB24, bit depth: 8, resolution: 140x140, fps: 25, color matrix: 470bg, yuv luminance scale: limited, scanorder: progressive
# Loading C:\Users\Selur\Desktop\photo_2.png using vsImageReader
clip = core.imwri.Read(["C:/Users/Selur/Desktop/photo_2.png"])
clip = core.std.Loop(clip=clip, times=1000)
# Input color space is assumed to be RGB24
clip = clip if not core.text.FrameProps(clip,'_Transfer') else core.std.SetFrameProps(clip, _Transfer=5)
clip = clip if not core.text.FrameProps(clip,'_Primaries') else core.std.SetFrameProps(clip, _Primaries=5)
# Setting color range to TV (limited) range.
clip = core.std.SetFrameProp(clip=clip, prop="_ColorRange", intval=1)


# convert clip to RGB
#clip = core.resize.Bicubic(clip, format=vs.RGB24, matrix_in_s="470bg")

input = clip
# build mask
mask = core.imwri.Read(["C:/Users/Selur/Desktop/photo_mask.png"])
mask = core.std.BinarizeMask(mask,threshold=16)
mask = core.std.Maximum(mask)
mask = core.std.Maximum(mask)
mask = core.std.Loop(mask, clip.num_frames)
mask = core.std.ShufflePlanes(mask, [0], vs.GRAY)

core.std.LoadPlugin(path="i:/Hybrid/64bit/vsfilters/Support/cv_inpaint.dll")
clip = core.cv_inpaint.InpaintNS(clip, mask, 3)
#clip = core.cv_inpaint.InpaintTelea(clip, mask, 3)

#clip = core.cv_inpaint.InpaintFSR(clip, mask)
#
clip = core.cv_inpaint.InpaintShiftmap(clip, mask)


# making sure frame rate is set to 25
clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)

# Output
clip.set_output()

InpaintNS gives me:
Output 1000 frames in 0.38 seconds (2653.32 fps)
InpaintTELEA gives me:
Output 1000 frames in 0.36 seconds (2783.15 fps)
InpaintFSR gives me:
Output 1000 frames in 77.50 seconds (12.90 fps)
InpaintShiftmap gives me:
Output 1000 frames in 375.55 seconds (2.66 fps)
I'm wondering why InpaintFSR quite a bit slower for me.


Cu Selur