Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > VapourSynth

Reply
 
Thread Tools Search this Thread Display Modes
Old 20th July 2020, 16:49   #1  |  Link
JarJarRuss
Registered User
 
Join Date: Jul 2020
Posts: 8
Conversion to and from Numpy

So I'm trying to use the bm3d plugin that Vapoursynth has. Right now, I have the image as a numpy array. The bm3d plugin requires a clip as an input. Is there a way to convert a numpy array to a clip and then back to a numpy array after bm3d?

Code:
from vapoursynth import core
from skimage import io
import mvsfunc as mvf
import numpy as np

I = io.imread('D:/Jia Lab/ACsN/Python/Python Test Images/TIRF_01_10ms.tif')
I = np.array(I[0])

# ERRORS
flt = mvf.BM3D(I[0], sigma=3.0, profile1="fast")
JarJarRuss is offline   Reply With Quote
Old 20th July 2020, 18:20   #2  |  Link
Myrsloik
Professional Code Monkey
 
Myrsloik's Avatar
 
Join Date: Jun 2003
Location: Ikea Chair
Posts: 2,250
1. Load images with imwri.Read() and that end of the problem should be solved
2. After the filter do "flt.get_frame(0).get_read_array(<plane number here>)"

That will return numpy usable views of the frame.
__________________
VapourSynth - proving that scripting languages and video processing isn't dead yet
Myrsloik is offline   Reply With Quote
Old 20th July 2020, 21:12   #3  |  Link
_Al_
Registered User
 
Join Date: May 2011
Posts: 163
something:
Code:
from vapoursynth import core
import mvsfunc as mvf
import numpy as np
clip = core.imwri.Read('D:/Jia Lab/ACsN/Python/Python Test Images/TIRF_01_10ms.tif')   #one frame clip
flt = mvf.BM3D(clip, sigma=3.0, profile1="fast")
npArray = np.dstack([np.asarray(flt.get_frame(0).get_read_array(i)) for i in range(3)])
#print(npArray)

Last edited by _Al_; 20th July 2020 at 21:22.
_Al_ is offline   Reply With Quote
Old 20th July 2020, 21:50   #4  |  Link
JarJarRuss
Registered User
 
Join Date: Jul 2020
Posts: 8
Unlike in the test example I posted, I did some preprocessing on the image after I read it in using the numpy library. Is there a way to convert the numpy array into a clip? I know I could probably save the numpy array as a tiff file and then read it in again with imwri but I'm trying to preserve some speed. Any options?
JarJarRuss is offline   Reply With Quote
Old 20th July 2020, 22:24   #5  |  Link
_Al_
Registered User
 
Join Date: May 2011
Posts: 163
you can do something like this, using ModifyFrame, which needs a placeholder vs clip with dimension as your numpy and guessing it is uint8 (RGB24 in vapoursynth, 8bit):
Code:
import vapoursynth as vs
from vapoursynth import core
import mvsfunc as mvf
import numpy as np
import functools


#you have your npArray here, npArray=I

clip_placeholder = core.std.BlankClip(width = npArray.shape[1], height = npArray.shape[0], format= vs.RGB24, length=1)

def get_vsFrame(n, f, npArray):
    vsFrame = f.copy()
    [np.copyto(np.asarray(vsFrame.get_write_array(i)), npArray[:, :, i]) for i in range(3)]
    return vsFrame
  
clip = core.std.ModifyFrame(clip_placeholder, clip_placeholder, functools.partial(get_vsFrame, npArray=npArray))
flt = mvf.BM3D(clip, sigma=3.0, profile1="fast")
npArray = np.dstack([np.asarray(flt.get_frame(0).get_read_array(i)) for i in range(3)])
Using ModifyFrame() because not sure how to make vapoursynth clip out of just frame

Last edited by _Al_; 20th July 2020 at 22:33.
_Al_ is offline   Reply With Quote
Old 21st July 2020, 17:30   #6  |  Link
JarJarRuss
Registered User
 
Join Date: Jul 2020
Posts: 8
So I did this and got an error on the last line. I've posted the error below. I think the problem lies within flt.get_frame(0) but not sure how to resolve it. The clip description before and after bm3d is also shown.

Quote:
Shape of Numpy Array
(256, 256)
VideoNode
Format: RGB24
Width: 256
Height: 256
Num Frames: 1
FPS: 24
Flags: NoCache IsCache

###########
VideoNode
Format: RGB24
Width: 256
Height: 256
Num Frames: 1
FPS: 24
Flags: NoCache

Traceback (most recent call last):
File "test.py", line 28, in <module>
I = np.dstack([np.asarray(flt.get_frame(0).get_read_array(i)) for i in range(2)])
File "test.py", line 28, in <listcomp>
I = np.dstack([np.asarray(flt.get_frame(0).get_read_array(i)) for i in range(2)])
File "src\cython\vapoursynth.pyx", line 1244, in vapoursynth.VideoNode.get_frame
vapoursynth.Error: too many indices for array
JarJarRuss is offline   Reply With Quote
Old 22nd July 2020, 01:08   #7  |  Link
_Al_
Registered User
 
Join Date: May 2011
Posts: 163
You are using I[0], just a part of it, instead of whole array I. But for RGB numpy array you need all three planes.
But isn't skimage format in numpy already? This should be enough:
Code:
npArray  = io.imread('D:/Jia Lab/ACsN/Python/Python Test Images/TIRF_01_10ms.tif')
then also use:
for i in range(3):
or
for i in [0,1,2]:
does not matter, but that error was caused using that I[0]
_Al_ is offline   Reply With Quote
Old 22nd July 2020, 15:37   #8  |  Link
JarJarRuss
Registered User
 
Join Date: Jul 2020
Posts: 8
Makes sense. Made that change. Now using two frames. Code is below. Getting a new error now though. I tried changing the length parameter from 1 to 2 but the error still persists (pasted below code). Just for reference, I transposed the original numpy array to get a shape of (256, 256, 2) instead of (2, 256, 256). If I don't to that the clip ends up having a height of 2 and nothing prints out when I try to print the shape of the numpy array after dstack.

Code:
I = io.imread('D:/Jia Lab/ACsN/Python/Python Test Images/TIRF_01_10ms.tif')
npArray= np.array(I[0:2])
npArray = np.transpose(npArray).copy() ## to get the shape (256, 256, 2) instead of (2, 256, 256)


clip_placeholder = core.std.BlankClip(width = npArray.shape[1], height = npArray.shape[0], format= vs.RGB24, length=2)

print(clip_placeholder)

def get_vsFrame(n, f, npArray):
    vsFrame = f.copy()
    [np.copyto(np.asarray(vsFrame.get_write_array(i)), npArray[:, :, i]) for i in range(3)]
    return vsFrame

clip = core.std.ModifyFrame(clip_placeholder, clip_placeholder, functools.partial(get_vsFrame, npArray=npArray))
print(clip)

flt = mvf.BM3D(clip, sigma=3.0, profile1="fast")

new = np.dstack([np.asarray(flt.get_frame(0).get_read_array(i)) for i in range(2)])
Quote:
Traceback (most recent call last):
File "zTest.py", line 29, in <module>
new = np.dstack([np.asarray(flt.get_frame(0).get_read_array(i)) for i in range(3)])
File "zTest.py", line 29, in <listcomp>
new = np.dstack([np.asarray(flt.get_frame(0).get_read_array(i)) for i in range(3)])
File "src\cython\vapoursynth.pyx", line 1244, in vapoursynth.VideoNode.get_frame
vapoursynth.Error: index 2 is out of bounds for axis 2 with size 2
JarJarRuss is offline   Reply With Quote
Old 22nd July 2020, 22:15   #9  |  Link
JarJarRuss
Registered User
 
Join Date: Jul 2020
Posts: 8
Just to make sure. The numpy array I'm converting is one where the third dimension is the number of frames not the channel. There's no channel dimension as this is a GRAY16 image.
JarJarRuss is offline   Reply With Quote
Old 23rd July 2020, 04:48   #10  |  Link
_Al_
Registered User
 
Join Date: May 2011
Posts: 163
If its grayscale, then you do not need three planes, just one plane only, using shape like this for example:
Code:
[[169 168 167 ... 114 117 106]
 [166 168 168 ... 117 116 117]
 [168 170 169 ... 116 120 118]
 ...
 [ 43  55 124 ... 142 140 138]
 [ 45  44  41 ... 149 148 152]
 [ 39  43  44 ... 168 169 169]]
Code:
#you have your npArray here, shape it accordingly
clip_placeholder = core.std.BlankClip(width = npArray.shape[1], height = npArray.shape[0], format= vs.GRAY8, length=1)
def get_clip(n, f, npArray):
    vsFrame = f.copy()
    [np.copyto( np.asarray(vsFrame.get_write_array(0)), npArray )]
    return vsFrame
  
clip = core.std.ModifyFrame(clip_placeholder, clip_placeholder, functools.partial(get_clip, npArray=npArray))
flt = mvf.BM3D(clip, sigma=3.0, profile1="fast")
npArray = np.asarray(flt.get_frame(0).get_read_array(0))
_Al_ is offline   Reply With Quote
Old 23rd July 2020, 05:58   #11  |  Link
_Al_
Registered User
 
Join Date: May 2011
Posts: 163
I tested it now and I have some weird results, sometimes it works and sometimes does not, something needs to be fixed while getting vsFrame if just using one plane.
_Al_ is offline   Reply With Quote
Old 23rd July 2020, 18:04   #12  |  Link
_Al_
Registered User
 
Join Date: May 2011
Posts: 163
ok, tried it again changing ModifyFrame calling "get_vsFrame" not "get_clip" and it works, perhaps some name conflict,
also changed it to GRAY16:
so input npArray is shaped like this, 16bit integers:
Code:
[[43360 43011 42745 ... 29113 29930 27196]
 [42587 43012 42965 ... 29834 29803 29883]
 [42949 43469 43358 ... 29583 30636 30165]
 ...
 [11099 14036 31621 ... 36331 35814 35311]
 [11492 11147 10472 ... 38043 37966 38878]
 [10095 10958 11336 ... 43084 43227 43198]]
Code:
import vapoursynth as vs
from vapoursynth import core
import mvsfunc as mvf
import numpy as np
import functools

#you have your npArray input here

clip_placeholder = core.std.BlankClip(width = npArray.shape[1], height = npArray.shape[0], format= vs.GRAY16, length=1)

def get_vsFrame(n, f, npArray):
    vsFrame = f.copy()
    np.copyto( np.asarray(vsFrame.get_write_array(0)), npArray )
    return vsFrame
  
clip = core.std.ModifyFrame(clip_placeholder, clip_placeholder, functools.partial(get_vsFrame, npArray=npArray))
flt = mvf.BM3D(clip, sigma=3.0, profile1="fast")
npArray = np.asarray(flt.get_frame(0).get_read_array(0))
#flt.set_output()

Last edited by _Al_; 23rd July 2020 at 18:10.
_Al_ is offline   Reply With Quote
Old 23rd July 2020, 22:16   #13  |  Link
JarJarRuss
Registered User
 
Join Date: Jul 2020
Posts: 8
I think this works for what I'm doing. Thank you so much!
JarJarRuss is offline   Reply With Quote
Old 28th July 2020, 17:29   #14  |  Link
JarJarRuss
Registered User
 
Join Date: Jul 2020
Posts: 8
As a follow up to this, what if I wanted to do the same thing with multiple frames. So instead of the input being (256, 256, 1), the input numpy array is (256, 256, 10). So 10 frames in this case. Tried tinkering around with the code but I kept getting some array index problems. Thanks in advance.
JarJarRuss is offline   Reply With Quote
Old 29th July 2020, 00:49   #15  |  Link
_Al_
Registered User
 
Join Date: May 2011
Posts: 163
You can try to loop that line with ModifyFrame:
Code:
for i in range(0,10):
    clip = core.std.ModifyFrame(clip_placeholder, clip_placeholder, functools.partial(get_vsFrame, npArray=npArray[:, :, i]))
    flt = mvf.BM3D(clip, sigma=3.0, profile1="fast")
    new_npArray = np.asarray(flt.get_frame(0).get_read_array(0))
    #here you add your new_npArray into your new_stack_npArray, whatever you  do it

Last edited by _Al_; 29th July 2020 at 00:59.
_Al_ is offline   Reply With Quote
Old 29th July 2020, 19:11   #16  |  Link
JarJarRuss
Registered User
 
Join Date: Jul 2020
Posts: 8
I would do this but I specifically want to use the VBM3D functionality that mvf.BM3D provides. So the input clip should be multiple frames. I guess the question is if there is a way to make a (256, 256, 10) numpy array into a 10-frame clip and then back to an array? That way I can use VBM3D instead of denoising each frame separately through a loop.
JarJarRuss is offline   Reply With Quote
Old 29th July 2020, 22:03   #17  |  Link
_Al_
Registered User
 
Join Date: May 2011
Posts: 163
yes, try this:
Code:
clip_placeholder = core.std.BlankClip(width = npArray.shape[1], height = npArray.shape[0], format= vs.GRAY16, length=10)

def get_vsFrame(n, f, npArray):
    vsFrame = f.copy()
    np.copyto( np.asarray(vsFrame.get_write_array(0)), npArray[:, :, n] )
    return vsFrame

clip = core.std.ModifyFrame(clip_placeholder, clip_placeholder, functools.partial(get_vsFrame, npArray=npArray))
flt = mvf.BM3D(clip, sigma=3.0, profile1="fast")
and further script it depends, what you need , stacked numpy again?
something like:
Code:
new_arrays = []
for i in range(0,10):
    new_arrays.append(np.asarray(flt.get_frame(i).get_read_array(0)))
npArrays = np.dstack(new_arrays)
del new_arrays
or one liner
Code:
npArrays = np.dstack([np.asarray(flt.get_frame(i).get_read_array(0))  for i in range(10)])
there might be easier way to append/stack numpy array right away, check numpy.stack
or if final shape of npArrays was wrong try to stack it as you do, you get a general idea how to get one frame array: np.asarray(flt.get_frame(i).get_read_array(0))

Last edited by _Al_; 29th July 2020 at 23:01.
_Al_ is offline   Reply With Quote
Reply

Tags
clip, numpy, python

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 21:50.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2020, vBulletin Solutions Inc.