Convert video node frame to numpy array or pillow image object (python)

[jlw_4049]

I was wondering if someone could point me in the right direction. I've been struggling with it a portion of the day.

Currently I'm only able to output greyscale images from the frames.

I was attempting to use code like this

rgb_clip = vs.core.ffms2.Source(video.mkv) #1frame clip

#number of planes , for rgb should be 3 anyway
planes = rgb_clip.format.num_planes

#making numpy array from vapoursynths videonode
list_of_arrays = [np.array(rgb_clip.get_frame(0).get_read_array(i), copy=False) for i in range(planes)]
numpy_array = np.dstack(list_of_arrays)

But it is not what I need

[_Al_]

Not sure what you aim for. Video should be in rgb, maybe your really is, to go frame by frame - vs frame to np array and back to vs frame:

import numpy as np
#from PIL import Image
#import cv2

def format_shuffles(n,f):
npArray = np.dstack([np.asarray(f.get_read_array(p)) for p in range(3)]) #latest vs: f[p] instead of f.get_read_array(p)
#work here with npArray image
#hsv = cv2.cvtColor(npArray, cv2.COLOR_BGR2HSV)
#PIL_img = Image.fromarray(npArray, 'RGB')
#or work with PIL image than convert it back to np array
#npArray = np.array(PIL_img)
f_out = f.copy()
[np.copyto(np.asarray(f_out.get_write_array(p)), npArray[:, :, p]) for p in range(3)]
return f_out
clip = core.std.BlankClip(format=vs.YUV420P8) #here loading YUV video with whatever source plugin
clip = clip.resize.Point(format=vs.RGB24, matrix_in_s = '709')
clip = core.std.ModifyFrame(clip, clip, format_shuffles)
clip.set_output(0)

[jlw_4049]

Thank you. I plan to make a viewer of sorts to view a video frame by frame and modify the script and visually see the changes

So this should help me. I got a 14 hour shift today so likely cannot test until tomorrow.

[_Al_]

You can just use view.py (https://github.com/UniversalAl/view) ,
just design your script in some python editor and just run the script

import vapoursynth as vs
clip = vs.core.lsmas.LibavSMASHSource('source.mp4')
clip = ... some filters ....
clip.set_output()
if __name__ == "__main__":
import view
view.Preview(clip)

it is a good idea to have that preview set in that if block, if you forget that there and actually using that py script for encoding , that block would not run

[jlw_4049]

I actually played around with view.py for a bit. I did get it to work by editing the code, but it does not work as it stands. So I figured there was no support for it anymore.

I see now that you are the developer haha!

So, I still haven't had a chance to get to my computer thanks to work, but I plan to try to play around with the code tonight/tomorrow after I'm off. I'll respond back here the results, also I'll look at view.py again!

[_Al_]

Yes, it was for API3, Vapoursynth version R55 (not sure 100%) and below, not for API4, I thought because it is run in Python, fix would be easy for anyone, because it shows errors. Not using it, I use other versions which are not published. I'll look into it.

[Quadratic]

Thank you. I plan to make a viewer of sorts to view a video frame by frame and modify the script and visually see the changes
.

https://github.com/Irrational-Encoding-Wizardry/vs-preview
https://github.com/quietvoid/vspreview-rs

[jlw_4049]

Not sure what you aim for. Video should be in rgb, maybe your really is, to go frame by frame - vs frame to np array and back to vs frame:

import numpy as np
#from PIL import Image
#import cv2

def format_shuffles(n,f):
npArray = np.dstack([np.asarray(f.get_read_array(p)) for p in range(3)]) #latest vs: f[p] instead of f.get_read_array(p)
#work here with npArray image
#hsv = cv2.cvtColor(npArray, cv2.COLOR_BGR2HSV)
#PIL_img = Image.fromarray(npArray, 'RGB')
#or work with PIL image than convert it back to np array
#npArray = np.array(PIL_img)
f_out = f.copy()
[np.copyto(np.asarray(f_out.get_write_array(p)), npArray[:, :, p]) for p in range(3)]
return f_out
clip = core.std.BlankClip(format=vs.YUV420P8) #here loading YUV video with whatever source plugin
clip = clip.resize.Point(format=vs.RGB24, matrix_in_s = '709')
clip = core.std.ModifyFrame(clip, clip, format_shuffles)
clip.set_output(0)


I tried this but it didn't result in anything.


import numpy as np
#from PIL import Image
#import cv2

def format_shuffles(n,f):
npArray = np.dstack([np.asarray(f[p]) for p in range(3)]) #latest vs: f[p] instead of f.get_read_array(p)
print(npArray)
#work here with npArray image
#hsv = cv2.cvtColor(npArray, cv2.COLOR_BGR2HSV)
#PIL_img = Image.fromarray(npArray, 'RGB')
#or work with PIL image than convert it back to np array
#npArray = np.array(PIL_img)
f_out = f.copy()
[np.copyto(np.asarray(f_out.get_write_array(p)), npArray[:, :, p]) for p in range(3)]
return f_out
clip = core.lsmas.LWLibavSource(r"C:\Users\jlw_4\Desktop\rugrats test\The Rugrats Movie (1998).mkv")
# clip = core.std.BlankClip(format=vs.YUV420P8) #here loading YUV video with whatever source plugin
clip = clip.resize.Point(format=vs.RGB24, matrix_in_s = '709')
clip = core.std.ModifyFrame(clip, clip, format_shuffles)
format_shuffles(0, clip)
clip.set_output(0)


print(npArray) doesn't return anything. Because the function isn't actually running as far as I can tell.

[_Al_]

format_shuffles(0, clip)

should be:
format_shuffles(0, clip.get_frame(0))

simplest basic opencv player, it will play without delay frame by frame:

import cv2
import numpy as np

clip = core.lsmas.LWLibavSource(r"C:\Users\jlw_4\Desktop\rugrats test\The Rugrats Movie (1998).mkv")
#assuming clip is loaded correctly from mkv
rgb = core.resize.Point(clip, format=vs.RGB24, matrix_in_s = '709')
for f in rgb.frames():
img = np.dstack([np.asarray(f.get_read_array(p)) for p in [2,1,0]]) #or f[p] ...
cv2.imshow('movie', img)
cv2.waitKey(1)

[jlw_4049]

https://github.com/Irrational-Encoding-Wizardry/vs-preview
https://github.com/quietvoid/vspreview-rs

Thanks for the reply. I did look these over, I'm not looking for a completed GUI to do this. I just am trying to understand the conversion/utilize it in a very basic window in python to complete some projects

should be:
format_shuffles(0, clip.get_frame(0))

simplest basic opencv player, it will play without delay frame by frame:

import cv2
import numpy as np

clip = core.lsmas.LWLibavSource(r"C:\Users\jlw_4\Desktop\rugrats test\The Rugrats Movie (1998).mkv")
#assuming clip is loaded correctly from mkv
rgb = core.resize.Point(clip, format=vs.RGB24, matrix_in_s = '709')
for f in rgb.frames():
img = np.dstack([np.asarray(f.get_read_array(p)) for p in [2,1,0]]) #or f[p] ...
cv2.imshow('movie', img)
cv2.waitKey(1)


This set me on the right track. I do have this working now at least. I appreciate the information so far!