Make "The Colors of Motion" style image

[MonoS]

Hi, some months ago i made this image http://www.mediafire.com/file/onhrbgcil8j0n67/full.png (it requires 0.5GB of RAM to be opened) to emulate this site https://thecolorsofmotion.com/

I used this script to made the image:
import vapoursynth as vs
core = vs.get_core()

src = core.lsmas.LWLibavSource("movie.mkv")
start = 0
pixel = core.fmtc.resample(src, 1,1, kernel="gauss", csp=vs.YUV444P16)

stacked = pixel[start]
concat = None
for i in range(1,pixel.num_frames):
if(i % 17280 == 0):
if(not concat):
concat = stacked
else:
concat = concat + stacked

stacked = pixel[start+i]
else:
stacked = core.std.StackHorizontal([stacked, pixel[start+i]])

last = stacked

res = core.fmtc.resample(concat, concat.width, 1080, kernel="point").fmtc.matrix(fulls=False, fulld=True, col_fam=vs.RGB, mat="709")

resl = core.fmtc.resample(last, last.width, 1080, kernel="point").fmtc.matrix(fulls=False, fulld=True, col_fam=vs.RGB, mat="709")

res = core.std.Splice([res, resl], True)

res = core.imwrif.Write(res, "PNG48", "movie %01d.png")

res.set_output()


But it was very slow, occupied a lot of ram didn't use almost none of the CPU, also i had to use the for range else it would never finish.

Is there a better way to make this kind of image?

[richardpl]

mpv MOVIE -vf "scale=1x1,format=rgba,scale=1x800,setsar=1/1,tile=1280x1:overlap=1279:init_padding=1279"

[MonoS]

Hi, i tried your script, but it play the resulting file, i instead want a single png with the whole final file.

[richardpl]

You will need to drop or average frames, because typical movie can have a lot of frames. And stacking all of them into one very huge picture is not efficient.

[foxyshadis]

Directly munging and merging frames is absolutely the wrong way to go about this. I wish I had the time to put together a full script, but you should be collecting results with something like:

frames = []
width = 1920
height = 1080
stacked = core.std.BlankClip(width=width,height=1,format=vs.YUV444P16,length=1)
for i in range(1,src.num_frames):
frames.append([core.std.PlaneStats(src).PlaneStatsAvg,core.std.PlaneStats(src,plane=1).PlaneStatsAvg,core.std.PlaneStats(src,plane=2).PlaneStatsAvg])
skip = src.num_frames % height
for i in range(1,src.num_frames,skip):
colorline = core.std.BlankClip(width=width,height=1,format=vs.YUV444P16,length=1,color=frames[i])
stacked = core.std.StackHorizontal(stacked,colorline)

I know there's a way to write into a frame without creating new ones every time, but at least that's less intensive than the method you were using.

This doesn't actually work though; nothing in VS core seems to work as documented. I don't blame you for barking up the wrong tree; it's pretty difficult to actually use the more interesting parts of VS from the documentation. I'm hoping I can fix that this summer.

[richardpl]

Hi, i tried your script, but it play the resulting file, i instead want a single png with the whole final file.

You will need to first calculate number of frames in input movie:

ffmpeg -i MOVIE -c:v copy -an -f null -

Let it be for this example 135407.

Than divide that number with width of output image: 135407 / 1280 = 105.786..

Take next higher number which is 106, and put it into first tile filter:

ffmpeg -i MOVIE -vf "scale=1x1,format=rgba,tile=106x1,scale=1x1,scale=1x800,setsar=1/1,tile=1280x1" output.png

And after few minutes you get final image.

[MonoS]

Hi richardpl, i'm not interested in averaging the frames, i'm interested in the whole picture even if it takes half an hour.

I've modified your script changing the tile to "1x1" and tile to "{framenumber}x1" and it worked it worked producing the desired image after about 30s for a 25m 720p video.

Regarding foxyshadis script, i've not tried yours because you said that it doesn't works, also i can't quite grasp the differences between our two solution, doesn't we do the same at the end?

[richardpl]

Sorry but producing such huge images is just insane. And that webpage you linked does not have images that big.

[MonoS]

it is correct that the site i linked doesn't have such huge images, but i like that they contains all the frames, no particular reason :)

By the way, i've produced the image from the same file, once with ffmpeg and the other with my original script, they gave me quite different colors
http://www.mediafire.com/file/cp59lvelkobq225/output%20files.zip
I can't grasp if it is due to the kernel used (is a gaussian kernel a good choice, or will bilinear be better?) or due to colorspace differences.

[richardpl]

Probably because format=rgba filter should be first:

ffmpeg -i MOVIE -vf "format=rgba,scale=1x1,scale=1x800,setsar=1/1,tile=1280x1" output.png