Automatically changing Images while displaying video streams

[a1s2d3f4]

I have some videos that I have been manipulating with avisynth. I also have a bunch of PNG files that have some information related to the videos, which I would like to display underneath as static images, changing them at regular intervals.

I have a 1920x604 video clip (itself made up of two different video streams).

Here is what I currently have in the relevant part of my code:

image1 = ImageSource(general_path_to_images+"0001.png").BilinearResize(1920,476,120,100,3050,700).ConvertToYV12() #1st system on page
StackVertical(StackHorizontal(video_stream_a, video_stream_b),image1).DelayAudio(audio_delay_sec)


This works great in that I get the video + the image that I want to see.
BUT, I'd like this image to automatically change every 4.9 seconds (which would be every 293.706 frames at my video framerate of 59.94, so, sometimes every 293rd frame, and sometimes every 294th frame) and go through the following changes:



image1 = ImageSource(general_path_to_images+"0001.png").BilinearResize(1920,476,120,100,3050,700).ConvertToYV12() #1st system on page
#after 4.9 secs change to...
image1 = ImageSource(general_path_to_images+"0001.png").BilinearResize(1920,476,120,720,3050,700).ConvertToYV12() #2nd system on page
#after 4.9 secs change to...
image1 = ImageSource(general_path_to_images+"0001.png").BilinearResize(1920,476,120,1220,3050,700).ConvertToYV12() #3rd system on page
#after 4.9 secs change to...
image1 = ImageSource(general_path_to_images+"0001.png").BilinearResize(1920,476,120,1820,3050,700).ConvertToYV12() #4th system on page
#after 4.9 secs...
#Increment image counter +1 (making it 0002.png)
#loop through these changes again
#Continue till no more images, at which point set image to display "Black.png"


#NB, if it helps, I could use a "constant" increment for my slider down the vertical of the image. Right now it is 100, 720, 1220, 1820. I could make it into 100, 680, 1260, 1840, which increments it by exactly 580 each time

I am really hoping there exists something like a loop for changing variables (not looping clips) in AviSynth.

Thanks.


p.s. I had started a related discussion in the thread below, but since I am no longer trying to use the original PDF as my image source, I'm not continuing to post there.
http://forum.doom9.org/showthread.php?p=1457394#post1457394

[Gavino]

I am really hoping there exists something like a loop for changing variables (not looping clips) in AviSynth.
Well, there is GScript (see my sig).
But doing it that way you would end up with a large number of ImageSource instances, likely running out of memory. How many different images are there?

[Gavino]

Thinking further, I've figured out a way to approach your problem without proliferating ImageSource instances, using GScript.
n = 294 # frames per image change
nImages = ... # number of distinct images
images = ImageSource(general_path_to_images+"%04d.png", start=1, end=nImages).ConvertToYV12()
i1 = images.BilinearResize(1920,476,120,100,3050,700) #1st system on page
i2 = images.BilinearResize(1920,476,120,720,3050,700) #2nd system on page
i3 = images.BilinearResize(1920,476,120,1220,3050,700) #3rd system on page
i4 = images.BilinearResize(1920,476,120,1820,3050,700) #4th system on page
top = StackHorizontal(video_stream_a, video_stream_b)
c = i1.Trim(0, -1).Loop(n)+i2.Trim(0, -1).Loop(n)+i3.Trim(0, -1).Loop(n)+i4.Trim(0, -1).Loop(n)
GScript("""
for (i=1, nImages-1) {
c = c + i1.Trim(i, -1).Loop(n)+i2.Trim(i, -1).Loop(n)+i3.Trim(i, -1).Loop(n)+i4.Trim(i, -1).Loop(n)
}
if (c.frameCount < top.FrameCount) {
c = c + ImageSource("black.png", end=0).ConvertToYV12().Loop(top.FrameCount-c.FrameCount)
# assumes black.png is 1920x476
}
""")
StackVertical(top, c).DelayAudio(audio_delay_sec)
It assumes each image is shown for exactly 294 frames.
Modifying it to showing some for 294 and others for 293 is left as an exercise. :)
It would involve keeping track of the time and frames 'used so far' on each iteration of the loop, and setting n each time to
either 293 or 294 as appropriate.

[a1s2d3f4]

Man, you are, like, a genius. Thanks!

I'll try to figure out how you are doing what you are doing with your code so I can tweak the frames per image change value.

I have a feeling that I'll be using GScript a lot more for my next "enhancement" to this project (which I should describe in a different thread).

Amazing.

[a1s2d3f4]

I did a very simple hack so far (I can be very dull at times to figure out a real solution):

GScript("""
for (i=1, nImages-1) {
possible_offset = 1
if (i%10==0) {
possible_offset = 2
}
c = c + i1.Trim(i, -1).Loop(n)+i2.Trim(i, -1).Loop(n)+i3.Trim(i, -1).Loop(n)+i4.Trim(i, -1).Loop(n-possible_offset)
}
if (c.frameCount < top.FrameCount) {
c = c + ImageSource("black.png", end=0).ConvertToYV12().Loop(top.FrameCount-c.FrameCount)
# assumes black.png is 1920x476
}
""")


I need to do better, since this will drift out of sync gradually.

[Gavino]

Yes, if you work it out, then without any compensation, the drift is about 0.3 (0.294) frames per section. Each iteration of the loop does 4 sections, so drift is 1.2 frames per iteration. Hence in 10 iterations, it's 12 frames. Your mod corrects by 11 in total (9x1 + 1x2), so all you need is to add an extra case where possible_offset=2.
possible_offset = ((i%10==0 || i%10==5) ? 2 : 1)
Of course, it would be simpler if you could run at exactly 60fps. :)

Edit: Actually, thinking again, a simpler (and exact) solution is to use ChangeFPS to remove the unwanted frames after the construction, ie add the following before the final StackVertical:
c = c.AssumeFPS(60).ChangeFPS(60000, 1001)
Then you can use the original loop code without any modification.

[a1s2d3f4]

As before, perfect. Thank you so much.

Actually, my other part to this was to figure out a transparent yellow narrow vertical rectangle (height of 476, width 30) that moves across the imported image, to show exactly with it what part of the image corresponds to the video at each point in time. That is, it moves across each image in exactly 4.9 seconds, its left edge starting at X=50, and ending the run with X=1890, moving by about (1890-50)/293.7 = ~6.264 pixels each frame, or else, ca 25 pixels every 4 frames, then resets to the beginning and does it all over for the new image).
I thought of posting that as a separate post, initially, but the problem will be so tied in to this code on this thread that I don't know how to post it. (I just need to learn about some kind of "overlay" code).

Do you think it's ok to add it on here, even though it wasn't in the OP?

[Gavino]

I think it's fine to continue on this thread, since, as you say, the solution needs to build on the earlier code and it's really an extension of the original problem.

I'm sure it can be done, but I need to think a bit longer about the most efficient way to do it.

[Gavino]

OK, if I've understood your requirements correctly, this should do what you want.
It replaces all the code starting from the GScript call in the earlier code.
Requires MaskTools v2.
GScript("""
for (i=1, nImages-1) {
c = c + i1.Trim(i, -1).Loop(n)+i2.Trim(i, -1).Loop(n)+i3.Trim(i, -1).Loop(n)+i4.Trim(i, -1).Loop(n)
}
strip = BlankClip(c, length=n, color=color_yellow)
mask = Animate(strip, 0, n-1, "MakeMask", 50, 1890)
opacity = 0.5 # adapt to suit
c = c.Overlay(strip.Loop(4*nImages), mask=mask.Loop(4*nImages), opacity=opacity)
if (c.frameCount < top.FrameCount) {
c = c + ImageSource("black.png", end=0).ConvertToYV12().Loop(top.FrameCount-c.FrameCount)
# assumes black.png is 1920x476
}
""")
c = c.AssumeFPS(60).ChangeFPS(60000, 1001)
return StackVertical(top, c).DelayAudio(audio_delay_sec)

function MakeMask(clip base, int x) {
# create a mask for base that is opaque between x and x+30
sx = string(x) sx2 = string(x+30)
base.mt_lutspa(expr="x "+sx+" < 0 x "+sx2+" > 0 255 ? ?", relative=false)
}

[a1s2d3f4]

Truly, if I ever start video processing business, you are hired. So perfect.

Works exactly like I wanted it. For anyone else's benefit, download masktools from here: http://manao4.free.fr/masktools-v2.0a46.zip


Of course, at this point I really start to see the CPU cycles count. If before, I was able to watch the project in real time in VirtualDub by "decimating frame rate by 2", with this yellow bar I can't even get it to playback smoothly if I decimate by 25!

I guess to overlay images is not a trivial matter.
Looking through this code, I don't see how overlaying an actual yellow bmp, instead, would help. Seems like you have the most optimal solution.

NB:
This project could be used for anything that uses a video of an event, and some concurrent sensor data from something relevant. As long as you have a way of converting your data to an array of images display the data as some sort of graph/diagram, with a constant time on the X axis, you can use this template to quickly display all of this together, in sync.

This is definitely not for this thread, but I was actually wondering if there exists a plugin for AviSynth/VirtualDub, where you can basically click on the frame with the mouse, and it sends some kind of control (think textedit protocol and PDF files), which can be used to sync back to the original data....endless possibilities.



Again, thanks!

[Gavino]

I guess to overlay images is not a trivial matter.
Looking through this code, I don't see how overlaying an actual yellow bmp, instead, would help. Seems like you have the most optimal solution.
I can see two possible (and complementary) ways to speed it up:
1) Precreate the mask clip in a separate script - it's only 294 frames long, but unless you have enough memory for all of it to fit in Avisynth's cache, its frames are constantly being recreated for each section (each time calling function MakeMask with a different value of x for each frame).
2) Use mt_merge instead of Overlay, which is known to be fairly slow. You would need to incorporate your chosen opacity factor directly into the mask (replace 255 by the value of 255*opacity in function MakeMask).

Incidentally, I have now realised it's possible to do the construction of the images clip without using GScript at all, using this code:
...
top = StackHorizontal(video_stream_a, video_stream_b)
c = Interleave(i1, i2, i3, i4).AssumeFPS(1).ChangeFPS(n)
strip = BlankClip(c, length=n, color=color_yellow)
mask = Animate(strip, 0, n-1, "MakeMask", 50, 1890)
opacity = 0.5 # adapt to suit
c = c.Overlay(strip.Loop(4*nImages), mask=mask.Loop(4*nImages), opacity=opacity)
c = c.frameCount < top.FrameCount ?
\ c + ImageSource("black.png", end=0).ConvertToYV12().Loop(top.FrameCount-c.FrameCount) : c
# assumes black.png is 1920x476
c = c.AssumeFPS(60).ChangeFPS(60000, 1001)
...
I don't think this would speed it up much, but it's a simpler and more elegant way to do the same thing. GScript's for-loop is very powerful, but if you find yourself explicitly looping over all the frames of a clip, the work can often be done by choosing a suitable combination of basic Avisynth functions instead.

[a1s2d3f4]

Interesting - no more GScript!

Well, I replaced the relevant part of the code with the above, and here is the error I got:

---------------------------
VirtualDub Error
---------------------------
Avisynth open failure:
Splice: Video framerate doesn't match
(my.avs, line 129)
---------------------------
OK
---------------------------

Line 130 is your

c = c.frameCount < top.FrameCount ?
\ c + ImageSource("black.png", end=0).ConvertToYV12().Loop(top.FrameCount-c.FrameCount) : c


I guess this makes sense, since first you set frame rate to 294fps with ChangeFPS(n) and then you try to concatenate a clip where fps is 24 (by default ImageSource fps==24). So, I basically, added fps=n to ImageSource("black.png", end=0)


c = c.frameCount < top.FrameCount ?
\ c + ImageSource("black.png", end=0, fps=n).ConvertToYV12().Loop(top.FrameCount-c.FrameCount) : c



Works as it should now, but you are right, if it sped it up a bit, I am not feeling it :)

I'll look through your 2 suggestions - looks like they should help.

Thanks.

[Gavino]

I guess this makes sense, since first you set frame rate to 294fps with ChangeFPS(n) and then you try to concatenate a clip where fps is 24 (by default ImageSource fps==24). So, I basically, added fps=n to ImageSource("black.png", end=0)
Yes, that's the right solution.
Sorry I didn't spot the need for that.