Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > Avisynth Usage

Reply
 
Thread Tools Search this Thread Display Modes
Old 23rd March 2016, 19:56   #1  |  Link
Seedmanc
Registered User
 
Join Date: Sep 2010
Location: Russia
Posts: 47
Transposing a 2D videostream into a 3D object (frame count as a depth)

I got a strange idea once, what if you could represent your average videostream that has width, height and duration as a box with the same dimensions, but having its z-axis along the video length? Such a "box" would have all video frames inside of it. Imagine that when you're watching a video, you're looking at this box from its front side, going deeper into it as you're progressing along the time axis.


With me so far?

Now, what if we rotated the box 90 degrees around its width or height axis? The video length would become the new video height or width respectively and vice-versa. You would be looking at a collection of every frame's side pixel columns, and watching the video would move you along the original video frame pixel lines (sorry if my wording is confusing).


In order to implement this I wrote the following script (Avisynth+)

Code:
AVISource(“C:\source.avi”) 
converttorgb32

original = last 
last.blankclip(length = original.width, width = original.framecount, height = original.height) 

scriptclip(""" 
  
  for ( i= 0, original.framecount - 1) { 
    selected = original.freezeframe(0, original.framecount - 1, i) 
    layer(selected.crop(original.width - 1 - current_frame, 0, 1, 0), x = i) 
  }  

""")
(this one transposes video length to width).

You can see how the video looks like after such transformation here: https://www.youtube.com/watch?v=Ut00lonRqVk (left to right is beginning to end of the original)
Here's also the same method used to transpose length to height: https://www.youtube.com/watch?v=-K5b2BKI0ew
Compare to the original video: https://www.youtube.com/watch?v=-YKuxZgDamM (mine was trimmed to speed up processing).

The catch is though that despite the simplicity of the script it runs extremely slowly (originally at 0.2 fps for a 672x384 video, later sped up to ~1.5-3 fps by replacing overlay with layer and switching from Avisynth to AVS+). Speed is also inversely proportional to the original video length, because in order to construct one frame of the output the script has to access every frame of the input.

Interesting thing is, this conversion is reversible, you can get the original video back by applying the transformation again (although it will be reversed and mirrored, but those filters are lossless as well).
This opens such possibilities as applying a spatial only filter to the temporal domain and vice-versa. I already tried upsampling FPS with a neural network upscaler as well as upscaling with MFlowFPS. Results weren't that much impressive, although in the first case it was curious to notice that parts of the frame were moving independently of each other.

Another point to make is how switching dimensions, temporal with spatial, lets you see the whole timeline of the video at once, even though only for a column of pixels at the moment. To remove this limitation we need to add up a dimension instead of replacing one with another - to go 3D.

I took the already transposed width-to-length video and processed it with another script, overlaying each frame with a diagonal shift to achieve 3D looks.
Code:
AviSource("D:\width-transposed.avi").converttorgb32.trim(0, 639)

a = last
a.blankclip(source_framecount, a.width + 640, a.height + 640/2, "RGB32", color=$888888)
scriptclip("""
        for (i = 0, 639) {
            frozen = a .freezeframe(0, a.framecount - 1, i).crop(current_frame, 0, 0, 0)
            layer(frozen , x = i, y = i / 2,  level = round(256.0 * (640 - i) / 640)) 
        }
 """)
Now the video actually looks like a 3D box, with the original playing on its left side, while you can see all the frames at once: https://www.youtube.com/watch?v=2vUihdaMbwk
Well, could, you can't see through an object, so I added some transparency to its front edge.

Why am I writing all this? To share my experience and discoveries made thanks to Avisynth and to give my insight on the concept of observing N-dimensional objects from a N+1 dimensional space. Remember 4D tesseract in Interstellar, seeing 3D space in all timeframes at once? Here we see all instances of a 2D video by going 3D.

Since this is more of a "share thoughts" topic than a "get help" one I'm not asking anything by it, but perhaps you can share some cool non-conventional things you did with video by Avisynth too, or maybe give ideas on speeding up such transformations.

Last edited by Seedmanc; 23rd March 2016 at 20:01.
Seedmanc is offline   Reply With Quote
Old 24th March 2016, 11:48   #2  |  Link
Sparktank
47.952fps@71.928Hz
 
Sparktank's Avatar
 
Join Date: Mar 2011
Location: Planet Express, Inc.
Posts: 788
i knew looking in this thread would keep me awake.

you got me with the right keywords: z-axis, transpose, tesseract, Interstellar, 3D, the!

my little folder of art grows every year.

Really great stuff to play with.

Quote:
Originally Posted by Seedmanc View Post
perhaps you can share some cool non-conventional things you did with video by Avisynth too
Well, not by me, but I did ask so things can get rolling and a lot of fun stuff did happen.
Usually it starts out with someone doing something on youtube where other software was used first.

Your tesseract project reminds me of a version of the "video fingerprint" I wanted to replicate.
http://forum.doom9.org/showthread.php?t=167769
I need to really sit down one day and sort out examples and update the first post with the different scripts and plugins, coplete with examples.

Some weird video thing I forgot I asked about. :lol:
http://forum.doom9.org/showthread.php?t=172475
Layering with only maxY.

There's still plenty of other things out here, I have subscribed to.
Some fun with Spline100 mod, iirc.

I am definitely going to have to spend some time with this over teh weekend.
__________________
Win10 (x64) | GPU Caps Viewer v1.32.0.0
Crucial M500 240GB SSD | Kingston SSDNow V300 (Marvell) 120GB | NVIDIA GeForce GTX 750 Ti | R375.95 (Nov 18, 2016)
NTSC | DVD: R1 | BD: A
Sparktank is offline   Reply With Quote
Old 25th March 2016, 04:13   #3  |  Link
junh1024
Registered User
 
Join Date: Mar 2011
Posts: 16
If you want to increase the FPS, install SVP for avs, use

Quote:
super=SVSuper("{gpu:0}")
vectors=SVAnalyse(super, "{}")
SVSmoothFps(super, vectors, "{ algo: 23 }", url="www.svp-team.com", mt=2)
and be done with it. Better than Interframe, Mflow, and that russian one.
junh1024 is offline   Reply With Quote
Old 25th March 2016, 04:52   #4  |  Link
Sparktank
47.952fps@71.928Hz
 
Sparktank's Avatar
 
Join Date: Mar 2011
Location: Planet Express, Inc.
Posts: 788
Yeah, I thought about adding SVp to it. I really like how they put the script together for you.
You can customize with InterFrame, same libraries. But you have to know how first.

Right now, I'm looking at converting a scene from Interstellar to a bunch of PNG's.
Crop the borders for scenes in 1920x804 (2.388:1) WMOD=32 HMOD=4.
Then add a transparent/alpha border to those frames.
Keep the IMAX scenes intact.
Run it through something like ffmpeg to concat all the images, maintain alpha transparency, and make a lossless RGB32 intermediate file to run through this.

It would be nice to see the side view of the Interstellar clip. Showing absolute changes in height for different aspect ratios present in the bluray.
Give an excellent illustration. But the only way I can see that working is to show transparency for the non-IMAX frames instead of black or colored bars.
The black bars would look like it's just part of a very dark IMAX scene, as there are a lot of dark frames/scenes.

I could pick a flashy magenta or neon green that isn't present anywhere in the film as a last resort.

I managed to get ImageMagick to create transparent borders.

Code:
# South, image moves south to create transparent border for TOP=138
"C:\Apps\Image\ImageMagick\convert.exe" "Interstellar RGB32 [278].png" -background none -gravity south -extent 1920x942
\ "Alpha-south [278].png"
Borders should be:
a) gone
b) black or white or grey (depending "what" is viewing it.)
Saving it and looking at it differently will show you the borders indeed transparent.

In XnView, I made the backgrounds all green.
Then when I select/highlight the pictures, I can definitely confirm transparent borders.
la la:


So somehow...
take all the cropped with alpha borders....mix with full framed IMAX...turn it on its side and make a pretty wallpaper...
or something to that effect.

It would actually be great for making custom soundtrack.
Take clips from the bluray, work it up like "tesseract" as a video to match time length of a track from the OST and then start listening on MPC-HC instead of Foobar2000.
This would be a hit at any gatherings or parties. :laugh: If anyone at them listened to soundtracks, that is.
__________________
Win10 (x64) | GPU Caps Viewer v1.32.0.0
Crucial M500 240GB SSD | Kingston SSDNow V300 (Marvell) 120GB | NVIDIA GeForce GTX 750 Ti | R375.95 (Nov 18, 2016)
NTSC | DVD: R1 | BD: A

Last edited by Sparktank; 25th March 2016 at 04:56.
Sparktank is offline   Reply With Quote
Old 25th March 2016, 19:08   #5  |  Link
Seedmanc
Registered User
 
Join Date: Sep 2010
Location: Russia
Posts: 47
Sparktank, yeah I remember that thread about video footprint, I recall I was trying to do a similar thing in runtime, but it ground to a halt as it progressed further into video, haha. As for the lighter color thing with SW, does it have any practical uses, or it's just an effect? I reckon, you can't recover two streams once you mixed them this way, or can you? I wonder if there was a way to mux several video streams into one without increasing size or framecount, with the ability to demux them after compression (perhaps lossily). Increasing density, so to say.
nh1024, I need to increase speed of my scripts, this has nothing to do with frame interpolation. If only there was a way to make Avisynth store entire videostream in memory, perhaps that would have made it faster.

Speaking about colors and dimensions, I remember doing sort of image recognition through combination of masks. I was able to detect eyes on the picture with a good enough success rate.
In short, I profiled the eye image, then limited first in the brightness domain with mt_binarize, then in the frequency one with mt_edge and finally in colors with MaskHS.
Combined masks provided decent detection rate:

if not for some false positives. Still I think it was pretty cool for just a bunch of masks in Avisynth, given how you would usually require neural networks for image recognition.
Full article with scripts, if you're interested.
Seedmanc is offline   Reply With Quote
Old 25th March 2016, 23:15   #6  |  Link
raffriff42
Retried Guesser
 
raffriff42's Avatar
 
Join Date: Jun 2012
Posts: 1,251
Stuff like the Eadweard Muybridge motion studies might look cool in 4D, if you could see through the frames to get the "human centipede" effect.
raffriff42 is offline   Reply With Quote
Old 25th March 2016, 23:53   #7  |  Link
Sparktank
47.952fps@71.928Hz
 
Sparktank's Avatar
 
Join Date: Mar 2011
Location: Planet Express, Inc.
Posts: 788
Quote:
Originally Posted by Seedmanc View Post
or it's just an effect?
That's entirely for effect. If you encode it like that, it's all pretty much what you see.

A friend of mine started streaming to twitch her gameplays. So at one point I told her to stare at something for 5 seconds without moving. Haha. I just need to dig up a war movie now and plaster the two together.

I just need to catch a few streams of her doing insane things before making the video.
__________________
Win10 (x64) | GPU Caps Viewer v1.32.0.0
Crucial M500 240GB SSD | Kingston SSDNow V300 (Marvell) 120GB | NVIDIA GeForce GTX 750 Ti | R375.95 (Nov 18, 2016)
NTSC | DVD: R1 | BD: A
Sparktank is offline   Reply With Quote
Old 3rd April 2016, 10:57   #8  |  Link
Seedmanc
Registered User
 
Join Date: Sep 2010
Location: Russia
Posts: 47
The other time I decided to put a test to Avisynth's runtime capabilities and made a script that analyzed a gameplay recording, recognizing certain situations and generating a subtitle template with timestamps, character names and screenshots so that the translator could later just fill it with text. That was one massive script, however it ran at around 450 fps for 720p, bottleneck being the decoder I think. Script's still here http://pastebin.com/rtJi5cD9
Seedmanc is offline   Reply With Quote
Old 6th January 2017, 19:56   #9  |  Link
Seedmanc
Registered User
 
Join Date: Sep 2010
Location: Russia
Posts: 47
application in MRI imaging

I was messing around trying to improve my script that overlays all frames of a video to achieve 3D look, when I realized that this is basically the inverse of how MRI scanners work.

So I went to youtube, got myself an MRI sequence video and built a volumeric body model using its frames: (NSFW-ish image)

The script is as follows, you don't have to export the whole video to still images as an intermediate step anymore, although it would have improved the responsiveness.
Code:
dss2("E:\01 MRI Scan Male Body (480p_30fps_H264-128kbit_AAC).mp4").trim(0,1500)
  
converttorgb32
last.blankclip(height=round(last.height*1.9)).overlay(last)
 turnright
skewrows(1)             #rotate the model to achieve isometric 3D look
 turnleft
crop(0,height-width,0,0)
flipvertical 
 

m= converttoyv12.mt_binarize(76)  # build a mask to remove the background
m=mt_logic(m,converttoyv24.maskhs(170,100,coring=false) .converttoyv12 .mt_inpand ,mode="min").converttoy8
mask(last, m.converttorgb32) # put it to the alpha channel
 
original=last 

last.BlankClip(width=last.framecount-1+last.width, color=$888888)

scriptclip("""
        for (i = original.framecount-1, current_frame, -1) {
            frozen = original.freezeframe(0, original.framecount - 1, i)   
            layer(frozen , x = i,level=128)               
        } 
 """)
crop(300,0,0,0)
By playing with the mask radius and layering opacity you can even see the underlaying tissues behind the skin, kinda like in that scene in Hollow Man.
Unfortunately it takes a good minute for the script to reload and it renders at a fraction of FPS, so the interactivity is rather limited. If only there was a way to massively accelerate this processing, it would have became a very useful tool.
Seedmanc is offline   Reply With Quote
Old 6th January 2017, 21:14   #10  |  Link
davidhorman
I'm the Doctor
 
Join Date: Jan 2004
Posts: 1,331
I'm not sure whether you'll really care to know, but that video wasn't made from an MRI scan. If you're currently eating and/or are squeamish, you may not wish to continue reading. You have been warned!

A donated cadaver was frozen in ice/gelatin, then images were taken. Between each image, a 1mm slice was removed from the ice. All those blue flashes are, I believe, cracks and other things in the ice.

It's called the Visible Human project.

I'd guess the problem with your script is that, each time, it's looping from frame 0 to (current_frame). I may have an experimental plugin that can greatly improve this.
__________________
My AviSynth filters / I'm the Doctor

Last edited by davidhorman; 6th January 2017 at 21:21.
davidhorman is offline   Reply With Quote
Old 6th January 2017, 21:22   #11  |  Link
Seedmanc
Registered User
 
Join Date: Sep 2010
Location: Russia
Posts: 47
Yeah right I kinda saw that in a description of a similar video but that eluded me. Should have taken notice however considering that the images were colored, something an MRI can't do.
Regardless, the idea is the same, as MRI can produce similar kind of images minus the coloring.
Seedmanc is offline   Reply With Quote
Old 6th January 2017, 21:30   #12  |  Link
davidhorman
I'm the Doctor
 
Join Date: Jan 2004
Posts: 1,331
I got a scriptclip error when I tried your script (line 2, column 60), but in any case I can't figure out where, in your clip, the frames are shifted along to make the "stack". Hint?
__________________
My AviSynth filters / I'm the Doctor
davidhorman is offline   Reply With Quote
Old 6th January 2017, 21:41   #13  |  Link
Seedmanc
Registered User
 
Join Date: Sep 2010
Location: Russia
Posts: 47
davidhorman, this is a script for Avisynth+, which has the "for" construction incorporated instead of being separate in GScript. You can probably replicate the functionality by installing GScript and putting it inside of ScriptClip.

As for stacking, that's what the For script is, well, for. For every requested output frame it accesses all frames of the source starting with the end of the source clip and up to the requested frame number. Then it overlays every frame with a shift equal to the iterator (the frame number in the original).

Thus the exceptionally low fps, since to construct a single frame it needs to process all frames. It gets faster close to the end though.
Seedmanc is offline   Reply With Quote
Old 7th January 2017, 00:08   #14  |  Link
Gavino
Avisynth language lover
 
Join Date: Dec 2007
Location: Spain
Posts: 3,377
Quote:
Originally Posted by Seedmanc View Post
Code:
...

original=last 

last.BlankClip(width=last.framecount-1+last.width, color=$888888)

scriptclip("""
        for (i = original.framecount-1, current_frame, -1) {
            frozen = original.freezeframe(0, original.framecount - 1, i)   
            layer(frozen , x = i,level=128)               
        } 
 """)
crop(300,0,0,0)
...
If only there was a way to massively accelerate this processing, it would have became a very useful tool.
A variant of the trick from the 2009 thread "Sticking" overlay might help.
The idea is to reuse each partial stack in the construction of the next rather than building each one from scratch.
For this to work, you will have to generate the clip frames in reverse order, so run the following script, save the result and either play it backwards (if possible) or run another reverse script to generate the final result.

For the script, replace your scriptclip section with the following code: (warning - untested!)
Code:
original = original.Reverse()
stack = last # the blank clip
ScriptClip("""
  frozen = original.freezeframe(0, original.framecount - 1, current_frame)   
  stack = layer(stack.SelectEvery(-1), frozen, x=current_frame, level=128)               
  return stack
""")
The performance gain comes from the expectation that at each frame, the previously generated frame (obtained via SelectEvery(-1)) will be preserved in the Avisynth cache.

The frames have to be generated in strict linear order, so it probably needs to be run single-threaded.
__________________
GScript and GRunT - complex Avisynth scripting made easier

Last edited by Gavino; 7th January 2017 at 00:11.
Gavino is offline   Reply With Quote
Old 7th January 2017, 01:44   #15  |  Link
davidhorman
I'm the Doctor
 
Join Date: Jan 2004
Posts: 1,331
Dammit, Gavino, ruining my big reveal...

...because the plugin I've been developing has just such a "sticking overlay" thing built-in. As with the above, it means the script as to be run strictly in frame order to give the correct result (although I might make this optional as I've now discovered it can lead to interesting results if misused in just the right way).

I'll post the script and the (beta) plugin if anyone's interested, but I'll leave it out for now because it's long and complicated, and so is the plugin, for which I haven't finished the documentation. I also have no idea if it's compatible with Avisynth+.

Here, though, is the result (source was https://www.youtube.com/watch?v=dPPjUtiAGYs)

https://youtu.be/ZkkUNFXaYyk

It runs at about 20fps on my computer (i7-2720QM, 2.20GHz Quad Core). He came out a bit green because I didn't bother inpanding the mask.
__________________
My AviSynth filters / I'm the Doctor

Last edited by davidhorman; 7th January 2017 at 02:17. Reason: replaced with a slightly cleaner version
davidhorman is offline   Reply With Quote
Old 7th January 2017, 05:00   #16  |  Link
poisondeathray
Registered User
 
Join Date: Sep 2007
Posts: 3,097
Quote:
Originally Posted by davidhorman View Post

...because the plugin I've been developing has just such a "sticking overlay" thing built-in. As with the above, it means the script as to be run strictly in frame order to give the correct result (although I might make this optional as I've now discovered it can lead to interesting results if misused in just the right way).

I'll post the script and the (beta) plugin if anyone's interested, but I'll leave it out for now because it's long and complicated, and so is the plugin, for which I haven't finished the documentation. I also have no idea if it's compatible with Avisynth+.

Here, though, is the result (source was https://www.youtube.com/watch?v=dPPjUtiAGYs)

https://youtu.be/ZkkUNFXaYyk

It runs at about 20fps on my computer (i7-2720QM, 2.20GHz Quad Core). He came out a bit green because I didn't bother inpanding the mask.

That's is pure awesomeness!!!!


I'd love if you posted the plugin
poisondeathray is online now   Reply With Quote
Old 7th January 2017, 06:08   #17  |  Link
Sparktank
47.952fps@71.928Hz
 
Sparktank's Avatar
 
Join Date: Mar 2011
Location: Planet Express, Inc.
Posts: 788
Quote:
Originally Posted by davidhorman View Post
I'll post the script and the (beta) plugin if anyone's interested,
Yes, yes, yes, please!
__________________
Win10 (x64) | GPU Caps Viewer v1.32.0.0
Crucial M500 240GB SSD | Kingston SSDNow V300 (Marvell) 120GB | NVIDIA GeForce GTX 750 Ti | R375.95 (Nov 18, 2016)
NTSC | DVD: R1 | BD: A
Sparktank is offline   Reply With Quote
Old 7th January 2017, 12:16   #18  |  Link
smok3
brontosaurusrex
 
smok3's Avatar
 
Join Date: Oct 2001
Posts: 2,375
Would that be bunch of 2d planes with pictures mapped to them? (I have some blender skilz)
smok3 is offline   Reply With Quote
Old 7th January 2017, 13:52   #19  |  Link
Seedmanc
Registered User
 
Join Date: Sep 2010
Location: Russia
Posts: 47
Gavino, this is most impressive. I've been thinking about reusing the previous frame, but I could never figure out how. The SelectEvery(-1) is some black magick, it doesn't even work on its own alone (crashes Avspmod), only in this script. Interesting that both in the editor and in Avsmeter the framerate is highly erratic and often hang up, but when actually encoding the result it's a smooth fast ride.

smok3 I wish I had experience in 3D editor. My actual goal after all these experiments in Avisynth is to be able to create a full 3D model out of all video frames to be able to rotate and look through it freely, unhindered by a video editor which is not even meant to work with 3D.
Seedmanc is offline   Reply With Quote
Old 7th January 2017, 14:02   #20  |  Link
StainlessS
HeartlessS Usurer
 
StainlessS's Avatar
 
Join Date: Dec 2009
Location: Over the rainbow
Posts: 5,157
Quote:
The SelectEvery(-1) is some black magick, it doesn't even work on its own alone (crashes Avspmod)
I think the SelectEvery(-1) is mistake, should be SelectEvery(1,-1)
Have not tried that script.

But this works as expected [with the (-1) I get 1 black frame].
Code:
Colorbars.KillAudio.ShowFrameNumber
SelectEvery(1,-1)
SSS="""
    subtitle(String(current_frame)+"]")
"""
ScriptClip(SSS)
return last
EDIT: From docs
Quote:
SelectEvery (clip, int step_size, int offset1 [, int offset2 [, ...]])
Both step_size and offset1 are non optional.

EDIT: This also works as expected
Code:
Colorbars.KillAudio.ShowFrameNumber
SSS="""
    SelectEvery(1,-1)
    subtitle(String(current_frame)+"]")
"""
ScriptClip(SSS)
return last
but without the 1,-1, fails in Scriptclip.
[Shows static ShowFrameNumber as 107891 on entire clip of 107892 frames, ie last frame of clip]
Is it supposed to do that ?, Maybe it is black magic

EDIT: Checked source args string, "SelectEvery " = "cii*"
Actually 1 compulsory clip, 1 compulsory int, and zero or more optional int, so maybe it is truly dark magic by the dark lord Gavino
EDIT: So the docs are strictly speaking incorrect compared to implementation.
__________________
I sometimes post sober.
StainlessS@MediaFire ::: AND/OR ::: StainlessS@SendSpace

"Some infinities are bigger than other infinities", but are any of them infinitely bigger ???

Last edited by StainlessS; 7th January 2017 at 14:29.
StainlessS is offline   Reply With Quote
Reply

Tags
avisynth+, gscript, runtime, transition

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 17:14.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2017, vBulletin Solutions Inc.