fauxD: 2D to stereo-3D conversion in real-time

[eslave]

Yes, that's right, fauxD will pulfrich your videos into pseudo-stereo-3D in real-time (at most SD and sub/near-SD resolutions). I know there are sophisticated stereo-3D buffs out there just waiting to flame me for offering this, but I think fauxD is far too entertaining to keep to myself any longer.

The sense of 'presence' offered by this technique can be quite impressive under the right conditions. Imagery in the frame needs to be moving horizontally for the best effect (just a few pixels can often be enough). Rapid camera and subject motions will defeat this technique, and so will a stationary camera staring at an unchanging scene.

By default fauxD will produce red-cyan anaglyphs from your videos, but by changing one argument this can easily support other stereo frame-types; such as Over-Under, Side-By-Side, or Row-Interlaced (for shutter-glasses). See the documentation or script comments for more details.

So dust-off those old paper or LCD glasses and enjoy something in 3D!

-eslave

I finally got around to gathering some samples (red-cyan anaglyph only). Admittedly most are 'best-case' images. If they look a little washed-out its because I accidentally left the brightness Tweak() on.




I've been asked if this works with 3D games or in-game videos. This will work with the videos, but only when imagery is moving laterally. If you want to actually play or record your 3D games in stereo-3D then check www.iz3d.com/t-dcdriver.aspx and use FRAPS to record it.

If you need something for still images I've had decent results with 'redblue3D' at sglx.web.ur.ru/converter/ and funny results with make3d.stanford.edu. Bear in mind its been years since I've tried 3DCombine and that I haven't tested every still image conversion program.

v0.12 re-posted 1/15/2009 This time with the correct source-code. That's what I get for rushing.

[setarip_old]

Hi!

Naive question perhaps but, can you convert this to a VirtualDub filter?

[eslave]

setarip_old,

I'm certain a vdub port of fauxD itself is possible, and it should be relatively trivial, but its not on my to-do list this week. I'll look into it more, but I've never explored the vdub API - so don't hold your breath. (I'm not much for developing UI elements either.)

The only foreseeable stumbling block is fauxD's dependency on MVTools. I would need to get motion data from a depan logfile, or by accessing AviSynth+MVTools programmatically, or some other means of informing fauxD as to which way the camera is panning for each frame.

In the meantime fauxD's source is available, and from what I recall vdub/mod will accept AVS scripts, correct? (Maybe you need ffdshow? I just dragged&dropped fauxD.avs onto vdubmod and it opened for me.) Of course, my current plugin doesn't always like random (re-)seeks, being rapidly re-opened, and other things one might be prone to do while editing within vdub.

I'll play with the idea for a few days and see what I can do.

-eslave

Update: I think I'm going to table this idea for a while. FauxD is not stand-alone, nor is it mature enough to be hacked into a vdub filter at this point. Even if I did force it into the form of a vdub filter it would probably crash your editor every 5 minutes - hardly what one would consider a 'contribution to the community'. Similarly, there would be very little point in creating an AviSynth front-end filter for vdub, as vdubmod already supports AVS scripts. I'm not giving up on this idea, its just going to take a different approach than I had originally thought.

[setarip_old]

I'll play with the idea for a few days and see what I can do.

Thank you ;>}

[rfmmars]

Indeed I am very interested in your work, can't wait to play with it.

Richard

photorecall.net

[leeperry]

looks great, thanks!

but how can I use it in ffdshow ?
I've renamed fauxD.avs to .avsi and I've tried fauxD() but ffdshow won't load at all(I've got the latest MVtools and I've copied the content of the archive in the /plugins/ dir)

[eslave]

leeperry,

Please read the html document under "Script Usage"; look for "Input method". You should not need to rename or move any files to use fauxD within ffdshow.

Just un/comment one or two lines in the PROLOGUE section (see the "Method #X" comments), then paste 2 lines into ffdshow's avisynth config dialog (and enable it's checkbox):

import("d:\somepath\fauxD.AVS")  #extension doesn't really matter
return last  #(optional) just in case

After making these changes every file I open in MPC gets processed through fauxD (and anything else enabled in ffdshow's filter chain). Did you install ffdshow with avisynth support? I'm not sure if every build includes that capability, and I think the installer asks you as well.

If you really want both methods to be available at all times (without re-editing) you could make two copies of fauxD.avs - one for ffdshow and another for Drag&Drop/SendTo.

Be careful not to open fauxD.AVS with an ffdshow enabled application when ffdshow's internal avisynth is configured for fauxD. The application would attempt to fauxD() your content twice and either crash, throw an error, or run incredibly slowly.

Let me know if you still have difficulties.

edit1: Sorry, I've just re-read this post (with the benefit of sleep) and realized I should have consulted my reference first as the doc doesn't offer any real info in its present state. I'll need to update it soon.

edit2: New documentation posted 1/4/2009.

-eslave

[leeperry]

so what I've done so far :

-fix the "D:\av\fauxD\" paths in fauxD.avs to my own path
-comment "Import("fname.txt")"
-uncomment "source=last"
-I've got mvtools2.dll v2.3.1.0

I get an error msg that "MVSuper" is unknown(line 74 in fauxD.avs)...so I've taken it out as you said...but then it says "MVAnalyze" is unknown

[mikeytown2]

leeperry, you have mvtools 2 correct?
http://avisynth.org.ru/mvtools/mvtools2.html

edit...
remove the V from MVSuper so its MSuper. Do the same for the rest of the MVTool function calls.

[eslave]

Ok, I've revamped the documentation and made it as clear as I possibly can with respect to the input methods. (There is a picture, too.)

Leeperry, if you're still getting errors with the names of MVTools functions (MVSuper, MVAnalyse) then I think you may have a bad filename or incomplete path specified somewhere, preventing the plugin from loading. To the best of my knowledge those are the correct function names. (I'm looking at the MVToolsV2 doc right now.) I rename my copies of the MVTools DLLs with their version number so I can keep track of what I'm doing. Please double-check the paths and filenames used on the LoadPlugin and Import lines of your script.

You may also want to hit CTRL-ALT-DEL and check the 'Processes' list (show all) to see if any stalled instances of your video player are still 'running'. If so kill all instances of the player and wait a few seconds before trying again. I have seen numerous flashes of the fauxD vector dialog occur, sometimes for a up to a minute, after killing stalled player instances.

To clarify the "input method" issue I should say your input can come from anywhere, you just need to get it into a clip variable named source near the top of the PROLOGUE section. The rest of the fauxD script (whether 'opened' or 'imported') will then do all the work and return the final clip. You should not need to create or modify any explicit calls to fauxD() yourself in the script's normal 'configuration' (as released) for it work properly. Conceivably issues with color-spaces and pixel formats with respect to ffdshow may arise, but I rather doubt that's the case unless you went randomly checking boxes in ffdshow at some point.

Let me know how it goes.

-eslave

[rfmmars]

Even with the last update, it still wouldn't work. I had to change all of the MVtools call to Mtool~. Still didn't see any results, but exported into Vdub, all is well. below is my simplified AVS so people can get started.
http://img136.imageshack.us/my.php?i...8004728bk5.jpg

A dialog box appears but with no information????? EDIT: I must be going blind, I could have sworn there was no data when I did the screen grab.

Code
...........................................

source = DirectShowSource("c:\3D-test.avi")

LoadPlugin("c:\AVS\Plugins\AGC.dll")
LoadPlugin("c:\AVS\plugins\removegrain.dll")
LoadPlugin("c:\AVS\Plugins\mt_masktools.dll")
LoadPlugin("c:\AVS\Plugins\warpsharp.dll")
Loadplugin("c:\avs\plugins\gradfun2db.dll")
loadplugin("c:\avs\plugins\DeGrainMedian.dll")
loadplugin("c:\avs\plugins\Smoothdeinterlacer.dll")
LoadPlugin("c:\avs\plugins\Convolution3d.dll")
LoadPlugin("c:\avs\plugins\FFT3DFilter.dll")
LoadPlugin("c:\avs\plugins\VagueDenoiser.dll")
LoadPlugin("c:\avs\plugins\depan.dll")
LoadCPlugin("C:\avs\plugins\AVSCurveFlow.dll")
LoadPlugin("C:\avs\plugins\MVTools2210.dll")
LoadPlugin("C:\avs\plugins\fauxD.dll")

Import("C:\avs\fauxD\fauxD_util.avsi")

####################################################################

scth = 40 #scene change threshold (on a scale of 255, I believe) lower=less tolerance
use_plugin = true #use fauxD function from plugin or AVS
stereo_mode = 0 # 0 = Red-Cyan Anaglyph, 1 = Over-Under (Right Above, Left Below), 2 = Side-By-Side (Left, Right), 3 = shutterglasses (pure AVS, Left=Field1)

flat = source.ConvertToYV12





#----- PREP WORK (Analysis) ---------------------------------

analyze = (source.height%2==0) ? source : source.ConvertToRGB32.CropBottom(1) #adjust to make height even
analyze = (analyze.height%2==0 && !(analyze.height%4==0) && !(analyze.height%8==0)) ? analyze.CropBottom(2) : analyze #try to make height mod4/8
analyze = (analyze.height%2==0 && !(analyze.height%4==0) && !(analyze.height%8==0)) ? analyze.CropBottom(2) : analyze #again if needed to make height mod4/8
analyze = (analyze.height%8==0) ? analyze.verticalreduceby2.verticalreduceby2 : analyze #quarter height
analyze = (analyze.height%4==0) ? analyze.verticalreduceby2 : analyze #half height

analyze = analyze.ConvertToYV12 # SHOULD be YV12 at this point for MVAnalyse


vectors = MSuper(analyze).MAnalyse( blksize = 16 , search = 5 , searchparam = 8 ) #BETTER SEARCH
global depan = MDepan(analyze, vectors) #convert motion vectors into DEPAN clip format (add zoom and rot options for complete vectors)

#----- PULFRICH (Interpret GM & fuse frame-pairs as stereo) ---------------------------------
framestep = 1
#TESTING previous frame (1 frame late)
global current = flat.KillAudio #current frame (now)
global previous = UnalignedSplice(BlankClip(current,framestep), current).KillAudio #previous frame (1 frame late)
global mvthresh = 0.0 #disabled - failed attempt at ignoring 'false-positive' motion
global reverse_stereo = false #for the sake of completeness, affects AVS functions only

global avs_mode = (stereo_mode == 1) ? "OverUnder" : (stereo_mode == 2) ? "SideBySide" : (stereo_mode == 3) ? "InterleavedShutter" : "AnaglyphRedCyan" # convert integer to string (function name) for fauxD/ScriptClip
use_plugin = (stereo_mode == 3) ? false : true #'native' shutterglasses are only supported through the AVS function
stereo = (use_plugin && (stereo_mode!=3)) ? fauxD(flat.converttoRGB32, depan, stereo_mode) : AudioDubEx( ScriptClip(flat, """ FauxD(GetPan(depan), avs_mode) """) , flat.KillVideo )





#----- EPILOGUE (Show me already!) ---------------------------------



flat = flat.ConvertToYV12 #for YDifferenceFromPrevious
stereo = stereo.MatchPixelFormat(flat)
out = ConditionalFilter(flat, flat, stereo, "YDifferenceFromPrevious()", ">", String(scth), false ) #faster than RGB?

# do any FINAL post-processing here
out = (stereo_mode == 3) ? out.Tweak(Bright=30) : out #brighten final output (for shutterglasses) AUTO

out = out.ConvertToRGB24 #for TMPGEnc, VFApi, or any other application limited to 24-bit RGB (8-bits per color/channel)
return out #the END (if everything worked)


#----- SANITY CHECK (Yes, I know this looks stupid.) ---------------------------------

return MessageClip("fauxD Error: Sanity Check!", 320, 240, shrink=false, text_color=$FF0000)

...............................................................................

Richard

photorecall.net

[eslave]

At the risk of repeating myself, please double-check your filenames and paths. I see you still have "MVTools2210.dll" on your LoadPlugin line. Unless you've renamed your file to match then AviSynth will not load the plugin. However, if you're seeing vector data in fauxD's dialog then things should be functioning properly.

I'll admit that I've never really tried using fauxD.AVS together with other scripts (such as LSF/SeeSaw). There may be unforeseen issues with all the global/ScriptClip stuff (its still relatively new to me). I had considered stream-lining the main script for this release and leaving a heavily commented copy for testers, but I felt that would only add to the confusion.

I'll take a swack at trying fauxD in Vdubmod later today and see what happens.

I apologize for all the unnecessary configuration problems. My original intent was to make it easier for casual testers to modify, but the results appear to be quite different from my expectations.
I'll bet you can't tell this is my first release!

-eslave

[rfmmars]

Originally Posted by eslave

At the risk of repeating myself, please double-check your filenames and paths. I see you still have "MVTools2210.dll" on your LoadPlugin line. Unless you've renamed your file to match then AviSynth will not load the plugin. However, if you're seeing vector data in fauxD's dialog then things should be functioning properly.

I'll admit that I've never really tried using fauxD.AVS together with other scripts (such as LSF/SeeSaw). There may be unforeseen issues with all the global/ScriptClip stuff (its still relatively new to me). I had considered stream-lining the main script for this release and leaving a heavily commented copy for testers, but I felt that would only add to the confusion.

I'll take a swack at trying fauxD in Vdubmod later today and see what happens.

I apologize for all the unnecessary configuration problems. My original intent was to make it easier for casual testers to modify, but the results appear to be quite different from my expectations.
I'll bet you can't tell this is my first release!

-eslave

What I did was rename a copy of my MVtools.dll "Version 2.31 to match your "loadplugins script" , I could not find the version you had listed.

I am very happy with the first results, just wanted to make it as easy for others to get started.

Many thanks,

Richard

[leeperry]

ok cool, something that works with a simple call in ffd would be really awesome

[Fizick]

sorry for function names confusing

[eslave]

Originally Posted by Fizick

sorry for function names confusing

Don't worry about that, its my own fault for living in the past. I really should have taken a closer look at 2.31 before my very first post. I still don't know what my 'hang-on-exit' issue was, but it was most assuredly my fault as well. (Obviously I never called any MVtools functions.)

Unless something goes wrong I'll have a new version of fauxD posted later this evening. (Compatible with the current version of MVTools, 2.31) I still need to do some verification testing and fix the docs.

-eslave

[setarip_old]

@eslave

May I ask if you've made any progress regarding converting this to a VirtualDub filter?

[morsa]

I found one problem in this implementation of the Pulfrich Effect

http://en.wikipedia.org/wiki/Pulfrich_effect

The effect should only be applied on lateral motions, and from what I saw, this script/plugin doesn't take that into account.Vertical panning/motion also has the effect on it which is not a good thing.
Is there any possibility of using just displacements on the X axis?

[eslave]

Well, that took far more testing and head-scratching than I had anticipated. Now with 16 different ways to call fauxD (together with its AVS doppleganger) its very easy to overlook something. I've added look-ahead for scene-changes, an 'always-on-top' checkbox, and fixed the 'flat-on-seek' issue rfmmars pointed-out. I've also removed all the 'V's from the MVTools function names for compatibility with Fizick's current version. (Thanks leeperry, mikeytown2, and rfmmars for repeatedly trying to point that out to me.)

Look-ahead will produce 2 duplicate stereo frames in sequence. My alternative was to use reverse stereo on the first frame, thereby furnishing a sense that 'every frame is different', but frequent false-positives caused eye-strain.

@morsa
Speaking of eye-strain, which I make little mention of in the doc, I know pulfrich should only be applied to purely horizontal video (with the camera moving at a prescribed rate), but I am not in control of the camera motions. True, I make no exceptions for vertical motion, I only care about pan direction ('displacements on the X axis'). Yes, this is an oversimplifcation with respect to pulfrich, but its the best I can do in realtime. I simply fuse 2 adjacent frames as stereo depending on pan direction. I wrote the ZeroDepan function hoping to apply some stability (vertical, rotation, zoom) to the source clip while preserving horizontal motion, but I haven't made any real effort to employ such a technique yet. I refer to scenes with strong vertical motion, or anything else that causes severe eye-strain, as "pulfrich-unfriendly". If you really want to see some depth in a video with only vertical motion then turn the video 90 degrees before processing. (Of course you'll need to watch the final output at the same orientation.)

@setarip_old
I've updated my earlier post. I still don't have a time-frame for a vdub port at the moment. Sorry.

@Shinobu
Oh my! Please don't get your expectations too high. This is purely a 'trick' as far as 3D-freaks are concerned. Also, if you happen to have 2 projectors maybe you can try polarized glasses.


I really appreciate all the help and testing that all of you have done, but I'm nearing the end of this current 'fit of creativity'. I'm not stopping development, but I want to actually enjoy fauxD for a little while before breaking it again. Also, my new toy has just arrived and I'll probably be playing with that for at least a few days. So don't be concerned if I suddenly get a little 'quiet' around here. However, if I do notice that something serious is broken I'll try to correct it asap.

-eslave

[Shinobu]

i don't expect to high, i'm just interested in testing this "trick".
as for the 2 projectors, i think i'll get a 3-dlp 1080p (when i'll have the money) instead of buying another 720p just to test 3D ^^.

by the way, i have an old 22" crt display, a 24" lcd and the 1-dlp projector, so i'll tell you witch one is the best renderer for your plugin (i'm going to test it with film and animation bdr).