Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > Avisynth Development

Reply
 
Thread Tools Search this Thread Display Modes
Old 9th July 2011, 05:49   #1  |  Link
Youka
Registered User
 
Youka's Avatar
 
Join Date: Mar 2011
Location: Germany
Posts: 64
FLuaG - OpenGL rendering on video stream

After some time of karaoke effect making, tools like VSFilter (WinGDI) and Overlua (Cairo) weren't enough anymore, so the next step should be an OpenGL filter for Avisynth - and here it is.
FLuaG calls scripts in scripting language Lua to render with OpenGL on video frames or modify audio data.


Downloads:
Github

Usage:
Code:
FLuaG(clip c, string video = void, string audio = void, bool onscreen = false, bool samples32 = false, string userdata = void)
  • c: Clip with video and/or audio.
  • video: Lua script, extended with OpenGL and other utility functions, for working on video frames.
  • audio: Lua script for working on audio samples.
  • onscreen: OpenGL render context visible? (needed for graphic cards which don't support window offscreen rendering)
  • samples32: Samples with size of 32 bit instead of 16 bit?
  • userdata: AVS variable names for sending data to Lua.
Video presentation

Todo:
  • Lua functions
    • FFT
    • TGA loading & saving
    • Outline path utilities
  • Others
    • Examples: advanced karaoke and drawing
    • Documentation: extended + german version


Changelog:

Version 0.5 (11.12.2011):
  • Userdata (AVS variables to Lua)
  • Option for 32 bit samples
  • More utility functions (math and print path)
Version 0.4 (24.11.2011):
  • New process structure (similar to Avisynth Filter SDK)
  • OpenGL 2.1 support
  • Audio access
  • various fixes
Version 0.3 (13.07.2011):
  • Just one OpenGL context for the whole progress (frame-wise before)
  • flDrawPath's internal tesselation combine function call doesn't change color to black/alpha=0 if there're no alternative colors
Version 0.2 (09.07.2011):
  • Faster frame processing
  • Compiled for .NET 3.5
  • Function fixes: math.on_line, math.in_triangle, glReadPixels, flDrawPath
  • Color and pixel data from range 0-255 changed to 0-1
  • OpenGL 1.3

Last edited by Youka; 21st May 2012 at 12:17.
Youka is offline   Reply With Quote
Old 9th July 2011, 08:27   #2  |  Link
tin3tin
Registered User
 
tin3tin's Avatar
 
Join Date: Mar 2005
Posts: 366
Thank you. Looking forward to tinker with it.

[the download links seems dead atm.]
__________________
DVD slideshow GUI(Freeware).

Last edited by tin3tin; 9th July 2011 at 08:30.
tin3tin is offline   Reply With Quote
Old 9th July 2011, 09:33   #3  |  Link
Youka
Registered User
 
Youka's Avatar
 
Join Date: Mar 2011
Location: Germany
Posts: 64
Fixed.
Youka is offline   Reply With Quote
Old 9th July 2011, 10:31   #4  |  Link
Gavino
Avisynth language lover
 
Join Date: Dec 2007
Location: Spain
Posts: 3,431
Thanks for the new version, Youka.

Looking through the source code, I've spotted a minor efficiency glitch in GLWindow.cpp:
Code:
	valid = wglChoosePixelFormatARB(hdc, iAttributes, fAttributes, 1, &pformat, &num);
	if(!(valid && num >= 1))
		//8x AA
		iAttributes[23] = 8;
		valid = wglChoosePixelFormatARB(hdc, iAttributes, fAttributes, 1, &pformat, &num);
		if(!(valid && num >= 1))
			//4x AA
			iAttributes[23] = 4;
			valid = wglChoosePixelFormatARB(hdc, iAttributes, fAttributes, 1, &pformat, &num);
			if(!(valid && num >= 1))
				//2x AA
				iAttributes[23] = 2;
				wglChoosePixelFormatARB(hdc, iAttributes, fAttributes, 1, &pformat, &num);
Even if the first call to wglChoosePixelFormatARB (for 16x AA) succeeds, it is still called a further three times, because of a lack of {} on the 'if' statements. The indentation doesn't match what the code actually does.
__________________
GScript and GRunT - complex Avisynth scripting made easier

Last edited by Gavino; 9th July 2011 at 11:21. Reason: use narrower code window
Gavino is offline   Reply With Quote
Old 9th July 2011, 12:15   #5  |  Link
Youka
Registered User
 
Youka's Avatar
 
Join Date: Mar 2011
Location: Germany
Posts: 64
Arg, you're right. Tried in past with a computer without graphic card, so i didn't noticed it + forgot in my last check the GLWindow class ^^"
Will add brackets immediatly!

Thx for reporting.

Last edited by Youka; 9th July 2011 at 12:21.
Youka is offline   Reply With Quote
Old 13th July 2011, 15:37   #6  |  Link
Youka
Registered User
 
Youka's Avatar
 
Join Date: Mar 2011
Location: Germany
Posts: 64
Version 0.3 released.

There're just 2 small changes.
  • If internal combine function of tesselation of flDrawPath was called without colors in given path table, it changed color to black/alpha=0 for following triangles. Fixed now, that color doesn't change in this case.
  • OpenGL context was recreated for every frame. Now it creates one context for whole progress. Reuse of textures, display lists, etc. possible -> faster processing.
Because of the second change you can do a lot of shit now if you forget to disable something. See internal frame initialization and understand the problem:
Code:
glMatrixMode(GL_TEXTURE)
glLoadIdentity()
glMatrixMode(GL_PROJECTION)
glLoadIdentity()
glMatrixMode(GL_MODELVIEW)
glLoadIdentity()
glViewport( 0, 0, flGetVideoWidth(), flGetVideoHeight() )

glClearColor(0, 0, 0, 0)
glClearDepth(1.0)
glClearStencil(0)
glClearAccum(0, 0, 0, 0)
glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT | GL_STENCIL_BUFFER_BIT | GL_ACCUM_BUFFER_BIT)

local frame = --Gets video frame pixels
glDepthMask(false)
glDrawPixels(flGetVideoWidth(), flGetVideoHeight(), GL_RGBA, frame)
glDepthMask(true)

--Calls registered functions from user in right order!!!

frame = glReadPixels(0, 0, flGetVideoWidth(), flGetVideoHeight(), GL_RGBA)
--Sets video frame pixels (frame)
But it's much faster than before


Added a small example how to make karaoke effects.
Advanced creations of me i cannot post here or upload to YT regrettably because of music and video material


Documentation just reaches until OpenGL 1.1, but 1.3 is available.
Documentation is more work than the source, so it will take some time, but will continue it.


Because of the plan to allow audio access:
Video and audio runs in 2 threads, so i cannot bring it together in one function. Another developer used ffmpeg for loading audio data separatly... will think about it.

Last edited by Youka; 13th July 2011 at 16:25.
Youka is offline   Reply With Quote
Old 13th July 2011, 17:40   #7  |  Link
Gavino
Avisynth language lover
 
Join Date: Dec 2007
Location: Spain
Posts: 3,431
Quote:
Originally Posted by Youka View Post
Version 0.3 released.

OpenGL context was recreated for every frame. Now it creates one context for whole progress.
The code that does this (in GLWindow::GLWindow) will only create the context if running on an OpenGL implementation that supports multisampling.
Code:
	if(CheckMultisampling())
	{
		...

		//Set pixel format
		SetPixelFormat(hdc, pformat, &pfd);

		//My OpenGL context to current window
		cx = wglCreateContext(hdc);
	}
Shouldn't the call to wglCreateContext be outside the conditional part?
__________________
GScript and GRunT - complex Avisynth scripting made easier
Gavino is offline   Reply With Quote
Old 14th July 2011, 04:39   #8  |  Link
Youka
Registered User
 
Youka's Avatar
 
Join Date: Mar 2011
Location: Germany
Posts: 64
Quote:
Originally Posted by Gavino View Post
The code that does this (in GLWindow::GLWindow) will only create the context if running on an OpenGL implementation that supports multisampling.
Code:
	if(CheckMultisampling())
	{
		...

		//Set pixel format
		SetPixelFormat(hdc, pformat, &pfd);

		//My OpenGL context to current window
		cx = wglCreateContext(hdc);
	}
Shouldn't the call to wglCreateContext be outside the conditional part?
Most graphic card's today have OpenGL >1.3 and are supporting multisampling, so it's not such bad, but will change it.
(always the GLWindow part, ~grrr~)

Last edited by Youka; 14th July 2011 at 08:53.
Youka is offline   Reply With Quote
Old 14th July 2011, 10:15   #9  |  Link
Gavino
Avisynth language lover
 
Join Date: Dec 2007
Location: Spain
Posts: 3,431
Quote:
Originally Posted by Youka View Post
Most graphic card's today have OpenGL >1.3 and are supporting multisampling, so it's not such bad
True, but as the previous code worked without multisampling capability, you might as well continue to support this.

Incidentally, the reason I was so quick to spot this issue (and the previous one) is that I am working on something similar (without Lua) for my own purposes, and I had already made a similar change (create the context only once) in my own code.

Other points which you might find helpful:
- Some graphics cards/drivers (like my laptop!) will not render to an invisible window. You can use a pbuffer context (where supported) to avoid this problem.

- Instead of copying to and from an intermediate buffer when passing pixel data between Avisynth and OpenGL, you can use glPixelStorei(GL_[UN]PACK_ROW_LENGTH, pitch/4) with a format of GL_BGRA_EXT in glReadPixels and glDrawPixels. (This probably makes your PixelHandler class redundant.)
__________________
GScript and GRunT - complex Avisynth scripting made easier
Gavino is offline   Reply With Quote
Old 14th July 2011, 11:24   #10  |  Link
Youka
Registered User
 
Youka's Avatar
 
Join Date: Mar 2011
Location: Germany
Posts: 64
Quote:
Originally Posted by Gavino View Post
Incidentally, the reason I was so quick to spot this issue (and the previous one) is that I am working on something similar (without Lua) for my own purposes, and I had already made a similar change (create the context only once) in my own code.
It's mysterious that for every frame in avisynth the same context has to be make the current again. You know why?

Quote:
Originally Posted by Gavino View Post
Other points which you might find helpful:
- Some graphics cards/drivers (like my laptop!) will not render to an invisible window. You can use a pbuffer context (where supported) to avoid this problem.
Never heard of such a case. Know PBOs but weren't needed until know (slower than standard). Maybe add to next version with VBOs.

Quote:
Originally Posted by Gavino View Post
- Instead of copying to and from an intermediate buffer when passing pixel data between Avisynth and OpenGL, you can use glPixelStorei(GL_[UN]PACK_ROW_LENGTH, pitch/4) with a format of GL_BGRA_EXT in glReadPixels and glDrawPixels. (This probably makes your PixelHandler class redundant.)
The BGRA format is an official part of OpenGL 1.2, so i'd chosen the way by swapping RGBA because of my old PC (without graphic card) didn't support it + won't become much faster.
Youka is offline   Reply With Quote
Old 14th July 2011, 11:49   #11  |  Link
Gavino
Avisynth language lover
 
Join Date: Dec 2007
Location: Spain
Posts: 3,431
Quote:
Originally Posted by Youka View Post
It's mysterious that for every frame in avisynth the same context has to be make the current again. You know why?
I found that this was not necessary, per se. Perhaps something in the Lua part (which I don't have) screws with the context? However, in order to support multiple calls to FLuaG in the same script, it would be necessary to re-establish the current context for each frame anyway.

Quote:
Never heard of such a case.
The OpenGL spec allows rendering of non-visible pixels to be skipped (the "pixel ownership test"). And I found I had to change to using a pbuffer to see anything on my laptop (NVidia GeForce FX Go5200).
__________________
GScript and GRunT - complex Avisynth scripting made easier

Last edited by Gavino; 14th July 2011 at 11:53.
Gavino is offline   Reply With Quote
Old 19th July 2011, 11:54   #12  |  Link
Gavino
Avisynth language lover
 
Join Date: Dec 2007
Location: Spain
Posts: 3,431
Quote:
Originally Posted by Youka View Post
It's mysterious that for every frame in avisynth the same context has to be make the current again.
Revisiting this, it occurs to me that the reason you are seeing this could be multithreading. The OpenGL context is specific to a single thread, but depending on your encoder (or if using a MT version of Avisynth), GetFrame() may be called from a different thread than the constructor.
__________________
GScript and GRunT - complex Avisynth scripting made easier
Gavino is offline   Reply With Quote
Old 12th November 2011, 13:23   #13  |  Link
EmuAGR
Registered User
 
Join Date: Jan 2008
Posts: 17
FLuaG x64 v0.5

Updated v0.5 x64 build: FLuaG64.dll

v0.4
FLuaG64.dll


v0.3
I've compiled FLuaG v0.3 for AviSynth64 (MSVC++ 10): FLuaG64.dll
The source code is almost the same, just converted inline ASM to intrinsic functions for MSVC to compile fine in x64. Replace PixelHandler.h in the Source folder and libs\Lua_5.1.4_static.lib with a suitable Win64 VC lib from here: Lib for MVSC++ 10

Last edited by EmuAGR; 14th December 2011 at 21:53.
EmuAGR is offline   Reply With Quote
Old 22nd November 2011, 13:06   #14  |  Link
Youka
Registered User
 
Youka's Avatar
 
Join Date: Mar 2011
Location: Germany
Posts: 64
(Release next)

Last edited by Youka; 24th November 2011 at 07:32.
Youka is offline   Reply With Quote
Old 22nd November 2011, 21:50   #15  |  Link
Büke
Registered User
 
Join Date: Jul 2010
Posts: 4
You are awesome Youka! This opens worlds of possibilities for AviSynth.

I am especially looking forward to fragment shader support.

Multiple inputs would be nice too, but I imagine, one workaround would be StackVertical() for video...
Büke is offline   Reply With Quote
Old 24th November 2011, 08:07   #16  |  Link
Youka
Registered User
 
Youka's Avatar
 
Join Date: Mar 2011
Location: Germany
Posts: 64
FLuaG 0.4 release (see first post).
Youka is offline   Reply With Quote
Old 24th November 2011, 12:03   #17  |  Link
Gavino
Avisynth language lover
 
Join Date: Dec 2007
Location: Spain
Posts: 3,431
Thanks for the update, Youka.

Quote:
Originally Posted by Youka View Post
Version 0.4 (24.11.2011):
  • New process structure (similar to Avisynth Filter SDK)
Does this mean scripts written for the earlier version will no longer work and must be adapted for the new interface?
__________________
GScript and GRunT - complex Avisynth scripting made easier
Gavino is offline   Reply With Quote
Old 24th November 2011, 12:26   #18  |  Link
Youka
Registered User
 
Youka's Avatar
 
Join Date: Mar 2011
Location: Germany
Posts: 64
Regrettably yes, but it's not really work to change this and i will not change it in future anymore.

Last edited by Youka; 24th November 2011 at 12:38.
Youka is offline   Reply With Quote
Old 25th November 2011, 19:50   #19  |  Link
tin3tin
Registered User
 
tin3tin's Avatar
 
Join Date: Mar 2005
Posts: 366
Truely great work Youka! Very nice with all the included examples, but why did you leave out your previous 3D OpenGl examples?

I've got to investigate further myself, but is it possible in this version to make video 3D transitions over a video background?
__________________
DVD slideshow GUI(Freeware).
tin3tin is offline   Reply With Quote
Old 26th November 2011, 13:09   #20  |  Link
Youka
Registered User
 
Youka's Avatar
 
Join Date: Mar 2011
Location: Germany
Posts: 64
My old examples are all rewritten and included into the new ones, so there's nothing missing except the video cube.
Making a video 3D transition over a video background won't be a problem but it might be you mean working with more than one video stream. That will come with clip array input to FLuaG soon, but for now you have to use some tricks.
Quote:
Originally Posted by Büke View Post
Multiple inputs would be nice too, but I imagine, one workaround would be StackVertical() for video...
Youka is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 05:46.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.