Building professional video editor using AviSynth?

[Gavino]

I've attached a pdf with some suitable pictures and desciptions of the intended structure and some pseudo-code as to how the script might be generated. Would appreciate any comments - I'm trying to second guess what methods might be more or less efficient.
I haven't had time to study everything in the paper yet, but here are some quick thoughts.

When rendering multiple secondary timelines, instead of creating vertical sections cutting across all the timelines, I think it would be both simpler and more efficient to render each secondary timeline in turn, each building on the result of the previous ones. Thus your example would end up looking something like:
# construct primary timeline:
result = ... + ... + ...
# add 1st secy timeline:
result = result.Trim(0, t1) + Overlay(result.Trim(t1, t2), ApplyTransition(c3, c4)) +
\ result.Trim(t3, t4) + Overlay(result.Trim(t4, t6), c5) + result.Trim(t6, 0)
# add 2nd secy timeline:
result = result.Trim(0, t2) + Overlay(result.Trim(t2, t5), c6) + result.Trim(t5, 0)

Advantages:
simpler to generate - can treat each timeline independently and uniformly (important once it comes to adding other classes of timelines, eg audio, subtitles).
more efficient script - less Overlay instances

Other random thoughts on efficiency:
- generate a single blank clip (or one per clip size/properties) and reuse wherever needed;
- overlay can be replaced by simple splice when overlaying clip has same size/properties as underlying clip (and no mask used).

[JonE]

When rendering multiple secondary timelines, instead of creating vertical sections cutting across all the timelines, I think it would be both simpler and more efficient to render each secondary timeline in turn, each building on the result of the previous ones.
Good point ... certainly gets around the horrendous looking overlay-within-an-overlay ... consider idea stolen :)
generate a single blank clip (or one per clip size/properties) and reuse wherever needed;
Hadn't considered that, but yes it does make sense. Presumably, BlankClip uses the same amount of memory/cpu regardless of its duration, so just create appropriately formatted BlankClip's with duration equal to overal project duration then I know I can gaurentee to always be able to trim out a length that I need.

BTW, whats the maximum line length AVISynth can handle?

TTFN,
Jon

[Fizick]

whats the maximum line length AVISynth can handle?

TTFN,
Jon

There is no explicit limit.
I use 1-line pseuodoclip in MVTools for motion vectors stream.

[tin3tin]

Here's an idea. Both the NLEs OpenShotVideo and Kdenlive are using MLT as "engine" in Linux. What they do is to produce a script(from the timelines) which is passed by mlt(melt).

I think OpenShotVideo is coded in python, so that should be portable(if the author likes the idea), however porting mlt might be more difficult. Instead of porting Mlt, I guess the script produced by OpenShotvideo could be passed into an avisynth script instead.

Here's a description of the Mlt format(very timeline friendly format): http://www.mltframework.org/twiki/bin/view/MLT/MltXml (http://www.mltframework.org/twiki/bin/view/MLT/MltXml)

[JonE]

My main target involves AVISynth scripts which offer different things to MLT, particularly with respect to effects and filters. However that doesn't mean the main timelines and I guess overlays couldn't be output in an MLT style XML but I haven't got the time to look in detail - please don't read that as being not bothered, simply that unless someone expert with the MLT system can offer advice then I don't think anything would get done.

KDEnlive's facility for pre-trimming media before placement on the timeline is good and a variation of it is certainly something that every editor should have.

TTFN,
Jon

[Baxich]

Hi everybody, it has been a couple of years that I was thinking about creating an AviSynth NLE GUI, but at the time I didn't know how to get info from AviSynth in Delphi, and an example app caused some problems in Delphi I used - I don't remember the details... Anyway, about ten days ago I tried that ShowVideoInfo (an example how to use avisynth.h in Delphi), it worked OK and then I tried to encapsulate those C functions into an Delphi object (class in C++)... And so it started...

So far I have built a small OOP application (I don't see much point of sending you this program at this almost fetus stage - but if somebody insists, ali will send it) that can import video through AviSynth, automaticaly make thumbnails, preview videoclip by moving a slider (no real playback yet, audio is still "killed"), set in and out points for each videoclip in "bin", create a track, add trimmed (with in and out points) videoclips from bin to a track, apply Mohan's filter ReTouch (I like it's "watercolors" so I wanted to see if I can properly apply this filter through my program), affect opacity of a clip, preview sequence in my editor (with visualization - choose histogram type and channels - all through internal AviSynth filters) and export to AVS file.

What are my goals?
To create a free Windows AVS NLE GUI.

Why Windows?
Because Linux has several free NLE programs, and it seems that they won't be ported to Windows, and Windows basically has none (I don't count almost useless Windows Movie Maker or horrible ZweinStein or whatever-is-it-called video editor)! And Windows is still much more popular than Linux, and I believe this will still be the case in the next five years, if not ten or even more. I agree - we don't need yet another AviSynth GUI - for Linux, but we desperately need one for Windows!

What tool?
I started building it in Delphi 6 Personal. Someday it might be ported to Linux through Lazarus, but as I said I'm not considering Linux right now. I must say that I'm actually really astonished that no one has yet built a normal free NLE for Windows. I actually blame Microsoft for that - for many years they had only that rubbish MFC framework. Borland has brilliant VLC framework, but because it is not from Microsoft, people gave up Borland tools - and they are (or at least were) superior RAD's. I'm really sad to see that almost nobody uses Delphi anymore, and it is one of the best Windows desktop application development tools - programs made in Delphi stable and fast, and development time is short. There are few more reasons for Delphi - this is free version, I'm don't know other languages/frameworks nearly good as Delphi, and also development of visual components in Delphi is very easy - and I'll need to make a ton of them.

What features?
So I want to create a powerfull NLE as AviSynth GUI, which means importing, trimming & cutting media, multitracks, applying AviSynth effects in realtime (or almost realtime) - my program will because of that be usefull even to AviSynth pros because this will be the easiest way to find proper values for effect parameters - one could import their AviSynth script, ad an effect, tweak parameters, and then continue writing his script in a favourite text editor.

I intend to make this program without some common artificial limitations. For example, in my program one will be able to apply effect/filter on everything:
- on clip in a library
- on a foler/bin in a library (it will affect all clips inside that bin - this can be useful if you import interlaced video from camcorder so you can apply deinterlacing, resizing and color correcting to all clips sourced from camcorder)
- on s clip inside a track
- on a track
- on a sequence
Effects will be inherited - if one puts an effect on a sequence, it affects all tracks in that sequence and clips inside them. If you put an effect on a track, it will affect all clips inside a track. This will be possible because all of these objects (except folder/bin in a library - this does not even exist yet) are descendants of a class TClip. And TClip knows that if it has a source TClip, or a parent track, that it first asks it's source or parent to apply their effects and only after that the clip applies it's own effects. Anyway, my program will demonstrate a beauty of object-oriented programming - create object rules and let objects play with themselves. ;-)


So with this post I just wanted to inform you about my plans. I might need a help from you in near future. I have already read some interesting points in this topic that will help me in making my program.

[tekNerd]

I would love that very much.

There is a project started in C# https://sourceforge.net/projects/manamanator/, but I cant make it work...it is very alpha though...

[Baxich]

There is a project started in C# https://sourceforge.net/projects/manamanator/, but I cant make it work...it is very alpha though...
This is one of the reasons I chose Delphi (or better to say I stick with Delphi) - no dependencies, no special DLLs are required, no .NET framework (and exactly "this and this" version of framework), run time errors cannot bring down a program (for example, I remember that applications made in older versions of Visual Basic would close on basically every run-time error - even those totally benign - a Delphi program many times survives even "access violation" errors).

I also tried that program and didn't work for me either. But what is unbelievable to me is that the author of that program didn't even put a single screenshot of his program - basically keeps us wondering if his program is actually real!

[tin3tin]

@ Baxich
Sound great. I'm looking forward to follow your progress.

Some more thoughts on avisynth script editing combined with a NLE interface:
In DVD slideshow GUI there is a very simple approach to Avisynth scripts. They can be imported as slides(clips), the order can be rearranged and the individual scripts can be edited(trim etc.) in AvsP. When AvsP is closed after editing the script clip is updated in DSG. However DSG doesn't have an interface suited for real NLE work, but the thinking of each clip as individual scripts could be added to existing NLEs.

Now Pitivi (http://www.pitivi.org/wiki/Main_Page) is now beeing ported to windows by Andoni Morales(of LongoMatch (http://www.longomatch.ylatuya.es/index.php) fame). Check it out here (https://bugzilla.gnome.org/show_bug.cgi?id=601982).

Pitivi is based on Gstreamer which Andoni is porting to windows as well. Check it out here (http://www.gstreamer-winbuild.ylatuya.es/doku.php).

Now if it is possible to load Avisynth scripts into Gstreamer with this (http://gstreamer.freedesktop.org/wiki/AviSynth) wrapper. Pitivi should become able to load avisynth scripts. From there I guess it should be a trivial matter to add a patch to Pitivi(in python - I think?) which allows users to open avisynth sourceclips in AvsP for editing - and updating those clips in Pitivi after AvsP is closed.

But that is of course entirely depending on the ability to import avisynth scripts into Gstreamer. Anyone knows if this actually is possible with that wrapper?

[EDIT: Found som more on that wrapper here (http://blogs.gnome.org/uraeus/2009/08/24/summary-of-this-years-gstreamer-summer-of-code/): Support for Avisynth filters in GStreamer – Ruslan Ijbulatov has been working steadfastly on this and solving issues as they have arisen. The current code allows you to recompile any Avisynth element towards GStreamer, for use in GStreamer pipelines, with just some slight tweaking of the original code. While the original goal was to allow the use of these plugins unmodified this turned out to be untenable during the course of the project. Our new API though should make it easier to make avisynth filters cross platform. The code to enable the Avisynth filters are currently GPL as it contains code under that license from Avisynth, but we hope to get permission for a relicensing the parts of Avisynth code in there to the LGPL, so that if the avisynth plugin writers are willing we can get access to those plugins under a license better fitting GStreamer.
So it's aim/result isn't to let Gstreamer to import Avisynth scripts...]

[Baxich]

@ Baxich
Sound great. I'm looking forward to follow your progress.

Some more thoughts on avisynth script editing combined with a NLE interface:
In DVD slideshow GUI there is a very simple approach to Avisynth scripts. They can be imported as slides(clips), the order can be rearranged
This is already possible in my program - it makes (almost) no difference to me whether a file is AVI or AVS - it is loaded with the same function.


and the individual scripts can be edited(trim etc.) in AvsP. When AvsP is closed after editing the script clip is updated in DSG.
Yes, this is a good idea. For a while I was thinking about integrating an AVS editor into my program, but I dropped that idea because it would take too much time and there is no point of developing an editor if I can't make it better than some existing editor.

Also, the whole point of a visual NLE is to hide underlying code from users. Because of that I'm thinking about introducing "User experience modes" - in beginner's mode user practically wouldn't be aware that there is AviSynth under all that, and in Expert mode user would see code generated for every imported clip, would be aware of clip/track/sequence internal names, would be able to put a "custom" effect (by directly entering AviSynth code) etc.

So, yes, I will use external AVS Editor. If I forget about this feature, please remind me later. :) At this moment a priority is to develope a class that represents a sequence, and to make a timeline that can work with a sequence (multiple tracks).

Now Pitivi (http://www.pitivi.org/wiki/Main_Page) is now beeing ported to windows by Andoni Morales
I won't wait for that. And if it won't have support for AviSynth it will lack much of the so wanted and needed features. My goal is to use as much AviSynth's features as possible.


BTW are you the author of DVD slideshow GUI? I like your program - it was actually one of the reasons why I decided to continue developing my program, because it gave me hope that Windows NLE built on top of AviSynth is not SF. ;)

[tin3tin]

BTW are you the author of DVD slideshow GUI?
Yes I am - unfortunately I coded it in wxBasic, which has been pretty much dead the last 4-5 years. So making a NLE interface is impossible in wxBasic - there are simply too many functions missing. :rolleyes:

What do you mean by 'SF'?

[Gavino]

SF = Science Fiction

[Baxich]

Yes I am - unfortunately I coded it in wxBasic, which has been pretty much dead the last 4-5 years. So making a NLE interface is impossible in wxBasic - there are simply too many functions missing. :rolleyes:
Oh... I hate Basic... It is a horrible programming language (except maybe VB.Net, but I don't have any experience with it). I started learning to program with Basic (QBasic) in DOS, but I was very frustrated with what it can do, I tried Turbo Basic which was better, but still unsatisfactory, and then I discovered Turbo Pascal - really great language at the time and superfast compiler, and you could write almost any program (it has built in assembler, if needed). And transition to C/C++ is not hard.


Back to the NLEs... I am having doubts - should video and audio tracks be separated (as in Adobe Premiere), or should user be able to freely arrange them as he or she wishes (I suppose as in Sony Vegas - although I practically didn't work in it)? The difference is whether a sequence class holds one list (vector) of tracks for both audio and video, or I have two lists of tracks, one for video, and one for audio. It's not a big difference to me at this point, but if I choose one solution now, it will be pain in the a.s to change it later, when all or most features will be done. I like video and audio tracks separated (I worked mostly in Premiere), but I would like to hear other opinions.

[Gavino]

I would prefer to have both possibilities, so that a given track can be:
- video only
- audio only
- video+audio (so cutting a section affects both in sync)

[vlada]

I think you should implement both options. To allow usage of your NLE to both novice and advanced users. So if I import a movie, it should load both video and audio and show them separately, but keep them locked together. If I move or modify video track, the audio keeps in sync. But allow a simple command to separate the tracks.

I think it is a good idea to allow grouping and locking tracks. The basic groups would be input movie and whole timeline. But also allow creating of user's defined groups, which then can be locked/unlocked.

Do you understand my idea?

[Baxich]

I see now that I might have badly explained what I have doubts in. Clips in the time-line (in a sequence) will be video-only and audio-only clips, and also video/audio clips in which audio and video will be linked (they will be synced, they will move one with the other, cut or roll one with the other etc., but they will be represented with one bar in a video-track and one bar in an audio track) - as in Premiere. User will be able to separate them if he or she wants that.

What I was actually asking affects the view/organization of a time-line... In Premiere you have audio tracks (at the bottom of time-line), and video tracks (above audio tracks). I believe that Vegas doesn't make that distinction - you can freely "mix" audio and video tracks in time-line - or am I wrong? For example, in Premiere:

| Video track 03 | |----------------clip 3 - logo overlay------------------|
| Video track 02 | .|-clip2-|...........|-clip - video only-|.....................
| Video track 01 | ............|---clip1---| .............................................
---------------------
| Audio track 01 | ............|---clip1---| .............................................
| Audio track 02 | .|-clip2-| ................. |-clip - audio only-|.............


In some other editor you might have (Vegas?):

| Video track 03 | |----------------clip 3 - logo overlay------------------|
| Video track 02 | .|-clip2-|...........|-clip - video only-|.....................
| Audio track 02 | .|-clip2-| ................. |-clip - audio only-|.............
| Video track 01 | ............|---clip1---| .............................................
| Audio track 01 | ............|---clip1---| .............................................


So how do you think a timeline should be organized? Or you have some other concepts?

[JonE]

IMO the requirements of Basic and Advanced users require quite different thinking. Carefully think about how each type will use the system i.e. exactly what jobs will they try and do ?

A basic user should by default be hidden from as much detail as possible - question every item you intend to display and try to think "do they _really_ need to know this".

So, editing their wedding video, or cutting out the ads from a TV broadcast, or uploading their mobile phone footage to YouTube, are they likely to want to split out audio ? I doubt it. So, have a single timeline - add second timeline only if they choose to add an effect (e.g. picture-in-picture). Only add an audio timeline if they choose to add a seperate audio source etc. That way the interface expands to suit the user.

However, this approach is likely to be wearisome for advanced users, so by default I'd wade in with four video timelines + two audio timelines (plus a text overlay timeline) and automatically add any video file's audio to the first audio timeline, by default linked to the video clip(internally managed still as a single file to help maintain sync).

Personally, if I were to display a library of available files that I could drop onto a timeline, I prefer a nice text list (filename, frame rate, image size, blah blah blah) but a novice user will almost certainly be expecting a nice thumbnail they can pick instead.

There are lots of other differences, but I came to the conclusion that it is better to write two versions ("Lite" and "Pro" - yeuk!) - by putting as much of the underlying functionality into common libraries there isn't that much more effort.

Final question is "how pro should you go" ? Part of the power of AviSynth is its being script based, so any attempt to totally wrap it in a gui is just going to hamper the advanced user.

TTFN,
Jon

PS : Although so far not a problem, could we leave out discussions of programming languages ? Only I've seen too many threads hijacked, turned into wars and the thread closed. The best programming language is the one used by whoever actually sat down and wrote the program.

[Baxich]

I think it is a good idea to allow grouping and locking tracks. The basic groups would be input movie and whole timeline. But also allow creating of user's defined groups, which then can be locked/unlocked.

Do you understand my idea?
I suppose by "tracks" you actually mean audio/video clips in a timeline?

To me a track is "one line" in a timeline - a track can hold one or more clips. Audio tracks can hold audio clips (or audio-part of an audio/video clip), video tracks can hold video clips (or a video-part of an audio/video clip).

A sequence consists of one or more audio tracks and one or more video tracks. Actually, that is a Premiere limitation - I think that I can make audio-only and video-only sequences which can be very useful when editing video (video-only sequence could for example hold photos with transitions).

[Baxich]

So, editing their wedding video, or cutting out the ads from a TV broadcast, or uploading their mobile phone footage to YouTube, are they likely to want to split out audio ? I doubt it. So, have a single timeline - add second timeline only if they choose to add an effect (e.g. picture-in-picture). Only add an audio timeline if they choose to add a seperate audio source etc. That way the interface expands to suit the user.

However, this approach is likely to be wearisome for advanced users, so by default I'd wade in with four video timelines + two audio timelines (plus a text overlay timeline) and automatically add any video file's audio to the first audio timeline, by default linked to the video clip(internally managed still as a single file to help maintain sync).
I have already addressed this issue - my program will have several experience modes so yes - the total beginner experience mode could consist only of one video and one audio timeline (or track, if we use Premiere's terminology). I'm not sure about a "text overlay" track - maybe... I suppose it wouldn't be that hard to achieve that - text titles in bin would be marked as "text titles" so a program would automatically put them into text track, and video/audio in those tracks.

And new sequence i pro-experienced would have 3-4 video tracks and several audio-tracks.

This is not a problem to achieve.

Personally, if I were to display a library of available files that I could drop onto a timeline, I prefer a nice text list (filename, frame rate, image size, blah blah blah) but a novice user will almost certainly be expecting a nice thumbnail they can pick instead.

This can be configurable - so yes, it can default to thumbnail+clip name for beginners, and to thumbnail+extensive info for pro's, with an option of not displaying a thumbnail.


There are lots of other differences, but I came to the conclusion that it is better to write two versions ("Lite" and "Pro" - yeuk!) - by putting as much of the underlying functionality into common libraries there isn't that much more effort.

There won't be two versions of my program, that's for sure. It will have two (actually more than two) experience modes/levels.

Final question is "how pro should you go" ? Part of the power of AviSynth is its being script based, so any attempt to totally wrap it in a gui is just going to hamper the advanced user.
The idea is to create a full-featured NLE that can output .AVS. I will be satisfied if I make that.

User will be able to have an AVS script as an input audio/video file, and also the user will be able to add an AviSynth script/file as "custom effect" so through that an advanced user will basically be able to wrap any functionality of AviSynth he or she wishes. The resoulting output AVS script probably won't be as optimizes as if the user does everything by hand, but I don't see that as a problem.

[vlada]

Baxich> Sorry for the confusion of terms. Of course I meant clips, not tracks. I see that your plan is very similar to my idea.

According to the order of tracks (first video or mixed), why not leave it up to the user? The track (from a programmer's point of view) will be an audio or video object. So you should be able to change their order as you like.

[Baxich]

According to the order of tracks (first video or mixed), why not leave it up to the user? The track (from a programmer's point of view) will be an audio or video object. So you should be able to change their order as you like.
From a programmer's point of view it is easier to implement separate video (layered with AviSynth's layer function) and audio tracks (MixAudio), and also is marginally faster (speed difference probably wouldn't be detectable, but it will be there! :D).

But yes - it's might be the best to have only one list (vector) of tracks, and then sort them if user want's separated audio and video tracks like in Premiere.