Command Line Compressor - AVS2AVI

[Emp3r0r]

EDIT DaveEL:
Latest version of avs2avi currently is http://daveel.leffe.dnsalias.com/avs2avi-131a.zip
Just to save you reading the entire thread.


I am looking for a commandline style program that would take a input.avs file and a target output.avi and then prompt you with this: http://jvance.com/images/videocompression.gif

I've already attempted the project myself but I still don't know enough C++ or the VCM interfaces to take "store.c" from DVD2AVI and create the app. Maybe something like this already exists or someone thinks it wouldn't be that hard to whip it up.

Additionally, Nic provided me with the following help: "For encoding to avi, you need to learn the AVI API & IC API to do it yourself. Look up commands like AVIGetStream & ICCompress. Basically you do an AVIReadStream from an avs file. That gives you the YUY2 frame in a buffer, then call ICCompress to get the compressed frame & then use AVIWrite (I think :) DVD2AVI is the place to look :) )"

Thanks Nic and anyone else who may provide help.

[int 21h]

Look to M4C sources.

[Emp3r0r]

Thanks and I looked at the code but I just found out in my C++ class that you can set a method()=value and it kinda blew my mind :p but anyway, my c++ skills are horrible so I think I will wait for DirectX.NET

[[Toff]]

Here is a surprise for you ;) in attachement.

It should work with RGB24, RGB32, YUY2 and YV12 avs source.

It compress well with DivX 3.11, but it crash at the end with DivX5 and XviD, on the last :
AVIFileRelease(pSRCfile);
I don't now why ATM, I will look at it later. The file is still readable anyway at start :D

Source are included, if any vfw guru want to look at it ?

EDIT : Attachement removed (see below for new version)

[[Toff]]

I found what the crash problem is.
Line 195 :
if(pDSTCompVideoStream)
AVIStreamRelease(pDSTVideoStream);
Must be :
if(pDSTCompVideoStream)
AVIStreamRelease(pDSTCompVideoStream);

No crash anymore.

Even if the avi seems perfectly playable there is still a problem at the end of the stream (an invalid chunk, that's what vdub say when using the hexeditor and Riff chunk tree)
The problem is also present in DVD2AVI.

EDIT : Oups, a new problem fixed all frames were written as KF, new upload soon. It's seems that RGB support is broken, I must test with a stable avisynth version.

[[Toff]]

OK here is 1.01 version

Tested with avisynth 2.06, RGB24,32,YUY2 works compressing to Xvid, DIvX5 or DivX 3.11.

Tested with avisynth 2.5alpha YUY2, YV12 works.
But RGB seems broken with 2.5 alpha.
I can post a short clip If you want.
EDIT : no bug, I must put ConvertToYUY2 before ConvertToRGB24

Ctrl-C terminate properly now.

EDIT : Attachement removed (see below for new version)

[Emp3r0r]

Your awesome! I appreciate your work! I will start testing right away. Thanks again.

[Emp3r0r]

Great work [Toff]! This is exactly what I was looking for! If you ever get time could you display the FPS during the encode? Also, how do learn how to read/create/modify a codec parameters file?

PS: or just tell it to use xvid since i can change those parameters in the registry.

[Emp3r0r]

I guess I must wait for mod to approve your version 1.01?

[[Toff]]

>I guess I must wait for mod to approve your version 1.01?
Yes, just have to wait.

>If you ever get time could you display the FPS during the encode
Yes that should be easy, I will add an estimated time also.

>Also, how do learn how to read/create/modify a codec parameters file
The file that is created is a raw copy of the codec state so there is no general method to create/modify them. Each codec has is own format.

>or just tell it to use xvid since i can change those parameters in the registry
I think that's possible to make a special case for xvid :)

There is still a little problem at the end of the file, when you use your media player in loop mode it doesn't loop.
The problem is also present in DVD2AVI.
I let you test.

[int 21h]

I'll take a look at this later and see about adding some stuff and perhaps compiling with Intel for speed increases (when I'm sober :) )

[int 21h]

I hope [Toff] doesn't mind, but I went ahead and implemented a limited two pass functionality. It works fine, but its limited in that it currently doesn't support the load/save function, if you supply load/save on the commandline and also supply the twopass argument, it should dump you out with the usage. (Sorry I forgot to update the usage printout also)

To activate two pass mode you just supply something like

avs2avi source.avs destination.avi -2

This will, in succession ask for compression codecs, the first will be for the first pass, and the second for the second pass (it will confirm them and display). If they don't match (i.e. you select XviD during one pass and DivX for the second, the program will close, printing an error).

Version number is incremented to 1.10.

I tested this briefly and I'm 99% certain it works and I didn't break anything existing.

http://phreak404.virtualave.net/avs2avi_110.zip

[[Toff]]

No problem, feel free to improve.
Let's go test 2 pass encode now.

[Emp3r0r]

I must be in heaven. You guys are great! I did some testing with DVD2XCD and I think everything is now in place to create a new GKnot type program since GKnot is very outdated. Thanks, for the work.

[io]

Originally posted by [Toff]
Even if the avi seems perfectly playable there is still a problem at the end of the stream (an invalid chunk, that's what vdub say when using the hexeditor and Riff chunk tree)
The problem is also present in DVD2AVI.
i stumbled over the same problem using AviFile. the files created seem to be fine, but virtualdub shows an error in its hex editor and some apps did not like these files. i donot think that it is a bug in the code, because the same behavior is present in other programms using AviFile (especially. the VFW examples from microsoft!).

as far as i can see, there is a block added after the RIFF AVI chunk (directly succeeding the AVI legacy index), but it only contains a portion of the last frame. i guess this is for some strange padding reasons, but i would like to get rid of this stuff!

probably the only way to solve this, is to follow virtualdub/vidomi and code your own version of AviFile. *sigh*

[[Toff]]

Thanks, for your comment io.
So I know it's not entirely my fault.

Here we go for v1.11 with :
- 2 pass load/save mode.
- fps display, and remaining time (not very accurate, don't know why)

avs2avi_111.zip (http://christophe.paris.free.fr/temp/avs2avi_111.zip)

No use default codec parameter (from registry) without prompt at the moment.

[int 21h]

I had a v1.11 also, but I've incorporated my additions in to yours once again, so here is v1.12

Changelog
---------

ReWrote Option Parsing (Won't mean much to most of you, except you can put your flags in whatever order you want now, source, and destination still need to be 1st and 2nd respectively)
Added -q to the flag list, use this to get a 'quiet' mode and supress the frame by frame output (gains about 1fps)
Cleaned up program run header a bit to look sexier ;)
Fixed potential crash if invalid codec state file was supplied, or if there was a problem opening a file for dumping the state
Compiled with Intel compiler (program now requires at least MMX to run), modest speed gain should be seen by some

[unplugged]

- Surprised about so simple and so effective! (quality+speed toghether with MPEG2DEC3 YV12 and AVS 2.5a)
- Now I'll use VirtualDub only to find frames boundaries :p, command-line is more quick to setup!!
- Should be bundled with Avisynth

Thanx Toff!

[MaTTeR]

Any chance of multi-threading for the future? *grin* I only ask because otherwise it's going to run much slower on SMP boxes compared to Vdub. Good work guys, can't wait to do some testing.

[int 21h]

I'll look into it, I'm not familiar with how to synch the threads.

[MaTTeR]

I started reading some OpenMP documentation several months ago just to see how difficult it would be to parallelize threads. Well, not being a coder myself, I don't half to tell you how completely lost I was after 3 days of reading:rolleyes: Thx for looking into it though, I'm sure Acaila would appreciate it as well.

[-h]

CreateThread() through pure win32 isn't that horrible from when I last used it. CreateMutex() will make synchronization pretty easy, since you only have to keep decoding 1 frame ahead of encoding.

I would say create your encoding/decoding threads, and have the encoding thread polling for a mutex with the current required frame number. Once that mutex appears, encode the frame and free the mutex, then create another mutex signifying that the frame has been successfully encoded. Decoding should poll for the success mutex created by the encoding thread and proceed to decode another frame (or however many in advance).

There are probably better ways to do it, but this is a pretty quick and easy way to get encoding/decoding threads in your system.

-h

[gamr]

this is kewl! back to cli! now for the fun of writing some bat files to detect if there is an avs in a directory and if so process it if not sleep, this will be fun :)

im getting weird results with the eta tho,

Pass 1/1 frame 1160/135572, (4.53fps) 00:05:57.233

obviously that should be 5hrs not 5mins. also is there any chance of getting current filesize/estimated final filesize?

[[Toff]]

>gamr
>im getting weird results with the eta tho,
>Pass 1/1 frame 1160/135572, (4.53fps) 00:05:57.233
Yes it's very possible as I haven't much test it.

>also is there any chance of getting current filesize/estimated final filesize?
For 1 pass mode and 2nd pass it can be done.
But estimated final filesize will may be not accurate. (as my crazy eta)

>int 21h
>ReWrote Option Parsing (Won't mean much to most of you)
>...
Thanks, it was needed ;)

>MaTTer
>Any chance of multi-threading for the future
I think there can be 1 thread for avs frame reading, and 1 thread for encoding.

Hey ! 60 ko ! it's starting to be bloated :D
Let's go test this 1.12 version now.

[sh0dan]

Great, great program - it's so simple and works so well!

Getting this however:
Pass 1/1 frame 273/1038, (10.63fps) 00:00:-14.-466

Seems a bit strange to me ;)

How about audio handling? Passing uncompressed audio should be fairly easy. Export to WAV could also be a nice feature.

(Another thing: It seems like overkill to use the intel compiler - there really isn't anything to optimize, as far as i can see).

[int 21h]

Originally posted by sh0dan
(Another thing: It seems like overkill to use the intel compiler - there really isn't anything to optimize, as far as i can see).

I just always use it, its performance, even for anything that appears unoptimizable, usually gives better results on my machine than the MS one, so that's what I use. Feel free to recompile to whatever you'd like :)

[sh0dan]

Sorry - didn't mean to offend you!

It's a great tool! I already use it exclusively for performancetesting.

[int 21h]

I wasn't offended :)

[int 21h]

I've got the functions seperated out so that I can thread them, just having trouble synching my threads now, I'll have to read up on Mutex and Event stuff.

Just thought I'd let you guys know though, I am working on it :)

[[Toff]]

Great !

I let you know that I have fixed the ETA bug.
Now I'm working on using a codec (default registry settings) by giving his fourcc.
Something like :
avs2avi mymovie.avs myXvidMovie.avi -c xvid

An idea for synchro:
* Init:
mysemaphore = CreateSemaphore(NULL,0,1,NULL);

* Reader :
Read frame
ReleaseSemaphore(mysemaphore, 1, NULL);

* Writer (in is own thread) :
WaitForSingleObject(mysemaphore, INFINITE);
Write frame

Something strange with windows mutex:
doing :
WaitForSingleObject()
WaitForSingleObject()
in same thread doesn't block the thread.

[int 21h]

Originally posted by [Toff]
Something strange with windows mutex:
doing :
WaitForSingleObject()
WaitForSingleObject()
in same thread doesn't block the thread.

That explains exactly the problem then. I had a WaitForSingleObject(hMutex,INFINITE) and it never stopped one of my threads from going...

[[Toff]]

I've already exprimented this problem on an old multiplatform (Linux/Win32) project. And I remember very well differences between windows mutex and pthread mutex (I have spent enought time to find the bug on win32 when I first develop more than half of the project on linux)

[mandm]

where can I get a avs to test the program ?


thanks

M&M

[int 21h]

Originally posted by [Toff]
Great !

I let you know that I have fixed the ETA bug.
Now I'm working on using a codec (default registry settings) by giving his fourcc.
Something like :
avs2avi mymovie.avs myXvidMovie.avi -c xvid

An idea for synchro:
* Init:
mysemaphore = CreateSemaphore(NULL,0,1,NULL);

* Reader :
Read frame
ReleaseSemaphore(mysemaphore, 1, NULL);

* Writer (in is own thread) :
WaitForSingleObject(mysemaphore, INFINITE);
Write frame



If you use Semaphore, how does the Reader know to stop reading until the writer is ready for another frame? It should be something like this right?

ReadFrame
WaitUntilWriteReady
WriteFrame
WaitUntilReadReady

etc.

I had some other coding to do tonight so I didn't give it as much time as I would have liked, I'll mess around with it more tomorrow.

[[Toff]]

int 21h : If you use Semaphore, how does the Reader know to stop reading until the writer is ready for another frame? It should be something like this right?

Yes you are right, if reading a frame is fastest than compressing it we are going to have big problem, so one way is to use 2 semaphores (initialized differently) :

* Init:
hSemWaitRead = CreateSemaphore(NULL,0,1,NULL);
hSemWaitWrite = CreateSemaphore(NULL,1,1,NULL);

* Reader :
WaitForSingleObject(hSemWaitWrite, INFINITE);
Read frame
ReleaseSemaphore(hSemWaitRead, 1, NULL);

* Writer (in is own thread) :
WaitForSingleObject(hSemWaitRead, INFINITE);
Write frame
ReleaseSemaphore(hSemWaitWrite, 1, NULL);


Here is v1.13
avs2avi_113.zip (http://christophe.paris.free.fr/temp/avs2avi_113.zip)
What's new ?
- Use codec by fourcc, can be used by external program to encode in xvid. The external program first tweak xvid registry key configuration. Then use the "-c xvid". (sould be usefull to Emp3r0r)
- Fixed ETA
- Display current size of AVI

I'm missing space for estimated final filesize.


mandm : where can I get a avs to test the program ?
You must do it yourself ;)
For that you need Avisynth first : http://www.avisynth.org

[octapus]

Hey , it's a very nice work you have done. I was wondering what those lines mean inside function
void InitializeDestination(const char* filename)
.
.
.
.

BITMAPINFOHEADER rgb = bmihVideoSource;
rgb.biCompression = BI_RGB; // Force to RGB ? (doesn't accept YUY2, or YV12)
rgb.biBitCount = 24; // Force 24 ? (doesn't accept 16 or 12)
rgb.biPlanes = 1; // Force 1 ?
.
.
.
.
.


And another thing which has been said some times. Time remaining has to output correctly.

Thanks again.

[octapus]

I'm sorry i didn't read before i post (about remaining time).
-Fixed ETA.

[stax76]

I did some testing with DVD2XCD and I think everything is now in place to create a new GKnot type program since GKnot is very outdated.

I hope people will like your new GKnot type program better as mine ;) , like you said before people will have a hard time to replace trusty GKnot with another programm, especially when it's a .NET program. When the new GKnot is finished, everbody will use it..., but I know it is a lot hard work to code all the features people expect from the new version, I already did most of it :D

[int 21h]

Version 1.14

What's new?

Multi-threaded, decode/encode/display are now seperate threads, this improves performance on even non multi-processor machines.

[mandm]

Well in M&M after a lot of test
semaphores - signaling object - timers...


the easiest way was to generate 2 threads and to put a
sleep (0)
where you want to switch between them...

of course only one thread is active at time...

... instead if you have a multiprocessor :eek: ...
... you should have a sync at a variable level
(but I do not have a MP to test it...:( )


At the end we have abbandoned the synched multi thread version as there was no effective advantage against the actual release (where we use threads but I cannot think we can realy say we "multi thread" :o )


Can anyone tell me how do you encode avs ? or where I can find them ?

M&M

[[Toff]]

This 1.14 version work great at least with 1 processor.
I have added a priority flag on my version, but as it is a small change, I'm waiting for other features to release it.

@mandm
You don't "encode" avs.
avs are script files that allow video processing.
To use avs script you must first install it http://www.avisynth.org

Take a video source on your hard drive, an AVI file (we call it "mymovie.avi"), you want to resize it before encoding. You create a text file with your prefered text editor :
AVISource("mymovie.avi")
BicubicResize(384,288)
Call it mymovie.avs
To preview the result open it in any player or virtualdub.
Then you can do your encoding :
avs2avi mymovie.avs resizedmovie.avi

You can open the avs file in nearly any application that support avi.
Look at the avisynth web site for more details it's worth a try.

[int 21h]

Small Update, I've got some primitive audio support working right now via BeSweet, having some issues with BeSweet's progress display and info records but it is producing valid mp3s from an audio stream specified in an avs file via AudioDub() and WavSource().

[[Toff]]

octapus :
I was wondering what those lines mean inside function
void InitializeDestination(const char* filename)

BITMAPINFOHEADER rgb = bmihVideoSource;
rgb.biCompression = BI_RGB; // Force to RGB ? (doesn't accept YUY2, or YV12)
rgb.biBitCount = 24; // Force 24 ? (doesn't accept 16 or 12)
rgb.biPlanes = 1; // Force 1 ?

I don't really understand myself why you need to use this settings even if you really use different parameters.
Even if you have YUY2 (16bits) AVIStreamSetFormat will fail with rgb.biBitCount = 16;

@int 21h:
Here is a link with the priority flag.
http://christophe.paris.free.fr/temp/avs2avi_priority.cpp
Does your audio mod. works with VBR MP3 ?

[int 21h]

Ok, updated to include that flag on the thread.

The audio system I have setup works with any parameter of BeSweet including VBR MP3 and many,many other settings :D

I will keep working on the audio stuff, we have to get DSPGuru's permission before using his .dll anyways, I've already messaged him, but am waiting for a reply.

Ok, from what I can see in Debug, the way this info thing is working seems totally wrong, I think its a dll error, so I'm just going to use my own progress indicator.

[MaTTeR]

Originally posted by int 21h

Multi-threaded, decode/encode/display are now seperate threads

Many many thx. I had no idea you guys would whip something together this fast. My time is limited right now unfortunately but I'm anxious to test it out this weekend.

[Emp3r0r]

like you said before people will have a hard time to replace trusty GKnot with another programm, especially when it's a .NET program. When the new GKnot is finished, everbody will use it..., but I know it is a lot hard work to code all the features people expect from the new versionWell, everyday things are getting easier. Most of the hard stuff is already done. The way I see it, it is now as simple as:

1) Select your DVD Drive
2) Select the streams
3) Get besweet settings or DSC source sound
4) Create avisynth script
5) Get codec settings
5) Run avs2avi

Now I imagine it must be kept as flexible as possible... something I always liked about gknot. All that needs to be done is the gathering of variables and using automation to the max. The next big thing will be a program that uses besweet, avisynth and avs2avi in an appropriate manner.

What's neat is that all these things get better everyday making everyone's life easier, including whoever creates the next great gknot.

[Back on topic]
I am so grateful for the updated version of avs2avi. Thanks [Toff] and int 21h. I'll test immediately. Anyone with some fps numbers on a 2CPU machine?

[int 21h]

New version. [Waiting for DSPGuru's ok for BeSweet.dll usage]

What's new

Can encode .wav files into VBR/CBR mp3 files via BeSweet.dll, just add the .wav file to your Avisynth script like this:

AudioDub(mpeg2source("C:\vobs\test.d2v"),WavSource("c:\vobs\test.wav"))

(AC3 support may come, it requires another parameter on the commandline, and some other changes, easier if Avisynth team implements a new source filter for it :) )

Fixed potential ETA bug that popped back up after AVS2AVI became multi-threaded.

LAME parameters for BeSweet encoding added to commandline, extended commandline is now avs2avi sourcefile destinationfile [flags], if flag -a is set, Audio will be processed, the following parameters can be set along with that flag:

--abr: ABR Encoding mode
-b: CBR low bitrate (or VBR low bitrate)
-B: VBR High Bitrate
-m: Mode (S,J,M,F)
--alt-preset: enables alt-presets
--preset: enables -presets
--r3mix: use r3mix
-V: VBR Quality (0-9)
-G: Gain for BeSweet to add (default is auto-find)


Two pass XviD with an mp3 would be

avs2avi xvid.avs xvid.avi -2 -a --alt-preset 192

ComingSoon: Hopefully muxxing support.

[[Toff]]

I have noticed a problem in "Ctrl-C" handling to stop the program during the process.
The display thread lock the program.
changing:
while(current_frame > 0)
to:
while(current_frame > 0 && !Stop)
must fix the problem.

[int 21h]

Ok fixed. I have permission from DSPGuru, but I am going to wait a little and see if he fixes the bug I found, and also his answer to another request.

[int 21h]

Originally posted by int 21h
ComingSoon: Hopefully muxxing support.

This will take awhile, mostly because I don't know the specifics of converting a file into the stream required for the AVI. And from what I can see on how AVIMux, Vidomi, and Nandub all do it, its very complicated.