Command Line Compressor - AVS2AVI

[DaveEL]

UPDATE: small bug fixed please download again.

Ok B-Frames code is now done.

I know it works on Divx5 and should also work on Xvid as that seems to produces the same output when it wants to drop an output frame.


This also "fixes" (make it act the same as virtualdub rather then a normal vfw application) b-frames even if you dont use the resume feature.

One small bug in the code currently is that at the end of the encode it should send the codec some blank frames to flush the codec of the last frames (currently they will not be in the output). I want to try and find some more information (im going to have to read the virtualdub source as i dont think its documented) on if the codec needs to be signaled in some other way that the source is finished.

http://www.ecs.soton.ac.uk/~del100/avs2avi.zip

DaveEL

[JimiK]

Hello, for me DaveELs version does not work in 2pass. And I posted a possible AVS2Avi-problem here:
http://forum.doom9.org/showthread.php?s=&postid=214686
Edit: Sorry, maybe it was not nice to say it that way. Great work, didn't mean to offend anyone.

Best regards,
JimiK
P.S. Pictures can be found a few posts before that one.

[DaveEL]

Originally posted by JimiK
Hello, for me DaveELs version does not work in 2pass.

Fixed i forgot to reset the bframe counter between passes.
http://www.ecs.soton.ac.uk/~del100/avs2avi.zip

DaveEL

[DaveEL]

Added support for flushing the codec after encoding bframes

http://www.ecs.soton.ac.uk/~del100/avs2avi.zip

DaveEL

[DaveEL]

Edit: Another bug fix please download again.

It now pauses at the last keyframe (not next) and so stops when you press ctrl-c.

DaveEL

[DarkNite]

So far, so sweet! Keep up the excellent work!

I can't wait to see avs2ogm ;) :D

[Wilbert]

Could you provide some documentation in the next version? Ie: how does it work (do you need VDubMod, it works with DivX5 and Koepi's built of Xvid (does it also work with other Xvid builts without doing any color conversions), where do you have to install it, what happens to the audio, etc.

[DaveEL]

Originally posted by Wilbert
Could you provide some documentation in the next version? Ie: how does it work (do you need VDubMod, it works with DivX5 and Koepi's built of Xvid (does it also work with other Xvid builts without doing any color conversions), where do you have to install it, what happens to the audio, etc.


If you read this forum post thats about all the documentation that exists appart from if you run it with no parameters it prints the usage information.


Basic usage: avs2avi [AVS Input File] [AVI Output File]

No it doesnt need any other applications.
It should work with any codec.
The Xvid Enhancements im working on now (not in the current build as they dont work yet) will require any xvid build since 23 April 2002.
install it anywhere you want.
encode the audio with what ever tool you want and mux it with any tool you want. (it doesnt do audio).


BTW i posted yet another new build to fix thread sync issues in that last build when you paused the encode.
Edit: still not fixed im trying something else now to try to make it work but its gonna take a little while (mainly cos im also doing the xvid enhancements on the same version).
If someone else wants to look at the code and try to find the problem please do (my brain hurts when i think about threads for too long).

DaveEL

[int 21h]

I'm still working on the audio support, its slowed down a little bit because I've not had much time, but I'll keep people updated on its progress.

[DaveEL]

Hey guess what..
yes thats right ANOTHER new build

i think i finally have the thread issues sorted.

And a little preview of two pass pause and resume.
Only currently does the first pass in xvid.

How to test
install an xvid build.
enable resuming and select first pass in xvid.
pause (ctrl-C) and resume as many times as you like.
When you finish you should have a correct stats file.
WARNING Dont use mv hints this will NOT work.

Ill work on the 2nd pass soon but:
1) i want a break
2) i want to make sure whats here works (such as the deadlock in the threads which was so much harder to fix as i id written a whole load of new code which i tried to debug but was working fine).

Also ill try to seperate out the xvid specific code and perhaps if i get time add divx 2 pass support as well (doesnt look hard).

DaveEL

[[Toff]]

About the thread lock problem I think that the best way is to use 2 writing loop.
The first loop just write the frame in synch with the read loop.
(it doesn't write bframe). It doesn't buffer frame.

The second loop is not synchronized with the reading thread. It write frames corresponding to the delay produce by bframe.

Using this way I don't think you need a buffer.

Then to add Ctrl+C handling you define a "safe" point in the writing loop where you test for the Stop flag and you break. (depending on where you put this test you will need to release the semaphore before breaking).


Currently with your version I have problem when I don't pause ;) some frame are missing.
I have only tested one pass encoding at the moment.
But when I pause/resume 1 or several time it work well.

Also could you increment version number, or at least give a number to your zip so we know about the version we talk.
You can also put your name or pseudo, if you want.

For my part, I have done some experimentation in the xvid code. I can call custom function using the vfw interface. So maybe it could be interesting to add some function to make xvid start at a defined point in the stat files. And add a function to save full codec stat (with 2 pass info).
But it's true that it will need to be added officially to be really used.

[DaveEL]

Originally posted by [Toff]
About the thread lock problem I think that the best way is to use 2 writing loop.
The first loop just write the frame in synch with the read loop.
(it doesn't write bframe). It doesn't buffer frame.

The second loop is not synchronized with the reading thread. It write frames corresponding to the delay produce by bframe.

Using this way I don't think you need a buffer.

Then to add Ctrl+C handling you define a "safe" point in the writing loop where you test for the Stop flag and you break. (depending on where you put this test you will need to release the semaphore before breaking).




Not sure i get what you mean here the thread locking was a problem whereby the stop even was read by the two threads at different points and so appropirate semaphores are not released. nothing to do with compression or saving of data. I think i can correct it by taking the semaphores before testing the stop variable.


Currently with your version I have problem when I don't pause ;) some frame are missing.


Hmm are you using b-frames and if so how many and how many frames are missing.



I have only tested one pass encoding at the moment.
But when I pause/resume 1 or several time it work well.

Very Strange that would expalin why i didnt see it i tested the pause and resume code dont think ive done a run through in one go yet.


Also could you increment version number, or at least give a number to your zip so we know about the version we talk.
You can also put your name or pseudo, if you want.

Will add info to next version

For my part, I have done some experimentation in the xvid code. I can call custom function using the vfw interface. So maybe it could be interesting to add some function to make xvid start at a defined point in the stat files. And add a function to save full codec stat (with 2 pass info).
But it's true that it will need to be added officially to be really used.
All you need is a variable to select to start position but its easy to do outside of xvid so i wouldnt (and didnt) bother.

DaveEL

[DaveEL]

Currently with your version I have problem when I don't pause some frame are missing.
I have only tested one pass encoding at the moment.
But when I pause/resume 1 or several time it work well.


Cant replicate that problem tried both with and without bframes.
Earlier version did have that problem with b-frames (did you upgrade between tests?)
Anything before i posted "Added support for flushing the codec after encoding bframes" suffered from that problem after thatShould be ok.

DaveEL

[[Toff]]

Hmm are you using b-frames and if so how many and how many frames are missing.
Encode in 1 pass without pause :
Xvid, 3 b-frame, input : 501 frame, output : 492 frame
It doesn't change anything if I change the b-frame number or even without bframe.

I don't really understand the buffering part?
Is it possible to do something like that without buffering :


void EncodeFrame()
{
...
// 1st loop
for (DWORD frameNumber = StartFrame;
frameNumber < SRCStreamInfo.dwLength && !Stop;
frameNumber++)
{
ICSeqCompressFrame(...)
if(not_delay_frame)
AVIStreamWrite(...)
else
BFrameNum++;
}

// 2nd loop (write delayed frame)
for (DWORD frameNumber = SRCStreamInfo.dwLength
frameNumber < SRCStreamInfo.dwLength+BFrameNum && !Stop;
frameNumber++)
{
ICSeqCompressFrame(...)
AVIStreamWrite(...)
}
}


Also estimated time has a strange behaviour. Not very important at the moment but I hope it doesn't impact on anything.

EDIT:
did you upgrade between tests?
The exe date in zip is 26/11/2002 18:00

[DaveEL]

Originally posted by [Toff]
Encode in 1 pass without pause :
Xvid, 3 b-frame, input : 501 frame, output : 492 frame
It doesn't change anything if I change the b-frame number or even without bframe.

Weird cant see how that would work if your on the current build like i say i cant make it do that here. The only time i had problems was an old build with serious threading issues (the first time i tried to rewrite the threading cos i couldnt make it work). Ill check the buffer is being flushed correctly at the end.

I don't really understand the buffering part?
Is it possible to do something like that without buffering :

The B-Frames and the buffering code are seperate.
The buffer allows me to stop at the last keyframe.

Im going to move the linked list buffer routines outside of that function so its easier to read.

Ill try to get all this done tomorow

Also estimated time has a strange behaviour. Not very important at the moment but I hope it doesn't impact on anything.


Its due to the buffer i think should be possible to fix i think it doesnt recalc some of the stuff till it commits from the buffer to the file.

DaveEL

[[Toff]]

Originally posted by DaveEL
The B-Frames and the buffering code are seperate.
The buffer allows me to stop at the last keyframe.


Ok I haven't think about that.
It's more clear in my head know.

I'm going to test another way.
You write all frame the normal way in destination file.

When you break the destination file look like this :
I P P P P P I P P P P
| break here


When you reopen it, FrameStart = last_KF (you can get last KF with AVIStreamPrevKeyFrame) and you overwrite the 4 old frame.

So that way you don't need buffering.

I will tell you if it work.

[[Toff]]

He ! He !
My last idea doesn't work :o

But I needed to try.

So yes we need buffering !


Im going to move the linked list buffer routines outside of that function so its easier to read.
Ill try to get all this done tomorow


Yes, I wait for tomorrow.

[jonny]

Hi

I've problems with the "-l" option (tested on 1.13 and 1.15 with same results).

I'm using a simple avs script (containing only the Version() function)



This is what i do:



avs2avi ver.avs ver.avi -s 1p.bin
i select DivX 5.02
i set Quick Config CLI with:

-b21 800 -key 250 -log "C:\divxpass.log" -mv "C:\mvinfo.bin0" -dr 12,2,2000,10,20 -sc 50 -pq 5

This do a 1st pass encode.


avs2avi ver.avs ver.avi -s 2p.bin
i select DivX 5.02
i set Quick Config CLI with:

-b22 800 -key 250 -log "C:\divxpass.log" -mv "C:\mvinfo.bin0" -dr 12,2,2000,10,20 -sc 50 -pq 5

This do the 2nd pass encode.


The file ver.avi is created correctly.


Now the problem, i want to make the 1st, then the 2nd pass using 1p.bin and 2p.bin:

avs2avi ver.avs ver.avi -l 1p.bin

A window appear with this message:
"Could not open log file for reading"
I think this is a DivX Codec message... it seems that the parameters are not set correctly.
avs2avi quit with this message:
AVIStreamSetFormat failed : Impossibile trovare il file specificato.

"Impossibile trovare il file specificato" in english mean: "unable to find the specified file"

Is this a bug or i'm missing something?

Cheers
jonny


PS: parameters saved in 1p.bin and 2p.bin seems to be generated correctly, i've compared the codec data with the data generated by my DivX5Enc and are equal.


Edit: i haven't this error all the times, sometimes the 1st pass go and the second give me errors...

[DaveEL]

Ok new build up should fix the missing frames problem (the frames were missing if you didnt ask it to enable resuming).

Code should be easier to read now (new files xvid.cpp and xvid.h for xvid specific stuff and moved the linked list code.)

DaveEL

[[Toff]]

Yes problem fixed here.

So if I resume now we can pause/resume :
- Encoding done in 1 pass
- Encoding done in 2 pass but only during 1st pass and without the -2 flag

Also we have a little jumping frame counter problem caused by buffering.

[DaveEL]

Originally posted by [Toff]
Yes problem fixed here.

So if I resume now we can pause/resume :
- Encoding done in 1 pass
- Encoding done in 2 pass but only during 1st pass and without the -2 flag

Also we have a little jumping frame counter problem caused by buffering.

Small problem with the xvid code change mmioFourCC('X','V','I','D') to mmioFourCC('x','v','i','d') in xvid.cpp and then it should work.

sorry

im working on (and nearly done i think) the second pass. Give me a couple of days (or possibly till next week as my birthday is soon so im gonna be very drunk this weekend :) )

Need to look at the frame counter stuff.

DaveEL

[[Toff]]

Hey ! Hey ! no problem, take your time...

I was just summarizing so others people don't say "hummm! there is a bug here it doesn't work ..."
But if it's not implemented it can't work :D

Need to look at the frame counter stuff.
I have 2 idea/solutions to try to solve this problem.
- Use a 2nd frame counter that is increased when we read and use this counter for dynamic display.
- Use the old way wait for KF with no buffering.

Happy birthday Dave ! :p

[DaveEL]

Originally posted by [Toff]

I have 2 idea/solutions to try to solve this problem.
- Use a 2nd frame counter that is increased when we read and use this counter for dynamic display.
- Use the old way wait for KF with no buffering.



Done in my codebase earlier today. change the value of frames done and output size when things are added to the buffer rather then when saved to the avi stream. will be in my next release. :)

DaveEL

[DaveEL]

Ok as i spent so much time fixing the threading code i thought i should make it work properly. So can someone with a multiprocessor machine test to see if this is faster for me

http://www.ecs.soton.ac.uk/~del100/avs2avi.zip

DaveEL

[DaveEL]

version 1.20 http://www.ecs.soton.ac.uk/~del100/avs2avi.zip


*Fixed Threads AGAIN (ok so i broke them but that's not the point).
*Implemented resuming on xvid 2nd pass (internal only no mv hints file).

Ill work on getting the -2 option working with resuming soon.

DaveEL

[Dark-Cracker]

@DaveEL

very good work the possibilty to resume the encoding process is a damn good idear :) good luck for the support of the hintme function.

[[Toff]]

Originally posted by DaveEL
Fixed Threads AGAIN (ok so i broke them but that's not the point).

I see a little problem here using xvid, 1 pass encode, 3 bframe.
The encoding thread lock the program at end. (I just see "done read", but the program continue).

Also why SRCData is now an array ? Is it for threading issue ?

I will try to look at it this weekend.
No, don't look at it it's your birthday ;)

[DaveEL]

Originally posted by [Toff]
I see a little problem here using xvid, 1 pass encode, 3 bframe.
The encoding thread lock the program at end. (I just see "done read", but the program continue).

Also why SRCData is now an array ? Is it for threading issue ?

I will try to look at it this weekend.
No, don't look at it it's your birthday ;)

Damm the threads still lock. Ill take another look i HATE threads.

by making SCRData an array on multi processor system it should be a lot faster as one thread can decode while the other encodes.


DaveEL

[DaveEL]

ok this *should* work

Change
while (!ReleaseSemaphore(hSemWaitWrite, 1, NULL) && (frameNumber < SRCStreamInfo.dwLength+BFrameNum) && !Stop) Sleep(100);

to
while (!ReleaseSemaphore(hSemWaitWrite, 1, NULL) && (frameNumber < SRCStreamInfo.dwLength) && !Stop) Sleep(100);

in EncodeFrame.

DaveEL

You might also want to make the sleep shorter needs some testing to find a good value.

[[Toff]]

When you look in details you can see some frame ordering problem with xvid (25112002).
It works pefectly only if I check "Packed bitstream" & "DX50 B-VOP compatibility".
You can easily see the problem using avisynth ShowFrameNumber function.

But I don't know if it's an encoding or decoding problem.

I've quickly tested the 2nd pass pause/resume, it works here.

Also I've simplified a bit (I think) the thread synchronization.

http://christophe.paris.free.fr/temp/avs2avi_121.zip

[Emp3r0r]

IIRC, the older version allowed you to run avs2avi to just save codec settings. Now when you do the "-s out.txt", it also encodes. Is there anyway to run avs2avi to just save codec settings?

[Beave]

HI, I always get this error:

AVIFileOpen failed to create destination file "d:\...avi"

I tried with avs2avi 1.20, 1.21, 1.13 with gui on 2 different computers with the included test.avs files. it happenes after i choose the codec and the commandline window opens. It only prints the avs2avi infos until
Destination:
* Filename: D:\DVD (...).avi

What could that be?

[DaveEL]

Emp3r0r : just press ctrl-c when it starts encoding :)

Beave : What is the command you are running?

Edit: whoops didnt read the gui bit
sounds like the gui isnt quoting paths with spaces in them. Ttry using a file with no spaces in the path.

DaveEL

[Beave]

ok, I tried with avs2avi the following lines:

D:\DVD\avs2avi\bin>avs2avi test.avs test.avi
D:\DVD\avs2avi\bin>avs2avi.exe test.avs test.avi
D:\DVD\avs2avi\bin>"avs2avi.exe" "D:\DVD\avs2avi\bin\test.avs" "D:\DVD\avs2avi\bin\test.avi"

They are all not working. After the Videocompression window pops up i choose divx 5.02 in quality mode (tried xvid and cinepak as well), click ok and the command line window says:

AVIFileOpen failed to create destination file D:\DVD\avs2avi\bin\test.avi : Das System kann die angegebene Datei nicht finden.

Do I need additional options? Do I need anything else installed? I have avisynth 2.07 on one of my computers and 2.5 alpha on the other. Both report the same thing.

[DaveEL]

Originally posted by Beave
ok, I tried with avs2avi the following lines:

D:\DVD\avs2avi\bin>avs2avi test.avs test.avi
D:\DVD\avs2avi\bin>avs2avi.exe test.avs test.avi
D:\DVD\avs2avi\bin>"avs2avi.exe" "D:\DVD\avs2avi\bin\test.avs" "D:\DVD\avs2avi\bin\test.avi"

They are all not working. After the Videocompression window pops up i choose divx 5.02 in quality mode (tried xvid and cinepak as well), click ok and the command line window says:

AVIFileOpen failed to create destination file D:\DVD\avs2avi\bin\test.avi : Das System kann die angegebene Datei nicht finden.

Do I need additional options? Do I need anything else installed? I have avisynth 2.07 on one of my computers and 2.5 alpha on the other. Both report the same thing.

hmm weird looks ok to me.
1) what version of windows are you running
2) does it work with other avs files rather then the test ones?

DaveEL

[Beave]

I'm using Win XP SP1. On my second computer I even have a fresh install. I tried with other avs as well, same thing.
I even tried with a windows98 version running inside VMWare.

here is the full window:

AVS2AVI v1.13 (c) 2002 Christophe PARIS (christophe.paris@free.fr)
Contributions by: int21h

Thanks to Avery Lee author of VirtualDub and Chia-chen Kuo author
of DVD2AVI for providing great source code of inspiration.

Source:
* Filename: D:\DVD\avs2avi\bin\testlong.avs
* Bit depth: 32
* FOURCC: none (RGB)
* Frames: 650
* Resolution: 320x240
Compressor:
* Name: DivX Pro 5.0.2 Codec
* FOURCC: DIVX
Destination:
* Filename: D:\DVD\avs2avi\bin\test.avi

AVIFileOpen failed to create destination file D:\DVD\avs2avi\bin\test.avi : Das
System kann die angegebene Datei nicht finden.

[[Toff]]

I'm not a divx5 user and I haven't made more test before today where I found there is really problems with divx5.
I've try a lot of thing (having only 1 instance of the codec at one time), but it won't work for 2pass in loading settings mode.
I must miss some initialization things somewhere.

But at least it work well with 1 pass encode.


Now when you do the "-s out.txt", it also encodes. Is there anyway to run avs2avi to just save codec settings?


Yes I can be done. I just want to (try to) fix divx5 problem first.

[Beave]

Same thing happenes with XVID as well, so its not only a DiVX problem. Acually none of the codecs work here so far.
What I find really weird, is that it even doesn't work on my other machine but no one else seems to have a problem. So how come both my computers don't work, but all the others work fine?

Why is the program acually asking for the avi file, if it isn't created yet?

[Emp3r0r]

Beave, I have found that problem to only occur when something else is also using the codec. Do you have virtualdub going or something else?

[DaveEL]

New release been meaning to do this for a while but had loads of work to do.

You can now resume encodes when doing 2 pass (with -2)

http://www.ecs.soton.ac.uk/~del100/avs2avi.zip


BTW im not sure xvid + resuming + 2 pass + bframes all work together (any combination except all is fine) as the stats file format seems to have changed slightly for bframes. Ill try to look into this in the new few days. Then ill start bothering the xvid team to change xvid config struct so i can do:
1) mv hints
2) 2nd pass external mode.
with the resuming support.


After that im gonna try to tidy up the code a bit (remove the globals) so i can implement an idea i saw on the xvid developer mailing list about doing very fast encoes on dual cpu boxes. (should be faster then the normal approach of doing the encode on one cpu and decode on the other like we do now).

DaveEL

[Emp3r0r]

I can't tell you how much I appreciate avs2avi. Here is what I came up with as part of the program I am working on. Image attached

[mfluder]

Hi,

I updated my GUI to support new avs2avi features. I attached it to this post.

mfluder

[sh0dan]

@Beave: Delete resume.codec - for some reason resume seems to be enabled per default.

[DaveEL]

Yes the key resuming is enabled is a bit weird (if you want to start a resumable encode you use -r and then it auto resumes after that)

I think it would be better to change it to this if noone minds


All encodes are started with resuming support (if you dont pause it resuming should make no difference to the encode).

if you specify -r it will resume the previous encode.

Additionally in the .codec file i think the file names should be stored but they can be overiden on the command line

So you end up with:

to start an encode

avs2avi inputfile outputfile [options]

To resume that encode later

avs2avi -r


Also -s should not encode just save the parameters
Loading support could then be either
avs2avi -l file.codec
which will use the options and filenames you used to save the codec file
or
avs2avi inputfile outputfile -l file.codec
which will use the files on the command line and the options from file.codec


DaveEL

(Emp3r0r: I see your devloping in c#, have you worked out how to send SIGINT (Ctrl-c) to avs2avi yet? I cant seem to find any way to do it)

[Beave]

hmm, i can't find any resume.codec on my system.
I just unzipped all those files in avs2avi.zip and run them on the test files and on some other *.avs I had on my system.
There has never been any setting files or something simular created in the avs2avi directory. I also searched the rest of my hd for "resume".

Any more Ideas?

On my new system i actually only installed all of GKnot and the Divx Codec plus OggDS and XCD Filters. Maybe one of those things collide with avs2avi?

[Emp3r0r]

I was thinking of a way to do this while reading your post and it came to me right before I read Originally posted by DaveEL

(Emp3r0r: I see your devloping in c#, have you worked out how to send SIGINT (Ctrl-c) to avs2avi yet? I cant seem to find any way to do it) I haven't tried it yet but I'm assuming you just create a StreamWriter and then pipe it to the StandardInput of the Process.

anyway I hook a System.IO.StreamReader to System.Diagnostics.Process.StandardOutput to read the console. I'll play with sending "ctrl-v" by possibly creating a System.IO.StreamWriter and then writing to System.Diagnostics.Process.StandardInput

[Emp3r0r]

hmm, i can't find any resume.codec on my systemI think you can create that file manually using -s "resume.codec". I haven't tested but I think the resume.codec is created automaticaly when you use -r and pause the encoding.

[seewen]

I've tried many times to re-encode DIvX (for Kiss DP-450), in 2-pass Mode with DivX 5 Pro.

It works very weel @ 75%

But the 25% left, The process stops after the 1st pass (it's blocked, but the computer continue to work hard).

Someone have an idea about this (I use always the same method. And I use only the cmd line, no GUI ) .

Thanks

EDIT : I use AVS2AVI 1.21, only with "-2" cmd.

[MaTTeR]

Originally posted by DaveEL
So can someone with a multiprocessor machine test to see if this is faster for me On my AMD dualie the speed seems and CPU usage seems to be almost on par with VirtualDub. I haven't had a chance to do any "real" benchmarking yet but first results seem very promising to say the least. Excellent work guys!

The only problem I'm having with a few of the versions here is that avs2avi is locking up during the first pass for some reason. CPU usage reamins very high but no encoding is actually taking place according to Debug View. At some point during the 1st pass of XviD the CLI is reporting bogus ETA and FPS information, at the start of the encode it seems to be just fine though. I'll report back more after some further testing this week. Many thx again for all the efforrts here:)

[Beave]

I tried with -s resume.codec. The only difference is that it now won't ask for the Codec Settings. The errormessage:
AVIFileOpen failed to create destination file test.avi
remains the same.

If I delete it again it will pop up a window and ask for the Codec, that's all. Error stays the same.

Is there anything written to the registry?
Maybe somebody could give me a debug version of it, so I could test it?