parallel encoder beta6a: The fastest encoder in town

[708145]

I'd like to announce a new release of ELDER (http://www.funknmary.de/bergdichter/projekte/index.php?page=ELDER), a parallEL encoDER.

Current state (ELDER_beta6b.exe (http://www.funknmary.de/bergdichter/projekte/ELDER/ELDER_beta6b.exe)) includes:
* a GUI for all modes
* SMP and windows cluster support (distributed encoding)
* xvid 2pass
* xvid 2pass with improved rate control (qcomp)
* xvid with crf parameter (will do a 2pass with same bit distribution as x264)
* x264 1pass crf
* x264 1pass abr
* x264 2pass
* avisynth input
* mp4 output

You'll need PXPerl. There is a copy of the original page (http://www.bumpersoft.com/Programming/Interpreters_and_Compilers/D_10987_index.htm) up here: PXPerl-5.8.7-6.exe (http://www.funknmary.de/bergdichter/projekte/ELDER/PXPerl-5.8.7-6.exe)

bis besser,
T0B1A5

[bond]

sounds very interesting. is there a speed/quality tradeoff and if yes, how big is it (any values?)

[708145]

sounds very interesting. is there a speed/quality tradeoff and if yes, how big is it (any values?)

If you follow the links, you'll find a bit of info.
As the RC is very simple for now, quality should be lower with this alpha.

edit: As of version alpha4 the size is hit quite accurately and the quality drop is very small. But please test for yourself :)

This will be fixed for the betas. I just wanted to have a working version out ASAP so you guys can start messing with it.

For speedups I wait for ppl testing it and telling me. I have access to SMPs and clusters running Linux, but I can't test on Windows SMP or multicore.

My guess: speedup should be _very_ linear, like 1.9x-fold on 2 cores and 3.9-ish on 4 cores.

bis besser,
T0B1A5

[708145]

Hey guys. It's been over a week and I got almost no feedback!

You don't need HT or SMP to test it. Just download and run it with your favourite avs script.

But for sure I'd be interested in speedup results of HT and SMP systems as well :P

bis besser,
Tobias

[scb]

Hi 708145!
Thank you in advance for Elder for win32!!! Would love to test the alpha version on my windows/smp setup but I have no idea how to install the thing :\

Would you (or some other kind person) let me know exactly how to get it installed, set it up and run it? I promise to report the results here.

tia!

[unskinnyboy]

Hi 708145!
Thank you in advance for Elder for win32!!! Would love to test the alpha version on my windows/smp setup but I have no idea how to install the thing :\

Would you (or some other kind person) let me know exactly how to get it installed, set it up and run it? I promise to report the results here.

tia!
I haven't tested ELDER4XVID myself, but from what I can see it isn't any difficult. The downloadable file is in .tgz format which is Unix Gzip/Tape Archive and can be decompressed by the latest WinRAR if you have it installed. Once you open the folder ELDER4XVID_alpha1 inside the .tgz, read the file README.txt which should answer your questions. Nothing to install. Extract the contents of the archive into a folder.

[scb]

thanks unskinnyboy

Got it. OK....reading through the readme file...still need help.

* needs a PERL interpreter to run; tested under cygwin...do I just google PERL cygwin, find it and install it? Does it have an installer?

Have been using GK, never used mencoder, so have some problems there.

I figure some of the ELDER parameters can be had from the *.avs via GK.

Here are the parameters that I'm unsure about:
-f number of frames; this is temporary as it can be obtained using the avs (gk?)
-c number of chunks; should be set a bit larger than maxpar<-- huh?
-p 1st pass options to mencoder<-- no idea
-q 2nd pass options to mencoder; must end with "bitrate="<-- no idea

Pretty sure (somewhat sure? hmmm...no idea) that the aspect ratio would be part of mencoder options? As above, I've no idea how to use mencoder so have no idea how to set aspect ratio to desired target of 1.333. Also, how/where do I set the output size to the desired 512x384?

As audio conversion needs to be done separately, is there a SMP-capable audio converter anyone knows of? (this is not ELDER-related per se, just me trying to get faster conversion)

tia for any help! (apologies if my alpha-teste newbieness grates)

[unskinnyboy]

* needs a PERL interpreter to run; tested under cygwin...do I just google PERL cygwin, find it and install it? Does it have an installer?

I suppose so. Elder.pl, as you can see, is written in Perl and Perl is an interpreted programming language, so you need a Perl interpreter. Cgywin is the environment Tobias tested this in. You need not have Cygwin installed.


Here are the parameters that I'm unsure about:
-f number of frames; this is temporary as it can be obtained using the avs (gk?)
-c number of chunks; should be set a bit larger than maxpar<-- huh?
-p 1st pass options to mencoder<-- no idea
-q 2nd pass options to mencoder; must end with "bitrate="<-- no idea

You will understand this better if you look at the example in the README:

* encode blafu.avs with 30000 frames to a 10MB XviD using 32 chunks and 2 CPUs:
./ELDER.pl -a blafu.avs -c 32 -m 2 -b 10485760 -f 30000 -p "- -ovc xvid -xvidencopts pass=1" -q "- -ovc xvid -xvidencopts pass=2:bitrate=" -o blafu.avi

-f --> you can get this by loading the avs into VirtualDubMoD, pressing Ctrl+-> and going to the end of the file. If you want to encode till the end of your movie, then take the last frame number as the input to -f.
-c --> to divide the movie into chunks while encoding to enable parallelization, I *think*.
-p --> your first pass encoding options. If you look at the example above, it is "- -ovc xvid -xvidencopts pass=1".
-q --> your second pass encoding options. As per the example, it is "- -ovc xvid -xvidencopts pass=2:bitrate=". As you can see, it ends with "bitrate=".

To know more about the encoding options & to answer your aspect ratio questions, check the MPlayer documentation (http://mplayerhq.hu/DOCS/man/en/mplayer.1.html#CODEC%20SPECIFIC%20ENCODING%20OPTIONS%20(MENCODER%20ONLY)).

[scb]

thanks again unskinnyboy

yeah...checked the example before my last post. Figured I could get the -f value from gk but you're method would work perfectly, too.

Thanks also for the mencoder doc link :)

ok...so....
Grab and install ActivePerl here: http://www.dzsoft.com/getperl.htm (the 'With MSI Installer' version, I'd guess?)

Unpack all the alpha version files to it's own directory

Rip dvd to h/d, open prompt, type command (still trying to figure out the exact command), hit 'enter' and let it run?

Question: Will ELDER/mencoder automatically find everything it needs for the XviD codec or do I have to put a file into the ELDER folder?

Regarding selection of the AR=1.333 and 512x384 issues....this looks like something I'd need to add to the -p and -q parameters (?). Found this in the mencoder doc (thanks for the link):
"aspect=<x/y | f (float value)>
Store movie aspect internally, just like MPEG files. Much nicer solution than rescaling, because quality is not decreased. MPlayer and a few others players will play these files correctly, others will display them with the wrong aspect. The as- pect parameter can be given as a ratio or a float- ing point number."

Looks like I'd need to add aspect=1.333 somewhere in the mencoder parameter string (?)

For the sizing, looks like I'd also add this to the mencoder parameters:
-vf scale=512:384

Took a look at videohelp.com's page in hopes of finding a guide but nada on mencoder.

Ugh....hate being too tech-illiterate to just set it all up and get to the testing part :(

It'd be much appreciated if someone better (more experienced) than me with this package could just give me the info to get going on this....

Other notes....on the project page, known issues:
'fps is hardcoded to 25 (will be fixed in alpha2)'....does this mean I'd get a max of 25fps per cpu (50-ish fps total, possibly 98fps for the 2cpu with HT-enabled)? Am currently getting around 77fps 1st pass, 58fps 2nd pass using GK.

'in the input avs there needs to be a single "#trim()" line to tell ELDER where to fill in trim commands (fixed in alpha2)'....so need to open *.avs in notepad and add this parameter at the top? What (if anything) do I need to put in the ()?

Anyway, again, if anyone feels like writing up a quick guide for 'How to install and test ELDER4XVID alpha for windows', it'd be SOOOO appreciated!!

I'm now gonna go take a couple aspirin and lay down for a little while hehe

[708145]

thanks unskinnyboy for the explanations :)


yeah...checked the example before my last post. Figured I could get the -f value from gk but you're method would work perfectly, too.


The -f is becoming obsolete in alpha2 since I learned how to get that info from the avs.

edit: unfortunaltely this is harder than I thought... so bear with me as I'm trying to solve that issue.


Grab and install ActivePerl here: http://www.dzsoft.com/getperl.htm (the 'With MSI Installer' version, I'd guess?)

Rip dvd to h/d, open prompt, type command (still trying to figure out the exact command), hit 'enter' and let it run?

Question: Will ELDER/mencoder automatically find everything it needs for the XviD codec or do I have to put a file into the ELDER folder?


With active perl I guess you have to remove all "./"s from the script. Apart from that your plan sounds right :)

no extra xvid needed, it's built-in in mencoder.exe


Looks like I'd need to add aspect=1.333 somewhere in the mencoder parameter string (?)

For the sizing, looks like I'd also add this to the mencoder parameters:
-vf scale=512:384


set aspect as mencoder option.
I'd suggest resizing in the avs script. So you could use GKnot to set up the avs for you.


Ugh....hate being too tech-illiterate to just set it all up and get to the testing part :(
It'd be much appreciated if someone better (more experienced) than me with this package could just give me the info to get going on this....


I try :P


Other notes....on the project page, known issues:
'fps is hardcoded to 25 (will be fixed in alpha2)'....does this mean I'd get a max of 25fps per cpu (50-ish fps total, possibly 98fps for the 2cpu with HT-enabled)? Am currently getting around 77fps 1st pass, 58fps 2nd pass using GK.

'in the input avs there needs to be a single "#trim()" line to tell ELDER where to fill in trim commands (fixed in alpha2)'....so need to open *.avs in notepad and add this parameter at the top? What (if anything) do I need to put in the ()?


OK. I'll publish alpha2 later which doesn't have this restriction.
You don't have to care about the trim() and not about the fps.


Anyway, again, if anyone feels like writing up a quick guide for 'How to install and test ELDER4XVID alpha for windows', it'd be SOOOO appreciated!!

I'm now gonna go take a couple aspirin and lay down for a little while hehe

and about the guide: I try to make this whole program much more user friendly. Some things (like the -f) were just for my convenience so I could release ELDER sooner.

testing is tough :D

bis besser,
Tobias

[708145]

The -f is becoming obsolete in alpha2 since I learned how to get that info from the avs.

OK. I'll publish alpha2 later which doesn't have this restriction.
You don't have to care about the trim() and not about the fps.


I'm sorry I couldn't get rid of the -f option.
I'll write a proper changelog tomorrow but as you can see some issues have been ironed out already with alpha2.

The download link is in the first post of this thread.

bis besser,
Tobias

[scb]

Hi 708145

Which PERL interpreter do you recommend to try?
I've never done any real avisynth stuff so not entirely sure about how to edit one (except that I know they open in notepad), what the resize command would be, etc, etc (guess I could read up on it, though).

Will try to make a first go at it this weekend.

I'm just thrilled by the prospect of finally being able to take advantage of smp for xvid! Thank you 708145!

PS- If anyone out there would care to make an actual guide, it'd be SOOOO helpful ;)

[708145]

Hi 708145
Which PERL interpreter do you recommend to try?
I've never done any real avisynth stuff so not entirely sure about how to edit one (except that I know they open in notepad), what the resize command would be, etc, etc (guess I could read up on it, though).


Well I recommend cygwin since it allows to run Unix-style scripts. But I guess any PERL interpreter would do (if the "./" were deleted from the script).
The reason why I like to leave them in is that I want the same ELDER script to run on both linux and windows.

About the avisynth: You can use GKnot and let it generate the avs for you.

bis besser,
Tobias

[708145]

As audio conversion needs to be done separately, is there a SMP-capable audio converter anyone knows of? (this is not ELDER-related per se, just me trying to get faster conversion)


I intend to add audio encoding at a later stage once video is working.

bis besser,
Tobias

[stephanV]

I've created a little guide which (hopefully) explains the settings and what programs are needed to run ELDER4xvid.

http://www.geocities.com/grotesteph/ELDER/guide.html

have fun

[Kostarum Rex Persia]

Wow,very good work, stephanV, keep going. Thank you for that good guide.

[708145]

I've created a little guide which (hopefully) explains the settings and what programs are needed to run ELDER4xvid.
http://www.geocities.com/grotesteph/ELDER/guide.html


Thank you very much :D

bis besser,
Tobias

[708145]

I just released alpha3! :D

alpha3:
* avimerge.exe added; merging step activated
* distribution style changed: one part all exe files the other the scripts
* changelog added
* comments added
* warning if chunks are too small added
* size and bitrate options added

bis besser,
Tobias

[DarkNite]

Nice additions, will test if they're broken or not as soon as current encode is finished.

[708145]

Nice additions, will test if they're broken or not as soon as current encode is finished.

How did you know it's broken :]
I fixed a few things in alpha4... at least in runs fine for me.

alpha4:
* sometimes wrong location for trim(): fixed
* StageC didn't start: fixed
* automatic selection of number of chunks if not specified
* delete intermediate files

bis besser,
Tobias

[708145]

alpha5 is up.

alpha5:
* modify perl for compatibility with Active Perl
* OS dependent defaults
* added cygwin1.dll to the exe bundle; needed by avimerge.exe
* added x264.exe to the exe bundle; needed to get framecount
* correct handling of pathes on both linux and windows
* get number of frames from avisynth; option -f not needed anymore

It will not use more than one thread currently :(
But please test for correctness. Thanks.

bis besser,
Tobias

[708145]

just released beta1:
* fixed bug when trim() was used in the input avs
* use xvid_encraw for 1st pass (stage A and stage B)
* consistent naming for temporary files
* parallelism broken

bis besser,
Tobias

[IceM]

Verry promising thread title i must say :)

My first question before i begin testing this application (which wont be today caus of a lack of time, but will be on a short notice), is the output result similar as which you would get with the same encode in VirtualDub?

[708145]

Verry promising thread title i must say :)

My first question before i begin testing this application (which wont be today caus of a lack of time, but will be on a short notice), is the output result similar as which you would get with the same encode in VirtualDub?

Output should be (almost) the same. If it is really the case testing is there for :P
The bit distribution should also be very similar to what xvid 2 pass does, but I can't guarantee it is the same. Some early testers even claimed that bit distribution would be better than default xvid :D But again: testing needed on this issue.

beta2 will output to mp4 instead of avi (which beta1 does) => a difference to Vdub.
Don't wait for beta2 for testing though since it will take me some time to implement all the changes I plan for this release :)

bis besser,
T0B1A5

[708145]

Don't wait for beta2 for testing though since it will take me some time to implement all the changes I plan for this release :)


beta2:
* added installer
* muxing to mp4
* show progress
* debug output suppressed
* compile perl scripts on windows or ship perl.exe with ELDER
* improved defaults for chunksize

bis besser,
T0B1A5

[Badness430]

perl ELDER4XVID.pl -a test.avs -m 4 -s 1297055000 -o test.mp4

Im using this and I still only get 25% CPU usage.

System:
Dual Xeon 3.6ghz
Windows XP Pro x64
2 gb ddr2 memory.

Xeon 3.6's are dual core, so when I look in performance, I see the 4 cpu columns, but only one is in use, hence the 25% cpu usuage. What am I missing =)

[708145]

perl ELDER4XVID.pl -a test.avs -m 4 -s 1297055000 -o test.mp4

Im using this and I still only get 25% CPU usage.

System:
Dual Xeon 3.6ghz
Windows XP Pro x64
2 gb ddr2 memory.

Xeon 3.6's are dual core, so when I look in performance, I see the 4 cpu columns, but only one is in use, hence the 25% cpu usuage. What am I missing =)

Well with alpha5 parallel execution on windows broke :( and is still not fixed.
I think other things are more urgent... like testing for usability and output quality.
Trust me: I'll take care of the parallel scaling soon. It works on msys, cygwin and native linux. Just windows doesn't. :rolleyes:

Does it produce correct output? Are you happy with the bitrate distribution?

bis besser,
T0B1A5

[Badness430]

Well with alpha5 parallel execution on windows broke :( and is still not fixed.
I think other things are more urgent... like testing for usability and output quality.
Trust me: I'll take care of the parallel scaling soon. It works on msys, cygwin and native linux. Just windows doesn't. :rolleyes:

Does it produce correct output? Are you happy with the bitrate distribution?

bis besser,
T0B1A5

I'll have run a full test. When I saw the only %25 I stopped encoding. Regular virtualdubmod usually gives me 45-50% doing what I dont know. I was looking for something to give me full use of my Xeons

[708145]

I'll have run a full test. When I saw the only %25 I stopped encoding. Regular virtualdubmod usually gives me 45-50% doing what I dont know. I was looking for something to give me full use of my Xeons

I understand. Avisynth is being run in the same process as xvid in ELDER => 1core. It will actually give better scaling when it finally works.

I started hacking in preliminary || support for windows... hang on.

bis besser,
T0B1A5

[708145]

I started hacking in preliminary || support for windows... hang on.


I feel dirty due to this hack... but it works here.

beta3:
* parallelism fixed

bis besser,
T0B1A5

[Badness430]

I feel dirty due to this hack... but it works here.

beta3:
* parallelism fixed

bis besser,
T0B1A5

OK. I'm impressed now. My system is kicking fully percent on both CPU's and both cores.

Processing 2007 frames on 4 threads in 10 minutes. So thats about 8000 frames in 10 minutes. Wow. Typical HD tv show, which is what I mess with a lot, is 60000 frams on avg. At this rate this show will only take about 75 minutes.

When its done, I'll post quality comparison results and how everything finished up tomorrow.

[Sharktooth]

Impressive... i just started a test encode on an athlon x2 3800+ and the FPS are almost doubled!
more info when it's done.

[Badness]

OK. I'm impressed now. My system is kicking fully percent on both CPU's and both cores.

Processing 2007 frames on 4 threads in 10 minutes. So thats about 8000 frames in 10 minutes. Wow. Typical HD tv show, which is what I mess with a lot, is 60000 frams on avg. At this rate this show will only take about 75 minutes.

When its done, I'll post quality comparison results and how everything finished up tomorrow.

OK. Where does it put the file. I did a search accross my whole system and I dont see the file indicated in the -o switch.

D:\Work>perl ELDER4XVID.pl -a rome.avs -m 4 -s 1297055000 -o rome.mp4
detected fps = 23.98
detected frames = 68197
using 4 threads to encode 34 chunks with approx. 2006 frames each.
Total size will be 1297055000 bytes.
### starting process 0
### starting process 1
### starting process 2
### starting process 3
### Stage A spawned

Finished up with this.
### Stage C completed
### chunks merged
### temporary folder deleted
### another job well done

Normal chunk processing in between, didnt see the need to paste all that.

But I dont see the output file anywhere. I did a search accross my whole system to be sure and its not there.

[708145]

OK. Where does it put the file. I did a search accross my whole system and I dont see the file indicated in the -o switch.

D:\Work>perl ELDER4XVID.pl -a rome.avs -m 4 -s 1297055000 -o rome.mp4

But I dont see the output file anywhere. I did a search accross my whole system to be sure and its not there.

It *should* end up in the same dir where you issued the command. But I guess I know why it failed.
I pass the merge command to mp4box via commandline and it might well be that it is too long since there is a very tight character limit in cmd.exe.
Could you do a quick test for me?
add -c 8 -f 8000 to your commandline. It will do a small part of your clip and then mux. I hope it works then. If yes it is very likely that we can blame cmd.exe once again :)

bis besser,
T0B1A5

[Badness]

If I add a path on with the -o switch it comes out. Otherwise I'm not sure where it is going =) Maybe it is going into the sub folder that gets created and then it gets deleted with the temp files.

[Badness]

If I add a path on with the -o switch it comes out. Otherwise I'm not sure where it is going =) Maybe it is going into the sub folder that gets created and then it gets deleted with the temp files.

Well, I take that back. It worked once on a small clip but I havent been able to get it to work since.

D:\Work>perl ELDER4XVID.pl -a thresh.avs -m 4 -s 593691000 -o d:\thresh.mp4

Should drop a file in the root of my D Drive but no go =(

[708145]

Well, I take that back. It worked once on a small clip but I havent been able to get it to work since.

D:\Work>perl ELDER4XVID.pl -a thresh.avs -m 4 -s 593691000 -o d:\thresh.mp4

Should drop a file in the root of my D Drive but no go =(

did you try "-c 8" yet?
I tested up to "-c 20" and it worked.

bis besser,
T0B1A5

[Badness]

did you try "-c 8" yet?
I tested up to "-c 20" and it worked.

bis besser,
T0B1A5

Running it with -c 8 now. Ill post results in the morning.

[Badness]

Running it with -c 8 now. Ill post results in the morning.

OK, it worked with the -c 8.

Now I have tried to load the resulting file into virtualdubmod and avimux and they wont accept the file. How can I mux the audio into the file?

[708145]

OK, it worked with the -c 8.


Good to hear that. Can you play it correctly? I ask because mp4box messed up some of my test encodes.


Now I have tried to load the resulting file into virtualdubmod and avimux and they wont accept the file. How can I mux the audio into the file?

The output is mp4. Tools for avi won't work :P
There is a forum on doom9 dedicated to containers and muxing. Or just look at the mp4 FAQ: http://forum.doom9.org/showthread.php?t=62723

bis besser,
T0B1A5

[Kaur]

Are there any plans to support avi output?

[708145]

Are there any plans to support avi output?

Well if enough people request it and somebody shows me a CLI tool that can mux/concatenate .m4v files into avi then yes :)

bis besser,
T0B1A5

[Kaur]

I've tried converting m4v to avi with ffmpeg and then joining them with vdub and fixing the resulting avi with divfix.
It works fine but every now and then the file crashes the player.
I have tested bsplayer and wmp, crashes both if you seek too much.
Any thoughts/comments?

[shpitz]

As audio conversion needs to be done separately, is there a SMP-capable audio converter anyone knows of?

actually there is a version of LAME that is multithreaded. it is called LAME MT.

check it out at here (http://softlab.technion.ac.il/project/LAME/html/lame.html)

[708145]

beta4a is out. It fixes a few bugs with muxing and is based on the new modular code base.

bis besser,
T0B1A5

[chilledoutuk]

how does this compare to the multithreaded versions of XVID?

[708145]

how does this compare to the multithreaded versions of XVID?

ELDER is independent of the MT versions of xvid. I'll add resuming so that even if your PC crashes you can pick up your encode with almost no overhead.
(actually resume is already in beta4a but not functional)

In short:
* ELDER should be much faster and scale much further (I still aim at 60-fold speedup on 64 compute nodes)
* ELDER will support fault tolerance due to spontaneous reboots or deliberate pausing due to gaming sessions :)
* ELDER will feature a very nice and improved rate control similar to x264's qcomp => better overall quality
* ELDER will eventually get a high efficiency high fps mode (as some of you might recall I prototyped this under the name xvid50p last year)

bis besser,
T0B1A5

[Revgen]

I just tested this version with with my AMD 4600+ and I got a boost over single thread. This version didn't cut my video short like the X264 beta3 version did. Hopefully you can fix the problem in that version.


Single Thread (using -m 1): 15.69FPS - Better performance than vfw version.

Multi-Threaded: 23.77FPS - about a 51% improvement over -m 1.


Here's my commandline

perl ELDER4XVID_start.pl -a E:\3s-Goo~1\goofs2.avs -m 2 -b 1500 -o E:\3s-Goo~1\goofs.mp4 -q "-quality 6 -hqacp -inter4v -max_bframes 2 -trellis -qpel -vhq 4 -rated -rd_bf"


EDIT:I tried using -c 4, but it would only use 2 chunks.

[708145]

This version didn't cut my video short like the X264 beta3 version did. Hopefully you can fix the problem in that version.

Single Thread (using -m 1): 15.69FPS - Better performance than vfw version.
Multi-Threaded: 23.77FPS - about a 51% improvement over -m 1.

EDIT:I tried using -c 4, but it would only use 2 chunks.

nice.
How long is your clip that ELDER does not accept 4 chunks? I can only imagine this happening with less than 2000 frames or so.

Please use a longer clip for speed testing, best would be at least 16K frames. This way the load is balanced a lot better. :)

BTW: beta4 for x264 is almost ready :D

bis besser,
T0B1A5

[Revgen]

It's a 973 frame test clip.

Okay, so that's it.