View Full Version : parallel encoder beta6a: The fastest encoder in town


708145
1st October 2005, 18:18
I'd like to announce a new release of ELDER (http://www.funknmary.de/bergdichter/projekte/index.php?page=ELDER), a parallEL encoDER.

Current state (ELDER_beta6b.exe (http://www.funknmary.de/bergdichter/projekte/ELDER/ELDER_beta6b.exe)) includes:
* a GUI for all modes
* SMP and windows cluster support (distributed encoding)
* xvid 2pass
* xvid 2pass with improved rate control (qcomp)
* xvid with crf parameter (will do a 2pass with same bit distribution as x264)
* x264 1pass crf
* x264 1pass abr
* x264 2pass
* avisynth input
* mp4 output

You'll need PXPerl. There is a copy of the original page (http://www.bumpersoft.com/Programming/Interpreters_and_Compilers/D_10987_index.htm) up here: PXPerl-5.8.7-6.exe (http://www.funknmary.de/bergdichter/projekte/ELDER/PXPerl-5.8.7-6.exe)

bis besser,
T0B1A5

bond
1st October 2005, 18:56
sounds very interesting. is there a speed/quality tradeoff and if yes, how big is it (any values?)

708145
1st October 2005, 19:16
sounds very interesting. is there a speed/quality tradeoff and if yes, how big is it (any values?)

If you follow the links, you'll find a bit of info.
As the RC is very simple for now, quality should be lower with this alpha.

edit: As of version alpha4 the size is hit quite accurately and the quality drop is very small. But please test for yourself :)

This will be fixed for the betas. I just wanted to have a working version out ASAP so you guys can start messing with it.

For speedups I wait for ppl testing it and telling me. I have access to SMPs and clusters running Linux, but I can't test on Windows SMP or multicore.

My guess: speedup should be _very_ linear, like 1.9x-fold on 2 cores and 3.9-ish on 4 cores.

bis besser,
T0B1A5

708145
11th October 2005, 14:46
Hey guys. It's been over a week and I got almost no feedback!

You don't need HT or SMP to test it. Just download and run it with your favourite avs script.

But for sure I'd be interested in speedup results of HT and SMP systems as well :P

bis besser,
Tobias

scb
11th October 2005, 17:55
Hi 708145!
Thank you in advance for Elder for win32!!! Would love to test the alpha version on my windows/smp setup but I have no idea how to install the thing :\

Would you (or some other kind person) let me know exactly how to get it installed, set it up and run it? I promise to report the results here.

tia!

unskinnyboy
11th October 2005, 18:30
Hi 708145!
Thank you in advance for Elder for win32!!! Would love to test the alpha version on my windows/smp setup but I have no idea how to install the thing :\

Would you (or some other kind person) let me know exactly how to get it installed, set it up and run it? I promise to report the results here.

tia!
I haven't tested ELDER4XVID myself, but from what I can see it isn't any difficult. The downloadable file is in .tgz format which is Unix Gzip/Tape Archive and can be decompressed by the latest WinRAR if you have it installed. Once you open the folder ELDER4XVID_alpha1 inside the .tgz, read the file README.txt which should answer your questions. Nothing to install. Extract the contents of the archive into a folder.

scb
11th October 2005, 19:03
thanks unskinnyboy

Got it. OK....reading through the readme file...still need help.

* needs a PERL interpreter to run; tested under cygwin...do I just google PERL cygwin, find it and install it? Does it have an installer?

Have been using GK, never used mencoder, so have some problems there.

I figure some of the ELDER parameters can be had from the *.avs via GK.

Here are the parameters that I'm unsure about:
-f number of frames; this is temporary as it can be obtained using the avs (gk?)
-c number of chunks; should be set a bit larger than maxpar<-- huh?
-p 1st pass options to mencoder<-- no idea
-q 2nd pass options to mencoder; must end with "bitrate="<-- no idea

Pretty sure (somewhat sure? hmmm...no idea) that the aspect ratio would be part of mencoder options? As above, I've no idea how to use mencoder so have no idea how to set aspect ratio to desired target of 1.333. Also, how/where do I set the output size to the desired 512x384?

As audio conversion needs to be done separately, is there a SMP-capable audio converter anyone knows of? (this is not ELDER-related per se, just me trying to get faster conversion)

tia for any help! (apologies if my alpha-teste newbieness grates)

unskinnyboy
11th October 2005, 20:41
* needs a PERL interpreter to run; tested under cygwin...do I just google PERL cygwin, find it and install it? Does it have an installer?

I suppose so. Elder.pl, as you can see, is written in Perl and Perl is an interpreted programming language, so you need a Perl interpreter. Cgywin is the environment Tobias tested this in. You need not have Cygwin installed.


Here are the parameters that I'm unsure about:
-f number of frames; this is temporary as it can be obtained using the avs (gk?)
-c number of chunks; should be set a bit larger than maxpar<-- huh?
-p 1st pass options to mencoder<-- no idea
-q 2nd pass options to mencoder; must end with "bitrate="<-- no idea

You will understand this better if you look at the example in the README:

* encode blafu.avs with 30000 frames to a 10MB XviD using 32 chunks and 2 CPUs:
./ELDER.pl -a blafu.avs -c 32 -m 2 -b 10485760 -f 30000 -p "- -ovc xvid -xvidencopts pass=1" -q "- -ovc xvid -xvidencopts pass=2:bitrate=" -o blafu.avi

-f --> you can get this by loading the avs into VirtualDubMoD, pressing Ctrl+-> and going to the end of the file. If you want to encode till the end of your movie, then take the last frame number as the input to -f.
-c --> to divide the movie into chunks while encoding to enable parallelization, I *think*.
-p --> your first pass encoding options. If you look at the example above, it is "- -ovc xvid -xvidencopts pass=1".
-q --> your second pass encoding options. As per the example, it is "- -ovc xvid -xvidencopts pass=2:bitrate=". As you can see, it ends with "bitrate=".

To know more about the encoding options & to answer your aspect ratio questions, check the MPlayer documentation (http://mplayerhq.hu/DOCS/man/en/mplayer.1.html#CODEC%20SPECIFIC%20ENCODING%20OPTIONS%20(MENCODER%20ONLY)).

scb
11th October 2005, 21:21
thanks again unskinnyboy

yeah...checked the example before my last post. Figured I could get the -f value from gk but you're method would work perfectly, too.

Thanks also for the mencoder doc link :)

ok...so....
Grab and install ActivePerl here: http://www.dzsoft.com/getperl.htm (the 'With MSI Installer' version, I'd guess?)

Unpack all the alpha version files to it's own directory

Rip dvd to h/d, open prompt, type command (still trying to figure out the exact command), hit 'enter' and let it run?

Question: Will ELDER/mencoder automatically find everything it needs for the XviD codec or do I have to put a file into the ELDER folder?

Regarding selection of the AR=1.333 and 512x384 issues....this looks like something I'd need to add to the -p and -q parameters (?). Found this in the mencoder doc (thanks for the link):
"aspect=<x/y | f (float value)>
Store movie aspect internally, just like MPEG files. Much nicer solution than rescaling, because quality is not decreased. MPlayer and a few others players will play these files correctly, others will display them with the wrong aspect. The as- pect parameter can be given as a ratio or a float- ing point number."

Looks like I'd need to add aspect=1.333 somewhere in the mencoder parameter string (?)

For the sizing, looks like I'd also add this to the mencoder parameters:
-vf scale=512:384

Took a look at videohelp.com's page in hopes of finding a guide but nada on mencoder.

Ugh....hate being too tech-illiterate to just set it all up and get to the testing part :(

It'd be much appreciated if someone better (more experienced) than me with this package could just give me the info to get going on this....

Other notes....on the project page, known issues:
'fps is hardcoded to 25 (will be fixed in alpha2)'....does this mean I'd get a max of 25fps per cpu (50-ish fps total, possibly 98fps for the 2cpu with HT-enabled)? Am currently getting around 77fps 1st pass, 58fps 2nd pass using GK.

'in the input avs there needs to be a single "#trim()" line to tell ELDER where to fill in trim commands (fixed in alpha2)'....so need to open *.avs in notepad and add this parameter at the top? What (if anything) do I need to put in the ()?

Anyway, again, if anyone feels like writing up a quick guide for 'How to install and test ELDER4XVID alpha for windows', it'd be SOOOO appreciated!!

I'm now gonna go take a couple aspirin and lay down for a little while hehe

708145
11th October 2005, 23:58
thanks unskinnyboy for the explanations :)


yeah...checked the example before my last post. Figured I could get the -f value from gk but you're method would work perfectly, too.


The -f is becoming obsolete in alpha2 since I learned how to get that info from the avs.

edit: unfortunaltely this is harder than I thought... so bear with me as I'm trying to solve that issue.


Grab and install ActivePerl here: http://www.dzsoft.com/getperl.htm (the 'With MSI Installer' version, I'd guess?)

Rip dvd to h/d, open prompt, type command (still trying to figure out the exact command), hit 'enter' and let it run?

Question: Will ELDER/mencoder automatically find everything it needs for the XviD codec or do I have to put a file into the ELDER folder?


With active perl I guess you have to remove all "./"s from the script. Apart from that your plan sounds right :)

no extra xvid needed, it's built-in in mencoder.exe


Looks like I'd need to add aspect=1.333 somewhere in the mencoder parameter string (?)

For the sizing, looks like I'd also add this to the mencoder parameters:
-vf scale=512:384


set aspect as mencoder option.
I'd suggest resizing in the avs script. So you could use GKnot to set up the avs for you.


Ugh....hate being too tech-illiterate to just set it all up and get to the testing part :(
It'd be much appreciated if someone better (more experienced) than me with this package could just give me the info to get going on this....


I try :P


Other notes....on the project page, known issues:
'fps is hardcoded to 25 (will be fixed in alpha2)'....does this mean I'd get a max of 25fps per cpu (50-ish fps total, possibly 98fps for the 2cpu with HT-enabled)? Am currently getting around 77fps 1st pass, 58fps 2nd pass using GK.

'in the input avs there needs to be a single "#trim()" line to tell ELDER where to fill in trim commands (fixed in alpha2)'....so need to open *.avs in notepad and add this parameter at the top? What (if anything) do I need to put in the ()?


OK. I'll publish alpha2 later which doesn't have this restriction.
You don't have to care about the trim() and not about the fps.


Anyway, again, if anyone feels like writing up a quick guide for 'How to install and test ELDER4XVID alpha for windows', it'd be SOOOO appreciated!!

I'm now gonna go take a couple aspirin and lay down for a little while hehe

and about the guide: I try to make this whole program much more user friendly. Some things (like the -f) were just for my convenience so I could release ELDER sooner.

testing is tough :D

bis besser,
Tobias

708145
12th October 2005, 01:39
The -f is becoming obsolete in alpha2 since I learned how to get that info from the avs.

OK. I'll publish alpha2 later which doesn't have this restriction.
You don't have to care about the trim() and not about the fps.


I'm sorry I couldn't get rid of the -f option.
I'll write a proper changelog tomorrow but as you can see some issues have been ironed out already with alpha2.

The download link is in the first post of this thread.

bis besser,
Tobias

scb
12th October 2005, 03:55
Hi 708145

Which PERL interpreter do you recommend to try?
I've never done any real avisynth stuff so not entirely sure about how to edit one (except that I know they open in notepad), what the resize command would be, etc, etc (guess I could read up on it, though).

Will try to make a first go at it this weekend.

I'm just thrilled by the prospect of finally being able to take advantage of smp for xvid! Thank you 708145!

PS- If anyone out there would care to make an actual guide, it'd be SOOOO helpful ;)

708145
12th October 2005, 09:25
Hi 708145
Which PERL interpreter do you recommend to try?
I've never done any real avisynth stuff so not entirely sure about how to edit one (except that I know they open in notepad), what the resize command would be, etc, etc (guess I could read up on it, though).


Well I recommend cygwin since it allows to run Unix-style scripts. But I guess any PERL interpreter would do (if the "./" were deleted from the script).
The reason why I like to leave them in is that I want the same ELDER script to run on both linux and windows.

About the avisynth: You can use GKnot and let it generate the avs for you.

bis besser,
Tobias

708145
12th October 2005, 09:28
As audio conversion needs to be done separately, is there a SMP-capable audio converter anyone knows of? (this is not ELDER-related per se, just me trying to get faster conversion)


I intend to add audio encoding at a later stage once video is working.

bis besser,
Tobias

stephanV
13th October 2005, 16:14
I've created a little guide which (hopefully) explains the settings and what programs are needed to run ELDER4xvid.

http://www.geocities.com/grotesteph/ELDER/guide.html

have fun

Kostarum Rex Persia
13th October 2005, 16:46
Wow,very good work, stephanV, keep going. Thank you for that good guide.

708145
13th October 2005, 19:13
I've created a little guide which (hopefully) explains the settings and what programs are needed to run ELDER4xvid.
http://www.geocities.com/grotesteph/ELDER/guide.html


Thank you very much :D

bis besser,
Tobias

708145
14th October 2005, 21:18
I just released alpha3! :D

alpha3:
* avimerge.exe added; merging step activated
* distribution style changed: one part all exe files the other the scripts
* changelog added
* comments added
* warning if chunks are too small added
* size and bitrate options added

bis besser,
Tobias

DarkNite
18th October 2005, 23:38
Nice additions, will test if they're broken or not as soon as current encode is finished.

708145
24th October 2005, 23:04
Nice additions, will test if they're broken or not as soon as current encode is finished.

How did you know it's broken :]
I fixed a few things in alpha4... at least in runs fine for me.

alpha4:
* sometimes wrong location for trim(): fixed
* StageC didn't start: fixed
* automatic selection of number of chunks if not specified
* delete intermediate files

bis besser,
Tobias

708145
30th October 2005, 17:43
alpha5 is up.

alpha5:
* modify perl for compatibility with Active Perl
* OS dependent defaults
* added cygwin1.dll to the exe bundle; needed by avimerge.exe
* added x264.exe to the exe bundle; needed to get framecount
* correct handling of pathes on both linux and windows
* get number of frames from avisynth; option -f not needed anymore

It will not use more than one thread currently :(
But please test for correctness. Thanks.

bis besser,
Tobias

708145
1st November 2005, 01:07
just released beta1:
* fixed bug when trim() was used in the input avs
* use xvid_encraw for 1st pass (stage A and stage B)
* consistent naming for temporary files
* parallelism broken

bis besser,
Tobias

IceM
2nd November 2005, 21:15
Verry promising thread title i must say :)

My first question before i begin testing this application (which wont be today caus of a lack of time, but will be on a short notice), is the output result similar as which you would get with the same encode in VirtualDub?

708145
2nd November 2005, 23:09
Verry promising thread title i must say :)

My first question before i begin testing this application (which wont be today caus of a lack of time, but will be on a short notice), is the output result similar as which you would get with the same encode in VirtualDub?

Output should be (almost) the same. If it is really the case testing is there for :P
The bit distribution should also be very similar to what xvid 2 pass does, but I can't guarantee it is the same. Some early testers even claimed that bit distribution would be better than default xvid :D But again: testing needed on this issue.

beta2 will output to mp4 instead of avi (which beta1 does) => a difference to Vdub.
Don't wait for beta2 for testing though since it will take me some time to implement all the changes I plan for this release :)

bis besser,
T0B1A5

708145
19th November 2005, 23:41
Don't wait for beta2 for testing though since it will take me some time to implement all the changes I plan for this release :)


beta2:
* added installer
* muxing to mp4
* show progress
* debug output suppressed
* compile perl scripts on windows or ship perl.exe with ELDER
* improved defaults for chunksize

bis besser,
T0B1A5

Badness430
21st November 2005, 15:23
perl ELDER4XVID.pl -a test.avs -m 4 -s 1297055000 -o test.mp4

Im using this and I still only get 25% CPU usage.

System:
Dual Xeon 3.6ghz
Windows XP Pro x64
2 gb ddr2 memory.

Xeon 3.6's are dual core, so when I look in performance, I see the 4 cpu columns, but only one is in use, hence the 25% cpu usuage. What am I missing =)

708145
21st November 2005, 21:00
perl ELDER4XVID.pl -a test.avs -m 4 -s 1297055000 -o test.mp4

Im using this and I still only get 25% CPU usage.

System:
Dual Xeon 3.6ghz
Windows XP Pro x64
2 gb ddr2 memory.

Xeon 3.6's are dual core, so when I look in performance, I see the 4 cpu columns, but only one is in use, hence the 25% cpu usuage. What am I missing =)

Well with alpha5 parallel execution on windows broke :( and is still not fixed.
I think other things are more urgent... like testing for usability and output quality.
Trust me: I'll take care of the parallel scaling soon. It works on msys, cygwin and native linux. Just windows doesn't. :rolleyes:

Does it produce correct output? Are you happy with the bitrate distribution?

bis besser,
T0B1A5

Badness430
21st November 2005, 23:26
Well with alpha5 parallel execution on windows broke :( and is still not fixed.
I think other things are more urgent... like testing for usability and output quality.
Trust me: I'll take care of the parallel scaling soon. It works on msys, cygwin and native linux. Just windows doesn't. :rolleyes:

Does it produce correct output? Are you happy with the bitrate distribution?

bis besser,
T0B1A5

I'll have run a full test. When I saw the only %25 I stopped encoding. Regular virtualdubmod usually gives me 45-50% doing what I dont know. I was looking for something to give me full use of my Xeons

708145
22nd November 2005, 00:01
I'll have run a full test. When I saw the only %25 I stopped encoding. Regular virtualdubmod usually gives me 45-50% doing what I dont know. I was looking for something to give me full use of my Xeons

I understand. Avisynth is being run in the same process as xvid in ELDER => 1core. It will actually give better scaling when it finally works.

I started hacking in preliminary || support for windows... hang on.

bis besser,
T0B1A5

708145
22nd November 2005, 00:54
I started hacking in preliminary || support for windows... hang on.


I feel dirty due to this hack... but it works here.

beta3:
* parallelism fixed

bis besser,
T0B1A5

Badness430
22nd November 2005, 02:02
I feel dirty due to this hack... but it works here.

beta3:
* parallelism fixed

bis besser,
T0B1A5

OK. I'm impressed now. My system is kicking fully percent on both CPU's and both cores.

Processing 2007 frames on 4 threads in 10 minutes. So thats about 8000 frames in 10 minutes. Wow. Typical HD tv show, which is what I mess with a lot, is 60000 frams on avg. At this rate this show will only take about 75 minutes.

When its done, I'll post quality comparison results and how everything finished up tomorrow.

Sharktooth
22nd November 2005, 04:34
Impressive... i just started a test encode on an athlon x2 3800+ and the FPS are almost doubled!
more info when it's done.

Badness
22nd November 2005, 05:52
OK. I'm impressed now. My system is kicking fully percent on both CPU's and both cores.

Processing 2007 frames on 4 threads in 10 minutes. So thats about 8000 frames in 10 minutes. Wow. Typical HD tv show, which is what I mess with a lot, is 60000 frams on avg. At this rate this show will only take about 75 minutes.

When its done, I'll post quality comparison results and how everything finished up tomorrow.

OK. Where does it put the file. I did a search accross my whole system and I dont see the file indicated in the -o switch.

D:\Work>perl ELDER4XVID.pl -a rome.avs -m 4 -s 1297055000 -o rome.mp4
detected fps = 23.98
detected frames = 68197
using 4 threads to encode 34 chunks with approx. 2006 frames each.
Total size will be 1297055000 bytes.
### starting process 0
### starting process 1
### starting process 2
### starting process 3
### Stage A spawned

Finished up with this.
### Stage C completed
### chunks merged
### temporary folder deleted
### another job well done

Normal chunk processing in between, didnt see the need to paste all that.

But I dont see the output file anywhere. I did a search accross my whole system to be sure and its not there.

708145
22nd November 2005, 12:18
OK. Where does it put the file. I did a search accross my whole system and I dont see the file indicated in the -o switch.

D:\Work>perl ELDER4XVID.pl -a rome.avs -m 4 -s 1297055000 -o rome.mp4

But I dont see the output file anywhere. I did a search accross my whole system to be sure and its not there.

It *should* end up in the same dir where you issued the command. But I guess I know why it failed.
I pass the merge command to mp4box via commandline and it might well be that it is too long since there is a very tight character limit in cmd.exe.
Could you do a quick test for me?
add -c 8 -f 8000 to your commandline. It will do a small part of your clip and then mux. I hope it works then. If yes it is very likely that we can blame cmd.exe once again :)

bis besser,
T0B1A5

Badness
22nd November 2005, 22:57
If I add a path on with the -o switch it comes out. Otherwise I'm not sure where it is going =) Maybe it is going into the sub folder that gets created and then it gets deleted with the temp files.

Badness
23rd November 2005, 23:03
If I add a path on with the -o switch it comes out. Otherwise I'm not sure where it is going =) Maybe it is going into the sub folder that gets created and then it gets deleted with the temp files.

Well, I take that back. It worked once on a small clip but I havent been able to get it to work since.

D:\Work>perl ELDER4XVID.pl -a thresh.avs -m 4 -s 593691000 -o d:\thresh.mp4

Should drop a file in the root of my D Drive but no go =(

708145
24th November 2005, 00:41
Well, I take that back. It worked once on a small clip but I havent been able to get it to work since.

D:\Work>perl ELDER4XVID.pl -a thresh.avs -m 4 -s 593691000 -o d:\thresh.mp4

Should drop a file in the root of my D Drive but no go =(

did you try "-c 8" yet?
I tested up to "-c 20" and it worked.

bis besser,
T0B1A5

Badness
24th November 2005, 03:51
did you try "-c 8" yet?
I tested up to "-c 20" and it worked.

bis besser,
T0B1A5

Running it with -c 8 now. Ill post results in the morning.

Badness
24th November 2005, 14:36
Running it with -c 8 now. Ill post results in the morning.

OK, it worked with the -c 8.

Now I have tried to load the resulting file into virtualdubmod and avimux and they wont accept the file. How can I mux the audio into the file?

708145
24th November 2005, 16:12
OK, it worked with the -c 8.


Good to hear that. Can you play it correctly? I ask because mp4box messed up some of my test encodes.


Now I have tried to load the resulting file into virtualdubmod and avimux and they wont accept the file. How can I mux the audio into the file?

The output is mp4. Tools for avi won't work :P
There is a forum on doom9 dedicated to containers and muxing. Or just look at the mp4 FAQ: http://forum.doom9.org/showthread.php?t=62723

bis besser,
T0B1A5

Kaur
29th November 2005, 01:47
Are there any plans to support avi output?

708145
29th November 2005, 02:22
Are there any plans to support avi output?

Well if enough people request it and somebody shows me a CLI tool that can mux/concatenate .m4v files into avi then yes :)

bis besser,
T0B1A5

Kaur
29th November 2005, 10:33
I've tried converting m4v to avi with ffmpeg and then joining them with vdub and fixing the resulting avi with divfix.
It works fine but every now and then the file crashes the player.
I have tested bsplayer and wmp, crashes both if you seek too much.
Any thoughts/comments?

shpitz
1st December 2005, 02:50
As audio conversion needs to be done separately, is there a SMP-capable audio converter anyone knows of?

actually there is a version of LAME that is multithreaded. it is called LAME MT.

check it out at here (http://softlab.technion.ac.il/project/LAME/html/lame.html)

708145
1st February 2006, 03:39
beta4a is out. It fixes a few bugs with muxing and is based on the new modular code base.

bis besser,
T0B1A5

chilledoutuk
1st February 2006, 15:03
how does this compare to the multithreaded versions of XVID?

708145
1st February 2006, 18:52
how does this compare to the multithreaded versions of XVID?

ELDER is independent of the MT versions of xvid. I'll add resuming so that even if your PC crashes you can pick up your encode with almost no overhead.
(actually resume is already in beta4a but not functional)

In short:
* ELDER should be much faster and scale much further (I still aim at 60-fold speedup on 64 compute nodes)
* ELDER will support fault tolerance due to spontaneous reboots or deliberate pausing due to gaming sessions :)
* ELDER will feature a very nice and improved rate control similar to x264's qcomp => better overall quality
* ELDER will eventually get a high efficiency high fps mode (as some of you might recall I prototyped this under the name xvid50p last year)

bis besser,
T0B1A5

Revgen
4th February 2006, 21:50
I just tested this version with with my AMD 4600+ and I got a boost over single thread. This version didn't cut my video short like the X264 beta3 version did. Hopefully you can fix the problem in that version.


Single Thread (using -m 1): 15.69FPS - Better performance than vfw version.

Multi-Threaded: 23.77FPS - about a 51% improvement over -m 1.


Here's my commandline

perl ELDER4XVID_start.pl -a E:\3s-Goo~1\goofs2.avs -m 2 -b 1500 -o E:\3s-Goo~1\goofs.mp4 -q "-quality 6 -hqacp -inter4v -max_bframes 2 -trellis -qpel -vhq 4 -rated -rd_bf"


EDIT:I tried using -c 4, but it would only use 2 chunks.

708145
4th February 2006, 23:51
This version didn't cut my video short like the X264 beta3 version did. Hopefully you can fix the problem in that version.

Single Thread (using -m 1): 15.69FPS - Better performance than vfw version.
Multi-Threaded: 23.77FPS - about a 51% improvement over -m 1.

EDIT:I tried using -c 4, but it would only use 2 chunks.

nice.
How long is your clip that ELDER does not accept 4 chunks? I can only imagine this happening with less than 2000 frames or so.

Please use a longer clip for speed testing, best would be at least 16K frames. This way the load is balanced a lot better. :)

BTW: beta4 for x264 is almost ready :D

bis besser,
T0B1A5

Revgen
5th February 2006, 01:16
It's a 973 frame test clip.

Okay, so that's it.

Revgen
5th February 2006, 06:14
Okay, here is the retest.

I've now encoded a 16501 frame clip.

Using these settings:

perl ELDER4XVID_start.pl -a E:\3s-Goo~1\goofs2.avs -m 2 -b 1500 -c 4 -o E:\3s-Goo~1\goofs.mp4 -q "-quality 6 -hqacp -inter4v -max_bframes 2 -trellis -qpel -vhq 4 -rated -rd_bf"


Results:

Single-Threaded (using -m 1): 17.18 FPS

Muti-Threaded: 32.35 FPS - About an 88% performance increase!:eek:


I can definitely say that this is the fastest XVID encoder I've used to date for my dual-core CPU. Heck, 32.35 FPS is fast enough to record TV shows in realtime. Too bad this isn't a vfw encoder.:( Or else I'd definitely use it for PVR purposes with my TV tuner.

708145
5th February 2006, 18:32
I've now encoded a 16501 frame clip.
Single-Threaded (using -m 1): 17.18 FPS
Muti-Threaded: 32.35 FPS - About an 88% performance increase!:eek:


great :D
Could you play a bit with the settings, i.e. using more chunks and "-m 3"?
I'm in particular interested in:
m2 c8
m2 c16
m3 c8
m3 c16

I still target 1.95x speedup on dual core with ELDER :)
And the final goal is 60x speedup on 64nodes :sly:
I still have a few performance increasing tricks left :p


I can definitely say that this is the fastest XVID encoder I've used to date for my dual-core CPU. Heck, 32.35 FPS is fast enough to record TV shows in realtime. Too bad this isn't a vfw encoder.:( Or else I'd definitely use it for PVR purposes with my TV tuner.

As this is not what I intent to use ELDER for, I might not code it. But in principle it ain't difficult and ELDER is GPL. So everyone can add this ;)
Instead I have plans to build a live streaming app on top of ELDER which will be for pay only. But if there is no "market" at the time of release it might well end up as GPL as well ;)

bis besser,
T0B1A5

Revgen
6th February 2006, 06:33
great :D
Could you play a bit with the settings, i.e. using more chunks and "-m 3"?
I'm in particular interested in:
m2 c8
m2 c16
m3 c8
m3 c16


Here are the results:

-m 2 -c 8 = 31.92 FPS - About 86% faster than -m 1

-m 2 -c 16 = 31.19 FPS - About 82% faster

-m 3 -c 8 = 32.17 FPS - About 87% faster

-m 3 -c 16 = 30.96 FPS - About 80% faster


It seems that 16 chunks hurts performance with this setup. It doesn't seem that -m 3 helps or hurts performance. I'd say the -m 3 -c 8 setting is about equal to -m 2 -c4. The result was only a couple seconds shy.

708145
6th February 2006, 08:05
-m 2 -c 8 = 31.92 FPS - About 86% faster than -m 1
-m 2 -c 16 = 31.19 FPS - About 82% faster
-m 3 -c 8 = 32.17 FPS - About 87% faster
-m 3 -c 16 = 30.96 FPS - About 80% faster

It seems that 16 chunks hurts performance with this setup. It doesn't seem that -m 3 helps or hurts performance. I'd say the -m 3 -c 8 setting is about equal to -m 2 -c4. The result was only a couple seconds shy.

Many thanks :D
-c 16 should be faster if done right (tm). So I need to tune it a bit for the next release ;)

bis besser,
T0B1A5

chilledoutuk
8th February 2006, 12:22
if i understand correctly would it not be possible to have a network of computers running slave clients and using this method of encoding distribute the job to all the computers you have running the client thus you could build lots of cheap computers and encode really fast.

are there any plans for something like this?

708145
8th February 2006, 13:39
if i understand correctly would it not be possible to have a network of computers running slave clients and using this method of encoding distribute the job to all the computers you have running the client thus you could build lots of cheap computers and encode really fast.

are there any plans for something like this?

Sure, it was the prime reason for me to start ELDER development. I will do a linux based cluster mode and everybody else is invited to add a windows based cluster mode if there is demand.
<dream>Encoding a full 2hour movie in less than 30 seconds would be awesome :D</dream>

But wait a bit for that since muxing, parameter passing and rate control issues are topmost on the TODO.

bis besser,
T0B1A5

Kostarum Rex Persia
8th February 2006, 15:38
708145, I must ask this. You, before 3 or 4 days, told us that you can't develop a GUI for Elder.

708145
8th February 2006, 15:54
708145, I must ask this. You, before 3 or 4 days, told us that you can't develop a GUI for Elder.

Huh? I said here that it is on my list (refer to the TODO list on my wiki) for a long time
http://forum.doom9.org/showthread.php?p=780977#post780977

If someone volunteers to do a GUI I'd appreciate that very much :D But integration into MeGUI would be a better solution IMO.
BTW, does MeGUI work in mono?

chilledoutuk
8th February 2006, 17:04
yes it would be easier to integrate it into megui

708145
17th February 2006, 01:21
beta4c is out :D

please check if parameters work and if the muxing goes all right.
if the quality is constant and it hits target size in the ELDER encodes then I can move along to
* the linux port
* better scaling with more CPUs
* improved rate control for xvid

708145
17th February 2006, 12:48
As I get continuous complaints about crossposting of ELDER announcements, I'll abandon this thread and update the other thread only:
http://forum.doom9.org/showthread.php?t=102119

The future ELDER builds will still contain XviD, x264 (plus even more codecs when time comes) but a single place to announce new builds makes things easier.

I don't know the mods opinion but I guess threads discussing specific issues for the XviD branch should still go into the XviD forum.

Thanks for your attention.
bis besser,
T0B1A5

bond
17th February 2006, 12:55
i would say its ok to discuss the xvid enhancements in this thread and the x264 ones in the other

Revgen
17th February 2006, 17:27
i would say its ok to discuss the xvid enhancements in this thread and the x264 ones in the other

It was Neuron2 that told him he was crossposting and violating rule #8 in this post (http://forum.doom9.org/showthread.php?p=731453#post731453).

Fortunately for him, Neuron2 doesn't moderate either forum. Even though 708145 technically wasn't violating rule #8 since both topics were unique, the structure of both topics and thread titles seemed similar and obviously confused Neuron2 at the time.

LordIntruder
19th February 2006, 06:52
Well if enough people request it and somebody shows me a CLI tool that can mux/concatenate .m4v files into avi then yes :)

T0B1A5

Hi,

I should be very interested to have the AVI in output. Mp4 is a bit annoying. ;) Also by adding this AVI option and a GUI many people could test your program. I never saw here someone not willing to test a program that allows to cut the encoding time close to half and AVI at the end. ;) However at the current stage, no AVI ouput, you have to use command line, it does not attract the masses. But well it is already great you work at such a tool. :)

I didn't test your work for the moment, I'm using the multi-threaded Xvid core 1.2 beta from Koepi site and the Xvid dll from Syskin. With normal build, 1st pass is close to 100% and 2nd to 60%. Now 1st is still close to 100% and 2nd 75%-80%. I own an AMD 4200+.

Many thanks and cheers. :)

bond
19th February 2006, 12:38
if you want a crappy container (aka avi) simply demux the video stream from mp4 to avi (eg with yamb)

708145
19th February 2006, 12:58
I should be very interested to have the AVI in output. Mp4 is a bit annoying. ;) Also by adding this AVI option and a GUI many people could test your program. I never saw here someone not willing to test a program that allows to cut the encoding time close to half and AVI at the end. ;) However at the current stage, no AVI ouput, you have to use command line, it does not attract the masses. But well it is already great you work at such a tool. :)


Sure thing I'll add a GUI soon :) Or at least integrate ELDER into MeGUI although that's windows only AFAIK.

Could you tell me what exactly is annoying about MP4. Which editing/muxing/preview steps are essential for you?
Because from a developer and codec point of view AVI (and VFW) is really obsolete! So our time is better spent developing a working MP4 workflow than hacking in an ugly AVI support to advanced video codecs.


I didn't test your work for the moment, I'm using the multi-threaded Xvid core 1.2 beta from Koepi site and the Xvid dll from Syskin. With normal build, 1st pass is close to 100% and 2nd to 60%. Now 1st is still close to 100% and 2nd 75%-80%. I own an AMD 4200+.

Many thanks and cheers. :)

I didn't get the percentages. Do you mean utilization of your CPUs?

bis besser,
T0B1A5

Revgen
19th February 2006, 20:04
It's not so much that people like AVI, it's more that containers like MKV are a lot more flexible. I'd rather use MKV and use the original .ac3 file than have to re-encode to AAC and introduce further quality loss into my audio file.

This is easily solved by muxing the no-audio .mp4 encoded by ELDER and the original .ac3 with MKVMerge.

LordIntruder
20th February 2006, 18:11
Could you tell me what exactly is annoying about MP4. Which editing/muxing/preview steps are essential for you?
Because from a developer and codec point of view AVI (and VFW) is really obsolete! So our time is better spent developing a working MP4 workflow than hacking in an ugly AVI support to advanced video codecs.



I didn't get the percentages. Do you mean utilization of your CPUs?
T0B1A5

Hi,

Yes percentages were utilization of my CPU. Just an indication to what I got with the 1.2 beta and dll SMP stuff.

Yes I know AVI is obsolete but if you look what is used most, it is avi and mp3, even if there are much better codecs. And Divx players does not help at all to spread new formats/containers. :(

So my ask is to have the good old AVI available but of course MKV too, etc... Like that those that use mp4 are satisfied, those that use mkv are happy and those that use avi container too. Also with those kinds of containers supported, you will have a broader audience to test your program.

Of course the better would be to get rid of AVI, there is much better as solution but on the other hand we can't forget that avi is most used. But if we always support AVI, we will never get rid of it. ;)

However I understand if you have no time you only concentrate on mp4 rather than an absolete format. It is quite normal to focus on futur formats that on old ones forever and there are tutorials and explanations around here to convert mp4 to avi, as bond just recalls. :)

foxyshadis
20th February 2006, 23:31
AVI and MKV can both be muxed from MP4 just fine. Well, AVI requires a few more manual steps, but I suppose those steps wouldn't be so hard to automate (mux to mp4, demux big combined .264, mux to avc2avi. if it won't export a big combined .264, you'd have to merge each to an avi and join them, probably beyond the scope of this project). mkv, well, you have to mux your audio in manually anyway, so why not use the intermediate mp4 instead mkv?

Oh wait, this is the xvid forum, mux to avi would be having xvid_encraw output avi and merging them. Still, if it supports for xvid it'd have to support for x264 as well.

708145
16th October 2006, 15:21
Quite a lot of time passed and it is time for an update.

+ ELDER4XVID is continued to be developed as GPL.
+ The muxing works in beta5
+ The XVIDcrf ratecontrol is working in my tests
+ There is a GUI in beta5 :)

- muxing is .mp4 only, .mkv will be added shortly
- support for windows cluster is commercial
- still no audio support
- resume is still not functional
- still no support for theora and snow :p

It'll take a while to package the release because I switched the PERL distribution again.

bis besser,
T0B1A5

shpitz
16th October 2006, 16:10
thanks tobias.

can you give us a quick review of elder? what is the benefit of using it over the other solutions?

708145
16th October 2006, 16:20
can you give us a quick review of elder? what is the benefit of using it over the other solutions?

* It's faster.
* It'll support resume soon. So you can resume your encode after a crash or shutdown/pause deliberately when you want to do something else.
* It supports workstation clusters (well, the GPL version not yet)

bis besser,
T0B1A5

shpitz
16th October 2006, 16:36
thanks.

any benefits in terms of SMP?

708145
16th October 2006, 16:41
any benefits in terms of SMP?

yes. The first two points in my list.

708145
27th October 2006, 10:24
Hi folks!

Finally the first version of the beta5 series is up. :D
It includes a GUI (very simple prototype but maybe somebody finds it useful), correct muxing and more parallelism (as long as you don't mess with ELDER's autodetection).

Enjoy 400+ fps PAL encoding on your quad cores :p

PS: My wiki is down atm. Find the download link in the first post of this thread.

bis besser,
T0B1A5

708145
31st October 2006, 17:01
Finally the first version of the beta5 series is up. :D
It includes a GUI (very simple prototype but maybe somebody finds it useful), correct muxing and more parallelism (as long as you don't mess with ELDER's autodetection).

PS: My wiki is down atm. Find the download link in the first post of this thread.


Wiki is up again and the download section contains the PXPerl binary as well. I mirror it because some people had difficulties getting it from the original page.

bis besser,
T0B1A5

unknownone
1st November 2006, 19:31
Hi,

Is it just me or the beta 5series isn't out yet? I checked both the Tobiwiki and funknmary place and they only have beta 4d avail.

I been wanting to test it out but no gui until now so please upload the beta5 version. Thanks.

708145
1st November 2006, 22:54
Hi,

Is it just me or the beta 5series isn't out yet? I checked both the Tobiwiki and funknmary place and they only have beta 4d avail.

I been wanting to test it out but no gui until now so please upload the beta5 version. Thanks.

Well it's my provider. There was a robbery two weeks ago => wiki gone.
So I used a backup of mine and published ELDER beta5a.
Now they got the stolen RAID back => restored the status of before the robbery and _before_ the beta5 release :(

I'm updating again atm.
<edit>done. just the changelog is missing </edit>

bis besser,
T0B1A5

708145
6th November 2006, 11:59
ELDER beta5b is out:
* support for windows cluster
* GUI improved a bit
* xvid 2pass "qcomp" rate control... better bitrate distribution
* x264 1pass crf mode

I hope there are not too many bugs left.
Hang on for shutdown/resume and more rate control modes.

Download link is in the first post.

bis besser,
T0B1A5

Sharktooth
6th November 2006, 16:05
@Tobias: would you like to integrate ELDER into MeGUI?
Distributed encoding from scratch is taking me too much time and i absolutely need to port MeGUI to linux as fast as i can...

708145
6th November 2006, 18:29
@Tobias: would you like to integrate ELDER into MeGUI?
Distributed encoding from scratch is taking me too much time and i absolutely need to port MeGUI to linux as fast as i can...

Humm, don't know how much effort this would be.
But if you already have an avs for your encode then using my GUI is really simple.

bis besser,
T0B1A5

Sharktooth
6th November 2006, 18:42
yes. MeGUI creates an .avs and feeds it to the encoder.
we just need to integrate the gui functions into megui.

DDogg
6th November 2006, 22:43
Completes stage A, starts stage B, does 1 chunk and then all machines report:
stage-B result xxx ready "nothing to do"
sleeping

708145
6th November 2006, 23:02
Completes stage A, starts stage B, does 1 chunk and then all machines report:
stage-B result xxx ready "nothing to do"
sleeping

which mode and which command line or GUI settings?

DDogg
7th November 2006, 00:04
I just changed the paths in the GUI and pressed start. Just barely getting up to speed on elder and don't understand enough yet to report properly.

I wrote up a quickstart for people who want to try elder but can't figure out your docs. Please correct me if incorrect on any steps:

1> Install perl on master machine
2> Install elder on master machine
3> Share and map the elder directory, on the master machine, to a drive letter, let's say "M"
4> Create sub directories in elder directory named "support", and "source". In support directory put PXPerl-5.8.7-6.exe (the perl install file). Put some sample source in "M:\source". If copying vob's and a d2v, remember to edit the d2v in notepad and change the path to "M:\source" - also edit any avs you copied, or create from scratch a new source and avs in M:\source
5> On each slave machine map drive letter M to the shared elder directory on the master machine
6> On each slave, from m:\support, install perl. Verify the sample avs will play on all slave machines before proceeding. This means avisynth, dgdecode.dll and other called plugins will need to be in the avisynth\plugins directory on each slave.
7> To start the encode - On all machines to be used, master and slaves - from the M:\ run the spawn1.bat for each core available on the machine. So for a HT P4 you would run spawn1.bat once, and for a true dual run it twice. You will see a minimized cmd window for each time you run spawn1.bat
8> After that, on the master machine, run control.bat
9> Then run guixvid.bat on master machine - Move size slider to desired size, enter M:\source\your_input avs and M:\source\your_output.mp4. Press start

You should then see information scrolling in the spawned widows on all machines. Whether it will finish and mux I have not found out yet.

Also, when a job gets aborted, certain things that I have not figured out yet need to be cleaned up or things don't work correctly after that. Before you start the encode, make a copy of the elder directory. Then you can get a fresh start by replacing the elder directory that aborted with the fresh backup. I know this is not the proper way, but temporarily it may be of use.

foxyshadis
7th November 2006, 01:06
You can actually share out a folder with all the perl binaries/libraries, avisynth, plugins, sources, etc. If you do the d2v on the shared folder in the local machine, it's nearly as fast as reading right from the disk, and will automatically use the right folder. This way everything definitely has the same scripts, avisynth, perl, and so on. At least, that's the way I did it, although it makes for a bit of a cluttered folder. :p

I suppose it's easier to install things though, especially if some of the slaves are *nix.

DDogg
7th November 2006, 01:20
That was the only way I could figure out how to do it. Call me old and linear; I need my 1,2,3 :p Hey, did you get it to finish an encode?

foxyshadis
7th November 2006, 07:08
Haven't tried the new GUI one, I'll give it a spin. Next job: Install totally stateless daemons/services that will always be ready to encode and return chunks of video at any time. =D

708145
7th November 2006, 15:19
I just changed the paths in the GUI and pressed start. Just barely getting up to speed on elder and don't understand enough yet to report properly.

I wrote up a quickstart for people who want to try elder but can't figure out your docs. Please correct me if incorrect on any steps:


I couldn't reproduce your bug but stumbled over two others in the process... I'll need some time for fixing as this is a busy week and I can only spend a few minutes per day bug hunting.

Thank you very much for the usage guide. Much appreciated.

bis besser,
T0B1A5

708145
7th November 2006, 15:21
Haven't tried the new GUI one, I'll give it a spin. Next job: Install totally stateless daemons/services that will always be ready to encode and return chunks of video at any time. =D

well the pick_job script is a simple replacement of that. Probably needs to get more versatile and universal.

bis besser,
T0B1A5

708145
8th November 2006, 17:13
new beta5c is uploaded. I hope it fixes your bug DDogg.

Changelog:
* GUI: allows to select number of CPUs
* better load balancing (when -m is defined)
* cleaner code
* nicer text output: always prints encoded time and output size error
* first chunk was sometimes missing in x264 crf mode: fixed now
* added x264 abr mode
* added xvid crf2 mode: same size and bitrate distribution as x264's crf mode
* fixed a few rate control bugs... still undersizes sometimes

Next on the list:
* fixing remaining bugs
* improving the GUI: all modes, file selector, ...
* get pause/resume working
* integrate into MeGUI?
* add an xvid HFR mode (high frame rate: 50p and 60p)
* add x264 3pass

bis besser,
T0B1A5

DDogg
9th November 2006, 15:15
gave it a quick try and got the same thing so perhaps it is user error. It finishes stage A, says the import is done, and then says nothing to do.

If you will give me a procedure, or point out what exact information you need to diagnose the problem, I will get it for you.

708145
9th November 2006, 15:28
gave it a quick try and got the same thing so perhaps it is user error. It finishes stage A, says the import is done, and then says nothing to do.

If you will give me a procedure, or point out what exact information you need to diagnose the problem, I will get it for you.

The two outputs are in 2 different windows, right?
The nothing to do is in the compute process and the "import done" in the control process?

Try the following order:
0) make sure everything is installed but no ELDER process running
1) run shutdown.bat to clean the jobfile
2) run initjobs.bat to init the jobfile
3) run control.bat to start up the control job
4) spawn all clients with spawn1.bat... as many as you like (even on remote PCs)
5) start the GUI and issue your job

Important: Make sure your filenames don't contain spaces.
For testing just start with one control job and one compute job on your main computer.

bis besser,
T0B1A5

DDogg
9th November 2006, 18:31
followed the instructions exactly. Used one control and one spawn on master machine. Stage C completed. Results, the mp4 file was unplayable. Window said:
2nd pass fps = 36.94
total time for encoding: 735 seconds
overall fps = 20.4
now merging the chunks...
chunks merged
target: 101711872 bytes
encode: 56189830 bytes
error: 44.76%
another job well done
Source avs:
DGDecode_mpeg2source("M:\source\VTS_02_1.d2v",info=3)
ColorMatrix(hints=true)
crop( 0, 60, -2, -64)
LanczosResize(640,272) # Lanczos (Sharp)
trim(5000,20000)


Starting to think it is a gui problem. [] Give me a way to copy the cmdline from the gui.

708145
9th November 2006, 19:09
followed the instructions exactly. Used one control and one spawn on master machine. Stage C completed. Results, the mp4 file was unplayable. Window said:
2nd pass fps = 36.94
total time for encoding: 735 seconds
overall fps = 20.4
now merging the chunks...
chunks merged
target: 101711872 bytes
encode: 56189830 bytes
error: 44.76%
another job well done
Source avs:
DGDecode_mpeg2source("M:\source\VTS_02_1.d2v",info=3)
ColorMatrix(hints=true)
crop( 0, 60, -2, -64)
LanczosResize(640,272) # Lanczos (Sharp)
trim(5000,20000)


Starting to think it is a gui problem. This is an xvid encode with avi specifiied, why the forced mp4?

Currently it can only mux to mp4, so the GUI automatically adds the correct extension. I will add muxing to avi later (for the xvid part). Avi will not be supported for x264 for obvious reasons. I'm also thinking about writing .mkv.

But that's a huge error you get plus an unplayable file. I guess they are correlated. Errors for me are in the 6% range for short samples and <2% for complete movies.
How many chunks were used? I played with the autodetection formula and didn't test with many cases before releasing beta5c.

Of course testing the cli would help tremendously.

bis besser,
T0B1A5

708145
10th November 2006, 13:31
Starting to think it is a gui problem.


New beta5d is up. Now has a reset SMP button (please use before encoding to set up internal files) as well as a start SMP button to start the master.

Additional clients still have to be added by hand. If someone knows how to automate this on windows please tell me.

beta5d:
* added SMP control to GUI
* prints frames in the control window for xvid 2pass
* fixed a bug in chunk computation
* starts processes in lower priority

bis besser,
T0B1A5

DDogg
10th November 2006, 16:32
sorry, all i get is "nothing to do" in the windows. I think I did what you wanted, but to be sure give me a 1,2,3

708145
10th November 2006, 17:46
sorry, all i get is "nothing to do" in the windows. I think I did what you wanted, but to be sure give me a 1,2,3

OK.
There is a detailed usage guide in the README. I'll sum it up here:

1) as before, install everything and make sure ELDER is not running.
2) start the guixvid.bat
3) hit reset in the gui
4) select number of CPUs on your main PC
5) hit start SMP
6) optional: add more slaves via spawn1.bat on other PCs in the same directory
7) select size, encoding options, input and output and hit "add job"
8) wait for encode to finish / or add more jobs
9) hit "reset" after all is done and "quit"

EDIT: ouch, my bad. I commented out a line I shouldn't have :) I put a fixed version online. it now even starts a control process.
EDIT2: Just worked for me on a 15K frames sample (50fps overall).

bis besser,
T0B1A5

DDogg
11th November 2006, 01:32
<crap> #7 Probably should do a "Nothing in queue, you complete dummy" type of thing.

Results still not playable. /add:Hmm, nero showtime will play it, but MPC will not. Mpc will play my other mp4 samples. Probably a problem on my end, but could you check with MPC on your end?

stage C progress: 14/16 chunks done (87%) in 463 seconds. 13127 frames so far.
stage C progress: 15/16 chunks done (93%) in 495 seconds. 14195 frames so far.
stage C progress: 16/16 chunks done (100%) in 500 seconds. 15001 frames so far.
stage C completed
2nd pass fps = 52.82
total time for encoding: 500 seconds
overall fps = 30
now merging the chunks...
chunks merged
target: 83886080 bytes
encode: 53967994 bytes
error: 35.67%
another job well done

708145
11th November 2006, 13:04
Results still not playable. /add:Hmm, nero showtime will play it, but MPC will not. Mpc will play my other mp4 samples. Probably a problem on my end, but could you check with MPC on your end?

stage C progress: 14/16 chunks done (87%) in 463 seconds. 13127 frames so far.
stage C progress: 15/16 chunks done (93%) in 495 seconds. 14195 frames so far.
stage C progress: 16/16 chunks done (100%) in 500 seconds. 15001 frames so far.
stage C completed
2nd pass fps = 52.82
total time for encoding: 500 seconds
overall fps = 30
now merging the chunks...
chunks merged
target: 83886080 bytes
encode: 53967994 bytes
error: 35.67%
another job well done

:) So this means you did get it working. Great!
Could you now test with more CPUs (slave machines)? IOW the "optional" step 6)

I haven't used DirectShow based players in a while. I'll install mpc+ffdshow and see. I suspect an issue with mp4box.

After that we can address the huge undersize you experience. Is it a reasonable bitrate you ask for? Maybe the codec is saturating? Could be some error on my side as well although my test encodes come in at +-6% usually.
There are several ways to improve size accuracy just haven't implemented any yet. The final version of ELDER should get <1% error in all cases of at least a few minutes length.

bis besser,
T0B1A5

708145
11th November 2006, 15:33
both vlc and mpc can play the resulting mp4 here.

DDogg
11th November 2006, 15:37
I did set a full encode going after the 15k frame test completed. Used two machines with total 4 processes encoding. The result was a stall after stage A. Have you done a full, say 200k frame encode, on your end?

Wondering if you implementing some type of diagnostic log would help you?

As for the undersize, don't worry about that as I was doing 15k frames of an easily compressible source.

708145
11th November 2006, 17:22
I did set a full encode going after the 15k frame test completed. Used two machines with total 4 processes encoding. The result was a stall after stage A. Have you done a full, say 200k frame encode, on your end?

Wondering if you implementing some type of diagnostic log would help you?

I did a 150Kframes encode yesterday: 1.2% undersize.
But it was 2 CPUs only so the job control wasn't really tested.

It could be due to some race condition. I'll update the locking of the jobfiles and we'll see. And a diagnostics log would be a good idea.

708145
14th November 2006, 19:36
It could be due to some race condition. I'll update the locking of the jobfiles and we'll see. And a diagnostics log would be a good idea.

Jobcontrol is implemented with locking now. I also improved the GUI.

beta5e:
* add bitrate, size and crf selection to GUI
* GUI support for all modes
* print "frames so far" in all modes
* print output size error with sign
* locking on the jobfiles implemented

The locking should fix the sudden stop in encoding.

bis besser,
T0B1A5

Inventive Software
14th November 2006, 20:53
Shutdown/stop and resume would be magnificent Tobias! :D

708145
14th November 2006, 21:03
Shutdown/stop and resume would be magnificent Tobias! :D

Coming up soon. But I first want to sort out existing issues. Most of the code for pause/resume is already in functions.pl. But not working yet.

708145
15th December 2006, 14:41
ELDER beta5f is up.
It includes a x264 2pass mode as well as avi output in xvid mode.

Have a look at Changelog.txt for a full changelog.

bis besser,
T0B1A5

Al79
17th December 2006, 10:25
How did you know it's broken :]
I fixed a few things in alpha4... at least in runs fine for me.

alpha4:
* sometimes wrong location for trim(): fixed
* StageC didn't start: fixed
* automatic selection of number of chunks if not specified
* delete intermediate files

bis besser,
Tobias

:thanks: :thanks: :thanks: downloading ;), i hope i can give u a good feedback

708145
18th December 2006, 19:22
:thanks: :thanks: :thanks: downloading ;), i hope i can give u a good feedback

Sitting here and waiting for feedback especially about the GUI.
Please test the crf modes as well.

To keep you up to date:
* I'm internally debugging my codec n0153 and adding it to ELDER.
* ELDER4Theora still lacks muxing.
* x264_3pass is around the corner.
* mkv out is around the corner.
* I did not look into Snow encoding yet.
* beta6 series will finally bring pause/resume :D

By requesting certain features you can influence my priority list a little bit. It's Xmas wishlist time! :p

bis besser,
T0B1A5

708145
22nd December 2006, 16:25
Sitting here and waiting for feedback especially about the GUI.
Please test the crf modes as well.
* beta6 series will finally bring pause/resume :D


I couldn't wait: here is beta6a.
It is mainly a bugfix release for avs pathes and adds a new fps detection algorithm.
I hope to make some progress on the TODO list over christmas.

bis besser,
T0B1A5

experttech
23rd December 2006, 00:03
This tool looks good an I'd love to try it.

Can this work with a DV-AVI file as input? And de-interlace and resize it (Avisynth)

708145
23rd December 2006, 00:14
This tool looks good an I'd love to try it.
Can this work with a DV-AVI file as input? And de-interlace and resize it (Avisynth)

You can use every source which avisynth can read.
There are very good deinterlacers and resizers in avisynth :)

icezellion
25th April 2008, 22:15
Hey T0B1A5, will there be any more updates to the elder4XVID branch? This was a very promising app, I just never had the time to post. :)

708145
28th April 2008, 23:20
Hey T0B1A5, will there be any more updates to the elder4XVID branch? This was a very promising app, I just never had the time to post. :)

Well, ELDER was a fun project a few years ago when multicore CPUs were just appearing in consumer PCs and codecs had very inefficient SMP code built in.
Modern codecs are very efficient now and I think it is time to make the switch if you didn't already.
If you need xvid for some reason (standalone?) then you can just be happy with the speed xvid already has, or improve the ELDER code on your own, it's GPL.
I won't continue on ELDER as I'm very low on "video codec time" in general. If at all then I'll work on my codec invention JAVC which will set a new world record in codec inefficiency ;)

bis besser,
T0B1A5

PatchWorKs
12th May 2008, 10:04
I won't continue on ELDER as I'm very low on "video codec time" in general.
Can you please open a SF.net page for your adbandoned projects ?

Thanks in advice !

708145
12th May 2008, 17:22
Can you please open a SF.net page for your adbandoned projects ?


Why would an abandoned project need a SF page?
I'd think that when somebody comes along who wants to continue development he'll find a place to put the code.

bis besser,
T0B1A5