x264 parallel encoder

[708145]

Development continues! :)


Notice: Development of this branch of ELDER is frozen. I might include it to the main branch at a later time.


The main branch here http://forum.doom9.org/showthread.php?t=100766 contains x264 modes as well.
edit2006-12-15: the beta5 branch contains a x264 2pass with GUI now!
edit2007-06-29: beta6b released with x264 3pass support!

The old x264 only version can still be found here:
http://www.funknmary.de/bergdichter/projekte/ELDER/ELDER4X264_beta4d.exe

bis besser,
T0B1A5

[708145]

Changelog for beta1:
* initial release based on alpha5 of ELDER4XVID
* uses XVID 1st pass and X264 2nd pass
* fixed bug when trim() was used in the input avs
* use xvid_encraw for 1st pass (stage A and stage B)
* consistent naming for temporary files
* parallelism broken

bis besser,
T0B1A5

[Guest]

Please stop crossposting, per forum rule 8. Thank you. :readrule:

[Revgen]

By crossposting do you mean the thread (http://forum.doom9.org/showthread.php?t=100766) he posted in the Xvid forum?

That thread isn't the same subject as this one yet the titles describing the threads are the same.

This seems more like a rule #9 issue.

[708145]

Please stop crossposting

I don't expect x264 users to read the xvid forum. And this IS software for x264. That's why it makes sense to me to post it here.

Please note that the post in the xvid forum announces software for xvid. :)

bis besser,
T0B1A5

[708145]

By crossposting do you mean the thread (http://forum.doom9.org/showthread.php?t=100766) he posted in the Xvid forum?

That thread isn't the same subject as this one yet the titles describing the threads are the same.

This seems more like a rule #9 issue.

Feel free to propose a better title.

bis besser,
T0B1A5

[Revgen]

How about X264 parallel encoder for windows - beta1

That way It's differentiates itself from the xvid one.

[Sirber]

x264 can already "parallel" by using more than 1 thread. I don't see the point... :confused:

[Guest]

Same title and both recent posts gave a changelog for beta1. If it looks like a duck and quacks like a duck...

[708145]

Same title and both recent posts gave a changelog for beta1. If it looks like a duck and quacks like a duck...

Both topics changed :)
And btw, even the changelogs are slightly different.

The two scripts share some code but will share less in later revisions. For example I use the same 1st pass code for both ATM.

bis besser,
T0B1A5

[708145]

x264 can already "parallel" by using more than 1 thread. I don't see the point... :confused:

a) current parallelism does not scale far
b) current parallelism does not scale well

ELDER is supposed to scale approx. 60x on 64nodes.
But as I wrote before... current beta is is "parallelism challenged" due to an issue with Active Perl.

I will be able to sort out the scaling part myself but I need testers for the quality aspects of ELDER which are testable with one CPU machines easily by either using defaults or setting "-m 1" aka selecting to use 1 CPU max.

Another planned feature is pause/resume which should come handy for most users as well.

And I have to start yet another thread sometime as I plan to include SNOW encoding as well :D

bis besser,
T0B1A5

[foxyshadis]

I'm going to be getting a 4-core (or 8-core, haven't decided yet) server to upgrade our main office, when I do I'll give your stuff a test.

How exactly does it work? Does it split each frame into x blocks like x264? Does it split the movie (on 1st-pass-generated IDR chunks perhaps)? A more unique method?

[708145]

How exactly does it work? Does it split each frame into x blocks like x264? Does it split the movie (on 1st-pass-generated IDR chunks perhaps)? A more unique method?

I split the video scene accurately into chunks on the temporal axis. For more details refer to the project page :)

bis besser,
T0B1A5

[Hellworm]

I tried ELDERchap, but when I tried the elderchap_testrun.sh it tolds me in the first line:
cat: chapter.info: Datei oder Verzeichnis nicht gefunden
then it continues, but it seems to do nothing ( its ready in an instant ) and when I looked into the jobfiles they were empty.
To me it looks like something with gettting the chapter info from mplayer isnt working. ( the info is coming from mplayer, isnt it? )

[708145]

I tried ELDERchap, but when I tried the elderchap_testrun.sh it tolds me in the first line:

then it continues, but it seems to do nothing ( its ready in an instant ) and when I looked into the jobfiles they were empty.
To me it looks like something with gettting the chapter info from mplayer isnt working. ( the info is coming from mplayer, isnt it? )

elderchap is obsolete and only worked/s on linux.
please try elder.

bis besser,
T0B1A5

[Hellworm]

I'm sorry. I forgot to mention, that I am using Linux so I can only use elderchap.
So can you help with my Problem?

[708145]

I'm sorry. I forgot to mention, that I am using Linux so I can only use elderchap.
So can you help with my Problem?

gen_chapinfo.pl is supposed to print the chapter info (hence the name)
it will only work when the DVD is in the drive NOT with a ripped DVD on HD.

Anyway I suggest to wait for beta2 which I'll port to linux (wine) soon. Elderchap is not always frame accurate, ELDER is. :)

bis besser,
T0B1A5

[Hellworm]

I used a real DVD.
It's a shame that there won't be a native Linux program. :(
Will the port use wine for avisynth or will there be a possibility to use mplayers input?
If the actual program consists of perl scripts, it should be much easier to port it directly to Linux, than using wine.
Can you outline how the Programm is working? On the Project page is not that much Information.

Anyway, thank you for your efforts, I think elder is a program with great potential! :thanks:

[708145]

I used a real DVD.
It's a shame that there won't be a native Linux program. :(


Once avisynth 3.0 will be released for linux it will be linux native :)


Will the port use wine for avisynth or will there be a possibility to use mplayers input?
If the actual program consists of perl scripts, it should be much easier to port it directly to Linux, than using wine.


I never found out how to do frame accurate seeking in mplayer => :(


Can you outline how the Programm is working? On the Project page is not that much Information.


As written it splits the first pass into chunks of same duration (A). Then adds a correction stage (B) to find out correct scene changes on the chunk borders. The second pass is then conducted scene accurate and bitrate accurate (C) and an optional correction (D) is added to hit the target size more accurately.


Anyway, thank you for your efforts, I think elder is a program with great potential! :thanks:

Thanks for appreciating the work done so far. I hope to have beta2 ready next weekend. This is no promise though :D

bis besser,
T0B1A5

[Hellworm]

Once avisynth 3.0 will be released for linux it will be linux native :)

avisynth 3.0 is already available it's just that you can't do much with it ( you can already render fonts :D )

But anyway, I asked in mplayers irc and there really is no method of frame accurate seeking. I'll try to look in the mplayers source, perhaps I can add it myself, but I havent that much time so it could take a while.

As written it splits the first pass into chunks of same duration (A). Then adds a correction stage (B) to find out correct scene changes on the chunk borders. The second pass is then conducted scene accurate and bitrate accurate (C) and an optional correction (D) is added to hit the target size more accurately.

I read that ( I'm Sorry for my bad English, I meant describe, not outline ), what I exactly wanted to now is, how are the stages done, for example B: is it doing an additional first pass, around the borders, and then analyzing the stats file, or is it overlapping the chunks in stage A to get the info. Does bitrate accurate mean that every scene is done with the desired bitrate or is the second pass calculation of the codec done, to give every scene the same quality, but a different bitrate?
Perhaps you can write that details on the project's page? There isn't much info on it.

I hope to have beta2 ready next weekend. This is no promise though

Thats nice to hear, I'll test it when it's coming! :p

[708145]

I read that ( I'm Sorry for my bad English, I meant describe, not outline ), what I exactly wanted to now is, how are the stages done, for example B: is it doing an additional first pass, around the borders, and then analyzing the stats file, or is it overlapping the chunks in stage A to get the info. Does bitrate accurate mean that every scene is done with the desired bitrate or is the second pass calculation of the codec done, to give every scene the same quality, but a different bitrate?
Perhaps you can write that details on the project's page? There isn't much info on it.


I updated the Wiki:

ELDER Description (parallEL encoDER):
* ELDER Does a parallel 2 pass encode that tries to use the same bit allocation as normal 2pass
* First pass is divided into 2 stages:
* Stage A divides the stream into chunks of the same size
* Stage B corrects the scene changes on the chunk borders (encode from last I-frame of previous chunk to 2nd I-frame of current chunk).
* The second pass is divided into 2 stages:
* Stage C encodes a part of the chunks in parallel both scene and bitrate accurate (a bitrate is assigned to each chunk to give all of them approx. the same quality).
* Stage D corrects for size under- or overshoot

bis besser,
T0B1A5

[Hellworm]

I updated the Wiki:

Thats more informative, thank you! :thanks:

[708145]

new version is up.

beta2:
* added installer
* muxing to mp4
* show progress
* debug output suppressed
* compile perl scripts on windows or ship perl.exe with ELDER
* improved defaults for chunksize

bis besser,
T0B1A5

[bond]

detected fps = 25.00
detected frames = 300
using 2 threads to encode 1 chunks with approx. 300 frames each.
Total size will be 1075200 bytes.seems to be wrong, why 2 threads for 1 chunk? why not 2 chunks?

[708145]

detected fps = 25.00
detected frames = 300
using 2 threads to encode 1 chunks with approx. 300 frames each.
Total size will be 1075200 bytes.seems to be wrong, why 2 threads for 1 chunk? why not 2 chunks?

nope it's correct. 1 thread will just sit idle.
if there are too few frames ELDER will use fewer chunks.

edit: there are 2 defaults in the code: preferred chunksize (2000 frames) and minimum chunksize (350 frames).

bis besser,
T0B1A5

[bond]

ic, i now tested it with 1000 frames (1 core only) and the output only had 999 frames!

[708145]

ic, i now tested it with 1000 frames (1 core only) and the output only had 999 frames!

silent update to version beta2b

bis besser,
T0B1A5

[bond]

silent update to version beta2bfixed the framenumber problem :)

[708145]

new release :)

beta3:
* parallelism fixed

bis besser,
T0B1A5

[Revgen]

I tried this version with my AMD Athlon X2 4600+ and ActivePerl 5.8.7 Build 813 with -m set to 2 and the video encoded fast but not right.

My clip is 40 sec. long but ELDER only encoded 19 sec.

I'll do some more tests later today when I have more time.

[708145]

I tried this version with my AMD Athlon X2 4600+ and ActivePerl 5.8.7 Build 813 with -m set to 2 and the video encoded fast but not right.
My clip is 40 sec. long but ELDER only encoded 19 sec.
I'll do some more tests later today when I have more time.

Thanks for testing.

The mp4box version that ships with beta3 is buggy and does not mux correctly in some/many cases. If anyone knows a newer + more stable build please point me to it.

Plus I fixed a few other bugs in beta4... release soon: I'll just include latest mp4box, encraw and x264 and there you go. No need to test an ancient version of ELDER :P

bis besser,
T0B1A5

[708145]

just released beta4a of ELDER4XVID. The x264 version is right around the corner as well.

[blubberbirne]

Hi Tobias, alter Namensvetter :)

I love your Encoder :)

Image (http://img397.imageshack.us/img397/5058/elderx264beta3test4gd.jpg)

Speed is good :) With "normal" settings i get ~25fps, thats best score ever on my computer (X2 3800+)

Can't wait so see next Beta

[708145]

Hi Tobias, alter Namensvetter
Can't wait so see next Beta

Hey blubberbirne,

you didn't have to wait long it seems :) there is a new beta out NOW.
beta4b uses the new framework of ELDER. If all goes well there will be many more betas in the next couple of weeks (next in queue is an xvid crf mode and linux cluster mode)

bis besser,
T0B1A5

[blubberbirne]

wow,

i'm going to test it.

hope you write a gui in the near future. Creating batch files will coast a lot of time *g*

[708145]

hope you write a gui in the near future. Creating batch files will coast a lot of time *g*

Why is everybody asking for a GUI? :rolleyes:
It is on the TODO for a long time but I considered other things more important (like make it work).
But since this is the case now, I'll think about a GUI when I have some spare time. Maybe someone wants to contribute?

bis besser,
T0B1A5

[blubberbirne]

This batch doesn't work.....

perl ELDER4X264_start.pl -a G:\cut\Bettina_Cramer_Blitz-05_02_06.avs -m 2 -b 1150 -o G:\cut\Bettina_Cramer_Blitz-05_02_06.mp4 -q " --level 5 --ref 3 --mixed-refs --bframes 3 --b-pyramid --b-rdo --weightb --subme 7 --trellis 1 --analyse p8x8,b8x8,i4x4"

I get 0kbit bitrate. Filesize for a 2min clip is ~600kbit :helpful:

And here the same.......

perl ELDER4X264_start.pl -a G:\cut\Bettina_Cramer_Blitz-05_02_06.avs -m 2 -b 1150 -o G:\cut\Bettina_Cramer_Blitz-05_02_06.mp4 -q "--ref 10 --mixed-refs --bframes 3 --b-pyramid --b-rdo --weightb --filter -2,-1 --subme 7 --trellis 1 --analyse all --8x8dct --progress --no-psnr"

I think, there is a bug in your new version

EDIT:

http://img387.imageshack.us/img387/8877/bug5ph.jpg

[Kostarum Rex Persia]

Well, GUI is much more easy for everyday user.

708145, how much you can optimize(to the limit) encoding speed for dual-core machines? 5-10 %, perhaps. And, do you planning to introduce SSE3 support in your x264 parallel encoder?

Thanks for your answers.

[blubberbirne]

Now i leave the -q parameter an it works? Very interresting. Looks like beta4a is very buggy. wiht your old beta it works perfect.

[LRN]

For GUI, you should contact to developers of MeGUI, StaxRip and other frontends.

[blubberbirne]

Of course, it would be very nice, if megui can handle your decoder :)

I found the error while it don't work: i relplaced the x264 in you old beta with a newer one. this one can handle more settings. setting which youre release don't handle.

So, everthink works fine :)

[KWAGGA]

Hi there Guys! Yes I believe a Gui would be a great idea for this encoder. I am not scared of cmd line app, but for all the settings ect it would be sweet to do it via a gui! Maybe I will try making a Java Interface :) (gimme +5 years though *<>* )

Kw

[ndkamal]

What is the gain of performance with this encoder ?

[foxyshadis]

Since it's just a multi-pass command-line, it makes perfect sense to integrate it into MeGUI (once that settles down), instead of wasting a lot of time making a half-assed gui for it, no offense to Tobias. By the time they're done you should just be able to design a form with all the options, set it as a 3 or 4-pass job, and let it rip. No worrying about all the rest of the GUI messiness.

Mr. Rex, I don't think you can code SSE3 in Perl. ;)

Oh yeah, just in time for dual-core laptop coming tomorrow, sup.

[708145]

What is the gain of performance with this encoder ?

There are some speedup figures (up to 88%) in the xvid thread. Speedup is very likely the same with beta4b in both xvid and x264 mode.
Have a look here:
http://forum.doom9.org/showthread.php?p=780648#post780648

bis besser,
T0B1A5

[bkman]

Do I understand correctly that you avoid the discrepancy of joining separately encoded chunks? But the output stream isn't bit identical to the standard encoder in single thread mode, is it.

[708145]

Since it's just a multi-pass command-line, it makes perfect sense to integrate it into MeGUI (once that settles down), instead of wasting a lot of time making a half-assed gui for it, no offense to Tobias. By the time they're done you should just be able to design a form with all the options, set it as a 3 or 4-pass job, and let it rip. No worrying about all the rest of the GUI messiness.

Mr. Rex, I don't think you can code SSE3 in Perl. ;)
.

Yeah, adding it to MeGUI would be great. But I prefer to handle multi-pass in ELDER since I'm about to fix the resume feature which enables to cont. partial encodes with little overhead :D
MeGUI can't do that (yet).

@Mr. Rex: No need for SSE3 in the perl script. We are talking about optimizing a couple of ms! And as ELDER uses all existing codec optimizations it is real fast anyway.
A better optimization is an improved load balancing in 2nd pass which will come soon. Still having the 1.95x speedup on X2 in mind :p

bis besser,
T0B1A5

[708145]

Do I understand correctly that you avoid the discrepancy of joining separately encoded chunks? But the output stream isn't bit identical to the standard encoder in single thread mode, is it.

Not bit identical but if you look at the bitrate distribution throughout the video it is _almost_ the same.
The difference stems from the fact that a normal 2pass always has the chance to correct bitrate over/undershoot in the following frames. ELDER does the same _except_ on chunk borders! Hence a small difference.

The larger the chunks the smaller the difference will be.
The smaller the chunks, the better the load balancing and thus speedup.

I'm inventing a clever way to find an adaptive tradeoff right now :D But current ELDER defaults ain't that bad, IMO.

bis besser,
T0B1A5

[MeteorRain]

can this program be extended to a LAN with lots of computers encoding parallelly?

[708145]

can this program be extended to a LAN with lots of computers encoding parallelly?

Of course since this is the reason I started ELDER development.
The first ELDER version could do so but scaled poorly (~11x speedup on 24nodes)

But I will not do a windows cluster mode. Cluster mode will be linux only.

bis besser,
T0B1A5