parallel encoder beta6a: The fastest encoder in town

[Revgen]

Okay, here is the retest.

I've now encoded a 16501 frame clip.

Using these settings:

perl ELDER4XVID_start.pl -a E:\3s-Goo~1\goofs2.avs -m 2 -b 1500 -c 4 -o E:\3s-Goo~1\goofs.mp4 -q "-quality 6 -hqacp -inter4v -max_bframes 2 -trellis -qpel -vhq 4 -rated -rd_bf"


Results:

Single-Threaded (using -m 1): 17.18 FPS

Muti-Threaded: 32.35 FPS - About an 88% performance increase!:eek:


I can definitely say that this is the fastest XVID encoder I've used to date for my dual-core CPU. Heck, 32.35 FPS is fast enough to record TV shows in realtime. Too bad this isn't a vfw encoder.:( Or else I'd definitely use it for PVR purposes with my TV tuner.

[708145]

I've now encoded a 16501 frame clip.
Single-Threaded (using -m 1): 17.18 FPS
Muti-Threaded: 32.35 FPS - About an 88% performance increase!:eek:


great :D
Could you play a bit with the settings, i.e. using more chunks and "-m 3"?
I'm in particular interested in:
m2 c8
m2 c16
m3 c8
m3 c16

I still target 1.95x speedup on dual core with ELDER :)
And the final goal is 60x speedup on 64nodes :sly:
I still have a few performance increasing tricks left :p


I can definitely say that this is the fastest XVID encoder I've used to date for my dual-core CPU. Heck, 32.35 FPS is fast enough to record TV shows in realtime. Too bad this isn't a vfw encoder.:( Or else I'd definitely use it for PVR purposes with my TV tuner.

As this is not what I intent to use ELDER for, I might not code it. But in principle it ain't difficult and ELDER is GPL. So everyone can add this ;)
Instead I have plans to build a live streaming app on top of ELDER which will be for pay only. But if there is no "market" at the time of release it might well end up as GPL as well ;)

bis besser,
T0B1A5

[Revgen]

great :D
Could you play a bit with the settings, i.e. using more chunks and "-m 3"?
I'm in particular interested in:
m2 c8
m2 c16
m3 c8
m3 c16


Here are the results:

-m 2 -c 8 = 31.92 FPS - About 86% faster than -m 1

-m 2 -c 16 = 31.19 FPS - About 82% faster

-m 3 -c 8 = 32.17 FPS - About 87% faster

-m 3 -c 16 = 30.96 FPS - About 80% faster


It seems that 16 chunks hurts performance with this setup. It doesn't seem that -m 3 helps or hurts performance. I'd say the -m 3 -c 8 setting is about equal to -m 2 -c4. The result was only a couple seconds shy.

[708145]

-m 2 -c 8 = 31.92 FPS - About 86% faster than -m 1
-m 2 -c 16 = 31.19 FPS - About 82% faster
-m 3 -c 8 = 32.17 FPS - About 87% faster
-m 3 -c 16 = 30.96 FPS - About 80% faster

It seems that 16 chunks hurts performance with this setup. It doesn't seem that -m 3 helps or hurts performance. I'd say the -m 3 -c 8 setting is about equal to -m 2 -c4. The result was only a couple seconds shy.

Many thanks :D
-c 16 should be faster if done right (tm). So I need to tune it a bit for the next release ;)

bis besser,
T0B1A5

[chilledoutuk]

if i understand correctly would it not be possible to have a network of computers running slave clients and using this method of encoding distribute the job to all the computers you have running the client thus you could build lots of cheap computers and encode really fast.

are there any plans for something like this?

[708145]

if i understand correctly would it not be possible to have a network of computers running slave clients and using this method of encoding distribute the job to all the computers you have running the client thus you could build lots of cheap computers and encode really fast.

are there any plans for something like this?

Sure, it was the prime reason for me to start ELDER development. I will do a linux based cluster mode and everybody else is invited to add a windows based cluster mode if there is demand.
<dream>Encoding a full 2hour movie in less than 30 seconds would be awesome :D</dream>

But wait a bit for that since muxing, parameter passing and rate control issues are topmost on the TODO.

bis besser,
T0B1A5

[Kostarum Rex Persia]

708145, I must ask this. You, before 3 or 4 days, told us that you can't develop a GUI for Elder.

[708145]

708145, I must ask this. You, before 3 or 4 days, told us that you can't develop a GUI for Elder.

Huh? I said here that it is on my list (refer to the TODO list on my wiki) for a long time
http://forum.doom9.org/showthread.php?p=780977#post780977

If someone volunteers to do a GUI I'd appreciate that very much :D But integration into MeGUI would be a better solution IMO.
BTW, does MeGUI work in mono?

[chilledoutuk]

yes it would be easier to integrate it into megui

[708145]

beta4c is out :D

please check if parameters work and if the muxing goes all right.
if the quality is constant and it hits target size in the ELDER encodes then I can move along to
* the linux port
* better scaling with more CPUs
* improved rate control for xvid

[708145]

As I get continuous complaints about crossposting of ELDER announcements, I'll abandon this thread and update the other thread only:
http://forum.doom9.org/showthread.php?t=102119

The future ELDER builds will still contain XviD, x264 (plus even more codecs when time comes) but a single place to announce new builds makes things easier.

I don't know the mods opinion but I guess threads discussing specific issues for the XviD branch should still go into the XviD forum.

Thanks for your attention.
bis besser,
T0B1A5

[bond]

i would say its ok to discuss the xvid enhancements in this thread and the x264 ones in the other

[Revgen]

i would say its ok to discuss the xvid enhancements in this thread and the x264 ones in the other

It was Neuron2 that told him he was crossposting and violating rule #8 in this post (http://forum.doom9.org/showthread.php?p=731453#post731453).

Fortunately for him, Neuron2 doesn't moderate either forum. Even though 708145 technically wasn't violating rule #8 since both topics were unique, the structure of both topics and thread titles seemed similar and obviously confused Neuron2 at the time.

[LordIntruder]

Well if enough people request it and somebody shows me a CLI tool that can mux/concatenate .m4v files into avi then yes :)

T0B1A5

Hi,

I should be very interested to have the AVI in output. Mp4 is a bit annoying. ;) Also by adding this AVI option and a GUI many people could test your program. I never saw here someone not willing to test a program that allows to cut the encoding time close to half and AVI at the end. ;) However at the current stage, no AVI ouput, you have to use command line, it does not attract the masses. But well it is already great you work at such a tool. :)

I didn't test your work for the moment, I'm using the multi-threaded Xvid core 1.2 beta from Koepi site and the Xvid dll from Syskin. With normal build, 1st pass is close to 100% and 2nd to 60%. Now 1st is still close to 100% and 2nd 75%-80%. I own an AMD 4200+.

Many thanks and cheers. :)

[bond]

if you want a crappy container (aka avi) simply demux the video stream from mp4 to avi (eg with yamb)

[708145]

I should be very interested to have the AVI in output. Mp4 is a bit annoying. ;) Also by adding this AVI option and a GUI many people could test your program. I never saw here someone not willing to test a program that allows to cut the encoding time close to half and AVI at the end. ;) However at the current stage, no AVI ouput, you have to use command line, it does not attract the masses. But well it is already great you work at such a tool. :)


Sure thing I'll add a GUI soon :) Or at least integrate ELDER into MeGUI although that's windows only AFAIK.

Could you tell me what exactly is annoying about MP4. Which editing/muxing/preview steps are essential for you?
Because from a developer and codec point of view AVI (and VFW) is really obsolete! So our time is better spent developing a working MP4 workflow than hacking in an ugly AVI support to advanced video codecs.


I didn't test your work for the moment, I'm using the multi-threaded Xvid core 1.2 beta from Koepi site and the Xvid dll from Syskin. With normal build, 1st pass is close to 100% and 2nd to 60%. Now 1st is still close to 100% and 2nd 75%-80%. I own an AMD 4200+.

Many thanks and cheers. :)

I didn't get the percentages. Do you mean utilization of your CPUs?

bis besser,
T0B1A5

[Revgen]

It's not so much that people like AVI, it's more that containers like MKV are a lot more flexible. I'd rather use MKV and use the original .ac3 file than have to re-encode to AAC and introduce further quality loss into my audio file.

This is easily solved by muxing the no-audio .mp4 encoded by ELDER and the original .ac3 with MKVMerge.

[LordIntruder]

Could you tell me what exactly is annoying about MP4. Which editing/muxing/preview steps are essential for you?
Because from a developer and codec point of view AVI (and VFW) is really obsolete! So our time is better spent developing a working MP4 workflow than hacking in an ugly AVI support to advanced video codecs.



I didn't get the percentages. Do you mean utilization of your CPUs?
T0B1A5

Hi,

Yes percentages were utilization of my CPU. Just an indication to what I got with the 1.2 beta and dll SMP stuff.

Yes I know AVI is obsolete but if you look what is used most, it is avi and mp3, even if there are much better codecs. And Divx players does not help at all to spread new formats/containers. :(

So my ask is to have the good old AVI available but of course MKV too, etc... Like that those that use mp4 are satisfied, those that use mkv are happy and those that use avi container too. Also with those kinds of containers supported, you will have a broader audience to test your program.

Of course the better would be to get rid of AVI, there is much better as solution but on the other hand we can't forget that avi is most used. But if we always support AVI, we will never get rid of it. ;)

However I understand if you have no time you only concentrate on mp4 rather than an absolete format. It is quite normal to focus on futur formats that on old ones forever and there are tutorials and explanations around here to convert mp4 to avi, as bond just recalls. :)

[foxyshadis]

AVI and MKV can both be muxed from MP4 just fine. Well, AVI requires a few more manual steps, but I suppose those steps wouldn't be so hard to automate (mux to mp4, demux big combined .264, mux to avc2avi. if it won't export a big combined .264, you'd have to merge each to an avi and join them, probably beyond the scope of this project). mkv, well, you have to mux your audio in manually anyway, so why not use the intermediate mp4 instead mkv?

Oh wait, this is the xvid forum, mux to avi would be having xvid_encraw output avi and merging them. Still, if it supports for xvid it'd have to support for x264 as well.

[708145]

Quite a lot of time passed and it is time for an update.

+ ELDER4XVID is continued to be developed as GPL.
+ The muxing works in beta5
+ The XVIDcrf ratecontrol is working in my tests
+ There is a GUI in beta5 :)

- muxing is .mp4 only, .mkv will be added shortly
- support for windows cluster is commercial
- still no audio support
- resume is still not functional
- still no support for theora and snow :p

It'll take a while to package the release because I switched the PERL distribution again.

bis besser,
T0B1A5

[shpitz]

thanks tobias.

can you give us a quick review of elder? what is the benefit of using it over the other solutions?

[708145]

can you give us a quick review of elder? what is the benefit of using it over the other solutions?

* It's faster.
* It'll support resume soon. So you can resume your encode after a crash or shutdown/pause deliberately when you want to do something else.
* It supports workstation clusters (well, the GPL version not yet)

bis besser,
T0B1A5

[shpitz]

thanks.

any benefits in terms of SMP?

[708145]

any benefits in terms of SMP?

yes. The first two points in my list.

[708145]

Hi folks!

Finally the first version of the beta5 series is up. :D
It includes a GUI (very simple prototype but maybe somebody finds it useful), correct muxing and more parallelism (as long as you don't mess with ELDER's autodetection).

Enjoy 400+ fps PAL encoding on your quad cores :p

PS: My wiki is down atm. Find the download link in the first post of this thread.

bis besser,
T0B1A5

[708145]

Finally the first version of the beta5 series is up. :D
It includes a GUI (very simple prototype but maybe somebody finds it useful), correct muxing and more parallelism (as long as you don't mess with ELDER's autodetection).

PS: My wiki is down atm. Find the download link in the first post of this thread.


Wiki is up again and the download section contains the PXPerl binary as well. I mirror it because some people had difficulties getting it from the original page.

bis besser,
T0B1A5

[unknownone]

Hi,

Is it just me or the beta 5series isn't out yet? I checked both the Tobiwiki and funknmary place and they only have beta 4d avail.

I been wanting to test it out but no gui until now so please upload the beta5 version. Thanks.

[708145]

Hi,

Is it just me or the beta 5series isn't out yet? I checked both the Tobiwiki and funknmary place and they only have beta 4d avail.

I been wanting to test it out but no gui until now so please upload the beta5 version. Thanks.

Well it's my provider. There was a robbery two weeks ago => wiki gone.
So I used a backup of mine and published ELDER beta5a.
Now they got the stolen RAID back => restored the status of before the robbery and _before_ the beta5 release :(

I'm updating again atm.
<edit>done. just the changelog is missing </edit>

bis besser,
T0B1A5

[708145]

ELDER beta5b is out:
* support for windows cluster
* GUI improved a bit
* xvid 2pass "qcomp" rate control... better bitrate distribution
* x264 1pass crf mode

I hope there are not too many bugs left.
Hang on for shutdown/resume and more rate control modes.

Download link is in the first post.

bis besser,
T0B1A5

[Sharktooth]

@Tobias: would you like to integrate ELDER into MeGUI?
Distributed encoding from scratch is taking me too much time and i absolutely need to port MeGUI to linux as fast as i can...

[708145]

@Tobias: would you like to integrate ELDER into MeGUI?
Distributed encoding from scratch is taking me too much time and i absolutely need to port MeGUI to linux as fast as i can...

Humm, don't know how much effort this would be.
But if you already have an avs for your encode then using my GUI is really simple.

bis besser,
T0B1A5

[Sharktooth]

yes. MeGUI creates an .avs and feeds it to the encoder.
we just need to integrate the gui functions into megui.

[DDogg]

Completes stage A, starts stage B, does 1 chunk and then all machines report:
stage-B result xxx ready "nothing to do"
sleeping

[708145]

Completes stage A, starts stage B, does 1 chunk and then all machines report:
stage-B result xxx ready "nothing to do"
sleeping

which mode and which command line or GUI settings?

[DDogg]

I just changed the paths in the GUI and pressed start. Just barely getting up to speed on elder and don't understand enough yet to report properly.

I wrote up a quickstart for people who want to try elder but can't figure out your docs. Please correct me if incorrect on any steps:

1> Install perl on master machine
2> Install elder on master machine
3> Share and map the elder directory, on the master machine, to a drive letter, let's say "M"
4> Create sub directories in elder directory named "support", and "source". In support directory put PXPerl-5.8.7-6.exe (the perl install file). Put some sample source in "M:\source". If copying vob's and a d2v, remember to edit the d2v in notepad and change the path to "M:\source" - also edit any avs you copied, or create from scratch a new source and avs in M:\source
5> On each slave machine map drive letter M to the shared elder directory on the master machine
6> On each slave, from m:\support, install perl. Verify the sample avs will play on all slave machines before proceeding. This means avisynth, dgdecode.dll and other called plugins will need to be in the avisynth\plugins directory on each slave.
7> To start the encode - On all machines to be used, master and slaves - from the M:\ run the spawn1.bat for each core available on the machine. So for a HT P4 you would run spawn1.bat once, and for a true dual run it twice. You will see a minimized cmd window for each time you run spawn1.bat
8> After that, on the master machine, run control.bat
9> Then run guixvid.bat on master machine - Move size slider to desired size, enter M:\source\your_input avs and M:\source\your_output.mp4. Press start

You should then see information scrolling in the spawned widows on all machines. Whether it will finish and mux I have not found out yet.

Also, when a job gets aborted, certain things that I have not figured out yet need to be cleaned up or things don't work correctly after that. Before you start the encode, make a copy of the elder directory. Then you can get a fresh start by replacing the elder directory that aborted with the fresh backup. I know this is not the proper way, but temporarily it may be of use.

[foxyshadis]

You can actually share out a folder with all the perl binaries/libraries, avisynth, plugins, sources, etc. If you do the d2v on the shared folder in the local machine, it's nearly as fast as reading right from the disk, and will automatically use the right folder. This way everything definitely has the same scripts, avisynth, perl, and so on. At least, that's the way I did it, although it makes for a bit of a cluttered folder. :p

I suppose it's easier to install things though, especially if some of the slaves are *nix.

[DDogg]

That was the only way I could figure out how to do it. Call me old and linear; I need my 1,2,3 :p Hey, did you get it to finish an encode?

[foxyshadis]

Haven't tried the new GUI one, I'll give it a spin. Next job: Install totally stateless daemons/services that will always be ready to encode and return chunks of video at any time. =D

[708145]

I just changed the paths in the GUI and pressed start. Just barely getting up to speed on elder and don't understand enough yet to report properly.

I wrote up a quickstart for people who want to try elder but can't figure out your docs. Please correct me if incorrect on any steps:


I couldn't reproduce your bug but stumbled over two others in the process... I'll need some time for fixing as this is a busy week and I can only spend a few minutes per day bug hunting.

Thank you very much for the usage guide. Much appreciated.

bis besser,
T0B1A5

[708145]

Haven't tried the new GUI one, I'll give it a spin. Next job: Install totally stateless daemons/services that will always be ready to encode and return chunks of video at any time. =D

well the pick_job script is a simple replacement of that. Probably needs to get more versatile and universal.

bis besser,
T0B1A5

[708145]

new beta5c is uploaded. I hope it fixes your bug DDogg.

Changelog:
* GUI: allows to select number of CPUs
* better load balancing (when -m is defined)
* cleaner code
* nicer text output: always prints encoded time and output size error
* first chunk was sometimes missing in x264 crf mode: fixed now
* added x264 abr mode
* added xvid crf2 mode: same size and bitrate distribution as x264's crf mode
* fixed a few rate control bugs... still undersizes sometimes

Next on the list:
* fixing remaining bugs
* improving the GUI: all modes, file selector, ...
* get pause/resume working
* integrate into MeGUI?
* add an xvid HFR mode (high frame rate: 50p and 60p)
* add x264 3pass

bis besser,
T0B1A5

[DDogg]

gave it a quick try and got the same thing so perhaps it is user error. It finishes stage A, says the import is done, and then says nothing to do.

If you will give me a procedure, or point out what exact information you need to diagnose the problem, I will get it for you.

[708145]

gave it a quick try and got the same thing so perhaps it is user error. It finishes stage A, says the import is done, and then says nothing to do.

If you will give me a procedure, or point out what exact information you need to diagnose the problem, I will get it for you.

The two outputs are in 2 different windows, right?
The nothing to do is in the compute process and the "import done" in the control process?

Try the following order:
0) make sure everything is installed but no ELDER process running
1) run shutdown.bat to clean the jobfile
2) run initjobs.bat to init the jobfile
3) run control.bat to start up the control job
4) spawn all clients with spawn1.bat... as many as you like (even on remote PCs)
5) start the GUI and issue your job

Important: Make sure your filenames don't contain spaces.
For testing just start with one control job and one compute job on your main computer.

bis besser,
T0B1A5

[DDogg]

followed the instructions exactly. Used one control and one spawn on master machine. Stage C completed. Results, the mp4 file was unplayable. Window said:
2nd pass fps = 36.94
total time for encoding: 735 seconds
overall fps = 20.4
now merging the chunks...
chunks merged
target: 101711872 bytes
encode: 56189830 bytes
error: 44.76%
another job well done
Source avs:
DGDecode_mpeg2source("M:\source\VTS_02_1.d2v",info=3)
ColorMatrix(hints=true)
crop( 0, 60, -2, -64)
LanczosResize(640,272) # Lanczos (Sharp)
trim(5000,20000)


Starting to think it is a gui problem. [] Give me a way to copy the cmdline from the gui.

[708145]

followed the instructions exactly. Used one control and one spawn on master machine. Stage C completed. Results, the mp4 file was unplayable. Window said:
2nd pass fps = 36.94
total time for encoding: 735 seconds
overall fps = 20.4
now merging the chunks...
chunks merged
target: 101711872 bytes
encode: 56189830 bytes
error: 44.76%
another job well done
Source avs:
DGDecode_mpeg2source("M:\source\VTS_02_1.d2v",info=3)
ColorMatrix(hints=true)
crop( 0, 60, -2, -64)
LanczosResize(640,272) # Lanczos (Sharp)
trim(5000,20000)


Starting to think it is a gui problem. This is an xvid encode with avi specifiied, why the forced mp4?

Currently it can only mux to mp4, so the GUI automatically adds the correct extension. I will add muxing to avi later (for the xvid part). Avi will not be supported for x264 for obvious reasons. I'm also thinking about writing .mkv.

But that's a huge error you get plus an unplayable file. I guess they are correlated. Errors for me are in the 6% range for short samples and <2% for complete movies.
How many chunks were used? I played with the autodetection formula and didn't test with many cases before releasing beta5c.

Of course testing the cli would help tremendously.

bis besser,
T0B1A5

[708145]

Starting to think it is a gui problem.


New beta5d is up. Now has a reset SMP button (please use before encoding to set up internal files) as well as a start SMP button to start the master.

Additional clients still have to be added by hand. If someone knows how to automate this on windows please tell me.

beta5d:
* added SMP control to GUI
* prints frames in the control window for xvid 2pass
* fixed a bug in chunk computation
* starts processes in lower priority

bis besser,
T0B1A5

[DDogg]

sorry, all i get is "nothing to do" in the windows. I think I did what you wanted, but to be sure give me a 1,2,3

[708145]

sorry, all i get is "nothing to do" in the windows. I think I did what you wanted, but to be sure give me a 1,2,3

OK.
There is a detailed usage guide in the README. I'll sum it up here:

1) as before, install everything and make sure ELDER is not running.
2) start the guixvid.bat
3) hit reset in the gui
4) select number of CPUs on your main PC
5) hit start SMP
6) optional: add more slaves via spawn1.bat on other PCs in the same directory
7) select size, encoding options, input and output and hit "add job"
8) wait for encode to finish / or add more jobs
9) hit "reset" after all is done and "quit"

EDIT: ouch, my bad. I commented out a line I shouldn't have :) I put a fixed version online. it now even starts a control process.
EDIT2: Just worked for me on a 15K frames sample (50fps overall).

bis besser,
T0B1A5

[DDogg]

<crap> #7 Probably should do a "Nothing in queue, you complete dummy" type of thing.

Results still not playable. /add:Hmm, nero showtime will play it, but MPC will not. Mpc will play my other mp4 samples. Probably a problem on my end, but could you check with MPC on your end?

stage C progress: 14/16 chunks done (87%) in 463 seconds. 13127 frames so far.
stage C progress: 15/16 chunks done (93%) in 495 seconds. 14195 frames so far.
stage C progress: 16/16 chunks done (100%) in 500 seconds. 15001 frames so far.
stage C completed
2nd pass fps = 52.82
total time for encoding: 500 seconds
overall fps = 30
now merging the chunks...
chunks merged
target: 83886080 bytes
encode: 53967994 bytes
error: 35.67%
another job well done

[708145]

Results still not playable. /add:Hmm, nero showtime will play it, but MPC will not. Mpc will play my other mp4 samples. Probably a problem on my end, but could you check with MPC on your end?

stage C progress: 14/16 chunks done (87%) in 463 seconds. 13127 frames so far.
stage C progress: 15/16 chunks done (93%) in 495 seconds. 14195 frames so far.
stage C progress: 16/16 chunks done (100%) in 500 seconds. 15001 frames so far.
stage C completed
2nd pass fps = 52.82
total time for encoding: 500 seconds
overall fps = 30
now merging the chunks...
chunks merged
target: 83886080 bytes
encode: 53967994 bytes
error: 35.67%
another job well done

:) So this means you did get it working. Great!
Could you now test with more CPUs (slave machines)? IOW the "optional" step 6)

I haven't used DirectShow based players in a while. I'll install mpc+ffdshow and see. I suspect an issue with mp4box.

After that we can address the huge undersize you experience. Is it a reasonable bitrate you ask for? Maybe the codec is saturating? Could be some error on my side as well although my test encodes come in at +-6% usually.
There are several ways to improve size accuracy just haven't implemented any yet. The final version of ELDER should get <1% error in all cases of at least a few minutes length.

bis besser,
T0B1A5