x264 parallel encoder [Archive] - Page 3

Revgen

9th February 2006, 00:28

I don't think it makes any sense to do these tests on it with several known killer bugs, when a bugfix release should be coming shortly. Otherwise, of course results are going to be thrown way off.

I'm testing it so that it can be improved. This program is a beta after all.

On top of that, 708145 asked us to do these tests in his first post.

ChrisBensch

9th February 2006, 02:57

Edit: I noticed that I didn't include the stats file, when trying to use --stats "c:\temp\elder\154754\stageC-0.stats" in the CLI, x264 complains giving an error:

x264 [error]: ratecontrol_init: can't open stats file
x264_encoder_open failed

I'm not sure where to go next, I guess it's just time to wait for the next beta.

Original:

After messing around a bit, I decided to stop ELDER just when it starts to encode the first two chunks. Then I went into the stageC batch files and changed the x264 command line from the broken one to:

..\x264.exe --progress --keyint 240 --ref 5 --bframes 3 --b-pyramid --weightb --filter -2,-1 --subme 6 --trellis 1 --analyse all --8x8dct --no-psnr -B 4368 -o test0.264 stageC-0.avs

I then launched the batch files two at a time (selected two in Windows explorer and then hit the <ENTER> key, launched them both). I have to say I am a bit disappointed, both encodes ran fine but the speed was exactly the same as if it were using MeGUI. The first passes ran mighty quick, but the second pass was the same. Overall it's a speed up, but the second pass was the one I was hoping would have the most improvement.

blubberbirne

9th February 2006, 22:36

@708145

Hope you can get all x264 parameter working. i see big picture differents, because your script doesn't like all settings :(

708145

17th February 2006, 01:21

@708145

Hope you can get all x264 parameter working. i see big picture differents, because your script doesn't like all settings :(

beta4c is out :D

please check if parameters work and if the muxing goes all right.
if the quality is constant and it hits target size in the ELDER encodes then I can move along to
* the linux port
* better scaling with more CPUs
* improved rate control for xvid
* GUI

Revgen

17th February 2006, 01:58

I just tried it.

There seems to be a problem with Stage C. It seems to take about 5 secs while stage A takes about 15 min.

And sure enough, after the encode is done, there is no MP4 file in the directory where I specified with the -o command.

Here's my script

perl ELDER4X264_start.pl -a H:\BLAZING_SADDLES\VIDEO_TS\saddles.avs -m 2 -b 1500 -c 4 -o H:\saddle.mp4 -q "--keyint 240 --ref 5 --bframes 3 --b-pyramid --b-rdo --bime --weightb --subme 6 --analyse all --8x8dct --me umh --merange 20"

708145

17th February 2006, 02:06

There seems to be a problem with Stage C. It seems to take about 5 secs while stage A takes about 15 min.

after Stage A completes there should be some debug output. Could you paste this please. I think there are some wrong values (maybe zeroes) in the size array.

edit: I figured it out. It's my stupidity when reading the command line. I will fix it tomorrow.

Revgen

17th February 2006, 02:11

Debug Output?

Can you give me a hint? :confused:

EDIT:

Is this a new command that isn't in the Readme.txt?

blubberbirne

17th February 2006, 09:39

beta4c is out :D

please check if parameters work and if the muxing goes all right.
if the quality is constant and it hits target size in the ELDER encodes then I can move along to
* the linux port
* better scaling with more CPUs
* improved rate control for xvid
* GUI

sounds good :thanks:

i'll test later when i'm back from work :p

708145

17th February 2006, 12:51

Debug Output?
Can you give me a hint? :confused:

EDIT:
Is this a new command that isn't in the Readme.txt?

there are output lines starting with "debug:" in beta4c. But those values seem to be correct. The fault is in the command line parsing :rolleyes:

I'll update this. Hang on for beta4d. :)

bis besser,
T0B1A5

blubberbirne

17th February 2006, 14:03

perl ELDER4X264_start.pl -a G:\cut\x264test.avs -m 2 -b 3400 -c 6 -o G:\cut\x264test.mp4 -q " --ref 16 --mixed-refs --no-fast-pskip --bframes 3 --b-pyramid --b-rdo --weightb --filter -2,-1 --subme 7 --trellis 2 --analyse all --8x8dct --me umh --progress --no-psnr "

This one doesn' work

708145

17th February 2006, 16:53

This one doesn' work

As a quick workaround I suggest to change the defaults in ELDER4X264_acrf.pl to your settings and don't specify -p and -q on the command line.

If this is too much hassle for you then I suggest to wait for 4d.

blubberbirne

17th February 2006, 20:21

If this is too much hassle for you then I suggest to wait for 4d.

Ok, waiting for 4d :)

Revgen

18th February 2006, 05:56

Okay, I changed the default settings in the ELDER4X264_acrf.pl file to the ones I posted earlier that didn't work.

The video turned out fine. No muxing issues. Quality is much better. Video plays back at 23.98FPS in VLC player instead of 25.000FPS like last time. The length of the video is also correct.

However, I will wait to do speed tests and quality comparisons once the 4d version is released.

708145

18th February 2006, 13:25

The video turned out fine. No muxing issues. Quality is much better. Video plays back at 23.98FPS in VLC player instead of 25.000FPS like last time. The length of the video is also correct.

However, I will wait to do speed tests and quality comparisons once the 4d version is released.

Great! :)

I'm cleaning up the code a bit and reimplement parameter passing from scratch.

The quality tests are of prime importance since the scaling I can do myself since for the same -c value the output is bit identical independent of CPU count.

bis besser,
T0B1A5

Revgen

18th February 2006, 21:53

Okay I tried my goofs video clip, and this one still does have muxing issues. I'm not sure why though.

The video quality, framerate, etc are fine though.

Revgen

19th February 2006, 19:20

I decided to take a closer look at how my clips are being encoded to see why one muxes correctly and the other muxes incorrectly, and I found something interesting:

Goofs (unsucessfull mux):

Finishes 1st: Stage A-0
Finishes 2nd: Stage A-1
Finishes 3rd: Stage A-3 *
Finishes 4th: Stage A-2 *

Finishes 5th: Stage C-0
Finishes 6th: Stage C-1
Finishes 7th: Stage C-3 *
Finishes 8th: Stage C-2 *

Saddles (sucessfull mux):

Finishes 1st: Stage A-0
Finishes 2nd: Stage A-1
Finishes 3rd: Stage A-2
Finishes 4th: Stage A-3

Finishes 5th: Stage C-0
Finishes 6th: Stage C-1
Finishes 7th: Stage C-2
Finishes 8th: Stage C-3

*It seems to me what differentiates the Saddles clip from Goofs clip is that Saddles finishes all of it's stages sequentially while Goofs is insequential on the 3rd and 4th stage. Although ELDER does wait for the 3rd stage to finish before muxing or passing on to stage C, there might be a issue with the way the stages are stored that may be causing problems with muxing.

708145

19th February 2006, 22:34

*It seems to me what differentiates the Saddles clip from Goofs clip is that Saddles finishes all of it's stages sequentially while Goofs is insequential on the 3rd and 4th stage. Although ELDER does wait for the 3rd stage to finish before muxing or passing on to stage C, there might be a issue with the way the stages are stored that may be causing problems with muxing.

I'll try to reproduce this. I'm quite sure the muxing command line still is correct. But mp4box might use file timestamps to determine muxing order instead of parameter order? :confused:

Revgen

21st February 2006, 00:18

Any luck yet?

708145

21st February 2006, 07:16

Any luck yet?

Sorry I was held back by real life and work. Gonna try tonight :)

elfurbe

21st February 2006, 21:32

708145

21st February 2006, 21:59

Following the thread. Just wondering, you mention a Linux software for DVD converting on your site, but most of the development seems to be on this Windows project. I'm currently putting together a Linux video encoding cluster for academic purposes, and I'm very interested in your project for doing parallel x264 encodes from high-def MPEG2. Is there a current Linux version I can try out or is that coming after the Windows version works?

Good to hear that :)
The original linux project will not be developed further. But the port of the current ELDER code to linux happens via wine at the moment. When avisynth is available native for linux then the wine dependency will fall :D

I'll release the linux port when I think it works good enough to bother others with testing :p

What you could do now is set up an NFS so that all clients see a common directory. Further you'll need wine with avisynth 2.5.6 installed. Add all the avs filters you need as well. All the rest will ship in the linux package. :)

I'm really looking forward to testing on a cluster myself and tweaking scalability there since we have a very heterogeneous cluster wrt. speed (slowest node is ~10x slower than fastest node). But this speedup tweaking is a hobby of mine :)

bis besser,
T0B1A5

Revgen

11th March 2006, 17:12

How's the project going so far?

Were you able to detect the problem?

708145

12th March 2006, 17:03

How's the project going so far?
Were you able to detect the problem?

I located the muxing problem and the fix is underway... but no more ELDER today...

Revgen

12th March 2006, 17:05

Thanks. I won't bother you anymore about it.:D

Revgen

26th April 2006, 17:38

@708145

Have you looked into possibly using mp4creator from the mpeg4iptools suite instead of mp4box. It's harder to find, but it could be more stable than mp4box. I have the April 23, 2005 build and can send it to you if you're interested.

Revgen

18th May 2006, 14:55

Here's a quote from forum post http://forum.doom9.org/showthread.php?p=829027#post829027

A not so short note first:
ELDER is capable of just using 4 chunks, just use the -c option to specify number of chunks :p
More chunks is there to improve load balancing on several cores since the segments differ in encoding speed your total time would otherwise depend on the slowest part! More segments help with resume as well, aka shutdown during encode and continue later (maybe after some gaming session or TV capture or holidays or...)

A complete 1st pass was my pick which is the most accurate and reasonably fast.
But even when you got relative difficulty of the chunks how would you encode them without a stats file? I hope CBR is out of the question.

About muxing: I solved the muxing issue I had which had to do that mp4box used the timestamps of the chunks instead of the order given in the command line! :S

One open issue:
If someone wants to contribute a GUI for ELDER I can spend my time on the advanced rate control modes I plan like an CRF like mode for xvid and the high fps mode called xvid50p which is optimized for 50p and 60p encodes more and more people out there are doing. At same quality 50p consumes just ~20% more than 25p using xvid50p! So it's time for fluid motion!

bis besser,
708145

Sounds good. I like to encode my 720p (60p) HDTV caps alot since I've gotten it. This should make it easier to encode.

I'm awaiting the new release.;)

havix

3rd October 2006, 22:49

708145

4th October 2006, 00:13

Is anymore work being done on this? Also why was it decided that only Linux would have cluster support? This seems rather shortsighted since most people that have multiple machines will be Windows based.

I plan to get back on this project... when time permits.

I have nothing against a windows cluster mode. I just stated that I won't implement it but I'm open to contributions :)

bis besser,
708145

Sharktooth

4th October 2006, 15:12

it's getting its way into megui... :)

ChrisBensch

5th October 2006, 04:14

I can't wait...I have 4 machines at home waiting to crunch through my HD captures!

tomos

5th October 2006, 21:51

it's getting its way into megui... :)

yay :thanks:

will be so much faster encoding with this :)

Revgen

7th October 2006, 06:55

I plan to get back on this project... when time permits.

I have nothing against a windows cluster mode. I just stated that I won't implement it but I'm open to contributions :)

bis besser,
708145

I'm still here to test if you need me.:D I still have the same movie that didn't mux right the last time.

708145

16th October 2006, 15:31

I'm still here to test if you need me.:D I still have the same movie that didn't mux right the last time.

Thanks for all the testing.

I'm not sure how to proceed with ELDER4X264. It might go commercial.
I'll get the last release bugfixed nevertheless.

ELDER4XVID will remain in GPL-land and I will add SNOW and Theora encoding shortly :D Maybe someone can convince me to support Dirac. :p
Per hour I work on a commercial product I'll spend some time on the GPL side.

bis besser,
T0B1A5

Sharktooth

16th October 2006, 15:54

well... x264 is not a commercial product though...

708145

17th October 2006, 18:53

well... x264 is not a commercial product though...

The good thing about GPL is that the current state of the code will stay GPL. And I might even add some encoding modes later, but the ELDER fork has priority for me.

I'm not sure how to proceed with ELDER4X264. It might go commercial.
I'll get the last release bugfixed nevertheless.

So here it is: the bugfixed mux-enabled version of ELDER4X264.
Grab your copy of beta4d here:
http://www.funknmary.de/bergdichter/projekte/index.php?page=ELDER

changelog for beta4d:
* fixed muxing to mp4
* split the project from ELDER4XVID

bis besser,
T0B1A5

bkman

18th October 2006, 05:56

Tobias, do you think you can support single-pass (crf) parallelisation in Elderx264?

708145

18th October 2006, 06:48

Tobias, do you think you can support single-pass (crf) parallelisation in Elderx264?

Yes. That is one of the encoding modes I was speaking about here. But I can't tell when I'll get around to that as other ELDER parts (xvid, audio, live streaming) have higher priority for me.
OTOH, it is not much effort to hack it in.

And I might even add some encoding modes later, but the ELDER fork has priority for me.

bis besser,
T0B1A5

Revgen

19th October 2006, 17:00

The good thing about GPL is that the current state of the code will stay GPL. And I might even add some encoding modes later, but the ELDER fork has priority for me.

So here it is: the bugfixed mux-enabled version of ELDER4X264.
Grab your copy of beta4d here:
http://www.funknmary.de/bergdichter/projekte/index.php?page=ELDER

changelog for beta4d:
* fixed muxing to mp4
* split the project from ELDER4XVID

bis besser,
T0B1A5

Great! I'll test it eventually. Perhaps this weekend.

708145

21st November 2006, 13:31

Great! I'll test it eventually. Perhaps this weekend.

There is even more to test: The latest beta5 series of ELDER contains x264 modes. I'll integrate a keyframe accurate 2pass mode soon.
Oh, and there is a fancy GUI as well :)

bis besser,
T0B1A5

woah!

6th December 2006, 08:33

i have an issue with the import() function within the made up avs scripts :(

import("default.avs")

is all it loads in and not the path to the avs? so it doesnt find the script and exits.

so my stageA-0.avs has:

import("default.avs")
trim(0,15370)

my cmd log is below:

C:\ELDER>perl ELDER4X264_start.pl -a default.avs -o ELDER.mp4 -p "-b 2 -r 1 -m 4
" -q "-b 2 -r 5 -m 4"
The system cannot find the file specified.
### encode job 363586 set up successfully
### starting encode now

C:\ELDER>perl -w ELDER4X264_acrf.pl -t 363586 -a default.avs -o ELDER.mp4 -p -b
2 -r 1 -m 4 -q -b 2 -r 5 -m 4
detected fps = 59.94
detected frames = 122958
using 4 threads to encode 8 chunks with about 15370 frames each.
Total size will be 525146 bytes.
### starting process 0
### starting process 1
### starting process 2
### starting process 3
### stage A issued

C:\ELDER>perl -w ELDER4X264_c2.pl 363586
####### Stage A Progress: 1/8 chunks done (12%) #######
####### Stage A Progress: 2/8 chunks done (25%) #######
####### Stage A Progress: 3/8 chunks done (37%) #######
####### Stage A Progress: 4/8 chunks done (50%) #######
####### Stage A Progress: 5/8 chunks done (62%) #######
####### Stage A Progress: 6/8 chunks done (75%) #######
####### Stage A Progress: 7/8 chunks done (87%) #######
####### Stage A Progress: 8/8 chunks done (100%) #######
### Stage A completed
### Stage C generating jobs
Illegal division by zero at ELDER4X264_c2.pl line 52.

C:\ELDER>perl -w ELDER4X264_d.pl 363586

i looked at that line and it has this:

$chunktarget = ($totalbytes * $chunksizes[$i]) / $total_chunksize;

then my head exploded...

woah!

6th December 2006, 09:42

hmmm its now working but i dont really know what i did to solve it heh...

flat lined quadcore:

http://images.dr3vil.com/files2/default/100QX.jpg

708145

6th December 2006, 15:40

Wow! Nice.

I decided to include x264 2pass into beta5 in one of the next releases.

bis besser,
T0B1A5

woah!

7th December 2006, 02:46

its doing 2pass ok right now for me?

i added some default x264 settings to your ELDER4X264_acrf.pl and it is works great with all the settings i need at 210fps total on the 1pass for this setup at 448x336 reizing.

seems i have found my tool of choice thank you :)

http://images.dr3vil.com/files2/default/210fps.jpg

heres the 2pass result of 80fps total:

http://images.dr3vil.com/files2/default/80fps.jpg

rate control seems good aswell.

708145

7th December 2006, 11:32

its doing 2pass ok right now for me?
seems i have found my tool of choice thank you :)
rate control seems good aswell.

Yes it was working in beta4 for a long time. But the fact is I abandoned that branch. beta4 is/was only usable to hard core users like you. :p
OTOH there were vague plans to include it into the beta5 and 6 branch again (and even better :D) with a GUI, better rate control, pause/resume feature.
I decided a week ago to actually _do_ that. So you could already try to get beta5 working on your setup since I'll add x264 2pass real soon :)

bis besser,
T0B1A5

Al79

10th December 2006, 00:02

Yes it was working in beta4 for a long time. But the fact is I abandoned that branch. beta4 is/was only usable to hard core users like you. :p
OTOH there were vague plans to include it into the beta5 and 6 branch again (and even better :D) with a GUI, better rate control, pause/resume feature.
I decided a week ago to actually _do_ that. So you could already try to get beta5 working on your setup since I'll add x264 2pass real soon :)

bis besser,
T0B1A5

:cool: :o :thanks: Im waiting for that beta, really, thank you very muchhhhhhhhh tobias, ur great :sly: