x264 parallel encoder

[MeteorRain]

Of course since this is the reason I started ELDER development.
The first ELDER version could do so but scaled poorly (~11x speedup on 24nodes)

But I will not do a windows cluster mode. Cluster mode will be linux only.

bis besser,
T0B1A5
囧 only support linux?
BTW, is it possible to hack the x264 core and make it a C/S mode encoder? whose server is a AVS Reader + mp4/mkv/raw Writer, while the clients are the encoder thread?
in local mode, it works in namepipe, and in LAN, works on TCP/IP network, just like MySQL works?

;) just my thought :)

[Revgen]

I tried using the new version with more advanced settings than I used the last time, and I ran into probelems.

Using this script:

perl ELDER4X264_start.pl -a E:\3s-Goo~1\goofs2.avs -f 16501 -m 2 -b 1500 -c 4 -o E:\3s-Goo~1\goofs.mp4 -q "--pass2 --keyint 240 --stats ".stats" --ref 5 --mixed-refs --bframes 3 --b-pyramid --b-rdo --bime --weightb --subme 6 --analyse all --8x8dct --me umh --merange 20"

Gave me a 4.38 MB file for an 11minute 38 second video and the results are awful. Obviously, it's not encoding at 1500 kbps.

Is there a reason why?


EDIT: Okay I eliminated --pass2, --stats, and the --mixed refs parameters and the video doesn't look awful anymore. Unfortunately the muxing bug from the last version still remains. My file is 5:30 instead of 11:38.

This doesn't seem to be an mp4box problem, since ELDER4XVID doesn't have this issue.

[Hellworm]

But I will not do a windows cluster mode. Cluster mode will be linux only.

So, is the hole cluster going to be linux or is the server windows (because of avisynth)?
What will be streamed over the network, the already decoded raw video or the undecoded stream ?

[blubberbirne]

Most parameters of x264 doens't work :(
But, this paramaters are a must have.

Deklocking parameter is one of them

[708145]

EDIT: Okay I eliminated --pass2, --stats, and the --mixed refs parameters and the video doesn't look awful anymore. Unfortunately the muxing bug from the last version still remains. My file is 5:30 instead of 11:38.

This doesn't seem to be an mp4box problem, since ELDER4XVID doesn't have this issue.

The muxing command is almost the same (excpept for .264 and .m4v file extensions of the chunks) => I really think it is mp4box.
Maybe it handles different raw streams / different extensions differently.

As for the "missing" options: I know what goes wrong and it is #1 on the TODO :D

bis besser,
T0B1A5

[708145]

So, is the hole cluster going to be linux or is the server windows (because of avisynth)?
What will be streamed over the network, the already decoded raw video or the undecoded stream ?

Finally it will be avisynth3 on linux all the way. As a prototype I might go for 2.56 under wine. Either way it's the undecoded stream that is fed to the clients.

bis besser,
T0B1A5

[bond]

Unfortunately the muxing bug from the last version still remains. My file is 5:30 instead of 11:38.

This doesn't seem to be an mp4box problem, since ELDER4XVID doesn't have this issue.is everything in the created .mp4? are no parts missing? is the framerate correct?

when muxing raw .264 to .mp4 with mp4box you always need to set a framerate, otherwise mp4box will use 25fps

[Revgen]

Half of the "chunks" aren't muxed properly for some reason, so the file is cut in half. The video plays fine except for the missing parts, so it isn't a framerate issue.

[bond]

what do you mean by "chunks"? are frames missing?

[Revgen]

ELDER divides the video into "chunks" while encoding, which allows the encoding to be more balanced between the 2 cores when using multiple threads. Once these chunks are finished they are merged into one video. For some reason these chunks aren't being muxed properly. So, yes, frames are missing from the video.

[bond]

so elder propably feeds mp4box with a wrong or incomplete muxing command so it doesnt mux all of the chunks

btw are there more than one video stream in the mp4 created by elder?

[708145]

so elder propably feeds mp4box with a wrong or incomplete muxing command so it doesnt mux all of the chunks

btw are there more than one video stream in the mp4 created by elder?

Gonna print the mux command in the next beta for your convenience. Then you can tell me if the command line is wrong or mp4box is misbehaving.

bis besser,
T0B1A5

[Revgen]

Wow you're a genius!

There is indeed 2 video streams in the file. I didn't check for that.

Both streams look good. Here is how they are separated:

Sream 1 - Beginning of video, end of video

Stream 2 - First middle part of video, second middle part of video

I used 4 chunks when encoding so this is how they are separated.

[bond]

thats because for some reason mp4box recognized those streams to not use the same settings
its only allowed to merge streams which use the same settings, i think mp4box checks the SPS/PPS for that

maybe you can extract the raw streams from the mp4 and run the h264_parse tool from mpeg4ip over them
this will show you the structure of the headers (SPS/PPS) and if they are identical

[Revgen]

I ran h264_parse over both of the the streams. Stream 1 took a while and I was only able to copy the last of what was done. Stream 2 took a very short time amd ended with a message saying "couldn't find start code in buffer from 0"

[ChrisBensch]

After trying to get ELDER to work for quite some time, I've had no luck:

"perl ELDER4X264_start.pl -a D:\$VIDEO_WORK$\WORKING\test-elder.avs -m 2 -b 4000 -c 4 -o c:\temp\test-elder.mp4 -q "--bframes 3 --b-pyramid --filter -2,-1 --subme 1 --analyse none --me dia --progress --no-psnr""

I did create some batch files from the CLI provided by MeGUI and tried two simultaneous instances of x264 and I got about a 50% increase in speed (~8.5fps with MeGUI with two threads to 13fps with the batch tests). I'm really interested in ELDER because of this increase since most of my encodes are HD sources.

I've tried a whole bunch of times with various settings. I copied the CLI output from MeGUI and nothing seems to work. I've attached a picture of my command window in hopes that maybe someone can see something I'm doing wrong

[Revgen]

Try this and see if it works. Please don't ask why.;)

"perl ELDER4X264_start.pl -a D:\$VIDEO~1\WORKIN~1\test-~1.avs -m 2 -b 4000 -c 4 -o c:\temp\test-e~1.mp4 -q "--bframes 3 --b-pyramid --filter -2,-1 --subme 1 --analyse none --me dia --no-psnr"

[ChrisBensch]

Unfortunately, it didn't help, I get stuck at the same point. It appears to be stopping right when the "b" perl script is being called:

ex: perl ELDER4XVID_b.pl 776519

Something else to note, it's not detecting my FPS from the AVS script:

detected fps = format
detected frames = 2001

[Revgen]

Are there actually 2001 frames in your video, or is that a misread by ELDER?

2001 frames are too short for using 4 chunks. Try increasing it to about 8000 or even 16000 as "708145" suggests.

If it's a misread than use the -f numberofframesinyourclip before the -q "" parameter starts.

EG: your clip is 16000 frames then use:

"perl ELDER4X264_start.pl -a D:\$VIDEO~1\WORKIN~1\test-~1.avs -m 2 f- 16000 -b 4000 -c 4 -o c:\temp\test-e~1.mp4 -q "--bframes 3 --b-pyramid --filter -2,-1 --subme 1 --analyse none --me dia --no-psnr"

[ChrisBensch]

Yes, the 2001 frames is correct, but a mistake on my part. I had shrunk the frame amount for other testing. I'll increase the frame count to 16000 and try again. By the way, my setup is that on one hard drive I had elder then on another I had all of my video tools. I had to move the AVS script into the same dir and make an absolute reference to my .d2v within that script as things are on separate drives. By the way, the output from the earlier posted setup was terribly blocky and you could almost not even tell what the video was.

Another try coming...

[Revgen]

By the way, the output from the earlier posted setup was terribly blocky and you could almost not even tell what the video was.

I had this problem too but I solved it by removing --pass2 --stats and --mixed-refs

Perhaps there are other parameters that cause problems for ELDER. The only parameters that you are using that I didn't use was "--no-psnr" and "--filter". Perhaps these are causing problems.

[ChrisBensch]

Another try in progress, instead of moving and editing the AVS to have an absolute path to the .d2v, it just needed the absolute path (mpeg2source("d:\videowork\test.d2v") not (mpeg2source("test.d2v"). Seems to be working fine so far.

Edit: The job has finished. Using MeGUI on the HQ-Slow preset, I would get about 8.5fps on the first pass and then about 3.5fps on the seconde with my HD sources on my 2.8GHz dual core. With the same source and AVS, I'm getting about 14.3fps (with 2001 frames, redoing with 16K frames) with ELDER for the one pass. I would however, like to use two-pass like MeGUI but am unsure as to exactly how I would do that. I am just copying the cli from MeGUI for testing so that I have at least a somewhat reasonable comparison between the two.

[ChrisBensch]

One thing that would be nice is if you were somehow able to capture the average FPS for the whole process without having to sit and watch the output windows. Maybe write it to a file for info purposes?

[Revgen]

Is your encode the right length?

[ChrisBensch]

Before I go any further, I'm gonna sleep on it. My eyes are crossing as I've been up way to long to be doing this tonight. All I can say is ELDER is very interesting and I'll be back at it tomorrow. Thanks to all.

[bond]

I ran h264_parse over both of the the streams. Stream 1 took a while and I was only able to copy the last of what was done. Stream 2 took a very short time amd ended with a message saying "couldn't find start code in buffer from 0"i need the sequence and picture parameter at the beginning of h264_parses output

[Revgen]

i need the sequence and picture parameter at the beginning of h264_parses output

Okay I have the beginning sequence for Stream1.

I had to go into safemode command prompt form of windows to do it. My machine is way too fast in normal mode, hence their was no chance to pause the operation and copy anything.

[bond]

thx but we would need both for comparing if the streams use the same settings

btw passing the output of h264_parse to a txt is very easy via the following cmdl:
h264_parse raw.264 > h264_log.txt

[Revgen]

The stream2.txt in my previous post (post #65) has the beginning info for second stream.

Only stream1.txt (in the previous post) was cut short.

[bond]

hm they are identical, except the bitrate in the SEI, tough i dont think that mp4box checks the SEI

actually i have no idea what could be wrong on the mp4box side

maybe elder uses not -cat everywhere but -add maybe?

[foxyshadis]

No, everything is -cat, it's just one tiny loop over all pieces.

708145, is it really necessary to have so many "$pathtoencraw =~ s/\//\\/g;" type lines? Windows understands forward slashes fine.

[708145]

No, everything is -cat, it's just one tiny loop over all pieces.
708145, is it really necessary to have so many "$pathtoencraw =~ s/\//\\/g;" type lines? Windows understands forward slashes fine.

I never tested :shame:
I'll delete them then :)

[foxyshadis]

So I was scoping out the perl and found a few things. You might want to change:

$tmpdir=int(rand(1000000));

to

$tmpnum=int(rand(1000000));
$tmpdir="/var/tmp/".$tmpnum; # /tmp? /home/tmp? dunno.
#WINDOWS#
$tmpdir="%temp%\\".$tmpnum;
#WINDOWS#

(mkdir is one instance where / doesn't work). That way there's no write-permission issues. And also check for existence of the avs before going on to encode nothing (it hangs, I'm not quite sure where though). I'm tinkering with it to see if I can replicate revgen's issues.

[708145]

So I was scoping out the perl and found a few things. You might want to change:

$tmpdir=int(rand(1000000));

to

$tmpnum=int(rand(1000000));
$tmpdir="/var/tmp/".$tmpnum; # /tmp? /home/tmp? dunno.
#WINDOWS#
$tmpdir="%temp%\\".$tmpnum;
#WINDOWS#

(mkdir is one instance where / doesn't work). That way there's no write-permission issues. And also check for existence of the avs before going on to encode nothing (it hangs, I'm not quite sure where though). I'm tinkering with it to see if I can replicate revgen's issues.

good idea. but I have to change the pathto.. variables then as well to absolute paths.

maybe setting up a svn would be wise as well. this way motivated people like you can contribute directly :)

[Revgen]

Okay, I've tried ELDER on a different movie than what I was using earlier. And the encode turned out fine this time with no muxing issues. Very strange:confused:

I'm going to try to see if I can replicate the issue with some other sources.

[ChrisBensch]

Edit: After this was finished, I checked the video and it is extremely blocky and I'd say 100% unwatchable...

I've done a very small scale test to see the speed increase I can come to expect and I'm pretty impressed. I've posted the AVS and CLI I used below. The HDTV source is 1920x1080 that was capped from a DVR. I'm seeing a speed of about 25.12fps for the first pass and 14.10fps for the final pass with 100% cpu util with ELDER. As I stated in earlier posts, I was seeing a first pass of 8.5fps and second pass of 3.5fps @ 80% cpu util with MeGUI/x264.

Here is my AVS:

LoadPlugin("D:\$VIDEO_WORK$\bin\AviSynth Plugins\DGDecode.dll")
LoadPlugin("D:\$VIDEO_WORK$\bin\AviSynth Plugins\RemoveGrainSSE3.dll")
MPEG2Source("D:\$VIDEO_WORK$\WORKING\The Matrix Reloaded-HDTV.d2v")
Crop(2,2,-2,-6)
LanczosResize(1280,608)
RemoveGrain(mode=2)
Trim(20000,23999)

Here is the command line I used:

perl ELDER4X264_start.pl -a D:\$VIDEO_WORK$\WORKING\test-elder1.avs -m 2 -b 4000 -c 4 -o test-elder.mp4 -q "--keyint 240 --ref 5 --bframes 3 --b-pyramid --weightb --subme 6 --trellis 1 --analyse all --8x8dct --progress"

[Revgen]

Try removing --progress and see what happens

[ChrisBensch]

Just for the hell of it, I'm trying a different build of x264. The version included with ELDER is build 400 svn, I'm sub'ing it with build 397 from sharktooth.

Also, I noticed that during the encoding stages, the cli looks like this:

..\x264.exe --progress -B 0 -o test3.264 stageC-3.avs

I don't see any of the special options I passed to it on the cli???

[ChrisBensch]

The only difference of note when changing x264 builds is that the file was 80KB larger...

I'll try removing "--progress" and see what happens

[Revgen]

The "..\x264.exe --progress -B 0 -o test3.264 stageC-3.avs" is just a temporary stage. It is used in my encodes too.

It doesn't hurt to try another version of CLI, but IMHO the problem lies with ELDER.

[708145]

..\x264.exe --progress -B 0 -o test3.264 stageC-3.avs
I don't see any of the special options I passed to it on the cli???

My bad. I just noticed I did not rename the variable holding the passed settings everywhere. Thus the 2nd pass options don't get used :/
Expect a bugfix release in a few days, there is a lot to fix :p

edit: this also explains the extreme blockyness some of you experienced.

[ChrisBensch]

The "--progress" option was killing the video. Now my 4000 frame clip is ~80MB...and the quality is what I expect.

Edit: The "--no-psnr" option works fine too...as long as it's not the last option on the cli..
--no-psnr --8x8dct" worked where --8x8dct --no-psnr" did not work.

[ChrisBensch]

I was wondering what was going on...

My bad. I just noticed I did not rename the variable holding the passed settings everywhere. Thus the 2nd pass options don't get used :/
Expect a bugfix release in a few days, there is a lot to fix :p

edit: this also explains the extreme blockyness some of you experienced.

[ChrisBensch]

When I run my 23.976fps file through VLC, the Stream and Media Info window shows that my Frame rate is 25.000000...is that a muxing issue?

[Revgen]

It might be.

See bond's explanation (post #57) for more info.

[ChrisBensch]

gotcha!

[blubberbirne]

-q " --ref 16 --mixed-refs --no-fast-pskip --bframes 3 --b-pyramid --b-rdo --weightb --subme 7 --analyse all --me esa --no-psnr --8x8dct "

this option does not work for me. If i removed the --no-psnr parameter, everythink works fine ?!?!

[708145]

-q " --ref 16 --mixed-refs --no-fast-pskip --bframes 3 --b-pyramid --b-rdo --weightb --subme 7 --analyse all --me esa --no-psnr --8x8dct "

this option does not work for me. If i removed the --no-psnr parameter, everythink works fine ?!?!

did you try a newer x264 build. the one shipped is svn400.

[Revgen]

-q " --ref 16 --mixed-refs --no-fast-pskip --bframes 3 --b-pyramid --b-rdo --weightb --subme 7 --analyse all --me esa --no-psnr --8x8dct "

this option does not work for me. If i removed the --no-psnr parameter, everythink works fine ?!?!

I would suggest removing --mixed-refs. It gave me problems as detailed in post #52. Also remove the --no-psnr parameter, just to see what happens. If that doesn't work than maybe a different x264 build will work.

[Revgen]

I've decided to do a test and comparison using the movie I described in post #85. I used the exact same settings for both.

Test Settings:

For ELDER: perl ELDER4X264_start.pl -a H:\BLAZING_SADDLES\VIDEO_TS\saddles.avs -m 2 -b 1500 -c 4 -o H:\saddle.mp4 -q "--keyint 240 --ref 5 --bframes 3 --b-pyramid --b-rdo --bime --weightb --subme 6 --analyse all --8x8dct --me umh --merange 20"

For X264.exe: x264.exe --bitrate 1500 --keyint 240 --ref 5 --bframes 3 --b-pyramid --b-rdo --bime --weightb --subme 6 --analyse all --8x8dct --me umh --merange 20 --threads 2 --progress --output "H:\BLAZING_SADDLES\VIDEO_TS\saddles.mp4" "H:\BLAZING_SADDLES\VIDEO_TS\saddles.avs"


Test Results:


12.11 FPS W/ ELDER -m1 -c4
22.36 FPS W/ ELDER -m2 -c4 - About an 85% improvement over -m1 -c4

4.80 FPS W/ x264.exe ABR 1 thread
8.37 FPS W/ x264.exe ABR 2 threads - 74% improvement over ABR 1 thread


Quick Summary:

1) ELDER doesn't use the options that were given in the test settings. See the .txt attachments for more details.
2) The ELDER videos play back at 25 FPS while the x264.exe encodes playback at 23.976. This is probably the MP4Box issue that bond talked about in post #57.
3) The ELDER videos are 11min 0 secs while the X264.exe videos are 11:28 secs. The ELDER videos are shorter than they should be. This may be because of the 25 FPS playback issue.
4)The quality of the x264.exe encodes are better the ELDER videos. Possibly having to do with summary item #1 above.
5)The ELDER encodes are 128mb while the x264.exe encodes are 121mb.
6) It seems that ELDER scales better than x264.exe. But we can't be sure since it isn't using all of the options.


Extended Summary:

ELDER is pretty fast but that's because it isn't using all of the options. For example if you look at the .txt attachments created by h264_parse you'll see that ELDER used "subme 5" instead of the "subme 6" that I orginally specified. There are other options that aren't used either. Consequently the videos doesn't look bad, but just not as good as the x264.exe encodes. I have no idea why the ELDER videos are larger than x264.exe's encodes. It could possibly be due to summary item #1 above.

Conclusion:

It seems that ELDER scales well (compared to -m 1), but still has bugs and issues to iron out. Once these issues are ironed out I think we'll still see 80% to 90% scaling improvements or possibily more.