ffmpeg / libx264 networking

[nonymous]

Gents,

I think the current asnwer is "no" but I thought I'd ask anyway. Do any of you know if ffmpeg/libx264 support parallel processing over the network?

If not, is any development along those lines in the works?

If so, do they need help (I'd love to volunteer to help)

If not, would anyone care to join me in creating an IRC channel to discuss the feasability of it and approach on doing it?

Cheers!


BTW, I'm Linux based, don't care about windows and gui crap. Somebody else can do that. (Just because x264 is popular among the windows masses, let's not forget where it all started).

[RunningSkittle]

http://forum.doom9.org/showthread.php?t=117889

[nonymous]

Thanks, I actually had not seen that post. I actually want to do one on linux under either GNU licensing or similar with a team of 2 or 3 guys so we can actually put some design thoughts into it before we start coding. Maybe talk about the possibility of using xgrid or so...

[Dark Shikari]

I don't fully understand the point of networked encoding; realtime encoding can be done with x264 at any sane resolution with sufficient hardware, so the only reason to network would be to get a whole lot of encoding jobs done on a bunch of machines--in which case all you really have to do is have a simple job management system built out of a few scripts or something.

[nonymous]

I am not sure what you have in mind when you mention real-time encoding. How are you achieving real-time ?

I have been playing with HD re-encoding content (1920x1080), and granted my machine isn't the fastest (intel quad-core 2.5G, 4G RAM 800MHz, two different HD for source and destination so I don't waste time seeking), it takes me 22 hours to re-encode a 2 Hour feature - and that's only for 1 pass. (Yes, I'm using -threads and all my cores are used 99%).

I would imagine that maybe with different settings it would be possible to achieve faster speeds but I'm shooting for that balance highest-quality-possibe for target-size that gives me acceptable results.

So anyway, I know this sounds like duplicating an effort that has already been made but like I previously said, I want to start such a project and was wondering if anyone cared to join in the initial architecture. I'm sure that we could come-up with a load/node manager in the form of a library that could be integrated with x264 sources.

You are active in the development of x264 (correct me if I'm wrong). Wouldn't you possibly consider adding the services of such a library to x264 (assuming the changes to the core sources were minimal)?

[CruNcher]

I wished people would slowly realize how much Energy they are currently "wasting" on their encodes and how much they could save without loosing the crisp Look and Feel (putting every option on insane high is such a big energy "waste", especialy seeing Space Prices) :(

[Gabriel_Bouvigne]

I have been playing with HD re-encoding content (1920x1080), and granted my machine isn't the fastest (intel quad-core 2.5G, 4G RAM 800MHz, two different HD for source and destination so I don't waste time seeking), it takes me 22 hours to re-encode a 2 Hour feature - and that's only for 1 pass. (Yes, I'm using -threads and all my cores are used 99%).

side note: it's quite likely that you could achieve both higher speed and higher quality by using 2 passes at reasonable settings vs 1 pass at insane settings

[Dark Shikari]

I am not sure what you have in mind when you mention real-time encoding. How are you achieving real-time ?With fastest settings, 720p can be encoded at about 56fps on one core of a 3.2Ghz Core 2 system. With fast settings, 1080p/i can be done in realtime on a quad.
You are active in the development of x264 (correct me if I'm wrong). Wouldn't you possibly consider adding the services of such a library to x264 (assuming the changes to the core sources were minimal)?You shouldn't have to modify the core library at all--any such app would interface with x264, not change it, I think. But as I said I doubt its very useful--you might as well just do segmented encoding.

[nonymous]

Please pardon my ignorance but I am new to x264. When you say segmented, I assume that would consist in cutting the input in subsections for distributed processing. How would that subsection be defined? by the settings of keyint and min-keyint? or be the I frames on the input if the input wasn't raw?

I guess I should look at how the threading is done today and that would probably answer most of the questions I'm having now... But, the part that gets me confused is that the scheduler would then have to be aware about input video container format, just so it could parse and segment it.

What I was thinking is that libx264 already does that and it does it well. So why not let libx264 continue doing that but add a API layer between x264 and the host's native threads. So when x264 decided to spawn a new thread, it would call that library which in turn could either simply create a thread on the local host or decide to export that job to a remote host.

Pardon me if that is over simplified. I worked for years on a clustering system for N*vell but it had nothing to do with video so maybe I am over trivializing the part about the video which is new to me.

[audyovydeo]

Please pardon my ignorance but I am new to x264. When you say segmented, I assume that would consist in cutting the input in subsections for distributed processing. How would that subsection be defined? by the settings of keyint and min-keyint? or be the I frames on the input if the input wasn't raw?


Hello.
One way to do it would be (from x264 --longhelp) :


--seek <integer> First frame to encode
--frames <integer> Maximum number of frames to encode



I actually haven't yet tried this option out, because I've got my access revoked from the 2nd home computer.
The problem I anticipate is the race condition whereby N x264 instances access the source video file, whatever the network storage type.

cheers
audyovydeo

[akupenguin]

So when x264 decided to spawn a new thread, it would call that library which in turn could either simply create a thread on the local host or decide to export that job to a remote host.
x264's threads require shared memory. You don't get that over a network.

[Blue_MiSfit]

Have you looked into ELDER?

Last I checked it enabled clustering of one sort or another over a network...
~MiSfit

[Comatose]

The creator of http://helmer.sfe.se/ suggested this to me when I asked about video encoding on multiple machines:
If you want to do fast video encoding the opensource linux program
Cinelerra http://cinelerra.org works with distributed encoding, and
DqQueue.

I don't actually have the resources to try this, but I was curious :p
As I haven't tried it, I don't know if you can use x264.