How to handle queueing of video encoding?

[johnvegan]

I am working on developing a video streaming site where users can upload videos to the site (multiple videos at once using the uploadify jquery plugin).

Now, I am faced with the question of encoding the videos to FLV for streaming them online.

When should the video encoding process take place ? Should it take place immediately after uploads have finished (i.e redirect the user to upload success page, and then start encoding in the background using exec command for ffmpeg ?) However, using this approach, how do i determine if the encoding has finished successfully ? What if users upload a corrupt video and ffmpeg fails to encode it ? How do i handle this in PHP ?

How do i queue encoding of videos since multiple users can upload videos at the same ? Does FFMpeg has its own encoding queue ?

I also read about gearman and message queueing options such as redis and AMQP in another related SO thread. Are these one of the potential solutions ?

I would really appreciate if someone could give answers to my questions.

[LoRd_MuldeR]

I would assume that you would have a web-server, which handles the upload, and a number of worker nodes that do the encoding jobs.

So this is closely related to the classic producer-consumer problem, I think:
http://en.wikipedia.org/wiki/Producer-consumer_problem

Actually you have that twice: The web-server produces uploaded files that the worker nodes consume for encoding (as soon as there is a free node). And the worker nodes produce re-encoded files that the web-server (maybe the same one that handles the uploads, maybe a different one) consumes for storage and/or streaming.

For synchronization between different processes and machines, you will probably need something like MPI:
http://en.wikipedia.org/wiki/Message_Passing_Interface

[Ghitulescu]

I would write an applet to encode the file on user's computer, not only it will free up valuable resources on server, but it will (probably) lower the needed bandwidth for uploading.