A Parallel Processing DivX 5.0 Codec?

[DoC hEx]

I have three powerful computers, which I use to encode my DVD’s to DivX. Encoding a single DVD can take about 7-8 hours but if I where to split the load between 3 PC I could in theory get the encode time down to about 2-3 hours.

From my understanding of DivX processing (while limited) it would seem possible to break a DivX encoding session up into keyframe groups that could be sent to other computers for processing, because each keyframe is the start of a new section of video which doesn’t need information from the last frame. With a 100Mb network one PC would be fast enough to sever the rest the VOB’s needed in processing.

Are their any projects trying to do this? Because off the top of my head I don’t see why the encoding method used for SBC (DVD2AVI-Gknot-Nundub [Movie-Credits] process) couldn’t get broken down further so you would have, “MovieName-Frames-0-1500.d2v” with “MovieName-Frames-0-1500.stats” then “MovieName-Frames-1500-2890.d2v” with “MovieName-Frames-1500-2890.stats”, etc… till the end of the movie. These sections are then sent to a wait PC that would process them and send back a completed command and would look for the next section. Then at the end the server would append all the AVI sections together to form the complete movie.

Am I mad? Or would people find this useful?

DoC hEx

[TheWEF]

this isn't really useful unless you want to participate in an encoding race.
1 night, 1 computer: 1 movie.
1 night, 3 computers: 3 movies.
that's the solution, works really well...
;)
wef.

[MxxCon]

TheWEF, why don't you do exactly what you suggested?
encode 1/3 of all frames on each puter and then merge 3 avis into 1 complete

[ThePanda]

There were people working on something like that at projectmayo.com before, but I don't know what happened to it.

[ppera2]

Even with 100 MB LAN transferring large files is very slow.
Frameserving via LAN is out of question.

I think that only sensible idea is to split work by Vobs, as they are on DVD. It's easy to realize, just put DVD to each comp in order etc.

But if work in 2 pass, it will degrade quality. Other side: it requires lot of handwork.

Not much useful. Better sell 3 machine and buy one very fast.

I can finish encoding of average movie in 2 pass in 4-6 hours. Only 1350 MHz Athlon.

And your bill for electricity will be much lower :) .

[cordraconis]

A lot of hand(y)work indeed....

I think it would be interesting to do only the first pass on 3 PC, and then do the 2nd pass on one machine, with all the VOB's.

... but "Glue-ing" the 1st-pass LOG file's together can be a bit of a hassle. (Anyone wants to try that?) :D

About the previous post regarding the OpenDivX Cluster project at Projectmayo.com; It is no longer in the list of projects.

I subscribed to the project, and since then I have sent a few suggestions to Martin (Think that was his name), the "supervisor" of the Cluster project.
But I haven't got any replies anymore since then.

I think it has just bleeded to death.

Maybe within a few years, if I have time, I will learn to program C++ or something like that, and then I'll give it a shot with my own ideas.

:(

[ThePanda]

Originally posted by cordraconis

Maybe within a few years, if I have time, I will learn to program C++ or something like that, and then I'll give it a shot with my own ideas.
:(

Maybe within a few years movies will be encoding at DVD quality in 10 seconds at half the filesize, so you won't care about it anymore :D

[Doom9]

what all of you with dual cpus or networks full of pcs doing nothing should keep in mind is that 99% of the people encoding do not have these kind of facilities. Personally.. I couldn't care less if a soft was smp or not.. until every cpu consuming soft is smp capable I will never go for a dual cpu solution. And... those who took courses in operating systems or paralell programming know how damned hard it is to program for more than 1 CPU.. distributed programming is even much much harder.

just a few things to keep in mind...

[fu2k]

I guess it's just a matter of taste. I'm planning to _never_ go back to a single processor machine.

[Ozymandis]

I'm with you fu2k. I've had a lot of single processor systems and a lot of dual processor rigs (dual celery, dual p3s, now using two dual athlon MP setups (well, one dual MP one dual XP, same thing though).

Doom: I've taken Operating systems :) Fun class huh? :/ My OS prof agreed for the most part, but I did get into an argument with one of my comp. engineering professors over this actually...lol. While it's certainly true that without multithreading in an application, that application alone won't benefit from the additional processor, the OS can distribute applications across multiple CPUs. That by itself equates to nearly a doubling of speed for most heavy users who are running 10 things at once. What a lot of people who haven't used smp setups very much don't take into account is how smooth having 2 cpus makes the computer. Normally single CPU systems freeze frequently, if very briefly. With an smp system, this almost never happens. Everything runs incredibly fluidly, since one CPU can be locked with an aberrant process while the other one is chugging on its merry way, insuring that the interface is still responsive to the user.

Of course, as far as divx encoding goes, using the standard avisynth/virtual dub setup, I normally get approximately 80% CPU utilization. Subtracting 2% for smp overhead, which is probably a bit pessimistic since there's no threading being done which might lower the algorithmic efficiency of the codec, that still means that you're gaining 28% additional speed for divx encoding (by relieving the burden of mpeg2 decoding,mp3 encoding,etc), with the added benefit of being able to burn CDs, browse the web, play back files, play games, etc, all without decreasing the encoding speed much. All that, and the added cost to the system is only maybe another $250 or so, another 100 for the more expensive motherboard and 150 for another processor.

I don't ever intend to go with a single CPU computer again myself :)
Every time I've tried in the past I've hated the experience, even when going to a faster CPU than my two previous combined.

BESIDES, having those two huge fans at the top of my motherboard keep my corner of the room nice and cool :D

As far as distributing encoding across multiple machines...geez, it's mind-boggling how complicated the math behind even somewhat linear mpeg-4 encoding is. I'd hate to think about trying to coordinate all of that over a network of arbitrary size..geesh :P I doubt that'll ever happen, personally. Nobody would want to write it. If more speed is needed, it'd be -needed- by companies, and they could go with a threaded codec someday with an 8-way xeon system, or similar setup.

[cordraconis]

(Hmmm... seems very difficult today (and yesterday !) to post replies ...) :rolleyes:

O.K. let's try to *not* make a fool of myself, but here it goes; (shiver ...) :)

yes, it could be possible to do the encoding process on a lot of machine's together, and indeed, I guess you would have to be half-crazy to try to synchronize and control all that PC's *at once*, but maybe it could be interesting to let each PC do it's (part of the) job on it's own, and *periodicaly* synchronize everything. (with the right time interval, it could be reasonable, I guess ...).

I'll try to explain my line of thinking.
I heard somewhere, (could be in the OpenDivX Cluster mailing-list), that somebody tried to distribute the CD-ripping proces of Audiograbber.
Something like; Rip one WAV-file, and send it to a network-PC, let that PC convert it to MP3, and in the mean while, rip the next track of the CD and send it to another PC. Afterwards gather all MP3 files.
The program was some kind of "Platform" on each PC, that controls the "Audiograbber".

Sorry if I am wrong about this program, but I did not find/keep the post in which it was mentioned.

[tonygib]

I to have to agree with fu2k and Ozymandis on this one. Having also recently got a dual AMD system I hate going back to a single cpu at work. Everything Ozymandis said about the increase speed in encoding and the smoothness of system operation is spot on. Some software will only make partial use of the second CPU, while TMPGENC for example will use both to 100%. So like what has been said before, just sell two comps and get a super dual and then use a KVM switch between the two or encode 3 movies over night, one on each comp.

[philippas]

Hi, I think someone suggested that you can lock a process to one cpu, of the dual system, and use the other one for other processes. How can you do that ?
I thought that the load is always half in the one cpu and half to the other.

[tonygib]

philippas,

Yes you can control which/how many CPU's work on a given process, in the Task Manager on Win 2k or NT right-click the task and select Affinity, this will give you a box with upto 32 CPU's listed and a check box next to each one that you have in your system.

[philippas]

Cool Thanks!!!
I'm using windows XP pro,i guess it's exactly the same way.
I'll assemble my old dual mainboard soon and i'll try it under XP.

[funkton]

I was using flask .5xx and .6 beta, but found only with certain settings would it use 100% processor utilization. 65%-80% was the average.

I then tried out Vidomi. 100% processor utilization no matter what I was compressing to.

VOB to DivX 720x480 all quality features at "max" was letting me compress a 2:15 movie in 2hours using 2pass encoding.

VOB to SVCD 480x480 all quality features at "max" was letting me compress a 2:15 movie in 1hour. Around 55-60fps.

I love my dual system, and it wasn't too expensive.

Motherboard - $205 (Tyan Tiger 2460)
2x 1600+ MP - $168 each (or get the 1500+ XPs for $108 each)
2x 256MB Registered ECC (crucial) $50 each.

I won't ever go back to a single processor system.

[philippas]

Flask sucks big time. Although i thing everyone tried it at some point.
As for vidomi i don't know, i haven't tried it because of the thing with Avery Lee. Maybe i'll try it when i'll assemble my dual system. What's the speed comparing with DVD2AVI and avisynth+virtualdub for divx4 ?
Thanks for info.

[funkton]

Philippas,

I'll give it a try tonight, and get you the numbers tomorrow.

I'll do a comparison between Vidomi for DivX4 and DVD2AVI w/ avisynth+virtualdub for divx4.

Could some of the other people here please tell us what combo of software is your favorite for DivX4 & why?

Thanx,
Funkton

[philippas]

I would say that the fastest is DvD2AVI(on its own). Nick has written a software that enable u to do 2-pass divx4 encoding automatically. Then DvD2AVI->avisynth->virtualdub is second and close if not faster is xmpeg(which i don't recommend it, but i haven't tested the new version yet).
Thanks funkton, Waiting for the numbers tomorrow.

[dragonlz]

Every time I've tried in the past I've hated the experience, even when going to a faster CPU than my two previous combined.

Can you explain why? It seems to me at the same speed, one cpu is always better than 2 cpu's


And going back to why spliting the work onto multiple machines is not a smart idea. If you do a two pass encoding, bitrate is distributed for the movie AS A WHOLE. If you have a movie where all the action scenes are in the second part. When process the first half on one machine and second half on another, you'll end up having a nice looking first half and a crappy looking second half. totally defeats the purpose of doing a first pass.

[ppera2]

Originally posted by philippas
I would say that the fastest is DvD2AVI(on its own). Nick has written a software that enable u to do 2-pass divx4 encoding automatically. Then DvD2AVI->avisynth->virtualdub is second and close if not faster is xmpeg(which i don't recommend it, but i haven't tested the new version yet).
Thanks funkton, Waiting for the numbers tomorrow.

Yes, I agree that DVD2AVI is fastest. Of course you can't solve some DVD's with it.
But I have some problem with Nick's small programm: how to set different bitrate for first and second pass ? First pass should have much higher bitrate for good quality.

[philippas]

There's not need to set a different bitrate for the second pass.
If you rip the DvD's properly DvD2AVI can handle any dvd.

[dragonlz]

But I have some problem with Nick's small programm: how to set different bitrate for first and second pass ? First pass should have much higher bitrate for good quality.
I think DivX codec handles that internally. Just as how you'd encode in Divx4 in VDub, the user sets the same bitrate and the codec determines everything else.