CAFxX
19th December 2009, 07:47
Suppose I need to execute the same process a number of times on different inputs. Each run of the process is independent of the other runs. Now suppose that, for some reason, if I naively run those processes sequentially, some of the resources of my machine wouldn't be fully used (e.g. CPU-bound single-threaded process on a multiple-core machine).
To fix that I normally break up the list of processes to be executed in batches of M (M being the number of cores) and then I run them in parallel using & and wait (I'm talking about thousands of processes, therefore launchin all of them in parallel wouldn't be feasible)
This approach has a few problems, though. If, for some reason, one of the processes takes longer than the other, we'll have to wait for it. Also, this approach doesn't take into account automatically the other resources (e.g. RAM, I/O, etc.).
I guess it has to exist some infrastructure to batch a bunch of independent processes to be automatically executed as soon as there are some free resources... I bet it's absolutely trivial, but I really can't find it... any idea?
To fix that I normally break up the list of processes to be executed in batches of M (M being the number of cores) and then I run them in parallel using & and wait (I'm talking about thousands of processes, therefore launchin all of them in parallel wouldn't be feasible)
This approach has a few problems, though. If, for some reason, one of the processes takes longer than the other, we'll have to wait for it. Also, this approach doesn't take into account automatically the other resources (e.g. RAM, I/O, etc.).
I guess it has to exist some infrastructure to batch a bunch of independent processes to be automatically executed as soon as there are some free resources... I bet it's absolutely trivial, but I really can't find it... any idea?