x264 Multiple Instances Issue

[UltraTV]

I run multiple instances of single threaded x264 tasks through the command prompt. Frequently, x264 will complete a pass, print the [info] for the completed encode, then pause before returning to the command prompt until any other concurrently running instances complete their passes.

This becomes an issue when automating encodes via scripting. Let's say you're running 6 instances. One or two instances complete pass 1, automatically launching the second pass. The other four or five instances will complete, but before they return the process to the command prompt, they wait for the processes that are running second passes to complete. This means 4 encoding stations sit idle for 4 hours while two encoding stations complete their second pass.

When the instance freezes after pass 1, it can be killed as the stats file can still be read by pass two, however, if pass two freezes and is killed the resulting mp4 file is unusable.

Any word from the developers?

[nm]

Sounds strange. Do you use some encoding frontend, or how do you launch the x264 instances?

[UltraTV]

The issue occurs launching x264 instances from the command line, so no front end. I've toyed around with a script that auto launches new encoding tasks via the command prompt, and it suffers from the same issue.

This is using XP SP2, I haven't tried any other Windows systems.

[nm]

Do you specify a different stats file for each instance?

[UltraTV]

Yes, each task has a unique stats file

[UltraTV]

Just a quick update.

I also tried spawning instances from unique x264 binaries each in different directories, but still encounter the same issue.

I believe this occurs locally as well, but I am currently experiencing it encoding over samba.

[Nazgul]

This may be a stupid question, but I've got to ask. Why are you running multiple single-threaded instances of x264.exe instead of a single multi-threaded instance? Are these instances being distributed among multiple computers?

I ask mostly because I use x264 to encode batches of files as well, but I run the encodes in series rather than parallel, so to speak. All I use to automate it is a few handy windows batch files with some FOR loops in them, and it's been working great.

[UltraTV]

Not a stupid question at all, it's actually one I'm in the middle of asking myself :)

There are a few reasons. The first is that there is a quality hit, albeit negligeable and beyond my powers to see, when running multiple threads. Second, I'm running these tasks on 8 core machines and I found the speedup for running 8 threads is less significant than running 6 single threaded instances (the other two cores saved for avisynth decoding). I will go back and test running multiple threads (I'm currently toying with the idea of two instances with 4 threads each.) If for no better reason the problem is driving me a little nuts.

I'm still running tests to try and diagnose the behavior. The instances become attached to each other some how, so two can finish encoding but hold before exiting until one other instance finishes. It's always dependent on a single instance, never more than one.

I'm having some friends look at the source to see if there is code that is searching for some registry entry or something before exiting, but I'm not too hopeful. The only solution I can come up with for now is having each of the stations wait until the other stations are complete to launch the next pass. Hopefully this will keep all of the stations running in parallel rather than leapfrogging each other.

Thanks for your responses, hopefully we can get some attention from the devs before too too long :devil:

[Dark Shikari]

the speedup for running 8 threads is less significantTry running 12 or 16. There's a reason the default "threads=auto" is 1.5 * number of cores.

[squid_80]

What is your source material? An AVI file, avisynth script, piped data from another process ?

[UltraTV]

Dark Shikiri -
Try running 12 or 16. There's a reason the default "threads=auto" is 1.5 * number of cores.

Thanks for the input. I ran a test using threads with multiples of 1.5 and found that there's still a speed advantage using multiple instances over multiple threads.

6 threads - single instance - 5000 frames
avg. encode speed pass 1 = 64.58 fps
avg. encode speed pass 2 = 24.63 fps - cpu @ 97%

1 thread - 3 instances - 5000 frames
combined avg. encode speed pass 1 = 122.69 fps
combined avg. encode speed pass 2 = 21.12 fps - cpu @ 83%

1.5 threads - 3 instances - 5000 frames
combined avg. encode speed pass 2 = 26.53 fps - cpu @ 100%

After getting this data In I'll switch over to 1.5 threads for all of my encodes with multiple instances. Thanks for the heads up.

I think relying on multiple threads rather than multiple instances introduces too many bottlenecks, not the least of which being Avisynth's single threaded filters. Multithreading Avisynth scripts are still seriously unstable and cause x264 to crash regularly.

squid_80:

What is your source material? An AVI file, avisynth script, piped data from another process ?

Avisynth scripts.

[Dark Shikari]

After getting this data In I'll switch over to 1.5 threads for all of my encodes with multiple instances. Thanks for the heads up.Um. 1.5 threads?

Threads is an integer value :p

[squid_80]

squid_80:
Avisynth scripts.Soooo, what's in the scripts? Fft3dgpu by any chance?

[UltraTV]

Um. 1.5 threads?

Threads is an integer value

ok, i'm switching to 3 threads then.

Soooo, what's in the scripts? Fft3dgpu by any chance?

Nope, nothing as advanced as that. This behavior occurs regardless of the contents of the script, as best as I can tell. Some scripts use VMtoon and deen with mpeg2source, some scripts are using nothing except converttoyv12 with qtinput.

[talen9]

@UltraTV: I think you misinterpreted DS's previous sentence:

Try running 12 or 16. There's a reason the default "threads=auto" is 1.5 * number of cores.

This doesn't mean "use multiples of 1.5 as the number of threads" but "if I have 8 CPUs, the default is to use 1.5 * 8 = 12 threads", so 12 is the number you should go for ;)

[UltraTV]

This doesn't mean "use multiples of 1.5 as the number of threads" but "if I have 8 CPUs, the default is to use 1.5 * 8 = 12 threads", so 12 is the number you should go for

Thanks for the explanation. I'm not the quickest when it comes to these things :D

Anyway, just a heads up, I managed to get a build a debug version of x264 and noticed that it sticks on the "closing in file and out file"

http://bobamerica.tv/img/264debug.gif

[talen9]

I believe this occurs locally as well, but I am currently experiencing it encoding over samba.

Are you still running remotely or on network shared folders? In this case, since you're now saying that it stops (for what amount of time btw? seconds, minutes ... ?) when "Closing In/Outfiles", this actually *could be* the reason for the behaviour you're noticing, expecially if all your encodes are so short (very few frames -> the closing pause is very noticeable).

OTOH, this *still* could be some kind of thread lock ... but I'll leave further argumenting on this to more knowledged people :D

[UltraTV]

Are you still running remotely or on network shared folders? In this case, since you're now saying that it stops (for what amount of time btw? seconds, minutes ... ?) when "Closing In/Outfiles", this actually *could be* the reason for the behaviour you're noticing, expecially if all your encodes are so short (very few frames -> the closing pause is very noticeable).

OTOH, this *still* could be some kind of thread lock ... but I'll leave further argumenting on this to more knowledged people

After more testing, it occurs both locally and over the network, regardless of the number of frames in the video. We're still stumped over here, but my compatriot will dig deeper into the source code and see what he can find. A given instance hangs at the closing in and out phase until the instance it's attached to finishes, after which it is able to close the file and return to the command prompt. This can be anywhere from a few seconds to a few hours depending on the situation.

Of course, if anybody has any ideas to test I'm game.

[UltraTV]

We're running a massive encoding job and it seems like one of the new nightly builds has fixed the issue with x264 instances stalling. I'll keep my eye on this and try to confirm it with testing.

If one of the devs did fix it, thank you!

[UltraTV]

False hope. It's still an issue, although for some reason or another it seems to be less prevalent.