Any hardware to aid x264 encoding?

[mmace]

Hi guys, I'm going to be building a new PC soon, I will be doing a lot of x264 encoding as well as editing AVCHD from a camera.
Apart from an i7 CPU, is there any other hardware that will aid x264 encoding that doesn't cost more than a car?

Any help/advise would be appreciated

[Dark Shikari]

Another i7 CPU?





;)

[mmace]

you got any links to dual i7 motherboards?
could be interested in that!

[Chengbin]

you got any links to dual i7 motherboards?
could be interested in that!

No such thing unforunately.

[Audionut]

Updated XEONS will be here soon.

[photoguy123]

Another i7 CPU?
;)

Actually I have a big question mark as to how much this will help.

1 Core i7 cpu = 8 logical threads
2 Core i7 cpus = 16 logical threads
2 Core i7 XeonMP cpus = 32 logical threads

So what's the typical performance scaling factor of x264 with this much parallelism?

[Dark Shikari]

Actually I have a big question mark as to how much this will help.

1 Core i7 cpu = 8 logical threads
2 Core i7 cpus = 16 logical threads
2 Core i7 XeonMP cpus = 32 logical threads

So what's the typical performance scaling factor of x264 with this much parallelism?Even with threaded slicetype (coming soon), x264 will probably cap out before 32 logical threads in most cases. So just run two encodes at once ;)

[Blue_MiSfit]

Exactly!

That's what I do with 8-core servers that can't saturate all 8 cores with just one encode (due to other restrictions, not x264).

~MiSfit

[photoguy123]

Exactly!
...8-core servers can't saturate all 8 cores with just one encode (due to other restrictions, not x264).
~MiSfit

What other restrictions?

Core i7 has tons of memory bandwidth, and I use a RAM drive to eliminate disk IO as a bottleneck.

So I'm guessing it might be a limitation of algorithmic scalability rather than a limitation of hardware architecture.

Every problem can only be made parallel to a certain degree. A famous saying says some things like ray tracing happen to be "embarassingly parallel", other things not so much.

[asarian]

Even with threaded slicetype (coming soon), x264 will probably cap out before 32 logical threads in most cases. So just run two encodes at once ;)

I'm glad the subject came up. Been planning to buy an i7, too; also for x264 processing, mostly. And I can't find where I once read it, but I seem to recall a wiki or some such, which said the quality of the x264 rendering diminishes some with each extra thread.
Is that (still) true? Also, I read x264 and hyperthreading isn't an ideal solution (though the latter came from a somewhat questionable source).

If you could shed some light on the matter, I'd appreciate it.

[Sharktooth]

the quality drop is negligible (read: so low there will be no difference).

[Blue_MiSfit]

All that really matters is that the i7 is hugely faster for x264 encoding.

The other bottlenecks I was talking about are mostly decoder related (I think, but wasn't ever really able to confirm). I deal with 50-80mbps MPEG-2 sources, and 80-220mbps MJPG / ProRes sources, so it's kind of a different ballgame from encoding BluRay discs like most folks do.

~MiSfit

[SixKiller]

Maybe a CUDA - enabled build of x264 would help :) Along with 4 GTX 285 stuffed in or even 4 GTX 295. *Just dreamin'*

[photoguy123]

Maybe a CUDA - enabled build of x264 would help :) Along with 4 GTX 285 stuffed in or even 4 GTX 295. *Just dreamin'*

A company has already sunk 7 figures into making a CUDA based h.264 encoder and you can buy it right now.

So far it has produced unimpressive perf improvements, unimpressive output quality and flexibility, and unimpressive stability.

The jury is still out on how well this nut can be cracked.

[lexor]

Alternatively, use a CUDA decoder to offload decoding from CPU (as well as deinterlacing) and use GPU based filters (well, one filter that I know of, fft3dGPU) for denoising.

[Sagekilla]

I don't think chaining a CUDA decoder with FFT3DGPU would be a good idea. You'd be doing -a lot- of CPU <-> GPU communication. Think about it: The chain would look like CPU -> GPU -> CPU -> GPU -> CPU -> x264.

Bad idea.

[laserfan]

Hi guys, I'm going to be building a new PC soon...any other hardware that will aid x264 encoding that doesn't cost more than a car? Any help/advise would be appreciated'
By all means consider an Nvidia videocard with VP2 support, so you can take advantage of neuron2's tools for processing your videos.

[lexor]

I don't think chaining a CUDA decoder with FFT3DGPU would be a good idea. You'd be doing -a lot- of CPU <-> GPU communication. Think about it: The chain would look like CPU -> GPU -> CPU -> GPU -> CPU -> x264.

Bad idea.

Only if the volume is low. Someone with neuron2's tool should test it for us to really know. I don't believe it will be slower than CPU denoise with Fizzik's original 3dfft filter.

[Sagekilla]

Volume is low? Do you mean the encoding fps? If anything, it would be very little problem at low fps because if x264 is only crunching @ 4 fps with 100% CPU, then the bottleneck isn't CPU <-> GPU communication. It would be more problematic at higher speeds.

[lexor]

Volume is low? Do you mean the encoding fps? If anything, it would be very little problem at low fps because if x264 is only crunching @ 4 fps with 100% CPU, then the bottleneck isn't CPU <-> GPU communication. It would be more problematic at higher speeds.

By volume I mean the number of frames being pushed to the GPU. If the filter can saturate GPU at all times, overall result should be better than CPU+GPU mix for decode/denoise. The cost of communication is only expensive, if we communicate in little chunks. The more frames/data being pushed, the less important communication cost becomes.

[Sagekilla]

Except, how are you going to make sure it's going to process huge batches of frames? I don't know the avisynth internals, but I believe it only works as frames are requested, which would make your idea difficult to work.

Ideally, it would be great if you could make the VP2 decoder pass off frames directly to the FF3DGPU code running on the GPU, then have it sent back to the CPU (CPU -> GPU [VP2 -> Shaders] -> CPU). Methinks this would require some tight integration though ;)

[asarian]

Even with threaded slicetype (coming soon), x264 will probably cap out before 32 logical threads in most cases. So just run two encodes at once ;)
I've actually seen screengrabs posted of x264 not fully loading all 8 cores. Hyperthreading, I think, may have something to do with that, too. Mencoder_MT does not perform particularly well with hyperthreading, either.

Doing two x264 jobs sounds cute, but I rather just want 1 job done real fast. :) Hope the future will bring ways to achieve that.

[asarian]

the quality drop is negligible (read: so low there will be no difference).
I've always assumed it was. :) And with only 4-6 threads I'm sure it is. I wonder how it scales, though. When we're talking about 32 logical threads, wouldn't there come a point where it actually becomes an issue?

[lexor]

Except, how are you going to make sure it's going to process huge batches of frames? I don't know the avisynth internals, but I believe it only works as frames are requested, which would make your idea difficult to work.

Ideally, it would be great if you could make the VP2 decoder pass off frames directly to the FF3DGPU code running on the GPU, then have it sent back to the CPU (CPU -> GPU [VP2 -> Shaders] -> CPU). Methinks this would require some tight integration though ;)

I don't see how that's different from normal use of fft3dgpu, why would it slow down now?

[Sagekilla]

You're missing my point. He was talking about using GPU decoding and GPU filtering at the same time. GPU for only filtering is no problem cause you only have to send a frame to the GPU once, then get it back. Add in two filters and it's going back and forth a lot more.

[708145]

I've actually seen screengrabs posted of x264 not fully loading all 8 cores. Hyperthreading, I think, may have something to do with that, too. Mencoder_MT does not perform particularly well with hyperthreading, either.

Doing two x264 jobs sounds cute, but I rather just want 1 job done real fast. :) Hope the future will bring ways to achieve that.

Why wait for the future?
Have a look at x264farm and set up several "clients" on the same machine.

bis besser,
T0B1A5

[_DW_]

When we're talking about 32 logical threads, wouldn't there come a point where it actually becomes an issue?

This something a friend I where wondering about the other night. With 8 core cpu's coming out and the possibility of having up to 8 of those on a system. At what point does the threading issue become a problem?

I'm assuming it won't because of limitations in software and hardware.

[Chengbin]

We really need some new magic in CPUs. We can't add cores forever. CPUs are still very slow.

My encode of Kung Fu Panda from Bluray to 4GB 1080p is taking retardedly long. It is taking 22 hours for the turbo first pass on a 2.4GHz quad core machine. I can't imagine the time needed for the second pass, where the real settings come in (16 b, ref frames, subme 9, me tesa and merange 32, all partitions).

It will probably take 5 days based on experience (first pass usually 3.2x faster than second pass).

I can't imagine if I use RDRC. We really need some 100GHz single core CPUs.

[Sagekilla]

You'd stand to gain -a lot- more if you used some high quality preprocessing. In fact, for a fraction of the time you can get way more compression.

[Chengbin]

You'd stand to gain -a lot- more if you used some high quality preprocessing. In fact, for a fraction of the time you can get way more compression.

Please explain to a noob.

[Sagekilla]

Simple. Denoise the video a little bit, or try stabilizing the grain slightly and the bitrate required will drop significantly. MDegrain1 can do light denoising + stabilize it nicely, if you're not trying to go too over the top.

[Chengbin]

Simple. Denoise the video a little bit, or try stabilizing the grain slightly and the bitrate required will drop significantly. MDegrain1 can do light denoising + stabilize it nicely, if you're not trying to go too over the top.

I'm encoding Kung Fu Panda, and the source is Blu-Ray. No noise. But thanks, I'll do that for my future encodes, which is a lot. But I've encoded a lot too. Is the increased quality significant in a 700-800Kbps encode (movies)?

[Sagekilla]

That may be the case, but there could be some temporal fluctuations because it was lossily encoded. Since no commercial movie uses lossless compression, every movie out there (animated or otherwise) will contain some "noise." It might not be visible, but it's there. Try doing a small encode of a section with and without filtering. If it improves compression, use it. Otherwise, don't bother. Most movies, even animated ones, you can get some compression benefit from denoising though.

For 800 kbps encodes, yes, you can gain quality.

[Dark Shikari]

We really need some new magic in CPUs. We can't add cores forever. CPUs are still very slow.

My encode of Kung Fu Panda from Bluray to 4GB 1080p is taking retardedly long. It is taking 22 hours for the turbo first pass on a 2.4GHz quad core machine. I can't imagine the time needed for the second pass, where the real settings come in (16 b, ref frames, subme 9, me tesa and merange 32, all partitions).Wait--so if you use incredibly slow settings--the encode will go incredibly slow?! :rolleyes:

x264 can do 1080p in realtime with reasonable settings on a quad-core Core 2.

[Chengbin]

Any other tips in preprocessing to improves quality when encoding a video? I seriously don't want to reencode everything again because I missed something. me=tesa, merange=32, and subme 9 takes forever, making it take 4-5 hours to encode a movie.

Can MVDenoise 1 hurt compression in some cases?

To the OP, I am really sorry for hijacking this thread.

[Sagekilla]

Denoising will almost never harm compression. Except in some absurd case where you screwed up the motion of the video and made it more difficult to encode, you should always get an improvement.

Advice: Use --me umh --merange 16 (default) --b-frames 4 --b-adapt 2 --ref 4

Any more than that is just stupid. Your efforts are better spent in processing your source beforehand.

[Chengbin]

Denoising will almost never harm compression. Except in some absurd case where you screwed up the motion of the video and made it more difficult to encode, you should always get an improvement.

Advice: Use --me umh --merange 16 (default) --b-frames 4 --b-adapt 2 --ref 4

Any more than that is just stupid. Your efforts are better spent in processing your source beforehand.

Thanks, I'll make MVDenoise 1 defaulted.

I really don't care about encoding time, I care about quality, so I'm using me tesa,merange 32, subme 9, b adapt 2 with 4 b frames and 6 ref frames. I didn't waste $2000 on a quad core machine last year for nothing, this is the time to use it. I have a lot of time on my hand, and I have nothing better to do with my CPU anyway.

[LoRd_MuldeR]

I really don't care about encoding time, I care about quality, so I'm using me tesa,merange 32, subme 9, b adapt 2 with 4 b frames and 6 ref frames. I didn't waste $2000 on a quad core machine last year for nothing, this is the time to use it. I have a lot of time on my hand, and I have nothing better to do with my CPU anyway.

Then you must not complain about encoding time :rolleyes:

Anyway, some of the settings you are using are just insane. They take MUCH longer to encode and probably give no visible benefit over the settings suggested by Sagekilla.

Slower encoding settings don't necessarily mean "better" quality ;)

[Sagekilla]

subme 9, b-adapt 2, 4 b-frames and 6 refs is fine if you want quality. Those actually have a noticeable effect. --me tesa and --merange 32 is pointless. The quality gain for those two settings is so minimal you're wasting your time if you care about quality.

As I said, drop down to --merange 16 and --me umh (and lose < 0.1% compression) and do post processing. Your encodes will be much faster and much higher quality.

[Chengbin]

How come MVDenoise 1 slows down encoding like crazy? It is about 5 times slower than when it is turned off.

[J_Darnley]

Because MVDenoise is slow. Use the MVtools2 branch if you want to speed it up and/or safely multi-thread it.