x265 HEVC Encoder

[jlpsvk]

this is x264:
http://s12.postimg.org/4r368zrl9/SW_x264.png

and this x265:
http://s8.postimg.org/sldo3qj51/SW_x265.png

No matter what I do, the transition on the banner (from red to black - highlighted) is smoother on x264. On x265 you the the squares after the close-up. What do you advice?

My settings:

--crf 22 --preset slow --output-depth 10 --tu-intra-depth 2 --rdoq-level 1 --qg-size 16 --ipratio 1.3 --pbratio 1.2 --max-merge 2 --rc-lookahead 60 --ref 3 --min-keyint 23 --keyint 240 --colorprim bt709 --colormatrix bt709 --transfer bt709 --deblock -3:-3 --psy-rdoq 5

[MeteorRain]

Originally Posted by pingfr

Keeping in mind, that this is taxpayer money we're talking about here, as such: every single dollar/euro spent has to be properly justified and allocated.

So you have 2 options for the maximum performance per money,
1) i7-4790 + B85 mobo + 8GB RAM + PSU
2) 2xE5-2670 (Used) + C602 mobo + Some ECC RAM (Used) + Good PSU

First choice is about $450 without HDD, monitor and computer case.
Second is about $600 without these craps.
So if you have, let's say, 5 bluray to encode, distribute them to 5 different servers is going to maximize the speed. Given $3000, running 5 i7s will be faster than a single E7, let alone E7 being much more expensive.

The whole point is that, you'll be paying a lot more for the high density (the amount of space on the rack).

[pingfr]

Originally Posted by MeteorRain

So you have 2 options for the maximum performance per money,
1) i7-4790 + B85 mobo + 8GB RAM + PSU
2) 2xE5-2670 (Used) + C602 mobo + Some ECC RAM (Used) + Good PSU

First choice is about $450 without HDD, monitor and computer case.
Second is about $600 without these craps.
So if you have, let's say, 5 bluray to encode, distribute them to 5 different servers is going to maximize the speed. Given $3000, running 5 i7s will be faster than a single E7, let alone E7 being much more expensive.

The whole point is that, you'll be paying a lot more for the high density (the amount of space on the rack).

I think you misread it real fast.

Regarding my interest in regards to x265 in general, there are two things;

Professionally, like, at my work, as my job, it's impossible for me to bring-in "fleaky hardware" from dubitous sources without proper invoices, warranties or without a corporate contract directly from HP, Dell or any other major retailers because it's actually a budget that has to be allocated and voted in city hall's court. Which implies, what hardware I will eventually get my hands on depends on things that are out of my control.

Now what I have complete control over is what I do with my personnal funds, at home, as a leisure/hobby and what I have at the moment is a 3k budget to use as I see it at my own will and I have all reasons to believe, my best option at home is a Quad Xeon E7 v2 since I'm acquiring parts for reaaaaaaallly cheap compared to the official fares.

But then again if someone has an idea of "what performs better than what for cheaper than whatever else grabbed from over here rather than over there", please let me know. I'm all ears.

[kypec]

Originally Posted by pingfr

Professionally, like, at my work, as my job, it's impossible for me to bring-in "fleaky hardware" from dubitous sources without proper invoices, warranties or without a corporate contract directly from HP, Dell or any other major retailers because it's actually a budget that has to be allocated and voted in city hall's court. Which implies, what hardware I will eventually get my hands on depends on things that are out of my control.

Maybe I'll go a bit off topic here but did you also consider HEVC licensing fees apart from technological (better compression ratio) aspects of choosing HEVC over AVC? I don't want to spoil your fun and really hope that your project of digital movie library ends up successfully but I think you should engage lawyers as well into the process before spending too much time & electricity on encoding the content...

[MeteorRain]

Originally Posted by pingfr

I think you misread it real fast.

Regarding my interest in regards to x265 in general, there are two things;

Professionally, like, at my work, as my job, it's impossible for me to bring-in "fleaky hardware" from dubitous sources without proper invoices, warranties or without a corporate contract directly from HP, Dell or any other major retailers because it's actually a budget that has to be allocated and voted in city hall's court. Which implies, what hardware I will eventually get my hands on depends on things that are out of my control.

Now what I have complete control over is what I do with my personnal funds, at home, as a leisure/hobby and what I have at the moment is a 3k budget to use as I see it at my own will and I have all reasons to believe, my best option at home is a Quad Xeon E7 v2 since I'm acquiring parts for reaaaaaaallly cheap compared to the official fares.

But then again if someone has an idea of "what performs better than what for cheaper than whatever else grabbed from over here rather than over there", please let me know. I'm all ears.

Sorry for misreading your post.
For the professional part, unless you are running out of physical space, I'd not recommend using rack servers as they tend to be more expensive and less cost-effective. Buying multiple dell or hp i7s business level computer could be a good use of money, compared to those E5s (2x+ expensive with almost same performance).

For the hobby part, if you can get dirty cheap E7 then go ahead. The E5 that I mentioned was also a quite cheap option for you. Just take care of the noise control and power consumption.
For performance wise, google search "passmark <whatever cpu here>" and you can compare the score easily.

[pingfr]

Originally Posted by MeteorRain

Sorry for misreading your post.

No problems, no harm, I know you were trying to be helpful.

Originally Posted by MeteorRain

For the hobby part, if you can get dirty cheap E7 then go ahead.

400$ per CPU, not sure if that's "dirt cheap" but we're talking about E7 Xeons here at the fastest/highest frequency ones from that generation.

Originally Posted by MeteorRain

For performance wise, google search "passmark <whatever cpu here>" and you can compare the score easily.

Passmark tends to be "unreliable" specially if you're looking at multi sockets benchs, on the contrary I believe SPEC benchs are more reliable.

Regardless, thanks a lot for your input.

[MeteorRain]

Originally Posted by pingfr

400$ per CPU, not sure if that's "dirt cheap" but we're talking about E7 Xeons here at the fastest/highest frequency ones from that generation.

That's cheap enough. Usually you can only get an i7 with that money.

Originally Posted by pingfr

Regardless, thanks a lot for your input.

You're welcome. Hope it helps

[pingfr]

Originally Posted by MeteorRain

That's cheap enough. Usually you can only get an i7 with that money.

It's really a bargain, otherwise I wouldn't even bother to begin with.

It seems the best CPU power route from the worst to the best is:

E5-2699 v3 x2 -> E7-4890 v2 x4 = E7-4850 v3 x4 -> E5-4669 v3 x4 -> E7-8895 v3 x8 -> Nothing yet? maybe some high-end E5 or E7 v4 around Q3-Q4 2016 -> E5 or E7 v5 with AVX512 H2-2017.

Originally Posted by MeteorRain

You're welcome. Hope it helps

Anything helps.

[MeteorRain]

Originally Posted by pingfr

It seems the best CPU power route from the worst to the best is:

I actually don't have much idea on how much performance can you get with one of these. I'd personally go with cluster solution where it fits me better.

[foxyshadis]

Another datapoint you might be interested in: VMWare, both ESXi and Workstation, gives you native encoding speed. Benchmarks of KVM and Xen show the same. Virtualbox you take a hit, unfortunately, but other virtual solutions allow 99%+ native speed in x264 and x265. (One of the benefits of heavy number crunching is that you don't need to call into the kernel or do I/O much.) You could use that to set up a test cluster for working out how a real cluster would perform at work, and get experience in setting it up.

I know Multicoreware has demonstrated a cluster, but I don't know how polished vs spit and duct tape it is, since they don't seem to offer it for sale. Aside from that, the only one I know of is lancoder, though there's x264farm & ELDER that might be modified to call x265 instead of x264, and Media Encoding Cluster that uses ffmpeg. None of those have been touched since 2009-2010 though, still a bit of a gap for someone who wants to come in and fill it.

Maybe you can ask Netflix if they're willing to share part of their solution? You never know.

[2themax]

Originally Posted by LigH

Experimental UHD Bluray support (--uhd-bd) in x265 1.9+106-c8ec86965e54; is there anyone who can test and confirm?

This is very close to working. The only error I am getting is that GOPs are longer than they are specified to be.

[sneaker_ger]

x265 command-line? x265 log? Exact error message of your authoring software? (What software?)

[Vesdaris]

Originally Posted by x265_Project

Forget about turbo clock speeds... if you're running x265 you will hit the thermal limits of your chips, and there will be no turbo boost.

Why would he hit thermal limits having a proper cooling?

I can run my Ivy bridge 3770k @4.5 on air using a pretty high for these CPUs voltage (1.33V under 100% load -> unfortunately I haven't won the silicon lottery) and I still don't hit thermal limits. Ever.

[foxyshadis]

Originally Posted by Vesdaris

Why would he hit thermal limits having a proper cooling?

I can run my Ivy bridge 3770k @4.5 on air using a pretty high for these CPUs voltage (1.33V under 100% load -> unfortunately I haven't won the silicon lottery) and I still don't hit thermal limits. Ever.

I think the initial assumption was that it would be a pizza box (rack mount), which are much, much harder to cool that a standalone server, because there just isn't enough room. You don't see too many quad sockets that aren't racked these days. Only a heavy-duty watercooling can keep those running at full-tilt, and even that's not easy in the confines of the rack. Since it'll be standalone, though, it'll be loud but easily capable of being cooled.

Originally Posted by Vesdaris

Why would he hit thermal limits having a proper cooling?

I can run my Ivy bridge 3770k @4.5 on air using a pretty high for these CPUs voltage (1.33V under 100% load -> unfortunately I haven't won the silicon lottery) and I still don't hit thermal limits. Ever.

I'm guessing that you hit thermal throttling all the time, but you aren't aware of it. Thermal management is all done automatically. You would have to use an advanced performance profiling tool like Intel's Performance Counter Monitor to log the clock speed in small increments.

Your Core i7-3770K has a TDP of 77 watts, or roughly 19 watts per core. Many-core Xeons have higher TDPs, but much lower TDP per core (hence, the slower clock speeds). Fully saturating them with compute-intensive applications will heat them up good, and you are very unlikely to have enough thermal headroom to allow turbo mode to be engaged.

[Vesdaris]

Originally Posted by x265_Project

I'm guessing that you hit thermal throttling all the time, but you aren't aware of it.

You are guessing wrong. I'm always running a monitoring tool(hwinfo). I always stay below my CPU's thermal limits and I have turbo clock at it's max (clocks & performance).

[pingfr]

No fighting here boys! Keep it cool. (You see what I did there?).

[MeteorRain]

Originally Posted by Vesdaris

You are guessing wrong. I'm always running a monitoring tool(hwinfo). I always stay below my CPU's thermal limits and I have turbo clock at it's max (clocks & performance).

I guess the TDP is the maximum generated thermal, not the maximum evacuated thermal. So in most case if you max out your CPU, it will (regardless of cooler) limit itself to a certain point.
I know that some mobo can crack it a bit to have turbo boost running at all time. In this case the only requirement would be a decent sink and cooler.

[benwaggoner]

Originally Posted by pingfr

Passmark tends to be "unreliable" specially if you're looking at multi sockets benchs, on the contrary I believe SPEC benchs are more reliable.

Regardless, thanks a lot for your input.

Also, x265 makes heavy use of AVX2 instructions. So perf-per-clock gets a lot better with a chip that uses that. I wouldn't suggest buying anything pre-AVX for use with x265.

[benwaggoner]

Originally Posted by MeteorRain

I guess the TDP is the maximum generated thermal, not the maximum evacuated thermal. So in most case if you max out your CPU, it will (regardless of cooler) limit itself to a certain point.
I know that some mobo can crack it a bit to have turbo boost running at all time. In this case the only requirement would be a decent sink and cooler.

And we also know that lots of sustained multicore AVX2 operations (heavily used to good effect by x265) can also trigger thermal throttling, apparently irrespective to the level of cooling in action. I know MCW has been looking at this.

Even with that, we see big throughput increases (>50%) on c4.8xlarge instances versus c3.8xlarge instances for x265 encoding with recent builds.

C3.8xlarge: E5-2680v2 (Ivy Bridge), logical CPUs=32
C4.8xlarge: E5-2666v3 (Haswell), logical CPUs=36

While there are a few more cores, it's mainly going from v2 to v3.