GPU to convert a +300TiB library to HVEC?

[jfromeo]

Hello all.

I am building a rig to mainly use for encoding a whole media library of 360TiB to H265, with a projected reduction in size of around 50%. The rig will be used afterwards as a Plex server for the decoding on GPU.

I am aware there will be losses as it would be a lossy to lossy conversion (library is encoded on H264), but I would focus on quality rather than speed.

I discarded software encoding (CPU) as It would take several years to accomplish.

There are lots of info and tests with CPU encoding, but I cannot find much on GPU. I have to decide to buy among the AMD 7900XTX/9700XT and nVidia 5070/5080. My maximum budget on GPU is around 1.000€. The CPU I have in mind is the Ryzen 9 9950x3D (to be released)

Is nVidia better for this task or could i stick to AMD? How many simultaneous streams could I be encoding with both? Power draw? I plan to use Tdarr on Linux for it.

Thanks in advance.

[benwaggoner]

Originally Posted by jfromeo

I am building a rig to mainly use for encoding a whole media library of 360TiB to H265, with a projected reduction in size of around 50%. The rig will be used afterwards as a Plex server for the decoding on GPU.

I am aware there will be losses as it would be a lossy to lossy conversion (library is encoded on H264), but I would focus on quality rather than speed.

I discarded software encoding (CPU) as It would take several years to accomplish.

There are lots of info and tests with CPU encoding, but I cannot find much on GPU. I have to decide to buy among the AMD 7900XTX/9700XT and nVidia 5070/5080. My maximum budget on GPU is around 1.000€. The CPU I have in mind is the Ryzen 9 9950x3D (to be released)

Is nVidia better for this task or could i stick to AMD? How many simultaneous streams could I be encoding with both? Power draw? I plan to use Tdarr on Linux for it.

So, backing up a bit, what's your end goal with all of this? Getting 50% reduction with hardware recompression will be visually lossy in many shots, and you'll still need to store 180 TB. While what you want to do is practical, storage is cheap enough these days that it generally pencils out better to just get more storage rather than spend that money on encoding hardware, electricity, and time to do the recompression.

You can get 180 TB of drives for around $3000 these days. Which is a lot, sure, but if you add up the new GPU, the power bill, the time you'll spend managing and testing all of this, $3K can turn out to be a bargain surprisingly quickly.

[Selur]

I agree, doesn't really make sense to me either.

About the gpu encoding: if no machine learning stuff is needed and the aim is the hardware encoder, I would recommend using Intel instead. (additionally, maybe go for av1 instead hevc)

[Emulgator]

If quality: Concurring, keep it as-is. The effort might not be worth it, exactly as described by Ben and Selur.
Even for my semi-pro uses it comes out as the same balance sheet betweeen electricity cost plus loss of detail vs shaving some % TB.
Gaining some percent of storage space is quickly moot because safe space is to be provided anyway, here at a margin of 1.6 (RAID6 + Hotspare)
The recently developed codecs aim at distribution paths that have to save bandwidth at all costs, satellite broadcast being the driving party.
Well, those have to pay bandwidth dearly, and so have to be inclined to shave off some detail.
Maybe that the private and semi-pro vaults do not have to jump on the latest bandwagon
which will only accommodate sources at the cost of pre-cleaning them to reduce encoding complexity.
"Won't pay for less than 10Mbps streams..."
And if you must encode, yes, I have achieved good, say 80..90% detailed encodes from Intel 750 and nVidia 3080 GPUs
with drivers from 2024.
If I don't pixel-peep: No personal preference at the moment, both perform quite nice,
Intel eating even less electricity while doing it "under the hood".

[Z2697]

You should buy "dedicated" encoding cards rather than spending hundreds on Gaming Cards then only do encoding on them.
A big factor here is how many "parallel" streams you can encode, take Nvidia for example, you won't even get 3 encoding engines without going for the most expensive one (RTX 5090).
(While some may accept more streams, the overall speed cannot exceed the capability of the encoding engines)

For quality, AMD's hardware encoder, at least the old ones (before RX 9000 series), usually fall far behind Nvidia and Intel. New series? No one tested yet.

Although I don't think it's a good idea, but if I must recommend some consumer hardware that you can buy for "personal large-scale encoding", I'd suggest multiple low-end Intel Arc GPUs, the encoding engine is the same in high-end and low-end. (keep attention on generations though.) But team green and red don't really go low-end these days.
You can even have a little better quality with Intel hardware encoder compared to Nvidia's, based on some online tests.

[RanmaCanada]

Your only real option is Intel. Just build your server with an Intel CPU with an iGPU and use the embedded iGPU for encoding (Alder Lake or newer). As the others mention, a full 50% compression will make things look like garbage. There is no need to buy an expensive gaming card when Quicksync exists and is superior to everything else on the consumer level when it comes to hardware encoding.

[jfromeo]

Thank you all for the options.

I will reconsider adding more storage. Problem is that I am currently at 24x20TB with redundancy, and I run out of space to add another 4U JBOD.

I also have a low-power NAS on an Intel Xeon E-1245v6, it has an iGPU but I do not know if it would be fast enough.

I could add something similar to and Intel ARC A380, would that help?

[RanmaCanada]

I believe an Arc A380 would suffice, even an A310 really. The Xeon you have is Kaby Lake and will not suffice.

[benwaggoner]

Originally Posted by Z2697

You can even have a little better quality with Intel hardware encoder compared to Nvidia's, based on some online tests.

Do you have links to any you find compelling?

These things are very hard to test apples-to-apples.

[Z2697]

Originally Posted by benwaggoner

Do you have links to any you find compelling?

These things are very hard to test apples-to-apples.

https://rigaya.github.io/vq_results/

I don't 100% sure about the results too, they are indeed very hard to test apples-to-apples.
Somehow the listed speed of NVENC is wayyy slower than what I get.

[benwaggoner]

Originally Posted by Z2697

https://rigaya.github.io/vq_results/

I don't 100% sure about the results too, they are indeed very hard to test apples-to-apples.
Somehow the listed speed of NVENC is wayyy slower than what I get.

Thanks for the link. Yeah, there are doing only very basic tuning in those parameters. Unless you're looking at quality at the same bitrate AND similar encoding time, you're not really comparing what the encoders are capable of.

The sources are already compressed long-GOP content, which can have different performance impacts on different pipelines.

VMAF is the best single metric we have, but it still has significant error bars on it as well. It's very possible to tune an encoder to optimize for VMAF in ways that don't help subjective quality nearly as much.

It's hard to say much definitively from this, particularly about efficiency @ time.