Netflix To Re-Encode Entire 1 Petabyte Video Catalogue In 2016 To Save Bandwidth

[raffriff42]

http://hardware.slashdot.org/story/15/12/15/1558242/netflix-to-re-encode-entire-1-petabyte-video-catalogue-in-2016-to-save-bandwidth
http://variety.com/2015/digital/news/netflix-better-streaming-quality-1201661116/Over the past few months, Netflix has dared some of its employees in its Los Gatos offices with a special kind of challenge: Two TVs mounted side-by-side were playing the same TV show episode. One was coming straight from Netflix’s existing service, the other was based on a new bandwidth-saving technology that the company has been working on for four years. Anyone capable of pointing out the difference could win a bottle of champagne. But in the end, even eagle-eyed employees had to give up, and the prize went unclaimed...

Netflix’s service has been dynamically delivering ... based on a consumer’s bandwidth needs... which didn’t really make sense. “You shouldn’t allocate the same amount of bits for ‘My Little Pony’ as for ‘The Avengers,’” explained Netflix video algorithms manager Anne Aaron.

...a show like “My Little Pony” is not at all like the ‘Avengers’ — especially to a computer in charge of preparing media files for streaming. Animated shows like “My Little Pony” can be reproduced with relatively little data. An animated sky, for example, tends to be filled with the same shade of blue, and an animated pony that just stands around talking isn’t very complex to a computer either. The “Avengers” on the other hand is full of fast-paced action, which plays out in front of cityscapes and other environments with lots of visual details. “Noise is hard to encode,” said Aaron.

...This allows the company to stream visually simple videos like “My Little Pony” in a 1080p resolution with a bitrate of just 1.5 Mbps.

...they quickly realized that no two indie movies are alike, and that there can even be considerable visual differences within a season of the same TV show. “Each episode could be very different,” said Aaron. The result is a true title-by-title approach, where every single movie and TV show episode gets its own encoding settingsI predict the first wave, hand tweaked by NetFlix's top engineers, will look great. Then they will outsource the rest to who-knows-where, and the quality will be uh, "variable."

[wonkey_monkey]

So... what? They (re-)invented constant quality encoding...?

I guess there might be a little bit more to it than that, but it sounds like they originally had everything encoded at a flat 5800kbps (or possibly encoded so everything averaged 5800kbps), regardless of the content.

So not really sure what, if anything, is "new" here, as far as video encoding goes.

[luquinhas0021]

What?????????????? 1.5 Mbps are equal to 0.1875 MBps. This is for a 1080p video... Impressive! I just want know what is the fps of video you've talked about. If we consider a non-compressed 1080p RGB 888 non-3D 24 fps has bitrate of 145.8 MBps, the Netflix bitrate is quite strange. 777.6 times of compression... I only wish see the artifacts of this algorithm!

[foxyshadis]

You can download a sample of any YIFY release to see what 1080p@1.5 mbps looks like, if you don't want to do it yourself. It's not unwatchable, barely, but it's definitely not even in the same league as bluray or high-rate recompression. (Comparing against the uncompressed bitrate is pointless, though, that's just 101 Intro to H.264. Video compresses well, who'd have thunk it!)

I wish they'd use the extra bandwidth for more quality, since Netflix always looks noticeably more smooth, but at least they're trying to spend more cpu time on encoding to maximize their bitbudget.

[hello_hello]

So... what? They (re-)invented constant quality encoding...?

It doesn't seem like it, if the article is accurate. I don't understand what they're doing. Sounds like a method of analysing the video before it's encoded to work out the bitrate required for a particular quality.

http://variety.com/2015/digital/news/netflix-better-streaming-quality-1201661116/
Over the past couple of years, Netflix’s video algorithm engineers have worked on perfecting this more flexible approach towards video encoding...... The result is a true title-by-title approach, where every single movie and TV show episode gets its own encoding settings.

The article ended with this:
And then there is another crazy idea that could require the company to re-encode the entire catalog all over again: After finding the best setting for each single video, Aaron’s team is now thinking about even encoding each scene of a movie or TV show with different settings to account for higher information density during fight scenes and lower demands during slow moments of introspection.

[nevcairiel]

The article ended with this:
And then there is another crazy idea that could require the company to re-encode the entire catalog all over again: After finding the best setting for each single video, Aaron’s team is now thinking about even encoding each scene of a movie or TV show with different settings to account for higher information density during fight scenes and lower demands during slow moments of introspection.

Sounds like something any half-competent encoder should already do by itself. That said, professionally mastered video encodes do use per-scene tuning as well.

[Atak_Snajpera]

Indeed netflix has invented wheel again. This is probably nothing but cq mode with set max bitrate limit.

[Doom9]

I'd rather they do a few different quality level in function of the available bandwidth. I have an insane line and the ISP hosts a Netflix cache server so I wouldn't mind them streaming at blu-ray rivaling bitrates or 100mbit for 4K... I'm not saying it is for everyone, but given the capabilities, why downgrade what could be delivered in full quality. It's even worse with all those IPTV solutions.. the OTT ones use minimal bitrates to fit those last century DSL lines, the provider based ones are a bit better, but far away from the original Satellite signal that they could easily feed to their customers without any transcoding.

[wonkey_monkey]

given the capabilities, why downgrade what could be delivered in full quality

Someone's got to pay for the traffic. It's probably not worth their while offering 100mbit streams for the number of users that would care for the benefit and/or be willing to pay the higher prices.

[huhn]

720p with higher mbit like 1080p or UHD mbit would be a good start.

[foxyshadis]

Someone's got to pay for the traffic. It's probably not worth their while offering 100mbit streams for the number of users that would care for the benefit and/or be willing to pay the higher prices.

On the peering side (the backbone), the difference between a 5 GB movie and a 50 GB movie is negligible. Less than a penny more. On the last-mile side, costs are higher, but that's exactly why I pay my ISP that inflated monthly price; it makes no sense that they're so heavily quality restricted.

Storage costs are probably what would kill them, though.

[Doom9]

It's probably not worth their while offering 100mbit streams for the number of users that would care for the benefit and/or be willing to pay the higher prices.Unless you've tried, how are you supposed to know? I say give the people a choice rather than making the choice for them. I'm sure plenty of "oh, it's good enough for me" people would eventually come around if they get to experience the quality difference for themselves. What Netflix offers as 1080p and what you get on BluRay for instance - with today's large screen TVs, you'd have to be blind not to see it. People are willing to pay extra for extra quality after all.. both for streaming services as well as TV service.

[wonkey_monkey]

On the peering side (the backbone), the difference between a 5 GB movie and a 50 GB movie is negligible.

That may be so - I have no idea - but there's still only so much data one network card or one server can transmit. Raw bandwidth isn't the only cost. Even their power bills would go up.

Unless you've tried, how are you supposed to know?

Because why else wouldn't they have tried? If they thought they could make more money out of it, I'm sure they would (subject to the law of diminishing returns). But I can only assume they've figured out it's not worth it yet. Don't they also run the risk of raising the ire of ISPs - again - by upping bitrates? Just because you pay for a 100mbit/s connection, doesn't mean your ISP can actually supply that to you and all your neighbours at the same time from any service. If you did all start trying to stream 100mbit/s movies, the ISPs are probably going to start demanding compensation from Netflix, as I believe previously happened to either Netflix or one of their predecessors.

[kuchikirukia]

My Little Pony: Friendship is Magic is Flash animation. Don't be surprised that 1.5Mbps is fine.
Be surprised that season 1 was really good.

[kolak]

http://hardware.slashdot.org/story/15/12/15/1558242/netflix-to-re-encode-entire-1-petabyte-video-catalogue-in-2016-to-save-bandwidth
http://variety.com/2015/digital/news/netflix-better-streaming-quality-1201661116/I predict the first wave, hand tweaked by NetFlix's top engineers, will look great. Then they will outsource the rest to who-knows-where, and the quality will be uh, "variable."

I don't know how much Netflix does in-house (if any), but I know that there are few houses which do transcoding for them. What I also know is that Netflix is quite strict with their suppliers (compared to others). Few % rejected files and you loose your Netflix approval for years. In the same time- this is still transcoding on massive scale, so no one is going to pick x264 settings per title. This is simply impossible due to scale of the operation. What could be done is some sorting based on automated or human assisted "video compressibility measurement". This sounds doable. Other than this simply use CRF mode with max bitrate(s).

1PB is about 11000 hours of ProRes source, which is not that crazy number at all. 1000 (even more) hours per week can be done on 1 good server, so 10 boxes could do it in a about week time.

[SquallMX]

The new season of Daredevil was encoded using this "new bandwidth-saving technology", for the high 1080p mode, bitrate varies per episode from 4600 to 6200 kbps, not much different from the old fixed bitrate (5800), but the other resolutions are way more lower than before, the 720p mode when from 3300 Kbps to around 2000 Kbps, I wonder if they are using a lower CRF only for the highest quality mode? Also, they changed the audio codec too, before they were using AAC-HE or WMA10 at 64 or 96 Kbps, now they are using DD+ at 128 Kbps for stereo content, before DD+ was only used for 5.1 (at 192 Kbps).

Quality seems to be good at least during normal playback.