Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Video Encoding > High Efficiency Video Coding (HEVC)

Reply
 
Thread Tools Search this Thread Display Modes
Old 2nd January 2018, 23:53   #1  |  Link
ShogoXT
Registered User
 
Join Date: Dec 2011
Posts: 91
Intel(?) bug incoming

https://www.reddit.com/r/sysadmin/co..._bug_incoming/

This is going to really hurt Intel dominance. Results in up to 30 percent performance loss.

Does x264 or x265 happen to use syscalls? Probably not, right?
ShogoXT is offline   Reply With Quote
Old 3rd January 2018, 00:14   #2  |  Link
LoRd_MuldeR
Software Developer
 
LoRd_MuldeR's Avatar
 
Join Date: Jun 2005
Location: Last House on Slunk Street
Posts: 12,960
Any user-space application uses syscalls, mostly for memory (de)allocation and for I/O stuff, even though applications rarely do that directly. They use wrapper "system" userland-libraries, such as glibc (Linux) or kernel32 (Windows), which provide a "high level" interface to the functionality the application needs - and do the syscalls internally, where needed. The C Runtime adds yet another layer of abstraction between the application code and the underlying "system" libraries.

But, most important, unless the application is extremely I/O heavy, the syscall overhead is pretty much negligible.

(And for applications that are very I/O heavy, the syscall overhead always has been a problem! That's why people came up with ideas like mapping the MMIO addresses into user-space and thus passing by the kernel and its drivers altogether )
__________________
There was of course no way of knowing whether you were being watched at any given moment.
How often, or on what system, the Thought Police plugged in on any individual wire was guesswork.



Last edited by LoRd_MuldeR; 3rd January 2018 at 02:16.
LoRd_MuldeR is offline   Reply With Quote
Old 3rd January 2018, 06:16   #3  |  Link
ShogoXT
Registered User
 
Join Date: Dec 2011
Posts: 91
My mistake. I guess what I meant to ask is how much it is relied upon for performance. I'm more of a hardware guy, but I figured that link would be useful.

https://www.phoronix.com/scan.php?pa...5-x86pti&num=2

X264 is uneffected it seems. Not sure about x265.
ShogoXT is offline   Reply With Quote
Old 3rd January 2018, 15:34   #4  |  Link
LoRd_MuldeR
Software Developer
 
LoRd_MuldeR's Avatar
 
Join Date: Jun 2005
Location: Last House on Slunk Street
Posts: 12,960
Quote:
Originally Posted by ShogoXT View Post
My mistake. I guess what I meant to ask is how much it is relied upon for performance. I'm more of a hardware guy, but I figured that link would be useful.

https://www.phoronix.com/scan.php?pa...5-x86pti&num=2

X264 is uneffected it seems. Not sure about x265.
As said before, for CPU-bound applications – and clearly that applies to x264, x265 and pretty much any video encoder – syscall overhead is usually negligible.

For I/O-bound applications that do I/O in the "traditional" way, syscall overhead can become a bottleneck. And that's not a new problem at all! It just might get worse with KPTI (kernel page-table isolation) in place – on some older CPU's.

At the moment everything is speculation, so let's wait for the kernel patches to actually role out. But, as far as x264 and x265 are concerned, you probably won't see a difference – except maybe with "ultra-fast" presets


Newer CPU's support Process-Context Identifiers (PCIDs), which eliminates the need to flush the TLB and thus avoids most of the KPTI overhead. If you have one of those newer CPU's with support for PCIDs, you probably don't need to worry much.
__________________
There was of course no way of knowing whether you were being watched at any given moment.
How often, or on what system, the Thought Police plugged in on any individual wire was guesswork.



Last edited by LoRd_MuldeR; 3rd January 2018 at 23:03.
LoRd_MuldeR is offline   Reply With Quote
Old 4th January 2018, 01:01   #5  |  Link
LoRd_MuldeR
Software Developer
 
LoRd_MuldeR's Avatar
 
Join Date: Jun 2005
Location: Last House on Slunk Street
Posts: 12,960
Apparently not really limited to Intel:
https://googleprojectzero.blogspot.d...with-side.html

Quote:
Originally Posted by Jann Horn, Project Zero
We have discovered that CPU data cache timing can be abused to efficiently leak information out of mis-speculated execution, leading to (at worst) arbitrary virtual memory read vulnerabilities across local security boundaries in various contexts.

Variants of this issue are known to affect many modern processors, including certain processors by Intel, AMD and ARM. For a few Intel and AMD CPU models, we have exploits that work against real software. We reported this issue to Intel, AMD and ARM on 2017-06-01 [1].

So far, there are three known variants of the issue:

Variant 1: bounds check bypass (CVE-2017-5753)
Variant 2: branch target injection (CVE-2017-5715)
Variant 3: rogue data cache load (CVE-2017-5754)
Quote:
Originally Posted by Paul Kocher et al. in "Spectre Attacks"
Hardware. We have empirically verified the vulnerability of several Intel processors to Spectre attacks, including Ivy Bridge, Haswell and Skylake based processors. We have also verified the attack’s applicability to AMD Ryzen CPUs. Finally, we have also success fully mounted Spectre attacks on several Samsung and Qualcomm processors (which use an ARM architecture) found in popular mobile phones
So,there are at least three "variants" of the attack, one of which seems to be Intel-specific (for now), but the others apply to AMD and even ARM as well. So this is not even x86-specific.

Furthermore, as far as I understand, KPTI (kernel page-table isolation), the fix that was discussed to cause some slow-down of syscalls, only fixes one of the three issues...

Details:
* https://spectreattack.com/spectre.pdf
* https://meltdownattack.com/meltdown.pdf
__________________
There was of course no way of knowing whether you were being watched at any given moment.
How often, or on what system, the Thought Police plugged in on any individual wire was guesswork.



Last edited by LoRd_MuldeR; 4th January 2018 at 13:07.
LoRd_MuldeR is offline   Reply With Quote
Old 4th January 2018, 01:34   #6  |  Link
LoRd_MuldeR
Software Developer
 
LoRd_MuldeR's Avatar
 
Join Date: Jun 2005
Location: Last House on Slunk Street
Posts: 12,960
The basic idea is described in the following excerpt (from the "Meltdown" paper):

Quote:
Originally Posted by Moritz Lipp et al.
From a security perspective, one observation is partic-
ularly significant: Out-of-order; vulnerable CPUs allow
an unprivileged process to load data from a privileged
(kernel or physical) address into a temporary CPU reg-
ister. Moreover, the CPU even performs further com-
putations based on this register value, e.g., access to an
array based on the register value. The processor ensures
correct program execution, by simply discarding the re-
sults of the memory lookups (e.g., the modified register
states), if it turns out that an instruction should not have
been executed. Hence, on the architectural level (e.g., the
abstract definition of how the processor should perform
computations), no security problem arises.

However, we observed that out-of-order memory
lookups influence the cache, which in turn can be de-
tected through the cache side channel. As a result, an
attacker can dump the entire kernel memory by reading
privileged memory in an out-of-order execution stream,
and transmit the data from this elusive state via a mi-
croarchitectural covert channel (e.g., Flush+Reload) to
the outside world. On the receiving end of the covert
channel, the register value is reconstructed. Hence, on
the microarchitectural level (e.g., the actual hardware im-
plementation), there is an exploitable security problem.
TLDR: They use the "out-of-order execution" capabilities of the CPU to trick the CPU into speculatively reading some "forbidden" memory area into a temporary register. Later the CPU will realize that it actually was not allowed to read that data, so the speculative results are withdrawn. The "nominal" program execution seems correct. But the "forbidden" data is now in cache - which the CPU does not revert - from where it can be recovered using a cache side-channel attack
__________________
There was of course no way of knowing whether you were being watched at any given moment.
How often, or on what system, the Thought Police plugged in on any individual wire was guesswork.



Last edited by LoRd_MuldeR; 4th January 2018 at 13:13.
LoRd_MuldeR is offline   Reply With Quote
Old 4th January 2018, 02:34   #7  |  Link
froggy1
ffx264/ffhevc author
 
froggy1's Avatar
 
Join Date: May 2007
Location: Belgium
Posts: 1,454
Already on Linux kernel 4.14.11 which by default has PTI enabled and my /proc/cpuinfo reports "bugs : cpu_insecure". I do not see any slowdown in encoding speed here, neither in x264 nor in x265
__________________
ffx264--ffhevc--ffxvid

Last edited by froggy1; 4th January 2018 at 02:45.
froggy1 is offline   Reply With Quote
Old 5th January 2018, 08:52   #8  |  Link
Weyoun
Registered User
 
Join Date: Jul 2017
Posts: 11
The speed penalty for the patch is blown way out of proportion. The "30%" figure I've seen thrown around a lot is based on an artificial benchmark specifically designed to show this penalty and represents NO real world application I know of. x265 is bottlenecked computationally, and will not be effected.
Weyoun is offline   Reply With Quote
Old 5th January 2018, 13:53   #9  |  Link
Atak_Snajpera
RipBot264 author
 
Atak_Snajpera's Avatar
 
Join Date: May 2006
Location: Poland
Posts: 6,734
Quote:
Originally Posted by Weyoun View Post
The speed penalty for the patch is blown way out of proportion. The "30%" figure I've seen thrown around a lot is based on an artificial benchmark specifically designed to show this penalty and represents NO real world application I know of. x265 is bottlenecked computationally, and will not be effected.
Tell that to big servers' owners where I/O is first bottleneck.
EPYC will just destroy Xeon in these tasks after this security patch
No wonder M$ decided to migrate Azure to EPYC.

Last edited by Atak_Snajpera; 5th January 2018 at 13:55.
Atak_Snajpera is offline   Reply With Quote
Old 5th January 2018, 22:21   #10  |  Link
Atak_Snajpera
RipBot264 author
 
Atak_Snajpera's Avatar
 
Join Date: May 2006
Location: Poland
Posts: 6,734
https://np.reddit.com/r/pcmasterrace...atch_and_bios/
Atak_Snajpera is offline   Reply With Quote
Old 6th January 2018, 02:39   #11  |  Link
Weyoun
Registered User
 
Join Date: Jul 2017
Posts: 11
Quote:
Originally Posted by Atak_Snajpera View Post
Tell that to big servers' owners where I/O is first bottleneck.
EPYC will just destroy Xeon in these tasks after this security patch
No wonder M$ decided to migrate Azure to EPYC.
I'm pretty sure the professionals have dedicated encoding machines.
Weyoun is offline   Reply With Quote
Old 9th January 2018, 07:19   #12  |  Link
WhatZit
Registered User
 
Join Date: Aug 2016
Posts: 58
How Likely Is A Spectre/Meltdown Attack?

That title is what I googled, and this level-headed assessment came up:

https://www.networkworld.com/article...h-the-fud.html

So, if true, a low chance of attack on an unpatched system, but still a chance, especially if an unsecured web browser can somehow be coerced into running the exploits.

Otherwise, it's a lower consumer threat level than most other malwares, and sensible precautions could avoid it completely.

Since there are emerging reports that OS+BIOS patched systems have their NVMe SSD performance absolutely clobbered (SATA performance is merely slapped, amongst other hits elsewhere), the question to patch or not to patch becomes more than an obvious security issue. It becomes one of whether you consider the threat serious enough to devalue your hardware, and do you apply only the OS Meltdown patch, or the Spectre BIOS patch as well.

Last edited by WhatZit; 9th January 2018 at 07:28. Reason: Added NVMe Scary Story
WhatZit is offline   Reply With Quote
Old 9th January 2018, 13:47   #13  |  Link
Motenai Yoda
Registered User
 
Motenai Yoda's Avatar
 
Join Date: Jan 2010
Posts: 692
Quote:
Originally Posted by WhatZit View Post
Since there are emerging reports that OS+BIOS patched systems have their NVMe SSD performance absolutely clobbered (SATA performance is merely slapped, amongst other hits elsewhere), the question to patch or not to patch becomes more than an obvious security issue. It becomes one of whether you consider the threat serious enough to devalue your hardware, and do you apply only the OS Meltdown patch, or the Spectre BIOS patch as well.
Useless test and biased conclusions as those are tests that can change a lot even on phases of the moon.
2 of 3 benchs show no drop even on 4k, 1 bench has huge drop on every test, I'd blame that bench.
__________________
powered by Google Translator
Motenai Yoda is offline   Reply With Quote
Old 12th January 2018, 18:14   #14  |  Link
benwaggoner
Moderator
 
Join Date: Jan 2006
Location: Portland, OR
Posts: 2,758
Quote:
Originally Posted by Weyoun View Post
I'm pretty sure the professionals have dedicated encoding machines.
LOTS of encoding happens in the cloud today. Netflix has talked a lot about how their entire infrastructure is on AWS.

It's a pain to do in the cloud for high-touch craft encoding since uploading sources takes forever. But commercial content is already in the cloud.
__________________
Ben Waggoner
Principal Video Specialist, Amazon Instant Video

My Compression Book

Amazon Instant Video is hiring! PM me if you're interested.
benwaggoner is offline   Reply With Quote
Old 12th January 2018, 18:46   #15  |  Link
Blue_MiSfit
Derek Prestegard IRL
 
Blue_MiSfit's Avatar
 
Join Date: Nov 2003
Location: Los Angeles
Posts: 5,432
YEP! Massively parallel chunked encoding on cheap spot EC2 instances is a game changer.
Blue_MiSfit is offline   Reply With Quote
Old 13th January 2018, 09:24   #16  |  Link
Weyoun
Registered User
 
Join Date: Jul 2017
Posts: 11
Quote:
Originally Posted by benwaggoner View Post
Netflix has talked a lot about how their entire infrastructure is on AWS.
I have no sympathy for dumb companies that do stuff like this.
Weyoun is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 07:24.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2019, vBulletin Solutions Inc.