There’s currently some debate about whether there was an issue within the SuperMicro supply chain. Bloomberg publicized that they had become aware of compromised chips that Supermicro, an assembly partner used by Apple and Amazon, had been planted and could be used by the Chinese government.
While this is still being debated whether this is a valid report or not, let’s discuss what the possible ramifications might be:
For customers who purchase potentially vulnerable equipment?
For attackers who may find methods to use this vulnerability to their advantage?
Vince Kelly says
I realize that this may be a little bit of an investment in terms of reading but I think that the Bloomberg article below definitely does goes to the heart of the entire “open source, open protocol, open compute/white box” debate.
I actually know and worked with a rather large SI/Cloud provider (NOT any of the big four!:) who purchased *many* Supermicro platforms over the years – we actually had several debates about the wisdom of doing that. Certainly not gloating over anyone else’s misfortune here – I can’t imagine how panicked they must be right now.
The article details how the Chinese military was able to implant nearly microscopic (about the size of a grain of rice!) microchips disguised as simple DSP hardware onto, and in some cases even INTO the motherboards of Supermicro computer systems. The attack,(if it did actually happen) is extraordinary on several levels. First, the chip design was cutting edge technology – something that only a state-sponsored entity could afford to pull off. The chips were not only programmed to gain access to the operating system when it booted up but it was also programmed to establish a backdoor command and control channel so that the target could be manipulated remotely later on.
Second, up until now hardware based attacks were thought to be so difficult to pull off that it was felt to be impractical, especially given the fact that it’s far easier to spread malicious software than it is to design, build and install specialized hardware like the microchips discussed in the article and therefore the general consensus was that hardware based attacks represented only minimal risk.
The third extraordinary aspect of this attack was the method that the Chinese military used to implant the microchips. According to the article, there are two methods available to the intelligence community when attempting technology based infiltration like this. One method is called ‘interdiction’ which involves intervening at some point within the logistical system or the supply chain of the target. The other method is known as ‘seeding’ and involves infiltration at the earliest point of the manufacturing process. This method was believed to be more of a ‘shotgun blast’ attempt at infiltration – something that is so defuse and inexact at getting to a specific target that its use was thought to be almost beyond the realm of practicality or probability. And yet this was the method that the Chinese military successfully used to gain access to Apple, AWS and more than 30 other companies and governmental agencies.
I don’t think that the importance of this event can be understated because the repercussions from this form of attack call into question the stability, efficacy and confidence in the entire global supply chain system.
But what does this have to do with the open source/open platform debate? Supermicro is what is known as a ‘white box’ computer manufacturer and is THE poster-child for open source – everything. The entire white box ‘movement’ that Supermicro represents in my opinion came about as a kind of ‘rebellion’ against a perception of proprietary, over-designed, over-engineered and over-priced computer manufacturers products. This started to gain momentum (again in my opinion) when the hyper scale cloud providers and social media conglomerates like AWS and Facebook made it known that they were building their own computing platforms completely from COTS components. Facebook lead an open source based initiative that ultimately morphed into the Open Compute Project (OCP). An entire ecosystem of ‘cheap’, ‘generic’, COTS component integrators started manufacturing and selling ‘plain ol vanilla’ server hardware at rock bottom prices. Supermicro became a dominant player within that ecosystem by taking a non-proprietary, open source philosophy and growing it into a $3B business.
The problem however is that in their haste to drive costs out of their business, Supermicro also lost control of its supply chain. Even though Supermicro has facilities in San Jose, the Netherlands and Taiwan, it completely disintermediated its supply chain and outsourced all of its motherboard assemblies to Chinese component manufacturers. These component manufactures became the entry point where the nefarious microchips were embedded on the Supermicro motherboards.
The point of all of this is that open source is a good thing – when it’s applied in moderation – 🙂
In my opinion, Supermicro is a cautionary tale about the potential perils that can occur by over-relying on open source strategies. It’s a cautionary tale not because of the intricate levels of intelligence gathering and technical wizardry but because at times it can be difficult or even impossible to know with absolute certainty where all of the components that ultimately make up an open source system actually came from.
Proprietary approaches are also a good thing, when they are applied in moderation 🙂 The need for market differentiation can give rise to proprietary approaches that foster innovation and drive competition. But most importantly proprietary approaches by their very nature drive a level of control that open source systems simply cannot match – ask any airline pilot or passenger if they would prefer that the version of the flight control systems that are controlling their aircraft used a completely open source operating system or a ‘locked-down’ proprietary version. I think the answer would be pretty obvious.
What are the implications for customers who purchase potentially vulnerable equipment?
I think there are substantial implications. First, according to the article, this type of attack is *extremely* hard to detect. The chips themselves were disguised as simple DSPs and in some cases were even implanted *into* the motherboard itself. As a result really the only alternative for customers who purchased this equipment is to error on the side of caution – they must *assume* that the platforms that they have are contaminated. I don’t think that this necessarily means that they need to toss all of their platforms out – that may be economically infeasible for many companies. Instead, they should probably take a couple of steps immediately:
1. Identify where/ what locations the platforms are deployed into and if possible consolidate them together as much as possible (in order to keep an eye on them).
2. Identify what applications are running on those platforms and immediately start migrating any application that has any kind of value over to a more secure platform. Of course, this in and of itself could potentially create problems because now they would need to verify that the OS running on the Supermicro hasn’t already been modified/contaminated and hasn’t also in turn contaminated all of the applications files as well.
3. Once steps 1 and 2 have been accomplished, isolate the Supermicros as much as possible from the rest of the network infrastructure. Surround the Supermicros by deploing IDS/IPS and NGFWs around them. If they can’t physically move the boxes then put them on isolated L2 VLANs and closely monitor all traffic coming into and out of those segments.
For attackers who may find methods to use this vulnerability to their advantage?
…I think that the final point to be made here is this; There seems to be some debate about the accuracy of the article that has cropped up in the last 24 hours. Whether this actually happened or not, everyone would have to agree that the attack has been successful at least from the perspective that it’s instilled at least some level of suspicion and doubt in many people’s minds about the possibility of the story being true. And that, in my opinion is where the Chinese military have succeed. You don’t need to actually *attack* something or someone in order to cause mayhem – you just need to sow the seeds of doubt that you (the attacker) have just perpetrated a nearly undetectable infiltration and the potential victims *may* never know when you’ll decide to strike! 🙂
Duy Nguyen says
Hi Vince,
Great post, a lot of good points.
I don’t know much about this area of IT but reading your post and the article has me thinking a lot about what type of damage can be done. How would the government mitigate this type of attack? Having a competitive country have access to backdoors into the infrastructure of the 2 or 3 of the top 5 IT companies in the world. What kind of damage if it came down to it would this cause. Amazon and Facebook alone had millions upon millions of user’s personal private data. Exploiting Amazon alone would be disastrous for the US economy.
That being said, in my opinion, this article is not or has a high percent chance of not being true. I would think that there’s no way a company of Apple or Amazon level of sophistication would have this vulnerability go so long without identifying it or having it mitigated.
Dan Bilenker says
If true, this would present a security threat at the user level, but would also suggest the infection of global supply chains. As Vince stated, technology of this caliber is never achieved bu amateurs, and would require state sponsored backing. This is especially true if the supply chain has been compromised to the extent that is alleged.
What would be most alarming, is the notion of a Government successfully infiltrating a state based organization, and disseminating malicious product globally. The ramifications of a nation being able to disperse their reach that far, with no detection has scary implications. If this allegation is true, and indeed sponsored by the government, imagine what an advantage the Chinese Military would have in global conflict.