The fix for IT supply chain attacks

The fix for IT supply chain attacks:

As I’ve written previously, I’m very skeptical of Bloomberg’s report about the Chinese placing hardware spy chips on server motherboards used by U.S. companies. China is actively spying on U.S. businesses all the time, I believe, and has already stolen most of the intellectual property secrets they are interested in. The Chinese are on their way to becoming the world’s leading economic power, and manufacturing computer chips is a big part of that equation. I don’t think they would jeopardize that business so blatantly.

If any good is to come out of the Bloomberg article, it is bringing the problem of the supply chain to the forefront. If nearly every computer device and chip is made by potential adversaries, how can you ever be assured that what you are buying doesn’t have intentional bugs or even spying chips?

The supply chain is the aggregation of all entities that provide the products and services needed for other entities to provide their products and services to their customers. Theoretically, any entity can knowingly or unknowingly introduce insecurity that impacts the final product. This is the exact issue that the Bloomberg authors and their anonymous sources allude to: that a spy chip can be placed on motherboards that eventually get placed into servers used by foreign companies.

IT supply chain risk has always existed

This is not a new issue. …

Keeping the supply chain status quo is not an option

So, one solution is no solution: Keep things as-is. As far as we know, incidents of nations using supply-chain malicious inducements are rare. If a nation-state compromised the supply chain too routinely, none of the other nations would buy its chips. It would be a self-solving solution. We’ve made it so far, so good, using this “strategy.”

When do you use a detect-and-regulate supply chain strategy?

… Well, for one, the military already has programs to prevent supply chain issues for its most critical infrastructure. Many levels of the U.S. government have programs that look for malicious supply chain issues. That’s precisely why I don’t believe that we have a widespread issue of Chinese spying chips all over the U.S.

The question is at what level of the supply chain do we start requiring stricter oversight and monitoring? …

The opposite school of thought to the “keep the status quo” argument is that we need to check all computer devices for spying hardware, software and firmware. This can be done by government or industry groups (like the Underwriter’s Laboratories [UL] or Consumer Reports). The problem is that all governments want to spy on people — its own people, and those in other countries. Asking the government to make sure everything is secure and not spying is asking for the fox to guard the henhouse. At the same time, I’m not sure we can do what needs to be done without governmental involvement.

The supply chain security solution needs to be global

… Every nation needs a nationally created and funded regulatory group that can look for supply chain issues but isn’t directly governed by the government. It’s not perfect. It’s like asking the foxes to pay for the shepherds who protect the henhouse, but I don’t see any other realistic way for a supply chain security solution to actually work. Or we can keep the status quo and hope for the best.

(Via CSO Online)

I agree with the article in large part. I disagree that government action and international agreements are the way to address supply chain risks. It is vulnerable in a multitude of ways independent of hardware hacking like the Bloomberg report claims. Compromising hardware not only requires physical access but its own reliance on a supply chain.

I tend toward industry and market forces addressing all aspects of supply chain insecurity. Redundancy, resiliency, supplier diversity, quality assurance, and monitoring are best done by those with the most at risk. Governments are too mercurial, international agreements and treaties often are not worth the paper they are printed on, and special interests can introduce new risks into the equation through self interest and a lack of vision.

Also on:

Chinese Supply Chain Hardware Attack

Chinese Supply Chain Hardware Attack:

Bloomberg is reporting about a Chinese espionage operating involving inserting a tiny chip into computer products made in China.

I’ve written (alternate link) this threat more generally. Supply-chain security is an insurmountably hard problem. Our IT industry is inexorably international, and anyone involved in the process can subvert the security of the end product. No one wants to even think about a US-only anything; prices would multiply many times over.

We cannot trust anyone, yet we have no choice but to trust everyone. No one is ready for the costs that solving this would entail.

(Via Schneier on Security)

The story moved since poblication last week, but Bruce’s words still hold true.

Also on:

The Big Hack: How China Used a Tiny Chip to Infiltrate U.S. Companies – Bloomberg

The attack by Chinese spies reached almost 30 U.S. companies, including Amazon and Apple, by compromising America’s technology supply chain, according to extensive interviews with government and corporate sources.
— Read on

I like this quote from the TV show Elementary:

Are governments capable of evil? Yes. Of corse they are. All institutions are. But they are more capable of incompetence.

Apply a bit of Occam’s Razor as well and the puzzle gets a bit less scary.

Again, the news is still forthcoming so I may well eat my words.

Also on:

TaoSecurity: Network Security Monitoring vs Supply Chain Backdoors

The limitations of this approach are worth noting. First, if the intruders never activated their backdoors, then there would be no evidence of communications with C2 servers. Hardware inspection would be the main way to deal with this problem. Second, the intruders may leverage popular Internet services for their C2. Historical examples include command and control via Twitter, domain fronting via Google or other Web sites, and other covert channels. Depending on the nature of the communication, it would be difficult, though not impossible, to deal with this situation, mainly through careful analysis. Third, traditional network-centric monitoring would be challenging if the intruders employed an out-of-band C2 channel, such as a cellular or radio network. This has been seen in the wild but does not appear to be the case in this incident. Technical countermeasures, whereby rooms are swept for unauthorized signals, would have to be employed. Fourth, it’s possible, albeit unlikely, that NSM sensors tasked with watching for suspicious and malicious activity are themselves hosted on compromised hardware, making their reporting also untrustworthy.

The remedy for the last instance is easier than that for the previous three. Proper architecture and deployment can radically improve the trust one can place in NSM sensors. First, the sensors should not be able to connect to arbitrary systems on the Internet. The most security conscious administrators apply patches and modifications using direct access to trusted local sources, and do not allow access for any reason other than data retrieval and system maintenance. In other words, no one browses Web sites or checks their email from NSM sensors! Second, this moratorium on arbitrary connections should be enforced by firewalls outside the NSM sensors, and any connection attempts that violate the firewall policy should generate a high-priority alert. It is again theoretically possible for an extremely advanced intruder to circumvent these controls, but this approach increases the likelihood of an adversary tripping a wire at some point, revealing his or her presence.

— Read on

An assessment of the Bloomberg hardware compromise report which provides insights I hinted at but are better articulated here.

I remain skeptical this happened. It seems cheaper and easier to introduce fear, uncertainty, and doubt (FUD) into the supply chain than to actually compromise it (beyond what the Chinese supply chain already does to skim money). Again, time will tell.

Also on:

Notes on the Bloomberg Supermicro supply chain hack story

Notes on the Bloomberg Supermicro supply chain hack story:

Bloomberg has a story how Chinese intelligence inserted secret chips into servers bound for America. There are a couple issues with the story I wanted to address.

The story is based on anonymous sources, and not even good anonymous sources. An example is this attribution:

a person briefed on evidence gathered during the probe says

That means somebody not even involved, but somebody who heard a rumor. It also doesn’t the person even had sufficient expertise to understand what they were being briefed about.

The technical detail that’s missing from the story is that the supply chain is already messed up with fake chips rather than malicious chips. Reputable vendors spend a lot of time ensuring quality, reliability, tolerances, ability to withstand harsh environments, and so on. Even the simplest of chips can command a price premium when they are well made.

(Via Errata Security)

The truth on this story is still revealing itself. I do know that I already tire of it.

Robert Graham’s article is the strongest critique of the Bloomberg story I’ve read. My skeptical nature tends to agree with him until more facts are known.

Also on:

Supply Chain Security

USB flash drivers sent with Conext Combox and Conext Battery Monitor products, part of Schneider Electric’s solar power range, were “contaminated” during the manufacturing process, according to a security advisory released by the industrial equipment manufacturer.

Schneider Electric says that the USB media “may have been exposed to malware during manufacturing at a third-party supplier’s facility.”

Here’s the thing: people won’t remember who Conext is. They probably provide bits and bobs for a bunch of other companies, and this issue might impact those other companies (and consumers, obviously). But this will be associated with Schneider because they are they headliner.

Also on:

Supply Chain Attack Hits Organizations In South Korea

Supply Chain Attack Hits Organizations In South Korea:

Supply-chain attacks are very difficult to detect and prevent because malware comes from an outside source considered trustworthy. Contaminating the waterhole, though, is sometimes easier to achieve than going directly after the target, who may have strong defenses in place.

(Via BleepingComputer)

I would change this to say, “Supply-chain attacks are very difficult to detect and prevent because malware comes from an outside source assumed trustworthy, if considered at all.” This is true for most companies.

I appreciate a subtile approach regardless of a malicious actor’s malevolence:

The malicious actor made sure that the compromised version of the software did not spread to entities that were not of interest. For this, they set up the update server to send out the infected files only if their target was located within a specific range of IP addresses.

To avoid detection, the malicious update was signed with a valid certificate stolen from the remote solutions provider. It is unclear when this occurred, but researchers say that on April 8 they found a piece of malware that hid under the same stolen certificate.

With signed malware and access to the update server, all the threat actor had to do was to wait for a client to request a software update.

If the call came from the targeted IP range, the attacker sent the update server the malicious file packaged as “” When the update executed, so did the 9002 RAT inside it.

Considering that the update process is likely encrypted by default, catching this early in the kill chain is unlikely.

See Wired’s writeup on NotPetya.

Also on:

Global Supply Chains Are Dangerously Easy to Snap

Global Supply Chains Are Dangerously Easy to Snap:

… Global supply chains are, in fact, a national security issue, and one that has been neglected by planners for too long.

… When everything works, the supply chains allow distributors and retailers to keep minimal stocks, a model known as just-in-time. In the U.K., for example, many retailers only stock 24 hours’ worth of fresh produce. The system works so well that between 2010 and 2015, 52 percent of U.K.-based suppliers reduced their stock levels, while just 22 percent increased their stock. That, too, helps keep consumer prices low by saving on warehousing costs. At Tesco, Britain’s largest retailer, an orange, whether it’s from Argentina, Chile, Cyprus, Egypt, Israel, Morocco, Peru, South Africa, Spain, Swaziland, Turkey, or Uruguay, sets the consumer back a mere 30 pence (39 cents).

But if someone damaged the supply chains, all of this falls apart fast.

… Cyberattacks—whether perpetrated by a government or proxies—could wreak havoc in companies’ logistics systems, which organize the travel route of every product. Or, an adversary could sabotage harbor operations. For that matter, workers at harbors or distribution centers could simply go on strike. E-commerce is just as exposed as brick-and-mortar chains to supply chain disruptions.

Delivery disruptions are, of course, not a new concern. So limited were supplies during World War II that the British government was forced to maintain rationing after the war had ended. 

… But the real challenge may be nature. Britain’s sweltering summer heat, a byproduct of climate change, has already caused shortages of fans and air conditioners in a country unused to scorching weather. After Japan’s 2016 earthquakes, Toyota—one of the world’s largest automakers and a pioneer of just-in-time—had to suspend production at its Japanese factories because it couldn’t get the parts for its cars. In a 2015 report for the Hawaii Department of Transportation, Ian Robertson, an engineering professor at the University of Hawaii, recommends that in case of an earthquake or tsunami warning “every effort should be made to evacuate all ships and barges.” But Hawaiians rely on those very ships for their livelihoods. “Closure of any of the Hawaiian commercial ports for more than a week due to storm or tsunami inundation would severely affect the health and safety of island residents,” Robertson notes in the report.

Disruptions to global supply chains are, in fact, more devastating than a traditional military attack. “We assume that we’ll always have daily deliveries, and consumers have come to rely on it,” Marsden, the Cardiff University professor, noted. “But we only need to look at truck-driver strikes to understand the effects of disruptions to supply chains. The 2000 lorry-driver strike put the fear of God in the [U.K. Ministry of Defense].” Brazilians suffered a similar fate earlier this year, when a trucker strike caused food and fuel shortages. When President Michel Temer responded by sending in the Army, commanders discovered that they, too, were short on fuel. An adversary could bring a country to its knees without dispatching a single soldier.

… And exactly because an attack on global supply chains (not to mention natural disasters or animal or plant disease) is more likely than a military attack, this is not a paranoid scenario. On the contrary, we have been lucky that our fragile supply chains have not yet been hit. Navies, including Britain’s Royal Navy, protect global shipping traffic, but supply chains can be sabotaged by cyberattacks or attacks on harbors or distribution centers. By teaming up, governments and the private sector could better protect these lifelines.

… Governments and industry might also consider whether the global supply model is sustainable. No supply chain is completely secure, but localized production reduces vulnerabilities.

(Via Foreign Policy)

Also on:

3rd Party Supply Chain Security in the Tank

Casino Gets Hacked Through Its Internet-Connected Fish Tank Thermometer

Nicole Eagan, the CEO of cybersecurity company Darktrace, told attendees at an event in London on Thursday how cybercriminals hacked an unnamed casino through its Internet-connected thermometer in an aquarium in the lobby of the casino.

According to what Eagan claimed, the hackers exploited a vulnerability in the thermostat to get a foothold in the network. Once there, they managed to access the high-roller database of gamblers and “then pulled it back across the network, out the thermostat, and up to the cloud.”
(Via Hacker News)

I didn’t get a chance to write about this when it came out, but it’s dissemination came at an opportune moment. About 1 hour earlier I was using the Target breach as an example of third-party risks.

This story made an excellent follow-up.

Also on: