TSMC: Outbreak of Malware That Triggered Delays & Losses Caused by Software for New Tool
by Anton Shilov on August 9, 2018 3:00 PM EST- Posted in
- Semiconductors
- 16nm
- 10nm
- TSMC
- 7nm
TSMC announced this week that it suffered a computer malware outbreak, resulting in a roughly 3 day outage for parts of the fab while systems were restored. As a consequence of the downtime, the fab expects certain shipments delays and additional charges. Specifically, because of the interruptions and costs, the company’s Q3 revenue and gross margin will be 2% and 1% lower than anticipated respectively. TSMC later clarified that the outbreak was caused by “misoperation” during the software installation for a new piece of equipment.
What Happened?
TSMC’s personnel set up a new manufacturing tool on Friday, August 3, and then installed software for the device. The machine was not isolated and confirmed to be malware-free before connecting it to TSMC’s internal network. Consequently, the introduction of a malware-infected machine to TSMC's internal production network allowed the malware to quickly spread and infect computers, production equipment, and automated materials handling systems across TSMC’s fabs.
According to the chipmaker, the malware was a variant of the WannaCry ransomware cryptoworm. WannaCry, though over a year old at this point, still has the ability to propogate among any remaining unpatched systems, which is what happened here: the malware infected Windows 7-based machines “without patched software for their tool automation interface.” As a consequence, the affected equipment either crashed, or rebooted continuously, essentially being inoperable.
TSMC has been stressing that not all of its tools and automated materials handling systems were affected, and that degree of infection varied by fab. The company had to shut down infected equipment and apply patches. By 2 PM Taiwan time on Monday, 80% of the impacted tools had been recovered and TSMC said that it would mend all of them by Tuesday.
The Impact
Since the said tools are located across multiple fabs and are therefore are used to process wafers using a variety of process technologies for different customers, it is evident that the outbreak affected delivery schedules for many chips. As a consequence, the company had to notify its customers and reschedule their wafer delivery dates. Some of the delayed wafers will be delivered not on Q3, but in Q4, thus affecting product launch plans.
None of TSMC's well-known customers are currently commenting on the matter, but this event has occured with what's widely believed to be the ramp-up periods for new chips from Apple and NVIDIA. Since at least some of TSMC’s production tools were offline for four to five days, it is evident there will be impact, though it is hard to estimate how significant it will be.
What remains to be seen is how several-day outage of numerous semiconductor production tools is set to affect TSMC’s customers in general. After all, 2% of TSMC’s Q3 revenue is between $169 and $171 million and that is a lot of money. We will likely learn more about the effect of the malware outbreak in the coming months.
(ed: As an aside, I find it very interesting that this entire episode was essentially happenstance, rather than some kind of targeted attack as would typically be the case. WannaCry is over a year old and is self-propagating; so as a proper worm, it goes wherever it can, whenever it can. In fact with the release of patches over a year ago, WannaCry's primary function is done. So for TSMC this is the IT equivalent of stepping on a landmine from a long-forgotten war, and reinforcing the fact that advanced malware can be dangerous to the public long after it has done its job. -Ryan)
Related Reading:
Facebook
Twitter
RSS
41 Comments
View All Comments
ironargonaut - Friday, August 10, 2018 - link
I think worms should be classified legally the same as arson. Carelessly or intentionally lighting fires carries a huge penalty almost everywhere because of how easily fires can turn catastrophic. Rome burning, Chicago burning are examples from history. In this case it was TSMC burning a 100Million, added to the already burnt millions or billions from the main spread. If your worm infects a hospital and leads to a death you should be charged with murder by arson. Legal precedents are already set.eva02langley - Friday, August 10, 2018 - link
Well, once again, a third party software installed on a closed network infected the client IT infrastructure.That means the contractor was having the malaware and infected the client (TSMC). This is the typical scenario of cyber security.
The problem is that some closed network needs to have custom updates. Also, each of them needs to be checked before release on the network in case some software crash.
TSMC might have patch their systems, however maybe one of the patch that was incompatible with their network was not applied.
It is way more complicated than the simple "OMG TEH UPDATES FAIL!".
rocky12345 - Friday, August 10, 2018 - link
"the fab expects certain shipments delays and additional charges"So basically it was their fudge up by not properly checking the machine or if they did the tech that did check it most likely never actually checked it. So they are going to pass the costs off to their clients with additional charges. How is it their clients fault or problem that they fracked things things up. At this point if I was them I would be more worried about being sued or losing clients than being worried about trying to recoup their money from what it cost to do the clean up and get back up and running. Why should their clients pay for the clean up basically.
They screwed up so get the crap fixed so you can do business and keep your clients happy do not try to piss them off even more by over charging them.
I would also like to point out the only time Wannacry was known to crash systems was if it tried to install on Windows XP so are they admitting that they have a lot of Windows XP machines in their stables. If on Windows 7 it will run in the back round for a while as it locks your files down but it never touches windows itself bacically because if it was to actually crash your system to the point of non usable state how would it be able to put up that sheet on your screen telling you that you are pretty much fracked and of you want your family photo's and everything else back then pay the price or else you are hooped. Are they sure they actually had WannaCry and not something else. I have dealt with a lot of WannaCry infected machines and non of them ever crashed except XP machines.
fuji_T - Friday, August 10, 2018 - link
I think...if the answer to the problem was patching fab tools, they probably would have done it.Furthermore, updating a tool's software from Windows XP to Windows 7 generally isn't just insert the USB key, press F2 for the boot menu and boot from said USB key. A lot of the hardware and software is very closely tied together and upgrading to a new OS, depending on the vendor support would be very expensive or even impossible. Not every tool in the fab is going to be brand new with all the bells and whistles.
So, I'm not saying that TSMC is in the clear, but please do try and have some grace when making assumptions about what they can or cannot do.
As far as the "additional charges," there are a lot of charges beyond just customer loss like wafer scrap, etc. I'm being purposely vague with this post, but think of what can happen when manufacturing gets interrupted.
rocky12345 - Friday, August 10, 2018 - link
I was saying that the way they made it sound was it was all Windows 7 machines that got infected with this virus. I then went on to say that for the most part if a Windows 7 machine gets infected with this virus yes it will cause slow downs but that is because all of the user data is getting locked so the user will not be able to open those files without paying a fee. It would not be in the WannaCry's best interest to just completely make the system go into a total non working mode as in not being able to boot up at all because if they did how would they get their nasty little ransom demand posted all over the users screen.Yes there may be times that the system might just crash but that would have more to do with the hardware config and the software that is installed on those machines that totally crash. For Windows XP it will just crash if the virus tries to install or run not because it is a more secure OS but because it is such a old OS that it does not have what is required to let the virus do a auto install. The weird thing is the virus can actually be installed manually on XP and it will install then lock your files and then do the ransom demands.
As for TSMC maybe going to recoup their losses by charging their clients extra money again I say how is it TSMC's customer problem or fault that they dropped the ball here. Like I said before they should be more worried about keeping the clients happy because of delays and lack of product made and not worry about recouping the money spent on the clean up or at least hide it in the price sheet on future products and deals made.
iwod - Friday, August 10, 2018 - link
I often wonder which one is worst in an enterprise environment, macOS or Windows from a security and lockdown perspective.rocky12345 - Friday, August 10, 2018 - link
That's actually a very good question if there is anyone that is involved in that sector hopefully they can answer that question.JBrickley - Friday, August 10, 2018 - link
Well IBM seems to think macOS is far better. They have deployed 150,000+ and counting employee computers with Mac's. Saving them hundreds per Mac in licensing and support costs. They can buy the Mac's from Apple under their DEP system so they are zero touch. They ship the Mac still shrink wrapped to the employee straight from the factory. The employee opens the box connects power and connects it to the network (even Internet at home) and it phones home to Apple who due to DEP looks at the serial number and says, this is an IBM Mac so it redirects it to the IBM JAMF Pro servers which then enroll the Mac with the MDM. Then all the policies and configuration profiles are applied and software installed. The user starts seeing information displayed about the Mac@IBM program while they wait. It then pops up an IBM App store where they can install Microsoft Office, developer tools, Lotus Notes, etc., etc. The Mac's are encrypted and the keys escrowed into JAMF. The Self Service app provides all sorts of handy apps and scripts to fix stuff on your own. If you have to call the help desk they can remotely manage the Mac. This is worlds better than anything Microsoft is doing with Windows. The users rarely need to call for help and everything is heavily automated. The Macs are checking into the MDM on the corporate LAN and on the Internet and if the user does something they are not supposed to like enable something IBM wants disabled, it will either completely prevent the user from doing so or it will disable it when the Mac re-connects to the MDM on a regular check-in cycle. The Mac's also last longer than the PCs. On Mac's most things can be locked down with Configuration Profiles. The rest can be scripted. Apple keeps adding to the Config Profiles every year. Apple's new T2 64bit ARMv8 co-processor controls SSD encryption, provides a secure enclave, and supports Secure Boot. So you can lock them down so they cannot boot from USB and the boot cannot be infected by malware. This brings it much closer to being like an iPhone or iPad with hardware level security. The future will only tighten this security.All that is great but I don't see Apple macOS being used with manufacturing tooling and custom machines. That's a space ideal for Linux if there were easier developer tools and APIs. The reason Windows is used is because there are more developers who can code for it. Modern systems would use Win10 and C# applications to run the machines whereas old machines were WinXP / Win7 and VisualBasic / C#.
JBrickley - Friday, August 10, 2018 - link
Dig through the JAMF YouTube channel for JAMFNation Conferences, there a few IBM presentations talking about how they leveraged JAMF Pro to manage their Macs. Most of Silicon Valley is using Macs because they are building Linux based Cloud solutions and the Mac is Unix under the hood and plays very well in that space. There's a lot of different ways to manage them besides JAMF. Such as Chef, Puppet, alternative MDM's like SimpleMDM, Munki, etc., etc.Microsoft is becoming much more cloud developer friendly as of late because they see the threat that Apple and Cloud presents to Microsoft. So they are playing along with SQL Server being ported, better support in everything for Cloud tech. SSH/SSHd in beta for Win10, and the Linux Subsystem for Win10. Those last two go a long way to bringing developers to Win10 instead of Mac but it's still not there 100% yet. But the days of PC vs Mac are pretty much over.
JBrickley - Friday, August 10, 2018 - link
The problem is all the manufacturing tooling machinery that relies on old versions of Windows that haven't been patched against vulnerabilities. So whatever vendor provided the new tooling introduced a WannaCry variant into the internal production line network which then unleashed the worm across all the vulnerable machines and shutdown production. This is absolutely insane! These machines should not be running old vulnerable Windows operating systems, they should probably be embedded Linux and they should be patched. But patching these vulnerable Windows systems would probably break the tool just as bad as malware. It's really horrific how these very expensive machines are controlled by such god awful software running on ancient versions of Windows. Yes, it gets the job done but at what cost? So they hired programmers who could only deal with Windows instead of something a lot more rock solid. So sad... Really...