They do have backups but consider that a fab like GF's Fab 8 needs 80MW of power and dual 150KV lines. That's a lot of UPSes, flywheels, and diesel generators. And these aren't there to take a full power failure but rather to smooth transients. I expect Samsung is in the same ballpark and they have to weigh the risk of losing power against the certainty of the cost such a system would incur (installation/maintenance).
I was thinking the same thing. Deploying a sub-plant that's just enough juice to run a site for a few minutes wouldn't be prohibitively expensive when you consider the amount of lost revenue that results. Rarely are the power outages more than a few seconds, though the last major power outage at Sandisk\Toshiba was like 18 minutes which is a clear sign something major happened.
I'm sure they ran the numbers but I bet they did expect to have two (this second is tbd atm) with in a year. If they had expected this frequency it would have shifted the number for the execs to beef up their back up power. Also idk how it so bad in the first place. They at the very least have to have (unless there crazy) at least 15 seconds minimum of temporary instantaneous backup power. And at the point back up generators are not the expensive, like in the tens of millions for a billion dollar plant. Imo seems like a no brain to me and one most data center use. If google has a data center that need dual 150KV lines you can be sure they have a least one back up 150kv line.
Yeah I'm not sure there is any technology that could realistically provide that much power with no warning; even on-site lithium battery banks would probably take a few seconds to fully activate, and by then the wafers are destroyed. Pumped water, flywheels, combustion generators all would take long enough that it's not worth it.
The other thing people aren't thinking about is that Samsung has insurance for this; they get reimbursed for those destroyed wafers so it's not a total loss. The math to determine if the investment is worth it will have to factor in 1) cost of the redundancy equipment 2) whatever insurance premiums they would save (they cannot just cancel that coverage in case the system does not activate) vs 1) how much they pay now 2) what is the possibility of two incidents in a year exceeding their aggregate insurance cover and then the dollar amount being exposed to an uninsured loss.
I can guarantee you that Samsung has done this analysis and considered their options. The most economical choice is to not invest in a ton of electrical redundancy equipment that may or may not actually save anything, and instead put that money towards insurance coverage. Also remember just how many fabs they have - their insurance covers all of the fabs, while backup power would need to be installed at each location. The cost of equipment quickly goes through the roof.
"The other thing people aren't thinking about is that Samsung has insurance for this"
that's funny!! if they were a 'normal home owner' they'd get JUST ONE claim (at any location, not just this one), henceforth not one insurer would write them for any event at any location.
UPS and generators would be advisable for a site like that. Most places I've worked have had that much. It's a wonder why Samsung, of all companies, would not have that.
We have servers here that can last a few days as they have a huge gas tank, as big as a normal gast stations. They provide lot of CDN services so outage is not an option. Tesla also made a 100mW system for Aus, they will probably look into an LG one soon!
I would love to see a calculation/visualization of any commercially available power backup equipment capable of powering an entire modern fab for more than a few minutes.
"So the necessary 50 MW+ UPS is not really viable." lol where are you getting that from. Shocker you get a lot of "smaller" but still very "ups" systems going for the few minutes it takes to spin up the generators which they should have standing by on site... sounds like they don't though.
The function of UPS is to provide "very short term" power for dealing with spikes or emergency shut downs when power is out. So you are thinking UPS and backup generators, which all fabs have. However, fabs of this size suck power like a small nation and even backup power for emergency "short term" relief, in addition to "very short term" relief of UPS would already be very expensive. No company would invest in full time backup. The first thing in choosing fab location is power supply. Lastly, the damage to fabs is that its yield depends on continuous production and any shutdown, even planned, will be problematic. The losses are calculated in both WIP wafers as well as loss in yield curve.
You don't UPS an entire fab. As was stated, a modern fab facility uses as much juice as a small nation. In our fabs, we apply UPS solutions as needed for critical hardware that simply CAN'T go down. It would be incredibly expensive to find a battery that runs in excess of 100MW sustained output.
Because chip fabs use hundreds of Megawatts of power in operation. If you decide only to backup 'critical equipment', the list of critical equipment is 'all of it'.
They could do that. The Tesla battery pack in Australia is 100 MW and cost $66 mil and like a $4-5 mil cost to operate each year. It's got a 10 year lifecycle. I'm not sure if that'd be considered a good or bad investment. I guess depends how much money they lose each time they hit a power outage.
They had a 30 min power loss at their Pyeontaek fab and that cost them $43 mil. Today though, they have a healthy supply of chips, so probably going to be a smaller loss. Maybe a smaller battery backup for critical systems would be a cheaper and wiser investment. I don't know.
Like 50-60 MW to run a fab. The point isn't to keep the fab running, but give enough time to finish the task or reach a safe stopping point, that way you can stop production without killing any machines. Not sure how long 100 MW of power from batteries would last, but they can also tie that into twelve 4 MW Cat power generators. I guess they could ramp that up more and just run the fab on power generators, until normal power is restored.
Or tie the battery system to the grid with multiple redundant links, so that one substation going down won't take down their power.
Power isn't the issue. Cost to setup the backup power, generators, and maintenance is the issue. Whether it's all worth it or not.
You guys are aware that Samsung is free to produce as much DRAM and NAND as they want, right? If they wanted to cut their output they could do so in a less costly and disruptive way than to stage a power outage. It's not shady or illegal for Samsung to cut their output. If the power output were staged then it would be done by someone manipulating stock prices for personal gain.
But in fact, it's almost certainly just a power outage. These plants need tremendous amounts of power and no one's lives are at stake. It's not like a hospital. A backup power plan is implemented on a cost/benefit analysis. Sure it is possible for them to have a complete backup plan, but if the cost of such a thing is more than the expectation of loses from some lesser plan than it is a bad decision to implement it. A 2 to 3 day shut down at one of their fabs is not going to bankrupt Samsung so there's no reason to bleed the company's operating margins to more fully insure against such an event.
Some people (thankfully not a majority by any stretch) must view the world through the lens of conspiracy and underhanded plots. If such critters didn't exist the market for survivalist bunkers and associated zombie apocalypse supplies would be significantly smaller. They are economic drivers and push share prices upward when they help grease the wheels if industry and commerce. Kindly stop trying to bring them around to accept reality while I am still looking to pull in dividends and ride a golden chariot to retirement upon their clueless backs.
... defined as the one immediately following the information age; Once everyone can access and disseminate any information they want, they have the thought in their head that their beliefs are just as valid as other people's knowledge, and that "just throwing it out there" isn't that far from proving it.
I would be quite surprised if there wasn't some kind of insurance in place. While I don't think Samsung would orchestrate this, I am quite confident they'll profit on this.
No, it's run by reason. And reasonable people have a decent idea of when statistics works, and when not. Huge risks that are rare enough to have poor data to feed statistics, and where the victim may game the system easily since they have more knowledge of the underlying risks than the insurer... I seriously doubt insurers are going to rely very heavily on what normal people might call statistics in those cases - if they even insure them at all this would have been a very human-guided risk analysis (obviously using whatever poor statistics they have, sure).
From experience I can tell you an insurance claim against a local utility is only going to recoup about 40-50% of the total losses unless an investigation can show gross negligence in maintaining their equipment, then maybe more like 60%
I worked at the Samsung Fab in Austin when it started up...we had power outage caused by a rattlesnake bridging transformers. Cost was horrific both in wafers lost, downtime because tools were "stuck" in a mode and engineers/techs had to figure out how to reset them, and blown electronics on the equipment.
Seems to me it would be easier, and cheaper, to put battery backup/UPS on each tool to allow shutdown in a safe manner...allow the tool to finish a wafer (chamber tool) or rinse chemicals off of wafers (wet tools). What I do not know is the reaction time...how fast does backup have to kick in to "save" a 7 nm device?
the sceptic in me sees a profit gouge coming up weird how it seems to affect the same type of product all the time whos prices have been in freefall for most of the last year
We only hear about power outages in the news if they have a large impact. Most industries can just resume where they left off if they lose power for 10 minutes.
So you have bias due to the very specific information that is filtered thru the media. (And I should note there is nothing wrong with only reporting relevant news.)
Didn't they say back in January that they want to increase prices of RAM and SSD's by up to 40% as people are compelled to upgrade their computers now that Windows 7 reached EOL?
neither the article nor the comments addresses the fundamental question: how much time and power are needed to 'soft land' the plant? IOW, what are the pause points in the myriad processes where the in-process material will preserve until re-started? do none exist?
We’ve updated our terms. By continuing to use the site and/or by logging into your account, you agree to the Site’s updated Terms of Use and Privacy Policy.
Facebook
Twitter
RSS
48 Comments
Back to Article
FSWKU - Thursday, January 2, 2020 - link
Time for memory prices to go up again, I guess...nandnandnand - Thursday, January 2, 2020 - link
They should have these power outages more often.Kamen Rider Blade - Thursday, January 2, 2020 - link
Makes you wonder why they don't have a site-wide UPS (Un-Interruptible Power Supply)Alistair - Thursday, January 2, 2020 - link
contact Tesla :)close - Friday, January 3, 2020 - link
They do have backups but consider that a fab like GF's Fab 8 needs 80MW of power and dual 150KV lines. That's a lot of UPSes, flywheels, and diesel generators. And these aren't there to take a full power failure but rather to smooth transients. I expect Samsung is in the same ballpark and they have to weigh the risk of losing power against the certainty of the cost such a system would incur (installation/maintenance).quiksilvr - Friday, January 3, 2020 - link
Considering this happens at least once every couple of years you would think the ROI would make sense to set up such a thing.Samus - Friday, January 3, 2020 - link
I was thinking the same thing. Deploying a sub-plant that's just enough juice to run a site for a few minutes wouldn't be prohibitively expensive when you consider the amount of lost revenue that results. Rarely are the power outages more than a few seconds, though the last major power outage at Sandisk\Toshiba was like 18 minutes which is a clear sign something major happened.Skeptical123 - Friday, January 3, 2020 - link
I'm sure they ran the numbers but I bet they did expect to have two (this second is tbd atm) with in a year. If they had expected this frequency it would have shifted the number for the execs to beef up their back up power. Also idk how it so bad in the first place. They at the very least have to have (unless there crazy) at least 15 seconds minimum of temporary instantaneous backup power. And at the point back up generators are not the expensive, like in the tens of millions for a billion dollar plant. Imo seems like a no brain to me and one most data center use. If google has a data center that need dual 150KV lines you can be sure they have a least one back up 150kv line.Kakti - Saturday, January 4, 2020 - link
Yeah I'm not sure there is any technology that could realistically provide that much power with no warning; even on-site lithium battery banks would probably take a few seconds to fully activate, and by then the wafers are destroyed. Pumped water, flywheels, combustion generators all would take long enough that it's not worth it.The other thing people aren't thinking about is that Samsung has insurance for this; they get reimbursed for those destroyed wafers so it's not a total loss. The math to determine if the investment is worth it will have to factor in 1) cost of the redundancy equipment 2) whatever insurance premiums they would save (they cannot just cancel that coverage in case the system does not activate) vs 1) how much they pay now 2) what is the possibility of two incidents in a year exceeding their aggregate insurance cover and then the dollar amount being exposed to an uninsured loss.
I can guarantee you that Samsung has done this analysis and considered their options. The most economical choice is to not invest in a ton of electrical redundancy equipment that may or may not actually save anything, and instead put that money towards insurance coverage. Also remember just how many fabs they have - their insurance covers all of the fabs, while backup power would need to be installed at each location. The cost of equipment quickly goes through the roof.
FunBunny2 - Sunday, January 5, 2020 - link
"The other thing people aren't thinking about is that Samsung has insurance for this"that's funny!! if they were a 'normal home owner' they'd get JUST ONE claim (at any location, not just this one), henceforth not one insurer would write them for any event at any location.
dgingeri - Thursday, January 2, 2020 - link
UPS and generators would be advisable for a site like that. Most places I've worked have had that much. It's a wonder why Samsung, of all companies, would not have that.menting - Thursday, January 2, 2020 - link
there are UPS systems in place for all these companies, but the UPS only provides protection from power spikes and power outages in the seconds range.Sahrin - Thursday, January 2, 2020 - link
I think the DoJ's antitrust regulators would find that there is in fact a working UPS system.Or rather, they would if we had a government that cared about investigating businesses for corruption.
svan1971 - Friday, January 3, 2020 - link
Perhaps the US governments corruption needs to be dealt with before worrying about businesses in Korea.Skeptical123 - Friday, January 3, 2020 - link
Ahh yess everything in the country has to be perfect before they comment on anyone else. Thats wise.............Byte - Friday, January 3, 2020 - link
We have servers here that can last a few days as they have a huge gas tank, as big as a normal gast stations. They provide lot of CDN services so outage is not an option. Tesla also made a 100mW system for Aus, they will probably look into an LG one soon!Skeptical123 - Friday, January 3, 2020 - link
lol ya, this is the industry norm. Idk what so hard about this for people to get.Ryan Smith - Thursday, January 2, 2020 - link
Bear in mind that it's industrial fabrication, which requires a lot of power. So the necessary 50 MW+ UPS is not really viable.Cullinaire - Thursday, January 2, 2020 - link
I would love to see a calculation/visualization of any commercially available power backup equipment capable of powering an entire modern fab for more than a few minutes.Mccaula718 - Thursday, January 2, 2020 - link
UPS is only expected for minutes while waiting for generators to start up. That visualization would be interesting thoughSkeptical123 - Friday, January 3, 2020 - link
"So the necessary 50 MW+ UPS is not really viable." lol where are you getting that from. Shocker you get a lot of "smaller" but still very "ups" systems going for the few minutes it takes to spin up the generators which they should have standing by on site... sounds like they don't though.wr3zzz - Thursday, January 2, 2020 - link
The function of UPS is to provide "very short term" power for dealing with spikes or emergency shut downs when power is out. So you are thinking UPS and backup generators, which all fabs have. However, fabs of this size suck power like a small nation and even backup power for emergency "short term" relief, in addition to "very short term" relief of UPS would already be very expensive. No company would invest in full time backup. The first thing in choosing fab location is power supply. Lastly, the damage to fabs is that its yield depends on continuous production and any shutdown, even planned, will be problematic. The losses are calculated in both WIP wafers as well as loss in yield curve.JKflipflop98 - Thursday, January 2, 2020 - link
You don't UPS an entire fab. As was stated, a modern fab facility uses as much juice as a small nation. In our fabs, we apply UPS solutions as needed for critical hardware that simply CAN'T go down. It would be incredibly expensive to find a battery that runs in excess of 100MW sustained output.edzieba - Friday, January 3, 2020 - link
Because chip fabs use hundreds of Megawatts of power in operation. If you decide only to backup 'critical equipment', the list of critical equipment is 'all of it'.PeachNCream - Thursday, January 2, 2020 - link
Someone needs to contact Elon Musk to obtain one of his "visionary" power wall kits.khanikun - Thursday, January 2, 2020 - link
They could do that. The Tesla battery pack in Australia is 100 MW and cost $66 mil and like a $4-5 mil cost to operate each year. It's got a 10 year lifecycle. I'm not sure if that'd be considered a good or bad investment. I guess depends how much money they lose each time they hit a power outage.They had a 30 min power loss at their Pyeontaek fab and that cost them $43 mil. Today though, they have a healthy supply of chips, so probably going to be a smaller loss. Maybe a smaller battery backup for critical systems would be a cheaper and wiser investment. I don't know.
jordanclock - Friday, January 3, 2020 - link
I don't think you understand how much power it takes to run a fab.khanikun - Sunday, January 5, 2020 - link
Like 50-60 MW to run a fab. The point isn't to keep the fab running, but give enough time to finish the task or reach a safe stopping point, that way you can stop production without killing any machines. Not sure how long 100 MW of power from batteries would last, but they can also tie that into twelve 4 MW Cat power generators. I guess they could ramp that up more and just run the fab on power generators, until normal power is restored.Or tie the battery system to the grid with multiple redundant links, so that one substation going down won't take down their power.
Power isn't the issue. Cost to setup the backup power, generators, and maintenance is the issue. Whether it's all worth it or not.
Yojimbo - Friday, January 3, 2020 - link
You guys are aware that Samsung is free to produce as much DRAM and NAND as they want, right? If they wanted to cut their output they could do so in a less costly and disruptive way than to stage a power outage. It's not shady or illegal for Samsung to cut their output. If the power output were staged then it would be done by someone manipulating stock prices for personal gain.But in fact, it's almost certainly just a power outage. These plants need tremendous amounts of power and no one's lives are at stake. It's not like a hospital. A backup power plan is implemented on a cost/benefit analysis. Sure it is possible for them to have a complete backup plan, but if the cost of such a thing is more than the expectation of loses from some lesser plan than it is a bad decision to implement it. A 2 to 3 day shut down at one of their fabs is not going to bankrupt Samsung so there's no reason to bleed the company's operating margins to more fully insure against such an event.
Yojimbo - Friday, January 3, 2020 - link
BTW, by "you guys" I mean the various commenters who are suggesting Samsung staged a power outage for some reason, not the author of the article.PeachNCream - Friday, January 3, 2020 - link
Some people (thankfully not a majority by any stretch) must view the world through the lens of conspiracy and underhanded plots. If such critters didn't exist the market for survivalist bunkers and associated zombie apocalypse supplies would be significantly smaller. They are economic drivers and push share prices upward when they help grease the wheels if industry and commerce. Kindly stop trying to bring them around to accept reality while I am still looking to pull in dividends and ride a golden chariot to retirement upon their clueless backs.Hul8 - Friday, January 3, 2020 - link
Well we *are* living at the age of stupid.Hul8 - Friday, January 3, 2020 - link
... defined as the one immediately following the information age; Once everyone can access and disseminate any information they want, they have the thought in their head that their beliefs are just as valid as other people's knowledge, and that "just throwing it out there" isn't that far from proving it.HollyDOL - Friday, January 3, 2020 - link
I would be quite surprised if there wasn't some kind of insurance in place. While I don't think Samsung would orchestrate this, I am quite confident they'll profit on this.Speedfriend - Friday, January 3, 2020 - link
I think you will find that insurance contracts are set up so that they can't profit on this. Or do you think the insurance industry is run by idiots?HollyDOL - Friday, January 3, 2020 - link
insurance industry is run by statisticsemn13 - Friday, January 3, 2020 - link
No, it's run by reason. And reasonable people have a decent idea of when statistics works, and when not. Huge risks that are rare enough to have poor data to feed statistics, and where the victim may game the system easily since they have more knowledge of the underlying risks than the insurer... I seriously doubt insurers are going to rely very heavily on what normal people might call statistics in those cases - if they even insure them at all this would have been a very human-guided risk analysis (obviously using whatever poor statistics they have, sure).HollyDOL - Friday, January 3, 2020 - link
I am not saying they will profit more on the insurance pay, but insurance pay plus already expected NAND price rise... they won't come short.FullmetalTitan - Friday, January 3, 2020 - link
From experience I can tell you an insurance claim against a local utility is only going to recoup about 40-50% of the total losses unless an investigation can show gross negligence in maintaining their equipment, then maybe more like 60%hanselltc - Friday, January 3, 2020 - link
An outage a year keeps the price drop away I guesschris994 - Friday, January 3, 2020 - link
I worked at the Samsung Fab in Austin when it started up...we had power outage caused by a rattlesnake bridging transformers. Cost was horrific both in wafers lost, downtime because tools were "stuck" in a mode and engineers/techs had to figure out how to reset them, and blown electronics on the equipment.Seems to me it would be easier, and cheaper, to put battery backup/UPS on each tool to allow shutdown in a safe manner...allow the tool to finish a wafer (chamber tool) or rinse chemicals off of wafers (wet tools). What I do not know is the reaction time...how fast does backup have to kick in to "save" a 7 nm device?
FullmetalTitan - Friday, January 3, 2020 - link
Less than 1-2 cycles of the AC phasealufan - Friday, January 3, 2020 - link
the sceptic in me sees a profit gouge coming up weird how it seems to affect the same type of product all the time whos prices have been in freefall for most of the last yearHul8 - Friday, January 3, 2020 - link
We only hear about power outages in the news if they have a large impact. Most industries can just resume where they left off if they lose power for 10 minutes.So you have bias due to the very specific information that is filtered thru the media. (And I should note there is nothing wrong with only reporting relevant news.)
Gastec - Saturday, February 8, 2020 - link
Didn't they say back in January that they want to increase prices of RAM and SSD's by up to 40% as people are compelled to upgrade their computers now that Windows 7 reached EOL?FunBunny2 - Friday, January 3, 2020 - link
neither the article nor the comments addresses the fundamental question: how much time and power are needed to 'soft land' the plant? IOW, what are the pause points in the myriad processes where the in-process material will preserve until re-started? do none exist?Gastec - Saturday, February 8, 2020 - link
1 minute of power outage @ Samsung factory and now they all have the excuse to increase prices by 40%Gastec - Saturday, February 8, 2020 - link
Because of..."massive financial losses"