"unexpected power outage in the Yokkaichi province" These fab are some of the most sophisticated and costly factories in the world. A simple loss of grid power is by far one of their easiest issues to mitigate and is done so with if possible multiple power pant connections, batteries, and diesel generators. This is standard practice at facilities that cost a fraction of these chips fabs. As such regardless of the reason heads will roll
"easiest issues to mitigate and is done so with if possible multiple power pant connections, batteries, and diesel generators."
does "too big to fail" strike a familiar note? just because a company could build out sufficient back up power supply for a campus doesn't mean it's smart to do so. balanced against the cost of the back up supply is the expected occurrence of demand. IOW, if such a power loss was/is believed to be a once in a decade or more event, then the loss of product spread over that decade may be far less than the amortized cost of the back up supply. the engineers and bean counters decided that cost exceeded the probability. what matters is the actual cause, and why the event occurred. it may be that the event was within the designed probability envelope, and the companies are OK with the outcome.
OMG the siliness in this thread is out of control. Dude power loss is NOT a once in a decade event in japan. This is clearly a design failure. Dude stop pretending these fabs drew more power than half the country. Dude aircraft carriers have onboard power. Some clown actually said "there may not have been enough space in japan" these fabs cost more than aircraft carrier to build. are u seriously saying they didnt think about power outage when designing this? if thats the case every semiconductor should have multiple fabs out every year.
My brain is about boil at the sheer number of ppl and magnitude of stupidity in this thread. I am truly amazed at the number of ppl who can be this far out in cartoonland. I was wondering how can a whole country be stupid enough to fall for something like a bolshevik revolution. now i know.
It's ironic that someone who cannot consistently operate the shift key or type the word "people" in order to communicate effectively is losing it over the supposed stupidity of others. That's a lovely glass house you're living in over there, dude.
Dude, the issue is whether the cost/benefit analysis of a back up supply robust enough to save the campus *of this magnitude* (modified by the expectation of such an event) wasn't done or was ignored. As reported, there was a back up supply, just not one robust enough to last 13 minutes. You may not remember the Great Northeast Blackout, but I lived through it. Should every town have had a backup power plant? Holyoke did, in that the municipal plant was off-grid, so wasn't taken out in the cascade. Good planning or good luck?
This likely has nothing to do with grid power. See that small building to the Northeast. That's a power distribution facility. The problem occurred somewhere between there and the campus buildings, because billion dollar sites like this produce their own backup power in real time.
This likely has nothing to do with technical reasons at all.
"Western Digital Corp. shares logged their best weekly gain in nearly seven years Friday as investors considered a power outage at one of the digital storage device maker's flash memory production facilities to be good news given the glut in memory chip inventories."
Of course there is. You aren't talking gigawatts of power. You are talking at most several dozen megawatts. I've seen the backup power supplies for many industrial operations that draw far more power than a fab. You are talking a backup power solution that likely at most runs in to the low hundred million range for what is several billion dollars of equipment. Even ignoring the cost of the equipment that might be damaged, 6 exabytes of NAND production is in the ballpark of $150-200 million dollars worth of NAND down the drain (plus Toshiba's losses, which if the article is correct in speculation, is likely another $200-400 million in losses).
$66 million installation cost for 100mw of battery power, 300mwh storage (so 3 HOURS of backup power). Yes, the Tesla project was done well after whatever backup power system the fab complex has would have been installed. The point is, for what is clearly higher output than what the fab would need (again, the power requirements for the entire fab complex are likely to be in the 10-40mw range), it is drop in the bucket compared to the losses they suffered.
I am sure that the fab complex DOES have backup power sufficient to the task. What I am also pretty sure of is it isn't redundant (which it should be) and also very likely wasn't tested sufficiently to catch problems (as that would likely require shutting down the fab temporarily, maybe once a year, to test the system live). Most industries that have/rely on backup power do an annual plugs out test.
Looks like maybe WD/Toshiba didn't adhere to industry standards there.
Actually there is, you only need a few seconds of UPS service till the back up generators of such factories kick in. You usually have several protection levels for this stuff. For example it is not uncommon for such large facillities to have a complete discrete power plant or to be connected to more than one power plant/grid at a time in addition to UPS and generators.
Did you never wonder why industrial hardware has like 2-4 power connections, that is not just a redundant PSU, it enables you to have the hardware on just as many discrete UPS and powergrids to mitigate the likelyhood of failure.
I remember seeing the new some time ago that some flash manufacturers want to reduce their production because or prior over production and lots of flash still in stock at warehouses. This is 13min failure that could have been prevented, but instead somehow causes 3months of issues. This sounds like a way to avoid issues with markt price manipulation, but artificialy reducing the available amount of flash.
This should under no scenario have caused an issue similar to the flood back then in south east asia.
If the power goes out you do not suddenly start pumping polluted air into the clean rooms and making stuff unusuable for months.
Your use of the word dude is a clear indication about your knowledge of power. Aircraft carriers use direct nuclear power for one, so I'm not sure why you included that? And thinking they didn't have a power backup is another indication you have no idea how things work in a fab. Just the fact you have no clue how much power is used or how it's produced is another indication. Or the fact a backup can't produce enough power if you are right in the middle of production.
I'm sure they'll be contacting you soon to revamp power distribution sense you know so much about it.
A high end data-center can run 30kW racks. Why can a data center plan and economically execute full time backup power with that energy density but a fab can not?
Gunbuster is correct with his assumption. I worked at Argonne National Laboratory through college in the APS (Advanced Photon Source) building - which is a large synchrotron particle accelerator. It had 32, and recently expanded to 36 sectors. It's actually larger than Fermi lab. But nowhere near as large as the large haldron collider.
But collisions it does. The Argonne campus has its own power plant. So do fabs. As anybody who knows anything about power generations knows, power plants produce at capacity in real time, and this capacity is always beyond grid demand to account for spikes or sudden demand surges where power cannot be diverted elsewhere from the grid. This makes them inherently inefficient because the excess power can't be stored, so it is wasted in the form of simply not generating power that could be generated using the supplied pressure\heat\whatever type of fuel is being used. This is especially true with nuclear power production (although no sites like these use nuclear power production on-site, that is overkill) as it takes weeks, and even months with old designs, to roll back fuel processing and scale down power production. During the cooldown, they are super inefficient in the sense they COULD be making a lot more power for the same amount of fuel spent.
ANYWAY. This site clearly has a oil fuel power plant on site. I can spot it on the low-res campus photo in the top right. This plant was built to supply more than enough power to the site EVEN IF grid power seized. But built for, and producing, are two different things.
Something happened at the power processing facility, or something malfunctioned causing a blackout (which seems to be the case considering the length of time there was a power outage...they could divert grid power almost immediately like Samsung did during their recent production woes that lasted in the milliseconds.)
That said, this is hard to wrap my head around. These mistakes just don't happen. But mistakes can be manufactured. And mistakes that take down an entire campus of electricity for minutes almost has to be manufactured...even natural gas\diesel generators warm up and produce power in 20 seconds...
@Gunbuster - A high end data-center can run 30kW racks
Problem is in this case we're talking 2 orders of magnitude higher. In the tens of MW range, maybe even close to 100MW (GloFo Fab 8 needs 80MW *on average* as a reference). And supplying that from backup constantly until the power is restored in not something most industries ever have to deal with so people should stop using their UPS as a reference for anything. There's no real way to save any power during such incidents, there's no "power saving mode", no shutting down part of the line. The backup power has to provide exactly what the main did. Usually you manage to supply enough power to shut down the lines with no damage to the equipment but all the silicon in the pipeline just became trash anyway.
This happened to Samsung, it happened to Toshiba/WD. It's just AT commenters that seem to have the wrong career. AT became home to all of the "I'm an expert dude" type people.
And most data centers don't cost a billion to stand up. You scale up to provide services. Plenty of 10MW gensets to choose from. Sure they are the size of a semi trailer but that's a huge building. Line up 15 of them.
Clearly you have no idea how semiconductor manufacturing works. 13 minutes is an INSANELY long outage. Samsung had at 200ms (yeah, 0.2s) outage in 2017 that cost something like $50M in product losses.
Losing power means they lost: All RF etch tools - any wafers in those tools are scrap All fab exhaust/pumps/cooling water - any tools with sensitive environments scrap 100% All power to furnaces - temperature varied from spec, 100% scrap
What happened? Has SSD price fallen that quickly over past month and so "correction" is needed? Or was there a need to collect some insurance money? 13 minutes outage in *Japan* is from sci-fi category...
Yep, they needed a price correction. Planned out carefully, insurance will pay out fantastically and they'll essentially make a profit on all this (and even more profit on everything else sold for many years after) as prices shoot up. I think this lesson was learned from the HDD flooding a few years back. Obviously that was a real natural accident, this is likely the best possible "accident" they could come up with to simulate the effect (and with very precise control).
I'd love to see a documentary on how they did it. It couldn't have been easy to cause such a cascade failure in all the parts of such a complex setup. Wonder how they compensated some engineers from the power company (or the company itself, though that seems more a China thing).
I don't need a tinfoil hat for this one, I stay in Japan 3 months at a time...those people are not hapless idiots.
Of course not, not like they'd place diesel backup generators on coast where they know a tsunami will hit one day. Or the nuclear power stations that rely on the diesel generators working to not meltdown.
Nope, that would be the behavior of hapless idiots!
After reading the Micro article earlier, when they said production was high and demand low, my cynical side said, wait for the disaster to hit. A few hours later - this article shows up. Let's face it, once or twice is a coincidence, but this sort of thing happens far too often when supply of product X is exceeding demand of product X for it to be always just a coincidence.
"Western Digital says that the 13-minute power outage...will reduce its NAND flash wafer supply in Q3 by approximately 6 EB (exabytes), which is believed to be about a half of the company’s quarterly supply of NAND." Talk about write amplification!
It does not have to maintain anything. It has to last until safe shutdown of processing facilities. Everything is turned off at first moment, only emergency lights, elevators, air purification and actual machines that make chips. Of course there are acceptable looses as there are always some wafers in chambers being processed that can't be halted. But definitely something went wrong in this or someone without a shred of intelligence created the project that did not account for a long term power loss.
We are not talking about UPS there. A generator is not a stable source of power. It introduce a lot of noise. Also, silicon fabs use a huge amount of power in their process that no backup power can provide.
We don't know how this affect the would supply chain because we are not in the industry. Seeing smart idiots claiming they know better is just ridiculous.
Are you absolutely sure of that? How much power exactly does it take? ASML machines that actually make the product use pretty weak lasers, on the whole current best ones (strongest) use 250W. You don't want to evaporate metals there, you want to, so to speak, shine some light at the correct frequency - it's not a brute force approach but a finesse one. I'm sure much more power is used for filtering atmosphere, temperature control or simple lights then it is on actual lithography. The generator may not be stable power source but power lines are not stable too, you really think the power is not filtered, stabilised and converted as many times as needed? Besides, battery backup they had is not stable either, no matter what type of batteries the used, voltage decreases a lot during discharge, they must have a lot of stabilization going on, you think that would not be capable of stabilizing generators? The only explanation is they did not turn the generators on counting on power being restored or the whole "accident" is a scam created for unsold inventory, insurance claim and future prices increase. With only one "stroke" you get a lot of benefits.
Generators are not stable power sources? Well I am of to all power plants around the world telling them your conclusion that whatever they use is not stable and that we should replace it with solar panels even if it needs 50-100 the area of a nuclear plant to produce the same amount of energy under good conditions.
Battery- and generator-power can only support the security and absolutely essential systems for more than a few minutes. These fabs draw hundreds of megawatt a piece, and essentially any silicon that isn't fully finished is basically lost once the power goes out. The only way to be able to go off-grid for such fabs is an on-site power station, and that you'd need backups for that, because they also go down or need maintenance.
Are you absolutely certain that there are no points at which, in the hundreds steps process, production can not be halted? Does all the phases happen on one sealed chamber or does the wafers being processed travel between different chambers in the fab?
You don't understand. Even something simple like baking the wafers can't be interrupted. If the process takes 1 continuous hour, and you lost power for 13 minutes, then you lost everything. There's so many processes that you can't just shut down and start back up again in the middle.
Well, then if you are aware of that, then you create a solution that will deliver power for the required time. If not you are a moron, or you are counting on insurance to cover your ass.
Not him and late, but yes, absolutely certain. It's genuinely hard to overstate how insanely precise the timing and stability requirements of each step are when you're literally, LITERALLY talking "3 atoms is too thin, 5 atoms is too thick". The modern silicon fabrication process is genuinely one of humanity's greatest wonders. If you haven't seen it, a good little talk is "Indistinguishable From Magic": https://www.youtube.com/watch?v=NGFhc8R_uO4 Keep in mind that everything you here in there is now very old and obsolete tech too.
These facilities pay out the nose for power assurance from local utilities. They get the same guarantees as hospitals. They also have banks and banks of UPS systems, like thousands of batteries, and even then can only supply critical safety systems for more than a few minutes. The Samsung fab in Austin I live near has its own power substation in case you need a good indicator of how much power these facilities use. The Yokkaichi complex is about 3x that size
And how much time do you need to safely halt production? Or is that, that someone did not flip the switch hoping for power to be restored in moments and safety systems did not engage? Or these fabs really created in such a flawed way that simple power outage will destroy months of work even with all emergency systems and protocols in place and operational? Do they really only depend on insurance policy to cover looses?
You cannot halt production in silicon chips process. Besides, it takes days and weeks for a full process to complete. This Yokkaichi site probably uses 500-600 MWs - this power generation requires two large GE turbines /if gas electric power plant or a nuclear reactor is used :) :)/ plus reserve power of the same magnitude. Such turbines need several hours for start up time, and more than 24 hrs of reaching optimal condition, i.e. no power generators can help. Regardless, the memory industry is happy - accident will cause jump of prices to sustainable levels, memory prices went to down - everybody was on freeze since January - no capital expenditures, no hire, no budget increases, 25% furloughs etc. This will help for years to come.
Ok, it seems I was mistaken after all: http://energyskeptic.com/2014/interdependent-chip-... "Fab plants use up to 60 megawatts of power" Oh well, it is strange that so much power is required since the lasers lithography actually use are so weak (250W).
Better numbers than I found, thanks Mat9v. 60mw, is a lot. But it isn't an obscene amount. Ananke, as another commenter noted, many critical sites such as this produce their own power interfaced with the grid. It isn't terribly efficient, but it provides completely redundancy. That power is filtered so spike and sags are cleaned up, fluctuations in frequency, etc. If the campus power generation failed, it is grid tied and the fail over is to the grid, which can react in seconds (meanwhile the sagging voltage generated by throwing 60mw on to the local/regional grid during that time is filtered and supported by the on campus power filtering equipment, so the campus wouldn't see the voltage sag).
Or at least that SHOULD be how it is done. Obviously something went badly wrong, but you can see the power plant on the campus in the picture. They have one. They just either didn't have it online, or they were using it as backup and the grid as primary and whatever they had running for intermittent grid disconnection didn't support the campus power requirements until their power plant came up (IE bad design/no testing/no sufficient layers of backups).
yes i think thats why uss ronald reagon runs on tesla supplied batteries. oh wait.
cmon u can't be serious in asserting that a 13 minute power outage in japan is so rare or so expensive to deal with, that they just planned to live with it
Generators do not turn on instantly. Hell, even batteries do not start providing power instantly. Even if they are super speedy batteries which less than 1 second to start providing power that's long enough to wreck everything running. Super/ultracapacitors are required for nearly instant (<1 ms) emergency power delivery.
But these have low energy capacity, so my guess is that they might be employed to provide an "energy burst" for that first 0.5 to 1 sec, and then hand over the power delivery to the batteries. The batteries, in turn, can hand the power delivery to generators after a few minutes.
I have no idea if this a viable scheme though. Instead of employing a set of supercapacitors, batteries and generators of enormous size and mass just for emergency power, it might make more sense -and cost almost the same- to build a dedicated mini power plant on-site. Ideally it should require barely any maintenance and a skeleton staff, i.e something like a solar farm along with some other power source (like wind) for the solar downtime. It could be for emergency power and the rest of the time its power could be sold back to the grid - or as the main power source with the grid acting as backup power, depending on which option was more favorable.
Pretty sure I remember reading that new huge Tesla battery in Australia can kick on and provide power to the grid and stabilize voltage on the order of milliseconds. Like 50 milliseconds or less. So pretty sure these smart guys that designed such a fab would probably have something equivalent in place.
@Ej24 - That type of technology is relatively new TBH. Most battery control systems I've worked with have much slower control loops in the order of hundreds or thousands of ms due to their industrial background of most of the systems. Tesla however used repurposed vehicle control systems which have much faster control loops which is why Tesla has a moat there.
Oh please, "super speedy batteries"? I've got battery backup for my house and the lights don't even flicker when the power goes out. Heaven forbid a billion dollar facility can't do that.
"I've got battery backup for my house and the lights don't even flicker when the power goes out."
that's because, if you have an APC like mine, it's in line all the time, is connected to the PC, which means each and every machine in each building has to have a very scaled up APC. and each machine needs sufficient intelligence to be monitored by the APC controller to do that.
Hell 25+ years ago the large mainframe datacenter I worked in had no chance that batteries could sustain the processors, DASD, and more importantly, the chillers. A huge room full of batteries was just enough to keep it going for the THREE generators to fire up. Any 2 generators could sustain the load, the third was a backup. These were sort of medium size generators - the sort of diesels you might find in heavy construction equipment (slightly larger than semi truck types). Around the same time I also got to tour a major telecom interconnect site - they too had a huge room full of batteries to allow their generators to start - their generators used diesels the size of locomotive prime movers.
So the batteries did not provide them enough time to shut down anything "gracefully"? Why have batteries at all then, if they are good for nothing? Or did the batteries provide them enough power to shut down just the most critical systems, and the rest had to be left out?
Yes he is serious. Most companies don't have enough standby power for full operations, just for critical things. Considering this is a plant, critical is not full operations. I can tell for a fact that its easier to claim loss than worry about it. Case in point, a normal walmart does $100million a year..but only power emergency lights, servers for registers and cameras. Thats it. The back up power sits in a small closet. Freezers, coolers, etc are not powered at all.
Actually, most fabs cost somewhere in the ballpark of $3-5 billion. A typical first-world aircraft carrier of a modern navy costs $6-9 billion.
In comparison, the latest Gerald Ford US aircraft carrier is estimated to cost about $9 billion. While the Fab15 facility of TSMC was estimated to cost $9.3 billion.
Given that, perhaps they are more close to parity in terms of cost, rather than one being far costlier than the other.
As for backup power, it's probably not feasible for a fab of this size to have extensive battery/generator backups. Where as, some very modern aircraft carriers may have backup systems in place, but they are only meant for short periods and at minimal operating capacity (probably emergency lighting, basic navigation/systems, and perhaps some weaponry).
Backup systems for a mutli-facility fab like this probably would be very expensive to maintain and operate. Even if it did exist and work properly, I doubt it would hold out for 13 minutes. Most larger server facilities that have backup systems are designed to let the facility fail gracefully (basically enough time to save and power down, not run for hours on end until power is restored). Even, these larger server farms probably would still crash and fail if they ever had to rely on their backup power systems.
" Where as, some very modern aircraft carriers may have backup systems in place, but they are only meant for short periods and at minimal operating capacity"
considering that such boats are run by pressurized light water nucular reactors, how much of the boat still runs if that reactor goes legs up is the least of your problems. :)
US Aircraft carriers have two nuclear reactors, and a single reactor is sufficient to keep the ship moving fast enough to deploy and retrieve aircraft, although deployment is slower - perhaps only one launch at once, and perhaps a lower payload on the aircraft since the ship isn't able to move at full speed. But a stopped aircraft carrier is likely to become a sunk aircraft carrier.
A set of generators at my office stores two weeks of fuel for the diesel generators, and has contracts in place to deliver a full load of fuel on a day's notice. One of the two generators is sufficient to operate the facility, but one isn't enough to start the chillers which cool the facility while the load of the facility is on the generator, so if there is a generator failure on startup, and the chillers aren't started before the batteries run out, the data center is likely to overheat.
In a huge site like this one, there are bound to be generator failures, and the maintenance staff is not likely to be in the right place to change jumpers and get all the equipment not on UPS moved quickly. Chillers and motors have a large (but short) load on startup, and don't work well with UPS equipment, they need the momentum of a mechanical generator to get over the short large load.
A power outage can destroy parts that have been in manufacturing for months. Many steps in the process are non-restartable, but some are recoverable. That said, single-part fabs don't often take that long, and even if they did, half of quarterly production does sound high.
Honestly in this day and age it's criminal negligence by executive management they didn't have emergency power systems to sustain themselves in the event of a mains power failure. Every building should have had emergency batteries and generators.
It's not just the parts in production. Many of the production tools themselves will need to be recalibrated if not entirely refurbished to return to full production. Given the tolerances involved it can be an expensive and involved process. You don't really have backups handy for all your tools suddenly conking out at one time. Outside of situations like this it would be a waste that doubles the cost of production.
It takes about three months to process a 3D NAND wafer from start to the end. In other words, there was a quarter worth of wafers in the affected fabs, of which half are now expensive silicon discs due to them being in critical process steps that are not repeatable/repairable.
The problem with semiconductor manufacturing is that, until the last step is all finished, any interruption can turn weeks or months of work into scrap. Some of these wafers have to undergo not dozens, but hundreds of steps before they are done (and safe).
Seems very suspicious. Much less critical environments such as your typical cloud hosting services have layers of power redundancy. If an outage could cause this much damage, someone messed up big time designing and implementing their power redundancy and protection systems (which are generally routinely tested as well).
Sounds more like we needed a reason to bring SSD prices back up :(
Still seems shady, otherwise you'd expect this to happen every 2-3 years whenever there's a storm or poweroutage from earthquake or just power company maintenance going wrong.
Look at all the power outages in California/Texas/New York/Ohio with weather or maintenance failures. Redundancy is absolutely required with some sort of gas generators and some low power mode needed in the factory to preserve the systems rather than allow it to fail but still consume much lower power than full production.
Maybe pause manufacturing but keep temperature and pressure intact until non-backup power is restored.
Maybe they should invest in a Tesla battery system like Australia. Seriously though, battery backups are only meant to keep things running until the generators kick in. I would assume that their generators didn't work, or they were too cheap to invest in some to begin with. Especially if they're covered by insurance and can profit from higher NAND prices this is sure to cause.
You underestimate the amount of power a place like this needs. Battery backup is prob just for certain critical applications, and generators can't sustain the power output.
Battery backup (or mechanical energy from a flywheel) is only there to hold the load until generators spin up. That is why batteries may only last a few minutes at most. There's almost no reason to have batteries if you don't have generators. The disaster happens when the generators don't start.
Not really. But what one needs is the same thing a city would need. A power PLANT. Land being what it is in Japan space may be tight for something like that.
Regardless of power requirements they not only could but should have had in place emergency power. IMO it's fiscal negligence to not have full backup power capability that could either keep the factory going through an outage or at the very least allowed a controlled shutdown to minimize wafer loss.
You act as if Diesel generation systems are impossible to build and having emergency plans for something like this is impossible. There are entire countries that run on Diesel generators and some of those countries are not small. For about one acre of ground they could have had a cluster of diesel generators that could have supplied emergency power in the gigawatt range and it's literally trivial to purchase diesel delivery contracts that would have kept those generators running until the mains came back on upwards of months if necessary (though at that kind of timeframe the costs would get exorbitant).
Sure it would have cost several tens of millions of dollars for the generators and ongoing maintenance costs of several hundered thousand a year to keep them at peak form and working but frankly if I was invested in this company I'd be furious that management had been this negligent. They just lost a whole quarters revenue (and with repair and scrapping of the damaged stuff it's probably closer to 2Q's worth of revenue) due to fiscal negligence by management.
Any type of factory that is that dependent on power is going to have backup power systems in place for both short outages and much longer mains failures. That these fabs didn't is just unbelievable, what if someone knocked a power pole over down the road in a car accident? The exact same thing would have happened and that's a very common accident. They literally had no protection for an outage of more than a few minutes and that's just criminal.
That's some impressive armchair engineering. Why didn't Toshiba and WD consult you when they were building their multi billion dollar campus when you clearly know better than all of the engineers they hired?!
Sometimes decisions aren't a matter of engineering, but of economics. Clearly somewhere in Japan someone engineering a system that could supply all these plants and others. That takes care of the smart-alec "why didn't they consult you". They didn't need to when someone else already did all the work. What they didn't do is basically duplicate that system and the reason for that is simple economics. Just because it's called "semiconductor manufacturing" doesn't mean "bottom-line" is ignored. And in that vein "cheap electronics" is part of that. So let's ask ourselves the most important question. Are we willing to spend more for our electronics so plants like this can survive power outages for a reasonable amount of time?
The companies involved likely believed, at the time the plants were built, that because Japanese power was primarily provided through nuclear that the chances of a sustained outage were near impossible. The entire equation on that changed with the fukushima disaster and they should have adjusted.
Economics no doubt plays into the equation but you know why this type of disaster doesn't happen very often? Because all the other fabs have backup power systems in place. Anyone this dependent on power has backup systems and contracts in place, hell even my office building has such systems.
I think several people pointed out that the amount of power needed is in the area of 60MW. Meaning a feasible redundancy strategy like a back up grid or so would have solved the issue.
6EB sounds like a lot, but in the bigger picture it's not that much. You can get a 1TB Sabrent or Inland Phison E12 for $100, and that's retail pricing with dram, controller, etc. Meaning that 6EB is at most $6 million.
In contrast TSMC took a $550 million hit a few months ago from a defective chemical making Nvidia and Huawei chips go bad.
@azfacea, it's comments like yours why I rarely bother posting. You're better than that, aren't you? 6 comments in 1 article first attacks the victim of circumstances in the article and the next 5 attack other people's comments. Sorry if your home life is harsh but please don't take it out on us. Give it to a therapist instead.
Agreed, the AnandTech readership is in a tailspin. I don’t understand why you cannot simply implement upvote/downvote buttons to get rid of abusive posts like his? Or at the very least get some moderators. This community is dying.
I forgot to calculate the $100 part so yeah, up to $600 million.
It will actually be lower than $600 since a 1TB Toshiba SSD (Sabrent or Inland) retails for $100, but not that much lower. I would estimate somewhere between $400 to $500 million
Apparently, the power outage was quite widespread, and affected not just the fabs, but the entire area (and many thousands of residents of that province). That being said, I also couldn't help to think "what a coincidence", as Toshiba/WD had planned to reduce their 2D NAND output, and this was one way of getting it done, and, possibly, the insurance and power companies to pay for it. I don't work in that business, but I expect any insurance affected by this to hire a skilled investigative team before cutting a very sizable check.
Fukushima didn't have a chance no matter how well maintained the generators. While designed to survive an earthquake, they were never designed for a tsunami of that magnitude.
And to point that out most of the generators (primary and backups) were actually destroyed by the Tsunami, by the time they were able to bring in emergency generators from offsite the plant was already in meltdown.
It didn't actually happen. They hired Kubrick to film the whole thing. Those 6 exabytes were purchased by the US government to run simulations on alien technology in Area 51.
I know it is complicated and Expensive but any manufacturing business of that sort that can lose billions of dollars from few minutes of power outage should have some sort of backup or redundancy. There should be no excuse for planning that sort of precaution ahead of time. I know it's Japan and those kind of incidents are very rare but a business like semiconductor manufacturing can not leave any thing for luck.
I'd like to buy a wafer as decoration for my room @Toshiba, how much $ is it per Wafer? Seriously, there could be quite some demand for this from enthusiasts. (And how much would be a fully functioning wafer be worth?)
Depends on the manufacturing process how many layers the NAND has and some other stuff but it can cost as much as 12k usd for 300mm wafers, maybe more.
I actually have an uncut wafer from Dallas Semi. Not sure how the guy obtained them, but I was able to score one These are much older, smaller wafers than a modern fab works with, and the process is large enough that with a decent magnifier you can see the structure, but it's still neat. I need to frame it and build some sort of protection - they are VERY fragile. The guy was selling whole ones as well as broken, I have one of each, keep the whole one well wrapped and away from things like asshole cat who knocks everything on the floor. I'd say a old wafer like this is someone more interesting than a modern one, as you wouldn't be able to see much with the naked eye, compared to this old one.
just another example of why bitcoin mining happens in places with very large hydro power locally. of all the sources of dem teeny electrons screaming down the wire, them's the most reliable.
even if the prices will rise, you do understand that both of those companies will lose Billions of USD, as part of damaged goods, damaged machinery, losing major clients and contract and more. for those companies this is NOT an excuse but more of a Nightmare coming to materialization.
"for those companies this is NOT an excuse but more of a Nightmare coming to materialization. "
well, that assumes there is sufficient slack in current production elsewhere to fulfill the lost qtrly. production. IOW, the rest of the supply market has to have boatloads of unspoken-for production in process. near zero likelihood. it does take a qtr., apparently, end-to-end to get a chip out. depending on how long it takes to re-start the lines, what are the odds that other producers increasing production without firm contracts for product? near zero, yes?
and so on. lots of timing issues have to be answered w/resp. to how much production comes from others and how much moolah T/WD get from insurance.
It is nowhere near a billion in looses, let alone billions. Do the math, NAND sells to consumers at 0.10 / GB let alone what manufacturers pay. No equipment was damaged.
I worked at one of the largest datacenters in the world and its power draw would have been significantly more than the fabs in this article. In addition to direct connections to two different power stations we had batteries capable of keeping the power online long enough for our diesel generators which had a week supply of diesel and contracts with separate companies to do refills if needed. There are only two ways a loss like this could have happened. 1. They bribed a power company and sabotaged their own backup systems hoping to collect on insurance or 2. The executives were so totally incompetent they spent no money on proper disaster planning and should all lose their jobs.
Anyone who thinks 1. Isn’t possible should google “Enron California power fraud”
once again, with fervor: 1) what was/is the estimated probability of a 13 minute power outage and 2) what is the cost of building out a local power station (and maintaining its capability to power up and be on line fast enough to avoid loss of X% of in-process product) robust enough to run all of the campus for at least that long? I agree it's most likely no one ran the numbers for that scenario, since the back of the bar nap arithmetic made it way to costly.
no CxO is ever going to say, "I don't care what it costs, just give me a 1,000 (or X) megawatt power station that will go on-line in 10 milliseconds!!!!"
A little more to it than that. What is the results of a data-center going down, and how much would that cost? Same question applied to a semiconductor plant? Some might say the former is more forgiving, while the one's who did the final result agreed differently. i.e data-center loses more.
No way on earth a data center looses more. A really good DC would be active-active with another geo separate facility and experience zero loss. A step down you restore power and pull in your near realtime backups from a NVME storage array and be back up in a hours to a day with some database headaches. In any case no one with even a quarter of a brain would go near a data center if they got told "well backup power generation is just too hard and costly at our scale. If we loose commercial feeds we'll run on UPS for 10 minutes or so and then your compute load goes poof gone."
how odd, seems one fab of sorts for memory or nand or storage is being shut/taken down for various reasons and of course "massive amounts of inventory not yet totally accounted for will likely affect all consumers alike"
bad luck they say comes in streaks, that should mean wicked great is going to happen as well (but people are people, and do stupid people things)
so likely it will be just as pricing has finally started to come down most of the folks making this stuff are putting as little out as possible (at higher price) to try to "maintain" status quo (you want this .01c grain of sand, ok, it will cost..... 1000000% more (minimum) likely will break in a year or so because we saved some of that sand to use on your next .01c grain of sand (which is now .0000001c (shhhhh dont tell anyone)
Not really seeing any traditional backup genset looking gear on google maps view. Is it possible they were so cocksure on power delivery they don't have any? How on earth would you get that approved, its like not building bathrooms...
Do tell us oh wise engineer, say you got the contract to build out a fab. Your solution is for a 15 minute power hit is to take it on the nose and burn something like 100 million to half a billion in lost inventory and calibration/restart time? You're telling me a data center can be engineered to have backup battery/genset power with 30kW power density racks but a fab cant? I don't buy that for a second.
That's why corporation do have insurance contracts - in many many cases having an insurance and allocating the cost of the risk among many payers may be a lot more financially soundly than double or tripple backup - this is not a matter of saving human lives scenario, so the much cheaper insurance path makes more sense. And yes, the end user is part of the insurance cost allocation, essentially.
Gunbuster, this campus practically needs a power plant for 6 Fab sites at one location. It has to be using more than 500MW. Samsung's largest plant requires 200MW
So you allocate 5% of your build budget to backup power generation. They always tout fabs costing billions. 5% of your build that's 50 million per building to buy gear to keep your operation up in a disaster scenario. lets say 10 10 megawatt gensets per building. I'd say that's doable on 50 million. And do it per building so you dont have all your eggs for the campus in one basket...
As a business...why spend the money? The problem increases chip prices due to market forces and there is probably some support with insurance. They make more money over time by not investing in backup systems. If the industry or consumers are so concerned about it, then they can collectively spend the money for it.
Why build the building to earthquake code? As a business just loose production while you rebuild half the structure and pass all the cost on to your consumers? Insurance will support too derp a lerp. That's how it works right?
That's not a honest comparison. The business and building will survive a power outage with a 30 day disruption. An earthquake can destroy the entire fab and sink the company for years.
Accidents like this always happen when memory prices have fallen. They weren't having any luck with floods or fires this year so a convenient power outage makes sense. Ridiculous.
We’ve updated our terms. By continuing to use the site and/or by logging into your account, you agree to the Site’s updated Terms of Use and Privacy Policy.
Facebook
Twitter
RSS
148 Comments
Back to Article
azfacea - Friday, June 28, 2019 - link
wow a fab that gets destroyed for 3 months by a power outage lol. darwin award well deservedV900 - Friday, June 28, 2019 - link
Little harsh there! We have no idea what caused the outage, who was responsible or if it could have been prevented.Skeptical123 - Friday, June 28, 2019 - link
"unexpected power outage in the Yokkaichi province" These fab are some of the most sophisticated and costly factories in the world. A simple loss of grid power is by far one of their easiest issues to mitigate and is done so with if possible multiple power pant connections, batteries, and diesel generators. This is standard practice at facilities that cost a fraction of these chips fabs. As such regardless of the reason heads will rollFunBunny2 - Saturday, June 29, 2019 - link
"easiest issues to mitigate and is done so with if possible multiple power pant connections, batteries, and diesel generators."does "too big to fail" strike a familiar note? just because a company could build out sufficient back up power supply for a campus doesn't mean it's smart to do so. balanced against the cost of the back up supply is the expected occurrence of demand. IOW, if such a power loss was/is believed to be a once in a decade or more event, then the loss of product spread over that decade may be far less than the amortized cost of the back up supply. the engineers and bean counters decided that cost exceeded the probability. what matters is the actual cause, and why the event occurred. it may be that the event was within the designed probability envelope, and the companies are OK with the outcome.
azfacea - Saturday, June 29, 2019 - link
OMG the siliness in this thread is out of control. Dude power loss is NOT a once in a decade event in japan. This is clearly a design failure. Dude stop pretending these fabs drew more power than half the country. Dude aircraft carriers have onboard power. Some clown actually said "there may not have been enough space in japan" these fabs cost more than aircraft carrier to build. are u seriously saying they didnt think about power outage when designing this? if thats the case every semiconductor should have multiple fabs out every year.My brain is about boil at the sheer number of ppl and magnitude of stupidity in this thread. I am truly amazed at the number of ppl who can be this far out in cartoonland. I was wondering how can a whole country be stupid enough to fall for something like a bolshevik revolution. now i know.
PeachNCream - Sunday, June 30, 2019 - link
It's ironic that someone who cannot consistently operate the shift key or type the word "people" in order to communicate effectively is losing it over the supposed stupidity of others. That's a lovely glass house you're living in over there, dude.FunBunny2 - Sunday, June 30, 2019 - link
Dude, the issue is whether the cost/benefit analysis of a back up supply robust enough to save the campus *of this magnitude* (modified by the expectation of such an event) wasn't done or was ignored. As reported, there was a back up supply, just not one robust enough to last 13 minutes. You may not remember the Great Northeast Blackout, but I lived through it. Should every town have had a backup power plant? Holyoke did, in that the municipal plant was off-grid, so wasn't taken out in the cascade. Good planning or good luck?Samus - Monday, July 1, 2019 - link
This comment is spot on.This likely has nothing to do with grid power. See that small building to the Northeast. That's a power distribution facility. The problem occurred somewhere between there and the campus buildings, because billion dollar sites like this produce their own backup power in real time.
JanW1 - Monday, July 1, 2019 - link
This likely has nothing to do with technical reasons at all."Western Digital Corp. shares logged their best weekly gain in nearly seven years Friday as investors considered a power outage at one of the digital storage device maker's flash memory production facilities to be good news given the glut in memory chip inventories."
'Nuff said.
eva02langley - Monday, July 1, 2019 - link
We are talking about extreme power usage here. There is no backup for that.azazel1024 - Tuesday, July 2, 2019 - link
Of course there is. You aren't talking gigawatts of power. You are talking at most several dozen megawatts. I've seen the backup power supplies for many industrial operations that draw far more power than a fab. You are talking a backup power solution that likely at most runs in to the low hundred million range for what is several billion dollars of equipment. Even ignoring the cost of the equipment that might be damaged, 6 exabytes of NAND production is in the ballpark of $150-200 million dollars worth of NAND down the drain (plus Toshiba's losses, which if the article is correct in speculation, is likely another $200-400 million in losses).Example of backup "UPS" on a grand scale.
https://electrek.co/2018/09/24/tesla-powerpack-bat...
$66 million installation cost for 100mw of battery power, 300mwh storage (so 3 HOURS of backup power). Yes, the Tesla project was done well after whatever backup power system the fab complex has would have been installed. The point is, for what is clearly higher output than what the fab would need (again, the power requirements for the entire fab complex are likely to be in the 10-40mw range), it is drop in the bucket compared to the losses they suffered.
I am sure that the fab complex DOES have backup power sufficient to the task. What I am also pretty sure of is it isn't redundant (which it should be) and also very likely wasn't tested sufficiently to catch problems (as that would likely require shutting down the fab temporarily, maybe once a year, to test the system live). Most industries that have/rely on backup power do an annual plugs out test.
Looks like maybe WD/Toshiba didn't adhere to industry standards there.
whyaname - Friday, July 5, 2019 - link
Actually there is, you only need a few seconds of UPS service till the back up generators of such factories kick in.You usually have several protection levels for this stuff.
For example it is not uncommon for such large facillities to have a complete discrete power plant or to be connected to more than one power plant/grid at a time in addition to UPS and generators.
Did you never wonder why industrial hardware has like 2-4 power connections, that is not just a redundant PSU, it enables you to have the hardware on just as many discrete UPS and powergrids to mitigate the likelyhood of failure.
I remember seeing the new some time ago that some flash manufacturers want to reduce their production because or prior over production and lots of flash still in stock at warehouses.
This is 13min failure that could have been prevented, but instead somehow causes 3months of issues.
This sounds like a way to avoid issues with markt price manipulation, but artificialy reducing the available amount of flash.
This should under no scenario have caused an issue similar to the flood back then in south east asia.
If the power goes out you do not suddenly start pumping polluted air into the clean rooms and making stuff unusuable for months.
Dug - Monday, July 1, 2019 - link
Your use of the word dude is a clear indication about your knowledge of power. Aircraft carriers use direct nuclear power for one, so I'm not sure why you included that? And thinking they didn't have a power backup is another indication you have no idea how things work in a fab. Just the fact you have no clue how much power is used or how it's produced is another indication. Or the fact a backup can't produce enough power if you are right in the middle of production.I'm sure they'll be contacting you soon to revamp power distribution sense you know so much about it.
Gunbuster - Monday, July 1, 2019 - link
A high end data-center can run 30kW racks. Why can a data center plan and economically execute full time backup power with that energy density but a fab can not?Samus - Tuesday, July 2, 2019 - link
Gunbuster is correct with his assumption. I worked at Argonne National Laboratory through college in the APS (Advanced Photon Source) building - which is a large synchrotron particle accelerator. It had 32, and recently expanded to 36 sectors. It's actually larger than Fermi lab. But nowhere near as large as the large haldron collider.But collisions it does. The Argonne campus has its own power plant. So do fabs. As anybody who knows anything about power generations knows, power plants produce at capacity in real time, and this capacity is always beyond grid demand to account for spikes or sudden demand surges where power cannot be diverted elsewhere from the grid. This makes them inherently inefficient because the excess power can't be stored, so it is wasted in the form of simply not generating power that could be generated using the supplied pressure\heat\whatever type of fuel is being used. This is especially true with nuclear power production (although no sites like these use nuclear power production on-site, that is overkill) as it takes weeks, and even months with old designs, to roll back fuel processing and scale down power production. During the cooldown, they are super inefficient in the sense they COULD be making a lot more power for the same amount of fuel spent.
ANYWAY. This site clearly has a oil fuel power plant on site. I can spot it on the low-res campus photo in the top right. This plant was built to supply more than enough power to the site EVEN IF grid power seized. But built for, and producing, are two different things.
Something happened at the power processing facility, or something malfunctioned causing a blackout (which seems to be the case considering the length of time there was a power outage...they could divert grid power almost immediately like Samsung did during their recent production woes that lasted in the milliseconds.)
That said, this is hard to wrap my head around. These mistakes just don't happen. But mistakes can be manufactured. And mistakes that take down an entire campus of electricity for minutes almost has to be manufactured...even natural gas\diesel generators warm up and produce power in 20 seconds...
close - Tuesday, July 2, 2019 - link
@Gunbuster - A high end data-center can run 30kW racksProblem is in this case we're talking 2 orders of magnitude higher. In the tens of MW range, maybe even close to 100MW (GloFo Fab 8 needs 80MW *on average* as a reference). And supplying that from backup constantly until the power is restored in not something most industries ever have to deal with so people should stop using their UPS as a reference for anything. There's no real way to save any power during such incidents, there's no "power saving mode", no shutting down part of the line. The backup power has to provide exactly what the main did. Usually you manage to supply enough power to shut down the lines with no damage to the equipment but all the silicon in the pipeline just became trash anyway.
This happened to Samsung, it happened to Toshiba/WD. It's just AT commenters that seem to have the wrong career. AT became home to all of the "I'm an expert dude" type people.
Gunbuster - Tuesday, July 2, 2019 - link
And most data centers don't cost a billion to stand up. You scale up to provide services. Plenty of 10MW gensets to choose from. Sure they are the size of a semi trailer but that's a huge building. Line up 15 of them.whyaname - Friday, July 5, 2019 - link
@closeYou realize that is supposed to mean 30kW per rack?
scale that
Aside from that somebody very well posted already that backup power solutions for this scale exist.
Round - Tuesday, July 9, 2019 - link
Not related to the article, but regarding " how can a whole country be stupid enough to fall for something like a bolshevik revolution".We have almost half the US population teetering on the edge of a similar mental catastrophe right now. Intelligence is not distributed equally...
lazarpandar - Friday, June 28, 2019 - link
..What? Darwin awards are awarded for deaths.FullmetalTitan - Friday, June 28, 2019 - link
Clearly you have no idea how semiconductor manufacturing works. 13 minutes is an INSANELY long outage. Samsung had at 200ms (yeah, 0.2s) outage in 2017 that cost something like $50M in product losses.Losing power means they lost:
All RF etch tools - any wafers in those tools are scrap
All fab exhaust/pumps/cooling water - any tools with sensitive environments scrap 100%
All power to furnaces - temperature varied from spec, 100% scrap
azfacea - Friday, June 28, 2019 - link
clearly you have no idea how anything works.Eris_Floralia - Saturday, June 29, 2019 - link
Anandtech surely has spanded their reader coverage downwards recently.DyneCorp - Saturday, June 29, 2019 - link
Why don't you elaborate how he's wrong then, smart man?azfacea - Saturday, June 29, 2019 - link
go read the rest of these comments dumb manJedi2155 - Friday, July 5, 2019 - link
The conspiracy theorist in me thinks this is an attempt to artificially raise the market value of NAND.kgardas - Friday, June 28, 2019 - link
What happened? Has SSD price fallen that quickly over past month and so "correction" is needed? Or was there a need to collect some insurance money? 13 minutes outage in *Japan* is from sci-fi category...EliteRetard - Friday, June 28, 2019 - link
Yep, they needed a price correction. Planned out carefully, insurance will pay out fantastically and they'll essentially make a profit on all this (and even more profit on everything else sold for many years after) as prices shoot up. I think this lesson was learned from the HDD flooding a few years back. Obviously that was a real natural accident, this is likely the best possible "accident" they could come up with to simulate the effect (and with very precise control).I'd love to see a documentary on how they did it. It couldn't have been easy to cause such a cascade failure in all the parts of such a complex setup. Wonder how they compensated some engineers from the power company (or the company itself, though that seems more a China thing).
I don't need a tinfoil hat for this one, I stay in Japan 3 months at a time...those people are not hapless idiots.
Diji1 - Tuesday, July 9, 2019 - link
>those people are not hapless idiots.Of course not, not like they'd place diesel backup generators on coast where they know a tsunami will hit one day. Or the nuclear power stations that rely on the diesel generators working to not meltdown.
Nope, that would be the behavior of hapless idiots!
rrinker - Friday, June 28, 2019 - link
After reading the Micro article earlier, when they said production was high and demand low, my cynical side said, wait for the disaster to hit. A few hours later - this article shows up. Let's face it, once or twice is a coincidence, but this sort of thing happens far too often when supply of product X is exceeding demand of product X for it to be always just a coincidence.rrinker - Friday, June 28, 2019 - link
Of course you can;t edit here - I was referring to the MICRON story just a few items down.Yojimbo - Saturday, June 29, 2019 - link
And they hired Stanley Kubrick to film it on the moon...zodiacfml - Saturday, June 29, 2019 - link
beat me to this comment. it is a convenient outage for cheap memory recentlySivar - Friday, June 28, 2019 - link
"Western Digital says that the 13-minute power outage...will reduce its NAND flash wafer supply in Q3 by approximately 6 EB (exabytes), which is believed to be about a half of the company’s quarterly supply of NAND."Talk about write amplification!
jtd871 - Friday, June 28, 2019 - link
I can't imagine the power draw of that campus, but it probably makes it difficult to impossible to have enough standby capacity for an outage.azfacea - Friday, June 28, 2019 - link
r u serious? these fabs cost more than aircraft careers. they can't deal with a 13 min power outage cause it draws too much power ?? LOLIan Cutress - Friday, June 28, 2019 - link
Five fabs and an R&D center, outage was after the batteries also ran out.For perspective, the batteries at GF's leading fab can run the 1/3 of the systems for only a few minutes. That's the scale we're dealing with.
DigitalFreak - Friday, June 28, 2019 - link
Guess they've never heard of generators?imaheadcase - Friday, June 28, 2019 - link
That is not how power delivery works for a huge place like this. A generator can't maintain power load.mat9v - Sunday, June 30, 2019 - link
It does not have to maintain anything. It has to last until safe shutdown of processing facilities. Everything is turned off at first moment, only emergency lights, elevators, air purification and actual machines that make chips. Of course there are acceptable looses as there are always some wafers in chambers being processed that can't be halted. But definitely something went wrong in this or someone without a shred of intelligence created the project that did not account for a long term power loss.eva02langley - Monday, July 1, 2019 - link
We are not talking about UPS there. A generator is not a stable source of power. It introduce a lot of noise. Also, silicon fabs use a huge amount of power in their process that no backup power can provide.We don't know how this affect the would supply chain because we are not in the industry. Seeing smart idiots claiming they know better is just ridiculous.
eva02langley - Monday, July 1, 2019 - link
whole not wouldmat9v - Tuesday, July 2, 2019 - link
Are you absolutely sure of that? How much power exactly does it take? ASML machines that actually make the product use pretty weak lasers, on the whole current best ones (strongest) use 250W. You don't want to evaporate metals there, you want to, so to speak, shine some light at the correct frequency - it's not a brute force approach but a finesse one. I'm sure much more power is used for filtering atmosphere, temperature control or simple lights then it is on actual lithography.The generator may not be stable power source but power lines are not stable too, you really think the power is not filtered, stabilised and converted as many times as needed?
Besides, battery backup they had is not stable either, no matter what type of batteries the used, voltage decreases a lot during discharge, they must have a lot of stabilization going on, you think that would not be capable of stabilizing generators?
The only explanation is they did not turn the generators on counting on power being restored or the whole "accident" is a scam created for unsold inventory, insurance claim and future prices increase.
With only one "stroke" you get a lot of benefits.
whyaname - Friday, July 5, 2019 - link
@eva02langleyGenerators are not stable power sources?
Well I am of to all power plants around the world telling them your conclusion that whatever they use is not stable and that we should replace it with solar panels even if it needs 50-100 the area of a nuclear plant to produce the same amount of energy under good conditions.
eastcoast_pete - Friday, June 28, 2019 - link
Battery- and generator-power can only support the security and absolutely essential systems for more than a few minutes. These fabs draw hundreds of megawatt a piece, and essentially any silicon that isn't fully finished is basically lost once the power goes out. The only way to be able to go off-grid for such fabs is an on-site power station, and that you'd need backups for that, because they also go down or need maintenance.mat9v - Sunday, June 30, 2019 - link
Are you absolutely certain that there are no points at which, in the hundreds steps process, production can not be halted? Does all the phases happen on one sealed chamber or does the wafers being processed travel between different chambers in the fab?Dug - Monday, July 1, 2019 - link
You don't understand. Even something simple like baking the wafers can't be interrupted. If the process takes 1 continuous hour, and you lost power for 13 minutes, then you lost everything. There's so many processes that you can't just shut down and start back up again in the middle.mat9v - Tuesday, July 2, 2019 - link
Well, then if you are aware of that, then you create a solution that will deliver power for the required time. If not you are a moron, or you are counting on insurance to cover your ass.zanon - Monday, July 1, 2019 - link
Not him and late, but yes, absolutely certain. It's genuinely hard to overstate how insanely precise the timing and stability requirements of each step are when you're literally, LITERALLY talking "3 atoms is too thin, 5 atoms is too thick". The modern silicon fabrication process is genuinely one of humanity's greatest wonders. If you haven't seen it, a good little talk is "Indistinguishable From Magic": https://www.youtube.com/watch?v=NGFhc8R_uO4Keep in mind that everything you here in there is now very old and obsolete tech too.
FullmetalTitan - Friday, June 28, 2019 - link
These facilities pay out the nose for power assurance from local utilities. They get the same guarantees as hospitals.They also have banks and banks of UPS systems, like thousands of batteries, and even then can only supply critical safety systems for more than a few minutes.
The Samsung fab in Austin I live near has its own power substation in case you need a good indicator of how much power these facilities use. The Yokkaichi complex is about 3x that size
mat9v - Sunday, June 30, 2019 - link
And how much time do you need to safely halt production?Or is that, that someone did not flip the switch hoping for power to be restored in moments and safety systems did not engage?
Or these fabs really created in such a flawed way that simple power outage will destroy months of work even with all emergency systems and protocols in place and operational?
Do they really only depend on insurance policy to cover looses?
Ananke - Monday, July 1, 2019 - link
You cannot halt production in silicon chips process. Besides, it takes days and weeks for a full process to complete.This Yokkaichi site probably uses 500-600 MWs - this power generation requires two large GE turbines /if gas electric power plant or a nuclear reactor is used :) :)/ plus reserve power of the same magnitude. Such turbines need several hours for start up time, and more than 24 hrs of reaching optimal condition, i.e. no power generators can help.
Regardless, the memory industry is happy - accident will cause jump of prices to sustainable levels, memory prices went to down - everybody was on freeze since January - no capital expenditures, no hire, no budget increases, 25% furloughs etc. This will help for years to come.
magreen - Monday, July 1, 2019 - link
@Ananke. Thanks for the clear explanation. Nice to hear a straightforward answer from someone who knows what s/he's talking about.mat9v - Tuesday, July 2, 2019 - link
Ok, it seems I was mistaken after all:http://energyskeptic.com/2014/interdependent-chip-...
"Fab plants use up to 60 megawatts of power"
Oh well, it is strange that so much power is required since the lasers lithography actually use are so weak (250W).
azazel1024 - Tuesday, July 2, 2019 - link
Better numbers than I found, thanks Mat9v. 60mw, is a lot. But it isn't an obscene amount. Ananke, as another commenter noted, many critical sites such as this produce their own power interfaced with the grid. It isn't terribly efficient, but it provides completely redundancy. That power is filtered so spike and sags are cleaned up, fluctuations in frequency, etc. If the campus power generation failed, it is grid tied and the fail over is to the grid, which can react in seconds (meanwhile the sagging voltage generated by throwing 60mw on to the local/regional grid during that time is filtered and supported by the on campus power filtering equipment, so the campus wouldn't see the voltage sag).Or at least that SHOULD be how it is done. Obviously something went badly wrong, but you can see the power plant on the campus in the picture. They have one. They just either didn't have it online, or they were using it as backup and the grid as primary and whatever they had running for intermittent grid disconnection didn't support the campus power requirements until their power plant came up (IE bad design/no testing/no sufficient layers of backups).
azfacea - Friday, June 28, 2019 - link
yes i think thats why uss ronald reagon runs on tesla supplied batteries. oh wait.cmon u can't be serious in asserting that a 13 minute power outage in japan is so rare or so expensive to deal with, that they just planned to live with it
voicequal - Friday, June 28, 2019 - link
Obviously the generators failed to start or the control systems failed.Santoval - Saturday, June 29, 2019 - link
Generators do not turn on instantly. Hell, even batteries do not start providing power instantly. Even if they are super speedy batteries which less than 1 second to start providing power that's long enough to wreck everything running. Super/ultracapacitors are required for nearly instant (<1 ms) emergency power delivery.But these have low energy capacity, so my guess is that they might be employed to provide an "energy burst" for that first 0.5 to 1 sec, and then hand over the power delivery to the batteries. The batteries, in turn, can hand the power delivery to generators after a few minutes.
I have no idea if this a viable scheme though. Instead of employing a set of supercapacitors, batteries and generators of enormous size and mass just for emergency power, it might make more sense -and cost almost the same- to build a dedicated mini power plant on-site. Ideally it should require barely any maintenance and a skeleton staff, i.e something like a solar farm along with some other power source (like wind) for the solar downtime.
It could be for emergency power and the rest of the time its power could be sold back to the grid - or as the main power source with the grid acting as backup power, depending on which option was more favorable.
Ej24 - Sunday, June 30, 2019 - link
Pretty sure I remember reading that new huge Tesla battery in Australia can kick on and provide power to the grid and stabilize voltage on the order of milliseconds. Like 50 milliseconds or less. So pretty sure these smart guys that designed such a fab would probably have something equivalent in place.Jedi2155 - Friday, July 5, 2019 - link
@Ej24 - That type of technology is relatively new TBH. Most battery control systems I've worked with have much slower control loops in the order of hundreds or thousands of ms due to their industrial background of most of the systems. Tesla however used repurposed vehicle control systems which have much faster control loops which is why Tesla has a moat there.evernessince - Tuesday, July 2, 2019 - link
@SantovolOh please, "super speedy batteries"? I've got battery backup for my house and the lights don't even flicker when the power goes out. Heaven forbid a billion dollar facility can't do that.
FunBunny2 - Tuesday, July 2, 2019 - link
"I've got battery backup for my house and the lights don't even flicker when the power goes out."that's because, if you have an APC like mine, it's in line all the time, is connected to the PC, which means each and every machine in each building has to have a very scaled up APC. and each machine needs sufficient intelligence to be monitored by the APC controller to do that.
rrinker - Friday, June 28, 2019 - link
Hell 25+ years ago the large mainframe datacenter I worked in had no chance that batteries could sustain the processors, DASD, and more importantly, the chillers. A huge room full of batteries was just enough to keep it going for the THREE generators to fire up. Any 2 generators could sustain the load, the third was a backup. These were sort of medium size generators - the sort of diesels you might find in heavy construction equipment (slightly larger than semi truck types). Around the same time I also got to tour a major telecom interconnect site - they too had a huge room full of batteries to allow their generators to start - their generators used diesels the size of locomotive prime movers.Santoval - Saturday, June 29, 2019 - link
So the batteries did not provide them enough time to shut down anything "gracefully"? Why have batteries at all then, if they are good for nothing? Or did the batteries provide them enough power to shut down just the most critical systems, and the rest had to be left out?GreenReaper - Sunday, June 30, 2019 - link
To keep things going until the generators start up. They can keep going as long as there's fuel.imaheadcase - Friday, June 28, 2019 - link
Yes he is serious. Most companies don't have enough standby power for full operations, just for critical things. Considering this is a plant, critical is not full operations. I can tell for a fact that its easier to claim loss than worry about it. Case in point, a normal walmart does $100million a year..but only power emergency lights, servers for registers and cameras. Thats it. The back up power sits in a small closet. Freezers, coolers, etc are not powered at all.evernessince - Tuesday, July 2, 2019 - link
Walmart does not loose half it's product if the power goes out. Your comparison makes zero sense.Acreo_Aeneas - Sunday, June 30, 2019 - link
Actually, most fabs cost somewhere in the ballpark of $3-5 billion. A typical first-world aircraft carrier of a modern navy costs $6-9 billion.In comparison, the latest Gerald Ford US aircraft carrier is estimated to cost about $9 billion. While the Fab15 facility of TSMC was estimated to cost $9.3 billion.
Given that, perhaps they are more close to parity in terms of cost, rather than one being far costlier than the other.
As for backup power, it's probably not feasible for a fab of this size to have extensive battery/generator backups. Where as, some very modern aircraft carriers may have backup systems in place, but they are only meant for short periods and at minimal operating capacity (probably emergency lighting, basic navigation/systems, and perhaps some weaponry).
Backup systems for a mutli-facility fab like this probably would be very expensive to maintain and operate. Even if it did exist and work properly, I doubt it would hold out for 13 minutes. Most larger server facilities that have backup systems are designed to let the facility fail gracefully (basically enough time to save and power down, not run for hours on end until power is restored). Even, these larger server farms probably would still crash and fail if they ever had to rely on their backup power systems.
FunBunny2 - Sunday, June 30, 2019 - link
" Where as, some very modern aircraft carriers may have backup systems in place, but they are only meant for short periods and at minimal operating capacity"considering that such boats are run by pressurized light water nucular reactors, how much of the boat still runs if that reactor goes legs up is the least of your problems. :)
jhh - Monday, July 1, 2019 - link
US Aircraft carriers have two nuclear reactors, and a single reactor is sufficient to keep the ship moving fast enough to deploy and retrieve aircraft, although deployment is slower - perhaps only one launch at once, and perhaps a lower payload on the aircraft since the ship isn't able to move at full speed. But a stopped aircraft carrier is likely to become a sunk aircraft carrier.A set of generators at my office stores two weeks of fuel for the diesel generators, and has contracts in place to deliver a full load of fuel on a day's notice. One of the two generators is sufficient to operate the facility, but one isn't enough to start the chillers which cool the facility while the load of the facility is on the generator, so if there is a generator failure on startup, and the chillers aren't started before the batteries run out, the data center is likely to overheat.
In a huge site like this one, there are bound to be generator failures, and the maintenance staff is not likely to be in the right place to change jumpers and get all the equipment not on UPS moved quickly. Chillers and motors have a large (but short) load on startup, and don't work well with UPS equipment, they need the momentum of a mechanical generator to get over the short large load.
V900 - Friday, June 28, 2019 - link
A 13 minute power outage results in losing HALF THEIR QUARTERLY PRODUCTION?!?That sounds absolutely insane!
Sivar - Friday, June 28, 2019 - link
A power outage can destroy parts that have been in manufacturing for months. Many steps in the process are non-restartable, but some are recoverable.That said, single-part fabs don't often take that long, and even if they did, half of quarterly production does sound high.
rahvin - Friday, June 28, 2019 - link
Honestly in this day and age it's criminal negligence by executive management they didn't have emergency power systems to sustain themselves in the event of a mains power failure. Every building should have had emergency batteries and generators.desolation0 - Saturday, June 29, 2019 - link
It's not just the parts in production. Many of the production tools themselves will need to be recalibrated if not entirely refurbished to return to full production. Given the tolerances involved it can be an expensive and involved process. You don't really have backups handy for all your tools suddenly conking out at one time. Outside of situations like this it would be a waste that doubles the cost of production.Kristian Vättö - Friday, June 28, 2019 - link
It takes about three months to process a 3D NAND wafer from start to the end. In other words, there was a quarter worth of wafers in the affected fabs, of which half are now expensive silicon discs due to them being in critical process steps that are not repeatable/repairable.eastcoast_pete - Friday, June 28, 2019 - link
The problem with semiconductor manufacturing is that, until the last step is all finished, any interruption can turn weeks or months of work into scrap. Some of these wafers have to undergo not dozens, but hundreds of steps before they are done (and safe).CosmoJoe - Friday, June 28, 2019 - link
Seems very suspicious. Much less critical environments such as your typical cloud hosting services have layers of power redundancy. If an outage could cause this much damage, someone messed up big time designing and implementing their power redundancy and protection systems (which are generally routinely tested as well).Sounds more like we needed a reason to bring SSD prices back up :(
Ian Cutress - Friday, June 28, 2019 - link
Five fabs and an R&D center, outage was after the batteries also ran out.For perspective, the batteries at GF's leading fab can run the 1/3 of the systems for only a few minutes. That's the scale we're dealing with.
webdoctors - Friday, June 28, 2019 - link
Still seems shady, otherwise you'd expect this to happen every 2-3 years whenever there's a storm or poweroutage from earthquake or just power company maintenance going wrong.Look at all the power outages in California/Texas/New York/Ohio with weather or maintenance failures. Redundancy is absolutely required with some sort of gas generators and some low power mode needed in the factory to preserve the systems rather than allow it to fail but still consume much lower power than full production.
Maybe pause manufacturing but keep temperature and pressure intact until non-backup power is restored.
DigitalFreak - Friday, June 28, 2019 - link
Maybe they should invest in a Tesla battery system like Australia. Seriously though, battery backups are only meant to keep things running until the generators kick in. I would assume that their generators didn't work, or they were too cheap to invest in some to begin with. Especially if they're covered by insurance and can profit from higher NAND prices this is sure to cause.imaheadcase - Friday, June 28, 2019 - link
You underestimate the amount of power a place like this needs. Battery backup is prob just for certain critical applications, and generators can't sustain the power output.voicequal - Friday, June 28, 2019 - link
Battery backup (or mechanical energy from a flywheel) is only there to hold the load until generators spin up. That is why batteries may only last a few minutes at most. There's almost no reason to have batteries if you don't have generators. The disaster happens when the generators don't start.Threska - Friday, June 28, 2019 - link
Not really. But what one needs is the same thing a city would need. A power PLANT. Land being what it is in Japan space may be tight for something like that.http://energyskeptic.com/2014/interdependent-chip-...
And note he says "expensive" not impossible.
Also https://www.researchgate.net/publication/237511031...
Is 2.18 kW/m2
rahvin - Friday, June 28, 2019 - link
Regardless of power requirements they not only could but should have had in place emergency power. IMO it's fiscal negligence to not have full backup power capability that could either keep the factory going through an outage or at the very least allowed a controlled shutdown to minimize wafer loss.You act as if Diesel generation systems are impossible to build and having emergency plans for something like this is impossible. There are entire countries that run on Diesel generators and some of those countries are not small. For about one acre of ground they could have had a cluster of diesel generators that could have supplied emergency power in the gigawatt range and it's literally trivial to purchase diesel delivery contracts that would have kept those generators running until the mains came back on upwards of months if necessary (though at that kind of timeframe the costs would get exorbitant).
Sure it would have cost several tens of millions of dollars for the generators and ongoing maintenance costs of several hundered thousand a year to keep them at peak form and working but frankly if I was invested in this company I'd be furious that management had been this negligent. They just lost a whole quarters revenue (and with repair and scrapping of the damaged stuff it's probably closer to 2Q's worth of revenue) due to fiscal negligence by management.
Any type of factory that is that dependent on power is going to have backup power systems in place for both short outages and much longer mains failures. That these fabs didn't is just unbelievable, what if someone knocked a power pole over down the road in a car accident? The exact same thing would have happened and that's a very common accident. They literally had no protection for an outage of more than a few minutes and that's just criminal.
jordanclock - Friday, June 28, 2019 - link
That's some impressive armchair engineering. Why didn't Toshiba and WD consult you when they were building their multi billion dollar campus when you clearly know better than all of the engineers they hired?!Threska - Saturday, June 29, 2019 - link
Sometimes decisions aren't a matter of engineering, but of economics. Clearly somewhere in Japan someone engineering a system that could supply all these plants and others. That takes care of the smart-alec "why didn't they consult you". They didn't need to when someone else already did all the work. What they didn't do is basically duplicate that system and the reason for that is simple economics. Just because it's called "semiconductor manufacturing" doesn't mean "bottom-line" is ignored. And in that vein "cheap electronics" is part of that. So let's ask ourselves the most important question. Are we willing to spend more for our electronics so plants like this can survive power outages for a reasonable amount of time?rahvin - Monday, July 1, 2019 - link
The companies involved likely believed, at the time the plants were built, that because Japanese power was primarily provided through nuclear that the chances of a sustained outage were near impossible. The entire equation on that changed with the fukushima disaster and they should have adjusted.Economics no doubt plays into the equation but you know why this type of disaster doesn't happen very often? Because all the other fabs have backup power systems in place. Anyone this dependent on power has backup systems and contracts in place, hell even my office building has such systems.
whyaname - Friday, July 5, 2019 - link
I think several people pointed out that the amount of power needed is in the area of 60MW.Meaning a feasible redundancy strategy like a back up grid or so would have solved the issue.
zodiacfml - Saturday, June 29, 2019 - link
sadly but trueDyneCorp - Saturday, June 29, 2019 - link
Man you people are nuts.Alistair - Friday, June 28, 2019 - link
Wow, yeap sounds like they should invest in a Tesla battery storage system ;)Bravadu - Friday, June 28, 2019 - link
6EB sounds like a lot, but in the bigger picture it's not that much. You can get a 1TB Sabrent or Inland Phison E12 for $100, and that's retail pricing with dram, controller, etc. Meaning that 6EB is at most $6 million.In contrast TSMC took a $550 million hit a few months ago from a defective chemical making Nvidia and Huawei chips go bad.
Kristian Vättö - Friday, June 28, 2019 - link
6 exabytes = 6 billion gigabytesAt $0.10/GB, that means $600M
wishgranter - Friday, June 28, 2019 - link
that is the final retail price, we talking about manufacturing price of just wafers, not controller and etc.. So expect price in $0.02 - $0.03 /GBitazfacea - Friday, June 28, 2019 - link
600 mil is correct. if you think nand chips are 20-30% of ssd prices you don't have a clue.FullmetalTitan - Friday, June 28, 2019 - link
Literally every one of your comments here has been wrong, so maybe don't call out other peopleazfacea - Friday, June 28, 2019 - link
last time i am gonna shit on you. after this get lost.https://www.trendforce.com/price/flash
wholesale price for 3D TLC 256Gb ==== 3.1 USD
256 Gb = 32 GB
32GB ~= 3.2 USD ==>> price per GB 0.10 cents.
Got shat on by enough?
purerice - Friday, June 28, 2019 - link
@azfacea, it's comments like yours why I rarely bother posting. You're better than that, aren't you? 6 comments in 1 article first attacks the victim of circumstances in the article and the next 5 attack other people's comments. Sorry if your home life is harsh but please don't take it out on us. Give it to a therapist instead.MarcusMo - Saturday, June 29, 2019 - link
Agreed, the AnandTech readership is in a tailspin. I don’t understand why you cannot simply implement upvote/downvote buttons to get rid of abusive posts like his? Or at the very least get some moderators. This community is dying.Threska - Saturday, June 29, 2019 - link
Moderators would be better. Up/Down gets used too much as agree/disagree rather than merit..e.g Reddit.GreenReaper - Sunday, June 30, 2019 - link
Disagree.RealBeast - Friday, June 28, 2019 - link
Ouch -- that's gotta sting a little, even for large corporations like Toshiba/WDC.Bravadu - Friday, June 28, 2019 - link
I forgot to calculate the $100 part so yeah, up to $600 million.It will actually be lower than $600 since a 1TB Toshiba SSD (Sabrent or Inland) retails for $100, but not that much lower. I would estimate somewhere between $400 to $500 million
eastcoast_pete - Friday, June 28, 2019 - link
Apparently, the power outage was quite widespread, and affected not just the fabs, but the entire area (and many thousands of residents of that province). That being said, I also couldn't help to think "what a coincidence", as Toshiba/WD had planned to reduce their 2D NAND output, and this was one way of getting it done, and, possibly, the insurance and power companies to pay for it. I don't work in that business, but I expect any insurance affected by this to hire a skilled investigative team before cutting a very sizable check.voicequal - Friday, June 28, 2019 - link
An uncontrolled plant shutdown is actually a fairly dangerous situation. It's not something you want to test your insurance with.DyneCorp - Saturday, June 29, 2019 - link
Oh please, just stop. Jesus Christ. You people are nuts.DigitalFreak - Friday, June 28, 2019 - link
Now we know where all the Fukushima Daiichi electrical maintenance workers went!voicequal - Friday, June 28, 2019 - link
Fukushima didn't have a chance no matter how well maintained the generators. While designed to survive an earthquake, they were never designed for a tsunami of that magnitude.rahvin - Friday, June 28, 2019 - link
And to point that out most of the generators (primary and backups) were actually destroyed by the Tsunami, by the time they were able to bring in emergency generators from offsite the plant was already in meltdown.evernessince - Tuesday, July 2, 2019 - link
This is technically not true at all. Had they used a thorium based nuclear reactor, a nuclear meltdown would have been easily avoidable.KlfJoat - Friday, June 28, 2019 - link
Darwin Award is for removal from the gene pool. Death is a removal, sure. But so is castration, uterine ablation, and many other things.jrs77 - Friday, June 28, 2019 - link
Prices are falling... quick, let's fake an accident to cretae an artificial shortage to boost prices again.Seriously, this is getting old and needs to stop.
zodiacfml - Saturday, June 29, 2019 - link
exactlyYojimbo - Saturday, June 29, 2019 - link
It didn't actually happen. They hired Kubrick to film the whole thing. Those 6 exabytes were purchased by the US government to run simulations on alien technology in Area 51.Ashinjuka - Friday, June 28, 2019 - link
If Shinji had got Ramiel in one shot none of this would have happened.WE WERE COUNTING ON YOU SHINJI-KUN. 😫
Eliadbu - Friday, June 28, 2019 - link
I know it is complicated and Expensive but any manufacturing business of that sort that can lose billions of dollars from few minutes of power outage should have some sort of backup or redundancy. There should be no excuse for planning that sort of precaution ahead of time. I know it's Japan and those kind of incidents are very rare but a business like semiconductor manufacturing can not leave any thing for luck.Dark42 - Saturday, June 29, 2019 - link
I'd like to buy a wafer as decoration for my room @Toshiba, how much $ is it per Wafer?Seriously, there could be quite some demand for this from enthusiasts.
(And how much would be a fully functioning wafer be worth?)
Eliadbu - Saturday, June 29, 2019 - link
Depends on the manufacturing process how many layers the NAND has and some other stuff but it can cost as much as 12k usd for 300mm wafers, maybe more.rrinker - Monday, July 1, 2019 - link
I actually have an uncut wafer from Dallas Semi. Not sure how the guy obtained them, but I was able to score one These are much older, smaller wafers than a modern fab works with, and the process is large enough that with a decent magnifier you can see the structure, but it's still neat. I need to frame it and build some sort of protection - they are VERY fragile. The guy was selling whole ones as well as broken, I have one of each, keep the whole one well wrapped and away from things like asshole cat who knocks everything on the floor. I'd say a old wafer like this is someone more interesting than a modern one, as you wouldn't be able to see much with the naked eye, compared to this old one.FunBunny2 - Saturday, June 29, 2019 - link
just another example of why bitcoin mining happens in places with very large hydro power locally. of all the sources of dem teeny electrons screaming down the wire, them's the most reliable.PVG - Saturday, June 29, 2019 - link
sigh... And that's how you get an excuse to rise NAND prices all over again...Eliadbu - Saturday, June 29, 2019 - link
even if the prices will rise, you do understand that both of those companies will loseBillions of USD, as part of damaged goods, damaged machinery, losing major clients and contract
and more. for those companies this is NOT an excuse but more of a Nightmare coming to materialization.
FunBunny2 - Sunday, June 30, 2019 - link
"for those companies this is NOT an excuse but more of a Nightmare coming to materialization. "well, that assumes there is sufficient slack in current production elsewhere to fulfill the lost qtrly. production. IOW, the rest of the supply market has to have boatloads of unspoken-for production in process. near zero likelihood. it does take a qtr., apparently, end-to-end to get a chip out. depending on how long it takes to re-start the lines, what are the odds that other producers increasing production without firm contracts for product? near zero, yes?
and so on. lots of timing issues have to be answered w/resp. to how much production comes from others and how much moolah T/WD get from insurance.
evernessince - Tuesday, July 2, 2019 - link
It is nowhere near a billion in looses, let alone billions. Do the math, NAND sells to consumers at 0.10 / GB let alone what manufacturers pay. No equipment was damaged.SlashZerov - Saturday, June 29, 2019 - link
I worked at one of the largest datacenters in the world and its power draw would have been significantly more than the fabs in this article. In addition to direct connections to two different power stations we had batteries capable of keeping the power online long enough for our diesel generators which had a week supply of diesel and contracts with separate companies to do refills if needed. There are only two ways a loss like this could have happened. 1. They bribed a power company and sabotaged their own backup systems hoping to collect on insurance or 2. The executives were so totally incompetent they spent no money on proper disaster planning and should all lose their jobs.Anyone who thinks 1. Isn’t possible should google “Enron California power fraud”
GreenReaper - Sunday, June 30, 2019 - link
Let's be honest, 2) is just as possible - but of course "should" doesn't mean "will".FunBunny2 - Sunday, June 30, 2019 - link
"2) is just as possible"once again, with fervor: 1) what was/is the estimated probability of a 13 minute power outage and 2) what is the cost of building out a local power station (and maintaining its capability to power up and be on line fast enough to avoid loss of X% of in-process product) robust enough to run all of the campus for at least that long? I agree it's most likely no one ran the numbers for that scenario, since the back of the bar nap arithmetic made it way to costly.
no CxO is ever going to say, "I don't care what it costs, just give me a 1,000 (or X) megawatt power station that will go on-line in 10 milliseconds!!!!"
Threska - Sunday, June 30, 2019 - link
A little more to it than that. What is the results of a data-center going down, and how much would that cost? Same question applied to a semiconductor plant? Some might say the former is more forgiving, while the one's who did the final result agreed differently. i.e data-center loses more.Gunbuster - Wednesday, July 3, 2019 - link
No way on earth a data center looses more. A really good DC would be active-active with another geo separate facility and experience zero loss. A step down you restore power and pull in your near realtime backups from a NVME storage array and be back up in a hours to a day with some database headaches. In any case no one with even a quarter of a brain would go near a data center if they got told "well backup power generation is just too hard and costly at our scale. If we loose commercial feeds we'll run on UPS for 10 minutes or so and then your compute load goes poof gone."benzosaurus - Saturday, June 29, 2019 - link
Surprised it took this long, what with flash prices being low and all.Dragonstongue - Saturday, June 29, 2019 - link
how odd, seems one fab of sorts for memory or nand or storage is being shut/taken down for various reasons and of course "massive amounts of inventory not yet totally accounted for will likely affect all consumers alike"bad luck they say comes in streaks, that should mean wicked great is going to happen as well
(but people are people, and do stupid people things)
so likely it will be just as pricing has finally started to come down most of the folks making this stuff are putting as little out as possible (at higher price) to try to "maintain" status quo (you want this .01c grain of sand, ok, it will cost..... 1000000% more (minimum) likely will break in a year or so
because we saved some of that sand to use on your next .01c grain of sand (which is now .0000001c (shhhhh dont tell anyone)
Kastriot - Sunday, June 30, 2019 - link
Who cares they have dosh, but now we gonna pay their loses.eva02langley - Monday, July 1, 2019 - link
So basically, tomshardware is stating "to decline by six exabytes, or 1 billion gigabytes,"...1 000 000 000 gb X 0.22$ revenu/gb (roughly on amazon retail) = 10 000 000$ of revenue
I don't get it.
eva02langley - Monday, July 1, 2019 - link
Oups... 220 000 000$... post this too fast.Okay, still a nice sum, but I don't believe it is such a disaster.
For SSD nand, we are talking about 100 000 000$ for a 100$ tb drive.
Gunbuster - Monday, July 1, 2019 - link
Not really seeing any traditional backup genset looking gear on google maps view. Is it possible they were so cocksure on power delivery they don't have any? How on earth would you get that approved, its like not building bathrooms...eva02langley - Monday, July 1, 2019 - link
Running a server bank and making silicon are two different things... but only in the world of simpletons such comparison can occurs...Coming from an electric engineer by the way.
Gunbuster - Monday, July 1, 2019 - link
Do tell us oh wise engineer, say you got the contract to build out a fab. Your solution is for a 15 minute power hit is to take it on the nose and burn something like 100 million to half a billion in lost inventory and calibration/restart time? You're telling me a data center can be engineered to have backup battery/genset power with 30kW power density racks but a fab cant? I don't buy that for a second.Ananke - Monday, July 1, 2019 - link
That's why corporation do have insurance contracts - in many many cases having an insurance and allocating the cost of the risk among many payers may be a lot more financially soundly than double or tripple backup - this is not a matter of saving human lives scenario, so the much cheaper insurance path makes more sense. And yes, the end user is part of the insurance cost allocation, essentially.Adramtech - Tuesday, July 2, 2019 - link
Gunbuster, this campus practically needs a power plant for 6 Fab sites at one location. It has to be using more than 500MW. Samsung's largest plant requires 200MWGunbuster - Tuesday, July 2, 2019 - link
So you allocate 5% of your build budget to backup power generation. They always tout fabs costing billions. 5% of your build that's 50 million per building to buy gear to keep your operation up in a disaster scenario. lets say 10 10 megawatt gensets per building. I'd say that's doable on 50 million. And do it per building so you dont have all your eggs for the campus in one basket...Adramtech - Sunday, July 7, 2019 - link
As a business...why spend the money? The problem increases chip prices due to market forces and there is probably some support with insurance. They make more money over time by not investing in backup systems. If the industry or consumers are so concerned about it, then they can collectively spend the money for it.Gunbuster - Monday, July 8, 2019 - link
Why build the building to earthquake code? As a business just loose production while you rebuild half the structure and pass all the cost on to your consumers? Insurance will support too derp a lerp. That's how it works right?Adramtech - Tuesday, July 9, 2019 - link
That's not a honest comparison. The business and building will survive a power outage with a 30 day disruption. An earthquake can destroy the entire fab and sink the company for years.izmanq - Tuesday, July 2, 2019 - link
nah, this is just their excuse to raise priceBigDragon - Tuesday, July 2, 2019 - link
Accidents like this always happen when memory prices have fallen. They weren't having any luck with floods or fires this year so a convenient power outage makes sense. Ridiculous.jiji86 - Monday, July 8, 2019 - link
sounds like a secret plan to control the falling prices !!