"counteractive actions" sounds ominous. When things like this happen in a factory, it's very seldom just a yield problem. The testing of the chips is fine tuned to catch obvious issues. but, when some new issue shows up and yield tanks this is accompanied by some percentage of chips being shipped as good but with a problem missed by the testing. Once the problem is studied, some new test must be created to ensure shipping at high quality until the original problem is fully resolved. So, there is potentially a time window of material slipping out that is not of acceptable quality. This is usually when there is a need to talk to customers about "counteractive actions." If fully contained internally, then they simply crank through it to deliver promised quantities and no public statement is needed.
That said, it could simply be that output dropped and they need to cover themselves for a cost of operations hit... Temporary hit to margin. And temporary hit to supply. That is the best case scenario that does not require "counteractive actions."
It's hard to tell what they mean here, because it almost feels like sabotage when you consider photopolymers are common industrial chemicals. It's just mind boggling to wrap my head around how there was a "bad" batch. It's like getting a bad batch of epoxy resin at Home Depot. It just doesn't happen. And that's at a consumer scale where manufacturing tolerances are loose and batch testing is a fraction of a percent of production.
TSMC/Samsung/Intel have their own quality standards for every chem they use, whether that is ultra pure water, sulfuric acid, or photoresist.
These standards are not necessarily as high in the outbound testing from every individual supplier. i.e. Company A supplies H2SO4 with a spec of <x ppb of some contaminant, and a FAB may need it to be <y ppb, where y < x. Normally, this isn't an issue and that chem meets both specs 99% of the time, but if it falls in between those values, it may not be seen on the shipped quality report. It isn't feasible to do full analysis of every potential contaminant on every batch received, so these issues are found by tracing a failure cascade backwards to the source, hence the large impact and delayed response.
The "counteractive actions" is almost certainly going to be an audit request for the supplier in question, and a request to tighten testing standards to mitigate risk of future incidents.
As mentioned in my separate comment here, I am wondering if TSMC's internal QC was ratcheted down a bit or two to "increase productivity". TSMC has been very successful in being the first out of the gate with 7 nm, but that Fab cost a fortune, and all of TSMC's fabs (not just the 7 nm one) now have to earn that multi-billion $$$ investment back so TSMC can pay back the money they borrowed for it. Just like you, I also suspect a lot of SOP retraining and some personnel changes at the site.
Not sure how badly production problems will hurt NVIDIA. Yesterday's announcement pushed their stock a lot lower and seems to indicate softer Turing sales than expected (well, softer than NVIDIA expected, but no one looking at RTX pricing and TDP should be surprised).
Markets go up and down on a daily basis and the tech market had major hits to it in October, the difference is since then Nvidia is down even further and due to their own performance rather than the market. AMD peaked as the market grew but are still miles up on a year ago due to their performance being good and Intel is pretty flat on the year.
Compared to a year ago AMD are up 40%, Nvidia are down 46% and Intel are 3% down.
Until the RTX2060, the series was just priced WAY out of reality for 99% of gamers. I still don't know a single person who bought the RTX2080...
That's a stark contrast compared to the number of people I know cashing in their old cards for a RTX2060 - which is a proper value @ $350 when you factor in it includes about $60 in games and has some good overclocking headroom. It already trumps a $450 1070Ti.
Even the RTX2060 is a modest improvement. The RTX part is useless since it's too slow to actually use it, DLSS is limited to specific higher resolutions that unfortunately prevent it from being used in most cases, free games only matter if you care about those specific games or don't already have them, and overclocking headroom doesn't matter to most people (who don't overclock).
In practice, the 2060 appears to offer roughly comparable performance to the 1070 Ti (sometimes a bit faster, sometimes a bit slower) for around 80% of the price (as seen on Amazon). That's good, but doesn't offer a big enough improvement to entice anybody from the last few generations to upgrade. It's also a big price increase from the 1060 ($250), and still costs more than even the 1070 ($310).
I don't think it points to a public rejection of the raytracing technology - it's that they're expensive products sitting in the no-man's land between not having the feature at all, and having it be capable enough to play games with it at the kind of resolutions and fidelity that someone willing to pay that kind of money for a graphics card expect. I think most people, like myself, are waiting for gen 2 or 3 with acceptable performance, and support in a wider variety of games.
Based on the demonstrated performance I feel RTX should've been limited to the 2080 Ti primarily as a developer preview feature with wider availability promised when 7nm would let them launch it on enthusiast cards with at least 2x the current number of RTX cores. From there GTX 2080 should've been ~90% of the 2080 in conventional rendering but no ray tracing support, with the rest of the stack rebalanced down from there. The Jury is still out on DLAA and the value of the turing cores outside of ray tracing.
Gambling half your die area on aspirational features that won't benefit any existing titles was a huge risk; and unfortunately for NVidia one that appears to've failed for them. At this point the question I'd like an answer to is if they underestimated the amount of ray tracing throughput needed for good performance, or if v1 RTX cores fell significantly short of their expected performance numbers.
I'm sure they're selling a ton of TU104's in Tesla T4's, for deep learning/inferencing. In that sense, you can't really call the die area wasted. For games, yeah, but that's not the only use of these chips.
Unclear whether they had the volumes to justify a die specifically for this market, but possibly not.
RTX is the odd man out. I'm sure everyone in movie production is using it, but I'm inclined to agree they didn't have to put it in any dies besides their high-end TU102.
Tesla sales aren't as high as expected either. In their latest earnings adjustment announcement, NVIDIA cited slow Turing sales in general and that implies the entire range of products.
The RTX and Turing cores are roughly half the die area on a TU102/104/106 chip; the huge area is a primary reason why the RTX cards are so expensive and why at a given price point Turing is barely an upgrade over Pascal. It's only survivable for NVidia so far because AMD doesn't have anything competitive in higher performance segments. If Radeon VII or Navi perform as well in terms of absolute performance (even if still behind in perf/watt) as the rumor mills suggest the high prices and near uselessness of the RTX/Turing cores is going to put NVidia in a painful position until they can get their own 7nm chips out.
Talking about how the largest independent chip manufacturer is having issues due to chemical deviations seems like Tech hardware to me, without the products from companies like TSMC, the "tech hardware" you speak of would be rather limited.
Yeah, WTF? This is definitely more relevant than news about the companies' quarterly reports (although I'm not complaining about those articles, either).
I am a bit struck by the reason TSMC gives (below-spec photoresist chemical or chemicals). These chemicals are absolute essentials for any fabs photolithography. So, has TSMC gone a bit lackadaisical in the QC of its supplies, especially those that are mission-critical like this? With the entire output of a multi-billion $$$ fab depending on the purity of the chemicals used, standard operating procedure would be to QC each batch of chemical just before it's used, likely by in-line HPLC-MS or GC-MS. I am wondering if the pressure on TSMC to "make rent" on its enormous investment into 7 nm has led to risk-taking in form of laxer QC, which then leads to problems such as this one.
He's still doing it then. I never enjoyed his drivel and glad he's gone. Such a pathetic individual. Always writing as if he's the master and the authority on every single topic, claiming superior knowledge and information, be it electronics, politics, psychology, etc. without providing a shred of evidence, calling others arrogant and yet couldn't take it when someone challenged him.
He has an illogical personal vendetta against some companies, which is downright childish, but whenever someone pointed it out and suggested to tone it down a bit, he'd get hostile in his passive, aggressive way, call that person "cattle" and then proceed with insults that he claimed to be "sarcasm".
We’ve updated our terms. By continuing to use the site and/or by logging into your account, you agree to the Site’s updated Terms of Use and Privacy Policy.
28 Comments
Back to Article
woggs - Tuesday, January 29, 2019 - link
"counteractive actions" sounds ominous. When things like this happen in a factory, it's very seldom just a yield problem. The testing of the chips is fine tuned to catch obvious issues. but, when some new issue shows up and yield tanks this is accompanied by some percentage of chips being shipped as good but with a problem missed by the testing. Once the problem is studied, some new test must be created to ensure shipping at high quality until the original problem is fully resolved. So, there is potentially a time window of material slipping out that is not of acceptable quality. This is usually when there is a need to talk to customers about "counteractive actions." If fully contained internally, then they simply crank through it to deliver promised quantities and no public statement is needed.That said, it could simply be that output dropped and they need to cover themselves for a cost of operations hit... Temporary hit to margin. And temporary hit to supply. That is the best case scenario that does not require "counteractive actions."
Samus - Wednesday, January 30, 2019 - link
It's hard to tell what they mean here, because it almost feels like sabotage when you consider photopolymers are common industrial chemicals. It's just mind boggling to wrap my head around how there was a "bad" batch. It's like getting a bad batch of epoxy resin at Home Depot. It just doesn't happen. And that's at a consumer scale where manufacturing tolerances are loose and batch testing is a fraction of a percent of production.Not a conspiracy theorist but...how?
FullmetalTitan - Wednesday, January 30, 2019 - link
TSMC/Samsung/Intel have their own quality standards for every chem they use, whether that is ultra pure water, sulfuric acid, or photoresist.These standards are not necessarily as high in the outbound testing from every individual supplier.
i.e. Company A supplies H2SO4 with a spec of <x ppb of some contaminant, and a FAB may need it to be <y ppb, where y < x. Normally, this isn't an issue and that chem meets both specs 99% of the time, but if it falls in between those values, it may not be seen on the shipped quality report. It isn't feasible to do full analysis of every potential contaminant on every batch received, so these issues are found by tracing a failure cascade backwards to the source, hence the large impact and delayed response.
The "counteractive actions" is almost certainly going to be an audit request for the supplier in question, and a request to tighten testing standards to mitigate risk of future incidents.
eastcoast_pete - Sunday, February 3, 2019 - link
As mentioned in my separate comment here, I am wondering if TSMC's internal QC was ratcheted down a bit or two to "increase productivity". TSMC has been very successful in being the first out of the gate with 7 nm, but that Fab cost a fortune, and all of TSMC's fabs (not just the 7 nm one) now have to earn that multi-billion $$$ investment back so TSMC can pay back the money they borrowed for it. Just like you, I also suspect a lot of SOP retraining and some personnel changes at the site.PeachNCream - Tuesday, January 29, 2019 - link
Not sure how badly production problems will hurt NVIDIA. Yesterday's announcement pushed their stock a lot lower and seems to indicate softer Turing sales than expected (well, softer than NVIDIA expected, but no one looking at RTX pricing and TDP should be surprised).HStewart - Tuesday, January 29, 2019 - link
NVidia stock was not only one hit - also AMD and Intel (Intel was not hit bad however)Current: NVidia down 4.11% AMD down 3.26% Intel down 0.60%
I think a lot of this is because of RTX sales and pricing is likely a large part of this
But it news about TSMC yield could be large part of it.
drunkenmaster - Tuesday, January 29, 2019 - link
Markets go up and down on a daily basis and the tech market had major hits to it in October, the difference is since then Nvidia is down even further and due to their own performance rather than the market. AMD peaked as the market grew but are still miles up on a year ago due to their performance being good and Intel is pretty flat on the year.Compared to a year ago AMD are up 40%, Nvidia are down 46% and Intel are 3% down.
Samus - Wednesday, January 30, 2019 - link
Until the RTX2060, the series was just priced WAY out of reality for 99% of gamers. I still don't know a single person who bought the RTX2080...That's a stark contrast compared to the number of people I know cashing in their old cards for a RTX2060 - which is a proper value @ $350 when you factor in it includes about $60 in games and has some good overclocking headroom. It already trumps a $450 1070Ti.
zodiacfml - Wednesday, January 30, 2019 - link
yesGuspaz - Wednesday, January 30, 2019 - link
Even the RTX2060 is a modest improvement. The RTX part is useless since it's too slow to actually use it, DLSS is limited to specific higher resolutions that unfortunately prevent it from being used in most cases, free games only matter if you care about those specific games or don't already have them, and overclocking headroom doesn't matter to most people (who don't overclock).In practice, the 2060 appears to offer roughly comparable performance to the 1070 Ti (sometimes a bit faster, sometimes a bit slower) for around 80% of the price (as seen on Amazon). That's good, but doesn't offer a big enough improvement to entice anybody from the last few generations to upgrade. It's also a big price increase from the 1060 ($250), and still costs more than even the 1070 ($310).
twtech - Tuesday, January 29, 2019 - link
I don't think it points to a public rejection of the raytracing technology - it's that they're expensive products sitting in the no-man's land between not having the feature at all, and having it be capable enough to play games with it at the kind of resolutions and fidelity that someone willing to pay that kind of money for a graphics card expect. I think most people, like myself, are waiting for gen 2 or 3 with acceptable performance, and support in a wider variety of games.DanNeely - Tuesday, January 29, 2019 - link
Based on the demonstrated performance I feel RTX should've been limited to the 2080 Ti primarily as a developer preview feature with wider availability promised when 7nm would let them launch it on enthusiast cards with at least 2x the current number of RTX cores. From there GTX 2080 should've been ~90% of the 2080 in conventional rendering but no ray tracing support, with the rest of the stack rebalanced down from there. The Jury is still out on DLAA and the value of the turing cores outside of ray tracing.Gambling half your die area on aspirational features that won't benefit any existing titles was a huge risk; and unfortunately for NVidia one that appears to've failed for them. At this point the question I'd like an answer to is if they underestimated the amount of ray tracing throughput needed for good performance, or if v1 RTX cores fell significantly short of their expected performance numbers.
mode_13h - Wednesday, January 30, 2019 - link
I'm sure they're selling a ton of TU104's in Tesla T4's, for deep learning/inferencing. In that sense, you can't really call the die area wasted. For games, yeah, but that's not the only use of these chips.Unclear whether they had the volumes to justify a die specifically for this market, but possibly not.
RTX is the odd man out. I'm sure everyone in movie production is using it, but I'm inclined to agree they didn't have to put it in any dies besides their high-end TU102.
PeachNCream - Wednesday, January 30, 2019 - link
Tesla sales aren't as high as expected either. In their latest earnings adjustment announcement, NVIDIA cited slow Turing sales in general and that implies the entire range of products.DanNeely - Wednesday, January 30, 2019 - link
The RTX and Turing cores are roughly half the die area on a TU102/104/106 chip; the huge area is a primary reason why the RTX cards are so expensive and why at a given price point Turing is barely an upgrade over Pascal. It's only survivable for NVidia so far because AMD doesn't have anything competitive in higher performance segments. If Radeon VII or Navi perform as well in terms of absolute performance (even if still behind in perf/watt) as the rumor mills suggest the high prices and near uselessness of the RTX/Turing cores is going to put NVidia in a painful position until they can get their own 7nm chips out.boozed - Tuesday, January 29, 2019 - link
Indeed, given the fraudcoin mining bust and optimistic GPU pricing, they're not going to struggle with a supply hit.imaheadcase - Tuesday, January 29, 2019 - link
Remember when anandtech posted tech hardware related news. I do.bji - Tuesday, January 29, 2019 - link
According to the headline of this article, they did so as recently as January 29, 2019 11:00 AM EST. What's your point?joeyudog - Tuesday, January 29, 2019 - link
According to his username, he'saheadcase.jordanclock - Tuesday, January 29, 2019 - link
How is a problem at the largest semiconductor fab in the world NOT tech hardware news?Mobile-Dom - Tuesday, January 29, 2019 - link
Talking about how the largest independent chip manufacturer is having issues due to chemical deviations seems like Tech hardware to me, without the products from companies like TSMC, the "tech hardware" you speak of would be rather limited.mode_13h - Wednesday, January 30, 2019 - link
Yeah, WTF? This is definitely more relevant than news about the companies' quarterly reports (although I'm not complaining about those articles, either).eastcoast_pete - Tuesday, January 29, 2019 - link
I am a bit struck by the reason TSMC gives (below-spec photoresist chemical or chemicals). These chemicals are absolute essentials for any fabs photolithography. So, has TSMC gone a bit lackadaisical in the QC of its supplies, especially those that are mission-critical like this? With the entire output of a multi-billion $$$ fab depending on the purity of the chemicals used, standard operating procedure would be to QC each batch of chemical just before it's used, likely by in-line HPLC-MS or GC-MS. I am wondering if the pressure on TSMC to "make rent" on its enormous investment into 7 nm has led to risk-taking in form of laxer QC, which then leads to problems such as this one.eddman - Wednesday, January 30, 2019 - link
If ddriver was still around, this is what he'd write: "There is no doubt Intel and/or the US government sent ninjas to sabotage TSMC."mode_13h - Wednesday, January 30, 2019 - link
You might enjoy this (ddriver on ddriver):https://www.phoronix.com/forums/forum/hardware/pro...
eddman - Wednesday, January 30, 2019 - link
He's still doing it then. I never enjoyed his drivel and glad he's gone. Such a pathetic individual. Always writing as if he's the master and the authority on every single topic, claiming superior knowledge and information, be it electronics, politics, psychology, etc. without providing a shred of evidence, calling others arrogant and yet couldn't take it when someone challenged him.He has an illogical personal vendetta against some companies, which is downright childish, but whenever someone pointed it out and suggested to tone it down a bit, he'd get hostile in his passive, aggressive way, call that person "cattle" and then proceed with insults that he claimed to be "sarcasm".
Manch - Wednesday, January 30, 2019 - link
Nvidia says no prob, we will sell these as 1160/50's LOLAchtung_BG - Wednesday, January 30, 2019 - link
TSMC fab14b produces 16/12 nm. defective are between 10k and 30k wafers.