Jeff Atwood, perhaps the most widely read programming blogger, has a post that makes a case against using ECC memory. My read is that his major points are:
- Google didn’t use ECC when they built their servers in 1999
- Most RAM errors are hard errors and not soft errors
- RAM errors are rare because hardware has improved
- If ECC were actually important, it would be used everywhere and not just servers. Paying for optional stuff like this is downright enterprisey
Let’s take a look at these arguments one by one:
1. Google didn’t use ECC in 1999
If you do things just because Google once did them, here are some things you might do:
A. Put your servers into shipping containers.
Articles are still written today about what a great idea this is, even though this was an experiment at Google that was deemed unsuccessful. Turns out, even Google’s experiments don’t always succeed. In fact, their propensity for “moonshots” means that they have more failed experiments that most companies. IMO, that’s a substantial competitive advantage for them. You don’t need to make that advantage bigger than it already is by blindly copying their failed experiments.
B. Cause fires in your own datacenters
Part of the post talks about how awesome these servers are:
Some people might look at these early Google servers and see an amateurish fire hazard. Not me. I see a prescient understanding of how inexpensive commodity hardware would shape today’s internet.
The last part of that is true. But the first part has a grain of truth, too. When Google started designing their own boards, one generation had a regrowth1 issue that caused a non-zero number of fires.
BTW, if you click through to Jeff’s post and look at the photo that the quote refers to, you’ll see that the boards have a lot of flex in them. That caused problems and was fixed in the next generation. You can also observe that the cabling is quite messy, which also caused problems, and was also fixed in the next generation. There were other problems, but I’ll leave those as an exercise for the reader.
C. Make servers that injure your employees
One generation of Google servers had infamously sharp edges, given them the reputation of being made of “razor blades and hate”.
D. Create weather in your datacenters
From talking to folks at a lot of large tech companies, it seems that most of them have had a climate control issue resulting in clouds or fog in their datacenters. You might call this a clever plan by Google to reproduce Seattle weather so they can poach MS employees. Alternately, it might be a plan to create literal cloud computing. Or maybe not.
Note that these are all things Google tried and then changed. Making mistakes and then fixing them is common in every successful engineering organization. If you’re going to cargo cult an engineering practice, you should at least cargo cult current engineering practices, not something that was done in 1999.
When Google used servers without ECC back in 1999, they found a number of symptoms that were ultimately due to memory corruption, including a search index that returned effectively random results to queries. The actual failure mode here is instructive. I often here that’s ok to ignore ECC on these machines because it’s ok to have errors in individual results. But even when you can tolerate occasional errors, ignoring errors means that you’re exposing yourself to total corruption, unless you’ve done a very careful analysis to make sure that a single error can only contaminate a single result. In research that’s been done on filesystems, it’s been repeatedly shown that despite making valiant attempts at creating systems that are robust against a single error, it’s extremely hard to do so and basically every heavily tested filesystem can have a massive failure from a single error (see the output of Andrea and Remzi’s research group at Wisconsin if you’re curious about this). I’m not knocking filesystem developers here. They’re better at that kind of analysis than 99.9 programmrs. It’s just that this problem has been repeatedly shown to be hard enough that humans cannot effectively reason about it, and automated tooling for this kind of analysis is still far from a pushbutton process. In their book on warehouse scale computing, Google discusses error correction and detection and ECC is cited as their slam dunk case for when it’s obvious that you should use hardware error correction2.
Google has great infrastructure. From what I’ve heard of the infra at other large tech companies, Google’s sounds like the best in the world. But that doesn’t mean that you should copy everything they do. Even if you look at their good ideas, it doesn’t make sense for most companies to copy them. They created a replacement for Linux’s work stealing scheduler that uses both hardware run-time information and static traces to allow them to take advantage of new hardware in Intel’s server processors that lets you dynamically partition caches between cores. If used across their entire fleet, that could easily save Google more money in a week than stackexchange has spent on machines in their entire history. Does that mean you should copy Google? No, not unless you’ve already captured all the lower hanging fruit, which includes things like making sure that your core infrastructure is written in highly optimized C++, not Java or (god forbid) Ruby. And the thing is, for the vast majority of companies, writing in a language that imposes a 20x performance penalty is a totally reasonable decision.
2. Most RAM errors are hard errors
The case against ECC quotes this section of a study on DRAM errors (the bolding is Jeff’s):
Our study has several main findings. First, we find that approximately 70% of DRAM faults are recurring (e.g., permanent) faults, while only 30% are transient faults. Second, we find that large multi-bit faults, such as faults that affects an entire row, column, or bank, constitute over 40% of all DRAM faults. Third, we find that almost 5% of DRAM failures affect board-level circuitry such as data (DQ) or strobe (DQS) wires. Finally, we find that chipkill functionality reduced the system failure rate from DRAM faults by 36x.
This is somewhat ironic, as this quote doesn’t sound like an argument against ECC; it sounds like an argument for chipkill, a particular class of ECC. Putting that aside, Jeff’s post points out that hard errors are twice as common as soft errors, and then mentions that they run memtest on their machines when they get them. First, a 2:1 ratio isn’t so large that you can just ignore soft errors. Second the post implies that Jeff believes that hard errors are basically immutable and can’t surface after some time. That’s incorrect. You can think of electronics as wearing out just the same way mechanical devices wear out. The mechanisms are different, but the effects are similar. In fact, if you compare reliability analysis of chips vs. other kinds of reliability analysis, you’ll find they often use the same families of distributions to model failures. Third, Jeff’s line of reasoning implies that ECC can’t help with detection or correction of hard errors, which is not only incorrect but directly contradicted by the quote.
So, how often are you going to run memtest on your machines to try to catch these hard errors, and how much data corruption are you willing to live with? One of the key uses of ECC is not to correct errors, but to signal errors so that hardware can be replaced before silent corruption occurs. No one’s going to consent to shutting down everything on a machine every day to run memtest (that would be more expensive than just buying ECC memory), and even if you could convince people to do that, it won’t catch as many errors as ECC will.
When I worked at a company that owned about 1000 machines, we noticed that we were getting strange consistency check failures, and after maybe half a year we realized that the failures were more likely to happen on some machines than others. The failures were quite rare, maybe a couple times a week on average, so it took a substantial amount of time to accumulate the data, and more time for someone to realize what was going on. Without knowing the cause, analyzing the logs to figure out that the errors were caused by single bit flips (with high probability) was also non-trivial. We were lucky that, as a side effect of the process we used, the checksums were calculated in a separate process, on a different machine, at a different time, so that an error couldn’t corrupt the result and propogate that corruption into the checksum. If you merely try to protect yourself with in-memory checksums, there’s a good chance you’ll perform a checksum operation on the already corrupted data and compute a valid checksum of bad data unless you’re doing some really fancy stuff with calculations that carry their own checksums (and if you’re that serious about error correction, you’re probably using ECC regardless). Anyway, after completing the analysis, we found that memtest couldn’t detect any problems, but that replacing the RAM on the bad machines caused a one to two order of magnitude reduction in error rate. Most services don’t have this kind of checksumming we had; those services will simply silently write corrupt data to persistent storage and never notice problems until a customer complains.
3. Due to advances in hardware manufacturing, errors are very rare
The data in the post isn’t sufficient to support this assertion. Note that since RAM usage has been and continues to increase at a fast exponential rate, RAM failures would have to decrease at a greater exponential rate to actually reduce the incidence of data corruption. Furthermore, as chips continue shrink, features get smaller, making the kind of wearout issues discussed in “2” more common. For example, at 20nm, a DRAM capiactor might hold something like 50 electrons, and that number will get smaller for next generation DRAM and things continue to shrink.
The 2012 study that Atwood quoted has this graph on corrected errors (a subset of all errors) on ten randomly selected failing nodes (6% of nodes had at least one failure):
We’re talking between 10 and 10k errors for a typical node that has a failure, and that’s a cherry-picked study from a post that’s arguing that you don’t need ECC. Note that the nodes here only have 16GB of RAM, which is an order of magnitude less than modern servers often have, and that this was on a older process node that was less vulnerable to noise than we are now. For anyone who’s used to dealing with reliability issues and just wants to know the FIT rate, the study finds a FIT rate of between 0.057 and 0.071 faults per Mbit (which, contra Atwood’s assertion, is not a shockingly low number)3. Another thing to note is that there are multiple nodes that don’t have errors at the start of the study but develop errors later on.
Sun/Oracle famously ran into this a number of decades ago. Transistors and DRAM capacitors were getting smaller, much as they are now, and memory usage and caches were growing, much as they are now. Between having smaller transistors that were less resilient to transient upset as well as more difficult to manufacture, and having more on-chip cache, the vast majority of server vendors decided to add ECC to their caches. Sun decided to save a few dollars and skip the ECC. The direct result was that a number of Sun customers reported sporadic data corruption. It took Sun multiple years to spin a new architecture with ECC cache, and Sun made customers sign an NDA to get replacement chips. Of course there’s no way to cover up this sort of thing forever, and when it came up, Sun’s reputation for producing reliable servers took a permanent hit, much like the time they tried to cover up poor performance results by introduce a clause into their terms of services disallowing benchmarking.
Another thing to note here is that when you’re paying for ECC, you’re not just paying for ECC, you’re paying for parts (CPUs, boards) that have been qual’d more thoroughly. You can easily see this with disk failure rates, and I’ve seen many people observe this in their own private datasets. In terms of public data, I believe Andrea and Remzi’s group had a SIGMETRICS paper a few years back that showed that SATA drives were 4x more likely than SCSI drives to have disk read failures, and 10x more likely to have silent data corruption. This relationship held true even with drives from the same manufacturer. There’s no particular reason to think that the SCSI interface should be more reliable than the SATA interface, but it’s not about the interface. It’s about buying a high-reliability server part vs. a consumer part. Maybe you don’t care about disk reliability in particular because you checksum everything and can easily detect disk corruption, but there are some kinds of corruption that are harder to detect.
4. If ECC were actually important, it would be used everywhere and not just servers.
Rephrased slightly, this argument is “If this feature were actually important for servers, it would be used in non-servers”. You could make this argument about a fair number of server hardware features. This is actually one of the more obnoxious problems facing large cloud vendors.
They have enough negotiating leverage to get most parts at cost, but that only works where there’s more than one viable vendor. Some of the few areas where there aren’t any viable competitors include CPUs and GPUs. Luckily for them, they don’t need that many GPUs, but they need a lot of CPUs and the bit about CPUs has been true for a long time. There have been a number of attempts by CPU vendors to get into the server market, but each attempt so far has been fatally flawed in a way that made it obvious from an early stage that the attempt was doomed (and these are often 5 year projects, so that’s a lot of time to spend on a doomed project). The Qualcomm effort has been getting a lot of hype, but when I talk to folks I know at Qualcomm they all tell me that the current chip is basically for practice, since Qualcomm needed to learn how to build a server chip from all the folks they poached from IBM, and that the next chip is the first chip that has any hope of being competitive. I have high hopes for Qualcomm as well an ARM effort to build good server parts, but those efforts are still a ways away from bearing fruit.
The near total unsuitability of current ARM (and POWER) options (not including hypothetical variants of Apple’s impressive ARM chip) for most server workloads in terms of performance per TCO dollar is a bit of a tangent, so I’ll leave that for another post, but the point is that Intel has the market power to make people pay extra for server features, and they do so. Additionally, some features are genuinely more important for servers than for mobile devices with a few GB of RAM and a power budget of a few watts that are expected to randomly crash and reboot periodically anyway.
Conclusion
Should you buy ECC RAM? That depends. For servers, it’s probably a good bet considering the cost, although it’s hard to really do a cost/benefit analysis because it’s really hard to figure out the cost of silent data corruption, or the cost of having some risk of burning half a year of developer time tracking down intermittent failures only to find that the were caused by using non-ECC memory.
For normal desktop use, I’m pro-ECC, but if you don’t have regular backups set up, doing backups probably has a better ROI than ECC. But if you have backups without ECC, you can easily write corrupt data into your primary store and replicate that corrupt data into backup.
Thanks to Prabhakar Ragde, Tom Murphy, Jay Weisskopf, Leah Hanson, and Ralph Corderoy for discussion/comments/corrections. Also, thanks (or maybe anti-thanks) to Leah for convincing me that I should write up this off the cuff verbal comment as a blog post. Apologies for any errors, the lack of references, and the stilted prose; this is basically a transcription of half of a conversation and I haven’t explained terms, provided references, or checked facts in the level of detail that I normally do.
One of the funnier examples I can think of this, at least to me, is the magical self-healing fuse. Although there are many implementaitons, you can think of a fuse on a chip as basically a resistor. If you run some current through it, you should get a connection. If you run a lot of current through it, you’ll heat up the resistor and eventually destroy it. This is commonly used to fuse off features on chips, or to do things like set the clock rate, with the idea being that once a fuse is blown, there’s no way to unblow the fuse.
Once upon a time, there was a semiconductor manufacturer that rushed their manufacturing process a bit and cut the tolerences a bit too fine in one particular process generation. After a few months (or years), the connection between the two ends of the fuse could regrow and cause the fuse to unblow. If you’re lucky, the fuse will be something like the high-order bit of the clock multipler, which will basically brick the chip if changed. If you’re not lucky, it will be something that results in silent data corruption.
I heard about problems in that particular process generation from that manufacturer from multiple people at different companies, so this wasn’t an isolated thing. When I say this is funny, I mean that it’s funny when you hear this story at a bar. It’s maybe less funny when you discover, after a year of testing, that some of your chips are failing because their fuse settings are nonsensical, and you have to respin your chip and delay the release for 3 months. BTW, this fuse regrowth thing is another example of a class of error that can be mitigated with ECC.
This is not the issue that Google had; I only mention this because a lot of people I talk to are surprised by the ways in which hardware can fail.↩
In case you don’t want to dig through the whole book, most of the relevant passage is:
In a system that can tolerate a number of failures at the software level, the minimum requirement made to the hardware layer is that its faults are always detected and reported to software in a timely enough manner as to allow the software infrastructure to contain it and take appropriate recovery actions. It is not necessarily required that hardware transparently corrects all faults. This does not mean that hardware for such systems should be designed without error correction capabilities. Whenever error correction functionality can be offered within a reasonable cost or complexity, it often pays to support it. It means that if hardware error correction would be exceedingly expensive, the system would have the option of using a less expensive version that provided detection capabilities only. Modern DRAM systems are a good example of a case in which powerful error correction can be provided at a very low additional cost. Relaxing the requirement that hardware errors be detected, however, would be much more difficult because it means that every software component would be burdened with the need to check its own correct execution. At one early point in its history, Google had to deal with servers that had DRAM lacking even parity checking. Producing a Web search index consists essentially of a very large shuffle/merge sort operation, using several machines over a long period. In 2000, one of the then monthly updates to Google’s Web index failed prerelease checks when a subset of tested queries was found to return seemingly random documents. After some investigation a pattern was found in the new index files that corresponded to a bit being stuck at zero at a consistent place in the data structures; a bad side effect of streaming a lot of data through a faulty DRAM chip. Consistency checks were added to the index data structures to minimize the likelihood of this problem recurring, and no further problems of this nature were reported. Note, however, that this workaround did not guarantee 100% error detection in the indexing pass because not all memory positions were being checked—instructions, for example, were not. It worked because index data structures were so much larger than all other data involved in the computation, that having those self-checking data structures made it very likely that machines with defective DRAM would be identified and excluded from the cluster. The following machine generation at Google did include memory parity detection, and once the price of memory with ECC dropped to competitive levels, all subsequent generations have used ECC DRAM.↩
If you take the most optimistic FIT rate, .057, and do the calculation for a server without much RAM (here, I’m using 128GB, since the servers I see nowadays typically have between 128GB and 1.5TB of RAM)., you get an expected value of .057 * 1000 * 1000 * 8760 / 1000000000 = .5 faults per year per server. Note that this is for faults, not errors. From the graph above, we can see that a fault can easily cause hundreds or thousands of per month.↩