Which error did that customer hit?

A customer says the product broke. Tonight, three proofs, in a console you administer. Name the error on that customer's request: the type or message the log stored, the time, and the release it belongs to. Show a different customer's request that did not record that error. Then show an error from an older release that is not counted as this release, or write that the log cannot name a release. If the name, the other customer, or the release requires the person who built the product, you have not seen the error. You have a count.

This is one customer and one release. It is not the price they were shown. Show the price they accepted is the screen, the amount taken, and a reversal aimed at one row. A charge can be wrong while the page never threw. It is not the sign-in. Show the login they did not make names an authentication, a refusal, and a different account that still gets in. A failed password is not a stack trace from the page they were on. It is not the second factor on the account. Show the factor they enrolled names the factor, a sign-in that does not present it, and a different account's factor that still works. An exception in the checkout is not an authenticator. Ending the session they still hold does not tell you which line threw.

None of what follows decides whether a statute covers your product, and none of it is legal advice. The papers below are the clearest public counts of what a failure log actually contained, and of what a bucket of error reports is. They do not tell you which exception your customer hit. They do not replace the log you can open. They do tell you the shape of a proof a founder can perform this week, which is a product question.

Yuan, Luo, Zhuang, Rodrigues, Zhao, Zhang, Jain, and Stumm, at the University of Toronto, studied user-reported failures of five systems and published the study at OSDI 2014 (Yuan et al., Simple Testing Can Prevent Most Critical Failures). The sample is 198 randomly selected failures. The population they sampled from is 17,216 reported failures. 198 is not 17,216. The five systems are Cassandra, HBase, HDFS, Hadoop MapReduce, and Redis. Table 1 prints reported, sampled, and catastrophic: Cassandra 3,923 / 40 / 2, HBase 5,804 / 41 / 21, HDFS 2,828 / 41 / 9, MapReduce 3,469 / 38 / 8, Redis 1,192 / 38 / 8. The reported column adds to 17,216. The sampled column adds to 198. The catastrophic column adds to 48. 48 of 198 is the table's total row of 24%. That 24% is not the abstract's claim about catastrophic failures.

The claim people quote is a different cut. Almost all (92%) of the catastrophic failures are the result of incorrect handling of non-fatal errors explicitly signaled in software. 92% is a share of the 48, not a share of the 198, and not a share of the 17,216. In 58% of the catastrophic failures, the underlying faults could have been detected by simple testing of error-handling code. In 35%, the faults fall into three trivial patterns: an empty handler or a handler that only logs, an abort on an overly general exception, and a handler whose comment says FIXME or TODO. Another 23% were wrong enough that statement coverage or a careful review would have caught them. 35 and 23 are the two pieces of that 58. Do not add 35, 23, and 58. Figure 5 prints the same catastrophic failures a second way: 92% incorrect handling, of which 35% are the trivial mistakes and 57% are system-specific, plus 8% that are not that incorrect-handling slice. 35 plus 57 is 92. 57 is not 58. Inside the trivial 35: errors ignored 25%, abort on an over-caught exception 8%, TODO in the handler 2%. 25, 8, and 2 add to 35. Inside the system-specific 57: easily detectable 23% and complex bugs 34%. That 23% is the introduction's other 23%. 34 is not 35.

Table 2 is symptoms, and its percents are not shares of the 48. Unexpected termination 74, of which 17 (23%) are catastrophic. Incorrect result 44, of which 1 (2%). Data loss or potential data loss 40, of which 19 (48%). Hung system 23, of which 9 (39%). Severe performance degradation 12, of which 2 (17%). Resource leak or exhaustion 5, of which 0 (0%). The left column adds to 198. The catastrophic counts add to 48. The right-hand percents are catastrophic failures divided by that symptom's row, and the paper says so. 17 of 74 is the 23% on the termination row. 19 of 40 is printed as 48%. 9 of 23 is 39%. Those percents do not add to 100, and they do not add to the total row's 24%. The 48% on the data-loss row is not the 48 catastrophic failures. The 23% on the termination row is not the 23% of catastrophic failures that were easily detectable, and it is not the 23% of all failures that need only one input event.

Four different 23s sit in this paper, and none of them is a setting in your console. Table 3: 23% of failures need a single input event, 50% need two, 17% need three, 5% need four, and 5% need more than four. Those five add to 100. Finding 1 says a majority (77%) need more than one input event, and most (90%) need no more than three. 77 is 100 minus the single-event 23. 90 is 23 plus 50 plus 17. 77 is not 90. Table 4 says configuration changes are an input in 23% of failures, and of those, 30% involve misconfigurations. 30% of that 23% is not 30% of 198, and the paper does not print the product. The remaining majority of configuration-change failures are valid changes that turned a feature on. A valid change is not a misconfiguration. Table 4's other inputs sum past 100 because most failures need more than one event: starting a service 58%, a client write 32%, an unreachable node 24%, adding a node 15%, a client read 13%, an intentional restart 9%, data corruption 3%, other 4%. The 58% that start a service is not the 58% of catastrophic failures detectable by simple testing.

The same integers come back as different measurements. Finding 7 says that for a majority (84%) of the failures, all of the triggering events are logged. Table 5's cumulative share of failures reproducible on two nodes is also 84% of all failures and 86% of catastrophic failures. The log 84% is not the two-node 84%. Table 5 is a cumulative distribution, not a bin: one node 37% of all failures and 43% of catastrophic, two nodes 84% and 86%, three nodes 98% and 98%, more than three 100% and 100%. 84 is not 84 minus 37. 98% on three nodes is not 100%. The introduction's "3 or fewer nodes" is that 98%, not a guarantee about the last 2 points, and the paper prints 98 and 100 rather than a remainder. 37% that fit on one node is not the 35% trivial error-handling pattern.

Two findings both print 77%, and they are not the same finding. Finding 1's 77% is failures that need more than one input event. Finding 9's 77% is failures reproducible by a unit test, 153 of 198, from Table 8. 153 is not the event-count 77. Table 8's rows add: Cassandra 73% (29 of 40), HBase 85% (35 of 41), HDFS 82% (34 of 41), MapReduce 87% (33 of 38), Redis 58% (22 of 38). 29 plus 35 plus 34 plus 33 plus 22 is 153. Redis's 58% is 22 of 38. It is not the 58% of catastrophic failures, and it is not the 58% of failures that start a service. They also manually reproduced 73 of the failures. 73 is not 73%, it is not 153, and it is not the 77%.

Table 6 is a third cut, and its arithmetic does not close. Deterministic failures: Cassandra 76% (31 of 41), HBase 71% (29 of 41), HDFS 76% (31 of 41), MapReduce 63% (24 of 38), Redis 79% (30 of 38), total 74% (147 of 198). The five numerators add to 145. The total line prints 147. 145 is not 147. Cassandra's denominator on this table is 41. Table 1 sampled 40 Cassandra failures, and Table 8's Cassandra row is 29 of 40. 41 is not 40. 74% deterministic is not 76% with an explicit failure message, and it is not 77% reproducible by a unit test. The non-deterministic remainder in Table 7 is 51, which is 198 minus 147, not 198 minus 145. Timing 27 (53%), atomicity violation 13 (25%), deadlock 3 (6%), lock contention 4 (8%), other 4 (8%). Those counts add to 51. Those percents add to 100. 51 is not 48 catastrophic failures.

What the log contained is Finding 6, Finding 7, and Finding 8, and they are three sentences. 76% of the failures print an explicit failure-related error message. 84% log all of their triggering events. The median number of log messages printed by each failure is 824. 824 is not 76 and not 84. A previous study the authors cite, on Apache httpd, PostgreSQL, SVN, squid, and GNU Coreutils, found explicit failure messages in 43% of failures. 43 is not 76. The paper's explanation for the gap is a surmise, not a second measurement. They also excluded 10 duplicated bugs from the original sample. 10 is not 48. The margin of error they print for 198 samples is 6.9% at 95% confidence. 6.9 is not a failure rate. Tickets are dated 2010 or later, and the priority field had to be Blocker, Critical, or Major. A minor ticket is not in the 198.

Aspirator is their checker, and its percents are not the 92. The abstract says over 30% of the catastrophic failures would have been prevented had Aspirator been used and the bugs fixed. The evaluation says 33% of the Cassandra, HBase, HDFS, and MapReduce catastrophic failures could have been prevented. Redis is C, and Aspirator is a Java bytecode checker, so Redis's 8 catastrophic failures are outside that 33% sentence. 33 is not 30, and over 30% is not 35%. The introduction says Aspirator identified 121 new bugs and 379 bad practices. 121 plus 379 is 500, which is the later sentence's "500 new bugs and bad practices." 115 false positives are beside that 500, not inside it. FindBugs, already running on these systems, checks for over 400 rules. 400 is not 500. They reported 171 bugs and bad practices to developers. 143 have been confirmed or fixed: 73 fixed and 70 confirmed. 73 plus 70 is 143. 17 were rejected. The paper says the others have not received a response, and it does not print how many others that is. The 73 that were fixed are not the 73 failures the authors reproduced by hand. Analysis finishes within 15 seconds per system, with a memory footprint under 1.2 GB, on one laptop. 15 seconds is not 15 nodes. 1.2 GB is not 1.2%.

A count of reports is not a count of customers, and the second paper is about that distinction. Glerum, Kinshumann, Greenberg, Aul, Orgovan, Nichols, Grant, Loihle, and Hunt, at Microsoft, describe Windows Error Reporting at SOSP 2009 (Glerum et al., Debugging in the (Very) Large). An error, in their definitions, is a single event. A bug is a root cause that results in one or more errors. A bucket is a collection of error reports likely caused by the same bug. The ideal they name is one bug per bucket and one bucket per bug, and they write that they know of no such algorithm. Labeling runs on the client. Classifying runs on the server. Most error reports consist of no more than a bucket identifier. A dashboard that shows 500 is not 500 customers until you can open one report and name one person, and this paper is the reason not to pretend otherwise. The installed base in the abstract is a billion machines, and the reports collected over ten years are billions. A billion machines is not a billion reports. One pair of database servers, later described as one pair of SQL servers, records the errors. One pair is not twenty front-end servers, and it is not six query servers.

The Vista sentence is easy to add and must not be added. Programmers found and fixed over 5,000 bugs isolated by WER in Vista beta releases, after they had found and fixed over 100,000 bugs with static analysis and model checking, and before general release. The 5,000 are not inside the 100,000. The 100,000 were already fixed when the 5,000 were found. 100,000 lines of code in !analyze is a third number: roughly 100,000 lines implementing some 500 bucketing heuristics, with roughly one heuristic added per week. 500 heuristics is not 500 reports, and 100,000 lines is not 100,000 bugs. 47% of kernel crash reports result in a direction to a software update. 47% is kernel crashes, not all reports, and it is not Yuan's 48 catastrophic failures. A hang, in their trigger list, is a failure to process user input for 5 seconds. 5 seconds is not 5,000 bugs. One correlation they print: 96% of the computers faulting in a Windows event-tracing bug were running one third-party driver, and that driver's market share on all other error reports was well below 96%. 96% of the faulting computers is not a 96% market share. They ran one hypothesis across 10,000 memory dumps. 10,000 dumps is not 10,000 customers.

Bucketing misses in both directions, and the Office table is an upper bound they are explicit about. For Office 2010 internal testing, the share of reports in a second bucket plus the share in a one-hit bucket: Excel 17.2% and 4.4%, combined 21.6%. Outlook 7.7% and 10.0%, combined 17.7%. PowerPoint 30.6% and 6.2%, combined 36.8%. Word 19.3% and 8.8%, combined 28.1%. Each combined cell is the sum of its two neighbors. The prose says as many as 37% for PowerPoint. 36.8 is not 37, and 37 is not a fifth product. One-hit buckets are their assumption that a real bug is hit more than once, so a bucket with one report is counted as a miss. That is an assumption, not a proof that the single report was wrong. Figure 18 ranks labeling heuristics on a sample of 300 million error reports: module offset 67.95%, module name 9.10%, program name 5.62%, program version 4.22%, program timestamp 1.90%, exception code 1.02%, module timestamp 0.89%, module version 0.34%, and all other labeling heuristics 8.96%. Those nine add to 100. 67.95 is not 96. The first stage of their four-stage protocol stops in over 90% of error reports, because the service already has enough data and only the bucket label was sent. Over 90% is not 92%, and a label is not a minidump.

Do the three proofs on one real report, in the tool that stored it. First, the error. Open the record for that customer: the type or message, the time, and the release or version the event carries. A chart of "errors today" with no release and no customer is a total. Second, the other customer. A different account's request in the same window did not record that error. If the tool cannot show you one customer without showing you everyone's stack as the way to find it, write that, and do not tell the first customer you have isolated theirs. Third, the release. An error whose version is the previous release is not evidence that this release throws. If the log has no version, the sentence you are allowed is that you cannot name the release, not that the release they are on is the cause. Asking a model to explain the screenshot is not one of the three. The model did not store the event.

Three moves feel like the proof and are a second problem. Restarting the process, because the count dropped, while the record was deleted with it. Grouping every exception into one bucket because the tool you were given has no release field, and then describing the bucket as the bug. Reading the paper's 92% to the customer as if it were the share of their session that was mishandled. The first destroys the record. The second collapses the other customer's event into theirs. The third is a fraction from five storage systems in a 2014 sample, and it is not a rate on your invoice. A price they accepted is show the price they accepted. The amount is not the stack. Which environment a deploy reached is which environment got that change. A release name is not an environment name. The prompt a model was sent is which prompt did that request send. An exception is not a system prompt.

The path is ordinary when an admin can open one person's record without opening the next person's stack as the method. Gigbase is a multi-tenant workspace with subscription plans, role-based access for agencies, admins, and team members, and a dedicated admin dashboard. The shape that matters here is an admin, in an account you administer, who can open one person's record. That case is not an error ledger. Nothing in the description is a headcount, a duration, or a result figure. If the only control on the admin screen deletes the tenant, you do not have this proof. You have a kill switch.

There is a version of this page where hiring the path is the wrong purchase. Nothing is in production, because the product does not have customers yet. You have no product. You are still choosing who will build, and the document in front of you is a term sheet. The criteria for that choice sit on the page written for a top saas development company. That page is criteria. It does not rank anyone, including us. The only fact in the room is 92%, or 47%, or 76%, and you wanted that number to be your incident rate. It is not. A customer has already seen another customer's data. Naming one exception does not close a leak. Or you wanted a lawyer. Hire the lawyer.

When the report is real and the console cannot show one customer, the missing object is specific. An admin action, in an account you administer, that shows one error, the release it belongs to, a different customer who did not hit it, and an older release that is not this one. A startup software development company that expects this will have put that action next to the rest of account admin, before a chart of totals is the way you discover the vendor holds the log. One that needs a call in order for you to see which customer threw is telling you who holds the record. If the first slice is a fixed capability, an AI integration sprint is the shape where the acceptance sentence can include the three proofs. If you do not yet know which system records the exception, a two-week AI readiness assessment maps the systems. The findings are yours. They are not a rate.

Before you tell them you know what broke, do the three proofs and write the time. Name the error and the release. Show a different customer who did not hit it. Show an older release that is not counted as this one, or write that the log has no release. Do not read a bucket count as a customer count, and do not ask a model to remember a stack. A bucket is a hypothesis that reports share a cause. The record you can open is the page you can defend.