Skip to content
Insights · Engineering

Which environment got that change?

Name the environment a change reached. Show a change that stayed elsewhere is not called production, and that another client did not receive it.

Scroll
EngineeringOct 20, 202611 min readBy Salman Naqvi, Founder & CEO
Which environment got that change?

A change is somewhere. The client heard production. Tonight, three proofs, in a repository or a deploy log the agency or the client can open. Name the environment the change reached, and the artifact that landed there: the environment's name, the commit or digest, and the time. Show a change that exists only on a different environment and is not described as this one. Then show a different client's environment that did not receive it, or write that the log cannot name a second environment. If the name, the other environment, or the other client requires a laptop you do not hold, you have not named where it went. You have a branch.

This is one change and one environment. It is not the ref the workflow runs. Name the action the workflow runs is the action, the uses ref, and whether that ref is a full SHA. A pinned action can deploy to the wrong environment, and a correct environment can run a floating tag. It is not the digest that is running. Name the commit they are running is the digest in production against the commit you shipped. A matching digest does not say whether staging also has it, or whether a second client received it. It is not the required check. Name the check the merge required is which checks can block a merge. A green check is not an environment name. It is not who was allowed to merge. Name who merged their change is the account that landed the commit. The account is not the environment the job was allowed to touch.

None of what follows decides whether a contract makes the agency the sender, and none of it is legal advice. The page and the paper below are the clearest public statements of what an environment gate actually checks, and of how often a multi-tenant platform's incidents were on the customer's side rather than the platform's. They do not name this client's environment. They do tell you the shape of a proof an agency can perform before it tells a client the change is live.

GitHub's documentation of deployment protection rules is the console language, and the integers on that page are small enough to swap (GitHub, Deployments and environments). Any number of GitHub Apps-based protection rules can be installed on a repository. A maximum of 6 can be enabled on any one environment at the same time. 6 enabled is not "any number" installed. You can list up to six users or teams as reviewers. Only one of the required reviewers needs to approve the job for it to proceed. Six listed is not six approvals. One approval is not one reviewer on the list. If you turn on the setting that prevents self-review, the user who started the deployment cannot approve it, even if they are a required reviewer. A person who is both the author and the only reviewer is a different failure from a list of six.

The wait timer is a third control, and its unit is minutes. The time must be an integer between 1 and 43,200 (30 days). 43,200 is the minute count the page equates to 30 days. A timer set to 30 is 30 minutes, not 30 days, unless you typed 43,200. 1 is the minimum, not one day. Wait time does not count toward billable time. A delay is not a reviewer, and it is not an approval. Required reviewers and wait timers, on GitHub Free, Pro, or Team, are only available for public repositories. A private repository on those plans does not gain the gate by being called production. Deployment branches and tags are a different availability sentence: all public repositories, and private repositories on Pro or Team. Do not read one plan note onto the other control.

Branch rules have a hole that looks like a lock. The options are no restriction, protected branches only, and selected branches and tags. Protected branches only means branches with protection rules. If no branch has a protection rule, all branches can deploy. The restrictive label, with an empty protection list, is not a restriction. Selected names are matched against GITHUB_REF. The page's example is releases/*, which matches a ref whose name begins with releases/. A rule for refs/pull/*/merge also allows workflows triggered by pull_request. Wildcard characters will not match a slash. release/* does not match a name with a second slash. release/*/* does. A star is not a slash. Administrators can bypass protection rules by default. You can configure the environment so they cannot. "By default" is not "cannot." Custom protection rules are in public preview on that page. A preview is not a rule you have enabled. Environment secrets are available only to jobs that reference the environment, and if the environment requires approval, the job cannot read those secrets until one required reviewer approves. A job that does not name the environment does not get the secret. Self-hosted runners are not an isolated container even when the job uses an environment. The secret on a self-hosted runner is the same class of secret as a repository secret, which is the page's own warning. Environment variables are a different object, read through the vars context, not through secrets.

Those sixes are not a study of incidents. The study is Zhou, Lou, Zhang, Lin, Lin, and Qin, at Microsoft, on 210 real escalations from an anonymized company-wide platform they call ProductA (Zhou et al., An Empirical Study on Quality Issues of Production Big Data Platform). A footnote says the survey was conducted in 2012 and may not reflect the current platform. 2012 is not 210. ProductA serves thousands of customers from hundreds of teams, on hundreds of thousands of commodity servers, with tens of thousands of jobs a day. Those are scale sentences. They are not the sample. The 210 escalations were discussed in 2,196 emails and 188 incident records. 2,196 is not 210. 188 is not 210. The paper does not print the size of the gap between those two counts, so this page will not invent it.

Causes, in their Table II, are the cut that matters for "whose environment." Hardware faults 44 (21.0%). System-side faults 76 (36.2%): code defects 44 (21.0%), design limitations 20 (9.5%), operation faults 12 (5.7%). Customer-side faults 78 (37.2%): code defects 10 (4.8%), operation faults 21 (10.0%), misuse 47 (22.4%). Unknown, missing evidence, 12 (5.6%). 44 plus 76 plus 78 plus 12 is 210. 21.0 plus 36.2 plus 37.2 plus 5.6 is 100. Hardware's 44 and system code-defect's 44 are both 21.0%, and they are different rows. Do not add them into 88 of one kind. System-side 36.2% and customer-side 37.2% are close. 36.2 is not 37.2. A sentence that says "the platform" or "the client" without opening the record is choosing between two shares the paper could not tell apart by size. 94.4% were classifiable. 94.4 is 100 minus 5.6, and it is not 37.2.

Misuse is not the whole customer side. 47 (22.4%) is misuse, and it breaks into incorrect client configuration 15 (7.1%), improper job parameters 9 (4.3%), and improper system assumptions 23 (10.9%). 15 plus 9 plus 23 is 47. The percents 7.1, 4.3, and 10.9 add to 22.3, and the row prints 22.4, which is what rounding does. 22.3 is not a second measurement. Customer operation faults are 21 (10.0%), a different 21 from hardware's machine-outage prose. Customer code defects are 10 (4.8%). 10 is not 15. The paper's surprise is explicit: 37.2% (78) were still customer-side even though customers escalate when they think the problem is not theirs. 78 is not a reason to tell your client the failure is theirs. It is a reason to show which side the change was on.

Symptoms are a different table, and the counts do not close on 210. Connection failure 18 (8.6%). Job slowdown 57 (27.1%). Job failure 95 (45.2%). Wrong result 18 (8.6%). Service unavailable 12 (5.7%). Those five counts add to 200. 200 is not 210. The percents are shares of 210: 95 of 210 is the 45.2%, and 57 of 210 is the 27.1%. This page will not invent a sixth symptom row for the escalations those five rows do not cover. Job failure 45.2% is a symptom. Customer-side 37.2% is a cause. 45.2 is not 37.2. The two 18s, connection failure and wrong result, are both 8.6% and are not the same 18. Service unavailable prints 12 (5.7%). The system-side prose also prints service outages as 5.7% (12), of which 83.3% (10 of 12) were in the front end, 8.3% (1 of 12) in an extent node, and 8.3% (1 of 12) in the compiler. 10 plus 1 plus 1 is 12. Those front-end outages happened at only three time points. 3 is not 10. Identical pairs of 12 and 5.7% are not, by themselves, proof that the symptom row and the cause sentence are the same twelve incidents.

Severity is a fifth cut. Very high 7%, high 37%, medium 30%, low 22%, very low about 4%. The paper prints "about 4%," so do not force the five to 100. Very high must be engaged within 15 minutes and mitigated within 1 hour. 15 minutes is not 1 hour. Very low can wait a release cycle. A release cycle is not 1 hour. 37% at high severity is not the 37.2% customer-side cause. The 0.2 is the difference, and it is also a different table. Fault-tolerance cost is a sixth cut, from 40 weeks of traces, not from the 210: about 0.9% to 2.6% of time wasted by fault tolerance. About 42% to 75% of that wasted time is vertex invocation, and 9% to 20% is storage read errors. 40 weeks is not 210 escalations. 0.9 is not 21.0. The paper says more than half of escalations are short-lived and can be mitigated by resubmitting the job. More than half is not 45.2%, and it is not 37.2%.

Numbers in the related-work section are other papers, and they are not ProductA. Yin and colleagues studied 546 misconfigurations. Gray found administrator errors in 42% of failures in high-end mainframes. Patterson and colleagues reported more than 50% of failures in telephone networks and Internet systems due to operator errors. Vishwanath and Nagappan found 8% of servers had at least one hardware incident in a year. Benson and colleagues examined 8,684 forum problems, with a delay of 20 to 110 hours. Chou and colleagues found bugs remaining in the Linux kernel an average of 1.8 years. 546 is not 210. 42% is not 37.2%. 8% of servers is not 21.0% of escalations. 20 hours is not 15 minutes. 1.8 years is not a wait timer. Quote them as the paper's citations, or do not quote them.

Do the three proofs on one real deployment, in the tool that ran it. First, the environment. Open the job and read the environment name, the artifact, and the time. A branch called main is not an environment called production. Second, the change that did not arrive. A commit that is on staging, or on a preview, is not described as the production deploy. If the log has no second environment, write that you cannot show the absence, and do not tell the client the change is "only" in production. Third, the other client. A different client's environment did not receive this artifact. If the agency runs one shared cluster and cannot show a boundary, write that, and do not say the clients are isolated. A screenshot of a workflow file with no run beside it is a definition. The run is the proof.

Three moves feel like the proof and are a second problem. Renaming the branch, because the client said production and the branch is now called that, while the job still targets the old environment. Turning on "protected branches only" when no branch is protected, and describing the empty list as a lock. Reading 37.2% to the client as if their incident had been classified. The first is a label. The second is the documentation's own hole. The third is 78 of 210 escalations on an anonymized platform in a 2012 study, and it is not this deploy. Which error one customer hit is which error did that customer hit. An exception is not an environment. The prompt a request sent is which prompt did that request send. A system prompt is not a deployment target.

The path is ordinary when a deploy record names an environment without using the next client's data as the way to prove the boundary. Inflectiv Helios is a multi-assistant platform with prompt monitoring, dashboards, and a containerized deployment. The shape that matters here is a deployment that can be told apart from another run of the same system. That case is not an environment ledger. Nothing in the description is a headcount, a duration, or a result figure. If the only control promotes every environment at once, you do not have this proof. You have a single switch.

There is a version of this deploy where hiring the path is the wrong purchase. The work is not sold under the agency's name, because there is no client yet. The client asked to meet the people who will write the code, and a subcontract cannot answer that. The room only has a slide of an architecture. The buyer only wants a template of a workflow file. Or the buyer wants a legal opinion on who is liable if the wrong environment was updated. The criteria for choosing a firm sit on the page written for a top saas development company. That page is criteria. It does not rank anyone, including us. Hire the lawyer for the liability question.

When the change is real and the log cannot name an environment, the missing object is specific. A job record, in an account the agency or the client can open, that names the environment and the artifact, shows a change that stayed on a different environment, and shows another client's environment unchanged. A white label software development partner that expects this will put the repository and the environments in the client's organization, or in the agency's with a login the client can be shown, before the first deploy. One that needs a call in order for you to see where a change went is telling you who holds the environment. If the first slice is a fixed capability, an AI integration sprint is the shape where the acceptance sentence can include the three proofs. If you do not yet know which accounts the current work already deploys into, a two-week AI readiness assessment maps the systems. The findings are yours. They are not a percentage from ProductA, and they are not a GitHub plan.

Before you tell the client the change is live, do the three proofs and write the time. Name the environment and the artifact. Show a change that did not reach it. Show another client that did not receive it, or write that the log has no second environment. Do not rename a branch and call it an environment, and do not read a platform-wide percentage as this deploy. The run, with the environment on it, is the page you can defend.

Questions

Three records, in a repository or a deploy log the agency or the client can open. The environment name and the artifact that landed there. A change that exists only on a different environment, not described as this one. A different client's environment that did not receive it. If you cannot open the record, write that.

No. GitHub lets an environment list up to six users or teams as reviewers, and only one of those reviewers has to approve. Six names is not six approvals. A wait timer is a separate control, measured in minutes, and it is not a reviewer.

When the work is not sold under the agency's name, when the client asked to meet the builders, when the room only has a slide, when the buyer only wants a template, or when the buyer wants a legal opinion.

Find this useful? Tell Google to show you more of it.

Let's put AI to work in your business.

A 30-minute call. You bring the workflow or the roadmap — we'll tell you what's feasible, what it costs, and what we'd build first.

Book a call→