Most ISP network outages are caused by changes. Not infrastructure failures: changes. A firewall rule added without testing that blocks legitimate subscriber traffic. A BGP configuration update that inadvertently withdraws a prefix. A firmware upgrade that introduces a regression in CGNAT behavior. The pattern is consistent enough to have a name in operations circles: change-induced failure.
Change management is the set of procedures designed to reduce the probability of change-induced failures by requiring planning, review, testing, and rollback preparation before changes are applied to production infrastructure. It does not eliminate risk, but it converts unplanned risk into managed risk.
The Change Categories
Not all changes carry equal risk and not all changes warrant the same process overhead. Defining change categories allows the process to be proportionate:
Standard changes are pre-approved, low-risk, routine changes with a documented procedure. Adding a subscriber account, updating a DNS record, replacing a failed physical port with a spare configured identically. These changes are executed by any trained engineer following the documented procedure without additional approval. The record is the procedure log or ticket.
Normal changes are any change that is not standard or emergency. They require a change request, review by a second engineer or the NOC lead, approval before implementation, and a scheduled maintenance window. Normal changes are the category that generates most of the process overhead and most of the value: this is where the firewall rule that blocks subscriber traffic gets caught in peer review before it reaches production.
Emergency changes are changes required to restore service during an active incident, where the normal review and approval timeline is not compatible with minimising subscriber impact. Emergency changes are implemented with the approval of whoever is the most senior available person in the moment, documented immediately after implementation, and reviewed in a post-implementation review within 48 hours.
The Change Record
Every change, standard, normal, or emergency, should produce a record. The minimum information: who requested the change, who approved it, what exactly was changed (configuration extract or description specific enough to reverse the change if needed), when it was implemented, what the rollback procedure is, and what the outcome was.
For normal changes, the rollback procedure must be documented before the change is approved, not after it is implemented and has gone wrong. "We can roll back by reverting to the previous config" is not a rollback procedure. The specific commands or steps needed to restore the previous state, verified against the actual pre-change configuration, is a rollback procedure.
This record serves two purposes: it enables diagnosis when a change-induced problem is reported (the first question in any incident is "what changed recently?"), and it constitutes CTDISR-2025 audit evidence. Section 9's network security controls and Section 17's compliance monitoring both implicitly require that network changes are tracked, authorised, and documented. An auditor who asks for evidence that network changes are controlled and documented should receive a change log, not a blank look.
Maintenance Windows
Planned maintenance that involves temporary service disruption should occur during defined maintenance windows when subscriber impact is minimised. For a residential ISP, the lowest-impact window is typically late night to early morning on weekdays. For a corporate CIR ISP, the window must be negotiated with each client and documented in the SLA.
Advance notice to subscribers and corporate clients before maintenance windows is both a customer relationship practice and, for some operators, a regulatory obligation. Maintain a record of maintenance notices sent: the notice date, the maintenance date and time, and the subscriber groups or clients notified. This record is evidence of operational discipline and provides context when a subscriber complaint about a planned maintenance window arrives after the fact.
Post-Implementation Review
For significant normal changes and all emergency changes, a brief post-implementation review (a 15-minute conversation or a short written summary) documents: did the change achieve its intended outcome, were there any unexpected effects, was the rollback procedure needed and if so how did it work, and what should be done differently next time. This review is how change management improves over time rather than being a static procedure applied uniformly regardless of what the change experience revealed.
For operators who need change management procedures documented as runbooks that operations staff can follow consistently, RunBook AI generates structured change management procedures from descriptions of current practice. For the broader NOC operations build including change management as an integrated operational process, NOC Enablement & Monitoring covers the full design. For the CTDISR evidence audit trail that change records contribute to, CTDISR Audit Readiness includes operational documentation review as part of gap assessment.