Problem Management in ISP Operations: From Reactive to Proactive

Incident management and problem management are different disciplines that address different questions. Incident management asks: how do we restore service as quickly as possible? Problem management asks: why does this keep happening, and how do we stop it from happening again?

Most ISP operations are reasonably good at incident management once the NOC is built and the escalation paths are functioning. Almost none practice problem management formally, which is why the same incidents recur, the same subscribers call about the same issues repeatedly, and engineers spend a significant portion of their time solving problems they have solved before.

The Distinction That Matters

An incident is a specific service disruption event. A problem is the underlying cause of one or more incidents. The distinction matters because the resolution of an incident, restoring service, does not address the problem. A BGP session that flaps every three weeks due to an undersized hold timer gets resolved repeatedly as an incident while the problem, the misconfiguration, generates another incident three weeks later.

Problem management is the process of identifying patterns in incidents, investigating root causes, and implementing permanent fixes rather than repeated temporary restorations. It requires a different operational mindset than incident management: it is deliberate, scheduled, and operates on a longer timescale than the urgency of incident response.

Identifying Problems from Incident Patterns

The input to problem management is incident data. Review closed incident records monthly and look for patterns: incidents that recur on the same infrastructure, incidents affecting the same subscriber or subscriber group repeatedly, incidents with similar symptoms or root causes, and incidents that take longer to resolve than similar incidents elsewhere in the network.

A recurring incident is the clearest signal of an underlying problem: if the same equipment is generating the same alert and requiring the same intervention more than twice in a month, there is a problem that incident management alone is not solving.

For ISPs with NOC monitoring platforms, this analysis is supported by the alert history: which hosts generate the most alerts over time, which alert types appear most frequently, and which infrastructure elements appear in the most incident tickets. NOC Intelligence provides this pattern analysis across the alert stream, surfacing the recurring patterns that indicate underlying problems.

Root Cause Analysis

When a problem is identified, root cause analysis (RCA) determines why the underlying condition exists rather than just what the symptom is. A structured RCA approach uses the five-whys technique: ask why the incident occurred, then why that condition existed, then why that condition was not detected or prevented, and so on until you reach a root cause that is addressable.

For a link that flaps repeatedly: why did the link flap? The BGP session dropped. Why did the session drop? The hold timer expired before a keepalive was received. Why was the keepalive late? CPU utilisation on the remote router spiked during a traffic event. Why did CPU spike? The router is undersized for the current traffic volume. Why is it undersized? The capacity planning trigger was not triggered before the router reached its limits. The root cause is the capacity planning process failing to catch the need for an upgrade, not the BGP configuration.

The Known Error Record

When a root cause is identified but not yet permanently fixed, a known error record documents the problem, its workaround, and the status of the permanent fix. This record serves two purposes: it prevents engineers from spending diagnostic effort re-investigating a problem that is already understood, and it tracks the outstanding permanent fixes that need resourcing and scheduling.

The known error record is a simple document or ticket: description of the problem, affected infrastructure, workaround currently in use, root cause identified, permanent fix required, responsible owner, and target implementation date. Review the known error record monthly to ensure permanent fixes are progressing rather than accumulating indefinitely.

Problem Management as CTDISR Evidence

CTDISR-2025 Section 5 (Incident Response) and Section 17 (Compliance Monitoring) both benefit from documented problem management activity. An auditor reviewing your incident management framework who sees evidence of root cause analysis and permanent fix tracking sees an organisation that is not just reacting to failures but learning from them, which is evidence of a maturing security and operations programme.

Post-incident reviews, which are the mechanism that feeds incident learnings into problem management, are explicitly consistent with CTDISR-2025's requirement for lessons learned documentation following significant incidents. For operators building the full operational and compliance documentation framework, CTDISR Audit Readiness covers how operational documentation contributes to the evidence package. For documenting problem management procedures and RCA templates as formal runbooks, RunBook AI generates structured documentation that the NOC team can use consistently.