The national regulator, Ofcom, has this morning hit BT with a fine of £17.5m for being “ill-prepared to respond to a catastrophic failure of its emergency call handling service last summer” (here), which readers may recall occurred after a technical fault (“complex software issue”) resulted in 14,000 emergency calls being disrupted. The disruption lasted for 10.5 hours.
Just to recap. BT previously revealed that a software bug had caused a “caching issue“, which resulted in impacted calls (i.e. those made to both 999 and 112) not being routed correctly and the user’s call being disconnected. At the time, a “robust temporary fix” was put in place to rectify this, and they’ve since implemented something more permanent.
The incident, the first in nearly 90 years, has already sparked the introduction of further safeguards to help “bolster the resilience” of the 999 emergency call handling system (here). But Ofcom has also continued to conduct its own investigation, which has now concluded.
According to the regulator, on Sunday 25th June 2023, BT experienced a network fault that affected its ability to connect calls to emergency services between 06:24 and 16:56. During the incident, nearly 14,000 call attempts – from 12,392 different callers – were unsuccessful.
The Three Stages of BT’s 999 Fault
Phase 1, from 06:24 to 07:33
During the first hour, BT’s emergency call handling system was disrupted by what was later found to be a configuration error in a file on its server. This resulted in call handling agents’ systems restarting as soon as a call was received; agents being logged out of the system; calls being disconnected or dropped upon transfer to the emergency authorities; and calls being put back in the queue. BT was initially unable to determine the cause of the issue and attempted to switch to its disaster recovery platform.
Phase 2, from 07:33 to 08:50
The first attempt to switch to the disaster recovery platform was unsuccessful due to human error. This was a result of instructions being poorly documented, and the team being unfamiliar with the process. The incident grew from affecting some calls to a total outage of the system.
Phase 3, from 08:50 to 16:56
The rate of unsuccessful calls decreased once traffic was migrated successfully to the disaster recovery platform. However, usual service was not fully restored initially as the disaster recovery platform struggled with demand.
Ofcom’s investigation found that BT “did not have sufficient warning systems in place” for such incidents, “nor did it have adequate procedures for promptly assessing the severity, impact and likely cause of any such incident or for identifying mitigating actions“. The regulator also found that BT’s “disaster recovery platform had insufficient capacity and functionality” to deal with a level of demand that might reasonably be expected.
The incident also caused disruption to text relay calls, which meant people with hearing and speech difficulties were unable to make any calls, including to friends, family, businesses and services. This left deaf and speech-impaired users at increased risk of harm.
Suzanne Cater, Ofcom’s Director of Enforcement, said:
“Being able to contact the emergency services can mean the difference between life and death, so in the event of any disruption to their networks, providers must be ready to respond quickly and effectively.
In this case, BT fell woefully short of its responsibilities and was ill-prepared to deal with such a large-scale outage, putting its customers at unacceptable risk.
Today’s fine sends a broader warning to all firms -– if you’re not properly prepared to deal with disruption to your networks, we’ll hold you to strict account on behalf of consumers.”
Despite the fact that no confirmed cases of “serious harm” have been identified as a result of the incident, Ofcom correctly makes clear that the potential degree of harm was still “extremely significant” and hence the large fine. The fine includes a 30% reduction as a result of BT’s admission of liability, full cooperation, willingness to engage, system improvements, regular updates and agreement to settle the case.
BT now has two months to pay the fine, which will then be passed on to HM Treasury. But the hope is that something like this will never happen again. We have contacted BT for a comment.