Hardware vendor differences led to Rogers outage, says Rogers CTO

Article content

Rogers chief executive officer Tony Staffieri today appeared alongside Ron McKenzie, the company’s newly appointed chief technology officer, before the House of Commons Standing Committee on Industry and Technology to answer questions from regulators regarding its network outage that affected millions of Canadians on July 8.

Advertisement 2

Article content

Rogers CEO Tony Staffieri (left) and Rogers CTO Ron McKenzie (right) appeared before regulators from the House of Commons Standing Committee on Industry and Technology to address the July 8 Rogers network outage. Source: Hosue of Commons livestream screencap

Staffieri began by outlining four steps Rogers is currently undertaking to improve its network’s resiliency.

  1. Building an enhanced reliability plan. Rogers is working with the government and competitors on a formal agreement to ensure emergency services stay up at all times.
  2. Separate wireless and wireline traffic, which Staffieri said could cost around C$250 million. He further said that if the Rogers and Shaw merger deal is approved, this could be completed in half the time.
  3. Rogers will be investing C$10 billion over the next 10 years to build out its networks.
  4. It will also conduct more stringent testing. Additionally, it will conduct a thorough review of the incident with its partners and will share the results with competitors once it’s complete.

On Friday, July 22, Rogers sent a 39-page response to inquiries from the Canadian Radio-television and Telecommunications Commission (CRTC). The report attributed the failure to a deleted routing filter in its core networks that overwhelmed router capacity, rendering them incapable of directing traffic. The full report has been redacted for privacy and competitive reasons.

The error managed to slip through testing. At the hearing, McKenzie described the incident as unprecedented.

Advertisement 3

Article content

“We have extensive–in this particular program over two years–full rigour, full review, full analysis that was done,” said McKenzie. “This part of the implementation was a part of a seven-stage program and this was stage six; the previous five, no issue whatsoever. So this was truly an unprecedented incident.”

The root cause was in the different ways equipment from different vendors processed the update. Rogers uses hardware from a mix of vendors to build its networks, and it didn’t anticipate that the removal of a routing filter would not be handled identically on hardware across manufacturers.

“There was no belief at the time, no information at the time that there [the update deployment] was going to be any issue. And what happened was when the code change was executed–and the filter removed–the behaviour of the equipment in the way it’s designed between one vendor and a second vendor is very different,” explained McKenzie.

Not only did the outage leave millions of Canadians without connectivity, but also impacted 911 emergency services across the country. This became the key topic of discussion at the hearing. In response, Staffieri said Rogers will have a plan prepared.

“We will have failover measures so that in the event that there is an outage and it doesn’t fall over to our alternate network, it will fall over to one of our competitor’s networks within seconds,” Staffieri said.

The untimely outage occurred when the company is trying to persuade regulators to approve its acquisition of Shaw Communications for C$26 billion. Rogers had already failed to address competition concerns in its mediation with the Canadian Competition Bureau, which is seeking to block the deal.

Advertisement 4

Article content

As a part of its resiliency plan, Rogers wants to separate its wireless and wireline traffic to confine the scale of a similar outage in the future. Staffieri said that this expensive but necessary endeavour could cost around C$250 million, but can be completed in half the time if it’s able to acquire Shaw Communications.

IT World Canada has reached out to Rogers for a detailed explanation but did not receive a reply at the time of publication.

In the earlier portion of the hearing, François-Philippe Champagne, Minister of Innovation, Science and Industry, demanded telecommunication providers arrange an agreement of mutual assistance. Additionally, he asked all telcos to establish a communications protocol with the government and customers when a failure occurs.

The outage caused a major leadership change at Rogers. Last week, Ron McKenzie replaced Jorge Fernande as the new Rogers CTO effective immediately. Staffieri expressed his confidence in Ron McKenzie as Rogers’ new chief technology officer at the hearing.

More to come…

The post Hardware vendor differences led to Rogers outage, says Rogers CTO first appeared on IT World Canada.

This section is powered by IT World Canada. ITWC covers the enterprise IT spectrum, providing news and information for IT professionals aiming to succeed in the Canadian market.

Advertisement 1

Comments

Postmedia is committed to maintaining a lively but civil forum for discussion and encourage all readers to share their views on our articles. Comments may take up to an hour for moderation before appearing on the site. We ask you to keep your comments relevant and respectful. We have enabled email notifications—you will now receive an email if you receive a reply to your comment, there is an update to a comment thread you follow or if a user you follow comments. Visit our Community Guidelines for more information and details on how to adjust your email settings.

link

Leave a Reply

Your email address will not be published. Required fields are marked *