Thailand’s number two telco, DTAC, has suffered three massive network outages in the space of as many weeks.
With confidence shaken and an inquiry by the regulator looming, the beleaguered Telenor subsidiary called a press conference to convince the public that it was all down to improbable odds rather than mismanagement or incompetence.
DTAC CEO Jon Eddy Abdullah apologized for the series of outages.
The entire network collapsed on 21 December because of a technical issue with the migration of the HLR from one database to another. The migration was part of a two-year network modernisation plan signed off with Ericsson, that commenced in Q4 of last year, to replace the 18-year old 2G radio and back haul network with a modern one ready for 3G or LTE.
Abdullah said his technical people and Ericsson’s are now confident that they understand the root cause of the outage.
On 5 January, a switch in the South of Thailand failed, but what usually takes five minutes to restart took hours because of the ongoing HLR migration. The problem spread nationwide when a parameter change aimed at helping users come back online faster inadvertently knocked iPhone 4 users off the network, something that was unforeseen either to DTAC or Ericsson.
Abdullah has ordered a doubling of HLR database capacity before further migration takes place, which is expected to take six weeks.
On 8 January in the latest major network crash, a car accident severed one of two redundant fiber cables to the South and two minutes later, a bush fire cut the other cable. DTAC plans to build triple redundancy into critical network paths from now on.
Abdullah said that the chances of this happening were down to unbelievably bad luck and ruled out sabotage. “There are easier ways to cut a fibre optic cable,” he pointed out.
DTAC has already paid out 300 million baht ($9.5 million) in compensation in the first outage, 50 million baht ($1.5 million) in the second and is still calculating the level of compensation for the third.
But the real damage is to the company’s reputation and it remains to be seen how the churn and number portability figures tally up when the dust settles.
DTAC has committed 40 billion baht ($1.26 billion) in network investment over the next three years. Abdullah was asked why DTAC was investing so much given that its concession ends in 2018 and the previous, aborted, draft 3G licence specifically prohibited re-use of old 2G network infrastructure. He responded that only 20% of its users were now on 3G (850-MHz) and the company and its concession holder, state owned CAT Telecom, has a commitment to bring the best technology to all its subscribers, whether 2G or 3G.
Pressed if the network upgrade was a bid to to pre-empt and shape the new regulator’s 3G licensing terms and conditions, Abdullah only said, “It is the right thing to do. Whether we have six years left or two years left, it is the right thing to do.”
Abdullah said he expects 2.1-GHz 3G licensing in Thailand in Q3 this year and that the company has ample free cash, generating around 5 to 7 billion baht ($158 million to $221 million) every quarter, to invest in a licence and network.
Later, after meeting with DTAC officials, National Broadcasting and Telecommunications Commission vice-chairman Colonel Settapong Malisuwan announced a formal investigation and requested access to log files to see if the network crash was due to negligence or a technical issue as claimed. No immediate sanctions would be levied though further migration is prohibited pending the outcome of the investigation.
Commissioner Dr Prawit Leesathapornwongsa said that while DTAC had effectively offered 48 hours compensation for what was a four to six hour outage, consumers could still sue the operator in civil courts if they believed their consequential damage was higher offered help in doing so.
More worrying was his statement that in the future, any database migration would have to be submitted to the NBTC six months in advance for approval.