3 Hong Kong, the mobile arm of Hutchison Telecom, said a software glitch in one of the IP network routing components led to a temporary disruption of its 3G mobile data service last Friday, according to a preliminary report the company submitted to the regulator.
The incident occurred at 11:05 on June 29, when 3 Hong Kong detected alarms in the AAA server (authentication, authorization, and accounting), which caused a degradation of its 3G mobile data services for nearly three hours.
3 Hong Kong said although its vendor had performed immediate recovery action after the incident was reported, the service degradation in the AAA Server still persisted. At 12:45 the operator decided to bypass the AAA Server so as to restore 3G mobile data services first.
Upon the completion of the bypass action at 13:15, the degraded service started to improve gradually and mobile data service returned to normal at 13:55, 3 Hong Kong said in the report to OFCA.
Around 40,000 to 50,000 of its 3G customers experienced difficulty in accessing 3G mobile data service throughout the period, while other mobile services such as voice and SMS remained unaffected, 3 Hong Kong said.
The operator said the service degradation was caused by a bug in the software of the AAA server that handles IP assignments of new mobile data connections and IP- MSISDN mapping to other platforms in its 3G network, resulting in the reduction of the number of unallocated IP addresses in the IP Pool.
As a result, when there was new attempt of IP allocation from 3G mobile data users, no IP address was available for allocation to the users. Thus the request of the relevant 3G mobile data users was rejected if such users hit the affected IP Pool, 3 Hong Kong said in the report to OFCA.