The addition of a new software routine caused this week’s failure of Research in Motion's Blackberry messaging service in North America.
In its first statement explaining the outage, the company said the introduction of a new software routine that was supposed to optimise system cache memory led to the problems.
"The system routine was expected to be non-impacting with respect to the real-time operation of the BlackBerry infrastructure, but the pre-testing of the system routine proved to be insufficient," Research in Motion (RIM) said in a statement.
Things were made worse by the poor performance of the company's back-up system.
"Although the backup system and failover process had been repeatedly and successfully tested previously, the failover process did not fully perform to RIM's expectations in this situation and therefore caused further delay in restoring service and processing the resulting message queue," it said.
In pointing to the software fault the company also said it had ruled out security or capacity issues or the failure of hardware or core-software.
RIM, which apologised to customers for the outage, said its examination of the problem was not yet complete and that more information would be provided as it becomes available.
"RIM has already identified certain aspects of its testing, monitoring and recovery processes that will be enhanced as a result of the incident and in order to prevent recurrence," it said.
Find your next job with computerworld UK jobs