An IT contractor has reportedly been suspended by the London Stock Exchange following a huge network hit between Monday night and Tuesday morning.
The network was floored, knocking a key Linux trading platform out of action and bringing a halt to the largest technology transformation at the exchange in 25 years.
The police and the Financial Services Authority are aware the incident. The City AM newspaper said it had been informed of the suspension of the LSE IT worker, but the exchange today declined to comment on staff matters.
The stock exchange’s Turquoise dark pool, or anonymous trading venue, was taken offline for two hours on Tuesday morning, and later that day the LSE said it was delaying a key migration to open source systems until next year.
The LSE is conducting its own internal investigation, but has refused to state whether the outage was an accident, or whether it suspects a disgruntled employee. The FSA declined to comment.
One large shareholder in the exchange told the Financial Times that he suspected sabotage by an employee, because the LSE had cut staff numbers by more than 10 percent. “There are still probably a few disgruntled individuals who are in the departure lounge at the moment,” he said.
An unnamed city source told City AM that when the problem happened “there was an announcement about the platform change planned later the same day". The source speculated: “It looks like someone may have been trying to undermine this.”
While the LSE declined to comment, in a statement rushed out to the markets on Tuesday night it referred to both an "error" and "suspicious circumstances".
Some onlookers suspected the problem was an accident resulting from poor management of a contractor. A senior trader that had spoken to the LSE told the Financial Times : “It seems someone, a contractor, had access to the data centre and did something they shouldn’t have done.”
But last month, the LSE made the important step of allowing more IT personnel into its datacentre, expanding co-location beyond trading firms and to software firms – to install hosted systems and their own market data distribution platforms. It is not known if any of this raft of new companies entering the datacentre played a part in the outage.
The exchange is grappling with a major technology change and is also changing its IT skills base.
It is moving away from its existing TradElect platform, based on Microsoft .Net, and requires C++ and Linux skills instead. It is also thought to be moving increasingly towards blade servers and away from traditional stacks and Unix mainframes, changing expertise again.
Although the system is live on Turquoise, the cash markets migration that would have happened this month is considered a far bigger move. LSE chief executive Xavier Rolet has staked much of his leadership success on making the platform the fastest in the world, following a series of embarrassing outages on TradElect that took place before his tenure.
Until the Linux switchover is made, TradElect will continue to run. The system delivers trading speeds of just under two milliseconds (2,000 microseconds), around sixteen times the latency of the new Linux platform. It is also around five times slower than the LSE’s largest rival in Britain, Chi-X, and at least eight times slower than NASDAQ in New York.
The LSE has vowed to make the system run as soon as it is ready. It is understood to be increasing server and networking hardware capacity rapidly in order to cope with the high volumes of trading expected. While this means delays, market participants told Computerworld UK they welcomed the fact the LSE was not launching the platform before it was confident the datacentres were ready.
The new system will handle 50,000 messages per second, though market observers said traders increasingly expect up to 300,000 to be manageable. Some newer technology in development can reportedly manage a million messages per second, though this measurement is in testing, and live environments are more challenging.
In spite of the outages, in a monthly report the LSE today said it “sits at the heart of the world's financial community, offering international business unrivalled access to Europe's capital markets”.