Submission to the LegCo Panel on Financial Affairs
Market Disruption on the 28^thMay 2002

Introduction

1. A temporary disruption of the Derivatives markets occurred on 28^thMay 2002 following a power interruption and subsequent failure of the back-up Uninterruptible Power Supply ("UPS") system. This paper reviews the cause of the disruption and the actions that were taken as well as further follow-up actions that are underway to avoid a repetition of such an incident in the future. HKEx is committed to improving the reliability and robustness of its mission-critical systems, such as HKATS, as well as the systems that support them.

Description of the Market Disruption

2. There were two aspects to the market disruption on 28^thMay 2002, namely the temporary closure of the Derivatives markets due to failure of the derivatives trading system ("HKATS") and the duplicated trade information in the clearing system for the stock options market.

3. The sequence of events leading to the temporary closure of the Derivatives markets on 28^thMay 2002 was as follows:

3.1 The primary data centre of the Derivatives markets located in Asia Pacific Finance Tower ("APFT") experienced a sudden temporary power interruption at 3:12 p.m. The UPS system in APFT did not function as expected and the computer equipment in the APFT data centre suffered a power black out.

3.2 A site fail-over for all derivatives related systems ("HKATS") housed in APFT to the Wanchai secondary site was initiated at 3:22 p.m.

3.3 A Hong Kong Exchanges Contingency Management Team ("ExCMT") meeting was convened immediately to deal with the incident. Following HKEx's Market Contingency Plan and after consultation with the SFC, ExCMT decided at 3:45 p.m. that the Hong Kong equity related Derivatives markets (and the Dow Jones Industrial Average Index Futures) market would not be re-opened. This is in accordance with the guiding principle that, once the underlying cash market is closed, the related Derivatives markets should also be closed within 15 minutes. The close relationship between the Derivatives markets and the underlying cash market would make it difficult for the Derivatives markets to continue to operate in the absence of the cash market for extended periods for the following reasons:

• Price discovery function cannot be effectively performed in the Derivatives markets in the absence of the underlying Cash market;

• Investors may be forced to trade without market information or with incomplete data on the underlying stocks;

• Many trading strategies involving simultaneous trading in both the Derivatives and Cash markets cannot be conducted. The existence of these strategies is critical to the continued maintenance of a reasonable price relationship between the two markets.

3.4 Following completion of the site fail-over at 3:53 p.m., ExCMT decided that the non-Hong Kong equity related Derivatives markets, comprising the HIBOR Futures, the EFN Futures and the International Stock Futures and Options, would be re-opened at 4:30 p.m., allowing for a 30-minute notice period to be given to Exchange Participants

3.5 The maintenance vendor, Enviro-Tech Engineering Limited ("Enviro-Tech"), fixed the failed UPS system later in the evening. The UPS system resumed its normal function at 1:00 a.m., 29^thMay 2002.

3.6 On completion of the UPS testing, HKATS was restored to its original configuration on 1^stJune 2002, i.e. the primary data centre at APFT resumed its role as the primary production site and the Wanchai site resumed its role as the secondary site

4. During the site fail-over process, an operational error was made in the preparation of the clearing system for the stock options market, resulting in duplicated trade information in that system. The details are outlined as below:

4.1 The stock options clearing system ("TOPS"), which is located at the Quarry Bay data centre, was unaffected by the power interruption at APFT;

4.2 Due to a miscommunication during the HKATS site fail-over process, trades executed before the power interruption were re-sent from the HKATS to the TOPS system and, as a result, the trades executed before the power interruption were duplicated in the TOPS system and had to be reversed;

4.3 As a consequence, the stock option market participants' day-end job and, in particular, margin calculation and exercise / assignment results were delayed by about six hours.

5. It is worth noting that the UPS has been maintained by an external vendor, Enviro-Tech. The service records show that the last maintenance check was conducted on 29^thApril 2002. The records also show that the UPS was last activated on 20^thMay this year due to a fluctuation in the power supply and the system takeover was successful on that occasion, without any disruption to HKATS.

Root Cause of the UPS Failure

6. HKEx has commissioned an independent certified electrical and mechanical consultant, Ove Arup & Partners Hong Kong Limited ("Ove Arup"), to conduct a detailed investigation. According to the investigation report, the UPS failed because the inverters in the UPS which inverted D.C. power to A.C. power could not provide the same frequency and voltage. This ultimately caused the shut down of the UPS. According to Ove Arup's judgment, the malfunction of the inverters was probably due to the inverters which were not adequately adjusted at factory, at the first installation, or they lost the proper adjustment afterwards.

7. HKEx has accepted Ove Arup's conclusion and also agreed to publish its investigation report on the HKEx website.

Improvement Actions

8. HKEx is working on the following areas of improvements: UPS, derivatives system site fail-over procedures, IT operations, and contingency procedures.

8.1 UPS: The recommendations from Ove Arup are described in detail in Appendix 1. In brief, the short term recommendations relate primarily to the proper calibration and adjustment of the UPS units; the medium term recommendations include replacing the UPS system with a new system to better meet the requirements of the data centre in which the HKATS system is maintained and a new central system to monitor the UPS system; and the long term improvements relate mainly to environmental issues, such as the provision of back-up generator and improvement to the air-conditioning system. HKEx has accepted all the recommendations from Ove Arup.

HKEx is committed to complete all short-term actions by the end of July 2002 and all medium-term items by the end of October 2002. HKEx will also immediately start a review of how best to address the long-term recommendations.

8.2 Derivatives system site fail-over procedures: following a review of the site fail-over procedures, improvements set out below have been implemented:

• The "Derivatives System Site Fail-Over Procedures' document was reviewed and revised;
• A desktop walkthrough had been conducted to verify the revised procedures; and
• A regular data centre disaster drill for HKATS has been scheduled on 6^thJuly 2002. The revised site fail-over procedures will be rehearsed to ensure its accuracy and completeness.

8.3 IT Operations: HKEx's improvement programme in IT operations is outlined as follows:

8.3.1 In the course of the past six months, HKEx has embarked on a programme to improve the quality of its IT operations. This includes

• New processes jointly developed and implemented with the HKATS network provider to improve network stability;
• A new support model with the HKATS application provider to improve the stability of the HKATS trading platform;
• A range of measures on the operations of the data centres (such as error detection and rectification mechanism, status checking, monitoring and reconciliation steps); and
• A critical software vendor quality management standard, which includes quality requirements, quality plans and quality control mechanisms, for reinforcing quality control and assurance of IT outsourcing projects or system/services provided by critical software vendors.

8.3.2 HKEx will continue this improvement programme which will be expanded to include comprehensive reviews of HKEx's mission critical systems. The comprehensive review of the HKATS system will start in July 2002 and will be completed in stages in February 2003. HKEx will also be commencing a review of other mission critical systems from October 2002.

8.3.3 Besides, HKEx is planning to engage an external consultant to review its overall IT requirements and environment in order to come up with a recommendation on a quality management and certification framework for its IT system development and management. Once the framework is selected, HKEx will proceed to develop an implementation plan for achieving such certification.

8.4 Contingency Procedures: HKEx had detailed contingency procedures in place and closely followed them in the handling of the incident on 28^thMay 2002. HKEx has identified room for improvement, especially in communication and information dissemination to Exchange Participants and the market regarding the market suspension and trading resumption arrangements. Shortly after the incident, we immediately implemented corrective measure to ensure only designated authorised executives are allowed to perform such communication functions. In addition, we are currently reviewing the following areas for further improvement :

8.4.1 System failover and market readiness -

• the feasibility of reducing the time taken for the failover between the primary and back-up sites; and

• the feasibility of shortening the lead time for system readiness/notification to Exchange Participants

8.4.2 Trading resumption policy - to clarify and make public the principles for the resumption of trading (if so decided) following a trading disruption; and

8.4.3 Communication to the public - to improve the procedure relating to market communication, including in particular Exchange Participants, if trading disruption occurs and the subsequent decisions relating of market operations (such as whether, and if so when, the market would be resumed following a temporary market closure).

HKEx will consult the SFC on the revised contingency procedures before finalisation. The trading resumption related principles will be made public so that the market will have a clear understanding of HKEx's policy. The revised procedures will be implemented by the end of July.

APPENDIX 1

Ove Arup's Recommendations

Short-Term Improvements

After review of the existing installation, we (i.e. Ove Arup) recommend to carry out the following short-term improvements which should be targeted to complete in one month.

Medium-Term Improvements

We recommend the following medium-term improvements, which should be targeted to complete in three to four months.

1. The existing UPS system should be replaced with a new system which should meet the latest demanding requirements for a data centre serving a mission critical facility like stock exchange. In particular, the UPS system should include the following provisions:

• The battery backup time should be 30 minutes.
• A battery monitoring system should be installed.
• The batteries should be installed on proper battery cabinets or racks.
• The maintenance contract should include all the basic services plus additional works such as annual dummy load test, thermal scan test, quarterly measurement of battery conductance, comparison of voltage of each battery block and record of alarm logs monthly.

2. A new central monitoring system to monitor the UPS system, the battery system, the room temperature, the fire alarm etc. should be installed.

Long-Term Improvements

Subject to the availability of space and detailed design, the following long-term improvements are suggested for consideration.

APPENDIX 2

Ove Arup's Detailed Explanation of the Terms

Inverter- Inverter is one of the major components of UPS. In normal operation, A.C. power is supplied to the rectifier, which converts A.C. power to D.C. power and charges the batteries. At the same time, the inverter inverts D.C. power to A.C. power and provides power to the load. In case there is no incoming power or the power is fluctuated, the converter will shut down. D.C. power will be obtained from the batteries and the inverter will continue to invert D.C. power to A.C. power and support the loads.

Rectifier- A major component of UPS, which converts A.C. power to D.C. power.

Synchronise- means to have same frequency and voltage. When the inverter inverts D.C. power to A.C. power, it needs to make sure that the A.C. power it provides has the same frequency and voltage of the bypass supply before it connects the power to the output. In a parallel UPS system, the inverters of the units also need to make sure that their frequency and voltage are the same before they connect the power to the output. If it cannot synchronise with the bypass supply or other unit, it will shut down.

Bypass supply- An alternate incoming A.C. power supply to the UPS. When there are some problems on the rectifier or the inverter such as rectifier overload, which will cause rectifier shut down, the UPS output will connect to the bypass supply to continue provide A.C. power to the load.

Common battery breaker- A device to turn on/off the power connection between two sets of batteries serving two different UPS units.