NCDEX’s experience of moving a LIVE exchange environment to a hosted data centre

NCDEX recently became India’s first exchange to operate its Data Center from an uptime certified Tier IV Facility established at the Yotta Data center in Navi Mumbai. Being a regulated market infrastructure institution, the exchange had to keep a laser-sharp focus on availability, reliability, and resilience.

Unlike other institutions where system downtime may result only in business and / or reputation loss, for a regulated market infrastructure institution like NCDEX, it goes much above and beyond. NCDEX is a leading AgriCommodity Exchange of India with over 85 percent market share in agri derivatives market. Many of the agri commodities are only traded on NCDEX and it’s the only platform of its kind available in the country for trading of agri commodities. In many commodities, the trading volumes on the Exchange are equal to or greater than the production of the commodity for the said year. This indicates the extent of participation on the exchange and the consequent criticality of the exchange ecosystem from a national perspective. Hence any system downtime or disruption in services of the exchange has adverse impact not only on the exchange business but also on the larger market ecosystem and national economy.

Given the above, it is imperative for NCDEX to have a laser sharp focus on ensuring that the trading platform and related services are available uninterrupted, are fault tolerant and highly resilient. The move to Yotta is a significant upgrade of the Data Center Infrastructure in this direction.

With this move, NCDEX is now India’s first and the only Exchange to operate its Data Center from uptime certified Tier IV Facility. “Tier IV is the highest globally recognized certification for data centers. It certifies the facility as being designed and built to the highest levels of reliability and resiliency, supported by completely fault tolerant critical infrastructure for every critical system, distribution pathway or component that supports the facility’s IT operations thereby providing 100 % uptime,” says Viral Davda, CTO, NCDEX.

The NCDEX Strategy for moving to a hosted data centre
A detailed study was conducted to prepare a comprehensive business case for this move. This included the justification for moving from on-premise to a hosted DC and also for selection of Yotta as our Partner. Multiple rounds of deliberations were done internally within the management and also with the Technology Advisory Committee of the board. The selection process itself took more than three months.

Commodity Exchanges have longer trading hours as compared to equities and other markets. NCDEX operates from 9 a.m. until 9:30 p.m. By the time, post market End of Day (EOD) processes and daily backups are done, it’s almost 2:30 a.m. and we have to start our system for the next trading day from 5:30 a.m. “Therefore, we have very little room for any changes, upgrades and / or routine maintenance on the weekdays. This leaves us with only weekends for such activities including activities as complex as shifting of our Data Center,” explains Mr. Davda.

The exchange had 500+ devices in the data center and shifting all of them and testing it over a weekend was extremely risky and also in case of any issues like damage in transit etc. or any issues during the setup, the team would not have enough time for a smooth roll back. Considering the risk involved and the impact that it would have, a complete lift and shift was ruled out.

Another approach was to set up a parallel data center. This would minimize the risk significantly but would have a huge set up cost. To balance the risk and the overall cost, the exchange decided to adapt a hybrid approach where all the critical systems were set up in parallel and a few where the risk was acceptable were lifted and shifted on the final weekend of migration.

The hybrid model of moving the data centre
So, how was the overall experience and step-by-step approach adapted for ensuring successful transition of the data centre? The overall experience was very smooth. “We traded from our existing on-premise Data Center in Kanjurmarg on Friday, did a member mock trading session from Yotta on Saturday morning and started Live Trading from Yotta from Monday. On Monday, members were able to connect and trade from Yotta as any normal trading day without even a jitter! We also completed our post market End of Day (EOD) processes on time without any issues,” narrates Mr. Davda.

However, a lot of planning, testing and practice went into ensuring a seamless transition.  The strategy of setting up a parallel data center for critical systems helped a lot. The exchange took the handover of Yotta DC in December 2020 and set it up as an extended LAN of the existing Data center. An ultra-critical upgrade of the core trading system was getting implemented back then.

In Feb 2021, the core trading system was successfully upgraded. Post that, the work on setting up the parallel data center at Yotta was commenced. This was followed by rigorous testing – functional, integration, regression, performance and failover testing. “By July 2021, we completed our test cycles and started running Yotta DC as a parallel set up with complete system start up, intraday as well as EOD processes being run on a daily basis,” states Davda. It was like running two parallel exchanges together minus the member involvement in Yotta. This required a tremendous amount of effort and support not only from Technology but also from all the other departments involved in day to day operations of the exchange. This made the teams involved very comfortable with running daily operations from the new data center at Yotta.

Looking at the nature of NCDEX’s business, compliance and security requirements, Yotta has carved out a separate, enclosed space inside the facility which is isolated from other server halls. “In a way, we have created a mini data center within the data center, which is being accessed only by their employees and based on their approval, our technician can access it. We are also offering them the scalability, as the exclusive server hall also has the scope of future expansion,” states Kamal Goel, EVP – IT & Chief Evangelist, Yotta.

Looking at the requirement of a low latency network, Yotta directly brought the telecommunications to their server hall. The server hall dedicated for NCDEX has direct connectivity from multiple telcos with no loop, POP or amplifier. “This is not only giving the customer better performance but also helping them meet regulatory compliance. The telcos are directly getting terminated in their server halls which provide them better control, management, and compliance as per their business requirement,” says Goel.

The migration was done in phases during which the NCDEX team undertook multiple tests with no downtime. During the whole exercise, Yotta team was shadowing them, while monitoring the devices, maintaining the performance of the network and giving them the required uptime. As and when migration progressed, Yotta provided the support to bring the hardware inside the premises, deploy them as per their design, and implement software applications as per their workflow.

During the migration phase, the team from Yotta extended support in terms of keeping a buffer stock of equipment like cables, switches, firewalls, and servers. “We even offered them the equipment to test. Apart from logistics support, we have worked extensively with their OEMs and partners so that we have better SLAs and spare stock at our data center to ensure the services are not getting hampered at any given point of time. We have signed a 6-hour city replacement SLA with the vendors for the products that come with a warranty. This means any hardware that is being used has a SLA, wherein parts can be replaced within 4-6 hours. We are even managing their replacement material, wherein the moment there’s a fault login, we track it to the closure,” states Goel.

From August 2021, NCDEX started exposing Yotta Data Center to the members and conducted five external mock trading sessions over the weekends to enable the members to connect, verify and ensure that they are able to trade seamlessly from Yotta. During these mocks, we also ensured that all other external stakeholders like clearing banks, data feed vendors etc. are also able to connect to Yotta and perform their routine operations.

Once it was made certain that all the pre-conditions were met and NCDEX was ready to Go Live with Yotta, a comprehensive Go Live strategy was formed, documented and agreed at all levels including the Board committees. The Go-Live strategy documented the detailed sequence of events to be performed during the weekend of cutover i.e. from Friday EOD until Monday. It also listed down the plan of action required to be executed in case there was a need to revert back and also the failover plan required to be executed in case of issues on the first trading day from Yotta. All along the transition journey, the regulator, Governing Board and the Market was kept informed.

To summarize, meticulous planning, extreme collaboration and rigorous practice was the success Mantra.

Seamless transition for the members
Shifting the Data Center also required members to shift their point to point connectivity to the new Data Center at Yotta. Timing all the member links shifting together with shifting of the Data Center was not practical. To ensure minimum inconvenience to the members, a three-pronged strategy was adopted :

1) Made backup arrangements to allow members whose link shifting is not yet completed to continue connecting to our existing Data Center using their existing point to point links. Exchange ensured traffic diversion from its existing Data Center to Yotta.

 2) All the interface points for members and all other external stakeholders were kept unchanged. This ensured that each and every one connecting to the exchange from outside did not require to do any changes from their end. For the external world, connectivity to the exchange remained the same – whether the exchange operates from the existing Data Center or Yotta, its seamless experience for members.

 3) Conducted multiple mock trading sessions from Yotta before Go-Live. This gave an opportunity for members to test and certify that they are able to trade from the new Data Center without any issues.

Trading from the DR site
The exchange has a regulatory mandate to do Live Trading from the DR site twice a year. This practice will continue. “To ensure that the data is correctly replicated between Yotta and the DR site, we have done multiple rounds of testing before going Live with Yotta. We have tested that we are able to start our next trading day from the DR site using the EOD of state of previous trading day. We have also tested our ability to failover to our DR site intra-day from Yotta within 45 minutes. All these tests helped us to certify our RPO and RTO requirements set by SEBI. For us, regulatory guidelines are non-negotiable and we have taken every step to ensure that all regulatory guidelines are being met at Yotta,” says Mr. Davda.

Finally, what are the benefits that NCDEX foresee after moving to a hosted DC Vs On-premise – both to the Exchange and to the Market? First and the most important advantage with the new Data Center at Yotta is uptime. The fail-safe design of the data center guarantees 100% uptime thereby ensuring that the Exchange provides an uninterrupted access of the trading platform to the Market.

Second, the presence of leading Telecom operators ensures wider connectivity choices for the Members and a shorter time around time for network link provisioning.

Third, a design PUE ratio of 1.4 will ensure a massive reduction in power consumption thereby reducing the overall carbon footprints of the Exchange.

Finally, moving to a professionally managed hosted DC will also help our technology team to focus more on bringing technology innovations to the business rather than spending time on managing the physical data centre, which will now be managed with the experts in this field.

On a longer term strategic perspective, this move will also enable the Exchange to offer Co-Location facility to the members as and when it is allowed by SEBI. Yotta’s NM1 has the scale and also the expertise which can be leveraged to offer such a facility to the Market.

Yotta Infrastructure Solutions has set up a world class facility at Panvel. Located in 18-Acre, Yotta Navi Mumbai Datacenter Park, Yotta NM1 is the world’s second largest Uptime Institute certified Tier IV Facility with 7000 Racks capacity, 50 MW power and 15-layer physical security. Yotta NM1 boasts of 4 redundant fiber paths, each brining in leading Telecom operators, Internet exchanges and ISPs – that connect Yotta NM1 to the world. Designed as a fault tolerant facility, Yotta NM1 is a fail-safe datacentre that guarantees 100% uptime, providing continuous power and cooling to the server racks and other IT equipment. Yotta’s NM1 DC therefore ticked all the boxes and was the ONLY data center service provider who is uptime Tier IV certified and was committing 100 % uptime.