Online payments hit a snag with software upgrades

With increasing software complexity, online transactions often hit a snag due to some or the other software upgrade. The question that remains is why despite so many incidents, companies are unable to resolve this issue? By Pupul Dutta

Picture this, you are trying to recharge your mobile, have paid the money and suddenly the system says, “Oops, we are sorry, the recharge couldn’t happen this time”. It’s a common problem that we all have faced at some point or the other. The question that arises is why despite so many instances, companies are unable to resolve this one problem. Also, what is the root cause of this which leads to loss of time and energy for not only the customers, but also results in delays in crediting back the payments that were incorrectly debited earlier.

Recently, while recharging his/her pre-paid mobile connection through the site Paytm (a recharge website), an Express Computer employee faced a similar problem where the transaction could not be completed due to software upgrade. Though the company sent a system generated apology mail, the issue (why it happened) remains unanswered. Express Computer tried contacting Paytm for a better understanding of the problem but the company refused to comment.

Components of transaction software
As more and more businesses move online, there is a growing need for robust payment systems that can efficiently capture a customer’s order on an e-commerce site.

According to a study, unfortunately in India, the acquiring bank/net banking trends indicate that over 24% of pre-paid orders fail to capture payment as compared to less than 1% in the USA. “In November 2013, the payment systems collectively (acquiring banks and net banking option) was down for a total of 46 hours which is almost 6.4% of the month. Payment aggregations are working overtime to integrate multiple acquiring banks and perform intelligent routing to improve success rate. Yet these payment gateways are helpless when a bank’s net banking option is down,” explains Mrinal Chatterjee, corporate VP – technology & founding team member, ShopClues.com.

Most e-commerce websites use a slew of software solutions, either developed in-house or outsourced from some vendor. The components of software used by these e-tailers are: shopping cart software, secure sockets layer and network components.

While, “shopping cart” (also called a “shopping basket” or simply a “basket”) is a software used by a merchant to assist consumers with making purchases online — allowing them to accumulate a list of items for purchase; SSL is used to encrypt information sent between the consumer and merchant, and between the merchant and e-commerce payment gateway. According to PCI DSS (Payment Card Industry Data Security Standard) Requirement 4.1, payment card data has to be protected during transmission over open, public networks.

Network components, on the other hand, provide connectivity and communication between different systems (for example, between application and database servers), and between the merchant, consumer, and e-commerce payment processor.

Failed transactions or downtime
With changing times, businesses have evolved with greater reliance on the web to attract clients, communicate with suppliers and generate revenues. Due to this transition, the cost of failed online transactions can be significant: a single hour of downtime could cost a retailer thousands of dollars in lost sales.

According to a Parallel Data Laboratory report available online, software failures and human error account for about 80% of failures (in online transactions). The incidents of software failures on the web suggest that a large number of non-malicious failures occur during routine maintenance, software upgrades and system integration. “It is however, unclear whether these failures are mainly due to system complexity, inadequate testing and/or poor understanding of system dependencies. Often site owners tend to be vague about the underlying cause of the failure which leads to huge business losses,” the report notes.

Nitin Gupta, Co-founder and CEO, PayU, a payment gateway company, says, “There could be multiple reasons leading up to an unsuccessful transaction. One prime reason for failed transactions is downtime either at the merchants’ site or payment gateway or at the bank site itself. However, in most cases, failed transactions occur from the bank side where reasons could vary from insufficient funds, incorrect password, to faulty server issues.”

What can be done?
E-commerce payment processors often provide software to the merchant to interface with the merchant’s shopping cart software and to facilitate collection and transmission of consumers’ payment card data.

“For any company, it is very important that before they choose a gateway, they should evaluate it on the basis of the health of the gateway, the global probability of success of the transactions on the basis of payment method and bin (bank identification number — it is the first four to six digits of a credit card that identifies the institution issuing the card), probability of success of transactions on the basis of past transaction history, and many more parameters. This switching helps to limit failed transactions and deliver industry best conversion rate,” Gupta asserts.

However, the question that remains unanswered is why despite so many incidents, companies are unable to resolve this particular roadblock? Answers Jitendra Gupta, CEO and Founder of Citrus Payment Solutions, “Generally, companies do take care to upgrade their system during mid-night and avoid any fluctuations. But, certain times, as bank systems are large and multi-functional in nature, they need to take immediate measures to fix a specific issue. To speak specifically about Citrus, we always upgrade our systems when traffic is almost non-existent reducing any impact on the merchant’s business.”

According to Symantec, while making a financial transaction over the web, many a times, the system tends to undergo an automatic update. During this time, the effect can be seen directly by the end user as the system goes in downtime mode. “In such situations having a fall back environment for the complete payment gateway application (web, application and DB) is the best way to avoid a potential downtime. The fall back environment would ensure the application is highly available (HA) and the customer can continue to transact on the failback environment while the primary environment is getting upgraded and is unavailable,” notes Huzefa Motiwala, country head- Presales for Information Management, India & SAARC, Symantec.

So, while software remains a dominant cause of site outages, this does not necessarily imply that software quality has gotten worse over the years. Also, somewhere it is evident that these outages are almost impossible to prevent given somewhere something will go wrong in this long trail of transactions.

Gupta rightly observes that while software complexity had increased dramatically, software fault rates held constant. The increase in software complexity relative to the other components of the system might be the reason why software is a dominant cause of outages.

Top reasons for failure
Some of the common reasons for failure of an online transaction are: buffer overflows, weak authentication or session credentials and security misconfiguration among many others.
What happens during buffer overflow is that when an application tries to store more data than the capacity of a buffer (temporary data storage area), excess data overflows into adjacent buffers, which corrupts or overwrites valid data. Due to this, modification can happen in system configurations, files can get damaged and change in confidential data becomes possible.

Secondly, if there are weak passwords, exposed protocols and services, the transactions become more vulnerable to attackers which leads to a failed transaction, due to a mismatch in the data fed earlier and in the current session.

Lastly, it is important for any e-commerce vendor to define secure configurations and apply it to the entire e-commerce environment, including: servers, applications, network components (e.g., routers and firewalls), and logging/monitoring mechanisms. Commonly exploited vulnerabilities include weak or unchanged vendor default passwords and system settings, and insecure remote access settings.

Moving On…
For any e-commerce merchant, bank or payment gateway, it is important to follow a few best practices like; in internet technologies it is crucial for software development teams to be in sync with the deployment teams. By doing this, they can better plan how the feature should be implemented and can guide the infrastructure teams when things go wrong. Secondly, always plan for the unexpected — product development and infrastructure teams need to come together to define how upgrades should be rolled out so that the service requires no downtime. Lastly, one needs to understand one thing that internet never sleeps. “There is someone, somewhere always using your service. The infrastructure teams and CTOs need to create an application infrastructure where software upgrades can be done without a downtime. This can be achieved by making all changes backward compatible and also planning a staggered upgrade where some of the servers are upgraded whereas the others are not and roll over the users to upgraded servers without affecting them,” explains Chatterjee of Shopclues.com.

In summary, it is important for all involved in software development as well as the infrastructure teams to work together to develop a process where software can be upgraded without impacting users. Software has a tendency of introducing errors. Do thorough testing with all critical-use cases. Stagger release of upgrades so that not all users are affected in case something goes wrong. Finally, plan upgrades where it does not require any downtime.