SagePay's monster wobble... On the third day of sale week, UK retailers start to weep

Black Friday week? Is that a thing? Er, not for all retailers

What started as a Saturday afternoon nap for SagePay turned into a three-day snooze fest, angering retailers that were as of last night still struggling to process sales in a peak shopping week.

The online payment service, which boasts about letting retailers get on with the business of retailing while it manages multi-channel payments on their behalf, went for a lie-down at around 16.30GMT on 18 November, customers told us. Sage Tweeted this later in the day.

The company issued updates in the next hours confirming transactions were still not going through as normal. "We apologise for the impact to processing, a full root cause analysis will be completed when normal services resumes,” it said.

It is not yet clear what wiped the old Protx network (Sage slurped Protx in 2006) off the face of the internet – a source speculated a DDoS which remains unconfirmed – but it returned on Saturday at nearly 10pm GMT “hiding behind Akamai but under a different IP,” our source claimed.

One SagePay business specialising in e-commerce store hosting told us it saw “tickets flood in to our support desk and customer frantically calling about lost business during huge marketing events”.

"Black Friday shopping week" (yes, it was new to us as well) had begun the day before the outage hit and is set to conclude this coming Friday, the day after Thanksgiving in the US, 24 November.

El Reg was further told by the customer that SagePay had switched back to its original IPs later that night after Akamai announced its IP space, but performance and the ability to use the payment system “suffered badly” until mid-day Sunday. Then Sage wrote this:

The SagePay status page has since carried a green tick to indicate no further service interruptions, but some customers flocked to Twitter to disagree.

Another customer opined that akamai.prolexic.com settings added by SagePay on Saturday were causing the issue – and suggested a potential workaround, which apparently worked for some but not for others. We have asked Akamai for comment.

For others the problems persisted into Monday evening.

The web hosting firm said his retail customers were unable to process transactions for around 24 hours, "after spending tens of thousands on marketing".

This morning, there were still isolated reports from customers saying they had issues with the service, though many said their services had swung back into action.

Sage did not answer questions about what took the service down but instead sent us a prepared statement:

"After a disruption we have now returned to a normal service. If any customers are still experiencing issues we are encouraging them to get in touch and working round the clock to support them. Our customers are our first priority and we will continue to monitor."

As El Reg has always said, outages will happen, but it's the way a cloud company deals with them that sets the grownups apart. We've again asked the kids at Sage if they will answer technical questions about the outage. ?


Biting the hand that feeds IT ? 1998–2017