I thought I'd share a cautionary tale about something that happened to my website on Friday, in case anyone else has something similar.
We have had, for a number of years, a public donations page on the site. All it requires is that you go to the page, enter the amount you wish to give, and then click the Donate button. That then launches a Stripe Checkout session, during which you'll fill in card details, address, and so forth. It works, though to be honest, it's not been used a huge amount, as most of our donations come via a separate page for memebrs who are signed in to the site.
Until Friday. About 1930 I opened my 'Charge for Stripe' app to see what donations we'd had. And there was page after page after page of failed transactions for £1.00.
A look at the server logs showed a huge number of hits being made to the public donations page. People were using our page to test card numbers, before (presumably) using them for more serious fraud later on. I knew this sort of thing happens - it's like trying to buy a can of coke with a stolen card, before using it to buy an iPhone - but I didn't realise the scale.
My first attempt at mitigation was to add a redirect to the index page. That didn't stop it - the hits were still coming, so clearly people had cached the page, or were simply setting the referer and posting directly to the checkout script.
So, I tweaked the checkout script, adding a test for a session id, which every logged in user will have, and if there wasn't one, returning an HTTP 403 Forbidden response.
That stopped any new Stripe sessions being started, but there's still plenty of tidying up to do.
According to the logs, the first of these attempts was at 2351 the day before; there were two attempts at 0037, then things really got going with 94 at 0038, and 158 at 0039.
By the time I spotted and stopped this, at 1950, a total of 113,520 customer records had been created in our Stripe dashboard, which makes finding actual customer entries a chore; thankfully Stripe tells me these entries can be deleted in bulk if I send them a CSV file, which is a lot better than doing it 20 at a time via the dashboard.
Between when I updated the script to return a 403 error message, and when I checked about 17 hours later, requests were still being fired at the script; I should have monitored this sooner, but the load on the server was minor, it was Friday night, and I expected them to give up when the script kept returning an error. In fact, over that period, we had 670,000 attempts to start a checkout, from over 5,000 different IP addresses!
I've now got a fail2ban rule in place to block these IP addresses, but as I say - the scale of this was quite astonishing, certainly for someone like me who's never had this sort of attack on a server before.
I've now implemented rate-limiting for checkout sessions, with different time limits depending on whether someone's signed in or not. Stripe advises other types of friction, like a Captcha, may be worth considering. I've also beefed up logging, and a script runs every 20 minutes to check the number of callbacks to our Stripe webhook in the current hour, against the average for the last 24 hours. If there's a substantial increase, it alerts me and changes a setting in the database to stop public access to the checkout scripts.
Fortunately, we're lucky in that none of the card testing attempts worked, so no one was charged anything that we have to refund. I'm just left with a massive pile of customers that need deleting, and some updated scripts.
But, I figured it might be worth sharing this for anyone else who's got some sort of payment system. It's great to make it easy for people to donate to your project - but make it too easy and you too may suffer an attack of the card testers.
We have had, for a number of years, a public donations page on the site. All it requires is that you go to the page, enter the amount you wish to give, and then click the Donate button. That then launches a Stripe Checkout session, during which you'll fill in card details, address, and so forth. It works, though to be honest, it's not been used a huge amount, as most of our donations come via a separate page for memebrs who are signed in to the site.
Until Friday. About 1930 I opened my 'Charge for Stripe' app to see what donations we'd had. And there was page after page after page of failed transactions for £1.00.
A look at the server logs showed a huge number of hits being made to the public donations page. People were using our page to test card numbers, before (presumably) using them for more serious fraud later on. I knew this sort of thing happens - it's like trying to buy a can of coke with a stolen card, before using it to buy an iPhone - but I didn't realise the scale.
My first attempt at mitigation was to add a redirect to the index page. That didn't stop it - the hits were still coming, so clearly people had cached the page, or were simply setting the referer and posting directly to the checkout script.
So, I tweaked the checkout script, adding a test for a session id, which every logged in user will have, and if there wasn't one, returning an HTTP 403 Forbidden response.
That stopped any new Stripe sessions being started, but there's still plenty of tidying up to do.
According to the logs, the first of these attempts was at 2351 the day before; there were two attempts at 0037, then things really got going with 94 at 0038, and 158 at 0039.
By the time I spotted and stopped this, at 1950, a total of 113,520 customer records had been created in our Stripe dashboard, which makes finding actual customer entries a chore; thankfully Stripe tells me these entries can be deleted in bulk if I send them a CSV file, which is a lot better than doing it 20 at a time via the dashboard.
Between when I updated the script to return a 403 error message, and when I checked about 17 hours later, requests were still being fired at the script; I should have monitored this sooner, but the load on the server was minor, it was Friday night, and I expected them to give up when the script kept returning an error. In fact, over that period, we had 670,000 attempts to start a checkout, from over 5,000 different IP addresses!
I've now got a fail2ban rule in place to block these IP addresses, but as I say - the scale of this was quite astonishing, certainly for someone like me who's never had this sort of attack on a server before.
I've now implemented rate-limiting for checkout sessions, with different time limits depending on whether someone's signed in or not. Stripe advises other types of friction, like a Captcha, may be worth considering. I've also beefed up logging, and a script runs every 20 minutes to check the number of callbacks to our Stripe webhook in the current hour, against the average for the last 24 hours. If there's a substantial increase, it alerts me and changes a setting in the database to stop public access to the checkout scripts.
Fortunately, we're lucky in that none of the card testing attempts worked, so no one was charged anything that we have to refund. I'm just left with a massive pile of customers that need deleting, and some updated scripts.
But, I figured it might be worth sharing this for anyone else who's got some sort of payment system. It's great to make it easy for people to donate to your project - but make it too easy and you too may suffer an attack of the card testers.
