On February 16th, 2024 a developer was happily going about their life. They'd built a small static site as a side project 4 years ago and had hosted it for free on Netlify (a popular fully-managed cloud service) since then. It got some traffic (~200 daily visitors) but the monthly traffic / bandwidth fell squarely in the free tier (<10 GB bandwidth per month).

Everything changed when the developer received their monthly bill - for $104,500.

In this post we're going to explore what happened and how to prevent it so your side projects don't financially ruin you.

The Static Site

The static site seems to be some sort of directory / blog that has links to some sorts of music files. The specifics aren't important but what is important is that some of these audio files might be kind of big.

• Site: Simple static site
• Traffic: 200 daily visitors, ~10GB bandwidth per month
• Monthly Cost: Free - always in Netlify free tier (free tier gives 100GB / month)

The $100k Bill

The site seems to have been targeted by some sort of DDOS. DDOS stands for a Distributed Denial of Service attack - this is basically when a ton of ~fake traffic is sent to a site / service which ends up overloading its systems.

This DDOS seems to have targeted a single 3.44 MB audio file and downloaded it a LOT (my estimates are ~55 million times - 190TB * 1000 GB/TB * 1000 MB/GB / 3.44 MB file).

• Total Bill: $104,500
• Estimated Bandwidth: ~190 TB
  • (Netlify charges $55 / 100GB bandwidth at time of writing)
• DDOS Peak: 02.16 transferred 60.7TB (costing ab $33,550)

The goal of an attack like this is usually to bring the site down so it can't work appropriately (like when you try to get tickets to an event as soon as they release but the site keeps dying). But here - in the land of "infinite" serverless scale - such an attack may not succeed. In fact the affect may be worse - racking up an infinite cloud bill that financially ruins the entity behind the project!

The developer was rightfully flustered and scared - this is a financially ruinous amount of money, costing more than most people make in an entire year for something they built for fun. So they started reaching out to support and making posts to get advice.

The Netlify billing team got back to them saying:

• They've confirmed DDoS so are reaching out to lower the bill
• Usually they cut bill to 20% of its cost -> $20k in this case
• But because bill is very high they will lower to 5% -> $5k in this case

$5k is certainly a lot better than $100k but even that is a huge, unreasonable amount for a static site - in particular one that's run as a side project. For reference, I host my static sites for ~$1 per month and all my projects put together typically cost < $100 per year.

Note: The Netlify CEO did reach out and waive the bill altogether. However we should expect this to be an edge case rather than the norm and thus is not a reliable way to defend yourself from such an outcome. If your DDoS does not reach the front page of Reddit / Hacker News then it's unlikely you'll get anything but the standard 20% discount - and even that is not guaranteed.

How to keep your Side Projects from Financially Ruining you

The main problem with this side project is that it had unbounded financial downside - it was possible (as we saw) for the bill to grow infinitely without any comparable upside. This is the case for most side projects - they don't make money so more traffic leads to more expense. This is fine if it's a hobby but once it leads to financial ruin it's clear the tradeoff isn't worth it.

Here are some ways that we can setup our side projects to avoid financial ruin:

Set a Spend Limit (cap expenses)

The first thing we can and should do is set a spend limit. This allows us to set say $100 per month as the most we're willing to pay for a given project / all our projects. Once expenses hit that limit, the service can shut down.

This is not ideal for businesses as this could lead to lost revenue but in the case of DDoS this may be preferable to uncapped expenses. For side projects which typically aren't profitable this is almost definitely what you want to do. A few lost days of availability is probably worth avoiding financial ruin.

Note: At time of writing, it appears Netlify does NOT offer this capability so maybe avoid it for now.

Set max resources (cap expenses)

Another way to manage your expenses is to put bounds on the resources you use. Without doing this, it's possible for you to get DDoSed, your infra to scale automatically, and you to hit your $100 cap in 10 minutes.

But if you have an idea of what your software needs to do and the kind of traffic it will receive you can actually constrain it a bit to weather such a storm better.

For example - you could set your site to only run on a max of 3 small machines at a time. This way it can handle the traffic you expect but if you get DDoSed you might only pay ~$10 per machine per month and the same $100 may last you all month.

This method often gets pushback because people are worried about not being able to scale for customer traffic spikes. This is almost never a real issue and saves you from unbounded expenses.

Setup DDoS Protection (prevent ~most attacks)

Many Cloud services have "built-in" DDoS protection. But clearly it doesn't always work (as we've seen here).

Moreover we have to acknowledge that the incentives of a cloud host and the incentives of the user may not always align so ~poor DDoS protection may be a feature not a bug.

• Users pay Cloud for usage
• DDoS spikes usage super high
• Cloud makes more money off user due to DDoS

Now I'm not casting blame and saying any cloud host is maliciously causing DDoS. I'm just pointing out that the incentives for them to fix this may not be particularly strong so there may be some willful ignorance / deprioritization happening.

To protect yourself from this, you need to reach for an option where the incentives to protect you from DDoS align with your own. This typically means a service (like Cloudflare) that specializes in this and makes money from it. If they fail to protect you, customers stop using them, so they want to protect you.

This one takes a bit more effort than the above to setup but is well worth it once you get a site with more traffic, visibility, and/or dangerous usecases (like large file downloads).

Move to Cheaper Hosting with Better Features

So we've already talked about how Netlify was clearly missing some key capabilities that could've prevented this (namely good ddos protection, spending limits, and resource limits). But another lever we can pull is to choose a Cloud Host that provides the same resources for less money. This means our $100 spend limit can go much further, serving 10x more traffic (or maybe even 50x).

I've personally run my sites on Google Cloud Run for the past several years and generally had a good experience. But I've also been researching other Cloud Hosting options and discovered I could get the same serverless container experience for about half the cost on Digital Ocean. So this idea of Cloud Host shopping to get the same power for far less money is top of mind.

Netlify is one of those new and shiny "we manage everything" serverless options. This can make it easier to get started but almost always has an exorbitant convenience cost baked in.

For example if we look at Data Egress costs (via GetDeploying) (the Bandwidth that cost this side project $100k), we see:

• Netlify: $550 / TB
• Google Cloud: $111 / TB (1/5 the price)
• Digital Ocean: $10 / TB (1/50 the price)

This means if this same site was on Digital Ocean, this DDoS bill would've shrunk from $100k -> $2k - and that's without changing anything else!

Last Resort: Post about it

The truth is DDoS will likely never be fully preventable. It's basically a game of cat and mouse. The cloud host services get better at defending against it and then the attackers find new ways to circumvent these protections.

This means that there will always be possible ways to DDoS your site no matter what you do - you've just got to set yourself up so you're okay when it does eventually happen.

Now if this does happen to you and you're stuck with a ruinous bill the next thing you can do is hail mary by making a public post about it, which is what this developer did.

This is not a guaranteed win. In fact I would argue this rarely works. But it's a chance and long-term this is how you create change in the world so it doesn't happen again.

• Chance to get discount
• Chance to improve the service (and similar services)
• Chance to get useful feedback / tips from other devs
• Helps entire industry learn / grow

Software Engineers have a lot of buying power when it comes to technology as they can steer their org on what to spend money on. No Software Engineer wants to get stuck with a bill like this so it's very likely such a story gets a lot of pushback (as it did here).

So if it does happen - make a post about it. It may not save you but it will save others and likely make a change.

Next

This was a very interesting (and scary!) story for me. I run dozens of side projects and build several new ones each year. The idea that one of these projects I built years ago could come back to financially ruin me and seriously affect my future is devastating. It's enough to make you not want to build side projects - which is one of my favorite things to do.

Luckily there are some relatively easy ways to avoid such an outcome. They require some extra work but the bad outcomes they prevent are well worth it.

Personally I've removed all my sites from Netlify and it seems a lot of other Software Engineers are, too. They really need to get their shit together and none of my side projects are worth a $100k bill. That said they are not alone in this - many other platforms also need to get their shit together. But I'm sure this will be a wake up call for them as well.

References:

• Reddit Post
• Hacker News post

If you liked this post you might also like:

• How I host my Server-Side-Rendered F# site on Google Cloud for less than $1 per month
• The HAM Stack - A Simple Scalable Tech Stack for building modern web apps fast and cheap
• Instagram's tech stack will surprise you