· … views

Delete your data


Some of us build systems which are based on having some type of data. Maybe you develop e-com web-site and you store information about goods, prices and couriers. Maybe you develop maps and store information about different organizations, shops and roads. Or maybe you develop some type of social network when you store users’ content. It doesn’t matter. What does matter is that you don’t need all the data you have.

Data cost

Let’s explore lifecycle of some data unit in arbitrary backend system:

  • firstly, you create data unit. Maybe you update it a few times, send to different places where you process changes, maybe notify someone.
  • then usually data start being available to end users. They can see it in your mobile app or on a website. Usually that means that that data generates some type of profit for you. User orders something and you have a bit of gain. User explore other users’ content, spend more time on your website and see more adds.
  • at some point data becomes stale. Maybe in a month or 10 years but it is usually (data used for testing, former clients’ data, just something which us not used anymore). Not all the time of course. Some pieces of data should be there all the time: goods-bangers or some successfull organizations on the maps. But the other, I suppose, you usually archive and forget about it.

In my experience almost all the time you don’t hard delete such stale data. What you do instead is mark it deleted (soft delete), skip in the business logic. You pretend it’s not there.

How much you pay for such data?

Let’s suppose that you have 1’000’000 objects each 1MB (yes, quite big objects), half of which is not used anymore.

Half? Isn’t that too much?

Of course it is not. I saw systems where 80% of data wasn’t used at all.

Usually we think about cost of storing the data (which is not a lot: 1Gb is around 0.1$). So having 1Gb of archived data is pretty cheap. So what’s the problem?

First of all, we usually don’t store the data once. One item can be stored in some golden source database, replicas, in a backend service which sends this item to all other backend services, their databases and replicas, in dozens of RAM-caches in dozens of other services. Having 1GB in a source of truth could be 20GB across all the system (which is 2$ now). Now multiply that on amount of stored but not used data (1k$ paid once and some amount of money to renew that regularly).

But data cost is not only about storing it. Data tends to be amplified in many ways.

You need to move data around. In order item to arrive into some system it needs to travel through few network calls here and there. Network is pretty cheap this days. But on scale it could cost something. If we need to move each object 5 times, we could pay [0.01/GB * 500 * 5 =] 25$ every time when we update our system state. If you do that every minute you pay extra 36k$ a day (and 13.1M$ a year).

Next we need to process stored data and therefore spend some CPU on it. We can approximate it using formula like:

X = wasted CPU time = unused-data * time-per-data-unit

X = 500'000 MB * 2 ms/MB = 1'000'000 ms

And if we do 5 different processing steps we eventually waste 1.4 CPU-hours. If we assume that one CPU-hour is around 0.05$, we pay 0.07$ for one such cycle. What if we repeat it once every 10 minutes on each of our 100 machines? We pay additional 360k$.

And of course you want to be resilient, robust. You don’t want to fail tomorrow just because you didn’t buy additional TB of memory. So you pay twice as we counted just in case you need double amount of resources.

According to our napkin math, we could be wasting around $13.5M a year on network traffic and CPU time alone. And that’s before accounting for storage, replication, memory capacity, operational overhead, and the additional resources we provision for resilience.

The actual number depends heavily on the architecture and infrastructure costs, but the underlying point remains: unused data can be surprisingly expensive.

Of course that example is a bit exaggerated. The model is way too oversimplistic. Usually out systems are much more complicated. But I need it to be scary and simple enough to show you what I mean.

Other problems

We estimated only explicit costs. Now let’s talk about all other implicit stuff which we introduce into our system.

Since we don’t spend some resources processing unused data our system generally works faster (fast systems do less work). That opens different possibilities. For instance as we already discussed we could just save them. Or more possibly we could use them for other purposes: start update the data every 5 minutes, not 10. And now all the changes arrive faster which could potentially simplify system testing and problems fixing, or make user experience better. Or you could be prepared for growth without expanding costs.

Next understanding your system can be quite difficult. If you familiar with data-oriented design you know that before introducing some changes you need to understand your data and its’ patterns. You can find correct but useless information if half of your data is not used. It is genuinely a false signal. You need to understand only your working set, not all the data. But here it’s necessary to think beforehand that some of data isn’t used and patterns found in it are not representative. It could be less easier to do if you don’t need to think about that at all.

We also don’t do DOD explicitly all the time. Sometimes it’s also a small thing around the way how we write code and what assumptions we could do. Suppose you have such data flow in your system:

A -> B -> C -> D
          |
          E -> F -> G

A — items source. Items could be active and disabled. Maybe we have only 5% of inactive data and that’s 100Mb in total, same as nothing. And you pass that data downstream to all other clients. What all the code logic in all those services need to do is to remember that they could receive inactive item and they need to filter it out. You write if (item.IsActive()) all the time just because there is no other way. But if A would be so kind to filter those items before sending data to B, a lot of lines of code and, what is more important, brain fuel can be saved. Just because we can remove some semantic load. Great data allows you to do stricter assumptions and invariants which eventually leads to better performance.

Also having a lot of garbage means that your abstractions are leaked. In our last example if services after A do know about disabled items that means that they know too much about data lifecycle. Good design build clean boundaries so the irrelevant details aren’t available where they don’t need to be.

Sometimes good cleanup will lead to not having to do some work! My friend told me about a case when they thought about sharding 1.5TB database. But after some research they deleted 80% of data instead. 5 years later no sharding still needed.

But we need that data…

The thing which I hear a lot is that inactive/archived data is actually needed. Maybe one day it will be resurrected and used again.

I don’t believe such excuses. I heard them too much. But I can propose a few alternatives.

You don’t need to actually delete the data if you worry about using it. You just need to stop treat it like a useful one. Separate one from the other. Build some cold storage and use it only when you need it. Or just stop its’ propagation to every corner of your system. Ideally filter it closer to its’ source.

And you don’t need to build some type of complex system to deal with that. One meticulous cleanup in a few years should be enough to make your system (and your life) much better.

So delete your data.