The COGS Problem

This post is for engineering leaders specifically, but anybody looking for tactics and strategies to reduce their cloud spend should be able to pull out some nuggets from here.

Run an engineering department for a while, and it’s a bit astounding what becomes interesting. While I was once fascinated by new technologies and software, eventually my interests moved more into making sure the technology we were building was meshing and fitting with the rest of the goals of the company. I still love to hack on tech and still write lots of POCs or fun side projects, but for the day job, I spend the time on the intersection of business and technology – and it can actually be quite enjoyable. 

People outside of engineering generally care about your technology implementation to a very small degree – like almost none. Customers don’t buy your programming language choice, your platform choice or even your org chart make up. In the end somebody is measuring the return on investment (ROI) for engineering, and ideally the first person doing that is the engineering leader.

A key metric that contributes to ROI is Cost of Goods Sold (COGS). COGS is the cost component of Gross Margin (GM), which also can be manipulated by pricing/packaging and even by the way revenue is recognized. So, while total GM is a collaboration between Finance and Engineering, COGS can usually stay a little more within just the realm of engineering. However, there are certainly some rules around support staff costs, tooling budgets and some miscellaneous items that need to be accounted for (and will not be covered here). 

For an online business, the biggest component of COGS is likely cloud costs. At AWS RE:invent this year, that was a hot topic. The market was certainly beginning to change and the methods of measuring and benchmarking companies (particularly those in the tech sector) transitioned from growth at all costs to efficient growth – meaning costs need to be in control. The expo floor had around a dozen vendors who promised to save me huge amounts on cloud costs. 

Before you jump into spending money on cost control, take a step back. It’s not often that spending money to save money is the best first step. Here’s some thoughts to effectively review and improve your cloud expenditures. 

Stop Dumb Spending

First, stop with dumb spending. This may seem obvious, but for many organizations I’ve talked with, it’s just not being done. It can come off like “eat your vegetables,” but in reality you may have some really easy things to clean up to save a bunch. Even if it’s not a too sizable of savings, good hygiene in these areas will yield improve discipline for further steps. Dumb spending includes things like 

• Data lifecycle - Data that is saved with no retention or lifecycle (e.g. how many s3 buckets just grow and grow?, Do you prune old DB data?)
• Forgotten resources - Do you have resources/systems/machines that are running and not used? Perhaps implement a reaper of some kind to kill resources that aren’t tagged properly or are older than a certain date. Did you leave up a snapshot of an old RDS instance? Do you have an extra replica still running? 
• Overprovisioning - What utilization are you getting on your resources? If you’re running a 32 core 1TB of RAM node, do you need to? In my past, I’ve found machines with 32 GB of RAM running a single JVM and the JVM heap size was set to 16GBs. That’s…quite wasteful.
• Recommendations from the provider. The cloud provider likely has tools such as Cost Explorer that will make some recommendations on when to buy committed resources or sizing.

Yeah, that stuff seems easy. Cool. Just do it. Maybe you’ll see some impact. 

Unit Costs

Next is where the real work starts. You need to know what your unit cost is. It could be what it costs to host a customer, what it costs per mobile app user, per transaction on the platform, per minute spent on the software, etc. I can’t tell you exactly what your unit cost is, but you need to figure it out. At my last role, we knew the unit cost for a credit, a job, a build, and several other dimensions. At another company, the VP of Product knew the cost per agent deployed and knew that if sales sold something below that cost, there was no way to make up that delta – even at high volume.  

To illustrate this a bit:

Let’s say an agent costs $76 a year for a customer. This would likely be on average and some may be higher and some could be lower depending on the region of the world it’s installed in, the amount of data/usage going through the agent, the complexity of usage, or other factors. But, say that $76 is basically agreed upon by finance, accounting, engineering and product. Cool. Now if you have Gross Margin goals that you’ll get benchmarked against (and your company does have these), you can know how you stack up. 

If you’re out of line with your GM goals, you have two options. 

1. Make the per unit cost cheaper.
2. Price high enough to hit your GM targets with this unit cost (including things like discounting if applicable).

In reality, you’ll probably end up doing a combination of both.

Moving from $76 an agent to $71 an agent, may be pretty simple. Things like the ideas outlined in Stop Dumb Spending could plausibly get you that much savings. Maybe you can even get to $70 or $69 dollars annually for that agent cost. 

Now, if your finance team says the goal really needs to be about $55 annually per agent, you have some serious work to do. 

When working on this unit cost, you must know how you scale. Do you get cheaper when you add more units of work? Is it cheaper per unit to run 300 than 50? If so, getting more customers is a win-win move and should be prioritized. If you scale your costs basically linearly, then you have no real economies of scale and need to work on the  individual unit or change the design/architecture of the system to gain economies of scale. 

Unit cost should be measured and tracked over time. This will show you everything you need moving forward in terms of progress on the savings mission. Also, don't forget to account for static costs in your unit cost calculation (like that shared k8s cluster, shared DBs, backups, baseline network costs, and more).

Infrastructure Optimization

Once you have your unit costs, look at infrastructure first. Often it’s faster to move/change infrastructure than redesign an entire application (or dozens/hundreds of microservices). Things that can be advantageous for infrastructure:

1. Right size your systems. There are several tools on the market to do this, but even just having access to traditional load graphs can get you started.
2. Consider moving to AMD or ARM if you’re still running on Intel chips. ARM is considerably cheaper and in many cases outperforms other chips. The Graviton 2 nodes were amazing, at least in the exposure I have seen. This may require new image maintenance pipelines or processes, but still, saving 9-20% on nodes is probably worth it.
3. Optimize network flow. I continue to be surprised by how much of a cloud bill is network transfer. It’s the secret 20%+ of your bill. If you can make services less chatty, you’ll save a bunch. If you can compress traffic or payloads, you can save a bunch. If you can peer network, or get a network transfer specific volume discount with your provider, do it. 
4. Implement continued capacity monitoring to see if your behaviors have changed.  An optimized infrastructure setup can quickly become outdated if user or application behavior changes -- even in small ways.
5. Use spot instances (or interruptible instances on GCP). These are much cheaper, but come with some overhead in management. If you can use them, your costs will drop considerably. If this is a bigger architectural change, it might be work looking at a cost/benefit analysis to bring in the service teams as well. 
6. Safeguarding things – like can you have infinite loops in lambdas? How fast do you find them? Do you know if somebody spins up multiple replicas for a large database? Have you turned on every feature in cloud watch and cloud trail and now are paying lots to store/process events you're not using?

Sidebar: Don't go multi-cloud for a savings play. This has the overhead of multiple clouds, which includes security, technology understanding, and to make yourself portable, you're often leaving several of the best and most efficient cloud resources off the table. If you're in multiple clouds, consolidating down to one should allow some leverage for a steeper discount. Cloud providers usually will offer some competitive or migrations credits if the workloads are sizable.

Architectural Changes

You can absolutely make architecture changes earlier or in parallel to any other efforts. Depending on your product maturity and market-fit, this could be a bigger or smaller win. Often these get weighed against a product roadmap, so the trade-offs can be more complex. Major things I have seen cause extra spend:

1. New versions of software often are more optimized. As an example, moving from the last Java LTS to the latest Java LTS shaved about 10% off of CPU requirements for virtually every JVM application that adopted it. 
2. Data Design. Have a data schema for the most common things that move throughout your services. This seems like a no-brainer, but a lot of modern distributed applications grow up rather organically and as new services are added, they are each done in isolation. This can result in things like “account” having different fields or the same fields but they’re named differently. The end result is massive amounts of ETL, error/exception handling and overhead just to have two services work with the same logical construct. Making this change is difficult, but will save you in the long run.
3. Algorithm optimizations. If you have hot spots in your overall distributed application, can you optimize them? Better algorithms beat better hardware in the long run. Sometimes adding indexes, or denormalizing a database can really ensure faster reads and thus cheaper costs. Other times, better uses of caching or compression might be the ticket. 
4. Use redundancy appropriately. Not storing the same thing 10 times might be cheaper. It also might be more expensive to store it once and only once. Architectural recommendations are a little more difficult to make generically, since application and business context provides the answers. 
5. Spot instances. See above.

Conclusion

Tools exist to help with cloud cost management. They often help first with things like buying reserved instances, savings plans, or committed use discounts. Those are great, you should be doing that – but that’s a tip of the iceberg in terms of a real understanding of costs of your cloud applications. Some will dig in and find buckets without retention policies or oversized instances. Some do this with awesome tech like eBPF under the hood and even profile the application specific calls of a JVM or python. Other providers have secondary markets for reserved instances and capacity.

The mindset of COGS is important for software leaders. It’s also great for engineers. If you can show a saving or improvement in 24-48 hours, that’s amazing feedback. Developers love feedback. Show them the work they’ve done to retire a service or optimize a query matters. They’ll do more of it for that dopamine hit of knowing the work had a measurable impact. 

As a leader, understand your costs and how it flows into the rest of the business for benchmarks and measurements. The other members of your executive team will thank you.