Hype cycle of (data) platform efficiency

Cloud and AI have greatly accelerated time-to-market and technology/architecture evolution, yet the hype cycle is still applicable. History doesn't repeat itself, but it often rhymes.

1. Built-in components provided by the Cloud provider (such as Lambda, RDS, ALB, Kinesis, Redshift)
2. Pure DevOps mode (a lot of freedom to spin up infra components per project) for each team; many infra components are treated as blackboxes without enough insights or any optimizations
3. Data volume grows by 20x, # user grows 30x, # workflow grows 100x, … (but the costs for SaaS and/or serverless go out of control)
4. Switch to the more specialized vendors such as Snowflake/Databricks, AutoMQ/Redpanda, PlanetScale, NeoCloud… to replace the mega Cloud provider for better cost/performance and richer features
5. New critical/flagship use cases (such as low latency, large/complex analytics) emerge
6. Introduce niche technology/vendor such as Ray, StarRocks/ClickHouse, DuckDB, Temporal, Qdrant
7. Increase investment to customize additional features on top of the vendor/OSS solutions; shift just a few heavy services to self-hosted components
8. Create critical features on top of OSS, bare metal or home-grown & highly-specialized components for the top-n use cases
9. Data volume grows by another 5x, # user grows 8x, # workflow grows 20x, …
10. Acquire a new line of business and start the consolidation again, loop back to [1] or [4] 

There are a lot of best practice videos in Youtube and guidances provided by the vendors, but it is still quite difficult to adopt and apply those into the real work. Why? One possible side effect of the Cloud and AI acceleration is - engineers are overwhelmed by tight deadlines and product features, they don't have time or mental bandwidth to valid or understand the database or k8s internals, therefore they put a system in place which just works but does not scale. Dr. Vogels also said in his re'Invent keynote:

Your customers are never going to tell you that your database engineers are doing amazing work and they love what they've done. Only you understand that work that goes into it. Most of what we build nobody will ever see. And the only reason why we do this well is our own professional pride in operational excellence. That is what defines the best builders. They do the things properly even when nobody's watching.

It is becoming hard to appreciate the above pride especially when your stakeholders don't care and your chain of command is not watching. Have you ever/often heard from the product/service teams that they have a few weeks to launch, please don't tell them that they need to partition the table or redesign the query shape; or they don't believe that they need the "over optimizations" you suggest. Everything goes OK in the first 18~24 months, because initially the data/transaction volume is not big enough to cause incidents; or it is easy to scale up hardware to hide the problem. But after this period, you either have scaled to the biggest node type or your cost:perf ratio has gone so terribly, yet you still run into higher-than-expected latency and frequent reliability incidents. Some product/service teams are willing to fix the tech debt, but they have very low confidence in the special or advanced data technology or design patterns, or they'd rather cope with incident outage than proactively take maintenance window to fix their data model & query shape because the "just works" data architecture is quite difficult to evolve/refactor without a big surgery. In such a scenario, therefore right-sizing + container scheduling is usually the preferred "optimization" approach, while the application logic and data model change is avoided/delayed unless a code yellow/red is summoned.

At this point (between [3] and [4]), some companies "buy" and "migrate" to a different vendor for the 1 or 2 most expensive use cases to temporarily stop the cost madness. But just after 18 months, the cost spike comes back again because the new system is being swamped by the suboptimal/bad workloads again (because folks are too busy to learn internals and apply the design vigor).

Instead of buying the way out, some other companies have stronger engineering determinations to fix the inefficient patterns. At this stage, the granular ownership and provenance are required to hold certain teams accountable. As the data & engineering maturity improve over time, [6] & [7] become the standards & testaments of the engineering excellence.

[8] is the next level of efficiency if the engineering team is equipped with stable accumulation of talent and technical confidence. This capability can finally tame the next wave of significant growth without the rapid cost spike trajectory. By this phase, the buy-vs-build balance/preference has reached a new balance.

[10] is often triggered by major M&A(s) which bring the different infra, processes, and platforms into the established/paved landscape. The cycle of consolidation, migration and optimization starts again. A few companies can avoid [2] and start from [4] again, yet most companies may have to go through [2] phase again.

Which loop have you observed? [1]->[2]->[3]->[2] is very convenient and super expensive. For the data folks who still hold the pride of engineering excellence, hopefully you and me can still strive for [7] & [8] even when nobody's watching.