Building a Lightweight Formula Execution Engine for Analytics Platform

I recently had a very interesting problem to solve: how to safely store and parse user inputted formulas for our analytics platform. At first glance, it sounds simple. Take a formula, evaluate it, return a number. Something like: (meta.spends + (ads.spends - 50)) * 100 My first instinct was the obvious one: just evaluate the string. That idea lasted about 30 seconds. Because the moment you try to do this in a real system, things get complicated very quickly: • Users can input anything which is a security risk • Formulas need to be reusable and debuggable • Metrics depend on other metrics so ordering matters • You need control over functions and allowed operations I explored existing options, but they were either unsafe (like "eval"), too limited or too heavy for what we needed. I realized we had to build a lightweight execution engine, so here's what we ended up doing: 1. Parse the formula into an Abstract Syntax Tree (AST) 2. Evaluate it in a controlled environment (no arbitrary execution) 3. Extract dependencies to understand which variables are required 4. Support custom functions like SUM, AVG, etc. But the most interesting realization came later. Formulas are not isolated, they form a graph. One metric depends on another, which can depend on another in turn. Suddenly, this becomes a dependency problem. Which means: • You need to resolve execution order • You need to detect cycles • You need to think like a query planner, not just a parser What started as "just evaluate a formula" turned into designing a small, safe, and extensible computation layer. And this is something I have noticed repeatedly. The interesting problems are not always the big ones. They are the ones that look simple enough to ignore, until you try to build them properly. #engineering #systemdesign #python #analytics #backend #softwaredevelopment

Define PEG grammar -> use pyparsing to gen AST -> ast evaluator

To view or add a comment, sign in

Explore content categories