[Part 1/4] The Low-Code vs Pro-Code Debate Has a Hidden Variable — and Most Platforms Get It Wrong

Part 1 of 4: Why "Same Engine" Is the Question That Actually Matters

This is Part 1 of a 4-part series exploring how Palantir Foundry's architecture settles the build-vs-buy debate for real-time streaming pipelines. The series covers:

📡 Part 1 — The hidden variable in low-code vs pro-code (this post) 🔧 Part 2 — The pre-packaged Flink applications beneath the UI ⚖️ Part 3 — How other platforms handle engine parity (spoiler: most don't) 🔍 Part 4 — The debugging dividend and what it means for build-vs-buy

The fork every team hits 🔄

You're building a streaming data pipeline. Maybe it's sensor telemetry from a factory floor. Maybe it's airline booking transactions that need fraud detection in real-time. Maybe it's retail purchase events feeding inventory alerts, or customer care interactions being evaluated against SLA thresholds.

Whatever the domain, the data engineering team faces the same fork:

Path A: Use the platform's visual, low-code tools. Drag-drop transforms. Point-and-click configuration. Ship it by Friday.

Path B: Write custom code. Full control. Full flexibility. Ship it in six weeks.

Every data platform on the market now promises both paths. "Low-code AND pro-code!" is the universal sales pitch. And on the surface, it sounds like you're getting the best of both worlds.

But there's a question hiding beneath that promise that almost nobody asks:

Do the low-code tools and the pro-code tools run on the same execution engine?

If the answer is no — and for most platforms, it is — then you're not choosing between convenience and control. You're choosing between two fundamentally different runtime systems with different performance characteristics, different scaling behaviour, different monitoring surfaces, and different failure modes.

That's not a choice between authoring styles. That's a choice between architectures.

What "same engine" actually means 🔧

After spending months implementing a real-time streaming alerting pipeline on Palantir Foundry — ingesting high-frequency data, transforming it into time series, and evaluating dozens of condition-based alert rules — I discovered something that isn't obvious from the outside:

Foundry's low-code and pro-code features compile to the same execution engine: Apache Flink.

When you build a streaming pipeline in Pipeline Builder (the visual, drag-drop interface), it doesn't run on some lightweight scripting runtime. It compiles into a Flink job — with Flink operators, Flink checkpoints, Flink state management, and Flink exactly-once semantics.

When you build the same pipeline in a Code Repository (writing Python, Java, or TypeScript), it also compiles into a Flink job — running on an identical Flink cluster, with the same infrastructure, the same fault tolerance, and the same performance characteristics.

One engine. Two authoring experiences. Zero performance gap.

The four-hop pattern 📡

To make this concrete, here's the generalised data flow that applies across streaming use cases — from manufacturing to financial services to retail:

Four jobs. Each one handles a distinct responsibility in the pipeline. And here's the thing — every one of these four jobs runs as an Apache Flink application in Palantir Foundry, regardless of whether you built it with the visual tools or wrote it in code.

Jobs 1 and 2 are typically authored by your data engineering team — either visually in Pipeline Builder or as code in a Code Repository. The choice is about developer preference and transform complexity, not about runtime performance.

Jobs 3 and 4 are where it gets interesting. These aren't user-authored at all. They're pre-packaged Flink applications that Palantir ships as part of the platform — sealed JARs that handle time series ingestion and alert evaluation.

But here's the key: they're still Flink. The same Flink that runs your Pipeline Builder transforms. The same Flink that runs your Code Repository implementations. The same checkpointing, the same metrics, the same diagnostic surfaces.

Why this matters 💡

When the entire pipeline — user-authored and platform-managed — runs on a single execution engine, three things become possible:

1. You can start with low-code and graduate to pro-code without re-architecting the pipeline. The downstream jobs don't know or care which tool authored the upstream transforms.

2. Performance tuning is a single discipline, not a platform-specific skill per tool. Learn Flink metrics once, apply everywhere.

3. The platform's pre-packaged components aren't black boxes running on some mystery runtime. They're Flink jobs you can inspect, monitor, and tune with the same tools you'd use for your own code.

That third point — the ability to look inside the pre-packaged components — turns out to be far more valuable than I expected. But I'll save that for Part 4.

Coming up next ➡️

In Part 2, I'll crack open those pre-packaged Flink applications — Eddie, Epoch, Object-Sentinel, and others — and show exactly what each one does, what stages it contains, and which UI you interact with to configure it. It's the layer most Foundry users never see, and it's the most architecturally interesting part of the platform.

Have you ever hit the wall where a platform's low-code tools couldn't keep up at scale — and had to rewrite in code? I'd love to hear about it in the comments.