In quest of workflow engine

Workflow engine space is very crowded. It's typical to see a list of engines growing every year. The funny part is that they all start with the same requirements to "orchestrate" tasks but end up claiming to cover even BPMN 1.0 elements. What exactly drives engineering teams to write one more new shiny workflow engine versus contributing to already available is a very interesting topic to bring up. There are projects that start on a single codebase but eventually forked into a separate development stream with their strengths and weaknesses (Flowable vs Comunda). It's all about having their own playgrounds.

We all have been at least once on the below GitHub repository. For the sake of completeness ness, this curated list is just a starting point for exploration. Coming to this material the first time, we'd advise allocating at least two weeks full time to research tools. And by no means panic or regret, if you already have your own in-house grown workflow engine - it's of no surprise. Let us guess, it does orchestrate tasks and you had not needed anything else up until now. The same equally applies to "it works but does not scale", "too much effort to support", "ongoing refactoring", "no clean separation from the core product code" or "we'd like to re-architect our product as a part of our yearly innovation initiatives", etc. Happy days, your research will bring you to that GitHub page.

The basic idea, despite all the verbiage, is simple. At the glance...

I know only Java/Python/NodeJS

Seriously? These days, a decision is hard to be driven by just using an argument like this. The majority of wild software is not homogeneous. AI/ML is all over the place and python is in a dominant position. Despite the fact, there are attempts to expand other programming languages' capabilities, the python data science stack is ways ahead. If you're willing to sacrifice in functionality just to align low-level technology across the board, or because your team knows how to code only in, let's say, Java, you have more serious issues to solve in the product architecture. Noisy microservices design paradigm is knocking on your door and having a workflow engine as a service makes programming language choice less severe, and lightweight client language-specific SDK to it won't heart your teams productivity. The main message here is to review all available options despite the programming language or technical stack they based on.

Embedded vs (aa)Service

Just to draw the border what is considered as a Service. If business model, model tracking artifacts like versioning, deployment, runtime processes, etc. are all external and lightweight client is used to accessing we can call "as a Service". This works well as long as the business process isn't too deep in dealing with the product internals. Usual flow assumes dynamic variables for a workflow gathered in advance and sent to workflow server for processing passing workflow model version. These days, due to the popularity of orchestrating distributed services the workflow could be encapsulated in any service and not necessarily in a central business rule storage.

XML/JSON/YAML/Markdown

That's how it started with BPMN v1.0 - Appendix A (240 pages) and continued to v2 (400 pages). It's by design assumes you'd need an army of certified solution architects to get it running. Sure thing, they aren't going to edit XML directly - there are few dozen UI reincarnations. Without special UI, running the text-based diff is like using an ax to hammer screws. It is not developer-friendly format: git says "no". Server-side runtime versioning is, usually, coming out of the box.

Many in-house workflows also claim to be very configurable in open file format standards (JSON, YAML, Y). Unfortunately, they don't solve the versioning problem in any way, large workflow is hard to read and understand, no inheritance, viewer tools depicts software weakness rather than solving graph visualization.

Custom DSL is primarily focused on addressing product business logic customization on a per-customer basis. Or, if you want, change product behavior without touching the product code. It's one of the arguments to not go as-a-Code way. This customization requirement might outweigh the beauty of everything as a code concept, in particular, if feature-complete DSL is a hard requirement. For more lightweight DSL you can get away with it. You should remember workflow-as-a-code can interpret any BPMN, DNM, or whatever else notation but the other way around is not feasible. Hard to believe anyone is in a position to write one but philosophical talks exist.

Workflow-as-code

So-called workflow-as-a-code allows to orchestrate microservices calls and, while traditionally this logic expressed in plain code, now the tendency is to get it orchestrated by workflow engine because it has so many interesting features you even don't know (author realizes the topic non-triviality in the summary). For the sake of KISS, sometimes retry and the circuit breaker does the trick with the benefits of having custom security handling on top, clean read, and future modification path. You also get some nice features choreographing services in an asynchronous way. A hybrid approach is also very welcome. The red warning is to get total orchestration so microservices become library functions with REST APIs and lose business capabilities. Difficult to test, debug, extend. It's a sort of distributed monolith + microservices greedy approach that are clear winners for antipatterns.

The reality is having workflow as a code does make the development team's life easier. It's a very well-known environment, your favorite IDE, same code reviews, versioning, no additional tooling required. It's very collaborative. An actual workflow can be packaged along with service capability or moved to a central workflow repository.

The main disadvantage of workflow-as-a-code is that any change in the workflows requires project code to rebuild. So external on-the-fly/after-the-fact workflow customization is not possible or limited.

Workflow-as-code can be written using another workflow-as-code. Chicken-or-egg problem. Reverse engineering nested workflows you'll inevitably hit just a code. Honestly, parts of that code might be more useful used independently rather than all together in the context of a workflow engine.

Paranoid everything-a-workflow

Technically speaking, any code line can fail. Then it retries, waits, triggers a circuit breaker, calls fallback, etc. Is it a trigger to consider a workflow engine? Well. We've been building services that encapsulate logic handling call-fail scenarios for ages. It's not something extremely unique for microservices architecture too. It's all about trade-offs, a balance between simple intuitive vs twisted sophisticated.

We don't want to underestimate the workflow engine software. It does solve challenges you most likely don't want to solve as a part of your product development just because it's not your core business, although certainly a good feature for it.

For those in the sinking boat count how many times the word "complex" appears in the workflow-related articles (5, 4, 3, 2). You got it. Add workflow engine to your system and your engineering life will not be the same again.

Past performances don't necessarily indicate future results. Cite and refer to the post at your own risk.