The Case for Pause/Continue in Workflow Automation

All of us in the capital markets space are constantly collecting and managing data. Whether that data comes from a clearing firm, a CeFi exchange like Binance via APIs, or the blockchain via Web3, the challenge of efficiently handling it never goes away.

One of the companies I'm involved with, Clarity-Analytics , pulls in hundreds of thousands of trades and other data from Binance , Deribit , Coinbase , and others every day. Data collection takes about 12 hours, and part of it is time-critical because it involves point-in-time data — meaning you're taking a snapshot at exactly midnight UTC. Missing this snapshot makes reconciliation difficult.

The Problem

We've all had those challenging days when tech fails so spectacularly that you want to throw a server into the Chicago River. One of Clarity's components that communicates with Binance got stuck. Long story short, Clarity's data collection was still running as midnight UTC was approaching. It was crucial to capture those point-in-time snapshots for the new day. However, both data collections couldn’t run simultaneously — Binance would ban us for making too many API calls per second.

The Solution

In a recent release of XonaSoftware 's automation product, we incorporated a new "Pause/Continue" feature that does exactly what it says: allows you to pause a workflow and continue it later. By "pause," we mean the automation allows any currently executing tasks to complete but does not start any new tasks. When you continue the workflow, new tasks start exactly where it left off.

So, we paused the collection that was running, let the new workflow that started at midnight UTC collect all the point-in-time data, then paused it. We then resumed the workflow from the prior day, allowing it to run to completion before unpausing the midnight workflow. The pause/continue feature saved the day.