I've tried many workflow automation tools and nothing beats the #rstats package {targets}. It's pure R, pretty simple (while being feature-rich, great support for parallel execution and multi-lingual workflows, this is calling Python+DuckDB). It *trounces* the likes of make (bash), and #python pkgs dagster and prefect in my tests. Check it out!
If you like targets you might be interested in T, a programming language I’m building made specifically to run pipelines and which makes it easy to work with R, Python and Unix tools (Julia support is coming): https://www.garudax.id/posts/brodriguesco_github-b-rodriguestlang-activity-7444689439681060864-0sE0?utm_source=social_share_send&utm_medium=member_desktop_web&rcm=ACoAAA-iNjcB8dVLnuPhmchEjJLFCRhniEk0qxg
I couldn’t agree more. I think back to many projects from the past 10 years and re-imagine how much simpler they could have been with the discipline that targets enforces. It’s such a solid package. Can hardly think of a case where I wouldn’t use it these days for any pipeline, simple to complex.
Maybe mention the author Will Landau. Software of this level of complexity does not quite fall out of blue sky.
Professor Robin Lovelace, I've been using {targets} to build data engineering pipelines and it works like a charm! It is just one of my preferred R packages. Thanks for sharing your thoughts on this.
If you you have a workflow that takes anything more than a few minutes to complete, and involves multiple inter-dependent changes (basically any meaningfully large and complex #datascience project), getting your work into this structure can yield large benefits. It takes some getting used to as Malcolm Morgan found (I forced him to use it and he thanks me month later ; ) but for big pipelines seeing the dependency graph and live updates to it is priceless, see image below which auto-updates as the pipeline runs 😍 . All with free and #open software, this is industrial-grade data engineering pipeline automation for free!