Rstats Package {targets} for Workflow Automation

I've tried many workflow automation tools and nothing beats the #rstats package {targets}. It's pure R, pretty simple (while being feature-rich, great support for parallel execution and multi-lingual workflows, this is calling Python+DuckDB). It *trounces* the likes of make (bash), and #python pkgs dagster and prefect in my tests. Check it out!

  • No alternative text description for this image

If you you have a workflow that takes anything more than a few minutes to complete, and involves multiple inter-dependent changes (basically any meaningfully large and complex #datascience project), getting your work into this structure can yield large benefits. It takes some getting used to as Malcolm Morgan found (I forced him to use it and he thanks me month later ; ) but for big pipelines seeing the dependency graph and live updates to it is priceless, see image below which auto-updates as the pipeline runs 😍 . All with free and #open software, this is industrial-grade data engineering pipeline automation for free!

  • No alternative text description for this image

If you like targets you might be interested in T, a programming language I’m building made specifically to run pipelines and which makes it easy to work with R, Python and Unix tools (Julia support is coming): https://www.garudax.id/posts/brodriguesco_github-b-rodriguestlang-activity-7444689439681060864-0sE0?utm_source=social_share_send&utm_medium=member_desktop_web&rcm=ACoAAA-iNjcB8dVLnuPhmchEjJLFCRhniEk0qxg

Like
Reply
Dr. Kenneth Moore

Optimisation & Decision Analytics for Education & Complex Organisations

1mo

I couldn’t agree more. I think back to many projects from the past 10 years and re-imagine how much simpler they could have been with the discipline that targets enforces. It’s such a solid package. Can hardly think of a case where I wouldn’t use it these days for any pipeline, simple to complex.

Maybe mention the author Will Landau. Software of this level of complexity does not quite fall out of blue sky.

Like
Reply

Professor Robin Lovelace, I've been using {targets} to build data engineering pipelines and it works like a charm! It is just one of my preferred R packages. Thanks for sharing your thoughts on this.

Like
Reply
See more comments

To view or add a comment, sign in

Explore content categories