Chinese Yellow Pages | Classifieds | Knowledge | Tax | IME

ML workflow and pipeline orchestration

  • Kale – Aims at simplifying the Data Science experience of deploying Kubeflow Pipelines workflows.
  • Flyte – Easy to create concurrent, scalable, and maintainable workflows for machine learning.
  • MLRun – Generic mechanism for data scientists to build, run, and monitor ML tasks and pipelines.
  • Prefect – A workflow management system, designed for modern infrastructure.
  • ZenML – An extensible open-source MLOps framework to create reproducible pipelines.
  • Argo – Open source container-native workflow engine for orchestrating parallel jobs on Kubernetes.
  • Kedro – Library that implements software engineering best-practice for data and ML pipelines.
  • Luigi – Python module that helps you build complex pipelines of batch jobs.
  • Metaflow – Human-friendly lib that helps scientists and engineers build and manage data science projects.
  • Couler – Unified interface for constructing and managing workflows on different workflow engines.
  • Valohai – Simple and powerful tool to train, evaluate and deploy models.
  • Dagster.io – Data orchestrator for machine learning, analytics, and ETL.
  • Netflix Genie – Genie developed by Netflix is an open-source distributed workflow/task orchestration framework.

https://github.com/pditommaso/awesome-pipeline

https://neptune.ai/blog/best-workflow-and-pipeline-orchestration-tools

Please rate this


Comments are closed here.