• Prefect: Orchestrate workflows - article
• hydra: Manage configuration files - article
• Weights & Biases: Track and monitor experiments - article
• pre-commit plugins: Automate code reviewing formatting - article
• poetry: Python dependency management - article
• DVC: Data version control. - article
• BentoML: Deploy and serve machine learning models - article

• workshop branch focuses on Hydra, Prefect, and Weight & Biases along with explanations.
• bentoml_demo branch focuses on BentoML along with explanations.

• src: consists of Python scripts
• config: consists of configuration files
• data: consists of data
• notebook: consists of Jupyter Notebooks
• tests: consists of test files

1. Install Poetry
2. Set up the environment:

make setup
make install_all

3. To persist the output of Prefect's flow, run

export PREFECT__FLOWS__CHECKPOINTING=true

To run all flows, type:

python src/main.py

To run the process flow, type:

python src/main.py flow=process

To run the segment flow, type:

python src/main.py flow=segment