No Description

Peter Fogh 1e0d8dcade ran pipeline with DVC 0.40 1 year ago
.dvc 2733a3655d Updated DVC to 0.37 and crated new config settings 1 year ago
.gitignore 83c1e1db7c Process to TSV, Separate dataset into train and test-sets, transform to data to feature metrics, trained the model, and evaluated the model 1 year ago
Dvcfile 1e0d8dcade ran pipeline with DVC 0.40 1 year ago
README.md eae73caa62 Updated README to new folder dep/out for dvc run commands. And tested them 1 year ago
conda_env.yml 2733a3655d Updated DVC to 0.37 and crated new config settings 1 year ago
conf.py da1b4ca01c AA-445 changes all DVC stages to depent and output wrt. folder instead of files. 1 year ago
download_xml.dvc 1e0d8dcade ran pipeline with DVC 0.40 1 year ago
download_xml.py d8d96fb6a7 Added download_xml.dvc stage and removed wget and tar bash commands from README 1 year ago
evaluate.py da1b4ca01c AA-445 changes all DVC stages to depent and output wrt. folder instead of files. 1 year ago
featurization.dvc 1e0d8dcade ran pipeline with DVC 0.40 1 year ago
featurization.py da1b4ca01c AA-445 changes all DVC stages to depent and output wrt. folder instead of files. 1 year ago
split_train_test.dvc 1e0d8dcade ran pipeline with DVC 0.40 1 year ago
split_train_test.py da1b4ca01c AA-445 changes all DVC stages to depent and output wrt. folder instead of files. 1 year ago
train_model.dvc 1e0d8dcade ran pipeline with DVC 0.40 1 year ago
train_model.py da1b4ca01c AA-445 changes all DVC stages to depent and output wrt. folder instead of files. 1 year ago
xml_to_tsv.dvc 1e0d8dcade ran pipeline with DVC 0.40 1 year ago
xml_to_tsv.py d8d96fb6a7 Added download_xml.dvc stage and removed wget and tar bash commands from README 1 year ago

Data Pipeline

Legend
DVC Managed File
Git Managed File
Metric
Stage File
External File

README.md

DVC and Dask use case

This repository contains the description and code for setting up DVC to use a remote computer server using dask. Note that this use case relay on the original DVC tutorial and its code found here https://dvc.org/doc/tutorial.

How to set up the use case

Prerequisites

The use case have the following prerequisites:

  1. A remote server with:
    1. SSH installed.
    2. A unix user you have the username and password for.
    3. A folder for your remote shared DVC cache, my is at /scratch/dvc_project_cache/.
    4. A folder for your remote DVC data directories, my is at /scratch/dvc_users/[REMOTE_USERNAME]/.
    5. A Dask scheduler installed and running at port 8786, see http://docs.dask.org/en/latest/setup.html for a guide.
    6. A MLflow tracking server installed and running at host 0.0.0.0 and port 5000, with mlflow server --host 0.0.0.0 --file-store /projects/mlflow_runs/.
  2. A local SSH keyfile (ssh-keygen), which have been copied to the remote server, with ssh-copy-id [REMOTE_USERNAME]@[REMOTE_IP].
  3. An open SSH port-forward to the Dask scheduler and MLflow tracking server from your local machine to the remote server, with ssh -L 8786:[REMOTE_USERNAME]@[REMOTE_IP]:8786 -L 5000:[REMOTE_USERNAME]@[REMOTE_IP]:5000 [REMOTE_USERNAME]@[REMOTE_IP].
  4. Set up local DVC development repository (following https://dvc.org/doc/user-guide/contributing/) with a conda environment:
    1. Fork https://github.com/iterative/dvc on Github.
    2. git clone git@github.com:<GITHUB_USERNAME>/dvc.git
    3. cd dvc
    4. conda create -n py36_open_source_dvc python=3.6
    5. conda activate py36_open_source_dvc
    6. pip install -r requirements.txt
    7. pip install -r tests/requirements.txt
    8. pip install -e .
    9. pip install pre-commit
    10. pre-commit install
    11. which dvc should say [HOME]/anaconda3/envs/py36_open_source_dvc/bin/dvc and dvc --version should say the exact version available in you local DVC development repository.
  5. Configure you DVC globally (e.g. using the --global flag) for you local machine - note that I call my remote server "ahsoka":
    1. conda activate py36_open_source_dvc
    2. dvc remote add ahsoka ssh://[REMOTE_IP]/ --global
    3. dvc remote modify ahsoka user [REMOTE_USERNAME] --global
    4. dvc remote modify ahsoka port 22 --global
    5. dvc remote modify ahsoka keyfile [PATH_TO_YOUR_PUBLIC_SSH_KEY] --global
    6. dvc remote add ahsoka_user_workspace remote://ahsoka/scratch/dvc_users/[REMOTE_USERNAME]/ --global
    7. These globally configured DVC remotes are used by the DVC config file in the Git repository, see .dvc/config, to specify project specific remotes for the DVC cache and DVC data workspace.

Use case

This use case of DVC and Dask has been set up as follow.

On your local machine do the following:

  1. Clone this test repository from my Github: git clone git@github.com:PeterFogh/dvc_dask_use_case.git
  2. Install the Conda environment for this repository - note the new enviroment must point to your local DVC development repository:
    1. conda env create -f conda_env.yml, which have been create by the following commands (executed the 26-04-2019):
      1. conda create --name py36_open_source_dvc_dask_use_case --clone py36_open_source_dvc
      2. conda install -n py36_open_source_dvc_dask_use_case dask scikit-learn
      3. conda activate py36_open_source_dvc_dask_use_case && pip install mlflow matplotlib
      4. conda env export -n py36_open_source_dvc_dask_use_case > conda_env.yml
    2. Check dvc version matches your development repository version: conda activate py36_open_source_dvc && which dvc && dvc --version and conda activate py36_open_source_dvc_dask_use_case && which dvc && dvc --version
  3. Reproduce the DVC pipeline: dvc repro - which have been specified by the following DVC stages:
    1. conda activate py36_open_source_dvc_dask_use_case
    2. dvc run -d download_xml.py -d conf.py -o remote://ahsoka_project_data/download_xml/ -f download_xml.dvc python download_xml.py
    3. dvc run -d xml_to_tsv.py -d conf.py -d remote://ahsoka_project_data/download_xml/ -o remote://ahsoka_project_data/xml_to_tsv/ -f xml_to_tsv.dvc python xml_to_tsv.py
    4. dvc run -d split_train_test.py -d conf.py -d remote://ahsoka_project_data/xml_to_tsv/ -o remote://ahsoka_project_data/split_train_test/ -f split_train_test.dvc python split_train_test.py
    5. dvc run -d featurization.py -d conf.py -d remote://ahsoka_project_data/split_train_test/ -o remote://ahsoka_project_data/featurization/ -f featurization.dvc python featurization.py
    6. dvc run -d train_model.py -d conf.py -d remote://ahsoka_project_data/featurization/ -o remote://ahsoka_project_data/train_model/ -f train_model.dvc python train_model.py
    7. dvc run -d evaluate.py -d conf.py -d remote://ahsoka_project_data/featurization/ -d remote://ahsoka_project_data/train_model/ -o remote://ahsoka_project_data/evaluate/ -m eval.txt -f Dvcfile python evaluate.py
  4. Show DVC metrics dvc metrics show -a.
  5. Visit MLflow tracking server webUI from your local browser at http://localhost:5000/ to see the results of the pipeline.

Problems with MLflow for the use case

  • MLflow artifacts do not support our SSH setup. mlflow.log_artifacts() do not support files saved on the remote server. Artifact files must be located at a directory shared by both the client machine and the server using the methods described here. Read https://github.com/mlflow/mlflow/issues/572#issuecomment-427718078 for more details on the problem. However, we can circumvent this problem using Dask to executed the MLflow run on the remote server. Thereby, both the client and the MLflow tracking server has not problem reading and writing to the same folder, as the they are executed on the same machine.