A minimalistic integration of DVC with a simple Jupyter Notebook.
Using this guideline, you can keep working in a notebook while enjoying most of the benefits of data and model versioning.
For more information, see the README in the example project.
|DAGsHub-Official 9a91531027 Update 'README.md'||3 months ago|
|.dvc||3 months ago|
|data||3 months ago|
|metrics||3 months ago|
|.gitignore||3 months ago|
|Example.ipynb||3 months ago|
|LICENSE||3 months ago|
|README.md||3 months ago|
|eval.dvc||3 months ago|
|models.dvc||3 months ago|
|process_data.dvc||3 months ago|
|requirements.txt||3 months ago|
|DVC Managed File|
|DVC Managed File|
A minimalistic integration of DVC with a simple Jupyter Notebook. Using this guideline, you can keep working in a notebook while enjoying most of the benefits of data and model versioning.
env/directory. Git is already configured to ignore it.
pip install -r requirements.txt
pip install jupyter && jupyter notebook.
This project structure is as an example of how to work with DVC from inside a Jupyter Notebook.
This workflow should enable you to enjoy the full benefits of working with Jupyter Notebooks, while getting most of the benefit out of DVC - namely, reproducible and versioned data science.
The project takes a toy problem as an example - the California housing dataset, which comes packaged with scikit-learn. You can just replace the relevant parts in the notebook with your own data and code. Significantly different project structures might require deeper intervention.
The idea is to leverage DVC in order to create immutable snapshots of your data and models as part of your git commits. To enable this, we created the following DVC stages:
data/raw/, versioned in
data/processed/, versioned in
models/, versioned in
metrics/metrics.json, versioned as part of the git commit and referenced in
Unlike a typical DVC project, which requires you to refactor your code into modules which are runnable from the command line, In this project the aim is to enable you to stay in your comfortable notebook home territory.
So, instead of using
dvc repro or
dvc run commands, just run your code as you normally would in
We prepared special cells (marked with green headers) inside this notebook that let you run
dvc commit commands on the relevant
DVC stages defined above, immediately after you create the relevant data files from your notebook code.
dvc commit computes the hash of the versioned data and saves that hash
as text inside the relevant
.dvc file. The data itself is ignored and not versioned by git, instead being versioned with DVC.
.dvc files, being plain text files, ARE checked into git.
So, to summarize, this workflow should enable you to create a git commit which contains all relevant code, together with references to the relevant data and the resulting models and metrics. Painless reproducible data science!
It's intended as a guideline - definitely feel free to play around with its structure to suit your own needs.
To create a project like this, just go to https://dagshub.com/repo/create and select the Jupyter Notebook + DVC project template.
Made with 🐶 by DAGsHub.