Are you sure you want to delete this access key?
title | description |
---|---|
Version Control and ML Experimentation Tutorial with DagsHub – Project Setup | Delve into version control and track machine learning experiments with DagsHub. This tutorial guides you through data exploration, experiment tracking with DagsHub and DVC, and creating a model to classify Stack Exchange questions on machine learning, offering practical workflow enhancements at each step. |
This level of the tutorial covers setting up our project.
This includes the following tasks:
venv
and installing the needed requirements.If you are familiar with these steps, you can skip to the next level, where we train some models and set up data and model versioning.
...is really easy. Just sign up.
Then, after logging in, create a new repo, simply by clicking on the plus sign and create a repository in the
navbar.
Done with repo creation. On to project initialization.
Create a directory named dagshub_tutorial
for the project somewhere on your computer.
Open a terminal and input the following:
cd path/to/folder/dagshub_tutorial
git init
Now, we will set the remote to our repo on DagsHub. This can be done using the following command:
git remote add origin https://dagshub.com/<username>/<repo-name>.git
Finally, let's create 2 folders, for each of our main project components.
mkdir -p data outputs
This will add a clear structure to the project.
We assume you have a working Python 3 installation on your local system for the following explanations.
!!! warning
To ensure that you do, you can open a terminal and type in python3 -V
.
See that this command succeeds and that you get at least version 3.7 - if it's smaller or if the command fails,
you should download the correct version for your operating system.
To create and activate our virtual python environment
using venv
,
type the following commands into your terminal (still in the project folder):
=== "Linux/Mac" ```bash python3 -m venv .venv echo .venv/ >> .gitignore echo pycache/ >> .gitignore
source .venv/bin/activate
```
=== "Windows" ```batch python3 -m venv .venv echo .venv/ >> .gitignore echo pycache/ >> .gitignore
.venv\Scripts\activate.bat
```
The first command creates your virtual environment - a directory named .venv
, located inside your project directory,
where all the Python packages used by your project will be installed without affecting the rest of your computer.
The second and third commands make sure the virtual environment packages and pycache are not tracked by Git.
The fourth command activates our virtual python environment, which ensures that any python packages we use don't contaminate our global python installation.
The rest of this tutorial should be executed in the same shell session.
If you exit the shell session or want to create another, make sure to activate the virtual environment in that shell session first.
To install the requirements for the first part of this project, simply download this requirements.txt into your project folder.
These are the direct dependencies:
dagshub==0.3.8.post2
dvc==3.49.0
dvc-s3==3.1.0
joblib==1.3.2
mlflow==2.8.0
pandas==2.1.2
scikit-learn==1.3.2
Now, to install them type:
pip3 install -r requirements.txt
We'll keep our data in a folder named, oddly enough, data
.
It's also important to remember to add this folder to .gitignore
!
We definitely don't want to accidentally commit large data files to Git.
The following commands should take care of everything:
mkdir -p data
echo /data/ >> .gitignore
wget https://dagshub-public.s3.us-east-2.amazonaws.com/tutorials/stackexchange/CrossValidated-Questions-Nov-2020.csv -O data/CrossValidated-Questions.csv
Let's check the Git status of our project:
$ git status -s
?? .gitignore
?? requirements.txt
Now let's commit this to Git and push to DagsHub using the command line:
git add .
git commit -m "Initialized project"
git push -u origin master
You can now see the setup files on your DagsHub repo. So far so good.
In the next level, we'll prepare our data processing code and use DVC to keep track of our data and model versions.
Press p or to see the previous file or, n or to see the next file
Are you sure you want to delete this access key?
Are you sure you want to delete this access key?
Are you sure you want to delete this access key?
Are you sure you want to delete this access key?