ML-Monitoring

In this project were going to create a simple stock prediction model anduse it to predict yahoo BSD stock. Once a model has been created we're going to monitor the model using an ML-Monitoring tool like evidently. We will be loading the initial model and testiing it on a weekly basis to get a data report.

In this project, we will create a simple stock prediction model using machine learning and monitor it using Evidently.

Steps:

• Collect data: We will collect historical stock data for Yahoo BSD stock. This data can be obtained from a variety of sources, such as Yahoo Finance or Google Finance
• Train the model: We will train a machine learning model to predict the future price of Yahoo BSD stock. There are many different machine learning algorithms that can be used for this task, such as linear regression, support vector machines, and random forests.
• Monitor the model: Once we have trained a model, we will need to monitor its performance over time. This is important because the stock market is constantly changing, and our model may need to be updated regularly to maintain its accuracy.

Evidently is an ML-Monitoring tool that can be used to monitor our stock prediction model. Evidently provides a variety of metrics for evaluating the performance of our model, such as data drift , model quality and test reports. It also provides a variety of visualizations that can help us understand the performance of our model in more detail.

To monitor our stock prediction model using Evidently, we will follow these steps:

• Load the initial model: We will load the initial model into Evidently using MLFlow. Evidently supports a variety of machine learning frameworks, including scikit-learn, TensorFlow, and PyTorch.
• Test the model: We will test the model on a weekly basis using new data that has been collected since the model was trained. Evidently will generate a data report that shows the performance of the model on the new data.
• Analyze the data report: We will analyze the data report to identify any areas where the model is struggling. If necessary, we can update the model or collect more data to improve its performance.

By following these steps, we can create a simple stock prediction model and monitor it using Evidently to ensure that it is performing well over time.

Repository Structure

This repository uses the following directory structure:

├── README.md
├── Data_reports/ 
├── model/ # Sample Model
│   ├── conda.yml
|   ├── MLmodel
|   ├── model.pkl
    \ python_env.yaml
├── generate_report.py
├── handler.py
├── config.ini
├── requirements.txt
├── data_report.ipynb

• The README.md file is the main documentation file for the repository. It should contain information about the project, such as its purpose, how to use it, and how to contribute to it.

• The Data_reports/ directory contains the jupyter notebook reports that are generated when the models are run weekly. The files are sorted based on data and time and it helps you find reports easily

• The model/ directory contains a sample regression model. The code contains a model creation function that load the model to mlflow. But if you'd like to skip all that and use a previous model, try this.

• The generate_report.py file contains the core logic for the project. It has functions to generate_model, load a streamlit app to interactively get user credentials and model uri, create various kinds of reports and commit them to your repository

• The handler.py file contains logic to load an mlflow experiment and a function to load data.

• The config.ini file is a configuration file that contains your User and model information.

• The data_report.ipynb file is a template jupter notebook report that is used to create weekly reports

How to Use

To replicte the project, follow these steps:

1. Clone the repo

git clone https://dagshub.com/Nikitha-Narendra/ML-Monitoring.mlflow

2. Create a virtual envionment and activate it

   • To create a python virtual environment

     python3 -m venv model_report
    .\model_report\Scripts\activate
   

   • To create a conda environment

   conda create env -n model_report python==3.11
   conda activate model_report
   

3. Install the dependencies

   pip install -r requirements.txt
   
   

4. To run the demo:

   a. In the terminal, go to the location of your cloned repository

   cd {location of directory}
   

   b. Start the streamlit app

   streamlit run generate_report.py
   
   

   c. This will open a streamlit server in your browser.

   d. Fill out the form in the web-app and click Submit.

5. A jupyter notebook with the current data and time will be logged in the dagshub repository under the Data_reports folder.

Note: If you do not have a pretrained model on hand, leave the field blank and a default model will be loaded for you.