The MIT Supercloud Dataset

Stream data with DDA:

from dagshub.streaming import DagsHubFilesystem

fs = DagsHubFilesystem(".", repo_url="https://dagshub.com/DagsHub-Datasets/dcc-dataset")

fs.listdir("s3://mit-supercloud-dataset")

Description:

Collection of parsed datacenter logs and time series data of hardware utilization from the MIT Supercloud system.

Contact:

Collection of parsed datacenter logs and time series data of hardware utilization from the MIT Supercloud system.

Update Frequency:

Data will be updated annually

Managed By:

Siddharth Samsi

Resources:

1. resource:
   • Description: The MIT Supercloud Dataset
   • ARN: arn:aws:s3:::mit-supercloud-dataset
   • Region: us-west-2
   • Type: S3 Bucket

Tags:

datacenter, HPC, cloud computing, workload analysis, energy, aws-pds

Tutorials:

1. tutorial:
   • Title: Introduction to the MIT Supercloud Dataset
   • URL: https://nbviewer.org/github/MIT-AI-Accelerator/MIT-Supercloud-Dataset/blob/main/notebooks/Intro-to-MIT-Supercloud-Dataset.ipynb
   • AuthorName: Matthew Weiss
   • NotebookURL: https://github.com/MIT-AI-Accelerator/MIT-Supercloud-Dataset/blob/main/notebooks/Intro-to-MIT-Supercloud-Dataset.ipynb
   • Services: SageMaker Studio Lab
2. tutorial:
   • Title: Generating the labelled dataset
   • URL: https://nbviewer.org/github/MIT-AI-Accelerator/MIT-Supercloud-Dataset/blob/main/notebooks/Intro-to-MIT-Supercloud-Dataset.ipynb
   • NotebookURL: https://github.com/MIT-AI-Accelerator/MIT-Supercloud-Dataset/blob/main/notebooks/Intro-to-MIT-Supercloud-Dataset.ipynb
   • AuthorName: Matthew Weiss
   • AuthorURL: https://github.com/MIT-AI-Accelerator/workload-classification
   • Services: SageMaker Studio Lab

Tools & Applications:

1. tools & applications:
   • Title: The MIT Supercloud Workload Classification Challenge
   • URL: https://github.com/MIT-AI-Accelerator/workload-classification
   • AuthorName: Benny J. Tang, Qiqi Chen, Matthew L. Weiss, et. al.

Publication:

1. publication:

   • Title: The MIT Supercloud Dataset
   • URL: https://arxiv.org/abs/2108.02037
   • AuthorName: Siddharth Samsi, Matthew Weiss, David Bestor, et. al.

2. publication:

   • Title: AI-Enabling Workloads on Large-Scale GPU-Accelerated System: Characterization, Opportunities, and Implications
   • URL: https://doi.ieeecomputersociety.org/10.1109/HPCA53966.2022.00093
   • AuthorName: Baolin Li, Rohin Arora, Siddharth Samsi, et. al.
   • AuthorURL: https://github.com/boringlee24/HPCA22_SuperCloud

3. publication:

   • Title: The MIT Supercloud Workload Classification Challenge
   • URL: https://arxiv.org/abs/2204.05839
   • AuthorName: Benny J. Tang, Qiqi Chen, Matthew L. Weiss, et. al.
   • AuthorURL: https://github.com/MIT-AI-Accelerator/workload-classification