No Description

Alexander Levin a454800e04 Merge branch 'method_parse' into 'master' 3 days ago
.dvc bd896b3eeb copy from github; 7 months ago
.ipynb_checkpoints bd896b3eeb copy from github; 7 months ago
analysis 9de45f715a moved to "analysis" 3 months ago
comment_cleanup 1399905151 upd: change model folder 1 week ago
data
data_archived
data_mini
deprecated c21e8f21f5 Merge branch 'master' into 'prep/comment_cleanup' 1 week ago
form 2b42a01e04 add graph downloader 1 week ago
graph c7055f5394 add_new_graph_version 1 week ago
methods_parse 0f0243fdca add_new_graph_version_ 4 days ago
models
models_scripts f34bd7bb4f Merge branch 'regex_for_graph_7' into 'master' 3 days ago
open_data
parser fb6374832a moved to auto_features 1 week ago
.gitattributes f23a7f09e6 git cloned from https://github.com/whatevernevermindbro/source_code_classification 8 months ago
.gitignore 41a86073ac added: data_archived 1 week ago
Makefile bd896b3eeb copy from github; 7 months ago
README.md ae877d8024 added: conventions 4 months ago
data.dvc c5a26b9007 update markup_data 3 days ago
data_archived.dvc 41a86073ac added: data_archived 1 week ago
data_mini.dvc 981c3da34a added: competitions_info_cleaned.csv 2 weeks ago
dvc.lock 319f6a2523 added: DVC YAML with regex stage of pipeline 2 months ago
dvc.yaml 1b6b911c09 upd: commented 1 month ago
metrics.csv 714e05aa83 added: SVM v6 2 months ago
models.dvc ef0814e233 add: model for comment cleanup added to dvc in model folder 2 weeks ago
nl2ml.pptx dc835f45bb upd: github sync; 6 months ago
open_data.dvc bd896b3eeb copy from github; 7 months ago
params.yaml a76a869340 added: templating (not tested yet); 2 months ago
params.yml 714e05aa83 added: SVM v6 2 months ago
requirements.txt dcb234b9ff init: requirements file 4 months ago

Data Pipeline

Legend
DVC Managed File
Git Managed File
Metric
Stage File
External File

README.md

Source Code Classification

This is a repo of the Natural Language to Machine Learning (NL2ML) project of the Laboratory of Methods for Big Data Analysis at Higher School of Economics (HSE LAMBDA).

The project's official repo is stored on GitLab (HSE LAMBDA repository) - https://gitlab.com/lambda-hse/nl2ml \ The project's full description is stored on Notion - https://www.notion.so/NL2ML-Corpus-1ed964c08eb049b383c73b9728c3a231 \ The project's experiments are stored on DAGsHub - https://dagshub.com/levin/source_code_classification

Project Goals

Short-Term Goal

To build a model classifying a source code chunk and to specify where the detected class is exactly in the chunk (tag segmentation).

Long-Term Goal

To build a model generating code by getting a short raw english task in as an input.

Repository Description

This repository contains instruments which the project's team has been using to label source code chunks with Knowledge Graph vertices and to train models to recognize these vertices in future. By the Knowledge Graph vertices we mean an elementary part of ML-pipeline. The current latest version of the Knowledge Graph contains the following high-level vertices: ['import', 'data_import', 'data_export', 'preprocessing', 'visualization', 'model', 'deep_learning_model', 'train' 'predict'].

Data Download

To download the project data and models:

  1. Clone this repository
  2. Install DVC from https://dvc.org/doc/install
  3. Do dvc pull data or dvc pull data. Note: if you are failing on dvc pull [folder_to_pull], try dvc pull [folder_to_pull] --jobs 1

Contents:

The instruments which we have been using to reach the project goals are: notebooks parsing from Kaggle API and Github API, data preparation, regex-labellig, training models, validation models, model weights/coefficients analysis, errors analysis, synonyms analysis.

nl2ml_notebook_parser.py - a script for parsing Kaggle notebooks and process them to JSON/CSV/Pandas.

bert_distances.ipynb - a notebook with BERT expiremints concerning sense of distance between BERT embeddings where input tokens were tokenized source code chunks.

bert_classifier.ipynb - a notebook with preprocessing and training BERT-pipeline.

regex.ipynb - a notebook with creating labels for code chunks with regex

logreg_classifier.ipynb - a notebook with training logistic regression model on the regex labels with tf-idf and analyzing the outputs

Comments vs commented code.ipynb - a notebook with a model distinguishing NL-comments from commented source code

github_dataset.ipynb - a notebook with opening github_dataset

predict_tag.ipynb - a notebook with predicting class label (tag) with any model

svm_classifier.ipynb - a notebook with training SVM (replaced by _svmtrain.py) and analyzing SVM outputs

svm_train.py - a script for training SVM model

Conventions:

  • Input CSV: encoding='utf-8', sep=',' and CODE_COLUMN has to be == 'code_block' in all input CSVs
  • Knowledge Graphs: GRAPH_DIR has to be in the following format: './graph/graph_v{}.txt'.format(GRAPH_VER)