nl2ml

Alexander Levin 2b35e06141
sync with GitLab
7c97b2a9a3
dvc data fix
1 year ago
82b810b974
last checkpoints added
1 year ago
97c5b91e60
code2vec folder added
1 year ago
2b35e06141
sync with GitLab
9 months ago
2b35e06141
sync with GitLab
9 months ago
ee690b0621
SVM conf matrix graph 3 and 3.1, validation chunk_size == 10
1 year ago
bdb584c714
Get rid of csv in Git LFS
1 year ago
2b35e06141
sync with GitLab
9 months ago
a4b1957697
in-code comments classification added
1 year ago
3e3cfc582f
Makefile
1 year ago
2b35e06141
sync with GitLab
9 months ago
ed844fb89f
Add files via upload
1 year ago
45a0bcfcdb
changed names
1 year ago
45a0bcfcdb
changed names
1 year ago
2b35e06141
sync with GitLab
9 months ago
999d24c49f
added: golden 884 set added;
1 year ago
2b35e06141
sync with GitLab
9 months ago
3f08f4b1c5
exploring..
1 year ago
fc9e9f9c28
ramazyant files added
1 year ago
fc9e9f9c28
ramazyant files added
1 year ago
2b35e06141
sync with GitLab
9 months ago
2b35e06141
sync with GitLab
9 months ago
38ffcf58fa
upd: logreg weights analysis moved to separate notebook
1 year ago
2b35e06141
sync with GitLab
9 months ago
0b9b0d2347
upd: graph_v4, graph_v5 models and data
1 year ago
2b35e06141
sync with GitLab
9 months ago
08a39d0906
open_data folder (with Python150k) added
1 year ago
2b35e06141
sync with GitLab
9 months ago
2b35e06141
sync with GitLab
9 months ago
7673348010
upd: clear outputs;
1 year ago
2b35e06141
sync with GitLab
9 months ago
2b35e06141
sync with GitLab
9 months ago
d703f657c1
getting inference for logreg v3 and v3.1;
1 year ago
2b35e06141
sync with GitLab
9 months ago
Data Pipeline
Legend
DVC Managed File
Git Managed File
Metric
Stage File
External File

README.md

Source Code Classification

This is a repo of the Natural Language to Machine Learning (NL2ML) project of the Laboratory of Methods for Big Data Analysis at Higher School of Economics (HSE LAMBDA).

The project's official repo is stored on GitLab (HSE LAMBDA repository) - https://gitlab.com/lambda-hse/nl2ml \ The project's full description is stored on Notion - https://www.notion.so/NL2ML-Corpus-1ed964c08eb049b383c73b9728c3a231 \ The project's experiments are stored on DAGsHub - https://dagshub.com/levin/source_code_classification

Project Goals

Short-Term Goal

To build a model classifying a source code chunk and to specify where the detected class is exactly in the chunk (tag segmentation).

Long-Term Goal

To build a model generating code by getting a short raw english task in as an input.

Repository Description

This repository contains instruments which the project's team has been using to label source code chunks with Knowledge Graph vertices and to train models to recognize these vertices in future. By the Knowledge Graph vertices we mean an elementary part of ML-pipeline. The current latest version of the Knowledge Graph contains the following high-level vertices: ['import', 'data_import', 'data_export', 'preprocessing', 'visualization', 'model', 'deep_learning_model', 'train' 'predict'].

Data Download

To download the project data and models:

  1. Clone this repository
  2. Install DVC from https://dvc.org/doc/install
  3. Do dvc pull data or dvc pull data. Note: if you are failing on dvc pull [folder_to_pull], try dvc pull [folder_to_pull] --jobs 1

Contents:

The instruments which we have been using to reach the project goals are: notebooks parsing from Kaggle API and Github API, data preparation, regex-labellig, training models, validation models, model weights/coefficients analysis, errors analysis, synonyms analysis.

nl2ml_notebook_parser.py - a script for parsing Kaggle notebooks and process them to JSON/CSV/Pandas.

bert_distances.ipynb - a notebook with BERT expiremints concerning sense of distance between BERT embeddings where input tokens were tokenized source code chunks.

bert_classifier.ipynb - a notebook with preprocessing and training BERT-pipeline.

regex.ipynb - a notebook with creating labels for code chunks with regex

logreg_classifier.ipynb - a notebook with training logistic regression model on the regex labels with tf-idf and analyzing the outputs

Comments vs commented code.ipynb - a notebook with a model distinguishing NL-comments from commented source code

github_dataset.ipynb - a notebook with opening github_dataset

predict_tag.ipynb - a notebook with predicting class label (tag) with any model

svm_classifier.ipynb - a notebook with training SVM (replaced by _svmtrain.py) and analyzing SVM outputs

svm_train.py - a script for training SVM model

Conventions:

  • Input CSV: encoding='utf-8', sep=',' and CODE_COLUMN has to be == 'code_block' in all input CSVs
  • Knowledge Graphs: GRAPH_DIR has to be in the following format: './graph/graph_v{}.txt'.format(GRAPH_VER)