Reformer Reproducibility Experiments

Fastai community entry to 2020 Papers With Code Reproducibility Challenge

Our Reproducibility Challenge Submission

• Our OpenReview paper submission to the challenge can be found here
• Our Weights & Biases Report, with interactive charts, is available here

Installation

Setup

If you don't already, its a good idea to install the package into a virtual environment

python3 -m venv my_env
source ./my_env/bin/activate

Install

Then you can install the package via pip:

pip install reformer-fastai

Or (even better) install latest version from github:

pip install git+git://github.com/arampacha/reformer_fastai.git

Contributing

This project used nbdev for all development, see their docs here to install nbdev and get started. Once you have nbdev installed we suggest you follow the suggested contributor workflow

Running Experiments

A pip installed version of this library is needed to run experiments. All experiments are run using the run_exp command, followed by the particular task name and then the parameters related to that task. run_exp --help will display a list of all parameters as well as a brief description. For brevity, an example of how to run a Reformer Language Model experiment is show below, a list of all experiment commands can be found here

Example: Reversible Language Model

Below is an example of the code used that generated the results in Section 4.4 "Effect of reversible layers" of our submission paper.

run_exp "lm_rev" \
        --n_epochs=10 \
        --bs=2 \
        --max_seq_len=4096 \
        --grad_accum=8 \
        --save_model=True  \
        --clip=0.5 \
        --seed=444 \
        --precision=2 \
        --do_wandb_logging=False \

Hyperparameters Used

The main hyperparameters used are documented in the Experiment Commands page and the Experiment Configs page. In addition, a full list of our hyperparameters can be found in the Run Sets tables of our Weights & Biases Report. To see these, navigate to the experiment of interests, click on the "Run Set" button under each chart and scroll across to find all hyperparameters.

Results

All full description of our results, including charts and tables can be found in our paper here on OpenReview. Our results are summarised as follows:

Claims around speed on longer sequences and reduced memory footprint were validated; as sequence length increased, Locality Sensitive Hashing ("LSH") Attention became faster and increasing the number of hashes improved performance. We could not achieve the performance of a traditional Transformer with Reformer. Some experiments were not run for as long as in the paper due to a lack of computational resources. Potentially the under-performance of our Reformer may be due to under-training, implementation differences or nuances in JAX vs Pytorch. Also, exploding gradients were encountered with mixed precision training and several model settings were found to be unstable depending on the random seed or learning rate.

Trained Models

All trained models from this project can be found in our Weights & Biases project here

Evaluation

An notebook example of running evaluation on one of our trained models saved on Weights & Biases can be found here

Project Links

• Our OpenReview paper submission
• Reformer Reproducibility Report on WandB
• Our project documentation
• Fastai forums thread
• Google doc used for early planning

Resources

Author's Code and Resources

• Reformer Paper
• Authors ICLR video
• Google Blog
• Authors code (TRAX)
• Reformer enwik8 model and training config

More Code

• @lucidrain’s Reformer code
• HuggingFace: Reformer source code
  • Reformer enwik8 configs
  • Reformer WMT14 en-de configs
  • Reformer Machine Translation example = SubwordTextEncoder tokenizer used for Machine Translation
• HuggingFace: Reformer notebook example
• HuggingFace: long sequences
• HuggingFace: Pretraining

Data

Tokenizers used with these datasets can be found here

enwik8

• enwik8.zip, raw data, 100mb
• Tensor2Tensor enwik8 data generator code, with train/dev/test split. File lengths:
  • Train: 89,621,832
  • Eval: 5,000,000
  • Test: 5,000,000
• Tokenier used: ByteTextTokenizer

WMT14

• WMT on HuggingFace Datasets
• Reformer pre-trained WMT14 vocab
  • Vocab size = 33300, from WMT14 model config
• Train Test split: newstest2013 for validation and newstest2014 for test, in consistence with Vaswani et al. (2017) - from https://arxiv.org/pdf/2009.02070.pdf
• Tokenizer used: SubWordTextEncoder

Explainers

• Yannic K explainer
• HuggingFace blog post
• Illustrating the Reformer blog post
• Reformer Deep Dive

Related

• Coursera Attention Models in NLP course, with Reformer co-author
• @hallvagi Attention blogpost
• The Transformer Family by @lilianweng
• A Family of Attention Mechanisms bu @lilianweng