Swift Core ML implementations of Transformers: GPT-2, DistilGPT-2, BERT, DistilBERT, more coming soon!

This repository contains:

• For BERT and DistilBERT:

• pretrained Google BERT and Hugging Face DistilBERT models fine-tuned for Question answering on the SQuAD dataset.

• Swift implementations of the BERT tokenizer (BasicTokenizer and WordpieceTokenizer) and SQuAD dataset parsing utilities.

• A neat demo question answering app.

• For GPT-2 and DistilGPT-2:

• a conversion script from PyTorch trained GPT-2 models (see our transformers repo) to CoreML models.

• The GPT-2 generation model itself, including decoding strategies (greedy and TopK are currently implemented) and GPT-2 Byte-pair encoder and decoder.

• A neat demo app showcasing on-device text generation.