open-bmb

CPM1

CPM1 is a generative Chinese pre-trained language model with 2.6 billion parameters.
The architecture of CPM1 is similar to GPT and it can be used in various NLP tasks such as conversation, essay generation, cloze test, and language understanding.

GitHub

License

Features

Large Corpus

2.6 Billiion parameters, trained on 100GB Chinese corpus

Chinese Vocabulary

Build the multi-granularity vocabulary with both characters and words

Stable Training

Use multiple GPUs to increase the batch size for more stable model

Performance

The model performs well on multiple few-shot/zero-shot tasks

Performance on zero-shot text classification

Performance on zero-shot and few-shot question answering

Zero-shot (zs) and one-shot (os) results on Question Answering (QA) datasets, including DuReader (Zhidao and Search) and CMRC2018, we did experiments on models with three different sizes: small (s), medium (m) and large (l).