简介 Brief Introduction


ZEN2 model, which uses N-gram to enhance text semantic and has 668M parameters, is adept at NLU tasks.

模型分类 Model Taxonomy

需求 Demand 任务 Task 系列 Series 模型 Model 参数 Parameter 额外 Extra
通用 General 自然语言理解 NLU 二郎神 Erlangshen ZEN2 668M 中文-Chinese

模型信息 Model Information


We open source and publicly release ZEN2 using our Fengshen Framework in collaboration with the ZEN team. More precisely, by bringing together knowledge extracted by unsupervised learning, ZEN learns different textual granularity information through N-gram methods. ZEN2 pre-trains the N-gram-enhanced encoders with large-scale datasets and special pre-training strategies. In the next step, we continue with the ZEN team to explore the optimization of PLM and improve the performance on downstream tasks.

下游效果 Performance

分类任务 Classification

Model(Acc) afqmc tnews iflytek ocnli cmnli
Erlangshen-ZEN2-345M-Chinese 0.741 0.584 0.599 0.788 0.80
Erlangshen-ZEN2-668M-Chinese 0.75 0.60 0.589 0.81 0.82

抽取任务 Extraction

Model(F1) WEIBO(test) Resume(test) MSRA(test) OntoNote4.0(test) CMeEE(dev) CLUENER(dev)
Erlangshen-ZEN2-345M-Chinese 65.26 96.03 95.15 78.93 62.81 79.27
Erlangshen-ZEN2-668M-Chinese 70.02 96.08 95.13 80.89 63.37 79.22

使用 Usage

模型下载地址 Download Address


加载模型 Loading Models


Since there is no structure of ZEN2 in transformers library, you can find the structure of ZEN2 and run the codes in Fengshenbang-LM.

git clone
from fengshen.models.zen2.ngram_utils import ZenNgramDict
from fengshen.models.zen2.tokenization import BertTokenizer
from fengshen.models.zen2.modeling import ZenForSequenceClassification, ZenForTokenClassification

pretrain_path = 'IDEA-CCNL/Erlangshen-ZEN2-668M-Chinese'

tokenizer = BertTokenizer.from_pretrained(pretrain_path)
model_classification = ZenForSequenceClassification.from_pretrained(pretrain_path)
model_extraction = ZenForTokenClassification.from_pretrained(pretrain_path)
ngram_dict = ZenNgramDict.from_pretrained(pretrain_path, tokenizer=tokenizer)


You can get classification and extraction examples below.

分类 classification example on fengshen

抽取 extraction example on fengshen

引用 Citation


If you are using the resource for your work, please cite the our paper for this model:

  title="{ZEN 2.0: Continue Training and Adaption for N-gram Enhanced Text Encoders}",
  author={Yan Song, Tong Zhang, Yonggang Wang, Kai-Fu Lee},
  journal={arXiv preprint arXiv:2105.01279},


If you are using the resource for your work, please cite the our overview paper:

  author    = {Junjie Wang and Yuxiang Zhang and Lin Zhang and Ping Yang and Xinyu Gao and Ziwei Wu and Xiaoqun Dong and Junqing He and Jianheng Zhuo and Qi Yang and Yongfeng Huang and Xiayu Li and Yanghan Wu and Junyu Lu and Xinyu Zhu and Weifeng Chen and Ting Han and Kunhao Pan and Rui Wang and Hao Wang and Xiaojun Wu and Zhongshen Zeng and Chongpei Chen and Ruyi Gan and Jiaxing Zhang},
  title     = {Fengshenbang 1.0: Being the Foundation of Chinese Cognitive Intelligence},
  journal   = {CoRR},
  volume    = {abs/2209.02970},
  year      = {2022}


You can also cite our website: