Skip to main content

Update a checkpoint vocabulary

Build vocabulary files for the new corpus, then point a complete training YAML at those files and the existing checkpoint. Existing token embeddings are mapped to the new vocabulary; newly added tokens receive initialized embeddings.

src_vocab: /path/to/new.src.vocab
training:
train_from: /path/to/checkpoint
update_vocab: true
reset_optim: states

For a separate target vocabulary, set tgt_vocab as well. The config validator requires reset_optim: states or all when updating a vocabulary. Keep tokenizer artifacts and vocabulary IDs consistent, especially for converted HF models.