How can I update a checkpoint's vocabulary?
New vocabulary can be used to continue training from a checkpoint. Existing vocabulary embeddings will be mapped to the new vocabulary, and new vocabulary tokens will be initialized as usual.
Run eole build_vocab as usual with the new dataset. New vocabulary files will be created.
Training options for the yaml file to perform vocabulary update are:
update_vocab: set this optionreset_optim: set the value to "states"train_from: checkpoint path