sentencepiece src 4b8098c91303b222b540e2aaee3cfbae8c188422607234e927930cf27d52b651 An unsupervised text tokenizer for Neural Network-based text generation The SentencePiece is an unsupervised text tokenizer for Neural Network-based text generation. It is an unsupervised text tokenizer and detokenizer mainly for Neural Network-based text generation systems where the vocabulary size is predetermined prior to the neural model training. SentencePiece implements subword units and unigram language model with the extension of direct training from raw sentences. SentencePiece allows us to make a purely end-to-end system that does not depend on language-specific pre/post-processing. angrypenguin <angrypenguinpoland@gmail.com> https://github.com/google/sentencepiece