RSPA (Recursive Sub-word Pruning Algorithm) is a new C++ tokenizer 38x faster than GPT-4's tiktoken.
It uses Branching Entropy for cleaner splits (e.g. Hyperactive -> Hyper, active). Trains on 500MB in seconds.
Code: https://t.co/ec64avQUF7
#AI#NLP#cpp#Tokenization
@SidMahadev_ @choehau_alen @Mrwhosetheboss@MKBHD@GyanTherapy As far as productive task is concerned its absolutely fine... moreover the feel is nice...typing is better than any other lap in the same price...
Negative: overpriced.. could expect an gpu in the other brands at the same price