Jan Tempus

Doing some research on tokenisation.

[Jan Tempus @janallant]

@unixpickle This would only allow one to make runs in the future faster. Or take the output of the clustering and ask codex to try and tease structure out of it for other tokenisers. As language is so structured I would not be surprised if one can build heuristics for good cuts from the data

[Jan Tempus @janallant]

@unixpickle This means I would construct a second graph where the vertices correspond to constraints and an edge is if they are tight simultaneously. I would then probably run spectral clustering on this graph and see if there is any structure.

[Jan Tempus @janallant]

@unixpickle For the tokeniser of Pride and Prejudice how many constraints did you start with and how many did you start with and end with?

[Jan Tempus @janallant]

@unixpickle Super cool blog post!

[Jan Tempus @janallant]

@unixpickle So due to how bpe works it forces strings such as "like elephants" to be a single token or one splits it into "lik" "e e" "lephants". And if i remember correctly such vocabularies containing the second set of tokens lead to worse model performance.

[Jan Tempus @janallant]

@unixpickle I think dropping the pretokenizer would be interesting. Especially as the pretokenizer is added to stop bpe adding "e e" as this happens quite often in the english language.
Also with your technique how well does it generalize?
As I think overfitting is an issue

[Jan Tempus @janallant]

@unixpickle I found a lot more success with interior point methods and in the end PDLP. PDLP does have the issue it does not give back a vertex of the polytope, but one can set the cross over one gets a vertex. Also gurobi recently had a talk on tricks for cross over

[Jan Tempus @janallant]

@unixpickle Paper i mentioned https://t.co/n6xZgBM7xi

[Jan Tempus @janallant]

@unixpickle

[Jan Tempus @janallant]

@unixpickle The main issues with these hierarchies is that for each level the number of constraints grows polynomially with the level. So applying them naively one would add |variables|^level constraints. See https://t.co/V9hrlo4E6q for a really good introduction to these techniques

[Jan Tempus @janallant]

In our new paper, we reinterpret tokenisation as a problem in high-dimensional geometry (100M dims to be precise!), which we can solve efficiently to get a globally near-optimal tokeniser! Our method consistently improves language models over BPE. See 🧵for details.

[Jan Tempus @janallant]

@unixpickle So there are a number of hierarchies, Sum of Squares, Sherali Adams, Lovász-Schrijver which are automatically tighten the relaxation, and most (if i remember correctly) strengthened relaxations come from one of them

[Jan Tempus @janallant]

@AlexShtf @tpimentelms @mosektw @gurobi No so we didn't consider the various integer program solves. With the measurements and some tricks I do believe I remove the integrality gap, so I don't believe they are necessary. But I personally just really like linear programming

[Jan Tempus @janallant]

@guilhermeotina @tpimentelms No so the the relaxation becomes tighter as the vocabulary scales. At the largest vocabulary size we considered 262144, they were all effectively integral

[Jan Tempus @janallant]

@sasuke___420 As in how we precisely tokenise the data? I used unigram tokeniser with constant weights such that everything is hugging face compatible. But you can take any algorithm which finds the shortest path. Just fix the vocab and the problem is now a shortest path problem

[Jan Tempus @janallant]

@evijit Yes, very much so. Already sent the paper to someone thinking about tokenisation for DNA. I would however expect that one will want to alter the graph, so like having lower/upper bounds on the the length of the tokens. And potentially tweak the objective