@nicdunz i think they mistakenly stolen my keyboard autocompletion 🫢 but in my keyboard its available globally and in any textfield you can try it here https://t.co/3quivcahbM
@MainzOnX i have given each dataset 256 tokens and said to ai to summarize then mapped it to represented token given represented token in input and summarized representation in output trained on it it correctly learned this 1000 encoded tokens giving only the encoded token in input it...
@MainzOnX during first prefill the l1 encodes for example 256k tokens into 1k representations like vqvae does for imgs into latent speed is normal the l2 and l3 uses sliding window its uses sliding window untill a next l1 representation for example its uses the same 1k l1 tokens kv
@MainzOnX textual encoding, for example the encoded representation is like a term of an english language where the term is only a single word and its definition is sometimes over a complete essay i use exactly this method an l1 representation is like a term and the and the 256 encoded
@MainzOnX this was not about precision but i did this so that i can train individual parameters separately i think you know the recent sakana individual blocks training but my approach was single parameter training and i used 3 staged attention 1 paragraphs, 2 sentences, 3 phrases words