@she_llac i have had some issues sometimes depending on the wording but most of the time fable is rather effective at helping; just can't let it touch raw data because that does make it go boom very often.
@iamgrigorev (embeddings and nothing else). so you do small corpora, find the knee where geometry starts to settle, then you probe and profile the layers and see where they move in concert or quickly diverge, those divergence points being where the representation has solidified.
@iamgrigorev it's kind of an accepted fact that stable interlayer circuitry builds representation and then utilizes it; esp in the mid layers. and the embedding layer being isolated is also a strong boon towards isolating exploding residuals
@iamgrigorev i do wonder why they stick with arbitrary placements across the board when it is plainly easy to place the blocks on meaningful breakpoints in uneven amounts; everything works the same and it's a simple bootstrap pass at <5% corpora to find affinity