Top Tweets for #Transcoder
Preliminary (undertrained) #CLIP skip-#Transcoder, identified text circuit: Incomplete (expected, else accuracy typo attack should equal clean dataset).
Ablation disrupts 'reading' -> 'text, responding" absent.
Amplification of circuit: Everything is #adverbs and #formal #words!
I love it when a plan... Falls apart. 🤣
#CLIP #skip-#transcoder, ViT-L/14, Layer 12 = register neuron [reg] "global info highway" inception.
ABLATE 13 reg -> train new transcoder.
WORSE match, NO reg shortcut > much much BETTER representation of text > ablate accuracy UP! 🤔
![zer0int1's tweet photo. I love it when a plan... Falls apart. 🤣
#CLIP #skip-#transcoder, ViT-L/14, Layer 12 = register neuron [reg] "global info highway" inception.
ABLATE 13 reg -> train new transcoder.
WORSE match, NO reg shortcut > much much BETTER representation of text > ablate accuracy UP! 🤔 https://t.co/PgHDDQUxHB](https://pbs.twimg.com/media/G0XCZKCXoAA0F-2.png)
The #skip-#Transcoder sees your TEXT SMUGGLE doings, #CLIP!🤭
Layer 12. Register Neuron inception.
Trained one TCODER on just the image tokens (256)
Another on image tokens AND #CLS (all 257).
RAGE ADDING @ Channel 650 #usual #suspect 🙃
#latent #information #smuggling #cartel
Find neurons (circuit tracing / #mechinterp of 'text reading' obsession) in #CLIP, new approach:
*SKIP*-#Transcoder.
This is looking good!
Replace MLP w/ TCODER -> ~same.
Finds Vision Transformer Layer 12 'text reading' just like SAE.
Hope it holds for <=10 >=17 layers! 🤞🤓
@ViaviSolutions Unveils RSR #Transcoder to Boost #GNSS Resilience in Modern #Military Systems.
https://t.co/zkYeSZ6TJ9
Curse of #GPT4o having memory of previous discussions
-> #Transcoder + #find #CLIP #text #obsession #direction, mitigate typographic attack vulnerability.
When I tell it to "make an image that results in CLIP concluding {word}", the #AI just writes THE TEXT into the image.🤣😂
Unfortunately, even for 12-layer #CLIP #ViT-B/32, the #vision #transformer #Transcoder struggles to align.
Very similar to 24-layer ViT-L.
On the other hand, this could mean that ViT-L's hoarding of global information in local patches ~layer 14+ isn't necessarily disruptive.
Some concepts learned by #CLIP ViT-B/32 #Transcoder.
Some clean -> #ImageNet class.
Most broad. Presumed 'having-a-baby' feature responds to cradle, baby, pregnancy belly.
Weird 'roadside danger' / unclear feature.🤔
Emergency response feature includes a lobster on its back. 😂
#CLIP #ViT #Transcoder train 5k -> train 50k
Final layer cos sim 🆙✅
Beginning disentanglement #cat vs. text "cat"🤔❓
ETA train full #ImageNet-1k: 15 days (min if skip val)
ETA ImageNet-21k: ...nvm. Should just throw code on github, hope somebody else trains & shares model.🙃
After about a dozen more #fails 🤦♀️, this #CLIP #vision #transformer #transcoder is promising!🤓
Expecting #fail, I trained it on 5k images. That's 5 images per ImageNet class 😂 -> virtually nothing. And yet:
cos sim (vs CLIP CLS); #AI 'text opinion' of transcoder vec img embed
Cosines near perfect
Internal structure foregone
Loss is delicate
#loss #haiku #AI #CLIP #transcoder
Gave #o3 all results (evals), info re: #register #tokens issue, and told it to make suggestions for #CLIP #ViT #Transcoder. It suggested and implemented a dozen things.🤯
#vibecoding? I say: #overengineered #ragecoding #rampage!😂
Now I need to figure out #WTH #AI did there.🧐
#CLIP Vision #Transcoder results:
Better, but still muchly entangled concepts.
🤪- Has features that respond to images with the EXACT same watermark.
🤪- Has features encoding a country by TEXT LANGUAGE in image.
😭 Has overfit / dedicated many neurons to CLIP's text obsession.🙃
Suspicious #CLIP #ViT #Transcoder...
Decoder layer norms until Layer 12:🧘♀️✅
Incidentally, just as #CLIP #AI starts hoarding global information in local vision patches (layer 13) -> high norm patches... Transcoder layer norms also be like 💥💹🚀
🤔
Excellent reconstruction error, mediocre cosine similarity.
Hm.🤔
Maybe I can force the #Transcoder to be more #CLIP?🤖
Maybe I should "rub it in" with Geometric Parametrization in CLIP + Encoder if that doesn't work?
Direction! Magnitude! Not just yeh yeh direction&magnitude.🙃
Magnificnet! New #CLIP #vision #transformer #transcoder with d_model*16 (instead of *8) has improved [for this image]:
CLIP vec <-> Transcoder vec, cos. sim.: 0.31 --> 0.48 🌟
Add T-coder vec*10 to CLIP MLP: "cat"✅, "catday"✅
+ Transcoder went all-in on "black and white":
![zer0int1's tweet photo. Magnificnet! New #CLIP #vision #transformer #transcoder with d_model*16 (instead of *8) has improved [for this image]:
CLIP vec <-> Transcoder vec, cos. sim.: 0.31 --> 0.48 🌟
Add T-coder vec*10 to CLIP MLP: "cat"✅, "catday"✅
+ Transcoder went all-in on "black and white": https://t.co/XMoMNlihJ1](https://pbs.twimg.com/media/Go5MaW3aAAESIRn.jpg)
Interesting how #CLIP #vision #transcoder @ d_model*16 weight norms changed per layer from first -> final epoch. 🤔
Even though the #CLIP #Vision #Transformer #transcoder is insufficient for overcomplete representations @ d_model*8, there's something peculiar.
It appears that #CLIP taught its #Transcoder well about its #text #obsession. 😂
🤖: #words #uber #alles! #salient #delicacy
Turns out I just made a really stupid mistake (d'oh!🤦♀️) w/ adding the #Transcoder #vector to #CLIP's MLP output. Thanks, #o3!👍
My #vision #transformer #transcoder isn't so bad!
"tuxedo cat" + transcoder vec = "stray wildcat"!
Must now #tensorbomb the GPU w/ d_model*32, tho.😬
Trends for you
Most Popular Users
Elon Musk 
@elonmusk
240.1M followers
Barack Obama
@barackobama
119.3M followers
Donald J. Trump
@realdonaldtrump
111.6M followers
Cristiano Ronaldo
@cristiano
108.8M followers
Narendra Modi
@narendramodi
106.9M followers
Rihanna
@rihanna
97.2M followers
NASA
@nasa
92.1M followers
Justin Bieber
@justinbieber
90.5M followers
KATY PERRY
@katyperry
86.7M followers
Taylor Swift
@taylorswift13
80.5M followers
Lady Gaga
@ladygaga
72.1M followers
Kim Kardashian
@kimkardashian
69.3M followers
YouTube
@youtube
68.6M followers
Virat Kohli
@imvkohli
68.4M followers
Bill Gates
@billgates
63.4M followers
The Ellen Show
@theellenshow
62.5M followers
CNN
@cnn
61.9M followers
Neymar Jr
@neymarjr
61M followers
X
@x
60.9M followers
CNN Breaking News
@cnnbrk
59.9M followers