Nejm Benessaiah

Anthropologist specialising in power, ecology and cyborg relations.

[Nejm Benessaiah @Nejben77]

My latest op-ed in http://t.co/Debf8v01w2: Limits to Freedom of Speech? When Legalism Replaces Humanism http://t.co/XDNVJcfbkZ

[Sayash Kapoor @sayashk]

I had way too much fun using Claude to create a reply video.

- Extremely fun to be able to convert scattered thoughts and ideas into a song + video

- Creating this took <1 hour of my time (in occasionally steering the model in between other things)

- But it took ~10 hours of wall clock time for the agent to generate the video + iteratively refine it based on feedback

- In terms of AI usage: it required only a small number of Suno credits and 10% weekly usage of a Claude Max subscription (I was using Opus 5.5)

[Nejm Benessaiah @Nejben77]

@rowancheung This quote is totally out of context, he says that there was initially a lot of fear around AI but people correctly tend to view them as tools.

[Rowan Cheung @rowancheung]

Wild week of AI developments.

Huge announcements from Microsoft, Bard, DALL-E 3, Amazon, YouTube, Viral AI art, Google, OpenAI, Emory University, Wiz Research, the UK government, DeepMind, Intel, UPS, and The Authors Guild.

Here's everything you need to know (thread):

[Jim Fan @DrJimFan]

Let's reverse engineer the phenomenal Tesla Optimus. No insider info, just my own analysis. Long read:

1. The smooth hand movements are almost certainly trained by imitation learning ("behavior cloning") from human operators. The alternative is reinforcement learning in simulation, but that typically leads to jittery motion and unnatural hand poses.

There're at least 4 ways to collect human demonstrations:
(1) A custom-built teleoperation system - I believe this is the most likely means used by Tesla team. Open-source example: ALOHA, a low-cost bimanual robot arm and teleoperation system by Stanford AI Labs (https://t.co/8iXpiHVEjS). It enables very precise, dexterous motions, such as putting AAA batteries into a remote or manipulating contact lens.

(2) Motion Capture (MoCap): apply the MoCap systems used for Hollywood movies to capture the fine-grained motions of hand joints. Optimus' 5-finger hand is a great design decision that enables a direct mapping - there is no "embodiment gap" from human operators.
For instance, a demonstrator can wear a CyberGlove (https://t.co/S8hxErsEuU) and grasp the cubes on the table (as shown in video). CyberGlove will capture the motion signals & haptic feedback in real-time, which can be re-targeted onto Optimus.

(3) Wearing gloves & markers can be clumsy. An alternative way to do MoCap is through computer vision. DexPilot from NVIDIA enables marker-less and glove-free data collection. The human operator simply uses their bare hands to perform the tasks. 4 Intel RealSense depth cameras and 2 NVIDIA Titan XP GPUs (yeah, 2019 work) translate the pixels to precise motion signals for robot learning.

(4) VR Headset: turn the training room into a VR game, and let humans "role play" Optimus. Use the native VR controller or CyberGlove to control the virtual Optimus hands. This has the advantage of scalable remote data collection - annotators from around the world can contribute without coming onsite.
VR demonstration technique appeared in research projects like the iGibson home robot simulator, an initiative that I participated in at Stanford: https://t.co/eyI4ORkH6G

Above 4 are not mutually exclusive. Optimus could use a combo of them for different pros & cons.

2. Neural Architecture. Optimus is trained end-to-end: videos in, actions out. I'm quite sure it's implemented by a multimodal Transformer with the following components:

(1) Image: some variant of efficient ViT, or simply an old ResNet/EfficientNet backbone (https://t.co/L6PLTQJnGA). The block pick-and-place demo doesn't require sophisticated vision. The spatial feature map from the image backbone can be tokenized easily.

(2) Video: two ways. Either flatten the video into a sequence of images and produce tokens independently, or have a video-level tokenizer. There're numerous ways to efficiently process video pixel volumes. You don't necessarily need Transformer backbones, e.g. SlowFast Network (https://t.co/qDdXzqwJQp) and RubiksNet (https://t.co/CQU8D7TZgx, my paper at ECCV 2020, efficient CUDA shift primitives).

(3) Language: it's not clear if Optimus is language prompted. If it is, there needs to be a way to "fuse" the language representations into perception. FiLM is a very lightweight neural network module that serves this purpose (https://t.co/VI4TpgQ22V).
You can think of it intuitively as a "cross attention" of language embedding into the image-processing neural pathway.

(4) Action tokenization: Optimus needs to convert the continuous motion signals into discrete tokens for the autoregressive Transformer to work. A few ways:
- Directly bin the continuous values for each hand joint control. [0, 0.01) -> token #0, [0.01, 0.02) -> token #1, etc. This is straightforward but could be inefficient due to the long sequence length.
- The joint movements are highly dependent on each other, which means they occupy a low-dimensional "state space". Apply VQVAE to the motion data to obtain a shorter-length, compressed token set.

(5) Putting the above pieces together, we have a Transformer controller that consumes video tokens (optionally with language modulation), and outputs action tokens, one step at a time. The next frame from the table is fed back to the Transformer, so it knows the consequence of its action. That gives the self-corrective ability shown in the demo.

I believe the architecture is most similar to:
- Google RT-1: https://t.co/dpuon1bqU6
- NVIDIA VIMA: https://t.co/Tn3L63uGrv

3. Lastly, I'm genuinely impressed by the hardware quality. The motions are fluid, and the aesthetics is amazing as well. As I mentioned above, it's such a great decision to follow human morphology closely, so that there is no gap in imitating humans.

Atlas from Boston Dynamics only has simple gripper-style hands. In the long run, Optimus' bi-dexterous, 5-finger hands will prove far superior in daily tasks.

Congrats to @Tesla_Optimus team & @elonmusk 🎉! I look forward to seeing the bots roam Mars some day 🦾

[Nejm Benessaiah @Nejben77]

News: most carbon offset schemes are ‘junk’ https://t.co/iAm8uQ9Ngg

[Jon Finger @mrjonfinger]

Testing out @HeyGen_Official translation on French and German. I don’t speak either language so let me know if it sounds natural if you do.
I hope if you pay you can turn off the color correction.
It didn’t work on my phone so I had to upload on my pc.
https://t.co/FMJp9sJEBI

[CarlSaganDotCom @carlsagandotcom]

A lesson on the greenhouse effect with Carl Sagan.

🎥: Carl Sagan testifying before Congress in 1985 on climate change

[Paul Williams @freemonotheist]

Oxford University has published a map showing the oldest universities in the world:
1. Ez-Zitouna University in Tunisia (737)
2. Al-Qarawiyyin University in Morocco (859)
3. Al-Azhar University in Egypt (972)
4. University of Bologna in Italy (1088)
5. University of Oxford in England (1096)
The first and oldest three universities in history were in the Arab/Muslim world. https://t.co/qTft4NQhAM

[Middle East Institute @MiddleEastInst]

As mentioned in our Climate Week panel on Water Management &amp; Environmental Sustainability by Dr. Logan Cochrane - check out "Progress," a podcast about sustainability with perspectives on &amp; from the Middle East. https://t.co/y73tWH6Emk

[Marco Mascorro @Mascobot]

A new European AI regulation proposal would make any "American opensource developer" that hosts an "unlicensed LLMs" on GitHub &amp; available in Europe liable for "€20,000,000 or 4% of worldwide revenue"

https://t.co/qxA7TyJLsO

[Aakash Gupta @aakashgupta]

The first half of the week in AI has been INSANE 🤯

Wendy's Drive-Thru AI
AI to detect Parkinson's
Meta: Multi-Sensory Vision
Palantir Stock Moons on AI
OpenAI: New Text to Shapes
Microsoft &amp; AMD Team on AI
Google I/O: TONS of New AI features

Here's what you need to know 🧵

[Nejm Benessaiah @Nejben77]

Open source is outpacing Google and OpenAI in solving major open problems related to LLMs with faster, customizable, and more private models. #opensource #LLMs #Google #OpenAI https://t.co/fhNqVTp70B

[Dr. Émile P. Torres (they/them) @xriskology]

On this week’s episode of “WTF is TESCREAL?,” I’d like to tell you a story with a rather poetic narrative arc. It begins and ends with Ted Kaczynski—yes, THAT GUY, the Unabomber!—but its main characters are Eliezer Yudkowsky and the “AI safety” community.

A long but fun🧵.

[@emilymbender.bsky.social @emilymbender]

How is this even legal?

[Travis Fischer @transitive_bs]

ChatGPT now has access to every episode of the Lex Fridman Podcast 🔥

Every amazing interview, every nugget of wisdom, every guest 🤯

Really excited to share this ChatGPT Plugin that I've been working on with @rileytomasek and https://t.co/UpsJxGBfEW

In-depth examples below 👇

[@emilymbender.bsky.social @emilymbender]

Okay, so that AI letter signed by lots of AI researchers calling for a "Pause [on] Giant AI Experiments"? It's just dripping with #Aihype. Here's a quick rundown.

&gt;&gt;

[Jonny Miller @jonnym1ller]

I challenged chatGPT to invent a brand new language... and holy sh*t 🤯

[Nejm Benessaiah @Nejben77]

Stephen Wolfram claims that chatGTP + Wolfram Alpha plug-in can create new knowledge. This is huge, meaning that chatGPT no longer recycles information from the web https://t.co/gXJl9EZIXc via @YouTube

[Jason Hickel @jasonhickel]

This image from today's IPCC synthesis report is brutal. The existing policy trajectory represents a profound failure of our governments, and of our international political system. We need much more aggressive mitigation and much stronger international cooperation.

[Nejm Benessaiah @Nejben77]

@MushtaqBilalPhD Doesn’t work for me, I just get ‘document contains no extractable text’ for pdf or html