Michael

Robotics Software ML Engineer 🇨🇦

[Jim Fan @DrJimFan]

Let's reverse engineer the phenomenal Tesla Optimus. No insider info, just my own analysis. Long read:

1. The smooth hand movements are almost certainly trained by imitation learning ("behavior cloning") from human operators. The alternative is reinforcement learning in simulation, but that typically leads to jittery motion and unnatural hand poses.

There're at least 4 ways to collect human demonstrations:
(1) A custom-built teleoperation system - I believe this is the most likely means used by Tesla team. Open-source example: ALOHA, a low-cost bimanual robot arm and teleoperation system by Stanford AI Labs (https://t.co/8iXpiHVEjS). It enables very precise, dexterous motions, such as putting AAA batteries into a remote or manipulating contact lens.

(2) Motion Capture (MoCap): apply the MoCap systems used for Hollywood movies to capture the fine-grained motions of hand joints. Optimus' 5-finger hand is a great design decision that enables a direct mapping - there is no "embodiment gap" from human operators.
For instance, a demonstrator can wear a CyberGlove (https://t.co/S8hxErsEuU) and grasp the cubes on the table (as shown in video). CyberGlove will capture the motion signals & haptic feedback in real-time, which can be re-targeted onto Optimus.

(3) Wearing gloves & markers can be clumsy. An alternative way to do MoCap is through computer vision. DexPilot from NVIDIA enables marker-less and glove-free data collection. The human operator simply uses their bare hands to perform the tasks. 4 Intel RealSense depth cameras and 2 NVIDIA Titan XP GPUs (yeah, 2019 work) translate the pixels to precise motion signals for robot learning.

(4) VR Headset: turn the training room into a VR game, and let humans "role play" Optimus. Use the native VR controller or CyberGlove to control the virtual Optimus hands. This has the advantage of scalable remote data collection - annotators from around the world can contribute without coming onsite.
VR demonstration technique appeared in research projects like the iGibson home robot simulator, an initiative that I participated in at Stanford: https://t.co/eyI4ORkH6G

Above 4 are not mutually exclusive. Optimus could use a combo of them for different pros & cons.

2. Neural Architecture. Optimus is trained end-to-end: videos in, actions out. I'm quite sure it's implemented by a multimodal Transformer with the following components:

(1) Image: some variant of efficient ViT, or simply an old ResNet/EfficientNet backbone (https://t.co/L6PLTQJnGA). The block pick-and-place demo doesn't require sophisticated vision. The spatial feature map from the image backbone can be tokenized easily.

(2) Video: two ways. Either flatten the video into a sequence of images and produce tokens independently, or have a video-level tokenizer. There're numerous ways to efficiently process video pixel volumes. You don't necessarily need Transformer backbones, e.g. SlowFast Network (https://t.co/qDdXzqwJQp) and RubiksNet (https://t.co/CQU8D7TZgx, my paper at ECCV 2020, efficient CUDA shift primitives).

(3) Language: it's not clear if Optimus is language prompted. If it is, there needs to be a way to "fuse" the language representations into perception. FiLM is a very lightweight neural network module that serves this purpose (https://t.co/VI4TpgQ22V).
You can think of it intuitively as a "cross attention" of language embedding into the image-processing neural pathway.

(4) Action tokenization: Optimus needs to convert the continuous motion signals into discrete tokens for the autoregressive Transformer to work. A few ways:
- Directly bin the continuous values for each hand joint control. [0, 0.01) -> token #0, [0.01, 0.02) -> token #1, etc. This is straightforward but could be inefficient due to the long sequence length.
- The joint movements are highly dependent on each other, which means they occupy a low-dimensional "state space". Apply VQVAE to the motion data to obtain a shorter-length, compressed token set.

(5) Putting the above pieces together, we have a Transformer controller that consumes video tokens (optionally with language modulation), and outputs action tokens, one step at a time. The next frame from the table is fed back to the Transformer, so it knows the consequence of its action. That gives the self-corrective ability shown in the demo.

I believe the architecture is most similar to:
- Google RT-1: https://t.co/dpuon1bqU6
- NVIDIA VIMA: https://t.co/Tn3L63uGrv

3. Lastly, I'm genuinely impressed by the hardware quality. The motions are fluid, and the aesthetics is amazing as well. As I mentioned above, it's such a great decision to follow human morphology closely, so that there is no gap in imitating humans.

Atlas from Boston Dynamics only has simple gripper-style hands. In the long run, Optimus' bi-dexterous, 5-finger hands will prove far superior in daily tasks.

Congrats to @Tesla_Optimus team & @elonmusk 🎉! I look forward to seeing the bots roam Mars some day 🦾

[Karol Hausman @hausman_k]

🚨 New (Offline) RL Method 🦾 🚨

Introducing Q-Transformer https://t.co/YPxfVc6fF3 - new RL approach that works at scale with large models and many tasks.

This is the best method we found so far that works with demos and autonomous (also negative) data at large scale. 🧵

[Bindu Reddy @bindureddy]

Why Large Language Models Hallucinate and How to Reduce it

If you are a power user of ChatGPT you have probably been bitten by the hallucination bug. The LLM lulls you into getting comfortable with it and then springs a convincing but totally made-up story, playing you for a fool.

These hallucinations, like dreams, are LLMs fabricating narratives. So why do these LLMs hallucinate and how do you prevent it.

Here are a few reasons

Data Sparsity: This is the #1 reason for hallucination. GPT-4 for example doesn't have access to recent data as it was trained in 2021. Ask it a question, that pertains to a recent topic, and it is likely to hallucinate as it doesn't have the data for the right answer. The model is generalizing from what it has learnt but that may very well be inaccurate

Not supervised learning: LLMs don't have a "ground truth" or a set of correct examples. While the RLHF process tries to steer the LLM towards more correct answers. The base training isn't a supervised learning process and this makes things challenging as the model can't tell what is "correct" and what's not

Short-term context: The model architecture has a fixed-length context window, meaning it can only "see" a certain number of tokens at a time. If important context falls outside this window, the model may lose track of it, leading to errors.

No real-time feedback loop: Like humans LLMs don't have a real-time feedback look and don't instantly learn from mistakes. The good news is we can refine or fine-tune models with human feedback and make them hallucinate less.

So how do you prevent these hallucinations and are future LLMs less likely to hallucinate?

While there is no easy way to guarantee the LLMs will never hallucinate. The following techniques mitigate it to some extent

Prompt Design: Simple prompt engineering and design will reduce hallucination. For example, adding the following to your prompt help: "Provide a factual answer based on scientific evidence."

Fine-tune for a specific domain: The model can be fine-tuned on a narrower dataset that is highly reliable and relevant to the domain where hallucinations need to be minimized.

Contradiction checks: LLMs can be prompted to self-contradict themselves and then they are further prompted to recognize the contradiction and mitigate it. This falls into the category of advanced prompt engineering

Retrieval Augmented Generation: This is a common technique used in Enterprise LLMs. At Abacus, we use this routinely. You are basically looking up the relevant documents that contain the answer in a search index first and then feeding the search results to an LLM to formulate the final answer. Since the LLM is forced to find the answer in the information it was sent, it hallucinates much less.

Human In the Loop: A human expert can always check the answer before it gets used. This is a labor intensive option which isn't ideal

While the above techniques work on trained LLMs, the following two techniques can be applied during LLM training

Data Re-weighting: Assign higher weights to reliable and verified data during LLM training, effectively making the model pay more attention to them

Longer Context Windows: Extending the model's memory can help it maintain context over longer passages, reducing the chance of hallucinations.

So while there are several easy ways to mitigate and almost completely remove hallucinations if you are working in the Enterprise context, it's much harder in the AGI context. This is a very hot topic in AI research and several researchers are still working on it.

[Milan Jovanović @mjovanovictech]

Do you want to learn Clean Architecture and DDD?

I prepared 45 lessons for you 🔥

𝗖𝗹𝗲𝗮𝗻 𝗔𝗿𝗰𝗵𝗶𝘁𝗲𝗰𝘁𝘂𝗿𝗲 𝗙𝘂𝗻𝗱𝗮𝗺𝗲𝗻𝘁𝗮𝗹𝘀

- Project setup - https://t.co/x3BbJGUe9Q
- Minimal APIs - https://t.co/LGOXKtMFc6
- Dependency injection - https://t.co/xw1OrQJ6Dd
- CA + Document database - https://t.co/BvCZSMA42a
- Project setup from scratch - https://t.co/gjWTPz9BGL
- 4 Best practices for new project - https://t.co/sUROYi4uHG
- Structured logging - https://t.co/MOEhRDBEgn
- Message queues - https://t.co/Q15TljHawy

𝗗𝗼𝗺𝗮𝗶𝗻-𝗗𝗿𝗶𝘃𝗲𝗻 𝗗𝗲𝘀𝗶𝗴𝗻 𝗜𝗻𝘁𝗿𝗼𝗱𝘂𝗰𝘁𝗶𝗼𝗻

- Rich Domain model - https://t.co/p6WoHERBtd
- Entities - https://t.co/8jSXknZEJa
- Value objects - https://t.co/9qpqUMzF2F
- Aggregate root - https://t.co/d01rGxChrK
- Domain validation - https://t.co/emL6aCLI5r
- Domain model tradeoffs - https://t.co/BKSg0mwDSy
- Repository pattern - https://t.co/KwkP8ZN2k3
- Specification pattern - https://t.co/UJ6yRXDvM0
- Unit of work - https://t.co/Hf6IVDV8jC
- Smart Enums - https://t.co/QxUHyHUYil
- Snapshot pattern - https://t.co/LPYlk9xtqi
- Strongly typed IDs - https://t.co/QMd1kSuPnY
- Anemic Domain model - https://t.co/NkIlenXbFv
- DDD modeling - https://t.co/A6dia5vbEX
- DDD + EF mapping - https://t.co/z0rO0ogu35
- Incomplete aggregates - https://t.co/br2GX9mdhp
- Double dispatch - https://t.co/l8ZNuP5l7m

𝗖𝗤𝗥𝗦

- CQRS Fundamentals - https://t.co/ovdROa8VjL
- Validation /w Result - https://t.co/n9sZxs5iFf
- Validation /w Exception - https://t.co/uFYYKVcCmZ
- Read models - https://t.co/VUJzK0zWIL
- UoW pipeline - https://t.co/0YTrU0MSjJ
- The "Truth" on CQRS - https://t.co/GF3E2tPbhp
- CQRS Query side - https://t.co/ujH0QNSV5S
- Materialized views - https://t.co/2yBTibGi9f

𝗧𝗲𝘀𝘁𝗶𝗻𝗴
- Parameterized tests - https://t.co/IVJaOpnJUj
- Unit testing - https://t.co/qjMJIH0LMz
- Integration testing /w Docker - https://t.co/r6OJimak6D
- Architecture tests - https://t.co/sSbnicffOL

𝗗𝗲𝘀𝗶𝗴𝗻 𝗣𝗮𝘁𝘁𝗲𝗿𝗻𝘀
- Idempotent consumer - https://t.co/QI2dl3lk6P
- Saga pattern - https://t.co/XbVVBU8KG1
- Compensating transaction (Saga) - https://t.co/c8pnAb7UHn
- Domain events - https://t.co/tVaCBiuTAE
- Domain vs. Integration event - https://t.co/jHR6wBlkBZ
- Options pattern - https://t.co/ovI4WL7ttr
- Options pattern validation - https://t.co/IrmG1J18AS
- Decorator pattern - https://t.co/BeQP4Y1nVz

Hope these are helpful.

Have an awesome Sunday!

#dotnet

[Jerry Liu @jerryjliu0]

We’ve partnered with @wey_gu to create the world’s most comprehensive short course on using LLMs with Knowledge Graphs 🧑‍🏫 🌐

There’s a crazy amount of content in here (KG concepts -> code blocks -> demos).

It's all contained in a single Colab notebook: https://t.co/xGwxKvhEHz

[Bindu Reddy @bindureddy]

Linear algebra, deep learning, and GPU architectures are fundamentally interconnected

Linear algebra deals with matrices & tensors and is the branch of math that is applied in deep learning (DL). GPUs are used for training DL models.

Tensors and GPUs are a marriage made in heaven!

GPUs were first designed for graphics processing but they are remarkably well-suited to perform linear algebra operations

Tensors are multidimensional arrays that can represent anything from simple scalars to complex n-dimensional matrices. They're a unifying framework to represent data, weights, biases, and essentially all the numerical aspects of a neural network.

0th-Order Tensor: A scalar, a single real or complex number.
1st-Order Tensor: A vector, an ordered set of numbers. It's an n-tuple.
2nd-Order Tensor: A matrix, a grid of numbers arranged in rows and columns.
Higher-Order Tensor: Tensors of three or more dimensions, generalizing vectors, and matrices.

You can represent text, images, and videos as tensors and operate on these tensors to train the weights and biases of a neural network

Tensors support operations like addition, matrix multiplication, transposing, contraction, reshaping, and slicing

Each of the layers of the NN can be formulated as tensor operations and the fundamental algorithms like backpropagation and gradient descent apply these tensor operations

The cool thing about these tensors is that you enable parallelization of computations, and the cool thing about GPUs is that they are incredibly good at parrallelism. Unlike CPUs, which might have a few powerful cores optimized for sequential processing, GPUs have thousands of smaller, less powerful cores designed to execute operations simultaneously.

Tensors and GPUs work really well together

Parallelism: The operations on tensors, especially those of higher orders, can be broken down into many smaller, independent tasks. This suits the architecture of a GPU, where thousands of these tasks can be run in parallel.

Memory Hierarchy: GPUs have a hierarchical memory structure, including global, shared, and local memory. Effective use of this hierarchy while working with tensors can lead to substantial efficiency gains.

Optimized Libraries: Libraries like cuBLAS, cuDNN, and others offer GPU-accelerated implementations specifically designed for tensor operations. Frameworks like TensorFlow and PyTorch are built on top of these, abstracting away the complexity.

Batch Processing: Deep learning models often process data in batches, which means performing the same operations on a whole set of data points simultaneously. This again plays into the parallelism that GPUs provide. You can think of it as applying the same transformation to every element in a grid simultaneously.

Over time there have been a lot of software libraries and frameworks built to make building AI really simple and fast. You can iterate and experiment with your favorite NN using a couple of lines of code. In the background, the GPUs will crunch the tensor operations and extract patterns in your data magically.

Further reading and pic credit -
https://t.co/2Fr617YBDp
Why GPUs for ML - https://t.co/fvORANKbUL

[Teiva Harsanyi @teivah]

Challenging algorithms and data structures every programmer should try https://t.co/3JRAfmoNuE

[Brian Feroldi @BrianFeroldi]

10 handcrafted visuals by @safalniveshak every investor should see:

1. Stock Selection Framework

[Kirk Borne @KirkDBorne]

.@NASA scientists, engineers, and developers create software packages to manage space missions, test spacecraft, and analyze the petabytes of data produced by agency research satellites. Many of these programs are downloadable free of charge via #NASA’s Software Catalog: https://t.co/MWlE4yQ1AM

[Abacus.AI @abacusai]

A cheat sheet explanation of how Large Language Models work:

[Brett Adcock @adcock_brett]

The electric aircraft company I founded in 2018 raised $215M yesterday.

Investors include Boeing, Stellantis, United Airlines.

Posting the investor slide deck:

[elvis @omarsar0]

Memorization vs. Generalization is one of my favorite ML research topics.

One phenomenon of interest, referred to as grokking, is where models flip from memorizing to sudden generalization.

A big question LLM researchers are interested in answering is whether LLMs, when trained for longer, are generalizing or memorizing.

While evidence for both sides has been presented, it's still largely inconclusive. There are a lot of open questions regarding memorization and generalization.

A better understanding of these behaviors has a lot of implications when trying to further improve these systems from a practical perspective. Hence the emerging interest in the topic.

A new article by Google presents an interesting analysis of the topic. They look at the training dynamics of a tiny model and reverse engineer the solution it finds. They also provide an illustration of the emerging field of mechanistic interpretability.

Here is an interesting observation from the article:

"The periodic patterns suggest the model is learning some sort of mathematical structure; the fact that it happens when the model starts to solve the test examples hints that it’s related to the model generalizing"

There are many observations like these in different works but the interesting bit here is to understand why the model moves away from memorizing and what exactly is the generalizing solution.

They focus on understanding the mechanism of a one-layer MLP. The study dives deeper into scenarios and attempts to answer questions like "When does grokking happen?" and "Why memorization is easier than generalization?" and more.

One interesting goal of this research is to find and understand patterns that help automate uncovering algorithms learned by neural networks.

The analysis focuses on tiny models but the aim is clearly to work on the building blocks for future research on even larger models and a better understanding of grokking. The study contains interesting observations and even interactive charts. There are some good references as well. A highly recommended read.

(link in the replies)

[Cameron R. Wolfe, Ph.D. @cwolferesearch]

The KV-cache for self-attention is (arguably) the most effective/simple technique for speeding up LLM inference. I'm going to post a more extensive explanation tomorrow, but this writeup I found is super clear and even provides an implementation!

https://t.co/3G5MijbhFg

[Matt Shumer @mattshumer_]

Built a notebook that makes it dumb simple to fine-tune LLaMA 2.

Just load in a dataset, and run it!

https://t.co/KgMM5Lmgv7

[Cameron R. Wolfe, Ph.D. @cwolferesearch]

I just finished writing a survey on the history of open-source LLM research, spanning from the early days (e.g., OPT and BLOOM) to recent models like LLaMA-2. Here are three takeaways that seem to have the biggest impact on LLM quality…

Base models make all the difference. Pre-training is the first, most expensive, and most important step of training an LLM. If your base model is bad, then you can’t fix it with extensive fine-tuning. We learn from open-source LLM research (LIMA) that most of the LLM's knowledge base is gained during pre-training, while fine-tuning/alignment serves to learn the correct style/format.

Going further, major surges in open-source LLM research seem to be catalyzed by the release of powerful base models (e.g., LLaMA led to a wave of research on imitation learning). Plus, we see in a variety of papers that improving the underlying base model yields improvements after fine-tuning and alignment. Creating a good base model is difficult, but it makes all the difference!

Data is important. When we compare early attempts at open-source LLMs (e.g., OPT or BLOOM) to recent models like LLaMA-2, the largest difference between them is the data! LLaMA-2 is pre-trained over 2 trillion tokens of high-quality data, while OPT/BLOOM observe ~500M tokens of data during pre-training combined!

Additionally, we learn from LIMA and LLaMA-2 that, during both pre-training and alignment, the quality of data matters. Increasing the amount of data used for SFT has minimal impact, but ensuring the diversity and quality of this data (even if there are very few examples) massively benefits performance. Data quality and quantity matters across all steps of LLM training.

Alignment goes a long way. We see with LLaMA-2 that one of the largest remaining gaps between open-source and proprietary LLMs is the quality of alignment. Most open-source LLMs are fine-tuning using solely SFT over public sources of data that oftentimes lack in quality and diversity. On the other hand, closed-source models are aligned with SFT and RLHF over large proprietary datasets.

“Llama 2-Chat is the result of several months of research and iterative applications of alignment techniques, including both instruction tuning and RLHF, requiring significant computational and annotation resources.” - from LLaMA-2 paper

To close this gap, LLaMA-2-Chat models undergo an extensive alignment process, including multiple phases of SFT and RLHF. In total, 100K dialogue examples are used for SFT, as well as 1M instances of human feedback for RLHF. The resulting model is, by far, the highest-performing open-source LLM to yet be created, thus revealing the pivotal importance of fine-tuning and alignment.

[Michael @crazy_sanguine]

https://t.co/brjrPoMoP6

[Caglar @caglar_ee]

Video Lectures, CMU 16 299 Introduction to Feedback Control Systems spring 2022, by Chris Atkeson
https://t.co/Eh78dhhpFa

[Michael @crazy_sanguine]

https://t.co/oH0qDonnqO