LongliveLong

今天 Elon 宣布了 @xai - 目标 "理解宇宙"，估计这是 Tesla 的 "现实世界 AI" 的升级版。 xAI 将独立于 X Corp，与 Twitter & Tesla 协作更加紧密！从 https://t.co/Yx53ZiuRRu 可以了解到初创成员基本来自 DeepMind、OpenAI、Google & Microsoft Research、Tesla 以及 Toronto University，显然多大已经成了这次 AI 革命的摇篮。。网站也提到了会进一步引入创新技术，例如： Transformer-XL 能支持超长上下文 Context 的 Transformer https://t.co/htsO83SgKy Autoformalization 通过大语言模型，将语言、概念或理论转化为逻辑表达式、数学模型或者计算机算法 https://t.co/yuMP2sGBkO Memorizing Transformer 让语言模型能够在推理时简单地阅读和记忆新数据，从而立即获得新知识，无需修改训练权重 https://t.co/Bib3Fjg7By Batch Size Scaling 加速神经网络在并行设备上的训练速度 https://t.co/S1Mgl03PkN μTransfer 可以局部调整大语言模型参数的方法 https://t.co/QRnZyhf8f1 看完感觉 xAI 的真实目标是想接棒人类来发现"万物理论了"。。大家可以参加7月14号 twitter space 上的 AMA，问 Elon 关于 xAI 的任何问题🤔️

indigox's tweet photo. 今天 Elon 宣布了 @xai - 目标 "理解宇宙"，估计这是 Tesla 的 "现实世界 AI" 的升级版。

xAI 将独立于 X Corp，与 Twitter & Tesla 协作更加紧密！从 https://t.co/Yx53ZiuRRu 可以了解到初创成员基本来自 DeepMind、OpenAI、Google & Microsoft Research、Tesla 以及 Toronto University，显然多大已经成了这次 AI 革命的摇篮。。网站也提到了会进一步引入创新技术，例如：

Transformer-XL
能支持超长上下文 Context 的 Transformer
https://t.co/htsO83SgKy

Autoformalization
通过大语言模型，将语言、概念或理论转化为逻辑表达式、数学模型或者计算机算法
https://t.co/yuMP2sGBkO

Memorizing Transformer
让语言模型能够在推理时简单地阅读和记忆新数据，从而立即获得新知识，无需修改训练权重
https://t.co/Bib3Fjg7By

Batch Size Scaling
加速神经网络在并行设备上的训练速度
https://t.co/S1Mgl03PkN

μTransfer
可以局部调整大语言模型参数的方法
https://t.co/QRnZyhf8f1

看完感觉 xAI 的真实目标是想接棒人类来发现"万物理论了"。。大家可以参加7月14号 twitter space 上的 AMA，问 Elon 关于 xAI 的任何问题🤔️

11K

LongliveLong @longnameyfs

about 3 years ago

@indigox 治安问题越来越严重是个问题，其他还是挺舒服的

LongliveLong @longnameyfs

about 3 years ago

@catmangox 朋友端午去湖州旅游，759一晚的酒店入住排队1小时。

826

LongliveLong @longnameyfs

about 3 years ago

@jessyshen 加拿大一个20万人的镇也是样样齐全，基础设施跟城市看齐。

longnameyfs retweeted

Cameron R. Wolfe, Ph.D.

@cwolferesearch

about 3 years ago

In the wake of LLaMA, the deep learning research community quickly adopted the view that open-source LLMs will rule the future—reproducing open-source variants of proprietary models seemed to be easy and cheap. Is this the truth? Here’s a brief timeline of model proposals and what we know… LLaMA. The surge of open-source LLM publications started with the proposal of LLaMA, a suite of LLMs with sizes ranging from 7 billion to 65 billion parameters. Taking inspiration from Chinchilla, these LLMs are a bit smaller than their counterparts but are pre-trained extensively, allowing them to perform surprisingly well for their size (e.g., 13B model is comparable to GPT-3). Unlike closed-source models that are trained on a combination of public and proprietary data, LLaMA uses only publicly available data for pre-training, making it fully open-source and possible to replicate. LLaMA leak. After its proposal and release for research purposes, LLaMA weights were leaked to the public on 4Chan. This leak led many in the research community to begin doing research with the model, leading to a ton of different publications and proposals that we briefly overview below. A majority of these techniques fine-tune LLaMA on dialogue from ChatGPT (i.e., imitation models) and claim to achieve comparable performance, creating the view that creating open-source replicas of models like ChatGPT is actually quite easy. Alpaca is a fine-tuned version of the LLaMA-7B LLM. The fine-tuning process is based on self-instruct, in which instruction-following data is collected from a higher-performing LLM (i.e., text-davinci-003) and used for supervised fine-tuning. The entire fine-tuning process of Alpaca costs only $600 (including both data collection and fine-tuning). Vicuna is an open-source chatbot that is created by fine-tuning LLaMA-13B (i.e., comparable performance to GPT-3). Vicuna is fine-tuned using examples of user conversations with ChatGPT, and the entire fine-tuning process can be replicated for <$300, making chatbots more accessible for research purposes. Compared to Alpaca, Vicuna is more comparable to ChatGPT and generates answers with more detail and structure. Koala is a version of LLaMA-13B that has been fine-tuned on dialogue data from a variety of sources, ranging from public datasets to dialogues with other high-quality LLMs that are available on the internet. Compared to Alpaca, Koala is fine-tuned over dialogue data and evaluated more extensively (using a larger number of crown workers). GPT4ALL is a fine-tuned LLaMA-7B model that has been trained on over 800K chat completions from GPT-3.5-turbo. Along with releasing the code and model, authors of GPT4ALL release the 4-bit quantized weights of the model, which can be used to run inference on CPUs. The result is that we can actually host this model locally on a normal laptop. Massive Potential. The models mentioned above were all published in close succession and (in most cases) claimed to achieve results that were comparable to top models like ChatGPT or GPT-4. As a result, the LLM community quickly adopted the opinion that open-source LLMs will soon match the quality of the most powerful proprietary models. The Truth. When more targeted evaluations of these models is performed, we see that their performance is not as impressive as it seems. In reality, these models are good at matching the style and structure of models like ChatGPT when fine-tuned over a small amount of imitation data, but they lack the same knowledge base. Imitation models hallucinate more and perform poorly when subjected to a broader number of natural language benchmarks. Despite their limitations, these models were able to “trick” human evaluators via their style, despite being significantly less factual.

cwolferesearch's tweet photo. In the wake of LLaMA, the deep learning research community quickly adopted the view that open-source LLMs will rule the future—reproducing open-source variants of proprietary models seemed to be easy and cheap. Is this the truth? Here’s a brief timeline of model proposals and what we know…

LLaMA. The surge of open-source LLM publications started with the proposal of LLaMA, a suite of LLMs with sizes ranging from 7 billion to 65 billion parameters. Taking inspiration from Chinchilla, these LLMs are a bit smaller than their counterparts but are pre-trained extensively, allowing them to perform surprisingly well for their size (e.g., 13B model is comparable to GPT-3). Unlike closed-source models that are trained on a combination of public and proprietary data, LLaMA uses only publicly available data for pre-training, making it fully open-source and possible to replicate.

LLaMA leak. After its proposal and release for research purposes, LLaMA weights were leaked to the public on 4Chan. This leak led many in the research community to begin doing research with the model, leading to a ton of different publications and proposals that we briefly overview below. A majority of these techniques fine-tune LLaMA on dialogue from ChatGPT (i.e., imitation models) and claim to achieve comparable performance, creating the view that creating open-source replicas of models like ChatGPT is actually quite easy.

Alpaca is a fine-tuned version of the LLaMA-7B LLM. The fine-tuning process is based on self-instruct, in which instruction-following data is collected from a higher-performing LLM (i.e., text-davinci-003) and used for supervised fine-tuning. The entire fine-tuning process of Alpaca costs only $600 (including both data collection and fine-tuning).

Vicuna is an open-source chatbot that is created by fine-tuning LLaMA-13B (i.e., comparable performance to GPT-3). Vicuna is fine-tuned using examples of user conversations with ChatGPT, and the entire fine-tuning process can be replicated for <$300, making chatbots more accessible for research purposes. Compared to Alpaca, Vicuna is more comparable to ChatGPT and generates answers with more detail and structure.

Koala is a version of LLaMA-13B that has been fine-tuned on dialogue data from a variety of sources, ranging from public datasets to dialogues with other high-quality LLMs that are available on the internet. Compared to Alpaca, Koala is fine-tuned over dialogue data and evaluated more extensively (using a larger number of crown workers).

GPT4ALL is a fine-tuned LLaMA-7B model that has been trained on over 800K chat completions from GPT-3.5-turbo. Along with releasing the code and model, authors of GPT4ALL release the 4-bit quantized weights of the model, which can be used to run inference on CPUs. The result is that we can actually host this model locally on a normal laptop.

Massive Potential. The models mentioned above were all published in close succession and (in most cases) claimed to achieve results that were comparable to top models like ChatGPT or GPT-4. As a result, the LLM community quickly adopted the opinion that open-source LLMs will soon match the quality of the most powerful proprietary models.

The Truth. When more targeted evaluations of these models is performed, we see that their performance is not as impressive as it seems. In reality, these models are good at matching the style and structure of models like ChatGPT when fine-tuned over a small amount of imitation data, but they lack the same knowledge base. Imitation models hallucinate more and perform poorly when subjected to a broader number of natural language benchmarks. Despite their limitations, these models were able to “trick” human evaluators via their style, despite being significantly less factual.

505

471

367K

LongliveLong

@longnameyfs

Who to follow

Last Seen Users on Sotwe

Trends for you

Most Popular Users