Ben Tang

neural foundation models @ duke

[Ben Tang @bentang0]

@industriaalist @kevinhjin @har1shc @sselinmutluu

[Ben Tang @bentang0]

In the end neural foundation models will probably tell us less about AI, but more about ourselves: e.g. consciousness, being, shared representations between people.

[Ben Tang @bentang0]

I think of "The Unbearable Slowness of Being" paper.

Do our latent thoughts really flow at a bitrate>>10bps? And maybe it is through the process of instantiating our latent thoughts->words that help us understand things deeply.

Will latent thought->LLM really be a big unlock?

[Saturnin Pugnet @satpugnet]

The human brain is approaching its ChatGPT moment.

I spent 6 months writing Phase Lock: a manifesto on how brain computer interfaces can help align AI and preserve human agency.

If you only ever read one thing from me, read this.

[Ben Tang @bentang0]

Overall I am still super super bullish on neural foundation models but am not sure if noninvasive will get us there!

[Ben Tang @bentang0]

Or consciousness is an emergent property of the 86 billion neurons-the question is how? Study that with BCIs!

[🐾Eneasz Brodski @EneaszWrites]

the "Hard Problem" in a nutshell:

1. atoms aren't conscious
2. humans are made of atoms
3. humans are conscious
4. so at least one of 1-3 must be false

Panpsychists say #1 is false
Dualists say #2 is false
Illusionists say #3 is false
Materialists say #4 is false

[Cristian Garcia @cgarciae88]

Conscience is an emergent property, prove that it can only arise from cell-based neuron networks or go home.

LLMs most likely aren't conscious but the people arguing against it are doing a terrible job.

[Digi @digiii]

forget you’re in the race

[Scott @scottstts]

My god this is such a good speech that every SWE needs to hear. You know what? Every person should hear it

Keep the happy memories, eyes on the reality, be excited about the future. That’s the best that anyone can do

[Ben Tang @bentang0]

@samalanascience Wow good take!

[Ben Tang @bentang0]

Check out the project page: https://t.co/wCC004ODow
Paper: https://t.co/MJxkUYR556
Huge thanks to @zspald11 and @Cogan_G for their guidance and mentorship throughout this project!

(5/5)

[Ben Tang @bentang0]

A new state-of-the-art neural foundation model.

Excited to announce Pretraining for Sample-Efficient Neural Interfaces, a simple pretraining method for intracranial brain-computer interfaces!

(1/n)

[Ben Tang @bentang0]

The spatial encodings enable the model to generalize across diverse tasks and even to unseen subjects!

(4/n)

[Z Fellows @zfellows]

“We have a meaningful life when we do something that if we didn't do it, it wouldn't get done. That’s extremely meaningful and you should always aim for that”

— Peter Thiel

[Dmitry Rybin @DmitryRybin1]

There once lived a mathematician named Petrov.

Petrov was working on a problem.

The problem was very difficult, and so Petrov worked on it for twenty-three years.

At first Petrov was young and worked on the problem quickly. Then he grew older and worked on it more slowly. Toward the end, he hardly worked on it at all. He mostly just sat in front of it and looked.

The problem, meanwhile, wasn’t going anywhere either.

Every morning Petrov got up at half past seven, drank some tea, and sat down to work on the problem.

At twelve o’clock, Petrov would get up from the table and say:

“No.”

Then he would have lunch.

After lunch Petrov would sit down with the problem again, and at six o’clock he would say:

“No again.”

And so the day would pass.

One day Petrov’s wife asked him:

“Petya, what will happen when you solve the problem?”

Petrov was frightened.

He had never thought about it.

“When I solve it,” Petrov said, “then it will become clear.”

And he sat down to work on it again.

Five years later, Petrov’s wife asked him again:

“Well?”

“What?”

“Has it become clear?”

“Not yet.”

That was the end of the conversation.

Petrov’s colleagues treated the problem with great respect.

One professor would say:

“A very important problem.”

Another professor would say:

“An exceptionally important problem.”

A third professor said nothing, because he himself did not understand the statement of the problem.

But whenever anyone asked him about it, he would frown and say:

“There’s something considerably subtler going on there.”

For this, the third professor was held in especially high regard.

Petrov wrote seven papers about the problem.

The first was called “Some Remarks on the Question.”

The second was called “Some Further Remarks on the Question.”

The third was called “On the Impossibility of Certain Remarks on the Question.”

Petrov wrote the fourth in English, which made it seem especially serious.

Nobody read the fifth.

The sixth was read by one man in Canada, but it later turned out that he had confused Petrov with another Petrov.

Petrov decided not to publish the seventh.

It was the best one.

Several more years passed.

Petrov developed gray hair.

Then he developed more gray hair.

Then he developed less hair.

The problem did not change in the slightest.

One day, a computer appeared at the institute.

The computer was installed in a separate room.

At first Petrov paid no attention to it.

“A machine,” Petrov would say.

Then the computer began writing papers.

Petrov became interested.

Then the computer began proving theorems.

Petrov became concerned.

“What sort of theorems?” he asked.

“Various ones,” he was told.

Petrov did not like this.

“My problem is beyond it anyway,” Petrov said.

And, just in case, he closed the door to his office.

A week later, a young researcher named Sidorov knocked on Petrov’s door.

“Come in,” said Petrov.

Sidorov came in.

“Pyotr Nikolaevich,” Sidorov said, “the computer has solved your problem.”

Petrov looked at Sidorov.

Then at the window.

Then at Sidorov again.

“Which problem?”

“Yours.”

“I have lots of problems.”

In fact, Petrov had only one problem.

“That one,” Sidorov said.

“Impossible.”

“Possible.”

“It solved it incorrectly.”

“We checked.”

“Who did?”

“The computer.”

“The computer checked the computer?”

“Yes.”

Petrov thought about this.

“That’s not serious,” he said.

Sidorov then placed a printout in front of Petrov.

The printout was four hundred and eighty-three pages thick.

Petrov looked at the first page.

Then at the second.

Then at page four hundred and eighty-three.

On page four hundred and eighty-three there was a little square.

Petrov became very angry.

“I could have put a square there too,” he said.

“The square isn’t the point,” Sidorov said.

“Then why is it there?”

Sidorov did not know.

The next day there was a seminar.

The computer did not attend.

This reassured Petrov somewhat.

Sidorov gave the talk.

For two hours he explained the proof.

Petrov shook his head the entire time.

After the first hour, he said:

“There’s a mistake here.”

“Where?” Sidorov asked.

Petrov pointed.

It turned out to be the page number.

Twenty minutes later, Petrov said again:

“There’s a mistake here.”

This time it turned out to be a paper clip.

After that Petrov was silent for a while.

At the end of the seminar, the director of the institute stood up and said:

“Colleagues, an extraordinary event has taken place.”

Everyone applauded.

Petrov applauded too, because at first he had not understood what the director was talking about.

Then he understood and stopped.

The director continued:

“A problem that remained open for more than forty years has been solved.”

“Twenty-three years,” Petrov said.

“What?”

“I worked on it for twenty-three years.”

“Very good,” said the director. “Then you must be especially pleased.”

Petrov was not pleased in the slightest.

After the seminar, everyone went to have tea.

The computer was not given any tea.

Petrov saw a certain justice in this.

The next morning Petrov came to the institute as usual.

At half past eight he entered his office, took off his coat, sat down at his desk, and took out a notebook.

Then he remembered that the problem had been solved.

He closed the notebook.

A minute later, he opened it again.

Then he closed it again.

At ten o’clock Petrov went out into the corridor.

“Sidorov!” he shouted.

Sidorov came over.

“What am I supposed to do now?”

“What do you mean?”

“The problem is solved.”

“Take another one.”

Petrov was astonished.

“What do you mean, another one?”

“Another problem.”

Petrov looked at Sidorov as though Sidorov had suggested that he take another surname.

“That was my problem.”

“Now it’s solved.”

“That is exactly why it was mine.”

Sidorov did not understand and walked away.

Petrov returned to his office.

Twenty-three notebooks lay on his desk.

In the first notebook, it said:

“Let us try to prove the following.”

In the last notebook, it said:

“Let us try another way.”

Petrov stared at those words for a long time.

Then he turned the page.

The page was blank.

It was the first completely correct page in twenty-three years.

At noon, out of habit, Petrov said:

“No.”

But this time nobody had asked him anything.

A month later, the director summoned Petrov.

“Pyotr Nikolaevich,” the director said, “we need to talk.”

“About the problem?”

“No.”

Petrov immediately understood that the conversation was going to be bad.

“You see,” said the director, “computers solve a great many problems now.”

“I’ve noticed.”

“And they do it quickly.”

“That is their weakness.”

“Why?”

“They don’t have time to understand what they’re doing.”

The director wrote this sentence down on a piece of paper.

Petrov brightened.

But the director was only testing his pen.

“In any case,” the director said, “your position is being eliminated.”

“Why?”

“Optimization.”

“What does that mean?”

“It means your position no longer exists.”

Petrov thought about this.

“And me?”

“You still exist.”

“Without a position?”

“Yes.”

For a long time Petrov could not understand how a person could exist without a position.

Then he remembered that the problem had existed for twenty-three years without a solution, and this reassured him somewhat.

On his last day at work, Petrov packed his books into a box.

There were too many books.

So he left half of them behind.

Then he thought about it and left the other half behind too.

As a result, Petrov left the institute carrying an empty box.

At the door he met Sidorov.

“Pyotr Nikolaevich, where are you going?”

“Home.”

“What’s the box for?”

Petrov looked at the box.

“I don’t know.”

And he gave it to Sidorov.

At home his wife asked:

“Well, what happened today?”

“Nothing.”

“And the problem?”

“Solved.”

“Congratulations.”

“There’s nothing to congratulate me for.”

“Why?”

“I didn’t solve it.”

“Who did?”

“The computer.”

His wife thought for a moment and said:

“Then it’s a good thing you’re finally free.”

Petrov was frightened again.

He had never been free before.

The next day Petrov woke up at half past seven, drank some tea, and sat down at the table.

There was nothing in front of him.

He sat there until twelve.

At twelve, Petrov said:

“No.”

Then he had lunch.

After lunch he sat down at the table again.

At six o’clock he said:

“No again.”

This continued for another three days.

On the fourth day, his wife asked:

“What are you solving now?”

Petrov wanted to answer, but could not.

So he took a blank sheet of paper and wrote:

“Problem. Find a problem that a computer will not be able to solve.”

Petrov looked at what he had written and, for the first time in a long while, felt good.

At that moment, the telephone rang.

It was Sidorov.

“Pyotr Nikolaevich,” he said, “the computer has just come up with a list of seventeen such problems.”

Petrov put down the phone.

Then he walked over to the window.

The window was closed.

Petrov thought for a moment and decided not to open it.

It was the first decision the computer had not taken away from him.

Source: translation of Boris Bilich

[Ben Tang @bentang0]

@ShelbyOutHere @neuralink I've been working on self-supervised pretraining for more accurate and efficient BCIs. Will be posting about it next week!

[Reads with Ravi @readswithravi]

Computer scientist Sara Hooker on underestimating difficulty:

[Ben Tang @bentang0]

@randall_balestr This is so cool! Wow is iid masking really better?