Ali @salysnet - Twitter Profile

Pinned Tweet

3 months ago

neural networks start with a math paper. "a logical calculus of the ideas immanent in nervous activity". in 1943, warren mcculloch and walter pitts published a model of a neuron that took binary inputs, summed them, and fired 1 if the sum crossed a threshold, else fired 0. no learning, no weights adjusting, just fixed logic. it proved something important: a network of these simple units could compute and, or, and not(gates logic). that's the first proof that computation and neurons could be described in the same language. six years later, donald hebb gave the field its learning rule. his idea, "neurons that fire together wire together," said that the connection between two neurons should strengthen every time they activate together. this became the theoretical seed for basically every learning rule that came after, including gradient-based ones. hebb had no computer to test it on, but the idea outlived the era. frank rosenblatt turned this into a working machine in 1958. his perceptron added adjustable weights to mcculloch-pitts neurons. the key shift: instead of a human setting the weights, the machine updated them itself based on prediction error. that's the actual birth of supervised learning as a computational process, not just a theory. rosenblatt built physical hardware for it, the mark i perceptron, wired to a 20x20 grid of photocells so it could classify simple shapes. the press coverage at the time way overshot reality, comparing it to a walking, talking, self-aware machine. the perceptron's limitation turned out to be mathematical, not just practical. in 1969 minsky and papert proved a single-layer perceptron cannot solve xor operation, a basic non-linear problem. the fix, stacking layers, was theoretically obvious but practically unsolved: nobody had a reliable way to update weights across hidden layers. funding dried up. this is the first ai winter, and it lasted over a decade. the fix came from calculus, not new hardware. backpropagation, popularized by rumelhart, hinton, and williams in 1986 (paper - learning representations by back-propagating errors). it used the chain rule to push error gradients backward through every layer of a network, updating each weight in proportion to its contribution to the mistake. this is still the core training mechanism under basically every modern neural net, including transformers. the algorithm existed in pieces earlier (linnainmaa, werbos) but the 1986 paper is what made it standard practice. vision architecture developed in parallel. fukushima's neocognitron (1980) borrowed directly from hubel and wiesel's biological studies of the visual cortex, introducing the idea of stacking layers that detect local features and pool them into more abstract ones. lecun combined this structure with backpropagation in 1989 to build lenet, a working convolutional network that banks actually used to read handwritten digits on checks. this is the first time deep learning shipped in a real product. despite all this, neural nets stayed a secondary method through the 90s. support vector machines were cheaper to train, easier to reason about mathematically, and performed just as well on the small datasets available at the time. neural nets needed scale to prove their advantage, and scale wasn't there yet, not in data or in compute. 2012 is when scale finally arrived. krizhevsky, sutskever, and hinton entered alexnet into imagenet, training a deep cnn on gpus(two nvidia gtx 580) with relu activations and dropout for regularization. it beat the previous best by 10.8 percentage points(huge), a margin big enough that it couldn't be dismissed as noise. this is the moment deep learning shifted from a fringe method to the default approach in computer vision, and soon after, in most of ml. the next real architectural leap was recurrent nets hitting a ceiling. rnns and lstms could handle sequences, but they processed tokens one at a time, which made long-range dependencies hard to learn and training slow to parallelize. attention mechanisms were bolted onto rnns first, letting the model weigh earlier tokens directly instead of relying on a single compressed hidden state. but the problem of recurrence was there. then in 2017, vaswani et al. released "attention is all you need," removing recurrence entirely. the transformer used self-attention as the only mechanism for mixing information across a sequence, which meant full parallelization during training and much better scaling with data and compute. this single design choice is why gpt, sonnert, opus, and basically every modern llm exist. the line from mcculloch-pitts' threshold neuron to a transformer block is direct: better math for learning, better architecture for structure, and eventually enough compute to let both run at scale. this was bit about history of neural nets. how we got here? this was a brief answer.

salysnet's tweet photo. neural networks start with a math paper. "a logical calculus of the ideas immanent in nervous activity".

in 1943, warren mcculloch and walter pitts published a model of a neuron that took binary inputs, summed them, and fired 1 if the sum crossed a threshold, else fired 0. no learning, no weights adjusting, just fixed logic.

it proved something important: a network of these simple units could compute and, or, and not(gates logic). that's the first proof that computation and neurons could be described in the same language.

six years later, donald hebb gave the field its learning rule. his idea, "neurons that fire together wire together," said that the connection between two neurons should strengthen every time they activate together.

this became the theoretical seed for basically every learning rule that came after, including gradient-based ones. hebb had no computer to test it on, but the idea outlived the era.

frank rosenblatt turned this into a working machine in 1958. his perceptron added adjustable weights to mcculloch-pitts neurons.

the key shift: instead of a human setting the weights, the machine updated them itself based on prediction error.

that's the actual birth of supervised learning as a computational process, not just a theory. rosenblatt built physical hardware for it, the mark i perceptron, wired to a 20x20 grid of photocells so it could classify simple shapes. the press coverage at the time way overshot reality, comparing it to a walking, talking, self-aware machine.

the perceptron's limitation turned out to be mathematical, not just practical.

in 1969 minsky and papert proved a single-layer perceptron cannot solve xor operation, a basic non-linear problem.

the fix, stacking layers, was theoretically obvious but practically unsolved: nobody had a reliable way to update weights across hidden layers. funding dried up. this is the first ai winter, and it lasted over a decade.

the fix came from calculus, not new hardware. backpropagation, popularized by rumelhart, hinton, and williams in 1986 (paper - learning representations by back-propagating errors).

it used the chain rule to push error gradients backward through every layer of a network, updating each weight in proportion to its contribution to the mistake.

this is still the core training mechanism under basically every modern neural net, including transformers. the algorithm existed in pieces earlier (linnainmaa, werbos) but the 1986 paper is what made it standard practice.

vision architecture developed in parallel. fukushima's neocognitron (1980) borrowed directly from hubel and wiesel's biological studies of the visual cortex, introducing the idea of stacking layers that detect local features and pool them into more abstract ones.

lecun combined this structure with backpropagation in 1989 to build lenet, a working convolutional network that banks actually used to read handwritten digits on checks. this is the first time deep learning shipped in a real product.

despite all this, neural nets stayed a secondary method through the 90s.

support vector machines were cheaper to train, easier to reason about mathematically, and performed just as well on the small datasets available at the time.

neural nets needed scale to prove their advantage, and scale wasn't there yet, not in data or in compute.

2012 is when scale finally arrived. krizhevsky, sutskever, and hinton entered alexnet into imagenet, training a deep cnn on gpus(two nvidia gtx 580) with relu activations and dropout for regularization.

it beat the previous best by 10.8 percentage points(huge), a margin big enough that it couldn't be dismissed as noise.

this is the moment deep learning shifted from a fringe method to the default approach in computer vision, and soon after, in most of ml.

the next real architectural leap was recurrent nets hitting a ceiling.

rnns and lstms could handle sequences, but they processed tokens one at a time, which made long-range dependencies hard to learn and training slow to parallelize.

attention mechanisms were bolted onto rnns first, letting the model weigh earlier tokens directly instead of relying on a single compressed hidden state. but the problem of recurrence was there.

then in 2017, vaswani et al. released "attention is all you need," removing recurrence entirely.

the transformer used self-attention as the only mechanism for mixing information across a sequence, which meant full parallelization during training and much better scaling with data and compute.

this single design choice is why gpt, sonnert, opus, and basically every modern llm exist.

the line from mcculloch-pitts' threshold neuron to a transformer block is direct: better math for learning, better architecture for structure, and eventually enough compute to let both run at scale.

this was bit about history of neural nets. how we got here? this was a brief answer.

5

174

27

139

8K

Ali

@salysnet

Last Seen Users on Sotwe

Trends for you

Most Popular Users