I wrote about why more of that process knowledge should become public, and why Marin is organized around open development of frontier AI.
https://t.co/Qcu5R7kger
@OwainEvans_UK And in replicating the pathway from embedding to output, the student subtly replicates behavior which is correlated, such as a love for owls.
@OwainEvans_UK If both the teacher and student share the same base model, could it be a about the token embedding matrix being similar for both?
The circuit path from a token embedding to an output logit is being replicated by the student based on the data from the teacher.
But arguably, by changing where we choose to draw the encapsulation boundaries in our software, we can manage software complexity while still having the expressiveness to model complicated real-word behavior.
For example, in several Java codebases which adopt the "compile-time hierarchy of encapsulation matching the domain model", you will find 5 lines of core implementation hidden between a dozen different classes and interfaces.
My failed EdTech startup cost me $20k and 11 months. Here are 5 hard lessons I learned:
1. Validation isn’t enough
“Validate before you code,” they say. I did. I had a waitlist, even some verbal commitments to pay. But unless money actually hits your account month after month, it’s not validation. Worse, each customer wanted something different. As a solo dev, I couldn’t meet all the expectations. A waitlist means nothing unless people are truly paying and sticking.
2. Your initial network is everything
In the early days, speed of feedback is gold. If you’re building a dev tool and you know devs, feedback is quick. I was building for teachers, but I wasn’t in that world — no school, no college, no direct access. Build for the people you can reach. Bonus points if they’re active online.
3. B2B is brutal for a side hustle
I tried reaching out to universities. Between timezone gaps, job commitments, and the effort required for enterprise sales, it wasn’t feasible. B2B is a full-time game. If you can’t dedicate yourself to sales calls, follow-ups, and meetings — don’t go there part-time.
4. Some industries are just hard
Healthcare, education, energy, governance — these aren’t indie hacker-friendly. Long sales cycles, regulatory mazes, slow-moving institutions. People can easily find out hustles and lose interest. If you're not full-time or VC-backed, think twice before jumping in.
5. Don’t build for two users
I built for both teachers and students. Like marketplaces with buyers and sellers, these are hard to balance. You can't optimize for both equally. And adoption dies if one side finds it lacking. If you're a solo developer or a bootstrapped team focus on single-user products. It’s simpler, faster, and much easier to get right.
Most people don't actually know the lengths parents will go to try to raise an academic superstar. In this post, I will detail the life of the average thoroughbred in STEM PhD programs at a top university. The thoroughbred lives a difficult life full of enormous amounts of pressure. The thoroughbred's parents have oriented the next 18 years of their family life to evolve around the academic success of their children. The thoroughbred's parents don't simply move houses within their country so their kids can go to the best school in the district; they do a nationwide search to decide where to raise their children based on the schools in that area.
The thoroughbred's parents tell their kids that getting straight A's in school isn't enough because the kids in their class are "normal," and to cut it, they are going to have to strive far beyond what's taught in a classroom. They usually have various tutors starting in elementary school, do math and language courses after school, and engage in summer enrichment activities. They make sure their kids get into the gifted and talented programs in their kids' school, and if their kid doesn't make the cut, they hound the school as hard as possible to make sure their kid stays with the leaders of the pack.
Their parents give them extra homework during the summer so that they can test out of as many subjects as possible during the school year. Their parents know the algebra readiness exam is in 6th grade and that their child needs to score above a 90% to be able to take algebra 3 years early. They have their child prepare for this exam as early as their kid can handle the material. For these children, school should be a breeze, and they learn the real stuff during their studies outside of the classroom.
By middle school, they are spending summers at various math and science camps and doing STEM after school programs. I cannot stress how common math camp is. Most people I have met in STEM PhD programs have gone to math camp, and basically all know each other from their early days going to various math camps as kids. Moreover, in middle school, a lot of the parents start on SAT prep and hope they can do the bulk of their preparation before high school because in high school they have more difficult things to worry about. I know a lot of folks who got the SAT score they used for college in 8th grade. Some kids even have dubious non-profits that they started in middle school that they build up throughout high school in order to project sincere interest in outreach over a long time period—god forbid college admissions programs think you just created a non-profit to get into college. If the high school they want their kid to attend requires testing, they start their kids in test prep classes a few years prior to the high school admissions exam.
In high school, they are maxing out AP courses and taking the hardest possible courses available. Usually by junior year, they are taking at least one course at a nearby college. They are entering science competitions and scoring very well at the national and international level. Many of the students who do well at Intel science competitions or Science Olympiads have parents in that exact field of study who can help guide them towards more sophisticated ideas. I know someone who won the Intel science competition by doing a project in spectroscopy whose parents worked on spectroscopy professionally. That being said, the parents aren't doing the projects for them—they know that would ultimately hurt their child—they can just steer them towards actual cutting-edge science and tell them which projects are promising.
By high school, all of the thoroughbreds are together at various prestigious public and private schools. Scattered amongst the thoroughbreds are incredibly smart kids who got lucky, and a few people who are struggling in that academic situation who just got lucky during the admissions process. The kids who aren't thoroughbreds have no idea what's going on underneath the surface. They think the thoroughbreds are simply geniuses - they just have so much better mastery of the material and seem to learn everything more quickly than they do. They have no idea what they are stacked up against. They simply do their assignments, try to get good grades, and do a good job in the clubs at school.
If a thoroughbred finds themselves struggling in school for whatever reason, they get a tutor and work on it incredibly hard outside of school, though the parents would see that as a personal failure as they should already be so far ahead of their peers that it shouldn't be possible.
When the thoroughbreds apply to college, they end up all over, not just fancy institutions. This is primarily because colleges have unofficial admissions quotas for how many students they can admit from each high school. So the top 10% of the thoroughbreds from the top 10% of high schools fill up prestigious universities, and the rest go elsewhere. But do not fret; those who go elsewhere kill it in college and become academic superstars at their respective universities.
Once PhD programs come around, the thoroughbreds all end up back together. They all know each other from math camp, science competitions, and shared social circles from prestigious high schools. They have an academic base that's just incredibly hard to compete with if you did not have similar academic training. The additional dexterity you get with that much additional exposure to material is hard to overstate. Of the 50 students admitted to a physics PhD program at my university, most of their parents have PhDs, and all but one student took calculus in high school.
Their parents are not necessarily wealthy; they simply prioritized their child's education to an extent most families don't even realize is on the table. Very few people seem to realize just how far families are willing to go to ensure their kids succeed academically, and I hope this post shines some light towards what is going on under the surface of what it actually takes to raise a thoroughbred.
Simply horrible experience flying @airindia from Delhi to Amsterdam. Seats not reclining, screens working 50%, staff was totally insensitive to people with special needs like parents of toddlers.
This is the last time I will ever step into an @airindia flight, so you can keep your frequent flyer miles. They anyways dont get added to the user profile without bugs 😕
@TataCompanies@SingaporeAir I know you have big plans for Air India, but please atleast get the basics right!
Treat your passengers with some sensitivity, get their luggage back to them reliably. Fancy mobile apps and frequent flyer miles can wait.