Paramveer Singh

[Paramveer Singh @ParamveerS15896]

I am 23 working AI development!
Happy to connect and reach new developers !!!
Connect and show me your projects (^-^)

[Paramveer Singh @ParamveerS15896]

@harshendra_ai I built an AI that turns plain English into PCB boards 🤯

"ESP32 node, 5V regulator, decoupling caps" → a routed, fab-ready .kicad_pcb.

LLM planner → design critic → physics router → KiCad.
Runs on OpenAI, Gemini or Ollama.

⭐ + contributors wanted:
https://t.co/tnC2UE470n

[Paramveer Singh @ParamveerS15896]

@harshendra_ai Happy to connect 😊

[Paramveer Singh @ParamveerS15896]

@devboy_sg Let's connect !!

[Paramveer Singh @ParamveerS15896]

@neo_subhamoy Happy to connect !

[Paramveer Singh @ParamveerS15896]

@mayuri_3015 Happy to connect !

[Paramveer Singh @ParamveerS15896]

@SakshamMalhot27 Happy to connect !

[Paramveer Singh @ParamveerS15896]

@aadhyamoon Let's connect

[Paramveer Singh @ParamveerS15896]

I built an AI that turns plain English into PCB boards 🤯

"ESP32 node, 5V regulator, decoupling caps" → a routed, fab-ready .kicad_pcb.

LLM planner → design critic → physics router → KiCad.
Runs on OpenAI, Gemini or Ollama.

⭐ + contributors wanted:
https://t.co/tnC2UE470n

[Paramveer Singh @ParamveerS15896]

Thanks the team @MohammadShoeybi @MostofaPatwary @TheRealRPuri and all other authors for writing this encredible paper

[Paramveer Singh @ParamveerS15896]

Big models don't fit on one GPU. In 2019, NVIDIA researchers solved this with Megatron-LM, a technique that let them train an 8.3B parameter transformer — at the time, one of the largest language models ever trained. Here's how it works 🧵

[Paramveer Singh @ParamveerS15896]

Paper: "Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism" (2019) Authors: Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, Bryan Catanzaro 📄 https://t.co/cXmOzctg2m

[Paramveer Singh @ParamveerS15896]

Why this paper matters: Megatron-LM's tensor parallelism became a foundational technique used (directly or in spirit) across the industry for training today's largest LLMs. It's one of the papers that made "just make the model bigger" actually tractable.

[Paramveer Singh @ParamveerS15896]

They also found something subtle but important: careful placement of layer normalization in BERT-style models was critical to getting good results as model size scaled up — a detail that influenced how later large models were architected.

[Paramveer Singh @ParamveerS15896]

The results: they trained transformer models up to 8.3B parameters on 512 GPUs, sustaining 15.1 PetaFLOPs across the whole system at 76% scaling efficiency vs a strong singleGPU baseline. That efficiency number is the real headline parallelism usually leaks a lot of performance.

[Paramveer Singh @ParamveerS15896]

It's also designed to be complementary to pipeline parallelism (splitting the model layer-by-layer across GPUs). Combine both, and you get a system that scales to models far larger than any single technique could handle alone.

[Paramveer Singh @ParamveerS15896]

The elegant part: this requires no new compiler, no custom framework — just a handful of communication operations (all-reduce) inserted directly into native PyTorch. That simplicity is a big reason it became so widely adopted.

[Paramveer Singh @ParamveerS15896]

Why does that matter? Splitting inside a layer means each GPU only computes part of the math for that layer, then the results get combined. This keeps GPUs busy in parallel rather than sitting idle waiting for a full layer to finish elsewhere.

[Paramveer Singh @ParamveerS15896]

Megatron-LM's key idea is intra-layer model parallelism (aka tensor parallelism). Instead of splitting the model layer-by-layer across GPUs, it splits individual layers — the matrix multiplications inside attention and MLP blocks — across GPUs.

[Paramveer Singh @ParamveerS15896]

The core problem: as transformers grow past a few billion parameters, they simply don't fit in a single GPU's memory. You need to split the model itself across multiple GPUs — not just the data (that's "data parallelism," a different, older trick).