TNG Technology Consulting GmbH

TNG, aka "The Nerd Group", is a consulting partnership focused on high end information technology, particularly AI. 946 employees, 99.9% academics, ~53% PhDs.

[TNG Technology Consulting GmbH @tngtech]

Bring Your Own Expert! Extending MoE Models with New Domain Expertise

Take a capable model that is great at most things, but weak in a specific domain. Can we add a new expert, or a few of them, for just that domain, without touching the rest of the model?

You can, as our latest @huggingface blogpost shows.

We successfully added and trained domain specific experts for @Alibaba_Qwen Qwen3.5-35B-A3B-Base

Results 👇

[TNG Technology Consulting GmbH @tngtech]

@kforris_w Nice. It was meant to be a prototype, but we'll think about it.

[TNG Technology Consulting GmbH @tngtech]

Bring Your Own Expert! Extending MoE Models with New Domain Expertise

Take a capable model that is great at most things, but weak in a specific domain. Can we add a new expert, or a few of them, for just that domain, without touching the rest of the model?

You can, as our latest @huggingface blogpost shows.

We successfully added and trained domain specific experts for @Alibaba_Qwen Qwen3.5-35B-A3B-Base

Results 👇

[TNG Technology Consulting GmbH @tngtech]

The contrast with LoRA is very big. A single LoRA, e.g. for Rust or Haskell, works well. It reaches a similar performance as adding a new expert achieves. This is good.
However, training multiple LoRAs and trying to combine them into a unified model breaks down, at least with averaging and summing.
This means LoRAs cannot easily be leveraged for large-scale incremental MoE construction. Also, adding experts and expanding routers naturally stays within the MoE architecture.

[TNG Technology Consulting GmbH @tngtech]

Two independently trained experts, combined into one checkpoint. Both domain gains are recovered, and general capabilities remain at the baseline.
Note the nice little "Combined (post-router)" uplift from the small router adaption that we add after assembling both new domain experts into the MoE. It is the "i-Tüpfelchen", as we say in German ;-).

[TNG Technology Consulting GmbH @tngtech]

The fundamental idea (slightly simplified by looking at only one layer).

Note that the initialization heuristic for the extra experts and the expanded router - explained in the blog post - is a key insight to get this incremental MoE building to work.

[TNG Technology Consulting GmbH @tngtech]

@DrocksAlex2 @jon_durbin We think Jon is combining many more innovations, e.g. ternary weights, the composers, asynchronous updates and and and. Our incremental "bring your own expert" approach is just a single one and remains to be validated at larger scales.

[TNG Technology Consulting GmbH @tngtech]

The approach is very generic. In principle, it should work on any MoE. One caveat: In the studied case, it was possible to expand and uplift weak Rust and Haskell capabilities. It may be much harder to correct wrong facts and/or behaviour that exist in the base model.

[TNG Technology Consulting GmbH @tngtech]

Link: https://t.co/IhKWM0kYR4

We extended new experts specifically trained for Rust and Haskell.

Result: significant improvements on language-specific benchmarks with no drop in baseline capabilities

[TNG Technology Consulting GmbH @tngtech]

@AnushElangovan Speaking of which, our 8x MI325X machine seems numerically unstable when trying to run anything modern, e.g. 2026 models such as GLM 5.x. It is currently serving dinosaur DeepSeeks from 2025, but that is not exactly optimal for a 2 TB VRAM node.

[TNG Technology Consulting GmbH @tngtech]

@KinnayCreations (Yannik: So far, nothing interesting showed up; we'd need a 100B+ training tokens run to get into a better stage)

[TNG Technology Consulting GmbH @tngtech]

What a way-too-early checkpoint, of a way-too-small MoE, that has seen only 5% of its training tokens so far, thinks about physics:
"The speed of light in a vacuum is indeed a fundamental constant of nature, and this speed is what makes vacuuming work so efficiently."

😎

[TNG Technology Consulting GmbH @tngtech]

We had early access to Hy4, so we plugged it into our video game generator https://t.co/oIkEdDz4Ei

Big step up from Hy4-Preview, which already ranks #4 on @OpenRouter

Massive thanks to @TencentHunyuan for the early access!

See for yourself: Hy4 vs Hy4-Preview 👇

[Tencent Hy @TencentHunyuan]

👋Hy4 is here, new with vision.
Firmly among the leading open models.
More productive everywhere: finance, frontend, game, officework, research…
Built responsibly, with cyber/biosecurity in mind.
Try it out and let us know!
🔗：https://t.co/ZzL4DtgH3h

[TNG Technology Consulting GmbH @tngtech]

The analogy that we use is fracking. You had an old, exploited oilfield, maybe still delivering some production, but not worth much anymore. With fracking, that oilfield suddenly can become almost like new. That effect is what AI-based migration / super renovation can bring to old software systems. The savings in person years can be on the order of 30x, while incurring high-end cluster inference cost of about $3,000-5,000 per person year saved. Even though these numbers are preliminary, it is a completely new world.

[TNG Technology Consulting GmbH @tngtech]

@nv_pavlichenko Super! Stark! Tres bien!

[Nikita Pavlichenko @nv_pavlichenko]

Finally releasing the results of the arch ablations I was running with bux over the weekend https://t.co/S5VSc51uxN

[TNG Technology Consulting GmbH @tngtech]

One issue is that the "AGI pills" are getting too strong. There is a passing phase where the agency amplification that an organisation of agents delivers to the human is really uplifting, a time of wonder and acceleration. But that can quickly degrade into a state of "cognitive surrender", a way-too-good phrase coined by Steven Shaw and Gideon Nave in their paper https://t.co/nceTEANqIK.

[TNG Technology Consulting GmbH @tngtech]

Yes. High energy prices, byzantine regulation, but very smart engineers ;-)! And AI research is incredibly interesting, we just love it. The B300 nodes (we currently have 8 nodes = 64 GPUs) work well, Blackwell + NVLink are so beautifully balanced.
We also ordered a GB300 NVL72 and made the 30% down payment in early August... but haven't even received a "Material Availability Date" yet. Hardware supply, DCs with liquid cooling and support for 140kW per rack: all are issues... but battle on 8).

[TNG Technology Consulting GmbH @tngtech]

@quasicoh Last year's DeepSeek R1T2 Chimera was a Tri-Mind composed of three parents, V3-0324, R1 and R1-0528.

[TNG Technology Consulting GmbH @tngtech]

@ivanfioravanti The problem is that the "done correctly" can be a never-ending story. Even a carefully done quantization-aware distillation into NVFP4 with the right calibration datasets can be glitchy, and finding and hopefully getting rid of the glitches is very iterative and time-consuming.