only one catch: fields can't see each other. So it's great for extraction and classification (many short, independent fields from one input) and bad when a later field depends on an earlier one..
People don't realize Jev-style inference isn't just for trained classifiers.
It works with any regular LLM that outputs structured data.
Normally, the model types JSON token by token, sequentially.
Instead: Prefill the input once and cache the KV, fork one branch per field, decode all branches in a single batch, and let code build the JSON.
Latency drops from the sum of all fields to the longest one :)
@ZahirHamroune You can fine-tune the underlying LFM2.5-350M normally with Lora/SFT on your private prompt->output pairs (Liquid has TRL/Unsloth examples). Then load the fine-tuned weights into this inference code. The parallel decoding part is separate from training
Jev-inspired inference. 350M parameters. 63× faster !
Spent a few hours trying Jev-style inference with LFM2.5-350M.
63× faster on an L40S. 8x on MPS. No training (for now) just parallel decisions.
Code + weights on HF. Link below 👇
After co-inventing ChatGPT, I kept asking myself: why have superhuman chat models not led to AGI?
I’ve spent the last 2 years in stealth building a new way to train models (RLCD), and a new type of frontier AI model that we are releasing today: Jev
• 20-200x faster
• 40-400x cheaper (w/ output tokens free)
• Frontier composable intelligence optimized for decisions
AFAICT the shortest path to AI-based economic revolution
@icpp_pro I prefill the prompt once, reuse the KV/conv state, then score all allowed field values in parallel and assemble the json in Python.
So instead of autoregressive decoding over the whole output, it becomes a set of parallel classification-style decisions :)
@OpenAI is trusting @lawveAI to help bring ChatGPT for Law to life. We’re proud to be a launch partner.
We’ve worked with OpenAI to bring 270+ legal AI automations created by the https://t.co/1DF5Q3yPsH community directly into ChatGPT, across 22 practice areas and 35 jurisdictions.
To our contributors : your name is in there. Your expertise can now be discovered by lawyers and legal teams around the world!
Huge thanks to everyone who contributed, and to the OpenAI team for helping us bring Lawve to a much broader audience.
🔗 Add Lawve to ChatGPT: https://t.co/p6np0oT5zC