Vincent Bosch

LLM Company Law | Tech enthusiast

[Vincent Bosch @vincentbosch]

@ivanfioravanti @antirez Could you share your Autoresearch setup? I'd like to optimize GLM 5.2. Currently running it on my Mac Studio using MTPLX/oMLX native pp-kernels, but seeing improvements for Laguna got me eager to find out what performance we might be leaving on the table. Thanks in advance!

[the tiny corp @__tinygrad__]

The limiting factor of software engineering has never been code production. It is complexity management.

[Vincent Bosch @vincentbosch]

@advancedcskills Looks very interesting! Is this MCP publicly available by any chance? I am searching online for such tools, but cannot find anything like this one

[Vincent Bosch @vincentbosch]

@ivanfioravanti Would you be so kind to take the new dynamic quants in MLX-LM for a spin with Qwen 235B and share the sensitivities-file with us less URAM-blessed? 😇

[Vincent Bosch @vincentbosch]

@awnihannun Let’s populate a HF-repo with precomputed sensitivity files 😃

[Vincent Bosch @vincentbosch]

@awnihannun @N8Programs @LiMzba Are you saying that 3 bit with group size 32 would be better than 4 bit with 128 group size? Based on the evals given with the PR by barronalex for the mixed quants, I'd think that 4 bit is better?

[Vincent Bosch @vincentbosch]

@awnihannun @N8Programs @LiMzba Great work! Hopefully this brings us close to full precision feel. Could you maybe also upload a 4 bit variant with a group size of 128, for us URAM-poor M3/M4 Max users? 😇

[Vincent Bosch @vincentbosch]

@moskstraum21745 @ivanfioravanti If you can run the mixed 3_6 quant, you will also be able to fit a 4 bit MLX quant with q_group_size of 128, albeit barely. I quantized it myself today to compare both models. It seems the 4 bit is a bit better overall, but I must test some more and maybe some other quant configs

[Vincent Bosch @vincentbosch]

@osanseviero @ollama @lmstudio Great! This is exactly what I was hoping for 🙏🏻

Could you also upload the MLX-QAT with mlx-lm instead of mlx-vlm? For the people running it only for text-gen with an OAI-API compatible server? Thanks!

[Vincent Bosch @vincentbosch]

@anemll Out of curiosity, what is the largest model you ran on ANE? Would for example Mistral Small 2501/2503 or Gemma 3 be possible to run on ANE?

[Vincent Bosch @vincentbosch]

@UnslothAI Very interesting! Am I correct in that you examined the logits for a given prompt in full precision versus a 4-bit quantized version and then started comparing the logits with layer-wise keeping layer X in full precision and the rest in 4-bit? Yielding the most significant layers

[Vincent Bosch @vincentbosch]

@btibor91 According to the changelog update, Mistral Large 2411 will be released today: https://t.co/PXdcmAbFmA

[Vincent Bosch @vincentbosch]

@Prince_Canuma Since you're the master of porting models to MLX, I thought lets check whether you've got any plans to port Jamba 1.5 Mini to MLX? Results for RULER are out of the water, so it would make a great addition to the list 😃

[Vincent Bosch @vincentbosch]

@BramVanroy Looking at the model card, a 40B parameter version should also be coming. Looks interesting! Let me know how it stacks up against Mistral or other Dutch-speaking LLMs.

[Vincent Bosch @vincentbosch]

@alexandr_wang Very interesting benchmarks, thanks a lot for the hard work! If it’s possible, I am curious as to the results of the latest Command R and Command R Plus models (08-2024).

[Vincent Bosch @vincentbosch]

@seb_ruder Are you interested in scheduling a call or something? Then we could discuss the Dutch multilingual capabilities of CR(+) and whether I can contribute to it.

[Vincent Bosch @vincentbosch]

@BramVanroy @seb_ruder You’re welcome, and please let me know your experiences. Always curious about the findings of fellow Dutch-speakers with various LLM’s :-)

I am aware about the licenses and its implications. I mainly use the API’s, since the options for local models are still being explored.

[Vincent Bosch @vincentbosch]

@BramVanroy @seb_ruder Mind that Gemma 27B is very sensitive to the instructions. I must add “Antwoord uitsluitend in het Nederlands” to avoid English responses. Command R Plus was my previous go to model. Now it seems the updated Command R can replace it. Need to test the new CR+ model some more 2/2

[Vincent Bosch @vincentbosch]

@BramVanroy @seb_ruder I do indeed have experience with Llama 3.1 70b. Although it speaks Dutch quite well, I find its writing style too ‘popular’. My main line of work is legal text and corporate (legal) documents. Gemma 27B handles that fine, but the context length of 8192 is limiting. 1/2

[Vincent Bosch @vincentbosch]

@seb_ruder @BramVanroy I found the generation of Dutch from the previous Command R+ to be amongst the top of the line. The new Command R made a giant leap and can match Gemma 27b, which is great multilingually. I do agree that it feels a bit translated sometimes. Would love to help you improve Dutch!