Your virtual coding assistant who spots errors and unexpected behavior, does routine tasks for you and generates unit tests with meaningful values in real time.
Ever wished you can just generate your #unittests instead of painfully writing them? In this video 👇 Evelyn demonstrates of how to use https://t.co/5Hj9Zey9DL to speed up your daily development workflow 🚀✨
https://t.co/zUy3sXBUSG
#golang#java
New to LLM coding agents? 🤖 Our introduction covers the capabilities, limitations, and use cases of LLM agents for software development 👇
https://t.co/ElB3iTcpS4
Updated DevQualityEval v1.0 results are in 👀 Check out how our new king of cost-effectiveness (Google’s Gemini 2.0 Flash Lite) performed, and find out if Claude 3.7 Sonnet (Thinking) is worth the additional costs 👇
Insights of analyzing >100 LLMs for the DevQualityEval v1.0 (generating quality code) in latest deep dive
- 👑 Google’s Gemini 2.0 Flash Lite is the king of cost-effectiveness (our previous king OpenAI’s o1-preview is 1124x more expensive, and worse in score)
- 🥇 Anthropic’s Claude 3.7 Sonnet is the functional best model (with help) … by far
- 🏡 Qwen’s Qwen 2.5 Coder is the best model for local use
- Models are on average getting better at code generation, especially in Go
- Only one model is on-par with static tooling for migrating JUnit 4 to 5 code
- Surprise! providers are unreliable for days for new popular models
- Let’s STOP the model naming MADNESS together: we proposed a convention for naming models
- We counted all the votes, v1.1 will bring: JS, Python, Rust, …
- Our hunch with using static analytics to improve scoring continues to be true
All the other models, details and how we continue to solve the "ceiling problem" in the deep dive: 👇🧵
(now with interactive graphs 🌈)
Looking forward to your feedback :-)
OpenAI's o1-preview is the king 👑 of code generation but is super slow and expensive 😱 This and other insights of analyzing >80 LLMs in the deep dive blog post from the DevQualityEval v0.6 for generating quality code 👇
- OpenAI’s o1-preview and o1-mini are slightly ahead of Anthropic’s Claude 3.5 Sonnet in functional score, but are MUCH slower and chattier.
- DeepSeek’s v2 is still the king of cost-effectiveness, but GPT-4o-mini and Meta’s Llama 3.1 405B are catching up.
- o1-preview and o1-mini are worse than GPT-4o-mini in transpiling code
- Best in Go is o1-mini, best in Java GPT4-turbo, best in Ruby o1-preview
Please support our work for the community by liking and sharing this post! 🙏
All the details and how we will solve the "ceiling problem" in the deep dive https://t.co/TRo8GsVR28 (2x the content as the previous one!)
Execute only the tests you need 💡We see a 29% reduction in test execution times with just a basic approach. Details of the benchmark, example & guide: https://t.co/DcCujUNcvn
#LLM#observability 👀 Monitoring can help improve the performance of your LLM applications. Here’s what you need to know & the most useful tools for LLM observability 🔍
https://t.co/V0zfiCRhaC
Are you using #AI-powered tools in your #softwaredevelopment workflow❓ Aider is a good example that works well and even offers voice coding 🦾 Here’s our guide to using Aider:
https://t.co/XKGfQytyqF
How well do #LLMs generate code ❓ There’s only one way to find out: #benchmarking models for #softwaredevelopment tasks. Here’s a roundup of popular LLM benchmarks & insights into our take on the topic 🤓
https://t.co/nkUVxXKfz9
What metrics do you track when evaluating #LLMs? 👀 Here‘s an overview of complex statistical and model-based scorers 💡 Bonus: we also cover the #evaluation#frameworks that help you get started assessing #LargeLanguageModels. https://t.co/Y7AkEna0wT
Have you ever tried to fix performance issues in your #GoLang application but could not find why it was taking longer sometimes? 🚀 Instrumenting your application for #Go#tracing 💡might be what you need:
https://t.co/rY80MpEAsL
Do you #reuse code? ♻️ Optimizing code for #reusability helps drive down development effort and cost while improving quality. Here’s a list of the most important reusability best practices for #Java#coding:
https://t.co/j9b151WnYT
Confused by LLM evaluation? 😵💫 We can’t blame you. Our new series on LLM #benchmarking guides you through all you need to know about measuring #LLM performance:
https://t.co/z7TalXRPZl
Struggling with performance bottlenecks in your #GoLang app? 🤔 #Go#tracing to the rescue! Explore our comprehensive guide and conquer even the toughest optimization challenges 💪
https://t.co/rY80MpEAsL