Donald Kinghorn

Scientific adviser at @PugetSystems PhD Theoretical Chemistry, Computer nerd ... Mathematics, ML/AI, all that good stuff ... and Jazz Guitar!

[Donald Kinghorn @dbkinghorn]

@NoXeN_Studios @PugetSystems I'm way behind on writing! I've been doing internal AI work.
Thanks for the nudge! I want to get posts up on TGI,TEI, vLLM as followups for those posts but I don't have a timeline yet.

[Donald Kinghorn @dbkinghorn]

I put up 2 posts @PugetSystems This is the first 2 parts of an LLM server configuration series. Technical How-To's. The series will be the basis for an auto-config LLM server bare-metal deployment script. There's more coming...

https://t.co/X2eQKrdNhk

https://t.co/eE0ZxVnILL

[Donald Kinghorn @dbkinghorn]

I did some numerical computing testing @PugetSystems with the AMD Zen4 Tr PRO. Outstanding performance! HPL, HPL-MxP, HPCG and OpenFOAM. On TrPRO 7995WX, 7985WX, non-Pro 7980X and Xeon-w9 3495X. Charts with max perf and Amdhal's Law scaling. Enjoy!
https://t.co/fWnKZNvibg

[Donald Kinghorn @dbkinghorn]

@Barnacules @PugetMattBach I should point out... In the post I did about running Falcon 40b I was using simple sharding with transformers across 2 or 4 6000Ada's

I now use HuggingFace TGI server for running models and it's WAY better! It's Linux only?? best run in a container. Wonderful model server!

[Donald Kinghorn @dbkinghorn]

@Barnacules @PugetMattBach When you run you see the execution move from GPU to GPU. It's like having perf of 1 with memory of many. You do have to have enough memory for the largest "chunk" though.

Look up "sharding and model parallelism with transformers".

But save your pennies for better GPUs too :-)

[Donald Kinghorn @dbkinghorn]

@sumseq @YouTube Awesome!

[Donald Kinghorn @dbkinghorn]

@BenFranklin2019 @PugetSystems Oh yes! It's way beyond my reach. I feel fortunate that I get to work with it.

2xA6000 system would reduce that cost a fair amount. But still a lot of $

However, for some small organizations it might be a reasonable way forward!

[Donald Kinghorn @dbkinghorn]

I put up a post @PugetSystems answering the question,
Can You Run A State-Of-The-Art #LLMs On-Prem For A Reasonable Cost?

I had success using the STOA open LLM Falcon-40b with 2 RTX 6000 Ada GPUs.

I hope you find the post encouraging! --Don

https://t.co/sCh2LgE9r4

[Donald Kinghorn @dbkinghorn]

@Sentdex Hey Harrison! :-)
I'm working on setting up falcon-40b-instruct on a system @PugetSystems to experiment with.

May try a fine tunning on base ... wish me luck

[Donald Kinghorn @dbkinghorn]

I just put up a blog post @PugetSystems It's how to setup #Apache web server on #Ubuntu 22.04 to allow user sites in $HOME/public_html. It took me much longer than expected. Thus the blog post! Hopefully it will save someone else some time.

https://t.co/8cqaWUKV1L

[Donald Kinghorn @dbkinghorn]

Microsoft Build 2023 has finished. It was very good! Predictably it was focused on AI and Azure. The best talk by far in my opinion was Andrej Karpathy's "State of GPT" linked below.
https://t.co/A9ScEPc22O
@PugetSystems #ai #azure #microsoft #GPT #Karpathy

[Donald Kinghorn @dbkinghorn]

I added a post @PugetSystems with LOTS of links to talks at #NVIDIA #gtc23 . Great conference! The links are to NVIDIA On-Demand for sessions that I thought were interesting. You might like some of them too! (I'm still catching up on ones I missed!)

https://t.co/PjvCgfEX0l

[Donald Kinghorn @dbkinghorn]

I updated testing for P2P on 2 x 4090s. @PugetSystems As of driver 525.105.17 it is "properly" disabled on GeForce. Bummer.

For the updated results I have added another Appendix. Also, check the comments, several people have reported their updates too

https://t.co/9UPWafcw6k

[Donald Kinghorn @dbkinghorn]

@AndrewYNg Code hints, and I just started trying the integration with Bing for search/question answer

[Donald Kinghorn @dbkinghorn]

#NVIDIA #GTC23 Keynote was amazing!
The whole thing is a "highlight"

I was riveted to the screen. I think this is the most impressive GTC keynote I've seen!

https://t.co/ZLUlbNOsqJ

[Donald Kinghorn @dbkinghorn]

@SprayOnCopper @sumseq @PugetSystems I got 44.1 GFLOPS on the single socket 3495x using Intel's HPCG code form oneAPI

[Donald Kinghorn @dbkinghorn]

Did a bit of testing @PugetSystems on the #Intel Xeon-w9 3495x 56-core. HPL, HPCG, and NAMD Ubuntu 22.04 and oneAPI

Bests results ever with CPU! Even dual socket.

Great CPU for HPC/Sci/ML

Engineering sample, Intel says we can't publish results with those anymore :-)

[Donald Kinghorn @dbkinghorn]

@BenFranklin2019 @PugetSystems My feeling is, if you can use GPU then you should.
The new Xeon has BF16 and a "matrix unit" that I haven't checked out. It may be pretty good with Intel's build of PyTorch ??
I would target the CPU for engineering workloads that need dbl precision and lots of memory

[Donald Kinghorn @dbkinghorn]

@FelixCLC_ @BenFranklin2019 @PugetSystems The new Xeon has a couple of interesting features that I have not really checked out.
BF16 and a new matrix compute unit. I don't think it's going to compete with GPU but it's getting close. And if you need the memory it could be good.

[Donald Kinghorn @dbkinghorn]

@FelixCLC_ @BenFranklin2019 @PugetSystems I will make for a very nice workstation platform.
But, yes! If you can use GPU you should!