There are really 2 sides to HFT: the low latency aspect of placing orders quickly, and the bandwidth/HPC involved in parsing large amounts of historical data
Most systems I've replaced had a poor understanding of the computer science of one side or the other
@yenwod_ Doing it this way lets you hand off normalized data to someone else asap (or use it yourself) while also building up a historical record
Also having normalized data lets you check integrity with some dataframe library
@basedtensor More damning is that they hire web backend people from places like Google who proceed to fuck everything up because "I don't know how you guys do things in fiiiinance but at Google we ${moronic_idea}"
@Dirnchen Usually both were busted. But with backtests in particular RAM consumption is typically an issue.
Or they use some framework like Spark that has nothing to do with trading but someone wanted it on their resume
@chatocv12@blogagog I mmap compressed data to/from them and use them as if they were terrabytes of RAM. I’m not even sure if an EPYC board can hold 16TB of RDIMMs but it would be really expensive
@DanielleFong@CattelainFrano1 Even cheap CPUs have laser etched branding on the heat spreader. Why should a $200 nvme be any different?
Especially when these things burn out like lightbulbs and/or throttle if you hammer them