@penberg It's a delicate balance
From one side you get isolation, limited blast radius and better concurrency
But you have to live with schema syncing, distributed queries and file descriptor exhaustion challenges
M:N works too and you get the best, and also the worst of both!
@glcst@HSVSphere Wouldn't just setting the page cache size properly (or the mmap size if multiple processes use the db) deliver the needed memory-then-desk function but transparently?
@kellabyte Using how many client processes/threads?
Typically a cluster should perform better with more concurrent clients until it reaches its saturation point
But regardless, that throughput number sounds very ... low?
@lemire@Adev I would argue it is technological & governing advancement, power, expansion & wealth, in that order
And indeed there is a clear "current" advantage in the Western approach to tech and government. But history didn't end yet and new models will emerge, exactly like this one did
@Adev@lemire So I bring up the colonial past and you mention the state post the 70s. Who's straw manning?
I am not really sure what is being argued here? That European colonialism didn't happen? or that it wasn't an economic boost for Europe?
Both are simply false anyway
@Adev@lemire This is totally wrong. Small colonies existed for Scandanavia while Germany & Italy had vast colonies in Africa.
Pretending the systematic stealing of resources accompanied with oppressing the natives didn't happen or was inconsequential economically is bizarre to say the least.
@lemire Of course there are multiple other factors, but we can't escape the fact that England, France, Germany, Spain, Italy, Portugal and The Netherlands benefited a lot from their colonies and their resources, at the right moment too. Citing other examples doesn't negate this fact.
If you are building a really high performance data ingestion pipeline, what matters most is how many pages you are touching per tx and in which cache level (L1, L2, L3) would they fit.
WAL helps, but it will largely depend on how many records are written per page in a single tx
user1: why is this slow?
rails dev investigates: oh, it’s N+1 AR queries.
user2: why is this slow?
C dev investigates: oh, it’s N+1 malloc().
the similarity is uncanny.
@Iztok@brennandunn@patio11 This, the moment you need to set aside time to talk to a potential customer means a completely different engagement model than self-serve, and hence a completely different pricing structure is needed.
Spent a little time optimizing a hot route on a Ruby Roda + SQLite JSON API app
15 db queries: 1 refactored, 3 new indexes & all now use prepared statements
DB time was ~5ms per req, now ~500us per req
Most of the req time was in DB, reqs now are under 1 ms
Roda + SQLite FTW!