Introducing BCA — Blockchain Applied 👋
Clean, production-ready data access for #Cardano and #Bitcoin. On-chain data via cloud APIs and #BigQuery — no node required.
Built on HA Postgres · Kubernetes · open-source.
https://t.co/BacPA4nLr7
#blockchain#DataEngineering
Things worth saying out loud:
• SLAs only work if measured. The "median first-response time last month?" has an answer.
• The scheduler is memory-hungry.
• Integrate it. A ticket that can't trigger anything is just an email with actionable steps.
https://t.co/psoR5ruIna
🎫 #FOSS stack series, post 5 of 9: Zammad.
If OpenProject manages the work we plan, Zammad manages the work that finds us: customer emails, support requests, a failure report. It's an open-source helpdesk that does the unglamorous job of turning an inbox into a workflow.
• Triggers and schedulers for follow-ups, escalation, auto-replies
• A clean API so other systems can open tickets when they need a human
It runs on K3s with its HA Db. Inbound mail comes via Postfix; outbound via SMTP: email on [email protected], a ticket appears.
• External database, always. Don't run the bundled Postgres in prod.
• Seed data is fragile. First deploy migrates and seeds; the next one doesn't. Plan for it.
https://t.co/W7anKJO9kr
📋 #FOSS stack series, post 4 of 9: OpenProject.
We tried the usual project-management suspects and kept coming back to host it ourselves. OpenProject gives us work packages, Gantt charts, agile boards, time tracking, and roadmaps — all open source, all on our cluster.
• Roadmaps tied to real work packages, so they stay honest
• APIs everywhere — anything in the UI is automatable
Lessons:
• Rails memory is a thing. Tune Ruby GC, use jemalloc, lower Puma threads. We brought ours into a sane envelope with no measurable latency impact.
@IOHK_Charles Sad news about TapTools!
we are still going strong in the other direction: more and better services to the community. Though, we have chosen a rather cautious and slow trajectory.
What we'd tell our past selves:
• Failover is not free — plan for the reconnect, test it
• Use Sentinel-aware clients; hardcoding a master defeats the point
• Don't use Redis as a database
https://t.co/QWiC8dbmo7
⚡ #FOSS stack series, post 3 of 9: Redis HA with Sentinel.
PostgreSQL is for things we need to keep. Redis is for things we need to keep fast: auth tokens, per-user rate limits, session state, ephemeral caches.
One Sentinel pod refused to start after a config corruption — we fixed it with a small busybox pod that mounted the PVC, repaired the file, and exited. Not glamorous.
• HA ≠ backup. Patroni protects you from node failure, not from DROP TABLE.
• Connection pooling matters. A 5s DB failover becomes a 5m app outage with stale connections.
https://t.co/Ruaae1sR7j
🐘 #FOSS stack series, post 2 of 9: PostgreSQL HA with Patroni.
Almost everything stateful at BCA writes to Postgres — Ghost, OpenProject, Zammad, our own app data. So Postgres has to be boring. Boring means three nodes, automatic failover, no human in the loop.
The setup:
The architectural point: don't put etcd on the same failure domain as the thing it's protecting. Consensus only works if it survives what Postgres doesn't.
Lessons learned:
• Test failover BEFORE you need it. Kill the leader on a Tuesday afternoon and watch what your apps do.