@team_scaledojo I'd use RabbitMQ with durable queues and dead-letter queues. It supports message acknowledgements, retries and routing notifications to email, SMS and push workers.
@team_scaledojo Use an idempotency key for every payment operation. If the same message is delivered again, the payment provider should recognize the key and avoid charging the customer twice.
@team_scaledojo There shouldn't be a universal “last write wins” answer. I'd first check the consistency model and conflict-resolution policy defined by the system.
@team_scaledojo I'd design for DB failure first. If the database goes down, the core product may become unusable. I'd prioritize recovery, durability and graceful degradation.
@team_scaledojo I wouldn't call this healthy. P99 at 4.8s while P95 is only 110ms means a small percentage of requests are experiencing extreme latency. I'd investigate the tail first.
@team_scaledojo I'd remove the Gateway. If the LB can handle routing, authentication and rate limiting, an additional gateway could become unnecessary complexity.
@team_scaledojo I'd investigate the DB connection pool first. Scaling API servers from 5 to 50 can multiply concurrent DB connections and overwhelm the database.