Notify.lk

Notify.lk is a SMS gateway service with a lot of extra tools which can support local businesses to manage their customers only paying the cost of SMS.

[Notify.lk @notifylk]

This issue with the operator route is now resolved. All sender ids should resume as usual.

[Notify.lk @notifylk]

We are experiencing issues with an operator route. Portion of sender ids might experience delivery issues at the moment. We are working on solving the issue. Thank you for your patience.

[Notify.lk @notifylk]

The connection to RDS required TLS, and we made a mistake with the certificate configuration. This caused the services to fail connecting to the database until we fixed it.

The issue is was resolved by 1:52 AM and all services came back online.

3/3

[Notify.lk @notifylk]

We had a 22-minute downtime today from 1:30AM to 1:52AM affecting our API and dashboard.

This happened during our database migration from Google Cloud to AWS as part of our continued effort to provide a smoother, reliable service.

We’re sorry and appreciate your patience.

1/3

[Notify.lk @notifylk]

We recently decided to move our services from Google Cloud GKE to AWS EKS.

During the migration, we were replicating the database and preparing the final switch to AWS RDS.

2/3

[Notify.lk @notifylk]

There was an incident between 07:32 and 09:52, which caused a connection issue between Notify and an operator. This interrupted the delivery for a portion of Sender IDs. The matter is now solved we are investigating what caused the issue. Thank you.

[Notify.lk @notifylk]

The impact was limited to a small number of customers. We sincerely apologize for the inconvenience and appreciate your patience.

[Notify.lk @notifylk]

Status update: We identified and resolved an issue affecting SMS submissions (affected for a subset of sender ids) today starting around 7:00 AM SL time. The root cause was intermittent IPSec connection discrepancies between Notify and a provider site.

[Notify.lk @notifylk]

This caused one of the queues to hang, delaying message submissions for one operator on a single queue. The issue was fully resolved by 9:20 AM, and all pending messages were submitted (with a delay).

[Notify.lk @notifylk]

This is now resolved. The messages will start to deliver again as usual. Thank you for your patience and support.

[Notify.lk @notifylk]

Status update: there is a known connection issue with a single operator which might effect a majority of our sender id delivery. We are working on it and we expect it will be resolved within 30 minutes. Thank you for your patience.

[Notify.lk @notifylk]

Update: We have detected a delivery issue from an one operator end. Due to this, the SMSs whose operations works only with this operator might affected. We are working with them to resolve the matter.

[Notify.lk @notifylk]

This matter is now resolved. Queued SMS which were not sent before will be delivered as well. Thank you for your patience.

[Notify.lk @notifylk]

Service degrade alert: We have encountered a connection issue with one operator due to a technical failure. We are working with the provider to resolve it as soon as possible. Sender IDs depend only on this provider may experience delivery issues at the moment.

[Notify.lk @notifylk]

[Status update]: One of the major mobile operators having an issue with their SMPP services. Due to this matter there might be delivery issues for a few Sender IDs if the ID is only used via this route. Their IT team is working on a fix.

[Notify.lk @notifylk]

The root cause for this was a bug in our API rate-limiter. We are working on it to resolve it soon to avoid this from happening again. Thank you again!

[Notify.lk @notifylk]

We have detected an unusual length in our transactional queue and it made SMSs to be delayed by 5 minutes. We apologize you for the inconvenience this brought to your workloads. We have expanded the number of workers and everything back to normal as o f now. [1/2]

[Notify.lk @notifylk]

We are really sorry for any inconveniences this may have occurred to you or your users. We will investigate the cause for this Redis hang. Thank you!

[Notify.lk @notifylk]

Update: we detected a queue delay (about ~5 minutes) in transactional priority queues today around 11:00AM SL time. The issue was a hang in our Redis cluster and we were able to resolve it and get the queues back to speed within next 10 minutes.

[Notify.lk @notifylk]

Impact: transactional SMS were delayed to be pushed out from our servers for around 5-10 minutes.