Muzammil Bashir

Observability Architect. 10+ years optimizing complex systems. Stack: K8s, Grafana, Prometheus, Loki, Tempo, Mimir, Terraform, Python.

[Muzammil Bashir @mbashir360]

Who else is talking to Claude to learn more? To know more about the problem! to get to the bottom of things!

Not just generating code and hoping it will work!

[Muzammil Bashir @mbashir360]

Sometimes, doing things by hand feels more refreshing.

[Muzammil Bashir @mbashir360]

Yesterday, I set up a new VPS from scratch for a personal project.

Bought a new domain, spun up a new VM, set up SSL, and started prepping it for deployment. With so much automation and managed infra available, it was still enjoyable to redo all the steps manually.

[Muzammil Bashir @mbashir360]

@Samra_TC I have been facing problems with meezan bank and it wastes a good bit of time.

[Muzammil Bashir @mbashir360]

@Samra_TC @Samra_TC i know people who have been able to get it to work without visiting branch. But regardless of that, YOU will need the branch.

They will eventually create problems for you that will require a physical branch and then no branch ownes you because you are not registered.

[Muzammil Bashir @mbashir360]

Finally got OpenObserve working with Grafana. 🎉

The primary hurdle I faced was around authentication.

One thing that isn't obvious from the docs: metrics are queried using user credentials, not stream credentials. That took me a while to figure out.

[Muzammil Bashir @mbashir360]

I've been exploring OpenObserve to see if it can replace my usual LGTM deployments.

Next experiment: using Grafana as the frontend with OpenObserve as the backend.

I'll share what works, what breaks, and whether it's even worth doing. 👀

[Muzammil Bashir @mbashir360]

@devops_nk It's ok. If you really ended up learning AWS, Azure shouldn't be a problem.

[Muzammil Bashir @mbashir360]

On Grafana Cloud, I’ve configured alerts to notify me if either Prometheus or Loki goes down. It gives me piece of mind because now I have an independent signal path and it gives me higher confidence in my on-prem monitoring stack. (2/2)

[Muzammil Bashir @mbashir360]

Recently, I built a small tool I’m calling “prom-agent”. It’s essentially a lightweight Prometheus instance running in agent mode on an onprem cluster. It pulls a targeted set of metrics from my primary Prometheus (running onprem) , and ships them to Grafana Cloud. (1/2)

[Muzammil Bashir @mbashir360]

3. Efficiency: Metrics are far more storage-efficient than logs.

4/4

[Muzammil Bashir @mbashir360]

A friend told me this weekend they weren't sure if they actually needed metrics. "We already have logs."

Here's the thing: logs tell you what happened, but metrics tell you how the system is breathing.

Why you need both!!!!

(1/2)

[Muzammil Bashir @mbashir360]

2. Actionable insights: They support aggregation, e.g., average CPU utilization across cores.

3/4

[Muzammil Bashir @mbashir360]

You need both because they both complement each other.

Some metric pros:
1. Trends over time: Metrics reveal patterns, like daily signups or traffic spikes.

2/4

[Muzammil Bashir @mbashir360]

Optimized an expensive #Loki query for a client recently.

1. 50% faster end-to-end query time
2. 1/3 the data scanned
3. 1/5 the processing time

What do you think was the fix?

[Muzammil Bashir @mbashir360]

How are people adding observability to new apps? From group up or as an after thought?

[Muzammil Bashir @mbashir360]

@_jaydeepkarale You delete the corrupted checkpoint

[Muzammil Bashir @mbashir360]

Me in 2026: Still SSHing into a VPS just to read Discord bot logs like a caveman.

Fixed it real quick:
- Installed otel collector
- Dropped in the API keys
- Now streaming logs straight to Grafana Cloud

Stop checking logs in the terminal, your sanity will thank you.

[Shabeer Chauhan @shab_eer_]

@mooiharttt 😬😬