We had an Ubuntu web server that suddenly started making outbound HTTPS connections late at night.
Nothing was down.
The website was still working normally.
CPU and memory looked fine, so at first there was nothing serious to suggest the server had been compromised.
I logged in and started checking the running services.
One service caught my attention:
system-update.service
The name looked normal enough. On a Linux server, you can easily assume something like that belongs to the operating system.
I checked the actual service file.
Its ExecStart was pointing to:
/usr/local/bin/.system-update
That was where things stopped looking normal.
It was a hidden executable inside /usr/local/bin, running as a systemd service and configured with:
Restart=always
Meaning if the process died, systemd would simply start it again.
I checked the file.
It was a stripped 64-bit ELF binary.
Then I pulled the PID for the service and checked its network connections.
The process had an established connection to an external IP over port 443.
So now we had a suspicious binary, persistence through systemd and an active outbound connection.
The next question was how it got there.
I checked the timestamps.
The service had been created around 6:43 PM the previous evening.
Then I went through the authentication logs.
This was the part that really caught my attention.
A few seconds before the service was created, the www-data account had executed commands with sudo.
www-data is normally the account used by the web server.
It should not casually be copying executables into /usr/local/bin and enabling system services as root.
The logs showed it copying the hidden binary from /tmp, reloading systemd and enabling the new service.
At that point we isolated the server.
We did not immediately conclude exactly how the attacker got control of www-data, because the evidence we had did not prove the initial entry point yet.
It could have been the web application, a vulnerable plugin, bad sudo permissions or something else entirely.
That became the next part of the investigation.
But one thing was already clear:
system-update.service was not a Linux update service.
Someone had simply given it a boring name and hoped nobody would check what it actually executed.
And to be fair, if we had only looked at the service name, we probably would have passed it.
That is why when I see an unfamiliar Linux service now, I don’t stop at systemctl status.
I want to see the unit file.
I want to see the executable.
I want to know who created it and what that process is talking to.
Names can lie.
The path usually tells you more.