Talk about a new year starting with a bang. I guess there’s a first time for everything and this one certainly wasn’t pleasant. Honestly, I’m still grappling with my emotions as I write this. There’s paranoia, embarrassment, confusion. Not a very healthy mix of feelings for the bleakest month of the year—and also the reason it took me so long to finish this post. The shame just crept back in whenever I worked on it, even when I thought I was finally over it.
So what happened?
One of my hosts got hacked.
First things first, the breached system was a remote host from which I serve Hydrt and Quadrants, among some other static sites. Since I do not collect any user data, no user data was affected, which is good.
Other than those static files, I had also an instance of Umami running there for some privacy-respecting analytics. This instance only collected anonymous data such as installation and usage events without metadata and page views containing information such as system language, OS, browser, etc. but no IP addresses or any other form of personally identifiable information.
The instance only had two users, me and a friend of mine, whom I have already informed about the breach as soon as I realised it had happened. I also think this instance is the reason the breach could happen in the first place. More on that in a bit.
A Rough Timeline of Events
On January 3rd I noticed that a new version of Quadrants couldn’t be deployed because of an issue with connecting to the host and I couldn’t log into the server via SSH either. Accessing the Server via HTTP worked, my sites loaded normally, but he Umami instance threw a 502 error, indicating that it might not be running.
I reached out to the hosting provider to inquire about any ongoing incidents on their side and on January 6th, I received confirmation from the hosting provider that all systems were running normal.
In the meantime, I managed to register a new SSH key on the system and log in to discover that something was definitely off. My authorized_keys file had been cleared, the .bash_history and some configuration files had been deleted, and an unknown public SSH key was present in a new authorized_keys_disabled file.
Examining server-side backups, I was able to learn that the attack must’ve happened sometime between the 21st and 28th of December, while I was away from my systems for Christmas. I reached back out to support with my findings, inquiring if it could’ve been a configuration change or update gone wrong on their end, or whether I should assume the worst.
That same day, I received a reply that a successful attack was likely, and I should consider the system compromised. They confirmed that, oddly, the system had not been abused for crypto-mining, sending spam emails or hosting phishing sites and suggested it might’ve been a targeted attack to sabotage me as a person by a disgruntled co-worker. I felt so confused that besides the attempted lock-out, the deleted bash config files and the disabling of the Umami instance, nothing seemed amiss.
I immediately changed my password for the hosting provider and removed the unknown SSH key (even though the file was marked with _disabled) and asked support whether they would be able to provide me with some more insights on the attack. My user on the system had limited access, so I hoped they could provide some more info from system logs, etc. I also asked for some additional guidance on how to best proceed. In hindsight, I should probably have immediately nuked the system, but I didn’t, thinking that would perhaps destroy some evidence to better understand what had happened.
On January 8th, I deleted the system and set it up from scratch, despite not hearing from support. I did not install Umami again and decided that that host would be for static sites and apps only. On January 9th, Support confirmed that the Umami instance seems to have been running a vulnerable version of React.
The Likely Culprit
In the end of November and beginning of December 2025, a critical React vulnerability dubbed “react2shell” as discovered and disclosed. As the name implies, it allowed attackers to remotely execute code on the server. Very, very bad.
Since React and especially Next.js, which bundles its own version of React, are so widely used, malicious parties started exploiting the vulnerability almost immediately after it was disclosed. I typically don’t touch anything to do with React and Next.js if I can avoid it. However, as I outlined in my post on analytics solutions, Umami is what I begrudgingly settled upon after not finding a suitable alternative.
Umami is built using Next.js. I thought I had upgraded to a new version of Umami which included the fix for the critical vulnerability, but apparently I did not. Perhaps I got things confused with other Next.js deployments I fixed at work.
Whatever the reason, it is my own fault this happened and that makes it so much worse. It’s a personal failure, and just because I seem to have got away with a black eye and a bruised ego, doesn’t mean I take this lightly.
Nuke and Pave
As mentioned in the timeline above, I completely reset the compromised system using the suggested method provided by my hosting provider. I also decided to no longer run server-side applications under the same user as my static files. I chose convenience over security in doing so and this was the price for it.
In a way, I’m glad most of my projects don’t need any backend beyond static file hosting, as that significantly limits attack surface and maintenance pressure. I’m just a single person with a busy life, but as I’ve learned, running server-side software that is publicly accessible requires extremely timely updates. So any server-side tool I build increases my responsibilities and lessens the time I have to build fun new things.
Feeling Lost and Demotivated
As much as developing and hosting software feels great, this whole thing has put me in my place, and shown me how little I still know and understand about all this. I’ve been building software for more than a decade, and yet I feel like I’ve barely scratched the surface of what there is to learn.
Nobody can know everything, but trust that I am doing my best with what I do know—and I want to keep learning and exploring, despite this setback which made me feel quite lost, and to be honest, cost me a fair bit of motivation. I am tempted to stick to my client-only roots, but I am well aware that that’s not really an option if I want to keep growing. As much as can be done with just the browser these days, some things just aren’t possible without at least a simple backend running on a server.
If you’ve made it this far into this post, and you feel like you have some additional pointers for me on how to deal with and handle situations like this, feel free to reach out to me via Mastodon. I’d love to hear from you!