VoxiHost - Infrastructure failure – Incident details

This incident has been resolved.

We are currently finalizing our new Customer Panel. While the setup is taking a bit longer than expected, we remain partly operational accepting orders and delivering VPS services.

I expect to launch the new panel by the end of this week!

Best regards, Daniel Marszałkowski

CEO VoxiHost

I have deployed our front-page back and working! (New Design)
I'm still working on bringing back the customer panel!

Best regards, Daniel Marszałkowski
CEO VoxiHost

Hey, just wanted to drop quick update on the situation! I'm almost done with implementing new front-page, on the backend stuff i already setuped multiple backup solutions that preform global node backup every 1 day to prevent such situations...

Best regards, Daniel Marszałkowski
CEO VoxiHost

Emails were delivered from recovery@voxihost.pl, if you don't see one check your SPAM!

Update!

Working on sending emails to all customers. Please check your inboxes within 2-5h.

Update on the current situation

I have analyzed all VM disk dumps and, as mentioned before, I was only partially able to recover data from them. I am currently on a train to spend New Year's Eve with someone close to recharge. Before leaving, I managed to restore our mailing system, which means you can now contact us again at support@voxihost.pl. I have also created a dedicated email for recovery and compensation: recovery@voxihost.pl. From this address, users will receive more information regarding the next steps and their data. If I find the time, I will start sending informational emails within the next 2-3 hours, but I cannot promise that yet. I will do my best to be back at work as soon as possible, likely late on January 1st or on January 2nd. I sincerely apologize again for this entire situation. I will do everything in my power to bring everything back to life and compensate everyone affected.

Best regards, Daniel M. VoxiHost

Dear Customers, I am writing this with a heavy heart and complete honesty. I have been fighting this critical failure for the past 10 hours, but I have actually been on my feet for over 24 hours now. What started as a simple disk alert has ended in a broken Proxmox backend, affecting every single VPS and our own internal systems. I was deeply optimistic at first because the file sizes looked correct, but I have now discovered that many of them are severely corrupted.

Our internal systems, including the billing panel and database, have also been hit. Since our internal backups run on a 7-day cycle and the next one was due exactly at the end of today, some recent account changes or new registrations from the last few days are currently missing. However, I want to reassure you that your funds are safe. Once everything is stable, I will manually verify all transactions through our payment providers to ensure every credit and service is correctly restored.

Regarding your data, I am doing everything possible to recover what can be saved. For those whose data is unfortunately unrecoverable on our end, I will be sending an email shortly with a download link to your raw VPS disk image so you can perform your own analysis and potential recovery. For those whose servers were partially saved, I will provide a fresh, functional disk along with a compressed archive of your salvaged files.

As a one-man army building this hosting dream, this situation is devastating to me. I have reached a point of total exhaustion and I need to take a short break to sleep at least 2 hours and spend some time with my family for New Year’s Day. I need to recharge my mind to avoid making any critical mistakes during the rest of this recovery process. I will be back at work in full force very soon. I cannot express how sorry I am for this catastrophe and I thank you for your patience and for staying with me during this dark hour.

Best regards, Daniel M. VoxiHost

I'm sorry... i will try my best to find a way for compensation once this is all done.

After comprehensive diagnostics and data recovery operations, we have determined that the host system corruption is unrecoverable through repair. We are proceeding with a clean Proxmox reinstallation.

• All customer data has been successfully backed up to secure offsite storage

• Preliminary analysis indicates no customer VM data corruption

• RAID arrays verified healthy - no hardware failures detected

Root Cause: The cause of the corruption is still under investigation. Logs and diagnostics have been preserved for detailed analysis after service restoration.

Our Priority: Restoring all customer services as quickly as possible. Full post-mortem analysis will follow once services are back online.

ETA: Services resuming within 2-3 hours.

We can confirm that all customers data was safely extracted. This is yet to be confirmed later... no data on customers servers should have been affected as our alert system indicates that most VPS servers weren't touched by any error in terms of drive, and those that were went into read-only lockdown!

I'm still looking into fixing this as fast possible.

Daniel M.

CEO VoxiHost

We are currently recovering customer files, all data is being transferred to safe-drive. After that we will continue investigating.

We are currently running disk check, to verify the issue.