Hoggit
patreon
Apologies
🕑 Added 2019-10-02 19:33:43 +0000 UTC
Hey guys, I have some bad news.
We experienced a technical issue earlier today while I was trying to upgrade some server components that we use, and a mistake I made about 8 months ago while configuring our postgres database made itself apparent in the worst way: We lost data stored at https://patreon.hoggitworld.com
This includes both objectives and your slots.
I've corrected the error in the database that the patreon system was using, but also moved onto a managed database with automated backups.
I'm really sorry this happened and that you need to go fill in your data again. I've learned a bunch about the system we were using and have taken steps to make sure this doesn't happen again, but I still feel bad that my mistake has caused issues for you.
NOTE: No data was stolen, it was just deleted unintentionally. Your slots as they existed before will still exist, this did not affect the mission files for GAW or PGAW in any way. The only thing that was lost was what you had entered into patreon.hoggitworld.com
**Technical Details**
If you're not technical in nature, or don't care about the circumstances of the incident, you can skip this section.
We use Kubernetes on Digital Ocean to run most of our backend services, this includes the services that provide data about our current server state (some of you consume this for your own apps), the atwar.online domain, and recently the patreon.hoggitworld.com app.
Kubernetes uses docker-like containers, and anyone with experience with docker knows that storage in the running containers is ephemeral, meaning that as soon as the container stops anything added (or edited) to its storage is lost. In order to accommodate the necessity of persistent storage (databases), both docker and kubernetes solve this problem with volume-mounts (called persistant-volumes on kubernetes).
In kubernetes' case, you create a volume and then mount it on the pods you want to have persistent storage. When doing so you get to select a path the volume will be accessible from in the pod. I had mounted the volume to `/var/lib/postgresql` as this was where the docker container said that it would use when managing data. I did this 8 months ago for a small project that I thought I needed a database for, but ended up not using it and left the database around for future projects.
The patreon app that I started working on a few months ago needed a database, and lo-and-behold, I had set up a postgres instance that was not being used. So I hooked it up to that, and you guys started getting your rewards sorted out there.
Today, I noticed we had a kubernetes upgrade available to us in our provider. This is a fairly automated process, but all the pods (containers) needed to be restarted because you can't upgrade a kubernetes node in place. You create a new one, cordon off an old one, _drain_ the old one, and then delete the old one. This means that all your pods get deleted, and scheduled onto one of the new, upgraded nodes. This is all as expected.
8 Months ago when I set up the postgres mountpoint, I had not tested its persistence capabilities by adding something, killing the pod, and bringing it back up to make sure that all was well. It turns out all was not well, and I had mounted it such that all the data being stored into the mountpoint was not actually being stored on the persistent volume. When kubernetes did the upgrade, all the database information was lost.
I've corrected the mountpoint (mounting it to `/var/lib/postgresql/data` and providing an environment variable pointing to a directory under it), so our database in the cluster should not have this happen again (I also tested it by creating a database, killing the pod, and making sure the database was there when it came back up).
Feel free to ask any questions about this if it wasn't clear.
Comments
james mendes
I worked at a company that had that happen. Luckily I had just started taking backups of the DB's because they were not before i started working there.
Kevin Luff
I've gone to that link and I guess I submitted. It's one line and it won't let me submit anymore. However, The US F/A-18C on the Stennis is still not there. "[1st VFW] Nitro" please I've been asking for over a week now.
