Page 1 of 1

Did we lose a day of posts?

Posted: Fri Feb 03, 2017 9:40 pm
by RC45
That was a close one :)

Anytime you come back from a catastrophic hosting location failure intact its a good day :)

Way to have the backups tested.

Re: Did we lose a day of posts?

Posted: Fri Feb 03, 2017 9:45 pm
by hybrid
Yes, we lost around a days worth of posts since we had to go to backup.
Unfortunately, we weren't so lucky with the parts database.

Not sure if we will get any data back from that.... the best laid plans...

Re: Did we lose a day of posts?

Posted: Fri Feb 03, 2017 9:48 pm
by RC45
That's a bit of a fucker.

Different backup tools on the parts database?

Re: Did we lose a day of posts?

Posted: Fri Feb 03, 2017 10:04 pm
by hybrid
I just dump the main databases to file, then send them offsite.
The parts section used a completely separate DB that I overlooked.

I think I'm going to configure (very carefully) a full system backup once a month. I previously avoided it because it's so damn big and you can get a clean server up pretty quick these days.
In this case though, it would have saved our bacon.

So I will generate a full backup and see how big it ends up.

Re: Did we lose a day of posts?

Posted: Fri Feb 03, 2017 10:25 pm
by phildu31
Hi.
If you use MySQL, you can also setup a second database server and enable replication between the two servers.
The online database will be master for replication and the offline database will be slave for replication.
By doing so, all datas and actions will be mirrored.
But you still have to do complete backups at regular time ( in case of unlucky delete commands ... )
You can PM me for more informations if desired.

Re: Did we lose a day of posts?

Posted: Sat Feb 04, 2017 1:37 am
by hybrid
Yes I have experience with MySQL replication via MASTER/SLAVE and MASTER/MASTER.
Overkill for what we need - especially when the costs come from my own pocket.
We just need to make sure we don't leave any important stuff out of the backups, which is what happened here.

Re: Did we lose a day of posts?

Posted: Sat Feb 04, 2017 6:02 am
by RC45
Do you you need us to throw in a couple bucks to license something more robust than freeware or hosting company hosted database backup tools?

Re: Did we lose a day of posts?

Posted: Sat Feb 04, 2017 6:42 am
by hybrid
Thanks for the offer, but no I will just have to be more diligent with making sure everything is covered.
The backups are going to Amazon S3 storage, which is plenty robust enough.

Considering what happened, we haven't come out of it too badly really.

Re: Did we lose a day of posts?

Posted: Sat Feb 04, 2017 5:02 pm
by RC45
Well, as long as Amazon don't have a GitLabs moment :)

Re: Did we lose a day of posts?

Posted: Sun Feb 05, 2017 2:10 am
by hybrid
That would have to happen at the same time as our systems being wiped for it to be a problem.
If both those things happened, I think I would buy a lottery ticket afterwards!