Page 1 of 1
Shard revert due to database issue
Posted: Fri Feb 02, 2018 2:48 am
by Red Squirrel
Due to a database corruption issue we had to do a revert to Sun Dec 31 18:00:08 EST 2017. Given how dead the shard has been lately it only got noticed now so had to use a pretty old backup... but given the lack of activity not like much was lost anyway.
Sorry for the inconvenience this may have caused. I will investigate the cause of the corruption.
Archived topic from AOV, old topic ID:6740, old post ID:39199
Shard revert due to database issue
Posted: Fri Feb 02, 2018 5:17 pm
by ninja2007
should of left the characters wiped
i would of loved to start over from scratch with a bare server
Archived topic from AOV, old topic ID:6740, old post ID:39201
Shard revert due to database issue
Posted: Tue Feb 13, 2018 5:29 pm
by Red Squirrel
This happened again this morning.
There's a weird issue with my network where if something is pegging disk I/O too much, stuff starts to lockup. I was trying to sync the ethereum blockchain, I think that is what was causing issues.
Had to restore a backup from this morning. (Tue Feb 13 06:00:26 EST 2018)
I stopped the blockchain sync for now to see if the problem goes away just to rule it out. If it is that, then I will simply put it on a dedicated machine instead of a VM.
Archived topic from AOV, old topic ID:6740, old post ID:39203
Shard revert due to database issue
Posted: Sat Feb 17, 2018 11:24 am
by Red Squirrel
So this seems to keep happening. I really don't know what's going on but I won't really have much time to investigate for a while but I do need to look into this further. Nothing has actually changed for a long time so I really don't know why this keeps happening.
Archived topic from AOV, old topic ID:6740, old post ID:39216
Shard revert due to database issue
Posted: Sat Feb 17, 2018 11:37 am
by Red Squirrel
It seems the network link to the DB server is randomly dropping. Going to keep shard down while I do a ping test. It seems to only happen when there is heavy load on the file server. Right now there is a backup job running and it's just killing everything. This issue has been a bane of my existence, it seems backup jobs put a ridiculous load on the server and then it causes stuff to crash and I just don't get why.
Edit: Decided to bring shard back up as I wanted to test backup anyway. I will monitor ping test while it's up.
Backup from Sat Feb 17 06:00:08 EST 2018 restored.
Archived topic from AOV, old topic ID:6740, old post ID:39217