Possible shard revert / guilds crash issue

Red Squirrel · Post by **Red Squirrel** » Fri Oct 21, 2011 10:31 pm

It looks like there is an issue with guilds. I am investigating and there is the possibility of a revert.

I am hoping to be able to revert JUST the guilds, but I'm not sure if that will work.

I am leaving it running for now as the fix may be very minor, but if it's major I don't know what will happen.

Edit: Shard now back up using backup done on Fri Oct 21 05:23:14 EDT 2011. Currently investigating cause of crash so it can be fixed.

Archived topic from AOV, old topic ID:6157, old post ID:36794

ggkthx · Post by **ggkthx** » Fri Oct 21, 2011 10:43 pm

Guilds recreate just fine. Shrug.
I think it's awesome personally.

Archived topic from AOV, old topic ID:6157, old post ID:36795

Plas · Post by **Plas** » Fri Oct 21, 2011 10:44 pm

Red Squirrel wrote:It looks like there is an issue with guilds. I am investigating and there is the possibility of a revert.

I am hoping to be able to revert JUST the guilds, but I'm not sure if that will work.

I am leaving it running for now as the fix may be very minor, but if it's major I don't know what will happen.

If you need help testing it out let me know, I'm not doing much these days and I've actually been a professional QA Game Tester before.

Archived topic from AOV, old topic ID:6157, old post ID:36797

Red Squirrel · Post by **Red Squirrel** » Fri Oct 21, 2011 11:14 pm

Ok bad news, very bad news. I have to do a revert of the complete shard to the latest backup that ran early this morning.

I thought I fixed the backup script to not embed the database name into it, but for whatever reason, the backup had the name in it regardless. It's too big of a file to open in vim on the server so I could not actually check.

So when I did a restore to a temp DB so I can extract the data, it actually restored to the live DB while the shard was running. This would cause all sorts of phantom issues even if I forced everything on the shard to save, best not to take a chance with this, so I have to do a full restore.

The shard is down now and will be back up shortly. If my fix worked, the guild button should also be disabled. I'm getting a brain fart here and can't recall if there are other ways to access guilds, but please avoid using anything to do with guilds for the time being.

Any testing on TC1 to reproduce this would be much appreciated. The more info I have, the faster I can fix it and re-enable guilds again. Meanwhile I will try to reproduce the issue on dev.

Edit: Shard now back up using backup done on Fri Oct 21 05:23:14 EDT 2011

Archived topic from AOV, old topic ID:6157, old post ID:36802

Red Squirrel · Post by **Red Squirrel** » Fri Oct 21, 2011 11:56 pm

I think I may have found the cause. I need to do some more testing but things are looking good.

Archived topic from AOV, old topic ID:6157, old post ID:36803

Red Squirrel · Post by **Red Squirrel** » Sat Oct 22, 2011 1:12 am

I think I have the fix ready, but I don't want to make a change near bed time like this, so I will wait till tomorrow. The shard will not revert again, so you can resume as normal.

I checked the backup script and I guess I had missed the live one, so next time I try to restore a database like this, it should not be an issue. I will run a test during next downtime.

The fix has been applied to TC1, so please feel free to hammer it. Basically, anything to do with guild changes such as adding/removing players, guilds from alliances, etc...

Archived topic from AOV, old topic ID:6157, old post ID:36804

Plas · Post by **Plas** » Sat Oct 22, 2011 2:41 am

I would love to but I cant connect to TC1.

I select it from the server list, and then it just hangs on Connecting...

Is it down?

Archived topic from AOV, old topic ID:6157, old post ID:36805

ggkthx · Post by **ggkthx** » Sat Oct 22, 2011 7:48 am

I have same problem with TC1, though I know some people can connect to it.

Archived topic from AOV, old topic ID:6157, old post ID:36806

Death · Post by **Death** » Sat Oct 22, 2011 9:04 am

TC1 appears to be real slow right now. As it's hosted locally on red's machine that tends to happen from time to time when the system is using more resources.

Archived topic from AOV, old topic ID:6157, old post ID:36808

Blackadder · Post by **Blackadder** » Sat Oct 22, 2011 10:59 am

Well at least that explains the missing 200k and an excellent 98% jedi killer tally thats missing...

Hope you can sort it out though

Archived topic from AOV, old topic ID:6157, old post ID:36810

ggkthx · Post by **ggkthx** » Sat Oct 22, 2011 11:44 am

If only there were Jedi to kill.

Archived topic from AOV, old topic ID:6157, old post ID:36811

Blackadder · Post by **Blackadder** » Sat Oct 22, 2011 12:18 pm

I thought those things with the deathblade were jedi's?
by valor gate...
I dont stay alive long enough to see what they are lol

Archived topic from AOV, old topic ID:6157, old post ID:36812

Red Squirrel · Post by **Red Squirrel** » Sat Oct 22, 2011 1:14 pm

Wait, so there's other items that got lost too? We may have a much bigger issue then... Did they return with the revert?

Anyway, I will be releasing the guild fix to live shortly.

Archived topic from AOV, old topic ID:6157, old post ID:36813

ggkthx · Post by **ggkthx** » Sat Oct 22, 2011 1:25 pm

Oh yup, my bad, when they said "Star wars monsters and bosses have been removed" I thought that meant what it said instead of only the Star Wars content that dropped something useful being removed.

Archived topic from AOV, old topic ID:6157, old post ID:36815

ggkthx · Post by **ggkthx** » Sat Oct 22, 2011 1:26 pm

Red Squirrel wrote:Wait, so there's other items that got lost too? We may have a much bigger issue then... Did they return with the revert?

Anyway, I will be releasing the guild fix to live shortly.

Pretty sure it was the revert that lost the items.

Archived topic from AOV, old topic ID:6157, old post ID:36816

Blackadder · Post by **Blackadder** » Sat Oct 22, 2011 1:38 pm

Red Squirrel wrote:Wait, so there's other items that got lost too? We may have a much bigger issue then... Did they return with the revert?

Anyway, I will be releasing the guild fix to live shortly.

Its no biggie, i presume by revert you went to an earlier save, probably before i found the items and collected the gold....
Dont stress mate its fine..
Happened a lot on osi shards in the early days.....

Archived topic from AOV, old topic ID:6157, old post ID:36817

Red Squirrel · Post by **Red Squirrel** » Sat Oct 22, 2011 2:52 pm

Oh ok, yeah. I thought the items got lost before the revert, that scared me a little.

Now everything should be ok, I'll be monitoring the shard for a bit.

Archived topic from AOV, old topic ID:6157, old post ID:36818