Database corruption [resolved]

Red Squirrel · Post by **Red Squirrel** » Tue Nov 06, 2007 8:11 am

Looks like the whole database corrupted itself, I have no clue how but I'm quite pissed off at the instability of that database system. Shows that converting to SQL is a much more important task and I should stop putting it off

I'm at work now so I can't do crap all. I'll see if I can figure it out on my lunch hour (which is more like 15 minutes by the time I get home and eat then go back to work) but doubt it, so it will have to wait till like 7:00pm tonight. :/

Sorry about this.

Archived topic from AOV, old topic ID:1610, old post ID:10156

Red Squirrel · Post by **Red Squirrel** » Tue Nov 06, 2007 8:37 am

Ok I just did a blind database restore. I'm at work so I can't login to actually see how recent this backup is, someone let me know how recent it looks. If its super old or stuff is screwed up such as houses being gone, just let me know since I rather not run the shard if I'm only going to wipe everything after (to restore another backup, that is)

Archived topic from AOV, old topic ID:1610, old post ID:10159

dprantl · Post by **dprantl** » Tue Nov 06, 2007 8:59 am

Ok, so I just logged in and was 7/8th's of the way to killing dreadhorn. So I went ahead and killed him again and got out of there, then when the server saved it corrupted the DB AGAIN. And....

...

I got Crimson Cincture this time

I can't believe that, I'm going to shoot myself

And I need to go to work

Archived topic from AOV, old topic ID:1610, old post ID:10161

Red Squirrel · Post by **Red Squirrel** » Tue Nov 06, 2007 9:06 am

Wow is this ever fucked... I'll keep that backup knowing that its recent but definatly something is totally fucked, this will take a long time to fix since its impossible to pinpoint what is causing it.

If I can manage to reproduce this on TC1 it might be easier though. Was it the first world save after DreadHorn's death that did it?

Archived topic from AOV, old topic ID:1610, old post ID:10163

dprantl · Post by **dprantl** » Tue Nov 06, 2007 9:50 am

Red Squirrel wrote:Was it the first world save after DreadHorn's death that did it?

Yes.

Archived topic from AOV, old topic ID:1610, old post ID:10168

Shadow · Post by **Shadow** » Tue Nov 06, 2007 10:48 am

The world save was approximately, idk, 5 mins after the Dreadhorn drop that Altar did. Maybe the Dreadhorn spawn overloaded the server somehow? How often is Dreadhorn killed?

Archived topic from AOV, old topic ID:1610, old post ID:10169

dprantl · Post by **dprantl** » Tue Nov 06, 2007 11:42 am

There is no doubt that the death of dreadhorn is causing this. I bet if you restored the DB again and I just waited around with dreadhorn still alive for a world save, it would not corrupt. But I'm at work right now so I can't test this out.

Archived topic from AOV, old topic ID:1610, old post ID:10170

Red Squirrel · Post by **Red Squirrel** » Tue Nov 06, 2007 12:43 pm

I can already confirm since there was a few saves before it crashed again.

Nice thing about help desk is if I have a RPD connection opened with a dos console, it looks like work.

I just tried something, we'll see if it works.

Everyone is free to go play again, but avoid taking out pets as if it crashes again you might loose them.

I have to go back to work now.

Archived topic from AOV, old topic ID:1610, old post ID:10171

dprantl · Post by **dprantl** » Tue Nov 06, 2007 1:14 pm

Although this will suck because by the time I get back from work, my dreadhorn will have timed out and I will be kicked out of there

I will also not be able to test to see if killing him will reproduce the problem.

Archived topic from AOV, old topic ID:1610, old post ID:10172

Red Squirrel · Post by **Red Squirrel** » Tue Nov 06, 2007 1:50 pm

I'll arrange for you to get another key given the circumstances. I just hope my 5 minute fix worked so I can actually concentrate on my work without this stress.

Archived topic from AOV, old topic ID:1610, old post ID:10173

Dumples · Post by **Dumples** » Tue Nov 06, 2007 2:22 pm

Red Squirrel wrote:I'll arrange for you to get another key given the circumstances. I just hope my 5 minute fix worked so I can actually concentrate on my work without this stress.

Don't stress. Your real life job is more important.

Archived topic from AOV, old topic ID:1610, old post ID:10174

Death · Post by **Death** » Tue Nov 06, 2007 3:30 pm

Shadow wrote:The world save was approximately, idk, 5 mins after the Dreadhorn drop that Altar did. Maybe the Dreadhorn spawn overloaded the server somehow? How often is Dreadhorn killed?

Dreadhorn has been tested at least 7 times without this problem occuring. Of course, there were a few recent script modifications but it's just a cliloc which wouldn't do much. Would somebody care to explain who they were using to kill dreadhorn, and what exactly happened?

Archived topic from AOV, old topic ID:1610, old post ID:10175

dprantl · Post by **dprantl** » Tue Nov 06, 2007 3:47 pm

Seer Death wrote:
Shadow wrote:The world save was approximately, idk, 5 mins after the Dreadhorn drop that Altar did. Maybe the Dreadhorn spawn overloaded the server somehow? How often is Dreadhorn killed?
Dreadhorn has been tested at least 7 times without this problem occuring. Of course, there were a few recent script modifications but it's just a cliloc which wouldn't do much. Would somebody care to explain who they were using to kill dreadhorn, and what exactly happened?

I was killing dreadhorn. I used the same character as when we did it together. I did not see anything out of the ordinary until the save right after dreadhorn's death. I was already out of there and in my house. Then Red reverted to a backup in which I was almost done killing dreadhorn, so I killed him again and left again, and as soon as a save happened, the corruption happened again.

Archived topic from AOV, old topic ID:1610, old post ID:10178

dprantl · Post by **dprantl** » Tue Nov 06, 2007 3:51 pm

Actually Death, I can do it again right now if you give me the key (since I was kicked out while I was at work) with you watching if you want.

Archived topic from AOV, old topic ID:1610, old post ID:10179

Death · Post by **Death** » Tue Nov 06, 2007 3:52 pm

dprantl wrote:
Seer Death wrote:
Shadow wrote:The world save was approximately, idk, 5 mins after the Dreadhorn drop that Altar did. Maybe the Dreadhorn spawn overloaded the server somehow? How often is Dreadhorn killed?
Dreadhorn has been tested at least 7 times without this problem occuring. Of course, there were a few recent script modifications but it's just a cliloc which wouldn't do much. Would somebody care to explain who they were using to kill dreadhorn, and what exactly happened?
I was killing dreadhorn. I used the same character as when we did it together. I did not see anything out of the ordinary until the save right after dreadhorn's death. I was already out of there and in my house. Then Red reverted to a backup in which I was almost done killing dreadhorn, so I killed him again and left again, and as soon as a save happened, the corruption happened again.

I have a hunch.....did you teleport out of the room after the death or wait for the room to eject you? Also, did you disband your party before the room ejected you?

Archived topic from AOV, old topic ID:1610, old post ID:10180

dprantl · Post by **dprantl** » Tue Nov 06, 2007 3:53 pm

Seer Death wrote:
dprantl wrote:
Seer Death wrote:
Dreadhorn has been tested at least 7 times without this problem occuring. Of course, there were a few recent script modifications but it's just a cliloc which wouldn't do much. Would somebody care to explain who they were using to kill dreadhorn, and what exactly happened?
I was killing dreadhorn. I used the same character as when we did it together. I did not see anything out of the ordinary until the save right after dreadhorn's death. I was already out of there and in my house. Then Red reverted to a backup in which I was almost done killing dreadhorn, so I killed him again and left again, and as soon as a save happened, the corruption happened again.
I have a hunch.....did you teleport out of the room after the death or wait for the room to eject you? Also, did you disband your party before the room ejected you?

I went over to the island with the healer and used the gate to go to gate library. I did not wait. There was no party.

Archived topic from AOV, old topic ID:1610, old post ID:10181

Death · Post by **Death** » Tue Nov 06, 2007 3:54 pm

hmmm okay. You were the only person in there then?

Archived topic from AOV, old topic ID:1610, old post ID:10182

dprantl · Post by **dprantl** » Tue Nov 06, 2007 3:54 pm

Seer Death wrote:hmmm okay. You were the only person in there then?

Yes. In fact, I was the only person online at the time, apart from Counselor Shadow.

Archived topic from AOV, old topic ID:1610, old post ID:10183

Death · Post by **Death** » Tue Nov 06, 2007 3:55 pm

dprantl wrote:
Seer Death wrote:hmmm okay. You were the only person in there then?
Yes. In fact, I was the only person online at the time, apart from Counselor Shadow.

Hmmm k that narrows things down a bit then.

Archived topic from AOV, old topic ID:1610, old post ID:10184

Red Squirrel · Post by **Red Squirrel** » Tue Nov 06, 2007 4:04 pm

I think the cause was recipes, it generates an instance of the artifact then deletes it on the fly in order to get the hue, itemid and properties for display purposes. I had it set to delete the item on save, which turned out to corrupt the DB.

At least t hats my theory, I'm not 100% sure it was that.

Archived topic from AOV, old topic ID:1610, old post ID:10185

dprantl · Post by **dprantl** » Tue Nov 06, 2007 4:14 pm

Well I can tell you both times it crashed I got recipes...

Archived topic from AOV, old topic ID:1610, old post ID:10187

Death · Post by **Death** » Tue Nov 06, 2007 4:17 pm

Red Squirrel wrote:I think the cause was recipes, it generates an instance of the artifact then deletes it on the fly in order to get the hue, itemid and properties for display purposes. I had it set to delete the item on save, which turned out to corrupt the DB.

At least t hats my theory, I'm not 100% sure it was that.

Well seeing as we tested DH numerous times before and I tried to crash that part locally without success, I'm beginning to think that it is the new changes to the recipes as well. I'm trying to produce that locally right now.

Archived topic from AOV, old topic ID:1610, old post ID:10188

dprantl · Post by **dprantl** » Tue Nov 06, 2007 4:18 pm

Seer Death wrote:
Red Squirrel wrote:I think the cause was recipes, it generates an instance of the artifact then deletes it on the fly in order to get the hue, itemid and properties for display purposes. I had it set to delete the item on save, which turned out to corrupt the DB.

At least t hats my theory, I'm not 100% sure it was that.
Well seeing as we tested DH numerous times before and I tried to crash that part locally without success, I'm beginning to think that it is the new changes to the recipes as well. I'm trying to produce that locally right now.

Me too, I just did dreadhorn on TC1 and have manually added a recipe into my pack. Let's see if that does it in during the save, unless you put the fix into TC1 Red?

Archived topic from AOV, old topic ID:1610, old post ID:10189

Red Squirrel · Post by **Red Squirrel** » Tue Nov 06, 2007 4:20 pm

Yeah if fix is on live its also on TC1. I can't recall if I rebooted it though, think I did.

Archived topic from AOV, old topic ID:1610, old post ID:10190

Red Squirrel · Post by **Red Squirrel** » Tue Nov 06, 2007 6:13 pm

I'm home now, so trying to reproduce it on TC1 to confirm that my fix is actually what fixed it.

I was SO not productive at work today because of this, just can't think straight when I know a critical server is down and I can't do anything about it. Then again its been so dead lately so I doubt it affected too many people, though it does make us look bad to have such long downtime. I kinda blew the 5 9's on this one. Then again, OSI has had lot of issues too so we're not alone.

Archived topic from AOV, old topic ID:1610, old post ID:10194