Status Update

[Red Squirrel]

Just thought I would make an official status update on the state of the revamp and such.

As you may or may not know, the biggest hurdle has been to get the alternate data storage system to work well. While the MySQL was a success to a certain extent, there was a few performance issues, and other flaws with the way it worked, and I have decided to go with a totally custom storage engine, which is now nearing completion. This engine is better optimized for the type of data we are dealing with.

Once the data storage portion is finalized, tested for a while on TC1, and ready for production, it will be setup on live for some more testing before it is officially deployed.

Once the storage system is ready, it will then be released to live. This update will also include any other changes done to the test environment since the last shard update. And yes, this includes compatibility for the packet changes done by EA since the last update.

There will also be a few other changes such as the addition of the 7th character slot.

From this point on, we will try to do our best to concentrate on actual content. Unfortunately lot of the ML content will require lot of system revamps to code it properly. The property and craft system, for example, is a real mess and may need to be redone.

In the meantime we may also just add more events or other stuff to make sure there is always something to do. I also plan to add a dev environment so TC1 can be kept solely to mirror live. Anything that goes to TC1 is destined to go to live in a short period of time, as opposed to what I did with the SQL. Doing it this way will allow larger more complex updates to not go to TC1, so in the event a small change has to go to live, we can always do it directly to TC1 and release it right away (such as patching). So hopefully once we get back on our feet, we will be able to fully get up to date with OSI patching.

I am really hoping to have an update ready by mid July.

Archived topic from AOV, old topic ID:5141, old post ID:33137

[Red Squirrel]

I ran into a small roadblock in the file I/O procedure, was a very silly mistake, and I've overcome it. Things are moving rather smoothly.

The file write is working properly. For example, these are two items of serial 1 and serial 2 written to container 1:

00000000 00 00 00 01 00 00 00 0c 74 65 73 74 00 00 00 02 |........test....|
00000010 00 00 00 0d 74 65 73 74 32 |....test2|
00000019

There is an index for each container, in this case:

00000000 00 00 00 01 00 00 00 00 00 00 00 0c 00 00 00 02 |................|
00000010 00 00 00 0c 00 00 00 0d |........|
00000018

Using seekp / seekg and similar fstream functions I am able to move very quickly through a file by jumping right to the location needed.

I still need to code the snapshot feature and test, then I can modify RunUO's code to use this system instead of MySQL, and use the snapshots. The snapshots will ensure atomicity in the saved data, which we did not have with MySQL. It was just real time. The pro to that was very little to no revert if the server crashes, the con was the lack of atomicity, and it caused some issues when reloading the data. The snapshot based system may still incur some reverts upon a crash or bad shutdown, but at least the data will be atomic. I do not expect reverts to be any bigger then they are now. Depending on how fast the snapshots can be done, I think they can be done like every 5 minutes.

The beauty with this system is that it's fully custom coded so I can always make it better in the future.

Archived topic from AOV, old topic ID:5141, old post ID:33166

[Red Squirrel]

I have started implementing this system on the RunUO side of things, along with some major cleanup of the storage code to make non volatile object additions much easier in the future.

I don't want to promise any specific time this will be ready, as I tend to be busy with IRL stuff, but I really hope to have something ready soon if things go well. Some of the restructuring may prove harder then I hope, but we'll see as I tackle through.

Archived topic from AOV, old topic ID:5141, old post ID:33330

[Red Squirrel]

The RunUO implementation is really starting to come together. I did way more progress today then I expected and I still have 3 days of coding ahead of me. (I took today off)

The load procedure is done (but cannot be tested until save procedure is done) and save procedure logic is mostly done. I just need to add the code to make it all work, and I should be able to test a file -> SDE conversion, then SDE load. SDE = Serial Database Engine, my custom DBMS.

Still lot of smaller features to add like administrative commands and what not, but that is all minor stuff.

You can expect this to go live on TC1 very shortly if no roadblocks are encountered. This should be faster then MySQL as the custom DBMS has been coded to handle the type of data RunUO saves. MySQL was not being used properly as the data was not really normalized.

Archived topic from AOV, old topic ID:5141, old post ID:33469

[Red Squirrel]

Ran into a performance issue with the serialization portion, but I have a few options to optimize this further. Will play with it tomorrow. I also have Monday off. (Labour day)

I might just move on for now to at least get the system working, then optimize after. It's not super bad, but I could see it having trouble during champ spawns and stuff. Can only save 500 objects at once to memory, then process them. Was hoping to be able to do more like 5000 at a time, but it lags. Of course, this may work out better on the dual xeon box too.

Archived topic from AOV, old topic ID:5141, old post ID:33483

[Red Squirrel]

Got some progress done over the Thanksgiving long weekend. I managed to get the tcp link to work with the DB server, and wrote a class to be able to manipulate data at the byte level, because C# uses UTF16 by default on strings which makes things like Data IO very hard. My class solves this issue.

Right now it tries to load off an empty store and crashes, need to fix that so it can still work if the store is empty. I'm partially at the part where it can start to write data, but just to say I finished these in like the past half hour, so lot of testing and fixing needs to be done.

But we are getting closer.

Archived topic from AOV, old topic ID:5141, old post ID:33594

[Anonymous]

yeah yeah man I just got a landline ADSL connection for this yo`

Archived topic from AOV, old topic ID:5141, old post ID:33597

[Red Squirrel]

Another good development day today. Going to work for maybe another hour or so then call it a day. Church tomorrow.

I have it where it is writing data to the engine. It's not that fast with all the debug code but I think once I remove the debugs and also do more optimizations, it should work out nicely.

The load part is kinda messed atm though, but getting there.

Overall, it's not quite working yet, but I don't forsee any major roadblocks, just need to further debug and fix stuff.

Once it does work, I need to add more error checking code such as if the tcp connection gets broken and what not. Some other optimizations to do at that point too such as change the way the data is encoded. With SQL it was comma delimited numerical values that determined the ascii value of the chars. Takes up lot of space, but was required to ensure 100% compliance and no risk of SQL injections. Don't need to worry about this with the new engine.

Once this is all completed and running normally, I want to beta test it for a good month or so before going live. I just don't want to rush this end turns out there's a huge problem and then the shard has to be brought down, or reverted to some very far past date. But meanwhile, we'll be working on content, so this means when we do update, there will be some new content, possibly.

I really want to get the new boat system working, and we'll probably be forced to make it work for the new client to work, anyway. They did lot of changes to tiledata related stuff, and packets.

Archived topic from AOV, old topic ID:5141, old post ID:33605

[Red Squirrel]

Ran into a roadblock but I have a most likely fix for it. I can take the shortcut out, or do it properly, I'll do it properly. Should not take too much longer. A day or so.

I was planing to do this part anyhow... just was hoping not to do it right away. This will also help make things a little faster.

Archived topic from AOV, old topic ID:5141, old post ID:33606

[Anonymous]

your a champoion and if i could help I would

Archived topic from AOV, old topic ID:5141, old post ID:33607

[ggkthx]

Get a room.

Archived topic from AOV, old topic ID:5141, old post ID:33608

[Red Squirrel]

I stayed up till midnight last night debugging a memory leak but think I found the issue.

Today I worked on someone's PC so that put a delay. I've had the PC for a few days and was starting to get bothered so best to get that out of the way. I hate having to do this as it just wastes my time. I'll probably sit on it tomorrow while I continue working. If I give it back to the user right away then I'll be busy on the phone all night when they call me every 5 minutes.

Once I fix the memory leak and other issues in my bitstream class I will make the bytestream one (don't need bit level for network i/o) and then implement it.

Once I implement it, the new storage system should be functional and I will only be left with testing, and adding more error checking and some other administrative features. I also need to do some tweaking to speed things up, but going to the bytestream class not to mention having switched to a linked list, should speed things up greatly.

So things are moving, just lot of delays such as PCs I have to fix. Got two of them in the past few weeks. I wish I lived out of town sometimes...

Archived topic from AOV, old topic ID:5141, old post ID:33658

[Red Squirrel]

I'm free of PCs to fix, so been working on this a lot. I have two PCs coming in some undetermined time so that will be a big delay there...

Meanwhile taking advantage of the time I do have right now. The bitstream class is taking longer to fix then I thought but that's just because I've been getting lot of IRL delays. I'm making it way more efficient then it was before which I had to do before release, anyway, so it's not really lost time.

Archived topic from AOV, old topic ID:5141, old post ID:33666

[Red Squirrel]

Ok BitStream class is pretty much done, so I can move back to where I left off a few weeks ago,when I said days, I meant days of actual work.

Now I'm running the first test with the new code. I HOPE the crash issue was indeed what I thought it was (reaching index limit of a deque... converted my class to use a linked list which has no indexing to worry about internally).

I will let it run for an hour, if all goes well then it's a good sign. I will leave it overnight and go from there.

I still need to write byte based class similar to the BitStream one, but I'll do that later. Everything is coded to work with BitStreams, but they are not as efficient given I only need to work at the byte level. Anyway, technical stuff...

I'll update tomorrow the result of the first run with the new BitStream implementation. Really, if this works, we are VERY close. There is lot of side stuff left to do, but nothing all that complex, mostly housecleaning, and more error handling. Lastly, optimizations.

Archived topic from AOV, old topic ID:5141, old post ID:33668

[Red Squirrel]

That did not go so well. Apparently the "delete" keyword does not actually release the memory until the application is done executing. This shot down my whole new bitstream class implementation as it is based on new/delete.

Will be trying malloc, if that does not work, then back to the drawing board.

Though to speed things up I will just skip to making the bytestream class which should be able to use a string or something as the back end.

Archived topic from AOV, old topic ID:5141, old post ID:33670

[d.]

That did not go so well. Apparently the "delete" keyword does not actually release the memory until the application is done executing. This shot down my whole new bitstream class implementation as it is based on new/delete.

Will be trying malloc, if that does not work, then back to the drawing board.

Though to speed things up I will just skip to making the bytestream class which should be able to use a string or something as the back end.

I have read all of the posts in the thread and am still fairly clueless about what it all means.

Archived topic from AOV, old topic ID:5141, old post ID:33671

[Red Squirrel]

Haha yeah lot of technical stuff.

But rest assured things are being worked on.

I'm also thinking there is nothing wrong with my bitstream class. I'm just using it for a situation with way too much data involved. Every byte actually takes up 8 bytes internally. The data set I'm dealing with are HUGE.

Need to work on that bytestream class which will be much simpler to work with.

Archived topic from AOV, old topic ID:5141, old post ID:33672

[d.]

Haha yeah lot of technical stuff.

But rest assured things are being worked on.

I'm also thinking there is nothing wrong with my bitstream class. I'm just using it for a situation with way too much data involved. Every byte actually takes up 8 bytes internally. The data set I'm dealing with are HUGE.

Need to work on that bytestream class which will be much simpler to work with.

Is any of this really necessary? Compared to........content updates so we have a player base? I could understand all of this if the actual server needed some form of revamp to handle the massive player base, but we don't even have anything I'd feel comfortable calling a "player base" anymore.

Archived topic from AOV, old topic ID:5141, old post ID:33673

[Death]

Is any of this really necessary? Compared to........content updates so we have a player base? I could understand all of this if the actual server needed some form of revamp to handle the massive player base, but we don't even have anything I'd feel comfortable calling a "player base" anymore.

Yes and No. Although this means very little to players in terms of actual updates, it means a lot for the server in terms of maintenance and structure. Being able to save properly and efficiently without interrupting game play is a struggle which all shards have but which AOV strives to obtain.

Content is peanuts. It does not take long to implement content systems (implementing it correctly is what takes time). The save system has always been a challenge, one that is nearing completion. This is simply because no other shard has bothered to follow through with it, and instead has focused on content (Not a bad thing of course, but if you don't have the foundation to support it, you're going to run into big problems).

The shard is fully prepared to release content updates once the new shard core is revised and released. After all, there are more coders working on that portion (right now, there is only 1 active coder). Granted, it has taken a lot of experimenting and a lot of time which was not anticipated. That time has passed, and is now approaching upon the new era of AOV.

The shard started from scratch a long time ago and has built its player base, having started from nothing. The same can be said once it rebuilds itself. As long as interest exists in Ultima and its lore, the shard can grow again. When that happens, we will make it bigger and better than before. Sometimes big changes must be done, in order to grow and prosper.

Archived topic from AOV, old topic ID:5141, old post ID:33674

[Red Squirrel]

Pretty much this.

TBH if I had known the save engine would have been this involved, I would have kept it for last. There are lot of other engines that need to be redone such as the property system but I think at this point we will try to live with what we have and put out all the content we can, before playing with any major systems.

At the end this will be all worth it when/if we do have a large playerbase, at least the engine will be able to handle it without saves that take minutes at a time.

Archived topic from AOV, old topic ID:5141, old post ID:33675