Project Kibou: Server Development

[Spirited]

Introduction:
Dear supporters, colleagues, and friends,

First, I would like to thank you for your support over the years with my self-studies. I have been very fortunate to have such kind and caring readers interested in my work. Posting here just reminds me of the enthusiasm we both have shared for software development since the birth of my first project, Project Kibou, which brings us to the topic at hand.

If you're new to this community, let me introduce you to the newly refined idea: Kibou is a non-profit, self-study intended for demonstrations in interprocess communications at Saddleback Community College. The goal of the project is to create an expandable massive multiplayer online account, web, and game server. It achieves this by abstracting classes in base libraries, and then building upon that infrastructure through inheritance in the server projects, thereby increasing the modularity of each component of the project and creating a highly scalable server design.

A main objective for Project Kibou is to securely connect players to the game server and maintain a stable connection throughout their gameplay. Kibou uses IO completion ports along with threaded worker queues to ensure server reliability and efficiency. Red-black trees, skip lists, and other data structures are used to process network data and collections quickly and keep the response time of the server low. To protect player connections, the server implements algorithms such as the Carlisle Adams Stafford Tavares Cipher, Diffie-Hellman Key Exchange, and other private algorithms used for secure transfers between the servers and the client. These systems are safeguarded with brute-force attack prevention algorithms on the socket acceptance level (prior to connections being accepted by the server controlled by socket level callbacks). All clients that match the attack behavior are terminated and added to a temporary connection block list, keeping player passwords and server transfers secure.

Another objective of Project Kibou is to efficiently manage player and server data. Frequently accessed data, such as character information, is stored in a MySQL database and accessed using pooled connections. Static server resources only accessed upon server initialization, such as map access data, are stored as binary files on the host. Dynamic server resources, such as non-playable character dialogs, are stored as scripts and interpreted by the server’s scripting engine.

The final objective of Project Kibou is community. As a whole, the project operates as a private game service aimed at emulating a game world for player interactions. With permission from TQ Digital Entertainment and NetDragon Websoft, the project has rights to modify the Conquer Online game client binaries and game resources, used to connect to the account and game servers in the project. This agreement is only valid under terms of academic use. More information about the game storyline will come at a later date.

Alright, so a few months ago, I had to experience the passing of my grandmother (my only guardian), and though I didn't believe it would be relevant to my work, it was definitely affected. The very little enthusiasm I had remaining in C# has dispersed. C# is no longer a challenge or creative outlet for me, and I have no interest in continuing programming in C# as a self-study. I am exhausted after years of constant programming in the language, and I feel it is time I move on to bigger and greater things. The purpose of this project was met - I now have access to local internships and scholarships; therefore, my C# attempt at Project Kibou was closed.

Now, you might be asking "what's next". I can't say what yet, but magnificent events are falling into place, and it's time that I continue my work on a project that will utilize my new skill set. After talking with certain recruiters of my dream job, it has been decided that my best course of action is to peruse my interests: work on a project I'm passionate with that involves the type of work expected from me, in C++. After the past few months, I have learned more than I have learned in the past year and a half. I have drastically improved my skill set, and improved my server design by high magnitudes. So, in dedication to my grandmother and the efforts we put into my future as a computer scientist, and after spending months finalizing a new server architecture that is miles ahead of my current work, Project Kibou will continue. A public server will be created for beta testing, but will not be planned until Fall 2014. I am a full time student and part time tutor, so development might be slow at times; but please understand that new development news is always coming soon. Thank you so much for your patience and support.

Kind Regards,
Spirited

Development Log:

 Click here to expand

Infrastructure Development:

• IOCP Socket System [Complete]
• Pointer Arithmetic Packet Processor [Complete]
• Data Structures [In Development]
• Hash/Cipher Algorithms [In Development]
• MySQL Implementation [Complete]

Account Server Development:

• Packet Processing [Complete]
• Account Retrieval & Password Verification [In Development]

[InsomniacPro]

Great to see you overcome life's difficulties and get back into creating a server again. My condolences about the loss of a loved one. Ignore the trolls. Seeing as you are the only individual around with legal permission to do this, that makes it a very intriguing and unique project. I can't wait to see what you have in store. I have some questions by the way if that's fine. Have you completely started from scratch? What are you going to be using for database management? What patch are you aiming for?

[Spirited]

Thank you for your condolences. I wouldn't mind answering questions about the changes I've made. I have started completely from scratch; that means having to redesign and implement a socket system, packet processor, etc. from the ground-up. I was thinking about database management through an object-relational-mapper called litesql, but it was too slow. Instead, I'll be implementing a connection pool and stored procedures.

As far as a patch goes, that question is much trickier to answer. You see, my aim is not to recreate Conquer Online as it currently is or was at a patch, but to create the image of what I believe Conquer Online should have been at a patch. I want to stay true to what Conquer Online tried to be, but stay away from what it became. Technically, the base patch is 5615, but that really doesn't mean much since I'll be "fixing-up" the game so heavily.

[InsomniacPro]

Originally Posted by Spirited

Thank you for your condolences. I wouldn't mind answering questions about the changes I've made. I have started completely from scratch; that means having to redesign and implement a socket system, packet processor, etc. from the ground-up. I was thinking about database management through an object-relational-mapper called litesql, but it was too slow. Instead, I'll be implementing a connection pool and stored procedures.

As far as a patch goes, that question is much trickier to answer. You see, my aim is not to recreate Conquer Online as it currently is or was at a patch, but to create the image of what I believe Conquer Online should have been at a patch. I want to stay true to what Conquer Online tried to be, but stay away from what it became. Technically, the base patch is 5615, but that really doesn't mean much since I'll be "fixing-up" the game so heavily.

True, there is many features that made the game broken like it is now. I look forward to see what you have going on. If you need any help, which I'm sure you won't, gimme a holler. Good luck with the project man.

[Spirited]

Originally Posted by InsomniacPro

True, there is many features that made the game broken like it is now. I look forward to see what you have going on. If you need any help, which I'm sure you won't, gimme a holler. Good luck with the project man.

It's not really the features that broke the game, it's the game that broke the game. TQ did a great job in creating a successful MMORPG, but I'm just not satisfied. It feels broken to me because it doesn't have plot development, it doesn't have character development, and it gives me plenty of reasons not to play with the game economy. It's a bad game; a successful but bad game. If you have suggestions on how you'd improve Conquer Online, I do have a PM inbox - please don't hesitate to contact me whenever you want. I'm open for discussions, questions, suggestions, etc.

[Super Aids]

WHY AM I LAUGHING AT THIS, NOT YOUR THREAD FANG DW LOOOL JUST SOMETHING ELSE.....


Anyways best of luck with your project. Can't be bothered reading all this, I will keep a look at the progress though.

[Spirited]

Hey everyone. For the past week, I've been focusing on software designs for a MySQL implementation. The first (traditional) way of implementing MySQL, which is how most sources in the community have implemented it, is to create a connection every time you want to execute a database transaction. This usually takes between 50 and 200 milliseconds (at best). I decided this was way too long for my server, which is why I've spent time thinking of better solutions.

The first design I created used a stack data structure. When a connection was needed, it would pop a connection off the stack and push it back on after being used (so the connection could remain open and be recycled). This solution was very fast, but required locking the stack while it changed. This wouldn't scale the way I wanted it to, so I thought of a few additional solutions.

Out of the remaining solutions I wrote up, I decided on creating a pool that assigned a MySQL connection to a worker thread. When a worker thread needs a connection, the pool uses its thread id to retrieve an open MySQL connection. I used a red-black tree data structure to do this, since the tree structure is very balanced and won't be needing any restructuring during execution. This solution was the fastest, taking between 200 and 260 nanoseconds. Since the tree is thread-safe by design, there's no need for locks. This is a solution I'm finally comfortable with.

With a MySQL connection pool design finally implemented, I'll now be implementing the MySQL support classes and then moving on to the base classes required to implement Conquer Online's password cipher. More development is coming soon. Thanks for your support.

[SteveRambo]

What's wrong with the built-in connection pooling?
Also, does this really matter for a CO private server? I'm thinking/hoping that this was done just for educational purposes.
I can't really see why one would need to go and implement their own connection pooling system just for this really, a CO private server is not that database intensive.

[Spirited]

Originally Posted by SteveRambo

What's wrong with the built-in connection pooling?
Also, does this really matter for a CO private server? I'm thinking/hoping that this was done just for educational purposes.
I can't really see why one would need to go and implement their own connection pooling system just for this really, a CO private server is not that database intensive.

Well, first of all, there is no built-in connection pool because I'm using the C/C++ connector. My source is C++, not C#. The built-in connection pool that you're thinking of is for the .NET connector. It's also inefficient. Regarding your comment on "does it really matter", you tell me: how well has "does it really matter" done for this section?

I know it's against traditions in this section, but why do it the wrong way when you can do it the right way? Hard concept to grasp, I know, but why would I develop a server that just gets by when I can develop one that does laps around everyone else's? I mean, let's face it, I'm kicking *** right now. What is everyone else doing to catch up?

[InfamousNoone]

Originally Posted by Spirited

Well, first of all, there is no built-in connection pool because I'm using the C/C++ connector. My source is C++, not C#. The built-in connection pool that you're thinking of is for the .NET connector. It's also inefficient. Regarding your comment on "does it really matter", you tell me: how well has "does it really matter" done for this section?

I know it's against traditions in this section, but why do it the wrong way when you can do it the right way? Hard concept to grasp, I know, but why would I develop a server that just gets by when I can develop one that does laps around everyone else's? I mean, let's face it, I'm kicking *** right now. What is everyone else doing to catch up?

"does it really matter" has worked wonders for me.

[Spirited]

Originally Posted by InfamousNoone

"does it really matter" has worked wonders for me.

Roy, I know you put a lot of thought into your code. You're one of the people I can say who's also kicking ***. Your third source attempt, for example: you did a lot of really interesting and cool techniques that I actually learned from. You were the one that gave me the idea of using a data structure (dictionary at the time) in replacement of a switch statement. "Does it really matter" is a phrase I'd hear from the people who work on the Trinity source modifications.

[InfamousNoone]

no but it's true and I really stand by that statement, like right now with CO PS because john insists on using Entity Framework debugging certain database issues turns out to be a real ***** and I can't take it seriously because this is the exact reason why back in 2008 or whenever I first made a server I avoided databases. I chose something easy and unorthodox but something I knew I wouldn't shoot myself in with. It's clear ini files shouldn't be used as a core database mechanism, but for what I needed it for, it worked fine while everyone (at that time) always seemed to have some weird database related issues they could never fix constantly popping up one after another. So while my method was unorthodox, I still stand by it with the phrase "does it really matter."

there's no reason to over-complicate/abstract your code if the project is small enough that it doesn't call for it. Yes, ok, it's good practice to do so, but it can also be a huge waste of time. Your goal isn't to make a server, your goal is research and knowledge. Time isn't a constraint for you, this makes things very different.

[CptSky]

Originally Posted by InfamousNoone

no but it's true and I really stand by that statement, like right now with CO PS because john insists on using Entity Framework debugging certain database issues turns out to be a real bitch and I can't take it seriously because this is the exact reason why back in 2008 or whenever I first made a server I avoided databases. I chose something easy and unorthodox but something I knew I wouldn't shoot myself in with. It's clear ini files shouldn't be used as a core database mechanism, but for what I needed it for, it worked fine while everyone (at that time) always seemed to have some weird database related issues they could never fix constantly popping up one after another. So while my method was unorthodox, I still stand by it with the phrase "does it really matter."

there's no reason to over-complicate/abstract your code if the project is small enough that it doesn't call for it. Yes, ok, it's good practice to do so, but it can also be a huge waste of time. Your goal isn't to make a server, your goal is research and knowledge. Time isn't a constraint for you, this makes things very different.

I agree with Roy. Honestly, a CO2 server is far from being a resource-hog, or anything intensive. Plus, with the small community we have ? If your infrastructure can't support 100/200 CCU, meh, there is a big problem.

I've always used custom binary files for my databases and it worked well. I never had all the issues that others are seeing (because they don't know how to work with a SQL database). COPS v7 is the only source using MySQL, and without prepared statements or a connection-pool, and honestly, I'm sure it will do the job. Because if the design is good, you barely have 4-5 tables to touch from time-to-time. The rest is static data that can be loaded in memory because the footprint is so small...

Your project is a research project. Most servers were not done for education. That's the big difference. When you develop a server, you have time constraint because players want X, Y, Z now. You must do some trade-offs, hoping that it won't affect the gameplay. (Like my AI and my battle system in COPS v6, which wasn't designed properly because I had already delayed the opening...)

===

I would like to do a generic distributed architecture with specific services for CO2 (would hook the client and add a conversion layer there). If I do, I'll do it without trade-offs, because I won't have time constraint. But I don't expect server's owners to do the same.

[KraHen]

SQLite brought the best of the two mentioned worlds for me together in many projects, but you have to handle it with care and love.

[Spirited]

I suppose that's right, Conquer Online really doesn't need the resources, but it really doesn't make too much sense to me to cut corners like everyone else has. It just seems wrong to me. Maybe that's because I enjoy software architecture, maybe it's because I have ideas for the server that are resource intensive, I'm not really sure.... but there's no harm in doing things the way they should be done.