Chimera Private Server Development

[Spirited]

Originally Posted by ahmedali10

how long it takes untill we see your project in beta or alpha

I wouldn't hold your breath, but I am considering making a private branch and developing a server. Don't get me wrong, I'd love to make a server. It wouldn't be for months to a year though if I decide to. And even then, I've never delivered in the past. My project work always fell through. I have time to though, so maybe. Keep bugging me about it if you don't see progress, please. I need the reminders.

[turk55]

Originally Posted by Spirited

I wouldn't hold your breath, but I am considering making a private branch and developing a server. Don't get me wrong, I'd love to make a server. It wouldn't be for months to a year though if I decide to. And even then, I've never delivered in the past. My project work always fell through. I have time to though, so maybe. Keep bugging me about it if you don't see progress, please. I need the reminders.

He will do us all a favor if he did hold his breath. Just saying.

[suchecsgo]

Seems to be a nice project. I would like to talk to you via PM or prefered skype but your skype nick is not visible and you disabled the PM function on this board

[Spirited]

Originally Posted by suchecsgo

Seems to be a nice project. I would like to talk to you via PM or prefered skype but your skype nick is not visible and you disabled the PM function on this board

My account here is set to whitelist contacts. You should be able to now.
It's blocked by default to prevent people from spamming me with requests.

Originally Posted by turk55

He will do us all a favor if he did hold his breath. Just saying.

Almost spat out my tea on that one.

[.Xori^]

Originally Posted by Spirited

I wouldn't hold your breath, but I am considering making a private branch and developing a server. Don't get me wrong, I'd love to make a server. It wouldn't be for months to a year though if I decide to. And even then, I've never delivered in the past. My project work always fell through. I have time to though, so maybe. Keep bugging me about it if you don't see progress, please. I need the reminders.

I'd be really interested to see this happen.

[Spirited]

The Code
Yesterday, I mentioned I would be refining Chimera's packet system to no longer use reflection. I didn't expect to get to it so soon, but I was thinking about solutions yesterday and today and finished a packet reader and writer. The new reader and writer both help meet decode and encode interfaces, used by the client when receiving and sending data. The interfaces now look like this...

// Decoder defines a function that can be used to decode a byte buffer
// received by the server into a structure using TQ Digital's byte ordering
// rules. Must be met to be received by the server.
type Decoder interface { Decode(*Reader) interface{} }

// Encoder defines a function that can be used to encode a structure into a
// byte buffer to be sent to the client using TQ Digital's byte ordering
// rules. Must be met to be sent to the client.
type Encoder interface { Encode() *Writer }

The writer and reader are implemented using a byte.Buffer for the writer and a static byte array for the reader (slice split from the client's receive buffer). The methods are relatively uninteresting, but most importantly are used like this:

func Decode(read *packet.Reader) *Packet {
    p := new(Packet)
    p.Length     = read.Uint16()
    p.Identifier = read.Uint16()
    p.Account    = read.Seek(4).CString(128)
    p.Server     = read.CString(16)
    return p
}

func (p *Packet) Encode() *packet.Writer {
    write := packet.NewWriter()
    write.Uint16(p.Length)
    write.Uint16(p.Identifier)
    write.Uint32(p.Identity)
    write.Uint32(p.Token)
    write.Uint32(p.Port).Zero(4)
    write.CString(p.IPAddress, 16)
    return write
}

Test Results
As expected, this is much more performant than using reflection. For my decode tests, I tested decoding MsgConnect. When looking at my old reflection based system where structures were encoded and decoded automatically, 1000000 tests averaged about 1298 ns/op. Using the new reader where structures are decoded using explicit instructions, 20000000 tests averaged about 72.2 ns/op. Here are the below results:

> go test warry.io/chimera/lib/packet -bench .
BenchmarkDecodeMsgConnect-8             20000000              72.2 ns/op
BenchmarkDecodeUsingReflection-8         1000000              1298 ns/op
PASS
ok      warry.io/chimera/lib/packet     2.890s

Encoding is slightly slower due to the bytes.Buffer, and thus I might get back to that later. For my encode tests, I tested encoding MsgConnectEx. These were the results:

BenchmarkEncodeMsgConnectEx-8           10000000               191 ns/op
BenchmarkEncodeUsingReflection-8         2000000               722 ns/op

Conclusion
Obviously stated, don't use reflection in a high performance server. Removing reflection improved decode by a multiple of about 17. Encode is currently showing improvement by a multiple of about 4, so I'll work on that next (maybe). If you're interested in testing using Go, check out their testing package.
__________________

Edit:
It turns out that I wasn't thinking properly about the problem, and it didn't take long to come up with a better solution for packet encoding. Should have tried a simpler solution to start with. I replaced the bytes.Buffer with a static buffer of the target packet length. Here are the results:

BenchmarkEncodeMsgConnectEx-8           30000000              40.6 ns/op
BenchmarkEncodeUsingReflection-8         2000000               674 ns/op

Encoding now also shows improvement by a multiple of about 17. Cheers.

[Darach]

very good jop dude, keep it up <3 .

if you don't mind, to release all tool's that have been deleted.
it was use for client modifications and edit's tools.
it was useful tools and i need it some times <3 .

[Spirited]

Originally Posted by Darach

very good jop dude, keep it up <3 .

if you don't mind, to release all tool's that have been deleted.
it was use for client modifications and edit's tools.
it was useful tools and i need it some times <3 .

I don't believe I've released tools like that.
Only tools I know about are available here:

[KraHen]

As a sidenote, although this is a nice speed improvement, it might not actually be problematic in the real world. I've hosted servers for games that were 100% JSON based as far as the protocol goes and it still worked perfectly fine (with compression applied of course, so the bandwith increase wasn't really an issue, IIRC it was +5% or so), the majority of your processing power is doing way more complex things than serializing a buffer. Not saying it's not a useful improvement, but if the reflection based version makes it easier to maintain the code in the long run, it might be worthwhile considering using that - you need to reflect usually only once anyways and then cache the results and just fill in the changed values based on the type of the packet, although I'm not sure how one would go doing that in Go, I'm not very familiar with the reflection features that it provides.

[Spirited]

Originally Posted by KraHen

As a sidenote, although this is a nice speed improvement, it might not actually be problematic in the real world. I've hosted servers for games that were 100% JSON based as far as the protocol goes and it still worked perfectly fine (with compression applied of course, so the bandwith increase wasn't really an issue, IIRC it was +5% or so), the majority of your processing power is doing way more complex things than serializing a buffer. Not saying it's not a useful improvement, but if the reflection based version makes it easier to maintain the code in the long run, it might be worthwhile considering using that - you need to reflect usually only once anyways and then cache the results and just fill in the changed values based on the type of the packet, although I'm not sure how one would go doing that in Go, I'm not very familiar with the reflection features that it provides.

Go's reflection is nice, and they do use it in their own packages. I originally got the idea of reflecting packet structures from their , which uses reflection for abstract reads and writes. I also thought about using go's code generation functions for faster encoding/decoding. Basically, it would generate the decode and encode functions that you see in the current design. I mostly wanted a solution that made development easier, but I found these ideas just made it more difficult for people to understand what was going on and how to maintain the server. I wanted to scrap it for something that performed better anyways, so I went back to basics. It was mostly a decision around ease of development, and I may still write a code generator, but better performance is a good side outcome of all that.

[Xio.]

Originally Posted by Spirited

Go's reflection is nice, and they do use it in their own packages. I originally got the idea of reflecting packet structures from their , which uses reflection for abstract reads and writes. I also thought about using go's code generation functions for faster encoding/decoding. Basically, it would generate the decode and encode functions that you see in the current design. I mostly wanted a solution that made development easier, but I found these ideas just made it more difficult for people to understand what was going on and how to maintain the server. I wanted to scrap it for something that performed better anyways, so I went back to basics. It was mostly a decision around ease of development, and I may still write a code generator, but better performance is a good side outcome of all that.

Thought this is your solo project. Don't know anyone in the conquer community that uses Go.

[Spirited]

Originally Posted by Xio.

Thought this is your solo project. Don't know anyone in the conquer community that uses Go.

Chimera is open source, which means anyone could potentially use and develop on this. It could be that a select group of people from this community decide to learn Go (since it's really not that hard to pick up). It could be that people in my private group of friends decide to learn Go from it (like "Vander" who wrote GoConquer with me). I can see you learning Go and using it. I know you had interest in running Conquer servers off Raspberry Pis. Chimera does that. Just depends on people's interest in this... I suppose. Thanks for asking, btw. It's nice to see actual questions on this thread.

Note: I haven't pushed my recent revisions yet.

[Xio.]

Originally Posted by Spirited

Chimera is open source, which means anyone could potentially use and develop on this. It could be that a select group of people from this community decide to learn Go (since it's really not that hard to pick up). It could be that people in my private group of friends decide to learn Go from it (like "Vander" who wrote GoConquer with me). I can see you learning Go and using it. I know you had interest in running Conquer servers off Raspberry Pis. Chimera does that. Just depends on people's interest in this... I suppose. Thanks for asking, btw. It's nice to see actual questions on this thread.

Note: I haven't pushed my recent revisions yet.

Fair enough, point taken, just thought it isn't anymore as you removed the github repo.

Edit: Seems like I'm mixing up things here...

[Spirited]

Originally Posted by Xio.

Fair enough, point taken, just thought it isn't anymore as you removed the github repo.

Edit: Seems like I'm mixing up things here...

Yeah, it's a bit of a mess right now, but I'll fix that once this revision is done.

[Spirited]

@
Finished the revisions I was making to the packet system and cleaned up the repository. I also added some Visual Studio Code launch configurations so it can be debugged and executed from the IDE. Make file for setting up the project for the first time. You can access what I currently have for this revision here: .

I'm still working on the second revision as a whole, so it's not yet logging in like the last one was. Just starting to work on RPC revisions now. I used a basic TCP connection in my old revision, but I'm planning on implementing for a more bidirectional stream approach in this revision. That way, I can benefit from HTTP/2 (which wraps TCP) for multiplexed/parallel request processing. Not sure if it'll be a perfect match, but there's only one way to find out.

Edit: After speaking with Sedat and Smaethin, I think I'm convinced that gRPC is not the most appropriate solution for this. It adds too many project dependencies and headaches to be worth the hassle for RPC calls that don't actually need to be too performant. I'm going to default to go's official RPC package and just make a channel system for what I already had.