Register for your free account! | Forgot your password?

You last visited: Today at 19:51

  • Please register to post and access all features, it's quick, easy and FREE!


DMap -> BitArray | 10MB for 130 DMaps.

Reply
 
Old   #1
 
elite*gold: 67
Join Date: Aug 2014
Posts: 1,314
Received Thanks: 920
DMap -> BitArray | 10MB for 130 DMaps.

I'm using classes of Cpt.Sky's CO2_CORE_DLL to load raw dmaps, then I put the cells into a BitArray and serialize them using a DataContract Serializer to store them all in a single (14.7mb) file for fast loading.

All dmaps I have access to (130 from a 5017 client) consumed 11MB in total.

Here's a gallery that shows most of the source code so you can save time downloading and insult me and my code right away. Can't wait for Fang to tell me how its bad practice, flawed implementation and worthless shit

Gallery 1

I also added a class that holds all the data and let's you check the access for each cell using the x and y coordinates.

Probably not really interesting for most of you but thought I'd share it anyway. I used this system for the Raspberry Pi2 server I wrote out of boredom and don't need it anymore.

Source & Test console app.



Xio. is offline  
Thanks
1 User
Old 02/12/2016, 18:13   #2
 
elite*gold: 0
Join Date: Jul 2014
Posts: 402
Received Thanks: 534
Newest patch CO client has 267 compressed (.7z) DMap files with a total size of 2.72 MB ...


Best Coder 2014 is offline  
Thanks
1 User
Old 02/12/2016, 18:21   #3
 
elite*gold: 12
Join Date: Jul 2011
Posts: 7,052
Received Thanks: 3,384
I'll gladly tell you why it's bad practice, but firstly I have to say I that I used the same system when experimenting with the Raspberry Pi. For something as small as an embedded system, performance sometimes has to be scarified for good resource management. My primary concern is that you're not handling tile flags at all, nor are you handling elevation. That just means more computation per screen update if you choose to check for monster and item collisions. With the RP2 out with 1 GB of RAM, this map system really doesn't solve the problem correctly. A better idea would be to limit yourself to particular maps and load chunks of maps on demand. Perhaps use a more appropriate language and operating system for embedded systems as well.
Spirited is offline  
Thanks
1 User
Old 02/12/2016, 19:48   #4
 
elite*gold: 0
Join Date: Jul 2006
Posts: 1,933
Received Thanks: 664
Agreeing with Fang. The project itself though is not bad, just unfinished. Even though you really dont need this kind of optimization even on low end hardware, it's nice for an educational example, so kudos.


KraHen is offline  
Old 02/12/2016, 20:18   #5
 
elite*gold: 21
Join Date: Jul 2005
Posts: 9,193
Received Thanks: 5,311
Agreed. Optimization is fun with stuff like this for a research project and all but it wouldn't be what you'd want for a full scale server. Sooner or later you WILL want tile flagging (even a single byte per coord would be enough. 8 bits should handle all the flags you need as far as co is concerned) and then there's the question of height. Most servers wont need height data but that could change if you are going to implement any custom content or need specific checks.
pro4never is offline  
Old 02/13/2016, 02:33   #6
 
elite*gold: 12
Join Date: Jul 2011
Posts: 7,052
Received Thanks: 3,384
In addition to that, try to avoid await on server actions like the plague. I'm not saying you're doing this, but servers heavy in asynchronous actions usually perform very poorly. Over concurrency can lead to a whole mess of new problems, some not as easy to manage as others. Also, other server components rely on the map system, and having things like the map system load as an asynchronous action may cause race conditions for loading and accessing map data. I'm not sure if that influences your timing at all, but that may be something you should consider as well.
Spirited is offline  
Old 02/13/2016, 06:48   #7
 
elite*gold: 0
Join Date: Mar 2006
Posts: 413
Received Thanks: 42
dude! I appreciate your work unlike the other's out there! just keep telling and giving advice even telling you the worst thing and this person is even releasing any thread that is useful. unlike your's It kind a awesome!. Keep it up!
Soulfly25 is offline  
Thanks
1 User
Old 02/13/2016, 07:39   #8
 
elite*gold: 67
Join Date: Aug 2014
Posts: 1,314
Received Thanks: 920
Quote:
Originally Posted by Spirited View Post
In addition to that, try to avoid await on server actions like the plague. I'm not saying you're doing this, but servers heavy in asynchronous actions usually perform very poorly. Over concurrency can lead to a whole mess of new problems, some not as easy to manage as others. Also, other server components rely on the map system, and having things like the map system load as an asynchronous action may cause race conditions for loading and accessing map data. I'm not sure if that influences your timing at all, but that may be something you should consider as well.
My Raspberry Pi server uses async/await, the TPL and custom threadless tasks using TaskCompletion Source heavily. Here's some stats (running on my PC for Perfmon):
Loading takes 487/48/52 GC calls, picture shows stats after jumping around for 10min killing 4k bidrmen.


Tasks provide ContinueWith, WhenAll and other methods that let you easily synchronize the run order and dependencies. They are expensive in terms of allocations - especially threaded tasks, but if you cache task results you can cut down the allocations and GC pressure a ton.

I'm also doing things like updating the screen in a Parallel.ForEach with Aggregation and a single lock per partitioner on the final aggregation step. That again allocates quite a bit of ressources so I'm caching the threadlocal array. All in all, I'm making heavy use of pooling to reduce the GC presure and it works really well.

As you can see on the above screenshot, time spent collecting garbage is merely 0.07% of the entire CPU time spent on the server.

I am still wasting buffer objects on the socket system, somehow I'm not recycling all of them, once that's resolved it should show even less GC calls on Gen0.

The goal of this experiment was to reduce the allocations to a bare minimum using pooling while keeping the memory usage way down without stressing the ARM Quad Core CPU too much. Allocations are especially costly on the Pi, creating a new 850 byte array for example takes 17x longer compared to my PC.

Quote:
Agreed. Optimization is fun with stuff like this for a research project and all but it wouldn't be what you'd want for a full scale server. Sooner or later you WILL want tile flagging (even a single byte per coord would be enough. 8 bits should handle all the flags you need as far as co is concerned) and then there's the question of height. Most servers wont need height data but that could change if you are going to implement any custom content or need specific checks.
Also since I had two rPi's I tried to make the DMap/Pathfinding run on the other Pi and do some UDP communication between them to offload the load. Never really got far into that but basic access checks were working

The DMap System originally had 3 layers. 1 Ground, 1 Mob and 1 for Items. I switched to a single layer to reduce the memory usage while implementing a own BitArray like class based on Int+byte. The byte would allow me to store all flags without the overhead of having 3 bitarray layers. Never got that done though. Was an interesting project while it lasted.

Edit: By the way fang, I'm surprised you answered without insulting me. I appreciate that.

Quote:
Newest patch CO client has 267 compressed (.7z) DMap files with a total size of 2.72 MB ...
My file isn't compressed just serialized. Should load faster in theory. In my test it loaded all the maps from the serialized file in 480ms.

Quote:
dude! I appreciate your work unlike the other's out there! just keep telling and giving advice even telling you the worst thing and this person is even releasing any thread that is useful. unlike your's It kind a awesome!. Keep it up!
Thanks
Xio. is offline  
Old 02/13/2016, 08:13   #9
 
elite*gold: 12
Join Date: Jul 2011
Posts: 7,052
Received Thanks: 3,384
You have 106 threads running on a Raspberry Pi. 102 physical threads... on a Raspberry Pi. That's still 106 logical paths of concurrency. Here comes the insult... are you retarded? End of insult, thanks. That's absolutely horrible, how can you call that performance? You only have one person on the server and you have that many threads. Please stop. Stop doing whatever you're doing, drop ******* everything, and read a book or two on threading and concurrency. This is just as bad as those egy sources using a hundred+ threads for receiving data. You aspire to be a good programmer, so stop being ****. Do something about it and improve your base knowledge set. I'm so tired of seeing people do this. Read a book on threading; read a book on software design and style while you're at it as well. Holy **** this is bad. I want to see people do well here, so please spare me and spend some work in self-study.

Quote:
Originally Posted by Soulfly25 View Post
dude! I appreciate your work unlike the other's out there! just keep telling and giving advice even telling you the worst thing and this person is even releasing any thread that is useful. unlike your's It kind a awesome!. Keep it up!
And you... there's no way to say this kindly. You need to shut the **** up. You appear everywhere telling intelligent people to screw off. You tell people that our constructive criticism is wrong, when it's flat-out not. You tell people we don't release things, but do you realize that your drunk source wouldn't run if it weren't for the people on this thread plus many others? Stop it. You're just as bad as everyone we criticize. This isn't even negative criticism. Just stop. Can that be the slogan for this thread? Thank you.
Spirited is offline  
Old 02/13/2016, 08:24   #10
 
elite*gold: 67
Join Date: Aug 2014
Posts: 1,314
Received Thanks: 920
Quote:
Originally Posted by Spirited View Post
You have 106 threads running on a Raspberry Pi. 102 physical threads... on a Raspberry Pi. That's still 106 logical paths of concurrency. Here comes the insult... are you retarded? End of insult, thanks. That's absolutely horrible, how can you call that performance? You only have one person on the server and you have that many threads. Please stop. Stop doing whatever you're doing, drop ******* everything, and read a book or two on threading and concurrency. This is just as bad as those egy sources using a hundred+ threads for receiving data. You aspire to be a good programmer, so stop being ****. Do something about it and improve your base knowledge set. I'm so tired of seeing people do this. Read a book on threading; read a book on software design and style while you're at it as well. Holy **** this is bad. I want to see people do well here, so please spare me and spend some work in self-study.
Okay so, first off, I'm running this on my PC as I said. Also I noted that this is the global count of all threads being used by all applications that run on the .net framework. Its a global number, not only the server...

The server uses 22 threads. 16 for the scheduler (thats mostly idle and blocking so not consuming cpu time) 1 for the outgoing packet queue and 1 for item drops. the other 2 threads for the sockets, login and game. 1 main thread and the last one must dynamically created by the parallel foreach or w/e.



here fixed the counters for you

Now, that's more how I know you.
Xio. is offline  
Old 02/13/2016, 08:26   #11
 
elite*gold: 12
Join Date: Jul 2011
Posts: 7,052
Received Thanks: 3,384
Quote:
Originally Posted by Xio. View Post
Okay so, first off, I'm running this on my PC as I said. Also I noted that this is the global count of all threads being used by all applications that run on the .net framework. Its a global number, not only the server...

The server uses 22 threads. 16 for the scheduler (thats mostly idle and blocking so not consuming cpu time) 1 for the outgoing packet queue and 1 for item drops. the other 2 threads for the sockets, login and game. 1 main thread and the last one must dynamically created buy the parallel foreach or w/e.

Now, that's more how I know you.
Thank ***. You gave me a near aneurysm.
What is the scheduler?
Spirited is offline  
Old 02/13/2016, 08:32   #12
 
elite*gold: 67
Join Date: Aug 2014
Posts: 1,314
Received Thanks: 920
Quote:
Originally Posted by Spirited View Post
Thank ***. You gave me a near aneurysm.
What is the scheduler?
A kind of threadpool that queues actions to execute at a given time. Not my code though, 3rd party class.
Xio. is offline  
Old 02/13/2016, 11:48   #13
 
elite*gold: 0
Join Date: Mar 2006
Posts: 413
Received Thanks: 42
Quote:
Originally Posted by Spirited View Post
And you... there's no way to say this kindly. You need to shut the **** up. You appear everywhere telling intelligent people to screw off. You tell people that our constructive criticism is wrong, when it's flat-out not. You tell people we don't release things, but do you realize that your drunk source wouldn't run if it weren't for the people on this thread plus many others? Stop it. You're just as bad as everyone we criticize. This isn't even negative criticism. Just stop. Can that be the slogan for this thread? Thank you.
Ahhh Okay. My Name is Jeff
Soulfly25 is offline  
Old 02/13/2016, 20:09   #14
 
elite*gold: 0
Join Date: Dec 2012
Posts: 1,621
Received Thanks: 834
Regarding what Spirited said about using async and Task. Remember that what you really do is putting one task from one thread to another, in which using too many asynchronous calls can lead to performance issues due to that every time you create a task you have to allocate memory for the task, then request it to be added to the thread pool and a whole lot other things. Let's say you have 4 threads available and all threads execute a task asynchronous. How do you think that would perform? It would not perform as well as you'd believe because all four threads will most likely be assigned with the asynchronous tasks through the thread pool and you might as well just have executed it synchronously; note I'm talking about the threads available in the processor which usually equals the amount of cores you have. Of course different CPUs allows for thread execution differently, but it's always a good thumb of rule to assume you only have the amount of cores available as threads.


Super Aids is offline  
Reply



« Troy-Co 5095 Website ! | XSCO 2.0+ full »

Similar Threads
[Release] Football DMap (Built With DMap Editor)
06/29/2014 - CO2 Weapon, Armor, Effects & Interface edits - 4 Replies
Here you go guys, My first release. I have noticed alot of servers have added football to there tournaments. Here is a proper DMap for you guys (Works with both new clients and older clients) Screenshot http://i.epvpimg.com/nvGId.png Self Installing DMap
Dmaps
02/23/2010 - CO2 PServer - Discussions / Questions - 10 Replies
I have serious no clue how to make the dmaps in 5017 lotf =D. Anyone have the codes or can give me a hint? :handsdown:
Dmaps
08/16/2009 - CO2 PServer - Discussions / Questions - 23 Replies
Can anyone explain to me what they are and what they do I have a slight idea of what they are used for but i wanna be sure.
DMaps
06/21/2005 - CO2 Main - Discussions / Questions - 1 Replies
Could someone attach some of their modified maps please? Thanks!
Dmaps which 1's are which
06/04/2005 - CO2 Main - Discussions / Questions - 7 Replies
does anyone have like a list of what map belongs to which .Dmap file(s), or is there any ways to find out that?



All times are GMT +1. The time now is 19:51.


Powered by vBulletin®
Copyright ©2000 - 2018, Jelsoft Enterprises Ltd.
SEO by vBSEO ©2011, Crawlability, Inc.

Support | Contact Us | FAQ | Advertising | Privacy Policy | Abuse
Copyright ©2018 elitepvpers All Rights Reserved.