Inspirations from SRO_DevKit
No, this is not an out of season April fools joke! However, it's a very long and probably boring read, and since there's nothing ready to use yet, I'll save most people the time by mentioning that upfront, so feel to check back later once something is released.
I went on another SRO nostalgia trip the past month. I started looking back at a lot of my old released projects and dug up all my unreleased code. Literally 100s of projects for different things as I was on a quest to become a better reverse engineer. That quest still hasn't changed, only the game that I happen to be focusing on at the time, but here I am again.
I'm taking another short term look at SRO with new perspectives and knowledge now. Maybe I'll do some new stuff again or maybe I'll just get bored in a few months and move on, who knows. For now though, I've found a few things to have fun with and share with the community, so here's another TL;DR post from me because I've found something interesting again.
In this thread, I'm going to use florian's
project as the base source of inspiration. It was actually his blog posts that gave me a few new ideas for things I wanted to do (will be covered in other posts later on), but all that information is contained in the SRO_DevKit anyways, so it's easier to just reference the devkit.Since this is my first post in a while, I feel the need to also give special thanks to @
, @, @, and @ for their community contributions as well. Those are the main names I continuously see when it comes to dev posts I've read, so thank you for your work! I don't mean to leave anyone else out who has also made contributions to the community either, it's just those seem to be the primary recurring names I see across the forums. I'm sorry ahead of time if I've missed anyone else, but you also have my thanks as well regardless.Before I start, I'd like to also mention there's no usable code/project to share with the public yet in this post. I'm still putting things together and testing everything out. Basically, I'll only release something if I feel it's ready to be released and I'm happy with the quality of it. Anyone that wants to make their own version of what I'm about to describe certainly can though!
The main purpose of this thread is to share what I think is untapped potential with the SRO_DevKit project. I'll also be referring to this thread as the prerequisite for potential future projects, because their foundation will be built up using the stuff I will be talking about here. Basically, I've learned so much stuff over the past 10 years, there's no simple way to just share that knowledge, so I'm going to apply as much of it as I can in a practical manner.
[Initial Context]
As I mentioned earlier, I was revisiting SRO with the intent on seeing just how different things would look now with 10 more years of experience. I've learned a lot and have made even more projects/tools for Path of Exile during that time than I ever did with SRO, which is pretty crazy since I have so much SRO work done. I had started looking through community projects and some of the work that's been done ever since the VSRO files were released.
florian's blog post about memory based movement was what initially got me hooked. I still love bot development, and while I did have both packet and non-packet bot stuff for SRO back in the day, none of my old SRO stuff remotely compares to what I was able to accomplish with Path of Exile, so I've certainly raised the bar for myself moving forward.
Reading the blog posts greatly helped me make sense of things I encountered in the past, but didn't quite understand at the time. It saved me a lot of time from having to reverse them now, which isn't really a problem, rather it's just so time consuming and there's a lot to do.
Since VSRO 188 is a static version, I was able to catch up on SRO stuff and get back to doing things pretty quickly, despite the long time away. There's a lot of solid information provided between the blog posts and the SRO_DevKit project itself. As I started applying my new knowledge and perspective to this stuff, something dawned on me.
I don't think the community has realized yet just how much of a gold mine of information the SRO_DevKit actually is.
There's a bit of a nuanced topic here, so let me explain. The actual implementation of the SRO_DevKit is rigid due to binary compiler compatibility, hardcoded to VSRO 188, and really inefficient to work from when trying to reverse the game. It's a good modding platform though, but I feel there's other ways of doing things to achieve the same, which brings me to reference some of the new stuff I've learned over the years. In other words, it's the ideas behind SRO_DevKit and the way SRO works that makes it possible for there to still be unrealized potential when it comes to reversing the game.
[New reversing tools?]
I've come to see in my post-SRO days how important amazing reversing tools are. Being able to automate work each client update is important in saving you time and sanity. Furthermore, being able to apply your tools to different SRO versions allows you to understand the differences between them and potentially catch issue in your project because you have more targets to test across. As such, I've taught myself how to make some pretty crazy tools.
To illustrate my perspective as to why I feel SRO_DevKit isn't effective as a reversing tool right now, let's look at what it takes to understand how to iterate the entities in the game world. Doing some digging in the devkit, you'll piece together the following (obviously with some stuff snipped out for this post):
Thinking aloud, g_pGfxEttManager tells me there's a global variable that holds the pointer to a CEntityManagerClient instance, so it's like a singleton. Great, makes life easy. Let's say I want to understand the entire CEntityManagerClient object, so I dereference that pointer in x32dbg. I'd see something roughly like this (I should note I'm using memory from TRSRO, since I don't have VSRO setup locally yet)
Now, referencing the code and using some already known low level MSVC/C++ knowledge:
- I know CObj should be 4 bytes because it has a vtable, and no other data members
- CObjChild states it has 16 bytes from offset 4 (based on the reclass name) followed by a list
- Doing some math, I know 0x4 + 0x10 means that list should be at 0x14
- 'vc80-stl.cpp' has a test case that verifies n_list should be 0xC bytes
- CEntityManager has 4 padding bytes followed by the map of entities
- I could probably compile the code in vs 2005 and use sizeof to verify the map size and offset of 'entities', to make sure I didn't miss anything or made a mistake, but I think I'll be able to make my point now
From a reversing point of view, this is inefficient, error prone, and just too much work for something this simple. Furthermore, due to how glorious C++ is, trying to write context aware memory dumps (basically make logs of the values of everything at any point in time) is super tedious and time consuming. I think I've seen some libraries that help, but that's just more complexity and work.
CEntityManagerClient is a tiny class, and there are substantially larger classes where this same process has to be perform on. SRO_DevKit does make use of the sizeof/offset of enforcement checks, which is great, but it's manual work, and not very helpful all the time, because the real CEntityManagerClient should be larger than what you can deduce by looking at the devkit source files (in VSRO it's 0x48 bytes, in TRSRO where the image is from, it's 0x3C bytes).
Having to manually go through and update layouts and fix offsets between different sro versions or game updates feels like an impossible task, and probably is to a degree right now. It's possible that I missed it, but I don't recall seeing any reclass tools for users to use to help advanced the object layouts, but even if there were, it would still be nice to have easily customizable tools to create your own efficient workflow. I myself have never used reclass; I always code my own tools for that stuff, so having to use that would hinder my reversing workflow.
So, what's the alternative? Jumping to the end result of my current project's output, it would be this (C# code if it's not obvious, also for TRSRO because I can't dump memory of a running VSRO yet, and I'm not giving ISRO $29/4 weeks just to be able to login)
The 'NativeMap' types were manually labeled, but everything else was code generated. By default, I simply split each structure along 4 byte alignment, since that's what MSVC does in 32-bit, so most fields will end up being 4 bytes, but can be ushorts/bytes, and perhaps even a double (8 bytes) so I'd have to manually split those up. In any case, most of the layout revering work is done simply by making everything 4 bytes, and then going through and adjusting things as needed.
The cherry on top from using a language supporting reflection like C#, is that the memory output from the image above (values might not exactly match, since memory changes) can easily be generated.
I've found being able to create memory logs like this greatly helps speed up reversing while giving you the ability to use diff/search tools to find recurring pointers or to match things across different classes. For example, let's say you dumped a bunch of memory and then search for '0x272A53C4'. If it appears in some log at some particular field, you know that field is a pointer to the current 'CEntityManagerClient' instance. Or, if you find a pointer in the client you don't know what it is, you can see if any of your dumped data contains it to investigate more.
[An important side note]
I'm running the client in a debugger, which is a must for reversing memory stuff, so I have debug heaps enabled. That means client performance tanks a lot, but you can see uninitialized memory (0xBAADF00D) rather than random memory values to get an idea of what's real data and what's not, as well as being able to see various C++ container types a lot more easily and various memory markers.
For example, in text listing above, I know CEntityManager_Data::_0014 is really either an u/short or two bytes followed by 2 bytes of what most likely is padding. Had I run the client without debug heaps, you'd get random values there and it might look like a pointer, or a float, or just random nonsense. The only way you'd know the real type would be to breakpoint on access the field and check the accessing assembly. That's impractical to do for every field, so having every bit of help really counts.
That is why debug heaps are so critical in reversing memory based stuff. It eliminates most of the guesswork when it comes to finding unused padding bytes, but it's not foolproof, since you don't know if something is simply uninitialized at that point in time, or is really just unused padding bytes. Checking CObjChild_Data in other types reveals the bytes are actually used.
Lastly, two more examples of just how important this stuff is. Consider the following image:
To most people, if you saw this memory layout, you'd start to think in terms of each part. First is a pointer, followed by 3 unset or padding uints, followed by 2 uints. However, this is a very obvious C++ container, either a std::string or std::wstring. By visiting the first pointer, you'd have seen it's a std::string because the text was not in wide character format.
In older SROs, yes, it's been posted that std::n_string is being used, which has 4 bytes prepended, but being able to understand containers and differences across Visual Studio versions is required regardless. PoE was similar back when I started with it, and over time as Visual Studio updated, so minor adjustments had to be made. It's a learn once, use forever type of skill though.
Consider the following C++ container, what is it?
If you mistakenly reversed this as a float, list, vector, then two uints, you'd be missing a pretty important container, because the stuff contained in the list/vector might seem really weird. However, knowing it's the specific C++ container it is, you can use the right code to easily access the data inside.
Given enough experience reversing this stuff, you'll start to be able to instantly recognize such memory patterns, and things speed up a lot and you have a much clearer view of structures.
[Building up from C#]
Back to the output of the tool before the side note, what's great about this approach, is that you can then take snapshots of memory, perform some action, take a new snapshot and then use a text compare tool against the logs to figure out what's changing where to narrow down where you might need to focus your next efforts reversing/debugging.
Obviously, you'll have to write custom code to dump container contents, but that's just part of reversing so it's not a big deal. Since it's an external tool and C# is quick to recompile and run, you can easily make changes to the layouts as you reverse memory, rerun and then almost instantly see your results.
In essence, the tool is taking on the same role as something like ReClass, except it's providing a different workflow, one that I've used for years to reverse very large parts of games, so I know how efficient the process is. In addition, it's super easy for the community to use, just download the tool, specify the process, then take a snapshot. Edit struct layouts as you reverse them, upload the changed code, then everyone can easily stay up to date with minimal headaches.
The benefits don't stop there either. Using C# as your foundation, as a meta language in a sense, you can then use reflection to code generate C++ code. That's right, some of the best C++ code is actually just code generated from a different language, as you'll be able to easily make changes and rework things without having to do a bunch of tedious or error prone manual changes.
I actually wrote a C++ structure generating tool based on C# struct layouts for PoE, and it killed me just how long it took for me to understand why I should never have been writing so many manual structure layouts in C++ for years. It's far better to do it in something like C#, get your fast and easy productivity tools and memory dumps, then code generate whatever language you need from it.
[What are you doing and how does it work?]
What I'm doing is auto-magically generating the estimated structure layouts of the main objects of the game. Silkroad does not use an entity component system architecture, rather just your typical old school OOP design (which make sense because it was coded a long time ago).
For example, let's say I wanted to find the current player's HP/MP. Well, that information is stored somewhere in class that holds that type of information, which is CICharactor. However, the CICharactor object is then stored in another class for the type that the object actually is. For example, your character's object is CICPlayer. However, what about monsters? Their CICharactor data would be instead stored in a CICMonster object instead, and that memory layout would obviously be different.
By understanding SRO's OOP design, thanks in part to the dynamic RTTI information, it is possible to not only know the sizes of game structures, but also their exact ordering as it pertains to how each object is laid out in memory. I've not yet directly used the compile time rtti system florian blogged about (still on the todo list), but just knowing that existed and seeing some of the names helped me realize the other (?) system in place that can be leveraged to do neat things. Also, I'm just referring to it as rtti because it makes sense, I'm not sure what it actually is, if it's a typeid system or something.
Anyways, back to the example, rather than taking a CICMonster object as an entire singular object and applying some offset to find the hp, and then taking an entire CICPlayer object and applying a different offset to find the hp, you can instead just find the offset of the hp in a CICharactor object, and then magically know where it should be regardless if you have a CICMonster or CICPlayer object because you know the exact class layout from RTTI.
A practical example of this can actually be seen from
:Using the new system I am talking about, CICCos would actually be (and this is the exact VSRO 188 layout, since I can generate it, I just can't dump memory yet)
So "unsigned int MaxHP; //0x0458" would actually fall inside "CICharactor_Data", which is expected because as I mentioned a few sentences ago, hp/mp data is in CICharactor, but CICharactor might be at a different offset depending on the object type (in reality, most objects share a similar hierarchy, it's the differences between sro versions that matters more). Hopefully that all makes sense now. It's a pretty big deal, reversing wise to have this type of information available.
The really neat thing is, this system is used for a lot of things. florian has a blog post about the client's state machine based process system. If you go through CGame, you can get the current process, so when you're in the game world, CPSMission is the active process, you can then use this system to dump the object and work out additional information, besides the known time information. For example, I've found an instance to a GWeatherManager and
ACObjectMgr object, which I can then dump the same exact way to work out everything.
Going back to my initial example of getting entities, the following C# code:
produced this output in jangan where new characters spawn:
The main attraction to this type of SDK generation, is that you then only have to match naming conventions across each SRO version, and your main code is going to pretty much work the same way as long as the SRO features you are coding for are the same.
For example, I have running code on TRSRO right now. When I get VSRO up and running, I just need to label the CEntityManager.:Entities and CIGIDObject::Name fields as well as use the larger old C++ native containers (behind the scenes) and the example code will work exactly the same.
That means developing across different SRO versions is way more accessible, so in theory, most VSRO work can be ported to other versions because there's not a massive difference in the entirety of the underlying game, so a lot of work is cut down. By diffing the rtti info across versions, vsro, csro-r, trsro/isro, you can get an idea of which classes/functionality is missing, and code accordingly.
The "fun" project I wanted to make, was a new autoselect feature for ISRO as it's been long removed. Now that I know how to iterate entities and find info such as type, hp, etc..., it's just a matter of finishing the project by either using packets or injected code to perform the selecting action.
Of course, this is only represents a small portion of what SRO_DevKit does, so it's not a replacement for it by any means or even a fair comparison. I'm only focused on the memory layout of objects right now, so coding support for virtual functions or binary compatibility with vs 2005 is just not a priority. However, you'd be able to generate better C++ structures to work from that were the correct size and ready to be understood in ways not currently possible with any tools I've seen in the community, so it'd help supplement it greatly I think.
In my opinion though, I'd rather look at rewriting the entire client as opposed to trying to reverse the entire thing to be able to fully mod it, but that's another topic for another day.
The last thing to talk about for this section is the "how does it work" part.
This is not compilable as-is (not hard to make it if you were really determined), but is the main program that shows exactly what I'm doing to generate the files
RTTI_Layout_Generator.cs
As a VSRO 1.188 example, I'm basically finding and parsing information like this from the client:
Then, using a bit of creativity to make magic, I build the rtti hierarchy as a composite object, and then create separate data files for each object for people to update. Using reflection, I also have a function in my project to make sure no one breaks the size of anything by checking against the saved rtti data in the json file.
While it's technically "working", I've already had to make a few revisions to address issues where the structs I generated needed to use a different format to make diffing between game updates as easy as possible, and making sure this can become a community tool people can contribute to without having to jump through any hoops. This requires regenerating all files, which obviously overwrites any work, so I need the format to be finalized before starting up community work on it. That's also why I've generated an x32dbg labeling script for the client, as it certainly helps give more context to a lot of functions that use these types.
If you've made it this far or just happened to read this part, the following label script is for the original VSRO 188 client (MD5: DEDDDC44FD43914E2D4E2861C5ABA29C) so don't run it unless you know what you're doing, because you can mess up your existing x32dbg label database and lose a lot of work.
Instead, you can visit a few addresses first to make sure everything looks right, and then run it on a clean client (after backing up and moving your old x32dbg file database of course). ALT + S to open script, right click inside the program, Load Script -> Paste (obviously copy the code to clipboard first), then spacebar to run while the client is paused or at the main entry point.
I had to use gists for the script, because this post would have otherwise broken the 100k thread character limit as the script itself is nearly 70k characters.
[Closing]
Phew, that was a lot of words. It seems the older I get, the more of them I tend to use
This stuff is exciting to me right now because I had wanted to start doing some new memory based things with the client, and now I can. It feels like there's a lot of new potential for stuff, so I'll just have to see where things end up as I familiarize myself with Silkroad again and start revisiting old projects.
On the todo list for this stuff is find more global pointers holding classes to dump information from (basically, trace into the Allocate_ functions, looking for 'this' to be stored in a static address), general testing to make sure I have as efficient and easy to use workflow as possible, and finally setting up a community project for people to start reversing or just generate structures for the SRO_DevKit to further advanced that awesome project.
I know this wasn't a super flashy thread, and is probably a confusing/hard to understand topic, but I think it'll start to make more sense and become more apparent how useful this stuff can be once the next set of projects rolls out. As I mentioned before, I'll be referring to this as the prerequisite thread, so I have to get it out of the way as I'm building upon this foundation for now.
