General mods work by modifying the code that the client is using in memory. First, a little primer on general code..
The programmers over at Devcat write the game in C++. Then they run it through something called a compiler to get the actual, runnable files that we use to play. The compiler takes this nice, easy to read C++ code and translates it into the language that our computers understand. This language is pretty difficult for humans to read, but its possible.
Mods just modify the code that the game is reading in memory. For example, the data folder mod. That exists in the game already, its just disabled. The mod that enables it modifies the games memory to re-enable it.
Pake mods, mods that interfere with the packets themselves work a bit different.
The game client communicates to the server in little bursts of data, called packets. When you move, you send a certain packet, telling the server what you want to do. The server works its magic, and if the move was valid, sends packets to all other characters around you, telling them about your move. Then their clients show you moving.
Mabipake 'hooks' the game's functions for sending and receiving these packets. This means that instead of a packet being read ONLY by the client, it is detoured into Mabipake first. Mabipake can do whatever it wants to the packet before the client ever sees it. Mabipake also 'hooks' the game's send function, so it can inject its own packets into normal communication. These packets can be malicious, as the client didn't create them to be compatible with the server.
The server understands some packets that were never meant to be used, or were used a long time ago. These packets are huge security holes in the server. They often aren't checked for malicious activity or can be used to give the player a huge advantage. Two examples are the status modification packet and the invisibility packets.
I found a packet, long ago, that let me activate any status on my character. From exp boosts to demigod status. It was quite an amazing find, of course it was just a few months until others figured out how I did it through guessing until they got it, so now its patched up and no longer recognized by the server.
Another interesting packet was the pet hide packet. I was looking at the packets pets used and was trying to apply them to my actual character.. Then I tried the pet 'fetch' packet, it allows pets to become invisible for a while, while they fetch an item for you. The server didn't check if you were a pet when you sent that packet, so I was able to become completely invisible using this. Of course, again, it was only a matter of time before others figured out how I did this, and because of abuse it is now patched up.
If you have any questions or I wasn't clear about something, feel free to PM me or ask here.