Custom GFXFileManager - Create your own PK2-Format

florian0 · 03/04/2017, 12:57

Hello beloved, dead community,

i think its time for some fresh sources. I've reverse engineered the GFXFileManager. For the ones among us who don't know what this is: This thing handles all PK2-Container related stuff, meaning it is used to read and write files in and to the PK2-Container. The launcher and the server also use it to access files on the disk.

Yes.

. But his codebase was super ugly. (Sorry Drew <3).

For now, its doing what I am using it for: Easy development
All files are loaded from the disk, resulting in your SRO_Folder looking like this!

(Note: The Media.pk2 is still existing in a minified version because edxLoader requires it.)

So, having its interface reverse engineered, you now can:

Create your fully customized container format
Make a universal PK2 Editor that does not has to deal with the container-structure
Replace the container with plain disk access (for easier development)
Create wrappers for different file-types or names and replace entire file-formats (like introducing a compressed image format, maybe?)
...

IFileManager.h

Code:

#pragma once 

#include <Windows.h>
#include <stdio.h>

#include "shit_t.h"
#include "CJArchiveFm.h"
#include "searchresult.h"
#include "result_entry_t.h"
#include "DialogData.h"
#include "eCallbackState.h"

#define FM_VERSION 0x1007


typedef BOOL (__cdecl *error_handler_t)(HWND hwnd, const char *message, const char *caption);
typedef void (__cdecl *foreach_callback_t)(CALLBACK_STATE, result_entry_t *, void*);


#define SHOW_ERROR(msg, caption) \
	do { \
		if (error_handler) { \
			if (!error_handler((HWND)1, msg, "false")) \
				MessageBoxA(hwnd, msg, caption, MB_OK); \
		} else { \
			MessageBoxA(hwnd, msg, caption, MB_OK); \
		} \
	} \
	while (0)



class IFileManager
{
public:
	virtual int Mode(void) = 0; //returns the container-mode (1 for CP, 2 for CW)
	virtual int ConfigSet(int, int) = 0; //
	virtual int ConfigGet(int, int) = 0; //

	//
	// Container
	//

	// Create a new container
	// Parameter:
	// - filename: filename of the container
	// - password: password for accessing the new container
	virtual int CreateContainer(const char *filename,  const char *password) = 0;

	// Open an existing container
	// Parameter:
	// - filename: filename of the container
	// - password: password required for accessing the container
	// - mode: unknown, maybe for read and write access
	virtual int OpenContainer(const char *filename, const char* password, int mode) = 0;

	// Close the current container
	// No Parameter
	virtual int CloseContainer(void) = 0; //


	// Returns 1, if this instance has opened a container
	virtual int IsOpen(void) = 0; //



	virtual int CloseAllFiles(void) = 0; //Similar in both implementations

	// Returns the MainModule-handle
	virtual HMODULE MainModuleHandle(void) = 0;


	virtual int Function_9(int) = 0; //CPFileManager returns -1


	//
	// Files
	//

	
	// Open a file inside the container using a path
	// Parameter:
	// - filename: filename, relative to current dir or absolute path inside archive
	// - access: 0 for open-existing, 0x80000000 for open and share_read, 0x40000000 for create_always
	// - unknown: not used for original CPFileManager
	// Return:
	// Handle of opened file (can be any number or pointer) or -1 if opening is was unsuccessful
	virtual int Open(const char *filename, int access, int unknown) = 0; //


	// Open a file inside the container using the CJArchiveFm-class
	// Parameter:
	// - fm: A valid pointer to the CJArchiveFm-class
	// - filename: filename, relative to current dir or absolute path inside archive
	// - access: 0 for open-existing, 0x80000000 for open and share_read, 0x40000000 for create_always
	// - unknown: not used for original CPFileManager
	virtual int Open(CJArchiveFm* fm, const char *filename, int access, int unknown) = 0;

	virtual int Function_12(void) = 0; //return -1
	virtual int Function_13(void) = 0; //return 0

	virtual int Create(const char* filename, int unknown) = 0; //
	virtual int Create(CJArchiveFm * fm, const char * filename, int unknown) = 0; //
	

	// Delete a file by name
	// Parameter:
	// - filename: name of file to delete
	virtual int Delete(const char *filename) = 0; //

	// Close file by handle
	// Parameter:
	// hFile: Any handle or pointer identifiying this file
	virtual int Close(int hFile) = 0; //

	// Read a number of bytes from file
	// Parameter:
	// hFile: Any handle or pointer identifiying this file
	// lpBuffer: pointer to reserved memory for read operation
	// nNumberOfBytesToWrite: size of lpBuffer
	// lpNumberOfBytesWritten: pointer to memory, will contain the number of bytes read from the file
	virtual int Read(int hFile, char* lpBuffer, int nNumberOfBytesToRead, unsigned long *lpNumberOfBytesRead) = 0;

	// Write a number of bytes to file
	// Parameter:
	// hFile: Any handle or pointer identifiying this file
	// lpBuffer: pointer to reserved memory for read operation
	// nNumberOfBytesToWrite: size of lpBuffer
	// lpNumberOfBytesWritten: pointer to memory, will contain the number of bytes written to the file
	virtual int Write(int hFile, const char* lpBuffer, int nNumberOfBytesToWrite, unsigned long *lpNumberOfBytesWritten) = 0;




	// The next two functions are bugged, see issue #19
	// Get the full path of the executable
	virtual char* CmdLinePath(void) = 0;

	// Get the executable
	virtual char* CmdLineExe(void) = 0; //




	// Unknown function that gets two variables
	virtual shit_t* getShit(shit_t* shit) = 0; //get shit

	// Unknown function that sets two variables
	virtual int setShit(int a, int b) = 0; //set shit


	//
	// Directory Management
	//

	// Create a new directory in the container
	// Parameter:
	// - name: name of the directory
	virtual int DirectoryCreate(const char* name) = 0;

	// Delete a directory in the container
	// Parameter:
	// - name: name of the directory
	virtual int DirectoryRemove(const char* name) = 0;


	virtual bool ResetDirectory(void) = 0; //
	virtual bool ChangeDirectory(const char* dirname) = 0; //
	virtual int GetDirectoryName(size_t buffersize, char* Dst) = 0; //


	virtual int SetVirtualPath(const char *path) = 0; //set root
	virtual int GetVirtualPath(char *dest) = 0; //similar on both impl


	//
	// Searching
	//

	// Start a new search for files
	// Parameter:
	// - search: result structure representing a handle for the search
	// - pattern: matching pattern for a list of files
	// - entry: the output-structure for the first entry of the resulting list of files
	virtual searchresult_t* FindFirstFile(searchresult_t* search, const char* pattern, result_entry_t* entry) = 0;

	// Get the next file entry in the search result list
	// Parameter:
	// - search: result structure representing a handle for the search
	// - entry: the output-structure for the next entry of the resulting list of files
	virtual int FindNextFile(searchresult_t* search, result_entry_t* entry) = 0;

	// Free the search result
	// Parameter:
	// - search: result structure representing a handle for the search
	virtual int FindClose(searchresult_t* search) = 0;


	//
	// File Information
	//

	virtual int FileNameFromHandle(int hFile, char* dst, size_t count) = 0;
	virtual int GetFileSize(int hFile, LPDWORD lpFileSizeHigh) = 0;
	virtual BOOL GetFileTime(int hFile, LPFILETIME lpCreationTime, LPFILETIME lpLastWriteTime) = 0;
	virtual BOOL SetFileTime(int hFile, LPFILETIME lpCreationTime, LPFILETIME lpLastWriteTime) = 0;
	virtual int Seek(int hFile, LONG lDistanceToMove, DWORD dwMoveMethod) = 0;

	//
	// Others
	//

	virtual HWND GetHwnd(void) = 0;
	virtual void SetHwnd(HWND) = 0;
	virtual void RegisterErrorHandler(error_handler_t callback) = 0;

	virtual int ImportDirectory(const char *srcdir, const char *dstdir, const char *directory_name, bool create_target_dir) = 0;
	virtual int ImportFile(const char *srcdir, const char *dstdir, const char *filename, bool create_target_dir) = 0;
	virtual int ExportDirectory(const char *srcdir, const char *dstdir, const char *directory_name, bool create_target_dir) = 0;
	virtual int ExportFile(const char *srcdir, const char *dstdir, const char *filename, bool create_target_dir) = 0; // create_target_dir is unused

	// Returns: 0 on found, -1 on not found
	virtual int FileExists(char* name, int flags) = 0; //

	// Shows an open file dialog
	virtual int ShowDialog(DialogData *data);
	virtual int ForeachEntryInContainer(foreach_callback_t cb, const char *filter, void *userstate);
	virtual int UpdateCurrentDirectory(void) = 0; //
	virtual int Function_50(int) = 0; //returns zero in both impl.

	// get the version of this file manager
	virtual int GetVersion(void);

	//prompt error if version mismatch 
	virtual int CheckVersion(int version);

	virtual int Lock(int) = 0;
	virtual int Unlock() = 0;


private:
	void loop_container_content(foreach_callback_t cb, const char *filter, void* userstate);

	
};

Hints and Warnings:

It is developed based on the GFXFileManager in VSRO 1.188. Other versions and locales can work, but its likely they won't. Nonetheless, having the source you can easily make it work for other versions.
Feel free to report bugs on GitLab (please check if your bug has been reported already, and please put a meaningful description of your problem.)
Feel free to request stuff
I've never seen a good, community driven, silkroad project and I doubt it will ever exist. Anyway, feel free to contribute code on GitLab.
Please keep in mind: VSRO was compiled on Visual Studio 2005 (Platformtoolset 80). Microsoft does not care about ABI-compatibility between toolsets. While GFXFileManager seems entirely separated from the Main codebase, any newer toolset than 80 might break the DLL. Visual Studio 2010/Toolset 100 seems to work fine till now.
Compiling on "Debug" will break the ABI, no matter what version you are using! Always compile on "Release"!
I've even implemented some of the bugs the original file manager had

Complete Source:

License:

. No warranties whatsoever!

Download: This is a library. There is no use in a compiled version. Download the source on GitLab.

Have fun.

~~XerK..~~ · 03/04/2017, 13:18

thank you dud

XxGhostSpiriTxX · 03/04/2017, 13:57

thanks!

devil770 · 04/12/2017, 19:08

How does it work ? any guides ?

xxnukertube · 04/14/2017, 04:45

Quote:

Originally Posted by florian0

Hello beloved, dead community,

i think its time for some fresh sources. I've reverse engineered the GFXFileManager. For the ones among us who don't know what this is: This thing handles all PK2-Container related stuff, meaning it is used to read and write files in and to the PK2-Container. The launcher and the server also use it to access files on the disk.

Yes.

. But his codebase was super ugly. (Sorry Drew <3).

For now, its doing what I am using it for: Easy development
All files are loaded from the disk, resulting in your SRO_Folder looking like this!

(Note: The Media.pk2 is still existing in a minified version because edxLoader requires it.)

So, having its interface reverse engineered, you now can:

Create your fully customized container format
Make a universal PK2 Editor that does not has to deal with the container-structure
Replace the container with plain disk access (for easier development)
Create wrappers for different file-types or names and replace entire file-formats (like introducing a compressed image format, maybe?)
...

IFileManager.h

Code:

#pragma once 

#include <Windows.h>
#include <stdio.h>

#include "shit_t.h"
#include "CJArchiveFm.h"
#include "searchresult.h"
#include "result_entry_t.h"
#include "DialogData.h"
#include "eCallbackState.h"

#define FM_VERSION 0x1007


typedef BOOL (__cdecl *error_handler_t)(HWND hwnd, const char *message, const char *caption);
typedef void (__cdecl *foreach_callback_t)(CALLBACK_STATE, result_entry_t *, void*);


#define SHOW_ERROR(msg, caption) \
	do { \
		if (error_handler) { \
			if (!error_handler((HWND)1, msg, "false")) \
				MessageBoxA(hwnd, msg, caption, MB_OK); \
		} else { \
			MessageBoxA(hwnd, msg, caption, MB_OK); \
		} \
	} \
	while (0)



class IFileManager
{
public:
	virtual int Mode(void) = 0; //returns the container-mode (1 for CP, 2 for CW)
	virtual int ConfigSet(int, int) = 0; //
	virtual int ConfigGet(int, int) = 0; //

	//
	// Container
	//

	// Create a new container
	// Parameter:
	// - filename: filename of the container
	// - password: password for accessing the new container
	virtual int CreateContainer(const char *filename,  const char *password) = 0;

	// Open an existing container
	// Parameter:
	// - filename: filename of the container
	// - password: password required for accessing the container
	// - mode: unknown, maybe for read and write access
	virtual int OpenContainer(const char *filename, const char* password, int mode) = 0;

	// Close the current container
	// No Parameter
	virtual int CloseContainer(void) = 0; //


	// Returns 1, if this instance has opened a container
	virtual int IsOpen(void) = 0; //



	virtual int CloseAllFiles(void) = 0; //Similar in both implementations

	// Returns the MainModule-handle
	virtual HMODULE MainModuleHandle(void) = 0;


	virtual int Function_9(int) = 0; //CPFileManager returns -1


	//
	// Files
	//

	
	// Open a file inside the container using a path
	// Parameter:
	// - filename: filename, relative to current dir or absolute path inside archive
	// - access: 0 for open-existing, 0x80000000 for open and share_read, 0x40000000 for create_always
	// - unknown: not used for original CPFileManager
	// Return:
	// Handle of opened file (can be any number or pointer) or -1 if opening is was unsuccessful
	virtual int Open(const char *filename, int access, int unknown) = 0; //


	// Open a file inside the container using the CJArchiveFm-class
	// Parameter:
	// - fm: A valid pointer to the CJArchiveFm-class
	// - filename: filename, relative to current dir or absolute path inside archive
	// - access: 0 for open-existing, 0x80000000 for open and share_read, 0x40000000 for create_always
	// - unknown: not used for original CPFileManager
	virtual int Open(CJArchiveFm* fm, const char *filename, int access, int unknown) = 0;

	virtual int Function_12(void) = 0; //return -1
	virtual int Function_13(void) = 0; //return 0

	virtual int Create(const char* filename, int unknown) = 0; //
	virtual int Create(CJArchiveFm * fm, const char * filename, int unknown) = 0; //
	

	// Delete a file by name
	// Parameter:
	// - filename: name of file to delete
	virtual int Delete(const char *filename) = 0; //

	// Close file by handle
	// Parameter:
	// hFile: Any handle or pointer identifiying this file
	virtual int Close(int hFile) = 0; //

	// Read a number of bytes from file
	// Parameter:
	// hFile: Any handle or pointer identifiying this file
	// lpBuffer: pointer to reserved memory for read operation
	// nNumberOfBytesToWrite: size of lpBuffer
	// lpNumberOfBytesWritten: pointer to memory, will contain the number of bytes read from the file
	virtual int Read(int hFile, char* lpBuffer, int nNumberOfBytesToRead, unsigned long *lpNumberOfBytesRead) = 0;

	// Write a number of bytes to file
	// Parameter:
	// hFile: Any handle or pointer identifiying this file
	// lpBuffer: pointer to reserved memory for read operation
	// nNumberOfBytesToWrite: size of lpBuffer
	// lpNumberOfBytesWritten: pointer to memory, will contain the number of bytes written to the file
	virtual int Write(int hFile, const char* lpBuffer, int nNumberOfBytesToWrite, unsigned long *lpNumberOfBytesWritten) = 0;




	// The next two functions are bugged, see issue #19
	// Get the full path of the executable
	virtual char* CmdLinePath(void) = 0;

	// Get the executable
	virtual char* CmdLineExe(void) = 0; //




	// Unknown function that gets two variables
	virtual shit_t* getShit(shit_t* shit) = 0; //get shit

	// Unknown function that sets two variables
	virtual int setShit(int a, int b) = 0; //set shit


	//
	// Directory Management
	//

	// Create a new directory in the container
	// Parameter:
	// - name: name of the directory
	virtual int DirectoryCreate(const char* name) = 0;

	// Delete a directory in the container
	// Parameter:
	// - name: name of the directory
	virtual int DirectoryRemove(const char* name) = 0;


	virtual bool ResetDirectory(void) = 0; //
	virtual bool ChangeDirectory(const char* dirname) = 0; //
	virtual int GetDirectoryName(size_t buffersize, char* Dst) = 0; //


	virtual int SetVirtualPath(const char *path) = 0; //set root
	virtual int GetVirtualPath(char *dest) = 0; //similar on both impl


	//
	// Searching
	//

	// Start a new search for files
	// Parameter:
	// - search: result structure representing a handle for the search
	// - pattern: matching pattern for a list of files
	// - entry: the output-structure for the first entry of the resulting list of files
	virtual searchresult_t* FindFirstFile(searchresult_t* search, const char* pattern, result_entry_t* entry) = 0;

	// Get the next file entry in the search result list
	// Parameter:
	// - search: result structure representing a handle for the search
	// - entry: the output-structure for the next entry of the resulting list of files
	virtual int FindNextFile(searchresult_t* search, result_entry_t* entry) = 0;

	// Free the search result
	// Parameter:
	// - search: result structure representing a handle for the search
	virtual int FindClose(searchresult_t* search) = 0;


	//
	// File Information
	//

	virtual int FileNameFromHandle(int hFile, char* dst, size_t count) = 0;
	virtual int GetFileSize(int hFile, LPDWORD lpFileSizeHigh) = 0;
	virtual BOOL GetFileTime(int hFile, LPFILETIME lpCreationTime, LPFILETIME lpLastWriteTime) = 0;
	virtual BOOL SetFileTime(int hFile, LPFILETIME lpCreationTime, LPFILETIME lpLastWriteTime) = 0;
	virtual int Seek(int hFile, LONG lDistanceToMove, DWORD dwMoveMethod) = 0;

	//
	// Others
	//

	virtual HWND GetHwnd(void) = 0;
	virtual void SetHwnd(HWND) = 0;
	virtual void RegisterErrorHandler(error_handler_t callback) = 0;

	virtual int ImportDirectory(const char *srcdir, const char *dstdir, const char *directory_name, bool create_target_dir) = 0;
	virtual int ImportFile(const char *srcdir, const char *dstdir, const char *filename, bool create_target_dir) = 0;
	virtual int ExportDirectory(const char *srcdir, const char *dstdir, const char *directory_name, bool create_target_dir) = 0;
	virtual int ExportFile(const char *srcdir, const char *dstdir, const char *filename, bool create_target_dir) = 0; // create_target_dir is unused

	// Returns: 0 on found, -1 on not found
	virtual int FileExists(char* name, int flags) = 0; //

	// Shows an open file dialog
	virtual int ShowDialog(DialogData *data);
	virtual int ForeachEntryInContainer(foreach_callback_t cb, const char *filter, void *userstate);
	virtual int UpdateCurrentDirectory(void) = 0; //
	virtual int Function_50(int) = 0; //returns zero in both impl.

	// get the version of this file manager
	virtual int GetVersion(void);

	//prompt error if version mismatch 
	virtual int CheckVersion(int version);

	virtual int Lock(int) = 0;
	virtual int Unlock() = 0;


private:
	void loop_container_content(foreach_callback_t cb, const char *filter, void* userstate);

	
};

Hints and Warnings:

It is developed based on the GFXFileManager in VSRO 1.188. Other versions and locales can work, but its likely they won't. Nonetheless, having the source you can easily make it work for other versions.
Feel free to report bugs on Github (please check if your bug has been reported already, and please put a meaningful description of your problem.)
Feel free to request stuff
I've never seen a good, community driven, silkroad project and I doubt it will ever exist. Anyway, feel free to contribute code on Github.
Please keep in mind: VSRO was compiled on Visual Studio 2005 (Platformtoolset 80). Microsoft does not care about ABI-compatibility between toolsets. While GFXFileManager seems entirely separated from the Main codebase, any newer toolset than 80 might break the DLL. Visual Studio 2010/Toolset 100 seems to work fine till now.
Compiling on "Debug" will break the ABI, no matter what version you are using! Always compile on "Release"!
I've even implemented some of the bugs the original file manager had

Complete Source:

License:

. No warranties whatsoever!

Download: This is a library. There is no use in a compiled version. Download the source on Github.

Have fun.

what about blowfish

florian0 · 04/14/2017, 11:28

Quote:

Originally Posted by xxnukertube

what about blowfish

It's not planned to implement the original filemanager any time soon. There is no gain in recreating a class for the blowfish-encrypted-pk2.
This lib is meant for implementing custom formats in the first place.
If you want to create a wrapper around the original file manager, you can simply use the interface and instanciate the original file manager inside your wrapper.

~~Dev.Bedo~~ · 04/14/2017, 13:42

thank dude

Spidy. · 05/23/2017, 03:02

#Sticked

Snow* · 05/23/2017, 13:37

Quote:

Originally Posted by Spidy.

#Sticked

Wondering why this guide got sticked

Exo · 05/23/2017, 13:51

Quote:

Originally Posted by Snow*

Wondering why this guide got sticked

Because it's a good release. One of the best releases in years. He put efforts in something that is not average.

Snow* · 05/23/2017, 14:18

Quote:

Originally Posted by Exo

Because it's a good release. One of the best releases in years. He put efforts in something that is not average.

I do appreciate his efforts, but it's not a good reason to stick the thread.

Exo · 05/23/2017, 14:20

Quote:

Originally Posted by Snow*

I do appreciate his efforts, but it's not a good reason to stick the thread.

I'd say: This release (good ones) that get buried in the section just because of the lack of skill from the community deserve more light so perhaps more people would be encouraged to contribute. This is development.

~~Kai·~~ · 05/23/2017, 16:36

Quote:

Originally Posted by Spidy.

#Sticked

Maybe making a sticked thread that contains all the good ones threads related to development would be better for showing appreciation for everyone's efforts,
and not leading sticky section to get crowded,
also easier for users to find everything in one place.

ZeonNETWORK · 12/03/2018, 08:40

can you re-upload

florian0 · 12/03/2018, 11:53

Quote:

Originally Posted by ZeonNETWORK

can you re-upload

How can I reupload something thats not even down?