[Release] Less buggy RDB editor

[glandu2]

Some time before, I made a RDB editor but if I remember correctly it was not very stable and didn't supported SQL loading/saving.

This time I optimized this tool to load faster (less than 5sec for db_item.rdb) and added support for reading from a SQL Server (or PostgreSQL if someone use that server for just data mining but I don't think ^^)

I was not able to check if the SQL feature works with Ms SQL Server as I have only PostgreSQL on my slow computer.

To use it, you must load first a DLL that describe the RDB file / SQL table / CSV file you want to load. Then load the file. This allow to add file format without changing anything to the GUI and the underlying reader.

For SQL, you must provide the data source name (use the "configure ODBC" button to configure a data source, the data source name is the first column)

There are also the sources of all Database description DLL in RpzRdbBase-dev.zip, you can make your own using any C compiler (this should not be too hard as all files are pretty much the same but the content about RDB files)

The RappelzRDBToolQt need Qt's dll that are in QtDlls4.8.1.zip if you don't already have it. It contains QtCore4.dll, QtGui4.dll, msvcr100.dll and msvcp100.dll. The GUI save last files locations in RappelzRDBToolQt.ini, to be able to find the last path used to load & save RDB/CSV/DLLs, it's safe to delete it.

For more information about the library, see
The second RpzRdbBase.DLL in this thread (at the end of the first page) should be compatible with this one. (But slower than this one)

Some of RDB description DLL are outdated, feel free to implement one for new 8.1 RDBs (as well as new RDB files).

Also note that only 32 bits DLLs are supported, so you have to compile as a 32 bits DLL, not 64 bits. Same for the ODBC driver, it must be a 32 bits one. (available at be sur to download and install the x86 one)

[Modamer9]

nice tool ... and very fast form other tool editor.

but How I can change structure dll?

[glandu2]

You need a C or C++ compiler for that. (Whatever the compiler name, you can use mingw or VC++ or another compiler, but the provided files were compiled with msvc)

Basicaly, you will always use another file and change it's value.
For example, I will recreate StringDatabase.cpp from MixCategoryDatabase.cpp (which correspond to RDB files db_string.rdb and db_mixcategory.rdb)
Here the code for MixCategoryDatabase.cpp:

 Spoiler

#include "../Base/DataType.h"
#include "../Base/ExportDLL.h"
#include <stdlib.h>
#include <string.h>

#ifdef __cplusplus
extern "C" {
#endif

#pragma comment(linker, "/EXPORT:registerDBStructure=_registerDBStructure@8")
void EDATABASEDLL DLLCALLCONV registerDBStructure(FieldDescriptor **dfmPtr, int *sizePtr) {
	FieldDescriptor df[] =
	   {{1, TYPE_INT16, "id"},
		{1, TYPE_INT8, "mix_id"},
		{1, TYPE_INT32, "local_flag"},
		{1, TYPE_INT16, "high_category_id"},
		{1, TYPE_INT16, "middle_category_id"},
		{1, TYPE_INT16, "low_category_id"},
		{1, TYPE_INT32, "category_text_id"},
		{1, TYPE_INT32, "formal_id"},
		{1, TYPE_INT32, "result_id"}};

	const int size = sizeof(df) / sizeof(FieldDescriptor);

	FieldDescriptor *dfm = (FieldDescriptor*) malloc(sizeof(FieldDescriptor)*size);
	memcpy(dfm, df, sizeof(FieldDescriptor)*size);

	*dfmPtr = dfm;
	*sizePtr = size;
}

#ifdef __cplusplus
}
#endif

Then change the returned structure from the registerDBStructure function.
Actually, the code that describe the db_mixcategory.rdb is that:

FieldDescriptor df[] =
   {{1, TYPE_INT16, "id"},
    {1, TYPE_INT8, "mix_id"},
    {1, TYPE_INT32, "local_flag"},
    {1, TYPE_INT16, "high_category_id"},
    {1, TYPE_INT16, "middle_category_id"},
    {1, TYPE_INT16, "low_category_id"},
    {1, TYPE_INT32, "category_text_id"},
    {1, TYPE_INT32, "formal_id"},
    {1, TYPE_INT32, "result_id"}};

You need only to change that more than 60% of the time. For the other 40% you need also to change that, but you will need to provide some information about columns ordering in the SQL table or default values, or ...

To begin, you know that db_string.rdb contain these field:

int name_string_size;             //This value tell the reader how much bytes to read from the "name" field
int value_string_size;             //Same for "value" field
char name[name_string_size];
char value[value_string_size];
int code;
int group_id;

//Here are unknown field which seems to be always 0. Be we need them too to be able to read and write a full record.
int unknownValue0;
int unknownValue1;
int unknownValue2;
int unknownValue3;

Having that, you can make the FieldDescriptor array.
The first column is the field size.
The second is the flag field. It contains information about the RDB field, if it should be in SQL table, it's type, ...
The third is the name of the field. Use the same name as in the SQL table if you want to be able to read SQL table as well.

So for the fields of db_string:
- The first field tell the size of a string. So it's type is TYPE_VARCHAR_SIZE. Also, we don't want it in the SQL table, so we will need to add TYPE_SQLIGNORE. These to token are flags. You can find them all in DataType.h. This field is just to know the size of a string, so we don't need to set a field name.

Then you have: MAKEINDEXEDVAR(0, 1), TYPE_VARCHAR_SIZE | TYPE_SQLIGNORE, ""
MAKEINDEXEDVAR indicate that the variable has an index, which will be needed when we will describe the associated string which use this field to know it's size. For this first field description, we use 0 as the index.

Use MAKEINDEXEDVAR like that: MAKEINDEXEDVAR(index, size) where size is in number of TYPE_VARCHAR_SIZE. Except for arrays, it's always 1 (it's not the size in byte, but rather the number of element)
Flag are concatenated using the pipe operator, that is |
The name "" is just an empty name as we don't need one.

So for now we have:

FieldDescriptor df[] =
	{{MAKEINDEXEDVAR(0, 1), TYPE_VARCHAR_SIZE | TYPE_SQLIGNORE, ""}};

Another possible and maybe easier way to lay out these data is:

FieldDescriptor df[] = {
	{MAKEINDEXEDVAR(0, 1), TYPE_VARCHAR_SIZE | TYPE_SQLIGNORE, ""}
};

As we only have a double bracket at the begin of the array and at the end. So when adding a new field description, we must not copy them as well, we will see that for the second field.

The second field is almost the same as the first one, except that we need to provide a different index to be able to differentiate them after. So we use 1 as the index.

Then we have:

FieldDescriptor df[] =
	{{MAKEINDEXEDVAR(0, 1), TYPE_VARCHAR_SIZE | TYPE_SQLIGNORE, ""},
	 {MAKEINDEXEDVAR(1, 1), TYPE_VARCHAR_SIZE | TYPE_SQLIGNORE, ""}};

Notice the bracket, we didn't duplicated them all.
Another way is to write that:

FieldDescriptor df[] = {
	{MAKEINDEXEDVAR(0, 1), TYPE_VARCHAR_SIZE | TYPE_SQLIGNORE, ""},
	{MAKEINDEXEDVAR(1, 1), TYPE_VARCHAR_SIZE | TYPE_SQLIGNORE, ""}
};

There, we just have to copy the first line for the first field (but I haven't thought about it)

Then we have a different field, this one is the "name" field which is a string. This string is not a fixed length string, so we don't use an array of character, but rather a "varchar" (as in SQL). So it's type will be TYPE_VARCHAR_STR. This field is also in the SQL table, so we don't provide TYPE_SQLIGNORE. We use "name" as it's name.
There the MAKEINDEXEDVAR indicate different things than before. The index is which field to use to know the string size. The "name" field use the first field (name_string_size) which we have provided an index of 0. So we use here 0 as the index for the "name" field. When the RDB will be read, name_string_size will be used to know the size of "name".
The provided size is 64, as the number provided to varchar when creating a SQL table. The name field was of type "varchar(64)" in my SQL table, so I use the same number (but it could be more or less, provided that all string are not larger than this number)

So we have:

FieldDescriptor df[] =
	{{MAKEINDEXEDVAR(0, 1), TYPE_VARCHAR_SIZE | TYPE_SQLIGNORE, ""},
	 {MAKEINDEXEDVAR(1, 1), TYPE_VARCHAR_SIZE | TYPE_SQLIGNORE, ""},
	 {MAKEINDEXEDVAR(0, 64), TYPE_VARCHAR_STR, "name"}};

Or:

 Spoiler

FieldDescriptor df[] = {
	 {MAKEINDEXEDVAR(0, 1), TYPE_VARCHAR_SIZE | TYPE_SQLIGNORE, ""},
	 {MAKEINDEXEDVAR(1, 1), TYPE_VARCHAR_SIZE | TYPE_SQLIGNORE, ""},
	 {MAKEINDEXEDVAR(0, 64), TYPE_VARCHAR_STR, "name"}
};

The next field, "value" is the same, but the varchar max size is not 64 but 4096. And the index of the field telling the size of the "value" field is the second one which have an index of 1 instead of 0. We also use "value" as it's name.

So we have:

FieldDescriptor df[] =
	{{MAKEINDEXEDVAR(0, 1), TYPE_VARCHAR_SIZE | TYPE_SQLIGNORE, ""},
	 {MAKEINDEXEDVAR(1, 1), TYPE_VARCHAR_SIZE | TYPE_SQLIGNORE, ""},
	 {MAKEINDEXEDVAR(0, 64), TYPE_VARCHAR_STR, "name"},
	 {MAKEINDEXEDVAR(1, 4096), TYPE_VARCHAR_STR, "value"}};

Or:

 Spoiler

FieldDescriptor df[] = {
	 {MAKEINDEXEDVAR(0, 1), TYPE_VARCHAR_SIZE | TYPE_SQLIGNORE, ""},
	 {MAKEINDEXEDVAR(1, 1), TYPE_VARCHAR_SIZE | TYPE_SQLIGNORE, ""},
	 {MAKEINDEXEDVAR(0, 64), TYPE_VARCHAR_STR, "name"},
	 {MAKEINDEXEDVAR(1, 4096), TYPE_VARCHAR_STR, "value"}
};

The next field, "code", is a 32 bits integer. We only have one value (ie: it's not an array) so the size will be 1. We will not use MAKEINDEXEDVAR as we don't need an index for this field (in fact, we use this macro only for TYPE_VARCHAR_SIZE and TYPE_VARCHAR_STR types)
The type will be TYPE_INT32. We also add another flag, TYPE_FLAG_KEY, so when we will create the table in SQL, we will use this field as the primary key, as the code is what Rappelz use to search a value in this table. (that's optionnal)
And we use "code" as the field name

So we have:

FieldDescriptor df[] =
	{{MAKEINDEXEDVAR(0, 1), TYPE_VARCHAR_SIZE | TYPE_SQLIGNORE, ""},
	 {MAKEINDEXEDVAR(1, 1), TYPE_VARCHAR_SIZE | TYPE_SQLIGNORE, ""},
	 {MAKEINDEXEDVAR(0, 64), TYPE_VARCHAR_STR, "name"},
	 {MAKEINDEXEDVAR(1, 4096), TYPE_VARCHAR_STR, "value"},
	 {1, TYPE_INT32 | TYPE_FLAG_KEY, "code"}};

Or:

 Spoiler

FieldDescriptor df[] = {
	 {MAKEINDEXEDVAR(0, 1), TYPE_VARCHAR_SIZE | TYPE_SQLIGNORE, ""},
	 {MAKEINDEXEDVAR(1, 1), TYPE_VARCHAR_SIZE | TYPE_SQLIGNORE, ""},
	 {MAKEINDEXEDVAR(0, 64), TYPE_VARCHAR_STR, "name"},
	 {MAKEINDEXEDVAR(1, 4096), TYPE_VARCHAR_STR, "value"},
	 {1, TYPE_INT32 | TYPE_FLAG_KEY, "code"}
};

Then we have other fields which are all 32 bits integer. The only difference is that we don't declare them as primary keys, so we don't provide TYPE_FLAG_KEY. (There should be always at most one primary key, if there are)

So we have at the end:

FieldDescriptor df[] =
	 {{MAKEINDEXEDVAR(0, 1), TYPE_VARCHAR_SIZE | TYPE_SQLIGNORE, ""},
	 {MAKEINDEXEDVAR(1, 1), TYPE_VARCHAR_SIZE | TYPE_SQLIGNORE, ""},
	 {MAKEINDEXEDVAR(0, 64), TYPE_VARCHAR_STR, "name"},
	 {MAKEINDEXEDVAR(1, 4096), TYPE_VARCHAR_STR, "value"},
	 {1, TYPE_INT32 | TYPE_FLAG_KEY, "code"},
	 {1, TYPE_INT32, "group_id"},
	 {1, TYPE_INT32 | TYPE_SQLIGNORE | TYPE_CSVIGNORE, "unknownValue0"},
	 {1, TYPE_INT32 | TYPE_SQLIGNORE | TYPE_CSVIGNORE, "unknownValue1"},
	 {1, TYPE_INT32 | TYPE_SQLIGNORE | TYPE_CSVIGNORE, "unknownValue2"},
	 {1, TYPE_INT32 | TYPE_SQLIGNORE | TYPE_CSVIGNORE, "unknownValue3"}};

Or:

 Spoiler

[code]FieldDescriptor df[] = {
{MAKEINDEXEDVAR(0, 1), TYPE_VARCHAR_SIZE | TYPE_SQLIGNORE, ""},
{MAKEINDEXEDVAR(1, 1), TYPE_VARCHAR_SIZE | TYPE_SQLIGNORE, ""},
{MAKEINDEXEDVAR(0, 64), TYPE_VARCHAR_STR, "name"},
{MAKEINDEXEDVAR(1, 4096), TYPE_VARCHAR_STR, "value"},
{1, TYPE_INT32 | TYPE_FLAG_KEY, "code"},
{1, TYPE_INT32, "group_id"},
{1, TYPE_INT32 | TYPE_SQLIGNORE | TYPE_CSVIGNORE, "unknownValue0"},
{1, TYPE_INT32 | TYPE_SQLIGNORE | TYPE_CSVIGNORE, "unknownValue1"},
{1, TYPE_INT32 | TYPE_SQLIGNORE | TYPE_CSVIGNORE, "unknownValue2"},
{1, TYPE_INT32 | TYPE_SQLIGNORE | TYPE_CSVIGNORE, "unknownValue3"}
};[code]

We could also compact the last unknown fields, but if we discover what they mean, we will have to reseparated them so I didn't compacted them in an array.
This would be that:

FieldDescriptor df[] =
	 {{MAKEINDEXEDVAR(0, 1), TYPE_VARCHAR_SIZE | TYPE_SQLIGNORE, ""},
	 {MAKEINDEXEDVAR(1, 1), TYPE_VARCHAR_SIZE | TYPE_SQLIGNORE, ""},
	 {MAKEINDEXEDVAR(0, 64), TYPE_VARCHAR_STR, "name"},
	 {MAKEINDEXEDVAR(1, 4096), TYPE_VARCHAR_STR, "value"},
	 {1, TYPE_INT32 | TYPE_FLAG_KEY, "code"},
	 {1, TYPE_INT32, "group_id"},
	 {4, TYPE_INT32 | TYPE_SQLIGNORE | TYPE_CSVIGNORE, "unknownValues"}};

Or:

 Spoiler

[code]FieldDescriptor df[] = {
{MAKEINDEXEDVAR(0, 1), TYPE_VARCHAR_SIZE | TYPE_SQLIGNORE, ""},
{MAKEINDEXEDVAR(1, 1), TYPE_VARCHAR_SIZE | TYPE_SQLIGNORE, ""},
{MAKEINDEXEDVAR(0, 64), TYPE_VARCHAR_STR, "name"},
{MAKEINDEXEDVAR(1, 4096), TYPE_VARCHAR_STR, "value"},
{1, TYPE_INT32 | TYPE_FLAG_KEY, "code"},
{1, TYPE_INT32, "group_id"},
{4, TYPE_INT32 | TYPE_SQLIGNORE | TYPE_CSVIGNORE, "unknownValues"}
};[code]

Where we tell that we have an array of 4 unknown values which are not referenced in the SQL table and we don't want them either when using a CSV file.

(actually the CSV file is more a TSV file, but this extension is not well recognized, so I use CSV. These files are readable with any text editor and you can open them in a spreadsheet program with the tabulation as the separator (nothing else else it will mess up columns. Also be sure to disable any quotes. If you have some rows with more than the expected number of columns, it's probably that you forgot to tell you spreadsheet program to ignore quotes)

Finally, to compile the new file (StringDatabase.cpp here), create a DLL.
With the command line version of msvc2010 (it's probably the same for all msvc versions) it's:
cl.exe -D_CRT_SECURE_NO_WARNINGS -DBUILDING_DATABASE /DWIN32 /D_WINDOWS /W3 /Zm1000 /EHsc /GR /MD /O2 /Ob2 /D NDEBUG <files-to-compile> <files-to-link> /link /DLL /OUT:<desired-dll-name>.dll

Where here <files-to-compile> is only StringDatabase.cpp (for this example, else use the file name you want)
<files-to-link> is RpzRdbBase.lib and <desired-dll-name>.dll is StringDatabase.dll

I will edit this post later to provide more explanation about compiling as a DLL. But if you have an IDE (like codeblocks or MS Visual Studio C++), juste create a new DLL project (dynamic library) and use StringDatabase.cpp as the only file to compile. (There is no main function as it's just a DLL). If your IDE generate a file like main.cpp file with content inside it (for DLL initializatino for example), you can just remove all that code and replace with the code that would be in StringDatabase.cpp.

It's best to name your DLLs as <rdb_type>Database.dll as the GUI show by default all file which end with Database.dll and not the others.

[glandu2]

I forgot to add the databases description DLL sources, so here they are.
I also added a working codeblocks project.

You have to download codeblocks with mingw for the project to work (here: )

You can also use cmake, where the source directory is where .cpp, .h and RpzRdbBase.lib are, and the build directory is a subdirectory (like "build").
to use cmake, configure the source dir and build dir, then click on configure then generate. Now open a command prompt and go to the build directory. then type "nmake" or "make" then enter and wait ^^

Using cmake can be easier if you have a compiler already installed (default location is even better for cmake to recognize it)

[M>M]

nice tool indeed

i have made a dll for the new rdbs with it :P

[glandu2]

You used mingw or Ms VC++ ?

It's also possible to create a DLL that read a text file, to enable reading new RDB without having to create a new DLL, but just a new text file describing it's content. I never done that, but that should not be so hard to do. The only problem that might arise is if some other RDB don't use a flat format. Actually there is one that is not flat that use something like that:

struct {
int sub_struct_number;
struct sub_struct[sub_struct_number]
}

where sub_struct contains the data. (but I don't remember which RDB is made like that)

[edit] I fount it, I was talking about SkillTreeResource

Is there any new RDB using this type of structure instead of plain flat data layout ?

[TealSky]

Perhaps it's different when writing a c++ editor?

But I have not encountered such a thing in my endeavors with editing rdbs, the structure I use for SkillTreeResource is:

    public Int32 job_id { get; set; }
    public Int32 skill_id { get; set; }
    public Int32 min_skill_lv { get; set; }
    public Int32 max_skill_lv { get; set; }
    public Int32 lv { get; set; }
    public Int32 job_lv { get; set; }
    public Single jp_ratio { get; set; }
    public Int32 need_skill_id_1 { get; set; }
    public Int32 need_skill_lv_1 { get; set; }
    public Int32 need_skill_id_2 { get; set; }
    public Int32 need_skill_lv_2 { get; set; }
    public Int32 need_skill_id_3 { get; set; }
    public Int32 need_skill_lv_3 { get; set; }
    public Int32 cenhance_min { get; set; }
    public Int32 cenhance_max { get; set; }

    public void ReadFile(BinaryReader br)
    {
        this.job_id = br.ReadInt32();
        this.skill_id = br.ReadInt32();
        this.min_skill_lv = br.ReadInt32();
        this.max_skill_lv = br.ReadInt32();
        this.lv = br.ReadInt32();
        this.job_lv = br.ReadInt32();
        this.jp_ratio = br.ReadSingle();
        this.need_skill_id_1 = br.ReadInt32();
        this.need_skill_lv_1 = br.ReadInt32();
        this.need_skill_id_2 = br.ReadInt32();
        this.need_skill_lv_2 = br.ReadInt32();
        this.need_skill_id_3 = br.ReadInt32();
        this.need_skill_lv_3 = br.ReadInt32();
        this.cenhance_min = br.ReadInt32();
        this.cenhance_max = br.ReadInt32();
    }

[glandu2]

Ok. It's maybe another one, or GALA changed it's structure to match other ones. (but it's not C++, I talk only about the way data is layered in the RDB file ^^)

It's the way to find how many times to call "ReadFile" in fact that is not the same as other RDBs. In my sample code, ReadFile would only read "sub_struct" element.
__________________________________________________ ______________

As I received some question about how the GUI works, here is some informations:

Opening a file:

To open a RDB file, you need to first open a DLL file that describe the RDB you want to open. For example, to open db_string.rdb, you need to open StringDatabase.dll. In fact, the DLL name are the SQL table name without Resource.

Then you open the RDB file.

It's the same for every type of file (CSV, RDB or opening using a SQL table).

After that you can edit fields inside the GUI or/and save data to RDB/CSV or to a SQL server.

Using a SQL server:

For SQL, the difference is that you don't open a file but a table using a connection to your server.

To be able to use that, you must configure a SQL server native client for ODBC. When configuring a new data source (using odbcad32.exe or click on the button about configuring ODBC in my client), remember it's name. On the SQL configuration window from my GUI, type the data source name under the server type (which will be almost always Microsoft SQL Server) then if you have, an username and password (leave empty if there is no password or no account)

Then in the input dialog about the SQL table, you will need to indicate the full table name, including the database. For example, to open the database dbo.StringResource which is in Telecaster, you will need to enter "Telecaster.dbo.StringResource" (the database name is not needed if it's the same as the data source's default database, but if you are unsure, add the database name)

/!\ When writting to a SQL table, the targeted table is first dropped (all data is discarded), then the data you want to add to the SQL server is inserted. (so if you're unsure about the resulting table, use a table name that is not used (for example to test StringDatabase, you could use dbo.StringResourceTest))

[TealSky]

Originally Posted by glandu2

Ok. It's maybe another one, or GALA changed it's structure to match other ones. (but it's not C++, I talk only about the way data is layered in the RDB file ^^)

It's the way to find how many times to call "ReadFile" in fact that is not the same as other RDBs. In my sample code, ReadFile would only read "sub_struct" element.

Oh in C# the "read_file" is called repeatedly WHILE the hex_file (rdb) is being read. (this is done with WHILE(textReader.Peek() != -1) So as long as the structure is right the "read_file" will be called once per row, then you just tell it per row read add to datagridview.

It's nice to see the contrast between available coding languages, hopefully it will bring more diversity. Thanks for all your work Glandu2 ^^

[glandu2]

I didn't want to mean that, what your are doing is for flat file, all records are after the one before ^^ For these file, with any language you do the same thing (but write it using the word of the language)

But before at least, there was a file that was like that.

Normal flat file are like that:
char header[128];
int record_number;
Record records[record_number];

where you with your function, read un Record at a time and until the end of file (peek() == -1)

But the file I talk about was like that:

char header[128];
int record_number;
MasterRecord records[record_number];

struct MasterRecord {
  int subrecord_number;
  SubRecord record[subrecord_number];
}

Where SubRecord contain the actual data. So you had to have 2 loop to read the file.

I remember well that behavior, as in my RDB editor, I tried to be as modular as possible, and I couldn't make a generic way to read all RDB files including this file without implementing some sort of substructure. So I decided to implement a special way to deal with this file as it's the only one that was like that (implementing a way to deal with substructure would be much complicated, too complicated for just one file ...)

But maybe the db_skilltree.rdb file has changed to be like the other one, that is a normal flat file, thus there is no need to have 2 loop as before.

[edit] I just tested the new db_skilltree.rdb file from official 8.2 client, and it still need a double loop as before, it just has a new column.