[Release]Advanced Data Converter

[funhacker]

IMPORTANT
Please be sure to download most up to date patch along with application.rar, as the latest patch (2012-05-06) contains a much faster conversion system)


This is an updated and much more flexible data converter than my previous one.
Note though, due to its complete flexibility to the point you could use this for almost anything, it runs much slower than the previous one.

However the previous one is not capable of converting the cq_goods to shop.dat.

Please don't forget to change the config file before using it to match your MySQL settings.

Structure Files

• ItemAddition
• Itemtype
• LevelExp(Not Encrypted)
• MergeInfo
• Shop

How To:

Convert cq_itemtype to Itemtype.dat

• Follow instructions on cq_itemtype table design. Located below.
• Download Application.rar
• Download ItemtypeStructure.rar
• Extract files from rar files.
• Click and drag the "ItemtypeConfig.ini" into the "ConvertData.exe"
• Wait until the program shuts down, you should see it display how many bytes have been written to the file as it proceeds.

Convert cq_goods to Shop.dat

• Follow instructions on cq_goods table design. Located below.
• Download Application.rar
• Download ShopStructure.rar
• Extract files from rar files.
• Click and drag the "ShopConfig.ini" into the "ConvertData.exe"
• Wait until the program shuts down, you should see it display how many bytes have been written to the file as it proceeds.

Table Designs:

cq_itemtype:

 Spoiler

Primary Key = id
Allow Null = NONE
Decimals = 0 FOR ALL

Field Name	|Type		|Length	|Signed/Unsigned|Default Value
----------------+---------------+-------+---------------+-------------
id		|int		|4	|Unsigned	|0
name		|char		|16	|N/A		|Empty String
req_profession	|tinyint	|1	|Unsigned	|0
level		|tinyint	|1	|Unsigned	|0
req_level	|tinyint	|1	|Unsigned	|0
req_sex		|tinyint	|1	|Unsigned	|0
req_force	|smallint	|2	|Unsigned	|0
req_dex		|smallint	|2	|Unsigned	|0
req_health	|smallint	|2	|Unsigned	|0
req_soul	|smallint	|2	|Unsigned	|0
monopoly	|tinyint	|1	|Unsigned	|0
weight		|smallint	|2	|Unsigned	|0
price		|int		|4	|Unsigned	|0
id_action	|int		|4	|Unsigned	|0
attack_max	|smallint	|2	|Unsigned	|0
attack_min	|smallint	|2	|Unsigned	|0
defense		|smallint	|2	|Unsigned	|0
magic_atk_min	|smallint	|2	|Unsigned	|0
dodge		|smallint	|2	|Signed		|0
life		|smallint	|2	|Signed		|0
mana		|smallint	|2	|Signed		|0
amount		|smallint	|2	|Unsigned	|0
amount_limit	|smallint	|2	|Unsigned	|0
ident		|tinyint	|1	|Unsigned	|0
gem1		|tinyint	|1	|Unsigned	|0
gem2		|tinyint	|1	|Unsigned	|0
magic1		|tinyint	|1	|Unsigned	|0
magic2		|tinyint	|1	|Unsigned	|0
magic3		|tinyint	|1	|Unsigned	|0
magic_atk_max	|smallint	|2	|Unsigned	|0
magic_def	|smallint	|2	|Unsigned	|0
atk_range	|smallint	|2	|Unsigned	|0
atk_speed	|smallint	|2	|Unsigned	|0
hitrate		|int		|4	|Unsigned	|0
monster_type	|int		|4	|Unsigned	|0
target		|smallint	|2	|Unsigned	|0
able_mask	|tinyint	|1	|Unsigned	|0
exp_type	|tinyint	|1	|Unsigned	|0
emoney		|int		|4	|Unsigned	|0
official1	|smallint	|2	|Unsigned	|0
official2	|smallint	|2	|Unsigned	|0
official3	|smallint	|2	|Unsigned	|0
official4	|smallint	|2	|Unsigned	|0
official5	|smallint	|2	|Unsigned	|0
official6	|smallint	|2	|Unsigned	|0
official7	|smallint	|2	|Unsigned	|0
official8	|smallint	|2	|Unsigned	|0
official9	|smallint	|2	|Unsigned	|0
official10	|smallint	|2	|Unsigned	|0
official11	|smallint	|2	|Unsigned	|0
soul_value	|int		|4	|Unsigned	|0
description	|char		|128	|N/A		|Empty String

cq_goods

 Spoiler

Primary Key = id
Allow Null = NONE
Decimals = 0 FOR ALL

Field Name	|Type		|Length	|Signed/Unsigned|Default Value
----------------+---------------+-------+---------------+-------------
id		|int		|4	|Unsigned	|0
ownerid		|int		|4	|Unsigned	|0
itemtype	|int		|4	|Unsigned	|0
shoptype	|int		|4	|Unsigned	|2
sorttype	|varchar	|64	|N/A		|EmptyString
itemdes1	|varchar	|64	|N/A		|EmptyString
itemdes2	|varchar	|64	|N/A		|EmptyString
itemdes3	|varchar	|255	|N/A		|EmptyString

Defining Structures:

MainConfig File

Used to define main settings of conversion, such as MySQL login, expected output file size and the source/structure file locations.

Must Contain the Following:

• Heading "[Main]"
• Field and Value for "MySQLLogin"
• Field and Value for "MySQLPassword"
• Field and Value for "MySQLHost"
• Field and Value for "MySQLDatabase"
• Field and Value for "SourceFileName"
• Field and Value for "StructureText"
• Field and Value for "ExpectedFileSize" (can just be set to 0 if not sure)

Example File ("ShopConfig.ini"):

 Spoiler

[Main]
MySQLLogin=test
MySQLPassword=test
MySQLHost=localhost
MySQLDatabase=ACME
SourceFileName=%current_directory/ShopSource.ini
StructureText=%current_directory/ShopStructure.txt
ExpectedFileSize=ShopCount/c|*|3000

Source File

Used to define source fields and their values.

Must Contain the Following (For each field):

• Heading "[FieldName]"
• Field and Value for "Type" - Method used for getting data
• Field and Value for "String" - the 'aquisition' string used for aquiring data

Special characters:
'|' = Separates blocks of string, can be used to input source fields into MySQL strings, or to separate a field name and value. Its use depends on the type of data retrieval being used.

Example File ("ShopSource.ini"):

 Spoiler

[ShopID]
Type=MySQLQuery
String=select distinct ownerid from cq_goods;

[ShopCount]
Type=SourceValue
String=ShopID|Count

[ShopType]
Type=MySQLQuery
String=select shoptype from cq_goods where ownerid = |ShopID/c| limit 1;

[Itemtype]
Type=MySQLQuery
String=select itemtype from cq_goods where ownerid = |ShopID/c|;

[ItemCount]
Type=SourceValue
String=Itemtype|Count

[SortType]
Type=MySQLQuery
String=select sorttype from cq_goods where itemtype = |Itemtype/c| and ownerid = |ShopID/c|;

[ItemDescription1]
Type=MySQLQuery
String=select itemdes1 from cq_goods where itemtype = |Itemtype/c| and ownerid = |ShopID/c|;

[ItemDescription2]
Type=MySQLQuery
String=select itemdes1 from cq_goods where itemtype = |Itemtype/c| and ownerid = |ShopID/c|;

[ItemDescription3]
Type=MySQLQuery
String=select itemdes1 from cq_goods where itemtype = |Itemtype/c| and ownerid = |ShopID/c|;

Structure Text

Used to define the structure of the output file.

Must contain at least 1 line of text, at least 1 group and at least 1 field.

Special Characters used:
'@' = Used after a "DataType" definition
'#' = Used after an "ArraySize" definition
'[' & ']' = used to enclose the name of a 'SourceFieldValue'
'/c' = used to get the currently selected value in the source field
'/n' = used to select and get the next value in the source field
'|' = used before defining the 'WriteCount' which is how many times the field/group is written to the file before moving on
',' = used to separate fields within a group
'-' = used to separate groups within a group
'(' & ')' = used to enclose a 'ConditionalStatement' for a conditional group
'^' = used to separate multiple conditional statements/groups
'{' & '}' = used to enclose the fields/group to use when the conditional statement is true
'!=','==' ,'<=','>=' = used to evaluate within a conditional statement

Example File (ShopStructure.txt):

 Spoiler

uint@[ShopCount/n]|1|1
uint@[ShopID/n]|1,int@0|1,int@0|1,int@0|1,int@0|1,uint@[ShopType/n]|1,uint@[ItemCount/n]|1|[ShopCount/c]-([ShopID/c]!=1207){uint@[Itemtype/n]|[ItemCount/c]|1}^([ShopID/c]==1207){uint@[Itemtype/n]|1,chararray@64#[SortType/n]|1,chararray@64#[ItemDescription1/n]|1,chararray@64#[ItemDescription2/n]|1,chararray@256#[ItemDescription3/n]|1|[ItemCount/c]}

Patches:

Applying Patches:
Download all patches,
Extract in order of date (Patch[YYYY-MM-DD])
Override existing files. - Note if you made any custom changes to structure files, backing up is suggested.

2012-04-20
Files: ConvertData.exe, ShopConfig.ini, ShopSource.ini

ConvertData.exe
Changed how the ConvertData.exe loads the config file. Primarily the "ExpectedFileSize" value now allows for more advanced math than before.
The "ExpectedFileSize" still only calculates left to right without any order of precedence but it now allows for bracketed math. An example is how the Shop.dat size is calculated: 4|+|(ShopCount/c * 28)|+|(NonMallItemCount/c * 4)|+|(MallItemCount/c * 452)
This could not be achieved in the previous version.
Fixed a minor error in the /n method of source fields. (May cause future problems)
Changed the display of the command prompt, updating is also done every second now instead of 200times/second, this gives a very slight performance increase.

Screenshot of new status screen:

 Spoiler

ShopConfig.ini
Updated the "ExpectedFileSize" value which will calculate the file size correctly, even if you have duplicate entries.
An example is if you have:
id-itemtype-ownerid
1-190000-1207
2-190000-1207

ShopSource.ini
Included some new fields that were needed for the "ExpectedFileSize" field of the newly updated ShopConfig.ini.
Updated some existing fields, to prevent duplicate entries.

NewFields:

 Spoiler

NonMallItem

 Spoiler

MySQLQuery

select 
distinct ownerid,itemtype 
from cq_goods 
where ownerid != 1207;

MallItem

 Spoiler

MySQLItem

select
distinct itemtype
from cq_goods
where ownerid = 1207;

NonMallItemCount

 Spoiler

SourceValue

NonMallItem|Count

MallItemCount
SourceValue

 Spoiler

MallItem|Count

UpdatedFields:

 Spoiler

Itemtype

 Spoiler

MySQLQuery

select 
distinct itemtype,ownerid
from cq_goods
where ownerid = |ShopID/c|;

2012-05-06
Files: ConvertData.exe, DataConverterLibrary.DLL, ShopSource.ini, ItemtypeSource.ini

ConvertData.exe
Made it so the application doesn't close immediately after conversion was completed. You must push a button to exit once done, this was implemented since now conversion is over a thousand times faster.

Screenshot of new status screen (Creating a 45MB shop.dat from cq_goods with 101,364 entries in just 9seconds!)

 Spoiler

DataConverterLibrary
Added new "Enum" - UpdateFrequency
This determines how often the "AquisitionString" should be called, if "None" it will be called once then all the required data will remain in memory, such types like this will be "ShopCount" or "ShopID".
If "Continuous" it will be called each and every time the field is accessed, this will have a heavy load on the cpu if repeated too much.
If "EachRetrieval" it will be called each time the /n key is used on a field, this will have a moderate load on the cpu if repeated too much.

ShopSource.ini
Added a new field, and updated all existing fields, this should be as optimised as possible for speed.

NewFields:

 Spoiler

Items

 Spoiler

Continuous-MySQLQuery

select 
distinct itemtype,ownerid
from cq_goods
where ownerid = |ShopID/c|;

Updated Fields:

 Spoiler

ShopID

 Spoiler

None-MySQLQuery

select distinct ownerid from cq_goods order by ownerid, itemtype;

ShopCount

 Spoiler

None-SourceValue

ShopID|Count

NonMallItem

 Spoiler

None-MySQLQuery

select distinct ownerid,itemtype from cq_goods where ownerid != 1207;

MallItem

 Spoiler

None-MySQLQuery

select distinct itemtype from cq_goods where ownerid = 1207 order by itemtype;

NonMallItemCount

 Spoiler

None-SourceValue

NonMallItem|Count

MallItemCount

 Spoiler

None-SourceValue

MallItem|Count

ShopType

 Spoiler

None-MySQLQuery

select distinct shoptype,ownerid from cq_goods order by ownerid;

Itemtype

 Spoiler

None-MySQLQuery

select distinct itemtype,ownerid from cq_goods order by ownerid, itemtype;

ItemCount

 Spoiler

EachRetieval-SourceValue

Items|Count

SortType

 Spoiler

None-MySQLQuery

select distinct sorttype,itemtype from cq_goods where ownerid = 1207 order by itemtype;

ItemDescription1

 Spoiler

None-MySQLQuery

select distinct itemdes1,itemtype from cq_goods where ownerid = 1207 order by itemtype;

ItemDescription2

 Spoiler

None-MySQLQuery

select distinct itemdes2,itemtype from cq_goods where ownerid = 1207 order by itemtype;

ItemDescription3

 Spoiler

None-MySQLQuery

select distinct itemdes3,itemtype from cq_goods where ownerid = 1207 order by itemtype;

2012-05-11
Files: ConvertData.exe, DataConverterLibrary.DLL, ItemtypeConfig.ini, ItemtypeSource.ini, ItemtypeStructure.txt, ShopConfig.ini, ShopSource.ini, ShopStructure.txt, ItemAdditionConfig.ini, ItemAdditionSource.ini, ItemAdditionStructure.txt
Folders: Output, StructureFiles, SourceFiles

ConvertData.exe
Capable of converting multiple files at once now. Just highlight a group of Config files, click and drag onto the exe. Then wait until all are finished, hit any key.
More threading enabled, making multiple conversions even faster now.
Capable of saving files to preferred file names, as per the config "OutputFileName" value.

Screenshot of new status screen (Creating 3 new files, using ItemAdditionConfig/ItemtypeConfig/ShopConfig)

 Spoiler

DataConverterLibrary
Changed the MySQL handling to allow for multiple connections, needed to handle multiple simulataneous file conversions.

ItemAddition Structure Files
Added a conversion structure for cq_itemaddition -> ItemAddition.ini, should work with current table design.

New Folders
Added for file management, updated config files accordingly.

[Sir*DeviL]

Nice Release GJ bro!

[DyjgK64J451Jhv0]

good one , keep going would be nice to release one for conquer too

[funhacker]

Originally Posted by DyjgK64J451Jhv0

good one , keep going would be nice to release one for conquer too

It will work for conquer, just need to make the structure files.

New Patch Available!

2012-04-20
Files: ConvertData.exe, ShopConfig.ini, ShopSource.ini

ConvertData.exe
Changed how the ConvertData.exe loads the config file. Primarily the "ExpectedFileSize" value now allows for more advanced math than before.
The "ExpectedFileSize" still only calculates left to right without any order of precedence but it now allows for bracketed math. An example is how the Shop.dat size is calculated: 4|+|(ShopCount/c * 28)|+|(NonMallItemCount/c * 4)|+|(MallItemCount/c * 452)
This could not be achieved in the previous version.
Fixed a minor error in the /n method of source fields. (May cause future problems)
Changed the display of the command prompt, updating is also done every second now instead of 200times/second, this gives a very slight performance increase.

Screenshot of new status screen:

 Spoiler

ShopConfig.ini
Updated the "ExpectedFileSize" value which will calculate the file size correctly, even if you have duplicate entries.
An example is if you have:
id-itemtype-ownerid
1-190000-1207
2-190000-1207

ShopSource.ini
Included some new fields that were needed for the "ExpectedFileSize" field of the newly updated ShopConfig.ini.
Updated some existing fields, to prevent duplicate entries.

NewFields:

 Spoiler

NonMallItem

 Spoiler

MySQLQuery

select 
distinct ownerid,itemtype 
from cq_goods 
where ownerid != 1207;

MallItem

 Spoiler

MySQLItem

select
distinct itemtype
from cq_goods
where ownerid = 1207;

NonMallItemCount

 Spoiler

SourceValue

NonMallItem|Count

MallItemCount
SourceValue

 Spoiler

MallItem|Count

UpdatedFields:

 Spoiler

Itemtype

 Spoiler

MySQLQuery

select 
distinct itemtype,ownerid
from cq_goods
where ownerid = |ShopID/c|;

[DEADP00L]

Nice release Fun, Glad to see there are still one or two people with an IQ releasing things that they create from scratch rather than "zomg a pet from another client lets release it:

[nazeila84]

i cant use that application...im using netframwork 3.5 n 4.0

[funhacker]

New Patch! 2012-05-11, allows for multi-threading and multiple file conversions all at once!

Originally Posted by nazeila84

i cant use that application...im using netframwork 3.5 n 4.0

Did you drag the "Config" file into the exe or are you trying to open the exe?


--------------------------------------------------------------------------------------

2012-05-06

 Spoiler

Files: ConvertData.exe, DataConverterLibrary.DLL, ShopSource.ini, ItemtypeSource.ini

ConvertData.exe
Made it so the application doesn't close immediately after conversion was completed. You must push a button to exit once done, this was implemented since now conversion is over a thousand times faster.

Screenshot of new status screen (Creating a 45MB shop.dat from cq_goods with 101,364 entries in just 9seconds!)

 Spoiler

DataConverterLibrary
Added new "Enum" - UpdateFrequency
This determines how often the "AquisitionString" should be called, if "None" it will be called once then all the required data will remain in memory, such types like this will be "ShopCount" or "ShopID".
If "Continuous" it will be called each and every time the field is accessed, this will have a heavy load on the cpu if repeated too much.
If "EachRetrieval" it will be called each time the /n key is used on a field, this will have a moderate load on the cpu if repeated too much.

ShopSource.ini
Added a new field, and updated all existing fields, this should be as optimised as possible for speed.

NewFields:

 Spoiler

Items

 Spoiler

Continuous-MySQLQuery

select 
distinct itemtype,ownerid
from cq_goods
where ownerid = |ShopID/c|;

Updated Fields:

 Spoiler

ShopID

 Spoiler

None-MySQLQuery

select distinct ownerid from cq_goods order by ownerid, itemtype;

ShopCount

 Spoiler

None-SourceValue

ShopID|Count

NonMallItem

 Spoiler

None-MySQLQuery

select distinct ownerid,itemtype from cq_goods where ownerid != 1207;

MallItem

 Spoiler

None-MySQLQuery

select distinct itemtype from cq_goods where ownerid = 1207 order by itemtype;

NonMallItemCount

 Spoiler

None-SourceValue

NonMallItem|Count

MallItemCount

 Spoiler

None-SourceValue

MallItem|Count

ShopType

 Spoiler

None-MySQLQuery

select distinct shoptype,ownerid from cq_goods order by ownerid;

Itemtype

 Spoiler

None-MySQLQuery

select distinct itemtype,ownerid from cq_goods order by ownerid, itemtype;

ItemCount

 Spoiler

EachRetieval-SourceValue

Items|Count

SortType

 Spoiler

None-MySQLQuery

select distinct sorttype,itemtype from cq_goods where ownerid = 1207 order by itemtype;

ItemDescription1

 Spoiler

None-MySQLQuery

select distinct itemdes1,itemtype from cq_goods where ownerid = 1207 order by itemtype;

ItemDescription2

 Spoiler

None-MySQLQuery

select distinct itemdes2,itemtype from cq_goods where ownerid = 1207 order by itemtype;

ItemDescription3

 Spoiler

None-MySQLQuery

select distinct itemdes3,itemtype from cq_goods where ownerid = 1207 order by itemtype;

2012-05-11
Files: ConvertData.exe, DataConverterLibrary.DLL, ItemtypeConfig.ini, ItemtypeSource.ini, ItemtypeStructure.txt, ShopConfig.ini, ShopSource.ini, ShopStructure.txt, ItemAdditionConfig.ini, ItemAdditionSource.ini, ItemAdditionStructure.txt
Folders: Output, StructureFiles, SourceFiles

ConvertData.exe
Capable of converting multiple files at once now. Just highlight a group of Config files, click and drag onto the exe. Then wait until all are finished, hit any key.
More threading enabled, making multiple conversions even faster now.
Capable of saving files to preferred file names, as per the config "OutputFileName" value.

Screenshot of new status screen (Creating 3 new files, using ItemAdditionConfig/ItemtypeConfig/ShopConfig)

 Spoiler

DataConverterLibrary
Changed the MySQL handling to allow for multiple connections, needed to handle multiple simulataneous file conversions.

ItemAddition Structure Files
Added a conversion structure for cq_itemaddition -> ItemAddition.ini, should work with current table design.

New Folders
Added for file management, updated config files accordingly.

[funhacker]

Update:
New conversion files available. You can now convert your cq_grade table to output a MergeInfo.ini
Download the from this post or the first!

[funhacker]

Update:
New conversion files available. You can now convert your cq_levexp to output an unencrypted Levelexp.dat
Download the from this post or the first!

[D4RK_4NG3L]

Originally Posted by nazeila84

i cant use that application...im using netframwork 3.5 n 4.0

i have the same problem but is the reason that i don't have chinese language installed !!
i still have some items not translated ..
and yes i dragged the "Config" file into the ConvertData.exe

[pengsaizhen]

i follow the tut, but no function,no output.....
plz,,help me...how to fix it...