[Tool] edxNVMViewer Beta

Page 3 of 4

08/01/2009 23:39 pushedx#31

Quote:

Originally Posted by InvincibleNoOB

Drew,Do you provide any license for your code? I'd like to use the nvm.cpp in a commercial project of mine. :)

Thanks, I should have mentioned that but I forgot. All code is released to the public domain and can be used for whatever you want in any way you need. I just can't guarantee support for it or be associated with any liability from using it.

For a "legal" license, it's pretty much like the zlib license but without any additional restrictions:

Quote:

This software is provided 'as-is', without any express or implied
warranty. In no event will the authors be held liable for any damages
arising from the use of this software.

Permission is granted to anyone to use this software for any purpose,
including commercial applications, and to alter it and redistribute it
freely.

That's it. :)

For anyone confused on the last line, "redistribute it freely" means the ability to redistribute it with no strings attached from me. The code is yours now.

One important thing though is any of the 3rd party libraries I've used have their own licenses to be mindful of. I don't use anything that's viral like GPL though. "My code", relating to the NVM stuff, is covered by the license I just described.

08/02/2009 09:49 hack0r89#32

this is very nice drew!
i see it very usefull for pathfinding. so a point on that:
i used A* before on user created map because A* has to know for each point if its usable or not.
i think with your code its possible to automaticly create high detailed maps for this algo.
maybe i've missed a thing but it seems by now its just possible to detect the real unusable parts just by guessing the bounding from the entities.

can you make a part where the whole map is displayed not the real way, but a black-white version with usable terrain? maybe i'd do it on my own with your really great code! but will take me lots of more time than you, because you just know your code.
would be great!

Again: Great work, as always!

08/02/2009 22:57 maxbot#33

Amazing. As usual.

(the fact that this is open-source and the application itself)

08/03/2009 03:43 pushedx#34

Quote:

Originally Posted by hack0r89

maybe i've missed a thing but it seems by now its just possible to detect the real unusable parts just by guessing the bounding from the entities.

Right, that is one of the major flaws I mentioned about this. You really need a BSR/BMS loader to build path data based on the entity geometry to get a full view of the map. You can at least approximate collisions though using the AABB, so that should be "good enough" for coarse collision detection. You just have to remember to take care of the trees like I did. You do get a collision packet in Silkroad, so you can simply find the closest entity and you know which direction you need to move to get around it as well. Something like that might work well for collision correction.

Quote:

can you make a part where the whole map is displayed not the real way, but a black-white version with usable terrain? maybe i'd do it on my own with your really great code! but will take me lots of more time than you, because you just know your code.

The viewer is just a visualizer of the data. The specific terrain drawing code is this:

Code:

// Terrain
{
	float texoffset = 0;
	terrainDisplayList = glGenLists(1);
	glNewList(terrainDisplayList, GL_COMPILE);
	glEnable(GL_TEXTURE_2D);
	glPolygonMode(GL_FRONT_AND_BACK, GL_FILL);
	for(int y = 0; y < 96; ++y)
	{
		for(int x = 0; x < 96; ++x)
		{
			DWORD tid = nvm.GetTextureIndexAt(95 - x, y);
			if(textureMap.find(tid) != textureMap.end())
			{
				glBindTexture(GL_TEXTURE_2D, textureMap[tid]->texture);
			}
			else
			{
				glBindTexture(GL_TEXTURE_2D, 0);
			}

			glBegin(GL_QUADS);
			glTexCoord2f(0.0f + texoffset, 0.0f + texoffset); glVertex3f((x + 0) * 20.0f, nvm.GetHeightAt(96 - x - 0, y + 0), (y + 0) * 20.0f);
			glTexCoord2f(1.0f - texoffset, 0.0f + texoffset); glVertex3f((x + 1) * 20.0f, nvm.GetHeightAt(96 - x - 1, y + 0), (y + 0) * 20.0f);
			glTexCoord2f(1.0f - texoffset, 1.0f - texoffset); glVertex3f((x + 1) * 20.0f, nvm.GetHeightAt(96 - x - 1, y + 1), (y + 1) * 20.0f);
			glTexCoord2f(0.0f + texoffset, 1.0f - texoffset); glVertex3f((x + 0) * 20.0f, nvm.GetHeightAt(96 - x - 0, y + 1), (y + 1) * 20.0f);
			glEnd();
		}
	}
	glBindTexture(GL_TEXTURE_2D, 0);
	glEndList();
}

If you don't want textures, you could just make that code:

Code:

// Terrain
{
	float texoffset = 0;
	terrainDisplayList = glGenLists(1);
	glNewList(terrainDisplayList, GL_COMPILE);
	glEnable(GL_TEXTURE_2D);
	glPolygonMode(GL_FRONT_AND_BACK, GL_FILL);
	for(int y = 0; y < 96; ++y)
	{
		for(int x = 0; x < 96; ++x)
		{
			glBindTexture(GL_TEXTURE_2D, 0);
			glBegin(GL_QUADS);
			glVertex3f((x + 0) * 20.0f, nvm.GetHeightAt(96 - x - 0, y + 0), (y + 0) * 20.0f);
			glVertex3f((x + 1) * 20.0f, nvm.GetHeightAt(96 - x - 1, y + 0), (y + 0) * 20.0f);
			glVertex3f((x + 1) * 20.0f, nvm.GetHeightAt(96 - x - 1, y + 1), (y + 1) * 20.0f);
			glVertex3f((x + 0) * 20.0f, nvm.GetHeightAt(96 - x - 0, y + 1), (y + 1) * 20.0f);
			glEnd();
		}
	}
	glEndList();
}

If you want the entire height map at once, to feed to somewhere else, all you have to do is called the GetHeightMap function, whose code is:

Code:

float const * edxNVM::GetHeightMap() const
{
	return pimpl->nvm.heightmap;
}

So, you already have access to the entire heightmap data. The format is a 97 x 97 array of floats, so you can also refer to the GetHeightAt function for accessing it since it's a one-dimensional array representation of two dimensional data (most efficient to implement!)

That should be all you need really. The drawing code shows how its setup, 20 units between tiles so you'd have to write your own interpolate function to get the height at any position in the terrain that does not fall on a grid line.

08/03/2009 10:49 hack0r89#35

thx 4 answering
what i think is intresting, that all data related to collision detecting is in the pk2 (->clientsided)
i found this by blocking all packets to the client after i made a walk to horizont
result: the char walks on and on and just stoppedat water, trees or buildings

but also the height limits the usable path. do you know, on which height the terrain is no longer usable? or are there special entities to tell the client where it cant walk?

you made your app to use squares of a few pixels. on this map you draw the terrain and entities. the entities can also be rotated so the block more squares.
what i want to do is, make a map of the whole world of squares (in a few parts of course) and check for evry square if there is an entitie, water or the maximum height (or vice versa: get all blocking objects and set the squares to non usable)

the problem will be e.g. the walls: the entitie is above the gate through it...so it will be detected as non usable. so i'd need more info on the collission detection.

and a request: can you comment your structs in your code?
just a little comment what will be filled into it. otherwise its really hard to read and understand

08/26/2009 13:54 pushedx#36

Just a little update. I've been working on rewriting my PK2 API and needed to test it. I updated the edxNVMViewer to use it now. The improvements should be easily seen! I noticed 1/2 less memory usage as well as a noticeable increase in speed when accessing the PK2s.

Attached is a binary package that you can use to compare the old version and the new version. No source is available of the new stuff since it's under heavy development still.

[Only registered and activated users can see links. Click Here To Register...]

edxNVMViewer is still an inactive project right now while I finish up my other active projects, edxSIikroadLoader, edxSilkroadProxy, and a few other things. However, since most of my projects are using the same core code, it's important I update things like the PK2 API when needed to make sure its not bringing down the quality of the projects using it.

I have no ETA on when I'll be doing updates on this project, but it is not by any means "abandoned", just has a very low priority to all of the other things that are more useful to developers that I am working on.

If you run the demo, let me know how it goes! I still have to setup a new pk2 testing demo using the new version, so this is kind of like an early preview test.

09/01/2009 14:57 hack0r89#37

do you have something like a reference what is in a bsr file?
i mean how to parse it
you have some comments there lik" skip over first block"
but i want to learn more about the bsr

09/01/2009 15:50 pushedx#38

Quote:

Originally Posted by hack0r89

do you have something like a reference what is in a bsr file?
i mean how to parse it
you have some comments there lik" skip over first block"
but i want to learn more about the bsr

Nope, that code is setup that way because I haven't parsed the BSR file format. I only figured out that layout for obtaining the AABB briefly looking at them. There are some parts to the file that didn't make sense to me and I couldn't quite get them parsed right so I just did the necessary stuff for the viewer.

Your best bet is just to reverse the file format. Tracing the files data as they are processed in sro_client, that's what I do to reverse the formats. You could also edit them and reimport them into the PK2, but that's a bit time consuming and not always convenient.

09/01/2009 18:25 hack0r89#39

thanks...
but how did you got that there is an AABB?
because for sro its not usefull to have such a BB, because it needs precise collision detection.
i really want to get this collision system. if i was JM i'd included a polygon into the bsr telling the non walkable area...maybe something like that is there...i'll try to find the point where sro decides that the char is stucked...

something else:
i ported your viewer to delphi and made it a 2D view...
it seems to be correct but there are 2 things i'm not sure if they are right.
i attach it.
could you take a look there and tell me if i'm wrong or your 3D view is not that exact?

the problems (use the default area. so just press load and paint):
1) the object "cj_jang_gate" (big one in the middle) is shown in the debug output of yours but not at the terrain...its really big...is this right?
2) same for the etc shop (cj_etc)
3) the trees seems to be wrong...check the positions for the trees near the portal...i set the size to 50*50. rotating them around the middle of the square/rectangle

i asume that the X,Y,Z coords given are the middle of each object, is this right?

09/01/2009 20:25 pushedx#40

Quote:

Originally Posted by hack0r89

thanks...
but how did you got that there is an AABB?
because for sro its not usefull to have such a BB, because it needs precise collision detection.
i really want to get this collision system. if i was JM i'd included a polygon into the bsr telling the non walkable area...maybe something like that is there...i'll try to find the point where sro decides that the char is stucked..

You would need to parse the BMS files for the exact model collision info. The BSR seems to be more like "containers" for the models to help determine faster collision detection. I.e., if the player is not in the AABB of any entities, it can't be colliding with anything, so it's a necessary optimization. As for how I got the AABB, I was just dumping data and then I saw something that really stuck out and it turned out to be the AABB. It was like a lightbulb moment really.

Quote:

1) the object "cj_jang_gate" (big one in the middle) is shown in the debug output of yours but not at the terrain...its really big...is this right?
2) same for the etc shop (cj_etc)
3) the trees seems to be wrong...check the positions for the trees near the portal...i set the size to 50*50. rotating them around the middle of the square/rectangle

Nice program!

1, 2. I think mine is just bugged. I had noticed a few coordinate problems in my code and the orientation in the game. I tried to fix most of them, but there are still problems. The thing in the middle though looks right (it's that stone floor you walk on!)
3. I noticed a few things seemed wrong in mine as well. It might have to do with specific model settings or additional flags I've not handled yet.

Quote:

i asume that the X,Y,Z coords given are the middle of each object, is this right?

I think that's how they have their models oriented, but I don't know for sure. It would make the most sense though, but it might possibly be at the bottom middle of the mode too. Not too sure yet.

This stuff deals highly in 3D modeling and graphics theory, of which I'm not that well versed in. So I know there are going to be things missing or not done right, but trying to reverse a games propitiatory graphics formats is not easy! It's a lot of trial and error and just playing with stuff until it all looks right. I certainly got started on the right path, but as you can see there's still a lot more to do.

You will want to look at the different "layers/zones" when it comes to collisions, I think they have predefined zones to help reduce collision checks. That would make the most sense for the server to reduce the collisions it has to check each update loop in a region.

09/01/2009 21:36 hack0r89#41

i added a coord calculation to my programm. it seems you are totally right with evrything:
the edges of the objects ingame are exactly what i get in my program:
--> rotation, origin, AABB and position is correct

and i think and hope, that JM calculates collissions only by some space of the ground (a 2D calculation)
i'm currently trying to find the client's collission calculation...
there is something i noticed:
i blocked the stuck packet from server
i blocked the movement of the char (the place where the current position of the char is calculated by the moving speed and angle)
this happens:
i click on move. that char does nothing for some time
than it instantly teleports to the selected target, or to the object at what it would have stucked!
trying to find this...

i found something:
the function where it calculates the current X/Y/Z starts with this:

Code:

0089A370 <sro_clie.test>                   /$  83EC 38                    SUB ESP,38
0089A373                                   |.  53                         PUSH EBX
0089A374                                   |.  56                         PUSH ESI
0089A375                                   |.  8BF1                       MOV ESI,ECX
0089A377                                   |.  33DB                       XOR EBX,EBX
0089A379                                   |.  399E AC000000              CMP [ESI+AC],EBX
0089A37F                                       0F84 00010000              JE 0089A485
0089A385                                   |.  389E 94000000              CMP [ESI+94],BL
0089A38B                                   |.  0F84 E5000000              JE 0089A476
0089A391                                   |.  8B16                       MOV EDX,[ESI]
0089A393                                   |.  8B42 3C                    MOV EAX,[EDX+3C]
0089A396                                   |.  2B86 98000000              SUB EAX,[ESI+98]
0089A39C                                   |.  8B8E 9C000000              MOV ECX,[ESI+9C]
0089A3A2                                   |.  3BC8                       CMP ECX,EAX
0089A3A4                                   |.  73 16                      JNB SHORT 0089A3BC
0089A3A6                                   |.  8BCE                       MOV ECX,ESI
0089A3A8                                   |.  C786 9C000000 F4010000     MOV DWORD PTR [ESI+9C],1F4
0089A3B2                                   |.  E8 F9FCFFFF                CALL 0089A0B0
0089A3B7                                   |.  E9 AF000000                JMP 0089A46B

a bit below you'll find this:

Code:

0089A585                                   |.  3B05 F418D300              CMP EAX,[<Data>]
0089A58B                                   |.  75 05                      JNZ SHORT 0089A592
0089A58D                                   |>  BF 01000000                MOV EDI,1
0089A592                                   |>  8B06                       MOV EAX,[ESI]
0089A594                                   |.  8B0D 1CACF500              MOV ECX,[F5AC1C]                                 ;  sro_clie.00F9BCA0
0089A59A                                   |.  8B11                       MOV EDX,[ECX]
0089A59C                                   |.  8B52 30                    MOV EDX,[EDX+30]
0089A59F                                       50                         PUSH EAX                                         ;  DataPtr
0089A5A0                                       53                         PUSH EBX                                         ;  0
0089A5A1                                       8D4424 38                  LEA EAX,[ESP+38]
0089A5A5                                       50                         PUSH EAX
0089A5A6                                       8D46 4C                    LEA EAX,[ESI+4C]
0089A5A9                                       50                         PUSH EAX
0089A5AA                                       6A 01                      PUSH 1
0089A5AC                                       57                         PUSH EDI
0089A5AD <sro_clie.CheckCol>                   FFD2                       CALL EDX
0089A5AF                                       8BD8                       MOV EBX,EAX
0089A5B1                                       F7C3 00000010              TEST EBX,10000000
0089A5B7                                       5F                         POP EDI
0089A5B8                                       74 15                      JE SHORT 0089A5CF

The function at 0089A5AD (i named the position) is the one that checks for collisions.
it returns 0 if evrything is ok
it returns 1 if a collission is detected (and it sets the current X/Y/Z to the point where it happened)
there also seems to be the height check (i noticed some strings like "getnavmeshHeight")
i tried to reverse it but it is very complicated.
if you have some time to look at that function it will be great

Edit2: I have some updates:
the AABB seems not to be used for collision detection...i changed it to (-1000,-1000,-1000)|(-999,-999,-999) but there is still a collision
also: there are more objects than shown in the navmesh
i parsed the .o2 files of the map folder. but it seems only the objects with a collision are in the nvm

09/05/2009 12:13 hack0r89#42

ok my last answer was a bit long. so another one:
the parsing style cruor is using still works. look at the last page of the forum topic posted on page 2 of this thread
i used this as a basic to parse the bsm.
after the header there is an index:

Code:

    TBMSIndex=record
      vertex,bone,triangle:Cardinal;
      uk1:Array[0..1] of Cardinal;
      stat,uk2,HitBox:Cardinal;
      uk3:Array[0..6] of Cardinal;
    end;

the hitbox contains all points used for collision with and also walking on the object
just after the hitboxes (there is no other index for it) there are some data used for them. they are telling how to connect the points from the hitbox.
i figured out, that sro is creating lines from the object and checks, if the line, the player is walking on, and the line from the object hits-->collision
evry data is right there...
i have a method with that i can create a simple map, that shows the objects with its collidable lines
with this data i should also be able to get the lines, where the object starts. so bridges can also be detected and handled
the problem: very much data is unknown. i got them parsed, so i have the structure of it, but i dont know the meaning of them
i got the hitbox lines by trial and error testing all possible combinations, that seems to be serious and trying to find commonalities and differences

if you want, i'll post the code i have for parsing the bsm and my assumptions what are the lines...
but just if you want and have some time for spending it on this topic, since it will be a bit work to get them all together (have them split in some files in a quick and dirty test project)

BTW: you dont need to mess with the asm i send you:
i found out what it is doing:
it gets the line the player is walking on (sourceX/Y and targetX/Y is calculated before)
it transformes it so the object is at the coord origin and not rotated. than it checks this line against all lines of the object

the problem: the lines of the object are calculated somewhere else and not all are checked each time...e.g. a big tree: just the lines near the player are checked. but dont know where it gets those lines

PS: here is an image:
[Only registered and activated users can see links. Click Here To Register...]

Edit: Next Data i found:
Layer 3 divides the field into lines (layer 3 are just lines, not regions!)
some lines have a value, that blocks the char from passing. (first byte<>4)

09/07/2009 16:34 fanste#43

wow, not bad!

A little question to that:
After i looked for it in the files i have here i was the opinon, that the hitbox is defined in that bms file of an object (most objects are build out of many bms files) that is directly on the ground, but i only find some offsets at this position in that bms file, that is first named in the bsr file.
Is that correct or am i at the wrong position? (i have atm no programm here to test it or to write a lil test-app)

09/07/2009 17:02 pushedx#44

I've not had a chance to look into your posts hack0r, but that's a very nice screenshot showing the outlines! Also just to mention, the AABB doesn't have much use on the 'client side' of things, but I think it's more useful on the 'server side'.

If you consider the geometry for an AABB, it's really simple, just a box, very few verticies relatively speaking and so on. Imagine if you setup your map to simple contain the AABB boxes of all collidable entites. Now, you represent the player with its own AABB as well. As the player moves throughout the world, the main collisions are triggered by the collisions of the AABBs.

Once an AABB is actually triggered, you then perform a second collisions detection between your higher resolution models (bms) and based on that collision, you would then determine if there was a real collision to process or not. That would cut down greatly on resource usage because having a physics world of only boxes is very easy to process.

I was going to look into using [Only registered and activated users can see links. Click Here To Register...] as the collision detection library using these techniques eventually. The idea though, is to create a polygon soup model of the region, and if you used the AABBs of the objects, you can have a lot of them to check against without suffering any inefficiencies since each AABB only has 12 triangles to test against. It's one of the simplest 3D cases to test against!

I'll revisit your posts and this thread later though, right now I am working with some Lua stuff and updating some more of my network code. Nice work so far though :)

09/07/2009 18:50 hack0r89#45

i studied the nvm,bms,bmr files more
and found some things
first: there is a filename of a bsm after the name of the the object in the bsr
if it is there (length>0) than this bms file is used for collisons
if not-->no collisions

the aabb seems never to be used. even on server side it seems to be useless. i think server uses the same code as client...

the nvm zone 1 data: the extraarray you have in your code: each is an index in the entity list of the nvm file...
so client/server only checks in which zone 1 rectangle it is and loads these objects and checks them. there are also some values so it doesnt even have to check all lines of the object. but dont know what value it is and its not usefull for me. so i left it unknown

and i found more: there are lines, that are for collisions
there are lines, that are for entering an object (like a stair or bridge)
and there are triangles that descripe the floor of an object. so the char knows how to walk on

the collisions itself are all checked with no height-->2D check
if the char is on an object, just collisions with this objects lines are checked (go to jangan center and look for the lamps on the grey ground at the side to middle street
you can walk through them ;-) )

the zone 3 data are lines. collision lines with terrain (water, hills, canyons)
and these lines are exactly on the border of the zone1 rectangles

2 new screenshots from the bridge region:
[Only registered and activated users can see links. Click Here To Register...]
[Only registered and activated users can see links. Click Here To Register...]

there is still a few data unknown but i made a short documentation on the 3 file types for own work
will try to bring this to an end now...

PS:
[Only registered and activated users can see links. Click Here To Register...]
(with object triangles)
i highlighted the lamps i mentioned...
check it ;-)

Page 3 of 4