Register for your free account! | Forgot your password?

You last visited: Today at 17:52

  • Please register to post and access all features, it's quick, easy and FREE!


TQ Custom RLE Ecoded Images

Reply
 
Old   #1
 
elite*gold: 0
Join Date: Jul 2018
Posts: 15
Received Thanks: 9
TQ Custom RLE Ecoded Images

Hello Ya'll, looking for some help which is semi-relevant to CO2, but if this need to go somewhere else, let me know.

If any of you played Monster & Me 2.5 back in the day, the file structure for early MAM and CO are pretty similar. I've been working on a project to revive MAM, and have used quite a bit of the resources and information on this forum to great effect (Thanks everyone!). Unfortunately, I've hit a stone wall for nearly 2 weeks now, and want to see if ya'll have any thoughts or help to provide.

TQ has a semi-custom file extension called RLE, which obviously stands for run-length encoding. It follows standard structure for the encoded image data with packet headers and 2 byte length pixels. That's where the easy part ends. The header of the file doesn't follow any known image header standards. I am able to parse the image data and output an image, but the pixel coloration is completely wrong because it looks to be color-mapped. Unfortunately the color map is some blob of bytes that I just can't figure out how to parse, and without a proper header, it's extremely hard to know how and what to read!

Have any of you come across these before? I can upload samples of the files and the things I've managed to figure out so far if you want to take a look (parts of the header such as w/h and image data offset, the image data decoding algorithm etc).



Jonathis is offline  
Old 07/24/2018, 21:35   #2
 
elite*gold: 12
Join Date: Jul 2011
Posts: 7,006
Received Thanks: 3,358
Is it possible that the image portion is a 2D texture mapping encoded as a DDS file?


Spirited is offline  
Old 07/24/2018, 22:34   #3
 
elite*gold: 0
Join Date: Jul 2018
Posts: 15
Received Thanks: 9
I guess it's certainly possible, but I am not sure how to confirm it yet. My current assumption is that these were just a custom compressed TGA file, since that is what all the other 'sprites' are in the game.

All the image data is split into three files: data, character, pet WDF. I reached out to Cloud Wu (developer of Windsoul++, original engine for MAM CO and other TQ games) and got the full copy with all source code (all the ones posted online are missing any number of pieces). Was able to use that to extract the WDF files, sans file names since the .lst seem to be missing.

Most of data is in TGA/BMP/JPG format. The exception is 99% of character and pet is in this nasty RLE format, which accounts for some 40,000 files!!! There were literally only 2 files in character that were not RLE, and those were in TGA format.

Since this is back from 2001, I don't know that they were using textures inasmuch as converting data to a 2D bitmap. Windsoul++ only seems to handle bitmaps and not as textures, though I'm not completely sure. Reading the source database and packet structure, they actually send sets of Hue/Sat/Bright as a packet to dictate alternate colors, instead of having multiple copies clientside. Not sure if this has any bearing, but I found it at least interesting

I attached a rar of some of the sample files and the results of the RLE decoding of the image data. As you can see, the colors are horribly wrong, so don't mind that It is of note, every RLE file starts with 'ND' for some reason. Maybe it means 'NetDragons', who knows. Usually littleEndian would flip the letters, so maybe its just a bad guess haha.

--------------

BTW, if you want the full Windsoul++ copy, just let me know. it's a free game engine anyways, though quite aged now, so I'd be glad to pass out the correct version Cloud Wu sent me.
Attached Files
File Type: rar RLE.rar (58.7 KB, 10 views)
Jonathis is offline  
Old 07/25/2018, 00:07   #4
 
elite*gold: 12
Join Date: Jul 2011
Posts: 7,006
Received Thanks: 3,358
And it's not a wsprite from Windsoul++?
Code:
/*
 "               ..;;;;,                     ;;;,    
 '           ..;;;"` ;;:           ,;;;;;: ,;;;:.,;..          _/
 `     ,;..,;;"`    :;;'            :;;"`,;;;;;;"":;;         _/ 
       ;;;"  `:;;. ;;'           ..;;:  .;;,.;;:',;;"    _/_/_/_/_/
      .;;`   ,;;" .;"          ,;;;;;;" :;`',;",;;"         _/
     ,;;,:.,;;;  ,;:          :" ,;:` , `:.;;;;;'`         _/   
     ;;"'':;;:. .;; .          ,;;;,;:;;,;;;, ,;             _/
    :;;..;;;;;; :;' :.        :;;;"` `:;;;;;,;,.;.          _/
  .;;":;;`  '"";;:  ';;       '""   .;;`.;";:;;;;` ;,  _/_/_/_/_/
 ;;;" `'       "::. ,;;:          .;"`  ::. '   .,;;;     _/ 
 ""             ';;;;;;;"        ""     ';;;;;;;;;;`     _/
 
                        Windsoul++

                by 云风 (Cloud Wu)  1999-2001
				
		http://member.netease.com/~cloudwu 
 		mailto:
 
		请阅读 readme.txt 中的版权信息
		See readme.txt for copyright information.

		Description:		RLE Sprite 头文件
 		Original Author:	Cloud wind
		Authors:
		Create Time:		2001/6/1
		Modify Time:		2001/11/12

.:*W*:._.:*I*:._.:*N*:._.:*D*:._.:*S*:._.:*O*:._.:*U*:._.:*L*:._.:*/

#ifndef _WINDSOUL_SPRITE_H
#define _WINDSOUL_SPRITE_H

#include "wsaddon.h"

#define _ERR_RLESPRITE 'r'
#define WS_RLESPRTIEFILEERR MAKE_ERR(_ERR_RLESPRITE,'FLE')
#define BLIT_ALTERNATE		0x1
#define BLIT_PALETTE		0x5
#define BLIT_EDGE			0x6
 
typedef WPixel* WPalette;
  
struct WSpriteStruct;  

class WINDSOULADD_API WSprite : public WObject {
public: 
	WSprite() {}
	WSprite(WPalette pal,void *data) { Create(pal,data); }
	WSprite(const WSprite& sprite);
	~WSprite();
  
	void Create(const WPalette pal, void *data);   
	
	int GetW() const;
	int GetH() const;
	BLIT_FUNC DrawFunc(DWORD style) const;	// 获得 Draw 函数指针
	WObjStruct *Duplicate(WObjectHeap *heap) const;
	WSpriteStruct *GetSpriteStruct() const { return reinterpret_cast<WSpriteStruct *>(objdata); }	// 获取数据结构
	void SetUserData(DWORD userdata) const;	// 设置用户数据
	bool IsCover(int x,int y) const;		// 判断是否遮挡
	bool IsCover(WPoint p) const { return IsCover(p.x,p.y); }
	const WSprite& operator()(DWORD userdata) const { SetUserData(userdata); return *this; }
	const WSprite& operator()(WPalette addpal) const { SetUserData((DWORD)addpal); return *this; }
	WSERR Load(const char *filename,int *size=0);	// 读文件

	void *Zoom(int w,int h,int bSpecialPal) const;
};

class WINDSOULADD_API WSpritePak {
	void *m_Data;   
	int m_Direction;
	int m_Width;
	int m_Height;
	int m_Kx;
	int m_Ky;
	WPalette m_Pal;
	mutable WPalette m_AddPal;
	DWORD m_Size;
protected:
	int m_Frame;
	WSprite *m_Spt;
public:
	enum { GreyPal=0, RedPal=1, GreenPal=2, BluePal=3, YellowPal=4, BlackPal=5 };
	WSpritePak() : m_Spt(0),m_AddPal(0),m_Size(0) {}
	virtual ~WSpritePak() { Destroy(); }
	WSERR Destroy();								// 销毁
	bool IsValid() const { return (m_Spt!=0); }		// 判断是否有效
 	const WSprite* operator[](int dir) const { ASSERT(IsValid()); return m_Spt+dir*m_Frame; }
	WSERR Load(const char *filename,int skip=0,int size=0);	// 读文件
	void InitAdditionalPal() const;					// 初始化附加 Pal
	int GetDirection() const { return m_Direction; }// 读方向数
	int GetFrame() const { return m_Frame; }		// 读帧数
	int GetWidth() const { return m_Width; }		// 读宽度
	int GetHeight() const { return m_Height; }		// 读高度
	int GetX() const { return m_Kx; }				// 读X
	int GetY() const { return m_Ky; }				// 读Y
	DWORD GetSize() const { return m_Size; }
	const WPalette LightPal(int level) const;		// 返回亮度 Pal
	const WPalette ColorPal(int color) const;		// 返回彩色 Pal
private:
	virtual void ReadAdditional(void *header) {}		// 读附加信息
};
 
class WINDSOULADD_API WSpriteAni : public WSpritePak {
	int *m_Delay;
	mutable int m_CurrentFrame;
	mutable int m_DelayFrame;
public:
	struct Pair {
		short current;
		short delay;
	};
	WSpriteAni();
	~WSpriteAni();
	void Set(int frame=0);
	void Set(Pair &pair,int frame=0) const;
	int TotalFrame() const;
	const WSprite& NextFrame(bool *change=0) const;
	const WSprite& CurrentFrame() const {
		return m_Spt[m_CurrentFrame];
	}
	const WSprite& NextFrame(Pair &pair,bool *change=0) const;
private:
	void ReadAdditional(void *header);
};

#endif
Code:
#include "StdAfx.h"   
#include ".\wsprite.h"   
   
#define ERRORBOXE(c) { MessageBox(NULL,c,"提示[wastest]",MB_OK); return; }   
   
CWSprite::CWSprite(void)   
{   
    m_pData      = NULL;   
    m_pFrameHead = NULL;    
    m_pPal       = NULL;   
}   
   
CWSprite::~CWSprite(void)   
{   
    if(m_pData)   
    {   
        delete m_pData;   
        m_pData      = NULL;   
        m_pFrameHead = NULL;   
        m_pPal       = NULL;   
    }   
}   
   
void CWSprite::SetData(const char *filename)   
{   
    if(filename==NULL)   
        ERRORBOXE("文件名错误!");   
   
    WORD wastitle;          //文件标示 SP   
    WORD szHheard;          //文件头大小   
    long filesize;          //文件大小   
    int datasize;           //数据区大小   
    FILE *pf;   
   
    // 打开文件   
    pf=fopen(filename,"rb");   
    if(!pf)   
        ERRORBOXE("打开文件错误!");   
   
    // 获取文件大小   
    fseek(pf,0,SEEK_END);   
    filesize=ftell(pf);   
   
    // 检测文件是否是was文件   
    fseek(pf,0,SEEK_SET);   
    fread(&wastitle,2,1,pf);   
    if(wastitle!='PS')   
    {   
        fclose(pf);   
        ERRORBOXE("文件不是was动画文件!");   
    }   
   
    // 获取文件头的大小   
    fread(&szHheard,2,1,pf);   
   
    // 设置数据区   
    datasize=(int)(filesize-szHheard-4);   
    if(datasize<0)   
    {   
        fclose(pf);   
        ERRORBOXE("错误的文件!");   
    }   
    m_bIsLoading = 1;   // 开始加载数据   
    if(m_pData!=NULL)   
    {   
        delete m_pData;   
        m_pData      = NULL;   
        m_pFrameHead = NULL;   
        m_pPal       = NULL;   
        m_pLine      = NULL;   
        m_pLineData  = NULL;   
    }   
    if((m_pData = new BYTE[datasize])==0)   
    {   
        fclose(pf);   
        m_bIsLoading = 0;   
        ERRORBOXE("分配内存失败!");   
    }   
   
    // 取动画头信息   
    fread(&washead,sizeof(WASFILEHEADER),1,pf);   
    char buf[1024];   
    sprintf(buf,"动画的方向数:%d\n\r每个方向的帧数:%d\n\r动画的宽度:%d\n\r动画的高度:%d\n\r动画的中心 X:%d\n\r动画的中心 Y:%d\n\r",   
        washead.nDirection,washead.nFrame,washead.nHeigh,washead.nWidth,washead.xCenter,washead.yCenter);   
    MessageBox(NULL,buf,"提示",MB_OK);   
   
    // 取动画文件数据   
    fseek(pf,szHheard+4,SEEK_SET);   
    fread(m_pData,datasize,1,pf);   
    fclose(pf);   
   
    // 设置调色板指针   
    m_pPal = (WPixel*)m_pData;   
    m_pFrame = (unsigned int*)(m_pData+512);   
   
    // 结束加载数据   
    m_bIsLoading=0;   
}   
   
//**************************************************   
// 函数名称:DrawPal   
// 函数参数:   
//      pDstData  - 目标数据区    
//      nDstPitch - 目标数据区的行跨度    
// 作用:将本WAS动画的调色板绘制到目标数据区   
// 作者:foxer   
// 时间:2006-09-02   
//**************************************************   
void CWSprite::DrawPal(BYTE* pDstData,int nDstPitch)   
{   
    int x,y,m;   
    WPixel bufdata[20];   
    if(m_bIsLoading==1||m_pData==NULL)   
        return;   
    for(y=0;y<16;y++)   
        for(x=0;x<16;x++)   
        {   
            for(m=0;m<20;m++)   
                bufdata[m]=m_pPal[y*16+x];   
            bufdata[19]=WPixel(255,255,255);   
            for(m=0;m<19;m++)   
                memcpy(pDstData+(y*20+m)*nDstPitch+x*40,bufdata,40);   
            memset(pDstData+(y*20+19)*nDstPitch+x*40,0xFF,40);   
        }   
}   
   
//**************************************************************   
// 函数名:DrawAlpha   
// 函数参数:   
//      pDst      - 目标数据区   
//      nDstPitch - 目标数据区行跨度   
//      cx        - 目标数据区的宽度   
//      cy        - 目标数据区的高度   
//      x         - 需要画到目标的坐标X   
//      y         - 需要画到目标的坐标Y   
//      nDir      - 需要绘制的方向   
//      nFrm      - 需要绘制的帧   
// 作用:将本WAS动画的某方向的某帧绘制到目标数据区的某个坐标点   
// 作者:foxer   
// 时间:2006-09-02   
//**************************************************************   
void CWSprite::DrawAlpha(BYTE* pDst,int nDstPitch,int cx,int cy,int x,int y,int nDir,int nFrm)   
{   
    if(m_bIsLoading==1||m_pData==NULL)   
        return;   
   
    // 设置绘制参数   
    if(nDir>washead.nDirection||nFrm>washead.nFrame)   
        return;   
    int ntmp;   
    int sx,sy,scx,scy;   
    int dx,dy,dcx,dcy;   
    ntmp = m_pFrame[nDir*washead.nFrame+nFrm];   
    if(ntmp==NULL) return;   
    m_pFrameHead = (DATEHEARD*)(m_pData+ntmp);   
    sx=sy=0;   
    scx=m_pFrameHead->Width;   
    scy=m_pFrameHead->Height;   
    dx=x-m_pFrameHead->FramX;   
    dy=y-m_pFrameHead->FramY;   
    dcx=dx+m_pFrameHead->Width;   
    dcy=dy+m_pFrameHead->Height;   
   
    // 裁剪处理   
    if(x-m_pFrameHead->FramX>cx||y-m_pFrameHead->FramY>cy||x+m_pFrameHead->Width-m_pFrameHead->FramX<0||y+m_pFrameHead->Height-m_pFrameHead->FramY<0)   
        return;   
    if(dx<0)     // 左边出来鸟   
    {   
        sx=-dx;   
        dx=0;   
    }   
    if(dy<0)     // 上边出来鸟   
    {   
        sy=-dy;   
        dy=0;   
    }   
    if(dcx>cx)       // 右边出来鸟   
    {   
        scx-=dcx-cx;   
        dcx=cx;   
    }   
    if(dcy>cy)       // 下边出来鸟   
    {   
        scy-=dcy-cy;   
        dcy=cy;   
    }   
    m_pLine=(unsigned int*)(m_pData+ntmp+16);   
    BYTE t,a;   
    WORD px;   
    WORD *dpx;   
    WORD *dstline;   
    int spta,sptb,dpt;   
       
    // 开始绘制   
    for(int m=sy;m<scy;m++)   
    {   
        m_pLineData=(BYTE*)(m_pData+ntmp+m_pLine[m]);   
        dstline=(WORD*)(pDst+(dy+m-sy)*nDstPitch+dx*2);   
        int n=0;   
        while(*m_pLineData!=0x00&&n<scx)   
        {   
            t=*m_pLineData++;   
            if((t&0xC0)==0) // 若bit6~7=00   
            {   
                if(t&0x20)  // 若bit5=1   
                {   
                    if(n<sx)   
                    {   
                        n++;   
                        if(n>=scx) goto NEXTLINE;   
                        m_pLineData++;   
                        continue;   
                    }   
                    t=t&0x1f;   
                    px=m_pPal[*m_pLineData++].color;   
                    spta=(px&0xF81F)+((px&0x07E0)<<16);   
                    dpx=dstline;   
                    px=*dpx;   
                    sptb=(px&0xF81F)+((px&0x07E0)<<16);   
                    dpt=(spta*t+sptb*(32-t))/32;   
                    *dpx=(((dpt&0x7E00000)>>16)+(dpt&0xF81F));   
                    n++;   
                    dstline++;   
                    if(n>=scx) goto NEXTLINE;   
                }   
                else   
                {   
                    t=t&0x1f;   
                    if(n<sx)   
                    {   
                        if(n+t<sx)   
                        {   
                            m_pLineData+=2;   
                            n+=t;   
                            continue;   
                        }   
                        else   
                        {   
                            while(n<sx&&t>0)   
                            {   
                                n++;   
                                if(n>=scx)   
                                    goto NEXTLINE;   
                                t--;   
                            }   
                        }   
                    }   
                    a=*m_pLineData++;   
                    px=m_pPal[*m_pLineData++].color;   
                    spta=(px&0xF81F)+((px&0x07E0)<<16);   
                    while(t>0)   
                    {   
                        dpx=dstline;   
                        px=*dpx;   
                        sptb=(px&0xF81F)+((px&0x07E0)<<16);   
                        dpt=(spta*a+sptb*(32-a))/32;   
                        *dpx=((dpt&0x7E00000)>>16)+(dpt&0xF81F);   
                        t--;   
                        n++;   
                        if(n>=scx) goto NEXTLINE;   
                        dstline++;   
                    }   
                }   
            }   
            else if((t&0xC0)==0x40) // 若bit6~7=01   
            {   
                t=t&0x3f;   
                dpx=dstline;   
                while(t>0)   
                {   
                    if(n<sx)   
                    {   
                        n++;   
                        m_pLineData++;   
                        if(n>=scx) goto NEXTLINE;   
                        t--;   
                        continue;   
                    }   
                    dstline++;   
                    px=m_pPal[*m_pLineData++].color;   
                    *dpx=px;   
                    dpx++;   
                    t--;   
                    n++;   
                    if(n>=scx) goto NEXTLINE;   
                }   
            }   
            else if((t&0xC0)==0x80) // 若bit6~7=10   
            {   
                t=t&0x3f;   
                if(n+t<sx)   
                {   
                    m_pLineData++;   
                    n+=t;   
                    if(n>=scx) goto NEXTLINE;   
                    continue;   
                }   
                else if(n<sx)   
                {   
                    while(n<sx)   
                    {   
                        n++;   
                        if(n>=scx) goto NEXTLINE;   
                        t--;   
                    }   
                }   
                px=m_pPal[*m_pLineData++].color;   
                   
                while(t>0)   
                {   
                    dpx=dstline;   
                    *dpx=px;   
                    t--;   
                    n++;   
                    if(n>=scx) goto NEXTLINE;   
                    dstline++;   
                }   
            }   
            else        // 若bit6~7=11   
            {   
                t=t&0x3f;   
                while(n<sx&&t>0)   
                {   
                    n++;   
                    if(n>=scx) goto NEXTLINE;   
                    t--;   
                }   
                n+=t;   
                if(n>=scx) goto NEXTLINE;   
                dstline+=t;   
            }   
        }   
NEXTLINE:;   
    }   
}   
   
int CWSprite::GetDirection()   
{   
    return washead.nDirection;   
}   
   
int CWSprite::GetFrame()   
{   
    return washead.nFrame;   
}


Spirited is offline  
Old 07/25/2018, 02:09   #5
 
elite*gold: 0
Join Date: Jul 2018
Posts: 15
Received Thanks: 9
The file header is of a different structure, so it doesn't recognize it as a WSprite. Furthermore, the color map data seems to be of variant length whereas the load() for WS seems to expect a finite 512 bytes of colormap data.

Header:
Code:
/*
 "               ..;;;;,                     ;;;,    
 '           ..;;;"` ;;:           ,;;;;;: ,;;;:.,;..          _/
 `     ,;..,;;"`    :;;'            :;;"`,;;;;;;"":;;         _/ 
       ;;;"  `:;;. ;;'           ..;;:  .;;,.;;:',;;"    _/_/_/_/_/
      .;;`   ,;;" .;"          ,;;;;;;" :;`',;",;;"         _/
     ,;;,:.,;;;  ,;:          :" ,;:` , `:.;;;;;'`         _/   
     ;;"'':;;:. .;; .          ,;;;,;:;;,;;;, ,;             _/
    :;;..;;;;;; :;' :.        :;;;"` `:;;;;;,;,.;.          _/
  .;;":;;`  '"";;:  ';;       '""   .;;`.;";:;;;;` ;,  _/_/_/_/_/
 ;;;" `'       "::. ,;;:          .;"`  ::. '   .,;;;     _/ 
 ""             ';;;;;;;"        ""     ';;;;;;;;;;`     _/
 
                        Windsoul++

                by 云风 (Cloud Wu)  1999-2001
				
		http://member.netease.com/~cloudwu 
		mailto:
 
		请阅读 readme.txt 中的版权信息
		See readme.txt for copyright information.

		Description:		RLE Sprite 内部头文件
 		Original Author:	云风
		Authors:
		Create Time:		2001/6/1
		Modify Time:		2001/11/12

.:*W*:._.:*I*:._.:*N*:._.:*D*:._.:*S*:._.:*O*:._.:*U*:._.:*L*:._.:*/

#ifndef _WINDSOUL_SPRITE_INTERNAL_H
#define _WINDSOUL_SPRITE_INTERNAL_H

struct WSpriteStruct : public WObjStruct, memory<WSpriteStruct> {
	int w;									// 宽度
	int h;									// 高度
	BYTE **line;							// 第一扫描行指针
	WPalette pal;							// 调色盘
	DWORD userdata;							// 额外数据
	void *data;								// 数据指针
// 	WSpriteStruct *keyframe;				// 帧间压缩用的 key 帧指针

	WSpriteStruct() {}
	WSpriteStruct(const WSpriteStruct &sprite);
	~WSpriteStruct() {}
	WSpriteStruct& operator=(const WSpriteStruct &sprite);
	using memory<WSpriteStruct>::operator new;
	using memory<WSpriteStruct>::operator delete;
	void *operator new(unsigned size,WObjectHeap *heap) { return WObjStruct::operator new(size,heap); }
#ifndef _BCB
	void operator delete(void *p,WObjectHeap *heap) {};
#endif    
};     
 
struct WSpritePakFileHeader {
//	WORD id;
//	WORD len;	// 从 dir 开始一直到 style
	WORD dir;
	WORD frame;
	WORD width;
	WORD height;
	short kx;
	short ky;
//	WPixel pal[256];
//	DWORD offset[1];
};      

struct WSpriteAniFileHeader : public WSpritePakFileHeader {
	BYTE delay[1];
};

#endif
wsprite.cpp implementation of loading the 'RLE' file
Code:
WSERR WSpritePak::Load(const char *filename,int skip,int size)
{
	struct {
		WORD id;
		WORD len;
	} header;

	WFile f;
	int i,s;
	WSpritePakFileHeader *temp;
	DWORD *offset,offset_tbl;
	if (IsValid()) Destroy();
	m_Size=0;
	if (f.Open(filename)) return WS_RLESPRTIEFILEERR;
	f.Skip(skip);
	f.Read(&header,sizeof(header));
//	f.Read(&temp,12);	// 读出 id, dir, frame, style
	if (header.id!='PS' && header.id!='PA') return WS_RLESPRTIEFILEERR;
	temp=(WSpritePakFileHeader*)_alloca(header.len);
	f.Read(temp,header.len);

	m_Direction=temp->dir;
	m_Frame=temp->frame;
	m_Width=temp->width;
	m_Height=temp->height;
	m_Kx=temp->kx;
	m_Ky=temp->ky;
	ReadAdditional(temp);

	s=m_Direction*m_Frame;
	m_Spt=new WSprite[s];

	offset_tbl=s*sizeof(DWORD);
	offset=(DWORD*)_alloca(offset_tbl);
	
	if (size==0) size=f.GetSize();
//	headersize=sizeof(WSpritePakFileHeader)+(s-1)*sizeof(DWORD);
	m_Size=size-header.len-sizeof(header)-offset_tbl;
	m_Data=WAlloc(m_Size);
	int bytes;
	f.Read(m_Data,512,&bytes);
	if (bytes!=512) goto _error;

	f.Read(offset,offset_tbl,&bytes);
	if (bytes!=(int)offset_tbl) goto _error;

	f.Read((void*)((DWORD)m_Data+512),m_Size-512,&bytes);
	if (bytes!=(int)m_Size-512) goto _error;

	m_Pal=(WPalette)m_Data;

	for (i=0;i<s;i++)  {
		if (offset[i])
			m_Spt[i].Create(m_Pal,(void*)((DWORD)m_Data+offset[i]-offset_tbl));
		else {
			m_Spt[i].Create(0,0);
		}
	}

	return WS_OK;	
_error:
	Destroy();
	return WS_RLESPRTIEFILEERR;
}
Jonathis is offline  
Old 07/25/2018, 03:21   #6

 
elite*gold: 0
Join Date: Jan 2008
Posts: 1,394
Received Thanks: 1,055
Do you have a screenshot of what is supposed to be the real colors? It can help to map them in the palette of the file and understand the format.
CptSky is offline  
Old 07/25/2018, 04:53   #7
 
elite*gold: 0
Join Date: Jul 2018
Posts: 15
Received Thanks: 9
Quote:
Originally Posted by CptSky View Post
Do you have a screenshot of what is supposed to be the real colors? It can help to map them in the palette of the file and understand the format.
That, my good sir, is an excellent idea! It'll take a touch of hunting, to find an in game asset that matches one of the 30k monster images, but certainly doable. Now I at least have another avenue to start investigating without just staring at the hex editor for hours and trying random ideas.

Update: Seems like CptSky's suggestion was a no go. Pet colors in game are modified by the server provided hue/sat/bright with some sort of range/modifier. Raises the question on what colors these RLEs actually have. I can verify that there is 1 image for all 3 versions of each pet, and the above values are somehow applied to the RLE data. As for what the base image looks like, still a mystery!



Perhaps also of interest is the fact that the start of the image data encoding doesn't actually lineup with the number of pixels in the image. Drawing them from the end of the array results in a properly shaped images (with wrong colors), but that leaves a fair chunk of now decoded bytes following the same RLE syntax, but not used in the image. Some files don't have enough of these to cover the number of pixels, others have nearly 4x as many as the image pixels itself.
Jonathis is offline  
Old 07/27/2018, 17:03   #8
 
elite*gold: 0
Join Date: Jul 2018
Posts: 15
Received Thanks: 9
Starting to think this is a hopeless endeavor. I'd actually be tempted to pay someone to try and figure this out for me so I can focus my energies on other parts of this rebuild project. Surprised I got as far as I did with the image data. I've decoded 90% of the packets, and all the ciphers are functioning. This is the last step to writing a new game client that works with the server. What do ya'll think? Is it worth it? Not really familiar with the trading/black market aspect of the site though I've been trying to read the postings.
Jonathis is offline  
Old 07/27/2018, 18:39   #9
 
elite*gold: 12
Join Date: Jul 2011
Posts: 7,006
Received Thanks: 3,358
Quote:
Originally Posted by Jonathis View Post
Starting to think this is a hopeless endeavor. I'd actually be tempted to pay someone to try and figure this out for me so I can focus my energies on other parts of this rebuild project. Surprised I got as far as I did with the image data. I've decoded 90% of the packets, and all the ciphers are functioning. This is the last step to writing a new game client that works with the server. What do ya'll think? Is it worth it? Not really familiar with the trading/black market aspect of the site though I've been trying to read the postings.
Is being able to open those files? Why not just reuse the existing client and have it connect to your server? Is there a server.dat file you can modify?
Spirited is offline  
Old 07/27/2018, 18:59   #10
 
elite*gold: 0
Join Date: Jul 2018
Posts: 15
Received Thanks: 9
Quote:
Originally Posted by Spirited View Post
Is being able to open those files? Why not just reuse the existing client and have it connect to your server? Is there a server.dat file you can modify?
The project goal is to write an entirely new client as it hasn't been updated in many years (Since like 2001/2~). We have the original game server, account server, autopatch server, and database, and have been freely changing the database with our own tweaks. TQ actually provided the files to us themselves. Unfortunately, no source code and the client is buggy as hell and full of issues which really kills the experience.

We have all the clientside resources extracted from the WDF files, and ini files not packet into WDF. We have most of the packets decoded including the ciphers, cipher seed, and password cipher. The main stop now is that a lot of the game sprites are in these RLE files (Some 50k+ of them! yikes). All the rest are in TGA/BMP/JPG which we can read. Without a way to interpret the image data in the RLEs, we can't rewrite the client because all of the monsters, characters, and animated sprites in game all use this format.

We actually reached out to see if there was any of their prior programmers who could point us in the right direction. No followup as of yet =/
Jonathis is offline  
Old 07/27/2018, 22:35   #11

 
elite*gold: 0
Join Date: Jan 2008
Posts: 1,394
Received Thanks: 1,055
Can you post the code you have right now? And maybe upload more files to find correlation/do tests. Also, if you could find a specific RLE file, with the in-game color and the hue/sat/bri values.

I might take a look if I have some free time. I can't guarantee anything.
CptSky is offline  
Old 07/27/2018, 22:36   #12
 
elite*gold: 0
Join Date: Aug 2010
Posts: 984
Received Thanks: 1,057
Did you or anyone else attempt to reverse engineer the header structures you need from the client?
{ Angelius } is offline  
Old 07/29/2018, 00:00   #13
 
elite*gold: 0
Join Date: Jul 2018
Posts: 15
Received Thanks: 9
I have attached what I have so far as a RAR. Included my cpp code, a handful of RLE samples (all of the same monster), the current image output data I am getting for each, gifs of the actual monster with the proper colors (three variants, each color described in petColor.ini).

The same base RLE is used for each variant of the same pet, and different sets of colors are applied to three different zones of its image data. I assume this has something to do with the hue 32/96/128 values as well as the 8 range. This stays consistent for every single HSB values, though I am not yet clear on how it is implemented or what the original image looks like without the HSB changes.

The first 18 bytes of the file header has consistently followed this format:
bytes[2] "ND"
00
int width
int height
WORD (always 8)
WORD (always 1)
SHORT Image offset from end of file
The rest is all mystery data so far.

Here is my code thus far. Apologies for any lack of style or skill on the implementation >.<
Code:
#include <windows.h>
#include <iostream>
#include <fstream>
#include <sstream>
#include <iomanip>
#include <vector>
#include "FreeImagePlus.h"

using namespace std;

struct tColorRGBA
{
	unsigned char r;
	unsigned char g;
	unsigned char b;
	unsigned char a;
};

struct RLEheader {
	BYTE id[2];
	WORD u1;
	DWORD width;
	DWORD height;
	WORD eight;
	WORD one;
	WORD endOffset;
	WORD zero;
};

bool LoadRLE(string fileName);
tColorRGBA convertBytesToPixel(unsigned char* chunk);
bool write_truecolor_tga(string fileName, vector<tColorRGBA> data, unsigned width, unsigned height);


int main() {
	string fileName = "turtle0";
	LoadRLE(fileName);
	return 1;
}


bool LoadRLE(string fileName) {
	string rleFile = fileName + ".rle";

	RLEheader header;
	unsigned int bytesPerPixel;

	//RLE parts
	unsigned char packetHead;
	unsigned char *chunk;

	tColorRGBA pixel;
	vector<tColorRGBA> pixels;

	ifstream ifs(rleFile, ios::binary | ios::ate);
	ifstream::pos_type pos = ifs.tellg();
	int fileSize = pos;
	
	ifs.seekg(0, ios::beg);
	ifs.read((char*)&header, sizeof(RLEheader));
	int imageOffset = fileSize - header.endOffset;
	int dataSize = fileSize - imageOffset;

	ifs.clear();
	ifs.seekg(imageOffset);
	BYTE *data = new BYTE[dataSize];
	ifs.read((char*)data, dataSize);

	bytesPerPixel = 2;

	int bytesDone = 0;
	//Parse Image Data as packet header followed by 2*count bytes
	while (bytesDone < dataSize) {
		chunk = new unsigned char[2];
		packetHead = data[bytesDone];

		//Run of the same pixel
		if ((packetHead & 0x80)) {
			packetHead &= 0x7F;
			packetHead++;

			chunk[0] = data[bytesDone + 1];
			chunk[1] = data[bytesDone + 2];

			pixel = convertBytesToPixel(chunk);
			for (unsigned char b = 0; b < packetHead; b++) {
				pixels.push_back(pixel);
			}
			bytesDone += bytesPerPixel;
		}
		//Sequence of different pixels
		else {
			packetHead &= 0x7F;
			packetHead++;

			for (unsigned char b = 0; b < packetHead; b++) {
				chunk[0] = data[bytesDone + 1];
				chunk[1] = data[bytesDone + 2];

				pixel = convertBytesToPixel(chunk);
				pixels.push_back(pixel);
				bytesDone += bytesPerPixel;
			}
		}
		bytesDone++;
	}
	ifs.close();
	delete[] data;
	data = NULL;

	write_truecolor_tga(fileName, pixels, header.width, header.height);
	pixels.clear();

	return true;
}


tColorRGBA convertBytesToPixel(unsigned char* chunk) {
	tColorRGBA result;

	result.a = 255 * ((chunk[1] & 0x80) >> 7);
	result.r = (chunk[1] >> 2) & 0x1F;
	result.g = ((chunk[1] << 3) & 0x1C) | ((chunk[0] >> 5) & 0x07);
	result.b = (chunk[0] & 0x1F);

	// Convert 5-bit channels to 8-bit channels by left shifting by three
	// and repeating the first three bits to cover the range [0,255] with
	// even spacing. For example: http://www.ryanjuckett.com/programming/parsing-colors-in-a-tga-file/
	//   channel input bits:        4 3 2 1 0
	//   channel output bits: 4 3 2 1 0 4 3 2
	result.r = (result.r << 3) | (result.r >> 2);
	result.g = (result.g << 3) | (result.g >> 2);
	result.b = (result.b << 3) | (result.b >> 2);

	return result;
}


bool write_truecolor_tga(string fileName, vector<tColorRGBA> data, unsigned width, unsigned height)
{
	int startPos = 0;
	FIBITMAP *bitmap = FreeImage_Allocate(width, height, 32);

	//Most of the images are 'flipped', but not all. method to identify direction not yet found
	int row = height - 1;
	for (unsigned y = 0; y < height; y++) {

		BYTE *bits = FreeImage_GetScanLine(bitmap, row);
		for (unsigned x = 0; x < width; x++) {
			int pixelPos = ((y)* width) + x + startPos;
			tColorRGBA pixel = data.at(pixelPos);

			// Set pixel color
			if (pixel.r == 0 && pixel.g == 0 && pixel.b == 0) bits[3] = 0;
			else bits[3] = 255;
			bits[2] = (char)pixel.r;
			bits[1] = (char)pixel.g;
			bits[0] = (char)pixel.b;

			// jump to next pixel
			bits += 4;
		}
		row--;
	}

	string tgaFile = fileName + ".tga";
	FreeImage_Save(FIF_TARGA, bitmap, tgaFile.c_str(), 0);

	return true;
}

@ { Angelius }
We haven't yet tried to reverse engineer this yet. My reverse engineering skills in this vein are rather rudimentary. I've poked around a bit, but am by no means knowledgeable enough to gleam anything useful. I have started to read up a bit and start teaching myself some of the aspects of it, but I think it'll be a minute before I am quite there. At most I found where the client identifies an RLE file by the ".RLE" extension, but after that it becomes to me a garbled mess! I was hoping to figure this out just by parsing the file data directly, but perhaps I need to bite the bullet in the long run...


I appreciate any bit of input and suggestions ya'll provide. I'm sure it'll crack eventually, despite those pessimistic moments of defeat.
Attached Files
File Type: rar RLE Decode Project.rar (1.48 MB, 11 views)
Jonathis is offline  
Old 08/01/2018, 16:21   #14
 
elite*gold: 0
Join Date: Jul 2018
Posts: 15
Received Thanks: 9
Hey every, just wanted to give you a quick update...

I think I've nearly cracked it! Although my colors are off, the images are finally crisp and clean. The shadows are perfect, and the edges of the monsters are properly opaque. There are two issues to address. First, some of the images are flipped. There is probably a flag somewhere I am missing which dictates the direction of the bitmap. Second and most importantly, my colors are still off. This is likely due to my conversions of HSV to RGB. On that point, let me explain my findings...

1) Color map offset seems to be at about 22. I did a lot of brute force testing and tweaking here, and seem to be getting consistent results with that assumption.

2) The color map is stored in HSV not RGB! This really threw me for a loop, but when I noticed my R values seemed to drop around ranges of 32/96/128 for the turtles, which matches the expected values provided by the server hsv changes, it hit me that the RLE would be in HSV. This makes sense because, if you are going to modify the image anyways, why convert rgb>hsv>rgb and waste processing power, especially in 2001. Save the operations and directly store in HSV.

3) Since the pixel packet is two bytes, what does each byte mean? After playing around, I found one byte represented the alpha/opacity of the pixel. Making this assumption I found one of the bytes resulted in the perfect expected alpha. Everything not the image was clear, the edges of the monster and its shadow somewhat seethrough, and the inner pixels solid.

This is super exciting. Obviously the current colors you will see below are wrong. I am fairly certain the Hue is correct, so I will have to play with my Saturation and Value calculations some more and see what results I get.

Here some samples:


Jonathis is offline  
Thanks
3 Users
Old 08/06/2018, 10:37   #15
 
elite*gold: 0
Join Date: Dec 2012
Posts: 1,615
Received Thanks: 832
This is really interesting. MAM was godlike.


Super Aids is offline  
Thanks
1 User
Reply

Tags
decoding, image, rle, tq



« Packet update class "10014" and name packet "SpawnPlayer" with source alex 3D | hi »

Similar Threads
MLE AND RLE is has been Detected
08/03/2009 - Grand Chase Philippines - 24 Replies
detect na poh ung ating moonlight engine at radical engine kailan poh mag kakamerun ng bagong engine!!!??



All times are GMT +2. The time now is 17:52.


Powered by vBulletin®
Copyright ©2000 - 2018, Jelsoft Enterprises Ltd.
SEO by vBSEO ©2011, Crawlability, Inc.

Support | Contact Us | FAQ | Advertising | Privacy Policy | Abuse
Copyright ©2018 elitepvpers All Rights Reserved.