You need to edit some files in the .ini folder... if it's a status effect like a halo or whatever you will need to add stuff into statuseffect.ini (the 5165 version changed the structure but should still be easy enough) There was also I think 3deffect.ini and some other files in there that can help you control your effects (basically links effect id's to specific .dds effect files). You will generally need a .c3 file also (the one that goes along with your .dds file. Find the one your original effect referenced and link off to it)
Then just call the effect from your source.
I haven't done custom skills or anything like that but I made a custom halo (without overwriting an existing one) that way. Shouldn't be tooooo much trouble.