Defining structure memory with variable size arrays

[funhacker]

So I had finished the data library (library that's used mostly just for IO with files, not processing/drawing anything) and then when I thought there must be a way to convert a block of memory to a byte array so a quick google turned up Pointers and using

[noparse][StructLayout(LayoutKind.Explicit)][/noparse]

However when using the Marshal services to create a byte array or create from a byte array I read you have to use the following for strings/arrays:

[noparse][MarshalAs(UnmanagedType.ByValTStr, SizeConst = SIZE)][/noparse]

Obviously you can only use hard coded values for this, but this presents a problem for an array of tiles as the tiles array can range from 1 entry up to the maximum bytes allowed for a class/struct/array

So I'm wondering would I really have to set a constant size for this array? Wouldn't it work especially since I don't need to know the length as no more data comes after this array.

Also another question related to this is, the Tile object has an optional value "Object Type/ID" so is there an attribute I can use to say not to use the field if there is no value in it?

[Zeroxelli]

Hmm, I haven't had to use it that much, but as far as I know the size has to be a constant. As far as not using up the entire size goes, that may be okay, as long as the rest of the elements are 0/null. But you may want to double check that.

As far as grabbing a byte array from memory, why do you need anything but pointers? You know that the element type is a byte, so can't you just read the specified number of bytes into an array using pointers? If you don't know the length, it becomes slightly more tricky, and you probably would want to check each element to see if it's null, and if so, that would be the end of the array. If you're not careful, you can create a memory leak or corrupt the stack.

[funhacker]

Sorry I did reply again saying:
I read up more, it seems that I don't need to define the length of an array if it's a ref type for example. Is this correct? Since obviously the length of the array is part of the object data (although I wouldn't want the length written)

I also thought maybe

int? objectID; would make it that if there null it won't have any memory in this area allocated otherwise it would, is that correct?

As for why I'm doing this, once I understand it all a little better it will make loading/reading files a lot faster, because I load an array of bytes from a file that matches the same structure defined and parse it getting the class/structure in return, and I can do the reverse by turning my class/structure into a byte array that is defined to match the file format.
Just saves having to write save/load methods for all the types of files TQ incorporates.

[Zeroxelli]

As far as I know, null is more of a placeholder to ensure that the value is "safe" to be reassigned or the likes. Even if the value is null, for an int? you will still have to allocate 4 bytes of memory for the reference and possibly 4 bytes for the value. A nulled variable should not be taken out of the memory until GC makes its way to it, at which point, yes, the entire 8 bytes will be freed.

[funhacker]

Originally Posted by Zeroxelli

As far as I know, null is more of a placeholder to ensure that the value is "safe" to be reassigned or the likes. Even if the value is null, for an int? you will still have to allocate 4 bytes of memory for the reference and possibly 4 bytes for the value. A nulled variable should not be taken out of the memory until GC makes its way to it, at which point, yes, the entire 8 bytes will be freed.

Do you know of any attribute/way of making it so that you can have a variable in a class/structure but it's optional? Like the Object ID in this case.

I'm starting to think I should just expand and make another class that inherits from base tile which has this extra value...

[Zeroxelli]

Originally Posted by funhacker

Do you know of any attribute/way of making it so that you can have a variable in a class/structure but it's optional? Like the Object ID in this case.

I'm starting to think I should just expand and make another class that inherits from base tile which has this extra value...

The best way is probably to use interfaces. Make two interfaces, one of the two containing the member that's optional.

[CptSky]

To define a complete structure with variable fields, they have to be at the end of the structure.

        [StructLayout(LayoutKind.Sequential, Pack = 1)]
        private struct Entry
        {
            public fixed Byte Name[MAX_NAMESIZE];
            public Int16 Amount;
            public Int32 Delay;
            public Int32 LoopTime;
            public Int32 FrameInterval;
            public Int32 LoopInterval;
            public Int32 OffsetX;
            public Int32 OffsetY;
            public Int32 OffsetZ;
            public Byte Unknown1; //ZeroFill?
            public Byte ColorEnable;
            public Byte Level;
            public Byte Unknown2; //ZeroFill?
            public fixed Byte Parts[1];
        };

Parts, being a Byte* ptr will be casted correctly to retrieves all the variable-length parts.

[funhacker]

Originally Posted by Zeroxelli

The best way is probably to use interfaces. Make two interfaces, one of the two containing the member that's optional.

Thanks that's what I ended up going with anyway

Originally Posted by CptSky

To define a complete structure with variable fields, they have to be at the end of the structure.

        [StructLayout(LayoutKind.Sequential, Pack = 1)]
        private struct Entry
        {
            public fixed Byte Name[MAX_NAMESIZE];
            public Int16 Amount;
            public Int32 Delay;
            public Int32 LoopTime;
            public Int32 FrameInterval;
            public Int32 LoopInterval;
            public Int32 OffsetX;
            public Int32 OffsetY;
            public Int32 OffsetZ;
            public Byte Unknown1; //ZeroFill?
            public Byte ColorEnable;
            public Byte Level;
            public Byte Unknown2; //ZeroFill?
            public fixed Byte Parts[1];
        };

Parts, being a Byte* ptr will be casted correctly to retrieves all the variable-length parts.

But when using the Interop services to convert the structure to a byte array will it return the length of an array that's within the structure? (I don't want this)

[CptSky]

Originally Posted by funhacker

[...]But when using the Interop services to convert the structure to a byte array will it return the length of an array that's within the structure? (I don't want this)

Variable-length arrays are kind of hack. But used as a normal feature. The sizeof operator will return the size of the array with the char[1] at the end, not the actual real size. You must precompute the real size to alloc/copy the buffer. So, yes, it'll be the length of the structure.

For copying structure to byte*, you should check my CO2_CORE_DLL where I use them. It'll gives you example.

[funhacker]

Just thought I'd put this class here as it might help someone that comes across this in a search

using System;
using System.Collections.Generic;
using System.Linq;
using System.Runtime.InteropServices;
using System.Text;

namespace MapCore
{
    internal static class ByteManager
    {
        public static byte[] GetBytes(object input)
        {
            //Declare variables
            int size;
            byte[] result;
            IntPtr resultPtr;

            //Set the size of array
            size = Marshal.SizeOf(input);

            //Setup result array
            result = new byte[size];

            //Setup the result pointer
            resultPtr = Marshal.AllocHGlobal(size);

            //Copy the structure to the pointer
            Marshal.StructureToPtr(input, resultPtr, true);

            //Copy the pointer memory into the result array
            Marshal.Copy(resultPtr, result, 0, size);

            //Free up the used memory
            Marshal.FreeHGlobal(resultPtr);

            //return the result
            return result;
        }

        public static T FromBytes<T>(byte[] input)
        {
            //Decalre variables
            int size;
            IntPtr inputPtr;
            T result;

            //Create new instance of result
            result = Activator.CreateInstance<T>();

            //Get a size
            size = Marshal.SizeOf(result);

            //alocate memory
            inputPtr = Marshal.AllocHGlobal(size);

            //copy over the byte array into the pointer
            Marshal.Copy(input, 0, inputPtr, size);

            //Setup the result object
            result = (T)Marshal.PtrToStructure(inputPtr, result.GetType());

            //remove the pointer from memory
            Marshal.FreeHGlobal(inputPtr);

            //return the result
            return result;
        }
    }
}

Basically just converts any object into a byte array and back again
there are no null checks though so best you handle that before calling it xD

Edit
---------------------

Here's an example of me using this to create a DMap from bytes and turn a DMap into bytes

public static byte[] GetBytes(DMapData input)
        {
            //Declare variables
            List<byte>
                result;
            byte[]
                bufferOne,
                bufferTwo;

            //setup result list
            result = new List<byte>();

            //Get the input data
            bufferOne = ByteManager.GetBytes(input);

            //Setup second buffer
            bufferTwo = new byte[bufferOne.Length - 4];

            //Copy over the array
            Array.Copy(bufferOne, bufferTwo, bufferTwo.Length);

            //add buffer to list
            result.AddRange(bufferTwo);

            //Cycle through each tile
            for (int tileIndex = 0; tileIndex < input._tiles.Length; tileIndex++)
            {
                //Add bytes of tile data to list
                result.AddRange(ByteManager.GetBytes(input._tiles[tileIndex]));
            }

            //return result
            return result.ToArray();
        }

        public byte[] GetBytes()
        {
            //return bytes of this structure
            return GetBytes(this);
        }

        public void FromBytes(byte[] input)
        {
            //Declare variables
            byte[]
                buffer;
            DMapData
                tempData;
            int
                position = 0,
                temp,
                size;

            //setup second input array
            buffer = new byte[Marshal.SizeOf(typeof(DMapData))];

            //Copy over array
            Array.Copy(input, buffer, buffer.Length-4);

            //Add to the position
            position += buffer.Length - 4;

            //Get dmap data
            tempData = ByteManager.FromBytes<DMapData>(buffer);

            //Setup tiles array
            tempData._tiles = new ITile[tempData._width * tempData._height];

            //Cycle through each tile
            for (int tileIndex = 0; tileIndex < tempData._tiles.Length; tileIndex++)
            {
                //If there isn't enough bytes for expanded type
                if (input.Length - position - 6 >= 0)
                {
                    //Get the int16 after the 6 bytes (determines if there is an object there)
                    temp = BitConverter.ToInt16(input, position + 6);
                }
                else { temp = 0; }

                //Set size
                size = temp > byte.MaxValue ?
                    Marshal.SizeOf(typeof(TileExpandedData)) :
                    Marshal.SizeOf(typeof(TileData));

                //Set the buffer
                Array.Copy(input, position, buffer, 0, size);

                //Determine type of tile data
                if (temp > byte.MaxValue)
                {
                    //load current tile as an expanded type
                    tempData._tiles[tileIndex] = ByteManager.FromBytes<TileExpandedData>(buffer);
                }
                else
                {
                    //Load current tile as a regular type
                    tempData._tiles[tileIndex] = ByteManager.FromBytes<TileData>(buffer);
                }

                //increase position
                position += size;
            }

            this = tempData;

        }