Thread: Intialize An Array From A File?

Hello,

I'm wondering if there's a way to be able to initialize an array directly using a binary file, e.g. if I had:-

Code:

unsigned char buffer[4096];

Is there a way of linking 4096 bytes of data from a file rather than specifying each individual byte within the source code? I've noticed that compilation time increases dramatically when I do this as the compiler has to parse each and every byte.

My current workaround is to compile the program without initializing the array, locating the array in the executable output with a hex editor and overwriting the empty space.

I don't mind if there are compiler-specific extensions that you are aware of.

Not sure what you mean. You want to have 4096 bytes put in there from the .exe file?

If so, just open the file and read 4096 bytes into that buffer.

Well that would involve having code in your program to read from a file that you then have to cart around with the executable.

I'm specifically referring to the linker phase of program construction. I would like it so that the only thing about the array that the compiler evaluates is that it is n bytes long (if that is necessary), then the linker should be able to associate the symbol generated by the compiler with the contents of a file.

You could say that the resource data present in Windows executables is an example of this, although that isn't directly accessible to code without using external functions to retrieve it.

No you don't have to cart anything extra around. Just have the executable read itself into the array. If that's what you want. I'm still a little puzzled.

I don't know if the linker will help -- the linker is not really a very sophisticated tool (or I guess if it is I have completely missed it to date).

But this may be a false efficiency: would you rather have this slow down once (at compile time) or every single time you run the program (and reading 4K from a file while the program is running is almost certainly slower than reading 4K from the file that is being compiled during compile-time).

False efficiency or I'm not reading it right. I'm sure retrieving the executable from the disk drive and setting up whatever internals to start it running is way more expensive than reading in 4K at the very beginning of the program. Therefore the 4K read is negligible in the scheme of things.

nonoob, I mean that when the program starts those 4096 bytes are already initialized with data.

Say I wanted to store a 4KB picture. I could have an external .bmp file and read it into memory at run-time, or I could have it so that the executable already contains it and refers to it directly via a pointer without any further set-up or use of any functions.

To expand on what I was saying about Windows resources, the resource compiler (e.g. Microsoft's rc.exe) would pack the bitmap up into a binary resource directory and then the linker would attach this directory inside the executable. This I assume requires that the linker knows what the resource directory is, rather than some random object.

tabstop, I don't really have a problem just sticking it into the source code, but it just seems a bit silly that most of what is supposed to be a human-readable document is taken up by lines and lines of data that no-one needs to see, which then has to be interpreted by the compiler back into its original form.

Hmmm. How about defining a global variable,

extern unsigned char[];

which is properly initialized in another module... ".obj" or ".lib" whatever is used these days. The linker puts them all together.

Indeed. As I am following this thread my Googling has also reached the same conclusion.
Any suggestions for a tool to convert files to linkable .obj files?
I'm half-tempted to look up the COFF format and write one at this stage.

I think any compiler can produce .obj files as its target. Probably does anyhow temporarily. gcc has some command-line flag, but I don't know how to in Visual Studio.

Ok. So you have a 4k picture. You store it in an array format, reading it with a proper program (like matlab). You copy that array into a file. Like image.c. There you just initialize an array like unsigned char image[] = { your array/pic}; Then you have to make your array look like val1, val2, ...., valn so it can be initialized in a C syntax. You can make a simple C program to do that for you. Then you compile the file and have an .o. Then you use that .o with your main program and you are done. I suppose that is what nonoob meant?
I don't think there is a better way to do this in order to gain maximum performance

EDIT: the flac is -c for gcc to make an object (.o). You just put gcc -c my.c and you get a my.o

I'd write a little tool to read a .jpg file or whatever, and generate

Code:

unsigned long image[] = { 0x00000000, 0xFFFFFFFF, ... };

This will be compiled into a separate .obj that's to be linked later.

That's probably the most compact. Yeah it's still a strange workaround to do what you want.

Originally Posted by nonoob

I'd write a little tool to read a .jpg file or whatever, and generate

Code:

unsigned long image[] = { 0x00000000, 0xFFFFFFFF, ... };

This will be compiled into a separate .obj that's to be linked later.

That's probably the most compact. Yeah it's still a strange workaround to do what you want.

Well, it would be a nice practice, but note that reading a compressed image like a jpg isn't that easy. But you could find an appropriate program on the net.
I don't find it strange though. It is EXACTLY what you want. The only bad thing is that you have to make the array in an appropriate format to initialize the array. But there is really no easier way either....
To be more clear. You don't really want to initialize the array from a file, thus from information stored in the HDD. You want to initialize it with information from the RAM. Even more precise, you don't want to initialize the array, you just want to have a pointer to the information you need.

In my years, whenever something like this was needed, we would just write a utility to generate a custom .h and .c file.

Code:

//custom_data.h
extern unsigned char g_custom_data[];
// custom_data.c
unsigned char g_custom_data[] = {0x01, 0x02, 0x03, ..... };

Then compile and link as usual. Using bytes as the data type prevents any endianess issues.

gg

I did actually look up the COFF format and wrote a converter
It's very basic, but from testing with a debug build program it does the trick:-

Code:

// requires Win32 Platform SDK for structs and constants
#define WIN32_LEAN_AND_MEAN

#include <stdio.h>
#include <time.h>
#include <windows.h>

int main(int argc, char **argv)
{
	char szFilename[256], *szPtr;
	int bLong = 0, iData;
	unsigned long ulSize;
	FILE *fpIn, *fpOut;
	IMAGE_FILE_HEADER header;
	IMAGE_SECTION_HEADER section;
	IMAGE_SYMBOL symbol;

	printf("Binary File -> Object File Converter\n");
	if (argc < 3 || argc > 4)
	{
		printf("Usage: bin2obj <filename> <symbol> [output filename]\n");
		return -1;
	}

	fpIn = fopen(argv[1], "rb");
	if (!fpIn)
		return -1;

	fseek(fpIn, 0, SEEK_END);
	ulSize = ftell(fpIn);
	fseek(fpIn, 0, SEEK_SET);
	header.Machine = IMAGE_FILE_MACHINE_I386;
	header.NumberOfSections = 1;
	header.TimeDateStamp = time(NULL); // really necessary?
	header.PointerToSymbolTable = sizeof(header) + sizeof(section);
	header.NumberOfSymbols = 1;
	header.SizeOfOptionalHeader = 0;
	header.Characteristics = 0;
	memset(&section, 0, sizeof(section));
	memcpy(section.Name, ".data", 5);
	section.SizeOfRawData = ulSize;
	section.PointerToRawData = sizeof(header) + sizeof(section) + sizeof(symbol);
	section.Characteristics = IMAGE_SCN_MEM_READ | IMAGE_SCN_MEM_WRITE | IMAGE_SCN_CNT_INITIALIZED_DATA;
	memset(&symbol, 0, sizeof(symbol));
	// determine short/long symbol name storage
	if (strlen(argv[2]) <= 8)
		memcpy(symbol.N.ShortName, argv[2], strlen(argv[2]));
	else
	{
		symbol.N.LongName[1] = 4;
		section.PointerToRawData += strlen(argv[2]) + 5;
		bLong = 1;
	}

	symbol.SectionNumber = 1;
	symbol.StorageClass = IMAGE_SYM_CLASS_EXTERNAL;
	// prepare output filename
	if (argc == 4)
		strcpy(szFilename, argv[3]);
	else
	{
		// append extension to input filename
		strcpy(szFilename, argv[1]);
		szPtr = szFilename + strlen(szFilename) - 1;
		while (szPtr > szFilename && *szPtr != '.')
			szPtr--;

		if (szPtr > szFilename)
			strcpy(szPtr, ".obj");
		else
			strcat(szFilename, ".obj");

	}

	fpOut = fopen(szFilename, "wb");
	if (!fpOut)
		return -1;

	fwrite(&header, sizeof(header), 1, fpOut);
	fwrite(&section, sizeof(section), 1, fpOut);
	fwrite(&symbol, sizeof(symbol), 1, fpOut);
	if (bLong)
	{
		ulSize = strlen(argv[2]) + 5;
		fwrite(&ulSize, 4, 1, fpOut);
		fwrite(argv[2], strlen(argv[2]) + 1, 1, fpOut);
	}

	while ((iData = getc(fpIn)) != EOF)
		putc(iData, fpOut);

	fclose(fpIn);
	fclose(fpOut);
	return 0;
}