Thread: String fun

ok matsp

what is this whole section supposed to do? (Im sorry i thought i was decent at C... but im being blown out of the water)

Code:

for(i = 0; i < NFILES; i++)
	{
		short *chunkptr = bigbuf;
		// Fill the big buffer with data - in this case, just some sequencial numbers!
		for(j = 0; j < ARRAYSIZE(bigbuf); j++)
			bigbuf[j] = i << 8 | (j & 0xff);

		sprintf(name, BASENAME, i);
		// Fill the buffer with data. 
		f = fopen(name, "wb");
		if (f == NULL) 
			stop("Couldn't open file");

		
		while(chunkptr < &bigbuf[BIGNUM]) {
			if (fwrite(chunkptr, WRITESIZE, 1, f) < 1)
				stop("Failed to write to disk");
	
			chunkptr += (WRITESIZE / sizeof bigbuf[0]);
		}
		fclose(f);
	}

Originally Posted by chico1st

ok matsp

what is this whole section supposed to do? (Im sorry i thought i was decent at C... but im being blown out of the water)

In short: It fills "bigbuf" with some data that is then written in chunks to a file.

I've added some extra comments to the code-section.

Code:

for(i = 0; i < NFILES; i++)
	{
		short *chunkptr = bigbuf;
// We discussed this one earlier - ARRAYSIZE is a macro that "automagically" figures out 
// how big an array is
		// Fill the big buffer with data - in this case, just some sequencial numbers!
		for(j = 0; j < ARRAYSIZE(bigbuf); j++)
			bigbuf[j] = i << 8 | (j & 0xff);

		sprintf(name, BASENAME, i);
		// Fill the buffer with data. 
// Remove the abve comment - it's from a previous version of the code. 

// Open the file to write to...
		f = fopen(name, "wb");
		if (f == NULL) 
			stop("Couldn't open file");


// So, while chunkptr isn't at the end of the buffer. 		
		while(chunkptr < &bigbuf[BIGNUM]) {
// Write one "WRITESIZE" portion of data.
			if (fwrite(chunkptr, WRITESIZE, 1, f) < 1)
				stop("Failed to write to disk");
	
// This is incrementing the pointer to the next location in bigbuf. Since a pointer 
// is incremented by the size of the element, we divide the "size" in bytes by the
// size of each element - otherwise we'd increment by too much. 
			chunkptr += (WRITESIZE / sizeof bigbuf[0]);
		}
// Close the file :-)
		fclose(f);
	}

If you have further questions, let me know. I agree that it's not the most easy to read code I've ever written - it's hard to write generic, easy to read and efficient code - usually you can pick any two of those :-)

--
Mats

if i have to switch the byte order of all the data, from big endian to little endian, this well take forever to execute wont it?

I'm not sure i have to do that.. im going to find out tonight.... all i know if that my current binary output file doesnt work.

i have a thread open to deal with this.

http://cboard.cprogramming.com/showthread.php?t=92493

Originally Posted by chico1st

Code:

if (f[i] = fopen(FileName, "r")==NULL){

to

Code:

f[i] = fopen(FileName, "r");
		if (f[i] ==NULL){

Operator precedence problem.

oh ok
now that makes sense... i guess it should have been in brakets

just as an aside... if i had done this in something newer... like C# would it have been easier... or is C# not much of a console language?

so you are saying C# is a dead language... ok ...GO ANSI C!

oh man

My program doesnt work, but ill post it just to keep people updated.
i had to convert Header[] and Data[][] from big endian to little endian and here is where i am now:

Code:

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <winsock2.h>

#define RowsInArray 10000
//#define MAXNumberOfFiles 20

#define ByteSwap5(x) ByteSwap((unsigned char *) &x,sizeof(x))

void ByteSwap(unsigned char * b, int n)
{
   unsigned char temp;
   register int i = 0;
   register int j = n-1;
   while (i<j)
   {
      temp = b[i];
	  b[i] = b[j];
	  b[j] = temp;
	  i++;
	  j--;
   }
}


int main(){
	FILE *f[20];
	FILE *fbin;

	double Header[5]={3,1,1000000,0,0.00015625};
	char FileName[255];
	char BaseName[] = "Chan%.0d.csv";
	int i,j,k;
	int n=5;
	register int m;
	short NumberOfFiles = 1;
	//int RowsInArray = 10000;
	int TotalRows = 128000000;
	unsigned short *Data[RowsInArray];//[RowsInArray][NumberOfFiles];
	unsigned short DataPoint, DataPt2;

	//open binary file
	fbin = fopen("BinaryData.log", "wb");
	if (fbin==NULL){
		printf("Couldn't open binary file\n");
		exit(1);
	}
	
	//convert header
	for (m=0; m<n; m++) {
		ByteSwap5(Header[m]);
	}

	fwrite(Header, sizeof(Header[0]), 1, fbin);

	//open csv files
	for (i=0; i<NumberOfFiles; i++){
		//create filename to open
		sprintf(FileName, BaseName, (i+1));
		
		//open files
		f[i] = fopen(FileName, "r");
		if (f[i] ==NULL){
			printf("Couldn't open csv file\n");
			exit(1); 
		}
	}


	//dynamically make array to hold data before placing in binary
	for (i = 0; i < RowsInArray; i++)
		Data[i] = (void *)malloc(sizeof(short) * NumberOfFiles);



	//loop until all data taken
	for (k=0; k<(TotalRows/RowsInArray); k++){
		//put data in array
		for (i=0; i<NumberOfFiles; i++){
			for (j=0; j<RowsInArray; j++){
				fscanf(f[i], "%d", &DataPoint);
				//DataPoint = 1;
				DataPoint = DataPoint/2;
				DataPt2 = ntohs(DataPoint);
				Data[j][i] = DataPt2;
			}
		}

		fwrite(Data, sizeof(Data[0]), 1, fbin);
	}


	//free malloc'd Data
	for (i=0; i<NumberOfFiles; i++)
		free(Data[i]);

	//(Data) was not malloc'd just its elements were


	//Close csv Files
	for (i=0; i<NumberOfFiles; i++)
		fclose(f[i]);

	//close Binary
	fclose(fbin);
}

What machine are you running this on? Alpha? If it's an x86, it is little endian in the first place...

Don't use "register" keywords - if they have ANY meaning to the compiler, it will probably make the code worse.

Your byteswap doesn't look quite right - if you have 5 doubles, you'll need to swap 8 bytes at a time, not all 40 bytes - that would swap all five of the doubles to reverse order too. So your doubles:

Code:

{3,1,1000000,0,0.00015625}
would end up
{0.00015625, 0, 1000000,1,3}

Is that what you want?

sizeof(Data[0]) is the same as sizeof(short *), whcih is most likely 4 (or 8 on a 64-bit machine) - is that how much data you want to write each time?

--
Mats

ok...i just killed a lot of time working on this byte swap thing, after doing a test which said it was big endian... but it turns out i just read the result wrong... so its little endian.. so here is the updated code.

Code:

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <winsock2.h>

#define RowsInArray 1000
//#define MAXNumberOfFiles 20

int main(){
	FILE *f[20];
	FILE *fbin;

	double Header[5]={3,1,1000000,0,0.00015625};
	char FileName[255];
	char BaseName[] = "Chan%.0d.csv";
	int i,j,k;
	int n=5;
	register int m;
	short NumberOfFiles = 1;
	int TotalRows = 128000;
	unsigned short *Data[RowsInArray];//[RowsInArray][NumberOfFiles];
	unsigned short DataPoint;
	short DataPt2;
	//open binary file
	fbin = fopen("BinaryData.log", "wb");
	if (fbin==NULL){
		printf("Couldn't open binary file\n");
		exit(1);
	}
	
	
	fwrite(Header, sizeof(Header[0]), 1, fbin);

	//open csv files
	for (i=0; i<NumberOfFiles; i++){
		//create filename to open
		sprintf(FileName, BaseName, (i+1));
		
		//open files
		f[i] = fopen(FileName, "r");
		if (f[i] ==NULL){
			printf("Couldn't open csv file\n");
			exit(1); 
		}
	}


	//dynamically make array to hold data before placing in binary
	for (i = 0; i < RowsInArray; i++)
		Data[i] = (void *)malloc(sizeof(short) * NumberOfFiles);



	//loop until all data taken
	for (k=0; k<(TotalRows/RowsInArray); k++){
		//put data in array
		for (i=0; i<NumberOfFiles; i++){
			for (j=0; j<RowsInArray; j++){
				fscanf(f[i], "%d", &DataPoint);
				//DataPoint = 1;
				DataPoint = DataPoint/2;
				DataPt2 = (short) DataPoint;
				Data[j][i] = DataPt2;
			}
		}

		fwrite(Data, sizeof(Data[0]), 1, fbin);
	}


	//free malloc'd Data
	for (i=0; i<NumberOfFiles; i++)
		free(Data[i]);

	//(Data) was not malloc'd just its elements were


	//Close csv Files
	for (i=0; i<NumberOfFiles; i++)
		fclose(f[i]);

	//close Binary
	fclose(fbin);
}

i want to write the whole Data, 2D array to the binary file.
doesnt fwrite(Data, sizeof(Data[0]), 1, fbin); mean write Data , broken into chunks of size SHORT?

Your current fwrite writes sizeof(short *) bytes from Data - which is probably 4 bytes.

Note: sizeof gives the size of the datatype, not the size of something you've allocated - if you use malloc to fetch a 64 byte block of data into a "int *p", then the sizeof(p) would still be 4 bytes - because the size of a pointer is 4 bytes.

The next problem is that your data is discontiguous (and I may have "helped" make that happen). You really need to allocat one big block of data - you can still use the same Data declaration, but your call to malloc should really be for one chunk of memory- there's absolutely no guarantee that two calls to malloc for X bytes gives you two chunks of memory next to each other - in fact, it's MORE likely that they are not, as most of the time, malloc will add a small amount of "overhead" to each data-block, and use this for "housekeeping data" (e.g. the size of the block allocated, so that when it's freed, we can know the size of the block in the free-list, etc).

Something like this:

Code:

...
        short *AllData;
...
        AllData = malloc(sizeof(short) * NumberOfFiles * RowsInArray);
        if (AllData == NULL)
             // Do some error handling and bail out  - not enough memory.
	//dynamically make array to hold data before placing in binary
	for (i = 0; i < RowsInArray; i++)
		Data[i] = &AllData[i * NumberOfFiles];

Now you've got one big chunk of memory, split into portions.

You probably also want to check your "fscanf" to make sure you're not writing "rubbish" to the output file (not sure if that would make it better or worse, but if you at least handle the case of too-short file, or erroneous data, it would help a bit to diagnose any problem that may occur because of it).

--
Mats

oh thats a pretty cool method that makes perfect sense to me.

I was looking at your old code, with the fancy string chunk thing. I dont think im going to try that, i understand this code that i have now really well. so ill just polish this up. Ill clean up my code then upload it.

if i add an if statement to see if my values that i read in from my spreadsheets in ok, will that slow down the compilation time?

like this
if (DataPoint<0 || DataPoint>660000){

2 billion times