Thread: Doubt with struct and malloc

Good afternoon everyone,

I need a struct where its internal fields are dynamically allocated and the number of records is also dynamically allocated.

My question is that I have no idea how it is possible to implement this!

Imagine the struct below, each field will have a different size and will vary according to the input text.

Example: Field1 can receive 1000 bytes or 3000 bytes

The number of records in the struct will start at 0, and will increase as a record is registered.

Code:

struct {
    char
      Field1[   20],
      Field2[   20],
      Field3[ 1001],
      Field4[ 1001],
      Field5[ 1001],
      Field6[30000];
} Fields[50000];

I know how to use malloc to increase each field.

Code:

Field1 = malloc(101);

But how do I increase the number of records in the struct?

I also don't know if I can have for example different sizes in each record, since in a simple struct of char field[100] this is true for all records.

Record 1 = Field1 = 100 bytes
Record 2 = Field1 = 600 bytes

Can anyone help?

Olá, meu compatriota!! Bem... vamos ficar no inglês pro povo não reclamar...

When you declare a structure its size is fixed on the sum of sizes of its types (there is some "alignment" side-effects, but lets get this out of discussion for now), so:

Code:

struct S { char a[10]; char b[10]; };

Declares a struct with 20 bytes (chars) in size. These sizes are fixed for a (10 chars) and b (10 chars).

If you want variable sizes you must allocate space dynamically with malloc or calloc, and later, free them with free. You can "reallocate" the space previously allocated using the realloc function.

Note this new structure, like this, for example:

Code:

struct S {
  char *a;    // pointers to dynamically allocated buffer.
  char *b;

  size_t a_size;  // to keep track of the sizes.
  size_t b_size;
};

Now we don't have contiguous bytes for a and b as the arrays provides, since what we are storing in the structure is the pointers of those dynamically allocated buffers. The need to keep the sizes is because the compiler don't know how many bytes will be allocated and pointed by these pointers and you'll, probably, need that info later.

Notice, as well, that your Fields example is 1.54 GiB long, at least (not counting any improbable aligment side-effects): (20*2 + 1001*3 + 30000)*50000 = 1652150000 bytes. In some systems this will not even compile.

Let's suppose you have to allocate a 20 char buffer. You do it easily like this:

Code:

  char *p = malloc( 20 );
  // we should test if p isn't NULL afterwards.

And now you want to reallocate this buffer to 200 bytes. You should do this like:

Code:

  char *q = realloc( p, 200 );

Why the second pointer q? Because, like malloc, realloc can fail, returning NULL, but not changing the previous allocation. If you do:

Code:

  p = realloc( p, 200 );

And realloc fails, you'll have memory leakage in your hands. So, using a second pointer you have the opportunity to test for the failure without loosing the reference to the original buffer.

Is this enough to answer your question? (I can explain in pt-BR).

[]s
Fred

I know this answer will be a little bit over the top, but here is what I would do....

It's half way to an in-memory database

Code:

#include <stdio.h>
#include <stdlib.h>
#include <string.h>


#define N_FIELDS 5


struct Record {
    char *field[N_FIELDS];
};


struct Table {
   struct Record *records;
   int records_used;
   int records_allocated;
   int grow_by;
};


static struct Table *table_new(int grow_by) {
    if(grow_by < 0) return NULL;
    struct Table *table = malloc(sizeof(struct Table));
    if(table == NULL) return NULL;


    table->grow_by = grow_by;
    table->records = NULL;
    table->records_allocated = 0;
    table->records_used = 0;
    return table;
}


static int table_add_record(struct Table *table) {
   // Grow the table of if needed
   if(table->records_used == table->records_allocated) {
      printf("Adding %d more records\n", table->grow_by);
      struct Record *new_records = realloc(table->records,sizeof(struct Record)*(table->records_allocated+table->grow_by));
      if(new_records == NULL)
         return -1; // Failed
      table->records = new_records;
      table->records_allocated += table->grow_by;
   }


   // Add the new record to the end (should really check for errors...)
   int this_record = table->records_used;
   for(int i = 0; i < N_FIELDS; i++)
       table->records[this_record].field[i] = NULL;
   table->records_used++;
   return this_record;
}


static void table_set_field(struct Table *table, int rec_no, int field_no, char *value) {
   if(field_no >= N_FIELDS) return;
   if(table->records[rec_no].field[field_no] != NULL)
      free(table->records[rec_no].field[field_no]);
   if(value != NULL)
      table->records[rec_no].field[field_no] = strdup(value);
   else
      table->records[rec_no].field[field_no] = NULL;
}


static void table_print(struct Table *table) {
   printf("%d out of %d records used\n",table->records_used, table->records_allocated);
   for(int i = 0; i < table->records_used; i++) {
      printf("%3d:",i);
      printf(", %s",table->records[i].field[0]);
      for(int j = 1; j < N_FIELDS; j++) {
         printf(", %s",table->records[i].field[j]);
      }
      printf("\n");
   }
}


static void table_free(struct Table *table) {
   for(int i = 0; i < table->records_used; i++) {
      for(int j = 0; j < N_FIELDS; j++) {
         free(table->records[i].field[j]);
      }
   }
   free(table->records);
   free(table);
}
//===========================================================================
int main(int argc, char *argv[]) {
   struct Table *t = table_new(5);
   // As a test, add 30 records
   for(int i = 0; i < 28; i++ ) {
      int rec_no = table_add_record(t);
      table_set_field(t, rec_no, 0, "a");
      table_set_field(t, rec_no, 1, "b");
      table_set_field(t, rec_no, 2, "dog");
      table_set_field(t, rec_no, 3, "cat");
      table_set_field(t, rec_no, 4, "mouse");
   }
   table_print(t);
   table_free(t);
   return 0;
}

Originally Posted by flp1969

Code:

char *q = realloc( p, 200 );

Why the second pointer q? Because, like malloc, realloc can fail, returning NULL, but not changing the previous allocation. If you do:

Code:

p = realloc( p, 200 );

And realloc fails, you'll have memory leakage in your hands. So, using a second pointer you have the opportunity to test for the failure without loosing the reference to the original buffer.

Great tip!
Thank you for your attention!


Anyway,
Can I assign different bytes to each record without problems? As I asked here?

Code:

Record 1 = Field1 = 100 bytes = Fields[0].Field1
Record 2 = Field1 = 600 bytes = Fields[1].Field1

Notice that above I show 2 records but each of these records has different bytes! Because in a normal struct of type char[100] when I set the field size, I am setting it for all records that are created in the struct!

Another doubt, which I also mentioned in the question, how do I add 1 more record? Because I think that before I increase the field space, I need to increase the record qty!

Thanks

Originally Posted by marcelo.br

Great tip!
Anyway,
Can I assign different bytes to each record without problems? As I asked here?

No you can not. A 'struct' can only have on fixed size - the size returned by 'sizeof'.

Originally Posted by marcelo.br

Can I enter a size for each field here? example:

Code:

char *Database[][6];
Database[0][0] = malloc 100 bytes
Database[1][0] = malloc 500 bytes
Database[2][0] = malloc 800 bytes

That won't work. You want to define a structure to hold you fields, like this:

Code:

#include <stdio.h>
#include <stdlib.h>


struct record {
   char *fields[6];
};


int main() {
  struct record *records;


  // Create a table for 100 records
  records = malloc(sizeof(struct record)*10);
  for(int i = 0; i < 10; i++) {
    // Give each field a semi-ramdom length
    records[i].fields[0] = malloc(100+10*rand()%100);
    records[i].fields[1] = malloc(100+10*rand()%100);
    records[i].fields[2] = malloc(100+10*rand()%100);
    records[i].fields[3] = malloc(100+10*rand()%100);
    records[i].fields[4] = malloc(100+10*rand()%100);
    records[i].fields[5] = malloc(100+10*rand()%100);
  }


  // How to release everything
  for(int i = 0; i < 10; i++) {
    for(int j = 0; j < 6; j++) {
       free(records[i].fields[j]);
    }
  }
  free(records);
  return 0;
}

Originally Posted by hamster_nz

That won't work.

Ok, if it doesn't work with struct and not with matrix.
Is there something else I could use in C that would work?

Because there are fields that are extremely huge, and others that take only a few bytes.

And it would be a huge waste of memory for me to allocate for example 500Mb of memory for all the records, since most of them will use 100 bytes!

Example: A database with 1000 records
Record 1:
Title field: Uses 100 bytes
Description Field: Using 500Mb

Record 2:
Title Field: Expends 90 Bytes
Description Field: Expending 200 Bytes

Just think, I will have to allocate 1000 x 500Mb bytes just because out of these 1000 records some 20 use 500Mb or some 980 use very few bytes!

If I can't use struct, nor matrix, how can I do this?

Why do you need to load all records in memory? What if you have 1000000 of records, each with 128 KiB? Do you really expect to allocate 128 GiB of memory?
Do each record have meaning other than as array of chars?

Originally Posted by marcelo.br

Ok, if it doesn't work with struct and not with matrix.
Is there something else I could use in C that would work?

Because there are fields that are extremely huge, and others that take only a few bytes.

And it would be a huge waste of memory for me to allocate for example 500Mb of memory for all the records, since most of them will use 100 bytes!

Example: A database with 1000 records
Record 1:
Title field: Uses 100 bytes
Description Field: Using 500Mb

Record 2:
Title Field: Expends 90 Bytes
Description Field: Expending 200 Bytes

Just think, I will have to allocate 1000 x 500Mb bytes just because out of these 1000 records some 20 use 500Mb or some 980 use very few bytes!

If I can't use struct, nor matrix, how can I do this?

If you look closely at that last code I suppled, it does let you do what you want.

* The memory for each field is mallc()ed, and can be any size.

* A record has 6 fields.

* You can malloc() (and maybe realloc()) storage for as many records as you require for your dataset.

So each record needs storage for 6 pointers (48 bytes), and the storage allocated for it's fields, plus 7 times the overhead required by your malloc() implementation (e.g. something like 8 bytes per allocation)

Originally Posted by aghast

You don't tell us what you are trying to do, so I'm going to focus on that for a bit. Here are some common scenarios:

I have attached images for everyone to understand

I don't mind using struct, matrix, or any other kind of resource that I don't know the name of!

What I don't want is to allocate 500Mb for a record field that is only using 10 bytes! This is because another record in this same field consumes 500Mb.

As I show in the question
Record1->Field0 = 10 bytes
Record2->Field0 = 500Mb
Record3->Field0 = 100Mb

As I show in the images there are 4 fields in each record as an example

Notice that in the 4th Field, Steve spent 1Gb in bytes, and I don't want to allocate 1Gb in Marcelo's register, for example, just because Steve spent more bytes.


This is a simple program, like, cadaster, tasks, i.e., a program where we have records and each record some fields!


I only know how to do this using a struct or a matrix, I don't know any other way to do this. But allocating a gigantic memory for example 100,000 records where some 50 records spend so many bytes and the other 99950 records spend up to a maximum of 200 bytes is a bit of a waste


Can you help?

Originally Posted by hamster_nz

This sounds like the sort of problems for which actual databases exist

But I still can't see how your problem hasn't already been answered - maybe I need to be more explicit.

Here is an example that allocates either 100 or 1000000 or 20000 for data to be held in field[2]:

But that is exactly what I wanted to know!
If it was possible for me to have malloc with different sizes, as you showed in char *data.


I will now try to make a test of it here!