Thread: delete all lines that have more than one occurrence on a file

Hi all,

I have a input csv file, the lines are like this (in.csv):

Code:

0123456789;CUST098WZAX;35
0123450123;CUST056TVZE;37
0123458989;CUST034XMKS;67
0123456789;CUST098WZAX;35
0123458989;CUST034XMKS;67

And i want to remove all the lines that appears more that ones in the file, this file should produce this (out.csv):

Code:

0123450123;CUST056TVZE;37

I try this code but i have some trouble to fixe removing the first occurrence. your help is welcome.

Here is my code :

Code:

#include <stdio.h>
#include <stdlib.h>
#include <string.h>

struct somehash {
    struct somehash *next;
        unsigned hash;
        char *mem;
};

#define THE_SIZE 100000

struct somehash *table[THE_SIZE] = { NULL,};

struct somehash **some_find(char *str, unsigned len);
static unsigned some_hash(char *str, unsigned len);

int main (void)
{
    char buffer[100];
    struct somehash **pp;
    size_t len;
    FILE * pFileIn;
    FILE * pFileOut;

    pFileIn  = fopen("in.csv", "r");
    pFileOut  = fopen("out.csv", "w+");

    if (pFileIn==NULL) perror ("Error opening input file");
    if (pFileOut==NULL) perror ("Error opening output file");

    while (fgets(buffer, sizeof buffer, pFileIn)) {
            len = strlen(buffer);
            pp = some_find(buffer, len);
            if (*pp) { /* found */
                fprintf(stderr, "Duplicate:%s\n", buffer);
                }
            else    
        {       /* not found: create one */
                    fprintf(stdout, "%s", buffer);
                    fprintf(pFileOut, "%s", buffer);
                    *pp = malloc(sizeof **pp);
                    (*pp)->next = NULL;
                    (*pp)->hash = some_hash(buffer,len);
                    (*pp)->mem = malloc(1+len);
                    memcpy((*pp)->mem , buffer,  1+len);
                }
        }

return 0;
}

struct somehash **some_find(char *str, unsigned len)
{
    unsigned hash;
    unsigned short slot;
    struct somehash **hnd;

    hash = some_hash(str,len);
    slot = hash % THE_SIZE;
    for (hnd = &table[slot]; *hnd ; hnd = &(*hnd)->next ) {
        if ( (*hnd)->hash != hash) continue;
            if ( strcmp((*hnd)->mem , str) ) continue;
                break;
        }

    return hnd;
}

static unsigned some_hash(char *str, unsigned len)
{
    unsigned val;
    unsigned idx;

    if (!len) len = strlen(str);

    val = 0;
    for(idx=0; idx < len; idx++ )   {
            val ^= (val >> 2) ^ (val << 5) ^ (val << 13) ^ str[idx] ^ 0x80001801;
    }

    return val;
}

Regards.

Originally Posted by Salem

Are you open to using different tools for the job?

Code:

$ cat wibble.txt
0123456789;CUST098WZAX;35
0123450123;CUST056TVZE;37
0123458989;CUST034XMKS;67
0123456789;CUST098WZAX;35
0123458989;CUST034XMKS;67
$ perl -e '%h=();while(<>){chomp;$h{$_}++;} foreach $i ( keys %h ) { print "$i\n" if ( $h{$i} == 1 ); }' wibble.txt
0123450123;CUST056TVZE;37

Thanks Salem, but i wan't to do it in c

WIBBLE!

That is all.

Soma

Originally Posted by phantomotap

WIBBLE!

That is all.

Soma

Thanks phantomotap for the joke.

Regards.

Originally Posted by oogabooga

Or

Code:

$ cat wibble.txt
0123456789;CUST098WZAX;35
0123450123;CUST056TVZE;37
0123458989;CUST034XMKS;67
0123456789;CUST098WZAX;35
0123458989;CUST034XMKS;67
$ sort wibble.txt | uniq -u
0123450123;CUST056TVZE;37

Thanks but as i said before, i want to do this in c (in shell it's very simple as you post)

Originally Posted by oogabooga

To answer your question about doing it in C, however, Salem has shown the way.
Just add a count member to your struct. Set it to one when you add a node. Increment it when the same value is seen again.
Then loop through the table and print those members that have a count of 1.

Additionally,

What's with the "some" prefix to your names? What does it mean?
Calling perror does not exit, but you certainly don't want to continue if the files don't open.
Do you need "w+" (write AND read) access or just "w" for "out.csv"?

You're right it's a typo "w" is ok. Thanks for the tips. problem is now solved.

Regards.