xml file parsing in C

hi,
is it possible to parse an XML file in C so that i can fulfill these requirements :
1) replace all "<" and ">" signs inside the body of tag by a space, e.g. :
Example 1:

Code:

<foo> blabla < bla </foo>

becomes

Code:

<foo> blabla bla </foo>

Example 2:

Code:

<foo>> blablabla </foo>

becomes

Code:

<foo> blablabla </foo>

2) Remove all extra spaces at the end of every line of the XML file
3) Replace all special characters ( Unicode or Hexadecimal characters) by a space

I mean the XML file is not well formed if there are "<" and ">" signs a little bit everywhere,
it is not a valid file in that case, so i do not think the use of a parser would be appropriate in that case. (How would the parser react when it encounters a < that does not correspond to the beginning of a tag ???)

Do you have an idea on how i can write a program to deal with these requirements ?
Technical environment is : Unix, KSH, and C (gcc)

I am thinking of using the "sed" command instead, i can get rid of the extra spaces and replace the special characters but i still do not know how to deal with the extra ">" and "<" signs.

Thanks for your help.

Do you have any more bad examples?

For example,
if > is preceded by anything other than a letter, delete it
if < is followed by anything other than a letter, delete it.

thanks Salem for replying.
Unfortunately there are many more bad examples, so your solution would not work.

Does anyone else have an idea ?

Well it's hard to come up with ideas if you don't post more examples to work with.

The first rule of parsing XML is, don't parse it yourself. Find an XML parsing library.

It is impossible for an XML document to contain < in the text, if it is well-formed, so your program should just throw an error on these. What if you find < and > in the text? Do you want to replace these with spaces?

I agree.

If it's that broken, perhaps you should be looking at what broke the file so horribly in the first place rather than trying to fix it here.

Well, this is what I have done so far : I have written a ksh file that calls
the command SED 3 times, in fact for each requirement as specified above:

2) sed -e 's/ *$//' old_file > file_with_no_space
3) sed -e 's/$E5/ /' -e 's/$F6/ /' (etc ...) file_with_no_space file_with_no_special_characters

1) This is where I am still looking for a way to do it. How can I replace only the extras < and > that I find
on every line ?

I thought about an algorithm, since there is only one tag / line, it means you can only have 2 times
the character '<' and the character '>'.
If I find more than 2 times the character '<' that means i have to remove the ones that are not in the first position
or the last position.
Same thing with the '>' character.

how would you do that in C or with the 'sed' command ? Or eventually in conjunction with the 'awk' and 'grep' command ?
Also, how fast is it, knowing the file size varies from 5 mn to 300 mn ?

thanks for your help

Quote:

Originally Posted by lonbgeach

Also, how fast is it, knowing the file size varies from 5 mn to 300 mn ?

Okay, I'm showing my ignorance here, but how much is a "mn"? I've never heard of this before. B, KB, MB, GB, TB, DB but mn????

Quote:

Originally Posted by Kennedy

Okay, I'm showing my ignorance here, but how much is a "mn"? I've never heard of this before. B, KB, MB, GB, TB, DB but mn????

oops, I meant MB of course :)

No, you meant mega nuggets.

Can you use a language which has native support for regular expressions, rather than calling sed all the time? Anyways, here're some regular expressions that may help (written in perl, but I think the RegEx syntax is relatively standard accross languages):

Code:

if($input_xml_sample =~ /\<([A-Za-z0-9_\-]{1,40})\>(.*?)\<\/\1\>/s ) { # \1 refers to contents matched within the 1st () group $tag_name = $1; #get contents from first () group $tag_contents = $2; # get contents from 2nd () group $tag_contents =~ tr/\<\>/ /; # change < and > to spaces $tag_contents =~ s/\&lt\;/ /g; # get rid of < Repeat for > $tag_contents =~ s/\&\#[0-9]{1,5}\;/ /g; # matches decimal specials, i.e. &#453 $tag_contents =~ s/\&\#x[0-9a-fA-F]{1,5}\;/ /g; #matches hexaecimal specials, i.e. ¤ $tag_contents =~ s/^ +//; # get rid of leading spaces $tag_contents =~ s/ +$//; # trainling spaces }

These regex are just example and assume that at least the tags are in the right format though. I.e. tag opening and closing name must match and only contain alphanumerics with no spaces.

That, of course, is broken. You're expecting tags not to have any attributes? What about CDATA sections? Tag names must be less than 41 characters long? No colons, unicode characters, in tag names?

coder8137 : thanks for the effort !
I will try to use that code.
Do you have any suggestion on how to use it to check every line ? I mean, I need to put that code inside a loop.
Thanks for your help much appreciated.

Quote:

That, of course, is broken. You're expecting tags not to have any attributes? What about CDATA sections? Tag names must be less than 41 characters long? No colons, unicode characters, in tag names?

It was a regular expression example based on the data in the top post, not a full XML praser. I expected whoever wants to use it to apply common sense and adjust it to whatever their needs may be. For example, if you want attributes, an example could be...

Code:

$input_xml_sample =~ /\<([A-Za-z0-9_\:\-]{1,40})[^\>]{0,120}\>(.*?)\<\/\1\>/s

Now the tag name only matches the 1st word (+ colons). Presuambly, there will then be a space and then the arguments. If you need colons.... add it to the list. If you don't like 40... pick a higher number. The reason I put in limits is because he stated that his file is not proper XML to begin with. He mentioned speed also; if you omit the limits now, and just use *s and +s, your regex is a lot worse for bad XML.

Regarding unicode. I don't see any unicode in the example at the top, but he mentioned speed. No point to introduce unicode when not needed, as this will slow things down dramatically. [A-Za-z0-9] will match 62 characters. The unicode equivalent will probably match 1,000 times more characters.

Lastly, regarding CDATA. Parsing an XML file is more than 1 RegEx. If you want to use unicode, the 1st step is probably to get all the character encodings right in whatever programming language you chose. You would then probably be served best by removing all comments from the XML. Then you might remove all CDATA. Then you might do whatever else you have to do. And then you can parse it.

Quote:

Do you have any suggestion on how to use it to check every line ? I mean, I need to put that code inside a loop.

Depends on your file. You said that there is only one tag per line, right? So, if you use perl, you can simply read in one line at a time and use the regex. Add an else clause to the if statement, to work out when it doesn't match. Read the above regarding Comments and CDATA, which need to be cleared 1st. If you want to use C and sed, you can again read one line, but getting $1 and $2 from sed will probably be harder. To help you any more, I'd need to know what language you will use and/or a file sample.

The tag do not have attributes, there is no CDATA so it's all good.
The idea is good but the following code does not work with Ksh.

Code:

<bla> body </bla>

also we can have an empty tag, like this :

Code:

</bla>

How do you loop in a XML file
and store the opening tag into $1 and the body of the tag inside $2 for treatment ?

Code:

if($input_xml_sample =~ /\<([A-Za-z0-9_\-]{1,40})\>(.*?)\<\/\1\>/s ) { # \1 refers to contents matched within the 1st () group $tag_name = $1; #get contents from first () group $tag_contents = $2; # get contents from 2nd () group $tag_contents =~ tr/\<\>/ /; # change < and > to spaces }

thanks for helping

empty tag should be like this

Code:

<bla/>

I don't think you can do this with ksh (becuase I don't think it allows you to refer to what previously matched in the same pattern, i.e. \1). But since you were just going calling sed all the time before, you could just call an awk or perl script that will do everything in one go. I don't know much about awk, but I pieced together a perl script from the examples I posted earlier. A lot of unix installations come with perl. Calling it with ./my_script.pl should be enough.
Anyways, you didn't cofirm that you only have 1 tag pair / line, but I assumed this is so based on what you said earlier. If you use it, make sure to test it on your files. The script will harshly replace extra <s and >s with spaces or altogether if they are the last or first thing in the xml_contents_string.

This script won't normally print to STDOUT, but straight to file... change if you need. Will
warn/die if something is completly wrong (i.e. a line where nothing matched), but not if some data is discarded bc there is more than 1 tag pair perl line.

Code:

#! /usr/bin/perl # or try /usr/local/bin/perl if above doesn't work use warnings; use strict; open(INPUT_FILE, '<my_xml_file') or die "Can't open input file: $!"; open(OUTPUT_FILE, '>my_xml_output_file') or die "Can't open output file: $!"; #the above will truncate output file, if it exists my ($input_xml_sample, $tag_name, $tag_contents); my $line_num = 1; while($input_xml_sample = <INPUT_FILE>) { # reading one line at a time. This assumes strictly one tag pair per line and no more! # Anything not part of the tag or the contents will be discared # Example: " <some_tag:abc> Some random ><useless> text > all </some_tag:abc> " # Bad: " <just_an_open_tag> on this line " # Bad: " some text <just_a:closer /> " (Only tag name will make it to file) if($input_xml_sample =~ /\<([A-Za-z0-9_\-\:]{1,40})\>(.*?)\<\/\1\>/s ) { # \1 refers to contents matched within the 1st () group # does not handle tag atrributes. If Tages contain anyhting other than # A-Za-z0-9_-: or is longer than 40 than edit above group $tag_name = $1; #get contents from first () group $tag_contents = $2; # get contents from 2nd () group $tag_contents =~ tr/\<\>/ /; # change < and > to spaces $tag_contents =~ s/\&lt\;/ /g; # get rid of < $tag_contents =~ s/\&gt\;/ /g; $tag_contents =~ s/\&\#[0-9]{1,5}\;/ /g; # matches decimal specials, i.e. ǅ Change as needed!! $tag_contents =~ s/\&\#x[0-9a-fA-F]{1,5}\;/ /g; #matches hexaecimal specials, i.e. ¤ $tag_contents =~ s/^ +//; # get rid of leading spaces $tag_contents =~ s/ +$//; # trainling spaces print OUTPUT_FILE "<$tag_name>$tag_contents</$tag_name>\n"; } elsif($input_xml_sample =~ /\<([A-Za-z0-9_\-\:]{1,40}) {0,10}\/\>/ ) { # empty tag, i.e. "<tag />" print OUTPUT_FILE "<$1 />\n"; } else { if($input_xml_sample ne "\n") { warn("Couldn't use line $line_num at all!"); } } $line_num++; } close(INPUT_FILE); close(OUTPUT_FILE);

> since there is only one tag / line
See, more vital info

How about this

Code:

#!/usr/bin/perl -w use strict; while ( <> ) { chomp; if ( /<(\w+)>(.*?)<\/\1>/ ) { my ($tag,$content) = ($1,$2); $content =~ s/[<>]//g; print "<$tag>$content</$tag>\n"; } else { print "$_\n"; } } $ example, input in red, output black $ ./foo.pl <foo> blabla < bla </foo> <foo> blabla bla </foo> <foo>> blablabla </foo> <foo> blablabla </foo> <foo/> <foo/>

coder8137 :
thanks very very much ! I am using your solution and learning perl in the same occasion !
That is great, it works.
I modified it so that it allows tags with dots in the names like

Code:

<aa.bb>

and opening tags because there is only one pair of tag per line but you can also find lines
with opening tags only

Code:

<aa>

and of course closing tags

Code:

</aa>

Now, how do you pass a file as argument instead of having it hard-coded inside the file ?
Also, I want to return error codes in case one step failed, how do i do that ?

Here is a sample of XML file to deal with :

Code:

<?xml version="1.0" encoding="ISO-8859-1"?>   <lot> <facture> <Bloc.pointeur.ligne> <Nombre.exemplaire.facture>01</Nombre.exemplaire.facture> <Numero.client.fpl>117225</Numero.client.fpl> <Rang.client.fpl>02</Rang.client.fpl> <Numero.periode.fpl>0602</Numero.periode.fpl> <Numero.sequence.fpl>01</Numero.sequence.fpl> <Code.recto.verso>1</Code.recto.verso> <Code.MOF>2</Code.MOF> <Code.MSE>0</Code.MSE> <Numero.secteur>056 </Numero.secteur> <Code.CED>E</Code.CED> <Critere.tri.secondaire>056 </Critere.tri.secondaire> <Critere.tri.secondaire>11042 </Critere.tri.secondaire> <Critere.tri.secondaire>1 </Critere.tri.secondaire> <Critere.tri.secondaire>0 </Critere.tri.secondaire> <Critere.tri.secondaire>117225 </Critere.tri.secondaire> <Numero.logo.fpl>110</Numero.logo.fpl> <Structure.cle>3</Structure.cle> <Cle.banalisee>110420204201</Cle.banalisee> <Adresse.1er.enreg>00000001</Adresse.1er.enreg> <Adresse.dernier.enreg>00000001</Adresse.dernier.enreg> <Code.exploitation.fpl>98</Code.exploitation.fpl> <Date.exigibilite>17012007</Date.exigibilite> <Code.tache>1</Code.tache> <Code.anomalie.F18>N</Code.anomalie.F18> <Modele.impression>4</Modele.impression> </Bloc.pointeur.ligne> <Bloc.entete.cachet.logo.siret> <Desig.agence>MAIRIE DE: </Desig.agence> <Desig.agence>BLOMAC </Desig.agence> <Adresse.agence>6 RUERIE </Adresse.agence> <Adresse.agence>117MAC </Adresse.agence> <Tel.fax> <Libelle.tel.fax> FAX 04.8.32 </Libelle.tel.fax> <Telephone.fax>TEL5 </Telephone.fax> </Tel.fax>

thanks a lot, this is a great active and helpful community !

Quote:

How about this

Nice; alot easier to integrate with. Mine was more of a stand alone script. Just \w doesn't match the magic colon, which I left out first time around too, but it is quite common in real XML, as Rashakil Fol pointed out, hehe.

Quote:

Now, how do you pass a file as argument instead of having it hard-coded inside the file ?

If you use Salem's script, pass the data to STDIN. If you use mine, just change the open line to read:

Code:

open(INPUT_FILE, "<$ARGV[0]") or die "Can't open input file: $!";

And then call it with:

Code:

./my_script.pl some_file_name

CAUTION: The argument is not checked before opening the file. Therefore, this code is not designed for random input. In other words, it's vulnerable to abuse. You could use a regular expression or 2 to validate the input, if this bothers you.

You can get error / return values by either capturing STDOUT or STDERR and checking it. Are you going to call from C?

I am not going to call it in C, but it will be part of a chain of batches.
That is why I need to dal with return error codes.
How do you do that in Perl ?

Also, one requirement is to delete all extra spaces at the end of the lines, but the spaces between the tags should remain of course.
I thought that if I would just keep this line it would work but it does not, it also removes the space between tags :

Code:

$tag_contents =~ s/ +$//; # trainling spaces

I removed the line

Code:

$tag_contents =~ s/^ +//; # get rid of leading spaces

In other words :

Code:

<bla> Once upon a time </bla>

should remain as is and not be modified to

Code:

<bla>Onceuponatime</bla>

I just want to get rid of the extra spaces after the closing tag, on each line.

Thanks !

ok, i got it, i removed the wrong line. hehe

Please help me regarding the error codes.
Thanks

For an easier way to do error handling, you can use the exit() function in perl. exit(0) for success, exit(1) for failure... like C. (You'd have to replace warn()) I don't know much about ksh, but generally echo $? or similar, should return the exit value of last command on a shell.

re: xml file parsing in C

You may also want to check out the Expat XML Parser (written in C). It may give you some ideas or you may be able to use some of the code.

http://expat.sourceforge.net/

Sam

I got more precisions regarding the special characters :

Code:

<Cle.controle.TOP.2>bå</Cle.controle.TOP.2> <Numero.organisme>ö3M</Numero.organisme> <Cle.controle.TOP.2>bæ</Cle.controle.TOP.2> <Numero.organisme>¤0O</Numero.organisme>

I think there are basically 4 special characters :

Code:

å ö æ ¤

Which characters are these ? Unicode ? Ascii ?
I want to suppress them (replace by a space).
Is there a general way to deal with these special characters instead of listing them all in my code ?

Code:

$tag_contents =~ s/\ö\å\æ\¤\/ /g;

I have not tested this line yet by the way.

Thanks in advance for your help

Perhaps your special characters would be better dealt with by

Code:

<?xml version="1.0" encoding="UTF-8"?>

Though that depends on what is next in the set of things to do, and can it really cope with extended characters of some sort.

> $tag_contents =~ s/\ö\å\æ\¤\/ /g;
A character range would catch more, say

Code:

$tag_contents =~ s/[\x80-\xff]/ /g;

and is a lot less reliant on the interpretation of 'funny' characters.

You should be able to run my script with
./foo.pl input.xml > output.xml

Code:

use XML::Parser

http://www.xml.com/pub/a/98/09/xml-perl.html

The problem with the existing tools is that they're good at reading valid XML files.
They're not so good at fixing XML files which are broken.

You could try an HTML parser. They're good at broken stuff.

Quote:

I think there are basically 4 special characters

The XML file claims to be iso-8859-1. You cannot just use it as if it were UTF-8. In iso-8859-1 all characters are made up of one byte. In UTF-8, the non-ASCII characters (i.e. letters with accent marks / diacritics, like the ones you posted) are made up of 2 bytes. So in iso-8859-1, if you know the hex value of the characer you are looking for, you can always easily match it, as long as you don't play around with the encoding.

You can try what Salem posted, but I don't think it will work very well, because the dash ( - ) is only supposed to be used for simple sequences (like a-g for example) as far as I know. Here's some more things you can try:

Code:

$tag_contents =~ tr/A-Za-z0-9\.\_ / /c; # This will space out all characters except those listed! Add to the list anything you need # to keep.

Or, to do things more correctly, you can use Encode/Decode. ( http://perldoc.perl.org/Encode.html ) First decode data from iso-8859-1. Then you should be able to match all your special characters using their unicode if you like. But also, if you decode data from iso-8859-1 and then encode it as ASCII, anything non ASCII should automatically be replaced with "?". Instead of using Decode explictly, you can try to just change the calls to open if you use PerlIO.

Code:

open my $in, "<:encoding(iso-8859-1)", $infile or die; open my $out, ">:encoding(ascii)", $outfile or die; # See Docs