Command to strip out comments in source files

I am trying to find a command to strip out comments in C++ source files. I suspect it would be sed, but I don't know how to use it yet and don't have time to learn it right now. Any sed wizard care to spare a few seconds?

I need to strip out both "//" and "/* ... */" style comments. If possible, "#if 0/#endif" ones, too.

Thanks

Do you use a lot of #defines and #includes? You could maybe use the actual preprocessor. Otherwise I'm not a sed wizard either

What do you mean by use the actual preprocessor?

I am trying to strip out comments because my partner would only allow me to open source our program (a chess engine) without the comments in a source file (since there is quite a bit of private discussions going on in the comments about novel ideas for the chess engine that he prefers to be kept private for now, but doesn't mind the code being open).

Ah I see what you mean now. I don't think that's a good idea because I need to do it for every release... I think it's "messier" than a sed solution, too.

My guess would be that there are A LOT of comments that do not relate to any "new ideas", whilst a few comments do. How about working through the code and marking those comments that you don't want to publish with a marker, and then write a small application that strips only those comments?

--
Mats

The discussion is actually quite scattered, throughout the whole file, because a lot of ideas are being discussed, and the comments are put where they are relevant.

Not sure if this will do what you want

Quote:

---

Originally Posted by man gcc

-fpreprocessed
Indicate to the preprocessor that the input file has already been
preprocessed. This suppresses things like macro expansion,
trigraph conversion, escaped newline splicing, and processing of
most directives. The preprocessor still recognizes and removes
comments, so that you can pass a file preprocessed with -C to the
compiler without problems. In this mode the integrated
preprocessor is little more than a tokenizer for the front ends.

---

Quote:

---

Originally Posted by cyberfish

The discussion is actually quite scattered, throughout the whole file, because a lot of ideas are being discussed, and the comments are put where they are relevant.

---

Yes, but my point was that a large portion of comments probably do not relate to new ideas, but rather relate to what this code actually does, and thus are useful for others to help improve your code. Presumably part of your release in the open source is to gain other from peoples experience, and to let others learn from you? Completely stripping ALL comments will reduce the readability.

Of course, it may be that your code doesn't have much in the way of comments, EXCEPT for the comments on new ideas - in which case please ignore my post here.

I don't think pre-processing is a great idea - it will drag in all the header files that the project uses. Those header files are (at least in part) compiler and OS dependant, so whilst it will remove the comments, it will also make the code completely unportable.

Likewise for any adaption that uses something like:

Code:

---

#if WIN32
...
#else
...
#endif

---

--
Mats

> since there is quite a bit of private discussions going on in the comments about novel ideas for the chess engine
I'd suggest you start a local Wiki for that, and leave the comments in the code documenting the facts rather than the ideas.

Otherwise, as you say, it's going to get messy (and error prone with it).

Ah I see.

Quote:

---

-fpreprocessed

---

That kind of works, but it applies "#define", and doesn't remove "#if 0/#endif" blocks. It also seriously messes up the indentation.

Quote:

---

Yes, but my point was that a large portion of comments probably do not relate to new ideas, but rather relate to what this code actually does, and thus are useful for others to help improve your code. Presumably part of your release in the open source is to gain other from peoples experience, and to let others learn from you? Completely stripping ALL comments will reduce the readability.

---

That is a good point, but I think the source is pretty much self-documenting. No complex algorithms and such. Just simple evaluation terms.

Quote:

---

I'd suggest you start a local Wiki for that, and leave the comments in the code documenting the facts rather than the ideas.

---

That is good idea, except it will be quite a bit of work, so I'd rather just find something to strip out the comments.

I suppose I can write some C++ code to do it... but would prefer just a sed command. (or I can take the time to learn sed, of course :)).

Seriously, do it with sed. The time you've spent looking for alternatives you could have just learnt sed and done it already :p

I tried to learn it just now. The problem is... I don't even know regular expressions, so I have to learn that, too.

They're not that hard, plus they're fun! And very, very useful. Check out http://www.regular-expressions.info/

Code:

---

int main()
{
    printf("/* Make sure you deal with this case. */");
    return 0;
}

---

Or maybe

Code:

---

if ( 3 //**/ 3
== 1 ) {
    printf( "C\n" );
} else {
    printf( "C++\n" );
}

---

> That is good idea, except it will be quite a bit of work, so I'd rather just find something to strip out the comments.
A reliable comment stripper will be quite a lot of work.

Is your code in source control? Do you have branches and merges?
Keeping track of where your ideas are is going to seem like hard work to me.
Also, are you going to be checking in code with only fresh ideas added in comments?

sed 's@//.*@@g'
A simple approach, which will do fine for 99% of the C++ comments.
But there are a few special cases where this is exactly the wrong thing to do, and it's those which will really tax your skill / time / patience.

You might want to look at the doxygen approach, which tags comments.
So comments which you want to remove, always look like this (for example).

Code:

---

/*@@IDEA
*/

---

Parsing for these, and only these would be a lot easier than a generic comment stripper.

Quote:

---

Code:

---

    printf("/* Make sure you deal with this case. */");

---

---

Ah, that I didn't think of.

The code is not in source control.

Quote:

---

Code:

---

sed 's@//.*@@g'

---

---

That takes out the "//" comments, but I need to take out the "/* ... */" ones, too, which I heard is more difficult (multi-lines).

I am thinking about just writing a simple one myself in C++, as it doesn't really need to be fool proof. As long as it works for my code... :).

My ghetto solution, with a consecutive-blank-lines-stripper -

Code:

---

#include <iostream>
#include <fstream>
#include <string>

#include <cassert>
#include <cctype>

const int max_length = 1024*1024*10; //10 MB

int main(int argc, char **argv) {
        int file_length = 0;
        char *buffer = new char[max_length];
        std::string out_data;
       
        std::ifstream in_file(argv[1], std::ios::in | std::ios::binary);
       
        if (!in_file) {
                std::cerr << "Error opening file: " << argv[1] << std::endl;
                return 1;
        }
       
        if (in_file.read(buffer, max_length-1)) { //if the read was successful without hitting an EOF...
                std::cerr << "File too big." << std::endl;
                return 1;
        }
       
        file_length = in_file.gcount();
       
        buffer[file_length] = 0;
       
        std::string in_data(buffer); //make it a string so we have nice functions to play with
       
        //first replace every "\r" by space
        for (int i = 0; i < in_data.size(); ++i) {
                if (in_data[i] == '\r') {
                        in_data[i] = ' ';
                }
        }
       
        int state = 0; //0 = normal, 1 = ignore until end of line, 2 = ignore until "*/", 3 = ignore until "#endif"
       
        int skip_next = 0;
       
        for (int i = 0; i < in_data.size(); ++i) {
               
                if (skip_next) {
                        --skip_next;
                        continue;
                }
               
                switch(state) {
                        case 0:
                                if ((i + 2 <= in_data.size()) && (in_data.substr(i, 2) == "//")) {
                                        state = 1;
                                } else if ((i + 2 <= in_data.size()) && (in_data.substr(i, 2) == "/*")) {
                                        state = 2;
                                } else if ((i + 5 <= in_data.size()) && (in_data.substr(i, 5) == "#if 0")) {
                                        state = 3;
                                } else {
                                        out_data.push_back(in_data[i]);
                                }
                                break;
                        case 1:
                                if (in_data[i] == '\n') {
                                        out_data.push_back('\n');
                                        state = 0;
                                }
                                break;
                        case 2:
                                if ((i + 2 <= in_data.size()) && (in_data.substr(i, 2) == "*/")) {
                                        out_data.push_back('\n');
                                        state = 0;
                                        skip_next = 1;
                                }
                                break;
                        case 3:
                                if ((i + 6 <= in_data.size()) && (in_data.substr(i, 6) == "#endif")) {
                                        out_data.push_back('\n');
                                        state = 0;
                                        skip_next = 5;
                                }
                                break;
                        default:
                                assert(false);
                }
               
        }
       
        int n_count = 0;
        for (int i = 0; i < out_data.size(); ++i) {
                if (n_count > 1 && (out_data[i] == '\n' || out_data[i] == '\t' || out_data[i] == ' ')) {
                        continue;
                } else {
                        if (out_data[i] != '\t' && out_data[i] != '\n' && out_data[i] != ' ') {
                                if (n_count > 1) { //get all the junk we threw away back!
                                        int j = i;
                                        while (out_data[j] != '\n') {
                                                --j;
                                        }
                                        for (++j; j < i; ++j) {
                                                std::cout << out_data[j];
                                        }
                                }
                                n_count = 0;
                        }
                        if (out_data[i] == '\n') {
                                ++n_count;
                        }
                        std::cout << out_data[i];
                }
        }
}

---

It's unbreakable until you try to break it :).

EDIT: code edited for the consecutive-blank-lines-stripper.

I've never put it to the test, but I have a program called rmccmt on my Linux machine ("remove c/c++ comments").

I do have a program which calculates how much source code consists of comments. I tested it pretty thoroughly, and I could post it here if you wanted, but I suspect you'd be able to figure it out yourself without too much difficulty.

@cyberfish: About your code: consider looking at isspace().

Quote:

---

Originally Posted by dwks

@cyberfish: About your code: consider looking at isspace().

---

isspace() is locale-specific, but the language itself isn't. I think it's better to explicitly check for the characters which are declared by the standard to be "whitespace."

Quote:

---

I've never put it to the test, but I have a program called rmccmt on my Linux machine ("remove c/c++ comments").

---

That is strange. I have Ubuntu 8.04 and don't have it. Not even in the APT repository.

*edit* ah nvm, it's in liwc */edit*

*edit2* It messes up indentations, too, for some reason, and doesn't catch "#if 0/#endif" Guess I will just use mine :). */edit2*