Hello all
i released my web crawler as open source , done in C++ .
you can use it as you wish please check out:
driller-cpp-web-crawler - c++ web crawler / spider - Google Project Hosting