Thread: Large scale crawler with C/C++???

  1. #1
    Registered User
    Join Date
    Oct 2006
    Posts
    1

    Question Large scale crawler with C/C++???

    Hello all,

    I have been given the task of overseeing the development of our companys new project which is the design and implementation of a large scale crawling, processing and serving system in a distributed environment. Tasks include implementing search algorithms, data mining, improving relevancy or search results, managing terabytes of data and scaling algorithms to work on very large data sets, and serving search results using a large network of LINUX servers. The twist is that the company wants to use C and C++ in the developement rather than Java which I am proficient in.

    My problem is that I have only worked with small scale models and most of my experience is in Java. I would appreciate it if anyone can provide me some information on the subject and point me in the right direction

    Thanks

  2. #2
    For Narnia! Sentral's Avatar
    Join Date
    May 2005
    Location
    Narnia
    Posts
    719
    You got hired for a job programming, and you don't even know C/C++? Sorry, but that's too damn funny!
    Videogame Memories!
    A site dedicated to keeping videogame memories alive!

    http://www.videogamememories.com/
    Share your experiences with us now!

    "We will game forever!"

  3. #3
    semi-colon generator ChaosEngine's Avatar
    Join Date
    Sep 2005
    Location
    Chch, NZ
    Posts
    597
    Quote Originally Posted by Sentral
    You got hired for a job programming, and you don't even know C/C++? Sorry, but that's too damn funny!
    newsflash, C and C++ are NOT the beginning and end of software development. What most developers tend to forget is that the actual language is merely the concrete expression of your design and algorithms. It's important, but not as important as the ideas themselves.

    joshskender, are you looking for info on C++ itself or on crawlers/search algorithms? C++ is really not that dissimilar from java. Think about how you'd design this in java (without the java standard lib). The classes and packages (namespace in C++) you'd design in java can probably be written very similarly in C++.

    Besides, if you are "overseeing" this project, then your main priority is the design and architecture of this system. The actual implementation and coding should be done by your C++ code monkeys!
    "I saw a sign that said 'Drink Canada Dry', so I started"
    -- Brendan Behan

    Free Compiler: Visual C++ 2005 Express
    If you program in C++, you need Boost. You should also know how to use the Standard Library (STL). Want to make games? After reading this, I don't like WxWidgets anymore. Want to add some scripting to your App?

  4. #4
    Cat without Hat CornedBee's Avatar
    Join Date
    Apr 2003
    Posts
    8,895
    A company hiring a Java programmer and then putting him in charge of a C++ project is still funny.
    All the buzzt!
    CornedBee

    "There is not now, nor has there ever been, nor will there ever be, any programming language in which it is the least bit difficult to write bad code."
    - Flon's Law

  5. #5
    semi-colon generator ChaosEngine's Avatar
    Join Date
    Sep 2005
    Location
    Chch, NZ
    Posts
    597
    Quote Originally Posted by CornedBee
    A company hiring a Java programmer and then putting him in charge of a C++ project is still funny.
    true, but Sentral was implying he was unqualified as a programmer because he didn't know C or C++. At least, that's the way I read it.

    besides, as I've said, anyone at team lead level or above doesn't need to be a language expert. They should be designing in the problem domain, not the language domain. The problem with C and C++ programmers in particular is that they tend to think in the language domain first.

    Look at this board for example. the majority of the topic list concerns syntactical and language issues.
    "I saw a sign that said 'Drink Canada Dry', so I started"
    -- Brendan Behan

    Free Compiler: Visual C++ 2005 Express
    If you program in C++, you need Boost. You should also know how to use the Standard Library (STL). Want to make games? After reading this, I don't like WxWidgets anymore. Want to add some scripting to your App?

  6. #6
    For Narnia! Sentral's Avatar
    Join Date
    May 2005
    Location
    Narnia
    Posts
    719
    I have been given the task of overseeing the development of our companys new project which is the design and implementation of a large scale crawling, processing and serving system in a distributed environment. Tasks include implementing search algorithms, data mining, improving relevancy or search results, managing terabytes of data and scaling algorithms to work on very large data sets, and serving search results using a large network of LINUX servers. The twist is that the company wants to use C and C++ in the developement rather than Java which I am proficient in.
    This makes it even funnier! That's some complex crap you talk about. You are the one who got employed, therefore, you should now exactly what is expected of you. And you expect someone to help you? Isn't that why mommy and daddy payed for college? I have to laugh! When you need help, you join a programming forum, and expect to get an answer? So funny....I have no sympathy for you at all.
    Videogame Memories!
    A site dedicated to keeping videogame memories alive!

    http://www.videogamememories.com/
    Share your experiences with us now!

    "We will game forever!"

  7. #7
    Hardware Engineer
    Join Date
    Sep 2001
    Posts
    1,398
    You got hired for a job programming...
    No. He wasn’t hired as a programmer. He was hired to manage the project. I’ve worked for non-technical managers, and it’s not a problem if they are good managers.

    Your lack of big-project experience might be a bigger issue than the lack of C/C++ knowledge. I assume that you have some management experience.

    Be honest about your lack of specific knowledge with your superiors and your subordinates.

    Hire good people and learn from them. You are going to need help from more experienced programmers to develop a schedule & budget. Fire the first subordinate that tries to “snow” you!

    Break the project down into sub-projects. Make sure your schedule has plenty of milestones…. Most projects get behind schedule, and the earlier you realize that you are in trouble, the better!

    Nail-down the goals/requirements/specs with your managers. Document any changes in the requirements along the way…. Changes and additions will impact the schedule, and you’ll need to justify schedule-slippage.

    If you don't know what you're doing, make sure you deal with people who do.

    Pournelle's Law

  8. #8
    For Narnia! Sentral's Avatar
    Join Date
    May 2005
    Location
    Narnia
    Posts
    719
    No. He wasn’t hired as a programmer. He was hired to manage the project. I’ve worked for non-technical managers, and it’s not a problem if they are good managers.
    I'm pretty sure he was hired as a programmer.Hhe knows Java, but he doesn't know C/C++, which is soooooooo funny!
    Videogame Memories!
    A site dedicated to keeping videogame memories alive!

    http://www.videogamememories.com/
    Share your experiences with us now!

    "We will game forever!"

  9. #9
    (?<!re)tired Mario F.'s Avatar
    Join Date
    May 2006
    Location
    Ireland
    Posts
    8,446
    Quote Originally Posted by Sentral
    This makes it even funnier! That's some complex crap you talk about. You are the one who got employed, therefore, you should now exactly what is expected of you. And you expect someone to help you? Isn't that why mommy and daddy payed for college? I have to laugh! When you need help, you join a programming forum, and expect to get an answer? So funny....I have no sympathy for you at all.
    I'll be blunt.

    I suspect Sentral that you have no professional experience in the programming world, I also suspect you have no team management experience or, as a matter of fact, even actually understood one thing of what was asked by the OP.

    I suggest you participate in teams thrown together in a hurry where each programmer comes from a completely different background simply because there's no time for anything better or even the resources to pay for anything better.

    I suggest also you come off that high chair of yours where the world is perfect and companies only do perfect and right choices as far as hiring or project analysis is concerned. Then you will be in the Real world where small low budget companies are given huge contracts, where your best programmer may just leave you for another company with a month's notice because they get their own office or a yellow porsche instead of a red one, and you are left with an empty space on your already short list of good programmers. Where winning a project on a highly competitive world means lying to the customer saying that the company has all the needed resources and know-how.

    Then you will pitty the middle-man like the original poster that (may not be his own case, mind you) just a few days ago was an expert and respected programmer and today was thrown into a new language and into new responsabilities where the chances of him being considered incompetent are as high as low certainly will be the tolerance of his superiors in case he fails.
    Originally Posted by brewbuck:
    Reimplementing a large system in another language to get a 25% performance boost is nonsense. It would be cheaper to just get a computer which is 25% faster.

  10. #10
    semi-colon generator ChaosEngine's Avatar
    Join Date
    Sep 2005
    Location
    Chch, NZ
    Posts
    597
    Quote Originally Posted by Sentral
    I'm pretty sure he was hired as a programmer.Hhe knows Java, but he doesn't know C/C++, which is soooooooo funny!
    Sentral, I don't know why you find this so amusing. By your own admission you've been programming in C++ for just over a year. As far as I'm concerned, you don't C++ either.

    Mario's right, stop acting like a child.
    "I saw a sign that said 'Drink Canada Dry', so I started"
    -- Brendan Behan

    Free Compiler: Visual C++ 2005 Express
    If you program in C++, you need Boost. You should also know how to use the Standard Library (STL). Want to make games? After reading this, I don't like WxWidgets anymore. Want to add some scripting to your App?

  11. #11
    Registered User
    Join Date
    Aug 2006
    Posts
    74
    Good luck to the original poster. I would help, but I don't have the knowledge to do so.

    For the rest of you guys you are kinda goofy.

  12. #12
    Reverse Engineer maxorator's Avatar
    Join Date
    Aug 2005
    Location
    Estonia
    Posts
    2,318
    Quote Originally Posted by Sentral
    I'm pretty sure he was hired as a programmer.Hhe knows Java, but he doesn't know C/C++, which is soooooooo funny!
    It is normal. People who look at C and C++ and think it is not easy enough for them, decide to use an easy language.
    "The Internet treats censorship as damage and routes around it." - John Gilmore

  13. #13
    and the hat of int overfl Salem's Avatar
    Join Date
    Aug 2001
    Location
    The edge of the known universe
    Posts
    39,660
    This isn't going to go anywhere.

    @joshskender
    Talk to your management
    Hire competent people to assist you.

    There is a lot you need to know, and you're not going to get it for free in a couple of message replies on a message board.
    If you dance barefoot on the broken glass of undefined behaviour, you've got to expect the occasional cut.
    If at first you don't succeed, try writing your phone number on the exam paper.

Popular pages Recent additions subscribe to a feed

Similar Threads

  1. Format Precision & Scale
    By mattnewtoc in forum C Programming
    Replies: 1
    Last Post: 09-16-2008, 10:34 AM
  2. [Large file][Value too large for defined data type]
    By salsan in forum Linux Programming
    Replies: 11
    Last Post: 02-05-2008, 04:18 AM
  3. Scale in perfmon tool
    By George2 in forum Tech Board
    Replies: 0
    Last Post: 02-03-2008, 01:54 AM
  4. Representing a Large Integer with an Array
    By random_accident in forum C Programming
    Replies: 3
    Last Post: 03-03-2005, 08:56 PM
  5. Representing a Large Integer with an Array
    By random_accident in forum C++ Programming
    Replies: 3
    Last Post: 03-03-2005, 12:23 PM