Large scale crawler with C/C++???

Hello all,

I have been given the task of overseeing the development of our companys new project which is the design and implementation of a large scale crawling, processing and serving system in a distributed environment. Tasks include implementing search algorithms, data mining, improving relevancy or search results, managing terabytes of data and scaling algorithms to work on very large data sets, and serving search results using a large network of LINUX servers. The twist is that the company wants to use C and C++ in the developement rather than Java which I am proficient in.

My problem is that I have only worked with small scale models and most of my experience is in Java. I would appreciate it if anyone can provide me some information on the subject and point me in the right direction

Thanks

You got hired for a job programming, and you don't even know C/C++? Sorry, but that's too damn funny!

Quote:

---

Originally Posted by Sentral

You got hired for a job programming, and you don't even know C/C++? Sorry, but that's too damn funny!

---

newsflash, C and C++ are NOT the beginning and end of software development. What most developers tend to forget is that the actual language is merely the concrete expression of your design and algorithms. It's important, but not as important as the ideas themselves.

joshskender, are you looking for info on C++ itself or on crawlers/search algorithms? C++ is really not that dissimilar from java. Think about how you'd design this in java (without the java standard lib). The classes and packages (namespace in C++) you'd design in java can probably be written very similarly in C++.

Besides, if you are "overseeing" this project, then your main priority is the design and architecture of this system. The actual implementation and coding should be done by your C++ code monkeys! :D

A company hiring a Java programmer and then putting him in charge of a C++ project is still funny.

Quote:

---

Originally Posted by CornedBee

A company hiring a Java programmer and then putting him in charge of a C++ project is still funny.

---

true, but Sentral was implying he was unqualified as a programmer because he didn't know C or C++. At least, that's the way I read it.

besides, as I've said, anyone at team lead level or above doesn't need to be a language expert. They should be designing in the problem domain, not the language domain. The problem with C and C++ programmers in particular is that they tend to think in the language domain first.

Look at this board for example. the majority of the topic list concerns syntactical and language issues.

Quote:

---

I have been given the task of overseeing the development of our companys new project which is the design and implementation of a large scale crawling, processing and serving system in a distributed environment. Tasks include implementing search algorithms, data mining, improving relevancy or search results, managing terabytes of data and scaling algorithms to work on very large data sets, and serving search results using a large network of LINUX servers. The twist is that the company wants to use C and C++ in the developement rather than Java which I am proficient in.

---

This makes it even funnier! That's some complex crap you talk about. You are the one who got employed, therefore, you should now exactly what is expected of you. And you expect someone to help you? Isn't that why mommy and daddy payed for college? I have to laugh! When you need help, you join a programming forum, and expect to get an answer? So funny....I have no sympathy for you at all.

Quote:

---

You got hired for a job programming...

---

No. He wasn’t hired as a programmer. He was hired to manage the project. I’ve worked for non-technical managers, and it’s not a problem if they are good managers.

Your lack of big-project experience might be a bigger issue than the lack of C/C++ knowledge. I assume that you have some management experience.

Be honest about your lack of specific knowledge with your superiors and your subordinates.

Hire good people and learn from them. You are going to need help from more experienced programmers to develop a schedule & budget. Fire the first subordinate that tries to “snow” you!

Break the project down into sub-projects. Make sure your schedule has plenty of milestones…. Most projects get behind schedule, and the earlier you realize that you are in trouble, the better!

Nail-down the goals/requirements/specs with your managers. Document any changes in the requirements along the way…. Changes and additions will impact the schedule, and you’ll need to justify schedule-slippage.

Quote:

---

If you don't know what you're doing, make sure you deal with people who do.

Pournelle's Law

---

Quote:

---

No. He wasn’t hired as a programmer. He was hired to manage the project. I’ve worked for non-technical managers, and it’s not a problem if they are good managers.

---

I'm pretty sure he was hired as a programmer.Hhe knows Java, but he doesn't know C/C++, which is soooooooo funny!

Quote:

---

Originally Posted by Sentral

This makes it even funnier! That's some complex crap you talk about. You are the one who got employed, therefore, you should now exactly what is expected of you. And you expect someone to help you? Isn't that why mommy and daddy payed for college? I have to laugh! When you need help, you join a programming forum, and expect to get an answer? So funny....I have no sympathy for you at all.

---

I'll be blunt.

I suspect Sentral that you have no professional experience in the programming world, I also suspect you have no team management experience or, as a matter of fact, even actually understood one thing of what was asked by the OP.

I suggest you participate in teams thrown together in a hurry where each programmer comes from a completely different background simply because there's no time for anything better or even the resources to pay for anything better.

I suggest also you come off that high chair of yours where the world is perfect and companies only do perfect and right choices as far as hiring or project analysis is concerned. Then you will be in the Real world where small low budget companies are given huge contracts, where your best programmer may just leave you for another company with a month's notice because they get their own office or a yellow porsche instead of a red one, and you are left with an empty space on your already short list of good programmers. Where winning a project on a highly competitive world means lying to the customer saying that the company has all the needed resources and know-how.

Then you will pitty the middle-man like the original poster that (may not be his own case, mind you) just a few days ago was an expert and respected programmer and today was thrown into a new language and into new responsabilities where the chances of him being considered incompetent are as high as low certainly will be the tolerance of his superiors in case he fails.

Quote:

---

Originally Posted by Sentral

I'm pretty sure he was hired as a programmer.Hhe knows Java, but he doesn't know C/C++, which is soooooooo funny!

---

Sentral, I don't know why you find this so amusing. By your own admission you've been programming in C++ for just over a year. As far as I'm concerned, you don't C++ either.

Mario's right, stop acting like a child.

Good luck to the original poster. I would help, but I don't have the knowledge to do so.

For the rest of you guys you are kinda goofy.

Quote:

---

Originally Posted by Sentral

I'm pretty sure he was hired as a programmer.Hhe knows Java, but he doesn't know C/C++, which is soooooooo funny!

---

It is normal. People who look at C and C++ and think it is not easy enough for them, decide to use an easy language.

This isn't going to go anywhere.

@joshskender
Talk to your management
Hire competent people to assist you.

There is a lot you need to know, and you're not going to get it for free in a couple of message replies on a message board.