Thread: Personal project vs open source project vs commercial project

In the wide open source world and in the commercial world, is it always so complicated trying to get large projects built?

The impression I get is that tools are selected and then you use those tools, so issues from trying to compile all of the source on a different toolchain are bound to happen. Generally the key to success is to find out how the project wants you to build. Take for example the nightly for MPC-HC. If you wanted to build part of that from scratch you better use MSVC 2010 SP1, just like the Help menu from the executable suggests.

Keep in mind, you had to build all of the libraries around it for the first time as well. I'm not keen on telling people what to do, but now that you built Boost and Qt you might as well use those in the future in your own stuff and whatever else you decide to contribute. It'll eliminate a few steps.

Originally Posted by Sorin

In the wide open source world and in the commercial world, is it always so complicated trying to get large projects built?

I wouldn't describe myself as being part of the "open source" world or as the "commercial world", but never mind. I've worked on my share of largish projects (hundreds of man-years of effort, and more).

Generally the complexity of the build process for a large project depends on the groundwork that is done - or not done - up front. Decisions need to be made about what programming languages will be used, what tool chains (compilers, etc), build tools (make, etc), what IDEs, will be used, how the project itself is architected (eg directory structures, what parts build into libraries, etc), what third-party products (toolkits, etc) will be used, how parts written in different languages will interact, how parts compiled with different build chains will be linked together, etc etc.

The reason that this is easy for small or "personal" projects is that one person is in control - and can unilaterally decide to use one language, one build chain, one build method, one IDE, a consistent directory hierarchy, etc etc.

The reason this is harder for larger projects is that requirements might take a long time to lock down, they might change, there might be a number of contributors who each set up different parts of the overall system, there might be some development phases that occur in parallel, etc etc. The better the preparation for all those things, the less difficult this is in the long run. However, it is a very rare large project - in fact, I've yet to hear of one - for which all relevant factors can be locked down, or that never need to be changed, up front.

In some large projects, a unilateral decision might be made to restructure, and fix problems that have evolved over time. Such decisions are rare, because the effort to fix the problems can be significant. This makes such things unattractive to the development team (who are usually more interested in doing new work rather than the grind of tidying up crusty old work, and often have performance agreements linked to new functionality rather than to doing things well), customers (who will experience a hiatus in delivery, and will not understand the need for a delay), for managers (who have to try to convince a customer of the need for a hiatus), and for those who hold the purse strings (who tend to be less than appreciative at paying the salaries of a team while that team is not developing things to meet customer requirements). The more common decision is to simply limp along without fixing problems.

Having looked at the organisation of a number of large open source projects, I suspect the same difficulties arise in those (albeit, not necessarily with financial imperatives).

Building strategies and their tools haven't evolved much, despite the increased complexity of software projects. Despite well-intentioned efforts in standardization, we are still expected today to adopt strict toolchains and procedures that differ greatly from project to project. Build automation tools are a complex web of real added-value solutions mixed with completely useless and unnecessary ones that were created simply because someone though they had a better idea of how a project should be built (despite my love for Boost, this is a jab at the moronic boost.build, among others).

It's chaos in build automation land. Meanwhile project complexity has been on the constant rise, not affording to wait for this important aspect to software development to smarten up.

This is transversal across all forms of development. Open source and commercial projects all suffer from this problem. It's just easier to ignore in commercial projects because it's sort of accepted that developers (read, workers) must make use of the company tools, and the population at large won't ever have access to the source code. It doesn't matter though; it's almost certain that building strategies and tools are often arbitrary choices and each company you work for will force you into a new way of doing things. And you will always wonder if this shouldn't and couldn't be a better world.

In any case let's repeat Whiteflags core idea: You do EXACTLY as it reads in the instructions.

You do this, and problems will become much, much less. Who knows, maybe you will even build it on first try!