Personal project vs open source project vs commercial project

I wanted to continue with a theme from this thread, but didn't want it to go too far off topic in it.

The subject of build difficulty in different projects came up, so I wanted to see if I can get some fuller discussion from those more experienced in the real world.

Up until a few weeks ago, my experience had solely been school and personal projects. None of these were very complicated. A handful of source files in a project; hit build/compile; run it; the end. Nothing complicated. Then late last year I started working on my first gui project - hand-coded, bare, Win32. A little more complicated (about 30 source files, 11k lines), as I had to add a few libraries to the linker settings, but still a pretty simple build (I primarily use Code::Blocks IDE with TDM-GCC compiler, although I do have VS2012 Ultimate as well). So that had been my most complicated experience until a few weeks back.

But then, a couple of weeks ago, I tried my hand at compiling something bigger. I wanted to get some experience helping out on an open source project so I thought I'd look at Code::Blocks. (kinda hard...trying to figure out how the 1100 files and 200,000 lines all fit together...). I've used it for 4 years, so I figured if I was going to attempt to contribute to anything, it could be that. Why not? But to contribute, you obviously need to be able to build the program from source to test changes and catch crash spots, etc. So I download the source, and long story short, I fumbled around for 2 days trying to get it to work, but no go. I was going to just give up, but something someone said in the thread I linked made me go back to it. They said that the complicated collection of source files, compiler and linker settings, etc. is standard fare for commercial projects. I figured if my goal is to eventually do this for a living, I may as well get used to it. So I picked it back up, and after 4 days of plugging away at it, finally got it to work (I've got it set up now so that it's as simple as right click -> svn update, switch to IDE and hit build).

I've been messing around with boost for 2 days trying to get it built, but it seems that Windows + MinGW is problematic. I've been to over a dozen different sites, but still no luck. Many others appear to have issues with such a combination as well.

I've also been trying to get QT to build as well, but again, no luck with using a MinGW based Windows setup.

In both cases, the common route and easiest compatibility seems to be Visual Studio and its compiler. That's fine, as I have that, but I want to get those things working with my other IDE too. So I guess, once again, I'll keep plugging away at them both to try and get them working.

I thought running Windows with gcc (MinGW) was fairly common and would be well documented for most cross platform projects. Is this not true? Am I just in an oddball configuration?

Now my questions are:

In the wide open source world and in the commercial world, is it always so complicated trying to get large projects built? I understand that cross platform software is going to involve more than its fair share of fiddling around with build configuration files to properly set up for different platform and compiler combinations, but...dang. Is it always like this? Am I just a noob simply needing more hands on experience with this?

What have your guys' experiences been on big projects? Smaller projects? Open source projects? Commercial/proprietary projects? Build and compilation experiences from years back? From more recent times? I imagine this gets easier with time, with issues still cropping up now and again; is this true?

Please, let's hear some thoughts and experiences from those of you who are veterans to those of us who are very much not veterans. I haven't really seen much discussion about this sort of thing anywhere that I've been to.

Quote:

---

In the wide open source world and in the commercial world, is it always so complicated trying to get large projects built?

---

The impression I get is that tools are selected and then you use those tools, so issues from trying to compile all of the source on a different toolchain are bound to happen. Generally the key to success is to find out how the project wants you to build. Take for example the nightly for MPC-HC. If you wanted to build part of that from scratch you better use MSVC 2010 SP1, just like the Help menu from the executable suggests.

Keep in mind, you had to build all of the libraries around it for the first time as well. I'm not keen on telling people what to do, but now that you built Boost and Qt you might as well use those in the future in your own stuff and whatever else you decide to contribute. It'll eliminate a few steps.

Quote:

---

Originally Posted by Sorin

In the wide open source world and in the commercial world, is it always so complicated trying to get large projects built?

---

I wouldn't describe myself as being part of the "open source" world or as the "commercial world", but never mind. I've worked on my share of largish projects (hundreds of man-years of effort, and more).

Generally the complexity of the build process for a large project depends on the groundwork that is done - or not done - up front. Decisions need to be made about what programming languages will be used, what tool chains (compilers, etc), build tools (make, etc), what IDEs, will be used, how the project itself is architected (eg directory structures, what parts build into libraries, etc), what third-party products (toolkits, etc) will be used, how parts written in different languages will interact, how parts compiled with different build chains will be linked together, etc etc.

The reason that this is easy for small or "personal" projects is that one person is in control - and can unilaterally decide to use one language, one build chain, one build method, one IDE, a consistent directory hierarchy, etc etc.

The reason this is harder for larger projects is that requirements might take a long time to lock down, they might change, there might be a number of contributors who each set up different parts of the overall system, there might be some development phases that occur in parallel, etc etc. The better the preparation for all those things, the less difficult this is in the long run. However, it is a very rare large project - in fact, I've yet to hear of one - for which all relevant factors can be locked down, or that never need to be changed, up front.

In some large projects, a unilateral decision might be made to restructure, and fix problems that have evolved over time. Such decisions are rare, because the effort to fix the problems can be significant. This makes such things unattractive to the development team (who are usually more interested in doing new work rather than the grind of tidying up crusty old work, and often have performance agreements linked to new functionality rather than to doing things well), customers (who will experience a hiatus in delivery, and will not understand the need for a delay), for managers (who have to try to convince a customer of the need for a hiatus), and for those who hold the purse strings (who tend to be less than appreciative at paying the salaries of a team while that team is not developing things to meet customer requirements). The more common decision is to simply limp along without fixing problems.

Having looked at the organisation of a number of large open source projects, I suspect the same difficulties arise in those (albeit, not necessarily with financial imperatives).

Building strategies and their tools haven't evolved much, despite the increased complexity of software projects. Despite well-intentioned efforts in standardization, we are still expected today to adopt strict toolchains and procedures that differ greatly from project to project. Build automation tools are a complex web of real added-value solutions mixed with completely useless and unnecessary ones that were created simply because someone though they had a better idea of how a project should be built (despite my love for Boost, this is a jab at the moronic boost.build, among others).

It's chaos in build automation land. Meanwhile project complexity has been on the constant rise, not affording to wait for this important aspect to software development to smarten up.

This is transversal across all forms of development. Open source and commercial projects all suffer from this problem. It's just easier to ignore in commercial projects because it's sort of accepted that developers (read, workers) must make use of the company tools, and the population at large won't ever have access to the source code. It doesn't matter though; it's almost certain that building strategies and tools are often arbitrary choices and each company you work for will force you into a new way of doing things. And you will always wonder if this shouldn't and couldn't be a better world.

In any case let's repeat Whiteflags core idea: You do EXACTLY as it reads in the instructions.

You do this, and problems will become much, much less. Who knows, maybe you will even build it on first try!

What grumpy said is correct, i.e. the build process is as simple as the authors have made it upfront by careful planning. Code size doesn't really have much to do with it, I'd say dependencies themselves are more at fault. In any case, I find that projects that use a CMake build system are typically very painless. I've compiled a 12,000 file project on 4 different platforms without much trouble.