Starting a large project, tips?

[Alpo]

Hello all,

I've had a great idea for a project that I've been wanting to start for a long time now. I had been putting it off until I could find libraries that had all the functionality my project needed, and until I had enough confidence in my coding. Plus it will undoubtedly swallow all my free time (and soul :P).

I'm in a position now where I believe I have all the tools, and am confident in my ability to use them. However I've never developed a large code base by myself. My previous unrelated attempts tell me that I'm not great at designing.

I would like to get a general sense of how everyone starts a project? Is there some systematic approach that you follow for planning? Is there any foolproof method for design that will keep me from needing to redo everything down the line?

[Elysia]

There is absolutely no foolproof way. But if you want tips, I'd say you should use a tried-and-true methodology such as Agile Programming.
One way could be to start with modelling your system in UML (at least a class diagram) as best as you can, then get to work.
Pick some features you want. Code until it's done.
Next, pick some more features you want, do the necessary refactoring, code until it's done.
Repeat.

Another way is just to go with extreme programming (XP). Just get coding. Implement features one at a time and refactor your system as necessary.
Many methodologies are made in mind with many people working together and if you're just a lone programmer, those methodologies will be overkill for you.
My favorite method is just XP. But class diagrams might be somewhat handy in a large project in order to figure out and cut dependencies as much as possible (minimizes rewrites).
And don't use structs - use classes with proper setters and getters. My experience is that many structs will end up as classes in the end and that saves you from rewriting code.

[whiteflags]

Yeah agile programming can really help. Particularly practices like test driven development and simple design, which tend to feed each other. Since, if you use test driven development, you are thinking about how at least one other class interacts with your stuff (the unit test cases) which would encourage you to design what you need and refactor a lot.

[Mario F.]

1.
If your project involves a database, start with the data model (DRM). This will help immensely in defining your object model. In fact, there are tools that can design your object model based on the data model, cutting down on the time spent designing your project.

2.
Don't get caught with feature fever. Take your large project and try to divide it in stages. Adding new features in every stage. Each development stage (including the first) should end with a working program, so that you keep your drive to code. But trying to add too much too soon to your project will increase significantly the development time, which is one of the major factors behind people never ending their personal projects.

3.
Don't code around the clock. You'll burn out. No matter your initial drive and that intense feeling of getting something done, you'll get at most 3-5 weeks of intense work before you suddenly and inexplicably lose interest. Get one or two weeks of intense work, but then start slowing down, taking your time to do other things. You'll keep your project for much longer this way.

4.
Start your project from the very beginning with some Project Manager and Bug Tracking tool (it's imperative it supports version control). I personally use Redmine for all my C++ and C# projects. Since it's a Ruby software and installation requires some knowledge of Ruby distribution models, I have it installed through a Bitnami package, which is a simple Windows point and click installation. Even if you don't plan to expose your code to open source or other forms of software development collaboration, it will help to keep you and your project on track. And if you later do decide to expose your project, you already have an infrastructure. Use this tool religiously and you may actually have some hope of finishing your project. One of the main issues with abandoned projects is lack of direction. These tools help you establish a roadmap and document the development process.

5.
If you haven't already, gain an interest in programmable interface solutions, like Python or other similar languages (e.g. Lua). Study XML or other similar document format solutions (e.g. JSON). A scripting language and a document format markup, both, go a long way to solve modern software development problems that are harder to code and maintain in the programming language of choice.

[Alpo]

Thanks guys, I had worried the question was too broad for people to try to answer, but there are great ideas here.

One way could be to start with modelling your system in UML (at least a class diagram) as best as you can, then get to work.
Pick some features you want. Code until it's done.
Next, pick some more features you want, do the necessary refactoring, code until it's done.
Repeat.

Yeah agile programming can really help. Particularly practices like test driven development and simple design, which tend to feed each other. Since, if you use test driven development, you are thinking about how at least one other class interacts with your stuff (the unit test cases) which would encourage you to design what you need and refactor a lot.

This sounds like the method I will be following, my project sort of naturally breaks up into subsystems, so that should help with the refactoring. Plus I like the idea of combining testing with documentation in test driven design, it would seem to make keeping track of things easier.

Start your project from the very beginning with some Project Manager and Bug Tracking tool (it's imperative it supports version control). I personally use Redmine for all my C++ and C# projects. Since it's a Ruby software and installation requires some knowledge of Ruby distribution models, I have it installed through a Bitnami package, which is a simple Windows point and click installation. Even if you don't plan to expose your code to open source or other forms of software development collaboration, it will help to keep you and your project on track. And if you later do decide to expose your project, you already have an infrastructure. Use this tool religiously and you may actually have some hope of finishing your project. One of the main issues with abandoned projects is lack of direction. These tools help you establish a roadmap and document the development process.

I will find one and use it, thank you Mario.

I'm also planning to use XML for the data side of things, as my project requires a lot of data. I haven't used JSON but I'll look into it to see if it could be better.

[MutantJohn]

I fail to see why agility has anything to do with programming but my advice would be to map everything out before you start. I have several notebooks full of designs and maps, as my preferred method is pen and paper.

[phantomotap]

I would like to get a general sense of how everyone starts a project?

O_o

I spend hours thinking; then, I spend hours coding; then, I spend hours testing.

Lather. Rinse. Repeat.

Seriously, I do more programming in the shower than I do on the computer.

Is there any foolproof method for design that will keep me from needing to redo everything down the line?

o_O

Don't be afraid of redoing everything. Keep your tasks modular; you'll be fine redoing things a bit at a time.

*shrug*

Actually, the buzzword bullcrap of Agile programming and other umbrella methods exist pretty much only to formalize saying "Don't fret the changes.".

[Edit]

I'm also planning to use XML for the data side of things, as my project requires a lot of data. I haven't used JSON but I'll look into it to see if it could be better.

If you plan on serializing anything that looks like arrays, you should really look into JSON formatted data.
[/Edit]

Soma

[Elysia]

XML looks way better than JSON, and if you have a good library for serialization and deserialization that supports the complexity of XML, then there is no reason to favor JSON over it--it's all a matter of preference. Of course, I wouldn't be mentioning this id I didn't think XML is better than JSON. XML is more human friendly while JSON is more parser friendly.

[stahta01]

4.
Start your project from the very beginning with some Project Manager and Bug Tracking tool (it's imperative it supports version control).

I second the use of a version control system.

If you pick Git as the version control system, I have found out the hard way that Windows users really need to start with a good ".gitattributes" file before adding your source files to the Git repo.

Tim S.

[whiteflags]

I fail to see why agility has anything to do with programming but my advice would be to map everything out before you start. I have several notebooks full of designs and maps, as my preferred method is pen and paper.

If you weren't being dismissive, Extreme Programming is really the only one methodology that focuses on how to program. It makes extensive use of test driven development and pair programming. The other methodologies are really more management focused and talk a lot about how projects, iterations, and releases are planned.

[MutantJohn]

If you weren't being dismissive, Extreme Programming is really the only one methodology that focuses on how to program. It makes extensive use of test driven development and pair programming. The other methodologies are really more management focused and talk a lot about how projects, iterations, and releases are planned.

Lol I was being silly about the nomenclature. Calling it "agile programming" sounds kind of... strange. I don't even know what the agile method is all about, to tell the truth.

I thought the general rules were, use self-documenting variables names and write everything down. There've often been times I look at this massive source code tree and every leaf of code is absolutely unreadable because there's a function and the intermediate steps aren't explained.

I think it helps because if you go back and reexamine code at a later date, commenting out every section of code will pay off more than being agile, whatever that means. I assume agile just means you write software that's highly modularized so it's receptive of change and updates ("agile" in the sense that it's light on its feet and rolls with the flow).

I've also seen both extremes of distributing code amongst files. I've seen way too many files being used and I've seen 40,000+ lines of mesh code in one file. There is a happy medium that must only exist in the creator's mind.

Something I wish I got to try is pair programming. I've never once really worked on a project with another person before. It's such a lonely life for the King of the Beasts...

[grumpy]

If you weren't being dismissive, Extreme Programming is really the only one methodology that focuses on how to program. It makes extensive use of test driven development and pair programming. The other methodologies are really more management focused and talk a lot about how projects, iterations, and releases are planned.

Having used several approaches, and assessed systems developed by others, in my time, I wouldn't agree with any of that. XP (and Agile in general) is focused on one notion of programming - specifically avoiding up-front design of the system as a whole, supporting local change (refactoring, etc) - the latter being an element of, as Soma said, not sweating the small stuff.

Most other methodologies do cover planning in more detail than Agile, but it is hardly fair to suggest they do not cover programming. The difference is that substantial elements of planning are required to occur before the programming activities, whereas Agile strips those activities out in order to get to the programming earlier.

Where I have seen Agile techniques applied to large projects, they have been tailored with more up-front emphasis on planning, requirements capture, and design (or, more specifically, architecture) before the coders are unleashed (or, in one case, as a significant activity in parallel with the Agile development). In other words, as projects get larger, Agile techniques seem to draw in more of the other methodologies that advocates suggest are not needed.

[phantomotap]

The other methodologies are really more management focused and talk a lot about how projects, iterations, and releases are planned.

O_o

Forcing pair programming on introverted people is nothing but management for managements sake; the practice in those conditions is nothing more or less than misapplication of a tool.

I only bring this up because I have the exact same problem with development methodologies that I have with the "Oh. My. God. OOPS." mindset. The best tools in the world only cause problems when your forced to use them where they don't fit.

The other methodologies, every single one, are combined of many of the exact same tools as Extreme Programming. The methodology umbrella is just a way of saying "Use exactly these tools and no others.". Nonsense. Use what works for a given environment regardless for which umbrella a tool is known.

Soma

[whiteflags]

I think it helps because if you go back and reexamine code at a later date, commenting out every section of code will pay off more than being agile, whatever that means. I assume agile just means you write software that's highly modularized so it's receptive of change and updates ("agile" in the sense that it's light on its feet and rolls with the flow)

Well, it kind of does mean that. I don't want to pretend that you don't get the idea, but really what agile does is shorten development and feedback loops so that the client, the person you write code for, can make changes. It doesn't mean that you don't write comments, documentation, or design. It also doesn't really mean that everything is modularized; the client has to make decisions on features, but by shortening development loops you basically talk with the client more, and have the chance to undo mistakes from bad communication or because the client figures out that they didn't really want that, etc., before they become expensive or hard to undo.

In an agile method you have to deliver a working program at the end of the iteration with whatever feature(s) you worked on, and that's it. The client reviews that (but the exact way that happens is varied, in XP it can be automated with acceptance testing). After N iterations you get the whole program.

Individuals really only benefit from agile because it generates a lot of deliverables quickly. Since it's Alpo's first big project, he might find success if he adopts test driven development and works feature by feature. It helps prevent things like burnout.

I only bring this up because I have the exact same problem with development methodologies that I have with the "Oh. My. God. OOPS." mindset. The best tools in the world only cause problems when your forced to use them where they don't fit.

I am not going to be Mr. Agile if that is what you are about to accuse me of.

[phantomotap]

I am not going to be Mr. Agile if that is what you are about to accuse me of.

o_O

I'm fine to assume you were lightly speaking.

Some people though actually do think that employing a given methodology solves management, direction, and development problems.

I was warning Alpo off the mindset that a given tool would, instead of could, solve problems.

[Edit]
That is, if Alpo tries on Extreme Programming only to a poor fit he should absolutely not stick with it for no better reason than methodology.

Look at the deliverable you mentioned. Having functional milestones may be perfect for whatever Alpo has of of his project. On the other hand, repeatedly trying to retrofit missed requirements in components may cause Alpo to develop a "I can't get this right." mindset killing motivation.

You may not have truly intended what I inferred, but you did claim much for a particular methodology over others because it works for you. The tools you use may not work for Alpo.
[/Edit]

Soma