How do I plan and run a software project

I've had programming as my hobby for almost thirteen years now. Mostly python programming (and now lately C). I'm self-taught and have acquired all I know through internet and books.
One thing I have a problem with and got a hard time finding information about, is how to plan and maintain a software project in a structured way. Maybe that's a key feature developers learn in school, but isn't sexy enough to write tutorials about... I don't know. It must however, be of great importance.
I've got a history in writing code with no planning at all. Just inventing and altering the code as I go. That usually ends up in a mess of course. At least I think so, since the code gets so much harder and harder to maintain and work with after time. With no experience in larger software projects, I tend to go for some poor choices (obviously). I think it could be avoided to some extent with proper planning.
I've tried to plan my progress too, but I don't have a clue how to do it. I usually end up getting stuck speculating it all up, and nothing gets done. I can spend hours and hours with a little piece of code, wondering if it is the right way to do it. Altering it back and forth until I just get sick of it.
Obviously no planning and some work getting done are better than nothing gets done. But there must be a better way which suits me and makes me more productive.

I really could need a helping hand here, pointing me in the right direction. Do you know a book or tutorial for me to get my hands on. How do you do it? There are probably billions of ways to do this on, but I haven't found one yet.

Please help, thank you.

/FB

Quote:

---

Just inventing and altering the code as I go. That usually ends up in a mess of course. At least I think so, since the code gets so much harder and harder to maintain and work with after time.

---

Me too. ...I try to minimize the problems with enough comments and version control..but the difference is far less satisfying from what I'd expect after that.
As it is still a hobby for me, I think there is some difference between what we should do and what large software projects do .
Lets see what the others have got to advise us on this... :)

It's a little different for everyone... Or more accurately it's a personal pattern one falls into over time. I rather suspect it's a little different for each of us.

When working from "scratch" I generally start by thinking about what I want and how it has to work. Or, when working from specifications I will spend time making sure I understand exactly what's being asked of me. It is true that no programmer, no matter how experienced or intelligent, can ever code the solution to a problem they do not understand.

Once I "get it", I approach it like a puzzle, where you know the final outcome from the picture on the box (program specifications) and have all the pieces (as coding skill) still in the box... The task is to decide in ever smaller steps what has to happen to achieve the desired result.


Continuing the puzzle analogy...
A puzzler (and my better half is an avid jigsaw puzzle fan) will look at the picture... lets say it has a sky, a barn, an old car, and a farmhouse... and will begin by sorting pieces that look like they belong together into piles sorted by categories. For coding this is a process of breaking the overall task down into it's smaller component parts... "edges.c", "sky.c", "barn.c", "car.c", etc. until we understand the big lumps that we need.

The puzzler's next step is usually to set up the edges of the puzzle ... for us that's deciding the GUI elements of the program. They are wrapping the project in a boundary... so are we.

Next it's on to working on a specific part of the puzzle... my BH always does the sky first... I generally start with the main window or main menu.
As you work the tasks get smaller and smaller until you find yourself actually placing pieces of the puzzle to complete a specific section...

So you keep working from large sections down to individual pieces, until you end up dropping that last piece into place.

Of course the coding process continues beyond the puzzler's last piece, because now we're going to continue on and make changes and refinements until the code is "optimal"...


On large projects, just like large puzzles, this is not a 10 minute process, it may take days or months. For example: I recently did a project to convert thousands of playlists in various unicode formats to a standardized UTF8-M3U format. The project seemed simple enough at first, just read the file and re-write it... then I started understanding how different the source files were and complexity poured in by bucketfulls. In the end the project took nearly a month and when we gathered all the playlists and ran the program across them, it completed in about 6 minutes... Which is pretty typical from my experience.

Quote:

---

Continuing the puzzle analogy...

---

...what you describe assumes that the player never makes a catastrophic mistake...
..As a result of such mistakes, I currently have 3 trunks for a single project....all started from scratch when something goes horribly wrong in the others....but still contains some very good and useful code....
In the puzzle, it can be described as ..say...placing the house in the puzzle correctly...and mixing up the sky and the river.

What should be the best way to keep the house in order but still correct everything...while coding ?

Fader_Berg,

Code is by definition hard to maintain. It's inevitable that being alone with your project, means sooner or latter you will lose track of your code, being able to only focus on small pieces and often being unaware of the implications it may have on other parts of the code (or even not realizing other parts of the code could contribute to your current code and you are just reinventing the wheel, often several times on the same project). Our brains just can't cope with the amount of information we often put in our code.

Strategies to avoid this invariably always revolve about documenting your code. Not commenting... documenting. You should start by choosing tools for this very thing. The advantage of working in a team is that more minds can cover more knowledge about the project code. But being alone, your only hope is to be very focused on documenting your code and build a Wiki of sorts of all your source code.

Similarly the structure of your code probably needs to change. You aim is to be developing code that is predictable in your mind. That is, you don't know the details of every module you have developed, but you know how you developed it. You know that if your module reads a file from disk than the function is named readfile(), or that if some struct or variable is exposed by your module those name follows a naming pattern you adopt consistently.

The choice of IDE will also have a tremendous impact on your ability to code more efficiently. Tools that help you analyze code very fast, learning about what functions, variables and structs are exposed, and where, will contribute to minimize the feeling of getting lost. But ultimately it always happens. If you never leave a section of your code until you are finished with it and know you followed a predictable coding strategy, you'll not gain any more encyclopedic knowledge about your project. But you will make sure you won't need it.

... as much.

For myself, it helps when I generate a document to keep track of my thoughts and the program process. For anything but the smallest of projects, I spend a bit of time planning before I even start writing any code. I figure out exactly what I want the program to do. Then I start roughly determining the functions I would need to implement the program.

For me, a flow chart does wonders in giving a clear visual representation of the program flow. With this, I am able to determine the most efficient logic and program flow before I write any code. I found that this saves me the headache of getting partially through a program only to realize that "this should have been done like this" and then confusing myself by moving code around and suddenly getting lost in pages of code that I jumbled around in an attempt to fix mistakes from poor planning.

It's critical, for larger programs, to keep the planning document updated as I continue developing the code. Even with the best planning, there are unforeseen details that must be incorporated, and it helps to carefully track these instances. Of course, there is such a thing as planning too much - a delicate balance must be found through experience and personal preference.

Quote:

---

Originally Posted by manasij7479

...what you describe assumes that the player never makes a catastrophic mistake...
..As a result of such mistakes, I currently have 3 trunks for a single project....all started from scratch when something goes horribly wrong in the others....but still contains some very good and useful code....
In the puzzle, it can be described as ..say...placing the house in the puzzle correctly...and mixing up the sky and the river.

What should be the best way to keep the house in order but still correct everything...while coding ?

---

It makes no such assumption... many times both my BH and I have taken sections of our respective projects apart and redone them.

Basically you have a choice, you can work till you've made a mistake and just give up and start over... OR... you can learn to fix the mistakes and carry on. For me mistakes are learning opportunities... not personal failings... I take advantage of them and use them to better understand the language and coding in general. On the other hand, fixing mistakes is a point of pride and I use those moments to confirm my skills and enjoy my accomplishment.

-- This is where writing programs in a highly modular structure is very helpful. By writing self-containted code in fairly small blobs, it becomes much easier to fix a broken-blob or three.

One example of this is the typical case of Windows switch() statments in the message handlers... Most people pile hundreds and hundreds of lines of code into them and when it gets broken they are lost to find the problem. What I do is to write dozens of functions, each doing it's own task and call them from the switch(). Changing or replacing things becomes very easy when you are just rewriting a small function... certainly less of a problem than replacing chunks of code in switch() cases.

Another example is that of breaking large functions down into smaller ones... Very often the smaller blobs can be replaced or edited without messing with the major function. Just as often you find you can use the sub-functions elsewhere as well.

Thank you for your response Tater. I appreciate it.

When I try to picture what I want, I often get overwhelmed by the complexity of it all. I'll need this and that... a never ending reqest list. It gets so mind boggling that my passion for it simply dies.
I tend to restrict the picture so that I get a start- and end point somewhere. But knowing about certain coming improvements gives me a hard time prioritizing which solutions to go for. What do you - with your experience - recon is the best, going straight for the picture or ease up (or have in mind, for that matter) for coming improvements as you go? Should I overcome that complex big picture before I start - and don't restrict it - no matter what.

Where do you start? I usually start with program that runs (a body or main() if you prefer) and then try to add functionality needed to it. I don't know if this is the right way to do it. It feels like it restricts my way of thinking sometimes. Should I code the things I need independently and then assembly them togehter?

I'm lost...

Here's a true cliche: writing maintainable code means writing code that is easy to read, understand, and modify. There are bunch more true cliches about what that means specifically (meaningful variable names, document, etc).

Here's some stuff I used to do that I don't do any more along the same lines, because of having to deal with my own mess:

• Re-using variables: This is like, having a long function with an int variable "tmp", that gets used various places. I actually used to re-use variables I was done with that did have a meaningful name -- from the original context, lol, and that did not at all apply to the new one. But I figured that was okay if it was just checking a return value for error or something like that, that is really obvious and occurs in one or two lines. C89 declarations kind of encourage that, especially if you are lazy. My other excuse was that it is not worth a whole new 4 bytes (or worse, a char array!) just for one purpose. I now think I am very wrong about that: it is worth it and it does make code easier to follow when you read it over.
• Not using makefiles: I only created a makefile when I had to distribute something, and then I would use GNU autotools, which largely does for you. Otherwise, I would always compile from the command line. If the project was more complex, I put a few compiler calls together in a shell script. Of course, a makefile is sort of exactly that, and they are not much harder to write. But above and beyond that, maintaining makefiles gives me a concrete perspective on the whole thing that I did not pay as much attention to before, and certainly can help provide a perspective when you come back to it later. I think a lot of IDE's can automate this for you too; I've never tried that and don't know whether I would consider it a good thing. Which raises a question I'll leave dangling: could depending on elaborate IDE's hamper the clarity of your code?

WRT the second point, this is partially about how you organize/compartmentalize in terms of file scope. If you use file scope well in C, you can get most of the "clear and understandable" advantages of formal OOP (where a file would equal a class, appropriate use of static to create "private" functions and variables, function names with matching prefixes/suffixes, etc) if you also applies the principles of good OOP (what's a class for?).

At any point where you are considering clarity and think, "I don't have to be totally anal about this, only an idiot would be confused", err on the side of the idiot, you will appreciate yourself later and it probably won't slow you down that much. Which brings me to one more point: IMO, not clarifying as you write because you don't want to slow down not just contributes to less maintainable code, it is also a "burn out" mode of programming. I enjoy it a lot more, in the long run, when I am constantly having to slow down, fine tune and clarify. I'll admit my output rate is not as high as it once was, but what I am outputting now is also of higher quality.

Quote:

---

Originally Posted by Fader_Berg

Where do you start? I usually start with program that runs (a body or main() if you prefer) and then try to add functionality needed to it. I don't know if this is the right way to do it. It feels like it restricts my way of thinking sometimes. Should I code the things I need independently and then assembly them togehter?

---

That is a great question, and relates to what I was trying to say about file scope and OOP (post #9). I would guess it is really a question of experience: the more you have written, the easier it is to intuit what you are going to need, beyond main(), before hand. The first thing I write is usually not main() -- main is the second thing, because I need it to test whatever object file I've started on. And the main() ends up short and sweet, mostly making use of and pulling together the other pieces. Again, learning to use a manual "make" system of some sort probably encourages this.

That separation of parts -- modularity -- is a basic principle of good software design, methinks. Better 5 short object files than one long spagetti style main.c! You don't have to know what they will all be at the beginning of course (but it's good to try).

Another thing about files, modularity, and clarity is that clear discrete parts give you an obvious jumping off point to document. As Mario alludes to, documenting is not just commenting: if you leave yourself a separate text file schematizing how everything works, it can be worth a thousand comments. If you want an example, I did this for someone here last week:

http://cognitivedissonance.ca/src/inetServer/

That's an htmlified version of the project tarball. The README.schema is an example of what I mean. That code is very fresh and the project needs some conceptual tweaking before it moves forward, but the organization is good, I think.

Quote:

---

Originally Posted by Fader_Berg

Thank you for your response Tater. I appreciate it.

When I try to picture what I want, I often get overwhelmed by the complexity of it all. I'll need this and that... a never ending reqest list. It gets so mind boggling that my passion for it simply dies.

---

Think in smaller blobs... Just like the puzzler usually starts with the outside edges, I usually start with the UI. When working on the UI I couldn't care crap how I'm going to open that network connection or save that settings struct... These are other issues for another time. Then when working inside the ui, I don't care crap how I created the window, I'm focused on it's controls, etc. It's a mental tunnelling down process where you decide the larger issues but spend your real mental horsepower on what's immediately before you... Then like the puzzler, I will pick a part of the program --the house-- "house.c" and work on it... I couldn't care less about the barn or the sky... I'm working on the house.

If you work with sufficient modularity you will soon discover that all you really need to care about is what's between the closest set of braces.

Quote:

---

Where do you start? I usually start with program that runs (a body or main() if you prefer) and then try to add functionality needed to it. I don't know if this is the right way to do it. It feels like it restricts my way of thinking sometimes. Should I code the things I need independently and then assembly them togehter?

I'm lost...

---

I usually start with a pen and paper writing a plan...

Thank you Mario. That gives me some new perspectives on things.
I have never been good at documenting the code. Much because I don't know how it's done. I can imagine it's good to do it though, 'cause that's what everyone says.
I've really tried to do it, but I get frustrated when I'm only repeating what the code tells me. Sometimes I've read what I wrote just the next day and found out that it gives me nothing more than the code itself (and I can't say my code is self-explaining all the time). Obviously I'm doing something wrong. I have read other peoples documentation on their code, and they all put it very well. Me, on the other hand, can't seem to find anything of importance to say. To document the code seems like a skill of its own.

I'll have to try to learn those things and is there tools to help me out here, I would be glad. What tools do you have in mind?

Regards

/FB

Quote:

---

Originally Posted by CommonTater

Think in smaller blobs... Just like the puzzler usually starts with the outside edges, I usually start with the UI.

---

hmm... Are you sure that's where you start? Seems like the last place one should want to start. How do you deflect the problem of being inadvertently coercing your business logic that should otherwise be entirely independent of any UI decisions?

I used to start with the UI when I was like 12. Since then...well, let's just say that's where a project ends. Not where it starts.

Quote:

---

Originally Posted by Fader_Berg

Thank you Mario. That gives me some new perspectives on things.
I have never been good at documenting the code. Much because I don't know how it's done. I can imagine it's good to do it though, 'cause that's what everyone says.
I've really tried to do it, but I get frustrated when I'm only repeating what the code tells me. Sometimes I've read what I wrote just the next day and found out that it gives me nothing more than the code itself (and I can't say my code is self-explaining all the time). Obviously I'm doing something wrong. I have read other peoples documentation on their code, and they all put it very well. Me, on the other hand, can't seem to find anything of importance to say. To document the code seems like a skill of its own.

I'll have to try to learn those things and is there tools to help me out here, I would be glad. What tools do you have in mind?

Regards

/FB

---

It's not what you see in the code you just wrote... It's a matter of coming back to it in 6 months or 5 years, after moving on to other things and having to recall what you did. More than once I've looked at code I wrote years before and can't for the life of me figure out what I was doing... So now everything gets commented, lists of functions are created and dated for the last time the function was changed. Between the documentation and comments I've never found myself in that situation since I started.

Quote:

---

Originally Posted by Mario F.

I used to start with the UI when I was like 12. Since then...well, let's just say that's where a project ends. Not where it starts.

---

Way I figure it --being 12 1/2 years old and all-- the UI design sets a general direction for all that follows. If I don't understand the problem well enough to present it graphically on screen, I certainly don't understand it well enough to write proper code to solve it.

Besides, in Windows API... ain't much gonna happen until you've got a working UI... it's all message driven and the messages are keyed to the main UI window... so, unless you're silly enough to write the whole darned thing without testing it...

Hmm... You kind flip software analysis upside down. But if it works for you, it works for you.

Quote:

---

Originally Posted by Fader_Berg

To document the code seems like a skill of its own.

---

If the amount of otherwise great software with totally atrocious API documentation is any indication, you are probably right. Sometimes I think programmers try to outdo each other WRT just how elliptical, non-sensical, and genuinely unhelpful they can be in the documentation.

Quote:

---

Originally Posted by CommonTater

It's not what you see in the code you just wrote... It's a matter of coming back to it in 6 months or 5 years, after moving on to other things and having to recall what you did.

---

I know that... I've got a problem to write comments and documentation in a clearifying and meaningful way. When I get back to my comments (or documentation) in a couple of months, I just go "what the f...". They are useless, no matter how much effort I put into them.

Quote:

---

Originally Posted by Mario F.

Hmm... You kind flip software analysis upside down. But if it works for you, it works for you.

---

Analysis is different than writing code...

In the early phases before "int main (void)" I work from large to small... deciding requirements as I go. So by the time I've tunnelled into the problem far enough to understand it properly, I've pretty much already designed the UI and at least the larger blobs.

Then in the writing phases it's a matter of "I need this working before I can get that working" so the order of things changes quite a bit. As I said, in windows you aren't going to do much of anything without a working UI...

Quote:

---

Originally Posted by Fader_Berg

I know that... I've got a problem to write comments and documentation in a clearifying and meaningful way. When I get back to my comments (or documentation) in a couple of months, I just go "what the f...". They are useless, no matter how much effort I put into them.

---

LOL... then in all due respect... you really do need to work on that...

One tack, taken from my high school English teacher, is to "figure out how to best represent a given concept in the fewest words"... His "simpler is always better" approach has helped me a lot over the years.

Thank you MK27 for your tip... You've got me right there, with the re-using variables and rather using scripts than make files. I'm buying your concept straight off.

You're more than welcome to share more thoughts.