Understand Agile by Understanding How It’s Not Waterfall

Introduction

Many engineers and managers have a really hard time with the Agile project management methodology (the “new way” of building things). They desperately want to have Agile projects, but implementing the methodology is a struggle. One way that I have found to be useful in understanding and implementing Agile is by doing a compare and contrast with the Waterfall methodology (the “old way” of building things).

However, most people do not understand the Waterfall methodology; the very thing that Agile is attempting to fix in order to make developing software more effective and manageable. If you do not understand what the ineffective way is, how can you successfully avoid it?

The Waterfall Straw Man

A common symptom of misunderstanding Waterfall is the ritual building and burning of the Waterfall Straw Man. If you work in software long enough, you will hear all about the Waterfall Straw Man. You’ll hear a collection of arguments about how some action/process/etc. is bad because it is “too waterfall”!

  • We can’t predict the end date; that’s waterfall.
  • We can’t plan out all the features or tasks; that’s waterfall.
  • We can’t have meetings; that’s waterfall.
  • We can’t make document; that’s waterfall.

Waterfall becomes the culmination of every ridiculous or bad practice we’ve experienced or heard about in a software project. It’s the boogeyman of managing software projects. On some teams even the accusation of “waterfall” is enough to kill any process.

The Facts about Waterfall

What is Waterfall really? The funny thing is that Waterfall is actually a very successful project management method throughout the world in manufacturing and construction. The following is the simplified waterfall approach:

  1. Identify a Project
  2. Plan the project
  3. Implement the Plan
  4. Deliver Tangible Output to the Customer

That’s it! That is waterfall! It actually has several benefits. One is that comprehensive planning up front allows effective resource planning. Also, discovering and fixing problems early on is less costly than dealing with them later. You might argue,

“Well obviously the problem is that Waterfall is not Customer centric.”

Wrong. Including the customer in the planning of a waterfall project is not uncommon. You might instead say something like the following,

“Well obviously the problem is that Waterfall doesn’t have Sprints/Standup Meetings/Backlogs/etc. !”

And you would be wrong. Waterfall can have any of those things. Those processes and components are popular in but not specific to Agile.

The Waterfall method successfully builds furniture, cars, bridges, and skyscrapers every day throughout the world. Sometimes, a customer is unhappy with the result but Waterfall gets it right many times.

The Case for Agile

So then, why use Agile if Waterfall is so great? Waterfall is very poor at managing change; which we have learned is exceptionally common in software development (nobody really knows why). A central tenet of Waterfall is that plans are made upfront, complete, and comprehensive. Changing a plan later is rare and exceptionally costly because a waterfall plan is so large scale.

You can read the Agile Manifesto to see what the creators intend Agile to be. Knowing their intent, I suggest these three, concrete improvements that Agile makes to Waterfall:

  1. Iterative Processes
  2. Frequent Delivery of Tangible Output

The Agile method would be applied in this manner:

    1. Identify a project.
    2. Plan what you know as best you can; Include the customer in the planning.
    3. Execute the project plan to deliver some tangible output to the customer quickly for validation.
    4. Discover new information from (a) your development and (b) the customer upon your delivery.

… Iterate over steps 2-4 until the desired output is delivered.

Wait! All that planning sounds an awful lot like Waterfall. Of course it does! Making plans was never the problem with Waterfall. Inability to change plans based on new information is the problem with Waterfall.

Scrum, Kanban, Xtreme Programming, and other Agile methodologies have a lot of additional behaviors and processes, but they are all built on this frame: (1) Iterative process and (2) frequent delivery of tangible output.

Conclusion

Do not fall into the trap of burning the Waterfall Straw Man. It benefits no one. In fact, I have seen some software teams implement Agile so poorly that they run into more problems than if they just implemented a straightforward Waterfall plan. At least then they would have planned before coding, interacted with the customer at least once, and worked to deliver what they promised in their plan. Better than nothing!

Agile is absolutely the way to go when building Software. It also works in other projects where change management is important (which is a surprising number of projects). Take the time to understand Project Management so that you can benefit from these proven practices.

Understand Agile by Understanding How It’s Not Waterfall

Deploying to Production with Git

On your last day at any job, it is fun to go and change a bunch of things and then leave it all with your colleagues and say “Peace Out!” One thing you could do is rewrite/rework the project’s build and deploy process. Here is a way you could do it using Git with a Django project using Nginx and UWSGI (and yes, I did this all on my last day :).

The process is (1) automate our build process with a Makefile, (2) setup a Git repo on the live server to push to, and (3) use a Git hook to automatically call our Makefile targets.

Build with a Makefile

First things first, lets write a Makefile because they are very helpful. You can replace a Makefile with some other script or set of scripts, but I find Makefiles to be a very good idea on Unix based systems. You just need something to automate the your build process.

Here is an excerpt that is pretty similar to the Makefile I used:

# Makefile for building and deploying
#

UWSGI=/etc/init.d/uwsgi
NGINX=/etc/init.d/nginx

deploy: dependencies clean minified_static_files

restart: $(UWSGI) $(NGINX)
	$(UWSGI) restart
	$(NGINX) restart

stop: $(UWSGI) $(NGINX)
	$(NGINX) stop
	$(UWSGI) stop

dependencies: dependencies.pip
	pip install -r dependencies.pip # Or requirements.txt

resources:
	python resources_build.py # Minifies static files.

minified_static_files: resources
	python manage.py collectstatic # Collect into static_files/

clean:
	@-rm -rf static_files/
	@-find . -name '__pycache__' -exec /bin/rm -rf {} ;
	@echo 'Successfully Cleaned!'

.PHONY: clean resources dependencies restart stop deploy

You can put any useful commands that you run often in your project during development or when deploying to production. Put this in the root of your project directory or anywhere else in your Git project so you will not lose it.

Setup Production Git Repo

Lets use a Bare Git Repository. Log in to your production server, create a new directory, and initialize it as a bare git repository.

mkdir prod-repo
cd prod-repo
git init --bare

You will need to add this repository to your Git remotes on your local machine. The command looks something like this:

git remote add production ssh://username@www.yourserver.com:PORT/path/to/prod-repo

FYI: The PORT is whatever port you run your ssh server on.

Post-Receive Git Hook

Now lets setup a post-receive git hook on the production bare git repository that will call your makefile (or other automatic script) once a push has been received.

vim prod-repo/hooks/post-receive

A git hook file is any executable script, so you can write it in bash, sh, python, ruby, etc. Lets keep it simple and use sh.

#!/bin/sh
#
# SOURCE_DEST is whatever directory you have configured
# uwsgi look for your app in.  This is where Git will put
# the new source files that you push to this repo.

# Variables
SOURCE_DEST=/path/to/source
GIT_DIR=/path/to/prod-repo

# Update the HEAD to latest commit
git --work-tree=$SOURCE_DEST --git-dir=$GIT_DIR checkout -f

cd $SOURCE_DEST

# Run make targets
make deploy
make restart

# Fix permissions for Code
chown -R www-user $SOURCE_DEST
chgrp -R www-user $SOURCE_DEST

Putting it all Together

To update production, just issue a git push command to your production remote:

git push production master

This will push your changes on git and run your post-receive git hook script which calls your Makefile targets. Customize this to fit your needs. You can easily add in targets to run database migrations, compile coffeescript, pre-process CSS for SASS or Less, run unit tests, etc. The sky is the limit. It would also be a good idea to use a git tag each time before you push to production. Consider using some client-side git hook to accomplish that 🙂

Deploying to Production with Git