If you’ve been following this series of blog posts about why so many Agile projects seem to deteriorate into waterfall, you know that I believe failure to completely close iterations (or sprints) is a major reason why. If tasks continually spill over from one iteration into the next, the system is never stable enough to demo, and without demos, the feedback loop between the developer team and the Product Owner is broken. Without a rapid feedback loop, Agile doesn’t exist. The project is just a waterfall project with weekly or biweekly status meetings.
What can developer teams do to ensure that iterations close with functioning software that allows the feedback loop to run smoothly?
The most important thing you as a developer can do is to embrace change, as the motto of the Extreme Programming (XP) movement says. Don’t accept change. Don’t tolerate change. Embrace it. Understand that change is the very basis of the evolutionary and incremental approach to software development that is at the heart of Agile methodologies.
What does this mean?
Well, consider that most useful business functionality in an application takes a long time to develop. A single complete use case may take a month or more. And as you consider all the use cases in the system, you begin to see patterns emerging. Perhaps use case 1 requires a way to validate and persist complex incoming data to a database. Use case 20 requires a way to report invalid data to a compliance authority. Use case 35 requires you to deal with the response from the compliance authority. So you begin to think about how you can avoid having to rework what you do for use case 1 to incorporate the later use cases.
This is the road to perdition.
If you want all these use cases to “come together” all at once into a grand spectacle of software delivery, you’ll find it increasingly difficult to show progress along the way. There are too many interlocking pieces that all need to work in order for the whole thing to work. This is how work spills over from one iteration to the next – because the chunks of work you’re taking on are too big.
Instead, embrace change.
Know that some of what you do in use case 1 will likely be reworked for use case 20. This is a good thing, not a bad thing – because it allows you to chunk up your work into smaller pieces that are functional but with limited feature sets.
You may ask: but doesn’t the cost of the rework add up and inflate the project cost?
Yes, a little, but if you diligently use Agile engineering techniques like Test Driven Development (TDD), Continuous Integration (CI), and so on, the cost of change will be greatly reduced, and ultimately the cost will be less than the cost of eliminating the feedback loop. Think of it this way: if you incrementally develop and constantly demo use case 1, the Product Owner may discover that use case 20 needs modification, and that use case 35 goes away. Now you’ve actually reduced cost by using an evolutionary approach. Eliminating the short feedback loop is one of the most expensive things you can do on a software delivery project.
As a member of a developer team, take the iteration planning meetings (IPMs) very seriously. Your focus should be on working with the Product Owner to break down use cases into user stories and tasks that you and your team can complete in an iteration. Keep repeating to yourself: fully functional but not necessarily fully featured. Accept that the feature set is going to grow and evolve, but always keep the system functional so that whatever limited feature set is implemented can be demoed and incorporated into the feedback loop.
For example…
So, let’s take the example of use case 1 above: Validate and persist complex incoming data into our database. Will this fit into a 2-week iteration? Most probably not. So start small. Let’s get some valid data into the database. This needs a couple of tables and some data persistence objects. Will that fit into 2 weeks? Yes, probably, with some time left over. OK, so let’s use the time left over to do 5 of the 120 total validations we will have to do. Decouple the validation from the persistence, because that makes it easier to do each piece – but it’s better software design anyway. How do we report the validation errors with no UI yet? How about writing them to a file? Sure, you’ll end up throwing away the file-writing code, but once again, you’re practicing better software design. You’ve separated performing the validations from reporting the validations. And you can immediately start demoing your validations to the Product Owner.
(Hint: don’t simply write directly from the validators to a file. Design a validation reporting interface. Implement the interface first as a file, and later as whatever it actually needs to be. Use mock testing frameworks to test-drive the design. This decoupling and abstraction is good design whether you’re using Agile or not.)
Later, you will hook up the validations to the persistence. Then you will deal with how to handle invalid data. Then you will deal with showing validation errors via a UI. All the while, you have full functionality but not necessarily full feature sets. Each piece you develop is small, well-tested, and isolated. You then begin to interconnect them (don’t forget integration testing!) to deliver full feature sets.
[My colleague Robert Pantall wrote a blog post on how he used this technique of accepting controlled rework and incremental discovery to rewire his living room. It’s a fun read.]
Fully Functional, Not Necessarily Fully Featured
Keep in mind that you don’t get to unilaterally decide how the use cases get broken down into small chunks of functionality. You work with the Product Owner to do this, so that each small piece serves some business purpose. It’s a back and forth negotiation between you and the Product Owner.
Everything you do in an IPM should support this goal of evolutionary development that fits into the iteration. Be prepared to estimate quickly so you know whether work will fit. Don’t be afraid to say that work won’t fit and needs to be even more finely chunked down. Speak up. Have a dialog. Work with the product team. Keep the feedback loop running continuously.
After the IPM, when you’re working on the stories or tasks, you may run into unexpected trouble that lengthens the story completion time. Surface this immediately to the product team. You may need to break work down even more. That’s fine. Keep reminding yourself: fully functional, not necessarily fully featured.
I do want to emphasize something at this point. An evolutionary and incremental approach to software development is not a license to hack and slash. Saying you’re using incremental development does not give you an excuse to abandon architectural vision, best practices for design, continuous code improvement by refactoring, or coding standards. You must still design a system for its intended lifetime, following enterprise architecture best practices. Adopting incremental development only lets you make some short-term compromises for the benefit of keeping the software continuously functional. As the software evolves, it must evolve towards your architectural vision and continue to maintain its design and code integrity. As I showed above in the example, if you do this correctly, you’ll mostly be writing throw-away implementations to well-designed, long-term interfaces, and later building permanent production-ready implementations of the same interfaces.
I hope I’ve shown you that change is your friend if you truly adopt incremental delivery. It isn’t something to be feared or managed or avoided. Instead, it’s what makes it possible to develop complex business functionality while at the same time allowing the Product Owner to touch and use the software and to offer continuous feedback. Properly managed through correct Agile engineering techniques, the cost of the required rework fades into insignificance compared to the cost savings of the continuous feedback loop.
Don’t fear change. Embrace it.
Next episode, I’ll focus on what the Scrum Master or Project Manager can do to keep Agile projects from descending into waterfall.