Whatever Happened To Software Engineering?

map-clip-artI’ve been working some on my mailing list to help Agile teams do better and one of the recurring topics from readers has been how to fit “normal” software engineering principles into an Agile framework. How can you do database design, for instance, if every 2 weeks the database is actually being used?

Of course there’s an easy answer: we incrementally do things in Agile. So, for instance, you might use all those database design skills, but a little at a time, instead of all at once. The first sprint you sketch out the tables, maybe add a few fields. Second sprint you change up the tables some, add some more fields. Third sprint you’re working a bit with cardinality. And so on. The concept is that we do the same stuff, only a bit at a time instead of becoming a bottleneck.

This is a good enough answer, and it works for most stuff. But I think it’s also masking a conflict that many people have: software is supposed to be engineering, not just little dips and drabs of stuff added in here or there. To hear many YAGNI pundits and others, you don’t actually do anything until the last minute, and then only in support of the exact thing you’re working on. Most engineering disciplines, however, encourage you to solve the entire problem. If you’re building a bridge, architect the bridge first, dang it. If you’re going to the moon, you’d better have some technical work around the entire trip. You don’t just launch into orbit and figure it out from there. Building incrementally is fine. But there exists fields of inquiry where there’s a long sequential process of refinement over the entire problem domain — one in which you gain execution advantages by working the entire problem at once. We seem to either have tossed this fact away or are purposefully ignoring it.

I began noticing a problem. When we talk about these things, people shut down. I suspect many folks in software engineering, especially Agile coaching, don’t have an engineering background! This can lead to a severe disadvantage when dealing with certain areas:

  • Modeling I don’t see many teams sketch, much less model. That’s a shame, because visual information is a much more effective way to discuss technical matters. Add in a bit of formal training, say 30 minutes, and teams can sketch in UML. Then you can link diagrams. There’s something to be said for lightly sketching problems on the whiteboard. Take a picture if you want to keep it. Start using a modeling tool if you realize that you’re having a group discussion around highly technical stuff. You can sketch, you can use UML, you can use a modeling tool, all without having to become a waterfall BDUF project. Really, you can. It’s the only way to go for complex projects.


  • Process Analysis Does anybody remember structured process analysis? Not to see most teams. The way most people teach Scrum and Agile is that a list of stuff appears — I guess from the sky, brought down by a dove to the Product Owner. The team only works on the stuff in front of it. Don’t spend a minute thinking about the big picture! After all, your project could end at any point in time, and you don’t want to spend one minute on things that you’ll never need.


    Of course, there ARE projects that could end at any moment, but for most of us, we’re brought on to address some kind of system: a website, a business problem, an internal need, and so on. The team, and project, has some cohesion. You’re going to be here in six months, and you’re going to be working on the same thing. For those kinds of projects, spending some time doing process analysis is a no-brainer. You get back much more than you put in. I’m not talking about anything waterfall-like. I’m simply talking about building a process model, over time, of who does what with the system and why. This can help you cut to the chase and decide which stories should be prioritized first. It can help you define and have a common understanding of your stories. It can help the Product Owner make economic decisions about the backlog, and it can help the team interject creativity into the solution, instead of just being a bunch of order-takers. Great stuff for couple hours or so each sprint.

  • Lean Startup Another one of those end-to-end, outside-the-team areas where the team and the Product Owner need to work. What hypotheses is this work supposed to be testing? What are the revenue streams? How are we addressing the gap between market and PO? In a startup world, all these things are critical — and there’s a sequential engineering-type endeavor that can take you from point A to point B. It’s a mistake not to do it.

When we teach things like good architecture or proper database design, many times we view the developer as the center of the universe and the engineering practice as something that’s completed in toto before any other work can occur. This has caused tremendous pushback from Agile teams, rightly so, because it creates handoffs, bottnecks, and unecessary documentation. But are we overreacting the other way? Isn’t there a place for long-format, sequential, detail-oriented work, even in Agile teams in complex domains? Especially in Agile teams dealing in complex domains?

Each of these has a few areas in common: 1) they’re about more than the work directly in font of the team, 2) they’re about having the experts, the team, assist the organization in its work, 3) they’re sequential, 4) they build on themselves as work gets done, 5) they’re not the work itself, and 6) they end up discovering things for the PO and the organization that wouldn’t be discovered otherwise. Because of these attributes, they tend to “fall between the cracks” when teams adopt Agile practices. They are traditionally part of what people employ engineers to do.

How about you? Are there engineering processes you miss seeing in your Agile teams? What are you doing to make up for it?

Share
If you've read this far and you're interested in Agile, you should take my No-frills Agile Tune-up Email Course, and follow me on Twitter.

One thought on “Whatever Happened To Software Engineering?

  1. Cyrille

    Hi,
    “Most engineering disciplines, however, encourage you to solve the entire problem. If you’re building a bridge, architect the bridge first, dang it. If you’re going to the moon, you’d better have some technical work around the entire trip.”

    This shows a profound misunderstanding of the iterative development promoted by Agile practices. It *is* about solving the entire problem as soon as possible, for example with a “walking skeleton” in a few days or weeks. “Don’t spend a minute thinking about the big picture!” is not the mindset at all, the big picture matters a lot. Many agile teams do “Spikes”, or “Model Storming” specifically to quickly have a big picture.

    What’s hard is to cut the entire things into very thin slices such as each slice is still a slice of the *entire working* thing. If you were to build a bridge, then you’d first deliver a structure that can resist the expected load. Then from there you’d add the ability for cars to drive on it. Then you’d make it beautiful, and so on. Obviously you’d never design half of the bridge first, then the other half.

    “working the entire problem at once” sure is a good idea. It just does not oppose what people do in Agile circles.

    It makes sense however sometimes to “gain execution advantages by working the entire problem at once”, for example when the material used is very slow to shape, like stone, or very crappy legacy code.

    The problem with software engineering is the illusion that software development is mature enough for actual engineering practices as done in other fields. This is either not true, or it is true and the current body of agile practices are its engineering practices, though quite empiricist :)

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

* Copy this password:

* Type or paste password here:

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>