Tag Archives: agile

F#, Mono, Agile, Architecture, DevOps

Several people have asked online lately, “Can you really write production code in F# and mono?”, “How are you doing architecture now that you’ve switched completely to F#”, and “Aren’t you doing some kind of weird thing now that doesn’t work with interactive websites?”

I usually don’t do coding blogs, because, frankly, after a while it all gets pretty boring. Same old bytes, just new bells and whistles on top. But my approach to architecture has changed in a major way over the last five years. Might as well document it.

Description of Problem Domain. For our sample project, let’s take Newspaper23. It’s a production system with a small number of users that’s been running for a few years and has evolved through my change in philosophy.

Newspaper 23 exists for one reason: I have a very short attention span. Websites are always trying to get me to hang around, click on related stories, sign-up for things, and so on. I find this very distracting. It’s very easy for me to spend a lot of time online that’s unproductive. So a few years ago I had an idea: since the news I consume is all from a dozen or two sites, why don’t I just harvest the titles and links from those sites and put them in a big list? Then I’m choosing what content to consume based on the site brand and the title of the article, not social cues or any other nonsense.

So I got out my credit card, went over to Amazon AWS, and fired up the site.

The Old, Old Way of doing things. The first time through, I did this in a “classic” manner. I sketched out a domain model, I elaborated a bit on a database model, I built a database model in MySQL, and I created a new C# app in Visual Studio and wrote a program to do everything that was needed — get some links, save the data about them, create a page with the links, do some “other stuff”.

The problem was, I was never really clear what the program should do. Should I collect the votes the article had from reddit? Don’t know, so I’ll throw that in there and save it. Might need it later. Should I provide the ability to comment, or vote? Don’t know. I’ll add space — after all, space is cheap — and if I need it, it’s there. Should there be some kind of authentication? Beats me. (Decided to skip that one)

Don’t get me wrong; none of this was a heavyweight or Big Design Up Front process. I’m only talking about an hour or two of doodling around with things. And the beauty of modern tools is that it really doesn’t matter whether your table has 5 fields or 10. Unless you’re writing the code to do something with the data, it’s just “out there”.

It did make the “link sucking” process a little more complicated, because with different sites I had to get different pieces of data. Fair enough.

What happened. After a couple of weeks of off-and-on coding, I got the program up and running, along with a unique system for identifying data on a web page to harvest. (It was a recursive RegEx language. Please, no “but you can’t use Regex on html!” It looked good.) The code base consisted of a data tier, some controller logic on top, and it seemed to work fine.

For a while.

But over a period of just a few days or weeks, it seemed like there were always these weird edge cases, both in the sites, and more disturbingly, in the tool version, setup, and configuration. Site X would change their format, and I’d be in the code changing the way I received links. Yes, I had hard-coded in the parameters for each site, but how difficult was it to change some hard-coded values once or twice a year? Except the more source code there was, and the more I touched the code, the more errors I got. And it seemed like for one reason or the other with this way of programming there was a boatload of code to wade through and I was always horsing around with it. [insert discussion here about TDD]

By using a MVC paradigm, I had written one piece of software that I could easily lay a bunch of different controllers down on. In fact, as we all know, I could develop a “controller library” over time that would take the data and do all sorts of wondrous and amazing things. That’s the beauty of a good model — it facilitates many different solutions.

I still think this might be the way to go, assuming 1) you use good testing practices, and 2) you’re going to constantly be under the hood in the code for a long time. But it introduced all kinds of problems related to having everything pass through one executable and code base. Did I really want to horse around with the code that grabbed links, for instance, simply because I was fixing the code that wrote the web page? Why was I putting all of my eggs in one basket? OOP is a great way of doing things, but it introduces a freaking huge amount of hidden interdependencies. Some times I wonder if 90% of the problems development teams have with programming is based on the fact that in the developer’s mind there is separation of model and controller, but that in the wiring itself, things can be all hooked together in completely insane ways, usually for what seemed like good reasons at the time. The black box of OOP has a lot of very sharp edges.

The Old Way of doing things. The first thing I did was ditch C#. It’s a great language, a better Java than Java, but it was the wrong paradigm for me. The second thing I did was start to build the app from the ground-up. Programming functionally required me to write functions that did things. This sounds blazingly obvious, but it’s actually a much different way of looking at things from creating a model and adding controllers on top. I still used the MySQL datastore — had to put stuff somewhere — but my code was a LOT smaller.

It also needed much less tweaking. Remember: this is personal programming. Instead of sitting on top of a model that could do many things and having to choose what to implement, which might cover calls all over the place, my code did a few things, and there was a clear path to the data for the things it did.

What happened. There was still that problem with everything running through one spot. If I wanted to tweak anything, anywhere, I’m opening up the entire codebase to make a change.

Interestingly, since this was revision 2, the code ran for much longer without needing tweaking. This meant that when I did get around to tweaking it, I had forgotten the build setup! So I’d have a 2-line change that would require screwing around with re-remembering the build-deploy environment. That might take several hours. Plus the sites were still hard-coded. The output was hard-coded. As much as I loved data-driven programming, why was everything so coupled?

Along this time I built HN Books, a social book site for hackers. It was all static. I found I could do a lot with static web pages and JSON files. Much more than most web programmers would believe.

Hmm. Fuctional programming. Static pages. Hmmm.

The way I do things now.

So now I have a new philosophy:

  • Third time through, I decided I write small programs. As small as I can. Instead of one big honking system, my system has four programs: GetLinks, UpdateArticleLibrary, CreateSelectedArticleList, and FormatSelectedArticleList. There’s a bonus utility, CheckXPath, which lets me check out XPath strings against websites to make sure they work (in case the site layout changes). I use as few external libraries, tools, or frameworks as possible. Over the years, I’ve found that every tool I picked up had 40 tons of features, of which I only needed 3. That meant that whenever I needed to do something simple that I had never done before, I had to wade through all sorts of forums, hearing about all sorts of arcane switches and other stuff. Screw that. I do not need to buy and operate a nuclear attack sub to go fishing from the pier. So — no Visual Studio, no MySQL, no complex dependencies at all. (Having said that, I actually dumped my custom link-sucking system and went with HtmlAgilityPack. I mean heck, once it’s installed, it’s just XPath. Time spent learning XPath is not as completely sunk as time spent learning WhizBang 7). Simplify.

  • My files read and write data on the local storage. I’m not doing transactions, therefore I don’t need a transactional datastore. It’s functional coding. Things come in, things go out. The O/S already has a wonderful tool for storing things. It’s called a file system, and unless I’m going to be moving around tens of thousands of these things, it’ll work just fine, thank you. Simplify.

  • I use the O/S to schedule when the programs run and to move things around. Instead of having one program that runs once a day, split the work up into a “pipeline” and have the O/S manage the pipeline. It’s already good at managing processes — it has all kinds of tools for it. Use them. Simplify.

  • I don’t “interact”. I process things at certain times. In the old days, I’d have an instance of my program spun up in Apache, waiting around in fastCGI for a client to come and connect. Then the program would do all sorts of things depending on which client it was, what the request was, and so on.

    After many, many years of coding like this, I had to ask myself: why? Why the hell am I doing it this way? 9 times out of ten I’m delivering the same content. 9 times out of ten the client is just reading stuff. 9 times out of 10 I’m creating database connections, cursors, and all sorts of other cruft — just to send the same stuff back down the wire as I sent 2 seconds ago. For the huge majority of the use cases I can imagine, even for interactive sites, there’s no difference at all to the user between a program that sits waiting for connections and a series of programs that updates data every few seconds. This is stupid. Don’t do this any more. Decouple. Simplify.


  • I write simple queries to monitor how things are moving through the pipeline. Quite frankly, the system is running so well I don’t need to monitor it, but if I did, I’d simply monitor how things flow through the pipeline. I could even make a nice html page with graphs. Don’t need to, but it’s an easy option. In fact, I’d argue that the only things interesting to me as a owner/maintainer would be things visible at a system level, not a programming level. Huge win here: no programming skill required to look at CLR innards, just O/S skills. Simplify.

  • Nothing exists as hard-coded data. It’s all either config files or command-line parameters. Right now as I bring up the page, I can see that Hacker News isn’t returning any data. If I wanted, I could probably figure out what the problem was and fix it (assuming it was fixable on my end) in about 10 minutes. No programming required. All I need is a shell. I AM re-coding the system, kinda. I decided that since I have 10-15 functions that are the same across each executable, it would make sense to create one Visual Studio solution and share a couple of library source files. Welcome back to the days of shared C header files! I’m also making the logging more robust. Right now I log everything. (I don’t read the logs, but they are there.) This is using up too much disc space. Every few months I have to clean it out. So a more fine-tuned logging system would be nice. Maybe. Maybe not. Simplify.

  • TDD? TDD? We don’t need no stinking TDD. With purely functional programming, there are only 3 mistakes I can make: Failure to break down transforms into understandable atomic units, failure to describe the transform correctly, and failure to validate the data. If I do all three of those correctly? There’s nothing to test. It’s like trying to test an SQL select statement. The idea doesn’t make sense. Simplify.

It’s very nice. Here’s the main function for the first program in the chain, GetLinks, that gets links from sites (duh):

Code Snippet
  1. [<EntryPoint>]
  2. let main argv =
  3.     try
  4.         let opts = parseCommandLineArgs (Array.toList argv)
  5.         if opts.verbose >= Verbosity.Normal then
  6.             printfn "Options Selected:"
  7.             printfn "%A" opts
  8.             printfn ""
  9.         let config = getConfigData opts
  10.         let outputDataHolder = {
  11.             outputSections = new System.Collections.Generic.Dictionary<string, outputSection>()}
  12.         let outputData = ripLinks opts config outputDataHolder
  13.         printfn "Processing Complete for %A" opts.siteUrl
  14.         let numberOfDataPointsMatchMessage =
  15.             if outputData.outputSections.Count>0
  16.             then
  17.                 let max = outputData.outputSections |> Seq.maxBy(fun x->x.Value.outputList.Count)
  18.                 let maxCount = max.Value.outputList.Count
  19.                 let min = outputData.outputSections |> Seq.minBy(fun x->x.Value.outputList.Count)
  20.                 let minCount = min.Value.outputList.Count
  21.                 if maxCount = minCount
  22.                 then
  23.                       maxCount.ToString() + " links with " + (outputData.outputSections.Count-1).ToString() + " pieces of additional information about each link gathered."
  24.                 else "Number of links, titles, and other Data DOES NOT MATCH.\r\n" + max.Key + "  has " + maxCount.ToString() + "entries while " + min.Key + " has " + minCount.ToString() + ".\r\nCheck your configuration file or run this program with the verbose option set /V"
  25.             else
  26.                 "\r\n\r\nTHERE WERE NO SECTIONS OUTPUT.\r\nPlease check your configuration file.\r\nTry running this program with the /V option"
  27.         System.Console.WriteLine(numberOfDataPointsMatchMessage)
  28.         0 // return an integer exit code
  29.     with
  30.         | :? UserNeedsHelp as hex ->
  31.             System.Console.WriteLine("You'd like help")
  32.             System.Console.WriteLine("Good luck with that")
  33.             System.Console.WriteLine("/V for verbose output")
  34.             System.Console.WriteLine("/S:<url> to set the target site url")
  35.             System.Console.WriteLine("/C:<filename> to set the config file used. Config file is expected to have Windows line returns \r\n")
  36.             System.Console.WriteLine("/O:<filename> to set the output file created/overwritten")
  37.             System.Console.WriteLine("/N:<integer> number of links desired")
  38.             System.Console.WriteLine("")
  39.             0


This is 50 lines of code. But it’s really only 4, lines 204-207. It gets the configs passed in. It creates a new container to hold the output data, it processes the link, and then it saves the data (which happens in the ripLink function. Should be down in this one. Ugh.) The rest of it is logging and help system stuff.

I could show you ripLinks, but it’s the same deal: 50 lines of code which are really about 10. It gets the links from the page using the “ripLinksOnAPage” function (clever naming, eh?), then it processes the links according to the config file.

Let’s look at how it breaks up.

wtf-blog-getLinks-fd

As you can see, there are really only 5 functions, ripLinks, http, loadHtml, recBuildLinksUp, writeLinksOut, and ripLinksOnAPage. The rest is either library calls or a few small helper functions. Take out the logging and some of the other broilerplate, and there’s maybe 50 lines of “real code” here.

We’ve done something simple, easily describable, and concrete. We took stuff from the file system, read a webpage, wrote stuff to the file system. Logged as we went along. That’s it. No need to solve world hunger. It has value.

This code has been running for more than a year with no modifications. I expect it to continue running forever. I’m done. Isn’t that nice?

Common objections.

It’s re-inventing the wheel. In most cases, I don’t need a wheel. I need a lug nut. Yes, I know what the wheel looks like, but the cognitive load of buying a big stack of wheels and carrying them around when I just need a couple of lug nuts? Tell you what. If I start thinking about round things to fit on cars, I’ll get a wheel. Otherwise I’m fine.

It’ll never scale. The funny part about this objection is the opposite is true: the more complex your stack, the more difficult and nuanced it is to scale. I can take this app and scale it out as far as I like. Heck, copy it to a CDN. We’re done. And for those of you thinking interactivity, think long and hard whether you need immediate feedback for the user or just something that changes every few seconds or once a minute. You could be spending a huge number of processor cycles worrying about concurrent clients when you really don’t need it.

It’s poorly-thought-out. My datastore? Line-delimited text files. Code complexity? Nothing more than a few hundred lines of code. Is there anything it can’t do? Not really. But (and this would be the objection I would have made several years ago) what if you want it to do something that required the data to change? Wouldn’t you have to re-jigger every piece of code in the pipeline?

Not really. First, because I’m using name-value pairs, I can always add data and not use it. So really, what would happen if I wanted some cool new feature would be 1) I’d change the data representation where it was created, 2) I’d add the code to store it, and 3) I’d add the code to use it. If you’ve worked in a functional environment, this is the way you make changes anyway. Nothing new here.

It’s not as cool as X. Probably not.

The deployment process is brittle. Actually, although I’m not continuously deploying what I write — there’s no need to automate it when it’s just me — I’m integrating DevOps directly into the solution. There’s no one part of this solution that’s “programming” and another part that’s “deployment”. It’s all integrated together. Instead of good coupling and cohesion at the function level, I have good coupling and cohesion at both the function and the executable level. Very cool stuff.

Continuing to add features. The very next thing I’m going to do is add/adjust the logging. Having to clean out the logs once every 3 or 4 months is a chore. Next up in my quest to eliminate distractions, I’ll probably go to the target site and rip the plain text and store it here. I’ve thought about adding voting and commenting, but it’s a personal site.

None of this will require a major change or a re-think of how the architecture works. Mostly the system just works and I don’t have to mess with it. O/S updates handle updating security and scaling problems. I worked a bit to make the solution, and now the solution works for me. And isn’t that the entire idea?

Share
If you've read this far and you're interested in Agile, you should take my No-frills Agile Tune-up Email Course, and follow me on Twitter.

Publishing Your Ebook For Nerds – Lessons Learned

Working on wrapping up my third e-book. It’s a book on how to make and manage effective backlogs, or the lists of things both teams and developers do. The first two were kind of a lark — although I was trying as hard as I could, I realized that this was something that was going to take a while to learn.

This time around my e-book is much larger, and I’m hoping to hit the mainstream, both in terms of content and quality. Most all organizations have the job of coordinating their work both globally and locally, and I’ve made the book as a lesson wrapped in a story. I figure either you’ll like the lesson or the story, maybe both.

After spending several days fighting tools, I thought it’d be good to capture what I’ve learned. If you’re a tech person who wants to write an e-book, this’ll help you get from words to e-book format.

Share
If you've read this far and you're interested in Agile, you should take my No-frills Agile Tune-up Email Course, and follow me on Twitter.

Whatever Happened To Software Engineering?

map-clip-artI’ve been working some on my mailing list to help Agile teams do better and one of the recurring topics from readers has been how to fit “normal” software engineering principles into an Agile framework. How can you do database design, for instance, if every 2 weeks the database is actually being used?

Of course there’s an easy answer: we incrementally do things in Agile. So, for instance, you might use all those database design skills, but a little at a time, instead of all at once. The first sprint you sketch out the tables, maybe add a few fields. Second sprint you change up the tables some, add some more fields. Third sprint you’re working a bit with cardinality. And so on. The concept is that we do the same stuff, only a bit at a time instead of becoming a bottleneck.

This is a good enough answer, and it works for most stuff. But I think it’s also masking a conflict that many people have: software is supposed to be engineering, not just little dips and drabs of stuff added in here or there. To hear many YAGNI pundits and others, you don’t actually do anything until the last minute, and then only in support of the exact thing you’re working on. Most engineering disciplines, however, encourage you to solve the entire problem. If you’re building a bridge, architect the bridge first, dang it. If you’re going to the moon, you’d better have some technical work around the entire trip. You don’t just launch into orbit and figure it out from there. Building incrementally is fine. But there exists fields of inquiry where there’s a long sequential process of refinement over the entire problem domain — one in which you gain execution advantages by working the entire problem at once. We seem to either have tossed this fact away or are purposefully ignoring it.

I began noticing a problem. When we talk about these things, people shut down. I suspect many folks in software engineering, especially Agile coaching, don’t have an engineering background! This can lead to a severe disadvantage when dealing with certain areas:

  • Modeling I don’t see many teams sketch, much less model. That’s a shame, because visual information is a much more effective way to discuss technical matters. Add in a bit of formal training, say 30 minutes, and teams can sketch in UML. Then you can link diagrams. There’s something to be said for lightly sketching problems on the whiteboard. Take a picture if you want to keep it. Start using a modeling tool if you realize that you’re having a group discussion around highly technical stuff. You can sketch, you can use UML, you can use a modeling tool, all without having to become a waterfall BDUF project. Really, you can. It’s the only way to go for complex projects.


  • Process Analysis Does anybody remember structured process analysis? Not to see most teams. The way most people teach Scrum and Agile is that a list of stuff appears — I guess from the sky, brought down by a dove to the Product Owner. The team only works on the stuff in front of it. Don’t spend a minute thinking about the big picture! After all, your project could end at any point in time, and you don’t want to spend one minute on things that you’ll never need.


    Of course, there ARE projects that could end at any moment, but for most of us, we’re brought on to address some kind of system: a website, a business problem, an internal need, and so on. The team, and project, has some cohesion. You’re going to be here in six months, and you’re going to be working on the same thing. For those kinds of projects, spending some time doing process analysis is a no-brainer. You get back much more than you put in. I’m not talking about anything waterfall-like. I’m simply talking about building a process model, over time, of who does what with the system and why. This can help you cut to the chase and decide which stories should be prioritized first. It can help you define and have a common understanding of your stories. It can help the Product Owner make economic decisions about the backlog, and it can help the team interject creativity into the solution, instead of just being a bunch of order-takers. Great stuff for couple hours or so each sprint.

  • Lean Startup Another one of those end-to-end, outside-the-team areas where the team and the Product Owner need to work. What hypotheses is this work supposed to be testing? What are the revenue streams? How are we addressing the gap between market and PO? In a startup world, all these things are critical — and there’s a sequential engineering-type endeavor that can take you from point A to point B. It’s a mistake not to do it.

When we teach things like good architecture or proper database design, many times we view the developer as the center of the universe and the engineering practice as something that’s completed in toto before any other work can occur. This has caused tremendous pushback from Agile teams, rightly so, because it creates handoffs, bottnecks, and unecessary documentation. But are we overreacting the other way? Isn’t there a place for long-format, sequential, detail-oriented work, even in Agile teams in complex domains? Especially in Agile teams dealing in complex domains?

Each of these has a few areas in common: 1) they’re about more than the work directly in font of the team, 2) they’re about having the experts, the team, assist the organization in its work, 3) they’re sequential, 4) they build on themselves as work gets done, 5) they’re not the work itself, and 6) they end up discovering things for the PO and the organization that wouldn’t be discovered otherwise. Because of these attributes, they tend to “fall between the cracks” when teams adopt Agile practices. They are traditionally part of what people employ engineers to do.

How about you? Are there engineering processes you miss seeing in your Agile teams? What are you doing to make up for it?

Share
If you've read this far and you're interested in Agile, you should take my No-frills Agile Tune-up Email Course, and follow me on Twitter.

Introduction to the Group Grope

The military is a wonderful experience in life. Where else can you learn that a “Cluster Fuck” is a bunch of people busily working hard at nothing intelligible or productive? It’s a profane, yet colorful description of a business process many of us see every day. (Nice people use the phonetic terms. “Hey, look at that Charlie Foxtrot!”)

Yesterday when doing my startup standup startup blog entry, I used another one of those terms, not heard as much, the group grope.

Ever been to a group grope? If you’ve spent any time in the business community, I’m willing to bet you have.

A group grope is when people, who do not bear responsibility, assemble for purposes of solving a problem which none of them have the information or skills to solve.

So that last time your team got together to talk about source control, when none of you have the power to make a decision about what to do, or even have experience using various tools? You know, where you left feeling pretty good, but with the vague feeling that things were unresolved? That’s a group grope.

Group gropes are fun, yet subtly frustrating. You get to take time off from work, see your friends, talk about some issues of importance, get caught up. That’s all great. The next step, and the key indicator of a good group grope, is when each person asks the other what they know or can do about the problem without getting any positive result. “I don’t know how marketing feels about that, Bob, have you spoken to them?” “Those guys in marketing are doing a great job, and I’ve heard it could be a problem, Frank? Have you met with them lately?”

It’s like a Scooby Do episode where the mystery is never solved. It’s just a bunch of kids wandering around asking questions of each other.

It feels good, there’s a lot of fun people involved, you get to explore areas you normally don’t go into, but at the end of the day you’re left empty and unfulfilled. There’s no long-term benefit.

You can short-circuit a group grope, sometimes, by simply announcing you are in one. “Is there anybody here responsible for X? Also, guys, don’t we need some folks who know Y and guys from over in the Z department? Without them I can’t see us making much progress, can you?”

Be sure to separate a group grope from a brainstorming session. Brainstorming sessions are there to create potential ideas about a problem. The meeting ends, you have a list. You probably also have some assimilation and analysis to do to turn the list into action. Group gropes end with everybody knowing a little more about problem X, mostly trivial stuff, but nothing else of value can be gained from it.

Sometimes when you’re in a group grope you can stop it. Many times you cannot. Many times it’s just best to lie back and think of England. But if you’re the one responsible for setting a meeting up, and somebody calls “group grope” on you, it’s a good time to either kill the meeting and let folks return to work — or figure out what the hell you’re all supposed to be doing that’s going to be useful to the company. Don’t confuse social fun and a sense of learning with accomplishing something valuable.

Share
If you've read this far and you're interested in Agile, you should take my No-frills Agile Tune-up Email Course, and follow me on Twitter.

Startup Standup Startup

standup-iconsI’ve been coworking at a nearby town and a couple of us decided to try to help encourage each of us in our startups.

Being a startup junkie and an Agile Coach, I thought, “Why not a startup standup?” Each week we meet in person. There, each of us announces what they accomplished in the last week, what they’re planning in the next week, and what they’re number one problem is. No classes, no group-gropes, no fluff. Immediately after the ten-minute standup we can each help each other if one of us has an obstacle that we know about. Plus we can help each other position our work mentally in order to focus more on the right stuff and less on the wrong stuff.

So how to describe what we should be talking about?

I pulled down a couple PG essays, a blog on the Lean Startup concept, and some notes Derek Sivers made while reading the Lean Startup book. But heck, it was still too much verbiage. Put together, it was WAY too much to expect noobs without context to plough through. I decided to cut a bit. So here’s an effort to make a crash course in what you should know and talk about during a weekly report of your activities in a startup. I liberally edited for clarity and brevity.

1. What is a startup? (From PG’s 2012 essay)

Not every company is a startup. Millions of companies are started every year in the United States. Only a tiny fraction are startups. Startups are companies that make something people want that have the ability to scale rapidly. They may or may not involve technology.

For a company to grow really big, it must (a) make something lots of people want, and (b) reach and serve all those people. Barbershops are doing fine in the (a) department. Almost everyone needs their hair cut. The problem for a barbershop, as for any retail establishment, is (b). A barbershop serves customers in person, and few will travel far for a haircut. And even if they did the barbershop couldn’t accommodate them.

Writing software is a great way to solve (b), but you can still end up constrained in (a). If you write software to teach Tibetan to Hungarian speakers, you’ll be able to reach most of the people who want it, but there won’t be many of them. If you make software to teach English to Chinese speakers, however, you’re in startup territory.

Most businesses are tightly constrained in (a) or (b). The distinctive feature of successful startups is that they’re not.

 

2. Do you need a cool idea?

No, you do not.

3. What are you concentrating on and talking about while you are developing your startup? You are creating and testing value and growth hypothesis. A value hypothesis is a concrete way to determine what you are doing has value to your customer. A growth hypothesis is a concrete way to determine how you gain new customers. These are both measurable, and you have to have real results that indicate success or failure for your hypothesis. That means heavy, direct interaction with the people you are supposed to be making things that they want. More Lean Startup goodness from Sivers’ notes:

Startups are human institution designed to create new products and services under conditions of extreme uncertainty.

The stories in the magazines are lies: hard work and perseverance don’t lead to success. It’s the boring stuff that matters the most.

Startups exist to learn how to build a sustainable business. This learning can be validated scientifically by running frequent experiments.

The goal of a startup is to figure out the right thing to build – the thing customers want and will pay for – as quickly as possible.

Too many startup business plans look more like they are planning to launch a rocket ship than drive a car. They prescribe the steps to take and the results to expect in excruciating detail, and as in planning to launch a rocket, they are set up in such a way that even tiny errors in assumptions can lead to catastrophic outcomes.

The customers failed to materialize, the company had committed itself so completely that they could not adapt in time. They had “achieved failure” – successfully, faithfully, and rigorously executing a plan that turned out to have been utterly flawed.

Instead of making complex plans that are based on a lot of assumptions, you can make constant adjustments with a steering wheel called the Build-Measure-Learn feedback loop. Through this process of steering, we can learn when and if it’s time to make a sharp turn called a pivot or whether we should persevere along our current path.

Validated learning is the process of demonstrating empirically that a team has discovered valuable truths about a startup’s present and future business prospects. It is more concrete, more accurate, and faster than market forecasting or classical business planning.

Learning is the essential unit of progress for startups. The effort that is not absolutely necessary for learning what customers want can be eliminated. I call this validated learning because it is always demonstrated by positive improvements in the startup’s core metrics. As we’ve seen, it’s easy to kid yourself about what you think customers want. It’s also easy to learn things that are completely irrelevant. Thus, validated learning is backed up by empirical data collected from real customers.

Learn to see every startup in any industry as a grand experiment. The question is not “Can this product be built?” In the modern economy, almost any product that can be imagined can be built. The more pertinent questions are “Should this product be built?” and “Can we build a sustainable business around this set of products and services?” To answer those questions, we need a method for systematically breaking down a business plan into its component parts and testing each part empirically.

One of the most important lessons of the scientific method: if you cannot fail, you cannot learn.

A true experiment follows the scientific method. It begins with a clear hypothesis that makes predictions about what is supposed to happen. It then tests those predictions empirically. Just as scientific experimentation is informed by theory, startup experimentation is guided by the startup’s vision. The goal of every startup experiment is to discover how to build a sustainable business around that vision.

A minimum viable product (MVP) is simply the fastest way to get through the Build-Measure-Learn feedback loop with the minimum amount of effort. The goal of the MVP is to begin the process of learning. Its goal is to test fundamental business hypotheses.

Most entrepreneurs approach a question like this by building the product and then checking to see how customers react to it. I consider this to be exactly backward because it can lead to a lot of waste. First, if it turns out that we’re building something nobody wants, the whole exercise will be an avoidable expense of time and money. If customers won’t sign up for the free trial, they’ll never get to experience the amazing features that await them. Even if they do sign up, there are many other opportunities for waste. For example, how many features do we really need to include to appeal to early adopters? Every extra feature is a form of waste, and if we delay the test for these extra features, it comes with a tremendous potential cost in terms of learning and cycle time. The lesson of the MVP is that any additional work beyond what was required to start learning is waste, no matter how important it might have seemed at the time. [Which may include actually building anything at all]

(Dropbox:) To avoid the risk of waking up after years of development with a product nobody wanted, Drew did something unexpectedly easy: he made a video. The video is banal, a simple three-minute demonstration of the technology as it is meant to work. It was all he needed at first to test a value hypothesis. Many entrepreneurs refuse to spend any time in development at all until some initial value and growth hypotheses have been tested in the real world and they have real metrics from the tests. (Anecdotes, like “I showed it to twenty people and they liked it” are not real metrics)

MVP can seem like a dangerous branding risk. Easy solution: launch the MVP under a different brand name. Experiment under the radar and then do a public marketing launch once the product has proved itself with real customers.

Prepare for the fact that MVPs often result in bad news.

The solution to this dilemma is a commitment to iteration. You have to commit to a locked-in agreement – ahead of time – that no matter what comes of testing the MVP, you will not give up hope.

A startup’s job is to
(1) rigorously measure where it is right now, confronting the hard truths that assessment reveals, and then
(2) devise experiments to learn how to move the real numbers closer to the ideal reflected in the business plan.

The failure of the “launch it and see what happens” approach should now be evident: you will always succeed – in seeing what happens. Except in rare cases, the early results will be ambiguous, and you won’t know whether to pivot or persevere, whether to change direction or stay the course.

Entrepreneurs need to face their fears and be willing to fail, often in a public way. In fact, entrepreneurs who have a high profile, either because of personal fame or because they are operating as part of a famous brand, face an extreme version of this problem.

I recommend that every startup have a regular “pivot or persevere” meeting.

Remember that the rationale for building low-quality MVPs is that developing any features beyond what early adopters require is a form of waste. However, the logic of this takes you only so far. Once you have found success with early adopters, you want to sell to mainstream customers. Mainstream customers have different requirements and are much more demanding. A pivot is required.

Startups don’t starve; they drown.

Startups have to focus on the big experiments that lead to validated learning. The engines of growth framework helps them stay focused on the metrics that matter.

Companies using the sticky engine of growth track their attrition rate or churn rate very carefully. The churn rate is defined as the fraction of customers in any period who fail to remain engaged with the company’s product. The rules that govern the sticky engine of growth are pretty simple: if the rate of new customer acquisition exceeds the churn rate, the product will grow. The speed of growth is determined by what I call the rate of compounding, which is simply the natural growth rate minus the churn rate.

Focus needs to be on improving customer retention. This goes against the standard intuition in that if a company lacks growth, it should invest more in sales and marketing. This counterintuitive result is hard to infer from standard vanity metrics.

4. So formulating and reporting on hard metrics resulting from direct user contact that prove or disprove our value and growth hypotheses is what we should focus on. Once we do that, what are some common mistakes? (From PG’s 2005 essay)

  • Release Early. get a version 1 out fast, then improve it based on users’ reactions. By “release early” I don’t mean you should release something full of bugs, but that you should release something minimal.
  • Keep Pumping Out Features. I don’t mean, of course, that you should make your application ever more complex. By “feature” I mean one unit of hacking– one quantum of making users’ lives better. [Which should directly come from your hypotheses testing results]
  • Make Users Happy. There are two things you have to do to make people pause. The most important is to explain, as concisely as possible, what the hell your site is about. How often have you visited a site that seemed to assume you already knew what they did? The other thing I repeat is to give people everything you’ve got, right away. If you have something impressive, try to put it on the front page, because that’s the only one most visitors will see. Though indeed there’s a paradox here: the more you push the good stuff toward the front, the more likely visitors are to explore further.
  • Fear the Right Things. Most visible disasters are not so alarming as they seem. Disasters are normal in a startup: a founder quits, you discover a patent that covers what you’re doing, your servers keep crashing, you run into an insoluble technical problem, you have to change your name, a deal falls through– these are all par for the course. They won’t kill you unless you let them.And in any case, competitors are not the biggest threat. Way more startups hose themselves than get crushed by competitors. There are a lot of ways to do it, but the three main ones are internal disputes, inertia, and ignoring users. Each is, by itself, enough to kill you. But if I had to pick the worst, it would be ignoring users. If you want a recipe for a startup that’s going to die, here it is: a couple of founders who have some great idea they know everyone is going to love, and that’s what they’re going to build, no matter what.
  • Commitment Is a Self-Fulfilling Prophecy. I now have enough experience with startups to be able to say what the most important quality is in a startup founder, and it’s not what you might think. The most important quality in a startup founder is determination. Not intelligence– determination. [In fact, intelligence, like funding, can be counter-indicative of success.]Time after time VCs invest in startups founded by eminent professors. This may work in biotech, where a lot of startups simply commercialize existing research, but in software you want to invest in students, not professors. Microsoft, Yahoo, and Google were all founded by people who dropped out of school to do it. What students lack in experience they more than make up in dedication. You can lose quite a lot in the brains department and it won’t kill you. But lose even a little bit in the commitment department, and that will kill you very rapidly.
  • There Is Always Room. So for all practical purposes, there is no limit to the number of startups. Startups make wealth, which means they make things people want, and if there’s a limit on the number of things people want, we are nowhere near it. I still don’t even have a flying car.
  • Don’t Get Your Hopes Up. This is another one I’ve been repeating since long before Y Combinator. It was practically the corporate motto at Viaweb.Startup founders are naturally optimistic. They wouldn’t do it otherwise. But you should treat your optimism the way you’d treat the core of a nuclear reactor: as a source of power that’s also very dangerous. You have to build a shield around it, or it will fry you.The shielding of a reactor is not uniform; the reactor would be useless if it were. It’s pierced in a few places to let pipes in. An optimism shield has to be pierced too. I think the place to draw the line is between what you expect of yourself, and what you expect of other people. It’s ok to be optimistic about what you can do, but assume the worst about machines and other people.This is particularly necessary in a startup, because you tend to be pushing the limits of whatever you’re doing. So things don’t happen in the smooth, predictable way they do in the rest of the world. Things change suddenly, and usually for the worse.

    Shielding your optimism is nowhere more important than with deals. If your startup is doing a deal, just assume it’s not going to happen. The VCs who say they’re going to invest in you aren’t. The company that says they’re going to buy you isn’t. The big customer who wants to use your system in their whole company won’t. Then if things work out you can be pleasantly surprised.

    The reason I warn startups not to get their hopes up is not to save them from being disappointed when things fall through. It’s for a more practical reason: to prevent them from leaning their company against something that’s going to fall over, taking them with it.

  • Speed, not Money. The way I’ve described it, starting a startup sounds pretty stressful. It is. When I talk to the founders of the companies we’ve funded, they all say the same thing: I knew it would be hard, but I didn’t realize it would be this hard.So why do it? It would be worth enduring a lot of pain and stress to do something grand or heroic, but just to make money? Is making money really that important?No, not really. It seems ridiculous to me when people take business too seriously. I regard making money as a boring errand to be got out of the way as soon as possible. There is nothing grand or heroic about starting a startup per se.So no, there’s nothing particularly grand about making money. That’s not what makes startups worth the trouble. What’s important about startups is the speed. By compressing the dull but necessary task of making a living into the smallest possible time, you show respect for life, and there is something grand about that.

I could probably tighten this up a lot further given some more time, but it’s definitely decreased in size from the 50 pages it started out as!

Links/Sources:

So 1st, definition of a startup: http://www.paulgraham.com/growth.html

Here are some notes from the book Lean Startup. Good stuff in here. http://sivers.org/book/LeanStartup

Value Hypotheses. http://thesquigglyline.com/2012/03/05/creating-and-testing-a-leanstartup-value-hypothesis-creating-and-testing-a-leanstartup-value-hypothesis/

Finally, some common mistakes. http://www.paulgraham.com/startuplessons.html

Share
If you've read this far and you're interested in Agile, you should take my No-frills Agile Tune-up Email Course, and follow me on Twitter.

Why I’m Leaving My Home Office

When I’m not helping a large client, I work from my home office. I’ve been working this way for around ten years.

Today I’m leaving my office for a while to try coworking.

Why would I leave the comfort of my home to travel an hour to the nearest town to work with a bunch of strangers?

Productivity, that’s why. Random human encounters, being around other people working, the social atmosphere of an office — all things you don’t find in a home office.

It occurs to me that if co-location is so good for Agile teams, it might be good for sole founders as well. And in this business, there are no universal rules, only what works and what doesn’t work.

So for the next month or so, I’m going to try coworking.

I’ll let you know how it goes.

Share
If you've read this far and you're interested in Agile, you should take my No-frills Agile Tune-up Email Course, and follow me on Twitter.

No, You Probably Can’t Work Remotely

Agile sucks when it’s about stuff you don’t like. Just ask the guys at Yahoo.

The good news is that there is no “standard” Agile. It’s best practices around iterative and incremental development. We write books and list out a bunch of stuff, people try them all, and we compare notes. Somethings apply most all of the time, so we keep those. Some apply every now and then, so we keep those too — but keep them in our back pocket. We’re always just interested in what works for us, right now.

The bad news is that a lot of stuff that works most of the time is stuff we might not like. Being an old dinosaur, I was never a fan of TDD. Didn’t care much for fluffy charts on the wall, or playing with crayons and glue again. Did enough of that in kindergarten. This thing where you get together standing up every day seems the height of lunacy. Why not just talk like normal human beings?

It turns out we do a really bad job of trying to figure out ahead of time whether something works or not. Many times when we don’t like something, we just don’t try it. Or we try it a little bit and then give up on it without “getting it”.

Which brings me to the number one thing that we’re finding in Agile that works most of the time but people don’t want to hear about: co-located teams.

Everybody and their brother wants to work remotely, you know, in their pajamas. And it’s slowly turning out that for many types of technology work, distributed teams aren’t going to cut it.

Yahoo is bringing all their remote workers back. Startups have long found that their odds of success are dramatically multiplied when they are physically together. These aren’t just business opinions, these are moves in the marketplace that we’re seeing from groups that succeed.

Big business is still trying to crack the nut of offshoring. Yes, it’s possible, but you always end up either breaking the work up in to such small pieces as to have done it yourself, or training up a bunch of experts in how your company operates who don’t work for you. Either path is problematic.

If you think about it, this makes sense. If you are working without a lot of interaction, it’s either because you have a job that’s minutely-defined or you are an expert in this customer, this domain, and can work with little interaction.

But wait! I can hear you say. We HAVE tools for remote work. Chat, Skype, and so forth. We have TONS of tools!

Here’s where it gets a bit tricky, and remember that I’m only trying to reverse engineer why we’re observing that it doesn’t work. I don’t have all the answers.

Technology development — indeed much of human existence — is a social construct. It’s about people, personalities, relationships, teams, and so forth. The technology and the problem have little to do with anything. “Give me a good team,” the saying goes, “and I can do most anything”

Lots of truth in that.

What we want technology development to be about, however, is moving bits of data around. I have a list of requirements in Excel. We have a project plan on the web. I can chat with you online to explain the way this code works.

Our working theory, mainly because we’re all bit-heads living in the digital age, is that with the correct bits flowing the correct way, things happen. So “friends” on facebook are kinda like friends, but kinda not. Email is kind of like having a conversation, but kinda not. I “know” Joe because we follow each other on Twitter, but not really.

Agile technology development, where you are changing the nature of how you work and the definition of success dynamically, has almost nothing to do with bits of data. So it’s not like having a video conference, or emailing a specification. It’s like making real friends, with real strengths and weaknesses, and learning to use each other’s characters in something greater than the sum of the parts.

There’s not an app for that.

I like Agile when it tells me things I want to hear: project management doesn’t have to be dull, endless meetings, we can work faster the closer we work with the customer, and so forth. I don’t like it so much when it tells me things I don’t want to hear, like in many situations I have to be sitting in a room with the rest of my team instead of in my pajamas.



I saw something a month ago I want to share.

I was observing a team of 8 people from across the hall. I couldn’t hear what was being said but I saw one them stop working and look confused. He looked around the room. Everybody else was busy. So he went to the flipchart and started sketching out his problem for himself. When I looked back a minute later two of them were working around the flipchart, the second guy pointing out things to the first guy.

For many minutes everybody else kept working, many with headphones on. Then one guy got to a stopping point, looked up, and saw the conversation. He gets up and walks over. Now there’s three guys talking and drawing.

They talk for a bit, then there’s four, then five people, all around the flipchart, all working on a solution. The fifth guy was interesting. He stands by the chart for a bit, then realizes the conversation is not for him, so he goes back to his work.

They reach a conclusion and everybody goes back to their seats except the first guy, who stays to make sure he’s captured the results.

This all took place in about ten minutes. No meetings planned, no time waiting around for introductions, no stop in the flow of work, no having to bring somebody else in, no hanging threads, no pre-planned agenda, no figuring out who to invite. Just a bunch of people working together organically, naturally.

An hour later they all went out to lunch together.

I don’t know how you do something like that electronically.



the commenting system is currently down. If you would like to comment, send me an email and I promise to publish your comment unedited

Share
If you've read this far and you're interested in Agile, you should take my No-frills Agile Tune-up Email Course, and follow me on Twitter.

Agile for Familes

One of the cool things about Agile is how it is a blend of several different disciplines — sociology, applied psychology, sales, marketing, branding, team-building, structured analysis, and so on.

This is also the toughest thing about Agile: it’s not a solid set of principles. It’s best practices around iterative and incremental development. And “best practices” will change over time and in different circumstances. That means that we’re constantly trying out new ideas and seeing what works and in what kinds of situations.

In fact, that’s also the coolest part of Agile. Agile is a community of learning where we’re always trying new things in different situations and reporting on whether they work or not. One set of folks comes up with a big set of best practices, the community tries them out, people publish their lessons, then we all learn.

I thought about that today — the good and bad parts of Agile — as I read about Bruce Feiler’s “The Secrets of Happy Families: Improve Your Mornings, Rethink Family Dinner, Fight Smarter, Go Out and Play, and Much More

Bruce has gone into the Agile community and used some of our best practices to apply to family life. Things like a stand-up, or goal-setting. He’s also done a high-level review of a lot of self-help stuff from all kinds of disciplines.

There are folks who might call such books facile — simplistic, formulaic pablum for people wandering from one self-help book to another.

I think if you view these kinds of books as somehow holding the answers to life, the universe, and everything, you’re probably right. But that’s way too high of a standard.

Life’s a buffet, not a Happy Meal. You get to try new things and see what works for you. Looks like Bruce has done a lot of the legwork in putting various ideas together for you and your family — some from Agile, some from other places. Try some, see what works for you. Perhaps then you can write the version 2.0 of this book; ideas that work most of the time with most families. And that’d be a good thing for the rest of us.


ADD: For those of you who are interested in hearing more about the book, I came across Bruce’s book through this NPR story that a friend shared on G+.

Share
If you've read this far and you're interested in Agile, you should take my No-frills Agile Tune-up Email Course, and follow me on Twitter.

Good Coaching Means Being an Example

In the Agile coaching game, many times we coaches work as close as we can to several teams at one time. We’re there to provide structure, feedback, and a mentored learning environment as teams move to more agility. And of course, like any consultant, we’re paid smart friends. The client’s needs come ahead of our own.

But the goal is for the teams to become more Agile: to work better, faster, and have a happier work life and life balance.

So I was wondering how to handle a question we received a few weeks back. A friend and I were doing some Intro to Agile courses, and during one of the free-form discussions somebody flagged me over.

“So which is it?” he asked, “You told me we were supposed to have the sprint close, retro, and sprint planning all in the same day. This other guy says you split them up.”

He pointed at my friend. He was clearly frustrated.

I looked at my friend. I felt that the advice he was giving was not-so-good, but it was also important not to argue randomly in front of the client. On one hand I felt we were going the wrong way. On the other hand if we were to make the training too chaotic, nobody would get anything out of it.

“Hey, i do it this way,” I said, “and here’s the reasons”

“We’ve always told them this other way,” my friend said.

My friend and I kinda looked at each other. Both of us wondering: are we going to get into a long, boisterous discussion here? Is this the place?

So I looked back to the guy. I said “Look, I still feel my way is better, but it’s not that important thing. I like strawberry ice cream, you like chocolate. Let’s do it the way it’s always been done and see how it goes. We can always adapt later.”

There are a lot of coaches that would have a problem with this. Every little thing they coach is terribly important, and it must be done a certain way.

I’m not one of those people, and more to the point, when we coach we need to be an example. Do we want our teams full of people who either a) disagree all of the time and refuse to back down, or b) are so apathetic they can’t have an opinion on anything?

Of course not. And so we shouldn’t model that behavior. We should have “fierce opinions, lightly held” — be willing to explain and advocate, at length, why we feel a certain way. But never let that get in the way of things moving forward or the spirit of the team. Keep perspective.

I think a lot of time coaches (and the rest of us) get hung up on the details of coaching and forget the purpose of coaching.

Share
If you've read this far and you're interested in Agile, you should take my No-frills Agile Tune-up Email Course, and follow me on Twitter.

Agile Video Series Kicks-Off

A month ago I was talking to a client when the subject of Agile reporting came up. There was a lot of discussion around tools and meetings and all sorts of complexity that might occur in a large Agile program.

Feeling a bit flustered, I walked away and started drawing on a nearby whiteboard. What’s the maximum amount of visual indicators you would need to run a large, complex Agile program? It’s a small, finite number, and maybe by just getting it all out on the wall it would be clear that the real work is in running the program, not all the tooling and pretty displays. You should be able to capture, report, and analyze program status in just a few minutes per day. The rest of your time should be spent, well, working.

Took about an hour to draw it all up, and folks liked it enough to put a big “Don’t Erase!” on the board. Last I saw it was still on there.

But what really sucked was that I could see myself having the same exact conversation with another client in a month or two’s time. And having to draw it all over again. Was there some way to capture these kinds of whiteboard chats so they could be reused and shared more easily?

Turns out yes, yes there was. Using a graphics tablet and some software, I could capture a whiteboard chat and not have to repeat it. Way cool.

So then I thought, what’s the most helpful question I could answer for folks on the net? Something that they would have a difficult time getting from others?

I felt the answer was “How to prepare for Agile Adoption” because there are so many opinions, it’s tried so many different ways, and vendors have a conflict of interest — many times they’d much rather take your business and hope they can straighten things out later than tell you up front you’re doing it wrong — especially if you’re willing to sign the contract with somebody else. By being a talking head I could offer up my years of experience and not have “skin in the game”. I’m just sharing what I’ve seen.

Well this was getting to be a lot more fun than electric cooking, so, of course, I had to do a few more. After all, what would be the fun in just two? Today I finished up “Scrum Vs. Kanban”, which is a look at the two methodologies and how to apply them in your organization.

I have a list of a dozen or so topics I’d like to do, but I’m not sure if I’ll be able to do all of them. This is a new format for me and I like it because it combines teaching, movie-making, and technology development.

For those of you interested in the business side of things, the videos provide a calling card for me, they point traffic at my micro-publishing site, Tiny Giant Books, and since they’re about management in general and not some specific technology, hopefully they should hang around on the web for a long time.

But the biggest reason to do these is that all the pieces just came together. I had the tools, the material was already put together, I had deep knowledge in an area that many might find useful, and it looked like an opportunity to help lots of folks. It was just a no-brainer kind of moment.

If you have any ideas for topics or feedback from the videos you’ve watched, let me know! I’m enjoying learning this new media, and it the more feedback I get the better the result for everybody.

Share
If you've read this far and you're interested in Agile, you should take my No-frills Agile Tune-up Email Course, and follow me on Twitter.