Category Archives: Quality Avenger

Crazy Idea #4723

I’ve been thinking some lately about Operating Systems, software, functional programming, DevOps, and composable services.

I think they all may be headed for a common future.

Currently, when we want to make things happen, we move between imperative-style coding, where state is scattered all over hell and Georgia, to complicated deploy scenarios, where services talk to each other over all kinds of protocols, to even worse server architectures, where they’re set up once they just kind of grow in place. All of these pieces are moving targets with various versions and constraints. As each layer gets more complicated — the language, the services, the protocols, the architecture — we end up with more and more complex work of keeping all the pieces together and running smoothly.

We’re seeing some patterns. Pure functional programming is composable. Actually, that’s the whole idea: you compose functions into a larger transform. The Unix O/S service model, where you pipe together things like ls, grep, awk, and so forth, is also composable. Composable pieces are scriptable and easily testable; you can “play them back” to verify correctness. In addition, server architectures are also becoming scriptable, where you describe the server architecture in script and then the script builds the platform.

So why not join this all together?

Picture this: A pure FP language where you could either compile the entire code as one piece, or automatically split it up into chunks and deploy separately. You could keep the code in one place and the only thing you’d need to tweak would be the chunking. Want one big app with an input and output? Click this button. Want a dozen apps that talk to each other over a common format? Click that button. Now layer in some DevOps on top of that where certain pieces would talk to other pieces on a schedule, or across a wire, and this format, scheduling, and protocol could be specified in the code (perhaps using attributes). You could meld this into a Puppet/Ansible-style system where not only do you code the solution, but you code the deployment as well.

So you end up with one source file(s), in the same language, that covers app, inter-app-communication, scheduling, and deployment. Learn one system you learn the whole thing. Heck, roll in JS deployment and html while you’re at it. Go back to the days where you coded in one place and the things you coded just ran — you didn’t have to worry about 43 different pieces of complicated wiring spread everywhere.

Ok, now somebody go make that happen :)

If you've read this far and you're interested in Agile, you should take my No-frills Agile Tune-up Email Course, and follow me on Twitter.

F#, Mono, Agile, Architecture, DevOps

Several people have asked online lately, “Can you really write production code in F# and mono?”, “How are you doing architecture now that you’ve switched completely to F#”, and “Aren’t you doing some kind of weird thing now that doesn’t work with interactive websites?”

I usually don’t do coding blogs, because, frankly, after a while it all gets pretty boring. Same old bytes, just new bells and whistles on top. But my approach to architecture has changed in a major way over the last five years. Might as well document it.

Description of Problem Domain. For our sample project, let’s take Newspaper23. It’s a production system with a small number of users that’s been running for a few years and has evolved through my change in philosophy.

Newspaper 23 exists for one reason: I have a very short attention span. Websites are always trying to get me to hang around, click on related stories, sign-up for things, and so on. I find this very distracting. It’s very easy for me to spend a lot of time online that’s unproductive. So a few years ago I had an idea: since the news I consume is all from a dozen or two sites, why don’t I just harvest the titles and links from those sites and put them in a big list? Then I’m choosing what content to consume based on the site brand and the title of the article, not social cues or any other nonsense.

So I got out my credit card, went over to Amazon AWS, and fired up the site.

The Old, Old Way of doing things. The first time through, I did this in a “classic” manner. I sketched out a domain model, I elaborated a bit on a database model, I built a database model in MySQL, and I created a new C# app in Visual Studio and wrote a program to do everything that was needed — get some links, save the data about them, create a page with the links, do some “other stuff”.

The problem was, I was never really clear what the program should do. Should I collect the votes the article had from reddit? Don’t know, so I’ll throw that in there and save it. Might need it later. Should I provide the ability to comment, or vote? Don’t know. I’ll add space — after all, space is cheap — and if I need it, it’s there. Should there be some kind of authentication? Beats me. (Decided to skip that one)

Don’t get me wrong; none of this was a heavyweight or Big Design Up Front process. I’m only talking about an hour or two of doodling around with things. And the beauty of modern tools is that it really doesn’t matter whether your table has 5 fields or 10. Unless you’re writing the code to do something with the data, it’s just “out there”.

It did make the “link sucking” process a little more complicated, because with different sites I had to get different pieces of data. Fair enough.

What happened. After a couple of weeks of off-and-on coding, I got the program up and running, along with a unique system for identifying data on a web page to harvest. (It was a recursive RegEx language. Please, no “but you can’t use Regex on html!” It looked good.) The code base consisted of a data tier, some controller logic on top, and it seemed to work fine.

For a while.

But over a period of just a few days or weeks, it seemed like there were always these weird edge cases, both in the sites, and more disturbingly, in the tool version, setup, and configuration. Site X would change their format, and I’d be in the code changing the way I received links. Yes, I had hard-coded in the parameters for each site, but how difficult was it to change some hard-coded values once or twice a year? Except the more source code there was, and the more I touched the code, the more errors I got. And it seemed like for one reason or the other with this way of programming there was a boatload of code to wade through and I was always horsing around with it. [insert discussion here about TDD]

By using a MVC paradigm, I had written one piece of software that I could easily lay a bunch of different controllers down on. In fact, as we all know, I could develop a “controller library” over time that would take the data and do all sorts of wondrous and amazing things. That’s the beauty of a good model — it facilitates many different solutions.

I still think this might be the way to go, assuming 1) you use good testing practices, and 2) you’re going to constantly be under the hood in the code for a long time. But it introduced all kinds of problems related to having everything pass through one executable and code base. Did I really want to horse around with the code that grabbed links, for instance, simply because I was fixing the code that wrote the web page? Why was I putting all of my eggs in one basket? OOP is a great way of doing things, but it introduces a freaking huge amount of hidden interdependencies. Some times I wonder if 90% of the problems development teams have with programming is based on the fact that in the developer’s mind there is separation of model and controller, but that in the wiring itself, things can be all hooked together in completely insane ways, usually for what seemed like good reasons at the time. The black box of OOP has a lot of very sharp edges.

The Old Way of doing things. The first thing I did was ditch C#. It’s a great language, a better Java than Java, but it was the wrong paradigm for me. The second thing I did was start to build the app from the ground-up. Programming functionally required me to write functions that did things. This sounds blazingly obvious, but it’s actually a much different way of looking at things from creating a model and adding controllers on top. I still used the MySQL datastore — had to put stuff somewhere — but my code was a LOT smaller.

It also needed much less tweaking. Remember: this is personal programming. Instead of sitting on top of a model that could do many things and having to choose what to implement, which might cover calls all over the place, my code did a few things, and there was a clear path to the data for the things it did.

What happened. There was still that problem with everything running through one spot. If I wanted to tweak anything, anywhere, I’m opening up the entire codebase to make a change.

Interestingly, since this was revision 2, the code ran for much longer without needing tweaking. This meant that when I did get around to tweaking it, I had forgotten the build setup! So I’d have a 2-line change that would require screwing around with re-remembering the build-deploy environment. That might take several hours. Plus the sites were still hard-coded. The output was hard-coded. As much as I loved data-driven programming, why was everything so coupled?

Along this time I built HN Books, a social book site for hackers. It was all static. I found I could do a lot with static web pages and JSON files. Much more than most web programmers would believe.

Hmm. Fuctional programming. Static pages. Hmmm.

The way I do things now.

So now I have a new philosophy:

  • Third time through, I decided I write small programs. As small as I can. Instead of one big honking system, my system has four programs: GetLinks, UpdateArticleLibrary, CreateSelectedArticleList, and FormatSelectedArticleList. There’s a bonus utility, CheckXPath, which lets me check out XPath strings against websites to make sure they work (in case the site layout changes). I use as few external libraries, tools, or frameworks as possible. Over the years, I’ve found that every tool I picked up had 40 tons of features, of which I only needed 3. That meant that whenever I needed to do something simple that I had never done before, I had to wade through all sorts of forums, hearing about all sorts of arcane switches and other stuff. Screw that. I do not need to buy and operate a nuclear attack sub to go fishing from the pier. So — no Visual Studio, no MySQL, no complex dependencies at all. (Having said that, I actually dumped my custom link-sucking system and went with HtmlAgilityPack. I mean heck, once it’s installed, it’s just XPath. Time spent learning XPath is not as completely sunk as time spent learning WhizBang 7). Simplify.

  • My files read and write data on the local storage. I’m not doing transactions, therefore I don’t need a transactional datastore. It’s functional coding. Things come in, things go out. The O/S already has a wonderful tool for storing things. It’s called a file system, and unless I’m going to be moving around tens of thousands of these things, it’ll work just fine, thank you. Simplify.

  • I use the O/S to schedule when the programs run and to move things around. Instead of having one program that runs once a day, split the work up into a “pipeline” and have the O/S manage the pipeline. It’s already good at managing processes — it has all kinds of tools for it. Use them. Simplify.

  • I don’t “interact”. I process things at certain times. In the old days, I’d have an instance of my program spun up in Apache, waiting around in fastCGI for a client to come and connect. Then the program would do all sorts of things depending on which client it was, what the request was, and so on.

    After many, many years of coding like this, I had to ask myself: why? Why the hell am I doing it this way? 9 times out of ten I’m delivering the same content. 9 times out of ten the client is just reading stuff. 9 times out of 10 I’m creating database connections, cursors, and all sorts of other cruft — just to send the same stuff back down the wire as I sent 2 seconds ago. For the huge majority of the use cases I can imagine, even for interactive sites, there’s no difference at all to the user between a program that sits waiting for connections and a series of programs that updates data every few seconds. This is stupid. Don’t do this any more. Decouple. Simplify.

  • I write simple queries to monitor how things are moving through the pipeline. Quite frankly, the system is running so well I don’t need to monitor it, but if I did, I’d simply monitor how things flow through the pipeline. I could even make a nice html page with graphs. Don’t need to, but it’s an easy option. In fact, I’d argue that the only things interesting to me as a owner/maintainer would be things visible at a system level, not a programming level. Huge win here: no programming skill required to look at CLR innards, just O/S skills. Simplify.

  • Nothing exists as hard-coded data. It’s all either config files or command-line parameters. Right now as I bring up the page, I can see that Hacker News isn’t returning any data. If I wanted, I could probably figure out what the problem was and fix it (assuming it was fixable on my end) in about 10 minutes. No programming required. All I need is a shell. I AM re-coding the system, kinda. I decided that since I have 10-15 functions that are the same across each executable, it would make sense to create one Visual Studio solution and share a couple of library source files. Welcome back to the days of shared C header files! I’m also making the logging more robust. Right now I log everything. (I don’t read the logs, but they are there.) This is using up too much disc space. Every few months I have to clean it out. So a more fine-tuned logging system would be nice. Maybe. Maybe not. Simplify.

  • TDD? TDD? We don’t need no stinking TDD. With purely functional programming, there are only 3 mistakes I can make: Failure to break down transforms into understandable atomic units, failure to describe the transform correctly, and failure to validate the data. If I do all three of those correctly? There’s nothing to test. It’s like trying to test an SQL select statement. The idea doesn’t make sense. Simplify.

It’s very nice. Here’s the main function for the first program in the chain, GetLinks, that gets links from sites (duh):

Code Snippet
  1. [<EntryPoint>]
  2. let main argv =
  3.     try
  4.         let opts = parseCommandLineArgs (Array.toList argv)
  5.         if opts.verbose >= Verbosity.Normal then
  6.             printfn "Options Selected:"
  7.             printfn "%A" opts
  8.             printfn ""
  9.         let config = getConfigData opts
  10.         let outputDataHolder = {
  11.             outputSections = new System.Collections.Generic.Dictionary<string, outputSection>()}
  12.         let outputData = ripLinks opts config outputDataHolder
  13.         printfn "Processing Complete for %A" opts.siteUrl
  14.         let numberOfDataPointsMatchMessage =
  15.             if outputData.outputSections.Count>0
  16.             then
  17.                 let max = outputData.outputSections |> Seq.maxBy(fun x->x.Value.outputList.Count)
  18.                 let maxCount = max.Value.outputList.Count
  19.                 let min = outputData.outputSections |> Seq.minBy(fun x->x.Value.outputList.Count)
  20.                 let minCount = min.Value.outputList.Count
  21.                 if maxCount = minCount
  22.                 then
  23.                       maxCount.ToString() + " links with " + (outputData.outputSections.Count-1).ToString() + " pieces of additional information about each link gathered."
  24.                 else "Number of links, titles, and other Data DOES NOT MATCH.\r\n" + max.Key + "  has " + maxCount.ToString() + "entries while " + min.Key + " has " + minCount.ToString() + ".\r\nCheck your configuration file or run this program with the verbose option set /V"
  25.             else
  26.                 "\r\n\r\nTHERE WERE NO SECTIONS OUTPUT.\r\nPlease check your configuration file.\r\nTry running this program with the /V option"
  27.         System.Console.WriteLine(numberOfDataPointsMatchMessage)
  28.         0 // return an integer exit code
  29.     with
  30.         | :? UserNeedsHelp as hex ->
  31.             System.Console.WriteLine("You'd like help")
  32.             System.Console.WriteLine("Good luck with that")
  33.             System.Console.WriteLine("/V for verbose output")
  34.             System.Console.WriteLine("/S:<url> to set the target site url")
  35.             System.Console.WriteLine("/C:<filename> to set the config file used. Config file is expected to have Windows line returns \r\n")
  36.             System.Console.WriteLine("/O:<filename> to set the output file created/overwritten")
  37.             System.Console.WriteLine("/N:<integer> number of links desired")
  38.             System.Console.WriteLine("")
  39.             0

This is 50 lines of code. But it’s really only 4, lines 204-207. It gets the configs passed in. It creates a new container to hold the output data, it processes the link, and then it saves the data (which happens in the ripLink function. Should be down in this one. Ugh.) The rest of it is logging and help system stuff.

I could show you ripLinks, but it’s the same deal: 50 lines of code which are really about 10. It gets the links from the page using the “ripLinksOnAPage” function (clever naming, eh?), then it processes the links according to the config file.

Let’s look at how it breaks up.


As you can see, there are really only 5 functions, ripLinks, http, loadHtml, recBuildLinksUp, writeLinksOut, and ripLinksOnAPage. The rest is either library calls or a few small helper functions. Take out the logging and some of the other broilerplate, and there’s maybe 50 lines of “real code” here.

We’ve done something simple, easily describable, and concrete. We took stuff from the file system, read a webpage, wrote stuff to the file system. Logged as we went along. That’s it. No need to solve world hunger. It has value.

This code has been running for more than a year with no modifications. I expect it to continue running forever. I’m done. Isn’t that nice?

Common objections.

It’s re-inventing the wheel. In most cases, I don’t need a wheel. I need a lug nut. Yes, I know what the wheel looks like, but the cognitive load of buying a big stack of wheels and carrying them around when I just need a couple of lug nuts? Tell you what. If I start thinking about round things to fit on cars, I’ll get a wheel. Otherwise I’m fine.

It’ll never scale. The funny part about this objection is the opposite is true: the more complex your stack, the more difficult and nuanced it is to scale. I can take this app and scale it out as far as I like. Heck, copy it to a CDN. We’re done. And for those of you thinking interactivity, think long and hard whether you need immediate feedback for the user or just something that changes every few seconds or once a minute. You could be spending a huge number of processor cycles worrying about concurrent clients when you really don’t need it.

It’s poorly-thought-out. My datastore? Line-delimited text files. Code complexity? Nothing more than a few hundred lines of code. Is there anything it can’t do? Not really. But (and this would be the objection I would have made several years ago) what if you want it to do something that required the data to change? Wouldn’t you have to re-jigger every piece of code in the pipeline?

Not really. First, because I’m using name-value pairs, I can always add data and not use it. So really, what would happen if I wanted some cool new feature would be 1) I’d change the data representation where it was created, 2) I’d add the code to store it, and 3) I’d add the code to use it. If you’ve worked in a functional environment, this is the way you make changes anyway. Nothing new here.

It’s not as cool as X. Probably not.

The deployment process is brittle. Actually, although I’m not continuously deploying what I write — there’s no need to automate it when it’s just me — I’m integrating DevOps directly into the solution. There’s no one part of this solution that’s “programming” and another part that’s “deployment”. It’s all integrated together. Instead of good coupling and cohesion at the function level, I have good coupling and cohesion at both the function and the executable level. Very cool stuff.

Continuing to add features. The very next thing I’m going to do is add/adjust the logging. Having to clean out the logs once every 3 or 4 months is a chore. Next up in my quest to eliminate distractions, I’ll probably go to the target site and rip the plain text and store it here. I’ve thought about adding voting and commenting, but it’s a personal site.

None of this will require a major change or a re-think of how the architecture works. Mostly the system just works and I don’t have to mess with it. O/S updates handle updating security and scaling problems. I worked a bit to make the solution, and now the solution works for me. And isn’t that the entire idea?

If you've read this far and you're interested in Agile, you should take my No-frills Agile Tune-up Email Course, and follow me on Twitter.

Whatever Happened To Software Engineering?

map-clip-artI’ve been working some on my mailing list to help Agile teams do better and one of the recurring topics from readers has been how to fit “normal” software engineering principles into an Agile framework. How can you do database design, for instance, if every 2 weeks the database is actually being used?

Of course there’s an easy answer: we incrementally do things in Agile. So, for instance, you might use all those database design skills, but a little at a time, instead of all at once. The first sprint you sketch out the tables, maybe add a few fields. Second sprint you change up the tables some, add some more fields. Third sprint you’re working a bit with cardinality. And so on. The concept is that we do the same stuff, only a bit at a time instead of becoming a bottleneck.

This is a good enough answer, and it works for most stuff. But I think it’s also masking a conflict that many people have: software is supposed to be engineering, not just little dips and drabs of stuff added in here or there. To hear many YAGNI pundits and others, you don’t actually do anything until the last minute, and then only in support of the exact thing you’re working on. Most engineering disciplines, however, encourage you to solve the entire problem. If you’re building a bridge, architect the bridge first, dang it. If you’re going to the moon, you’d better have some technical work around the entire trip. You don’t just launch into orbit and figure it out from there. Building incrementally is fine. But there exists fields of inquiry where there’s a long sequential process of refinement over the entire problem domain — one in which you gain execution advantages by working the entire problem at once. We seem to either have tossed this fact away or are purposefully ignoring it.

I began noticing a problem. When we talk about these things, people shut down. I suspect many folks in software engineering, especially Agile coaching, don’t have an engineering background! This can lead to a severe disadvantage when dealing with certain areas:

  • Modeling I don’t see many teams sketch, much less model. That’s a shame, because visual information is a much more effective way to discuss technical matters. Add in a bit of formal training, say 30 minutes, and teams can sketch in UML. Then you can link diagrams. There’s something to be said for lightly sketching problems on the whiteboard. Take a picture if you want to keep it. Start using a modeling tool if you realize that you’re having a group discussion around highly technical stuff. You can sketch, you can use UML, you can use a modeling tool, all without having to become a waterfall BDUF project. Really, you can. It’s the only way to go for complex projects.

  • Process Analysis Does anybody remember structured process analysis? Not to see most teams. The way most people teach Scrum and Agile is that a list of stuff appears — I guess from the sky, brought down by a dove to the Product Owner. The team only works on the stuff in front of it. Don’t spend a minute thinking about the big picture! After all, your project could end at any point in time, and you don’t want to spend one minute on things that you’ll never need.

    Of course, there ARE projects that could end at any moment, but for most of us, we’re brought on to address some kind of system: a website, a business problem, an internal need, and so on. The team, and project, has some cohesion. You’re going to be here in six months, and you’re going to be working on the same thing. For those kinds of projects, spending some time doing process analysis is a no-brainer. You get back much more than you put in. I’m not talking about anything waterfall-like. I’m simply talking about building a process model, over time, of who does what with the system and why. This can help you cut to the chase and decide which stories should be prioritized first. It can help you define and have a common understanding of your stories. It can help the Product Owner make economic decisions about the backlog, and it can help the team interject creativity into the solution, instead of just being a bunch of order-takers. Great stuff for couple hours or so each sprint.

  • Lean Startup Another one of those end-to-end, outside-the-team areas where the team and the Product Owner need to work. What hypotheses is this work supposed to be testing? What are the revenue streams? How are we addressing the gap between market and PO? In a startup world, all these things are critical — and there’s a sequential engineering-type endeavor that can take you from point A to point B. It’s a mistake not to do it.

When we teach things like good architecture or proper database design, many times we view the developer as the center of the universe and the engineering practice as something that’s completed in toto before any other work can occur. This has caused tremendous pushback from Agile teams, rightly so, because it creates handoffs, bottnecks, and unecessary documentation. But are we overreacting the other way? Isn’t there a place for long-format, sequential, detail-oriented work, even in Agile teams in complex domains? Especially in Agile teams dealing in complex domains?

Each of these has a few areas in common: 1) they’re about more than the work directly in font of the team, 2) they’re about having the experts, the team, assist the organization in its work, 3) they’re sequential, 4) they build on themselves as work gets done, 5) they’re not the work itself, and 6) they end up discovering things for the PO and the organization that wouldn’t be discovered otherwise. Because of these attributes, they tend to “fall between the cracks” when teams adopt Agile practices. They are traditionally part of what people employ engineers to do.

How about you? Are there engineering processes you miss seeing in your Agile teams? What are you doing to make up for it?

If you've read this far and you're interested in Agile, you should take my No-frills Agile Tune-up Email Course, and follow me on Twitter.

Stop Creating Content You Evil Bastard

Over the last few years, I’ve been somewhat of an internet startup/content junkie: I’ve bought dozens of domains and developed sites that do all sorts of things. Some are webapps. Some are targeted niche content. Some are a mix. Some are aggregation sites. I’ve tried to get a feel for most all the different ways to create content on the web. As I’ve tried these ideas, usually I post my experiences over on HackerNews to get some feedback from the technical community.

And do I have news for you. Whatever kind of content you create on the web, you’re an evil bastard. There’s just no getting around it.

Surprised? Here are a few ways you are unknowingly nefarious:

Registrar selection. Stay away from GoDaddy. Aside from having the morals of used-car salesmen, they’ll turn off your site in an instance if they get any complaints from the feds — even if you’re doing nothing wrong.

User cookies (third party and others). If you own several web properties, surely you’re not going to keep track of the same user across them, will you?

Javascript. Scripting is the gateway to insecurity. Sites who use scripting could be trying to break into your system. Your site should not rely on Javascript.

Email collection. Don’t even think of asking for an email address, because all email lists are always just spam tools.

Content ads. The average content ad on the average blog makes something like 3 pennies a month. But to hear readers tell it, you spent 2 hours writing that blog entry just so you can pitch iPods at them. They have no idea of how web value is created, so they are suspicious of anything that’s associated with money.

Source cloaking/reader customization. Like to put up one banner for folks visiting from Reddit and another for folks visiting from HN? Be very careful. Search engines and the public distrust sites that change their text based on visitor properties (although I think if instead of calling it an article you called it an app that would work?)

Blog spam. I used to be a freelance writer back in the late 80s. We’d read stuff, do some work, then publish other stuff. This was called writing. Sometimes you only read one other article, then you created a news story from it. Sometimes you read ten articles and did three interviews. Sometimes you might watch a TV report and write an article. It all depended on your deadline and the nature of the topic.

Nowadays if you read a couple of current news stories on Apple and then write that same kind of piece, it’s called “blogspam”. If you read a couple of stories on Apple, one story on interface design back in your college days, and the Steve Jobs biography last week, then write a story, it’s called original. The premise is that style counts for shit. It’s all facts If you are adding no new facts from what appeared in some other source (say Wikipedia), you’re spamming. To hear some tell it, the internet is a collection of facts all having authoritative/original sources that should be respected. Nobody cares about your unique way of looking at current news or idiomatic way of explaining facts we all know, especially if you have, gasp, ads on your site. You can post an asinine and ill-informed rant about Apple or Microsoft on your blog, we might love that, but don’t dare just tell us the same thing we could find out reading TechCrunch in your own style. Very bad.

Niche writing. My editor used to talk to his advertisers, read the demographics of his audience, and then assign me things to write about. Sometimes they seemed pretty trite or boring. But he was always joining up the audience, the advertising money, and interesting content. My job was to make it work. Today if you do the same thing, you’re somehow being dishonest or a sell-out. Writing in the same fashion that I spent years putting food on the table is considered a modern-day form of slavery for English majors (admittedly the rates are much lower, but I’m not sure that has anything to do with the way the topics and content are created.)

Link spam. Spend some time googling for blog articles that cover things you care about, then make informed and interesting comments on them, linking back to your site? (I don’t do this much at all, by the way) This is a terrible thing to do. You are actually visiting and reading somebody else’s content simply as a way to help yourself!

Pictures. I have a site where I tag and share my funny picture collection. (I started collecting interesting/funny images a couple of years ago. The collection got so big I had to do something with it.) Best I can figure, any sharing of pictures is terrible — except with your friends on Facebook, G+, or Twitter. You’re not crediting the photographer (good luck finding him/her), you’re not creating things yourself, only sharing, you’re not adding a bunch of detailed new material, only curating other people’s stuff. And so on.

Self-promotion. Mention and pump something you did, like I just did with my funny picture site above? Stand-by to be accused of being a self-promoter. On some aggregation sites, it’s even a bad thing to do to submit your own material. Heaven forfend! My youngest son Jack got a Facebook account a few years ago. When he posted things, he always clicked he liked them. It struck me as odd, but then I thought “Why not? Why would you post things you didn’t like?” But of course the faux pas he made was in actually saying something nice about his own contribution to the universe. That should only be done by others.

Fake user accounts. A large number of the major sites you use on the web have used fake user accounts some time in the past in order to make the site look more busy than it was. This is because nobody wants to use an empty site. (Good luck getting the actual numbers on this. Nobody will talk about it.)

It’s gotten to the point I could look at any site on the Internet and start making a list of how they’re not to be trusted and are screwing over the public — if I wanted to.

And let me assure you, there’s no feeling in the world like spending a week or two carefully and lovingly creating a site in order to do something nice for folks — only to have somebody online treat you as if you just knocked over a liquor store.

My internet friend Swombat recently tweeted something like “Hey! I got my first hate mail! Must be doing something right.”

He has a point: the problem here is that you can’t stay away from being declared immoral once your audience is big enough. It’s impossible. If you garner enough eyeballs, people will snark at you — your motives are poor, your methods are low, and your values are sick. If you keep getting more popular, people will start actively making arguments that you’re the devil. Hell, they’ll start following you around.

One of the first lessons I learned as a blogger is the high number of readers to participants on a blog article. You might have 100 people read, and only 1 will upvote it on an aggregation site or make a comment. The same thing is at work here. I’d guess somewhere around 1-in-1000 consumers of your content will be pissed at you. Maybe they had a bad day, maybe they misunderstood your purpose, maybe their puppy died, maybe they’re 12 and don’t understand what you’re saying, maybe they didn’t even read the article, maybe they just don’t like the cut of your jib. And here’s a huge list of reasons they can pull off the shelf to use when they vent. We’re just getting started. There are lots more.

The bigger you are, the badder you’re going to be.

(In defense of the people making critiques, many of these technicques have been overused and/or smeared by people on the web who have taken them too far and acted in poor taste. It’s reached the point that, in some consumer’s eyes, any use of these things is very suspicious. They don’t trust anybody. That’s a throw-out-the-baby-with-the-bathwater kind of thinking, but all too prevalent, sadly.)

If you've read this far and you're interested in Agile, you should take my No-frills Agile Tune-up Email Course, and follow me on Twitter.

Software: More Battlestar, Less Gunsmoke

When I was a kid growing up in the 1970s, we had a show on TV called “Gunsmoke”

Gunsmoke was a western, which meant folks dressed like cowboys. Our hero, Marshall Matt Dillon, struggled every week with capturing bad guys, cleaning the town up, and just generally doing cool cowboy stuff.

Gusmoke had been running on TV for 20 years by the 1970s. It went on to run another 10.

Even though I was just a kid, it didn’t take me long to figure out that Gunsmoke was mostly empty of content — every week it was the same old rehased plots, with guest stars showing up and always the same kind of ending to the show. Folks loved it, but there was nothing new happening. The guys at Gunsmoke were simply trying to see how many television shows they could make and sell before somebody shut them down.

This past week, major networks in the United States announced that they were shutting down a couple long-running soap operas. These things had been going for decades. “All my Children” ran for 41 years. “One Life to Live” ran for 43.

There’s a new type of show concept, however, used in “Babylon-5″ and “Battlestar Galactica” The show has individual episodes, but each episode is part of a long serialized story. The story has a beginning, middle, and finally, an end.

What a great thing it is to watch one of these shows! It’s something like a cross between a movie and a long novel. Over a period of years and dozens of hours, a long, intricate story plays out. When the end finally comes, it’s really neat.

As a small business owner, I used QuickBooks. I upgrade every few years, because QuickBooks makes me upgrade: they simply make their program break. If I want to continue using it, I have to pay. Doing business accounting hasn’t changed that much in the 15 years I’ve been a customer. After all, a double-entry bookkeeping system and chart of accounts are pretty standard stuff. But QuickBooks makes me pay anyway.

Likewise my tax program requires me to shell out money every year. While the tax code is always changing, completing taxes is basically running an expert system against the data I input. Such a system could easily be maintained in the public domain and would provide a useful service to everybody. Whatever I’m getting, by the 15th time I’ve purchased it, it’s doesn’t feel like it’s worth the money I’m paying.

Microsoft is making tens of billions of dollars from Windows and MS Office. Every couple of years, folks pay hundreds of dollars — for what? To continue to write letters? To have a file system with programs that run in a window when you click on them? Maybe really cool for the 1990s. Perhaps even a bit novel for the 2000s, But at some point enough already. The extra purchase isn’t worth any extra benefit.

I thought about this the other day as I listened to a friend of mine Tweeting about what a pain it was to install program X. He was convinced there was some great value to be had taking 3 or 4 hours of his time doing the upgrade. That may be true, but it got me to wondering: what if I tallied up all the time I’ve spent upgrading various pieces of software. Let’s forget the money for a second, assume I’m made of cash, what about the time involved?

Before I started the list, just thinking about it, it was pretty depressing. All those versions of SQL Server. All those versions of Windows. Of Office. Of various programming languages. Of various programming toolkits. Of various graphics programs and toolkits.

Wow! It could easily run into the hundreds of hours. Over a decade or two, perhaps thousands of hours. Each of those installs promised some new and shiny future: faster query times, better debugging, being able to mail merge a list of friends. These features sounded pretty cool when I read about them in a magazine or imagined how awesome they would be. But inevitably what I used them for was pretty much banal stuff: writing a letter, coding up a business tier, creating a button for a website. If I had a list of all the cool new features from each of those hundreds of installs — a list that would run into the tens of thousands of items — what, exactly, would I be using today to make somebody’s life better?

Somewhere there’s some guy who is able to create an application in 10 minutes and 3 lines of code with one arm tied behind his back using his favorite toolkit. This guy spent ten thousand dollars and three months learning how to do this. With this cool new awesome super-power, he will write exactly 1 app (if he’s lucky), and then next year it’ll be all new stuff. He will start over.

In between that time, there will be a lot of “noise”. Customers will have requirements. He’ll have to maintain existing systems. Perhaps he’ll be doing sales, or tweaking a website to bring in more contacts. New people might have to be trained. There will be seminars and business development assistance, customer relations, re-architecture discussions, and patches and upgrades to existing code.

I’m not the sharpest knife in the drawer, but it occurs to me that all of this “noise” is really what business is all about. It further occurs to me that in software, if you are making a solution for somebody, it pays to never make them completely happy. People are making a lot of money distracting us from the real purpose of what our technology tools are supposed to be doing. There is always be some magic bullet they’re brining out next quarter. All to do the one simple thing that we were basically happy with 20 years ago.

Everybody wants to do Gunsmoke and nobody wants to do Battlestar. They want to get on the gravy train, find the cash cow, and then milk the hell out of it. That would be a hoot, except I’m beginning to feel a lot like the cow.

Enough of this already. Write it up, make it configurable, publish the spec, and move on. End it. Do the right thing. Treat people as you would want to be treated.

More value, less flash. More substance, less glitter. More solutions, less fantasizing.

More Battlestar, Less Gunsmoke.

If you've read this far and you're interested in Agile, you should take my No-frills Agile Tune-up Email Course, and follow me on Twitter.

The Internet Manifesto

I want to authenticate biometrically with a nearby device — it could be clothing, jewelry, a pad, a screen, or a display/keyboard combination. If many are nearby, I want to authenticate to all of them, and I want them to coordinate to figure out where I am directing my attention and how to respond (such as a wall display tracking my hand gestures while a piece of clothing picks up my galvanic response and a piece of jewelry takes in my voice.)

Based on this authentication, I want my “profile” automatically/noiselessly loaded from the net — it should be stored in multiple places and also cached locally in case of outage.

My profile determines the places on the net where I store things and the places on the net where I am allowed to execute pieces of programs. Not that I know or bother with any of that.

The things that I store — say updated caches of current news, friend’s statuses, and other data — seamlessly sync with some device around my body, perhaps tending to be wearable things rather than physical ones. When I access these things, the device both attempts to get them from the net and updates my cache. If the net is down or up I shouldn’t know or care for non-interactive material.

I should have control over the types of material; interactive or static; the delivery format; text, video, or immersive; and the amount of material the machine “pushes” at me and the amount I am allowed to consume myself. I may decide to spend an hour reading news and commentary every morning in a flat format. That means plain text and a timer that limits that activity. Perhaps the machine will scale the volume of text dynamically to make sure I can finish in time (using semantic analysis.)

For quick replies, I want to simply say something. The machine should figure out whether to deliver my words as text or audio. I may decide to respond with a short video. The machine likewise will adapt to both sender and receiver. (Note that using something like Mechanical Turk, this is doable today)

For longer replies that require composition, I’ll continue to use a keyboard, at least until some interactive dialog system can be invented that will allow composition at a longer format.

If want entertainment, I’ll simply allocate some of my time each day to immersive interactive material. This can include all kinds of things like FB games, P2P stuff, movies, etc. The key here is that I make the management decision about my time usage separately from my consumption of the material. The net should serve me, not seduce me.

To interact with others, I simply say their name, then say what I want to tell them. The machine works out the details of where they are, what types of messages they can accept, etc. If I want to send them something, I simply describe it. “Here’s the picture of that awesome fish I caught last year” The machine knows this because as I create material I tag it with various phrases. Imagine plugging in your camera and a lightboard comes up with touch interface. Simply drag out the pictures and describe them as if you were showing them to a family member. The computer remembers and associates the tags. Same with text, audio, video, etc.

Apps are mostly dead — I don’t need artificial constructs to do most of my consumption, although to create unique types of things I might require customized interfaces. HTML is mostly dead — the consumption format I choose is vastly more nuanced than a drop-down box or a AJAX form. I am a human being, not a biological version of the browser. In fact, the idea of a “browser”, “operating system”, “application”, or “service” is nonsensical in a world where everything is integrated. I have needs to store and process things, my computer (whichever one I’m using, which is weird, because there’s never just one) manages where everything comes from, goes to, and is stored. Why do I care about any of that? I care about living, not computational details. Make it happen. Send me the bill.

If you've read this far and you're interested in Agile, you should take my No-frills Agile Tune-up Email Course, and follow me on Twitter.

App Store Roulette and Suicide Truckers

You walk into a casino. All around you are slot machines. Most folks sit with a vacant stare, feeding coins into the slots. Every now and then a little money comes out. The folks smile, then continue feeding money.

Less occasionally, lights flash, horns honk, and there’s a big payout. Maybe several hundred dollars. Maybe thousands (or more). Everybody stops what they are doing and gawks. “Gee! This could happen to me!”

And even more people are pulled into the pit to play the slots, each one thinking how he would spend the money he’d get if only he’d “hit it big”

Many developers look at this and shake their head. How could these folks be such suckers? Can’t they see how they are being manipulated?

Then they go write an app for an app store somewhere based on reasoning just as bad.

Continue reading

If you've read this far and you're interested in Agile, you should take my No-frills Agile Tune-up Email Course, and follow me on Twitter.

Media, Morals, and Money

I’m looking at the situation in Egypt, and when famous technology company founders go on TV and say something like “We don’t make moral judgments. We just run a business” I have no idea what the heck they are talking about.

I used to do high-level consulting. Still do, to some extent. Consulting was really good for keeping you on your moral toes. You had the client, the customer, your boss, and the person who writes you a paycheck, and many times these are all completely different people with completely different goals. Makes you learn to think on your feet.

I thought as I moved into the startup world that most of that ambiguity would go away, but I find it even more prevalent in the world of technology. I run a very small business — I write little apps and websites for folks — and I find I have to make moral decisions all the time. In fact, I don’t think it’s possible to run a technology company without being able to make good moral decisions.

Here are just a few of the moral questions I’ve faced:

Continue reading

If you've read this far and you're interested in Agile, you should take my No-frills Agile Tune-up Email Course, and follow me on Twitter.

The Web Control HTML 5 Left Out

On the web, we live in a binary world. Do you like something? Vote up or down. Are you through with this form? Push the button to submit it. List the books you recommend on your blog.

All of these things (and more) ask us to make a choice between A or B. We are either A or we are B.

But the real world doesn’t work that way. Yes, I like it, but not as much as that other thing. And yes, I’m kind of done with the form, but I’d also like to double check some of my answers and come back to them. And yes, those 4 books were the best Javascript books I know, but they rank completely differently if we’re talking about web programming in general.

That’s why I was happy to see this control group on a recent charity site:

It’s the slider list, a grouping of controls that lets you rank things against each other, instead of just saying yes or no.

The slider list, and the more complicated slider array, are extremely important tools. Without them we are stuck in a wed where Google presents us the “perfect” list of search results, where Facebook ranks the “perfect” list of status updates for us, and where our content aggregation sites give us the perfect rank of news stories.

But life isn’t perfect. Many times there is no one answer to what I like. Having a slider list or a slider array in HTML 5 would have been an awesome acknowledgement that the tyranny of binary thinking is over. But alas, it didn’t happen.

I’m a computer guy who also likes writing and learning about startups. If you liked this article, you’ll really like the review I did today on this cool startup book

If you've read this far and you're interested in Agile, you should take my No-frills Agile Tune-up Email Course, and follow me on Twitter.

To code quickly, you must quit coding

I did something yesterday that doubled my daily coding performance. It was easy, cheap, and made a tremendous difference in my life.

Some of you may already know what it is. For the rest of you, its going to sound wacky.

I stopped working.

Continue reading

If you've read this far and you're interested in Agile, you should take my No-frills Agile Tune-up Email Course, and follow me on Twitter.