Tag Archives: programming

F#, Mono, Agile, Architecture, DevOps

Several people have asked online lately, “Can you really write production code in F# and mono?”, “How are you doing architecture now that you’ve switched completely to F#”, and “Aren’t you doing some kind of weird thing now that doesn’t work with interactive websites?”

I usually don’t do coding blogs, because, frankly, after a while it all gets pretty boring. Same old bytes, just new bells and whistles on top. But my approach to architecture has changed in a major way over the last five years. Might as well document it.

Description of Problem Domain. For our sample project, let’s take Newspaper23. It’s a production system with a small number of users that’s been running for a few years and has evolved through my change in philosophy.

Newspaper 23 exists for one reason: I have a very short attention span. Websites are always trying to get me to hang around, click on related stories, sign-up for things, and so on. I find this very distracting. It’s very easy for me to spend a lot of time online that’s unproductive. So a few years ago I had an idea: since the news I consume is all from a dozen or two sites, why don’t I just harvest the titles and links from those sites and put them in a big list? Then I’m choosing what content to consume based on the site brand and the title of the article, not social cues or any other nonsense.

So I got out my credit card, went over to Amazon AWS, and fired up the site.

The Old, Old Way of doing things. The first time through, I did this in a “classic” manner. I sketched out a domain model, I elaborated a bit on a database model, I built a database model in MySQL, and I created a new C# app in Visual Studio and wrote a program to do everything that was needed — get some links, save the data about them, create a page with the links, do some “other stuff”.

The problem was, I was never really clear what the program should do. Should I collect the votes the article had from reddit? Don’t know, so I’ll throw that in there and save it. Might need it later. Should I provide the ability to comment, or vote? Don’t know. I’ll add space — after all, space is cheap — and if I need it, it’s there. Should there be some kind of authentication? Beats me. (Decided to skip that one)

Don’t get me wrong; none of this was a heavyweight or Big Design Up Front process. I’m only talking about an hour or two of doodling around with things. And the beauty of modern tools is that it really doesn’t matter whether your table has 5 fields or 10. Unless you’re writing the code to do something with the data, it’s just “out there”.

It did make the “link sucking” process a little more complicated, because with different sites I had to get different pieces of data. Fair enough.

What happened. After a couple of weeks of off-and-on coding, I got the program up and running, along with a unique system for identifying data on a web page to harvest. (It was a recursive RegEx language. Please, no “but you can’t use Regex on html!” It looked good.) The code base consisted of a data tier, some controller logic on top, and it seemed to work fine.

For a while.

But over a period of just a few days or weeks, it seemed like there were always these weird edge cases, both in the sites, and more disturbingly, in the tool version, setup, and configuration. Site X would change their format, and I’d be in the code changing the way I received links. Yes, I had hard-coded in the parameters for each site, but how difficult was it to change some hard-coded values once or twice a year? Except the more source code there was, and the more I touched the code, the more errors I got. And it seemed like for one reason or the other with this way of programming there was a boatload of code to wade through and I was always horsing around with it. [insert discussion here about TDD]

By using a MVC paradigm, I had written one piece of software that I could easily lay a bunch of different controllers down on. In fact, as we all know, I could develop a “controller library” over time that would take the data and do all sorts of wondrous and amazing things. That’s the beauty of a good model — it facilitates many different solutions.

I still think this might be the way to go, assuming 1) you use good testing practices, and 2) you’re going to constantly be under the hood in the code for a long time. But it introduced all kinds of problems related to having everything pass through one executable and code base. Did I really want to horse around with the code that grabbed links, for instance, simply because I was fixing the code that wrote the web page? Why was I putting all of my eggs in one basket? OOP is a great way of doing things, but it introduces a freaking huge amount of hidden interdependencies. Some times I wonder if 90% of the problems development teams have with programming is based on the fact that in the developer’s mind there is separation of model and controller, but that in the wiring itself, things can be all hooked together in completely insane ways, usually for what seemed like good reasons at the time. The black box of OOP has a lot of very sharp edges.

The Old Way of doing things. The first thing I did was ditch C#. It’s a great language, a better Java than Java, but it was the wrong paradigm for me. The second thing I did was start to build the app from the ground-up. Programming functionally required me to write functions that did things. This sounds blazingly obvious, but it’s actually a much different way of looking at things from creating a model and adding controllers on top. I still used the MySQL datastore — had to put stuff somewhere — but my code was a LOT smaller.

It also needed much less tweaking. Remember: this is personal programming. Instead of sitting on top of a model that could do many things and having to choose what to implement, which might cover calls all over the place, my code did a few things, and there was a clear path to the data for the things it did.

What happened. There was still that problem with everything running through one spot. If I wanted to tweak anything, anywhere, I’m opening up the entire codebase to make a change.

Interestingly, since this was revision 2, the code ran for much longer without needing tweaking. This meant that when I did get around to tweaking it, I had forgotten the build setup! So I’d have a 2-line change that would require screwing around with re-remembering the build-deploy environment. That might take several hours. Plus the sites were still hard-coded. The output was hard-coded. As much as I loved data-driven programming, why was everything so coupled?

Along this time I built HN Books, a social book site for hackers. It was all static. I found I could do a lot with static web pages and JSON files. Much more than most web programmers would believe.

Hmm. Fuctional programming. Static pages. Hmmm.

The way I do things now.

So now I have a new philosophy:

  • Third time through, I decided I write small programs. As small as I can. Instead of one big honking system, my system has four programs: GetLinks, UpdateArticleLibrary, CreateSelectedArticleList, and FormatSelectedArticleList. There’s a bonus utility, CheckXPath, which lets me check out XPath strings against websites to make sure they work (in case the site layout changes). I use as few external libraries, tools, or frameworks as possible. Over the years, I’ve found that every tool I picked up had 40 tons of features, of which I only needed 3. That meant that whenever I needed to do something simple that I had never done before, I had to wade through all sorts of forums, hearing about all sorts of arcane switches and other stuff. Screw that. I do not need to buy and operate a nuclear attack sub to go fishing from the pier. So — no Visual Studio, no MySQL, no complex dependencies at all. (Having said that, I actually dumped my custom link-sucking system and went with HtmlAgilityPack. I mean heck, once it’s installed, it’s just XPath. Time spent learning XPath is not as completely sunk as time spent learning WhizBang 7). Simplify.

  • My files read and write data on the local storage. I’m not doing transactions, therefore I don’t need a transactional datastore. It’s functional coding. Things come in, things go out. The O/S already has a wonderful tool for storing things. It’s called a file system, and unless I’m going to be moving around tens of thousands of these things, it’ll work just fine, thank you. Simplify.

  • I use the O/S to schedule when the programs run and to move things around. Instead of having one program that runs once a day, split the work up into a “pipeline” and have the O/S manage the pipeline. It’s already good at managing processes — it has all kinds of tools for it. Use them. Simplify.

  • I don’t “interact”. I process things at certain times. In the old days, I’d have an instance of my program spun up in Apache, waiting around in fastCGI for a client to come and connect. Then the program would do all sorts of things depending on which client it was, what the request was, and so on.

    After many, many years of coding like this, I had to ask myself: why? Why the hell am I doing it this way? 9 times out of ten I’m delivering the same content. 9 times out of ten the client is just reading stuff. 9 times out of 10 I’m creating database connections, cursors, and all sorts of other cruft — just to send the same stuff back down the wire as I sent 2 seconds ago. For the huge majority of the use cases I can imagine, even for interactive sites, there’s no difference at all to the user between a program that sits waiting for connections and a series of programs that updates data every few seconds. This is stupid. Don’t do this any more. Decouple. Simplify.

  • I write simple queries to monitor how things are moving through the pipeline. Quite frankly, the system is running so well I don’t need to monitor it, but if I did, I’d simply monitor how things flow through the pipeline. I could even make a nice html page with graphs. Don’t need to, but it’s an easy option. In fact, I’d argue that the only things interesting to me as a owner/maintainer would be things visible at a system level, not a programming level. Huge win here: no programming skill required to look at CLR innards, just O/S skills. Simplify.

  • Nothing exists as hard-coded data. It’s all either config files or command-line parameters. Right now as I bring up the page, I can see that Hacker News isn’t returning any data. If I wanted, I could probably figure out what the problem was and fix it (assuming it was fixable on my end) in about 10 minutes. No programming required. All I need is a shell. I AM re-coding the system, kinda. I decided that since I have 10-15 functions that are the same across each executable, it would make sense to create one Visual Studio solution and share a couple of library source files. Welcome back to the days of shared C header files! I’m also making the logging more robust. Right now I log everything. (I don’t read the logs, but they are there.) This is using up too much disc space. Every few months I have to clean it out. So a more fine-tuned logging system would be nice. Maybe. Maybe not. Simplify.

  • TDD? TDD? We don’t need no stinking TDD. With purely functional programming, there are only 3 mistakes I can make: Failure to break down transforms into understandable atomic units, failure to describe the transform correctly, and failure to validate the data. If I do all three of those correctly? There’s nothing to test. It’s like trying to test an SQL select statement. The idea doesn’t make sense. Simplify.

It’s very nice. Here’s the main function for the first program in the chain, GetLinks, that gets links from sites (duh):

Code Snippet
  1. [<EntryPoint>]
  2. let main argv =
  3.     try
  4.         let opts = parseCommandLineArgs (Array.toList argv)
  5.         if opts.verbose >= Verbosity.Normal then
  6.             printfn "Options Selected:"
  7.             printfn "%A" opts
  8.             printfn ""
  9.         let config = getConfigData opts
  10.         let outputDataHolder = {
  11.             outputSections = new System.Collections.Generic.Dictionary<string, outputSection>()}
  12.         let outputData = ripLinks opts config outputDataHolder
  13.         printfn "Processing Complete for %A" opts.siteUrl
  14.         let numberOfDataPointsMatchMessage =
  15.             if outputData.outputSections.Count>0
  16.             then
  17.                 let max = outputData.outputSections |> Seq.maxBy(fun x->x.Value.outputList.Count)
  18.                 let maxCount = max.Value.outputList.Count
  19.                 let min = outputData.outputSections |> Seq.minBy(fun x->x.Value.outputList.Count)
  20.                 let minCount = min.Value.outputList.Count
  21.                 if maxCount = minCount
  22.                 then
  23.                       maxCount.ToString() + " links with " + (outputData.outputSections.Count-1).ToString() + " pieces of additional information about each link gathered."
  24.                 else "Number of links, titles, and other Data DOES NOT MATCH.\r\n" + max.Key + "  has " + maxCount.ToString() + "entries while " + min.Key + " has " + minCount.ToString() + ".\r\nCheck your configuration file or run this program with the verbose option set /V"
  25.             else
  26.                 "\r\n\r\nTHERE WERE NO SECTIONS OUTPUT.\r\nPlease check your configuration file.\r\nTry running this program with the /V option"
  27.         System.Console.WriteLine(numberOfDataPointsMatchMessage)
  28.         0 // return an integer exit code
  29.     with
  30.         | :? UserNeedsHelp as hex ->
  31.             System.Console.WriteLine("You'd like help")
  32.             System.Console.WriteLine("Good luck with that")
  33.             System.Console.WriteLine("/V for verbose output")
  34.             System.Console.WriteLine("/S:<url> to set the target site url")
  35.             System.Console.WriteLine("/C:<filename> to set the config file used. Config file is expected to have Windows line returns \r\n")
  36.             System.Console.WriteLine("/O:<filename> to set the output file created/overwritten")
  37.             System.Console.WriteLine("/N:<integer> number of links desired")
  38.             System.Console.WriteLine("")
  39.             0

This is 50 lines of code. But it’s really only 4, lines 204-207. It gets the configs passed in. It creates a new container to hold the output data, it processes the link, and then it saves the data (which happens in the ripLink function. Should be down in this one. Ugh.) The rest of it is logging and help system stuff.

I could show you ripLinks, but it’s the same deal: 50 lines of code which are really about 10. It gets the links from the page using the “ripLinksOnAPage” function (clever naming, eh?), then it processes the links according to the config file.

Let’s look at how it breaks up.


As you can see, there are really only 5 functions, ripLinks, http, loadHtml, recBuildLinksUp, writeLinksOut, and ripLinksOnAPage. The rest is either library calls or a few small helper functions. Take out the logging and some of the other broilerplate, and there’s maybe 50 lines of “real code” here.

We’ve done something simple, easily describable, and concrete. We took stuff from the file system, read a webpage, wrote stuff to the file system. Logged as we went along. That’s it. No need to solve world hunger. It has value.

This code has been running for more than a year with no modifications. I expect it to continue running forever. I’m done. Isn’t that nice?

Common objections.

It’s re-inventing the wheel. In most cases, I don’t need a wheel. I need a lug nut. Yes, I know what the wheel looks like, but the cognitive load of buying a big stack of wheels and carrying them around when I just need a couple of lug nuts? Tell you what. If I start thinking about round things to fit on cars, I’ll get a wheel. Otherwise I’m fine.

It’ll never scale. The funny part about this objection is the opposite is true: the more complex your stack, the more difficult and nuanced it is to scale. I can take this app and scale it out as far as I like. Heck, copy it to a CDN. We’re done. And for those of you thinking interactivity, think long and hard whether you need immediate feedback for the user or just something that changes every few seconds or once a minute. You could be spending a huge number of processor cycles worrying about concurrent clients when you really don’t need it.

It’s poorly-thought-out. My datastore? Line-delimited text files. Code complexity? Nothing more than a few hundred lines of code. Is there anything it can’t do? Not really. But (and this would be the objection I would have made several years ago) what if you want it to do something that required the data to change? Wouldn’t you have to re-jigger every piece of code in the pipeline?

Not really. First, because I’m using name-value pairs, I can always add data and not use it. So really, what would happen if I wanted some cool new feature would be 1) I’d change the data representation where it was created, 2) I’d add the code to store it, and 3) I’d add the code to use it. If you’ve worked in a functional environment, this is the way you make changes anyway. Nothing new here.

It’s not as cool as X. Probably not.

The deployment process is brittle. Actually, although I’m not continuously deploying what I write — there’s no need to automate it when it’s just me — I’m integrating DevOps directly into the solution. There’s no one part of this solution that’s “programming” and another part that’s “deployment”. It’s all integrated together. Instead of good coupling and cohesion at the function level, I have good coupling and cohesion at both the function and the executable level. Very cool stuff.

Continuing to add features. The very next thing I’m going to do is add/adjust the logging. Having to clean out the logs once every 3 or 4 months is a chore. Next up in my quest to eliminate distractions, I’ll probably go to the target site and rip the plain text and store it here. I’ve thought about adding voting and commenting, but it’s a personal site.

None of this will require a major change or a re-think of how the architecture works. Mostly the system just works and I don’t have to mess with it. O/S updates handle updating security and scaling problems. I worked a bit to make the solution, and now the solution works for me. And isn’t that the entire idea?

If you've read this far and you're interested in Agile, you should take my No-frills Agile Tune-up Email Course, and follow me on Twitter.

My First Computer

In 1977, when I was 12, I was in the gifted program. I’m not sure exactly what you had to do in order to be “gifted”, but I assume it had something to do with having an higher-than-average IQ. It was a special class you took every so often that was supposed to help gifted students engage their minds more — stay challenged in the school system. Each year the teachers tried to come up with something to expand our little minds but it never seemed to work out the way it was planned. The teachers didn’t hit on much and we probably weren’t the best students either. But we all went through the motions anyway.

I remember one year we had a creative writing class with Mrs. Woollery. Mrs. Woollery had a bit of a mustache, so behind her back I gave her the nickname “Mrs. Wookie.” We were truly dreadful little writers, but the teachers were troopers and did as much as they could to encourage us. The highlight of the creative writing class for me was tearing apart all the girls’ fiction, which was almost as bad as my own (One short story I remember writing was about a moose that gained self-awareness and became a genius as a side of effect of an alien invasion. The aliens were the size of insects. Sort of a “Flowers for Algernon” meets “Independence Day.” Lets just say I’m not waiting on any awards for that one.) On a good day you could tear apart the saccharine and overly-described heartfelt efforts of 2 or 3 girls while genuinely trying to help them, and in the cruel world of Junior High where girls weren’t the most friendly of sorts that was something.

Mrs. Woolery never knew what to make of my strange short stories. But she kept encouraging me anyway. This was the way of the gifted program.

So I didn’t have my hopes up at the beginning of 7th grade when they asked us what we wanted to study, but I had to admit that this was something new. They had never asked us what we wanted to do. Instead of deciding on what to give us, they were going to let us decide!

I knew in an instant what I wanted.

A computer.

So I lobbied our teacher to get a computer that we could program.

I won, and this is what showed up a couple of weeks later.

Computer kit from the Radio Shack 1977 catalog

I still remember the little wires, light bulbs, and other assorted trinkets

Let’s just say that although I tried diligently to get as much as I could from the kit, the gifted program didn’t make me a computer expert that year, no matter what Radio Shack promised on the box. I also did not diagnose any illnesses or predict the weather. My fun with computers didn’t begin until a couple of years later in High School.

But Mrs. Woolery told me I did a good job anyway.

Congrats to the Radio Shack fan site for the catalog image! I could spend hours over there looking at all the things I envied growing up. Radio Shack was like Nirvana to me growing up; the Kingdom of the Gods. They had everything that was cool.

The Radio Shack guys at the local strip mall should have gotten a medal. I loved that place. Every day after delivering the papers I would be there programming on the TRS-80. Truth be told, my after school hours at Radio Shack a couple of years later was what really started me on the road to being a programmer.

Funny thing, I can’t remember them saying anything especially encouraging to me. They just kindly and graciously put up with my constant presence.

I sure miss Mrs. Wookie though.

If you've read this far and you're interested in Agile, you should take my No-frills Agile Tune-up Email Course, and follow me on Twitter.

Or to put the programming levels another way…

Here’s a refactor of the last post:

  • Advanced O/S and text editor usage
  • Dealing with symbols you don’t understand
  • Dealing with symbols you do understand
  • Dealing with complex groupings of symbols you understand
  • Creating your own symbols
  • Creating your own complex groupings of symbols
  • Dealing with meta symbols
  • Dealing with complex groupings of meta symbols
  • Creating your own meta symbols
  • Creating your own complex groupings of meta symbols
  • Creating your own language (allowing others to join your meta symbols together in a structured way)
  • Creating your own complex languages
  • Be a scientist

And no, it’s not like you are either in one level or another. It’s very typical to move among the levels depending on the nature of the project. The point is which level you are able to consistently work at given the need.

If you've read this far and you're interested in Agile, you should take my No-frills Agile Tune-up Email Course, and follow me on Twitter.

What Level Programmer Are You?

Everybody’s talking about how programming is the skill that we all are going to need. [Except those folks who might feel that most programming could be turned into wizard-like tools. Insert long discussion about Strong AI.]

But what’s a programmer? Is the guy who set up his own Apache Web Server a programmer? How about the guy who created a complex Excel spreadsheet? The guy who made his own RPG level? Minecraft players? When we say “Everybody is going to have to know programming” what, exactly, does that mean?

We need a set of programming levels.

  • Level 1, The Read-and-Type: This is the guy who can read how to set up a web server, go through the directions, and get the thing working — as long as nothing goes wrong. He knows how to use a text editor and basics of the file system he’s on. He also has to have some ability to move and manipulate files, like using zip or FTP. He can use abstract symbols in a structured, explained way along with having system skills.
  • Level 2, The Script Kiddie: Can write a linear script to tell the computer to do something. Able to learn and use symbols on his own. Should be able to script out a very simple solution to do something useful for himself in bash or VB if given enough time and written instruction.
  • Level 3, The Librarian: This is someone who has learned some libraries or APIs and have the ability to learn more. He knows that for whatever scripting language he is using, he needs to be able to connect libraries to it. If he had to, he is able to read manuals and do that on his own. He is also able to search for libraries on the net and use them. He is able to find and sometimes even create his own symbols for his job. (Can write a function or declare and use a struct). Most beginning system administrators fall around here. Also beginning users of advanced Macro systems, like Excel.
  • Level 4, Object-Dot-Method Guy: Beginning to learn coupling and cohesion, this person starts organizing their scripts into objects with public and private members and methods. This is your six-week-programming miracle, the introductory .NET programmer. He’s supposed to play with a wizard, know the libraries, and then construct something that fits inside of classes that are given to him. If he’s not using classes, he can create his own modules or code files. Some advanced users of scripting systems get this far. There are a lot of advanced Excel shops with stock brokers who have basically coded their own VBA application without really knowing it who are in this level.
  • Level 5, Multiple Paradigm Man: Once you are able to crawl, you get into all sorts of trouble. A common next step on the programming ladder is being able to work in multiple paradigms. You can code C#, but you also know how to build tables and write SQL. You can write acceptable C, but you also can work with Javascript and HTML. You’re not a master of either skill, but you’re able to understand different ways of looking at the same problem. You know you’ve reached this step when you stop saying “We could fix this in X” for every question where X is the language you first really understood.
  • Level 6, Architect Apprentice: This comes as you really begin to engage in the craft of programming. You begin to deeply understand the libraries and APIs that you use everyday for work. It’s not that you have memorized them. It’s that you understand the different patterns the architects use to accomplish their goals. Want to draw in Win32? Probably going to need a handle to some GDI resource. Maybe a few. You know that a handle is a UInt32 hash into a system table somewhere — a common paradigm for Win32. Want to add your own library to JQuery? There’s a format for any new library. You know how that works and can explain why it works that way. As you work your tools, you begin to start thinking like the people who put the tools together. You also begin to learn how to refactor your code to the point it rocks. You start learning what code quality looks like. You begin to understand.
  • Level 7, Architect Astronaut: The time comes for you to build your own API, platform, or library. Maybe it’s a large project that requires it. Maybe you just want to take your new knowledge out for a spin. While your system is designed to solve a real-world problem for users, your target audience is maintenance programmers at Level 5. At this phase you’re deeply into the Gang of Four, OOAD, templating, and best practices. Most of us make a mess the first few times into this area, not understanding that it’s easier to make something complex than it is to make it simple. During this phase, which can go on for a long time, you continually mistake creating more complex and flexible libraries with building higher quality systems. Many times you (and hopefully a few others on your team) understand what you’re doing, but to bring in somebody from the outside would take months of domain-specific training. Support personnel don’t stand a chance — most all support calls end up on the team’s desk.
  • Level 8, Meta Man: At some point, hopefully, you realize that much of the complexity you have put into architectures and frameworks can better be described as a Domain-Specific Language. Your target audience becomes programmers at Level 4. Yes, the coding complexity is greater, but the code itself is much shorter and the DSL opens up your code to go all kinds of places. In short, you become the writer of the script languages that beginning programmers use. Your systems are open to beginners through easy-to-use scripts and wizards. You can learn this skill separately, but most times after many large complex architectures, eventually you factor your way to a DSL. And then again. And again. Pretty soon you start seeing the pattern: well-understood problems lend themselves to DSLs.
  • Level 9, Functional Nirvana: Moving into the bonus round, you start really learning how to program by learning Functional Programming. Your target audience is maintenance programmers at Level 3. FP makes you a better imperative programmer, helps you easily kick out DSLs faster, and can massively reduce the amount of code it takes to solve a problem. It can also be highly-scalable. With some functional languages you can even hot-swap while the system is running. Just like with all these other advanced levels, at some point you start feeling getting a bit of nirvana going on. FP must be the answer to everything! By this time, however, you’ve been through several of these stages of euphoria, so this one isn’t as bad as the rest. The real world intrudes. FP, although less buggy and many times much faster, can be very difficult to code. Programmers have to carry a linker around in their head much of the time — they have to keep track of lots of symbols. Eventually you realize FP is also just a step on the path, much similar to OOAD.
  • Level 10, Language Oriented Designer: You view all programming as creating new programming languages. The only difference is what the new language is supposed to do. You understand how to create Object-Oriented languages, Functional languages, and simple untyped scripting languages. You know when each is appropriate. Your target audience is maintenance programmers at Level 2. You sit down for each project and ask “What kind of language do we need here?” At the end of your project you have a nice tool that both fixes the problem and is easily understood and extensible by entry-level programmers, freeing you up to move on to other things. You can easily make the case that most projects do not require language-oriented designers, but that’s a story for another day.
  • Mystery Level, Computer Scientist: This is either the pinnacle of programming or doesn’t exist on the spectrum at all, depending on your perspective. This is the science of programming. Lots of cool stuff here. Your target audience is everybody else in the universe. But the focus is not on making things people want, it’s advancing the state of knowledge. Because most other capability levels of programming are so people-based, these guys live in a different world, but a necessary one. How easy is it to cross back and forth between the two worlds? Beats me.

You could construct many alternate paths here — for instance, functional programming might come first for you, then OOP later on. You might also put in building your own compiler — wasn’t that fun in college? Or assembly. Or really understanding set theory and how it applies to both databases and programming data structures. Or writing your first true programming tool.

But this is the path I see most programmers follow.

ADD: Please read the (very short) follow-up for another look at this.

If you've read this far and you're interested in Agile, you should take my No-frills Agile Tune-up Email Course, and follow me on Twitter.

Programming is the new High School Diploma

It used to be there were four tiers of work in the United States. The first tier was for the truly uneducated: the illiterate. The second tier was for people who could be counted on to read and write and perform basic math: the high school graduates. Then there were folks who could be counted on to learn a lot more and take up positions of greater complexity: the college graduates. Finally there was a spot in the job market every so often for an expert.

Newsflash: the second and third tier are going away. In it’s place is a single tier: people who are literate and are able to control computers. And we’re nowhere near ready for the changes coming.

Programming is the new High School Diploma.

Sure, it might be another decade or so until the rest of the world catches on, but we programmers, especially those of us over 40, already see it today. Forty years ago computers were specialty devices found only in the most obscure places, now they are everywhere. Until strong AI is discovered, we know there are severe limitations of canned programming. When I was a kid, nobody had computers. Now my dentist’s office has at least a dozen. Hell the dentist himself has 3 or 4 he walks around with.

As as consumer of pop-media, there might be an app for everything, but the way an economy works is people putting together disparate things into something new. There’s never going to be an app for your job. Because if there was an app for your job, your job would be on your way to being replaced by robots or outsourced.

The new “middle tier” of the workforce looks something like this: literate, able to use math and write well, able to read a manual and instruct computers to take various actions depending on various conditions. Note that I’m talking about entry-level work. This job segment will take over “old” jobs from both the illiterate sector, and the college-educated sector. Eventually, as robotics comes to fruition, it will consume every job niche but the true experts.

I think a fair criticism of this essay is “there’s nothing new here,” and in a general sense, that’s true. After all, we all saw this coming. As technologists, the more technology we deploy, the less mundane jobs there are in the world. That’s a given. But the flip side of that — the more advanced jobs there are — hasn’t really sank in for most of us. After all, we’re already programmers. In a way, as programmers we’re stove-piped; we see just the system that we’re writing and (hopefully) the repercussions of that. We don’t sit on the other end of the table and see a dozen jobs working a dozen manual processes replaced by three jobs working two dozen computer programs. While we see the good that we do for one task, we have absolutely no insight into all those thousands of new multiple-application jobs (and the real-world implications for people) scattered all over the economy.

This really sank in for me when I read that although the United States leads the world in manufacturing, it does so with an ever-decreasing number of jobs. Nobody wants people to stand on an assembly line; they want people to tell the robots what to do. The robots work on the assembly line. When I thought of robots, for some reason I just thought about manufacturing cars, but that change happened in like, 1990. What has happened with computers in my lifetime is now happening with robots. In the next couple of decades, just like the last, seeing robots will go from a rare curiosity to an everyday occurrence. The only jobs left will be those that deal with computers. Dealing with computers; programming, scripting, reading manuals, and connecting interfaces and data; it’s the new reading and writing.

Progress is good, and I can’t wait until we get a world where the truly dehumanizing jobs are all gone, but while these jobs are coming online now, the education system is nowhere near understanding this new reality.

The political implications of this for older industrialized societies are vast. If there ever was a “red alert” we could put on a problem, this should be it, for what is about to happen is truly going to be unprecedented. We are going to end up with societies that have developed complex automated support and production systems that they themselves cannot manage. There’s simply too much work and too little competence. Instead, outsourced programmers will be used to supervise and control much of this modernization over the net.

This will bring on even more commoditization of business practices than we already have. While your dentist could use a nurse who is able to do data mining and various programming tasks to help him optimize his business practices (and gain an edge on his competition), he won’t be able to find one. Instead, he’ll pitch in with dozens of other dentists to buy a pre-canned support system overseas, with real, live, English-speaking operators. They’ll be able to do the same tasks, sure, but they won’t be part of his team, programming and working the business problem intimately in an effort to innovate. Instead they’ll just be a fancy vending machine. As services stratify, the opportunity to take many different technologies and merge them together, inherent in programming, will be lost to hundreds of thousands of businesses of all kinds.

The critical question that needs answering right now is: how do we make people of all education and age levels into somewhat competent programmers?

ADD: I hope I’ve described “programming” broadly enough for you to get the gist. I have a strong feeling we all could end up in a deep and long discussion about just what I mean by “programmer” That’s probably a post for another day. LISP experience not required. The key here is learning and using all sorts of already programmed systems in scripted ways that are all edge cases. Also, as much as I love computers, this is also not an advocacy essay. I am not advocating one thing or another nor asking for your approval. Things are changing. We must adapt.

If you've read this far and you're interested in Agile, you should take my No-frills Agile Tune-up Email Course, and follow me on Twitter.

Programmer as Market

Something has been bugging me about Microsoft and other tech vendors for some time. As I was reading and responding to a comment over on Hacker News it finally jelled in my mind.

The commenter wanted to know if TDD/BDD was useful in startups. Of course, the answer, like all answers to these questions, is “it depends”. But feeling a bit punchy, I just said “no”, then outlined why the answer was no.

I got a bit of pushback, much of it along the lines of “but TDD/BDD is _good_ development. Surely you’re not telling us to be bad developers!”

Yes, I am.

Fact is, startups are all about making things people want, not being a great developer. Your program is number 50 on the list of important things for you to do. You might write 10 lines of Javascript and make a million bucks. Or you might write the cleanest, most gorgeous code anybody has ever seen and make zero.

It has nothing to do with code. It has to do with finding and meeting customer needs at scale. In fact, the odds are 20-1 that your entire effort is worthless no matter what or how much you code, so don’t write code to be maintained. Don’t write code to be pretty. Write code to get it done as quickly as possible so you can move on to the real game: customers.

This doesn’t go over well with most folks. I know it didn’t go over well with me.

The problem is that we developers have become a market, and like all markets, people are trying to sell us stuff. These people do not have our customers in mind — for them we are the customer. Instead they have us in mind.

People try to sell us stuff based on whether or not we want to consume it, not whether or not it’s useful to the people as programmers we are trying to help. This creates a bit of perverse incentives.

Don’t like talking to people? No problem. You can read a hundred programming books and none of them will tell you that you need to do that — even though “talking to people” has to be the most important thing a programmer does. Like cool little widgets instead of fuzzy problems? No problem. There are myriad vendors out there who will sell you all sorts of cool little widgets. You can spend the next ten years becoming a widget expert.

You see, what I found with Microsoft — and what’s true in the industry as a whole — is that they could care less about helping me help other folks. What they want to do is meet whatever pre-existing prejudices I might have as a programmer. Wonder what cool features C# 5.0 will have? Wonder who in the real world cares aside from us programmers?

This feeling was a long time coming, and like every person that has been conned, it took me a while to wake up. What? You mean all those things I spent money for were useless?

Last week I went through my office cleaning up. You know how many tools and languages and other pieces of crap I’ve bought over the years? A zillion. Just how many calendar controls does one person need, anyway? Will there be some calendar control Apocalypse, where everyone will suddenly have to write their own day-planner?

I have spent hundreds of thousands of dollars on software, tools, books, and equipment that tell me what I want to hear — that programming is a complicated yet noble profession which involves mastering lots of technical details and being plugged into the latest technology. But programming is helping people, not any of that stuff.

The same goes for TDD/BDD — and a lot of other things. We get sold this idea that programming is high craftsmanship. That there are “good” programmers and “bad” programmers. The “good” programmers all do things this certain way.

Yes, some things work better than others. No doubt about it. But people who sell us stuff don’t necessarily give us all the times when their stuff shouldn’t be used. For them, it’s good for almost everything. They have no incentive to provide context to us, because many times the context in which what they are selling is useful is very small, but the context in which we might imagine their stuff is useful can be very large indeed.

Do you know how many hundreds of millions of dollars — perhaps billions — I have seen spent on enterprise tools which all boil down to “your programmers need help to do things the right way. Our tools will change their behavior so you don’t have to”

Never works, but still folks spend. Marketers are selling folks on a vision of what they would like the world to be, instead of what the world really is.

Yes, I would to live in a world where having some cool new set of programming tools would mean that I have an awesome startup, or that I’m able to help people simply by clicking a button somewhere, but that world doesn’t exist anywhere outside a seminar, conference or sales pitch. In the real world, real people grapple with fuzzy difficult questions, they don’t always get along, and they implement suboptimal solutions that are better than nothing.

I have never been in a situation helping somebody where having some little cool piece of tech actually helped. Sure, I’ve thought that having the tech helped. You betcha. I’ve been programmed by vendors to think “Gee, good thing I have X! This will make the problem much easier to solve!” but it’s never actually been the case. (The only exceptions I would make to that statement would be relational databases and web applications, which standardized information storage and application distribution)

On the contrary, I have seen hundreds of programmers and dozens of teams get caught wasting huge chunks of time with stuff somebody bought that is getting in the way of their work. I have seen teams who are basically paralyzed because of all the “help” in the form of tools, languages, tech, and methodologies they have purchased. I had one client that had a hundred people working on a problem. It had reached the point where they were thinking their problem was insolvable. After prototyping for a week, I showed them how to solve their problem very simply.

Their reply? This won’t work because it doesn’t match up with our tools! Not that the customer would hate it, or that it failed to solve the problem. They couldn’t implement it because their existing tools selection prevented them from solving the very problem the tools were supposed to help them solve!

I have good friends who constantly tweet about getting hung up by tool X or technology Y — instead of solving somebody’s problem. They keep feeling like that by mastering this latest tech somehow their productivity will shoot ahead. But it never does. There’s just a new set of magical tools and tech that come out next year. This is a treadmill that will never stop. Vendors will keep making things and promising that problems will go away, increasing the overall complexity and depth — as long as programmers have money to spend.

A lot of tools are very useful, no doubt. But we need to be extremely careful that we are not being taken into some kind of imaginary marketing world that doesn’t exist for anybody else just for purposes of feeling like we’re cool. We do so at our own peril.

Continue your journey with the book I wrote on being a ScrumMaster.

If you've read this far and you're interested in Agile, you should take my No-frills Agile Tune-up Email Course, and follow me on Twitter.

The Web App that’s not a Web App

I have to admit getting a few chuckles in over the past month.

A month or two ago, I introduced my hobby site, hn-books. Basically I took the hacker site I usually visit, found out what books hackers recommend to each other, put them in one spot. Now I’m going through them and reading all the good ones and reviewing them.

Fun stuff, but the really enjoyable part, at least professionally, is the response I got to the way the site was designed.

Several people complimented me on the speed of the app — great query time! Very responsive! Nice speed on the database times!

I appreciate that. App speed was so important to me that I decided not to make it an app.

Continue reading

If you've read this far and you're interested in Agile, you should take my No-frills Agile Tune-up Email Course, and follow me on Twitter.

To code quickly, you must quit coding

I did something yesterday that doubled my daily coding performance. It was easy, cheap, and made a tremendous difference in my life.

Some of you may already know what it is. For the rest of you, its going to sound wacky.

I stopped working.

Continue reading

If you've read this far and you're interested in Agile, you should take my No-frills Agile Tune-up Email Course, and follow me on Twitter.

Programming is for Stupid People

I had to give a presentation on F# last week at Code Camp and boy, was I afraid.

You see, I have been learning F# — and functional programming — for the last year or so, and it’s different. I mean way different from “normal” C# programming. Don’t get me wrong, I love it, but I had a hard time getting my head around it. Unlike C# and OOP, it seemed that no matter how much I learned, there was always some next level that I hadn’t reached.

I find that immensely rewarding. But I also find it pretty scary when I’m giving a presentation called “F# Ninja” Folks are going to be expecting a master of space and time, uber-lord of F#. Instead, they get me.

The day came, and I went out there and started my presentation. On the very first example, a guy raised his hand in the audience.

You know, you can do what you’re doing simply by using library X, he said.

I managed to get by the question — after all, the point wasn’t whether there was a library that could do something, the point was showing how the language worked — but the guy seemed to enjoy pointing out my “mistake”

It was an awkward moment.

Afterwards, several participants approached and asked questions. A couple even said it was the best presentation they had seen at Code Camp that day (I was last). But my friend was there too, and he had a whole bunch of questions about F#.

I was happy to answer. He had heard a few things about the language that were not true. He also pointed out how there were many other methods in many other libraries that do the same thing that you can do in F#. After all, all of Microsoft’s programming languages run on the same CLR, so it makes sense that if you can do it in one .NET language you can do it in another.

I got one of those feelings — it seemed to me that this guy had some attachment to justifying his not learning F#. Perhaps I misread him — but then again perhaps not. Either way, that’s fine. I’ve worked with really smart guys, and he seemed top of the pack. Amazing recall of various libraries and such.

Finally, out of exasperation, I said something like “Yes, I know you can do it like that, but you know what? I am an old and simple-minded programmer. If I don’t have to remember 4000 libraries — if, instead, I only need to remember the few dozen functions I’ve written — it’s easier on my mind”

That seemed to pacify him, and it also hit me as accidentally touching on a really deep truth:

Programming is for stupid people.

Continue reading

If you've read this far and you're interested in Agile, you should take my No-frills Agile Tune-up Email Course, and follow me on Twitter.

Secret Hacker Bookshelf

I received an email two weeks ago from a guy in the Philippines. He wanted to learn how to program and didn’t know where to start. Last week I was talking to a family member — he wanted to get into computers but didn’t know where to start. This week, 2 people came up to me and asked me questions about which books to read to learn startups and marketing.

It’s a common pattern. On the forum I visit, HackerNews, every few weeks somebody asks the same question — what are the best programming books? What are the best startup books? What are the best books on marketing? There are a lot of people asking, and the same questions are asked quite frequently. A quick search on Google lists dozens of questions about programming.

So. What do hackers recommend to each other?

Frankly, it gets old having to post comments recommending the same books over and over again. I know others feel the same way. But still, I’d like to help. So I decided to take all day today and find the best books from hacker discussions and list them here. Next time somebody asks me, I can just point them to this page. Who knows, if enough folks like the list, maybe I can keep it updated and expand on it.

Caveat Emptor: reading a good book on something fuzzy, like marketing or starting a business, is like having a beer with somebody at a bar. There’s lots of great ideas and great experiences to be learned. It’s also important to note that it’s you, not the authors, who is responsible for your life. Don’t fixate on any one book or author and go off hell bent for leather on what the author said. Instead, sample broadly, compare notes, learn both sides of the argument, then figure out how to use this new information to do things you want to do.

Having said that, this is a pretty incredible list and a pretty cool bunch of recommenders. If you have time, you should follow the conversations around some of these books. Many of the people commenting and many of the people writing these books have made millions or billions of dollars and would like to help you succeed too. And they’re not the traditional get-rich-quick, business porn, or self-help books that clutter up the marketplace. Lots of value here.

These books are listed by how hackers rate them, the vote count — books appearing higher on the list were voted by hackers as better than those lower. The programming section has several sub-sections that I haven’t broken out yet, but you can easily spot where one section ends and another begins.

As for some meta advice, if I were interested in buying one of these books, I’d probably read the pro and con reviews on Amazon, taking careful note of the con reviews (many times the pro reviews are fake). I’ve had pretty good luck using this technique, especially when I get there from a recommendation from a friend. And now you just gained a thousand hacker friends :)

Continue reading

If you've read this far and you're interested in Agile, you should take my No-frills Agile Tune-up Email Course, and follow me on Twitter.