Thoughts On ...

Business and IT converging?

A Computer World article talks about business and IT alignment/integration:

As more CIOs move toward business and IT alignment over the next 
several years, the makeup and structure of IT will change. IT and business unit
employees will work more closely together -- and in some cases, 
interchangeably.

(Via Slashdot)

I speculated on the roots of this split, and the eventual convergence of business and IT last year.

I believe that it is through such closer interaction and blurring (but not elimination) of distinction between business and IT -- and not the application of Agile to large classic IT projects where they are doomed to become their opposite -- is where the principles and practices of XP and related methodologies will have their lasting impact on software development.

Applying Technology is not Technology

Via Bruce Schneier:

I've long said that the fundamental problems in computer security are no 
longer about technology; they're about applying technology. Workshops 
like WEIS are helping us understand why good security technologies fail 
and bad ones succeed, and that kind of insight is critical if we're going 
to improve security in the information age.

Bruce referring to a conference (WEIS) that looks at the relationship between economics and computer security.

I think this is a great point, but Bruce doesn't go far enough. I think this can be generalized to: The fundamental problems in professional computing are no longer about technology but applying technology.

I'm sure that people who are still pushing the boundaries in things like AI, nanotech, etc. would beg to differ, but for the vast majority of people who are involved with technology (i.e. those supplying and using computer-based systems in the business world), the problem has shifted.

Not that long ago (i.e. within my time as a professional programmer), hard drive size, memory size, bandwidth and processor speed were forefront concerns for any business application design, often requiring workarounds or selection (or anticipation) of new technlogies. Now the discussion has shifted to how we effectively apply the tools and technologies at our disposal (I haven't heard anyone say something along the lines of "well we can do that if we just get more memory" in a long time). Even my phone has more processing power now than some of the desktop machines used by companies 10 years ago.

This is not to say our work is done. But rather that its just getting started. Now the challenge is to understand how to use the things we have effectively and efficiently (arguably a bigger challenge).

It also might explain the growing sense that CS degrees are insufficient to prepare people for a career as a professional programmer. As the fundamentals become more commodotized, the application becomes more central to success.

Knowledge Nodes

One could (and I guess I am about to) argue that the various communication tools that are central to Extreme Programming -- namely the practices: small team; pair programming; co-location; daily-standups; continuous integration; on-site customer; and iteration planning game -- are aimed at producing a situation where everyone knows everything about the design and features of the system being worked on.

As the number of people on a software team grows, it becomes harder to accomplish this goal, and so there becomes a need for people who are what I term knowledge nodes: these people seek to understand the entirety of the design and features being worked on, so they can communicate relevant bits of that information to others who then have partial (but presumably sufficient) knowledge to do their jobs. Further, it appears common (and a function of both culture and team-size) to break the two areas of knowledge (i.e. design and feature-set) from each other and create specialized nodes for each of them.

The name of role such a person plays on the team varies, but Project Manager, Analyst, Lead Developer, Lead Programmer and Architect are all commonly used. Each of these incline toward the design or the feature (i.e. technical and business) aspects of the information on a project-by-project basis.

What do the people who are good at these roles have in common? In my experience, they are good communicators: they are skilled at gathering, explain ing and sharing knowledge about the design and/or the features of a software system.

The interesting thing (or at the very least: the reason I started writing this today) is that even very small teams need at least one such person. Even on a team with a half-dozen people or so (such as the one I work with now), if the knowledge of the system fails to reach a knowledge node -- it is less likely to get shared with everyone.

Right or Wrong I have always seen myself in this role. I've also (right or wrong) had a tendency to feel like if I didn't know something about the design or features of the system I was working on it wasn't known.

(and so, yes this entire analysis may be nothing more than ego gratification of my self-perceived role on software projects)

So now that I've found yet another way to describe my role on a software project, so what? Well, for those of you who also find yourself in this (or working with someone playing this) role, I have some advice:

That while you may think that things are not known (objective) unless you know them (subjective) remember that's your ego talking (but it still may be true if your team has come to rely on it).

If you are not someone who likes to talk endlessly (I've heard the word 'pontificate' a few times in my career) about design and feature ideas, get your thoughts to someone who does -- they'll make the rounds for you.

On small teams, one or two such people are more than enough (too many, and there will be no work, just talk) -- but you do need at least one to forge a team (otherwise everyone will just work on their bit without knowing about the rest).

Don't confuse knowledge nodes with knowledge creators -- good teams have both, and often the same person isn't good at both.

Particularly (as in my case) people are rarely good at both at the same time. I find that I can share design and feature knowledge, or I can create it, but when I am doing a lot of one, I am doing very little of the other. (team-size, iterations and pairing help with this -- as does some self-discipline on my part to keep it balanced).

Finally, the (presumed) XP goal of everyone knowing everything is in pratice accomplishable by making sure the knowledge node(s) of the team know everything and then get it shared around. If you are playing this role, you have to gather *and* share. If you are disinclined to do so, you have a responsibility -- at least -- to get what you know to such a person, so they can share what you know about the system with everyone else. This use of knowledge nodes may be (I don't know I just musing here) the key to an effective software team.

How to Start an Interplanetary War

H.G. Wells apparently got it backwards.

The thought of firing a space-based railgun at an unsuspecting planet in order to observe the resulting "debris plume, looking for any evidence of life" sounds somewhat Onion-esque to me.

XP Exaggerated Redux

Here's a wayback machine link to what I believe was the best proceedings summary from XPUniverse 2001. (via Alan Fancis)

XPEX was a parody summary that (as Alan points out) came to us after many rounds of Guinness in the Hotel lobby (somewhere right after Alan declared that XP needed a Flag if it were to conquer the world, if I recall correctly).

There's a lot of in-jokes of course (some *very* dated to the conference), but a lot of more general pop culture and Extreme Programming community references. For fun, see how many you can find.

Big thanks to Alan for resurrecting this. I needed a brief light moment before I dive back into the crazy rewrite/refactoring that has been my sole focus for the past two weeks.

Job Postings

Those of you who read this site regularly may remember that I left ThoughtWorks last year to join a software team full-time.

Well, my team is hiring. Specifically, we are looking for a Reporting Analyst, and a Junior Developer.

If you think you might be interested (or know someone who else who may be), please feel free to follow those links to the job descriptions and follow the resume submission criteria there (the above point to Craigslist, they are also on Monster, here and here).

Please don't send anything directly to me -- while I will be closely involved in the hiring process, I am not managing the process itself, so sending something to me will probably just ensure that you don't get considered nearly as quickly or reliably.

I have refrained thus far from talking a lot about Redbox, and what I've been doing here. That may change in the near future, but for now, I will only say that we are building a great software system, we have a great team, and yes we are practicing Extreme Programming.

I look forward to hearing from you.

Competition

What if you're team is better than 99% of team's developing software? What if you are better at software development than 99% of the people who've ever tried?

What if you've climbed to the base camp of Mount Everest? Then you would be farther along the path than 99% of who've ever lived.

Now all you have to do is climb Mt. Everest.

Automatic Discipline

Summary: Given a choice between automation opt-out and automation opt-in, choose the former.

The spirit of build process automation is finding ways to remove reliance on individual discipline, memory and technique; replacing it with tools that help us to remember to do the right thing -- usually by ensuring that when we forget a critical step in the process, the effect is immediate and local (rather than deferred and team-wide). That is: when I mess up, I know right away, and only I am affected (usually by my changes not being integrated).

A lot of people don't like this. Some believe in the better nature of the team, and feel that its too restrictive to have certain process activities happen automatically -- particularly when those activities can be more detailed, or sophisticated if done manually. Others fear that this will slow them down too much (my experience is that its the opposite). I also have encountered others who simply don't like that much discipline constraining their actions. I question the professionalism of this point of view, since unless they make no mistakes, it means that they simply defer or shift their impact - usually to their teammates or customers, but to each his own (as long as I don't have to work with them).

But mainly its the first group that is the most challenging -- the better nature people. They argue that people need to be disciplined -- its part of being a professional. The thing is: I'd rather the discipline come early (in setting up the process) than be hoped for in the heat of the critical 11th hour bug-fix. By shifting the notion of discipline from "make sure you do x, y and z on each check-in" to: "maek sure that y and z happen automatically, that way if you forget x, the build will break -- oh and its more likely to happen if you put it off (so make sure you do it often)" we get discipline with teeth.

A recent example from my current situation illustrates the difference. Like many teams practicing continuous integration, our team rebuilds the database as part of the build. If a programmer makes changes to the database schema, those changes are not reflected in the code base unless they update and check-in a binary "back-up" of the database. This file is then used (if there are changes) to rebuild the database on the build machine, and other development machines. There is no way to overlook this step -- one can (with effort) circumvent it, but one can't simply forget since one's changes won't make it into the build, and it will probably break (having story tests helps here too, but that's a different entry).

We used to use a text representation, but by switching to the binary format, we greatly reduced our build time -- at the expense of indivdiual change tracking (before the changes to each database object were versioned in the source code repository). Recently, we decided to add back (but not use to rebuild) the generation of the text representation. Unfortunately, our first pass requires opt-in on the part of the programmer (one needs to remember to run the old script and annotate the changes on check-in, as well as run the binary update). Its opt-in because unlike before, those scripts aren't used for anything (other than documentation), so failure to update them will not fail the build.

I discussed this with the programmer who implemented the solution (his name is Brian), and we debated the merits of the alternatives. My preference is to make the script run as part of the official build (managed by CruiseControl), purely as a housekeeping step. His counter was that having it done by the programmer on check-in allows detailed descriptions of what changed. We haven't -- as of this writing -- changed it yet, however Brian informed me earlier this week that he woke up in a cold sweat the other night thinking that he had in fact forgotten to run this script, and is now working out how to include it in our build. (Thus, I have officially caused someone nightmares by talking about continuous integration!)

Its the right choice. The extra documentation potential is simply far outweighed by the risk of omission. By moving the execution into the build, we get an automated change-log of each database object (each one is stored separately allowing a diff), and the guarantee that it will happen if anyone wants their changes actually submitted into the code base.

Brian's intuition is understandable -- its a bit simpler to implement, and its tempting when one has a high regard for the quality of one's teammates (as we do) to rely on their professionalism, but experience shows that no one is perfect, someone will forget to run the script, and they'll forget when we need it the most: in the heat of a critical fix with time pressure.

The only answer that allows me to sleep at night: use the build process to keep ourselves honest, that's what its there for.

Free Features

Summary: Avoid the temptation to deem features "in scope" because of the illusion that they are free.

Its very common for customers, when asked if they want a story to include option x, to reply: "well, I'll take it if it doesn't cost anything" -- i.e. if it doesn't change the estimate.

While its tempting (especially if its small) to give in to this, its a mistake. Here's why:

First, if the option in question is needed, then its needed whether cheap or expsensive (we'll get to ROI in a minute), so this is a dead giveaway that the value of the feature has not been established (IOW, its likely something "cool" but not something valuable). Second, nothing is ever free, its either already there, or someone will spend time adding it -- time that could be used for something that is considered valuable.

Yes, there are features where if cheap enough, they could be included. Such marginal ROI features however are not generally when this situation arises (those usually float near the bottom-middle of the story stack). In such cases though, they should still have some point value attached (even if you have to throw a couple together to make a one point story).

So, why is this so common? I think the answer is due in part to the fact that many development efforts have slack -- hidden slack -- built in because of the all-to-common practice of negotiated estimation. Negotiated Estimation is where the software creator guesses at the time it will take (usually padding the result in anticipation of pushback), and the software customer (or a representative project manager) pushes back on the estimates to coerce the result into alignment with expectations. At the extreme this becomes not a negotiation, but a one-way edict: The software will be completed within this time. The result is a strange compromise (strange in that both compromises are known to the creator, not the customer): The schedule is sandbagged, and quality is sacrificied.

Many customers are experienced with this mode of development, and so push the creators to add features (assuming their estimates are conservative). The most pressure is brought by former programmers who padded their estimates and assume that others do too. This also produces another wasteful side-effect: Vague areas are filled in by the programmer; ambiguous business rules are guessed at, speculative features that seem to add completeness are included, and non-functional eye candy are lovingly tweaked -- all without the customer's involvement. This is an awful lot of inefficiency, for free.

However, if you subscribe to the principle (you may not) that the person who is going to use and pay for the software should be the one -- the only one -- to decide (suggestion and advice are something else) what functionality is included, then you can see how such reliance on the programmer to make the decision whether speculative "free" features are added is counter to this goal. Instead, I subscribe to the notion that its better to have slack visible (it should exist regardless), get all effort quantified (not negotiated!), and assume all features are out, unless explicitly included (with the inevitable ambiguities resolved via regular customer discusion). It makes for better expectations all around, it builds trust, it puts decidable changes (i.e. changing scope instead of trying to change estimates) in the hands of the decision makers, and -- at its best -- results in better quality software, with better work environments.

But that's just me.

Asking about Trust

[Note: Yet another old draft]

"Can you be trusted?"

It's a question we ask people all the time -- not necessarily out loud, but we do so nonetheless. We ask it by evaluating their actions, and their words. We look, rather than listen for the answer. More than their personalities, more than their motives, we look at their results.

This entry was drafted not long after I found myself wanting to literally ask someone this, but didn't. I reflected afterward: "Why not?" Why, almost as soon as the notion formed, was my inclination to reject the idea? It would be rude for one -- even politically inexpedient as I was a consultant and that person a client manager. It seemed pretty illogical for another -- seeming about as fruitful as asking someone if they are a liar.

Yet the more I thought about it, the more I liked the idea. I figured the rude part could be mitigated by careful phrasing, but that left the logical futility. Since the answer is going to be some form of, "Yes, I am trustworthy" why bother?

My conclusion was that the value (if any) would come, not from what the person answers, but how -- and what they did afterward. If his actions had shown the question had rattled him, or led to a change in behavior (I am assuming that it was prior behavior that led to my desire to ask in the first place), it could be a sign that the person is manipulating how they are viewed by others.

In short, I concluded that usually asking the question would be useless, but the surprise factor of throwing it right out there, might just rattle someone's cage enough for you to get an accurate glimpse inside.

I kinda wish now, I had asked. Maybe someday, I will.