A Computer World article talks about business and IT alignment/integration:

As more CIOs move toward business and IT alignment over the next 
several years, the makeup and structure of IT will change. IT and business unit
employees will work more closely together -- and in some cases, 
interchangeably.

(Via Slashdot)

I speculated on the roots of this split, and the eventual convergence of business and IT last year.

I believe that it is through such closer interaction and blurring (but not elimination) of distinction between business and IT -- and not the application of Agile to large classic IT projects where they are doomed to become their opposite -- is where the principles and practices of XP and related methodologies will have their lasting impact on software development.

Via Bruce Schneier:

I've long said that the fundamental problems in computer security are no 
longer about technology; they're about applying technology. Workshops 
like WEIS are helping us understand why good security technologies fail 
and bad ones succeed, and that kind of insight is critical if we're going 
to improve security in the information age.

Bruce referring to a conference (WEIS) that looks at the relationship between economics and computer security.

I think this is a great point, but Bruce doesn't go far enough. I think this can be generalized to: The fundamental problems in professional computing are no longer about technology but applying technology.

I'm sure that people who are still pushing the boundaries in things like AI, nanotech, etc. would beg to differ, but for the vast majority of people who are involved with technology (i.e. those supplying and using computer-based systems in the business world), the problem has shifted.

Not that long ago (i.e. within my time as a professional programmer), hard drive size, memory size, bandwidth and processor speed were forefront concerns for any business application design, often requiring workarounds or selection (or anticipation) of new technlogies. Now the discussion has shifted to how we effectively apply the tools and technologies at our disposal (I haven't heard anyone say something along the lines of "well we can do that if we just get more memory" in a long time). Even my phone has more processing power now than some of the desktop machines used by companies 10 years ago.

This is not to say our work is done. But rather that its just getting started. Now the challenge is to understand how to use the things we have effectively and efficiently (arguably a bigger challenge).

It also might explain the growing sense that CS degrees are insufficient to prepare people for a career as a professional programmer. As the fundamentals become more commodotized, the application becomes more central to success.

As the number of people on a software team grows, it becomes harder to accomplish this goal, and so there becomes a need for people who are what I term knowledge nodes: these people seek to understand the entirety of the design and features being worked on, so they can communicate relevant bits of that information to others who then have partial (but presumably sufficient) knowledge to do their jobs. Further, it appears common (and a function of both culture and team-size) to break the two areas of knowledge (i.e. design and feature-set) from each other and create specialized nodes for each of them.

The name of role such a person plays on the team varies, but Project Manager, Analyst, Lead Developer, Lead Programmer and Architect are all commonly used. Each of these incline toward the design or the feature (i.e. technical and business) aspects of the information on a project-by-project basis.

What do the people who are good at these roles have in common? In my experience, they are good communicators: they are skilled at gathering, explain ing and sharing knowledge about the design and/or the features of a software system.

The interesting thing (or at the very least: the reason I started writing this today) is that even very small teams need at least one such person. Even on a team with a half-dozen people or so (such as the one I work with now), if the knowledge of the system fails to reach a knowledge node -- it is less likely to get shared with everyone.

Right or Wrong I have always seen myself in this role. I've also (right or wrong) had a tendency to feel like if I didn't know something about the design or features of the system I was working on it wasn't known.

(and so, yes this entire analysis may be nothing more than ego gratification of my self-perceived role on software projects)

So now that I've found yet another way to describe my role on a software project, so what? Well, for those of you who also find yourself in this (or working with someone playing this) role, I have some advice:

That while you may think that things are not known (objective) unless you know them (subjective) remember that's your ego talking (but it still may be true if your team has come to rely on it).

If you are not someone who likes to talk endlessly (I've heard the word 'pontificate' a few times in my career) about design and feature ideas, get your thoughts to someone who does -- they'll make the rounds for you.

On small teams, one or two such people are more than enough (too many, and there will be no work, just talk) -- but you do need at least one to forge a team (otherwise everyone will just work on their bit without knowing about the rest).

Don't confuse knowledge nodes with knowledge creators -- good teams have both, and often the same person isn't good at both.

Particularly (as in my case) people are rarely good at both at the same time. I find that I can share design and feature knowledge, or I can create it, but when I am doing a lot of one, I am doing very little of the other. (team-size, iterations and pairing help with this -- as does some self-discipline on my part to keep it balanced).

Finally, the (presumed) XP goal of everyone knowing everything is in pratice accomplishable by making sure the knowledge node(s) of the team know everything and then get it shared around. If you are playing this role, you have to gather *and* share. If you are disinclined to do so, you have a responsibility -- at least -- to get what you know to such a person, so they can share what you know about the system with everyone else. This use of knowledge nodes may be (I don't know I just musing here) the key to an effective software team.

The thought of firing a space-based railgun at an unsuspecting planet in order to observe the resulting "debris plume, looking for any evidence of life" sounds somewhat Onion-esque to me.

XPEX was a parody summary that (as Alan points out) came to us after many rounds of Guinness in the Hotel lobby (somewhere right after Alan declared that XP needed a Flag if it were to conquer the world, if I recall correctly).

There's a lot of in-jokes of course (some *very* dated to the conference), but a lot of more general pop culture and Extreme Programming community references. For fun, see how many you can find.

Big thanks to Alan for resurrecting this. I needed a brief light moment before I dive back into the crazy rewrite/refactoring that has been my sole focus for the past two weeks.

Well, my team is hiring. Specifically, we are looking for a Reporting Analyst, and a Junior Developer.

If you think you might be interested (or know someone who else who may be), please feel free to follow those links to the job descriptions and follow the resume submission criteria there (the above point to Craigslist, they are also on Monster, here and here).

Please don't send anything directly to me -- while I will be closely involved in the hiring process, I am not managing the process itself, so sending something to me will probably just ensure that you don't get considered nearly as quickly or reliably.

I have refrained thus far from talking a lot about Redbox, and what I've been doing here. That may change in the near future, but for now, I will only say that we are building a great software system, we have a great team, and yes we are practicing Extreme Programming.

I look forward to hearing from you.

What if you're team is better than 99% of team's developing software? What if you are better at software development than 99% of the people who've ever tried?

What if you've climbed to the base camp of Mount Everest? Then you would be farther along the path than 99% of who've ever lived.

Now all you have to do is climb Mt. Everest.

Summary: Given a choice between automation opt-out and automation opt-in, choose the former.

The spirit of build process automation is finding ways to remove reliance on individual discipline, memory and technique; replacing it with tools that help us to remember to do the right thing -- usually by ensuring that when we forget a critical step in the process, the effect is immediate and local (rather than deferred and team-wide). That is: when I mess up, I know right away, and only I am affected (usually by my changes not being integrated).

A lot of people don't like this. Some believe in the better nature of the team, and feel that its too restrictive to have certain process activities happen automatically -- particularly when those activities can be more detailed, or sophisticated if done manually. Others fear that this will slow them down too much (my experience is that its the opposite). I also have encountered others who simply don't like that much discipline constraining their actions. I question the professionalism of this point of view, since unless they make no mistakes, it means that they simply defer or shift their impact - usually to their teammates or customers, but to each his own (as long as I don't have to work with them).

But mainly its the first group that is the most challenging -- the better nature people. They argue that people need to be disciplined -- its part of being a professional. The thing is: I'd rather the discipline come early (in setting up the process) than be hoped for in the heat of the critical 11th hour bug-fix. By shifting the notion of discipline from "make sure you do x, y and z on each check-in" to: "maek sure that y and z happen automatically, that way if you forget x, the build will break -- oh and its more likely to happen if you put it off (so make sure you do it often)" we get discipline with teeth.

A recent example from my current situation illustrates the difference. Like many teams practicing continuous integration, our team rebuilds the database as part of the build. If a programmer makes changes to the database schema, those changes are not reflected in the code base unless they update and check-in a binary "back-up" of the database. This file is then used (if there are changes) to rebuild the database on the build machine, and other development machines. There is no way to overlook this step -- one can (with effort) circumvent it, but one can't simply forget since one's changes won't make it into the build, and it will probably break (having story tests helps here too, but that's a different entry).

We used to use a text representation, but by switching to the binary format, we greatly reduced our build time -- at the expense of indivdiual change tracking (before the changes to each database object were versioned in the source code repository). Recently, we decided to add back (but not use to rebuild) the generation of the text representation. Unfortunately, our first pass requires opt-in on the part of the programmer (one needs to remember to run the old script and annotate the changes on check-in, as well as run the binary update). Its opt-in because unlike before, those scripts aren't used for anything (other than documentation), so failure to update them will not fail the build.

I discussed this with the programmer who implemented the solution (his name is Brian), and we debated the merits of the alternatives. My preference is to make the script run as part of the official build (managed by CruiseControl), purely as a housekeeping step. His counter was that having it done by the programmer on check-in allows detailed descriptions of what changed. We haven't -- as of this writing -- changed it yet, however Brian informed me earlier this week that he woke up in a cold sweat the other night thinking that he had in fact forgotten to run this script, and is now working out how to include it in our build. (Thus, I have officially caused someone nightmares by talking about continuous integration!)

Its the right choice. The extra documentation potential is simply far outweighed by the risk of omission. By moving the execution into the build, we get an automated change-log of each database object (each one is stored separately allowing a diff), and the guarantee that it will happen if anyone wants their changes actually submitted into the code base.

Brian's intuition is understandable -- its a bit simpler to implement, and its tempting when one has a high regard for the quality of one's teammates (as we do) to rely on their professionalism, but experience shows that no one is perfect, someone will forget to run the script, and they'll forget when we need it the most: in the heat of a critical fix with time pressure.

The only answer that allows me to sleep at night: use the build process to keep ourselves honest, that's what its there for.

Summary: Avoid the temptation to deem features "in scope" because of the illusion that they are free.

Its very common for customers, when asked if they want a story to include option x, to reply: "well, I'll take it if it doesn't cost anything" -- i.e. if it doesn't change the estimate.

While its tempting (especially if its small) to give in to this, its a mistake. Here's why:

First, if the option in question is needed, then its needed whether cheap or expsensive (we'll get to ROI in a minute), so this is a dead giveaway that the value of the feature has not been established (IOW, its likely something "cool" but not something valuable). Second, nothing is ever free, its either already there, or someone will spend time adding it -- time that could be used for something that is considered valuable.

Yes, there are features where if cheap enough, they could be included. Such marginal ROI features however are not generally when this situation arises (those usually float near the bottom-middle of the story stack). In such cases though, they should still have some point value attached (even if you have to throw a couple together to make a one point story).

So, why is this so common? I think the answer is due in part to the fact that many development efforts have slack -- hidden slack -- built in because of the all-to-common practice of negotiated estimation. Negotiated Estimation is where the software creator guesses at the time it will take (usually padding the result in anticipation of pushback), and the software customer (or a representative project manager) pushes back on the estimates to coerce the result into alignment with expectations. At the extreme this becomes not a negotiation, but a one-way edict: The software will be completed within this time. The result is a strange compromise (strange in that both compromises are known to the creator, not the customer): The schedule is sandbagged, and quality is sacrificied.

Many customers are experienced with this mode of development, and so push the creators to add features (assuming their estimates are conservative). The most pressure is brought by former programmers who padded their estimates and assume that others do too. This also produces another wasteful side-effect: Vague areas are filled in by the programmer; ambiguous business rules are guessed at, speculative features that seem to add completeness are included, and non-functional eye candy are lovingly tweaked -- all without the customer's involvement. This is an awful lot of inefficiency, for free.

However, if you subscribe to the principle (you may not) that the person who is going to use and pay for the software should be the one -- the only one -- to decide (suggestion and advice are something else) what functionality is included, then you can see how such reliance on the programmer to make the decision whether speculative "free" features are added is counter to this goal. Instead, I subscribe to the notion that its better to have slack visible (it should exist regardless), get all effort quantified (not negotiated!), and assume all features are out, unless explicitly included (with the inevitable ambiguities resolved via regular customer discusion). It makes for better expectations all around, it builds trust, it puts decidable changes (i.e. changing scope instead of trying to change estimates) in the hands of the decision makers, and -- at its best -- results in better quality software, with better work environments.

But that's just me.

[Note: Yet another old draft]

"Can you be trusted?"

It's a question we ask people all the time -- not necessarily out loud, but we do so nonetheless. We ask it by evaluating their actions, and their words. We look, rather than listen for the answer. More than their personalities, more than their motives, we look at their results.

This entry was drafted not long after I found myself wanting to literally ask someone this, but didn't. I reflected afterward: "Why not?" Why, almost as soon as the notion formed, was my inclination to reject the idea? It would be rude for one -- even politically inexpedient as I was a consultant and that person a client manager. It seemed pretty illogical for another -- seeming about as fruitful as asking someone if they are a liar.

Yet the more I thought about it, the more I liked the idea. I figured the rude part could be mitigated by careful phrasing, but that left the logical futility. Since the answer is going to be some form of, "Yes, I am trustworthy" why bother?

My conclusion was that the value (if any) would come, not from what the person answers, but how -- and what they did afterward. If his actions had shown the question had rattled him, or led to a change in behavior (I am assuming that it was prior behavior that led to my desire to ask in the first place), it could be a sign that the person is manipulating how they are viewed by others.

In short, I concluded that usually asking the question would be useless, but the surprise factor of throwing it right out there, might just rattle someone's cage enough for you to get an accurate glimpse inside.

I kinda wish now, I had asked. Maybe someday, I will.

Congratulations to my hometown Pittsburgh Steelers on their pending trip to the Super Bowl after a great run in the AFC playoffs. Go Steelers!

I have vivid (and fond) memories of the pandemonium in my hometown (about 40 miles outside of Pittsburgh) after the Steelers beat the Rams in Super Bowl XIV -- hard to believe that was 26 years ago.

Other Fond Steeler-related memories for me include: meeting (and having my picture taken) with Rocky Bleier when I was about 11, and playing at Three-Rivers Stadium for the conference championship for my senior year in High School (we lost 19-0, but it was a memorable experience nonetheless).

A less-fond, but still vivid memory stems from an encounter with my childhood hero Mike Webster. My dad and I had gone to a Steeler game, and I was so keen to get Webster's autograph that I talked my dad into waiting by the player's exit for what seemed like forever. Finally, he came out -- and walked right by me without acknowledging me in any way. That's hard on a little kid and I was really disappointed, but eventually got over it (he was still the greatest center who ever lived after all) -- especially later once I realized that he might have been too ill to notice (for more follow the link).

Anyway, I'm not known for following football too much, but I've really followed the NFL this season. I credit (blame?) Tivo and the HBO show "Inside the NFL" for changing that this year. Apparently, I picked a good season to pay attention. Watching the Steelers in Super Bowl XL will be a great finish.

Hopefully this year, Pittsburgh will get the mythic "One for the Thumb."

This site has been through several incarnations. For the earliest, Alan Francis guest-hosted it off his old site (twelve71). After getting a domain name, I hosted it for a time on the ThoughtWorks gigaton server (a sorta catch-all server with free space for employees). That was great (read: free), but the fact that it got used for all kinds of things made it less reliable than I wanted, so I went hunting for a permanent home. Again, Alan was there to help; this time in the form of recommending Bytemark. I've been there ever since. Great service, great price. Just the right balance of control and reliability that I was looking for.

With all that moving around and rebuilding, some links have rotted away, and there are some links out there that don't go where they should. I wanted to fix that. So I tackled the first one this week: Links to url's that point to Alan's old site. IOW, Once again, I looked to Alan for help.

Here's an example of the problem:

Keith Ray links to an early posting here. That link points to: http://www.twelve71.com/caputo/archive/000236.html. From the context of Keith's posting, its clear that link is meant for my take on exceptions (a rebuttal of an article from Joel Spolsky). The real page is actually named archives/000009.html. What to do?

There are two challenges: Getting traffic going to the folder on Alan's machine to come to mine, and redirecting archive links into their new location. Alan dealt with the former, and I the latter. Here's how we did it:

Alan was kind enough to add a permanent redirect from the folder to my site. He accomplished this easily by simply adding the following to his httpd.conf (Apache of course):

Redirect permanent /caputo http://www.williamcaputo.com #requests for /caputo folder redirected to site

If the directory/file structure on my site was the same, this would just work, but because I've changed some things since then, I had more work to do. The biggest difference is that my permanent links used to be under a directory called '/archive/' but now are located in '/archives/' -- so I added that folder. Next, the articles under the old folder had different names than they do now -- the version of MovableType that I use, annoyingly uses the index number of the database entry for the file name. This has changed in newer versions. Since I didn't want to copy the files to both, and the numbers have changed, I decided to handle this at the OS level via symlinks. However, symlinks (by default are not followed by Apache, so I had a change to make to httpd.conf too:

 <Directory /path/to/wwwroot/archive/>
        Options FollowSymLinks
</Directory>

This allows symlinks only in the archive folder. Now I can create a symlink in archive/000236.html that redirects to archives/000009.html. The only downside is that they actually show up as two different pages in my traffic stats, but I consider that an advantage: it will be easier for me to track how much traffic actually goes through the older "sites" (and possibly notify other websites to update links).

With that one out of the way, I still need to detect and handle dead links (i.e. 404 errors). I need to investigate log readers for that purpose (time is at a premium, suggestions welcome). I also need to deal with incorrect links (i.e. links that point to real pages on my site, but not the one's they are supposed to). Some of these are easier than others (more on those as I fix them).

This might seem like a lot of work. Granted, I could simply say: "That's the problem of those who linked to me -- they should keep links up to date" but that doesn't really help those trying to find their way here -- nor does it help me get more readers. In the end, that's just bad customer service... not something I believe in.

As a bonus, I got to learn a bit more about Apache. I just hope things don't change more -- keeping this page's working would then be quite a challenge.

Short entry today. Clark Sell has been doing a lot with Visual Studio Team System (VSTS), and he's posted a jumpstart kit (almost a portal/clearinghouse) to help people wend their way through getting started with it.

Good Stuff, thanks Clark.

Mike Roberts points out the difficulty of applying Agile methods in an organization with values that don't match those of Agile practioners. Specifically Mike describes the difficulty (but possibility) of using an Agile method when the organization doesn't align software development with business value. Why business value? Well, according to Mike this is what many feel Agile methods value:

The principles behind the Agile Manifesto state "Our highest priority is to satisfy the customer through early and continuous delivery of valuable software." I've often heard this shortened into a frequently used battle-cry of Agile: It's all about business value!.

("Business Value" here seems to mean some objective, measurable bottom-line goal)

Mike then goes on to note that in an environment where this "Business Value" is not the goal, Applying an Agile method successfully is much more difficult. Fortunately Mike also points to a way forward: "what's needed is a knowledge about what motivates every individual within the stakeholder community." I assert that this is what should be done in the first place.

I think Mike has (perhaps unintentionally) illuminated a flaw in Agile as commonly practiced these days. Namely: those Agilists who are running around issuing the battle cry of "Its about business value" are making software development about them rather than their Customer. I.e. they are violating the very principle they profess to practice.

If you don't feel its important to understand what your customer values, if you assume that they (or worse tell them that they should) equate valuable software with 'software of Business Value' of course you are going to have a very difficult time succeeding (unless they happen to agree), because you are working toward a goal other than that of your Customer.

For those people who believe this (I am not saying that this necessarily includes Mike) what I am about to say might seem profound (to everyone else it'll just be common sense): Customer satisfaction is not achieved by preaching to your customer that they must strive for "business value" -- or any other valuation that you have defined as important, but rather by understanding their goals and giving them software that achieves what they value, whatever that might be. IOW: 'Business Value' means 'What the Customer Values' -- no more, no less.

So, save yourself the trouble; leave this "Business Value" stuff out of it. If that happens to be the goal of your Customer then by all means that's how to measure the value the software you're creating (and certainly its good when one has such customers), but that does not imply that it must be the goal of one's Customer, or that there are not legitimate Customers who may value something else (particularly in big IT where Agile adoption is all the rage right now). That little detail seems to have gotten lost during the Agile diaspora.

The fact that "[S]atisfy the customer through early and continuous delivery of valuable software" has turned into: "It's all about business value" means that being about the Customer, or more generally the People (once a key Agile value) is no longer true. If this is the case, this reinforces my feeling that Agile is in the process of becoming its opposite, serving to reinforce the very problem it was once a reaction against: Programmers deciding what's important for their Customers, instead of asking them. Wrapping it up in a pretty Agile wrapper just soils the wrapper.

In short, its not about whether one's stakeholders are striving for "Business Value" that makes using an Agile method hard, its that getting one's stakeholders aligned is hard -- regardless of method. What allows one to deliver anyway is the ability to align one's effort with the customer's desires, not the other way round.

Not telling them what they must value would be a good start.

A few weeks ago, I mentioned that there is some buzz in the blogosphere that Agile and lisp may be converging as the future of programming at Microsoft. In the comments, Jeremy Miller asked me "how or where a functional language approach is advantageous" and I suggested that someday I might follow up on why.

Well, Sriram Krishnan has provided both a good start on answering that question, and pointers to a whole lot more resources, so for now at least I will simply refer you to him.

Enjoy!