William Caputo (logosity)

Design goals and Complexity

William E. Caputo — 2013-03-14T05:00:00.000Z

Programmers solve problems. How they solve each problem is a function of their skill, talent, knowledge & time. The resultant solution will resolve the problem with greater or lesser complexity in the design. Thus, I find thinking about how (and more importantly where) complexity is handled, a useful way to evaluate a design. For this I use the time-honored technique of having three levels:

Level Zero

(Below this level, the problem is not solved)

The programmer has provided a solution. However, the interface is a mine-field; click things in the wrong order, the program crashes. Or it works great, but you need to reread the documentation Every. Single. Time. Or the resultant surrounding workflow is a Rube Goldberg device. However it surfaces, the programmer has placed the complexity on the user.

Level One

The interface is well thought out and reasonably intuitive. Controls work as expected and the solution is robust - even bulletproof. The solution not only fits the larger context, it improves it. The code itself however, is hard to change (or even understand). It is not well-organized; Or is, but full of tricky, interleaved logic. Or over-engineered, over-patterned etc. Here, the programmer has placed the complexity on the maintainer.[1]

Level Two

The burden of managing complexity is on the current programmer. The person writing the code takes the time and employs the talent and skill necessary to find an elegant solution that reduces the code to the minimum complexity needed to solve the problem and does so in such a way that is understandable (and changeable) later: The user gains the benefits of a level 1 design, while maintainers are left with clean code to change.[2]

Implications

Level zero code is common; it's the default for new programmers and an uncomfortable amount of commercial software. The thing of it is: Too often this sort of code is derided (including by me) when there is both a de facto and a cost-benefit rationale for managing complexity this way. Short-term projects are one example; software written for one's own use is another.

Level 1 software can be very valuable and enduring. I tend to equate this type of code with the Hacker ethic in all senses of that term. And I don't mean that pejoratively - we all use hacked together solutions every day and much of the world's technology infrastructure is built on it. It is a practical and stable design level.

Having the skill (and the time) to write level 2 code is a rare and wonderful thing. Amidst the hyperbole, aspiring to be such a programmer is at the heart of the "Software as Craft" movement and is a worthwhile goal for anyone who aspires to be a professional programmer.

I don't believe level 2 is inherently better than a level 1 (or level zero!) - it's about context - however, I think that for regularly edited and changed code (i.e. much IT software), this level of skill is what's implicitly expected (if not gotten) by the customer. Ironically, the time needed is often the first thing that goes as a non-technical customer can only assess code quality based on level 1 considerations and so pushes for faster results because everything "looks" OK. Only later - when their investment can't be changed without major overhaul - do they realize there is something wrong. And so the cycle continues...

Summary

This model is handy in several situations including: Judging the quality of an actual solution; choosing among different solutions to a given problem; estimating - and even when to stop refactoring. And while I don't believe that all code should be worked until it exhibits level 2 quality, it is what I want others to think of the code I leave behind. And it is what I hope to encounter in theirs. That said, being a successful professional programmer requires (among many other things) the ability to write all three levels of code, and the judgment to know when each is appropriate to use.

[1] This may well be the same person who wrote the code. The essential characteristic of a level 1 design is that complexity is put off to future efforts, not the current one. Thus level 1 designs tend to be high in technical debt.

[2]How the programmer achieves this is a matter of personal preference and technique. I do not subscribe to the notion that any particular set of programming practices provides this - or inhibit it by their absence. It has always been the people, not their practices that is the essential determinant of quality. Anyone who says different, is selling something.

Expoloring the French Defense (G30 practice game at DRW)

William E. Caputo — 2013-01-19T06:00:00.000Z

Played an interesting practice game last Friday (1/12) evening with one of my fellow DRW chess teammates, Oliver Gugenheim. After my stupendous blunder last week, I'm interested in playing some practice games - both to drill my pre-move thinking process, and because there's nothing like a bad loss to motivate one to start playing again...

Oliver and I wound up exploring a sharp line of the French defense - an opening I have historically not enjoyed playing as white, and so had started learning more about the past week. Oliver (without us discussing it) obliged me by playing a line I had looked at that day so we went a good way into the "book" before (very quickly thereafter) reaching crazy territory.

The most interesting bit tho, is actually black's move: 9. ... f6. The conclusion I got from this analysis, is that 9. ... f5 is better (see below for more) and so this was a useful game for this analysis alone...

All in all, it was interesting to play, and gave me the opportunity to practice the things above... and it gave Oliver a chance to fend off a ridiculous attack (which is always satisfying if a bit scary at times). Here's the game and my notes (Time Control is G30 with 5 second increment):

Event:
Site:
Round:
Date:

White:
Black:
Result:

Side to move:
Last move: variations:
Next move: variations:

Move comment:

And so, QED on this idea. My conclusion: better off building an attack here as White's got the ball. Also, for a bishop sac to have any chance, white really needs another piece. Perhaps once the f-pawn were advanced and White has castled, the possibilty of lifting a rook with tempo might be enough to give the sac some teeth. It'd be interesting to see if I can find any Winawer games with a bishop sac on h7 (if I do, perhaps I'll write a follow-up; regardless, looking at how White attacks here should be fun.)

CICL Chess Match: Board 6 vs Northwestern

William E. Caputo — 2013-01-07T06:00:00.000Z

I play in the Chicago Industrial Chess League (CICL) for my employer DRW. I generally play one of the lower boards, but I do pretty well. Two nights ago, I played possibly the best chess game of my life... and lost.

How? By touching the wrong piece. I had a choice of two recaptures - one that would win the game, and one that would lose. If I had simply moved quickly, that would've been bad enough, but I actually noted the problem with the first recapture, and then did some sanity checks for the other move. Content that all was good, I started my move by picking up the bishop to be captured, AND THEN PICKED UP THE WRONG CAPTURING PIECE. I realized it immediately, but by then it was too late. I resigned a few moves later.

The worst bit is I found a strong move on 18. (see below, I'm playing white) - and I saw that position 3 moves earlier. I don't always play this well when I'm rested - let alone after a 12 hour work day... which of course just makes the blunder that much more painful.

Moral of the story? When you're winning, when you're tired (or both!). STOP! Check it again. Check each bit, write it down and check it a third time. Then if your sure, make the move.

This one burned. Hopefully it stung enough that I won't do it again.

Anyway, for your viewing pleasure, here is my wonder-blunder:

Event:
Site:
Round:
Date:

White:
Black:
Result:

Side to move:
Last move: variations:
Next move: variations:

Move comment:

Site Redesign

William E. Caputo — 2012-12-28T20:40:00.000Z

After an amazing amount of procrastination, I've finally managed to redesign the site. All old links still work (if they don't, please let me know by emailing me at 'bill at this domain name').

Why? Two reasons: 1) I found my old layout limiting for what I want to do and, 2) Something in my toolchain got borked and I wasn't able to update my main page. This last one was a godsend because I decided that - rather than troubleshooting the problem, I'd put that energy toward my redesign.

Notable changes:

No more MovableType. I got a lot of mileage out of using MT, but in the end I hated having to use a database to store what is essentially static data. Furthermore, the templates and even writing of entries was a bit too much of a flow-mismatch. I wanted to write copy and edit layout in vim and just push the changes to the server. Now I'm using docpad and I have pretty much exactly that workflow
My "Thoughts on..." blog is no longer the main page. Right now, the main page is pretty sparse, but as I add more projects (I have some in mind, it'll start to fill up some more. For now, it's an easy way to get at some of the more popular material on my site, and...
My dabbling in Fiction. A few years back I started writing flash fiction under my middle name ("Edward"). I haven't in a while, but I hope to start again. So, now that's here too.
Total style overhaul. Hope you like it, because I'm sticking with it for a while. My goal was minimalism. This site has always been about what I write. Therefore I want to make it easy to find the stuff here and read it with no distractions.
No comments. This may change at some point (I had embedded Disqus but removed it) as I'm going with the Oatmeal Philosophy - I'm struggling against several years of writers block. Feedback can be great, but it can also demotivate. I need to avoid demotivators as I ramp this up.

That's it now that I have put everything I could see I needed in place, I'm outta excuses. Whether that means I'll be writing more, we'll have to see...

Programmer (blood) types

William E. Caputo — 2012-11-10T20:10:13.000Z

A lot gets said about evaluating technical skills when building teams, but I find that to be relatively straightforward - what's trickier is understanding how a person prefers to work and so how much effort (i.e. change) will be needed for the team to gel once that person is added. Consequently, I spend a lot more time thinking about this than any specific skill a person might have.

It's as if each of us has a personality blood-type. Just as transfusing blood among individuals without taking account of their body chemistry can result in serious health issues, so too mixing otherwise competent individuals without regard to group chemistry can greatly reduce everyone's effectiveness. This is true, even if these same individuals would be very effective in different groups.

Like one's actual blood type, there are multiple factors involved, but I find three especially important. So, to run with the metaphor let's call them 'A' factor, 'B' factor and 'Rh' factor:

A Factor: Knowledge Sensitivity: A person has this trait if I expect the effort needed for them to understand the Situational Context of the team will be a significant factor in the gelling process. Considerations include: current knowledge, learning preferences (online research, training, books, etc), learning-speed, learning-skill, open-mindedness and how (and how easily) they'll learn from their teammates.
B Factor: Environmental Sensitivity [1]. This stuff matters to most everyone, but some are more sensitive to it than others. If it looks like it'll be a prominent part of the gelling process, they have this trait.
AB: Person has both traits.
O: Person has neither. I don't expect the person will require radical shifts in team cohesion to gel with the group (though they still may cause such shifts over time).
Rh Factor (+ or -): Tendency toward taking social initiative[2]. This one's hard to define. To get a sense of what I mean, the next time you're in a casual group discussion (ideally of peers), note the people who tend to pick the topics of conversations, suggest lunch destinations and so on. These people have this trait. When work topics arise, they're the ones proposing new ideas and things that need focusing on in the first place. Incidentally, if you believe you have this trait, you might need to reign it in to get a sense of who else has it (and a good habit for you to develop regardless).

I see these factors as neither strengths nor weaknesses. Nor as (necessarily) fixed traits. A successful team may contain (or require) people exhibiting any combination of these traits. Adding people who learn quickly or who care a lot about how customers are acting toward the group or are passionate about their work environment [3] is just as important as having deeply experienced people who are agnostic to the details of the work environment. Even when people's quirks or lack of knowledge might seem to be burdensome, our work is a creative activity not a mechanical one. Taking the effort to incorporate people who have a particular genius or ability into my teams has paid off more often than it's been a liability.

It is also important to understand this is my thinking process after it becomes clear the person is a plausible fit for the team. That means they've already passed the no assholes rule, otherwise appear to have the skills the group needs and I believe will get along reasonably well with the rest of the team outside the context of the goal at hand.

Of course, I also consider many other non-skill traits to be important - e.g.: risk aversion; time flexibility; tolerance for ambiguity, tool/practice prejudices and social neediness are all relevant - but regardless of which ones dominate a particular gelling process, what I'm always after is a better understanding of how the person will complement the rest of the group. Most importantly, reflecting on why people work well together helps me to do my job better. Hope you find it useful for that too.

[1] More precisely: How sensitive the individual is to forces acting within or upon the group; this important topic will get a separate blog entry hopefully soon

[2] Leadership in the narrow sense of desire to influence direction of a group of people.

[3] Tool and process preferences is an entire - large - subcategory that can even be considered an entire trait all it's own. We do so love our tool and practice religions!

Aretê

William E. Caputo — 2012-05-24T01:50:30.000Z

"I am not a good manager."

I say this to people often, and most of them think I am joking because... well, because people don't say things like that about themselves if they're true and also, I have cultivated some skill in building strong teams that accomplish what they are asked to do. [1].

But I am serious when I say that. for three reasons:

I really have to work hard to do the organizational parts of a manager's job (the reports, the budgeting, the note taking, the meetings, the scheduling, the selling of initiatives, etc)... and I don't enjoy any of it.
I don't do well at optimizing for efficiency.
I don't have much use for roles, particularly ones that tend to specialize activities and artificially segment the work needed to solve the problem (and the skill set of those seeking to do so).

Like most programmers, I suck at estimates, I am motivated first by a need to solve interesting problems, and only secondarily at reaping the benefits of doing so, and I have zero desire to "be in charge of others." I also hate process for process sake and generally piss off any project managers foolish enough to work with me (though I am good friends with several).

So, how do I build teams and software? By treating efficiency - and even the primary goal of the team - as a secondary effect, and optimizing instead for... for what?

Well, until recently I had been (in my head, because I didn't feel too comfortable saying this out loud) using the word: "happiness." Of the members of the team, of my boss, my employer, our customers and (importantly, but until recently neglectedly) me. Make all of these folks happy, and everything just works.

Uncomfortable because this is a tough sell to accounting-type folks - and anyone who prepares budgets. "Naive", "Crazy" and "Ridiculous" are what I expect to hear. And the reason I expect to hear it is because it seems really risky - even terrifying - to them when I say anything that implies I'm not thinking about cost and value and ROI, and all those other business terms.

But I am. I'm thinking about them all the time. I just don't agree with them on how to optimize them.

Also, "happiness" is not quite right: some people are quite happy to do nothing, others are only happy when they are padding their egos at the expense of others, and a whole lot of other types of "happiness" that I don't optimize for. No, it's a specific kind of happiness - especially inside the team - that I am trying to maximize. Joy of doing one's best, professionalism, craftsmanship, cultivating flow, the need for slack. All dancing around it. All not quite it...

I recently mentioned here (and on twitter and facebook) that I'm re-reading Zen & the Art of Motorcycle Maintenance by Robert Pirsig. And it's blowing me away. Again. I read it a really long time ago - so long ago, that I had forgotten all of the details and only remembered: "That book really moved me and shaped my thinking."

What's been blowing me away is realizing how much it has done that. And so, I've been expecting for days now to find in it somewhere something that supported my view since Pirsig's focus on Quality and Care (not to mention technology) are very similar to my feelings of "optimize for happiness." But it still hadn't felt quite right yet...

...until, today. And I found it: The Greek word: Aretê (translated as "virtue" or "excellence") is a central part of any course on Greek philosophy and I had several classes in college where it was discussed. I was waiting for it to come up in the book (it doesn't until chapter 29) and when it did, I realized I was getting closer. Then, this quote:

"Aretê implies a respect for the wholeness or oneness of life, and a consequent dislike of specialization. It implies a contempt for efficiency - or rather a much higher idea of efficiency, an efficiency which exists not in one department of life but in life itself." ~ Pirsig, Robert M. (2009-04-10). Zen and the Art of Motorcycle Maintenance (p. 360). Harper Collins, Inc.. Kindle Edition.

And like a key fitting a lock, there it was: I believe that teams (organizations!) should be optimized for Aretê; that teams should be staffed with those who - like Hector the tragic hero of ancient Troy - seek excellence (in their work and in achieving the team's goals) for it's own sake and are not happy unless they are free to pursue it; and finally that the team's success (which will still be a function of external perception of value) will be a natural outgrowth of this process and any attempts to shortcut it (e.g. in the name of efficiency) will actually serve to reduce the team's effectiveness.

With this compass in hand, I can see now that what at times appeared to be random objections to process changes and my novel (some would say crazy) alternatives and experiments over the years have really been about trying to keep everyone focused on maximizing the ability to pursue excellence.

So, now I'm saying it: If you want to build great teams who reliably ship results: Don't optimize for efficiency, optimize for the pursuit of Aretê.

[1] Those who've worked with me on those teams can attest, I've done so by getting the team building part - and my role in it - wrong a lot, but learning from it.

Logic Engineers

William E. Caputo — 2012-05-17T11:55:20.000Z

I don't like any of the names for people who create technology solutions for use in the real world. In no particular order:

Developer - Sorry, that's for photographic film (and who uses that anymore?) not software - my second least-favorite on this list.
Programmer - probably the one I use the most, but the mechanistic connotations: a glorified typist of code; and the implication is that someone else does the thinking, bothers many - leading to...
Architect - Horrible. Borrowed from physical construction, it brings in notions of rigidity of thinking and solution; my least favorite, often used by those who use 'Developer' to describe "those who do the thinking around here."
Coder - Probably my favorite; however implies that software is the focus instead of the problem to be solved.
Hacker - I don't mind this one, but it feels forced sometimes and of course carries negative connotations due to it's use in a security context; It does however, put the problem solving front-and-center and I like that;
Software Engineer - Part of my current title; pros: sounds professional and captures the applied science nature of the work; cons: again the implication is writing code, not general problem solving and often - though not in my case - is indicative of a very rigid, bureaucratic culture
Software Craftsman - En Vogue with those who see (as I do) that there is more to this work than just slinging code, I have several problems with it: software focus again, the lack of gender neutrality, and a certain pretentiousness (that might just be because some people sound that way when they talk about the movement - which I am positively disposed to, but not a part of)
Technologist - This one is popular with the start-up crowd; This is one of the few on this list that recognizes the holistic integrated nature of the work from the solution side; on the other hand, lacks action, professionalism and anything related to the problem (not a coincidence that the startup world likes it, because those that use it are often seeing the project as separating problem and solution too)
Sys Admin, DBA, network engineer, etc, etc - I.e. every other technology-related role that isn't primarily about writing software. This is my biggest concern of all as I believe this distinction (writing software and everything else) is horribly misguided, artificial and needs to go (that's another entry)

Am I just being pedantic? I'd say yes, except that I believe these terms all devolve from not seeing the holistic nature of solving problems with programmable machines and since I do see it that way, I want a term that encompasses all of it.

Thus Logic Engineer. Physicists have Mechanical Engineers to apply their ideas, why don't Logicians have Logic Engineers? Currently, we conflate this practical science with Computer Science which is also a pure science - or would be if we stopped doing that. I know that Engineer has professional accreditation connotations in many disciplines - and I'm mixed on what that would mean given the current state of technological practice today - but I believe this most closely maps to what we are doing:

Logician: figure out the rules and limits of valid reasoning.

Logic Engineer: figure out the rules and limits of applying logic to real-world problems. We are Logic Engineers.

Further, if we stopped slicing the world up into hardware and software practitioners, and started seeing that software is just liquid hardware (and vice versa); maybe we'd have something worth accrediting (though I don't think for a long while yet).

So, I'm gonna try it for a while... at least in my head.

Dedication

William E. Caputo — 2012-05-11T15:10:25.000Z

I've been rereading Zen & the Art of Motorcycle Maintenance. This quote really struck me:

"You are never dedicated to something you have complete confidence in. No one is fanatically shouting that the sun is going to rise tomorrow. They know it’s going to rise tomorrow. When people are fanatically dedicated to political or religious faiths or any other kinds of dogmas or goals, it’s always because these dogmas or goals are in doubt." ~ Pirsig, Robert M. (2009-04-10). Zen and the Art of Motorcycle Maintenance (p. 140).

Made me think of all the zealotry accusations levied at Extreme Programming practitioners about 10 years back... still seen in reaction to some of the more ardent claims of the primacy of TDD even today.

No conclusions, just suspect I will see the next fanatic a little differently next time I encounter one.

Why Teams

William E. Caputo — 2012-02-05T20:41:22.000Z

Jeremy Lightsmith recently asked on twitter:

"Why are teams important? I'm looking for an inspired answer."

I don't know about inspired, but here's my thoughts on the topic:

First, I believe teams (true teams, mind you, not simply groups of individuals under single managers) are capable of delivering better results than individuals - this is the economic (and in a business sense the only) justification for teams. If they can't outperform, they probably shouldn't exist.

Much of my thinking on why they can outperform has been deeply influenced by the book: "The Wisdom of Teams" and I won't rehash it here except to say that I believe the key is in the notion of "complimentary skills" - teams are more efficient because rather than having to staff a group of super-humans (you know the kind that typical HR review material seems to want to turn us all into) you can look for how one person's personality and skills work to reinforce and amplify the others.

And that brings me to my most compelling belief of all: It's in our nature. We are social creatures, we need to connect with others, to bond with them, to feel we are working together. Rather than spend our time lamenting our "shortcomings" and worrying about our individual performance (and rewards), forming solid teams means using all of our actual skills as they are to compliment our peers and deliver more than any of us can alone.

People simply work better with a group of people who complete them, whom they feel a deep connection to, whom they trust whom they admire and feel respected by. When you have that, people stop thinking about what's wrong with them, and start thinking about how they can help each other to succeed. And when you have all of that, you have a force to be reckoned with.

Is this easy to accomplish? No - and you need to accomplish all of that (respect, complimentary skills, trust, common goals, acceptance of each other as we are, etc) to achieve a true team. But I am convinced that the payoff for everyone - from the employer to the individuals involved is so worth it, that I won't willingly work any other way - and I reject any "best practices" or conventional wisdom that interferes with achieving that sense of team in the groups I work with.

Python Challenge answers 0 thru 4... in clojure

William E. Caputo — 2011-10-20T01:48:12.000Z

The Python Challenge is a nifty site that presents you with a series of puzzles that it asks you to solve using python; getting each answer allows you to move on to the next puzzle.

Python is a cool language and it's a good tool for this job¹ However, I'm learning clojure right now, so I thought it would be fun to try and solve a few of them in clojure. Here's my answers for challenges 0 thru 4 (warning: if you want to do these puzzles yourself, reading further now might ruin the fun)

Challenge #0 (the "Warmup")

Asks you to solve 2 to the 38th power:

(clojure.contrib.math/expt 2 38)

i.e. just use the exponent function in clojure contrib.

Challenge #1

This one throws some scrambled text at you and a clue on what the key is (ROT 2):

(defn translate [text]
  (let [lookup (vec (map char (range 97 123)))]
    (letfn [(letter? [c] (and (>= (int c) 97) (<= (int c) 122)))
            (shift-2 [c] (mod (+ 2 (- (int c) 97)) 26))]
      (apply str (map #(if (letter? %) (get lookup (shift-2 %)) %) text)))))

Create a lookup table of the chars, a predicate to test if a char is a letter. & a function to get the index of 2nd to next letter (the index loops, essentially making lookup as a ring buffer), then map across the given text, shifting by 2 if its a letter or just returning the char if its not.

Challenge #2

This one throws a big hunk of random data at you and suggests you pick out the 'rare' characters:

(defn filter-file [path]
  (let [fs (line-seq (clojure.contrib.io/reader path))
        lookup (set (map char (range 97 123)))]
    (apply str (mapcat #(filter lookup %) fs))))

A quick visual scan of the text led me to a strong hunch the "rare"² characters were lowercase alpha, so:

Re-use our lookup table from the last challenge; this time make it a set, then use the set to filter each line of the file denoted by 'path' (I first saved the text to a file to make it easier to work with); use mapcat to flatten the lines out (this has the effect of stripping empty lines altogether); apply str to the resulting sequence to get the answer.

Challenge #3

This one's a big hunk of text too, so a quick refactoring of our last solution results in a more abstract (and higher-order) function that takes a filter function as an additional parameter:

(defn filter-file [filter-fn path]
    (apply str (mapcat filter-fn (line-seq (io/reader path)))))

the filter from challenge #2 thus becomes an argument; partial works nicely here:

(filter-file (partial filter (set (map char (range 97 123)))) "path/to/file")

Now we can make a new filter for challenge #3. This one will need to find character patterns that look like this: ABCxDEF. We'll need grab x. This one just screamed regex at me, so here's a filter that gives us the answer:

#(second (re-find #"[^A-Z][A-Z]{3}([a-z])[A-Z]{3}[^A-Z]" %)))

An anonymous function³ that uses re-find to match: "not-cap followed by 3 CAPS followed by not-cap followed by 3 CAPS followed by not-cap"; the second element of the resulting vector (because we use parens to create a group) produces x; mapcat et al do the rest.

Two big assumptions/limitations here: assumes each target is on its own line, and that the target pattern wasn't on the beginning or end of the line (which was good enough to get the answer).

Challenge #4

This challenge requires one to follow a url call chain, passing a different number as the argument to a 'nothing' parameter each time. The resulting page text provides the next number to follow (and/or some noise to keep you on your toes) until eventually we get the answer.

This one gets kinda ugly.

This is the kind of problem scripting languages are made for (e.g. perl, python & ruby coders would all make short work of this problem). Still, it's possible to write procedural code in clojure, and it's still reasonably straightforward.

One decision I had to make is how to GET the url's - my weapon of choice for this sort of thing is clj-http:

(require '[clj-http.client :as client])
(require '[clojure.contrib.string :as string]

(defn follow-chain [base-url number]
  (let [result (:body (client/get (str base-url number)))
        idx (.indexOf result "and the next")]
    (cond
      (re-find #"^Yes" result) (do
                                 (println result)
                                 (follow-chain base-url (/ (Integer/parseInt number) 2)))
      (= -1 idx)               result
      :else                    (let [result-vec (string/split (subs result idx) #" ")
                                     next-number (last result-vec)]
                                 (println result)
                                 (recur base-url next-number)))))

Take the url as a base & the first number to follow; use client-http/get to grab the page; extract the body of the page; get the index of the phrase "and the next" using the java "indexOf" method - we'll use the index later to parse out the end of the text and get the next number...

...unless of course, we get text that tells us something else (like a message saying "Yes" and then instructing us to divide the last number by two and continue on as before) so...

...we set up a switch using the cond macro: If the result starts with "Yes" make a recursive call dividing the last number by two; if indexOf otherwise came up empty, that's our answer, so return it; else pick the next number out of the result by splitting the end of the string into a vector (using clojure.contrib.string/split) and recur (tail recursively call the function again).

The println's could be removed, although they were essential when figuring out what the code needed to do.

Conclusion

This was a fun exercise; clojure's holding up pretty well so far, though clojure would not be my weapon of choice for that last one; if I choose to do the next five, I'll post them in a future article.

Footnotes

[1] It's also the darling of the hipster crowd right now -- in many cases the same people who snubbed python when ruby was the hip language about a decade ago... python abides.

[2] The official challenge answers also tackle ways to deduce "rare"; knock yourself out

[3] #() defines a function where % %2 etc represent positional parameters; the (fn [arg]) syntax would work here too

My first clojure macro

William E. Caputo — 2011-10-13T01:23:06.000Z

I'm finally experimenting with writing macros in clojure. Learning macros is (for me at least) a 4 stage process:

Learn to use them (pretty straightforward)
Learn to read their implementations (including the quoting)
Learning to write them (in progress)
Learning when to write them (in progress)

Those last two are iterative; #4 is especially tricky -- the web is full of general considerations ("when a function won't do", "when you want new syntax", "when you need to make decisions at compile time", etc) - but actually making that judgment in practice, takes... well practice.

Hence this exercise. Anyway to the code:

Clojure offers the if-let and when-let macros that allow you to combine a let block with testing the binding for nil:

(when-let [a (some-fn)]  
   (do-something-with a))

(if-let [a (some-fn)]  
   (do-something-with a)  
   (otherwise-fn))

I found myself (on some real code) wanting to be able to do something similar with try:

(try-let [a (some-potentially-exceptional-fn)]
  (do-something-with a))

(try-let [a (some-potentially-exceptional-fn)]
  (do-something-with a)
  ArithmeticException ((println (.getMessage e)) 42)
  :else (do-something-by-default-fn)
  :finally (println "always"))

etc.

So I wrote this (non-hygenic) macro that seems to do the job:

(defmacro try-let [let-expr good-expr & {fin-expr :finally else-expr :else :as handlers}]
  (letfn [(wrap [arg] (if (seq? arg) arg (list arg)))]
  `(try (let ~let-expr ~good-expr)
    ~@(map #(apply list 'catch (key %) 'e (wrap (val %))) (dissoc handlers :finally :else))
    ~(if else-expr `(catch Exception ~'e ~else-expr))
    (finally ~(if fin-expr `~fin-expr ())))))

Thing is... I don't if it's a good idea or not. For one thing its not hygienic (it implicitly declares e that can be used in the handler clauses) though this seems the kind of case that sort of thing is for.

For another... I don't know if its correct. It seems to be (I've tested all the scenarios I can think of), but this is kinda like security -- I suspect anyone can write a macro that they themselves can't break, but that doesn't mean its correct.

Some things to note: - e is available to handler expressions
- the local function wrap allows for a complex expression or single value to be spliced in
- any number of handlers can be included
- ':else' (default) handler and ':finally' handlers are optional (as are any others!)

In short: I'm interested in any opinions/feedback that aim at learning steps 3 & 4 (writing and when to write). Fire away!

Where Style Rules Come From

William E. Caputo — 2011-10-08T13:59:48.000Z

From a larger tutorial on Common Lisp Programming Style, comes a nice list written by Peter Norvig & Kent Pitman surveying "where your 'Style Rules' come from":

Religion, Good vs. Evil "This way is better."
Philosophy "This is consistent with other things."
Robustness, Liability, Safety, Ethics "I'll put in redundant checks to avoid something horrible."
Legality "Our lawyers say do it this way."
Personality, Opinion "I like it this way."
Compatibility "Another tool expects this way."
Portability "Other compilers prefer this way."
Cooperation, Convention "It has to be done some uniform way, so we agreed on this one."
Habit, Tradition "We've always done it this way."
Ability "My programmers aren't sophisticated enough."
Memory "Knowing how I would do it means I don't have to remember how I did do it."
Superstition "I'm scared to do it differently."
Practicality "This makes other things easier."

( via Common Lisp Tips )

Stand-ups

William E. Caputo — 2011-09-24T20:07:17.000Z

I've been saying this for a while, now I'm just gonna write it down:

Stand-ups suck. Treat them as a last-resort.

The classic stand-up: each day a group of people stand huddled in a circle chanting: "His Name is Robert Paulson... His Name is Robert Paulson... His Name..." - no wait that's not it...

They stand in a circle bored out of their skulls while each in turn says what they did yesterday and what they are going to do today. A couple of people sorta care what everyone says (usually project managers, or customer reps) everyone else is interested in what maybe one or two people say. No one can go into enough detail to really educate anyone. Everyone's glad when its over.

Sometimes people attend many of these per day (cross-team stand-ups anyone?). A few might even make it a significant chunk of their day & job descriptions; I think of these people as "Meeting Moths" - attracted to meetings like moths to the flame.

STOP. DOING. THIS.

Instead:

Figure out who needs status and how often; GO AND GIVE IT TO THEM. It's a good bet they don't need to hear the minutiae of what code was written anyway & you're probably addressing that in existing stand-ups by making sure the programmers don't get too detailed killing any value they might've gotten from talking about what they are doing - dysfunctions beget dysfunctions...
Those who are doing actual tasks should be free to talk about them when they need to. In my experience this part of the stand-up gets short shrift when its scheduled and attended by those not doing the work. GET THEM OUT OF THERE. If you do, you might just see spontaneous, short discussions start happening daily among task completors (that's a real stand-up btw, just don't call it that or you'll ruin it by attracting meeting moths)
Move people next to each other who need to talk a lot. Conversely, if you already are around each other all day, you probably already know who did what and whose doing what (if not WORK ON COLLABORATING MORE)

If after you are doing all of these things, you still feel like you need a periodic daily meeting, then think for a while! Be creative! Don't just do what some book or consultant said to do, use your brain and solve the problem to everyone's benefit as best you can. Only then, if you still feel you need a stand-up, then fine; go ahead and schedule one for the things you still didn't address (but don't cover things already addressed!). The result is much more likely to be useful for everyone involved.

All periodic meetings including stand-ups evolve into wasteful time-sucks; They develop an inertia that supersedes whatever utility they initially might have had. Aggressively question their very existence; make the burden of proof be on keeping them, not ending them. Simpler still, kill em all and see what's not being done and then be creative on how to fix it.

Who knows you might just get away with one less meeting in your day.

Software Patents are Amoral

William E. Caputo — 2011-08-06T14:30:17.000Z

UPDATE: This article illustrates what I'm getting at very well: Google's position is weak while Apple's is strong - making patents a moral issue actually hurts Google's chances at reforming them. Change the rules, then change your strategy)

Martin Fowler just wrote a great piece on why Software patents are absurd and should be abolished. As Martin is so gifted in doing, he pulls in just enough background and history to educate you on the merits of patents in general, then shows clearly and simply why software patents are harmful to innovation and a pointless tax on established businesses.

In fact, I'd go so far as to say Martin's article (as is so often the case) is almost a perfect short introduction to the rationale for abolishing software patents... until he gets to the end of his argument and ruins is it by admonishing "my fellow programmers" for being "complicit in this tragedy." Martin rounds out this unfortunate paragraph with: "Any programmer who cooperates in getting a baseless patent should be ashamed of themselves."

Now I respect that this is clearly a moral issue for Martin and I'm sure that many (including those reading this) share his belief that it is unprincipled for a programmer to be against patents and yet cooperate in the application process. I also respect that Martin is a man of principle. However, I think he is wrong to hold this one - and ruined an otherwise great argument by introducing it into his reasoning.

And while I think he did so as an attempt to influence those faced with the situation, I think it's likely to have the opposite effect - pushing some to rationalize that patents are a good thing, so they can sleep at night when they choose not to quit their jobs to avoid the shame.

Fortunately, I didn't have this moral hangup when faced with a similar situation, and so I can still sleep at night (I'll just have to hope Martin still respects me after this). Because I don't think patents are inherently immoral - any more (or less) than engaging in running a business is inherently moral.

Now I recognize (and you should too) that what follows could be nothing more than mere rationalizing on my part, that Martin's dig hit a little too close to home and I am simply jumping on my (much smaller) soap box so I can continue to sleep at night. You can judge for yourself. My goal here is to share my experiences with software patents, and why I think they are amoral (while also a mistake that should be abolished forthwith).

Yes, I've signed patent applications. Yes, I'd consider doing it again. I believe (though I don't hold them) that some of them were even awarded. Here's how it went down:

At a critical time in my former employer's growth (For those that don't know my employment history: the business involved manufacturing of software-driven custom-built hardware devices) we were faced with a thorny - yet common - business situation: Namely, we were in competition with other companies to secure the same market.

Further, the nature of this business did not require patents to secure a "moat" around one's market share, but that was certainly a strategy that could be employed - though in my then and current opinion not nearly as effectively as how my employer was going about doing so which was by providing better service, securing better partnerships and scaling up production faster.

However, one of our competitors was attempting to secure a software patent for their machine. This patent (as described by one of my fellow programmers at the time as "patenting client/server") had all the failings that Martin describes (and then some), but our attorneys - to my skepticism - felt our competitor had a good chance of getting it approved with a few tweaks and could then use it against us, so our best strategy was to get our own patents to fend them off.

Without giving away details, a combination of patent application, patent purchase and architectural change was the suggested strategy. We executed that strategy in various ways and thus positioned ourselves appropriately if the (as I believed farfetched) doomsday scenario occurred and our competitor got their patent of client-server.

As I said, I thought the whole thing absurd. No one would allow a patent for client/server, the lawyers were overreacting. Still, I'd seen enough absurd patents to agree we were at risk, and so even though I thought it a colossal distraction for our small team, and overly paranoid, I worked to identify our own "original" inventions and how we could change our architecture to tap dance our way past their absurd patent if it were granted.

Now, despite my skepticism, they did get (a slightly narrower version) of that damn patent awarded, they did rattle their saber, and the changes the suits suggested worked extremely well to make the whole thing a non-event in my employers growth.

So, now you know my "shame"; how I sold my soul to the patent demons. Right? Keep in mind, I was instructed by the highest management of our company to do this and if I held the position that patents were inherently immoral the only principled decision was to resign. However, once I understood the situation, I accepted the charge because it was clear that it was best way to counter the absurd patent. By the way, I didn't hide my opinion - I even shared it with the CEO - and while he recognized the merits of my argument (and even that the whole patent thing should be abolished) he also clearly saw the need (as did I) to defend ourselves based on the reality that we were doing business in.

And this is the key point I'm trying to make: It wasn't an unprincipled decision for me because my employer wasn't doing anything unprincipled. They were simply playing the game correctly based on the rules in front of us all. To ignore this threat to the business would've been wrong; wrong for our success, wrong for our shareholders and wrong for our customers (because we had the better product).

This is not to say one cannot act immorally while playing by the rules - the business world is replete with such things - but the mere existence of patents is just a rule of the game we call business. If there's a moral element - it's in how they are used. It is my belief, that neither of the companies in my example behaved immorally (though our competitor was playing a weaker hand), and if the rules were changed, the outcome would've been the same (just cheaper for all concerned). But since the rules are there, they were used because that's what business is: A competitive game where those who use the rules most effectively win. That the stakes are high, and that many cheat, doesn't change that (though it obscures it for many) and it doesn't change that patents are part of the rule book and should no more be avoided than forgoing castling in chess because one thinks that moving two pieces should never be allowed.

Anyway, I hope for those who are faced with signing a patent application, I've done a little bit to ground your decision on the purpose of that patent, and not its mere existence. The latter is a naive position that fails to recognize the business landscape we all (well most of us) work in as professional software developers.

What we need to do is to change the rule book. What I would love to see is a more active attempt to get patents banned by the software industry - and I see no inherent hypocrisy if they continue to use them while working toward their extinction (not that I really believe they will, unfortunately).

In sum: Making patents inherently a moral question simply distracts and retrenches participants and plays right into the designs of those who have the advantage of the status quo. Martin's shaming of his fellow programmers was respectuflly offbase in my opinion as it ignores the much more important consideration of how patents are used while they exist.

Software is not an asset

William E. Caputo — 2011-04-30T19:28:21.000Z

Software is not an asset - I know everyone thinks it is, but its not; its a liability - its very existence generates cost. Every minute someone spends coping with it in order to adjust the behavior of hardware (which is an asset) is a net-credit on the balance sheet and a decrease in profitability. A necessary expense in many cases, but a cost nonetheless. Getting rid of it goes directly to the bottom line.

Software isn't a thing - its a form of communication. Just as no one would list meeting minutes, or recordings of the CEO's speech as company assets, software should be seen as stored communications (with hardware). Its value should be viewed in how it changes behavior; its very existence should be seen as creating risk, and its deletion as key to a healthy techno-ecosystem.

This is the essential argument behind why refactoring is worthwhile, and why its value isn't really comparable to writing code. Its also why its hard for non-programmers to see that value - because they think of programmer time as a credit, when really its can also be a debit.

My advice to managers (and everyone else): Want a sure fire way to add value to your organization? Reward and encourage deleting of unused code; fight for and insist that programmers be given time to refactor and otherwise eliminate unused code, and don't account for time spent doing so in the same way you do the creation of code.

My advice to programmers: Delete unused code at every opportunity and scale (function, class, file, project, organization); don't compare the effort to do so to writing code; work at all times to have your software feature-set reflect its actual, operational use.

Oil 101

William E. Caputo — 2011-04-17T16:24:48.000Z

As I watched the Deepwater Horizon oil spill unfold last summer, I found there were many fundamental questions about the oil industry that I didn't understand. So, I started reading. I read many websites. Of these, The Oil Drum was probably the most useful, but I was still struggling with some basic questions like: "How does a country benefit economically from letting an independent company take oil from its shores?" - and the sheer volume of contradictory opinion, punditry and proposed solutions, led me to conclude: "I need to dig deeper."

So, I bought a book, Oil 101 as it got excellent reviews, and seem to cover everything about oil from chemistry & history, to exploration, production & trading. I recently finished this book and while its not a quick read, it is very well-written, interestingly presented and comprehensive. Written from a neutral point of view in that it describes what is, not what ought to be, it gave me just what I wanted: a good grounding that I can now use for further education, and evaluation of where we're headed.

Unfortunately, the book has created a new problem for me: Nearly everything I now see being presented on this topic (again the Oil Drum is a notable exception) seems, simplistic and uniformed, aimed at supporting one political agenda or another and/or just plain out to lunch.

To take just one example: I stumbled across this article about how the Koch Brothers are manipulating the oil market.

Now, I'm no fan of the Koch brothers - particularly their support for the anti-union movements and the big sleight-of-hand trick that is the tea party - but this article presents 'facts' and jargon, not to educate, but to scare.

Take 'contango.' Investopedia defines it as: "When the futures price is above the expected future spot price. Consequently, the [futures] price will decline to the spot price before the delivery date." Got that? Simple, right? Riiight...

To make any sense of Contango, you need to understand the oil futures market (not to mention futures themselves), the spot market, global supply and demand (both now and at the time of the contract) how oil is delivered, etc, etc - all stuff covered in Oil 101, but curiously absent from this and every other article I've read that mentions the term - probably because its taken me months to build even a basic understanding of the topic, so an article can't even come close - even if the article writer understands it. Instead its summarized neatly as: "demand is expected to outstrip supply." Wow, so simple a child could understand it. Clearly those who use the term 'contango', just want to hide how simple it is in order to steal from the poor.

Oh, and the gist of the manipulation? The Koch's are artificially reducing supply by leasing four supertankers and storing oil in them (presumably because they are using Contango against us). Just how much supply have they deprived the world of by doing this? Let's do some math (all figures from Oil 101, which is about 2 years old now, but it'll get the point across):

The largest supertankers (unlikely that's what's being used here, but we'll go with it): are called ULCC's (Ultra Large Crude Carriers) and they can hold up to 560,000 DWT (that's Dead Weight Tonnage). Again, unlikely to be filled to the brim, but we'll use the number as is.

4 ULCC x 560,000 DWT ≈ 2.25 million DWT 7.5 barrels per DWT ≈ 16.8mm barrels

Wow, 16.8 million barrels of oil... that's how many days supply?

Not even one. Not even one for the US! In 2008, the daily consumption for the US was 20.9mm barrels per day (bpd). How much did the world consume in a day? 85.7mm bpd.

16.8 million barrels isn't even enough to feed all of 2008's US daily reactor capacity (17.5mm bpd). In other words, the heinous manipulation being described in the article - at worst - is shifting supply by a day.

In truth, its not even close to that, because ULCC's aren't likely to be used for that, not all of the DWT is actually oil, the oil being stored may not service as much demand, etc, etc.

So why is Koch industries spending money to lease tankers to store this oil? One guess is that its a hedge. They run refineries, and certainly buy oil to feed those refineries using futures contracts. By storing this oil, they can hedge those contracts with physical oil (much like one can hedge a stock futures position by holding actual stock). Another, possibility is that they are hedging against a short-term supply shortage for input into their refineries (e.g. maybe a couple weeks worth).

Not so evil sounding, once you put some numbers behind it. huh? Actually, it sounds to me just like the descriptions in the book of how global supply is stored and moved so we don't see occasional spikes of $15.00 gallon gasoline in the summer. In short: The refineries are doing their job.

Now as I said, the Koch brothers seem to have a lot of blood on their hands - but articles like the one linked, make that hard to spot when they simply stir up the rabble with BS. The sheer size of the oil-based economy is incomprehensible, the numbers are enormous. An oil tanker-full sounds like a lot, but it isn't even the proverbial drop in the bucket.

Reading this book has made it impossible for me to see the oil problem in simple terms. Nor are any of the "simple" solutions to it... simple. I'm not even sure 'problem' is even the right word anymore - it's like saying 'life' is a problem to be solved. True I suppose, but not very enlightening. No, oil is deeply entwined into the fabric of our lives, and as its details change, our lives - all of our lives - change with it. No-one - or everyone (take your pick) is to blame for that, it just is. Oil magnates and speculators make such great scapegoats - they're rich, remote - and very likely using their power for their own gain, but hanging them in the square won't fix things. Understanding the facts, so we can make our own decisions about how they are using their power (both right and wrong) will.

We all owe it to ourselves - and each other - to learn as much as we can about how our world works. I've joked that I'm one of only three people who've read this book that aren't in the oil industry (me, the book's editor, and proof reader), if you care about this topic - and want to make your own educated decisions about what's really going on - without resorting to simple solutions like boogeymen. You should join the three of us in reading this book.