Once Upon A Time, A Nonsequitur
“You can observe a lot just by watching.” — Yogi Berra
Late in 2006 I was sitting in Rodney Bliss’ office for our weekly investor debriefing. Rodney was the President of the company; as the Director of Technology I reported to him. In theory, Rodney ran the company and I ran the development team. In practice, the investor would change our priorities and expand the scope of the project every week and it was all we could do to keep up. Rodney had just returned from meeting with the investor, and we were going over the new list of features so that we could prioritize dependencies. And then he casually told me of another request that our investor had made.
“He wants me to put together a list of all our expenditures, from payroll to soda pop–”
“Aw, CRAP!” I interrupted.
“You’re going to get fired!”
“What…? No I’m not!” Rodney actually laughed. I just stared at him in horror. I couldn’t believe he couldn’t see it. It was so clear.
“Rodney, I’m not joking. You’ve got 2 months at best; less if he’s smart. Based on the way things have gone this year, I’m guessing right at about six weeks.”
“Dave, calm down. I am not going to get fired.”
Six weeks later, almost to the day, Rodney got called in for his exit interview.
Rippling Grass Is Always Caused By Wind Except When It’s Tigers
The thing was, I knew Rodney was going to get fired, but I didn’t know how I knew. It was a very chaotic time for all of us, and there was no one significant indicator that I could point out, but sitting in that office, I could see all these tiny little details suddenly adding up to a starkly clear pattern. I’d worked for half a dozen clients in as many years, and based on all of my experiences joining and leaving companies, this request from our investor was… odd. Not just odd, but totally out of keeping with his character and with the way he had behaved towards us all that year. All of these things added up instantly in my gut, if not my head, and what was odd suddenly became ominous. Our investor had always respected Rodney’s autonomy, and aside from some ineffable “leadershippy things” that Rodney did, budgetary discretion was the only thing Rodney actually did… at least on paper. And now the investor was taking that function over. I knew–knew–Rodney was going to get fired, just as soon as our investor put all the pieces together.
It took me seven years to figure out how I figured all that out. It was just last weekend, in fact. Rodney and I were arguing good-naturedly–we’ve stayed friends over the years–about a blog post he’d just written, titled And Sometimes You Just Get Lucky. Go read that post; I’ll wait. It’s a lot of fun to read and most importantly, Rodney is wrong and I can’t resist the temptation to point that out. Especially because it’s the kind of wrong where I get to point out how awesome Rodney really is.
Luck Is Not A Factor
For those of you who didn’t go read his post–for shame!–the TL;DR is that Rodney got lucky debugging a computer problem. He sat down and typed a few commands, and noticed something out of the ordinary. To everyone else in the room, it was ordinary–just a very large amount of free disk space. But Rodney was one of the first SWAT engineers at WordPerfect, and had been steeping himself in the engineering lore of the company for years. He just happened to know that
Shell.exe counted up free disk space when it started up. He just happened to know that computers at that time were occasionally experiencing growing pains, where the size of the place in memory for storing numbers was turning out to be too small–like the number of free bytes on disk, in an era where people still carried floppy disks that held less than 2 megabytes, but now you actually could have entire gigabytes(!!!) of disk space lying around–even, in this case, the 2.147 gigabytes needed to overflow a signed 32-bit integer. And lastly, but most definitely not least, he just happened to know the name and phone number of the guy at WordPerfect who wrote
Shell.exe in the first place.
Rodney says he got lucky. Rodney is wrong. He was no more lucky that day than I was at guessing he was about to get fired. A hundred tiny factors, measured in our brains against a hundred past experiences, and suddenly we knew that particular ripple in the grass was a tiger.
Cool Story Bro, But So What?
Well, for starters, Bro, start looking at the grass. Unless you’re down with getting eaten by a tiger. That’s very eco of you, man. I mean, hardcore. I can, like, totally respect that.
No, the thing is, if this isn’t luck, then maybe this is a force we can harness for good. Or at least maybe luck is something we can measure. You know, with science and stuff. Or something. Richard Wiseman set out to do just that, measuring the skills and abilities of people who considered themselves lucky against people who considered themselves unlucky. One of the tests he gave was to ask participants to count the number of images in a newspaper:
"I gave both lucky and unlucky people a newspaper, and asked them to look through it and tell me how many photographs were inside. On average, the unlucky people took about two minutes to count the photographs, whereas the lucky people took just seconds. Why? Because the second page of the newspaper contained the message: "Stop counting. There are 43 photographs in this newspaper." This message took up half of the page and was written in type that was more than 2in high. It was staring everyone straight in the face, but the unlucky people tended to miss it and the lucky people tended to spot it."
You can read Wiseman’s entire article here. It’s fascinating. He even changed the message to say “Stop counting and tell the experimenter you have seen this and win £250.” There was no change in the survey results–except that now the lucky people were going home £250 richer.
I don’t want to steal Wiseman’s thunder, but his bottom line is that lucky people relax more, and tend to be more open to noticing things even while they are focusing on accomplishing a task.
No Seriously, So What?
Dude, seriously. You need to relax. No, I mean literally. That’s the whole point of Wiseman’s article.
I read his article a long time ago–It’s ten years old now–and I’ve been fascinated with the kinds of pattern recognition our brains are constantly doing. I began to notice more and more that I was spotting tigers in the grass in the weirdest of places. I began to notice much more subtle tigers. I began to spot tigers from much further away with less and less data. When I called Rodney’s termination, I knew the investor personally and had a lot of information about him personally. Turns out that was mostly noise; the “tell” is an executive with insufficient personnel knowledge asking for a budget spreadsheet. Earlier this year, a manager at one of my clients commented offhand that he had to prepare a budget breakdown for an executive that nobody on the team had ever heard of. I sighed, and smiled wistfully, and told him that it had been nice working with him. When he asked me what I meant, I shrugged and said “that executive is going to have all of the contractors fired. In my experience, it takes”–waaaiit for iiiit–“about six weeks.” This time I was watching the calendar carefully, and I called it exactly to the day.
Okay, So Spidey Senses Are Neat, But Can You Use Them In Anger?
Specifically, the question I am putting in your mouth here is whether or not you can use this pattern recognition skill actively and aggressively, not just defensively. The answer is most definitely yes.
You can use it to hunt tigers.
A couple years ago I was working at a company that handled some sensitive data. I’m used to playing things fast and loose, by which I mean empowering team members and requiring them to be responsible. Unfortunately, we had a sysadmin who didn’t want anybody touching anything for fear of accidentally exposing our production data. He knew his stuff and I respected him, but our approaches to development were worlds apart when I first came on board, and part of us working out our kinks was having some splendid arguments about silly little things like “being able to do my job” and such. You know, trivia.
I remember we had a problem on our production database server that was just beyond bizarre. We had a fairly intensive query that we had been optimizing. A typical query returning a few hundred records would take 10 seconds, which we optimized and got down to half a second. The worst-case queries would return 10 or 20,000 records and would take 90 seconds or more. We optimized those and got them down to 2 seconds. Things looked really good. Except… well, every thirty minutes or so, the queries would take 30 seconds to run. It didn’t matter if they were worst-case queries, or typical ones, or a mixture; the database would just go away for 30 seconds and all the queries would wait before suddenly coming back all at once.
What I desperately wanted to do was log into the production server and watch it run. I knew our sysadmin was absolutely freaked out by the very thought of this because, well, I asked him, and he absolutely freaked out. So I checked everything I could in the code again. Then double-checked. Then triple-checked. The code was fine. There HAD to be something weird going on on the server.
Our sysadmin wasn’t totally unreasonable, he just absolutely refused to compromise the safety of our data. This is a good trait to have in a sysadmin! But it was early days for us working together, and we hadn’t quite learned how to do it. Work together, I mean. He kept asking me, “Tell me what you want to know about the server, and I’ll tell you.” And I kept responding by holding up my hands and saying “I don’t know. I’ll know it when I see it, but I won’t know it until I see it.”
Months later we would work out how to work together effectively, but at that point in time it literally took the intervention of a Vice President to work things out. The proposed solution was laughably simple. To the sysadmin, he said “do you see any reason Dave can’t watch over your shoulder and tell you what to type? He never touches the keyboard, and you have veto power over him looking into sensitive data.”
The sysadmin grumbled that this would be okay, but that he was sure it would be a total waste of time.
We sat down–well, he sat down. I was literally forbidden to be within arm’s length of the desk. I asked him to start looking around the server. Logfiles, running processes, network I/O, database load. As we were looking at running processes, I noticed that the database was running a reverse DNS lookup on an inbound query. I also knew that, for security purposes, none of our webservers had reverse DNS lookups enabled.
“Hey, how long is the timeout if reverse DNS fails?”
The sysadmin stared at the screen for a long moment. I couldn’t tell if he was embarrassed, or angry, but I could see that we had just seen what we’d come to see, and he was piecing together all of the implications in his head.
Finally, all the implications came together and he said, all at once: “Thirty seconds, there’s the query delay. And when it fails, it caches for 30 minutes, so there’s the problem frequency. And reverse DNS shouldn’t even be enabled on the database. Hang on, let me turn it off.”
Total elapsed time, from the time I walked out of the VP’s office with the sysadmin, to the time I returned to tell him we had found and fixed the bug?
Wow. Can You Ever Use It Defensively And Offensively At The Same Time?
In theory, yes. In practice I’ve only ever seen it done once.
Here’s the theory: if you see that ripple in the grass a long, long way off, but you can’t change your course, you know a tiger attack is coming. Tigers rely on the element of surprise, but you have time to prepare. How much time depends on how far away you saw the ripple. So you try to pick where you’ll make your stand, and you try to bolster your defenses, and when it becomes clear that you can’t avoid the tiger, you… relax and let it happen. As the blur of black and orange streaks up at you from the grass, you can learn a lot about a tiger in a few milliseconds if your life depends on it, and in those milliseconds, you can discover that tigers are often quite surprised by having their ambush turned into a deft counterattack.
That’s the theory, anyway. In practice… well, remember the first story I told, about Rodney? I called the ripple six weeks before the tiger attack. Rodney didn’t believe me at first, but as the weeks went by, other signs began to become clear. I don’t remember what finally convinced him–I think there was a planning meeting on the calendar, and I told Rodney that if I was right, the investor would cancel it, because it made no sense to waste time making plans with somebody you’re going to fire. Sure enough, the investor canceled the meeting at the last minute for “personal reasons”. More importantly, after a year of working with the investor, we knew he was very organized, and always moved appointments instead of canceling them. This time, the investor didn’t set a date to reschedule. When Rodney asked when would be a good time to reschedule, the investor said “let’s talk about that after next week.”
“Next week” was six weeks after I’d said he had about six weeks left.
Rodney went to his weekly meeting with the investor, calm and ready for another normal day of meetings, but also on the lookout for anything out of the ordinary. And there it was: this time there was an extra person in the room–the CFO. Rodney knew from corporate experience that he was there to be a witness… and that meant that this was going to be his exit interview. He sat down, and they sprang what they thought was going to be an ambush.
As the blur of black and orange streaks up at you from the grass, you can learn a lot about a tiger in a few milliseconds if your life depends on it…
I don’t know what happened in that room that day, but Rodney walked out of there smiling, and still very much the President of the company. Oh, they still wanted him gone, but instead of kicking him to the curb as planned, they ended up timidly asking him what terms he would like for them to come to some sort of arrangement regarding that…?
Rodney met the tiger… and then the tiger met Rodney.
But… that part of the story isn’t my story to tell. Which is why I sat him down and made him promise to tell the story on his blog someday soon.
“Dave, calm down. I am not going to get fired.”
Huh. I guess he was right.