Once Upon A Time, A Nonsequitur
“You can observe a lot just by watching.” — Yogi Berra
Late in 2006 I was sitting in Rodney Bliss’ office for our weekly investor debriefing. Rodney was the President of the company; as the Director of Technology I reported to him. In theory, Rodney ran the company and I ran the development team. In practice, the investor would change our priorities and expand the scope of the project every week and it was all we could do to keep up. Rodney had just returned from meeting with the investor, and we were going over the new list of features so that we could prioritize dependencies. And then he casually told me of another request that our investor had made.
“He wants me to put together a list of all our expenditures, from payroll to soda pop–”
“Aw, CRAP!” I interrupted.
“You’re going to get fired!”
“What…? No I’m not!” Rodney actually laughed. I just stared at him in horror. I couldn’t believe he couldn’t see it. It was so clear.
“Rodney, I’m not joking. You’ve got 2 months at best; less if he’s smart. Based on the way things have gone this year, I’m guessing right at about six weeks.”
“Dave, calm down. I am not going to get fired.”
Six weeks later, almost to the day, Rodney got called in for his exit interview.
Rippling Grass Is Always Caused By Wind Except When It’s Tigers
The thing was, I knew Rodney was going to get fired, but I didn’t know how I knew. It was a very chaotic time for all of us, and there was no one significant indicator that I could point out, but sitting in that office, I could see all these tiny little details suddenly adding up to a starkly clear pattern. I’d worked for half a dozen clients in as many years, and based on all of my experiences joining and leaving companies, this request from our investor was… odd. Not just odd, but totally out of keeping with his character and with the way he had behaved towards us all that year. All of these things added up instantly in my gut, if not my head, and what was odd suddenly became ominous. Our investor had always respected Rodney’s autonomy, and aside from some ineffable “leadershippy things” that Rodney did, budgetary discretion was the only thing Rodney actually did… at least on paper. And now the investor was taking that function over. I knew–knew–Rodney was going to get fired, just as soon as our investor put all the pieces together.
It took me seven years to figure out how I figured all that out. It was just last weekend, in fact. Rodney and I were arguing good-naturedly–we’ve stayed friends over the years–about a blog post he’d just written, titled And Sometimes You Just Get Lucky. Go read that post; I’ll wait. It’s a lot of fun to read and most importantly, Rodney is wrong and I can’t resist the temptation to point that out. Especially because it’s the kind of wrong where I get to point out how awesome Rodney really is.
Luck Is Not A Factor
For those of you who didn’t go read his post–for shame!–the TL;DR is that Rodney got lucky debugging a computer problem. He sat down and typed a few commands, and noticed something out of the ordinary. To everyone else in the room, it was ordinary–just a very large amount of free disk space. But Rodney was one of the first SWAT engineers at WordPerfect, and had been steeping himself in the engineering lore of the company for years. He just happened to know that
Shell.exe counted up free disk space when it started up. He just happened to know that computers at that time were occasionally experiencing growing pains, where the size of the place in memory for storing numbers was turning out to be too small–like the number of free bytes on disk, in an era where people still carried floppy disks that held less than 2 megabytes, but now you actually could have entire gigabytes(!!!) of disk space lying around–even, in this case, the 2.147 gigabytes needed to overflow a signed 32-bit integer. And lastly, but most definitely not least, he just happened to know the name and phone number of the guy at WordPerfect who wrote
Shell.exe in the first place.
Rodney says he got lucky. Rodney is wrong. He was no more lucky that day than I was at guessing he was about to get fired. A hundred tiny factors, measured in our brains against a hundred past experiences, and suddenly we knew that particular ripple in the grass was a tiger.
Cool Story Bro, But So What?
Well, for starters, Bro, start looking at the grass. Unless you’re down with getting eaten by a tiger. That’s very eco of you, man. I mean, hardcore. I can, like, totally respect that.
No, the thing is, if this isn’t luck, then maybe this is a force we can harness for good. Or at least maybe luck is something we can measure. You know, with science and stuff. Or something. Richard Wiseman set out to do just that, measuring the skills and abilities of people who considered themselves lucky against people who considered themselves unlucky. One of the tests he gave was to ask participants to count the number of images in a newspaper:
"I gave both lucky and unlucky people a newspaper, and asked them to look through it and tell me how many photographs were inside. On average, the unlucky people took about two minutes to count the photographs, whereas the lucky people took just seconds. Why? Because the second page of the newspaper contained the message: "Stop counting. There are 43 photographs in this newspaper." This message took up half of the page and was written in type that was more than 2in high. It was staring everyone straight in the face, but the unlucky people tended to miss it and the lucky people tended to spot it."
You can read Wiseman’s entire article here. It’s fascinating. He even changed the message to say “Stop counting and tell the experimenter you have seen this and win £250.” There was no change in the survey results–except that now the lucky people were going home £250 richer.
I don’t want to steal Wiseman’s thunder, but his bottom line is that lucky people relax more, and tend to be more open to noticing things even while they are focusing on accomplishing a task.
No Seriously, So What?
Dude, seriously. You need to relax. No, I mean literally. That’s the whole point of Wiseman’s article.
I read his article a long time ago–It’s ten years old now–and I’ve been fascinated with the kinds of pattern recognition our brains are constantly doing. I began to notice more and more that I was spotting tigers in the grass in the weirdest of places. I began to notice much more subtle tigers. I began to spot tigers from much further away with less and less data. When I called Rodney’s termination, I knew the investor personally and had a lot of information about him personally. Turns out that was mostly noise; the “tell” is an executive with insufficient personnel knowledge asking for a budget spreadsheet. Earlier this year, a manager at one of my clients commented offhand that he had to prepare a budget breakdown for an executive that nobody on the team had ever heard of. I sighed, and smiled wistfully, and told him that it had been nice working with him. When he asked me what I meant, I shrugged and said “that executive is going to have all of the contractors fired. In my experience, it takes”–waaaiit for iiiit–“about six weeks.” This time I was watching the calendar carefully, and I called it exactly to the day.
Okay, So Spidey Senses Are Neat, But Can You Use Them In Anger?
Specifically, the question I am putting in your mouth here is whether or not you can use this pattern recognition skill actively and aggressively, not just defensively. The answer is most definitely yes.
You can use it to hunt tigers.
A couple years ago I was working at a company that handled some sensitive data. I’m used to playing things fast and loose, by which I mean empowering team members and requiring them to be responsible. Unfortunately, we had a sysadmin who didn’t want anybody touching anything for fear of accidentally exposing our production data. He knew his stuff and I respected him, but our approaches to development were worlds apart when I first came on board, and part of us working out our kinks was having some splendid arguments about silly little things like “being able to do my job” and such. You know, trivia.
I remember we had a problem on our production database server that was just beyond bizarre. We had a fairly intensive query that we had been optimizing. A typical query returning a few hundred records would take 10 seconds, which we optimized and got down to half a second. The worst-case queries would return 10 or 20,000 records and would take 90 seconds or more. We optimized those and got them down to 2 seconds. Things looked really good. Except… well, every thirty minutes or so, the queries would take 30 seconds to run. It didn’t matter if they were worst-case queries, or typical ones, or a mixture; the database would just go away for 30 seconds and all the queries would wait before suddenly coming back all at once.
What I desperately wanted to do was log into the production server and watch it run. I knew our sysadmin was absolutely freaked out by the very thought of this because, well, I asked him, and he absolutely freaked out. So I checked everything I could in the code again. Then double-checked. Then triple-checked. The code was fine. There HAD to be something weird going on on the server.
Our sysadmin wasn’t totally unreasonable, he just absolutely refused to compromise the safety of our data. This is a good trait to have in a sysadmin! But it was early days for us working together, and we hadn’t quite learned how to do it. Work together, I mean. He kept asking me, “Tell me what you want to know about the server, and I’ll tell you.” And I kept responding by holding up my hands and saying “I don’t know. I’ll know it when I see it, but I won’t know it until I see it.”
Months later we would work out how to work together effectively, but at that point in time it literally took the intervention of a Vice President to work things out. The proposed solution was laughably simple. To the sysadmin, he said “do you see any reason Dave can’t watch over your shoulder and tell you what to type? He never touches the keyboard, and you have veto power over him looking into sensitive data.”
The sysadmin grumbled that this would be okay, but that he was sure it would be a total waste of time.
We sat down–well, he sat down. I was literally forbidden to be within arm’s length of the desk. I asked him to start looking around the server. Logfiles, running processes, network I/O, database load. As we were looking at running processes, I noticed that the database was running a reverse DNS lookup on an inbound query. I also knew that, for security purposes, none of our webservers had reverse DNS lookups enabled.
“Hey, how long is the timeout if reverse DNS fails?”
The sysadmin stared at the screen for a long moment. I couldn’t tell if he was embarrassed, or angry, but I could see that we had just seen what we’d come to see, and he was piecing together all of the implications in his head.
Finally, all the implications came together and he said, all at once: “Thirty seconds, there’s the query delay. And when it fails, it caches for 30 minutes, so there’s the problem frequency. And reverse DNS shouldn’t even be enabled on the database. Hang on, let me turn it off.”
Total elapsed time, from the time I walked out of the VP’s office with the sysadmin, to the time I returned to tell him we had found and fixed the bug?
Wow. Can You Ever Use It Defensively And Offensively At The Same Time?
In theory, yes. In practice I’ve only ever seen it done once.
Here’s the theory: if you see that ripple in the grass a long, long way off, but you can’t change your course, you know a tiger attack is coming. Tigers rely on the element of surprise, but you have time to prepare. How much time depends on how far away you saw the ripple. So you try to pick where you’ll make your stand, and you try to bolster your defenses, and when it becomes clear that you can’t avoid the tiger, you… relax and let it happen. As the blur of black and orange streaks up at you from the grass, you can learn a lot about a tiger in a few milliseconds if your life depends on it, and in those milliseconds, you can discover that tigers are often quite surprised by having their ambush turned into a deft counterattack.
That’s the theory, anyway. In practice… well, remember the first story I told, about Rodney? I called the ripple six weeks before the tiger attack. Rodney didn’t believe me at first, but as the weeks went by, other signs began to become clear. I don’t remember what finally convinced him–I think there was a planning meeting on the calendar, and I told Rodney that if I was right, the investor would cancel it, because it made no sense to waste time making plans with somebody you’re going to fire. Sure enough, the investor canceled the meeting at the last minute for “personal reasons”. More importantly, after a year of working with the investor, we knew he was very organized, and always moved appointments instead of canceling them. This time, the investor didn’t set a date to reschedule. When Rodney asked when would be a good time to reschedule, the investor said “let’s talk about that after next week.”
“Next week” was six weeks after I’d said he had about six weeks left.
Rodney went to his weekly meeting with the investor, calm and ready for another normal day of meetings, but also on the lookout for anything out of the ordinary. And there it was: this time there was an extra person in the room–the CFO. Rodney knew from corporate experience that he was there to be a witness… and that meant that this was going to be his exit interview. He sat down, and they sprang what they thought was going to be an ambush.
As the blur of black and orange streaks up at you from the grass, you can learn a lot about a tiger in a few milliseconds if your life depends on it…
I don’t know what happened in that room that day, but Rodney walked out of there smiling, and still very much the President of the company. Oh, they still wanted him gone, but instead of kicking him to the curb as planned, they ended up timidly asking him what terms he would like for them to come to some sort of arrangement regarding that…?
Rodney met the tiger… and then the tiger met Rodney.
But… that part of the story isn’t my story to tell. Which is why I sat him down and made him promise to tell the story on his blog someday soon.
“Dave, calm down. I am not going to get fired.”
Huh. I guess he was right.
“A hundred tiny factors, measured in our brains against a hundred past experiences, and suddenly we knew…”
I always called this personal skill “Empathy for Entropy”. The ability to take a large set of tiny data and push it upward. Great read and thanks for sharing!
I like that phrase. Sounds nicer than how I sometimes think of it: “being at one with the clutter”. 🙂 Thanks!
This is literally the most badass blog post I’ve ever read.
Heh. There are some WAY more badass posts out there, but thank you. The funny thing is, Rodney is nice, courteous, kind, patient… he’s just about the last person you’d ever expect to have a badass streak in him. But there it is. 🙂
I can’t believe you don’t think I look badass? I no longer drive a minivan! I’ve got 13 kids. I’m active in Church. I was an Eagle Scout and now I’m the Cub Master. .
Oh crap. . .
I really don’t look like a badass.
The story of that meeting where I walked in about to get fired and walked out with a better deal will be the subject of a blog post in November. I know that sounds like a long time, but on my blog http://www.rodneymbliss.com I do a weeklong extended post every two months.
I just finished one a couple weeks ago about having to move my family into a barn after giving away all my kids’ bikes. The next one will cover this experience and it’s scheduled for November.
I’ll let you know when I get ready to post it. (Or y’all could just follow my blog, but you know, no pressure. I wouldn’t pimp my blog on Dave’s blog. . .and. . .I’m out.)
Great stuff. The point I’m taking away here is that learning to trust yourself, to recognize and heed that inner voice, and to recognize patterns is worth as much or more than all the technical skill you bring to the table.
I’d say they’re both essential, and they play off each other. There’s a saying I heard a long time ago that’s stuck with me through the years: “information precedes inspiration”. I don’t know if it’s possible to have those intuitive flashes without a large corpus of technical skills. But I think I take your meaning: it IS possible to have that large corpus of technical skills and NOT have a well-trained conscience, and if you’re in that boat, then by all means, you should redirect your focus extremely heavily in favor of training yourself to listen to your instincts. Thanks!
A lovely blog post about a topic close to my heart. I call it X by sense of smell, where X is the current activity that we’re engaged in. I think debugging is going to be the situation that most people instantly relate to where they might not know exactly what the problem is but they have a general idea where to look and no clue why they know. As mentioned above, we gather this depth of experience and map it to emotions which results in hunches.
Let us continue down this path for a moment and re-evaluate free will.
“Because you didn’t come here to make the choice, you’ve already made it. You’re here to try to understand *why* you made it.”
I believe that the choices we make are actually just lagging indicators of our experiences and we retcon justification after the fact. The concept of well reasoned, rational decisions could be complete bullshit.
I’ve taken to looking inward with this worldview and it has helped immensely. I just had a conversation with one of the co-founders of Test Double (where I work) about why I was spiking some anxiety and the two of us figured out what was going on in my head together and addressed it. Uttering the phrase “yeah, that seems like something that would bother Zach” was incredibly freeing. It eliminated the possibility of me “toughing it out” while I cultivated a martyr syndrome destined to turn to resentment.
This is wildly profound. Thank you. I’m currently reading “Trickster Makes This World”, which opens with this profound and terrifying thought: “Myths are just stories, and stories are just lies, and lies are all we have.” Everything we think we know is really just a story, which is a faulty approximation, which is a lie about the world around us. That doesn’t mean we should throw out the concepts of reason, or science, or skepticism, or logic. The quote says it well: these lies are all we have.
Re: anxiety, I had a similar experience a couple years ago; I’ll have to share it in person (or at least more private) with you sometime. I find it rare and precious and extremely gratitude-worthy to work with people that can help me debug my own head. That’s awesome that your co-founder was able to work with you to ferret that out.
Brilliant! A really interesting and enjoyable post to consider.
Personally I have been struggling with trying to explain this concept to a few people after I realized one day that I had a habit of pre-selecting situations and experiences that will train this skill of noticing patterns [subsconsciously?] and turning it into real, useful information. Unfortunate side effect is that others often think I am performing magic, or “spooky” as one guy called it, which is entirely erroneous from where I am standing. More like a forgotten but cool ability humans have.
The key for me is to allow more information in every day but not to pre-process it, the idea being to store the experience in my memory rather than my ideas about what is important at the time. This allows for more connections in the future, so I do the processing on recall… when a past moment arises in my mind out of the blue but in response to what is happening now, and “a feeling” in the gut that says, pay attention there is something in this. Then by the awesome power of the rational (or irrational) mind, voila! I will just say or think something that turns out to be an entirely accurate prediction. Recently I was able to prove to myself that I had actually observed information that I hadn’t consciously recognised or thought, which prevented me from being hit by a car that was running the traffic lights. Cool.
Very cool! I have given plenty of thought to the “being lucky” side of just being open to what’s going on around us; I hadn’t considered that we might also be front-loading our intuition.
I love the “spooky” bit. I’ve had a few of those moments, and it’s extremely gratifying.
I’ve been talking about this post all day in rl. Hope it brings you some new followers 😉
Pingback: Do You Believe In Luck? | Rodney M Bliss