Skip to content

Artificial Intelligence, Big Data, and story-telling apes

I’ve been reading, and focusing on AI. In the fiction area, Gnomon is a complete mind-melt. One of the many premises of the book is that a “system” will run society. If Then takes it a step further, positing a system that runs multi-variate testing on communities to optimize itself.

In the popular science range, the best description of AI I’ve found is <a target=”_blank” href=””>The Master Algorithm: How the Quest for the Ultimate Learning Machine Will Remake Our World</a><img src=”//″ width=”1″ height=”1″ border=”0″ alt=”” style=”border:none !important; margin:0px !important;” /> – an intelligible history of machine learning and AI.

I listened to this You Are Not So Smart podcast on how biases are baked into AI, and what we might do about that.

And most recently, I’ve read this description of an online retailer’s efforts to predict whether a user will be profitable, based on all the attributes that retailer knows. It’s probably the state of the art in this sort of thing – it’s a fairly general problem, there online retailers have lots of data, and getting it right has big financial pay-offs. Instinctively, it feels right – a shop keeper recognizes signs that a customer might spend a lot of money, and knows that treating high value customers better will make them even more valuable. Though they get it wrong – remember the scene in Pretty Woman? A sex worker, acted by Julia Roberts, goes into an expensive boutique, and the shop assistants size her up and discourage her from shopping there; it’s only when the sex worker’s client (Richard Gere) shows up and they recognize his spending power that they treat her like a valued customer.

Pretty woman shop scene

Pretty woman shop scene

The fun bit is to see how hard this problem is, and how the economics work.

The paper describes a few ways the retailer has approached the “how can we predict the value of a customer” question. They’ve created over 130 different attributes, and looked for which ones were most predictive. Those 130 attributes sound like a lot – but many of them are really simple, like “how many orders have you placed”, “how often do you place orders” etc. And with those 13o attributes, they get fairly good results – around 75% accurate.

The second part of the paper describes how they’re trying to include “self-learning” algorithms, that look at all of the data they have, and use it to create a better prediction. The website tracks everything you do – surely, there must be a correlation between looking at certain products and spending more money (there is) – so how do you let the algorithm work that out?

It turns out to be possible, but the cost of training the system to find these behavioural correlations on fast-moving product catalogues is high – and with AI, that cost is often measured in money, not just time.

What does that mean?

It means that the AI would apply a statistical model to Julia Roberts’ web traffic, and unless she matches one of those 130 attributes they use to predict her life time value, the AI would assume she’s “just another customer”. However, if she then browses the expensive, high-fashion dresses category and spends a lot of money, the AI would not learn that customers who look at expensive, high-value customers are likely to spend a lot of money.

Why is this interesting? Because a state-of-the-art machine learning system, with significant commercial incentives, cannot economically learn to identify behaviour triggers without human intervention.

It boils down to “using traditional methods, we can consider perhaps a handful of attributes to predict your lifetime value. Current machine learning allows us to expand that to hundreds of attributes. But to learn from the thousands of subtle clues each of us leaves is currently orders of magnitude too hard”. It’s the difference between categorizing you by crude methods, less crude methods, or treat you like an individual.

And this brings me to the second topic here – human intelligence is very much tied up in story-telling. Sure, humans can manipulate symbols and abstractions, and use formal logic to prove or disprove things. But explaining phenomena in the real world is largely story telling. There’s an example I read somewhere about a finance editor on the TV news explaining the stock market behaviour, and using exactly the same underlying fact to explain opposite trends – something like “positive figures on the labour market lead to higher stock prices as the market anticipated stronger consumer spending”, and “positive figures on the labour market lead to lower stock prices as the market anticipated a rise in interest rates to curb inflation”.

If you read the academic paper on customer lifetime value prediction, the authors do some story telling – “it makes sense that high value customers look at the more recent products, because they are more fashion conscious”. Story telling apes observe things in the world – the rising of the moon, an erupting volcano, a pattern in the data – and we tell stories to explain those things. As we’ve built better models of the world, the stories have become more accurate (and therefore more useful); many stories have become quantifiable and predictable – we no longer believe the sun rises because a deity drags it across the sky on a chariot; instead we can calculate to the second what time the sun will rise thanks to Newton’s formulae.

So what is the point of this long-winded musing?

Whilst ecommerce sites are non-trivial, they are certainly not the most complex system you might imagine when considering the uses of artificial intelligence. And they have relatively clear outcomes, within a meaningful timeframe – you buy something or you don’t, and you usually do it within a fairly short time of visiting the site. And even at this scale, we struggle to identify actions that correlate cleanly with the outcomes using current machine learning techniques. We need story-telling apes to at least identify hypotheses for the A.I. to test.

If you try to expand the scope of AI to “look at human behaviours, and anticipate health outcomes”, or “anticipate criminal behaviour”, or “anticipate political choices” – we’re still a long way off.


Distributed agile development teams

The key to distributed, Agile software development is to get good velocity by making sure the work the developers pick up is “workable”. This means validating requirements before adding them to the backlog.

The last few projects I’ve managed have been larger than most we do in my company. We needed very specific technical skills, and simply couldn’t find them all in London or Amsterdam.  So, we bit the bullet, and brought in developers from suppliers, partners, other offices in the company, and freelancers working remotely. At one stage, the only people “in the office” were the QA lead, the delivery manager, and I running the development team – but all the developers were in different locations. 4 worked for a Polish specialist software shop; 4 were freelancers working from home, 2 worked in a different office, and 2 worked in our Ukranian tech hub. Oh, and our client and product owners were in a different country, and the user interface design team was in a different office.

I was discussing this experience with a friend yesterday – he has a co-located team, with the product owner 2 doors away, but was complaining about lots of suboptimal process issues.

I realized then that running a distributed team forces you to answer process questions early on, because problems that can be dealt with informally if you sit next to each other quickly become intractable if you’re remote. One example was that we agreed between everyone – client, delivery people, developers, QA – that the only source of truth was our task database (Jira, if you must know). If it isn’t in Jira, it doesn’t exist. If you want something, write it in a Jira ticket, and follow the Jira process to get it on the backlog. Developers had to deliver what the Jira ticket specified; QA had to verify that this is what the software did. This way, you have a crude but accurate view of what you have to do – x Jira tickets – and what you have done so far – y Jira tickets. My friend’s project, on the other hand, collected work items in user stories, screen designs, casual clarifications from the product owner, bug reports etc. His developers were complaining they were spending a lot of time picking through all those requests to work out what they actually had to deliver. My distributed team, on the other hand, could focus on ticking off their Jira backlog.

I know, this sounds the antithesis of Agile – I’ll get to that.

The second benefit of this approach was that I could ask the developers to reject any Jira ticket they didn’t believe was “workable”. Requirements could be expressed in whatever way the originator was happiest with – but unless a developer could pick it up and start coding in 10 minutes, they could reject it. Lots of bugs got rejected – if you don’t include steps to reproduce this bug, or actual versus expected behaviour, the developers can’t work on it. Lots of features got rejected for being unclear, or incomplete, or contradictory requirements.

Again – doesn’t sound very “Agile”, does it?

Velocity is the key to Agile.

In my experience, the key to implementing Agile is to build trust with the business that the team will deliver business value, at the expected quality level, for a reasonable cost. And that means establishing a decent level of velocity.

When I work with clients and we agree on “ways of working”, the deal I make is this.

You make sure I’ve got a prioritized list of workable items, and I make sure the team delivers the best possible level of velocity and quality.

Business people like this – they control the priority, and get the best possible value for their development buck. When they see that top priority requests do indeed get turned around in hours or days, and that the team is consistently delivering the priority items in the backlog, they become a lot more confident in Agile as a process.

Chunking is the key to velocity.

We’ve seen numerous studies on developer productivity – but in my experience, the key is to have as much time as possible with clear, complete problems to solve, of a size that allows progress every day. By clear and complete, I mean that a developer should have all the context available when they start work so they have to focus only on the technical design and implementation, not the requirements or UI design.

Most Agile methodologies recognize this – the “unblocking” part of the Scrum Master’s role, and the “I am blocked by” element of the Scrum stand-up, for instance, are designed to allow a developer to hand off tasks that aren’t executable.

But with a distributed team, this is much harder – the communication channels are much narrower, and much slower.

Avoiding blocks is the key to chunking

So, if we want developers to be able to work on executable chunks of work, we need to minimize the number of block they encounter. And in distributed teams, that means we give them complete, executable pieces of work, defined in one or two artefacts – a task, and a visual design, for instance, or a bug report, or a non-functional specification and test report.

However, often “requirements” (in whatever form) are not complete or executable.

Verifying requirements is the key to avoiding blocks

In “traditional” Agile, this is often done mid-sprint. It’s a bit of a pain, but the product owner sits with the team, and it’s their job to clarify. In distributed teams, this is much better done before the sprint starts – get everyone to verify each story before it goes on the sprint backlog, and reject anything that isn’t ready.

Business stakeholders appreciate this process. The clarification process often brings up problems in the product vision, or highlights prioritization mistakes, or shows up organisational issues that shouldn’t be solved by software.

Microservices in the enterprise – breaking out of the IT silo.

Microservices are entering the “early adopter” phase in large, established corporates.  I was talking to a friend who works for a large systems integrator, and he told me that this is now a common topic of discussion with CIOs. His client is a large insurer, and they are migrating the back-office IT systems to a microservices architecture. Interestingly, they are also using this as a lever for organisational change, creating “pillars” around core business areas, rather than departmental “layers”. The entire IT infrastructure is being transformed, with containerization and API gateways as key enablers, and the solution architecture team is now focusing on time-to-market as a key driver – their previous focus was on standards compliance and long-term strategic fit.

It sounded like a very cool project – not just turning around a super tanker, but reconfiguring it to something fundamentally different. So, when I asked which new customer propositions the insurer would deliver, I was confused when my friend said “oh, that’s not really a priority”. The project is driven by IT, and their goal is to be more responsive to the needs of the business, but the microservices solution will not extend to the customer touchpoints – the website, the call centre applications, etc. When I asked why not, the answer was a little complicated, but I think it boiled down to “the teams responsible for those touchpoints had large, complex software packages which they were afraid to change”.

Microservices can have a dramatic effect on back-office solutions – but the true value comes from transforming the way businesses interact with their customers, by opening up new business opportunities or channels. Seeing it purely as a “back-office” solution just means you get better at delivering yesterday’s service, rather than opening up tomorrow’s opportunities.

Compose first, build last.

I was brainstorming a new product with a client the other day. We had all sorts of amazing ideas, ranging from cool user-interface tweaks to (almost) entirely new business models. “Wouldn’t it be cool if…” was probably the most commonly uttered phrase.

And then we came back to ground. Brainstorming is fun – but we wanted to launch a product, and as quickly as possible. So we looked at what we could do by putting together existing services, frameworks and solutions, rather than building things from scratch. We agreed to settle on a common user interface framework, and to use content-as-a-service and commerce-as-a-service platforms; we will run the production site on Amazon Web Services, and we’ll use Gitlab to manage our code repositories and deployment pipelines. We think we can probably limit the amount of custom development to a couple of weeks – and most of that time will go to our secret sauce idea, rather than plumbing and housekeeping tasks.

It made me think of my first start-up, back in 1999 – an ecommerce site selling prints and frames. We hand-built an ecommerce engine, a pick/pack/dispatch module, a label printer for the warehouse, integration with our payment gateway, a basic CMS, and a product catalogue solution.

We had a designer invent our very own navigation metaphor, and our check-out journey was only loosely based on Amazon’s process. We built a custom “tooltip” solution when it became clear not everyone knew how to check out online. I spoke to some of the guys at and learnt about how they used analytics to see what worked and it blew our minds!

We rented a rack at a hosting provider in West London, and I physically wired in our server (we could only afford the one, to begin with). It took about a month to set up our credit card processing facility – the bank didn’t really have much faith in start-up ecommerce companies taking payment online. We built the site with lots of hand-crafted code – no frameworks for us!

So much has changed. We could build that same solution today mostly by composing existing solutions, for a fraction of the cost. The platform that took 5 of us 6 months to build would probably take 2 people a few weeks at most. The capital required would follow a similar trend – from tens of thousands for hosting, servers etc. to hundreds.

For a similar project today, we’d use one of the many ecommerce platforms (I like Solidius, but we’d probably end up with Magento or Shopify), and use all of the established design artefacts – navigation structures, page layouts etc; our user journeys would be like every other ecommerce site (and that would be a good thing!), and we’d host on Amazon or similar. All of our energy would have gone to the unique, distinguishing features and content which defined our proposition.

On the other hand…it seemed so easy to launch back then. We built it, they came. We did some PR, a little SEO, and our site got traffic. I remember the go-live date, and somehow, from somewhere, the first visitor arrived. They left after a short while, but we got a dozen or so orders on the first day. It was like magic! Today, it’s much easier to build things – but much, much harder to get people to pay attention. Let alone visit your site…sure, we’ve got Facebook ads, and Twitter sponsored posts, and Google ads, and the banner ad is still going strong – but you’re competing against every other entity in the world for attention. In recent business modelling exercises, we found that while the cost of “building” propositions (the design and technical aspects) has dropped around 10-fold in the last 15 years, the cost of acquiring customers online has risen by at least 10-fold, and in some areas by much, much more.

Enterprise innovation through escalating bets

A lot of my work involves working with large, established enterprises to find new ways to reach customers. Sometimes, that’s “just” marketing, sometimes it’s product development, and sometimes it’s business model design.

Enterprise innovation is challenging. Most established models for innovation in software build on the concept of “lean” and “iterative/incremental”. The most common point of view is that you get better results by experimenting and integrating feedback than by planning and designing in the absence of customer interaction. For a 3-person start-up, a “minimum viable product” can be very minimal, but for a company with billions of dollars in revenue, and a reputation to protect, “minimum viable product” might be a huge commitment.

The best model I’ve seen is with a client known for their innovation (no names, of course); I’ve called their approach “escalating bets”.

Escalating bets as a portfolio strategy

Our client creates a portfolio of innovation “bets”. Anyone who has an idea they believe in and can meet a fairly low initial investment decision bar can get a limited amount of time (typically 6 – 12 months) and money to prove their idea is a winner. It’s a small bet from an enterprise point of view – many companies spend that much on discussing why an idea shouldn’t go ahead.

Each idea has to have an agreed proof point for the first bet, usually related to whether the idea can attract customers. “We can get 1000 people to sign up”. “We can get at least 10% of our users to spend more than an hour a day on this”. “Our first users will recommend us to at least 1 other person”.

If the bet pays off – if the team hit their goals – the company places a second, larger bet. 9 – 18 months, roughly double the original budget. They have to agree another proof point – typically involving customer acceptance and some kind of business case. And so on, and so forth. I’m pretty sure you’ve used products delivered in this way.

Why do escalating bets work?

Innovation involves risk. Companies focus on risk to reputation, risk to brand, risk to strategic priorities, risk to margin – but the bigger risks in innovation are finding a market, and – very simply – execution. The way most companies approach innovation is to place a small number of large bets – large R&D projects, consulting engagements with specialist companies, large product development teams. As each bet is large, and involves the careers and reputations of several senior people, the bets tend to be fairly conservative – incremental innovations, close to the defendable core of the business.

And if one of those large bets fails, it tends to make the business more conservative. I know of a large IoT innovation project that ran out of steam after 2 or 3 years, and a significant investment. The project was large, with a lot of management attention, and – from my point of view – imploded under its own weight. Different departments wanted to impress their own priorities on the innovation. Decision making was consensus-driven and slow. The large number of internal and external stakeholders and participants made the communication overhead almost unmanageable.

The basic concepts were great, it was the sheer size of the bet that brought the project to its knees. This doesn’t mean “IoT” was a bad bet – as a competitor proved just a few months later.

Escalating bets have two benefits.

The large number of bets means you have a better chance of success – VC companies expect only a small number of their start-up investments to pay off. Running a portfolio of small-ish innovation projects gives you a better chance of finding a winner.

More importantly, though, a small project with a clear goal has a better chance of success in most companies than a large project with a high-level goal. “Take 3 people and find 1000 customers” is clear, not threatening to the other departments in the company, and focuses attention.


Sleep habits.

I found an article online about re-setting your sleep patterns. 

I’ve tried this myself, repeatedly – it’s a great way to re-set. I still struggle with insomnia sometimes, and this routine is what I follow when it all gets out of whack. The thing I’d add is that I make sure I’m properly hydrated before going to sleep – typically by drinking a pint of water half an hour before I go to sleep.

Working with large, distributed teams

My last few projects have involved large-ish (up to 50 or so) teams, spread across multiple locations. I’ve been reading about this, and had the occasional conversation with @seldo about how NPM runs its teams, and I’ve realized a few things.

Firstly – language is crucial. Invest in a common idiom!

My current development team has roughly 30% native English speakers; we have French, Polish, Romanian, Russian, Gujarati and Dutch speakers too. While everyone speaks great English, there is a lot of subtlety in language. Developers use a specific vocabulary when communicating with each other – is it a bug, or a defect? Is it a pull request or a review? What, exactly, is the front-end – is it HTML/CSS/JavaScript, or is it “the website”? Is a task “pending” or “open”? Having a common, shared language makes things easier – especially when agreeing the status of the project. Is work that’s “in review” done, or not done? Do you accept “80% done” as done?

But the business domain is where the real benefits are – making sure the team (and the client) all share the same language is always important (Evans’ Domain Driven Design emphasis this too), but with a distributed team, it’s crucial. I spent about half a day on Slack with a developer in a far-away land trying to work out how to reproduce a bug the client had logged – it turned out all 3 of us had a slightly different interpretation of “product” on the ecommerce site. You can work that out in 5 minutes face-to-face, but at a distance, this can become a huge time-sink.

The second issue is – ridiculously – audio quality. One of our offices has beautiful high ceilings, and stylishly sparse decor in some of the meeting rooms – as a result, the audio quality on audio conferencing is terrible. We’ve got great speakers from Jabra which help – but it’s really worth investing in the best quality audio kit when using Hangouts for a Scrum.

The third issue is a little more subtle. It’s a culture thing – and I’m not sure I always get this balance right. Especially when working remotely, it’s really important that developers get “workable” assignments – a clearly described problem to solve. If it’s a development task, the story/task/whatever should be ready for development – all the dependencies should be clear, the task should have enough requirement detail to allow the developer to work on it. If it’s a bug, there need to be clear steps to reproduce the bug, expected and actual behaviours etc. This is hard to achieve with a remote team – backlog grooming, sprint planning etc. is hard to do remotely, and we tend to use a more centralized model, with a small group of developers and managers assessing and assigning work. Some developers enjoy this – “I just want to code, give me my todo list and don’t bother me”. Others want to be part of the process, and feel ownership of their piece of work. It takes a while to find the balance here – but I’ve learnt to hire “heads-down, coding” people alongside developers who are more engaged in the process.


Customer centric businesses need to be able to iterate. Quickly

I’m seeing the concept of “customer centric” business in more of my work. Focusing on your customer’s experience is obviously a good thing.


I was chatting to a friend whose business is trying to become more customer centric. My friend is the lead for this project, and he’s encountering all the classic challenges – the data they have is in silos, different departments have a different interpretation of “customer centric” behaviour, and short-term sales targets don’t always support long-term customer focus. But his biggest frustration is the lack of velocity. His team is focused on a culture change – focusing the entire business on the customer requires a shift in perspective from the entire organisation. But they also need to deliver tangible change – new business processes, an update to the web platform (that’s how we got chatting), new reports for senior management etc. And this is where he’s stuck – his IT department has a 3 year planning horizon, and is currently scheduling new projects for a start in 2 years. This encourages every project to be huge – if you have to wait years to get your software, you want to make sure you get everything you might possibly need!

This, in turn, has made my friend very nervous about his project – if you get one attempt to change the business process (through a software project), you have to get everything right first time round. So, he’s investing in lots of research, and prototyping, and customer interviews, and KPI development. He sees this as a make-or-break event in his career, and he’s making sure he’s thought of everything before finalizing the software specification; he’s currently expecting the first release in around 3 years.

And that is a problem.

In 3 years, customers will have moved on. Our expectations are no longer shaped by TV, or how we’re treated in a chain restaurant, or the supermarket. Customer expectations are increasingly shaped by web-native experiences. Even though they may be ethically challenged, Amazon, AirBnB and Uber create expectations of  instant, seamless gratification (at a low price). It’s unlikely those experiences will remain frozen for the next 3 years – so designing a process based on today’s expectations is likely to lead to be outdated. And of course, we have no way of knowing what new things may emerge in the meantime – it took many large companies 3 to 5 years to make their websites work on mobile devices.

So, if you want to be “customer centric”, you need to accept that customer expectations are evolving faster than ever – and you need to be able to keep up.

Can an agency be a consultant?

Adweek published an article noting the rise of “consulting services” within agencies. I think it misses the point.

I’ve worked at both “agencies” and “consulting shops” – and I’ve worked on several engagements where agencies and consultancies worked together. I’ve spoken to many friends in agencies and consulting firms, and the fundamental difference is not one of scale, or “competence”, or job title – it’s a fundamentally different view of what matters.

My friends who work for consulting firms believe that “the business” matters – how it’s organized, where the value is created, how the product range fits into the market place, pricing models, how it interacts with suppliers and regulators. They see “the market” as a primarily statistical entity, with “segments” and “channels” and believe that marketing and advertising uses a magical process called “creativity” to convince those market segments to buy the goods and services.

Some of my agency friends believe that it’s all about “the brand” – how consumers perceive it, how to tell stories about the brand, how to reach new people who might love the brand, how to measure the brand’s impact. Many of my agency friends go further, thinking about how “the brand” contributes to sales – how do we convert love for the brand into sales, how do we measure that contribution, where can we open up new opportunities for people to interact and buy? They see the rest of the business primarily as a black box which provides money for marketing, and products or services to sell to the consumer, with a magical process called “operations” which somehow delivers all this stuff.

Okay, okay, I’m exaggerating. But not much.

This dichotomy – separating customers from “the business” – has always been a bit problematic, but the Internet is breaking down those walls ever faster. Advertising – taking attention from consumers and using it to push your message – is ever more difficult as media is fragmenting, consumers have more choice in both the information they can access and the products and services they can buy. Today, “brand” is about how you treat your customers, not about your logo, or your strap line, or your colour palette.

But a business that focuses purely on operational efficiency is doomed – you have to innovate, and find new markets, or you will have to compete on price at best, or become irrelevant at best. While there is obvious value in improving operational capabilities, real innovation combines operational capabilities with customer needs.

And that brings me back to the Adweek article.

Where agencies can add new value is not in “marketing consulting” – we have been doing that for ages. Instead, agencies can bring that deep understanding of how to create a link with consumers – a story, a delightful user experience, a slick and natural-feeling technical implementation – to the operational improvements you can get from Accenture of McKinsey.

Offices suck.

About 5 years ago, people realized that they had better IT from Google, Microsoft and LinkedIn than they got from their own IT. Their home devices were much nicer to use, and much easier to live with, than their work laptop. They got free, unlimited email from Google, with amazing search while their IT department limited their mailbox.


This caused a bit of a corporate revolution, commonly known as “BYOD” or “bring your own device”. My girlfriend works for a local council (hardly a trailblazer), and doesn’t get a work laptop – instead, she gets a budget and some limits, and goes and buys her own laptop.


The same will happen with IoT.


At home I can tell Alexa to turn up the heat, or dim the lights, or play some music. I can see what the weather’s going to be when I leave home by looking at my alarm clock, and I know that the heating will come on when people are home, and stop when they leave.


At work, the HVAC works on a timer. The lights are on or off (mostly on) because I switch them on or off. If I’m cold, I put on a sweater. If I’m hot – well, HR have asked me not to disrobe anymore. If my colleagues decide to play pumping dance music, I have the option of wearing headphones, suffering, or physical violence.

My home is a more productive environment than the office when I have to get stuff done.