Jekyll2023-03-20T17:51:49+00:00/feed.xmlNeville Kuyt’s blog.Words, pictures, thoughts, ramblings.Neville Kuyt“Estimating is hard. Maybe impossible. So we must do it.”2023-02-02T15:44:43+00:002023-02-02T15:44:43+00:00/2023/02/02/Estimating<h1 id="why-estimate">Why estimate?</h1>
<p>Software estimating has been a challenge for as long as I remmeber. At the beginning of my career, we used <a href="https://en.wikipedia.org/wiki/COCOMO">CoCoMo</a> and <a href="https://en.wikipedia.org/wiki/Function_point">function point analysis</a> - rigorous, evidence-based methods that nevertheless rarely yielded “accurate” estimates.</p>
<p>Since then, there’s been a lot of thinking/writing, especially in the context of Agile. Some people say <a href="http://www.ines-panker.com/2020/11/04/estimation-is-bs.html">it’s a delusion</a>, or <a href="https://mdalmijn.com/p/11-laws-of-software-estimation-for-complex-work">fraught with challenges</a>. Mike Cohn wrote a <a href="https://www.mountaingoatsoftware.com/presentations/agile-estimating">great book</a> on the topic.</p>
<p>But whether we like it or not, those who pay for the software will want to understand what it will cost, before they release the funds.</p>
<p>That’s the paradox. Most people I’ve worked with agree that estimating is hard or impossible. But in order for the work to go ahead, we need an estimate.</p>
<h1 id="the-cone-of-uncertainty">The cone of uncertainty.</h1>
<p>This problem tends to be biggest right at the beginning of the project/product. There’s an idea or a hypothesis, and some degree of confidence it will be valuable - sometimes described in financial terms, sometimes just as a broad outline.</p>
<p>So, the first question becomes “I have this idea, I think it’s valuable, what would it take to bring it to life?”. Often, the next step in that process is to flesh out the idea - to imagine what the interactions would be, what it might look like, how it would work. And then you have to figure out what it would take to build the idea.</p>
<p>And this is where you run into the <a href="https://en.wikipedia.org/wiki/Cone_of_Uncertainty">cone of uncertainty</a> - early in the lifecycle, there is so much uncertainty that any estimates are likely to be based on missing or inaccurate information. You don’t know what you’re building, who’s building it, what constraints they are working under, you don’t know what’s going to go wrong - or what’s going to go better than expected.</p>
<p>And yet, in order to make progress, you have to allocate resources - people, money, attention.</p>
<h1 id="turning-the-return-on-investment-roi-conversation-upside-down">Turning the Return on Investment (ROI) conversation upside down.</h1>
<p>So, how do you come up with even a broad outline of the cost (in time, people, money or whatever) at this stage?</p>
<p>I like to use a simple framework for evaluating ideas - <a href="https://www.svpg.com/four-big-risks/">Marty Cagan’s 4 attributes of risk</a>.
The idea is that you take each feature making up the bigger idea, at a fairly coarse level, and score them on 4 attributes:</p>
<ol>
<li>#Value# - will this bring value to users? If it’s a commercial product, would they pay for it?</li>
<li>#Usability” - can we make this something users can interact with, ideally with some degree of joy?</li>
<li>#Feasible# - do we know how to build this?</li>
<li>#Viable# - will this align with the organisations goals and constraints?</li>
</ol>
<p>The trick is to find the area where those for attributes overlap - that’s the subset of ideas that have value to users, will be usable, make business sense, and can be delivered.</p>
<p>Here’s an example from a (long-since defunct) start-up I worked for. It was an ecommerce site, selling custom framed prints online. We had thousands of ideas, ranging from obvious “must-haves” like a product catalogue, basket and checkout, to some fairly wild and exciting options like a virtual art gallery with digital sales people.</p>
<p>Here’s how we started figuring out our product strategy. This is an abbreviated form - there were about 20 ideas - and we spent an afternoon working on the evaluation between business, technology and design.</p>
<table>
<thead>
<tr>
<th><strong>Idea</strong></th>
<th><strong>Valuable</strong></th>
<th><strong>Usable</strong></th>
<th><strong>Viable</strong></th>
<th><strong>Feasible</strong></th>
</tr>
</thead>
<tbody>
<tr>
<td>Product catalogue</td>
<td>10</td>
<td>?</td>
<td>10</td>
<td>10</td>
</tr>
<tr>
<td>Visual preview of product</td>
<td>10</td>
<td>8</td>
<td>9</td>
<td>?</td>
</tr>
<tr>
<td>Basket</td>
<td>10</td>
<td>7</td>
<td>10</td>
<td>10</td>
</tr>
<tr>
<td>Checkout</td>
<td>10</td>
<td>?</td>
<td>10</td>
<td>?</td>
</tr>
<tr>
<td>Virtual gallery</td>
<td>?</td>
<td>8</td>
<td>?</td>
<td>?</td>
</tr>
<tr>
<td>Rich product content</td>
<td>8</td>
<td>10</td>
<td>4</td>
<td>6</td>
</tr>
</tbody>
</table>
<p>We could see immediately that there were some ideas with question marks - we simply didn’t have enough information to even guess. Our designer was concerned about the usability of the product catalogue - how could we help customers chose between the hundreds of thousands of products we had? For the “visual preview”, we weren’t sure how to build it - there were several options, but none looked like a great match. Checkout was complex because shipping options were an important consideration, but both the design and tech team needed more information in order to figure out how to make it work. The Virtual Gallery idea had a clear design approach, but our business stakeholders weren’t really sure if customers would value that feature, and the technical feasibility was unclear.</p>
<p>This exercise was quick, and we avoided diving into detail - the goal was to get through the list, so we knew where to focus our attention. We agreed collectively that the Virtual Gallery idea was a “not yet” concept. We agreed that the design and tech folk would work together on the checkout process to investigate options and find a workable solution - the value and viability were obvious. Finally, we agreed that the tech team would look at ways to deliver the “visual preview” function.</p>
<p>For areas where design <em>and</em> technology had given high scores for usable and feasible, we asked both teams to break down the idea to specific features - this ensured we didn’t have big misalignments. Each idea broke down into 6-10 features - so again, these were fairly quick conversations.</p>
<p>At this point - after spending hours, <em>maybe</em> days, but certainly not weeks, you should have a broad outline of the big ideas, and the way they break into features. Can you estimate against this? Well, you can try - and there’s certainly value in the discussion. But the estimate’s accuracy is unlikely to be high.</p>
<p>Instead, I’d rephrase the question. Instead of asking <em>“What will it take to deliver this idea/feature?”</em>, which suggests a degree of accuracy you can’t deliver at this point, you should ask <em>“What level of effort gives us a better than evens chance to deliver this idea/feature”</em>. Why is this better? Because you can answer this question much more quickly - by accepting the 50% error margin, you’re accepting the uncertainty. You avoid the urge to investigate every single edge case, and you can move much faster. When someone asks “What about ….”, you can find out if it makes a material difference to that 50% margin - in most cases it doesn’t, and you can move on.</p>
<p>The output of this process is a list of product ideas and features, with a “better than evens” estimate. For our ecommerce start-up, this process took about a week. But “better than evens” is not good enough - so we multiplied every estimate by a “risk factor”. We were pretty confident we had to deliver <em>all</em> of the features in order for the solution to be viable, so we didn’t want to risk any single feature blowing the budget - so our risk factor was high (we tripled the estimates). Your risk appetite may differ!</p>
<p>At the end, we had enough information to agree on the return on investment for each idea/feature.</p>
<p>We categorized them into 4 groups: “obvious wins” (high value, low cost), “hygiene factors” (high value, high cost, enabled core proposition), “obvious nos” (low value, high cost), and “further investigation”.</p>
<p>This was enough to secure our funding, and start work. We didn’t need to go into complicated work breakdown processes.</p>
<h1 id="estimates-as-non-functional-requirement">Estimates as non-functional requirement.</h1>
<p>So, that’s great - we had a shared understanding of what we were building, and had a (very) rough estimate of the effort required. We started work (lots of other things had to happen, like building a team etc. - that’s a story for another time!), and - as always - found out that our “estimates” were….wrong, even after we’d tripled the “better than evens” estimate. Or rather, we found that when we looked at individual features in detail, we uncovered different amounts of work than fit in the estimates. Some features were <em>much</em> easier than we’d expected, some much harder. The variance had many causes - sometimes, we’d just not understood the complexity. In some cases, the design folk came up with a <em>much</em> better option, that was harder to build.</p>
<p>The solution to this was <em>not</em> to re-do the estimates. It was to treat the estimate as a <a href="https://en.wikipedia.org/wiki/Non-functional_requirement">non-functional requirement</a> in the delivery process. So, rather than asking the team to design the best possible implementation of the idea (in all senses - user experience, visual design, technical design), we asked them to design the best possible implementation that <em>fit within the estimated budget</em>.</p>
<p>This is, of course, difficult - how can a visual designer assess the implementation cost of their work? How can a software engineer stick to a budget in the light of all the uncertainty they face? The good news is that while it’s difficult, it’s <em>much</em> easier than estimating accurately far in advance of having the detail. Generally, it’s easier to estimate small-ish pieces of work (features, epics) than estimating an entire application.</p>
<p>We also had teams work together during the design phase - visual designers, engineers, testers, product folk all collaborating on finding solutions that <em>fit in the budget</em>. This collaboration was incredibly powerful, and as the team built relationships and momentum, they squeezed a lot of functionality out of the time they had available.</p>
<h1 id="handling-the-unexpected">Handling the unexpected.</h1>
<p>Of course, there are cases where you simply cannot fit the work in the estimate - at least not without affecting usability or value. We handle that in the same way as many other non-functional requirements. When the team couldn’t figure it out, we had a review meeting where the team presented the options, and we collectively agreed which option we’d ask them to pursue. Sometimes, the option was “spend more time/money” - but sometimes, it was “reduce functionality”, or even “leave the feature out”.</p>
<p>This is not a magic bullet - but it makes the decisions explicit and transparent, and it takes the responsiblity away from the team.</p>
<h1 id="so-whats-an-estimate-anyway">So, what’s an estimate anyway?</h1>
<p>Our approach was not perfect - in some cases, we really did get it dramatically wrong in the up-front scoping, and even after we’d agreed to change the boundaries, the team took longer than the estimate (and re-estimate). But there were very few surprises, and overall we came in within the budget we’d set.</p>
<p>So, the way I see estimates is not “what will it take to deliver this idea/feature/change?”. My way of looking at estimates is “how much time and resources would give us a reasonable chance of getting this idea/feature/change done?”.</p>
<p>This changes the emphasis from expecting certainty and perfection, and acknowledges you’re trying to assess chances, rather than certainties.</p>Neville KuytEstimating software projects accurately is hard - especially when the project is poorly understood. And yet, it's often unavoidable. Here's a process I've used to break through that deadlock“My boss is terrible - shall I tell _their_ boss?”2022-11-09T15:44:43+00:002022-11-09T15:44:43+00:00/2022/11/09/My-boss-is-terrible<p>I am mentoring a few people, and in one of our conversations, my mentee complained about their line manager.
“He’s terrible - every time something is even slightly off track, he turns it into a crisis. We’re constantly fighting fires - and they’re mostly imaginary!”
My mentee was feeling very stressed - being in constant crisis is exhausting, and having to switch your attention every few days because there’s another fire to put out is terrible for productivity. I sympathized, and then my mentee asked if he should raise this with their boss’s boss - the skip level, or grandboss.</p>
<p>I invited them to follow that train of thought - what will the grandboss think when they hear this feedback?
“Hm. Well, I suppose they’ll want to know what they can do to help?”
That seemed optimistic.
“Don’t you think your grandboss has enough problems? Why would they consider this a priority? Most people have “my problem” and “not my problem” mental categories - why would your grandboss make this <em>their</em> problem?”
My mentee thought about this. “Well, because it’s affecting the team, and I can’t fix it”.
“OK, let’s say that is how it plays out. What’s your grandboss going to do next?”
“Uh - they’ll replace my boss, or get the training or something?”
That again seemed optimistic.
“Maybe, but don’t you think they will talk to your boss first, get their perspective?”.
I saw the lightbulb go on.</p>
<p>“Managing up” is difficult. Sometimes, you end up with a boss you can’t work well with. Early in my career, I had a boss who was incredibly warm and caring, very much focused on personal development - but they were not “technical”, and thus the technical decision making was terrible. This meant lots of rework, lots of integration problems, and a reputation with other teams that we were flakey. When I raised it with my skip level boss - a very smart, career-driven operator who went on to become the CEO - he told me he knew (we had a high level of trust after working together on some challenging deadlines).
“You think I don’t know? Your boss is a great human being, and has had a lot of success leading teams, but in this case, the work is too technical. I’m in regular meetings with you boss, and I’m aware of the problem.”
I looked at my grandboss, waiting for the next sentence.
My grandboss looked back.
I ventured a very convincing “Eh, so…?”
My grandboss sighed. “I’m aware of it. The situation persists. What do you conclude?”.
I thought for a moment. “Either you’re working on it and can’t tell me, or you’re not working on it?”.
“Okay, let’s see why I might not work on it?”
I hated it when he did this. I was like being back at school.
“Uhm…because there’s nothing you can do, or because you don’t think it’s the most important thing to work on right now. Maybe there are constraints I’m not aware of?”.
And so I found out that, yes, my grandboss was aware of the problem, that they had a plan, but couldn’t tell me what it was, and that it would take a few months to play out.</p>
<p>I was very lucky - I had a grandboss who I trusted, and who was able to have this conversation in relative openness. They could easily have gone to my boss and discussed the feedback - and that would have made my position much harder. They could have told me to stop complaining and fix my own problems. They could have asked me if this was just personal animosity, and whether I had any specific examples. They could have insisted it was not a problem at all.</p>
<p>In short, when talking to skip level bosses about your own boss, I’d be <em>very</em> circumspect. Think through the next few steps that will follow your comments, stick to facts, and make sure you have a trust relationship.</p>Neville KuytIs there a good way to complain to skip level bosses about your _own_ boss?Architecture, software, and business2022-08-19T15:44:43+00:002022-08-19T15:44:43+00:00/2022/08/19/Architects%20-%20again<p>I just finished the <a href=""Certified Digital Practitioner" certification course from the Open Group">https://certification.opengroup.org/examinations/dpbok/dpbok-part1</a>. It was pretty good - some of the material is presented as “indisputable fact”, when reasonable people (okay, me) might disagree. And to get the certification, you have to memorize those facts. But passing exams is a thing I’m good at.</p>
<p>The final chapter of the Digital Practitioner’s Book of Knowledge is about architecture. This is described within the context of “enduring enterprise” - large, long-lived organisations with multiple products, complex environments, and challenging communications. In one of the study materials, <a href="https://www.linkedin.com/in/charlestbetz/">Charles Betz</a> describes the work he did as an architect. Very little of it involved boxes and arrows - he writes: <em>“…the architecture team was a mechanism for synchronizing across the organization”</em>.</p>
<p>This came to mind in a conversation with a former colleague, who asked for advice on the career progression for one of their reports. The person in question is a talented developer, with a strong Computer Science background, a proven track record of delivery, and a somewhat abrasive personal style.
“He says he doesn’t want to do politics, or management - he just wants to be able to fix the technical decisions he disagrees with. He’s asked for a promotion to Technical Architect”.
This set off alarm bells. The idea that technical decisions aren’t deeply “political” is hard to reconcile with my experience. The very worst technology decision I recall was at a well-funded start-up, where the CEO “chose” the development platform because he was friends with the vendor CEO. In another case, there were two development teams who couldn’t work together for tiresome political reasons, and we had to design the entire solution to avoid any communication between the two (yes, we may have accidentally invented micro services).</p>
<p>So, we started talking about how architects “get things done”. Even in the most hierarchical organisations, an architect may have positional authority, but that rarely means much. Usually, the delivery teams have more political clout - after all that’s where the business benefit is going to come from. In the quarter century I’ve been doing this, I don’t recall making any big decisions without support from at least some, usually most, hopefully all, the impacted parties. I have a friend who is Chief Architect at a FTSE100 company - and they have never once decreed a solution without the consent of those affected.</p>
<p>Architecture should impact the world - otherwise, we’re just people with opinions. The impact comes not from <em>making</em> the decision, but from seeing that decision <em>carried out</em>. And carrying out the decision requires the participation of more than just the architect(s).</p>
<p>Getting things done, ultimately, means building a consensus, and an environment where people can disagree and commit. That’s hard to do by issuing mandates and decrees. It is, ultimately, political.</p>
<p>So, the developer who wants to be an architect so they can make decisions “without the politics” is in for a rude awakening. Sorry.</p>Neville KuytPositional authority, persuasion, communication - what make an architect effective?The Pyramid Principle2022-06-22T15:44:43+00:002022-06-22T15:44:43+00:00/2022/06/22/Pyramid-principle-redux<p>The Pyramid principle boils down communication into a simple, top-down structure which forces you to structure your thinking, and provides senior folk with a logical flow - and makes sure the most important point stands out.</p>
<h2 id="the-audience">The audience</h2>
<p>The more senior the person, the less likely they have time or attention to spare. If you send them an email, it may linger in their inbox for hours or days. If you give a presentation, they may be messaging on their phone. If they do pay attention, they are almost certainly most interested in the way your message relates to their interests. What do you want them to do? Why should they do it?</p>
<p>So, the Pyramid Principle requires you to start with the action you want them to take. I try to make that the subject of my emails - so, instead of “Update on the XYZ situation”, I aim for “Please approve extra budget for XYZ so we can unlock benefit ABC”.</p>
<p>This sounds easy - but it means you have to do the work to clearly state the recommendation, and that can be difficult.</p>
<h2 id="the-pyramid">The pyramid</h2>
<p>The recommendation, solution, request, whatever, is the top of the pyramid.</p>
<p>You support that with 3 or so mutually exclusive and collectively exhaustive items that make up the recommendation/solution. They typically answer questions like “why”, or “how”. You can keep these brief - a paragraph or two. This is the middle of the pyramid.</p>
<p>The base of the pyramid is the further information supporting the “why”, “what alternatives exist” or “how” questions.</p>
<p>If you’re writing this in a word processor or similar, you can use the Outliner tool to help structure your thoughts - I’ve got into the habit of using this.</p>
<p>As an example, I might do something like this:</p>
<h1 id="please-approve-extra-budget-for-xyz-so-we-can-unlock-benefit-abc">Please approve extra budget for XYZ so we can unlock benefit ABC</h1>
<h2 id="benefit-abc-will-bring-in-additional-revenue">Benefit ABC will bring in additional revenue.</h2>
<h3 id="customers-will-pay-more-for-abc">Customers will pay more for ABC</h3>
<h3 id="abc-will-attract-new-customers">ABC will attract new customers</h3>
<h2 id="the-team-have-a-credible-plan-to-deliver-xyz">The team have a credible plan to deliver XYZ</h2>
<h3 id="we-already-have-an-api-for-some-of-the-features">We already have an API for some of the features</h3>
<h3 id="our-suppliers-can-provide-additional-support">Our suppliers can provide additional support</h3>
<h3 id="weve-got-a-high-confidence-estimate">We’ve got a high-confidence estimate</h3>
<h2 id="it-fits-in-the-roadmap">It fits in the roadmap</h2>
<h3 id="the-product-team-consider-this-capability-strategically-important">The product team consider this capability strategically important</h3>
<h3 id="its-on-the-roadmap-for-next-year-but-we-can-pull-it-forward-with-extra-budget">It’s on the roadmap for next year, but we can pull it forward with extra budget</h3>
<p>By using the various heading styles to outline the argument, I can see whether I’m repeating myself, whether I’m contradicting myself, and where I need to clarify the argument. Doing this before writing the actual content helps me cut down the actual time to write the recommendation.</p>
<p>I got into this habit many years ago - and I’ve found it helps in conversation, as well as in written communication.</p>
<h2 id="situation---complication---question---answer">Situation - Complication - Question - Answer</h2>
<p>The final recommendation from the Pyramid Principle is to start the top of the pyramid with SCQA (situation - complication - question - answer).</p>
<h3 id="situation">Situation</h3>
<p>A brief description of the situation shows you understand the context in which you’re operating, you understand the audience and the world they live in.</p>
<p>Remember how I started this post with:</p>
<blockquote>
<p>You’re an expert in your field - a software engineer, perhaps - and you have lots of insights to share.</p>
</blockquote>
<h3 id="complication">Complication</h3>
<p>But your wouldn’t have a recommendation if everything is hunkey dorey! So, you describe the challenge you face.</p>
<p>At the start of my post, I wrote:</p>
<blockquote>
<p>But you find it difficult to get senior management to act on your ideas.</p>
</blockquote>
<h3 id="question">Question</h3>
<p>To address that challenge, you try to boil it down to a single question that will (help to) remove the complication.</p>
<blockquote>
<p>What can you do to improve?</p>
</blockquote>
<h3 id="answer">Answer</h3>
<p>Once you’ve phrased the question, you can provide an answer.</p>
<blockquote>
<p>You can learn to apply the Pyramid Principle to your communication style.</p>
</blockquote>Neville KuytYou're an expert in your field - a software engineer, perhaps - and you have lots of insights to share. But you find it difficult to get senior management to act on your ideas. What can you do to improve? You can learn to apply the Pyramid Principle to your communication style.Resilience, efficiency, hierarchies and networks2021-09-09T15:44:43+00:002021-09-09T15:44:43+00:00/2021/09/09/resilience-versus-efficiency<p>I was chatting with a friend the other day about the changes we’ve seen over the last few years, and how it doesn’t feel like we’ll go back to the status quo ante.</p>
<p>One of the things we both noticed is that the demand for “simple, clear communication” is at an all time high, and that many leaders seem to want to fill that demand. It’s equally clear that the systems we all depend on are visibly more complex - we’re seeing all sorts of <a href="https://jamesstuber.com/second-order-effects/">second-order effects</a> from pretty much everything. Global pandemic -> fewer driving tests -> shortage of HGV drivers -> shortage of goods in shops.</p>
<h2 id="manufacturing-supply-chains-logistics">Manufacturing, supply chains, logistics.</h2>
<p>The last 200 years - but especially the era since the 1980s - has brought incredible improvements in manufacturing, which in turn has brought down costs for most consumer goods. The most expensive physical purchases most house holds make other than their home (car, TV, white goods) have all either stayed flat (compared to inflation) or declined in price. Cars cost baasically the same as they did in 1990 - though the quality is much higher. TVs cost a fraction of what they cost in 1990.</p>
<p>This improvement has come through technological advancements - but mostly through logistics. Most of these items are no longer “manufactured” in the way we imagine - factory halls, with workers and robots creating items from raw steel, polymers, etc. Instead, they’re assembled from specialist suppliers who offer the best price/quality trade-off. Those specialist suppliers, in turn, have their own supply chains; the combination of global markets, sophisticated logistics and instant information sharing has created supply chains bringing the best, cheapest products together in just-in-time operations which require minimal capital investment in stocks, allow businesses to focus on core competences, and deliver products with great efficiency.</p>
<p>Those advances in logistics allow other businesses to operate more efficiently too - retailers hold far less stock than they used to. A building site near me is bringing bricks - low value, heigh weight - from Belgium (presumably because the price is still better than buying from the UK). My independent corner shop has goods from Poland, Turkey, India, the Phillipines, and South America.</p>
<h2 id="finance">Finance</h2>
<p>The second trend is the change in finance since the Reagan/Thatcher years. Most middle class people now have a pension pot which they invest at their own risk, and - in very broad terms - interest rates have lagged behind inflation. The stock and bond markets have been financialized to the point they barely reflect the real economy, and there’s a <em>lot</em> of money looking for a return.</p>
<p>This has provided finance for start-ups, funded whole new industries, and helped to achieve those efficiencies I described earlier.</p>
<p>But this capital is also desperately looking for safe, guaranteed income streams, and many of the institutions that allow our society to function can provide exactly that. Once you’ve signed up a customer, if you’re a utility, a telco, a transport company, an insurance company - that customer represents a fairly certain future revenue stream. All the low-hanging fruit has been picked though - but there’s more money than ever looking for a return thanks to the monetary easing since 2008.</p>
<p>And once an organisation becomes a financial asset, it has a legal duty to optimize shareholder value. And that means doing nothing that isn’t legally required. The era of deregulation means that this becomes an ever-lower bar.</p>
<h2 id="brexit">Brexit</h2>
<p>Lots of points of view on Brexit, but it’s hard to argue that it imposed barriers on collaboration between UK organisations and those in the European Union. While the barriers are (mostly) not about tariffs, they <em>are</em> about paperwork, which introduces friction. In a process where we’re optimizing the last few pennies out of the bill of materials for a washing machine, that friction can matter…</p>
<h1 id="fragility">Fragility</h1>
<p>So, what have we seen? Mostly, the big systems in the U</p>Neville KuytWe've become incredibly efficient in many areas of life. But that efficiency comes at the cost of resilience. How can we adapt?Software architects and engineers - where do you draw the boundary?2021-08-22T15:44:43+00:002021-08-22T15:44:43+00:00/2021/08/22/Architects,%20engineers%20and%20boundaries<p>I was talking to a friend about how software teams are structured, and specifically what the role is of software architects. It’s a fruitful topic of conversation - because it helps expose assumptions about teams, processes, and technologies. My friend has recently moved from the title of “software architect” to “technical lead”, as well as change jobs.</p>
<p>So, here’s what I’ve seen in the ways organizations use “architect”.</p>
<h2 id="architects-as-super-coders">Architects as super coders.</h2>
<p>The simplest case is that the organization runs out of career track for software engineers, and gives them a cooler-sounding title once they’ve got to the end of that track. In this case, software architects are generally exceptional coders, with deep understanding of the business - but their day job generally remains mostly coding. While this is inelegant, it’s perfectly valid - and I’ve seen it work well as a way of retaining engineers. Of course, the downside is that there is no deliberate attention to the other work an architect might do, or the skills required to do that work.</p>
<h2 id="architects-as-design-authority">Architects as design authority</h2>
<p>The other case is that the software architect takes every major decision on a project - they draw UML diagrams, and the engineers’ role is merely to turn those diagrams into code. One client who operated this way told me they got great productivity from relatively junior developers this way, and that it was a great way to ensure a consistent approach across a complex application. The architect was incredibly smart, but became a bottleneck fairly quickly; they then extended the architecture team, with each architect owning a microservice. That worked well - but once the team exceeded 5 or 6 micro services (and therefore architects), the coordination between the architects became problematic.</p>
<p>More worryingly, it became obvious that while the engineers had been efficient in turning the designs into working code, that code was not always connected with actual use cases - this was an agile team, with regular prioritization and feature pruning, and the architects were often unable to keep up with those changes, so the team built what was designed - but not always what was agreed with the business.</p>
<h3 id="boundaries-of-design">Boundaries of design</h3>
<p>With architects as the design authority, the obvious question is “where does design end? How much detail should architects get into?”.</p>
<p>What I’ve seen work well is architects owning the structure of the major subsystems and interactions between those subsystems. A “subsystem” in this model might equate to a microservice, or a “bounded context” in DDD. Agreeing how to carve the overall solution up into subsystems, how those subsystems should interact, and what the major architectural choices for those subsystems are (development language/framework, deployment architecture, build-and-deploy strategy etc.) definitely requires architectural thinking.</p>
<p>However, within those subsystems, I’d want the delivery team to be the primary decision makers, figuring out detailed implementation decisions as they go. Architects can advise and offer a governance structure - but they quickly become bottlenecks, and - more importantly - deprive the delivery team of agency. The delivery team should own the delivery and implementation choices - this drives up quality, improves maintainability, and</p>
<h2 id="architects-as-the-owner-of-risk">Architects as the owner of risk</h2>
<p>I used to run an architecture team at a digital agency. My approach was shaped by the circumstances - our projects were rarely very large (the largest had around 60 developers for 12 months), and were usually not at the very core of our clients’ organization (unlike SAP, for instance).</p>
<p>My approach was get the architects to <em>reduce risk for the delivery teams</em>.</p>
<p>Sometimes, that risk came from areas traditionally associated with software architecture - a complex application structure, lots of integration points, communication with many technical stakeholders; we drew <em>lots</em> of UML diagrams, had lots of meetings and conference calls, filled endless whiteboards.</p>
<p>Quite often, however, the overall design of the solution was not particularly risky - a web application, using a well-understood framework, with perhaps some SaaS integration points. In those cases, the architect would discuss the approach with the development team, but act as a sounding board and facilitator, rather than the design authority.</p>
<p>Those well-understood web applications, however, would regularly have some attribute which required dedicated risk mitigation. For instance, an ecommerce application with very short, very high load predictions (think selling tickets to one-off events with world-famous singers). Or we’d have to integrate a mobile app with a brand new IoT device, using a protocol that could be described as “under heavy development”. Or a user experience which navigated incredibly complex underlying data structures - but had to hide that complexity from the end user.</p>
<p>In those cases, I’d get the architects to take ownership of the high-risk attribute of the project, and integrate it with the “regular’ development team with the minimum of friction. That meant the teams would be able to make progress on the 80% of the project that was known and understood, and the architect was responsible for the 20% that was risky and confusing.</p>
<h2 id="architects-as-an-investment-in-the-future">Architects as an investment in the future</h2>
<p>The final mental model I like concerns time horizons. Every software project is a trade-off between short-term delivery versus long-term survival. Alastair Cockburn talks about <a href="https://blog.codinghorror.com/software-development-as-a-collaborative-game/">software development as a collaborative game</a>, one of whose goals is to get to play again. It’s not enough to be fast, have lots of features, a cool UI - you have to survive so you can play again.</p>
<p>And this is much easier if you can ask different people to worry about different things. I’ve seen engineering leads spin their wheels for days, and reversing decisions repeatedly, because they had to worry both about “getting the thing out the door” and “make sure we can handle this future, somewhat hypthetical, use case”.</p>
<p>Some of my clients have solved this by asking delivery teams to focus primarily on the short-term delivery challenges, and architects on looking at the future. Concerns for the future usually involve high-level system design and overall implementation quality. But they also involve looking at the product roadmap and figuring out how to deliver key new capabilities. “Our current release only handles one payment method, but in 6 months we need to handle 5 - let’s make sure we don’t have to throw away today’s version”.</p>
<h3 id="architects-and-the-sdlc">Architects and the SDLC</h3>
<p>It’s worth discussing the role of the software development lifecycle (SDLC) in this context. A key factor in software quality and maintainability is the way the team converts requirements into running code. A single developer, pushing code straight to production may be fast, but unit tests, static code analysis, BDD-style acceptance tests, peer reviews etc. will probably improve the quality (and thus reflect an investment in the future).</p>
<p>I’ve seen many architects shy away from these process questions - but I believe they’re a key aspect of “Architects as an investment in the future”. They ensure that the team build the muscle memory to deliver high quality software, but also that the process is tuned to the demands of the project.</p>Neville KuytWhat's the role of an architect in a modern software development project? Here's what I've learnt...Software teams - what does “good” look like?2021-08-22T15:44:43+00:002021-08-22T15:44:43+00:00/2021/08/22/What-does-good-look-like<h1 id="what-does-good-look-like">What does “good” look like?</h1>
<p>A question I hear often from clients, colleagues and friends is “what does a good software delivery team look like?”. This can feel a bit like football fans comparing players in their team with some Platonic ideal - “A decent goalkeeper would have stopped that!” - (or, indeed, their grandmother) - “My nan could have scored that goal!”.
And while most fans would agree the primary purpose of a football team is to win competitions, they often talk about intangible aspects at least as much - “They have spirit”, “They play entertaining football”, “A real community club”.</p>
<p>Similarly, when talking about software teams, the primary purpose is pretty clear - deliver the software the organisation requires. It’s harder to measure than “winning a competition” - is your team more or less efficient than your competitor? Is your quality higher or lower?</p>
<p>And of course, we see <a href="lots">https://blog.crisp.se/wp-content/uploads/2012/11/SpotifyScaling.pdf</a> of <a href="https://www.oreilly.com/library/view/software-engineering-at/9781492082781/">stories</a> about <a href="https://www.theguardian.com/technology/blog/2010/nov/22/facebook-developer-life-inside">software teams</a> in the big tech companies - as well as books like <a href="Accelerate">https://www.goodreads.com/en/book/show/35747076-accelerate</a>, which clearly shows how focusing on 4 metrics allows software teams to deliver value.</p>
<p>But - as ever - it’s not straightforward. “Good” is a trade-off between many different dimensions.
I will discuss some of these below.</p>
<blockquote>
<p>When I talk about a “team” below, I am referring to a “2-pizza” size team of 6-10 people. So, in a team of 100 engineers, I’d expect to have 10-15 teams, and each would hit most if not all of the characteristics below.</p>
</blockquote>
<h1 id="the-dimensions">The dimensions</h1>
<h2 id="throughput">Throughput</h2>
<p>Throughput - the amount of software the team can deliver in a given time - is one of the biggest pain points for software teams. There is no consistent way to measure this dimension - “story points per sprint” is intended to help the team improve the accuracy of estimating, not as an objective evaluation criterion. I’ve seen teams “optimize” this metric simply by estimating more points for a given piece of work.</p>
<h3 id="time-to-value-beats-throughput">Time to value beats throughput</h3>
<p>Often, when I ask business stakeholders about their perception of throughput, they say things like “We prioritized a straightforward change, and it took 6 months to be delivered”. The <a href="https://boards.straightdope.com/t/is-this-the-most-intellectual-joke-in-the-world/99894">pedant</a> in me immediatly jumps to “that’s not throughput, that’s flow”.
The other common complaint is something like “we’ve been working on this big project for ages now, and it feels like it will never be finished”. That’s much more about “time to value” and “alignment” than about the raw throughput of the team.</p>
<blockquote>
<h4 id="good-throughput">Good throughput</h4>
<p>A “good” team delivers business value regularly, and quickly. Typically, this means new features are released every few weeks, and high priority requirements flow to production in weeks, rather than months.</p>
</blockquote>
<h2 id="predictability">Predictability</h2>
<p>A closely related complaint is “we can’t tell when things will be done. Everything seems to take much longer than we expect, and we don’t understand why”.
This comes back to one of the problems with “Agile” - most organisations need some kind of bounds on the cost versus benefit of a project. “We don’t know how long it will take” makes the cost denominator of the return on investment calculation unknowable.
This becomes much more of a challenge when the team is signing up to “large” (> 6 month) commitments, or facing “large” (> 20%) variations in delivery costs.
And as the organisation and its technology becomes more complex, teams become less predictable because they have to navigate dependencies at design, build and run time. “We can’t finish our work until that team finishes theirs” can damage business confidence in the team.</p>
<blockquote>
<h4 id="good-predictability">Good predictability</h4>
<p>Good teams make small promises, isolate themselves from dependencies, and build a steady “release to production” routine.
Nearly every “good” team I’ve seen in action routinely delivers to production several times per month. They tend to be fairly restrained with their delivery commitments, and focus heavily on isolating themselves from dependencies.</p>
</blockquote>
<h2 id="relationships-and-alignment">Relationships and alignment</h2>
<p>Software teams tend to be “different” to the rest of the organisation - even in technology companies. They tend to have a slightly different culture - I worked with a software team in a finance company where the developers wore hoodies, and everyone else wore a suit. They often have different jargon, and different peers. It’s not uncommon for the software team to have career privileges the rest of the organisation doesn’t - salary levels, for instance, may be higher.
It’s also not uncommon for software teams to become a little isolated, or to think that every problem has a software solution. Individuals may appear arrogant, and people may start to focus on the behaviours of the team, rather than the outcomes - in one case, I had a client whose best engineer was put on a disciplinary program because they were routinely 20 minutes late into the office.</p>
<p>Another problem is that development teams may drift away from the overall organisation’s priorities. In one case, I saw a development team in a struggling retail company embark on a large-scale, strategic re-architecture programme, while the rest of the business was cutting costs, and desperately looking for ways to survive. The re-architecture was justified and necessary - but the timing was terrible.</p>
<p>In a positive example, one client had a weekly team update where they linked the previous week’s commercial performance with work done by the teams - “New feature X resulted in a 3% uplift in metric Y”.</p>
<blockquote>
<h4 id="good-alignment">Good alignment</h4>
<p>Good teams maintain a link between the organisation’s priorities and their own roadmap. They can explain how they contribute to the mission, and maintain good personal relationships with key stakeholders.</p>
</blockquote>
<h2 id="quality">Quality</h2>
<p>The next dimension is “quality”. A “good” team delivers software with “good” quality. “Good” is relative - a pacemaker has different quality goals than a casual mobile game. But when business stakeholders complain about quality, it’s often framed as a very specific, long-running complaint. In one case, the client complained that the website “has always been slow”. Everything else was great - but the site performance was a bug bear. The team had worked on it, but not given it enough attention. In another case, the team had let a bug slip through to production which caused a small number of orders to be priced incorrectly - the team had noticed and fixed the defect within hours, but the press had got hold of it, and the business was acutely embarrassed.</p>
<blockquote>
<h4 id="good-quality">Good quality</h4>
<p>Good teams deliver good quality - but they pay special attention to the issues the business stakeholders care about most.</p>
</blockquote>
<h1 id="time---past-present-unknowable-future">Time - past, present, (unknowable) future</h1>
<p>The most important thing about “good” is that you can’t get there in one go, and that it doesn’t stay the same over time. Some of the best teams I have worked with started out…well, not quite so good.
The single most important aspect of “good” is the ability to learn and improve over time.</p>
<p>At one client, there was an “exemplar” team. They’d been established early on, as a way of demonstrating “what good looks like”. The team lead was great - considered, experienced, pragmatic. The engineers were all very experienced, and worked well together. They released their first feature to production after just 6 weeks, and their product area had terrific feedback - and the business stakeholders loved working with them.</p>
<p>After a few months, though, the client said they wanted to “shake up the team”. They’d been doing great work, made many incremental improvements - but the other teams had caught up, and they’d gone from “exemplar” to “middle of the pack”.</p>
<blockquote>
<h4 id="good-teams-keep-improving">Good teams keep improving</h4>
</blockquote>
<p>Sustainable, consistent improvement is hard. The good teams I have worked with have managed it - usually by focusing on making one change at a time, based on empirical feedback, and keeping that routine going over months and years.</p>Neville KuytWhen we look at software teams and the way they work, "good" can be hard to capture. Here's what it looks like for me.Analysis + capital + execution = success2021-08-22T15:44:43+00:002021-08-22T15:44:43+00:00/2021/08/22/Analysis-capital-execution<p><a href="https://en.wikipedia.org/wiki/Succession_(TV_series)">Succession</a> is great - all the characters are horrible, there are some great one-liners, and the dynamics between those horrible characters are fascinating, each proceeding with its own grim logic.</p>
<p>In series 3, they introduce Lucas Mattson, played by <a href="https://g.co/kgs/ArPXHN">Alexander Skarsgård</a>; he’s the founder/CEO of a tech start-up. He’s a coder, and - surprise - not a very likable character. He’s not impressed with the money, status and luxury lifestyle the Roy family take for granted. In the season finale, he says something like “I’m bored with success. Analysis plus capital plus execution - it’s easy”.</p>
<p>I googled that phrase, but didn’t get any hits. So, let’s see what we think about this.</p>
<p>The most successful person I know defines success as “the world behaving in ways that please me”. For them, that means a good lifestyle - nice family, interesting job that leaves plenty of time for other things, enough money for it not to be a problem, nice place to live. For other people, “the world behaving in ways that please me” may focus on money, or status, or power. Importantly, for those people success is often <em>relative</em> - it’s not just that they have power, money or status; they must be the most powerful/influential/rich person.</p>
<p>Whatever your definition of success, it’s important to understand whether you see this as a fixed target - “retiring to an island in the Pacific”, or whether that target is relative - “the wealthiest person in my town”. Relative targets by definition mean you are competing against others, and that you may not retain your success for long.</p>
<p>Undoubtedly, my friend’s <em>analysis</em> of the situation was great - they realized what they needed to be happy, and several credible routes to get there - education, career choices, life decisions etc. I think you can see that with many other successful people - they’ve thought about the world, and created their own definition of success, <em>as well as credible paths to achieving that success</em>.</p>
<p>If you’re competing against others for success, your analysis has more impact than anything else. The richest people in the world (at least those who made their own fortunes) all had a unique perspective. Bill Gates was, apparently, a great coder - but so were the folk who built Lotus 1-2-3, the first web browser, and OS/2. Gates had an vision of the personal computer which bypassed both the immediate opportunities of spreadsheets and the corporate vision of selling lots of desktop boxes. In most areas of human endeavour, the people we know and remember are those who don’t just execute well, but see the world from a new angle.</p>
<p>Capital is the next thing - for most of us, “capital” is “time”. The decision on how we spend our time is the biggest capital allocation decision most of us make; from education to career, from family to exercise. A friend who works for an investment fund, and routinely invests scarily large sums of money tells me that their biggest worry is what they pay attention to - in other words, how they allocate time.</p>
<p>And it’s common to hear “work hard if you want to be successful” - this is certainly true. However, no matter who you are, the day only has 24 hours, and we need down-time to maintain physical and mental health. I think the <em>analysis</em> aspect is more important than the sheer amount of time (or other capital) you apply. However, once you get access to financial capital, the allocation question changes. All of a sudden, you’ll be competing against the smartest people in the world. Do you <em>really</em> think you can compete with Goldman Sachs investing in shares? You need an edge - an unfair advantage. And - again - analysis gives you that.</p>
<p>And then it’s execution. This is where the hard work pays off - both in learning and in delivering. One of the smartest people I know is incapable of actually delivering their ideas. They just lose interest, or cut corners and nothing lives up to its potential. I also know an incredible craftsman - someone who works with their hands, and creates absolutely astonishing work - who feels stuck because they can’t break out of their rut. They lack analysis…</p>
<p>So.</p>
<p>I like the model. Another way of framing it is “do the right thing” (analysis), “do the thing right” (execution) and “get help” (capital).</p>Neville KuytIn "Succession" S03E09, the Matsson character - start-up founder, smart, not a people person - says something like "I'm bored with success. Analysis plus capital plus execution - it's easy". That got me thinking...Resume advice2021-08-22T15:44:43+00:002021-08-22T15:44:43+00:00/2021/08/22/The-perfect-resume<p>Writing a good CV is hard. And you rarely get meaningful feedback - so how are you supposed to improve? Here’s what I tell folk asking for resume advice.</p>
<h3 id="the-recruitment-process">The recruitment process</h3>
<p>Recruiting is hard. Every organisation has its own wrinkles, but it’s usually a set of tradeoffs between the hiring manager, the budget holder, HR, and the recruitment team. They all have different objectives and constraints, and you need to understand how they interact.</p>
<h4 id="the-hiring-manager">The hiring manager</h4>
<p>The hiring manager has a staffing need; they work with the budget holder and HR to figure out how to meet that staffing need, and eventually decide to “go to market”. They write up a job specification (or reuse an existing one), outlining what they need. Often this includes a description of the role, and the requirements for skill, qualification and experience that they believe will identify a good candidate. HR typically review the job spec, and add further information - company overview, benefits, legal requirements, etc.</p>
<h4 id="the-budget-holder">The budget holder</h4>
<p>The hiring manager has to negotiate the budget for both the recruitment process and the ongoing employment costs. The budget holder is balancing this role against all the other demands on the budget; they want to get the “best” candidate that fits in the budget.</p>
<h4 id="hr">HR</h4>
<p>In this context, HR refers to the folk who look after policy and process, rather than the recruitment itself. They want to make sure that the hiring process complies with the law, and with company policies. HR folk generally don’t like “exceptions” - they want to be clear that the candidate, the process and the decisions comply with policy. This can affect compensation, terms and conditions, and the recruitment process.</p>
<h4 id="the-recruiters">The recruiters</h4>
<p>It’s very rare for hiring managers to run the day-to-day recruitment processes - they typically work with recruiters (either internal or external) who place the adverts, screen CVs and often have the first interactions with candidates. Recruiters generally present a short list of candidates to the hiring manager; usually 3-5 candidates make the short list. Recruiters often have financial incentives - their income depends on getting a candidate hired. They rarely have expertise in the technical areas they recruit for, though they will likely have a list of buzz words/technologies they scan for.</p>
<p>While many recruiters are lovely people, the incentives are clear - to earn as much money as possible, they need to get a short list of candidates with a good chance of getting hired through to the hiring manager. They should spend as little time as possible on compiling that short list (so they can work on more than one hire at the same time).</p>
<p>In most cases, the recruiter will look at the job specification with the hiring manager, agree a “sourcing strategy” (how and where they will look for candidates), and provide a short list based on the fit of each candidate with the job spec. For many roles, the recruiter will only talk to the candidate if they’re going to place them on the short list.</p>
<h4 id="how-do-recruiters-compile-the-short-list">How do recruiters compile the short list?</h4>
<p>For most roles, they will look at each CV and compare it with the job specification. If there’s a strong match, they’ll go on the “yes” pile; if there’s no match, the CV will go on the “no” pile. For a partial match, there is sometimes a “maybe” pile.</p>
<p>It’s not unusual for there to be dozens or hundreds of CVs for each role, and that initial screening process will rarely take more than a minute. Time spent reading a CV is less valuable to a recruiter than time spent talking to the customer, or to the two or three people on the short list. If the pile of CVs to be scanned is large, the recruiter is usually looking for a reason to say “no” as quickly as possible - they can always go round again if they really can’t get the short list together. And they don’t have to be particularly rational or compassionate in their approach. Typos and poor grammar can be a sign of poor attention to detail. A layout that makes it hard to read the CV can suggest poor communication skills. The use of Comic Sans may suggest the candidate isn’t professional.</p>
<h1 id="the-role-of-your-cv">The role of your CV</h1>
<p>Your CV has just one purpose: get on the “yes” pile at each stage in the screening process.</p>
<p>The first screen is almost certainly by someone who has a large pile of CVs to evaluate, probably only limited understanding of the business and/or technology domain in which you operate, and is just trying to match your CV against a job specification, so they get a credible short-list and can move onto the next role.</p>
<p>So, make it easy for them to see whether you’re a good fit, in 30 seconds or less.</p>
<p>Here are things you can do:</p>
<ul>
<li>Prominently display the job title you currently hold, or are applying for, and make it easy to see how much relevant experience you have. The first thing a screener looks for is typically experience level - most job specifications will specify this, and it’s a nice objective way to fill up the “no” pile.</li>
<li>Have a short summary, outlining your unique skills and experience; ideally, this should be a match for the job spec, but that may require more effort than you can afford. Read some job specifications (LinkedIn jobs is good for this), and look at the role description - then write a summary that matches this level of detail.</li>
<li>Outline your work history, backing up the unique skills and experience you mention in the summary. Ideally, include some outcomes - not just “I ran the engineering team”, but “I ran the engineering team, improving quality metrics by x% and increasing staff retention by y%”.</li>
<li>In most cases, you want a decent level of detail for the most recent 5-7 years, and can summarize anything older.</li>
<li>Try to keep the language crisp, precise and professional. This is, of course, culture-specific, but look for as many words you can remove from the resume without losing meaning. Consider <em>“Instigated a process for transformational change in our customer service strategy”</em> with <em>“Improved customer service, delivering an increase in revenue of x%”</em>.</li>
</ul>
<p>Here are some things I’d avoid if possible:</p>
<ul>
<li>Non-standard designs and layouts: you want the reader to spend their time on the content, not parsing a new visual language. This may not apply if your role is creatively focused - but otherwise, keep it simple.</li>
<li>Too short and cramped: there’s a lot of advice on keeping your CV to “just one page”. If you can do that - great! But don’t sacrifice readability - 2 pages is fine, as long as the most important content is above the fold on page 1.</li>
<li>Too long: remember, the reader needs to decide in 30 seconds. They won’t get to page 3.</li>
<li>Casual language: a CV is not a text to a mate, and nor is it a contract. You want to get the language just right. Don’t try for humour, and don’t make the reader work through jargon or acronyms.</li>
</ul>
<p>Once you’ve done all this, drafted a version of your new CV, put it away for a day or two, then go through and cut out all the content that doesn’t contribute to getting past that screener.</p>
<h2 id="i-cant-write-a-new-cv-for-every-job-opportunity">I can’t write a new CV for every job opportunity!</h2>
<p>I know, once you start looking, you’ll see lots of opportunities - can you really taylor your CV for each one?</p>
<p>Most people can’t. Depending on your background, you might have 2 or 3 versions, each emphasizing different aspects of your background - a friend has a “startup CTO” CV, a “technology consultant” CV and a “strategy consultant” CV.</p>
<p>So thinking about that summary, and the points you chose to highlight, are important.</p>
<p>If you make it too generic, you’ll end up on the “maybe” pile for lots of jobs, but probably won’t hit the “yes” pile. If you make it too specific, you may not get on the “maybe” pile - your background may be obviously not a good fit.</p>
<p>The best advice I can give you is to find half a dozen job specifications for jobs you want to do, and are qualified for, and to write your summary to be a great match for at least some of them.</p>
<p>So, good luck with this - the job hunt is tiresome. Write a good CV, and help recruiters put you on the “yes” pile, and the next steps are much easier.</p>Neville KuytI have hired hundreds of people in my time - software engineers, testers, managers, directors. Here's what I tell people when they write their CV - or resume, if you prefer!A brief thought on politics2021-07-22T15:44:43+00:002021-07-22T15:44:43+00:00/2021/07/22/the-state-of-the-world<p>Reading some of Isiah Berlin’s essays - he’s a great writer, and a fascinating thinker. Politics, of course, is much more of a narrative art than a science, and any simplification is likely to be wrong - but that doesn’t mean it isn’t helpful. And I keep coming back to this fundamental question - “How much of a society’s needs should be personal responsiblity, and how much should be shared?”.</p>
<p>Individual responsibility (should) bring freedom. If I’m responsible for my personal protection, I should be allowed to exert force on those who threaten me. Berlin, of course, writes eloquently about freedom - but he also points out that freedom is not universally positive, and freedoms may be incompatible. Your freedom to smoke limits my freedom to breathe clean air.</p>
<p>Collective responsibility usually means that individuals pool resources and freedoms. This often means there are winners and losers - childless people’s taxes pay for schools. Rich people’s taxes pay for unemployment benefits.</p>
<p>Since the Victorian age, in the West, the trend moved from “individual” to “collective” for many aspects of life. Schools, healthcare, transport, communications, policing - all became increasingly “collective”. In some cases, that meant the state provided those services; in some it meant the state regulated and created the legal frameworks in which those services could be delivered. Through most of the 20th century, this trend continued. In communist countries, the state took control of most of society’s services, but even in capitalist countries, the state owned and provided many services. It was common for the state to own key industries - in the UK, the state owned steel, chemicals, airlines, road transportation providers, energy extraction, utilities and telcos. The state also played a key role in providing housing, health, education and social care. In most cases, the state both operated those services - the railways were owned and operated by the state.</p>
<p>In addition, the state regulated many aspects of society - financial institutions were severely constrained in the risks they could take. This was a direct example of the collective taking responsibility for an issue; it obviously curtailed freedoms, in advance of a collective goal.</p>
<p>In the 1980s, in the UK and the US, this started to unravel. The great privatisation wave of the 1980s redefined what businesses the state should operate. The great deregulation waves redefined the areas where the state should curtail freedoms. In some ways, this push to liberalization and de-regulation has carried on - in many countries, sexual freedoms have increased, and many social issues are being deregulated (e.g. the legalization of cannabis). These changes have become fairly mainstream throughout Europe and the liberal democracies.</p>
<p>Where there is still no consensus, though, is about which services the state should not just fund, but actively provide. It’s common for transport, utilities and telecoms companies to be privatized to some degree. Education has long been a a mixture - in some countries, the state provides education for some, and funds it for others (often religious groups); in others, there’s a mix of state and privately funded education. Healthcare is a major example - while most Western democracies have some degree of state funding, the NHS is one of the few examples of state-provided healthcare.</p>
<p>There are still some questions about collective responsibility where people widely disagree. Many people believe the state should have only limited responsibility for “the undeserving poor” - asylum seekers, single mothers, layabouts. In the US, many people believe their personal safety depends upon their ownership of a gun. The role of the BBC - a collective provision of information and entertainment - is by no means settled.</p>
<p>So, reasonable people of good will can disagree on these topics. With a few exceptions (racists, mysogenists, homophobes, transphobes etc.), I believe most people agree broadly on what they want for society at a very high level - happy, healthy people leading meaningful lives in a sustainable, pleasant surrounding; freedom from war, disease, poverty and injustice. Where we disagree is how to achieve those goals, or what some of them mean, or indeed what is the right way to trade some of those objectives against each other. How much freedom will we trade for security?</p>
<p>And then we get to the real-politik. Most people have (very) limited attention spans for politics. They make up their minds not on results and facts, or even on policies and manifestos, but on how politicians make them feel. “We all want to be free” is translated into “this group wants to steal your freedom!”. “We all want to live in a sustainable world” becomes “this group is harming your environment”. “We all want to be safe” becomes “this group wants to harm you”. Much of our politics now seems to be about drawing lines between “good” and “bad”, using emotive and simplistic language. There’s an in-group who can do no wrong, and an out-group who can do no right. Anything that happened more than a few months ago is irrelevant or forgotten; anything that might take more than a few weeks to show results is too hard.</p>
<p>And this is all upsetting, but the consequence is that our ability to get things done, to make substantive changes, is declining. Every problem becomes a question of finding blame, of identifying the “good” and “bad” actors. Traffic in London is a nightmare, pollution is harming people, the city is unpleasant - “it’s the fault of the drivers/cyclists/mayor/NIMBYs/TfL/government”. Nobody wants to change, bear any costs, suffer inconvenience. I just about remember my parents getting a small grant to replace the coal-fired home heating system with a gas-fired boiler - but our ability to raise tax has reduced, and the tabloids will pounce on “giveaways” so even those measures are becoming harder.</p>
<p>So, with huge, collective challenges - the pandemic, climate change, various ecological disasters, growing resistance to antibiotics, the growth of disease caused by obesity, the economic disparities in some Western countries, political crises in the Middle East - our ability to get things done seems to be declining. We can “get things done” on a local and national scale, sometimes; different countries have demonstrated that during the pandemic. But large-scale challenges, requiring collective action across many societies? It’s hard to see many successes.</p>Neville KuytReading some of Isiah Berlin’s essays - he’s a great writer, and a fascinating thinker. Politics, of course, is much more of a narrative art than a science, and any simplification is likely to be wrong - but that doesn’t mean it isn’t helpful. And I keep coming back to this fundamental question - “How much of a society’s needs should be personal responsiblity, and how much should be shared?”.