Spotify and beyond
Join Murray Robinson and Shane Gibson in a conversation about the Spotify Model and matrix organisation structures. Squads, tribes, chapters and guilds. From functional silos to cross-functional product teams supported by chapters and guilds. Scaling with tribes. Turning the traditional hierarchy on its side. From functional factory manager to a servant leader. When it works well and when it goes wrong. Develop your own matrix model.
Shane Gibson
Welcome to the no-nonsense Agile podcast. I'm Shane Gibson.
Murray Robinson
And I'm Murray Robinson.
Shane Gibson
Today, we're going to talk about the exciting thing that is options for scaling,
Murray Robinson
Options for scaling, or going beyond the Spotify model.
Shane Gibson
Let’s start there—Spotify, when they were kind enough to publish how they worked at the time, referred to it as the Spotify model. Then, when the Agile world did terrible things with their article, they stopped publishing all their learnings. Thank you, Agile world, not. But the second thing is they said; It’s not a model. It's a bunch of articles on how they were working at that time. So while we all tend to refer to it as the Spotify model, we should recognise that they didn't believe it was a model and were quite unhappy that the Agile world started calling it that. But everyone calls it the Spotify model because we don't know what else to call it.
Murray Robinson
Management consultants are going to clients and saying, we will introduce the Spotify model. I'm talking about the big three, McKinsey, Bain and BCG. They're rolling it out everywhere, whatever it means. I saw them do it at one place, and it's not the same as Spotify. Also, I implemented a version of the Spotify model at the digital agency I was at, and it worked exceptionally well. So I'd like to talk about what I did and why it worked. Also, what I can see other people doing and get your views on it.
Shane Gibson
So, when I've worked with a couple of customers, we've gotten to that lucky stage where the initial team is rocking it. Now we want to scale the team out, but we want to add some more people. So, we had to scale in some way. We adopted some of the patterns from Spotify for scaling and experimented to see if they worked, and the majority of them did in the scenarios that we tried to use them. They’re valuable patterns. I’m interested to see how you use them compared to how we use them.
Murray Robinson
Most organisations are organised by function. This is what I would call the factory model. The idea is that your functional manager assigns work, and then you go and do it. They also do recruitment, training and quality assurance. You get a lot of delays and waste in this model as you're waiting for things to go from one function to another. In the Agile model, we've changed that model to work in cross-functional product teams. But if you implement Scrum, all you get is cross-functional product teams, scaled up with Scrums of Scrums or LESS. There’s no management, no support, no nothing. I find that a real problem because people do need support. In the digital agency, I moved everybody into long-lasting, cross-functional, client-focused teams. We brought the work to the team. They didn't ramp up for projects and ramp down again. They stayed together and did different types of work. I took the functional managers and put them on the side of the organisation. The Spotify model is a type of matrix organisational structure, and there are many other types of matrix organisational structure. You don't have to do what Spotify did. But it's changing the functional manager’s role from telling people what to do to coaching, mentoring, training, supporting and recruiting. An experienced software developer in a team would coach and mentor the more junior people, and an even more senior person would coach them. The most senior technical people will report to an engineering manager for coaching, mentoring and support. In the Spotify model, your engineers belong to the engineering chapter. That chapter recruits them, and they do training and knowledge sharing with that chapter. It’s people management plus a career path. But the chapter doesn't tell people what tasks to do because the team is self-managing. I found that it worked great because it provided a lot of support to the teams.
Shane Gibson
And the same thing for me was that we took a slightly different approach in one organisation because we couldn’t change the organisational hierarchy at that time. We couldn't introduce the concept of a chapter where the chapter lead is a pastoral care manager, the person in charge of the feeding and watering and healthiness of the people on the team. So we broke those two skills out of the roles. We had somebody that looked after them from an HR point of view, from pastoral care for the people working there. And somebody else that looked at it from a skills point of view. People tend to develop strong skills in data engineering, data visualisation, or facilitation and analysis in the data space. So when we scaled up our data teams, we made sure we had people who were good at each of these things. But we didn't leverage any of the learnings across the squads. So if we had somebody who was a junior in data engineering, we didn't have a way of helping them get more efficient or increase their expertise. If we had a person who was an expert coach in data engineering, then they could help the rest of the data engineers improve their skills. Spotify combines pastoral care and skills capability in one chapter lead role.
Murray Robinson
A chapter lead can’t look after more than ten people, so you need a hierarchy within the chapter if you have more than that. I implemented this by taking the existing functional hierarchy and turning it on its side. I turned the functional managers into servant leaders who didn't tell people what to do. I asked them to have a weekly half-hour one-on-one with people who reported to them in their discipline and provide an apprenticeship for junior people. I also got them to develop solutions for new work across teams. I always made it hands-on, and I expected everybody to be doing client work. Even the most senior engineering manager was expected to spend about half his time doing client work and about half his time doing pastoral care. As you got lower, there was not much pastoral care or sales work, but it's pretty different from having somebody a few levels up still doing hands-on work. So that’s one thing that's very different from the traditional functional model. It's also very different to focus on pastoral care. Operational managers are used to telling people what to do and allocating resources, which is a very different role.
Shane Gibson
I agree. If I see people who rock that role, they move away from saying, “this month I need you to”. Instead, they turn into servant leaders where they are covering both pastoral care and increasing skills. They'll say things like, how are you going? What do we need to do to help you get better at what you're doing? They're asking questions and then going away and making things happen, versus mandating and pointing the direction for the team to go.
Murray Robinson
The other thing they do, which I found very valuable, is to identify common problems across teams and bring them to the attention of people who can resolve them. So when teams get stuck with something, the chapter lead can say, “I've seen this over here, you've got the same problem, and they tried this, let's talk to them, and we'll look at what they did”. Or, I can see that we need to get some more people to help. Or there's this other group that's causing everyone a lot of problems. We need to go and meet with them. It’s constructive. I've seen them take the existing management in big organisations and make them chapter leads and tribe leads. The big consultancies say that you don't need as many managers because you have self-managing teams organising themselves. The results are better because they're not causing problems. So you can slash out a whole pile of managers with the Spotify model.
Shane Gibson
I like the idea that people change from being a manager to a leader. But yes, I can see how that could be used for evil to reduce the number of people in those management roles. On the other hand, I'm not unconvinced that there are always too many middle-level management roles that add no value to an organisation. One of the interesting things you said was that there is effectively a hierarchy within the chapter leads. I haven't got to the stage with a customer yet where we have 20 or 50 cross-functional teams. Maybe when you get to that level of scale, you need to bring a hierarchy into the chapter lead type of role. But I'm not convinced that's the thing to do. To me, it sounds like an anti-pattern. Because what we're saying is, when we look at the way Spotify draws the picture if it's a matrix, the goal of the matrix is not to have any hierarchy within the matrix.
Murray Robinson
It does have a hierarchy if you look at it more closely. The vertical dimension of the model is the product teams. The horizontal dimension is the functions or what they call chapters. Teams are called squads because they decided to try and have sexy names, and a group of squads is called a tribe. The idea is that you have 5 to 10 squads in a tribe. So a tribe is 50 to 150 people that are all focused on a common goal. Typically, a tribe has a common product. The tribe is led by the product owner, an Agile coach, and a technical leader in the Spotify model. So, you don't have a product owner on every team. You have one product owner for one product as you do in LESS and Nexus. That's how Scrum works. It doesn't scale up well like that. But that's the idea. In the Spotify model, you have these three amigos across the tribes. The tribes’ technical leader is the most senior hands-on technical person in one of the squads, and your Agile coach is the most senior hands-on Agile person. The tribe leads also work part-time with individual teams. So, there's a hierarchy there, and if you have multiple tribes, then there's another level above that. And then horizontally, it's a similar thing. It's all a matter of how many people you can look after. We know, for example, that a people leader can't provide pastoral support for more than about ten people.
Shane Gibson
So, if we look at it, it's giving us something based on the Theory of Constraints. When we have more than nine people working together, the lines of communication look like spaghetti, and we become inefficient. That's why Scrum teams and squads are somewhere between two and nine people. So, if we have three squads and each squad has nine people split between data engineering, visualisation, and analysis, there would be nine people in each chapter, which is the most a chapter leads going to be able to manage. So we have a constraint here.
Murray Robinson
I reckon you want to allocate at least an hour per person per week, preferably more. But imagine if you're trying to do a hands-on role and you've also got ten one on one sessions with people in your chapter each week. You couldn't do much more than that.
Shane Gibson
One customer I worked with before Spotify had an interesting approach. The person leading the charge wasn't hands-on; she wasn't an expert in the data space. But she was a bloody good leader, so she was the pastoral care lead for 26 people. She would spend an hour a week with each one figuring out what they needed. Then the rest of the time, she would be making that happen. But that wasn't the chapter lead from a skills point of view; somebody else was the skills lead for the engineering team. That's different to the Spotify diagram, but it's still the same constraint model. We are still asking, “what's the maximum number of people you can work with to do those things?”
Murray Robinson
I prefer to have leaders who do both roles - to provide expert mentoring and career development. I worked for an organisation that employed an HR person to have one-on-one meetings with the offsite consultants to make them feel good and hear their complaints. Unfortunately, it was low value because she couldn't mentor anyone or remove blockers for them.
Shane Gibson
I disagree. I prefer that people with skills focus on building skills, and people who are good at helping people focus on that. I've often seen people who are experts in skills who are not the best people. I've seen some other people who are good at being servant leaders but not technical. They don't get into the dirt. But I take your point that it depends. It depends on the people you have and your organisation. In my view, it's okay to have dedicated pastoral care leads and skill leads in one person who's doing both. You’ve got to pick a way you want to work and then experiment with it and see if it works.
Murray Robinson
When I'm promoting technical people, I don't look for the best technical person; I look for a good technical person who is good at helping others in the team. So I would pick the second-best technical person with good people skills because I want them to do both. For example, when we changed the functional managers to chapter leads, I coached them on how to improve their coaching skills, do one on one, and listen to all of the common problems that were coming up.
Shane Gibson
And I suppose it comes back to some stuff that we talked about in previous podcasts. There are four steps in a person's career. They start as a novice; they become a practitioner, an expert and then a coach. Let's call it the Shagility model. I will begin putting ‘model ’ on the end of everything I do and then publish it. For me, that jump from expert to coach is a big jump because that's where we jump from being the best at those skills to somebody who can teach somebody else how to be better at them. That's what you're talking about.
Murray Robinson
What I'm looking for in chapter leads is somebody who's a good coach. The biggest mistake people make is to take a senior manager who is a terrible coach and make them into a chapter lead because they want to do something with them. That's not a good outcome.
Shane Gibson
When somebody picks up an UnSAfe Agile framework, we tend to see that they make all the managers Senior Product Owners and Chapter leads. They’re not changing the way we work or the way the organisation behaves. We're just shuffling the slippers under different desks for different titles and expecting something else to change, which it never does.
Murray Robinson
I have seen a general manager become a product manager for a team of 100 people, and it worked very well for that team. I've seen people at that level become chapter leads and become very good at it, too. But not everybody's going to make it in that new world. Not every manager is going to be suitable for these roles because it's a pretty different role. A chapter lead is a servant leader and a coach. That's quite different for a general manager who's spent all their time playing politics and hasn't done any real work in their field for a long time. They're not the person for that role because they can't provide the coaching people need in the chapter.
Shane Gibson
I think that we have to be fair to those people. We're changing everything. We're pulling the rug out from under them. We're saying that the hierarchical organisation that we used to have isn't going to happen anymore. The way you get funding, the way work gets done, and the way you get promoted is changing. We need to support those people. The same as when I start working with a new team, there are often people in that team that I don't think will get with this new Agile way of working. I know they're going to find it uncomfortable, and I believe they will opt-out. I'm often surprised that they don't. They love it. Sometimes it's the change they've been looking for. Other people that I think are going to be okay. They're the ones that go, Hey, this isn't for me, I don't like this, I don't want to work this way. We need to put things in place to support them during that change. It's the same with the senior people in the organisation.
Murray Robinson
I agree, and this is where the coaches come in. As Agile coaches, people like us can provide people with that support and help the organisation put the right people into these roles in this new structure.
Shane Gibson
We had a conversation a couple of podcasts ago about what's wrong with managers. We discussed that we put coaching support for teams because we know the changes we're making will be significant. But we often don't put coaching support in for the executives and managers in the organisation. We expect them to somehow adapt without any help. We need that coaching role that helps at every part of the organisation because they're all changing.
Murray Robinson
Agile is a big change. People who think it's not are kidding themselves. Agile and design thinking and other new ways of working are like going on a diet and exercise programme when you’re unfit, unhealthy, and face a heart attack. It can lead to massive improvements. But you've got to be willing to put in the work to stay on the diet and exercise programme. And a lot of managers don't want to make the change. They want to get the badge without having to do the work.
Shane Gibson
There is nothing like a two day Scrum course to be an expert. You talked before about HR people coming in as pastoral care to have a lovely conversation, but they don't help the team when they need help. I’ve seen the same in the change space. We bring in change managers, and I've never worked out what most of them do. A coaching role is vital to help people with change, but the traditional Change Manager doesn't do that. There was no role for change managers in the Spotify model. It was inherent in all the roles. The chapter lead helps their squads and their chapter make the changes they need to.
Murray Robinson
The chapter manages a lot of change. They try different things and learn from each other, which could lead to them changing the roles and structure again. Spotify was continually experimenting and changing. They said that everything they published was at a point in time.
Shane Gibson
I'm pissed off about what the Agile world did with that stuff. Spotify stopped publishing things because it's been abused. Imagine how far they've gone, the experiments they've done, the changes they've made. Imagine the patterns we could adopt because they have value.
So we've got squads, cross-functional, self-organising teams that focus on delivering a product. And squads are in a tribe, which is a bigger group that is focused on providing a standard product outcome. Then we have a matrix going across them, which are chapters focused on skills. Then the last one is the Guild, a community of practice, where all the data engineers from all the squads get together on a semi-regular basis to cross-pollinate in a safe environment.
Murray Robinson
A chapter is a community of practice, but a guild is a special interest group. It comes and goes, depending on what people want to do. Special interest groups cross over chapters and tribes. For example, you might have a special interest group for AI or Crypto when there's no chapter for it. But it's much more informal. People get together to discuss things and share ideas. They don't have defined leaders like chapters.
Shane Gibson
There are no permanent guild leaders. Somebody may say, I'm interested in this, and I'm going to create a guild and have lots of people come along. If they’re interested, we'll keep running it. If I turn up to the guild showcase on my own, then the guilds will stop.
Murray Robinson
Guilds have lunchtime talks and things like that. That's the Spotify model, but we can go beyond the Spotify model. I don't like the language of squads, tribes and chapters. It's something they did to be different. I like calling tribes, product teams and chapters, disciplines, but you can call them whatever you want. I call guilds special interest groups because it's just new names for old things that have been around long before Spotify.
Shane Gibson
First of all, thank you for not using that horrible project manager word when you describe the roles and tribes. That's good. The terms that Spotify used have solidified in the market. It's interesting that when you hear those terms, you get a feeling that they're following the Matrix Model that was published. I have no problem using those terms, but I also have no problem with an organisation creating their own terms, as long as they define the boundaries for a team versus a bigger group.
Murray Robinson
Let's say you've got a tribe with 100 people that look after a product in an organisation that has 5000 people as Spotify does now. How do you divide up your organisation into different tribes? They can't all be externally facing. There's interesting work done on this by the team topologies people. You can take a great product, break it into different parts with different users, and assign a tribe to manage each piece. But then you will inevitably find that many of these tribes need an internal group to provide a platform and standard services for them that all the teams use. If every tribe does their own thing, then there's a lot of duplication. If this happens, you can set up an internal product platform team whose customers and users face the paying customer. And you can have other support teams with specialised skills, such as an AI team that provides a service to the other teams. There's a trap here because if you set up a lot of specialised teams, you go back to functional structure again and get blockages everywhere. You don't want to have too many of those.
Shane Gibson
Those internal product tribes should only be there when the tribe receiving that service thinks it has value. So Spotify said that any squad could use any product they want to support themselves. So they had a bunch of squads that used JIRA and a bunch of squads that used Trello, and a bunch of others that built their tracking systems. They allowed a range of variability and tools. Over time they found that when new squads stood up, they used what the other squads were using because it was a path of least resistance. It made sense to them. They didn't care that much. It delivered what they needed and wasn't something worth reinventing the wheel. Over time, they worked out that if another tribe looked after JIRA for them, that made their lives better. So they opted in for it. That organic creation of support tribes made sense. But there's a real risk that people will look at that model and say that we need to set up a JIRA squad on day one. That's not what it's about. It's about where the value needs to be created. Self-organising teams will generate that value.
Murray Robinson
I agree with you.
Shane Gibson
Instead of somebody's thinking, it should happen.
Murray Robinson
I agree with you. I worked for a large telco where the procurement group was incredibly influential because they were seen as a cost-saving group. They dictated a lot of things to everybody else. You had to go through them. You had to use their vendors no matter how shitty they were. You had to go through this slow bureaucratic process even if they rubber-stamped things 90% of the time. They should have been a service to the product teams, but they set themselves up as a police force and made everybody do what they said. That's precisely the opposite of what you want in a support tribe. You can't allow one of these platform teams to become this police force that orders everybody around. You need to avoid that.
Shane Gibson
Unless there is a need for that police force, the Treasury risk group has to report to the government in a bank. There are a bunch of laws and rules that have to be followed, like anti-money laundering. There are sometimes services or groups within organisations that need to be the police, but it's very rare. It's only in highly regulated organisations. But what tends to happen is that people create fiefdoms and get the power to police things. The other thing is that those tribes that serve tribes sometimes need help to optimise what they're doing. When a large organisation adopts cloud services, it needs to go through a certification and assurance process with security and privacy assessments. There's a bunch of work, especially in the central government, that needs to happen before data is stored in those systems for the first time. The challenge is that the process often takes a long while and costs a lot of money. You will often see a CNA process take two months and costs $50,000 for a Software as a Service product that costs $100 a month. That's not the fault of the people doing the CNA. When organisation data is stored in a new place, they need to ensure that it's safe. Depending on how far in the matrix they’ve got, it would be good if a chapter or a tribe did that work. Does it need to be two months for everything? How do we help make it faster without abusing the product or making the product worse?
Murray Robinson
In big organisations, you might have platform teams supported by other platform teams. That's what the team topologies people talk about. There are a few standard models. First, there’s the customer-facing product team. Then there's the internal product that serves the customer-facing team. Then there's a platform team, and then there's a special skills team. They also talk about leadership as a service. The key idea is that If you're an internal team, you think of the people you're providing a service to as your customers. They get to decide whether they use you or not unless you are a mandatory regulatory team. That's very different. It's common to have an ops team rename themselves as the DevOps tribe and prevent anybody from deploying anything unless it goes through their three-month process. That is entirely the opposite of what we want. I like to think of it as being a value stream. The people closest to your customer inside the organisation are your internal customers. That's the way you need to treat them.
Shane Gibson
Good things like the Spotify stuff and the DevOps processes can still be poorly used by people who want to control everything. One of the reasons the Spotify model became successful was it did cool videos with cartoons. You get the story in the context of what they're doing and the problems they're trying to solve. The other thing was that it looked simple. When we start drawing matrices that are complex and hard to understand, we should make them simpler. Your example is that we can create a platform tribe. Great. But then we could also create a front end platform tribe, a back end platform tribe, a middleware platform tribe and a Docker container platform tribe.
Murray Robinson
That's a bad idea.
Shane Gibson
We can take that simple concept and ruin it with complex hierarchies and control points. If you can't draw it and explain it simply, think about whether you need the picture to be that complex. That's another takeaway.
Murray Robinson
The Spotify model is a way to structure your organisation to help your Agile product teams be more effective. You don't do this by itself. You do it with XP, DevOps, Scrum, Kanban and LESS. The other day, I got into an argument where somebody said, Oh, you shouldn't do this; you should do LESS. My view is, let's do LESS, as well as this. If you're an engineer in a Scrum team, you've only got your team to help you. There's nobody else. In contrast, this model provides you with a whole community of practice and leadership and coaching to support you, which will help you be a lot more effective.
Shane Gibson
When you're a Scrum team, and you start delivering value faster than anybody else, and you’re having fun, then everybody wants to adopt that way of working. The people paying the bills want more of that behaviour, so you have to scale. If you haven't thought about some patterns you might experiment with to scale, you’re in trouble. Think about that early and say that when we scale, we might experiment with a matrix, and the matrix may adopt some of the patterns from Spotify. The other takeaway for me is when organisations like Spotify document and share what they did, can we not all be dicks and do bad things with it? We lose access to this learning when we do bad things with it. We should be careful how we use this good stuff that people were kind enough to share with us.
Murray Robinson
I agree with you, Shane. But interestingly, the people who are implementing and pushing this model most are the big three management consulting firms. There's usually one person in each country who knows what they're doing, and then the rest of them are general consultants who heard about Agile last week or last client. Now they're implementing the Spotify model. This lack of experience and ability to coach the executive team is a real problem. Because it doesn't work straight off, it all needs to be tweaked and customised for your situation. Executives need to be coached and mentored, and trained. I think it's a bad idea to have these general management consultants come in and implement the Spotify model.
Shane Gibson
Well, I suppose one good piece of news, though, is we haven't seen Spotify model 2.0 come out of them yet. So at least that hasn't happened.
Murray Robinson
Well, it's now the ING model, which everyone is talking about. The big consulting firms said they implemented the Spotify model at ING, but ING said they did something different., So anyway, it’s now a significant case study for them.
Shane Gibson
Do they have a cool cartoon video that I can watch that explains it to me in 15 minutes?
Murray Robinson
Fraid not. But you can read all about it in the Harvard Business Review that's much more serious.
Shane Gibson
It was not a cool video.
Murray Robinson
You're not the target audience.
Well, it sounds like we've sucked all the juice out of this one, Shane.
Shane Gibson
Excellent. Well, good to catch up as always, and I'll catch you next time.
Murray Robinson
Keep it real, Shane.
That was the no-nonsense Agile podcast from Murray Robinson n Shane Gibson. If you'd like help with Agile contact Murray at ev0lve.co that evolve with a zero. Thanks for listening.