Cultivating an Experimental Mindset in Your Organization

Interactive Transcript

spk_0 Welcome to HBR on Leadership. Case studies and conversations with the world's top business

spk_0 and management experts hand selected to help you unlock the best in those around you. I'm

spk_0 HBR Senior Editor and Producer Amanda Kursi. As a leader, you face uncertainty all the time.

spk_0 Experiments offer a way to test assumptions, but it's not enough to simply run them.

spk_0 Their value comes from designing them carefully and being willing to act on what they reveal,

spk_0 even when the findings up and your expectations. Here's HBR idea cast host Kurt Nickish with a

spk_0 conversation from 2020 about what leaders need to know to design rigorous experiments and then put

spk_0 the evidence to work. In science, the need for experimentation is cut and dry. You come up with a

spk_0 hypothesis whether it's about how storm clouds move or how cells in the body die and you set up an

spk_0 experiment to test it. There's a method, it's called the scientific method and you test it over

spk_0 and over again until you're sure that it's replicable and your answers are right or at least as

spk_0 right as they can be until new variables come to light or the landscape changes. In business,

spk_0 there isn't currently as much experimentation. Value has been placed on experience on the intuition

spk_0 of managers and leaders and that's a bad thing says today's guest. Even in the most innovative

spk_0 industries we can think of, more can be done to set up experiments, test the results and deliver

spk_0 better products and services to customers and this goes far beyond AB testing at tech giants.

spk_0 Our guest today is Stefan Tomkitt. He's a professor at Harvard Business School. He's the author of

spk_0 the book Experimentation Works, the surprising power of business experiments and he also wrote the

spk_0 HBR article building a culture of experimentation. Stefan, thanks for coming in. Thanks for having me.

spk_0 Just to start, pretend I'm a business leader. Make the case for me. Why do we need to experiment more

spk_0 in business? Well, first of all, it can generate a tremendous amount of value. Let me give you an

spk_0 example. Microsoft's Bing, which is its search engine. Sure. An employee working sort of at Bing

spk_0 came up with an idea on how to sort of display ads. The manager didn't think much of it

spk_0 and they kind of shelved it. But the employee insisted. At some point the employee decided just

spk_0 to launch an experiment to run a test, a control test. When you run the test, that little change,

spk_0 a few days of work generated more than $100 million of additional revenue and that year alone.

spk_0 Of course, more revenue going forward is, in fact, it was the most successful experiment that

spk_0 was run at Bing. What made the difference? Well, the difference was that the employee had the power,

spk_0 essentially, or the authority to run the experiment, to launch it, and to test it. It's the test

spk_0 that actually told you what works and doesn't work and not the manager. And not the manager.

spk_0 The problem is in a lot of innovation, especially when you're trying to predict customer behavior,

spk_0 we get it wrong most of the time. And so rather than trying to follow our intuition or our opinions,

spk_0 why not just run the test and let the test tell us what works and doesn't work?

spk_0 And what's the answer to that? Why aren't people doing it?

spk_0 Well, there are lots of reasons why not people are doing it at scale, especially.

spk_0 Some people are running simple experiments because they refer to an experiment as something like

spk_0 a trial. We're trying something. That's not really an experiment in the scientific sense.

spk_0 They don't do many of those because they either don't have the infrastructure to run many tests.

spk_0 They may not have the tools to do so, and maybe too expensive to run it.

spk_0 And then they may decide that, listen, we run a test and we get some results, and then nobody listens

spk_0 to us anyway. Right. Do managers overestimate the downside to experiments and underestimate the upside?

spk_0 I think sometimes they are too overly concerned about the risk of running the experiment.

spk_0 For good reasons, you have a lot of traffic. You may not want to launch something that results in

spk_0 a loss of customers visiting your websites, for example, if it goes down. If it goes down.

spk_0 And so if you don't have good stoppage rule, kill switches and things like that sort of in place.

spk_0 There may be a risk of erosion. It's also stepping into the unknown. And quite honestly,

spk_0 it takes humility to admit that I just don't know. And I'm walking into a meeting and we're

spk_0 launching this saying, and everybody has some hypothesis about what the outcome is going to look like.

spk_0 And just go into the meeting and tell everybody, listen, honestly, I don't know what's going to

spk_0 happen. So let's just find out. Even though I get paid more. We get paid. I mean, charge. Yes.

spk_0 I don't know either. Exactly. And the higher you go, the more you get paid. Yeah.

spk_0 The more senior you get, you know, you get paid to make tough decisions. And you want to be

spk_0 decision maker. And you're like a great sort of an organization that ticks a little differently,

spk_0 sort of, to do the sort of thing. By the way, I mean, it's not just the online world. It's also

spk_0 the physical world, you know, where companies are running experiments. And even there, you know,

spk_0 we have to make big decisions, sometimes very expensive decisions. And it's the experiments that

spk_0 can, in fact, adjudicate, you know, whether we want to do something or not. Calls, you know,

spk_0 big retailer and so forth. So calls hire a consulting company and the consulting company,

spk_0 basically it does a cost analysis. And they go to senior management and tell them, listen,

spk_0 we figured out that you can save a lot of money if you open your stores an hour later.

spk_0 Now here you are. You're running this company and you have to make a decision.

spk_0 Should we do that? Calculating the cost savings is easy. But the big question is, what's actually

spk_0 going to happen to our revenue? You know, our customers going to buy less if we open an hour

spk_0 later. So how do you make these kinds of decisions? You know, we can analyze and analyze, but we

spk_0 won't know until we actually do it until we run the test. And in this case, they did. And so they

spk_0 ran controlled experiments in which they sort of set up these tests, opening an hour later. And

spk_0 lo and behold, at the end, you know, the result was that it made much difference. So just so we're

spk_0 on the same page, like how do you go about setting up an experiment? Are there playbooks for this?

spk_0 Well, first of all, there are tools. A lot of companies that describe in the book,

spk_0 built their own infrastructure, built their own tools because when they got started,

spk_0 many years ago, the tools ran around. So you look at Amazon, a Microsoft, a Netflix, a booking.com,

spk_0 I mean, you go through them. And it was about a dozen or so. They decided to do it themselves.

spk_0 So they just, they knew that they had questions they wanted to answer and they just figured out

spk_0 a way to do that. They figured this is going to give them a competitive advantage. You know,

spk_0 if they can kind of go out and just test a lot. And they knew that they often get it wrong.

spk_0 And so so they started investing in infrastructure. And so at a place like Microsoft, for example,

spk_0 you have a very, very large group that basically runs the infrastructure. You know, something like the

spk_0 last time I checked it was something like 85, 90 people or so that are just sort of doing infrastructure.

spk_0 But the good thing that happened a few years ago is there are now third party tools as well

spk_0 that can do this both in the online spaces and in the brick and mortar spaces, which do sort of a

spk_0 lot of the heavy lifting for you, a lot of the statistical stuff and so forth. And so it's

spk_0 gotten a lot easier than say if you wanted to start the five or 10 years ago, developing a culture

spk_0 for this is probably a little bit different. I think it may be potentially harder than getting

spk_0 the tools and building the tools. Yeah. Because now we're dealing with behaviors, with beliefs,

spk_0 with norms and all sorts of things. How does this show up in companies if the culture for

spk_0 experimentation is not working? What do you what do you actually see and observe? Well, the classical

spk_0 example is they start running experiments. We have an experiments. We hand over the result to the

spk_0 group that asks to run the experiment and then nothing happens. Or they will start to challenge

spk_0 the experiment. Something must have gone wrong. I remember a story where you know an angry person

spk_0 actually called sort of one of the one of the tool vendors sort of in this space and complained

spk_0 about the tool being wrong. The person ran an experiment that actually showed if you give

spk_0 customers less choice in his setting, you get better performance. And that was kind of just counter

spk_0 intuitive because everything that he believed in up to this point is that you should give people more

spk_0 choices. And so he was really disturbed by the finding. And so he called them and complained that

spk_0 there's a flaw in the tool. Something the tool must be wrong because the result doesn't match

spk_0 sort of the experience that he's had and he's been doing this for a long time. And so you run it

spk_0 to that sort of thing which kind of underlines your point that experiments bring new insights that

spk_0 you just can't develop on your own. Correct. There's a company called Booking.com which most of us use.

spk_0 In fact, it's the biggest accommodations platform in the world. More than 1.5 million

spk_0 room nights are booked on the platform each day. It's a two-sided platform. This is what we call.

spk_0 It's got suppliers on one side which are hotel operators for example. And of course, it's got

spk_0 customers like us on the other side. And Booking.com runs a massive number of experiments.

spk_0 My estimates are and I'm probably on the low side they told me it's my estimates. It's over 30,000

spk_0 a year of experiments. And it's a really really fascinating company. It's also a highly successful

spk_0 company. Their gross profits are in the high 90s percent. And they don't really have any assets.

spk_0 They don't really own any accommodation. So it's a super competitive industry too. And so how do

spk_0 they get away with this? And the answer to this is there are not a lot of experiments. And they

spk_0 created an experimentation culture where almost running experiments is like breathing. You kind of

spk_0 do it every single day. I mean, you have to think about the numbers here. Even if I'm running

spk_0 a lower number of experiments, I mean, they're running more than 100 new experiments a day.

spk_0 You have to have an organization that can even come up with so many hypotheses.

spk_0 I mean, you mentioned the number of transactions that Booking.com does in a day.

spk_0 How key is that to being able to run experiments? Does that also work for places that just don't

spk_0 have data like that? Yes, it works for places that also have a lot less traffic. The underlying

spk_0 math changes, you know, sort of what you have to do algorithmically is very different. In fact,

spk_0 if you have very large sample sizes, you know, a lot of traffic, for example, you can really find

spk_0 one. You can sort of do very, very small changes and you can kind of pick up whether that change

spk_0 actually causes something to happen. As your sample size shrinks, you know, you're going to have to go

spk_0 for bigger changes. We call it the power of an experiment. You have to power an experiment,

spk_0 statistical power. And so I recommend for companies that are sort of smaller that maybe they kind of

spk_0 run experiments that are a little bigger. Now, what happens also, and this is something that

spk_0 actually happened at IBM, when they started to do this, they realized that they have

spk_0 way too many websites. So yes, they had very little traffic on some of these websites,

spk_0 but they didn't need all the websites. So it actually led to a process of consolidation. I said,

spk_0 listen, we don't really need all these things. So what we'll do is we'll consolidate and we get sort

spk_0 of more traffic on fewer websites. It's which that allows us to sort of run more experiments.

spk_0 I wondered if there are companies or industries outside of consumer facing tech or outside of,

spk_0 you know, scientific or our pharmaceutical companies where experimentation really feels for it.

spk_0 Well, I mean, the classical companies, I think, are sort of in the creative industries where

spk_0 the assumption is that everything is driven by creatives. Look at entertainment, for example.

spk_0 And look at what Netflix has done. So Netflix kind of flipped it around and, you know,

spk_0 they operate in the creative industry, but they're completely experimentation driven.

spk_0 Right. And I think it was a big wake up call for the entertainment industry, because, you know,

spk_0 when you go in and you run Netflix, you are part of their ecosystem, their experimentation ecosystem,

spk_0 they run a massive number of tests, because they want to find out what works and doesn't work. By the way,

spk_0 running the test and getting a result doesn't mean that you have to blindly follow what the

spk_0 result is, because sometimes they're good strategic reasons why you may not want to implement what

spk_0 the test tells you. Right. Or they're trade-offs to whatever benefit of. Yeah. For example, or maybe,

spk_0 you know, there may be a contractual violation or something like that. But what the test does,

spk_0 it actually adds transparency to the decision. So you cannot pretend that we're doing this because

spk_0 it's good for the customer or something like the or good for the viewer. It adds clarity that we

spk_0 understand from the test what's good for the viewer, but there may be other reasons why we may not

spk_0 want to do it. And adding that transparency to what you're doing, I think, is sort of the big value.

spk_0 And it allows a company like Netflix to operate really in the creative industry with a testing approach.

spk_0 Yeah. I don't want to diminish the value of creative talent, because creative talent is really

spk_0 important, but that doesn't create certainty in terms of decision making.

spk_0 To me, the creative talent and the intuition is an important part of experimentation,

spk_0 because it allows us to create hypotheses. You have to ask yourself, Kurt, where do these hypotheses come

spk_0 from? Yes. They're from people. Some people asking questions are having ideas. Yeah.

spk_0 Absolutely. When, so when I'm saying is they're running all these experiments, they're all

spk_0 hypotheses that came out of product groups and it's the people who come up with these hypotheses and

spk_0 so whether they get the idea as well, it's intuition. Sometimes it's inside surprising,

spk_0 customer surprises, things that thought that were true and then they observe something that

spk_0 doesn't quite fit sort of what they know. It's usability labs, so there's still, I mean,

spk_0 these companies all run qualitative research. But they do all the kinds of things that other

spk_0 companies do, but they do it for generating hypotheses, which are then rigorously tested,

spk_0 versus other organizations that generate the hypotheses and go directly from hypotheses to

spk_0 launch. Based on whoever is the best product speaker or makes the best case in a meeting rather

spk_0 than. Yeah. Yeah. Yeah. There's a word for that in the community called Hippos. Hippos. Yes.

spk_0 Highest paid persons opinion. Yeah. Hippos. And we all know that Hippos are very dangerous

spk_0 animals. I think a lot of executives are probably also not used to knowing how much experimentation

spk_0 to do. How do you know what to experiment on and how do you know what to let be? Yes. You have to

spk_0 empower people to make that decision. And the reality is right now, I think most organizations

spk_0 test to a little. So, you know, I don't think you should be too worried about testing too much.

spk_0 Okay. Yes. There's probably a point in which you test too much because you need an organization

spk_0 that can absorb all that knowledge or all sort of that, all those findings that are generated by

spk_0 all these tests. That's true. And we need to think about that, but I don't think that's the problem

spk_0 in most organizations right now, right now they're not doing enough. If you're bringing this into

spk_0 a company, do you try to do this company? Why? Do you try to start with a team or a division and

spk_0 scale it up from there? So, the different ways to organize your experimentation teams,

spk_0 the three models that I describe in the book, you know, one model is really more centralized

spk_0 approach. I basically have like a center, a group that's responsible for experiments and they're

spk_0 like a service organization where you can come from a business unit, you can commission

spk_0 experiment and they'll run it for you and they give you the results. That's interesting. That's

spk_0 uncertain how this is all going to work out. They don't they may not believe that the company's ready

spk_0 to do this at large scale. It probably simplifies training and it lets people, let's people dip

spk_0 their toe in without without really having to. Exactly. And you have a few experts and they kind of

spk_0 make sure that people don't do foolish things. Yeah. Then the another form is to have a decentralized,

spk_0 completely decentralized. So now we're shifting the autonomy, basically to people and allow pretty

spk_0 much anybody to run experiments and we don't centralize it anymore. And of course, there you have

spk_0 to trust people. You have to know that they're actually capable of doing this and it's a way of

spk_0 course to rapidly scale things. But what happens there is when you start to put all these, you spread

spk_0 all these sort of your experts around and they're always sort of through the company, they get very

spk_0 busy and you kind of lose the focus on building capabilities because you need to always kind of get

spk_0 better and better. And so there's no coordinated approach to this. Everybody kind of does their own

spk_0 thing. So what companies have found is they go from centralized to decentralized and they want to

spk_0 scale things. But then they realize that they need to have a more coordinated approach and then they

spk_0 create something which they call a center of excellence. And the center of excellence is kind of a

spk_0 hybrid model. Then where you have sort of a core group that actually is responsible for developing

spk_0 capabilities, experimentation capabilities, kind of know what tools to use and push the envelope.

spk_0 But at the same time, you take people out of that group and then place some sort of into the different

spk_0 organizational units that are doing this and they're basically there to help as well. And companies

spk_0 found that that's actually sort of a very good compromise because on one hand, you kind of empower

spk_0 people to do things on their own. At the same time, you actually have someone who centrally owns

spk_0 this capability as well. How do you know when it's really working? You know, the way you're really

spk_0 working, I think it's a cultural test. And I tell you, here's the test. You sit in a meeting and

spk_0 you're discussing a decision. And you know when it's working, either when someone asks, where's the

spk_0 experiment? Or when someone actually walks into the meeting and says, here is the experiment.

spk_0 When these kinds of discussions are happening every single day without you having to ask for

spk_0 these things, then you know things are kind of working. I call it, it's like running the numbers.

spk_0 You wouldn't, when you go into a meeting, you always expect people to do some financial analysis.

spk_0 It's almost a given. So it has to be like that. It has to be like running a financial analysis.

spk_0 It has to be a given that you kind of do a test. You run an experiment. Unless you've done it,

spk_0 you know, you know, we're not going to make a decision.

spk_0 Say you're an individual contributor. You may be a manager. You may be a front line worker.

spk_0 But you buy into this. Like you see the value of experiments. You want your organization to do more.

spk_0 What do you do to try to bring a culture of experimentation to a place that is still relatively new to

spk_0 what you can do as an employee is first of all, raise the awareness around sort of you. What does that mean?

spk_0 That means basically explaining sort of the people what sort of the value of the experiment

spk_0 what experiments are. But then also, I think at the same time is maybe try to do some of these

spk_0 things in the areas that you control. You know, yes, I see the difficulty sometimes. And I hear

spk_0 this from people saying, okay, I get you. But you know, there are two levels up. You know,

spk_0 I'm not sure that they do. So what can I do? So I always tell them start small, get going. And then

spk_0 this is what often what happens. I've talked to organizations that actually started this way and

spk_0 they got bigger and bigger. They said, you know, we started out and we run an experiment and we

spk_0 went to the meeting and we told people what the experiment sort of showed us and so forth. And

spk_0 understand the value of it. But you got to get started. Don't wait. What kind of manager is

spk_0 then the successful manager in a company that has a culture of experimentation? Because in the past,

spk_0 maybe it used to be people who had experience, people who had intuition. Now when you run experiments,

spk_0 what is the type of manager who excels and advances in an organization that has a culture of

spk_0 experimentation? So you can ask the question, if everything is adjudicated by experiments,

spk_0 by tests, what's the role of the manager? Anyway, I kind of break it down into sort of three

spk_0 different things that they should do. First role, I think of a manager is to set a grand challenge.

spk_0 What we don't want to do is we don't have an organization that just does experiments willingly,

spk_0 you know, with no direction. So there needs to be a grand challenge. A grand challenge, for example,

spk_0 could be, you know, we want to have the best user experience in the industry. And that grand

spk_0 challenge then can be broken down into different pieces, which then can be addressed with hypotheses,

spk_0 which are then tested. So you give them a directionality that needs to be a program,

spk_0 a systematic program that sort of aims for some big or goal. So that's the grand challenge.

spk_0 The second thing I think that managers need to do, especially in this kind of environment,

spk_0 they need to place the systems, resources, and organizational designs that allow for the

spk_0 large scale experimentation to happen. Things like that don't happen by themselves. You need to

spk_0 invest in tools. You need to make sure that you've got the right organizational design to start

spk_0 out with and maybe then change it when things don't work. So you have to think about that as well.

spk_0 And you need to make sure that sort of all the systems are in place. So someone like that

spk_0 employee at Microsoft can just kind of push a button essentially and just launch and run this thing.

spk_0 If it takes employees weeks and weeks to set up an experiment, what are the odds of them doing it

spk_0 at large scale? It's not going to happen. So you kind of make it easy as well. And you do the

spk_0 empower sort of people to do it to you to democratize experiments. And the third one is they need to be a

spk_0 role model. They need to live by the same rules. So when we go into a meeting and we propose

spk_0 a course of action and someone says, that's really nice. We'll run a test and let you know what happens.

spk_0 We need to then have the humility to say, let's do it and let's do it quickly. So we need to

spk_0 live the same way. We need to kind of do the same thing that we ask our employees to do. So that's

spk_0 a different style of leading. Stefan, thank you so much. Maybe we'll try some experimentation on

spk_0 this show as well. Thank you, Arjun. Great to be here.

spk_0 Stefan Tumpke is a professor at Harvard Business School. He's the author of the book,

spk_0 Experimentation Works, the surprising power of business experiments, as well as the HBR

spk_0 article, Building a Culture of Experimentation. HBR on leadership will be back next Wednesday

spk_0 with another hand-picked conversation from Harvard Business Review. If this episode helped you,

spk_0 share it with your friends and colleagues and follow the show on Apple podcasts, Spotify,

spk_0 or wherever you listen to podcasts. And while you're there, consider leaving us a review.

spk_0 And when you're ready for more podcasts, articles, case studies, books, and videos with the

spk_0 world's top business and management experts, find it all at hbr.org.

spk_0 This episode was produced by Mary Doe and me, Amanda Cursey.

spk_0 On leadership's team includes Marine Hoek, Rob Eckhart, Tina Toby Mack,

spk_0 Erica Trexler, Ramsey Cabaz, Nicole Smith, and Ann Bartholomew. Music is by Koma Media.

spk_0 Thanks for listening.

Cultivating an Experimental Mindset in Your Organization

Interactive Transcript

Topics Covered

Related Episodes

Reimagining Midlife: My 2x20 Project™ Update & the Power of Experiments

Stranger Things Was REAL: Inside the Montauk Project

the magician who invented the modern world: john dee and the lost birth of science

Essentials: Erasing Fears & Traumas Using Modern Neuroscience

Share Episode