The airport with 2 control towers

In this Episode

• Why was this “partner ship” sinking?
• What did the US company do that made things worse, not better?

Transcript

Tom Cooper 0:00
What the heck is wrong with Lotus tech? And it sounded bad. Lotus Tech had a track record of not performing.

Tom Cooper 0:14
Welcome to the episode of the software horror stories Podcast. I’m glad to have you join me today. Today’s story comes from a while back and I wanted to share it because the challenges that my clients were facing then are still real challenges that clients face today. The technology is changed in some cases, but the problems are still the same. Today’s episode is one that I think of as the airport with two control towers. This will become clear in a minute. So my client asked me, why can’t we ship faster and ship better code. This is a billion dollar telecommunications firm with a long history. And they had embarked on a big transformational project with a goal of a major product rollout. Over the years, they’ve been living on the profit profits from their cash cow. And they knew a major investment was needed to allow them to transition to the next generation of services. If they were going to stay viable. Let’s call my client future sync. Future sync was on a burning platform and they knew it. And while they still had some time That fire was getting closer, and they were beginning to feel the heat. Now in response, they had put out an RFP for a partner to help them implement a new technology that would equip them to go to the next level. Lotus tech, the winner of the RFP was a firm based in Asia that had been longing for a foothold in the US market, they were hungry to partner with an established brand in the US that would extend their credibility and grow their overall sales.

Tom Cooper 1:39
As a part of their response to the RFP, Lotus tech took an existing bare bones, IP DVR product that was working in another market and they customized it for my client. The needs of the US market were very different from the existing clients. So lotus tech needed to make substantial customizations to their codebase to make future sync happy. Now the partnership had been running for about two years when I met Oliver and Chloe. Oliver was the director of consumer operations and Chloe was a director of development

Tom Cooper 2:07
for the new platform, and they asked me to help figure out what wasn’t working. Now initially, there had been a lot of promise. In fact, the early proof of concept versions came up very quickly, the development team at Lotus Tech had been very happy to listen to future sync and future sync was happy to see how their ideas were making it into updates of the product on a regular basis. Now, as the program move forward, eventually those proof of concept versions became the real MVP. And that MVP turned into set top boxes that rolled out to customers homes. And of course as a minimum product. Future sink did have a large backlog of product ideas they were excited to one day see be turned into a part of the platform for customers to Oliver’s job was to be oversee the rollout and support the new product. And as with any new product touching real users, there were issues because there’s no such thing as software without bugs. There were some showstopper Bugs, bugs that kill the operations of the set top box completely. And there were some minor usability bugs. And then there were some bugs that were just hard to replicate. And those intermittent failures was probably the worst. And what Oliver told me was he just didn’t understand how Lotus tech kept failing to fix those bugs. Over and over again, Lotus Tech had promised to deliver updates to the code, but there were long standing bugs and big ones that didn’t seem to ever get fixed.

Tom Cooper 3:26
That Chloe, on the other hand didn’t have to worry a ton about bugs. But her role was to create a viable roadmap for the product that could woo customers from competitors, and make a compelling case that future sink was the entertainment of vendor of choice based on superior DVR technology. And like Oliver, Chloe was upset that Lotus tech couldn’t seem to follow through with their promised features. She carefully curated and productized issues or prioritized issues. She maintained a backlog she planned releases, she was doing what needed to be done. But Lotus tech kept dropping the ball, they would promise to deliver a feature. And then when a new release came out, not only would the feature not be there, but there were other features that Chloe never even asked for showing up in the product. Oliver’s operational challenges combined with Chloe’s failure to deliver what she was promising lead to problems with their overall strategy. Customers just wanted their set top boxes to work as reliably as a toaster without having to reboot the box or call support. And that was that interruption was preventing them from enjoying their entertainment which was the whole point of the thing to begin with.

Tom Cooper 4:31
Future sinks competitors kept rolling out new capabilities faster than future sink did. And not only were they falling behind, but this program designed to be the future of the company was in danger of delivering too little too late.

Tom Cooper 4:45
A frustration is what happens when we what we experience doesn’t match what we expected. Frustration is what happens when what we experience doesn’t match what we expect it and to suggest that Oliver and Chloe were frustrated. Well, that was a massive

Tom Cooper 5:00
understatement. Each of them had been called on the carpet by their leadership for failing to deliver. The CEO was concerned that if they couldn’t make big changes in the health of the program, they might need to start all over again. But this time, they had a lot less money and a lot less time to get off that burning platform and started the whole thing to begin with.

Tom Cooper 5:19
It was about this time that I was brought in, and the problem I was hired to help them explore is what is wrong with Lotus tech. And it sounded bad. Lotus Tech had a track record of not performing. They made promises they didn’t keep. They didn’t acknowledge they were dropping the ball. And they seem to be really, really unhelpful.

Tom Cooper 5:38
I’ve got a son with a tech degree. He’s early in his career, and I’ve been working with him on learning problem solving techniques. One lean tool I’m a fan of is the five why’s where you ask a series of questions about why a failure happened. And here’s an example of of using that technique for a technology problem. Problem statement, the company’s websites experiencing slow page load times. One, why is the website loading slowly? Because server response time is high. Second, why? Why is the server response time high? Because database queries take a long time to execute. Third, why why did the database queries take a long time? Because there’s a high volume of traffic on the website causing database overload. Fourth, why why is there a high volume of traffic on the website? Because a recent marketing campaign increased user traffic significantly? Five, the fifth one, why did the marketing campaign cause database overload because the website infrastructure was not scaled or optimized to handle that level of traffic load. So in this example, the initial problem the presenting problem was described as slow page load times. By asking the five why’s we uncover the root cause the lack of scalability and optimization in the website infrastructure to handle the increased traffic for the marketing campaign. When you address the root cause the scaling and optimizing of infrastructure, you can ensure that the website can handle those traffic spikes without significant performance issues.

Tom Cooper 7:00
The five why’s technique helps pinpoint the underlying issues and software related problems and guide you toward an effective solution. You see, often the problem we hear is only the first layer, like slow load times, we need to dig dig deeper to better understand the factors that are contributing to the behavior we’re seeing. So as we began to investigate the issues at future sink, we scheduled some meetings to better understand the problems, as they were seen by different team members. problems that were included here, were Lotus Tech was not delivering reliable code. Lotus deck was not delivering promised features, code was late, Lotus tech didn’t even seem to have any sense of accountability. The future saying they were getting mad, big man. Now, given this input, it would be easy to allow that to frame the problem as one of compliance with contract. But anytime you’re dealing with teams, you’ve got more than one perspective. And our as our assessment process allows us to go in and not make any assumptions at all. We assumed nothing. And we asked questions like, What is the goal of this effort? And if it was working perfectly, how would you know? And we asked many other basic questions like that.

Tom Cooper 8:11
We also explored the processes and the practices that were in place. Future Singh had some experience with software development, but they probably didn’t have enough embedded systems and API experience. They do have some and their project management experience left something to be desired too. And what Oliver was doing was tracking issues and planning features and they had a decent process in place to capture information and to track progress on specific items. Oliver had established review meetings where they met internally and externally with the Lotus tech team to discuss issues and issue status.

Tom Cooper 8:43
As often happens, some of Oliver’s issues weren’t really bugs, but they were feature requests, and Chloe was working on her new features list and her list and priorities didn’t always match Oliver’s priorities either.

Tom Cooper 8:55
Now, when we come back from the break, we’re going to talk about what we learned in our assessment when we included a larger team.

Tom Cooper 9:09
As a leader in a software team, you know, you want to help your team perform at the highest level. And you know that with the tools we have, it’s never been easier to write software than it is today. But it’s still hard. I’m Tom Cooper, and for the last decade, I’ve focused on coaching and training software leaders and their teams to help them make the small but important improvements to help them perform as one team and to achieve their potential. How much better would your results be if your team communicated clearly managed conflict well delegated work that stayed delegated, and then created an executed on great work plants. It shouldn’t have to be this hard to write great software. And the good news is it doesn’t have to be are you ready to take the next step? Head on over to brightfield group.com So you can learn more

Tom Cooper 10:05
So before the break, we talked about problem that future sink was experiencing. We talked about Oliver and Chloe and how they were super frustrated with low performance on the part of Lotus tech. Our assessment expanded to more future sink team members, and we added Lotus tech team members as well. In the first segment, we talked about the needs that future sync had the risks they were facing, and their reasons to look for a partner like lotus tech. At this point, let’s turn the corner and talk a little bit about Lotus tech and their goals. Lotus Tech had a solid engineering team and their leadership had given them a mandate for growth. They were charged with building a platform that could be very profitable for the company. They knew the potential market for this platform was far far bigger than one small regional player in the Midwest, us. Lotus tech love the concept of landing this deal with future sync because they could partner with a US based company and learn a ton about market problems and market needs. Now Lotus Tech’s leaders were hungry, and they were willing to make a bet on this deal that they could they could have even even if they lost money on this client, and they could still be profitable overall in the US market. It’s a well known loss leader strategy. prior to landing this RFP with future sync, Lotus Tech had zero penetration in the US for this category of product. They have related products with some success. But the US market is so big, that this that the leaders Lotus tech believe this partnership would open doors for them. And by having a referenceable client in the US, they could build on that to sell the next client. And by learning what US consumers wanted an entertainment systems, they could prioritize their features and build a roadmap that could allow them to really take advantage of the US market. And they already had a product in this category that was in production. So with an existing engine, all they had to do was build the US specific parts around it right?

Tom Cooper 11:50
Oh my goodness, because Lotus tech wanted to grow market share, and profitability was less important than the learning in the market access. They played hard when they tried to land the RFP, they offered future sync a huge discount in order to land the deal. Future sync, looking at the competing platforms, believed that this huge discount on this already less expensive offshore partner made this deal almost too good to be true from future sinks perspective. Unfortunately, things don’t always work out the way that we originally hoped. And Lotus Tech was not very happy with how things were turning out. It turned out that building a product for us consumers was a lot harder than they initially thought compared to the other market that they were already operating in US consumers had very complicated interests in and needs when it came to DVR technology. And in order to support those complex needs, that they needed. That hardware that was more than capable in other markets wasn’t able to keep up with the CPU and the memory needs of the more complicated databases that were required in the US market. So the cost of hardware was going to be higher than the low low tech originally planned. And then the cost of software development was going to be a lot higher than forecast to. Now at this point, they had poured a ton of investment into a money losing deal with this regional us player. And they’ve made some good progress on improving the platform. But they were very far from having a happy client they could use to reference for additional sales. In fact, that that super low price that they offered well, it was nowhere near covering the cost of development, and they were losing money as the whole thing went forward. And future sync in response was disrespectful, angry, even hostile.

Tom Cooper 13:35
In the face of these challenges, Lotus Tech had moved from version two to version four of their platform. The current release was the most complex software they ever built, had rich graphics and a set of programming and search capabilities that was even beyond their initial vision. They were making progress, but every step was like walking through concrete. Lotus text management was happy to see the product got better. But their ultimate goal remember was market share growth, and that goal seemed to be far away, no matter how hard they worked or how much progress they made on the technical side.

Tom Cooper 14:08
Future sync also felt like they were in a dysfunctional marriage on the way toward divorce. Why couldn’t Lotus tech just keep their promises?

Tom Cooper 14:16
Ben Franklin famously said the bitterness of poor quality remains long after the sweetness of low prices forgotten that super cheap discount they got when they put out that RFP. It didn’t seem like such a great deal now.

Tom Cooper 14:29
So how did we get here? What went wrong that took us off the path. While our assessment process gave us some clues because future sync had one set of goals. And Lotus tech shared some of those goals. But they also had goals that were much larger than future sync. If you and your partner have different goals, you’re going to run into some difficulties on your journey because when you want to turn left, they want to turn right and the assessment revealed that both Lotus tech and future sync were aligned in their passion. Each group was committed to building a great product even if they

Tom Cooper 15:00
The difference in what constituted a great product. Another positive was that both groups were committed to a long term relationship. As a part of our assessment, we asked penetrating questions about whether this was a relationship headed for breakup. And leaders on each side, although they were highly frustrated, expressed a willingness, even a commitment to long term collaboration. However, the gap between what was expected and what was experienced on both sides led to low trust, Lotus tech felt future sync was disrespectful, demanding and insulting. Future sync felt that Lotus tech wasn’t keeping their commitments, and they didn’t complete agreed upon goals in terms of time of completion.

Tom Cooper 15:41
Another finding for our assessment was a cultural disconnect. This is a big deal when we do offshoring, and not enough companies spend enough time thinking about this. Erin Meyer did an excellent job of discussing factors and cultural differences in her book, the Culture Map. And when we’re working with cross cultural teams, it’s often helpful to realize that what is so called normal to us is likely not normal to other people. I don’t have time to go into details in today’s episode, but there are eight factors in collaboration that are affected by cultural norms. And they include the way we prefer to communicate, how we evaluate information, how teams organize how decisions are made, how trust is created, how we manage disagreement, and how we see time management. In this case, we had a US company working with an Asian company. In seven of the eight factors, the default approach of the Americans was the opposite of the preferred approach for the lowest tech team. You can imagine, if disagreement and decision making processes were opposite for each partner, you’d have difficulties. In this case, it was almost every single area where they had differences. As a result, conflict resolution was not happening in a healthy way, each party tended to behave in ways that were harmful to relationship with the other party. It’s a vicious cycle. Every time there was a disconnect, it led to more distrust, more hurt feelings more frustration, and because each side found the other side baffling, there was no positive progress being made. And in fact, a great deal of damage was happening to both sides, just from the regular interaction.

Tom Cooper 17:11
George Bernard Shaw once commented that England and America are two countries separated by a common language, suggesting that even a substantial overlap between US and British vocabulary was not enough to help us understand each other. Here we are talking about two cultures, which didn’t even share the same alphabet, much less words. Like many project teams, these two teams use common terms like release quality, bugs, and features. Unfortunately, each team had a different understanding of exactly what was meant by each term. And worse, like many software teams, team members had a high degree of confidence that their definition of those words was the correct definition. So they were consistently miscommunicating. And they were confident that they were not miscommunicating. And digging deeper, while Chloe had a backlog of features was some priority established, she’d not been confident enough in Lotus texts execution. For her to be able to create a meaningful roadmap, she was disappointed that Lotus tech kept missing deliverables and could not imagine why. So we looked at future at feature definition. And we discovered that the product team at future sync and the development at Lotus tech didn’t even agree on what a specific feature ought to do. So it’s not surprising that the features as delivered didn’t match what was expected. And also the people who were directly involved in the meetings and the communication, they frequently didn’t have the authority to make decisions, or make commitments. And as a result, delays and further breakdowns and communication happened on a regular basis. So who’s in charge anyway, going back to the goals we talked about earlier, one of the key findings from the assessment was that future sink, did have a running product management team. And they felt they were in charge of all the decision making. And the exact same thing was true at Lotus Tech because Lotus Tech had rolled out a product in another market. And so they felt like this is our product. And we’re making enhancements for future sake. And future seeing felt like this is our product. And we’re teaching Lotus tech how to do it. And the Lotus tech product team was working to deliver updates to non US customers based on the development state. And they had a queue of requests from other customers as well as operational requirements for components they were hosting in other environments for other clients. And so their interests and needs were not exactly the same as future sinks. And the upshot of all of this was that future sink felt they had the authority to make product decisions, set priority and set schedule at the same time. So did Lotus tech and since Lotus Tech was the one writing the code, their priorities were represented and what was got released.

Tom Cooper 19:39
And this is why I talked about trying to land at an airport with two control towers. Future sync might say you’re clear to land but Lotus tech would say no, you’re not.

Tom Cooper 19:48
And this was complicated because Lotus Tech was from a culture that resolves conflict through indirect communication. Using an American lens future sync hadn’t understood the intent of the messaging from Lotus tech

Tom Cooper 20:00
leader, if they had been more intune, they would have been able to understand that when Lotus tech said things like that will be very difficult. What they meant was, there’s no way we’re going to do that. And then when Lotus tech didn’t deliver, future sinks team would react in a very confrontational way leading to more distrust and more problems in the relationship. Successful production operation of a system like this requires a sequence of things to work perfectly power, internet connection, available bandwidth, streaming, servers, databases, and more all have to work at a certain minimum performance level. If any of those items is out of place, the system doesn’t work the way it should. And because of the history, future sync tended to interpret any production interruption as the fault of the DVR software. Sometimes that was true. But our assessment determined that many times it wasn’t in fact, challenges with wiring installation failures and network congestion might be to blame. But the first assumption tended to be it was Lotus texts, software or hardware that was at fault. Rather than believing the overall team was committed to fighting the problems together, the environment was much more accusatory and blame oriented with Lotus tech, having the position where they needed to prove they were not at fault.

Tom Cooper 21:11
What a mess. How do we fix something like that? Well, one thing I tell teams in conflict with each other is that you didn’t drive this far into the ditch overnight, and you aren’t gonna get out overnight. But we can start steering back toward the road together right now. The things races major frustrations were just symptoms of the disease, not the disease itself, we dig in to learn what was driving the unhealthy behaviors, and the force behind what was interpreted as disrespect and the subsequent conflicts. If we fail to define the problem, well, no matter how smarter fix is, it won’t be good enough. When the leaders that future sink had looked at the problem, they were focusing on the symptoms, and whatever fix they worked on, at best, would have only had a minor impact on overall team performance. When you want to fix a problem, you have to identify the root cause and put real fixes in place. And we’re all super busy. And we have so much input that it’s a distraction from the important stuff. So we come up with a shorthand that allows us to not spend time talking about the details, we’ll talk about things like the bid the bugs, or release five is is late as a quick way to help us get above those details. It’s interesting. In World War Two fighter and bomber pilots were subjected to a phenomenon called Target fixation. In the middle of chaos, they would focus so intently on their target, that they had a tendency to ignore everything else, and then fly directly into the target rather than destroy it and move away from it. Sometimes when we’re problem solving, we have to watch out for target fixation. Once we’ve decided we know what the problem is, it’s very difficult for us to see related items or even see underlying issues that cause the thing we’re looking at. In this case, the problem was design defined as late releases poor quality and a bad partner. Once that idea was stuck in the heads of the leadership team at future sync, it was very hard to pull out of that dive and not just slam into the target. So the first thing we needed to do we get the senior leadership team together and present findings and open a discussion about what could happen next.

Tom Cooper 23:07
Once we broke through the noise of all that frustration and pain that the leaders had experienced, we helped reframe the problem to look at the things that were causing the pain, then we could take a different course of action. So that meeting with future sinks, leaders went well, they were interested in making the relationship work, because what’s their best alternative to negotiated agreement scrapping everything they’ve done declaring failure and picking a different partner. And remember, Lotus Tech was cheaper by far than everybody else. So while they blown a ton of money to get to this point, they weren’t in a position to go back to their senior executives and say, Hey, we need a whole lot more money to start from scratch because we screw this whole thing up, right.

Tom Cooper 23:48
And they wanted to make that work. And they were open to learning what they had done poorly and what they could do to fix things going forward. And the next step was to engage Lotus tech and get their buy in, we needed both sides to commit to what I was calling a relationship reboot, to acknowledge the missing parts and the errors that were made, and then make a commitment to solve the problem. And we also needed a way to make sure the teams remained on track. It’s so easy to get caught up in details, and it’s hard to take a few steps back to look at how the engagement is going at a higher level. We recommended they put a quarterly meeting in place with executives from both organizations to review progress and to provide a way to escalate issues that remained ongoing. See, without a recurring meeting like that, there’s no opportunity to go in and resolve these things that come up from time to time.

Tom Cooper 24:35
And finally, we talked about getting to reality what was really happening. When it came to the roadmap and the features we needed a process to align Lotus text thinking with future syncs ideas. We needed to get Lotus tech to be direct and honest about what was a priority for them and what was literally not a priority. We also recommended a number of tactical improvements in the way they did testing in tracking work and reporting and follow through just to the overall

Tom Cooper 25:00
offer management process.

Tom Cooper 25:02
The end of the day, we were able to rescue that relationship and progress was able to be made, thanks to the commitment of both sides to be willing to discover and fix their issues. And without that willingness, without an investment in that commitment to saying, Hey, we own part of responsibility, this problem, we need to fix it, the chances of that coming out in a good way were very, very low. In this case, also, both sides were unhappy with the other because neither of them was making the kind of progress they wanted to make. And they weren’t clear with each other about what they defined as success or how they were measuring that success. And as a result, each was plowing forward on what they thought the goal was. But because of that disconnect, each side was unhappy with the other. These things happen all the time in organizations, because we get so busy doing the work that we lose track on whether we’re actually still headed toward the correct destination.

Tom Cooper 26:00
How are things going with your team and your projects? These kinds of problems are fairly common. And the good news is that once we begin to look at some of the underlying causes, we often are able to make significant progress and quick progress as well. Once we began to work on building trust, and making and keeping promises that everybody could agree on, the details of things got felt a lot better. And the way things feel on a project is very, very valuable. It affects morale in a huge way. If I think your intent is a bad intent, I’m not going to trust you. I’m not going to clap collaborate with you, I’m not going to want to help you. But once we begin to start to build that shared expectations and shared trust, all of a sudden projects get a lot healthier and a lot happier.

Tom Cooper 26:52
Are you facing a software horror story? I’m your guy for action oriented workshops and executive coaching for technical leaders and their teams. Thanks for joining us for this latest episode in the software horror stories Podcast Series. For more information about brighto Group please check out www dot bright Hill group.com that’s www dot bright Hill group.com for bright Hill group. I’m Tom Cooper.