In this Episode

• Why was this “partner ship” sinking?
• What did the US company do that made things worse, not better?

Transcript

Tom Cooper 0:00
What the heck is wrong with Lotus tech? And it sounded bad. Lotus Tech had a track record of not performing.

Tom Cooper 0:14
Welcome to the episode of the software horror stories Podcast. I’m glad to have you join me today. Today’s story comes from a while back and I wanted to share it because the challenges that my clients were facing then are still real challenges that clients face today. The technology is changed in some cases, but the problems are still the same. Today’s episode is one that I think of as the airport with two control towers. This will become clear in a minute. So my client asked me, why can’t we ship faster and ship better code. This is a billion dollar telecommunications firm with a long history. And they had embarked on a big transformational project with a goal of a major product rollout. Over the years, they’ve been living on the profit profits from their cash cow. And they knew a major investment was needed to allow them to transition to the next generation of services. If they were going to stay viable. Let’s call my client future sync. Future sync was on a burning platform and they knew it. And while they still had some time That fire was getting closer, and they were beginning to feel the heat. Now in response, they had put out an RFP for a partner to help them implement a new technology that would equip them to go to the next level. Lotus tech, the winner of the RFP was a firm based in Asia that had been longing for a foothold in the US market, they were hungry to partner with an established brand in the US that would extend their credibility and grow their overall sales.

Tom Cooper 1:39
As a part of their response to the RFP, Lotus tech took an existing bare bones, IP DVR product that was working in another market and they customized it for my client. The needs of the US market were very different from the existing clients. So lotus tech needed to make substantial customizations to their codebase to make future sync happy. Now the partnership had been running for about two years when I met Oliver and Chloe. Oliver was the director of consumer operations and Chloe was a director of development

Tom Cooper 2:07
for the new platform, and they asked me to help figure out what wasn’t working. Now initially, there had been a lot of promise. In fact, the early proof of concept versions came up very quickly, the development team at Lotus Tech had been very happy to listen to future sync and future sync was happy to see how their ideas were making it into updates of the product on a regular basis. Now, as the program move forward, eventually those proof of concept versions became the real MVP. And that MVP turned into set top boxes that rolled out to customers homes. And of course as a minimum product. Future sink did have a large backlog of product ideas they were excited to one day see be turned into a part of the platform for customers to Oliver’s job was to be oversee the rollout and support the new product. And as with any new product touching real users, there were issues because there’s no such thing as software without bugs. There were some showstopper Bugs, bugs that kill the operations of the set top box completely. And there were some minor usability bugs. And then there were some bugs that were just hard to replicate. And those intermittent failures was probably the worst. And what Oliver told me was he just didn’t understand how Lotus tech kept failing to fix those bugs. Over and over again, Lotus Tech had promised to deliver updates to the code, but there were long standing bugs and big ones that didn’t seem to ever get fixed.

Tom Cooper 3:26
That Chloe, on the other hand didn’t have to worry a ton about bugs. But her role was to create a viable roadmap for the product that could woo customers from competitors, and make a compelling case that future sink was the entertainment of vendor of choice based on superior DVR technology. And like Oliver, Chloe was upset that Lotus tech couldn’t seem to follow through with their promised features. She carefully curated and productized issues or prioritized issues. She maintained a backlog she planned releases, she was doing what needed to be done. But Lotus tech kept dropping the ball, they would promise to deliver a feature. And then when a new release came out, not only would the feature not be there, but there were other features that Chloe never even asked for showing up in the product. Oliver’s operational challenges combined with Chloe’s failure to deliver what she was promising lead to problems with their overall strategy. Customers just wanted their set top boxes to work as reliably as a toaster without having to reboot the box or call support. And that was that interruption was preventing them from enjoying their entertainment which was the whole point of the thing to begin with.

Tom Cooper 4:31
Future sinks competitors kept rolling out new capabilities faster than future sink did. And not only were they falling behind, but this program designed to be the future of the company was in danger of delivering too little too late.

Tom Cooper 4:45
A frustration is what happens when we what we experience doesn’t match what we expected. Frustration is what happens when what we experience doesn’t match what we expect it and to suggest that Oliver and Chloe were frustrated. Well, that was a massive

Tom Cooper 5:00
understatement. Each of them had been called on the carpet by their leadership for failing to deliver. The CEO was concerned that if they couldn’t make big changes in the health of the program, they might need to start all over again. But this time, they had a lot less money and a lot less time to get off that burning platform and started the whole thing to begin with.

Tom Cooper 5:19
It was about this time that I was brought in, and the problem I was hired to help them explore is what is wrong with Lotus tech. And it sounded bad. Lotus Tech had a track record of not performing. They made promises they didn’t keep. They didn’t acknowledge they were dropping the ball. And they seem to be really, really unhelpful.

Tom Cooper 5:38
I’ve got a son with a tech degree. He’s early in his career, and I’ve been working with him on learning problem solving techniques. One lean tool I’m a fan of is the five why’s where you ask a series of questions about why a failure happened. And here’s an example of of using that technique for a technology problem. Problem statement, the company’s websites experiencing slow page load times. One, why is the website loading slowly? Because server response time is high. Second, why? Why is the server response time high? Because database queries take a long time to execute. Third, why why did the database queries take a long time? Because there’s a high volume of traffic on the website causing database overload. Fourth, why why is there a high volume of traffic on the website? Because a recent marketing campaign increased user traffic significantly? Five, the fifth one, why did the marketing campaign cause database overload because the website infrastructure was not scaled or optimized to handle that level of traffic load. So in this example, the initial problem the presenting problem was described as slow page load times. By asking the five why’s we uncover the root cause the lack of scalability and optimization in the website infrastructure to handle the increased traffic for the marketing campaign. When you address the root cause the scaling and optimizing of infrastructure, you can ensure that the website can handle those traffic spikes without significant performance issues.

Tom Cooper 7:00
The five why’s technique helps pinpoint the underlying issues and software related problems and guide you toward an effective solution. You see, often the problem we hear is only the first layer, like slow load times, we need to dig dig deeper to better understand the factors that are contributing to the behavior we’re seeing. So as we began to investigate the issues at future sink, we scheduled some meetings to better understand the problems, as they were seen by different team members. problems that were included here, were Lotus Tech was not delivering reliable code. Lotus deck was not delivering promised features, code was late, Lotus tech didn’t even seem to have any sense of accountability. The future saying they were getting mad, big man. Now, given this input, it would be easy to allow that to frame the problem as one of compliance with contract. But anytime you’re dealing with teams, you’ve got more than one perspective. And our as our assessment process allows us to go in and not make any assumptions at all. We assumed nothing. And we asked questions like, What is the goal of this effort? And if it was working perfectly, how would you know? And we asked many other basic questions like that.

Tom Cooper 8:11
We also explored the processes and the practices that were in place. Future Singh had some experience with software development, but they probably didn’t have enough embedded systems and API experience. They do have some and their project management experience left something to be desired too. And what Oliver was doing was tracking issues and planning features and they had a decent process in place to capture information and to track progress on specific items. Oliver had established review meetings where they met internally and externally with the Lotus tech team to discuss issues and issue status.

Tom Cooper 8:43
As often happens, some of Oliver’s issues weren’t really bugs, but they were feature requests, and Chloe was working on her new features list and her list and priorities didn’t always match Oliver’s priorities either.

Tom Cooper 8:55
Now, when we come back from the break, we’re going to talk about what we learned in our assessment when we included a larger team.

Tom Cooper 9:09
As a leader in a software team, you know, you want to help your team perform at the highest level. And you know that with the tools we have, it’s never been easier to write software than it is today. But it’s still hard. I’m Tom Cooper, and for the last decade, I’ve focused on coaching and training software leaders and their teams to help them make the small but important improvements to help them perform as one team and to achieve their potential. How much better would your results be if your team communicated clearly managed conflict well delegated work that stayed delegated, and then created an executed on great work plants. It shouldn’t have to be this hard to write great software. And the good news is it doesn’t have to be are you ready to take the next step? Head on over to brightfield group.com So you can learn more

Tom Cooper 10:05
So before the break, we talked about problem that future sink was experiencing. We talked about Oliver and Chloe and how they were super frustrated with low performance on the part of Lotus tech. Our assessment expanded to more future sink team members, and we added Lotus tech team members as well. In the first segment, we talked about the needs that future sync had the risks they were facing, and their reasons to look for a partner like lotus tech. At this point, let’s turn the corner and talk a little bit about Lotus tech and their goals. Lotus Tech had a solid engineering team and their leadership had given them a mandate for growth. They were charged with building a platform that could be very profitable for the company. They knew the potential market for this platform was far far bigger than one small regional player in the Midwest, us. Lotus tech love the concept of landing this deal with future sync because they could partner with a US based company and learn a ton about market problems and market needs. Now Lotus Tech’s leaders were hungry, and they were willing to make a bet on this deal that they could they could have even even if they lost money on this client, and they could still be profitable overall in the US market. It’s a well known loss leader strategy. prior to landing this RFP with future sync, Lotus Tech had zero penetration in the US for this category of product. They have related products with some success. But the US market is so big, that this that the leaders Lotus tech believe this partnership would open doors for them. And by having a referenceable client in the US, they could build on that to sell the next client. And by learning what US consumers wanted an entertainment systems, they could prioritize their features and build a roadmap that could allow them to really take advantage of the US market. And they already had a product in this category that was in production. So with an existing engine, all they had to do was build the US specific parts around it right?

Tom Cooper 11:50
Oh my goodness, because Lotus tech wanted to grow market share, and profitability was less important than the learning in the market access. They played hard when they tried to land the RFP, they offered future sync a huge discount in order to land the deal. Future sync, looking at the competing platforms, believed that this huge discount on this already less expensive offshore partner made this deal almost too good to be true from future sinks perspective. Unfortunately, things don’t always work out the way that we originally hoped. And Lotus Tech was not very happy with how things were turning out. It turned out that building a product for us consumers was a lot harder than they initially thought compared to the other market that they were already operating in US consumers had very complicated interests in and needs when it came to DVR technology. And in order to support those complex needs, that they needed. That hardware that was more than capable in other markets wasn’t able to keep up with the CPU and the memory needs of the more complicated databases that were required in the US market. So the cost of hardware was going to be higher than the low low tech originally planned. And then the cost of software development was going to be a lot higher than forecast to. Now at this point, they had poured a ton of investment into a money losing deal with this regional us player. And they’ve made some good progress on improving the platform. But they were very far from having a happy client they could use to reference for additional sales. In fact, that that super low price that they offered well, it was nowhere near covering the cost of development, and they were losing money as the whole thing went forward. And future sync in response was disrespectful, angry, even hostile.

Tom Cooper 13:35
In the face of these challenges, Lotus Tech had moved from version two to version four of their platform. The current release was the most complex software they ever built, had rich graphics and a set of programming and search capabilities that was even beyond their initial vision. They were making progress, but every step was like walking through concrete. Lotus text management was happy to see the product got better. But their ultimate goal remember was market share growth, and that goal seemed to be far away, no matter how hard they worked or how much progress they made on the technical side.

Tom Cooper 14:08
Future sync also felt like they were in a dysfunctional marriage on the way toward divorce. Why couldn’t Lotus tech just keep their promises?

Tom Cooper 14:16
Ben Franklin famously said the bitterness of poor quality remains long after the sweetness of low prices forgotten that super cheap discount they got when they put out that RFP. It didn’t seem like such a great deal now.

Tom Cooper 14:29
So how did we get here? What went wrong that took us off the path. While our assessment process gave us some clues because future sync had one set of goals. And Lotus tech shared some of those goals. But they also had goals that were much larger than future sync. If you and your partner have different goals, you’re going to run into some difficulties on your journey because when you want to turn left, they want to turn right and the assessment revealed that both Lotus tech and future sync were aligned in their passion. Each group was committed to building a great product even if they

Tom Cooper 15:00
The difference in what constituted a great product. Another positive was that both groups were committed to a long term relationship. As a part of our assessment, we asked penetrating questions about whether this was a relationship headed for breakup. And leaders on each side, although they were highly frustrated, expressed a willingness, even a commitment to long term collaboration. However, the gap between what was expected and what was experienced on both sides led to low trust, Lotus tech felt future sync was disrespectful, demanding and insulting. Future sync felt that Lotus tech wasn’t keeping their commitments, and they didn’t complete agreed upon goals in terms of time of completion.

Tom Cooper 15:41
Another finding for our assessment was a cultural disconnect. This is a big deal when we do offshoring, and not enough companies spend enough time thinking about this. Erin Meyer did an excellent job of discussing factors and cultural differences in her book, the Culture Map. And when we’re working with cross cultural teams, it’s often helpful to realize that what is so called normal to us is likely not normal to other people. I don’t have time to go into details in today’s episode, but there are eight factors in collaboration that are affected by cultural norms. And they include the way we prefer to communicate, how we evaluate information, how teams organize how decisions are made, how trust is created, how we manage disagreement, and how we see time management. In this case, we had a US company working with an Asian company. In seven of the eight factors, the default approach of the Americans was the opposite of the preferred approach for the lowest tech team. You can imagine, if disagreement and decision making processes were opposite for each partner, you’d have difficulties. In this case, it was almost every single area where they had differences. As a result, conflict resolution was not happening in a healthy way, each party tended to behave in ways that were harmful to relationship with the other party. It’s a vicious cycle. Every time there was a disconnect, it led to more distrust, more hurt feelings more frustration, and because each side found the other side baffling, there was no positive progress being made. And in fact, a great deal of damage was happening to both sides, just from the regular interaction.

Tom Cooper 17:11
George Bernard Shaw once commented that England and America are two countries separated by a common language, suggesting that even a substantial overlap between US and British vocabulary was not enough to help us understand each other. Here we are talking about two cultures, which didn’t even share the same alphabet, much less words. Like many project teams, these two teams use common terms like release quality, bugs, and features. Unfortunately, each team had a different understanding of exactly what was meant by each term. And worse, like many software teams, team members had a high degree of confidence that their definition of those words was the correct definition. So they were consistently miscommunicating. And they were confident that they were not miscommunicating. And digging deeper, while Chloe had a backlog of features was some priority established, she’d not been confident enough in Lotus texts execution. For her to be able to create a meaningful roadmap, she was disappointed that Lotus tech kept missing deliverables and could not imagine why. So we looked at future at feature definition. And we discovered that the product team at future sync and the development at Lotus tech didn’t even agree on what a specific feature ought to do. So it’s not surprising that the features as delivered didn’t match what was expected. And also the people who were directly involved in the meetings and the communication, they frequently didn’t have the authority to make decisions, or make commitments. And as a result, delays and further breakdowns and communication happened on a regular basis. So who’s in charge anyway, going back to the goals we talked about earlier, one of the key findings from the assessment was that future sink, did have a running product management team. And they felt they were in charge of all the decision making. And the exact same thing was true at Lotus Tech because Lotus Tech had rolled out a product in another market. And so they felt like this is our product. And we’re making enhancements for future sake. And future seeing felt like this is our product. And we’re teaching Lotus tech how to do it. And the Lotus tech product team was working to deliver updates to non US customers based on the development state. And they had a queue of requests from other customers as well as operational requirements for components they were hosting in other environments for other clients. And so their interests and needs were not exactly the same as future sinks. And the upshot of all of this was that future sink felt they had the authority to make product decisions, set priority and set schedule at the same time. So did Lotus tech and since Lotus Tech was the one writing the code, their priorities were represented and what was got released.

Tom Cooper 19:39
And this is why I talked about trying to land at an airport with two control towers. Future sync might say you’re clear to land but Lotus tech would say no, you’re not.

Tom Cooper 19:48
And this was complicated because Lotus Tech was from a culture that resolves conflict through indirect communication. Using an American lens future sync hadn’t understood the intent of the messaging from Lotus tech

Tom Cooper 20:00
leader, if they had been more intune, they would have been able to understand that when Lotus tech said things like that will be very difficult. What they meant was, there’s no way we’re going to do that. And then when Lotus tech didn’t deliver, future sinks team would react in a very confrontational way leading to more distrust and more problems in the relationship. Successful production operation of a system like this requires a sequence of things to work perfectly power, internet connection, available bandwidth, streaming, servers, databases, and more all have to work at a certain minimum performance level. If any of those items is out of place, the system doesn’t work the way it should. And because of the history, future sync tended to interpret any production interruption as the fault of the DVR software. Sometimes that was true. But our assessment determined that many times it wasn’t in fact, challenges with wiring installation failures and network congestion might be to blame. But the first assumption tended to be it was Lotus texts, software or hardware that was at fault. Rather than believing the overall team was committed to fighting the problems together, the environment was much more accusatory and blame oriented with Lotus tech, having the position where they needed to prove they were not at fault.

Tom Cooper 21:11
What a mess. How do we fix something like that? Well, one thing I tell teams in conflict with each other is that you didn’t drive this far into the ditch overnight, and you aren’t gonna get out overnight. But we can start steering back toward the road together right now. The things races major frustrations were just symptoms of the disease, not the disease itself, we dig in to learn what was driving the unhealthy behaviors, and the force behind what was interpreted as disrespect and the subsequent conflicts. If we fail to define the problem, well, no matter how smarter fix is, it won’t be good enough. When the leaders that future sink had looked at the problem, they were focusing on the symptoms, and whatever fix they worked on, at best, would have only had a minor impact on overall team performance. When you want to fix a problem, you have to identify the root cause and put real fixes in place. And we’re all super busy. And we have so much input that it’s a distraction from the important stuff. So we come up with a shorthand that allows us to not spend time talking about the details, we’ll talk about things like the bid the bugs, or release five is is late as a quick way to help us get above those details. It’s interesting. In World War Two fighter and bomber pilots were subjected to a phenomenon called Target fixation. In the middle of chaos, they would focus so intently on their target, that they had a tendency to ignore everything else, and then fly directly into the target rather than destroy it and move away from it. Sometimes when we’re problem solving, we have to watch out for target fixation. Once we’ve decided we know what the problem is, it’s very difficult for us to see related items or even see underlying issues that cause the thing we’re looking at. In this case, the problem was design defined as late releases poor quality and a bad partner. Once that idea was stuck in the heads of the leadership team at future sync, it was very hard to pull out of that dive and not just slam into the target. So the first thing we needed to do we get the senior leadership team together and present findings and open a discussion about what could happen next.

Tom Cooper 23:07
Once we broke through the noise of all that frustration and pain that the leaders had experienced, we helped reframe the problem to look at the things that were causing the pain, then we could take a different course of action. So that meeting with future sinks, leaders went well, they were interested in making the relationship work, because what’s their best alternative to negotiated agreement scrapping everything they’ve done declaring failure and picking a different partner. And remember, Lotus Tech was cheaper by far than everybody else. So while they blown a ton of money to get to this point, they weren’t in a position to go back to their senior executives and say, Hey, we need a whole lot more money to start from scratch because we screw this whole thing up, right.

Tom Cooper 23:48
And they wanted to make that work. And they were open to learning what they had done poorly and what they could do to fix things going forward. And the next step was to engage Lotus tech and get their buy in, we needed both sides to commit to what I was calling a relationship reboot, to acknowledge the missing parts and the errors that were made, and then make a commitment to solve the problem. And we also needed a way to make sure the teams remained on track. It’s so easy to get caught up in details, and it’s hard to take a few steps back to look at how the engagement is going at a higher level. We recommended they put a quarterly meeting in place with executives from both organizations to review progress and to provide a way to escalate issues that remained ongoing. See, without a recurring meeting like that, there’s no opportunity to go in and resolve these things that come up from time to time.

Tom Cooper 24:35
And finally, we talked about getting to reality what was really happening. When it came to the roadmap and the features we needed a process to align Lotus text thinking with future syncs ideas. We needed to get Lotus tech to be direct and honest about what was a priority for them and what was literally not a priority. We also recommended a number of tactical improvements in the way they did testing in tracking work and reporting and follow through just to the overall

Tom Cooper 25:00
offer management process.

Tom Cooper 25:02
The end of the day, we were able to rescue that relationship and progress was able to be made, thanks to the commitment of both sides to be willing to discover and fix their issues. And without that willingness, without an investment in that commitment to saying, Hey, we own part of responsibility, this problem, we need to fix it, the chances of that coming out in a good way were very, very low. In this case, also, both sides were unhappy with the other because neither of them was making the kind of progress they wanted to make. And they weren’t clear with each other about what they defined as success or how they were measuring that success. And as a result, each was plowing forward on what they thought the goal was. But because of that disconnect, each side was unhappy with the other. These things happen all the time in organizations, because we get so busy doing the work that we lose track on whether we’re actually still headed toward the correct destination.

Tom Cooper 26:00
How are things going with your team and your projects? These kinds of problems are fairly common. And the good news is that once we begin to look at some of the underlying causes, we often are able to make significant progress and quick progress as well. Once we began to work on building trust, and making and keeping promises that everybody could agree on, the details of things got felt a lot better. And the way things feel on a project is very, very valuable. It affects morale in a huge way. If I think your intent is a bad intent, I’m not going to trust you. I’m not going to clap collaborate with you, I’m not going to want to help you. But once we begin to start to build that shared expectations and shared trust, all of a sudden projects get a lot healthier and a lot happier.

Tom Cooper 26:52
Are you facing a software horror story? I’m your guy for action oriented workshops and executive coaching for technical leaders and their teams. Thanks for joining us for this latest episode in the software horror stories Podcast Series. For more information about brighto Group please check out www dot bright Hill group.com that’s www dot bright Hill group.com for bright Hill group. I’m Tom Cooper.

In this Episode

• Why your prior success may not scale
• Cloud is great – except when it’s not
• Who is your customer? Are you sure?

Transcript

Tom Cooper 0:00

Tom, I feel like I have a fire hose that shoots money. And I just point it at any problems I find. In today’s software horror story, we’re going to talk about what happens when a smart guy, a really smart guy creates a startup that gets acquired by a large firm, and then they put him in charge of a rewrite of their cash cow from one of their business units. Sounds great, right? This guy could bring his expertise. And while he had been held back by tight budgets and small staff before, now, he could have the resources to build something really amazing. What could possibly go wrong. I reminded of financial planning expert Dave Ramsey talking about how access to more money often borrowed money can lead to failure. In fact, at one point, he was a real estate developer who was worth more than a million dollars on paper until things took a wrong turn. And he spent over a year on the journey of losing everything to bankruptcy. One of the things he learned in that process was how having more money can amplify bad decisions. This worked out for him one time when he did a big promotion related to a quick service restaurant. He had a children’s book written that he wanted to distribute his kids meal prizes, and this was going to help with kids learning about money and help promote his business a win win, right? Unfortunately, the big prize that the kid in the book was saving up for was a special trip, a trip to space camp. While Space Camp sounds amazing. Dave’s team didn’t realize that the term Space Camp was a trademarked term. And he learned that after he printed a bunch of books, and he had them in distribution. That’s when he got the cease and desist letter from the trademark and copyright holder. And he had to recall every single book. And what was his take away. It was a dumb mistake. But since he was using his hard earned cash, and not investor money, or bank loans, the mistake was a lot smaller than it could have been. Sometimes having access to less money is actually a good thing. So let’s turn our attention back to Jamie, our CTO, and his so called firehose of money. As I mentioned, Jamie had been frustrated for some time because he had grand plans for expansion. But he’d been held back by a lack of ready cash. After his company was acquired by a larger company that had plenty of money, he had the freedom to overcome those limitations. And boy, did he ever. Jamie had made investments in infrastructure and design in engineering. He oversaw the technical details of the system, and he worked with the engineers on creating something amazing. It’s a great plan, right? smart guys sufficient resources, strong engineers, customers with significant problems and money to solve them. It’s a perfect storm. Except it wasn’t until the storm was a big one. But it turned out not to be so helpful of a storm. When I met Jamie, he was in the middle of overseeing the creation of a really cool product. One that was a cloud based rewrite of a dominant on premise tool. I go into the cloud meant that Jamie’s company could deliver a comprehensive solution to less tech savvy customers. Customers without a deep bench have to have technical skills. And it also meant that customers using the old solution could potentially convert to the new one relatively easily. And that was one of the problems began. Bill Gates famously once said, Success is a lousy teacher. It seduces smart people into thinking they can’t lose. Success is a lousy teacher, it’s seduces smart people into thinking they can’t lose that Jamie had been successful. And he was a very capable CTO, Jamie’s new boss in this arrangement, Hank, he had been super successful in selling that on premise solution to customers. But Hank and Jamie, they were in that group that Bill Gates described over their careers, they had been successful, and because of that success, they believed they could not lose. Thanks, secret sauce was creating a tool that was very useful for his customer base. And Hank figured out he could get to a higher price point if he offered customization to larger companies. Hank solution at the time was on prem software, and he had enough prospects ask him for custom options that he began to offer them some level of customizations. Over the years, what happened was that Hank ended up with lots of variations of his software, customer a wanted one thing, Customer B wanted a different thing. Hank had his developers add those customizations. Now, the problem with this approach is that unless you are very careful, you end up with a lot of incompatible software. Haha, thanks said no problem. We will make everything configurable. That way we ship the same source code, every customer problem solved. And he was right. He could do that. And the unexpected consequence was that there were more configurable options than anybody could ever test. And that meant that the customer was the one who discovered price problems and errors. With a small number of clients, maybe that was manageable. But unfortunately for Hank, his sales team found a lot of enterprise customers who wanted custom options that were specific to their particular needs. So Panke directed his engineers to build exactly what the customer asked for. And the support team spent endless hours troubleshooting, the predictably untested and buggy software. And now Hanks team was improved. the acquiring company brought in Jamie as CTO and empowered Hank and Jamie with the money to build that replacement for the legacy on premise enterprise software. And the overall overall idea was that they could potentially offer a SaaS product, one that could compete with other products, targeted not an enterprise customers, but at a large volume of mom and pop companies who would buy the software like an appliance to manage their business. Now, let me just ask you to pause for a second here, can you think of some differences between enterprise customers, and Mom and Pop customers, having worked in large and small companies over the course of my career, I can think of many differences. Small business operators are so busy trying to keep the business running and keep customers happy that the last thing they want to do is think about or make technology decisions. They need a product that’s as reliable and predictable as a toaster. And by the way, as cheap as a toaster. Now, if we keep that analogy going, enterprises already have a bunch of toasters and toaster ovens and regular full size ovens and commercial ovens to manage. They have experience in weird wiring ventilation, electrical demand and processes to manage all these different types of cookers, we’ll call them. And because of the variability of the world they live in, enterprise leaders want to be able to set up the new toaster like all the other toasters, so they asked for, or they demand that vendors deliver specific options that work with the way the other systems work. There’s an expression that says he who chases two rabbits catches neither. In this case, Hank and Jamie were hoping to chase enterprise and small business customers, SAS and custom software co customers. And you can imagine that they were having some difficulty finding their minimum viable product. And you would be right. It seems like no matter what they did, they could not seem to get to 100% of something. So they added resources, more team members, oh my goodness, the classic blunder. In 1975, Fred Brooks wrote the book, the mythical man month, and he discussed an idea we now call Brooks’s law, which says adding manpower to a late software project only makes it later. Fundamentally, late projects are not late because of a lack of engineers. Late projects are late because of poor planning, and poor feedback loops that raise problems to the awareness of leaders so that leaders can make things better in a timely fashion. Late projects are also late because of poor scope management, incomplete thinking, followed by incomplete learning, and an incomplete replanting. They’re late because leaders assume they know exactly what’s needed. And they don’t think they need to learn about what market customer or user needs, and how those might be different now from what they believe the customer market wants. And by the time our team was engaged, this project was just like the ones described in the mythical man month late and adding resources that made it later. And later, it’s been said that what got you here won’t get you there. And the reality is that what had helped Hank and Jamie be successful, really was getting in the way of where they wanted to go. For Hank, success had come with infinite customizations for enterprise customers. For Jamie, his big product had been less customized, and he’d been able to get a lot done with a small team. And unfortunately, for Hank and Jamie, they’d never created a SaaS product for unskilled users before. And they also never created a one size fits all product at a low price point either. They also hadn’t managed engineering teams anywhere near as large as they now had. What have worked for them in the past, simply would not work for them now. And going back to Bill Gates warning about seducing smart people into thinking they can’t lose. For this President and CTO. It was worse. Not only did they believe they could not lose. They believe that losing was an indication that the person who lost who was a loser. At one point, we had an opportunity to gain some insights from an influencer in the organization who had experienced a pretty public crash. One of my favorite authors is CS Lewis, who famously wrote The Chronicles of Narnia series, and Lewis talked about learning from experience. Lewis said, experience that most brutal of teachers, but you learn my god do learn this influencer had experience that Hank and Jamie just didn’t have. He had learned painful lessons only available through life experience. When I suggested that we might want to sit with him and ask some questions. Hank and Jamie were disinterested. In fact, Jamie literally said, Tom, that guy’s a failure. Why would I listen to him?

Tom Cooper 10:00

Now, the most comprehensive and complete lessons do come from personal experience. But the quickest learning can come from the experience of others. So Hank and Jamie were successful, they each had a pattern of success, which was preventing them from being curious about what might be different today from what had made them successful before. And when the project was running too slowly, they decided they needed more engineers, a lot more engineers, at time won’t permit me to date until all of the story and perhaps I can come back to Hank and Jamie in a future episode. But for today, I want to talk about our experience using cloud computing, there were a couple of cloud computing problems. Now it’s tempting to point a finger at the compute vendor and talk about how crazy expensive cloud is. And Cloud can be crazy expensive, it doesn’t have to be but it can be. So think about cloud computing. I remember a time in my youth when setting up an Internet facing system meant that we needed to have an internet connection, a firewall or router, a load balancer, capacity planning to tell us how much bandwidth CPU and memory we would need licenses for front and back end software operating systems, database capacity and licenses, facilities to house equipment, power cooling, and an operations team to oversee and run the gear. And in each of those functions, we needed an expert who could help us determine the exact specifications and help configure the thing, we’d have to spend a good amount of time planning and coordinating, and then installing and configuring and testing and then developing and knowing what hardware we needed. Even after we know that we didn’t have to source it, and it would take time to get the boxes shipped to us, then we had to coordinate with the facilities people to get it racked and powered. But today, today, any developer with a credit card can get started with a compute system at any time in just a few minutes. With a few well chosen clicks on the virtual configuration panel, we get speed to market options for scalability, flexibility, and we can ramp performance up and down without having to put new equipment in our racks. Not only that, but we can stand up new environments quickly. And we can prototype and test in virtual spaces, without difficulty in that part of cloud. That part’s amazing. What’s interesting is we still actually need all of those broad skill sets, skills, insecurity, capacity, planning, databases, licenses, all those things and more. And that developer with the credit card, he or she may have some of those skills, but probably not all of them. Without the planning and expertise, you’re going to stand up services you don’t need. And you’re going to forget ones that you should have. Plus you’ll probably have services misconfigured as well. It’s a risk. Now. That’s before we talk about computing and network and storage charges. I want to be clear, I’m not anti cloud, it’s a great option in so many situations. When we saw that Hank and Jamie had lots of challenges with cloud. I’d like to take a break here. When we come back, I want to talk about some great things that happened in this project.

Tom Cooper 13:00

There is no greater business investment than investing in your team. You’ve got an exceptional team, but you’re still struggling to unlock consistent, outstanding performance. You know, it shouldn’t be this hard to deliver value to your business. And we do too. Even the best team struggled to build a culture of collaboration and high performance without clear communication and write planning frameworks. brightfield group we deliver leadership coaching that empowers you and your software team toward peak performance. Our personalized coaching solutions help you build trust and increase efficiency across your development efforts. Head on over to bright Hill group.com To schedule a call today. So you and your team can break free from this rut of untapped potential, and start delivering the high business value you know you’re capable of.

Tom Cooper 13:54

Welcome back. Now we’re gonna talk about some good things that happen on this project. One of the biggest benefits that Hank and Jamie had was their DevOps crew, those guys did an amazing job of creating automation and setting standards that made it possible to see and know and track virtual assets and to get them consistently deployed. Having team members who can build solid infrastructure that’s critical. And having a DevOps philosophy where there are standards and where the system supports developers by helping them do the right thing that’s helpful in reducing overall complexity, and increasing consistency between environments. And that saves tons of debug time. And if you can automate those things, Wow, that’s good stuff. It’s challenging because none of that stuff is customer facing. And as I like to say, no one cares about the infrastructure until the toilets don’t flush. But fixing the drains after the house is built is a lot harder than designing it. Well, to start with this DevOps team, they had done an excellent job of creating automation and standards for how things were configured and deployed. They had created an elegant solution, and when their process pushed back on developers, it did slow Have them down a little bit. But it helped the developers be able to focus on the most important stuff and not get distracted by the infrastructure. And we often think about automation as the last part of the process, after we have the application working. But we might have made decisions in the design that make automation really hard and changing that at the late stages of the project. That’s challenging. In this case, the DevOps team thought of all kinds of things that helped organize their large virtual setup. And one thing that nobody thought about was a business process that would put limits on what could get deployed. After all, if you need something, why wouldn’t you let the developers create it and start using it. And that works just fine. Until the AWS Bill shows up. In the previous month, Jamie’s team had blown through the annual budget. For cloud spend, every minute of delay led to additional overspending, they all of a sudden found themselves in a crisis that needed immediate attention. That day, all meetings were canceled in favor of an immediate all team huddle, we got to shut everything down right now. And the whole group began logging into the console to kill things. It was a virtual server bloodbath. We’re in crisis, no sacred cows kill everything. And all day long, the team feverishly worked to kill dev and test systems, they things were shut down like crazy. Anything that was obviously not production that had to go right now. I want to point out they had client facing production infrastructure, and they had paying customers who were using services. But even in this situation, anything that was production was subject to review. Now, as happens from time to time, some services had been over provisioned. There were some cases where auto scalars had ramped up to many instances which were essentially idle and chewing up budget for nothing. How do you move with both urgency and with care? Well, let’s just say in this situation, mistakes were made. There were some self inflicted wounds. I do remember there was one primary production database server for a cash cow for the company. And that server had been provisioned on the largest possible server that Amazon offer had the most memory that Amazon offered the most CPU. If there was ever a case of throw hardware at that performance issue. This database service was the poster child. And man was that server expensive. The box was crazy costly. And now that a bunch of other services had been shut down, that one stuck out like a sore thumb. And even after shutting down all those services, hundreds of them, they were still over budget, everything was somebody to review. So we began to ask, Are you sure you need that much capacity for that server? The answer was, Well, no, but we’ve had performance problems before. And when we added capacity that helped. Alright, let’s get to work on analyzing it. Let’s get some profilers and tools and evaluate to identify bottlenecks. We don’t have time for that, let’s just reprovision on less hardware right now. So they did, they shut down the database, they cut the hardware in half and started the database back up again. And they waited, what was going to happen? Nothing, nothing at all happened, the application was just fine on half as much hardware has been provisioned. But of course, they were still way over their annual budget, and even half the size, it stuck out from a cost point of view. So they decided to cut it in half again. And you know what happened? Nothing. That app was perfectly fine on a database that had 25% of its original capacity. This is one example of how not having that discipline of capacity planning. It leads to cost overruns and difficulties. It’s important to have that as part of your system. One of the things that we encourage the team to do, as we went through this experience with him was a creative learning model that made time to do retrospectives. I’ve been involved in Agile software development, since I don’t remember probably 2003. And I’ve worked with all sorts of models for development within that ideology. And one thing I’ve learned is that because Agile is a philosophy more than it’s a set of practices, it’s common for teams to use the same agile terms but mean entirely different things. What we know is that high performing agile teams have high safe, high psychological safety. And all team members feel empowered to speak up and to admit when they failed, without fear of retribution for revealing their imperfection. A good stand up or a good retro is very similar to a lean approach of going to the gemba going to the place where work is done to see the work being done. Because the ones who are doing the work know exactly what’s going on. And no one else knows. No one else knows. Healthy leadership teams know they need to listen and learn from the ones who have the information and that’s the people doing the work going to the gemba and the entire team needs to hear about impediments and failed experiments to help make the entire team better. Unfortunately for Jamie and Hank, when they said retrospective, they didn’t mean anything like that at all. retros with Hank and Jamie tended to be the leaders summarizing the successes during the previous work period. And glossing over the failures, while looking for ways to point blame at somebody outside their department. I’m sad to report this is a pretty common practice in my experience, teams commonly are not created and maintained in a way that team members feel safe to share, or where leaders listen and learn from their people. Back in the day, I worked for a boss who believed that failure was bad and failure needed to be avoided. When we made mistakes that could not be hidden from other departments, she demanded, we write long post mortem reports describing our mistakes. But because we never use those reports to help the organization learn and change, we concluded that her purpose was simply to punish us for failing. Rather than using the lessons learned to help make us better at our jobs. She pushed us to take fewer and fewer risks. Productivity closed to a so called Safe crawl, filled with sign offs, double checks and approvals before any changes can be made. I just don’t think that was delivering value to the organization. And when it comes to retrospectives, one of our goals is to learn, we want to learn as a team. Because together we’re learning to create great software for the organization. President Lyndon Johnson said, you ain’t learned enough. And when you are doing all the talking, all too often leaders are talking when they should be listening. It’s simply not helpful. Instead of embracing humility, and recognizing that what we do not know is far bigger than what we know. The focus is on what we did right and how if things went wrong, there was someone other than me who was taking the blame. These behaviors led to a culture of throwing other teams or other departments under the bus. As long as there was a way to point attention to the errors of others, each leader was secure in their role. Also, there wasn’t an openness to discussing things that needed attention or even exposing problems so they can be discussed. So when we suggested they should make some time as a team to reflect on how we had managed to deploy enough services to be 10 times our plant capacity, that was deemed wasteful, because there were more urgent problems to be solved. I saw a clip this week of Elon Musk talking about challenges with engineering and he said one of the most common engineering errors is to optimize a thing that should not exist at all. Interestingly, I can recall hearing similar things from Peter Drucker and even computer science luminary, Donald Knuth. Knuth called that problem one of premature optimization. He said, if, if we believe we had defined the problem too early, our solution no matter how clever it was, would simply not solve the problem at all. Knuth argued and I think must does, too, that we need to keep digging until we really understand the root cause. Unless we got to the root cause of how we managed to deploy 10 times the expected services, it was likely to be repeated in the future. But the crisis now was over the spin was back within parameters, and the team could focus elsewhere. And they did, they focused elsewhere for about three months until once again, the AWS spend was far greater than the budget. And we repeated the entire fire drill all over again.

Tom Cooper 23:32

There was another issue we hadn’t addressed. I mentioned before, there was a disconnect between SAS and custom enterprise software, there was another factor we hadn’t considered. One of the big pushes was to compete with low cost providers. By giving an all you can eat low price point. I’ve heard it said in the race to the bottom, the big problem is somebody wins. By definition, the person selling at the lowest price is very likely to have the lowest profit margin. The problem of the thin profit margin means that there’s a low margin of error in your business to in this case, thinking specifically the fixed fee SAS customers. In order for the app to be sticky. The company needed customers to love it and to use it a lot. And here was the big disconnect. When users use cloud services more the cost for hosting that service goes up. If you advertise an all you can eat pricing model, the more customers use your services, the higher your costs, and the lower your profits. The natural incentives are flip flopped. Your the best situation to be in is when your incentives match the customers incentives and services like Stripe or PayPal, will generally charge a small fee plus a percentage of sales when transactions are at a smaller dollar volume. When retailers processing a bunch of small dollar purchases, unless the payment provider is getting a base payment in addition to the percentage of sale, they’re going to lose money on every transaction. The model that stripe and PayPal offer while I’ll confess I don’t love it gives their customer an incentive to have higher dollar transactions in order to get better rates. There’s an alignment of interest between the customer and the service provider. For this product, it was the inverse, the more the customer use the service, the less money the software company would make. In fact, as a practical matter, the beta sites were actually money losers. Ever hear the expression, we lose money on every transaction, but we make it up in volume. That’s exactly what was happening here. This is a strategic disconnect. It was caused by product management, not thinking this through before coming up with their business plan. We talked about how Hank insisted that it just about that every setting should be configurable. We also started to look at the workflow to see if there was an opportunity to reduce cost per transaction. We sat down with some of the engineers and asked, Tell me exactly how this process works. And we began drawing flow diagram showing all the round trips between the endpoints and the cloud servers. As we began to share publicly what we were learning, with each review cycle, we would hear well, that’s close, but you’re leaving something else out. And we went through a bunch of iterations of these diagrams before we had the opportunity to connect with Jamie. We told him that we had discovered for every single transaction, there were more than 25 round trips between endpoints in the cloud. And he said, That’s not right. That’s not how it works at all. But here’s the deal. Jamie’s a super smart guy, and he had tried to touch all the parts of the system. And I bet the last time he had looked at it, he was exactly right. But this system had grown beyond the point where any one person could really understand the whole thing. And there was no way in spite of his 80 plus hour work weeks, that Jamie could keep his finger on the pulse of every part of the system. He thought he knew how it worked, and he believed it sincerely. Yet, he was sincerely wrong. We showed him what we had learned. And then we kept digging. The last I recall, we had catalogued 47 round trips of network data between the client and the cloud, every round trip affected performance and it impacted costs. That meant it was virtually impossible to be able to make any money at all running this app with a low price point and an all you can eat service plan. I want to emphasize this team was made up of very smart people, they had immense talent, and yet they miss some critical items that lead to strategic failures. from a functional perspective and application, it was pretty cool, it cleverly solve real world problems that customers were dealing with every single day. In a future episode, we can explore some other aspects of how engineering was managed the complicated things, but for today, let’s recap some of what went wrong here. From today’s discussion, I think we can say there were four main areas that led to the challenges in today’s episode. First success was a lousy teacher, it convinced these smart people like Hank and Jamie they couldn’t lose. Number two, leaders thought the path to success at great scale required them to work harder and more repeating the lessons that had led them to success at a smaller scale. But Einstein said the problems of today can only be solved at a higher level of thinking than those which created them. In order for Hank and Jamie to be able to tackle these new problems they were facing. They had to think at a higher level, they needed to continue to grow their skills. But they had convinced themselves that was not necessary because they’d already learned the critical lessons needed to get them to future success. Man harder faster and more work was needed. At least that’s what they thought. And as a result, there was a focus on what was urgent, not what was important. With the endless work hours seemingly limitless travel and continual fire drills, leaders got too busy to understand what was happening at the edges. It was easy to get sucked into the details and not lead at a strategic level. And that was holding engineering back. Number three, when it came to delivering a product, there was a lack of clarity. Are we building a Sam’s Club or a Nordstrom? How would we measure success? What happened internally when leaders had conflicting definitions of success? And how would we resolve differences. And number four, DevOps had the philosophy and the promise of scalability and greatness. But that thinking and leadership that were present in that team was not reflected elsewhere. Areas like cost management capacity, planning and security were often overlooked. And there was little to no desire to create repeatable and scalable business processes. I have to tell you, this was a tough slog on this project, it was hard to be positive and optimistic. There were a lot of smart people who worked a lot of long hours trying to create something great. Unfortunately, the vision of the all you can eat low price offering well, it didn’t make it to market. The enterprise side did because there was more margin in those clients. But a lot of that effort they put into trying to launch that low price thing just didn’t work out at all. It’s a shame. I liked those guys, even if we often disagree.

Tom Cooper 29:46

Thanks for joining us in this latest episode in the software horror stories Podcast Series. For more information about bright Hill group, please check out www dot bright Hill group.com that’s www dot bright Hill group dot up for bright Hill group. I’m Tom Cooper and we’ll see you next time

In This Episode

Frank was an “idea guy” with a big dream. What happened when he depended on Oliver to help him get to launch?

• How good Product Management could help
• Why adding a project manager didn’t solve it
• Key mistakes to avoid

Transcript

Tom Cooper 0:00

Oliver said, I can do the work or I can document the work, pick one.

Tom Cooper 0:10 Hello, and welcome to another in our software Horror Story series. I’m Tom Cooper and today I’m telling the story I call live in the dream. As we enter act one of today’s three act play, we meet Frank. Frank was a big guy with big ideas. He was a visionary who dreamt of building a high impact product that would change the way the world thought about charitable giving, and could potentially make him wealthy in the process. And Frank had been dreaming of a big launch for a long time. And over the years, he managed to raise some money, and each time he raised money, he would make some incremental progress toward his overall plan. And each fundraiser raising round, he managed to take steps to get closer to making that a reality. And by the time we met Frank, Frank was on about iteration three of his dream. At this point, he had contracted with a team to build it for real. But he was frustrated because he wasn’t seeing the progress that he hoped for. And he asked us to come alongside the team and figure out what was working and what wasn’t working. Now, the first step was to understand what was real. And what was vaporware. And one tool we use for that is a product we call our foundation assessment. In the foundation assessment, we look at elements of how the business is actually connected to the software, what processes and methods are in place as they’re producing the software, and how the technical resources are managed. And our goal is to come up with a comprehensive risk analysis that helps prioritize which of these 15 key areas need the most attention to improve those outcomes. And we’ve seen that teams can sometimes miss out on even seeing important risks, and then they can fall into a trap that can raise costs and potentially threaten their project success that had happened here. We began our foundation assessment with Frank Frank, Lieutenant David and the technical team. And what did we learn? Well, one thing is Frank had the vision but the technical stuff. Well, Frank relied on others to make his vision a reality. And he didn’t really know what it took technically, to build a tool that could credibly be deployed and run and operated in a financial technology world. Now that should have been our first clue. And I have to say this story got more exciting from there. Now, we began to work primarily with David and Frank had hired David to be his right hand man, somebody who could handle the details. While Frank focused on the vision and the fundraising. David was a smart guy, and he had some experience in software development, but not a lot of experience in bringing a new product to market. David was doing the best he knew how but there were some limits on what David could make happen based on his experience and the tools that were available to him. And if you remember that Frank was bootstrapping his dream, he had raised some money, but it was angel investors really, he looked for ways to cut costs regularly. And one of those ways that he cut costs was he persuaded a friend who had some technical folks to donate some of their time in exchange for some equity in his in his venture. And that’s where we meet Oliver. Oliver is the tech lead. He’s responsible for implementation of Frank’s big idea that Oliver has support, because he’s got a group of offshore developers that are half a day’s worth of time zones away. And Oliver’s job is to come up with a scalable technical design and then guide that team to build the software. I have to say at this point that Oliver is no slouch than in an interview situation, it is clear that he really does know his technical stuff. He can think well, and his ideas that are really technically smart. There’s nobody better to see the talent oversee the technical stuff, right? Well, did I mention that Oliver was only available part time for this project? Yeah, that was an issue. Oliver worked for Frank’s friend and Oliver his day job was overseeing the operations for Frank’s friend’s company. Any operational challenges that came up meant that Oliver was missing an action until the crisis was over. And since Frank was paying his buddy and a combination of equity and a small amount of money, revenue generating work for Oliver, well, that went to the front of the line, and everything for Frank and David went to the back of the line. And Oliver, like a lot of deeply technical folks is mostly concerned with whether he understands the design and the implementation, and not nearly so much focused on documentation or even communicating with others about design or how things actually get built. As we dug into the realities of the project, we discovered that when Frank had worked on version one, and then later on version two of his dream, he had been using a locally installed version of a ticketing system when the money was starting to run out. To his credit, Frank did create backups of the data. And he did pass those on to Oliver for Oliver to use, so he would have a frame of reference for what technical work had gone before. Unfortunately, for Frank and for Oliver. It had been so long since they had done the previous iterations that the version of the ticketing tool Frank had used before was now so old, it would not run on a modern operating system. The ticketing systems application code was so old that it was no longer available from the vendor either. So Oliver was tasked with finding a way to make it work. His solution, which was creative was to stand up a virtual machine, running the old operating system and restore the application code and the data to that VM. It was a good plan because it allowed Oliver to see the tickets and the status of previous work in progress. But it was an island of data, Oliver didn’t want to put that VM on the public network due to operating system security issues, because the OS was unsupported. And his offshore developers didn’t have access to that system either. And they couldn’t use it. So Oliver didn’t want to make his daily operations dependent on an unsupported locally installed copy of the ticketing software, which that was the right decision. So Oliver stood up a cloud instance of a new version of the ticketing system for his offshore developers to use. And he did make some initial attempt to import tickets in history to the cloud version, but decided that mountain was too high to climb. So he ended up treating is the old way and the new way and use the old ticketing instance as a reference for himself. Nobody else had access to that. Now, as long as we’re talking about ticketing, I’ll note that Oliver followed a pattern I’ve seen many highly skilled engineers embrace, he didn’t create or update tickets related to work he was doing. I call it a pattern, but I think it’s really an anti pattern. When your most senior people don’t document planned work status, a work in progress or completed work, their work queue and their work, progress is a black box. This problem is made much worse when people get busy. All too often stressed and frustrated, senior engineers will say something like I can do the work, or I can document the work pick one. And because the senior engineers, the only one now who can do the work, we tolerate that lack of documentation from them. I can’t stop myself from pointing out how deadly This is to long term project success. I remember, in college, I worked with a guy who was so smart, he basically breathed algorithms in and out, he would write concise code that almost always would compile without errors. It was efficient. It met all the technical requirements. And it demonstrated his utter fluency in the language as he wrote, It was impressive. I remember I was in awe of John, and on the surface seems great, doesn’t it? It’s great to have efficient and accurate code. But we have to ask ourselves, what problem are we solving? are we solving the problem of can John write that code? Or can John make changes to that code? Actually, no. It’s true, that code doesn’t rust. But it’s also true that the details of your business problems always change. So the problem we’re solving is not can John write that code for us. But the problem we’re really trying to solve is can we build a system that makes business requirements today, and can be modified in the future by a competent, mid level engineer, because we can’t afford to pay John to constantly be maintaining the system. For what we pay a developer like John, we really need him to work on figuring out the hard stuff, not making minor changes to the small parts of an existing system. So that leads us to this point, brilliant technical developers and architects write simple code that elegantly solves the hard stuff. They read it in such a way that young engineers and mid level engineers can understand what they wrote and can make changes to it over time. This is a profoundly valuable thing. That’s all too rare in our software development. engineers who are smart, but don’t do that they’re short sighted. Because if you’re the only one who can modify the code, you’ll be doing that job forever.

Tom Cooper 8:40

This idea of writing elegant code that is simple was a level of thinking that was higher than Oliver was willing to apply to this problem space. And it was a huge limiting factor for Frank and for his code based. One last comment about John from college, I really liked him. And we were working on a team project he he helped me overcome a major limit in my thinking about how to write in machine language. John was super smart, and I’m really glad we work together. He helped me get better my math classes because he raised the bar in class, let me assure you. And I also remember that one of my work study jobs as I was trying to work my way through college was in as a computer lab at Proctor. Yes. This was in the dark ages when even computer science students couldn’t afford to own their own computers. You young uns need to be super grateful for Moore’s law. But one night I was in the lab. I saw John he was stressed he was tearing his hair out. I’ve never seen him struggle that much of her work and certainly not like this. After a while I went over to John and said, What are you working on? I’m sure that I thought it was some complicated graphical rendering algorithm or something. Yeah, back in my day, we didn’t have GPUs. We had to learn how to do 3d transforms using math. But I digress. So John, what are you working on? And this brilliant young man, the one who intends updated me with his computer language fluency said, Tom, it’s this English paper, it’s killing me. I had to laugh. John was so at ease with math and computers that it never occurred to me that everything wasn’t simple for him. That was a big life lesson for me. So let’s go back to Oliver. If given the situation Oliver was in, I really don’t blame him for the choices he was making. Ultimately, he had a lot of things going on and he was being measured on is it done? Rather than? Is it done in a sustainable way? Or even? Are we asking you to do the right things here? His bosses were only asking him this terrible question they would ask how much longer or is it done yet. And of course, given those pressures, Oliver is going to shape his work into small tasks, he can get done, so that he can report progress back on things being done, and his bosses will get off his back. So in addition to the operational support work, which was all of his day job, in his work for Frank Oliver had two categories of work, work that he had to do himself and work that he needed to oversee, that would be done by other people. In this case, Oliver needed to define work to be done by the offshore team and then incorporate their work into his work as a part of getting stuff done. Now, we’ve been talking so far about the work that Oliver was doing, personally, I want to shift gears for a minute and talk about the work he was supposed to be overseeing on the offshore team. On the plus side, the work they were supposed to be doing was documented, it was in a ticketing system, and progress was being logged, at least at some level in the tickets. But just because work is being tracked in a ticketing system, that doesn’t mean the work is being done efficiently, or the work is being done well. Someone has to have the job of organizing the work, assigning the work, checking the work, to make sure that it does what it’s supposed to do, and that it’s written in a way that can be maintained and then releasing and deploying the code. And Oliver clearly did not have time to do that. As time went on, Frank got more and more frustrated with what the what he saw as the lack of progress. And he complained to his friend about it. And so Frank’s friend brought in a project manager to address this problem. Now, Judy was a competent project manager with a track record of getting stuff done. Unfortunately, Judy didn’t have experience running software projects. And she tended to work in a waterfall methodology rather than an agile approach. This is linked to the problems we could talk about in terms of product management. But for now, I just want to talk about project leadership. Now, to her credit, Judy did her best to try to document the work schedule, the work and ship work. She knew her limits, and she didn’t try to get in the weeds of the software development details. And that was smart. But it also meant that she didn’t fully understand what blockers existed or the potential implications of the problems that developers were describing in those meetings. What she did bring was a sense of consistent process and reporting of progress. And that was very valuable, I think it’s easy to overlook the importance of seeing what’s going on and recognizing where things could be getting off track. Now, the offshore partner, provided a team of a BA, eight mid and low level developers and two development leads. The challenge we ran into here is that it wasn’t clear where duties responsibilities stopped and where the development teams responsibilities started. And that lack of role definition meant that stories can be assigned to a developer where the story wasn’t developer ready stories that had unclear requirements or that could not be tested. Now, the addition of the BA to the team helped somewhat, but the BA was hampered by product management challenges as well. And because of this fuzziness in responsibility, it became harder and harder to really understand how much progress was being made toward a shippable product. At this point in our story, I think it’s time for a break.

Tom Cooper 14:09

As a leader in a software team, you know, you want to help your team performing at the highest level. And you know that with the tools we have, it’s never been easier to write software than it is today. But it’s still hard. I’m Tom Cooper, and for the last decade, I’ve focused on coaching and training software leaders and their teams to help them make the small but important improvements to help them perform as one team and to achieve their potential. How much better would your results be if your team communicated clearly managed conflict well delegated work that stayed delegated, and then created an executed on great work plants. It shouldn’t have to be this hard to write great software. And the good news is it doesn’t have to be are you ready to take the next step? Head on over to brightfield group.com So you can learn more

Tom Cooper 15:06

Welcome back. So we were talking about who was on Oliver’s team. And one of the challenges was role definitions. Now, for those of you who were paying very close attention, when I outlined the roles for the team, you might have noticed that one of the role definition problems was a role that wasn’t present at all. On this team, there were no QA team members, none. And the fact that there were no QA resources meant that the project team defended on developers to deliver what the business people intended when they asked for the features. I gotta tell you, this is not a winning strategy. QA teams, when they work, well do all sorts of things with their goal being to break the software. Unfortunately, developers write code and they assume that users will follow the happy path. And when that code breaks developers often respond with Why would anybody do that. And that reminds me the joke about a QA engineer walking into a bar, he orders a beer. He orders zero beers, he orders 99,999,999 beers, he orders a lizard, he orders minus one beers, he orders a bit of our first real customer walks in and ask where the bathroom is the bar burst into flames killing everybody. developers don’t think like QA engineers. And the fact that Oliver was overseeing a team without QA meant that a lot of bugs are going to slip through the tests of the developers. So Judy was pushing the team to be accountable to move things forward and show progress, that progress on individual tickets can be super valuable. But that’s only one part of the puzzle. One of the things I reinforced for teams is we need to be focused on getting 100% of something done. If we almost finished a bunch of stuff, we have nothing we can ship. And it’s critical that whatever we are shipping delivers value on its own. Which leads me to the topic of Product Management. The purpose of product management is to understand the path to value, prioritize desired features, and work with engineering to make sure that what they deliver is 100% of something that delivers value to the customer. Far too many ships of developers have been wrecked on the rocks of shipping a lot of stuff that didn’t add up to value for customer. As we worked with Frank Oliver and David, we realized Another potential problem. We had a huge backlog of features, but there was no comprehensive high level plan for launch. We didn’t know how many more Sprint’s before we could ship 100% of something, and certainly anything more than a proof of concept. And this reminds me of the idea of a launch countdown in the American space program. The flight director would go down a checklist of all the departments that needed to sign off, go for launch, and then got confirmation in order to blast off. In this situation. We didn’t even have a list of departments much less than evaluation of whether the components were ready for launch. See, Frank, David and Frank did outline the major milestones but not in detail. And it’s a far cry from a developer’s assurance that code works on the happy path to the place where we have a robust, supportable application in production. The other challenge Frank was dealing with was that he was really married to his friends investment in the project. Frank’s team didn’t control any of the systems that held their intellectual property. And as Frank began to lose confidence in his friends team to deliver what was needed, Frank was in trouble. If his friend decided to hold the code hostage, Frank would be powerless to do anything about it. And it’s right about now that we turn the corner from act one of today’s play. In Act One, we worked to find a way to make the team healthy and productive. And as we worked with Frank, and David and Judy and Oliver, Frank began to see that that mission was not one that could be successfully completed. So now we turn to act two, how do we safely help move the software project from Oliver’s leadership to another team. I’m a big believer in distributed teams. I’ve seen a ton of successes with nearshore and offshore development. But it’s critical that decisions about your product and the control the tools that hold your intellectual property, those things must remain in your hands and not in the offshore developers hands. And next episode, we’ll talk about some of that. But after we worked with David Oliver and the rest of the team for several work several weeks, Frank asked us to shift our mission from help this team turned around to help me figure out a way to get this to another team without killing the project in the process. But then, on its own brought a whole new wave of challenges. The good news is that Frank and his friend had a good relationship and they came to an agreement that Oliver would help us get to an orderly shutdown, and that Oliver would provide documentation.

Tom Cooper 19:45

I’ll note here that Oliver was relieved because this was a side project for him anyway, and he definitely wasn’t used to leading teams collaborating or communicating with others to build bigger projects and he could build on his own. However, there is a long distance between I agree that you need documentation and actually writing the documentation. So we entered phase two that orderly shutdown with Oliver walk while we figure out a path to restart development with a new team. Now this phase two involves two major areas of work. First, we had to gain agreement on a project plan where Oliver and the rest of the team could provide everything needed to move the project elsewhere. Now, that’s a delicate dance because we needed to convince Oliver to do things for us that he never believed were needed in the first place. Oliver wanted out and he knew that pleasing us was his best hope of not having to deal with us any longer. But at the same time, he had a full time day job, so he was often distracted from the things that we cared about. And like many other senior engineers, Oliver seemed allergic to writing documentation. Now, the second area of work was related to the fact that Frank and David had never really created a measurable launch plan this product, they had a concept of what success would look like, but the devils in the details, and they needed help coming up with the details. So while we worked on one side of the team by creating a done list, for them to satisfy us that we could build the product independent of them, we also worked with the other side of the team, helping them dance with the devil, if you will, of all those pesky details, deciding what was an absolute must have for launch, what was nice to have and what needed to remain a fantasy, at least for the first few releases. Now, while we were doing these things, we also worked with an outsourcer. To help Frank find a reliable software development team in Eastern Europe. I’ll take a moment here to talk about finding developers. It’s true that as my friend Andy Hilliard says smart people are everywhere. Following that principle means that using the right outsourcing strategy can allow you to build a highly skilled team of smart people who don’t live in your city, or even in your country. Now, one way I’ve seen that go badly for clients is they’ll Google offshore C sharp developers, which gives you a jillion results. It also gives you a mismatch of individuals and real software development companies. And of course, the individuals and real software companies have different rates per hour. And so many are tempted by low price believing that because smart people are everywhere, the only relevant differences price per hour. That is not the case. And the path that I’ve seen most consistently lead to great results is to find a company who focuses on actually writing software for clients and not a roll your own committee of developers who offer their services at the lowest hourly rate. I don’t have time in this episode to talk about all the differences between these two options. But suffice to say the lowest dollar per hour is not always the best value. And while the outsourcer worked to connect frank with a great team who was ready to hit the ground running to tackle that engineering work, we worked on helping Oliver and Frank find their way through the end of phase two orderly shutdown, with an island phase three starting up again with a new team. So we worked on defining the set of things we needed to know. And some of those were things that Oliver would just know off the top of his head. Others were things that Oliver had discovered and subsequently forgotten. And a third category included items that he didn’t know and he was gonna have to figure out. We were looking for product requirements, architecture, design documents, current work processes, processes for handling documentation, deployment, a list of tools in active use to support the development, the runtime stack, and third party libraries, I think we had a list of about 150 items that we needed Oliver to work with, to help us get captured. And once we had a starter list, which of course grew as we learned more, we worked consistently with Oliver and with God to get to the place where we had documents that we found acceptable. We also had to work on a migration path where we could export data from Oliver system and put it in a system under Frank’s control. And that included source code management work, ticketing and workflow system design documentation, mock ups and product roadmapping data has a ton of stuff. In parallel with that we had regular meetings with David to review what was working, what was broken, and what was missing. And we also challenge David to make some tough choices to figure out what really was in that minimum viable product? And what could wait. And because you have to make tough decisions. It’s a painful experience. Of course, we want everything. We can’t have a lot, but we can’t have everything all at once. And did I mention that the path to launch doesn’t just include the engineering components? Even if engineering can deliver perfect code, which literally never happens? Product Management still has a good deal of work to do before handing the application off to customers. For example, what are they going to do for user documentation for training the sales team for determining operating parameters like where the system will be hosted? And who gets to call when things aren’t working? How are they going to get paid for software? How will customers report problems and ask for help? How are you going to manage user identity and password resets for heaven’s sakes? And often these things are completely overlooked until engineers say oh, the code is ready. And then the client begins to think about these things. But these are all in the space of product management to figure out as much work as we needed to do with Oliver in his team, we had a similar amount of work with Frank and David. One of the pain points was making sure the system was documented in a way that’s understandable to a moderately skilled senior engineer. This project had gone through multiple iterations with very little documentation. And to Oliver’s credit, he had reviewed the system, the code, the old tickets, and the intent behind the system to get a sense of the way it worked. But Oliver started with essentially no documentation, and he was more interested in doing the work than documenting the work. So he built more and more, and he became the expert on how the system worked. Some people might feel that this is an intentional manipulation on the part of the engineer like, oh, I can be secure in my job if you don’t know how anything works. I really doubt this is the case for most engineers, what often happens is that writing documentation, well, it’s boring, and it gets put off in favor of the fun and interesting work. And I really think that’s what happened here. So our conundrum was we needed Oliver to choose to not do the interesting stuff, but to choose to do the boring stuff. Now in our favor, was that Oliver was tired of this side gig and he was happy to get rid of it. Unfortunately, all of his day job was demanding, and he had to squeeze this work in on a project that was getting shut down. And it was not his priority. In our project review meetings, Judy would ask Oliver, which of these things can you get done this week? And Oliver would say I’ll do the high level architecture doc and the application workflows, diagrams. But the next week would come by and he would report Well, the architecture documents done but I didn’t get to the workflows. And that brings us to one of the hardest software problems teams ever deal with the definition of Done. When is a document done? How would you know, when it comes to documentation, the standard we pushed for was understandable to a moderately skilled senior engineer. So why that standard? Let’s break that down. Understandable? Well, that makes sense. What’s the point of documentation that’s not understandable to a senior engineer? Okay. So we’re not asking for documentation for someone with no experience at all, because that can make your documentation 10 times as big. But what about the moderately skilled part? I think this is actually a critical element. This is important because every senior engineer who’s being pressed for this, first of all believes that he or she is highly skilled, and many times that’s actually true. But while they have high skill in technical areas, they often are not highly skilled in written communication. And they tend to be terse believing that anybody who gets it will need a lot will not need a lot of details, because as they see it, these things are obvious. We push for moderately skilled, because we want to tell the author who’s documenting that they have to dumb it down from their level, to a level understandable to a smart person, but maybe not quite as smart as they are. So when Oliver reported the documents were done, we would review them. And often we would discover that what he thought was clear, was actually lacking contextual details that were relevant to that next guy. And that person was going to need that information. Which meant that Oliver had it on his done list, but we kept pushing things back from done to needing revision. And since this was work he didn’t like and he was fitting in time to work on it anyway, you might imagine his attitude toward those project review meetings. Well, it was not great. They started out with his gently expressed frustration, which increased to annoyance. And overtime, these conversations became more and more aggressive. It’s understandable why it happened. But it led to some tense meetings before we finally got to a minimum viable set of documentation. And frankly, that part of the engagement was not one of the more fun parts. It actually took us a couple of months to get to an orderly shutdown in this case. And now we’ll turn brief attention to act three of today’s play successful startup with a new engineering team. While we continued working on that orderly shutdown with Oliver Frank had contracted with an offshore development company so we could begin phase three. How we begin things often impacts how the overall effort is gonna go. Engineers depend on accurate information and leading a team well requires a good plan with clear expectations about roles and about measurements.

Tom Cooper 29:07

Because of the work we had done with Frank and Oliver, we were able to help David onboard that new engineering team. One of the first things we did with this team is something we call a calibrate workshop. In a calibrate workshop, we invest time at the beginning to establish the tools, the meetings, the measurements, the communications, and the cultural impacts that can drive changes in the way that we work and cause miscommunication. These processes build trust quickly, and they establish a sense of one cohesive team. In just a few days, we get everyone on the same page about how the project is going to run and gain agreements about what we’re going to do when things inevitably don’t go perfectly. And after the Calibrate workshop, we worked with David and the engineers to help them come up to speed on the software they were taking over. We also coached and mentored David and the team. We held them accountable to establish work patterns that would allow the new team to stand up a very should have the app in their lab and begin that critical new development work. With the right parameters in place, and a structure that supported them, well, that team did an amazing job. They were laser focused. And it was just a few weeks after the contract was signed, when the new team was assembled, they built a new development environment, they had a copy of the old code operating, and they were even able to push out improvements to the code in just a few weeks. In this new model, Frank’s team had control the technical details, the control the knowledge, base and control of the intellectual property. They control the development and test environments on server images that they owned and controlled. And these, these environments were readily available to the development team. And that startup of that engineering team was one of the fastest and most smooth experiences I’ve ever seen. I like to say that it’s never been easier to write software than it is today. But it’s still hard. There are so many moving parts. Software is great, but it’s not simple, even with so many advanced tools at our fingertips. So let me take a minute and talk about what we learned in this engagement. We certainly saw the value of documentation, ticketing, and clarity about the criteria that was being used to measure success. And I think there were two major factors that led up to Frank needing to change development teams. The first, Frank didn’t have a ton of clarity about what the market needed and what the market would accept, he could have saved himself effort and money. If he systematically embraced principles from the lean startup with a focus on invalidating assumptions and rapidly learning in that way, he would have avoided paying for engineering work that potential customers just didn’t care about. And the second is, he was really focused on trying to do it cheaply, he looked at costs rather than the idea of investment. And because he was spending on the expensive stuff of paying engineers to work on unproven concepts, he was blowing through his investors money fast, without a clear path of return. If he had been prototyping and doing the critical market research, he could have failed faster and would have learned more with less spending. Often people dramatically underestimate the cost of engineering. Some markets require services that are a lot more robust, and his target market of financial services. That’s definitely one of them. What does that mean? It means it’s a long path from well, it works if everything goes right to it’s really hard to break when it’s attacked for a bunch of different angles. Frank thought he had a production ready solution, when what he had really built was a complex proof of concept. And finally, I want to come back to something I touched on earlier in this episode, one critical mistake that I see teams make is in the area of engineering, Team creation or team selection, that time and again, I see leaders confuse cost with value. Just because a resource is less cost per hour, that doesn’t mean they’re going to deliver higher value. If your team members on average cost half as much, but it takes them twice as long, you spend the same, but it takes you longer to get to your destination. And that time to get across the finish line. That’s opportunity cost. So going cheap, may actually cost you more money in the long run. And let’s talk about this. Do you want to spend your time recruiting, developing and shepherding the engineering talent? Because whether you hire independent contractors or bring on people as employees, that’s exactly what you’re signing up to do. These days, it’s often a far better value to hire a software development company, which has the expertise in those areas, and can bring an entire tech team online just for you. Yeah, that can be more expensive on a per hour basis than hiring independent experts. But it can lead to significantly better overall performance. And aren’t you really more focused on how many features you can ship that customers will be happy to pay for than just getting the lowest dollar per hour? Maybe your investment ought to be dollars per customer valued feature, rather than dollars per hour. And that wraps up today’s software horror story. We encourage you to tune in next time we’re going to talk about a team CTO who said to me, I feel like I have a firehose of money that I’m constantly pointing at new problems. That one was quite the carnival ride. See you next time.

In this Episode

• We went for a Big Bang Cutover and it wasn’t pretty
• Our project goals were defined poorly, and we paid the price
• We found out the hard way that low accountability leads to failure
• The tool we were using was great… for a completely different use case
• Our vendor had never taken on a challenge as big as us, and it showed

But we found a path to victory!

Transcript

Tom Cooper 0:00

Tom, the reality is this project will never work. We’re doomed. The political stuff that’s for the suits to solve not for me, if they hadn’t come up with such a bad idea to begin with, they wouldn’t have this problem at all stinks to be them.

Tom Cooper 0:21

Well, hello, and welcome to the second episode of the software horror stories podcast series from BrightHill group.

My name is Tom Cooper, and I’m your host.

We’ve just kicked off this series called Software horror stories where I’m sharing examples and lessons learned over the years about how projects succeed, and how they fail. In each episode, my goal is to use these stories to identify the things that blew up, why they blew up, and what you could do about it.

As I mentioned in the last episode, the coaching and training that I’ve been doing for the last few years has really focused on helping leaders of software teams do a better job of running those projects. And one of the things that really sticks out to me is so many times these projects have similar failures and similar challenges. Creating this podcast series as a way of helping make sure that we’re understanding the themes that hit us over and over again.

I recently came across a client that was struggling with managing user IDs, and it reminded me of a blast from my past. Over the years, my kids have often heard me say experience is what you get when you did not get what you wanted. Man did I get a lot of experience on this project!

Today’s episode covers the saga of a well intentioned organization seeking to solve a real and a painful experience.

It was an example of a solid business problem you would expect would be a great project to handle using software.

We all know that automation is a great way to reduce errors and increase the speed of routine and detailed processes. So you think that combined with all these great intentions, and the current pain of trying to find a solution manually, this project had all the markers of a great success, right?

I went and took on the job in this new department, I knew that we were working on automating user ID creation and deletion management.

It was true that the company had a growing problem keeping up with all the systems, all the passwords and all the users. Also, I knew this project had been going on for a couple of years, and there wasn’t yet a scheduled go live date.

Now, over the course of my career, I worked for that company for about 10 years. And I was about halfway through those years when I joined this project. And I gotta tell you, little did I know what I was getting into. So here we are a team trying to slay a dragon.

We had so many systems and so much employee turnover, we just could not keep up with creating and deleting these usernames and passwords. There was so much change, we were having trouble even recruiting enough people to activate and to shut down those accounts. And worse than that, there was no centralized system to manage IDs. And every application had its own store of usernames, password, password expiration, and password complexity policies, it was chaos, and it was getting worse every day.

So the company knew we had a problem. Abig one. In fact, because of the scope of this issue a couple of years before I joined the team, some bright person was tasked with finding a solution to managing identities in a multi vendor environment.

At that time, we had at least three major flavors of tools and directories in place at the time. And those were sort of enterprise directories. We had a whole bunch of other application specific directories that were out there really was a mess. And then we found great news, they found the perfect product, at least according to the vendor salespeople. Now the product would need a little bit of customization for an enterprise this large, but this toolkit could provide us with a silver bullet one place to manage identities across the whole company, it was gonna be great. Oh, man, I’ve heard that before.

Sometimes things are not as they seem. And this was one of those cases.

Now, as I mentioned, by the time I got involved, the build and prep for rollout had been going on for literally a couple of years, I was young and optimistic and I tried to bring energy to the team. This was a challenge because, frankly, the progress kept getting stalled. And every attempt to test had resulted in discovery of more problems than ever before. And the scope of work. It was just getting bigger and bigger.

One day, I greeted to the team members with enthusiasm and I ran into a bit of a brick wall. One of the grizzled veterans on the team tried to help me come into contact with some reality. And looking back it’s funny that that guy was a pessimist. He was the kind of guy who was never happy. If you gave him $100 He would say you shouldn’t be giving me 200 bucks.

Over time that guy and I became friends. He called himself a realist. Every pessimist I’ve ever met, calls himself a realist makes me chuckle just to think about it.

When I was a kid, my grandfather used to read Winnie the Pooh to me and my sister and I love the story.

I love the characters and one of my favorite characters was er, the pessimistic donkey, who would say things like, I guess that’s the sort of thing you like, if you liked that sort of thing.

This grizzled veteran, he reminded me of Eeyore, in fact, I began to affectionately refer to him as a you’re just because of his attitude. Now, here’s what you’re told me to help get my attention. And to bring me back to Earth.

He said, “Tom, I think it’s cute that you’re excited about this. The reality is, this project will never work. Ever. But the VP has been so much of the company’s money on it, she can’t afford to see it fail. We are doomed.”

And while over the years, there were many times that Eeyore’s dire predictions were wrong. The longer I was involved with this project, the more I became convinced that Eeyore was right.

It was a death march. Now a death march is a project which participants believed to be destined for failure, or that requires a stretch of unsustainable overwork.

I think this team had been feeling that for some time, but it sounds awful. Was it true?

Well, I’ll tell you a bit more about this in a little bit.

But now back to the details of today’s story. There were some bad sides.

First, there was that issue of the vendors optimism that I mentioned before, the vendor who created the solution had implemented it at a number of other clients before they landed us as a customer.

Unfortunately, they’d never landed a client as complex as we were before.

This meant that we had to do a lot more debugging a lot of customizations and a ton of new feature development, a lot more than they ever planned for, and certainly more than they budgeted for.

In many ways. This experience is like the dog that chases cars, and finally catches one, what the world will she do with it?

Or maybe the fisherman seeking to catch the biggest fish ever. But what happens when the fish is so big that it starts to sink your boat?

That’s what happened to this vendor.

They didn’t have the skills, the team members the experience, or the money to fulfill the promises they made.

We had continuously fallen prey to false hope. And to the sunk cost fallacy. We spent too much to quit, we have to keep going.

Well, it wasn’t all bad news.

In the middle of this effort, that vendor ended up getting some much needed help.

You see, their toolkit was based on the right technology.

And because it was solving problems that other customers were having, that vendor got acquired by IBM.

Now on the surface, this seemed really useful because IBM had lots of experience and lots of money to put behind making this a success.

IBM promised us as their customer that they would get us across the finish line to run the toolkit.

We already had a big spend with IBM, and they really wanted to keep us as a customer.

But we had some internal challenges. This was a long time ago.

Did I mention that back in those days, we weren’t great at project management.

And despite our best efforts, we weren’t super clear about what does success really look like.

By the time we were hip deep in this project, a lot of ideas have been floating around about how this was going to be that silver bullet to solve all of our issues around user management on every single platform everywhere.

That scope would really run away from us.

And in keeping with that theme of poor project management, we really weren’t great at accountability either. This accountability problem, it cropped up in two specific places.

First, we were not good at holding our internal people accountable, we would often not set a due date, or not clearly defined what needed to be done or not assign it well to the right person.

We ended up with a lack of follow through in a lot of ways, and that was painful.

As bad as we were at holding ourselves accountable, we were lousy at holding the vendor accountable. So that speaks to the vendor and to our project management.

But wait, there’s more!

We had technical problems with the toolkit and with our custom software as well. One key technical challenge was the idea that we could synchronize identity across multiple directories. Now the good news is, by the time the project was under full steam, the company had made the decision to get rid of one of the three directories we had.

But that left us with two systems that needed to be integrated in neither of those vendors had a simple way of doing it. So what we’d need to do is synchronize all the IDs to this new service. And then we would just reconcile the differences between the new system and the existing systems. Simple, right? What could go wrong.

Also, to make this work, we couldn’t just add this system and not have it affect the other systems. This tool would require what I like to call a Big Bang cutover. For example, on a Monday we’d be using the current process and then overnight on Tuesday, we would turn on the new system for 100% of users.

Now if it worked, it’s great. If it didn’t work, then nobody could log into any systems at all. That’s a terrible idea. In fact, just as an aside, if somebody comes to you and says the only way to implement the This project is a big bank run. Because it has been my experience over the decades that big bank cut overs never work ever. I’ve never seen a big bank cutover actually work.

Tom Cooper 10:12

Finally, there was a key difference between our implementation and the implementations our vendor had done was successful customers.

100% of the successful customers had built their solution around this product as the authoritative source of data. And any other data source was subordinate to this data source.

Of course, our system was different. Ours required we treat our existing LDAP as authoritative and then sync that system to this one. So this was a project was large, it was complex, and it had never been done this way before our test planning was weak. And we kept discovering more and more areas to fix.

All of that combined means that every single go live attempt that we had made up to that point had been a total disaster, and the team was losing hope, which is why Eeyore told me we were doomed.

It looked like Eeyore might have been right.

This project was a death march and the VP could not abandon it politically, she had made too many promises.

IBM didn’t want to walk away, and the team was getting crushed.

Just talking about this brings back some painful memories.

I think it might be a good time to take a break.

As a leader in a software team, you know, you want to help your team perform at the highest level. And you know that with the tools we have, it’s never been easier to write software than it is today. But it’s still hard.

I’m Tom Cooper, and for the last decade, I’ve focused on coaching and training software leaders and their teams to help them make the small but important improvements to help them perform as one team and to achieve their potential. How much better would your results be?

If your team communicated clearly managed conflict well, delegated work that stayed delegated, and then created an executed on great work plants. It shouldn’t have to be this hard to write great software. And the good news is it doesn’t have to be, are you ready to take the next step?

Head on over to brighthillgroup.com, so you can learn more?

Tom Cooper 12:16

So far, this seems like an awful situation, doesn’t it? It was. Now this is the part of the episode where we turn our attention to what can be done to fix this.

So far seems pretty bad, right? Lots of problem areas, lots of potential failures. And with each month, we ended up with a bigger backlog of ideas, a higher sunk cost for the project and little hope of shipping anywhere in here on what might be considered on time.

The reality is that 70% or more projects fail, they fail to deliver within scope or within budget.

Some projects are complete failures and have to be scrapped entirely.

We don’t like to talk about what went poorly.

On this this particular project. By the time I joined, I think all of the important directional decisions had been made.

We had a well established pattern of slipped dates with low accountability that were leading us down the path to destruction. Now, knowing what I know now, there are a lot of things that I would do differently.

But back in those days, I was more of a worker bee than I was a leader certainly had almost no positional power. And I was still learning what it meant to lead people in the from the middle with relationships and influence skills rather than formal positional power.

Today, if a leader was in a project like that, and I was offering them some advice, I think I could offer several options to help. But by the time the project had literally been running for years without anything in production, I think this project was was pretty much doomed.

I think Eeyore was right about this one.

I’m reminded of that scene in The Princess Bride where the heroes are arguing with Miracle Max about whether Wesley is all dead or mostly dead. Sadly, by this time, our Wesley was all dead.

All we could do, as Miracle Max said was root through his pockets looking for loose change.

Sometimes your best option is to declare the project dead so you can move on.

And sometimes the reality is the best outcome is to kill the project, capture some great lessons learned and start looking at better options.

Now, at the time, even once I became convinced this can work for us, I didn’t have the influence or the authority to make that a reality. So much time had been wasted on this failing effort that it literally could have put the VPS career in jeopardy.

I’m grateful. I’m grateful for another leader in the organization who was a possibility thinker.

He was one who load loathed the idea of letting the company down and was looking for alternatives. Jeff was good about that.

Jeff asked what has become one of my favorite questions to ask teams. I asked this question all the time. Jeff asked, “Exactly what problem are we trying to solve?”

This question is great because it is so so easy to fall into the trap of all the little details of projects and miss the big picture. He also asked an important follow up question, which is, if we had a way to make this work, how would we know?

I love this question, too, because it gives us a measurement.

We needed a measurement that would show that we were either making progress or we weren’t making progress.

It reminds me to remember that we are all learning together.

This is a team effort.

It’s not that you have the one hero who comes in to save the day, we are working as a team to solve this.

I appreciated what Jeff had to say about helping us reframe what was going on.

As you’re working on projects, look around you who can be helpful, who has good insights, even Eeyore had a necessary message “We’re doomed.”

Truth telling is often unpopular.

Now. I do want to take just a moment to tell a quick story about another experience I had with your now as it happens, Eeyore was right about this one thing, but he had developed a habit of complaining, he had created a negative attitude.

Now many times I’ll have a team member who’s pessimistic say, I just can’t help it. It’s the way things are. It’s not possible to be positive when so many things are so negative. That’s life.

But I’ve got a funny little story about a your despite his claims that the deck was stacked against him, I was able to prove conclusively that His attitude was entirely under his control.

His attitude was a choice that he made.

Years ago, when my grandmother decided that she was no longer able to drive she asked me to sell her car for her. Now this car really was that proverbial used car driven by the little old lady who drove it to the grocery store and church on Sunday.

Now I knew that Eeyore was looking for a reliable car for his son to drive and I asked if your might be interested in buying it.

But there’s one thing I know about cars, cars need maintenance.

And there’s always a story to tell about a problem with a car, something breaks wiggles that rattles payment falls off.

When Eeyore said he was serious about buying the car, I knew I was going to write up a contract for it.

I knew I did not want to hear about that car ever again after the sale.

I also knew that Eeyore was well, he was tight with $1.

I wanted to protect myself.

Thankfully I had a rare flash of brilliance. When I wrote the purchase agreement. It literally said, “If the buyer ever complains about anything related to this car, he will immediately pay the seller $5 cash for each negative statement made.”

When he or saw the purchase agreement. He smiled broadly. He laughed loudly and then he signed the contract.

So you know what happened? He never ever said a negative word to me about that car, your drove that car for 160,000 more miles over the next several years.

Every single time. And I mean, every time I saw him, he would smile. And he would make a point of telling me what a great car that was and how well it was working for him. Every single time Eeyore’s complaining had become a habit and he could choose to be positive.

I have no doubt that car needed all kinds of work over the years. But he never mentioned that to me, he focused on the good parts. I jokingly called him Eeyore, but I want to point out that he is a man I respect. He works hard for his employer and he works hard to care for his family.

His attitude was something he could change.

But back to our project back to Jeff’s question, “what problem are we trying to solve?”

Well, we really figured out we had two problems that we had to solve

first, what do we need to do technically, to make this thing work? And

second, how do we handle the political fallout of a seven figure project failure?

Now, many of you might be saying, “Look, I’m the tech guy, the political stuff, that’s not my problem. That’s for the suits to solve, not me. If they hadn’t come up with a bad idea like that to begin with, they wouldn’t have had this problem at all.”

I will say it is super handy if you can just wash your hands of a problem set like this.

But if you want to be a top performer, solving political problems is at least as important as solving the technical ones.

So what do we do?

Well, the first thing was to set up a small tiger team to investigate alternatives.

Jeff and I began to brainstorm about the root of the problem we needed to solve and we brought in some subject matter experts who helped us unpack the mess.

At the outset, we needed to keep it hush hush. The full project team continued to work toward delivery according to the published plan, but a few of us began really digging into what problem are we actually trying to solve?

On the technical and software side we realized a great Do the problems we were we were trying to solve involve working around how the tool needed to be configured because of the way our environment worked. We were fighting the tool the whole time, because we insisted on trying to make a square peg fit into a round hole.

Now, we did use the “5 why’s” technique, and we uncovered that the core that we needed to address could be a lot simpler.

We didn’t see it initially, because we were trying to figure out how to use that one product in our specific environment.

You know what else we discovered? after we had been working on this project for a couple of years, we knew a lot more about the problem than when we first started.

We had rushed to start to build a solution before we fully understood the problem we were solving. Before we could conceive of a minimum viable product that was achievable. We started writing code.

Tom Cooper 20:51

On the business front that the death march efforts so far, it allowed us to gain a lot of clarity about the complexities of the business processes and what was really causing pain in those areas.

We simply had not understood these things at that level when we started the project.

And on the technical front, we had some good news too.

By that time, we were really down to two directories.

And the technology from the other vendors had gotten better as well. It was true, we could not find a one stop shop like this vendor that was supposedly going to fix everything. But we did find a collection of tools that could potentially be combined to get us across the finish line for the biz biggest business problems were facing.

Even better was that the tools we believe could rescue us were sold by vendors we had good relationships with and we’d had decent experience with them in a really limited fashion in other parts of the business.

So we had access to knowledge and to lab copies of the software.

Our tiger team squeezed in some time to prototype and test and fail and fail and fail, try again and test and eventually succeed on a limited scale.

One other key difference is that we could do this incrementally and not depend on that big bang cutover.

So after a couple of months of prototyping in the shadows, we became convinced that the tools could do what we needed at least most of it.

But we were still on the death march, right?

Unless we could convince the powers that be to agree that a this project needs to end be we have another option that could fix the business need, and see that we could find a way for our senior leadership to save face, that that money wasn’t a complete waste. If we couldn’t do that we were going to have to keep going.

Eventually, we got some good news on that front, because IBM was getting sick of dealing with all the headaches and the massive cost overruns. They had an incentive to get out of this deal too, and our tiger team was able to present our findings to the VP. And she began to negotiate with IBM about potential alternatives so that we would not take a complete loss on the money we’d spent now. Over the three plus years we were working on this project, we had spent multiple seven figures on licenses, professional services and internal labor.

Our winning play turned out to be the we shared that smaller risk bite size approach with a larger team, and then with the peers of the VP and IBM agreed to trade us out licenses for alternative software with a retail price close to what we’d already spent with them.

So that kind of got us off the hook. And one day, we were able to stop working on that original project and we pivoted to the new suite of tools and smaller milestones.

Over the next year, we managed to roll out a solution which allowed us to scale the number of accounts and users without having to hire more team members for manual provisioning.

We also finally had a solution which could automate that provisioning and de provisioning of user accounts.

Now, as I’m wrapping this up, it’s always easy to blame others for project failure.

The roots of failure for this project were first we didn’t understand what success looked like.

We also didn’t understand how complex the business processes were or how hard it was going to be to automate them. That optimistic vendor sales team, they made promises their tech team simply couldn’t back up.

We didn’t establish checkpoints to help us fail fast and learn fast.

And also culturally, it wasn’t safe to say this was going to be this is going to fail.

As I said before, experience is what you get when you didn’t get what you wanted. I gained a lot of experience in this season of my career. And that’s today’s software horror story.

Tom Cooper 24:24

Do you have a story we should share on a future episode of the podcast?

What are you learning from your software projects? Are you dealing with a frustration on a current project and you just like to talk about it? Let’s connect head on over softwarehorrorstories.com That’s one word softwarehorrorstories.com And you’ll find a link to book a free quick call no strings attached. I’d be happy to help!

In This Episode

• Executive alignment
• Realistic (not optimistic) estimates
• Transparency of promises made

Transcript

Tom Cooper 0:00
When it comes to planning our roadmap these days, we just pick the client, we want to disappoint the least. No one system for setting priorities has ever lasted more than one quarter. When we work with you, we need you to bring on more team members without slowing anything down.

Tom Cooper 0:21
Welcome back to the becoming a geek leader podcast.

My name is Tom Cooper, and I’m your host. This is the first episode of season five. And in this season, we’re going to tackle something a little different. For the past few years, I’ve been focusing my coaching and training work on helping software teams improve their performance. In one day, my friend Bobby Dewrell and I were talking swapping some stories. We realized there are a lot of teams out there who are struggling with the same kinds of problems.

I think it was Bobby suggested that we start telling these stories, and that’s the genesis of this season.

So because of Bobby’s input, today, I’m starting a new series on this podcast.

I’m calling these software horror stories, stories about teams and projects that simply could not go right. As I’ve worked with clients and collaborate with other experts like you, we’ve all seen these challenges.

In many cases, we’ve found solutions. And in some cases, we certainly didn’t. In each episode, I’ll introduce a situation and I’ll talk about what did or did happen, or could have happened to make things better.

During the season, we may even be able to hear from some of my friends who’ve lived through some of the same kinds of stories.

So why tell these stories, I mean, is it just to give you a cold shiver down your spine? Well, I’m hoping you’ll enjoy knowing you’re not the only one suffering through this.

Hopefully that you’ll be able to have some takeaways from these stories that will help you with your projects and your teams. Now, for many of you, this may be your first episode of a podcast from becoming a geek leader.

I want to introduce myself to you. Once upon a time there was a young boy who fell in love with technology. When he was 12, he bought his first computer with money he earned on his paper out.

He started learning tech and writing code when almost nobody even had a computer at home. And as a young professional every day, he worked on tech projects. Day after day, he watched project team struggle and projects fail. How could this be? These teams had smart people, they had good tools, they’re working on solvable business problems, these things should just work. He continued to wrestle with this problem until finally one day he discovered the core issue is that teams didn’t deal with the people part of the equation. They were great at the technical stuff, but they often failed at leading people through the process to get them to success. And he began to learn these things. And as he did his career took off. He moved up the ladder in large companies and in small companies. Eventually he was the VP of Products for a software as a service firm. But he wondered why didn’t anyone ever teach me this stuff. And so he decided to quit his job and do just that. For the last 10 years, he’s been working with software leaders and teams to help with people stuff like communication, delegation, resolving conflict and planning. And what he found was that these skills were critical to helping move forward the projects that people were just desperate to get done. So let’s get started with today’s software horror story.

Tom Cooper 3:24
I’m calling today’s story. Everyone needs a reservation to get in.

So this client, let’s call them entry, Inc. came to us in an unusual way. They already had a tool in production that they’d written to book reservations, users could go online or with an app, they could make a reservation for a particular facility.

And then they can make a payment.

Simple enough, right?

But, truth be told, it was not going well, the leaders of the company were pretty mad. They said engineering is way too slow. We think the CTO is failing us. And they told the CTO to go find some help to solve the problem, or they would solve the CTO problem. And so he hired us to help solve the problem. Interestingly, right after the contract was so signed by the CEO, he quit. The new CTO came on board, he realized that he had to work with us, even though he hadn’t picked us. That’s not a good place to start, is it? We sat down with engineering, and it was true, they did have problems. They were far behind, and they couldn’t give us a plausible plan for how they were going to catch up. Well, so that meant the company leaders were right, weren’t they? Not so fast. As we looked at the engineering process and the software team, overall, they were doing good work. They were using good tools. And generally they were following the kinds of processes that should have led them to good production of new software. But the customers and the internal stakeholders they weren’t happy. They said code quality is no good. Existing customers are getting mad because they’re not getting the features that they were promised new customers are coming on board. If that would have features work, and then the features would break it management said it took too long for new clients to come on board.

That just wasn’t working well.

People had this perception that the software was full of bugs, whatever that means.

When we dug in with engineering, and they said, every time they turned around product management and operations would bring them a different list of priorities.

Sometimes on a Monday, they’d say, drop everything and work on x. And Tuesday, they’d here Forget about X, we need to work on why they even told us, we can’t even schedule an entire sprint, because things are constantly changing, and pay off technical debt.

Who’s got time for that? We asked to look at the backlog. Man, was there a backlog.

But when we dug into the backlog, what we discovered was most of what was being called requirements.

They weren’t requirements at all, they were almost statements of ambition, make it easier to add new users.

And not only that, but there wasn’t a plan, even verify that the code that got written matched what the requirements were.

We sat down with product management, and they told us, our mission is to be a product company that that were in charge of deciding the SAS features that are needed, and that they were really frustrated because senior leadership kept demanding that they force in customizations, and that were specific to individual clients. And these facility owners would say, No, we have to have this feature or that feature.

And the senior leaders would say make it happen. Leaders would come to product management, they would literally say this is what has to get done. In fact, they tried to set up multiple times they tried to set up a system to to schedule or plan or think through product prioritizations. And product management eventually told us no one system for setting priorities has ever lasted more than a single quarter. They also said, sales, close the account, and then start talking to engineering. And they said we can’t create a predictable roadmap because we’re constantly having change. In fact, and I put this in the cold open when it comes to planning these days, we just pick the client, we want to disappoint the least. Now, at the core, the problem that our client felt they had was capacity that if they could just add additional engineers, they’d be able to catch up. And in fact, they asked us to come up with a plan where they could bring on new team members and train them and get them submitting working code. But there was a catch. They couldn’t afford to slow down development because they were already too far behind. So they wanted to double their capacity, but they didn’t want to have any impact on short term productivity. Man, I was going back over my notes as I was preparing this episode. And all these were ongoing challenges in this organization and so many other things that we just don’t have time to get into in this particular episode.

Now, if you hang on for just another couple of minutes, we’ll be digging into what we think would really have made a difference for this client. But first, there is no greater business investment than investing in your team. You’ve got an exceptional team, but you’re still struggling to unlock consistent, outstanding performance. You know, it shouldn’t be this hard to deliver value to your business. And we do to even the best teams struggled to build a culture of collaboration and high performance without clear communication and right planning frameworks. brightfield group, we deliver leadership coaching that empowers you and your software team toward peak performance. Our personalized coaching solutions help you build trust and increase efficiency across your development efforts. Head on over to bright Hill group.com To schedule a call today. So you and your team can break free from this rut of untapped potential and start delivering the high business value you know you’re capable of.

Tom Cooper 9:03
So let’s talk about how do you go forward in this kind of situation. And realistically the things that we’ve talked about so far in this episode have been a lot more about sort of the evidence of a problem than the core problem itself. I think about this when when you have the wrong strategy, there’s nothing you can do tactically, that’s going to make a difference. The reality is when it came to productivity engineering was thrashing. There were too many page faults, too many high priority items that were pushing back current work to the backlog and so it really doesn’t matter how many more engineers you add. If you keep overloading the work in progress, you’re going to have problems even if we were able to straighten out the inefficient curves in the engineering cycle to make a straight line path even if we could have improved software quality overnight. It entry Inc still would have been stuck they that wasn’t that the problem was related dirt directly to engineering or QA, it had much more to do with what they needed to do differently as an organization. There were three core areas that we think were fundamentally a challenge. One was executive alignment. at the executive level, they just were not on the same page about priorities and commitments. I love this concept that the first job of a leader is to define reality. The reality is this organization was pretty dysfunctional in the way that they were operating. And the company was growing fast. And they, they were trying to figure out how to scale the business as much as they could. But the key leaders in the senior part of the organization, they weren’t on the same page about what promises had been made to customer, much less what promises could even be kept. The measurements that were really driving the senior leaders had to do with closing sales and grabbing up the most facilities, they could, because they felt that strategically, doing a land grab was critical that they could get as many facilities as possible that that would be a big win for them. And different executives kept changing the priorities based on missed commitments, who’s yelling the loudest? Man, if you are setting your priorities by who’s yelling the loudest, that is not going to be a good day. And they didn’t. Also they didn’t have good information on which to make plans.

Since they didn’t have good information, they were flying blind.

Making decisions, when you don’t have good information, that is not a recipe for success, they just continued to make promises that the organization just had no way of keeping.

That is so devastating to morale, if you’re going to come to work every day, and you’re going to get hit in the head, you’re gonna fail, you’re going to be tripped, you’re going to be caught off guard to cut off the knees. That’s what was happening, because you could work really hard to try to get something done, only to find out that that was de prioritized. Or even that it was pushed so far back in the schedule, it might as well have been canceled. So that’s area one, executive alignment, the second area had to do with realistic estimates. Now engineering did have some skin in the game. But one of the things that was critically needed was to put effort into looking at what they were calling their backlog. And what was happening was engineering wasn’t getting good requirements. And these requirements, were not developer ready. They weren’t ready for developers to pick them up and start working on it. And so what was happening was development would get and requirements that would say, make it easier to onboard people. But what does that even mean? And and how would they figure out the complexity or approach that was needed, they were not able, based on the information that was in the tickets to create estimates, they couldn’t even create t shirt sizes for features. And if you want to make a plan about it, when you’re going to deliver it, you have to understand how complex is this? How hard is it going to be to build this and it doesn’t have to be an hours, but at least was some idea of the concept.

This meant that engineering was not going to be able to follow through because they had no idea when they picked up work, whether that work was going to be one hour, or 100 hours.

Finally, the third area that I think is critical in this situation is transparency of promises.

And support had made one set of commitments to customers.

Onboarding, the professional services aren’t they made another set of promises, a

sales had A third, and then

product management and then

engineering,

all these different commitments that were being made to each other, and to customer were floating around out there.

This was like a time bomb because every time somebody would turn around, there was an unmet need, or a promise that wasn’t being kept.

That is so frustrating, and, and frustrating for customer and frustrating to sales and everybody was mad.

That’s not a good way to be able to have a positive future for the organization.

Recognizing that these organizational groups inside the organization, they needed to begin collaborating, no one had a view of the priorities outside their group.

There was no process in place to help gain that visibility and to to really push forward for cooperation across the teams.

That’s one of the things that I think was foundationally a problem that when there was not communication across teams, it led to everybody kind of pointing fingers at each other.

Now until everybody could agree on on what was needed, and which I items on that list of what was needed were the highest priority.

They were going to continue this churn. And this, this is just going to leave them to so much thrashing and frustration and pain.

Now, of course, there were improvements in the engineering process, we could have made that starting with engineering saying, No, we do not accept that as a requirement because it’s not development ready.

Being able to define what would product management consider acceptable quality. And because none of that was defined, it was just going around in circles.

When you’re going in circles, figuring out how to go faster, turns out not to be all that helpful.

Now, what’s interesting to me is that the majority of the issues that this organization was facing, they were not related directly to technology.

One thing I’ve learned about planning and about business process, if you’re going to change the way that those things work, it takes time, it takes reminders, and it takes encouragement, there is no silver bullet, to getting people to change behavior.

The reality is, we often know what to do, but we do not do it.

Ouch, that hurts.

It’s true.

We know what things we need to do to get better to grow to stretch, but we just don’t do it.

That was the case for this organization. And we came back to leaders with a recommendation we said, here’s some changes we think that you should make.

If you’re going to be successful in making those changes, we recommend you have some external accountability to follow those new processes.

We felt confident that they could be making and keeping promises to each other and to customers quickly and they could begin to build momentum.

Interestingly, in this case, when we delivered our recommendations, they said, Well, you’re not offering the kind of help we’re looking for.

And they sent us on their way.

I do hope that they got some of those things sorted out, we wish them the best and we did go on our way.

Tom Cooper 17:01
So that’s today’s software horror story. Everyone needs a reservation. I hope you found it helpful. But I want to ask you for just a second.

How is your team doing with planning with priorities with communication? Are you investing in your team?

Head on over to bright Hill group.com To learn more about how to improve your results as a Software leader