Disaster Averted: An Impossible Project Succeeds – BGLS5E2

In this Episode

• We went for a Big Bang Cutover and it wasn’t pretty
• Our project goals were defined poorly, and we paid the price
• We found out the hard way that low accountability leads to failure
• The tool we were using was great… for a completely different use case
• Our vendor had never taken on a challenge as big as us, and it showed

But we found a path to victory!

Transcript

Tom Cooper 0:00

Tom, the reality is this project will never work. We’re doomed. The political stuff that’s for the suits to solve not for me, if they hadn’t come up with such a bad idea to begin with, they wouldn’t have this problem at all stinks to be them.

Tom Cooper 0:21

Well, hello, and welcome to the second episode of the software horror stories podcast series from BrightHill group.

My name is Tom Cooper, and I’m your host.

We’ve just kicked off this series called Software horror stories where I’m sharing examples and lessons learned over the years about how projects succeed, and how they fail. In each episode, my goal is to use these stories to identify the things that blew up, why they blew up, and what you could do about it.

As I mentioned in the last episode, the coaching and training that I’ve been doing for the last few years has really focused on helping leaders of software teams do a better job of running those projects. And one of the things that really sticks out to me is so many times these projects have similar failures and similar challenges. Creating this podcast series as a way of helping make sure that we’re understanding the themes that hit us over and over again.

I recently came across a client that was struggling with managing user IDs, and it reminded me of a blast from my past. Over the years, my kids have often heard me say experience is what you get when you did not get what you wanted. Man did I get a lot of experience on this project!

Today’s episode covers the saga of a well intentioned organization seeking to solve a real and a painful experience.

It was an example of a solid business problem you would expect would be a great project to handle using software.

We all know that automation is a great way to reduce errors and increase the speed of routine and detailed processes. So you think that combined with all these great intentions, and the current pain of trying to find a solution manually, this project had all the markers of a great success, right?

I went and took on the job in this new department, I knew that we were working on automating user ID creation and deletion management.

It was true that the company had a growing problem keeping up with all the systems, all the passwords and all the users. Also, I knew this project had been going on for a couple of years, and there wasn’t yet a scheduled go live date.

Now, over the course of my career, I worked for that company for about 10 years. And I was about halfway through those years when I joined this project. And I gotta tell you, little did I know what I was getting into. So here we are a team trying to slay a dragon.

We had so many systems and so much employee turnover, we just could not keep up with creating and deleting these usernames and passwords. There was so much change, we were having trouble even recruiting enough people to activate and to shut down those accounts. And worse than that, there was no centralized system to manage IDs. And every application had its own store of usernames, password, password expiration, and password complexity policies, it was chaos, and it was getting worse every day.

So the company knew we had a problem. Abig one. In fact, because of the scope of this issue a couple of years before I joined the team, some bright person was tasked with finding a solution to managing identities in a multi vendor environment.

At that time, we had at least three major flavors of tools and directories in place at the time. And those were sort of enterprise directories. We had a whole bunch of other application specific directories that were out there really was a mess. And then we found great news, they found the perfect product, at least according to the vendor salespeople. Now the product would need a little bit of customization for an enterprise this large, but this toolkit could provide us with a silver bullet one place to manage identities across the whole company, it was gonna be great. Oh, man, I’ve heard that before.

Sometimes things are not as they seem. And this was one of those cases.

Now, as I mentioned, by the time I got involved, the build and prep for rollout had been going on for literally a couple of years, I was young and optimistic and I tried to bring energy to the team. This was a challenge because, frankly, the progress kept getting stalled. And every attempt to test had resulted in discovery of more problems than ever before. And the scope of work. It was just getting bigger and bigger.

One day, I greeted to the team members with enthusiasm and I ran into a bit of a brick wall. One of the grizzled veterans on the team tried to help me come into contact with some reality. And looking back it’s funny that that guy was a pessimist. He was the kind of guy who was never happy. If you gave him $100 He would say you shouldn’t be giving me 200 bucks.

Over time that guy and I became friends. He called himself a realist. Every pessimist I’ve ever met, calls himself a realist makes me chuckle just to think about it.

When I was a kid, my grandfather used to read Winnie the Pooh to me and my sister and I love the story.

I love the characters and one of my favorite characters was er, the pessimistic donkey, who would say things like, I guess that’s the sort of thing you like, if you liked that sort of thing.

This grizzled veteran, he reminded me of Eeyore, in fact, I began to affectionately refer to him as a you’re just because of his attitude. Now, here’s what you’re told me to help get my attention. And to bring me back to Earth.

He said, “Tom, I think it’s cute that you’re excited about this. The reality is, this project will never work. Ever. But the VP has been so much of the company’s money on it, she can’t afford to see it fail. We are doomed.”

And while over the years, there were many times that Eeyore’s dire predictions were wrong. The longer I was involved with this project, the more I became convinced that Eeyore was right.

It was a death march. Now a death march is a project which participants believed to be destined for failure, or that requires a stretch of unsustainable overwork.

I think this team had been feeling that for some time, but it sounds awful. Was it true?

Well, I’ll tell you a bit more about this in a little bit.

But now back to the details of today’s story. There were some bad sides.

First, there was that issue of the vendors optimism that I mentioned before, the vendor who created the solution had implemented it at a number of other clients before they landed us as a customer.

Unfortunately, they’d never landed a client as complex as we were before.

This meant that we had to do a lot more debugging a lot of customizations and a ton of new feature development, a lot more than they ever planned for, and certainly more than they budgeted for.

In many ways. This experience is like the dog that chases cars, and finally catches one, what the world will she do with it?

Or maybe the fisherman seeking to catch the biggest fish ever. But what happens when the fish is so big that it starts to sink your boat?

That’s what happened to this vendor.

They didn’t have the skills, the team members the experience, or the money to fulfill the promises they made.

We had continuously fallen prey to false hope. And to the sunk cost fallacy. We spent too much to quit, we have to keep going.

Well, it wasn’t all bad news.

In the middle of this effort, that vendor ended up getting some much needed help.

You see, their toolkit was based on the right technology.

And because it was solving problems that other customers were having, that vendor got acquired by IBM.

Now on the surface, this seemed really useful because IBM had lots of experience and lots of money to put behind making this a success.

IBM promised us as their customer that they would get us across the finish line to run the toolkit.

We already had a big spend with IBM, and they really wanted to keep us as a customer.

But we had some internal challenges. This was a long time ago.

Did I mention that back in those days, we weren’t great at project management.

And despite our best efforts, we weren’t super clear about what does success really look like.

By the time we were hip deep in this project, a lot of ideas have been floating around about how this was going to be that silver bullet to solve all of our issues around user management on every single platform everywhere.

That scope would really run away from us.

And in keeping with that theme of poor project management, we really weren’t great at accountability either. This accountability problem, it cropped up in two specific places.

First, we were not good at holding our internal people accountable, we would often not set a due date, or not clearly defined what needed to be done or not assign it well to the right person.

We ended up with a lack of follow through in a lot of ways, and that was painful.

As bad as we were at holding ourselves accountable, we were lousy at holding the vendor accountable. So that speaks to the vendor and to our project management.

But wait, there’s more!

We had technical problems with the toolkit and with our custom software as well. One key technical challenge was the idea that we could synchronize identity across multiple directories. Now the good news is, by the time the project was under full steam, the company had made the decision to get rid of one of the three directories we had.

But that left us with two systems that needed to be integrated in neither of those vendors had a simple way of doing it. So what we’d need to do is synchronize all the IDs to this new service. And then we would just reconcile the differences between the new system and the existing systems. Simple, right? What could go wrong.

Also, to make this work, we couldn’t just add this system and not have it affect the other systems. This tool would require what I like to call a Big Bang cutover. For example, on a Monday we’d be using the current process and then overnight on Tuesday, we would turn on the new system for 100% of users.

Now if it worked, it’s great. If it didn’t work, then nobody could log into any systems at all. That’s a terrible idea. In fact, just as an aside, if somebody comes to you and says the only way to implement the This project is a big bank run. Because it has been my experience over the decades that big bank cut overs never work ever. I’ve never seen a big bank cutover actually work.

Tom Cooper 10:12

Finally, there was a key difference between our implementation and the implementations our vendor had done was successful customers.

100% of the successful customers had built their solution around this product as the authoritative source of data. And any other data source was subordinate to this data source.

Of course, our system was different. Ours required we treat our existing LDAP as authoritative and then sync that system to this one. So this was a project was large, it was complex, and it had never been done this way before our test planning was weak. And we kept discovering more and more areas to fix.

All of that combined means that every single go live attempt that we had made up to that point had been a total disaster, and the team was losing hope, which is why Eeyore told me we were doomed.

It looked like Eeyore might have been right.

This project was a death march and the VP could not abandon it politically, she had made too many promises.

IBM didn’t want to walk away, and the team was getting crushed.

Just talking about this brings back some painful memories.

I think it might be a good time to take a break.

As a leader in a software team, you know, you want to help your team perform at the highest level. And you know that with the tools we have, it’s never been easier to write software than it is today. But it’s still hard.

I’m Tom Cooper, and for the last decade, I’ve focused on coaching and training software leaders and their teams to help them make the small but important improvements to help them perform as one team and to achieve their potential. How much better would your results be?

If your team communicated clearly managed conflict well, delegated work that stayed delegated, and then created an executed on great work plants. It shouldn’t have to be this hard to write great software. And the good news is it doesn’t have to be, are you ready to take the next step?

Head on over to brighthillgroup.com, so you can learn more?

Tom Cooper 12:16

So far, this seems like an awful situation, doesn’t it? It was. Now this is the part of the episode where we turn our attention to what can be done to fix this.

So far seems pretty bad, right? Lots of problem areas, lots of potential failures. And with each month, we ended up with a bigger backlog of ideas, a higher sunk cost for the project and little hope of shipping anywhere in here on what might be considered on time.

The reality is that 70% or more projects fail, they fail to deliver within scope or within budget.

Some projects are complete failures and have to be scrapped entirely.

We don’t like to talk about what went poorly.

On this this particular project. By the time I joined, I think all of the important directional decisions had been made.

We had a well established pattern of slipped dates with low accountability that were leading us down the path to destruction. Now, knowing what I know now, there are a lot of things that I would do differently.

But back in those days, I was more of a worker bee than I was a leader certainly had almost no positional power. And I was still learning what it meant to lead people in the from the middle with relationships and influence skills rather than formal positional power.

Today, if a leader was in a project like that, and I was offering them some advice, I think I could offer several options to help. But by the time the project had literally been running for years without anything in production, I think this project was was pretty much doomed.

I think Eeyore was right about this one.

I’m reminded of that scene in The Princess Bride where the heroes are arguing with Miracle Max about whether Wesley is all dead or mostly dead. Sadly, by this time, our Wesley was all dead.

All we could do, as Miracle Max said was root through his pockets looking for loose change.

Sometimes your best option is to declare the project dead so you can move on.

And sometimes the reality is the best outcome is to kill the project, capture some great lessons learned and start looking at better options.

Now, at the time, even once I became convinced this can work for us, I didn’t have the influence or the authority to make that a reality. So much time had been wasted on this failing effort that it literally could have put the VPS career in jeopardy.

I’m grateful. I’m grateful for another leader in the organization who was a possibility thinker.

He was one who load loathed the idea of letting the company down and was looking for alternatives. Jeff was good about that.

Jeff asked what has become one of my favorite questions to ask teams. I asked this question all the time. Jeff asked, “Exactly what problem are we trying to solve?”

This question is great because it is so so easy to fall into the trap of all the little details of projects and miss the big picture. He also asked an important follow up question, which is, if we had a way to make this work, how would we know?

I love this question, too, because it gives us a measurement.

We needed a measurement that would show that we were either making progress or we weren’t making progress.

It reminds me to remember that we are all learning together.

This is a team effort.

It’s not that you have the one hero who comes in to save the day, we are working as a team to solve this.

I appreciated what Jeff had to say about helping us reframe what was going on.

As you’re working on projects, look around you who can be helpful, who has good insights, even Eeyore had a necessary message “We’re doomed.”

Truth telling is often unpopular.

Now. I do want to take just a moment to tell a quick story about another experience I had with your now as it happens, Eeyore was right about this one thing, but he had developed a habit of complaining, he had created a negative attitude.

Now many times I’ll have a team member who’s pessimistic say, I just can’t help it. It’s the way things are. It’s not possible to be positive when so many things are so negative. That’s life.

But I’ve got a funny little story about a your despite his claims that the deck was stacked against him, I was able to prove conclusively that His attitude was entirely under his control.

His attitude was a choice that he made.

Years ago, when my grandmother decided that she was no longer able to drive she asked me to sell her car for her. Now this car really was that proverbial used car driven by the little old lady who drove it to the grocery store and church on Sunday.

Now I knew that Eeyore was looking for a reliable car for his son to drive and I asked if your might be interested in buying it.

But there’s one thing I know about cars, cars need maintenance.

And there’s always a story to tell about a problem with a car, something breaks wiggles that rattles payment falls off.

When Eeyore said he was serious about buying the car, I knew I was going to write up a contract for it.

I knew I did not want to hear about that car ever again after the sale.

I also knew that Eeyore was well, he was tight with $1.

I wanted to protect myself.

Thankfully I had a rare flash of brilliance. When I wrote the purchase agreement. It literally said, “If the buyer ever complains about anything related to this car, he will immediately pay the seller $5 cash for each negative statement made.”

When he or saw the purchase agreement. He smiled broadly. He laughed loudly and then he signed the contract.

So you know what happened? He never ever said a negative word to me about that car, your drove that car for 160,000 more miles over the next several years.

Every single time. And I mean, every time I saw him, he would smile. And he would make a point of telling me what a great car that was and how well it was working for him. Every single time Eeyore’s complaining had become a habit and he could choose to be positive.

I have no doubt that car needed all kinds of work over the years. But he never mentioned that to me, he focused on the good parts. I jokingly called him Eeyore, but I want to point out that he is a man I respect. He works hard for his employer and he works hard to care for his family.

His attitude was something he could change.

But back to our project back to Jeff’s question, “what problem are we trying to solve?”

Well, we really figured out we had two problems that we had to solve

first, what do we need to do technically, to make this thing work? And

second, how do we handle the political fallout of a seven figure project failure?

Now, many of you might be saying, “Look, I’m the tech guy, the political stuff, that’s not my problem. That’s for the suits to solve, not me. If they hadn’t come up with a bad idea like that to begin with, they wouldn’t have had this problem at all.”

I will say it is super handy if you can just wash your hands of a problem set like this.

But if you want to be a top performer, solving political problems is at least as important as solving the technical ones.

So what do we do?

Well, the first thing was to set up a small tiger team to investigate alternatives.

Jeff and I began to brainstorm about the root of the problem we needed to solve and we brought in some subject matter experts who helped us unpack the mess.

At the outset, we needed to keep it hush hush. The full project team continued to work toward delivery according to the published plan, but a few of us began really digging into what problem are we actually trying to solve?

On the technical and software side we realized a great Do the problems we were we were trying to solve involve working around how the tool needed to be configured because of the way our environment worked. We were fighting the tool the whole time, because we insisted on trying to make a square peg fit into a round hole.

Now, we did use the “5 why’s” technique, and we uncovered that the core that we needed to address could be a lot simpler.

We didn’t see it initially, because we were trying to figure out how to use that one product in our specific environment.

You know what else we discovered? after we had been working on this project for a couple of years, we knew a lot more about the problem than when we first started.

We had rushed to start to build a solution before we fully understood the problem we were solving. Before we could conceive of a minimum viable product that was achievable. We started writing code.

Tom Cooper 20:51

On the business front that the death march efforts so far, it allowed us to gain a lot of clarity about the complexities of the business processes and what was really causing pain in those areas.

We simply had not understood these things at that level when we started the project.

And on the technical front, we had some good news too.

By that time, we were really down to two directories.

And the technology from the other vendors had gotten better as well. It was true, we could not find a one stop shop like this vendor that was supposedly going to fix everything. But we did find a collection of tools that could potentially be combined to get us across the finish line for the biz biggest business problems were facing.

Even better was that the tools we believe could rescue us were sold by vendors we had good relationships with and we’d had decent experience with them in a really limited fashion in other parts of the business.

So we had access to knowledge and to lab copies of the software.

Our tiger team squeezed in some time to prototype and test and fail and fail and fail, try again and test and eventually succeed on a limited scale.

One other key difference is that we could do this incrementally and not depend on that big bang cutover.

So after a couple of months of prototyping in the shadows, we became convinced that the tools could do what we needed at least most of it.

But we were still on the death march, right?

Unless we could convince the powers that be to agree that a this project needs to end be we have another option that could fix the business need, and see that we could find a way for our senior leadership to save face, that that money wasn’t a complete waste. If we couldn’t do that we were going to have to keep going.

Eventually, we got some good news on that front, because IBM was getting sick of dealing with all the headaches and the massive cost overruns. They had an incentive to get out of this deal too, and our tiger team was able to present our findings to the VP. And she began to negotiate with IBM about potential alternatives so that we would not take a complete loss on the money we’d spent now. Over the three plus years we were working on this project, we had spent multiple seven figures on licenses, professional services and internal labor.

Our winning play turned out to be the we shared that smaller risk bite size approach with a larger team, and then with the peers of the VP and IBM agreed to trade us out licenses for alternative software with a retail price close to what we’d already spent with them.

So that kind of got us off the hook. And one day, we were able to stop working on that original project and we pivoted to the new suite of tools and smaller milestones.

Over the next year, we managed to roll out a solution which allowed us to scale the number of accounts and users without having to hire more team members for manual provisioning.

We also finally had a solution which could automate that provisioning and de provisioning of user accounts.

Now, as I’m wrapping this up, it’s always easy to blame others for project failure.

The roots of failure for this project were first we didn’t understand what success looked like.

We also didn’t understand how complex the business processes were or how hard it was going to be to automate them. That optimistic vendor sales team, they made promises their tech team simply couldn’t back up.

We didn’t establish checkpoints to help us fail fast and learn fast.

And also culturally, it wasn’t safe to say this was going to be this is going to fail.

As I said before, experience is what you get when you didn’t get what you wanted. I gained a lot of experience in this season of my career. And that’s today’s software horror story.

Tom Cooper 24:24

Do you have a story we should share on a future episode of the podcast?

What are you learning from your software projects? Are you dealing with a frustration on a current project and you just like to talk about it? Let’s connect head on over softwarehorrorstories.com That’s one word softwarehorrorstories.com And you’ll find a link to book a free quick call no strings attached. I’d be happy to help!