You would think that large, experienced companies use the best of the best tools, processes and software development teams when starting new projects. Sadly, you would be wrong - not all do, and there have been several famous IT outsourcing fails in recent memory. From airline software built for $9/hour by remote developers, to an online patient records system that caused tremendous financial losses, these are the stories of mistakes we should learn from.
Boeing’s 737 Max and software development outsourcing
It’s often said that travel by plane is safer than most other means of transportation. Yet in this case, faulty software might have caused tragic consequences. While the investigation is still in progress and we don’t know the exact causes of these events, there were obvious signs of something wrong happening within the company. At the time, Boeing went through a series of lay offs affecting experienced engineers. They also negotiated with suppliers to cut costs. As such, a good chunk of their workforce was temporary, contracted for as little as $9 an hour, often working for large Indian companies. Choosing an outsourcing partner based solely on their pricing (rather than their experience or technological mastery) could have easily been one of the factors leading to two plane crashes.
That is not to say that Indian developers are inherently worse than any others. However, those who work for rates as low as those mentioned above are either inexperienced or undervalued, thus less likely to deliver high quality code. Using their services is more like gambling than making a rational business decision: the risks usually outweigh the potential gains. Even building a simple website through an unreliable contractor can cause major issues. Imagine your main online marketing and sales platform suddenly failing during a major seasonal advertisement campaign (e.g. during Christmas promotions, or in the early days of a new challenger on a very competitive market) because the website wasn’t optimized for high traffic. In this scenario, the campaign was a wasted investment and your business’ reputation has been tarnished. In other cases, consequences include legal issues, for example through badly implemented data protection.
Boeing made the mistake of contracting people who had no real background in aerospace engineering, yet they developed and tested software responsible for passengers’ lives (software for flight-display, and flight-test equipment). The company’s local engineers report on the low efficiency of such a development process, as the code written by the contractors to meet specifications often required corrections, sometimes over and over. Communication was made extremely difficult by distance and the language barrier.
Boeing disagrees with the opinion that the work of these low-paid developers is connected with the Lion Air crash (October 2018) and the Ethiopian Airlines crash (March 2019). Both disasters occured while the planes were climbing to their cruising altitudes, when the MCAS system, responsible for automatically preventing stalls, pointed them down when it shouldn’t have, likely because of a malfunctioning sensor. Engineers who worked on the Max software report that the company’s focus at the time was cost reduction. Senior engineers were told they weren’t needed at Boeing, as the company’s software was mature. Boeing preferred to provide high-level specifications to contractors and chose to trust them with problem-solving.
You know you’re in trouble once you start thinking about software engineers as a commodity. And don’t be fooled by the low hourly rates - someone is going to have to manage and oversee those offshore teams, and the cost of their work will make the entire thing seem less like a bargain.
Boeing has conducted simulations and hundreds of test flights to find the origin of their problem. They seem to have found it, and are in the process of introducing a fix. However, it might take them some time to regain the trust they lost upon the tragic deaths of 346 people.
Development costs of the State of Indiana’s failed welfare system
It’s worth mentioning that Indiana contracted a large, well-known US corporation for this job, one that has a generally good reputation and has been on the market for many years. And yet this time, the project was a failure and both sides of the contract sued each other, which is how the details of the project became public knowledge. The failed system cost $1.4 billion to develop. The cause of failure in a nutshell? Resistance to change.
Indiana wanted to update their welfare processing system through outsourcing. Despite similar projects having been attempted and failing in Texas and Florida, their technological partner accepted the contract. It’s likely that the size of the contract was a deciding factor in accepting it - even though it came with proportional risk. (Several smaller contracts are less risky than a single large one. Sometimes, it might be worth it to break your project up into relatively independent parts.)
On the side of Indiana, the main issue was their resistance to change. The state referred to what they wanted to fix as “the nation’s worst welfare system”. They intended to switch to a new delivery model, hoping to reduce costs and help with their history of regulation violations. Both of these goals are incredibly ambitious on their own, and moreso in tandem. They required full commitment from both the technological partner and Indiana over the duration of their 10-year contract.
In this case, the state took a controlling approach to the change mechanism. They did not approve most of the changes that their technological partner requested of them, even though the conditions of their contract didn’t accommodate reality, as new programs cropping up over the years expanded the amount of work to be done. Indiana also blocked their partner from putting more resources into the project, as those would mean added costs. The length of the contract complicated matters further. Time brought unexpected change, which the State of Indiana resisted. This could have only lead to disaster.
One of the lessons to take from this is that a massive contract, meant to prepare you for any eventuality, will never do its job. Something unpredictable will happen, and stubbornly holding onto old assumptions isn’t a good change management plan. This particular welfare system required ten years of collaboration between Indiana and their partner. It was simply unrealistic to think that the original contract would be enough after all that time. Policies can change in a matter of days, and expecting technology to remain stagnant over a decade is just insane. In the case of Indiana’s welfare system, the real victims were the taxpayers, whose money was wasted on this failed project.
We can’t claim that there’s a straightforward way to make sure a huge, 10-year-long project will progress without issue. However, Agile development practices could have prevented a lot of the issues that occurred here. Using a Time & Materials payment scheme (rather than a fixed price approach) could have made it easy to scale the project up or down, as needed, or introduce sudden changes. The technological partner in this story was as much at fault as Indiana: with their extensive business experience, they should have known they wouldn’t be able to meet the state’s expectations with the contract and conditions they agreed on.
Medical software development for the Cambridge University Hospitals NHS Foundation Trust
The goal was to introduce a new patient record system. The NHS in the UK has a track record of spending a lot of taxpayers’ money on similar failed projects, even though that type of software isn’t exactly rocket science. At Cambridge, a US corporation was hired to do the job. The system was supposed to let medical staff access patient records via mobile devices, to speed up their work. The results? A 20% drop in performance, which, when you’re dealing with people’s lives and health, is quite a lot.
The Trust was not prepared for how huge of a challenge it would be to properly implement their new system. Staff was confused and undertrained in using it. This is a common paint point in introducing new software to public institutions, often caused by poor UX/UI and a lack of product design experience on the development team, or by cutting expenses through omitting a proper user-centered design process, with research and prototyping. These mistakes can be easily made in good faith by people who are simply unfamiliar with the usual pitfalls of software development, and who work with a tight budget. But the truth is that using the right approach can lead to incredible savings. Well-designed software often doesn’t even require training - it can be simply introduced to staff, as it fits into their every-day processes and is easy to understand. The case of the Cambridge University Hospitals NHS Foundation Trust’s failure is an extreme example of how costly it can be to omit crucial parts of developing high-quality software.
The system would also produce inaccurate discharges for patients, meaning some wouldn’t receive follow-up care. Information found within wasn’t always up to date, confusing patients and personnel. The results were a dramatic operational cost increase and a failure to deliver the system’s benefits to those who needed them. This means that proper QA processes were not implemented, as industry standard testing should have easily caught these issues, allowing the project team to fix them.
Avoiding outsourcing failure
There’s no easy recipe for overcoming the sort of problems described above, as each had complex, systemic causes. What we can do, however, is learn from those mistakes and use smart preventive measures before any issues have a chance to appear. Here are several suggestions that can help you manage difficult software outsourcing projects:
- If you’re dealing with a project of overwhelming scope, divide it into chunks. You can use Agile principles and methods with great success, as they have built-in measures against inflexibility and sudden change.
- Don’t rely on a contract to protect your interests, particularly in a long-term project. Be a proactive communicator and provide your technological partner with the information they need to serve your business needs.
- Never choose a low price over quality of service. It’s just not worth it - you stand to lose your entire investment in the project over this risky gamble, and often, partnering with an experienced outsourcing agency can help you limit costs through better planning, efficient processes and good quality assurance.
- Make sure your development team has the knowledge to produce quality work for your particular market. This is especially important if you’re working on a project in a high-stakes industry, such as aviation or healthcare.
- Don’t cut corners by neglecting good design. Effective UX and UI are some of the most important elements of modern software. Use them to your advantage to save on training costs or to delight your users and boost conversion.
- Remember that change happens more quickly than we expect it to, and that unpredictable issues are bound to occur in every project. Don’t ignore this, and instead work with your technological partner to use the new circumstances to your advantage.
- Finally, remember about the importance of quality assurance and post-launch maintenance. Even a team of experts can’t predict user behavior perfectly, and most newly created apps aren’t optimized for handling a lot of traffic. Be ready to adapt to problems and opportunities as they arise.
Conclusions
It has hopefully become clear that some of the largest IT outsourcing fails have been caused not by trouble with outsourcing itself, but by bad management, communication and a great reluctance to fix problems if it meant increasing project costs. Particularly in large-scale projects, such an approach is simply unreasonable - it’s always good to have a buffer for unforeseen events. And even when the app you want to build isn’t as complex or costly as the ones mentioned in this text, remember that the best way to ensure its success is through building trust and a good relationship with your technological partner.
If you’re worried about problems with building your web app through outsourcing, talk to us. We’ve delivered dozens of commercial projects over the years, successfully, on time and within budget.
