Pick the right problem: How to Avoid AI Project Failure
AI is everywhere. There’s lots of opportunity for companies to cut boring busy-work by leveraging AI, resulting in efficiency savings and allowing staff to focus on high value activities. Many companies are embarking on projects to integrate AI with their business for this exact reason.
But as many as 95% of projects using Generative AI are failing (according to an MIT study), particularly ones focussed on Sales and Marketing use cases.
This can cause a problem for those trying to set an AI strategy and gain executive buy-in for it – is it possible to avoid this failure rate? We think so (even bespoke build projects), and here’s how you can ensure yours is one of the 5% that succeed:
1. Pick a problem with a provable return on investment:
The first step for any project, AI or otherwise, is to ensure you’re targeting an area that will result in a return on investment. For new technology such as AI, achieving an easily provable ROI is a surefire way to build momentum for an AI roadmap. Proving utility to the business is the first step when prioritising projects into an AI strategy.
Often, it’s the least sexy areas of business, particularly back-office admin, that are easiest to add value to with AI. Processes that deal with ambiguity have historically been difficult to automate, as computers required rigid, rules-based programming. With Generative AI, this is no longer a constraint. Areas of business that were previously difficult to automate are now significantly easier and cheaper to develop solutions for.
Look to streamline your operations by finding repetitive and time-consuming tasks that have been overlooked in the past due to perceived complexity. Picking a problem that is easy to evidence and easy to calculate a return on is by far the most important way to avoid project failure.
2. Pick a problem that LLMs are designed for:
GenAI and LLMs are tools with specific capabilities and, when used correctly, perform exceptionally well. At their core, LLMs are excellent a processing text (and Agents allow this text processing to affect the real world). However, they have some limitations:
LLMs aren’t designed to calculate: the famous test is to ask ChatGPT how many times the letter “R” appears in the word “strawberry”. While some of the latest models are much better at this, there are easier and cheaper ways to perform computation. Consider traditional data science or plain old automation before bringing in LLMs when you need to count or compute something.
LLMs aren’t designed to be 100% accurate: More commonly known as “being wrong”, LLMs can “hallucinate” incorrect and outright bizarre information. When they do, they do so confidently, making it difficult to tell when they’re right and when they’re wrong. Avoid this being an issue by picking tasks such as summarisation where you are feeding LLMs the data, and making sure your problem domain can handle some level of incorrectness from the AI.
LLMs aren’t designed to have up-to-the-minute information: LLMs are limited by their training data and only have knowledge of events up to a certain date (their knowledge cut-off). As with hallucinations, if you’re feeding the LLM the information, you can avoid this issue. Alternatively, you’ll need to provide a search tool to allow the LLM to find the information it needs.
You can ensure the success of your project by avoiding problems that require LLMs to do these tasks, and instead find problems that involve processing text in a way LLMs are designed for.
(see our article on ways to use AI for customer focussed companies for more ways LLMs can succeed)
3. Pick a problem that LLMs are good at (or good enough at for your purposes):
We’re still climbing the capability curve of GenAI – even if LLMs are designed for the task you’re looking for, they may not yet be good enough at it. The long-term direction of GenAI is clear: the technology will get better and cheaper. LLMs will “hallucinate” less and through agents they will be able to affect the world around them. Every metric, benchmark and task they currently do is set to improve – summarisation, ideation, coding, classifying will all get better. Our understanding of prompting and managing context will increase, resulting in better outputs. The cost of running models will come down, enabling usage on end-user devices and making its use economical for more activities.
However, for your AI project to be successful today, you must consider the current capabilities of AI. Even if GenAI is conceptually capable of your task, you should make sure it is good enough at it to warrant kicking off a project. Before committing to a large build effort, make sure to evaluate the AI model, prototype the application it will be integrated with, and continue development iteratively once proven.
The best way to think about this is “don’t try and build Netflix while everyone’s only got dial-up internet”. Some products need technology to reach a certain capability before being worthwhile. GenAI has a well-known level of capability at different tasks, and to succeed you must prioritise projects where AI’s current capabilities in the required tasks are sufficient for your purposes, and put on hold projects where AI is not yet capable.
Go forth and prosper prioritise
If you’ve found a problem that meets these three criteria, you’re off to a strong start and can prioritise solving this over other initiatives. If you avoid the hype by focussing on ensuring business value while making sure LLMs are the right tool, you’ll avoid being one of the 95% of failures.
If you’ve got a lot of problems that meet these criteria, consider using our AI Strategy grid to help choose between them (and check out our Determine service if you’d like some expert help). If you’re ready to go, we’re ready to help – we always recommend creating a prototype to prove your idea, and our Demonstrate service helps companies prove business value for AI ideas with working software.