AI transformation — success beyond data-quality perfection

Introduction

The argument: “You can’t create good AI solutions and benefit from the AI revolution if your company has bad data…” seems to be circulating in analytics circles. Some of the proponents often start their logic with: “garbage in → garbage out”. They go on to structure their argument around that if you build AI solutions on top of an IT stack with poor data quality, then those problems will propagate down and those AI solutions will also suffer.

This is of course true for many of the traditional AI and ML solutions, but completely misses the ball on how the new gen AI solutions can be applied to create value in an organization. More specifically, the argument shows a very limited understanding of the current AI revolution and the increasingly profound capabilities we are seeing from the generative AI models. I would argue it is perfectly possible to build strong generative AI solutions that work largely independently of existing IT infrastructure.

Consider the following scenario:

You manage a company with 10 case workers and one data warehouse analyst. The case workers receive applications related to payouts in paper envelopes, and on a case-by-case basis can decide to either approve or deny the payout. The data warehouse analyst makes reports and analysis of the workers and generally does a bad job. He creates a lot of garbage data and his reports go straight in the junk mail folder.

One day you decide to bring in a gen AI specialist and you make virtual agent that can perform the tasks of the case workers using an enhanced version of ChatGPT (or similar class of models) with a RAG of your current internal knowledge added to its capabilities. You observe that in 95% of cases the new virtual case worker is at par or better than the it’s human counterparts, and in 5% of the cases it is uncertain in its decision.

Based on this — and assuming your company operates in a region with favourable employee protection regulation — you fire 9 of your 10 case workers. (Perhaps more commercially minded workers could retrain as sales agents?) The last remaining case worker is then directed to spend his/her time scanning the payout applications into .PDF format and emailing them to the virtual case agent. (This low-level manual work still has a role to play, however the robots are coming for him in the not-too-distant future.) In addition, the case worker also still operates the last 5% of cases where the virtual agent was uncertain. Your data warehouse guy is still making garbage reports and a lot of your company data is totally wrong — but hey, who cares? You just used gen AI to make 90% of your case workers redundant.

So, what’s changed? Why can we now use AI models to transform operations differently that just a few years ago? The answer lies in a few key differences between the traditional models and the new generative AI models. These are especially related to how the models are trained, what data they used for training and what they can be used for. Below we highlight some of these differences.

Traditional Models

Traditional machine learning models are typically trained on carefully curated data that is very domain specific. Consider a company that is looking to build a churn model. This will require specific and detailed observations of the customers base, and importantly a flag to indicate whether they churned within some time interval. Usually, this data will be mostly internal customer data from various sources within the organisation, perhaps enhanced with some external data points.

Being dependent on a lot of internal data, such a model needs to have strong quality controls on the training data (and the data used for inference later) in order to provide reliable and trustworthy predictions. If the training data is bad, the predictions will be too. This is typically where we get the “garbage in garbage out” relation.

After training — assuming the company now wanted to use the churn model — they would typically first need to host it somewhere in their IT infrastructure and then they would need to feed it with the same type of data for inference — or in this case producing churn predictions. So, the data pipeline would need to carefully monitored for quality and consistency.

Continuing with the churn model example, the model would now also have a very limited scope, namely producing churn predictions. It would be not applicable for much else (I’m not implying that this is a bad thing — I implement lots of churn models with my clients because they can be incredibly useful, even with a limited scope.) But try using it this model to predict anything else and it would fail miserably.

Generative AI Models

Contrast this with one of the new generative AI models, for example GPT-4 by Open AI. Firstly, this model is pretrained by Open AI so you don’t really need to worry about training at all. (The model has been trained on a vast corpus of varied data, some sources claim over 1 trillion tokens, and at a cost which Sam Altman says to have exceeded $100 Million.) Because of this, it becomes clear then that the training requirements for traditional models are much stricter.

Secondly, to run inference (obviously depending on what the model is being used for ) the model can be used with an API and hosting of the model does not need to be considered. (Unless you are hosting your own model.)

Furthermore, the application areas of the new generative AI models are vastly larger than the traditional ML models. You can use these models to write and analyse text, generate video and sound content, etc. They don’t have one specific domain they excel in, but many, and maybe most importantly, they are rapidly advancing in their capabilities.

Consider the table below which is sourced from the technical report by Anthropic where they detail the performance of the Claude 3 model range:

The improvements from just GPT 3.5, GPT 4 and then in some cases Claude 3 are very strong. And we are just in the beginning phase of the generative AI cycle. Given the strong capabilities of these models and their rapid rate of improvement it should be clear that implementing them should be a matter of if, but “how soon can we start?!”

Conclusion

In summary, while data quality still is a strong requirement for traditional ML models, depending on the application area it doesn’t need to factor in the same way with generative AI models. Companies looking to stay ahead in the digital age should pursue AI transformation even if their data quality isn’t perfect.

I would also argue that the differences in how generative AI models are trained and applied makes them ideal to be used in transformation projects. Especially for cost cutting projects and efficiency increases in the near and medium term. Given their rapidly increasing capabilities, in the longer term they will likely become ubiquitous in most knowledge work.