021: What kind of AI solution do I actually need?

I’ve been a Data Scientist for more than 10 years now. I’ve seen a fair few ways that people use data as part of their businesses. I’ve spent my days building analytics solutions that solve different types of business problems. Through this I’ve learned some decision frameworks that have helped to decide what type of solution to deploy.

With the advent of Generative AI (GenAI), I’ve experienced a shift in how people build analytics solutions. This has brought some tension in how to adapt old decision making frameworks for this brave new world.

A common question I find is,

The short answer to this question is: NO.

The slightly longer answer is that traditional analytics and GenAI answer different questions.

I’ve always used the Analytics Pyramid as a reference for what type of problem to solve using analytics. More recently, I have added another metaphor – The Ladder of AI Autonomy – to address how much of the problem solving is delegated to AI.

I’ll explain my thinking in this essay and bring it all together with a simple decision framework at the end. This framework can be used to structure conversations where stakeholders want to “use LLMs” or replace traditional analytics workflows with fully autonomous solutions and don’t know what level of AI autonomy to choose or what the implications thereof can be.

Here are my thoughts.

The Analytics Pyramid

A Chief Analytics Officer I worked with always described analytics to non-technical stakeholders as a pyramid.

The base of the pyramid consists of the company’s data layer. This needs to be robust and reliable before one can build analytics. ‘Nuff said. End of. Fact. #GIGO.

Once the data layer is hardened, companies can start doing descriptive analytics. This builds on the data layer by answering questions like “who did what and when did they do it?”

Descriptive analytics has a backward-looking view of historic data.

In the popular Gartner Analytics Maturity Model, descriptive analytics is followed by diagnostic analytics. The distinction between the two is, generally, that descriptive analytics answers “what” type questions and diagnostic analytics answers “why” type questions.

I group them together, since the data people that I’ve worked with that are worth their salt, never answer a “what” question without delving into “why” questions – even if only at a surface level.

As one gains comfort in answering questions about the past, another layer is added to the pyramid called predictive analytics. This is where traditional Machine Learning (ML) comes in. It builds on the data layer and descriptive analytics by assuming that the historic data will translate into the future. It answers questions like, “what will happen in the next quarter?”, “who is going to default on the loan I gave them?” and “how should I spend my next budget?”

Predictive analytics essentially answers questions of the future.

What predictive analytics doesn’t do, is tell you what you should do. The estimates derived from traditional ML models are associative or correlative in their nature. Regression coefficients are interpreted as “if you keep all other things the same, y will change by this much if you adjust x by one unit”. That sentence only holds true if the data don’t change. Which they do, and why model monitoring, re-training and redeployment is a key skill for a Data Scientist.

The final layer of the traditional analytics pyramid is prescriptive analytics. This is where tools like Causal Inference and Recommendation Engines come in. It answers questions like, “does raising the minimum wage reduce employment?”, “does an email campaign increase sales?”, “would you have clicked on the ad if I hadn’t shown it to the user?”

It answers questions not of the past, present or future, but of alternative realities.

These alternative realities are created by creating counterfactuals. And to touch on why I don’t consider diagnostic analytics as its own separate category, this is where “why” questions truly need to be answered.

As one moves up the analytics pyramid, the timeframe and complexity of the problems solved changes: Past, present, future, <alternative universe>.

Also note that the analytics pyramid narrows towards the top. This represents the amount of uncertainty that is compressed at every level. At the data layer, you have millions of rows of data. At the prescriptive layer, you have as little as one decision.

The Analytics Pyramid is therefore useful to answer this question:

And it describes the type of intelligence required to solve that problem.

Moving on to modern AI the following question now exists for Data Scientists:

Firstly, what does modern GenAI do? Typical descriptions of tasks that we use GenAI for are:

• generate,

• reason,

• plan, 

• orchestrate,

• summarize,

• search,

• act,

• sense.

Some things, like generate, reason, act and sense could be argued as a natural next step in the pyramid since they produce new types of judgements (which is what each layer of the pyramid already does).

To me though, things like summarize, orchestrate and search don’t naturally slot in at the top of the pyramid since they manipulate/route existing information without adding a new layer of judgement.

What AI does do, without a doubt, is change who performs the work.

This requires another image to describe the place of modern AI in the Data Scientist’s Toolkit.

The Ladder of AI Autonomy

I’m borrowing the ladder image from Judea Pearl’s Causation Ladder. This ladder image is useful for our purposes, since it can also represent the degree of autonomy that AI has in decision making:

On the bottom rung of the ladder, humans execute everything. This is the status quo, pre-Transformer architecture and coding assistants.

On the second rung, humans are assisted by AI.

On the third, humans approve AI work.

On the fourth, humans supervise the AI while they work.

(The third and fourth rungs are effectively the “human-in-the-loop” rungs with differing amounts of human intervention opportunities)

On the fifth, the AI is fully autonomous.

The Ladder of AI Autonomy is therefore not an extension of the Analytics Pyramid.

GenAI therefore doesn’t extend the Analytics Pyramid, it is an enabling component that can be inserted into the existing layers of the Analytics Pyramid.

Here are a few examples of this in action at different layers of the Analytics pyramid.

1. At the descriptive analytics layer, you stay on the bottom rung of the ladder if you’re producing the dashboard that displays your company’s sales figures, but you allow the AI to climb a few rungs to find possible causes/descriptors of a recent decline.

2. At the predictive analytics layer, unless you are calculating the regression coefficients by hand, you start on the second rung by default. AI climbs a few more to describe possible drivers of the model’s answers.

3. At the prescriptive layer, you are already making “alternative reality” predictions such as product recommendations. AI can move to the top rung by negotiating with the customer, following up on prompts and executing orders.

Here is why the ladder image becomes even more powerful.

Ask yourself, what happens as you climb up a ladder?

Two things stand out:

1. you get to see more of the landscape around you, but

2. you introduce more risk.

Let’s discuss each in turn:

You get to see more of the landscape around you

As you climb up the ladder, the outcomes become less deterministic. It doesn’t mean that the outcomes are more stochastic (or, random), though. A fully-autonomous LLM is just as random as a fully-supervised one. Randomness is a property of the model, not of the autonomy rung. The loss of determinism has another source. What actually increases as you climb is that more of the option space becomes visible. This means that there are fewer constraints left to pin down a right answer for your current task.

Note that I say “a” right answer, not “the” right answer. Valid answers don’t magically resolve to action, since only one decision can realistically be actioned at a time.

Imagine a scenario where you have built a predictive model and tasked a LLM to identify improvements on it. It could generate a year’s worth of analyst work with a few prompts.

When AI is doing the selecting among a wider option space with looser constraints, the human's job shifts from verifying one deterministic output to adjudicating among several defensible ones – which is a categorically harder supervisory task.

Adjudicating one wide option space is already hard – adjudicating dozens of them, faster than any team can action, is a different and even more difficult problem. Especially if you don't have the proverbial "skin in the game" in understanding how a solution was derived.

When you need to judge between multiple defensible solutions, you're abstracted a level up from doing the actual work. All the traditional training most Data Scientists will have had have been geared towards building - not judging - at this scale.

As I've been using coding assistants like Claude Code more extensively and climbing the ladder of autonomy, I've also realised these two things:

1. To become someone who can exercise this type of judgement, you need to purposefully slow down and break the momentum that moving up the rungs of automation give you - a surprisingly unpopular thing to do in a time of "yay-we're-building-stuff"-excitement.

2. It is really difficult to exercise caution and carefully assess options presented by AI when the barrier to accepting the options are as low as simply clicking "Yes" multiple times a day.

Which brings us to the next reason that the ladder image is so powerful.

You introduce more risk

The higher you climb the autonomy ladder, the greater the leverage, but also the greater the consequences of failure. If you fall off the first rung of a ladder, you bruise your ankle. If you fall off the top rung, you need an ambulance – or worse.

Imagine a scenario where an agent is operating at the top rung of the ladder by negotiating autonomously and directly with a customer over a payment dispute. This simple situation introduces regulatory exposure that a human call centre agent equipped with an accurate dashboard of the customer’s payments never could.

At a lower rung, that same agent might only draft a settlement proposal for a human to review. If it hallucinates a resolution that goes against company policies, a human can still read it before anything happens. At the top rung, the agent is authorized to negotiate and settle without review. The same prompt that produced an embarrassing draft at rung three now produces an executed, irreversible refund at rung five.

Also, this person could lose trust in the channel if they were to find out after the fact that they were negotiating with an AI instead of a real human – regardless of their opinions about AI in general.

An AI that is compromised via malicious prompt injections could be manipulated into destructive actions – wiping a database or spending tokens that can bankrupt a company.

Higher autonomy therefore increases the need to manage operational risk, regulatory obligations, human trust and economic cost.

Each higher rung therefore requires additional guardrails to mitigate the increasing risk.

The ladder image makes it clear that introducing AI into a decision making framework makes it possible to see more possible answers. They don’t come without a cost, though, since you need to pay for the increased optionality with more stringent risk mitigation.

The decision framework

This distinction between “type of intelligence” and "degree of AI autonomy” brings us to a very simple and practical decision framework to answer the question posed in the title of this essay. Note that this framework doesn’t go into the specifics of AI guardrails, which is a topic for another day.

What kind of AI solution do I actually need?

1. Determine the analytical problem using the Analytics Pyramid.

   1. If you are generating data – you’re in the data layer.

   2. If you’re asking backward-looking “what” type questions – you’re in the descriptive analytics layer.

   3. If you’re asking forward-looking “what could happen” type questions – you’re in the predictive layer.

   4. If you’re asking alternative world questions about “what should happen” – you’re in the prescriptive layer.

2. Ask whether deterministic software solves it adequately.

   1. Answer yes if rules will suffice – I’ve not discussed this at length here, because it is somewhat self-evident. If you have a problem that can be solved with a simple rules-based approach, adhere to Occam’s Razor and go for the simplest solution that requires the fewest assumptions.

   2. Answer no if your analytical solution needs to be enabled with tools like generation, language, reasoning, interaction, synthesis or planning.

3. If you answer No to question #2, take out your Ladder of AI Autonomy and decide how much authority you want to delegate to the AI.

   1. If you want to do all the work yourself – you’re on Rung #1.

   2. If you want to do the work, but be assisted by an AI – you’re on Rung #2.

   3. If you want to let an AI do the work, but approve key decisions – you’re on Rung #3.

   4. If you want to let an AI do the work and you simply supervise it – you’re on Rung #4.

   5. If you want to give an AI all the reigns – you’re on Rung #5.

   6. Things to consider in your decision:

      1. How many options do I want to have available when delegating the task to AI?

      2. How many guardrails do I want to add to the problem solving process?

      3. The higher up you go on the ladder, the more visible options you will have from point 1 and the more guardrails you’ll need on point 2.

To the future

In my 10 years as a Data Scientist I have seen the use of data to solve analytical problems evolve from “dashboards for everything” through ambitious prediction and forecasting models during the pandemic all the way to the current day where many people want to understand “why” things happened using prescriptive tools.

What hasn’t changed is the type of questions that can be solved by the tools in the analytics toolkit. What has changed completely with GenAI is who performs those tasks. It also led to new risk mitigations to be invented. And, it’s opened interesting philosophical questions like “who is responsible if the AI makes irreversible decisions”. These are all factors that are continuously being explored at breakneck speed in academia and industry.

My simple decision framework provides a perspective on how to frame old problems in a way that enables a Data Scientist to tap into the benefits that GenAI can bring by using it as an enabling component instead of a replacement technology.

It also gives me a reason to pause and think how this framework will change in another 10 years’ time. What cracks will appear in the data layer as we move to a world where much of our data will be AI generated? Will the Analytics Pyramid grow new layers above the prescriptive? Will rungs on the AI Ladder collapse as trust in AI increases? Or, are there rungs beyond the “fully autonomous AI” that we are yet to uncover?

Whatever happens, I’m sure there’ll be frameworks to devise to make sense of it all – human intelligence-style.