Challenges.

Enable reliable human override

A stop or override fails when the system is busy, degraded, or already mid-action.

Detect when human intervention is needed

The system keeps going through cases a person should have reviewed.

Prevent AI from bypassing human approval

The agent finds a path that skips the approval a person was supposed to give.

Define safe operating boundaries

The system has no clear limit on what it may do without a person.

Limit autonomous decision-making

Autonomy covers decisions that should stay with a person.

Prevent cascading autonomous actions

One action triggers a chain the operator never approved.

Safely deploy long-running agents

An agent that runs for hours can drift from the task it was given.

Sandbox AI systems

The system can reach tools and data outside the environment it was given.

Restrict system capabilities

The deployed system can do more than the job requires.

Prevent escape from defined environments

The system leaves the network, account, or tool set it was placed in.

Safely isolate malfunctioning systems

A failing system stays connected and keeps acting on live work.

Prevent harmful AI actions

The system can take an action that injures people, property, or rights.

Detect dangerous behaviour

Harmful patterns in outputs or tool use appear and nobody is alerted.

Prevent foreseeable misuse

Ordinary use can turn the system toward a harm the builder already knew about.

Reduce severity of AI failures

When the system fails, the blast radius is the whole task instead of a contained miss.

Prevent unintended optimization

The system pursues a metric that was never the outcome people wanted.

Detect reward hacking

The system scores well by gaming the measure instead of doing the job.

Prevent specification gaming

A literal reading of the instruction produces a result nobody asked for.

Detect divergence between intended and actual behaviour

What the system does no longer matches the behaviour it was built for.

Prevent unauthorized API actions

An agent calls APIs the user never authorized.

Prevent prompt injection from triggering tools

Untrusted text in a page or file makes the agent run a tool.

Restrict agent access to sensitive databases

The agent can read or write records outside its task.

Detect malicious tool use

Tool calls that leak data or change systems look like normal work.

Resist adversarial inputs

Small changes to an input flip the model decision.

Prevent model extraction

Repeated queries let an outsider copy the model behaviour.

Prevent model manipulation

Training data or updates change the model toward an attacker goal.

Prevent jailbreaking

A crafted prompt gets the model to ignore its safety rules.

Prevent data exfiltration

The system sends private data out through a reply, a tool, or a log.

Prevent supply-chain attacks

A dependency, dataset, or plugin changes the system before it ships.

Detect compromised AI pipelines

Training or deployment steps are altered, so the shipped system is not the one that was reviewed.

Prevent unauthorized model access

People or services reach a model they were not allowed to use.

Verify agent identity

A caller can act as an agent without proving which agent it is.

Manage agent permissions

Agent access stays broad after the task that needed it is done.

Prevent privilege escalation

The agent gains rights beyond the account it was given.

Implement least-privilege AI access

The agent holds permissions for data and tools it does not need.

Secure model hosting

The host that serves the model can be reached or changed by outsiders.

Protect inference infrastructure

Requests, logs, or weights on the inference path are exposed.

Secure model updates

An update replaces the model without a check that it is the intended one.

Detect compromised AI infrastructure

A breached host, key, or pipeline keeps serving traffic unnoticed.

Prevent hallucinations

The system states facts that are not in its sources.

Detect fabricated information

Invented names, numbers, and citations pass as real.

Verify generated claims

Claims go out without a check against a source.

Ground AI outputs in reliable sources

Answers cite nothing a reader can trust or open.

Handle unusual inputs

Odd formatting, languages, or edge cases make the system fail.

Resist distribution shifts

Accuracy drops when live data no longer looks like the test set.

Maintain performance outside benchmarks

Scores on a benchmark do not hold on the real task.

Handle ambiguous inputs

The system guesses when the request could mean more than one thing.

Detect when AI does not know

The system answers with the same confidence when it has no basis.

Calibrate confidence

Stated confidence does not match how often the answer is right.

Communicate uncertainty

Users never see that the answer might be wrong.

Trigger appropriate human review

Uncertain or high-stakes cases never reach a person.

Detect degraded performance

Quality falls in production and no alert fires.

Identify abnormal behaviour

Outputs or actions drift from the pattern the system was cleared for.

Detect silent failures

Wrong answers look normal, so nobody notices the miss.

Establish meaningful failure thresholds

There is no agreed point at which a miss counts as a failure.

Reduce contradictory outputs

The same question gets incompatible answers.

Maintain behaviour across contexts

The system changes its answer when the setting changes and the facts do not.

Ensure repeatable decisions

Rerunning the same case produces a different decision.

Detect unexplained behavioural changes

Behaviour shifts after a deploy or prompt change with no recorded cause.

Measure real-world capabilities

Lab scores do not say what the system can do on the actual job.

Evaluate agentic capabilities

Tests never check what the system does once it can use tools and take steps.

Test long-horizon tasks

Short tests miss failures that only appear after many steps.

Identify emerging capabilities

New abilities show up in use before anyone has tested them.

Test dangerous capabilities

Nobody checks whether the system can do serious harm if it is asked or steered.

Evaluate refusal behaviour

Refusals are inconsistent, so blocked requests sometimes go through.

Test boundary conditions

The system is untested at the edge of its allowed use.

Identify unexpected failure modes

Failures outside the written test plan reach users first.

Run adversarial testing

The system is only tested with friendly inputs.

Run misuse testing

Likely misuse paths are never tried before release.

Simulate agent attacks

Attacks that use the agent tools are not rehearsed.

Test real-world scenarios

Tests use toy tasks instead of the situations users actually face.

Develop meaningful benchmarks

Existing scores do not measure the decision that matters.

Avoid benchmark gaming

The system is tuned to the test and weaker on the real task.

Evaluate domain-specific performance

A general score hides failure in the domain where the system will be used.

Compare systems under equivalent conditions

Comparisons mix different data, prompts, or tools and are still treated as equal.

Monitor deployed systems

Behaviour after launch is not measured.

Detect capability drift

What the system can do changes, and the last evaluation is treated as current.

Detect changing failure modes

New ways of failing appear after release and stay invisible.

Re-evaluate systems after updates

A model or prompt update ships without a new evaluation.

Follow legitimate instructions

The system drops or rewrites instructions a user was allowed to give.

Resolve conflicting instructions

Two valid instructions collide and the system picks one without saying so.

Resist malicious instructions

Hidden or hostile instructions override the user request.

Preserve instruction intent

A literal follow-through misses what the instruction was for.

Translate human goals into AI objectives

The objective the system optimizes is not the goal a person stated.

Prevent proxy optimization

The system hits a stand-in metric and misses the outcome.

Detect goal misalignment

The system goal and the person goal diverge and nobody sees it.

Maintain goals over long tasks

A long task drifts away from the goal it started with.

Learn user preferences

The system ignores stable preferences the user has already shown.

Handle conflicting preferences

Two preferences clash and one is silently dropped.

Represent stakeholder preferences

People affected by the decision are missing from what the system optimizes.

Avoid imposing unintended preferences

The system treats its own default as the user preference.

Improve human feedback quality

Ratings and edits are too noisy to teach the behaviour people want.

Prevent reward manipulation

The system raises its score without improving the outcome.

Detect feedback gaming

Feedback is exploited and the signal still looks healthy.

Align incentives with desired outcomes

What the system is rewarded for is not what people need it to do.

Resolve competing values

Two values conflict and the system picks one with no record of the choice.

Handle individual vs collective interests

A choice that helps one person harms the group, or the reverse.

Handle conflicting stakeholder goals

Teams or communities want different outcomes and the system serves only one.

Make value trade-offs explicit

A trade-off is made and the people affected never see it.

Assign responsibility for AI decisions

A harmful decision has no named owner.

Define developer responsibility

Builders are unclear which failures remain theirs after handoff.

Define deployer responsibility

The organisation running the system has no stated duty for its decisions.

Define user responsibility

Users are blamed or excused without a clear line for what they control.

Record AI decisions

Decisions are made and not stored in a form anyone can review.

Reconstruct agent actions

After the fact, nobody can replay what the agent did.

Preserve decision evidence

The inputs and reasons behind a decision are discarded.

Enable independent audits

An outside reviewer cannot inspect how decisions were made.

Challenge automated decisions

People affected by a decision have no way to dispute it.

Provide appeal mechanisms

A challenge has nowhere to go once it is raised.

Correct erroneous decisions

A wrong decision stays in force after it is found.

Escalate high-impact cases

Serious cases stay on the automated path.

Allocate liability for AI failures

Harm happens and no party is clearly liable.

Handle multi-party AI systems

Several organisations share the system and none owns the failure.

Address autonomous actions

Harm from an action the system took on its own has no liability path.

Establish remediation mechanisms

People who are harmed have no route to a fix or a remedy.

Track data origins

Training and input data cannot be traced to a source.

Track model versions

A decision cannot be tied to the model version that made it.

Track system actions

Actions are not linked to the system, time, and account that took them.

Reconstruct system history

The sequence of changes that led to the current system is gone.

Explain consequential decisions

A decision that affects someone is given with no explanation.

Identify decision factors

The factors that moved the decision are hidden.

Expose relevant limitations

Users are not told where the decision is weak.

Communicate decision uncertainty

A consequential decision is presented as certain.

Disclose AI capabilities

Users are not told what the system can actually do.

Disclose AI limitations

Limits are missing from the description people rely on.

Identify system boundaries

It is unclear where the system stops and a person or another tool begins.

Communicate operating conditions

The system is used outside the conditions it was cleared for, and that is never stated.

Explain model outputs

An output arrives with no account of why.

Explain agent actions

Tool use and steps are invisible to the person accountable for them.

Provide user-understandable explanations

Explanations exist, but a non-specialist cannot use them.

Validate explanation quality

Explanations can be fluent and still wrong.

Track content origin

Generated or edited content has no record of where it started.

Track data provenance

Data moves through the system without a record of its source.

Verify transformations

Edits and model passes are not checkable against the original.

Preserve provenance across systems

Origin is lost when content moves to another product.

Identify AI-generated content

Synthetic text, image, or audio is presented as human-made.

Disclose AI involvement

A person is not told that AI produced or shaped what they are seeing.

Inform users about AI interaction

Users talk to a system without being told it is AI.

Prevent deceptive AI representation

The system is presented as a person, an institution, or a source it is not.

Prevent confidential data exposure

Private data appears in an answer, a log, or a tool call.

Prevent model memorization of sensitive data

The model can reproduce personal or secret training text.

Prevent agent data leakage

An agent carries private data into a tool, a page, or a later turn.

Detect accidental disclosure

A leak happens and no control notices it.

Obtain meaningful consent

Data is used for AI without consent a person could understand.

Manage consent over time

Consent is collected once and never revisited as the use changes.

Handle secondary data use

Data collected for one purpose trains or feeds another.

Respect data-use restrictions

Contractual or user limits on data are ignored by the pipeline.

Enable private inference

A query reveals more about a person than the answer requires.

Enable federated AI

Training across parties exposes one party data to the others.

Enable secure computation

Joint analysis requires pooling raw personal data.

Minimize sensitive data exposure

More sensitive data is collected and kept than the task needs.

Prevent inference of sensitive attributes

The system infers health, belief, or other sensitive traits the user did not provide.

Protect against membership inference

An outsider can tell whether a person data was in the training set.

Protect against attribute inference

Outputs or model access reveal attributes that were never asked for.

Limit unintended profiling

Interaction data builds a profile the person did not agree to.

Improve data quality

Training and decision data is wrong, stale, or incomplete and still used.

Enforce data access controls

Too many people and services can read the data behind the system.

Limit data retention

Personal data is kept after the purpose and the allowed time have ended.

Manage training data responsibly

Training data is gathered and reused without a record of rights or limits.

Detect demographic bias

Outcomes shift by race, gender, age, or another trait that should not decide them.

Detect language bias

Quality drops for languages other than the ones the system was tuned on.

Detect cultural bias

The system treats one culture norms as the default.

Detect contextual bias

The same person gets a different outcome because of context that should not matter.

Prevent discriminatory decisions

A decision disadvantages a group that should have been treated the same.

Detect disparate outcomes

Group outcomes differ and the gap is never measured.

Prevent proxy discrimination

A neutral feature stands in for a protected trait and reproduces the harm.

Correct discriminatory behaviour

A biased pattern is found and the system keeps using it.

Improve underrepresented data

Groups with little data get worse decisions and the dataset is not fixed.

Identify representation gaps

Missing groups stay invisible in the evaluation.

Evaluate minority populations

Average scores hide failure for smaller groups.

Reduce cultural blind spots

The system misses meanings, names, and norms outside its main culture.

Improve AI for people with disabilities

The system assumes sight, hearing, speech, or motor ability a user may not have.

Expand multilingual AI access

People who do not use the primary language cannot get the same service.

Design low-literacy AI interfaces

The interface assumes a reading level many users do not have.

Design accessible AI interaction

The way people operate the system excludes assistive tech and alternate input.

Identify who benefits

Gains from the system go to groups that are never named.

Identify who bears costs

Costs fall on people who do not receive the benefit.

Prevent unequal access

Some communities cannot reach the system others use.

Measure distributional effects

Benefit and harm are reported as an average, so the split stays hidden.

Detect AI-generated misinformation

False claims produced by a model spread before anyone flags them.

Reduce false information generation

The system produces false claims a reader will treat as fact.

Prevent AI amplification of false claims

Ranking, summaries, or agents give false claims more reach.

Verify AI-generated claims

Generated claims are published with no source check.

Detect deepfakes

Synthetic faces, voices, or scenes pass as recordings of real people.

Authenticate images

A viewer cannot tell whether an image is a photograph or a generation.

Authenticate audio

A listener cannot tell whether a voice recording is real.

Authenticate video

A viewer cannot tell whether the footage happened.

Detect AI-powered persuasion

Persuasive systems steer people without a visible sign.

Prevent targeted manipulation

Messages are tailored to a person weaknesses.

Detect coordinated influence

Many accounts or agents push one narrative and look organic.

Protect vulnerable users

People who are easier to mislead get no extra protection.

Verify content origin

Content is shared with no way to check who made it.

Track content transformations

Crops, edits, and model passes erase the path from the original.

Establish authenticity

There is no reliable way to show a piece of content is authentic.

Maintain provenance across platforms

Origin data is stripped when content is reposted.

Measure AI amplification effects

Automated ranking and generation crowd out slower, sourced information.

Detect search manipulation

Search results are steered by synthetic or gamed content.

Detect recommendation manipulation

Recommendations are pushed off what the user asked to see.

Reduce synthetic content saturation

Feeds and search fill with generated content until real sources are hard to find.

Prevent blind acceptance of AI recommendations

People accept a recommendation because it came from the system.

Encourage appropriate human review

High-stakes recommendations are followed with no review.

Detect inappropriate reliance

Teams rely on the system for calls they should still make.

Design effective human checkpoints

Checkpoints exist, but people click through them.

Detect excessive AI dependence

Work stops when the system is unavailable because people no longer do it themselves.

Preserve human expertise

Skill fades because the system does the hard part every time.

Maintain independent decision-making

People stop forming a view before they see the system answer.

Prevent deskilling

Roles lose the practice needed to notice when the system is wrong.

Define human/AI roles

It is unclear which steps belong to a person and which belong to the system.

Optimize task delegation

Work is handed to the system that a person still does better, or the reverse.

Design effective handoffs

Context is lost when work passes between a person and the system.

Coordinate human and AI decisions

A person and the system decide in parallel and contradict each other.

Preserve critical thinking

People stop checking reasoning once the system drafts the answer.

Understand effects on learning

Learners use the system in ways that skip the learning.

Prevent cognitive offloading harms

Offloading routine thinking also offloads judgement people still need.

Maintain human judgement

The final call is effectively the system call, and a person only signs it.

Preserve meaningful choice

The interface funnels people into the option the system prefers.

Prevent coercive AI interaction

Prompts, timing, or tone push a person past a choice they would have refused.

Enable informed decisions

People decide without the facts the system already has.

Prevent AI from restricting user agency

The system narrows what a person can see, say, or do.

Define AI ownership

No role owns the system after it is deployed.

Establish AI approval processes

Systems go live without a recorded approval.

Assign risk responsibilities

Risks are listed and nobody is assigned to them.

Create AI governance structures

Decisions about AI have no forum, cadence, or authority.

Identify AI risks

Risks are found after an incident, not before deployment.

Assess risk severity

Risks are named without a judgement of how serious they are.

Define mitigation controls

A known risk has no control that reduces it.

Monitor residual risk

Risk that remains after controls is not watched.

Translate regulation into controls

A legal duty never becomes a check the team can run.

Demonstrate compliance

The organisation cannot show which control meets which duty.

Manage AI inventories

The organisation does not know which systems it runs.

Implement regulatory reporting

Required notices and records are missing or late.

Define AI assurance methods

Claims about the system are not backed by a method someone else can repeat.

Require independent assessment

The only review is the team own review.

Establish certification

There is no external bar the system has to meet.

Define AI audit frameworks

Auditors have no agreed questions, evidence, or scope.

Detect AI incidents

Harmful failures are not defined as incidents, so they are not caught.

Report incidents

An incident stays inside the team that saw it.

Investigate incidents

The cause is never established, so the same failure repeats.

Coordinate incident response

Response is split across teams and nobody leads it.

Measure labour displacement

Jobs change or disappear, and the people affected are not part of the plan.

Measure productivity effects

Productivity gains are claimed without a measure of who got faster and who got more work.

Measure market concentration

AI advantage piles up with firms that already have data, compute, and distribution.

Measure distribution of economic gains

Gains accrue to owners and vendors, not to the workers and customers who bear the change.

Assess effects on communities

A community lives with the system effects and has no say in them.

Assess effects on institutions

Schools, courts, clinics, or newsrooms change how they work because of the system.

Assess social cohesion

AI-mediated interaction weakens trust between groups.

Detect changes in human behaviour

People change how they act because the system rewards or nudges that behaviour.

Measure AI energy consumption

Training and inference draw power that is not counted in the product cost.

Measure water consumption

Cooling and power generation use water that does not appear in the product account.

Improve compute efficiency

The same result costs more energy and hardware than it needs to.

Measure AI infrastructure footprint

Data centres, chips, and networks add an environmental cost outside the app.

Prevent cascading AI failures

One system failure triggers failures in systems that depend on it.

Map cross-sector dependencies

Critical sectors rely on the same models and fail together.

Measure concentration of AI capabilities

A few providers hold capabilities everyone else depends on.

Detect large-scale behavioural effects

A widely used system shifts behaviour across a population, not just one user.

Close the digital divide

People without reliable access, devices, or skills are left out of the benefit.

Reduce unequal access to AI

Price, language, or region decides who can use the system.

Measure concentration of AI benefits

Benefits cluster with groups that already had an advantage.

Expand access to AI infrastructure

Compute, models, and data stay with organisations that can pay for them.