AI Governance Buying Series · Requirements

How to Define AI Governance Software Requirements

An AI assistant approved to summarise documents can present different risks when it gains access to customer records or permission to take action.

Before comparing platforms, define how your organisation uses AI and which decisions, changes and actions need oversight.

Key takeaway

You’ll learn how to turn your organisation’s AI use into software requirements: what to discover and record, which uses need closer review, what evidence supports approval, and when changes should trigger reassessment. This includes AI your teams build, buy or use through existing business applications.

Define the AI Uses, Decisions and Evidence Your Software Must Support

Part of the AI governance buying series. This page covers requirements. Explore other stages under Keep Reading when they are published.

AI governance requirements begin where a general risk record needs more detail about the AI being used. An enterprise or operational risk platform may support parts of this work. First establish the records, evidence and decisions your organisation actually needs.

Example: An AI assistant advises customers about eligibility for a service. The table shows how each discipline approaches the same use.

Question	Enterprise risk management (ERM)	Operational risk management (ORM)	AI risk and governance
What is assessed?	Business exposure and objectives.	Customer-service process failure and controls.	Suitability of the assistant for its approved purpose, model, information sources and permissions.
Evidence	Exposure trends and assurance.	Affected cases, control tests and corrective actions.	Task evaluations, unsupported answers, access tests and effectiveness of human review.
Response	Change the organisation’s exposure or strategy.	Repair the workflow and affected cases.	Restrict the assistant’s sources or actions, change its configuration and retest the affected use.
Software link	Risk, objectives and accountable executives.	Processes, incidents, controls and remediation.	Use, application, model version, data, tests, approval conditions and subsequent changes.

One customer eligibility assistant can require all three perspectives. The AI-specific requirement is to trace the approved use to the system’s behaviour and supporting evidence.

01 Define Your AI Scope

Start with how AI enters
and operates in your organisation

Your footprint may include employee use of external tools, AI features in purchased software, applications built on external models, internally developed models and agents. Each route gives you different visibility and control. Connect each important business use to the application, model or external service behind it.

Describe the business purpose, users, data access and permitted actions for each use. Include predictive systems, such as applicant scoring or demand forecasting, alongside generative AI. The regulatory consideration below explains the separate applicability decisions.

Employees Using External AI Tools

Finding a visit to an AI service does not establish the purpose of the use or whether sensitive information was involved. Investigate the activity and route the relevant use for review.

Requirement to define: What activity must be discovered, which existing tools can supply it, and how will findings become an assessed AI use?

AI Embedded in Purchased Software

A supplier assessment may not reveal which AI feature a team enabled, what data it accesses or how staff use its output. Link the feature and its approved purpose to the purchased application.

Requirement to define: How will enabled features, intended uses and provider information enter the record and stay current?

Internally Built Applications and Agents

A model record alone may miss that an agent can retrieve customer data, call tools or change a business system. One model can also support uses with different impacts.

Requirement to define: Which relationships must the record connect, and where will information about models, data, tools and permissions come from?

Start with representative uses. A complete inventory is a continuing activity, not a prerequisite for describing the first requirements.

02 Understand the Impact

Understand the impact of each use
and the oversight it deserves

An assistant that drafts a meeting summary plays a different role from a system that helps assess job applications or recommend customer credit limits. The significance of its contribution depends on the decisions it supports, the people affected and the scale of its use.

What Does AI Contribute To?

Describe where the output enters the process and how much weight it carries. A draft reviewed before use has a different effect from a score that determines which applicants reach an interviewer. Record whether people can meaningfully review or challenge the result.

Requirement to define: What must be recorded about the decision supported, permitted actions and points of human review?

Who Experiences the Impact?

An AI recruitment tool supports the hiring team while shaping which applicants progress. Describe what an appropriate outcome means for those people, how an unsuitable decision could be identified and corrected, and whether evaluations examine relevant differences between affected groups.

Requirement to define: How will the platform capture affected groups, relevant evidence and conditions placed on the use?

How Far Does Its Reach Extend?

A use may serve one team or thousands of customers. A shared model or data source may contribute to several applications, each with a different purpose.

Requirement to define: Which relationships must be maintained so a finding can be traced to other uses needing review?

What Assurance Fits the Impact?

Choose evidence for the actual task, such as suitable accuracy, fair treatment, handling of sensitive data and effective human review. Use the significance and reversibility of the outcome to decide how much evidence is needed before use.

Requirement to define: What evidence and review steps should each impact level require, and how will the reasoning behind that choice be retained?

03 Respond and Test

Decide how the AI may operate
and verify the response

Assessment should lead to a decision about use. A customer assistant may answer routine questions reliably but struggle with eligibility decisions. A sensible response could limit automated answers to routine enquiries and refer eligibility questions to a specialist.

Choose a response

Strengthen safeguards, narrow the permitted use, defer deployment, discontinue a use or accept the remaining risk within defined conditions. Record who can make the decision and what evidence supports it.

Requirement to define: Link the selected response, remaining risk, approval conditions and review date to the AI use. Distinguish planned safeguards from controls already verified.

Put it into effect

A decision to remove an agent’s ability to update customer records must reach the system managing that permission. Identify the action owner and how implementation will be confirmed.

Requirement to define: Assign actions and retain evidence that restrictions or safeguards actually operate.

Test the Actual Use

For a policy assistant, test answers against their sources, conflicting guidance, restricted documents and questions without a supported answer. For an applicant-scoring model, choose evaluations relevant to the applicants and decisions it supports, including differences between affected groups. Agree what would block approval.

Requirement to define: Which scenarios, acceptance criteria and results must remain linked to the application version, instructions and data sources assessed?

Locate the Control and Its Evidence

Decide whether the platform must perform evaluations, import results from technical tools or coordinate a review of supplied evidence. Access restrictions may be enforced elsewhere. For purchased AI, your team may test responses and settings while relying on the provider for underlying model evidence.

Requirement to define: Which systems enforce limits or perform tests, what evidence can be imported, and how are gaps recorded?

Close Findings Through Retesting

A finding may result in a narrower set of documents, a required human review or a pause while an agent’s permissions are corrected. Track the action and test the changed use before lifting conditions.

Requirement to define: Can findings, actions, retests and approval decisions be traced as one sequence?

04 Maintain Oversight

Keep oversight aligned
with how your AI evolves

An assistant approved to search policy documents may later gain customer data, external users or permission to act. Approval should remain tied to the use and conditions that were actually reviewed.

What Was Approved?

Record the business purpose, user population, application version, information sources, permitted actions and supporting evidence. Approval of a model alone does not approve every application built on it.

Requirement to define: How will the software preserve the approved scope and conditions for each use?

What Triggers Another Review?

A changed model, new data source, wider user population, expanded autonomy or adverse result may alter the basis for approval. Agree which changes need evidence, specialist review or temporary restrictions.

Requirement to define: Which changes and findings reopen the assessment, and how are they routed according to impact?

How Will You Learn About Change?

Identify the source for each trigger: deployment records for model updates, application settings for new access, and complaints or evaluation results for operating concerns. Specify what can be automated and what a person must confirm. Record the expected response so monitoring produces a decision or action.

Requirement to define: Which signals require action, who receives them, and how will the record show the review and response?

You're ready to talk to suppliers when you can

✓Show representative AI uses and where visibility is incomplete
✓Explain the impact and people affected by consequential uses
✓Name the response, controls and task-specific tests that matter
✓Describe what changes should reopen an approval
✓Identify evidence held by technical tools and AI providers

Turn these answers into a short buying brief: priority uses, the oversight decisions you cannot support today, the evidence available and the connections required. Mark needs as confirmed, assumed or open, and give unresolved questions an owner. This is enough to start useful supplier discussions while the inventory develops.

Planning reference: This guidance is for planning a software purchase. Which obligations apply will depend on your organisation's own circumstances.

Four Factors That Shape Your AI Governance Software Search

01 · How Regulatory Obligations Shape Your RequirementsApplicability

The EU AI Act makes “we need an AI risk assessment” an incomplete requirement. First establish whether the activity is in scope, the organisation’s role and the classification of the particular system and use. A purchased application does not settle those questions.

Keep the legal applicability decision separate from your internal risk rating. Record why obligations apply, their source and the evidence or action required. Different uses of the same technology may need different review paths.

For your software search, this means investigating whether assessments can reflect those differences and retain the reasoning behind them. A framework label or preloaded questionnaire gives you little assurance about that depth.

What to check
Choose a relevant AI use and identify the applicability questions still open. Specify who resolves them and which records, reviews and evidence the resulting obligations would require. Check current application dates separately.
02 · How ISO Standards Shape Your Governance ApproachStandards

ISO standards help clarify which governance work you want software to support. Start with the process your organisation intends to establish or improve.

ISO/IEC 42001 addresses an AI management system. Translate that into requirements for connecting the agreed scope and policies to responsibilities, decisions, evidence and improvement actions.

ISO/IEC 23894 provides AI risk management guidance. Decide how the software should support the path from identifying a risk to choosing a response and reviewing what remains.

ISO/IEC 42005 addresses impacts on people and society. Your assessment may need to retain who is affected and the evidence about those effects, alongside the organisation’s own business exposure.

What to check
Ask how a relevant assessment and its evidence would support your chosen approach. A standards mapping is useful, but you still need to establish how the process operates, what evidence it retains and what remains outside the platform.
03 · Governing AI When Capabilities and Impacts Are EvolvingChanging AI

AI oversight needs a way to handle capabilities the organisation has not yet assessed. An assistant approved to retrieve information may later offer to send messages, update records or act across several applications. Its availability does not establish that those uses have been approved.

Set boundaries for the information it may access, the decisions it may support and the actions it may perform. Establish who can enable new features or expand those permissions. Where your tools allow it, make expanded authority subject to review before activation.

For purchased AI, investigate control over updates and feature activation. Where changes cannot be deferred, determine what notice, testing opportunities and fallback arrangements you need.

Testing covers the conditions examined. Feedback from users and affected people may reveal impacts the original assessment missed. Define when limited evidence or an unexpected outcome means narrower use, additional review or a temporary pause, and what would allow that restriction to be reconsidered.

What to check
Look for support that connects provider changes and unexpected findings to affected uses, including concerns outside existing risk categories. Establish where permissions can be restricted and how the governance process confirms that action took effect. The key question is how quickly the organisation can recognise, assess and respond to change.
04 · Confusing Feature Coverage with Working OversightWorkflow gaps

An inventory, assessments, testing and action tracking may all appear on a feature list. The buying question is how information moves between them. Otherwise, your team may still have to identify which application a test result belongs to, find its approval and decide who should act.

Suppose a test finds that an AI assistant can retrieve documents beyond a user’s permissions. Your process needs to connect the finding to the affected application and data source, assign the access change, record a retest and resolve any restriction on use. If those steps depend on disconnected records, a completed action could leave the approval based on outdated evidence.

What to check
Use one realistic finding to define the connections your process needs. Specify what information passes between testing, governance and the system applying the control, who handles each handover and what evidence closes the issue.

Common questions

Can we define requirements before completing our AI inventory?: Yes. Start with representative uses and record where visibility is incomplete. Decide whether discovery itself is a priority requirement, then expand the inventory as teams and tools supply evidence.
Do we need AI governance software if we only use AI in purchased applications?: Possibly. You still need to know which features are enabled, how they affect your processes and what evidence the provider supplies. First check whether your existing risk, procurement and security systems can connect those decisions.
Should we start with a spreadsheet or a dedicated platform?: A spreadsheet can work for a small, known set of uses. The case for software grows when uses, versions, approvals and technical evidence change faster than the team can keep them connected and current.
Should we use a supplier’s RFP template for AI risk management software?: Yes, it can save time and highlight gaps. However, a supplier’s template may present its distinctive technical or functional capabilities as essential requirements. Adopting these unchanged can quietly reshape your priorities and exclude otherwise suitable platforms. Define your core needs first, then use the template to supplement them. Keep added requirements only where they address your organisation’s AI uses and oversight needs.
Who should help define our AI governance software requirements?: Bring together the people who own AI uses and decisions, plus risk, technology, security, privacy and procurement. Include the teams that can explain testing, provider evidence and where restrictions will actually be enforced.