Sourcing an AI solution

AI system objective and constraints

Agencies need to

• Clearly state the objective.
• Verify that the AI system is aligned to its objective and note that this objective may change over time.
• Understand and address any constraints.
• Consider all the potential consequences or impacts of the AI system (i.e. Has the agency modelled all potential consequences of the AI system). Consider and mitigate against potential unintended consequences.
• Ensure that the objectives and constraints can be updated for the life of the AI system. If this expertise does not exist internally, this capability can be negotiated with the service provider.

Performance metrics

Set clear requirements on performance of the AI system against its objectives. This includes metrics (or a process for agreeing metrics), reporting requirements, response process, and process for developing additional metrics over time.

Monitoring performance

AI systems need ongoing monitoring, maintenance and auditing to ensure they continue to meet their objectives. This will be determined by the scope and scale of the AI system. For example, requirements for a system used for more routine analysis are very different from those for a system making consequential automated recommendations.

Agencies need to:

• Be clear on who is responsible for performance monitoring and how it will be undertaken, including any independent reviews required to certify that an AI solution is in compliance with the AI Strategy, Policy and this User Guide
• Require the AI system to record the objectives and performance metrics relevant to each decision it makes. This is important because the objectives and performance metrics can change so the system needs to capture the relevant information at that point in time.

Data

Specifications need to address:

• Data requirements, including adherence to data standards and data quality requirements
• Privacy and information security requirements, including adherence to legislation and government policy
• Data breach and security incident notification and management processes
• Ownership, use, control, and distribution of data used and produced by the AI system (AI systems may use a mix of internal and external data to learn and function correctly. Once in use, AI systems will likely also generate additional data).
• How the system will monitor and correct data distribution problems
• Data storage, retention and disposal requirements, including when a contract ceases or is terminated. This need to be in line with the State Records Act 1998 (NSW) and Government Information (Public Access) Act 2019 (NSW)

that vendors must specify what information must be retained to enable reproducibility (agencies need to verify this). Examples include the training data, model parameters, and random seeds.

Rules-based AI systems

Service providers must provide the rules that underpin the AI system.

Pre-trained AI systems

Agencies need to understand:

• The objective used for prediction and decision making
• The training data set used so that the agency (or independent person) can determine if it is appropriate for the AI project
• If there is any historical bias and mitigation measures.

Predictive model

If the system incorporates a predictive model (whether rules-based or machine learning), the agency needs to model all possible outcomes. This allows the agency to test how the system may behave in a range of settings before release. 

Testing and scaling AI systems

Agencies may consider using AI experts to assess system performance on government data prior to selecting an AI solution.

Once the AI system is selected, agencies need to ensure the AI system is tested and scaled gradually in its operating environment. This testing is in addition to testing already conducted by the service provider on other datasets or in other circumstances.

The purpose of the testing is to ensure the AI system’s objective can be met and unintended consequences are mitigated prior to full scale implementation.

Change management