Sourcing an AI solution
Agencies may need to procure (build or purchase off-the-shelf) all or part of an AI system from external vendors. This section steps through key considerations for procuring AI solutions.
It is recommended that the agency engages relevant privacy experts, data scientists and procurement professionals to contribute to the development of the procurement specifications and evaluation of proposed AI systems.
NSW Government procurement landscape
The NSW Procurement Policy Framework defines procurement (or sourcing) as the end-to-end buying process from needs identification to market engagement, contracting and placing orders, managing contracts and service provider relationships, and disposing of government assets.
Designing procurement specifications
The outcomes and boundaries for an AI product need to be clearly articulated to suppliers/designers. This table sets out considerations when developing specifications for projects involving AI.
Consideration | Description |
---|---|
AI system objective and constraints |
Agencies need to
|
Performance metrics |
Set clear requirements on performance of the AI system against its objectives. This includes metrics (or a process for agreeing metrics), reporting requirements, response process, and process for developing additional metrics over time. |
Monitoring performance |
AI systems need ongoing monitoring, maintenance and auditing to ensure they continue to meet their objectives. This will be determined by the scope and scale of the AI system. For example, requirements for a system used for more routine analysis are very different from those for a system making consequential automated recommendations. Agencies need to:
|
Data |
Specifications need to address:
that vendors must specify what information must be retained to enable reproducibility (agencies need to verify this). Examples include the training data, model parameters, and random seeds. |
Rules-based AI systems |
Service providers must provide the rules that underpin the AI system. |
Pre-trained AI systems |
Agencies need to understand:
|
Predictive model |
If the system incorporates a predictive model (whether rules-based or machine learning), the agency needs to model all possible outcomes. This allows the agency to test how the system may behave in a range of settings before release. |
Testing and scaling AI systems |
Agencies may consider using AI experts to assess system performance on government data prior to selecting an AI solution. Once the AI system is selected, agencies need to ensure the AI system is tested and scaled gradually in its operating environment. This testing is in addition to testing already conducted by the service provider on other datasets or in other circumstances. The purpose of the testing is to ensure the AI system’s objective can be met and unintended consequences are mitigated prior to full scale implementation. |
Change management |
Consider the costs and impacts of transitioning from the current process to an AI system. |