Staged evaluation criteria
Design outcome-focused stage criteria to progressively uncover more information.
Staged outcomes
The testing stages in agile procurement aim to progressively uncover more information. Each procurement stage should be designed to achieve certain outcomes that start with a focus on the problem and progress to greater focus on the technical solution and supplier capability. These outcomes guide how evaluation criteria are applied and ensure the selection process fosters innovation.
Read more about stages in agile procurement.
The following is a guide for expressing the outcome of procurement stages. Buying teams will need to tailor this guide for their project, since an innovation buying pathway might repeat stages or skip them, depending on the types of solution and market maturity they are testing.
| Stage name | Purpose | Success indicators |
|---|---|---|
| Tell us | The proposal seems to solve the problem |
|
| Show us | The supplier seems to understand the operating environment and users |
|
| Prove it works | The solution is technically feasible |
|
| Build it with us | The solution and supplier are viable |
Solution represents the best value for money and there is substantial evidence that it is viable in terms of:
|
Learn about what should happen at each stage, including objectives, information needs, risk milestones, approvals, costs and processes.
How these outcomes shape evaluation criteria
The outcomes defined for each stage guide the development of evaluation criteria. This ensures that each proposal is assessed on relevant and meaningful aspects at the right time.
In the initial stages, criteria focus on assessing whether the proposal addresses the core problem and aligns with project objectives.
As the process moves forward, the criteria become more specific and detailed. They assess the depth of the supplier's understanding and the feasibility of their solutions. By the final stages, the criteria are geared towards assessing the overall viability of both the solution and the supplier. They focus on aspects such as technical specifications, cost-effectiveness and the ability to deliver successfully.
This staged approach allows for a flexible and adaptive evaluation process while protecting fairness at each stage. It accommodates a wide range of innovative proposals while ensuring that only the most viable and effective solutions are selected.
Outcome-focused evaluation criteria
Outcome-based evaluation criteria help ensure that solutions are assessed against the desired outcomes regardless of the procurement stage. They provide flexibility by referencing requirements, which can evolve from one stage to the next, rather than having requirements embedded in them.
Using this approach, it is possible to have a single set of evaluation criteria that applies to all stages. While this is advantageous, it is not a strict need. Project teams may prefer to identify specific criteria for certain stages to support the outcomes they have defined.
Recommended evaluation criteria for innovation
When focusing on outcomes, the evaluation criteria used by innovation buying projects tend to be quite similar. Buying teams choosing to start with a single set of criteria can tailor these recommended criteria for their buying project. The recommended criteria are:
- Solution functional capability: addresses the challenge and is superior or differentiated.
- Solution technical capability: can work with organisation systems, level of maturity and/or customisation needed.
- Understanding of the problem and users: reflects current state, considers needs of users.
- Supplier delivery capability: skills, experience, and human resources capacity. May include specific methods relevant to the project.
- Ability to work with NSW Government: can meet NSW Government policies and requirements including contract terms.
- Commercial viability: solution has commercial viability, is worthy of investment, and has a reasonable roadmap for development and/or certification.
- Cost/commercial model: total cost of implementing and maintaining solution, based on the proposed business model for the solution (ownership, operation, etc.).
Balancing flexibility and accountability
Flexibility is essential in innovation procurement. The evaluation criteria outlined in a procurement strategy tend to be high-level. A detailed evaluation plan complements this by refining criteria and methodology over time. Between the two, buying teams can achieve clarity for decision-makers while allowing adaptation as market insights emerge.
Some projects include weightings and indicators in the strategy; others defer these to the evaluation plan. Regardless of approach, an evaluation plan can be set up to evolve at each stage to stay aligned with project needs. In practice, this may mean governing changes to the document or issuing addenda at key milestones or gates. (Note: Guidance covering Evaluation plans is under development.)
Alternatively, evaluation can become more specific and technical without any changes to criteria. Instead, the level of detail suppliers are asked to provide can evolve, increasing the rigour with which proposals can be evaluated at each stage. Read more about level of detail on the next page.
Outcome-focused to technical detail
Initially, criteria should be focused on the desired outcomes. This allows suppliers to present different approaches to achieving those outcomes. As more information becomes available and the scope of solutions is better understood, either the criteria become more technical and detailed, or the level of technical detail assessed within the criteria increases. This shift helps refine the pool of proposals and ensures that the end solutions are not only innovative but also technically feasible and aligned with the project’s needs.
Sample text for procurement strategies
Buying teams can adapt the following sample text in their procurement strategy to communicate the outcome-focused and iterative nature of evaluation criteria.
‘This is an outcomes-based market approach. This means that, while the final intent is to implement a solution at scale, the nature of the end solution is not known. A range of different solutions may be put forward by suppliers in the first stage based on a problem statement.
Given the scope of solutions being assessed at subsequent stages is currently unknown, detailed evaluation criteria will be defined as the scope is defined in the evaluation plan for each stage.’