Evaluation criteria play a crucial role in any procurement process. This is especially true for innovation procurement.
Evaluation criteria are the bridge between the project objectives that will end up being formalised through an agreement and the problem to be solved. They connect the experts who can help determine whether a proposal effectively addresses the problem. They also underpin a defensible process for the selection of one or more proposals.
Flexible, outcome-focused evaluation criteria are important when there is uncertainty about the end solution. This is especially true for emergent markets where little is known about supplier capability and solution maturity.
At a high level, the steps involved in setting up an evaluation approach for innovation buying projects are:
be flexible with minimum/mandatory criteria
define the outcome of each procurement stage
ask for the right level of detail at each stage
access the right expertise to define a stage outcome and determine if it has been achieved
predict when evaluation criteria might need to be adjusted and plan for it.
This guidance helps buying teams navigate each step in a way that supports innovation.
When to define evaluation criteria
Buying teams should define evaluation criteria during the Plan phase of procurement, after completing the scoping steps, creating the challenge statement and documenting the Statement of Requirements.
While evaluation criteria are set in the Plan phase before starting the first Source phase, they need to evolve before further Source phases. An iterative approach is not 'set and forget' or a a 'free for all'. Rather, it strikes a balance between transparency, fairness and flexibility by adapting to new information and governing changes carefully.
Managing risk through alignment and expertise
To make informed decisions, buyers need alignment between requirements, evaluation criteria and the evaluation process.
To support alignment, buying teams should consider:
- the objectives and desired outcomes
- the context and importance of requirements
- what intelligence needs to be gathered
- why the evaluation criteria was adopted
- how to undertake the evaluation to ensure outcomes are protected.
To maximise alignment, the people who draft requirements and evaluation criteria should also serve on the Evaluation Committee (EC), ideally across all stages. The EC should include expertise from all key areas of the project to ensure balanced, thorough assessments. Diversity enables collaborative and informed decision-making, helps identify and address issues early, and ensures alignment with project goals, agency priorities, and regulatory obligations.
Risk from not including experts
Without the right expertise, several issues can arise. Expand the boxes below to learn about the risks:
Poorly framed requirements can cause the following issues with defining the evaluation criteria:
- scoring and mandatory criteria misaligned with business needs
- suppliers unable to meet criteria as they don't understand requirements
- time and budget pressures from needing to reframe requirements after process is underway.
Learn how to define a challenge statement and document your requirements.
The following risks can emerge from low quality evaluation criteria:
- overly rigid criteria stifling supplier innovation
- unclear or ambiguous criteria making evaluation consensus difficult
- procurement process lacking probity and falling out of compliance
- need for rework causing delays in project timeframes.
- inaccurate evaluations, through an incomplete understanding of critical aspects, biases or a lack of diverse perspectives
- project delays caused by the need to re-work, with possible impacts on budgets
- reduced confidence by approvers in the committee’s recommendations
- risk of technical failures or non-compliance where potential flaws are overlooked due to lack of technical or regulatory/legal expertise
- lost opportunities for innovative ideas and best practices.
Who to involve and why
Buying teams are generally expected to define the evaluation criteria and identify the Evaluation Committee as part of a procurement strategy. They should identify the key areas of expertise required, such as operational, technical, financial, regulatory, and risk management knowledge.
Expand the box to read about who to choose and the risks associated with specific gaps in expertise.
Contribution of this expertise to evaluation process:
- understand the business rules, business needs and business environments
- ensure solutions adhere to the requirements of the business.
What may happen if this expertise is missed
Business-specific requirements are not captured or understood in the evaluation process, with the following potential impacts:
- misaligned solutions that do not address critical business challenges or pain points
- incomplete or ineffective solution due to insufficient understanding of core business requirements
- increased need for rework or modifications during later stages, causing project delays and budget overruns.
Business environment considerations are not incorporated into the evaluation criteria, with the following potential impacts:
- solutions are selected without considering existing operational constraints, leading to integration issues and inefficiencies
- the scalability of the solution is compromised, resulting in limited long-term viability
- incompatibility with existing workflows, systems or processes, leading to disruptions in day-to-day operations.
Business area expertise is not present on the Evaluation Committee, with the following potential impacts:
- selection of a solution that fails to meet business needs or requires significant customisation after the fact
- important operational concerns, such as usability and alignment with business objectives are overlooked
- stakeholder dissatisfaction as expected business outcome or value is not delivered.
Business intelligence is not derived from the evaluation process, with the following potential impacts:
- lack of critical insights into how the solution will support or hinder future business goals and strategy
- missed opportunities to capture lessons learned for future projects, reducing the ability to continuously improve procurement and selection processes.
Contribution of this expertise to the evaluation process helps:
- understand the technologies being used, evaluated and implemented
- ensure solution aligns with internal IT architecture and roadmaps
- evaluate the technical feasibility and integration capabilities of proposed solutions.
What may happen if this expertise is missed
ICT domain specific business or technical requirements are not captured overall or for a discrete stage, with the following potential impacts:
- poorly defined requirements, especially current state systems environment and tech capability for the Prove it works and Build it with us stages
- procurement activity that duplicates solutions that already exist across NSW Government
- procurement activity that conflicts with ICT/product strategy roadmap for the function
- non-compliant solution (information security/cyber security)
ICT domain specific evaluation criteria is not identified for the overall buying project and mandatory criteria is not identified or agreed for discrete stages, with the following potential impacts:
- poorly defined evaluation criteria, especially tech capability, architecture, etc. for the proof and build stages.
- non-compliant solution
- increased ICT assurance risk
- increased risk of not achieving business outcome/benefits.
ICT domain specific expertise is not available for the Evaluation Committee, with the following potential impacts:
- non-selection of a promising solution or selection of a less capable/scalable/mature solution.
- non-compliant solution
- increased ICT assurance risk
- increased risk of not achieving business outcome/benefits.
Business technology and procurement domain-specific business intelligence are not derived from the engagement.
Beneficial market insights are missed if nobody is assigned to undertake a retrospective, synthesise findings and prepare learnings in the form of a case study/market research summary.
Contribution of this expertise to the evaluation process helps:
- manage and oversee the procurement process
- assess the financial viability and long-term sustainability of proposed solutions.
What may happen if this expertise is missed
Financial viability and cost-effectiveness aren’t thoroughly evaluated overall or for a discrete stage, with the following potential impacts:
- solutions selected aren't cost-effective or commercially sustainable, impacting budget allocations and financial health
- underestimation of total cost of ownership and long-term financial commitments.
Procurement and financial expertise are not available for the Evaluation Committee:
- selection of solutions with hidden costs or poor value for money
- increased risk of financial mismanagement and budget overruns.
Procurement and financial intelligence are not derived from the engagement:
- missed insights into cost-saving opportunities and financial efficiencies
- lack of detailed financial analysis may impact the ability to make informed investment decisions.
Contribution of this expertise to the evaluation process helps:
- guide the committee through identification and treatment of potential risks associated with the solutions and their implementation
- provide an independent point of view on how risks and probity concerns are managed in the evaluation process and make recommendations for defensible process.
What may happen if this expertise is missed
Potential risks are not identified, assessed or mitigated overall or for a discrete stage, with the following potential impacts:
- unable to account for unforeseen risks in structure evaluation process
- selection of solutions that have high-risk profiles without proper safeguards.
Risk management intelligence is not derived from the engagement, with the following potential impacts:
- missed opportunities to identify and learn from potential risks and vulnerabilities
- lack of proactive risk management may impact the project’s ability to adapt to changes and uncertainties.
Contribution of this expertise to the evaluation process helps:
- ensure the solutions comply with relevant regulations and standards
- oversee adherence to internal and external compliance requirements
- advise on contractual mitigation options for risks arising and identify contractual risks.
What may happen if this expertise is missed
Legal and compliance requirements are not considered or captured overall or for a discrete stage, with the following potential impacts:
- contractual and legal non-compliance, leading to potential legal disputes and penalties
- overlooked important legal obligations and contractual terms.
Legal and compliance expertise is not available for the Evaluation Committee, with the following potential impacts:
- selection of solutions that are non-compliant with legal and regulatory standards
- increased risk of legal liabilities and breaches of contract.
Legal and compliance intelligence is not derived from the engagement.
- missed insights that could help avoid legal pitfalls and ensure adherence to compliance standards
- lack of legal guidance may impact the project's contractual integrity and enforceability.
Contribution of this expertise to the evaluation process helps:
- understand the customer’s experience with the problem and potential solution
- ensure solution will solve the problem and be feasible from the customer’s perspective.
What may happen if this expertise is missed
Customer needs and expectations are not captured during the evaluation process, with the following potential impacts:
- selected solution does not solve the core customer problem, resulting in low uptake or outright rejection by users
- features and functionality aren't aligned with user needs, leading to frustration and disengagement
- significant customisation required after implementation to meet user requirements, causing delays and additional costs.
Usability and practical barriers are not evaluated, with the following potential impacts:
- solutions may be difficult to use, leading to inefficiencies, increased errors and low productivity
- extensive training required for end users, ultimately increasing the total cost of ownership
- long-term user support requirements escalate, raising operational costs and reducing the solution’s overall return on investment.
Customer-centric features and key pain points are overlooked in the evaluation criteria, with the following potential impacts:
- core customer issues aren’t resolved, resulting in minimal improvements in user experience and outcomes
- customers may avoid the solution in favour of user-friendly alternatives, leading to low usage
- poor user experience negatively impacts the long-term success and acceptance of the solution, leading to project failure or rework.
Customer insights are not leveraged during the evaluation, with the following potential impacts:
- missed opportunities to improve the solution’s relevance, usability and alignment with customer priorities
- future projects may lack lessons learned about customer engagement, reducing the overall effectiveness of procurement activities.
Contribution of this expertise to the evaluation process helps:
- ensure a deep understanding of end-user needs, pain points and the real-world environment in which the solution will be used
- assist in defining the business problem from the perspective of the end users, ensuring that the solution addresses key personas and their specific use cases.
What may happen if this expertise is missed
Critical end user needs, the business problem and requirements are assumptive, ambiguous, incorrect, omitted or not captured overall or for a discrete stage, with the following potential impacts:
- poorly defined requirements, especially definition of the current state, key personas and success criteria
- submitted solutions don’t address the key problem or specific use case.
End user and business operation specific evaluation criteria are not considered or identified for the overall buying project:
- poorly defined evaluation criteria, especially functional and user experience/usability assessment criteria
- increases risk of not achieving end user benefits such as adoption and usage targets.
HCD innovation specialist expertise is not sought or available for the Evaluation Committee:
- disconnection between defined requirements and success criteria
- evaluation assessment misses a promising solution or selects a less functional or usable solution
- increases risk of not achieving business outcome/benefits.
How to motivate Evaluation Committee members
Involving committee members in discussions about the ongoing challenges and objectives of the procurement process can help them feel more connected and invested in the success of the project. Evaluators will most likely become responsible for implementing, operating or using the end solution and should be invested in both achieving a procurement outcome they can work with and forming good relationships with their fellow committee members.
To keep the EC members engaged and motivated, it's important to recognise their contributions and regularly express appreciation for their efforts. Ensure they understand how their expertise directly impacts the success of procurement outcomes by highlighting specific examples of how their insights have influenced key decisions. Providing clear feedback on how their evaluations shape project progress and outcomes can foster a sense of ownership and satisfaction.
How to set evaluation criteria for innovation
Define evaluation criteria resources
Set flexible criteria early that allows suppliers time to meet mandatory criteria in later stages.
Design outcome focused stage criteria to progressively uncover more information.
Make procurement more accessible to suppliers by setting ‘just in time’ criteria.