Worked risk examples

Sabrina has mobilised a core buying team which includes a procurement advisor, an ICT advisor and a representative from the business function that is experiencing the problem. They have determined their innovation scope by following the Test and Buy Innovation guidance to engage stakeholders in problem shaping, eventually defining a challenge statement.

The team members are familiar with risks relating to the business function and to standard procurements (i.e. specification-based procurements). The procurement advisor recognises that innovation introduces new risks and the team does not feel equipped to evaluate or manage them.

Since this is the first time Sabrina has worked on an innovation buying project, she needs some innovation procurement support. 

By reading the guidance, she understands that supplier interaction is part of reducing the overall risk of the project. This ensures that both solutions and suppliers are fit for purpose before investing in and implementing the end solution. 

Sabrina can see the importance of quality supplier interactions at a few points: 

• Industry engagement could unearth new information about the problem space they are exploring. It can confirm if the problem has been solved elsewhere and ensure the challenge statement can attract good solutions.
• Allowing suppliers to seek clarifications directly from the business function about the challenge statement ahead of the pitch-fest gives them the opportunity to better customise their proposal and demonstrate their understanding of users and ability to work with a government-specific scenario.
• Working closely with the suppliers shortlisted for the Proof of Concept (PoC) will improve the quality of the solution developed. It will also test how well suppliers can work with the project team and users. 

Sabrina has some experience with running transparent and fair supplier interactions from previous buying projects. However, probity considerations were simpler in that project because all supplier communications were completely open and consistent.  

Sabrina decides she needs some more expertise and contacts the agency governance and risk team. They agree to provide an independent advisor to help identify specific risks and controls. They tell her that for a higher value or higher profile project, they would have recommended appointing an external probity advisor.

Sabrina meets with the independent advisor to go through the innovation mindset for risk management. They test out a few hypothetical risks and treatments to make sure they have a shared understanding of which risks will help drive innovation and how to manage them.

Read more about the probity-related risks that can be amplified when buying innovation.

With the help of the independent advisor, Sabrina narrows down the sources of probity risk that could apply to her project. 

She describes each risk in terms of (a) cause, (b) what may happen and (c) impact on objectives.

She identifies four key probity-related risks:

1. Intellectual Property (IP) – During early industry engagement, suppliers might reveal IP that is then incorporated into a Request for Proposal. This could lead to formal complaints, lost time and legal costs. It could also damage the NSW Government's reputation as an innovation and engagement partner.
2. Perceived advantage – A supplier with a stronger grasp of the challenge statement may ask more mature clarification questions. Their shortlisting for the next stage might be perceived by another supplier as preferential treatment. This could lead to a dispute of the outcome, delays to project timelines and damage to the NSW Government's reputation.
3. Perceived bias in technical requirements – Technical requirements documented in the Request for Proposals for the final stage (Scale implementation) could closely resemble one solution put through a Proof of Concept (PoC) and not the others. Suppliers of the other solutions may perceive that the scope and technical requirements have been biased towards something one supplier already offers. This could lead to formal complaints, lost time and legal costs. It could also damage the NSW Government's reputation as an innovation collaboration partner.
4. Perceived bias in evaluation criteria – Following the PoC stage, adjustments to the originally proposed evaluation criteria might be necessary to better align with the technical requirements. Suppliers may perceive these changes as biased towards a particular supplier. This could lead to formal complaints, lost time, legal costs and damage to the NSW Government's reputation.

Sabrina considers the controls that are already in place for the three probity risks. 

To help with this, she reads ahead to the Risk treatment step to see if any of the proposed treatments are already in place. Note: if they already exist, they are called controls.

Sabrina focuses on scoring residual risk (that is, the risk left over based on effectiveness of current controls). She uses her agency risk management framework to score the severity of the consequence if the risk occurs and the likelihood of the risk occurring. This gives her an overall rating for each risk.

The next step is to narrow down the risks that require further treatment. She refers to her agency's risk appetite statement and discusses key risks with project sponsors or decision-makers to confirm the acceptable level risks for her buying project. She notes the appetite may vary for different kinds of risks or different aspects of the project.

For each of the four risks from the Identify the risk step, Sabrina has the option to remove them entirely. 

She is aware, however, that true innovation depends on these risks being taken. She knows that not taking them could create a much bigger risk when implementing and investing in the end solution. Instead, she looks at ways to reduce the likelihood or consequence of each risk:

1. Intellectual Property (IP)

Reducing the consequence of compromising someone's IP may be challenging, but Sabrina identifies several measures to reduce the likelihood:

• She consults with her agency's legal team to adopt a default position on how IP will be used and clearly communicates this position before and during the engagement activity.
• She schedules a session to explain this position to anyone involved in supplier engagement and problem-framing, providing examples of potential IP issues.
• Sabrina asks the independent advisor from Step 2 to attend the engagement sessions and provide an independent view on whether any IP was compromised before the request for proposals is released. 

2. Perceived advantage 

Reduce the likelihood:  

• Sabrina foreshadows all clarification processes in an open, shared forum and gives suppliers the chance to object to any proposed steps in the process.
• She trains team members involved in the engagement activity by running them through scenarios to help them differentiate between clarifications and additional information.

Reduce the consequence:

• She ensures processes are defensible through good minute-taking and the presence of the independent advisor. 

3. Perceived bias in technical requirements

Reduce the likelihood:

• Sabrina plans for all co-design stage deliverables to be evaluated at the close of the state, and feedback given to all suppliers, to minimise surprise outcomes for suppliers in the next stage.
• She budgets for compensation of suppliers for their participation in a Proof of Concept (PoC).
• She clearly communicates when technical requirements will be documented and how the PoC stage will feed into their definition and notifies suppliers if any new information leads to changes in the initially communicated process.

Reduce the consequence:

• Sabrina ensures the development of technical requirements is defensible by documenting the reasons for focusing on a specific solution type and including the independent advisor in discussions to validate those records.

4. Perceived bias in Evaluation Criteria 

Reduce the likelihood: 

• Sabrina is transparent in supplier communication about the fact that the pathway may adjust.
• She proactively communicates the structure and rigour around any separate supplier interactions, earning the confidence of suppliers.
• She consults guidance on best practice engagement steps and uses tailored pro-forma content in market-facing documents and briefings to manage supplier expectations. 

Reduce the consequence: 

• Sabrina carefully justifies, minutes, and/or documents decision-making about supplier interactions and any changes to the procurement approach to ensure defensibility if a complaint should arise. This is effective against reputation impacts, but may not reduce time and/or cost impacts.
• She consults guidance on documentation of communication and decision-making and ensures the iteration plan sets up project gates that account for relevant reviews of information, well-governed decision-making and robust records.

Sabrina ensures all identified risks are documented and initial controls are in place before proceeding with the procurement and approvals. This includes compiling risks, assigning owners to monitor and review the risks and presenting this initial risk register to decision makers for review and approval. 

Sabrina revisits the risk register and proposed controls ahead of each supplier engagement to ensure all probity considerations are embedded into the structure and plan for each engagement. This includes ensuring the engagement plan includes clear guidelines for maintaining transparency and fairness. 

Sabrina reassesses and updates probity risks risk before starting each stage to reflect any changes. This includes reviewing past stages, updating the risk register and gathering any feedback for improvement. For example, learning that a participant is prone to complaints or litigation might cause Sabrina to increase the likelihood of probity risks, thereby increasing their profile. 

She also continuously monitors risks and controls throughout the procurement process. This includes regular scheduled reviews and maintaining detailed records of all risk monitoring activities including meeting minutes, updated risk registers and decisions made. She ensures that the risk management strategies are agile with the ability to adapt to evolving project conditions. 

Lastly Sabrina conducts a comprehensive review of the entire procurement process to evaluate risk management effectiveness. This ensures continuous improvement in risk management practices for future innovation procurements. 

Sabrina has included an ICT advisor in the core buying team, but also understands that more specific expertise may be needed as the team starts evaluating proposals and narrowing down solutions. She knows it will be important to engage any technical subject-matter experts (SMEs) early in the project, before the procurement strategy is committed.

The ICT advisor has flagged the project with some experts in key technical areas like cyber security, data privacy and artificial intelligence, but hasn't yet briefed them or asked them to contribute.

Sabrina recognises her own limited ICT expertise and is grateful to have Mason, an ICT business partner, as part of the core buying team. He has a deep understanding of the agency's technical landscape and can connect her to more specific expertise when needed. Since he is part of the core buying team, he has participated in mobilisation and has been briefed on the project and market approach.

Mason understands the project’s aims and notes that while there is no guarantee that a technology-based solution will emerge, it’s essential to prepare for the possibility. He conducts a market scan and identifies a few existing technologies that solve similar problems, providing valuable insights into the state of the market. 

Mason understands the importance of keeping the early stages of the project focused on the challenge and desired outcomes rather than on technical specifications. This approach encourages innovative proposals that are not limited by predefined technical requirements. However, Mason also points out that if the project does move towards a technology-based solution, numerous technical considerations and risks will need to be managed. 

Together, Sabrina and Mason outline the initial steps for engaging technical experts throughout the project. Mason serves as the primary ICT contact, providing ongoing support and advice, while also identifying other relevant technical experts within the agency as necessary. He conducts a thorough market scan to gather intelligence, sharing the findings with the project team to enhance their understanding of potential solutions and associated risks. Additionally, Mason identifies existing technologies that address similar challenges, and he emphasises the importance of risk awareness by highlighting potential technical risks like system integration challenges, cybersecurity threats and data privacy concerns. These risks will be continuously monitored and evaluated at each stage of the procurement process to ensure proactive management. 

For the procurement strategy, Mason and Sabrina agree to highlight one key overarching risk: the uncertainty about the final solution. This uncertainty poses a significant challenge because it means they cannot identify and manage specific technology risks at the outset. These risks include system integration, cyber security, data privacy and others. 

Mason and Sabrina document the risk, using the three-part structure:

Uncertainty about the end solution

• Cause – The inherent uncertainty about the end solution makes it difficult to pinpoint and address specific technology risks. These include system integration, cyber security, data privacy and more.
• What may happen – This uncertainty can lead to potential risks being overlooked or inadequately managed, resulting in unplanned work to address these risks as they emerge. In the worst-case scenario, it could lead to significant issues like a security breach or system integration failure.
• Impact on objectives – These unmanaged risks could affect project timelines, leading to delays and increased costs. Additionally, trust and collaboration with shortlisted suppliers may suffer if unexpected technical issues arise.

Mason and Sabrina agree that the project team will need to revisit the risk assessment at each stage of the procurement process. As technology solutions are proposed and shortlisted, they will need to define specific risks and develop appropriate mitigants. This approach ensures that as more information becomes available, they can identify and manage risks responsively. The detailed strategies for managing these risks are outlined in Step 6. Risk treatment below. 

Sabrina considers the controls already in place for technical risks. Principally, the multi-stage approach to procurement which provides a structured way to uncover new information about the potential solution.

She consults with Mason who feels that, while these existing controls are a strong base for identifying risks later on, they don't give any guarantee of uncovering specific technology risks. This means the residual risk might still be relatively high. Sabrina reflects this in her scores for the severity and likelihood of the risk, to get an overall rating.  

She then refers to the agency's risk appetite statement and discusses key risks with project sponsors or decision-makers to confirm the acceptable level of risks for her buying project. In this case, the risk is deemed to need further treatment. 

Mason and Sabrina agree it is not possible to remove this risk entirely, as the buying project would not attract innovation if they did. Removing the risk would introduce other, much bigger risks like investing in a solution that doesn't solve the problem effectively, provides less value for money or may be out of date very quickly. They focus instead on how they can reduce the likelihood and consequence. 

They reduce likelihood of missing technology risks as solutions are narrowed down by: 

• Inclusion of ICT representative – Including a representative from the central ICT team on the evaluation panel ensures that technical considerations are addressed from the onset. 
• Structured review activities – Defining a structured activity between each stage to review information from proposals and identify technology risks. This includes regular meetings to discuss potential risks and their management. 
• Architectural review board or similar governance body – Once enough information is available, consulting an architectural review board or similar governance body can provide expert insights and identify hidden risks. 
• Technical expertise in design – Seeking input from technical experts on the design of stages or contributing to the questions that are asked of suppliers. For example, questions can be tailored to elicit information about software integration considerations and a Proof of Concept (PoC) can test the supplier's ability to work within the operating environment. 
• Stage planning activities –Building in stage planning activities that allow those technical experts to revisit their input when more information is available after each stage. This ensures that the risk assessment is current and reflects the latest information. 

They also reduce the consequence by: 

• Internal ICT policies – Using internal ICT policies and protocols to review the supplier's systems ensures that any solution aligns with existing standards and practices. 
• Appropriate contracting framework – Using the appropriate ICT/digital contracting framework so that if a risk is missed within a stage, it will still be tackled as part of negotiating any agreement. This includes clauses that cover unforeseen technical issues and ensure they are addressed without additional cost or delay. 
• Early communication – Communicating the preferred contracting framework to suppliers early so they are less likely to be surprised by technical requirements even if they weren't part of the evaluation. Clear communication can help set expectations and reduce resistance to compliance. 
• Collaborative relationships – Maintaining collaborative relationships with relevant technical experts, ensuring they are briefed early and know about upcoming milestones. This way, if something is missed, they are more likely to be able to mobilise quickly to help fix it. 
• Proactive engagement – Regularly engaging with technical experts to gather feedback and insights. This continuous loop of communication helps in identifying and addressing potential issues before they escalate. 

For this risk, monitoring is not just about keeping track of the risk, it is actually one of the treatments. Sabrina and Mason expect to find new risks throughout the process. They build in steps to do so, consulting with technical experts before opening each stage. Sabrina schedules regular risk assessments at key milestones in the project to reevaluate and update the risk register. This ensures that emerging risks are identified and addressed promptly.

Sabrina and Mason establish a continuous feedback loop with technical experts and project stakeholders. This ensures that any new information or change in the project is considered in the risk management process. To assist with this, Sabrina develops a reporting structure to keep senior management and project sponsors informed of any new risks, their potential impacts and steps to mitigate them.

Sabrina and Mason conduct post-stage reviews to analyse the effectiveness of risk management strategies and adjust as necessary. This helps in refining the approach and improving the overall risk management process.  

As time goes by, the severity and likelihood of this risk should decrease as the preferred solution is narrowed down. Technical experts will be able to provide informed inputs to the process as new technical risks are identified. These can have more targeted controls and treatments.