Iterative market insights model
Buying teams should expect to build on market intelligence and refine insights throughout procurement stages when buying innovation. Every time suppliers share information or complete deliverables, there is new information about solutions and market capabilities which supplements intelligence gathered in the Discover phase.
A multi-stage procurement offers many chances to uncover unknowns. An iterative approach – that supports testing, learning and adjusting at each stage – boosts market insight, lowers risk and builds confidence in both solutions and suppliers.
This section outlines how to form a hypothesis about the state of the market and possible solutions. It also covers testing the hypothesis through staged deliverables and evaluations. Finally, it discusses how to apply lessons learned to future stages.
Discover phase
During the Discover phase, market research helps form a hypothesis about solutions the market is likely to put forward. It informs the sourcing approaches expected to get the best results, without assuming any one solution.
A hypothesis is an initial point of view based on the information available at the time. Hypotheses are important for innovation, since a buying team needs to be able to plan but also must be careful not to make too many assumptions.
Strong hypotheses are based on evidence. They allow for alternatives and should be reviewed regularly. This way, they can be validated or updated based on new information.
For a good starting hypothesis, buying teams should ensure they have a strong understanding of the problem space, understand as much as possible about the market and predict the sorts of solutions that suppliers might put forward. They should then use these market and solution insights to design the procurement stages. Read more about each by expanding the boxes below.
To focus your market research effort, it is important to first understand the problem space and the scope and scale of the opportunity. This includes knowledge of:
- the current state
- pain points and implications for stakeholders
- what a good outcome looks like
- how success is measured
- the benefit of success.
This may involve early engagement with industry to explore a problem space from the perspective of suppliers, who may have insights into causes of the problem not otherwise visible to buying teams.
Market research involves collecting information from various sources of market information to determine the solution types, categories, maturity levels and supplier capabilities available. A range of solution types should be considered, including:
- non-ICT solutions
- ICT solutions
- professional services
- contractors.
Buying teams may conduct desktop research, speak to colleagues or outsource insights to a third party. They could also carry out more direct forms of market engagement like a market sounding, an Expression of Interest (EoI) or a Request for Information (RfI).
Armed with market insights, buying teams should bring together subject-matter experts (SMEs) across business functions and technical domains to brainstorm solution ideas. This can be done separately for each source of market information, or once with all collective insights. The result of brainstorming may be identifying a need for further research.
A collaborative approach means all the SMEs involved will have a good understanding of the sorts of solutions and associated risks they will need to plan for in the Plan phase. This helps with documentation of requirements, evaluation criteria, testing stages and response schedules. Further, SMEs will start to understand how their roles complement each other for these tasks.
Some suggestions for engaging SMEs include:
- co-design ideation session (end users, ICT and design specialists, product owners)
- checking with SMEs involved in the procurement or implementation, such as legal, cyber security, data privacy, enterprise architect
- discussing ICT solution possibilities with an ICT procurement category specialist
- outlining the problem and ICT solution possibilities to align with ICT/digital strategy
- considering whether NSW Government has solved a similar problem elsewhere and consult with a representative
- sharing and discuss information from Gartner on ICT solution possibilities
- ensuring that SMEs understand whether they need to contribute to the challenge statement, statement of requirements and evaluation criteria.
Insights from market research will inform other steps in the Discover phase:
- Iteration plan – insights will help identify the best opportunities to revisit assumptions through testing and to manage these as decision points or gates.
- Buying pathway – a clearer view of iteration may mean the procurement pathway needs revisiting to accommodate testing methods and evaluation decision points.
- Business case – insights will help build the case for change, analyse options and refine cost estimates.
Insights will become valuable in the Plan phase of procurement:
- Requirements – insights will help identify critical information for supplier proposals, including technical constraints.
- Strategy – insights will inform:
- which suppliers are approached
- how they are reached
- the level of detail they are asked to provide
- how the opportunity is framed to encourage participation and attract innovation.
Good market insights from the Discover phase should equip buying teams to develop a procurement strategy that suits the level of competition and the maturity of the market. It may involve approaching a limited number of suppliers based on a small selection of promising solutions or an open market approach to allow for a wide range of possible solutions. The procurement strategy may also involve multiple testing stages to build confidence with less mature products, as part of one or more innovation pathways.
The buying team should consider which market insights might be useful to other business units and buying teams at the end of each stage and at the end of the procurement process. Learn about more information on provisions relating to sharing market insights.
Tell us stage
The Tell us stage covers the first round of sourcing activity, including an initial market approach and the evaluation of responses.
Careful planning for evaluation ensure the evidence gathered through the evaluation process contributes to the understanding of the market and potential solutions. Buying teams will need to be able to clearly articulated evaluate decisions and how insights were arrived at to justify decisions about later stages.
Expand the drop-down boxes to better understand how to refine market insights.
In the Tell us stage, evaluation should focus on:
- how well a solution can solve the problem
- how well suppliers seem to understand the current state and users
- whether the supplier appears to have the capability to deliver the solution and work with government.
Evaluation committees should not aim to pick a single winner at this stage. Instead, they should consider which solutions they would like to know more about and potentially test. The evaluation team may need to spend some time discussing how this will work in practice. Some considerations here include:
- limits of the number and type of solutions the project can support in a testing stage
- whether they will rank suppliers overall or in different categories of solution
- whether they will progress the top ranked solutions or all solutions that pass a certain threshold for key criteria.
Requirements should not be onerous for suppliers to participate in the Tell us stage. A low barrier to entry is important to attract innovative ideas from smaller suppliers and those without government experience. The burden of proof should be low in response schedules, especially if there are likely to be many proposals. Avoid requests for technical details or evidence at this time. These requests are more useful as more is understood about potential solutions in market and when fewer suppliers are participating. This also keeps the evaluation effort low at this first stage, when evaluators will be dealing with the highest number of proposals.
The buying team should carefully consider what they need to learn from the market to validate and build on their hypothesis. They should request this information in their Statement of Requirements and/or response schedules [coming soon].
Based on proposals from an initial market approach, the buying team should be able to narrow down:
- the categories or groups of solutions that appear to be able to solve the challenge
- whether the focus can be on whole solutions or needs to include partial solutions and/or partnerships
- solutions and suppliers that may be mature and not need significant testing
- solutions that are less mature and the types of testing needed to build confidence
- the capability and capacity of suppliers to implement solutions
- the level of experience of suppliers working with government
- the level of competition and/or uniqueness of solutions
- the types of risks that will need to be managed for the preferred solution categories
- the sorts of technical details that might be relevant at the next stage to further evaluation and/or manage risk.
Buying teams will rely on the insights from this stage to:
- determine if one or more viable solutions appear to be able to address the challenge and whether it is worth proceeding to another procurement stage
- validate or adjust the scope of the next procurement stage to ensure the type of testing or sourcing is suitable for the type of solution and any risks identified
- identify what additional intelligence, including technical details, may be requested of suppliers to qualify for the next stage
- agree on a supplier shortlist
- justify abandoning the procurement or returning to market with refined requirements or response schedules.
Show us stage
With a solid understanding of possibilities from the Tell us stage and an idea of supplier capability, buying teams may invite shortlisted suppliers to demonstrate promising solutions. These solutions could be tailored to meet specific challenges or suppliers may pitch how they could be customised. Formats may include demonstration, Proof of Value (PoV), trial, hackathons, pitch-fest and/or showcase.
Expand the drop-down boxes to better understand how to further refine market insights.
In the Show us stage, the evaluation committee should focus on the ability of the solution to solve a problem. This means increasing the emphasis on how well the supplier understands the problem and the users. Suppliers with a good understanding of the problem are more likely to be good delivery partners and achieve the desired outcomes.
Requirements set here should place a higher burden of proof and/or weighting criteria related to the problem and users.
Buying teams might start to request some technical details and capability information. The technical burden of proof should still be low, particularly if there will be a separate Prove it works stage.
Having seen supplier proposals in the Tell us stage, each buying team will have a unique view of how much more evidence they need to confidently narrow down the solutions to consider investing in.
The types of insights that might be collected or refined in this stage include:
- the type of technology
- the case for change, including benefits that can be used to formulate a business case for a subsequent stage/s
- functional capability and usability
- maturity and any case studies and success stories
- ownership of all parts of the proposed solution and potential gaps or dependencies
- supplier capability to deliver in the NSW Government environment (indicative)
- supplier financial viability (indicative)
- commercial models (indicative) to inform budget decisions.
The buying team may need to consider providing data, processes or system information to suppliers for context. This can help them deliver a more relevant and successful demonstration.
After evaluating the information collected at this stage, the buying team will have more clarity about:
- solution possibilities
- solution maturity
- range of potential suppliers
- potential benefits
- supplier capability and viability
- potential risks.
This intelligence may be used to:
- determine how well solutions address the challenge
- determine if the proposed benefits of the best solution proposals will justify a further stage including funding requests or a business case
- identify specific technical risk domains and corresponding technical experts who can help identify and plan for the management of technical risks in later stages
- determine what additional evidence is needed to prove a solution works, inform the testing approach for the next stage and write stage requirements
- determine which suppliers have the best understanding of the challenge and users to become good delivery partners and which should be invited to participate in further testing.
Prove it works stage
Equipped with the market insight from the Show us stage, buying teams may choose to invite shortlisted suppliers to submit proposals for a Prove it works stage.
This stage involves testing promising solutions in a limited reach, time-boxed experiment that proves technical feasibility, uncovers gaps and validates solution assumptions. The stage may also be known as a Proof of Concept (PoC), limited implementation or a trial if testing a Software as a Service (SaaS) solution.
The PoC is usually built in a sandboxed operational environment to test interactions with real data, business processes and actual systems. To lower the risk profile of a testing stage, the PoC could be run with anonymised data and separated from live systems.
Following or instead of a PoC, buying teams might run a pilot for a period to test with real users in an operational environment.
Expand the drop-down boxes to better understand how to test promising solutions.
Evaluation in the Prove it works stage should focus on the functional and technical feasibility of the solution and whether solutions are likely to achieve the anticipated benefits of solving the problem.
Through this stage, suppliers will showcase their methods and ways of working, enabling the buying team to evaluate their delivery capability. Suppliers may have given an ‘on-paper’ view of their delivery capability in earlier stages. Evaluators can now assess those claims in practice.
For a Proof of Concept (PoC) one or more suppliers may be asked to address one or more of the challenge use cases or introduce a new specific use case. The buying team must write a new Statement of Requirements and, if needed, corresponding evaluation criteria for suppliers to participate in this stage. The requirements will be narrower and more specific than in the Show us stage. Additional mandatory requirements may be introduced at this stage, such as cyber security and data privacy, depending on the risk profile of the testing.
Requirements and assessment criteria should not be onerous for suppliers to participate in the Prove it works stage. For example, it's not necessary to furnish support training or service level agreement information. Likewise, this stage is not as concerned with assessing value for money, due to its experimental nature. The intent is to test the solution in a real world environment so issues can be ironed out before committing to a full implementation.
The types of insights collected from proposals in this stage include:
- functional capability of the solution
- technical feasibility of the solution
- technical architecture of the solution
- validating any solution assumptions
- identifying potential gaps and issues
- adherence to ICT policies including cyber security and data privacy
- how the supplier works with the delivery team including ways of working, delivery methodology, project timelines and milestones, availability and competency of resources, handling of bugs and errors
- training/change management/business adoption effort
- indicative pricing that can be used to formulate a business case [link to business case page] for a subsequent stage or stages.
The buying team may need to provide suppliers with access to facilities, data, processes and systems. This gives them more context to deliver a successful experiment.
After evaluating the Prove it works stage, the buying team will have more confidence in their preferred solution and supplier and a good understanding of the potential risks for a full implementation. With this intelligence they can:
- determine if they need to undertake further testing to address gaps, issues, risks or uncertainty
- determine if they can proceed to Build it with us stage or Implement at scale stage
- prepare a final business case and/or request implementation funding.
Build it with us stage
Equipped with insights from the Show us and/or Prove it works stage(s), buying teams should consider whether further testing or iteration is required. If they have enough confidence in a solution, they may progress directly to implementation at scale. Depending on their risk assessment and confidence level, they may choose to invite shortlisted suppliers to submit proposals for a Build it with us stage. The aim of this stage is to learn more from shortlisted suppliers about the potential scope before implementation at scale. As such, there will be specific information needs driving the requirements for this stage.
The Build it with us stage involves defining the Minimum Viable Product (MVP) that can be implemented within a time-boxed project delivery schedule. It may deliver the foundational capability that will support incremental implementation, and/or the minimum functionality required to realise benefits. The stage may require co-development with internal teams and may be followed by a pilot to test the implementation with business users.
Expand the drop-down boxes to better understand how to learn more about potential full scale scope.
The buying team should write a new Statement of Requirements and evaluation criteria before inviting suppliers to propose solutions for the Build it with us stage. The requirements will be limited, narrow and specification based. There will be additional mandatory requirements introduced at this stage, such as cyber security and data privacy standards. Contract conditions will also be introduced. These outline minimum support and performance guarantees, such as:
- service level agreements
- bug/fault remediation
- dispute resolution process.
Evaluation at this stage will heavily emphasise the technical aspects of a solution, through higher weighting or the introduction of more criteria. The importance of pricing will also increase.
Since any solution making it into the stage is assumed to realise acceptable benefits, the scope of requirements for this stage should focus on Minimum Viable Product, with evaluation based on:
- completeness and quality of deliverables
- Requirements Traceability Matrix (RTM) to assess to what extent all requirements have been met
- business acceptance testing
- thorough pilot testing in a live operational environment
- training/change management/business adoption effort
- project delivery retrospective that considers how well the supplier demonstrated they could work with us
- scalability and effort.
The types of insights collected from proposals in the Build it with us stage include:
- commercial models including End User License Agreement (EULA) and reseller conditions
- pricing that can be used to assess value for money and formulate a business case (link to business case page) for scale implementation.
- functional capability
- technical feasibility
- operational capability
- usability
- potential gaps and issues
- bugs and fault remediation approach
- cyber security and data policy adherence
- technical, process and data architecture documentation
- extensibility and interoperability
- dispute resolution and mediation process
- how the supplier will work with us including ways of working such as co-development, delivery methodology, project plan including timelines and milestones, resources skills and competency, supplier financial viability, delivery capability, internal resource engagement and estimated effort.
The buying team will need to engage in discussions with suppliers and provide access to facilities, data, processes and systems for integration.
Buying teams at the end of this stage should understand:
- what it might take to implement a solution at scale
- whether the solution is technically and commercially viable
- whether they want to continue to work with the supplier.
They should also be able to showcase the Minimum Viable Product to obtain business, sponsor and end user feedback.
All of the above allow buying teams to plan for implementation and seek relevant approvals.
Implement at scale
Equipped with the market insight progressively gathered from all testing stages, buying teams should consider whether they have sufficient confidence to progress to a full scale implementation. Depending on their risk assessment, they may choose to commit funding and invite shortlisted suppliers to submit proposals for this stage.
Expand the drop-down boxes to better understand confidence considerations for a full scale implementation.
The requirements for implementing at scale will be specification-based. They are likely to be extensive and include dependencies and constraints. Written proposals will typically need to include commercial and technical documentation. Buying teams may ask suppliers to fill out a Requirements Traceability Matrix (RTM). This matrix shows if individual solution requirements are fully, partially or not met.
The buying team may have specific follow up questions for suppliers on areas such as:
- technical architecture
- delivery methods
- commercial models
- project timelines
- resource availability
- support models
- product roadmaps.
They may also engage with shortlisted suppliers to answer questions and provide more access to facilities, security, data, processes and systems for integration.
These discussions can give the evaluation team valuable insights into how well a supplier will work with the implementation team.
The buying team must write a new statement of requirements and evaluation criteria before inviting suppliers to propose solutions for the Implement at scale stage. Any mandatory requirements not yet evaluated would be introduced at this stage.
The types of insight collected from proposals in the Implement at scale stage include:
- pricing that can be used to assess value for money
- commercial models including End User License Agreement (EULA) and reseller conditions and ongoing costs
- functional capability
- technical feasibility
- operational capability
- usability
- potential gaps and issues
- bugs and fault remediation approach
- cyber security and data policy adherence
- technical, process and data architecture documentation
- extensibility and interoperability
- dispute resolution and mediation process
- how the supplier will work with us including ways of working such as co-development and delivery methodology
- project plan including timelines and milestones
- resources skills and competency
- supplier capability and financial viability
- delivery capability
- internal resource engagement and estimated effort.
The types of insight collected during a scale implementation project include completeness and quality of deliverables, through evaluation of:
- Requirements Traceability Matrix (RTM) to assess to what extent all requirements have been met
- business acceptance testing
- thorough post-delivery testing in one or more live operational environments
- training/change management/business adoption
- project delivery retrospective that considers how well the supplier worked with us and how this might be improved
- project roadmap
- solution capability roadmap.
Business-as-usual (BAU) operations
Once the solution is in operation it should be actively evaluated by all impacted stakeholders.
Expand the drop-down boxes to better understand active stakeholder BAU evaluation.
The product owner should capture all feedback on a continuous basis to generate insights. This feedback may include:
- user experience including reliability, usability and problem identification
- bugs, faults and technical issues
- support and service levels
- supplier responsiveness
- supplier resource availability and competency.
Product owners and ICT/digital strategists should keep an eye on the market. They should keep up to date with competitive and emerging technology trends. They also need to check for developments in supplier capabilities and roadmaps. This focus helps improve business outcomes.
Procurement should periodically review if the solution is still fit for purpose and offers value for money. This review would rely heavily on the product owner and the business function or users. It is an opportunity to consider if there are any business-as-usual problems that need to be addressed.
In alignment with wider agency and government strategies, there may be opportunities for ICT/digital consolidation, expansion or retooling.
The outcome of such activities may initiate the scope of a new market challenge.