When planning an AI solution, consider the following:
The increasing use of AI may pose new cyber security risks. Agencies will need to comply with the mandatory requirements as outlined in the NSW Cyber Security Policy to a target level of maturity appropriate for the level of risk involved with the processing and storage of data including personal and sensitive information before implementing an AI system in NSW. The Chief Cyber Security Officer (CCSO) has responsibility for leading a coordinated government response to cyber security failures including malware and ransomware attacks.
The Privacy Regulatory Landscape
Privacy regulation in Australia focuses on the handling of ‘personal information’. In NSW personal information is defined as information or an opinion (including information or an opinion forming part of a database and whether or not recorded in a material form) about an individual whose identity is apparent or can reasonably be ascertained from the information or opinion. Personal information could include a record of a name, address and other details, photographs and video footage, fingerprints or DNA samples. The handling of personal information is regulated at both the state and federal level.
In the context of AI, agencies need to consider whether information constitutes personal information at the time it is collected, as well as at the time when it is combined with other datasets (if applicable). This is because there is a risk of re-identification, for example data may not be very meaningful on its own, but when it is linked with other information, it may expose the movements and activities of an individual.
There are a range of legislative protections in place to protect personal data and maintain privacy:
- In NSW the Privacy and Personal Information Protection Act 1998 (NSW) (PPIP Act) applies to NSW Government agencies, NSW local councils, and public universities. The PPIP Act defines personal information and sets out 12 Information Protection Principles (IPPs) that govern the handling of personal information.
- Additionally, in NSW the Health Records and Information Privacy Act 2002 (NSW) (HRIP Act) applies to organisations that are health service providers or that collect, hold or use health information. This can include universities that undertake research, a gym that records information about your health, or even your physiotherapist. The Act defines health information and sets out 15 Health Privacy Principles (HPPs) governing health information.
- The Privacy Act 1988 (Cth) (Privacy Act) applies to most Australian Government agencies and some private sector organisations, including private universities and health service providers. There is an exemption for small businesses with an annual turnover of less than $3 million. The Privacy Act defines personal information for Commonwealth purposes and sets out 13 Australian Privacy Principles (APPs).
Additionally, the Privacy Act 1988 (Cth) provides greater protections to ‘sensitive information’ as it is subset of personal information, including genetic information, biometric information and biometric templates. The European General Data Protection Regulation (GDPR) requirements may impact your AI project. It may impose contractual conditions relating to compliance with GDPR requirements if your AI project is impacted. The Office of the Australian Information Commissioner (OAIC) and Information and Privacy Commission NSW (IPC) have information on how to understand the impact of the GDPR on your organisation.
The NSW Information and Privacy Commission has developed a suite of resources to support NSW’s voluntary data breach reporting scheme.
Best Practice - Privacy by Design
Privacy by design is the process of proactively identifying privacy risk during the development of a project or initiative so that risks can be mitigated as part of the design of the project. Privacy by design allows privacy to be “baked in” from the beginning so that your AI solution is privacy protective by default.
Consider these seven principles of Privacy by Design when rolling out an AI solution:
- Be proactive not reactive. Be preventative not remedial. Do not wait until there is a privacy breach to consider privacy. Anticipate risk and prevent/ address privacy invasive events before they occur.
- Privacy as the default setting. Think privacy first and foremost and ensure that personal information is automatically protected.
- Embed privacy into the design of your project. The result is that privacy becomes an essential component of the core functionality being delivered.
- Positive sum, not zero sum – think win/win. Can you find a solution which has the greatest benefit e.g. data generation and analytics with strengthened privacy feature?
- End to end security for full lifecycle protection. This ensures that all information is securely collected, used, retained, and then securely destroyed at the end of the process, in a timely fashion.
- Visibility and Transparency. Be open with stakeholders. Ensure the individual is made fully aware of the personal information being collected, and for what purposes. All the component parts and operations of the project should remain visible and transparent, to users and providers alike.
- Respect for user privacy. Keep it user centric.
Additional guidance on privacy by design is available through the IPC and the NSW Government’s Internet of Things Policy. The Office of the Australian Information Commission website has a guide to data analytics and the Australian Privacy Principles.
Agencies need to be aware of the following privacy obligations that may relate to an AI project:
A Privacy Impact Assessment (PIA) is a written assessment of an activity or function that:
- identifies the impact that the activity or function might have on the privacy of individuals
- sets out recommendations for managing, minimising or eliminating that impact.
A PIA ‘tells the full story’ of a project from a privacy perspective. It is essential to operate on a privacy by design basis. A PIA should be conducted early in project development to guide implementation.
PIAs can help assess the overall proportionality of a policy or project, that is, whether the use of personal information strikes an appropriate balance between the project objectives and the resulting privacy impacts. This is particularly important where individuals do not have a meaningful choice to provide the information (i.e. where the collection of information is by sensors of which they are unaware, is required by law, or is required to access essential government payments or services).
The IPC has published a Guide to Privacy Impact Assessments in NSW
a) Privacy Self-Assessment
The IPC has published Information Governance Agency Self-assessment Tools. These tools may be useful to self-assess privacy management in your organisation. The IPC recommends regular self-assessment.
- Privacy Management Plan – agencies must have a strategic planning document describing how the organisation will comply with the PPIP Act and HRIP Act. The IPC has a Guide on making Privacy Management Plans. Agencies can seek support to access and interpret the plan from the agency’s Privacy Officer.
- Privacy Collection Statement - section 10 of the PPIP Act requires agencies to inform citizens if they are collecting personal information, why it is being collected, what it will be used for and how citizens can view or amend their personal information. Agencies must make citizens aware before, or as soon as is practical after, the personal information is collected. There is a template Statement in the NSW Internet of Things Policy.
- Statutory Guidelines on Research - In NSW, Health and other information may be disclosed and used for research or statistical purposes. There are strict controls on the sharing and use of the data. The NSW IPC has resources on how information can be shared under the Privacy and Personal Information Protection Act 1998 and the Health Records and Information Privacy Act 2002.
- Public Interest Directions or a Code of Practice - Under section 41 of the PPIP Act, the Privacy Commissioner, along with the approval of the Attorney General, may make a Public Interest Direction to waive or make changes to the requirements for a public sector agency to comply with an Information Protection Principle (IPP).
- Statutory Guidelines on Management of Health Services and the Statutory Guidelines on Collection of Health Information – these two documents outline requirements for the use and disclosure of health information for the management of health services and the notification requirements when collecting health information about an individual from someone else.
Access to Information
The Government Information (Public Access) Act 2009 (NSW) (GIPA Act) provides individuals with a right to request access to their personal information held by a government agency:
- PPIP Act - if an agency collects or holds personal information it needs to make it accessible to the citizen and allow them to correct or amend their personal information as required. The IPC has a resource on how to handle requests for personal information under the PPIP Act.
- GIPA Act – agencies must consider the impact of having an AI system and the additional requests for aggregated data, the outcomes from the AI system and the subsequent decision making under the GIPA Act. An agency’s Privacy Management Plan (and possibly PIA) needs to have mitigation strategies in place if someone’s personal information is unintentionally released under a GIPA request.
Under the GIPA Act there is a presumption in favour of the disclosure of government information unless there is an overriding public interest against disclosure. The GIPA Act provides for a balancing of considerations in favour of and against disclosure, having regard to the public interest. This is known as the ‘public interest test’. Data informing an AI system and information about the AI system may be accessible under the GIPA Act, even where information is held by private sector contractors.
Provision of access to information is an important accountability and transparency measure that should be built into the design of the AI project. Under section 27 of the GIPA Act an agency must keep a register of government contracts that has (or is likely to have) a value of $150,000 (including GST) or more.
Agencies should further be aware of the following GIPA provisions as they relate to AI:
- s20 and 23 – mandatory obligations on agency heads to ensure that an agency’s information guides expressly include an inventory of both AI decision making tools and data sets as well as all policy documents that apply to those AI systems
- s121 – ensuring that where agencies outsource their service delivery functions, there is an immediate right to access to relevant information contained in records held by the contractor
- Part 3, Division 5 – regarding contract disclosure
- s114-115 – actions taken consistent with the GIPA Act in respect of disclosure of information in good faith are subject to an immunity from civil and criminal action
Managing a data or privacy breach
A data or privacy breach occurs when there is a failure that has caused (or has the potential to cause) unauthorised access to your organisation’s data. Breaches include hacking and malware, sending an email containing classified information to the wrong person, and loss of a paper record, laptop or USB stick.
Agencies should have a data breach management or response plan in place. NSW does not have a mandatory notifiable data breach reporting requirement, but the NSW Privacy Commissioner has a voluntary scheme in place. In addition, the Chief Cyber Security Officer (CCSO) has responsibility for leading a coordinated government response to cyber security failures including malware and ransomware attacks.
The OAIC has published guidance on data breach preparation and response. If the breach has cyber security aspects, agencies should report it to Cyber Security NSW.
Links to further information
The IPC has published Information Governance Agency Self-Assessment Tools. Agencies can use these tools to self-assess privacy management practices and information governance arrangements.
Visit the Information and Privacy Commission NSW website for guidance on implementing your privacy obligations under the PPIP Act and the IPPs and/or the HPIP Act and HPPs.
Visit the Office of the Australian Information Commissioner website for guidance on the Commonwealth Privacy Act.
Your department’s privacy and legal team can also assist with navigating privacy and legal obligations.
Stakeholder engagement is an important element in agencies being able to understand and respond to legitimate concerns of the citizens or communities who may impact or be impacted by an AI project or AI-informed decision. The acceptance of new technology by the community is aided by allowing an opportunity to provide feedback and ask questions to improve understanding and address concerns.
Effective stakeholder engagement begins at the planning phase of the project and must be revisited at every opportunity across the AI project. Engagement will help validate assumptions and ideas when designing and implementing an AI system, including identifying ambiguities or misunderstandings that may result in the AI system generating a false positive or false negative decision.
Stakeholder engagement must involve diverse users, where appropriate, including those from culturally diverse backgrounds, regional and rural residents, Aboriginal and Torres Strait Islanders, those with disability, young people, and senior citizens. Care should be taken to be culturally sensitive and respectful when engaging. Consulting your internal diversity groups or diversity managers is one way to start this process.
General information on running an effective stakeholder engagement process can be found in the Internet of Things Policy.
This section addresses the need for minimising risks associated with AI as part of the planning process. Information on general risk and compliance management practices can be found at Section 3.4 of the Internet of Things Policy.
A key risk is that the underlying data used to train the AI system, or that is used for AI analysis is not a representative sample (i.e. over or underrepresent different parts of the population) and/or that it contains historical bias. A data set that is incomplete, insufficient or contains historical bias will provide skewed results which will then be built into the AI system.
For example, AI systems designed to flag suitable job applications based on historical hiring activity will perpetuate any biases contained within those historical hiring decisions - the system learnt from what did happen, rather than what should happen.
Only explicit compensation for this potential bias in the data will correct the system’s behaviour. An agency can correct the bias by obtaining additional data to make the dataset truly representative.
Data is the essential input - its quality, sensitivity, and re-use potential shape the design of AI systems. This section provides advice on the appropriate use of data for AI purposes. It compliments existing data legislation and policies.
The following Standards have recently been developed by the International Standards body ISO/IEC (International Organisation for Standardisation/International Electrotechnical Commission) Committee SC42 on Artificial Intelligence:
- ISO/IEC 20546:2019 Information technology – big data – overview and vocabulary
- ISO/IEC 20547-2:2018 Information technology – big data reference architecture – Part 2: use cases and derived requirements
- ISO/IEC 20547-5:2019 Information technology – big data reference architecture – Part 5: Standards roadmap
The key data considerations for an AI project are:
- Engage with the community to make sure the data is representative: Existing customer/user data can be used to predict behaviour and personalise services. While this is efficient for both agencies and customers, it can have unintended consequences (e.g. limiting a customer’s options by using historical data).
Agencies can take the following steps:
- Understand citizen needs by creating personas to reflect the different citizen groups; identify the missing, underrepresented or misrepresented users in the data; and test how well the AI system will deliver fair outcomes for all users.
- Make sure the AI system is “fit-for-purpose” and tailored to the specific and local need.
- Leverage stakeholder experience in service delivery to understand nuances in the dataset, business context, and to clarify the objectives for the AI system.
- Learn from academic research centres, including well-established ethics, data management and compliance practices.
- Find alternative datasets that can achieve the same objective or disclose the new or secondary use of a dataset: AI systems may draw in multiple datasets from different sources to find new patterns and inform recommendations. An agency needs to determine if it can use the data for the AI system. This can be challenging for historical data that may have been collected for a different purpose.
Agencies can take the following steps:
- Use the most recent data available.
- Find alternative data or seek less sensitive input data (i.e. de-identified, key-coded) that can yield similar results
- Ensure personal information is managed in accordance with privacy legislation and community expectations
- Verify if consent is needed considering the potential uses and consequences
- Where possible, provide adequate notice to individuals, provide options to withdraw their consent and alternatives to receive the same quality of service
- If obtaining consent is impossible or impractical, provide timely disclosure, or clear communications of future use of data and service expectations.
- Use AI insights for public good – combining data sets for AI use can provide invaluable insights, but it also presents a risk of re-identifying personal information. Whilst there are situations where citizens want their data used (i.e. seamless service delivery as exemplified in the development of the “Tell Us Once”), citizens expect agencies to determine the level of sensitivity of the insights, use an appropriately secured system and follow the current privacy legislation and data policies.
Agencies can take the following steps:
- Only collect and use the data that is needed
- Consider how the reidentified data will be used. Will it be for the public good or detriment?
- Use the five safes framework to guide decision on the effective use of sensitive information
- Safeguard collected biometric data or sensitive data of vulnerable persons: in technology like visual recognition, highly sensitive biometric data is collected and used. Agencies may find themselves having a greater duty of care or confidentiality responsibilities.
Agencies can take the following steps:
- Learn from existing practices (e.g. Safeguarding the biometric data of children from their future employers and to safeguard health records from insurance companies).
- Ensure they abide by existing legislation – (e.g. destroy personal information as soon as the reason it was collected for is completed (Privacy and Personal Information Act 1998))
- Understand the context in which the data was collected and its quality: Agencies may seek another agency’s data for its AI system. However, there are complexities, particularly around privacy and consent that must be considered. For example, some data requires consent for use (e.g. health records) and some data is more sensitive than others (i.e. personal data such as annual income is more sensitive than aggregated data like traffic flow information). There may also be legislative restrictions on data/ information-sharing.
Agencies can take the following steps:
- Appoint a data custodian - agencies are responsible for the security of any potentially private or sensitive data used within an AI system. Agencies need to determine who, or what teams, are responsible for the ongoing security of this data, and who is responsible for acting in the case of a security breach? This needs to be decided before any system is deployed.
You may need to consider the legal implications of using an AI solution for decision making. Legislation can be specific about a specific officer holder being the person who is authorised to make decisions or issue requests for payment or rectification on behalf of the state. A preliminary list of legislation can be found in the AI Policy. Consult your agency’s legal team for further information.
AI projects will be subject to certain assurance processes in addition to existing assurance mechanisms. This is to ensure that AI projects are subject to the highest level of scrutiny and continue to regard privacy, security and data bias as key concerns.
Initially, all AI projects will submit project overviews to the AI Review Committee. High-risk projects will then also have to progress through the ICT Assurance Framework. The AI Review Committee is in place to assist agencies to manage risks developing from AI projects.