Using AI with customer data: the privacy questions that actually matter

Ask a business whether its use of AI is private and you'll usually hear some version of "they don't train on our data". On business and API plans that's mostly true. Google, Microsoft, OpenAI and Anthropic all say so in writing.

It's also the easiest question in AI privacy, and the one that tells you least. Customer information can reach an AI model through a tool you already pay for, a staff member's personal account, your own systems or a supplier. Each route has its own terms, its own retention rules and its own path overseas. Under Australian privacy law, the business that collected the information generally stays accountable for all of them.

This guide covers the questions that sit around training. We checked every vendor and regulator claim in it against the source on 12 October 2026.

Why AI privacy is harder than it looks

Using AI used to mean someone deciding to open a chatbot. Now it mostly arrives inside software you already use, often switched on by default. Gemini features are on by default in Google Workspace (Google). Copilot Chat is pinned to the navigation bar of the Microsoft 365 apps for work accounts (Microsoft). Shopify's Sidekick assistant is available from every page of a store's admin (Shopify).

That's only one route. Customer data can reach a model in at least four ways:

• through AI features built into the tools you already pay for
• through staff using personal AI accounts for work
• through your own website, app or automations calling a model directly
• through suppliers (agencies, contractors and software vendors) and the subprocessors they rely on

Each route sits under different terms. The person who chose a tool often isn't the person who configured it, and nobody may have looked at the AI settings at all.

Six questions that matter more than training

1. Which terms apply?

Every major AI provider runs two sets of rules. Business, enterprise and API customers get one set. People using the free or personal app get another.

Sources: OpenAI, Anthropic, Google Gemini API terms, Google Workspace and the Gemini app.

So the same person doing the same task can sit under very different terms depending on which account they're signed into. Personal Gmail is a good example. Google lists the regions where Gmail's smart features are off by default (the EEA, Japan, Switzerland and the UK). Australia isn't one of them, and Google says that when smart features are on, "your data may be used to improve these features" (Google).

2. How long is it kept?

Not training on data isn't the same as not storing it. OpenAI keeps API abuse monitoring logs for up to 30 days. On Anthropic's consumer plans, chats used for training can be kept for up to five years. Google keeps Gemini app chats that are selected for human review for up to three years. None of that is hidden, but it's rarely what people picture when they hear "private".

3. Can a person read it?

Human review is common on free and consumer tiers. Google's terms for the unpaid Gemini API say human reviewers may read and annotate inputs and outputs, and tell users: "Do not submit sensitive, confidential, or personal information to the Unpaid Services." On the business side, Google says Gemini chats on Workspace accounts aren't reviewed by humans (Google). Same company, same model family, different answer.

4. Where is it processed?

Mostly overseas, and mostly in the US. Microsoft says customers outside the EU "may have their queries processed in the US, EU, or other regions" (Microsoft). It has announced in-country Copilot processing for Australia, first targeted for the end of 2025 and now expected by the end of 2026 (Microsoft). OpenAI offers Australian data storage for eligible customers, but the processing itself isn't local. Overseas processing isn't unlawful. It does carry obligations, which we cover below.

5. Who else touches it?

AI features are often powered by someone else's model. Shopify's subprocessor list names OpenAI, Anthropic, Microsoft, Applied Compute and Fireworks.ai for conversational AI, covering "personal data if input to or included in a response by Shopify Sidekick". It lists Groq for AI services with a data scope of "all platform data". All six are in the US (Shopify).

That's normal, and Shopify discloses it. It also means a single AI feature can involve several companies you never signed a contract with. Look for what isn't documented too. We couldn't find a published statement from Shopify on whether Sidekick data is used to train models, or a store-wide switch to turn Sidekick off. That doesn't mean anything is wrong. It means it's a question to ask.

6. Is it making decisions about people?

Drafting a product description and deciding whether someone gets a refund are different kinds of AI use. The second carries a specific legal obligation from 10 December 2026.

What Australian law expects

Australian Privacy Principle 8 says that before you disclose personal information to someone overseas, you take reasonable steps to make sure they handle it in line with the Australian Privacy Principles. In practice that usually means a contract: the vendor's business terms, its data processing addendum and its subprocessor list. Section 16C of the Privacy Act then does the part most businesses miss. If the overseas recipient mishandles the information, the law generally treats it as if you did (OAIC). There are exceptions, but accountability staying with you is the starting point.

That's why business terms matter so much. A business plan gives you a contract to point to. A staff member's personal account gives you nothing.

The Office of the Australian Information Commissioner (OAIC) has been direct about public tools. Its guidance says organisations should "not enter personal information, and particularly sensitive information, into publicly available generative AI tools" (OAIC). It's guidance rather than law, but it shows where the regulator stands.

The bigger change is a new transparency duty in the Privacy and Other Legislation Amendment Act 2024, starting on 10 December 2026 (Federal Register of Legislation). If a computer program uses personal information to make a decision, or to do something substantially and directly related to making one, and that decision could reasonably be expected to significantly affect someone's rights or interests, your privacy policy has to say so. It needs to describe the kinds of personal information used and the kinds of decisions involved.

The OAIC released its guidance on the new duty on 30 September 2026, including updated APP 1 guidelines, a fact sheet and a flowchart (OAIC). Law firm Allens reads the guidance as covering everyday software, spreadsheets and chatbots included. It also notes that a human signing off doesn't take a tool out of scope if its output is a key factor in the decision, and that the duty sits with the business using the tool rather than the vendor that built it.

Some places we'd look first (our reading, not legal advice):

• a tool that automatically approves or declines refunds or returns
• a chatbot that decides whether someone qualifies for a trade or wholesale account
• automated fraud screening that cancels orders without a person looking

Two limits matter. This is a disclosure duty in your privacy policy, not a consent requirement or a ban. And it applies to APP entities. The small business exemption still stands: businesses with annual turnover of $3 million or less are generally outside the Privacy Act, unless an exception applies, such as providing a health service or trading in personal information (OAIC). The government's next round of privacy reforms was released as an exposure draft, and consultation closed on 18 September 2026. It isn't law yet.

Being exempt yourself doesn't mean your customers or clients are. If you handle their customers' data, expect their contracts to hold you to their standard.

Where it gets complicated in practice

AI that's on by default

Default settings are a decision someone else made for you. Google Workspace lets admins switch Gemini features on or off by service, organisational unit or group. Microsoft 365 admins can choose not to pin Copilot Chat. Where a tool has no switch, decide who can use its AI features with customer data, and write that down.

A personal account, used for work

This is the scenario that worries us most, because it's so ordinary. Someone exports a customer list and pastes it into a personal ChatGPT or Gemini account to plan a campaign, because that's the account they had open. There's no business contract behind it, the consumer terms apply and the data may be used for training. A written rule helps. Giving people a business account, so they never need the personal one, helps more.

Building on a model yourself

When your website, CRM or automations call an AI model directly, you choose the provider, the plan and sometimes the region. Business API terms generally exclude training, so the questions shift to retention, which data processing terms apply (Google, for example, processes paid Gemini API data under its data processing addendum) and how much personal information the model needs to see at all. Often the answer is very little. A model can work from counts, segments or customer IDs rather than names and email addresses.

Suppliers with access to your customers' data

Your accountability doesn't stop at your own staff. When a developer, agency or contractor works in your store, CRM or inbox, ask how they use AI with your data, which accounts they use and whether customer information ever goes into a model. Get the answer in writing, ideally in their terms.

When private or local models are worth it

For most work, a business-tier AI service with clear terms is enough. Health information, financial records or a client contract that requires data to stay onshore can change that. There are two main alternatives.

The first is an Australian-hosted cloud model. Amazon Bedrock offers an Australian cross-region profile for several Claude models that routes requests between Sydney and Melbourne. AWS itself notes that geographic profiles aren't the same as single-region data residency, so check the details for the model you use (AWS).

The second is an open-weight model you run yourself. OpenAI's gpt-oss models are released under the Apache 2.0 licence. The larger gpt-oss-120b runs on a single 80 GB GPU and the smaller gpt-oss-20b within 16 GB of memory (Hugging Face). Alibaba's Qwen3 models are also published under Apache 2.0 (Hugging Face).

Running a model yourself moves the privacy question rather than removing it. The data no longer leaves your environment, but you now own the hardware, the security and the updates, and a self-hosted model may trail the best hosted ones on harder tasks. For most businesses this makes sense for a specific sensitive workload, not as a blanket replacement.

A privacy check before you use AI with customer data

1. Map the routes. List the AI features in the tools you pay for, any personal accounts staff use, your own integrations and the suppliers who touch customer data.
2. Put everyone on business accounts. Make it a written rule that customer data never goes into a personal AI account, contractors included.
3. Turn off what you don't use.
4. For each tool, note how long data is kept, whether people can read it, where it's processed and which subprocessors are involved.
5. Send models as little personal information as the task needs. Counts, segments and IDs usually do the job.
6. Ask your suppliers how they use AI with your data.
7. Update your privacy policy to cover your AI providers and overseas processing. If you're an APP entity, describe any automated decisions before 10 December.
8. Decide which data, if any, justifies a private or local model.

Training was the easy question

The vendors have answered the training question well, at least for business customers. The rest is governance: knowing every route your customers' data takes into a model, choosing the terms on each one and being able to explain them when someone asks.

If you'd like a second pair of eyes on how AI is set up across your business, that's the kind of work we do in AI consulting.

This article is general information, checked against vendor and regulator pages on 12 October 2026. It isn't legal advice. Vendor terms change often, so check the current version before you rely on any of it.