Adding AI to software without putting data privacy at risk means deciding what the model is allowed to see, what it is allowed to do, and who checks the result. Leaks rarely come from the model itself: they come from data sent without sorting, forgotten access rights and an AI given too much autonomy. These three points are settled at the design stage, before the first line of code.
Many software vendors and companies want to add an AI feature to their product or internal tool, and hesitate because of customer data. Here are the concrete risks, the rules in France and Switzerland, and the design choices that let you move forward without exposing what must stay protected.
Why does AI expose a software product’s data?
A generative AI feature works by sending text to a model and displaying what it returns. All the risk lies in those two journeys: what goes out, and what comes back.
What the model actually sees
With every call, the model receives far more than the user’s question. It receives the instructions written by the vendor, the documents or records attached to help it answer, and often part of the conversation history.
In business software, these elements quickly include customer names, amounts, internal exchanges or health data. What looked like a simple writing aid actually carries part of your database.
So the first question is not “which model should we choose?”, but “what goes out, to whom, and for what purpose?”. Without a written answer, nobody can guarantee what a call contains.
The model provider may also keep these exchanges for a while, in its technical logs. That is something to check in its terms, before the first trial with real data.
Answers that say too much
The risk does not stop at sending. OWASP, a reference organisation in application security, lists sensitive information disclosure among the main risks of applications built on large language models, in its entry LLM02:2025 Sensitive Information Disclosure.
It names personal, financial and health data, confidential business information, security credentials and legal documents. The same entry points out that an instruction telling the model not to reveal certain data is not always followed, and can be bypassed.
In other words, politely asking the model to keep quiet is not enough. What it must not reveal, it must not receive.
The risk grows when several customers share the same feature. An answer built from one customer’s documents must never appear for another.
Documents that give orders
The third risk is less well known. In its entry LLM01:2025 Prompt Injection, OWASP describes indirect injection: external content, such as a web page or a file, can contain instructions that the model interprets and that change its behaviour.
In software that summarises incoming emails or analyses attachments, a booby-trapped document can therefore try to extract information or trigger an action. OWASP notes that these instructions do not even need to be readable by a human.
Features that read content from outside therefore deserve particular attention: incoming emails, public forms, documents uploaded by third parties.
The more the AI can act in your systems, the heavier this risk weighs. It is one of the reasons to keep a person in the loop, as we will see below.
AI and data privacy: which rules apply?
No law forbids using AI on business data. But the existing rules apply in full, in France as in Switzerland.
In France and the European Union
The GDPR applies as soon as an AI feature processes personal data: a defined purpose, data minimisation, security, information for the people concerned. The CNIL, France’s data protection authority, has published practical guidance sheets on AI explaining how to apply these rules when developing AI systems.
For a vendor or company integrating an existing model, the most concrete point is the contract with the model provider. It processes data on your behalf: it is generally your processor, with the clauses that implies.
We cover those clauses, and what to ask a vendor, in our article on custom business software or SaaS. The same questions apply to a model provider: where is the data processed, is it kept, is it used to train the model.
If the feature makes decisions that affect people, such as sorting job applications, the framework becomes stricter. Have it reviewed by a lawyer before development.
In Switzerland, current law already applies
The Swiss Federal Data Protection and Information Commissioner has made it clear: current data protection law is directly applicable to AI. Manufacturers, providers and operators of AI systems must make the purpose, operation and data sources of the processing transparent.
The Commissioner adds that a user interacting with a language model has the right to know whether they are talking to a machine, and whether the data they enter is reused, for example to improve the system.
For software used in Geneva or Lausanne, this shows up in the interface: a clear notice that the feature uses AI, and honest information about what happens to the data entered.
A French company serving Swiss customers may fall under both frameworks. It is better to design once for both than to fix things afterwards.
How do you design an integration that protects data?
In April 2024, ANSSI, France’s national cybersecurity agency, published security recommendations for a generative AI system. Its recommendation R7 is clear: take data confidentiality into account from the design stage. Here is what that means in a product.
What to settle before sending anything to a model.
Send less, and only what is useful
For each feature, list the data the model really needs to do the task. A ticket summary does not need the customer’s address; a category suggestion does not need the invoice amount.
When a piece of personal data adds nothing to the result, remove or replace it before sending. Some features can work this way without sending any personal data at all.
Write these rules down, feature by feature. They will serve the developers, your data protection officer and the customers who ask.
Respect existing access rights
Your software already knows who may see what. AI must not become a shortcut around those rights. ANSSI’s recommendation R8 asks for the need-to-know principle to be taken into account from the design stage.
This is especially true of assistants that query a document base. If a colleague cannot open a file, the assistant must not be able to summarise it for them: documents must be filtered according to their rights before they reach the model.
That filtering happens in your code, not in the instructions given to the model. An instruction can be bypassed; a query that never returns the document cannot.
Filter what goes in and what comes out
ANSSI’s recommendation R25 advises filtering user inputs and outputs. On the way in, you spot data that should not go out and attempts to misuse the feature.
On the way out, you check that the answer contains no information that should not be displayed, before showing it to the user. This check also protects against indirect injection coming from a document.
No filter is perfect. It adds to the other measures, it does not replace them.
These filters are tested like the rest of the code, with concrete examples of what they must block and what they must let through.
What role should AI play in the software?
The more autonomous an AI is, the more an error or an injection can cost. The level of autonomy is chosen feature by feature, according to what a mistake would cost.
The more AI acts alone, the stricter the frame must be.
Suggest rather than act
ANSSI’s recommendation R9 calls for ruling out automated use of AI systems for critical actions, whether critical for the business, such as a transaction or published content, or for the infrastructure. OWASP, for its part, recommends human approval for high-risk actions and the principle of least privilege.
In practice, AI prepares and a person approves. A reply to a customer is drafted, then reviewed before sending; an accounting entry is proposed, then confirmed.
This sometimes slows the user down a little. Above all, it prevents a plausible error from reaching a customer without anyone seeing it.
That is how the products we design work. On the project with Metroscope for EDF, Victor Michel, founder of the studio, designed alerts with recommended actions while leaving the decision to the teams. In the studio’s own tools, the person reviews and approves before anything is published.
Plan a fallback mode
An AI service can be unavailable, slow down or start answering badly. ANSSI’s recommendation R15 advises planning at least a workaround procedure so users can keep working.
In concrete terms, the AI feature must be able to be switched off without blocking the software. Users keep a way to do the task by hand, and the team can disable the feature for one customer or for all within minutes.
It is also protection against a provider changing its terms: the feature can be suspended while you find another solution.
Keep a record of every call
ANSSI’s recommendation R29 asks for all processing within the AI system to be logged. Without a log, it is impossible to know afterwards what was sent, what the model answered and what the user did with it.
The log also helps improve the feature: accepted, edited or rejected suggestions show where AI really helps. It must itself be protected and kept for a defined period, since it contains data.
How do you check before opening the feature to everyone?
A successful demo proves nothing about privacy. Two steps let you check before rolling out.
Test on real cases, including malicious ones
Build a set of cases taken from daily work, including the hard ones, and add deliberate attempts: a document with hidden instructions, a question fishing for another customer’s information, a request outside the intended role.
For each one, check what the model receives and what it answers. Replay the set at every change of model, provider or instructions, because safe behaviour can change from one version to the next.
Bring in someone who did not design the feature. They will more easily find the questions the designers never imagined.
Our article on where AI creates value in a B2B SaaS explains how to choose the task to assist; this test set comes right after, before development.
Open in stages
Start with a small group of volunteer users, on real data but a limited scope. Follow the log, the feedback and any incidents for a few weeks.
Only widen access once the data rules have been checked in real conditions. Tell your customers the feature is coming, what it does and what it does not do.
Keep the option to disable the feature for a customer who asks. Some large accounts will want to run their own analysis before opening it to their teams.
For companies in Geneva and French-speaking Switzerland, we run this scoping in person and remotely, as our AI consultant in Geneva page explains.
The simplest way to start is to talk it through: in 30 minutes, we look at the feature you have in mind and the data it would touch.
Sources
- ANSSI, Recommandations de sécurité pour un système d’IA générative (29 April 2024)
- OWASP, LLM01:2025 Prompt Injection
- OWASP, LLM02:2025 Sensitive Information Disclosure
- Swiss Federal Data Protection and Information Commissioner, La loi actuelle sur la protection des données est directement applicable à l’IA
- CNIL, Les fiches pratiques IA
