Private LLM or Cloud? GenAI Architecture for Israeli Banks and Insurers

TL;DR: Israeli regulation lets banks and insurers run GenAI in the cloud or on their own servers, under conditions: classify the data, keep sensitive data in Israel or with a provider that meets GDPR-level protection, encrypt, keep an audit trail, and manage every AI model, vendor models included, under a model risk framework. Choose the architecture per use case, based on the data it touches. For most knowledge work, RAG over a model in an Israeli cloud region is a practical start. Keep on-prem open-weight models for the most sensitive flows.

Last checked: September 2026. This is general information, not legal advice.

Why GenAI projects stall in banks and insurers

A business unit wants an assistant that answers from policies and procedures. The CISO asks where the data goes. Risk asks who validates the model. Compliance asks what the customer is told. Each question is fair, and the answers sit in documents from different regulators.

We told the story of an architecture that passed a CISO review in a single meeting. This article goes one level up: what Israeli financial regulation says, which architecture fits which use case, and how to roll it out.

What the rules say

Banks: Bank of Israel, Proper Conduct of Banking Business directives

• [Directive 362, "Cloud Computing"](https://boi.org.il/media/ndmbigpq/362_et.pdf) (last updated June 2026). Banks may use cloud services, material systems included. The board approves a cloud policy, every cloud service gets a risk assessment before contracting, and data is encrypted in transit and at rest, at minimum the data classified as sensitive. Sensitive data may go to a cloud outside Israel only if the provider meets GDPR-level protection. The directive excludes a "private cloud": infrastructure for the exclusive use of one bank, on or off premises.
• [Directive 364, "Management of IT, Information Security, and Cyber Protection Risks"](https://www.boi.org.il/media/0vvpnqtw/h2799.pdf). Published in November 2024, in force since May 2026, replacing Directives 357, 361 ("Cyber Defense Management") and 363. Section 43 requires classifying activities, processes and information assets by criticality and sensitivity. Section 61.7 requires an audit trail of who accessed what, from where and when.
• [Directive 369 on model risk management](https://www.boi.org.il/media/mcyfqkkp/h2792.pdf) (Hebrew title: "ניהול סיכוני מודלים"). In force since August 2025. It covers models that use or rely on AI, and vendor models. For AI it stresses fairness and bias, accountability that matches the level of human involvement, explainability, and full documentation, including the data used to develop and validate the model.

Insurers and other institutional bodies: Capital Market, Insurance and Savings Authority

• [Institutional Bodies Circular 2016-9-14 on cyber risk management](https://www.gov.il/BlobFolder/dynamiccollectorresultitem/2016-9-14/he/2016-9-14.pdf) (Hebrew title: "ניהול סיכוני סייבר בגופים מוסדיים") puts cloud use under the outsourcing rules and requires a dedicated risk assessment first. Sensitive or customer data may sit in a cloud abroad only with a provider checked against Israeli privacy regulations and the EU data protection directive, and sensitive data must be encrypted there. For multi-tenant systems it calls for encryption, masking or tokenization.

Everyone who holds personal data

• [Amendment 13 to the Privacy Protection Law](https://www.gov.il/BlobFolder/reports/guide_tikon13_professional/he/tikun%2013%20_170825.pdf) has applied since 14 August 2025. Banks and insurers must appoint a privacy protection officer and an information security officer. Our Amendment 13 guide covers the engineering side.
• The Privacy Protection Authority's draft guideline on AI systems (April 2025) calls a privacy impact assessment before using AI on personal data the best way to show compliance. It also asks for a policy on employee use of external GenAI tools: who may use them, which data may go in, prompt retention and opting out of training.

Where it is heading. In December 2025 an inter-agency team, including the Banking Supervision Department and the Capital Market Authority, published a final report on AI in the financial sector. It proposes treating a third-party AI engagement as outsourcing, keeps legal responsibility with the regulated body, and recommends real-time human involvement for material, high-risk decisions about individuals where no compensating controls exist, plus human oversight of the system as a whole.

Three architectures compared

Two details decide more than the table shows:

• Verify the Israeli region. Amazon Bedrock has run in AWS's Tel Aviv region since September 2025, and its cross-Region inference sends requests to other regions for processing. On any cloud, check each model's deployment type and log where each request ran.
• Read the vendor terms. Anthropic states that by default it does not train on inputs or outputs from its commercial products, the API included (Anthropic Privacy Center). Check retention and sub-processors for every vendor. Directive 362 holds the provider accountable to the bank, also when it relies on a secondary provider.

Our Enterprise AI assistant follows the middle column: it runs inside the client's Azure tenant, and its services talk over Private Endpoints.

Classify the data, then choose the model

Architecture follows data. Banks already classify information assets under Directive 364, so start there. For each use case, list the data classes it reads and writes, and who sees the output:

• Public and non-sensitive internal data (brochures, procedures): any of the three options.
• Confidential business data (pricing models, credit policy): an Israeli region or on-prem.
• Customer data and data of special sensitivity (health details in claims, financial status): mask or tokenize before the prompt, and process in Israel or on-prem.

In our experience, many tasks need the shape of a case and none of its identifiers: summarizing a claim, classifying a complaint, drafting a reply. Swap names and ID numbers for tokens before the model call, restore them only in the approved output, and scan inputs and outputs with DLP rules.

Insurance brings its own rules to every field. When we designed AIG Israel's digital claims system, we first mapped every rule and exception across five insurance types with agents, claims handlers and compliance staff (the AIG story). GenAI in claims needs the same groundwork.

RAG or fine-tuning?

Most banking and insurance use cases start with RAG: the model answers from documents retrieved at query time, and cites them.

Fine-tuning fits narrow, stable tasks: routing incoming mail, extracting fields from a standard form, a house style for letters. Train on masked or synthetic data where you can. The appendix to Directive 369 asks banks to keep and document the datasets used to develop and validate AI models, so a fine-tuned model adds a paper trail for its whole life.

Audit trail and a human in the loop

Directive 364 requires the audit trail to be protected from change and kept for a period that fits the data's sensitivity. For GenAI, log per request:

• user, role and channel
• IDs of the retrieved documents
• model version and prompt template version
• the output, and whether a person approved, edited or rejected it

Decide on prompt text on purpose. Prompts can hold customer data, which makes the log a sensitive database. One workable pattern keeps metadata in operational logs and full text in a separate store with tighter access.

For human involvement, we build three levels:

1. Assist. The model drafts and an employee decides. Oversight runs on samples.
2. Approve. Anything that reaches a customer or changes a decision waits for a logged human approval.
3. Act. The system acts alone only on low-risk, reversible steps, within set permissions.

The same logic sits behind our agentic AI model: human checkpoints at the high-stakes steps.

Model risk management for LLMs

Treat each LLM use case the way Directive 369 treats any model:

• Inventory. Model and version, prompts, retrieval sources, owner and materiality.
• Validation before release. An evaluation set from real cases, with pass thresholds for accuracy, grounding and correct refusals, signed off by reviewers outside the build team.
• Vendor models. The directive applies the same principles to them. Validate behavior on your own evaluation set and keep the vendor's documentation.
• Change control. Vendors retire model versions. Pin the version and rerun the evaluation set before any switch.
• Monitoring and fairness. Sample outputs on a schedule, track overrides and complaints, and test for bias where outputs affect customers.

A phased rollout

1. Readiness. Use cases, data classification, an architecture per use case, an employee GenAI policy and a model inventory. Our AI readiness assessment costs $5K-$15K.
2. Internal assistant on non-customer data. RAG over procedures, single sign-on, permission-aware retrieval and full logging. A RAG assistant connected to one or two systems costs $15K-$50K.
3. Employee tools on customer data. Masking, DLP, evaluation sets and validation under Directive 369.
4. Customer-facing drafts. Letters, claim summaries and service replies, each approved by a person.
5. Agents and decision support. A production AI solution costs $50K-$150K, with full validation and approval checkpoints.

Each phase earns the next. Our POC-to-production guide covers what breaks between a demo and a live system, and our AI consulting team can run the first two phases with you.

Frequently asked questions

Can Israeli banks use cloud-based LLMs? Yes, under Directive 362: a board-approved cloud policy, a risk assessment before each cloud contract, encryption, and GDPR-level protection for sensitive data processed outside Israel. Directives 364 and 369 apply as well.

Does Bank of Israel Directive 369 apply to generative AI? Directive 369 covers models that use or rely on AI, including vendor models, and has been in force since August 2025. Ask your risk function to record which GenAI tools it treats as models under the directive.

Do insurers need an on-prem LLM for customer data? Circular 2016-9-14 allows cloud use under the outsourcing rules and sets conditions for sensitive and customer data in clouds abroad. Whether on-prem is worth its cost depends on the data class and the use case.

Should a bank use RAG or fine-tuning? Start with RAG for knowledge that changes, answers that need a source and permission-aware access. Use fine-tuning for narrow, stable tasks, and document its training data under Directive 369.

If you are weighing GenAI under Bank of Israel or Capital Market Authority rules, start with one use case and the data it touches. Let's map it together.