Anonymization for Privacy-Safe RAG
Anonymize sensitive data before storage and retrieval in RAG pipelines to prevent PII leakage.
In agentic workflows, such as RAG and multi-step agents, use AI Developer Edition to anonymize sensitive data before storage and retrieval. This prevents PII leakage through embeddings, prompts, or outputs.
Anonymization removes who while preserving what happened. Names, identifiers, and contact details are replaced or suppressed, but the events, patterns, and relationships that make data useful for AI reasoning remain intact. This lets your agents learn from real data, retrieve relevant context, and produce accurate insights — without exposing the individuals behind the records.
The Problem with RAG and Sensitive Data
Documents with PII → Embeddings → Vector DB → Retrieval → LLM → Output
↑ ↑ ↑ ↑
Names, IDs PII in vectors PII in context PII in response
At every stage, real identities can leak:
- Embeddings encode PII into vectors that can be extracted
- Retrieved context passes real names/IDs to the LLM
- Agent outputs may include identities from retrieved documents
- Logs capture the full RAG pipeline with PII
The Solution - Anonymize Before RAG
Documents → Anonymize → Embeddings → Vector DB → Retrieval → LLM → Output
↑ ↑
Remove PII No PII possible
Keep utility Safe insights
How Anonymization Works
| Original | Anonymized | Utility Preserved |
|---|---|---|
| John Smith filed complaint#4521 | Person_001 filed complaint#4521 | Complaint pattern |
| sarah@company.com reported bug | user_001@example.com reported bug | Bug report context |
| SSN 123-45-6789 in record | [SSN_REDACTED] in record | Field structure |
| Employee E1001 earned $170,000 | Employee_A earned $170,000 | Compensation data |
Best Practices
- Anonymize before embedding. Never store real PII in vector databases. Anonymize documents before generating embeddings so that identities cannot be extracted from stored vectors.
- Preserve data utility. Keep patterns, relationships, and structure intact during anonymization. The anonymized data should support the same reasoning and retrieval quality as the original.
- Use consistent pseudonyms. Map the same person to the same pseudonym across all records. This preserves relationship tracking so agents can follow connections without seeing real identities.
- Test for information loss. After anonymizing, verify that the data still supports your use case. Run sample queries or training tasks to confirm that utility is preserved before deploying.
- Use irreversible anonymization for training. Apply one-way anonymization to any data used for model training. Training data should never be linkable back to real individuals.
- Document what was removed. Keep metadata about which fields were anonymized and what method was used. This supports compliance reporting and lets you audit your anonymization pipeline over time.
Example 1: Anonymize Documents for RAG Pipeline
Cursor Prompt
In Cursor’s AI chat, enter the following prompt:
Using AI Developer Edition Anonymization, anonymize all documents before storing them in my vector database for RAG retrieval. The agent should be able to retrieve and reason on content without exposing real identities.
Implementation
"""
Anonymization for Privacy-Safe RAG Pipeline
"""
import protegrity_developer_python
protegrity_developer_python.configure(
endpoint_url="http://localhost:8580/pty/data-discovery/v2/classify",
named_entity_map={
"PERSON": "NAME", "EMAIL_ADDRESS": "EMAIL", "SOCIAL_SECURITY_ID": "SSN",
"PHONE_NUMBER": "PHONE", "LOCATION": "ADDRESS"
},
masking_char="#",
classification_score_threshold=0.6,
method="redact",
enable_logging=True,
log_level="info"
)
# Original documents containing PII
documents = [
{
"id": "doc_001",
"content": "Sarah Johnson (employee E1001) filed a harassment complaint "
"against her manager David Chen on March 15, 2025. HR Director "
"Maria Lopez reviewed the case. Contact: sarah.j@company.com, "
"SSN: 123-45-6789."
},
{
"id": "doc_002",
"content": "Performance review for James Smith (E1002): Exceeded expectations "
"in Q4 2025. Salary adjustment from $130,000 to $145,000 approved "
"by VP Emily Brown. James's address: 456 Oak Ave, Chicago, IL."
},
{
"id": "doc_003",
"content": "Customer complaint from Robert Davis (account #ACC-789012): "
"Unauthorized charge of $4,500 on card ending 9012. Customer phone: "
"555-0198. Email: r.davis@email.com"
}
]
# Anonymize before vector storage using find_and_redact
anonymized_docs = []
for doc in documents:
anonymized_content = protegrity_developer_python.find_and_redact(doc["content"])
anonymized_docs.append({"id": doc["id"], "content": anonymized_content})
Anonymized Output
# What gets stored in vector DB:
{
"id": "doc_001",
"content": "Person_001 (employee EMP_A) filed a harassment complaint "
"against their manager Person_002 on March 15, 2025. HR Director "
"Person_003 reviewed the case. Contact: [EMAIL_REDACTED], "
"SSN: [SSN_REDACTED]."
}
{
"id": "doc_002",
"content": "Performance review for Person_004 (EMP_B): Exceeded expectations "
"in Q4 2025. Salary adjustment from $130,000 to $145,000 approved "
"by VP Person_005. Person_004's address: [ADDRESS_REDACTED]."
}
{
"id": "doc_003",
"content": "Customer complaint from Person_006 (account #[ACCT_REDACTED]): "
"Unauthorized charge of $4,500 on card ending [REDACTED]. Customer phone: "
"[PHONE_REDACTED]. Email: [EMAIL_REDACTED]"
}
RAG Retrieval and Reasoning
# Agent retrieves anonymized content and reasons safely
query = "What harassment complaints have been filed recently?"
# Retrieved context (anonymized)
context = vector_db.retrieve(query)
# → "Person_001 filed a harassment complaint against their manager Person_002..."
# Agent reasons without knowing real identities
response = agent.reason(context)
# → "There was a harassment complaint filed on March 15, 2025.
# HR reviewed the case. The complaint involved a manager-subordinate relationship."
# Insight provided without exposing Sarah Johnson, David Chen, or Maria Lopez
Example 2: Train & Test AI Agents with Real Data Without Exposing Identity
The Scenario
An AI agent needs to:
- Analyze customer feedback and support logs
- Learn patterns from real data
- Make predictions about customer behavior
- Without ever seeing who the data belongs to
Cursor Prompt
In Cursor’s AI chat, enter the following prompt:
Using AI Developer Edition Anonymization, prepare our customer support logs for agent training. Remove all identities but keep the patterns and insights intact so the model can learn from real behavior.
Implementation
"""
Anonymization for AI Training: Customer Support Logs
"""
import protegrity_developer_python
import pandas as pd
protegrity_developer_python.configure(
endpoint_url="http://localhost:8580/pty/data-discovery/v2/classify",
named_entity_map={"PERSON": "NAME", "EMAIL_ADDRESS": "EMAIL"},
classification_score_threshold=0.6,
method="redact",
enable_logging=True,
log_level="info"
)
# Real support logs with PII
support_logs = pd.DataFrame([
{"ticket_id": "T001", "customer": "Alice Wong", "email": "alice.w@email.com",
"issue": "Login failed after password reset", "resolution": "Reset MFA token",
"satisfaction": 4, "resolution_time_hrs": 2.5},
{"ticket_id": "T002", "customer": "Bob Martinez", "email": "bob.m@company.org",
"issue": "Billing discrepancy on invoice #4521", "resolution": "Credit issued",
"satisfaction": 3, "resolution_time_hrs": 48.0},
{"ticket_id": "T003", "customer": "Carol Chen", "email": "carol@startup.io",
"issue": "API rate limiting affecting production", "resolution": "Upgraded plan",
"satisfaction": 5, "resolution_time_hrs": 1.0},
])
# Anonymize PII columns by redacting each value
for col in ["customer", "email"]:
support_logs[col] = support_logs[col].apply(protegrity_developer_python.find_and_redact)
print("Anonymized Training Data:")
print(support_logs[["customer", "email", "issue", "satisfaction"]].to_string())
Output
Anonymized Training Data:
customer email issue satisfaction
0 Customer_001 user001@example.com Login failed after password reset 4
1 Customer_002 user002@example.com Billing discrepancy on invoice #4521 3
2 Customer_003 user003@example.com API rate limiting affecting production 5
What the Agent Learns without Identities
# Agent trains on anonymized data
model = train_support_model(training_data)
# Agent learns:
# Login issues → MFA reset resolves quickly (avg 2.5 hrs)
# Billing issues → Credits needed, longer resolution (avg 48 hrs)
# API issues → Plan upgrades, highest satisfaction
# Never learns: Alice Wong has login problems, Bob has billing issues
Use Anonymization for Safe Code Debugging with Real Logs in Cursor
The Scenario
Developers debug production issues using real logs. These logs contain:
- User emails and IDs
- Session tokens
- IP addresses
- Request parameters with PII
Cursor Prompt
In Cursor’s AI chat, enter the following prompt:
Using AI Developer Edition Anonymization, strip user data from these production logs before I send them to Cursor for debugging. Keep the error context intact.
Implementation
"""
Anonymization for Safe Debugging: Production Logs
"""
import protegrity_developer_python
protegrity_developer_python.configure(
endpoint_url="http://localhost:8580/pty/data-discovery/v2/classify",
named_entity_map={
"PERSON": "NAME", "EMAIL_ADDRESS": "EMAIL", "PHONE_NUMBER": "PHONE"
},
classification_score_threshold=0.6,
method="redact",
enable_logging=True,
log_level="info"
)
# Real production log with PII
real_log = """
[2025-03-15 14:22:33] ERROR - PaymentService
User: john.smith@company.com (ID: USR-78901)
Session: eyJhbGciOiJIUzI1NiIs...
IP: 192.168.1.42
Action: process_payment
Card: 4532-1234-5678-9012
Error: TimeoutException at PaymentGateway.charge()
Stack:
at PaymentService.process(PaymentService.java:142)
at OrderController.checkout(OrderController.java:89)
at RequestHandler.handle(RequestHandler.java:45)
Request: {"amount": 299.99, "customer_email": "john.smith@company.com"}
[2025-03-15 14:22:34] ERROR - PaymentService
User: sarah.jones@email.org (ID: USR-45678)
Session: eyJhbGciOiJSUzI1NiIs...
IP: 10.0.0.15
Action: process_payment
Card: 5411-9876-5432-1098
Error: TimeoutException at PaymentGateway.charge()
Stack:
at PaymentService.process(PaymentService.java:142)
at OrderController.checkout(OrderController.java:89)
"""
# Anonymize for safe debugging - redact all PII while preserving error context
safe_log = protegrity_developer_python.find_and_redact(real_log)
What Cursor Sees as Anonymized
[2025-03-15 14:22:33] ERROR - PaymentService
User: user_001@example.com (ID: USR-XXXXX)
Session: [SESSION_REDACTED]
IP: [IP_REDACTED]
Action: process_payment
Card: [CARD_REDACTED]
Error: TimeoutException at PaymentGateway.charge()
Stack:
at PaymentService.process(PaymentService.java:142)
at OrderController.checkout(OrderController.java:89)
at RequestHandler.handle(RequestHandler.java:45)
Request: {"amount": 299.99, "customer_email": "[EMAIL_REDACTED]"}
[2025-03-15 14:22:34] ERROR - PaymentService
User: user_002@example.com (ID: USR-XXXXX)
Session: [SESSION_REDACTED]
IP: [IP_REDACTED]
Action: process_payment
Card: [CARD_REDACTED]
Error: TimeoutException at PaymentGateway.charge()
Stack:
at PaymentService.process(PaymentService.java:142)
at OrderController.checkout(OrderController.java:89)
Cursor Debugging Output
Cursor AI Analysis:
─────────────────
The TimeoutException occurs in PaymentGateway.charge() for multiple users
at similar timestamps (14:22:33 and 14:22:34). This suggests:
1. Payment gateway is experiencing latency/outage
2. Not user-specific - affects multiple customers simultaneously
3. The call stack shows the timeout happens at the gateway level,
not in your application code
Recommendation:
- Add circuit breaker pattern in PaymentService.process()
- Implement retry with exponential backoff
- Add timeout configuration (currently using default)
- Set up health check for PaymentGateway endpoint
Result: Cursor diagnoses the issue perfectly without ever seeing real user identities.
Anonymization Methods
| Method | Description | Use Case |
|---|---|---|
| Pseudonymization | Replace with consistent fake values | RAG, training |
| Generalization | Replace specific → general (“New York” → “East Coast”) | Analytics |
| Suppression | Remove entirely ([REDACTED]) | Logs, debugging |
| K-Anonymity | Ensure each record matches K others | Publishing data |
| Differential Privacy | Add calibrated noise | Statistical queries |
Feedback
Was this page helpful?