Data Discovery on

Data Discovery Architecture

Tue, 20 Feb 2024 00:00:00 +0000

Data Discovery is a powerful, developer-friendly feature. For more information, refer to the complete body of the Data Discovery documentation.

Overview

Data Discovery Text Classification service advances data discovery and classification. It specializes in the detection of Personally Identifiable Information (PII), Protected Health Information (PHI), and Payment Card Information (PCI) within plain text and free-text inputs. Unlike traditional structured data tools, it excels in dynamic, unstructured environments such as chatbot conversations, call transcripts, and Generative AI (GenAI) outputs.

What's New

Mon, 01 Jan 0001 00:00:00 +0000

Data Discovery

Standardized v2 APIs for Classify for Text and Tabular data, and Transform.
New endpoints added for API docs, log level management, and version info.
Improved Context Provider and Pattern Provider AI models.
Updated Classify API default threshold to 0.7. The default threshold for v1.1 remains at 0.0 for compatibility.
Added usage metrics and per‑language accuracy metrics.
Extended PII detection to multiple Markdown dialects.

For more details, refer to What’s New in Data Discovery.

Prerequisites for Data Discovery

Tue, 20 Feb 2024 00:00:00 +0000

Ensure that the following prerequisites are met before running these examples for Data Discovery:

Docker CLI, Docker Compose, and Python are installed. For more information, refer to AI Developer Edition, Pre-requisites Guide.
For shell samples: Bash version greater than or equal to 5.1.8 and curl version greater than or equal to 7.76.1.
For notebook samples: JupyterLab version greater than or equal to 4.5.6.

Setting up Data Discovery

Tue, 20 Feb 2024 00:00:00 +0000

Use the containers to set up the Data Discovery components required for identifying sensitive data.

Open a command prompt.
Navigate to the cloned repository location for protegrity-ai-developer-edition.

Running the Data Discovery samples

Tue, 20 Feb 2024 00:00:00 +0000

Use the information in this section to run the Data Discovery samples provided in the data-discovery/samples folder. These samples demonstrate how to use the Data Discovery API for classification and redaction of sensitive information in text and tabular data.

Running Data Discovery

The example scripts under the data-discovery/ folder demonstrate classification and redaction using the Data Discovery v2 API. For more information about the Data Discovery APIs, refer to the section Data Discovery APIs.

Uninstalling Data Discovery

Tue, 20 Feb 2024 00:00:00 +0000

Open a command prompt.
Navigate to the cloned repository location.
Uninstall Semantic Guardrails if it is installed. For complete instructions, refer to Uninstalling Semantic Guardrails.
Navigate to the data-discovery directory.
```
cd data-discovery
```
Run the following command to remove the containers and images.
```
docker compose down --rmi all
```