Data Discovery

• 1: Data Discovery Architecture
• 2: What's New
• 3: Running the Data Discovery samples
• 4: Using the Data Discovery APIs
• 5: Uninstalling Data Discovery

1 - Data Discovery Architecture

Data Discovery is a powerful, developer-friendly feature. For more information, refer to the complete body of the Data Discovery documentation.

Overview

Data Discovery Text Classification service provides real-time classification of sensitive data in text inputs. It specializes in the detection of Personally Identifiable Information (PII), Protected Health Information (PHI), and Payment Card Information (PCI) within plain text and free-text inputs. Designed for unstructured text such as chatbot conversations, call transcripts, and GenAI outputs.

Architecture

For more information about the general architecture and working of Data Discovery, refer to General architecture of Data Discovery.

2 - What's New

Data Discovery

• Standardized v2 APIs for Classify for Text and Tabular data, and Transform.
• New endpoints added for API docs, log level management, and version info.
• Improved Context Provider and Pattern Provider AI models.
• Updated Classify API default threshold to 0.7. The default threshold for v1.1 remains at 0.0 for compatibility.
• Added usage metrics and per‑language accuracy metrics.
• Extended PII detection to multiple Markdown dialects.
  

For more details, refer to What’s New in Data Discovery.

Major Changes

• Added Jupyter notebooks examples
  • data-discovery/samples/jupyter/sample-classification-jupyter-text.ipynb
  • data-discovery/samples/jupyter/sample-classification-jupyter-tabular.ipynb
  • data-discovery/samples/jupyter/sample-redaction-jupyter-text.ipynb
    

For more information on these examples, refer to Notebooks.

3 - Running the Data Discovery samples

Use the information in this section to run the Data Discovery samples provided in the data-discovery/samples folder. These samples demonstrate how to use the Data Discovery API for classification and redaction of sensitive information in text and tabular data.

Running Data Discovery

The example scripts under the data-discovery/ folder demonstrate classification and redaction using the Data Discovery v2 API. For more information about the Data Discovery APIs, refer to Data Discovery APIs.

Note: A dedicated data-discovery/docker-compose.yml is provided to start only the Data Discovery service.

1. Open a command prompt.

2. Navigate to the directory where AI Developer Edition is cloned.

3. Launch data-discovery services. Refer to the docker compose setup page to know how to set up the package.

4. Run any of the example scripts from the data-discovery/ directory:

   Classification - text input

   python data-discovery/samples/python/sample-classification-python-text.py
   bash data-discovery/samples/bash/sample-classification-bash-text.sh
   

   Classification - tabular (CSV) input

   python data-discovery/samples/python/sample-classification-python-tabular.py
   bash data-discovery/samples/bash/sample-classification-bash-tabular.sh
   

   Redaction

   python data-discovery/samples/python/sample-redaction-python.py
   bash data-discovery/samples/bash/sample-redaction-bash.sh
   

5. View the output of the files processed on the screen. The output displays the classification labels or redacted text returned by the Data Discovery service.

Using Notebooks for Classifying and Redacting unstructured documents

The notebook demonstrates how to use the Data Discovery API with Python’s requests library to classify and redact sensitive information in unstructured text and tabular data. It submits sample data containing sensitive information to a local Data Discovery service for classification. It also shows how the Transform API replaces detected PII entities with standardized labels, for example, [PERSON] or [SOCIAL_SECURITY_ID].

1. Make sure you have the Jupyter notebook installed in your system.

2. Navigate to the directory where AI Developer Edition is cloned.

3. Run the following command to start Jupyter Lab.

   jupyter lab
   

4. Copy the URL displayed and navigate to the site from a web browser. Ensure that localhost is replaced with the IP address of the system where the AI Developer Edition is set up.

5. Open the example at:

   • data-discovery/samples/jupyter/sample-classification-jupyter-text.ipynb
   • data-discovery/samples/jupyter/sample-classification-jupyter-tabular.ipynb
   • data-discovery/samples/jupyter/sample-redaction-jupyter-text.ipynb

6. Run all cells and see the results of the execution interactively.

4 - Using the Data Discovery APIs

Data Discovery has three types of API Endpoints:

• Classify to identify, classify, and locate sensitive data.
• Transform to identify, classify, and transform sensitive data.
• Common APIs, the standard operational endpoints available on the service.

For more information about Data Discovery APIs, refer to the complete body of the Data Discovery documentation.

5 - Uninstalling Data Discovery

1. Open a command prompt.

2. Navigate to the cloned repository location.

3. Uninstall Semantic Guardrails if it is installed. For complete instructions, refer to Uninstalling Semantic Guardrails.

4. Navigate to the data-discovery directory.

   cd data-discovery
   

5. Run the following command to remove the containers and images.

   docker compose down --rmi all