Running the Data Discovery samples

Use the information in this section to run the Data Discovery samples provided in the data-discovery/samples folder. These samples demonstrate how to use the Data Discovery API for classification and redaction of sensitive information in text and tabular data.

Running Data Discovery

The example scripts under the data-discovery/ folder demonstrate classification and redaction using the Data Discovery v2 API. For more information about the Data Discovery APIs, refer to the section Data Discovery APIs.

Note: A dedicated data-discovery/docker-compose.yml is provided to start only the Data Discovery service.

1. Open a command prompt.

2. Navigate to the directory where AI Developer Edition is cloned.

3. Launch data-discovery services. Refer to the docker compose setup page to know how to set up the package.

4. Run any of the example scripts from the data-discovery/ directory:

   Classification - text input

   python data-discovery/samples/python/sample-classification-python-text.py
   bash data-discovery/samples/bash/sample-classification-bash-text.sh
   

   Classification - tabular (CSV) input

   python data-discovery/samples/python/sample-classification-python-tabular.py
   bash data-discovery/samples/bash/sample-classification-bash-tabular.sh
   

   Redaction

   python data-discovery/samples/python/sample-redaction-python.py
   bash data-discovery/samples/bash/sample-redaction-bash.sh
   

5. View the output of the files processed on the screen. The output displays the classification labels or redacted text returned by the Data Discovery service.

Using Notebooks for Classifying and Redacting unstructured documents

The notebook demonstrates how to use the Data Discovery API with Python’s requests library to classify and redact sensitive information in unstructured text and tabular data. It submits sample data containing sensitive information to a local Data Discovery service for classification. It also shows how the Transform API replaces detected PII entities with standardized labels, for example, [PERSON] or [SOCIAL_SECURITY_ID].

1. Make sure you have the Jupyter notebook installed in your system.

2. Navigate to the directory where AI Developer Edition is cloned.

3. Run the following command to start Jupyter Lab.

   jupyter lab
   

4. Copy the URL displayed and navigate to the site from a web browser. Ensure that localhost is replaced with the IP address of the system where the AI Developer Edition is set up.

5. Open the example at:

   • data-discovery/samples/jupyter/sample-classification-jupyter-text.ipynb
   • data-discovery/samples/jupyter/sample-classification-jupyter-tabular.ipynb
   • data-discovery/samples/jupyter/sample-redaction-jupyter-text.ipynb

6. Run all cells and see the results of the execution interactively.