Document datasets

1. Click the Datasets tab.
2. Select the dataset you want to classify.

1. Classify the dataset in this window.

Edit name

Tag domains the dataset belongs to

Tag the provenance of the dataset

Tag the level of data trust

Tag the current status of the data

Add a comment to the dataset

Document personal data

To access the personal data panel :

1. Click the Datasets panel in the Data Governance portal
2. Select a dataset
3. Select the Personal data tab in the dataset overview 

Tag dataset containing personal data

1. Check "contains personal data" box if the dataset contains personal data

Document the consent and / or configure anonymization process

1. Click to Edit Settings to document the consent and / or configure anonymization process

Tag the user consent field

1. Select the field specifying if the user consented to the use of his personal data

Tag the entry date field

1. Select the field specifying entry date
2. Specify the number of days after entry date data will be anonymized

Select fields to be anonymized

1. Select fields to be anonymized

Enable anonymization

1. Check "Enable anonymization" box to declare anonymization process
2. Click on "Enable anonymization" button to save settings and anonymization job will launch depending on scheduling

Anonymization job

An default anonymization process in scala is proposed by Saagie. The code source is available here : https://github.com/saagie/outis.

You can use it as is, change it or replace it with one of your process.

To use it, build jar and create Spark processing job on platform with command line : 

spark-submit \
--conf "spark.executor.extraJavaOptions='-Dlog4j.configuration=log4j.xml'" \
--conf spark.ui.showConsoleProgress=false \
--driver-java-options "-Dlog4j.configuration=log4j.xml" \
{file} -u hdfs_user -t metastore_url -d datagov_user -p $ENV_VAR_PASSWORD datasetsToAnonymized_url callback_url

where :

• hdfs_user = user to launch job - user must have right to write in hdfs
• metastore_url = url of the hive metastore (exp : thrift://nn1:9083)
• datagov_user = user to access to Data Governance on the platform with right "Access all datasets" (may be the same as hdfs_user)
• datasetsToAnonymized_url = url to obtain datasets to anonymized (exp : http://{IP_DATAGOVERNANCE}:{PORT}/api/v1/datagovernance/platform/{PLATFORM_ID}/privacy/datasets)
• callback_url = url to inform dataset is anonymized (exp : http://{IP_DATAGOVERNANCE}:{PORT}/api/v1/datagovernance/platform/{PLATFORM_ID}/privacy/events/datasetAnonymized)
• $ENV_VAR_PASSWORD : environment variable for password

You can dowload the last version of the jar here :  

outis-link-1.1.0.jar

Exceptions handling

No dataset anonymization if : 

• You don't provide an entry date
• You don't provide a list of fields to anonymize
• If you provide a string field as an entry date without a pattern to parse the data
• Dataset isn't csv or parquet files on table

No record anonymization if : 

• The value of the field to anonymize has a NULL value
• You try to replace a string field mark as a date without providing a pattern
• An error occured

The strings fields are anonymized by substitution (character by character) 

This covers these types : Byte, Short, Int, Long, Float, Double and BigDecimal.

You can find execution errors of the job in Error Output part of Job Logs.