Document datasets

Image RemovedImage Added

1. Click the Datasets tab.
2. Select the dataset you want to classify.

Image RemovedImage Added

1. Classify the dataset in this window.

...

Add a comment to the dataset

Document personal data

Image RemovedImage Added

To access the personal data panel :

1. Click the Datasets panel in the Data Governance portal
2. Select a dataset
3. Select the Personal data tab in the dataset overview 

Tag

...

dataset containing personal data

Image RemovedImage Added

1. Check "contains personal data" box if the dataset contains personal data

Document the consent and / or configure anonymization process

Image RemovedImage Added

1. Click to Edit Settings to document the consent and / or configure anonymization process

...

1. Select the field specifying entry date
2. Specify the number of days after entry date data will be anonymized

Select fields to be anonymized

Image RemovedImage Added

1. Select fields to be anonymized

...

1. Check "Enable anonymization" box to declare anonymization process
2. Click on "Enable anonymization" button to save settings and anonymization job will launch depending on scheduling

...

Anonymization job

Anonymization job

An default anonymization process in scala is proposed by Saagie. The code source is available here : https://github.com/saagie/outis.

You can use it as is, change it or replace it with one of your process.

To use it, build jar and create Spark processing job on platform with command line : 

spark-submit \
--conf "spark.executor.extraJavaOptions='-Dlog4j.configuration=log4j.xml'" \
--conf spark.ui.showConsoleProgress=false \
--driver-java-options "-Dlog4j.configuration=log4j.xml" \
{file} -u hdfs_user -t metastore_url -d datagov_user -p $ENV_VAR_PASSWORD datasetsToAnonymized_url callback_url

where :

• hdfs_user = user to launch job - user must have right to write in hdfs
• metastore_url = url of the hive metastore (exp : thrift://nn1:9083)
• datagov_user = user to access to Data Governance on the platform with right "Access all datasets" (may be the same as hdfs_user)
• datasetsToAnonymized_url = url to obtain datasets to anonymized (exp : http://{IP_DATAGOVERNANCE}:{PORT}/api/v1/datagovernance/platform/{PLATFORM_ID}/privacy/datasets)
• callback_url = url to inform dataset is anonymized (exp : http://{IP_DATAGOVERNANCE}:{PORT}/api/v1/datagovernance/platform/{PLATFORM_ID}/privacy/events/datasetAnonymized)
• $ENV_VAR_PASSWORD : environment variable for password

You can dowload the last version of the jar here :  

outis-link-1.1.0.jar

Exceptions handling

No dataset anonymization if : 

• You don't provide an entry date
• You don't provide a list of fields to anonymize
• If you provide a string field as an entry date without a pattern to parse the data
• Dataset isn't csv or parquet files on table

No record anonymization if : 

• The value of the field to anonymize has a NULL value
• You try to replace a string field mark as a date without providing a pattern
• An error occured

The strings fields are anonymized by substitution (character by character) 

This covers these types : Byte, Short, Int, Long, Float, Double and BigDecimal.

You can find execution errors of the job in Error Output part of Job Logs.