Big Brother AI: Machine Learning Is Coming For Your Privacy

In order to choose the parameters, machine learning methods use training data with the goal of minimizing the predictive error on the training data. For example, if the goal is to predict whether a person would respond well to a certain medical treatment based on their medical history, the machine learning model would make predictions about the data where the model's developers know whether someone responded well or poorly.

The model is rewarded for predictions that are correct and penalized for incorrect predictions, which leads the algorithm to adjust its parameters – that is, turn some of the“knobs” – and try again.

To avoid overfitting the training data, machine learning models are checked against a validation dataset as well. The validation dataset is a separate dataset that is not used in the training process. By checking the machine learning model's performance on this validation dataset, developers can ensure that the model is able to generalize its learning beyond the training data, avoiding overfitting.

While this process succeeds at ensuring good performance of the machine learning model, it does not directly prevent the machine learning model from memorizing information in the training data.

Because of the large number of parameters in machine learning models, there is a potential that the machine learning method memorizes some data it was trained on . In fact, this is a widespread phenomenon, and users can extract the memorized data from the machine learning model by using queries tailored to get the data .

If the training data contains sensitive information, such as medical or genomic data, then the privacy of the people whose data was used to train the model could be compromised.

Recent research showed that it is actually necessary for machine learning models to memorize aspects of the training data in order to get optimal performance solving certain problems. This indicates that there may be a fundamental trade-off between the performance of a machine-learning method and privacy.

Machine learning models also make it possible to predict sensitive information using seemingly nonsensitive data. For example, Target was able to predict which customers were likely pregnant by analyzing the purchasing habits of customers who registered with the Target baby registry.

Once the model was trained on this dataset, it was able to send pregnancy-related advertisements to customers it suspected were pregnant because they purchased items such as supplements or unscented lotions.

While there have been many proposed methods to reduce memorization in machine learning methods, most have been largely ineffective . Currently, the most promising solution to this problem is to ensure a mathematical limit on the privacy risk.

The state-of-the-art method for formal privacy protection is differential privacy . Differential privacy requires that a machine learning model does not change much if one individual's data is changed in the training dataset.

Differential privacy methods achieve this guarantee by introducing additional randomness into the algorithm learning that“covers up” the contribution of any particular individual. Once a method is protected with differential privacy, no possible attack can violate that privacy guarantee .

Sign up for one of our free newsletters

• The Daily ReportStart your day right with Asia Times' top stories
• AT Weekly ReportA weekly roundup of Asia Times' most-read stories

Even if a machine learning model is trained using differential privacy, however, that does not prevent it from making sensitive inferences such as in the Target example. To prevent these privacy violations, all data transmitted to the organization needs to be protected. This approach is called local differential privacy , and Apple and Google have implemented it.

Differential privacy is a method for protecting people's privacy when their data is included in large datasets.

Because differential privacy limits how much the machine learning model can depend on one individual's data, this prevents memorization. Unfortunately, it also limits the performance of the machine learning methods.

Because of this trade-off, there are critiques on the usefulness of differential privacy, since it often results in a significant drop in performance .

Due to the tension between inferential learning and privacy concerns, there is ultimately a societal question of which is more important in which contexts. When data does not contain sensitive information, it is easy to recommend using the most powerful machine learning methods available.

When working with sensitive data, however, it is important to weigh the consequences of privacy leaks, and it may be necessary to sacrifice some machine learning performance in order to protect the privacy of the people whose data trained the model.

Jordan Awan is Assistant Professor of Statistics, Purdue University

This article is republished from The Conversation under a Creative Commons license. Read the original article .