318 lines
14 KiB
Plaintext
318 lines
14 KiB
Plaintext
{
|
||
"cells": [
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "74682f1a",
|
||
"metadata": {},
|
||
"source": [
|
||
"# Done by Aviolat Charline, Bach Joachim and Marino Gabriel"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "ad0d40d6",
|
||
"metadata": {},
|
||
"source": [
|
||
"# Exercice 3 - Review questions"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "3e556a9d",
|
||
"metadata": {},
|
||
"source": [
|
||
"**a) Assuming an univariate input *x*, what is the complexity at inference time of a Bayesian classifier based on histogram computation of the likelihood ?**"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "8d2fb7ef",
|
||
"metadata": {},
|
||
"source": [
|
||
"For each class, we must compute the likelyhood, which is one calculus per class, so O(nb_class). Then, for each x we must compute the posteriori probability, which is looking into a pre-computed histogram (done in the training phase), so this is O(nb_class). The a priori probability only needs to be computed for each class, so O(nb_class). So, the total complexity of the Bayesian classifier is O(2 * nb_class * nb_x)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "99632770",
|
||
"metadata": {},
|
||
"source": [
|
||
"**b) Bayesian models are said to be generative as they can be used to generate new samples. Taking the implementation of the exercise 1.a, explain the steps to generate new samples using the system you have put into place.**\n",
|
||
" "
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "88ab64b2",
|
||
"metadata": {},
|
||
"source": [
|
||
"To generate a new sample, we need to create a y and a x. This can be done by firstly picking the class Ck randomly according to the a priori probabilities P(Ck). This means that, if there is two classes and P(C1) = 0.6 and P(C2) = 0.4, we take 60% of the time C1 and 40% of the time C2\n",
|
||
"\n",
|
||
"Then, we can pick a random x based on the probability density function p(x|Ck). This means we choose a class and, in the density function (like histogram), we take a random x based on the probablilities. If there is two x values possibles and one is distributed as 0.4 and the other 0.6, we will taxe x1 40% of the time and x2 60%"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "e2f611fe",
|
||
"metadata": {},
|
||
"source": [
|
||
"***Optional*: Provide an implementation in a function generateSample(priors, histValues, edgeValues, n)**"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 9,
|
||
"id": "14aba0f7",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"def generateSample(priors, histValues, edgeValues, n):\n",
|
||
" # pick a class according to the proba\n",
|
||
" # to do that, compute the different probabilities sum. This is done by creating intervals between 0 and 1. The size of those intervals represents the probability of the random\n",
|
||
" # number generator to land on it.\n",
|
||
" cumulative_probs = np.cumsum(priors)\n",
|
||
"\n",
|
||
" # take a random number and see in which interval it falls. The index of this interval will be the class we chose\n",
|
||
" chosen_class = 0\n",
|
||
" r = random.random() \n",
|
||
" for i, cp in enumerate(cumulative_probs):\n",
|
||
" if r < cp:\n",
|
||
" chosen_class = i\n",
|
||
" break\n",
|
||
" \n",
|
||
"\n",
|
||
" # The same logic is used to find the new x value. We take the proba of x given c and chose randomly weighted by those proba.\n",
|
||
" # we have to compute the \"probabilities\" differently, because the histogram is only the count of each x in the c.\n",
|
||
" # here, we kept the count instead of proba and when generating the random number, instead of chosing between 0 1 and 1 we chose between 0 and total_hist\n",
|
||
" # which does the same job in the end\n",
|
||
" total_hist = np.sum(histValues[chosen_class])\n",
|
||
"\n",
|
||
" cumulative_probs_hist = np.cumsum(histValues[chosen_class])\n",
|
||
"\n",
|
||
" # take a random number and see in which interval it falls. The index of this interval will be the class we chose\n",
|
||
" chosen_x_index = 0\n",
|
||
" r = random.uniform(0, total_hist) \n",
|
||
" for i, cp in enumerate(cumulative_probs_hist):\n",
|
||
" if r < cp:\n",
|
||
" chosen_x_index = i\n",
|
||
" break\n",
|
||
"\n",
|
||
" chosen_x = edgeValues[chosen_x_index]\n",
|
||
" \n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "ed8c4f6b",
|
||
"metadata": {},
|
||
"source": [
|
||
"**c) What is the minimum overall accuracy of a 2-class system relying only on priors and that is built on a training set that includes 5 times more samples in class A than in class B?**"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "4bb03365",
|
||
"metadata": {},
|
||
"source": [
|
||
"If we only take the priors, then the posterior probability only depends on it. The system will chose the highest posterior probability, so the highest prior because it is all it has. This means it will always choose the class A. If the repartition of the test set is the same as the training set, then always choosing A will give a 5/6 success rate, which will be all the correct A and all the missed B. If the test set is balanced, the success rate will be 50% because it will find all the A and miss all the B. Finally, if the system is unbalanced in the other way, the success rate will only be the portion of the A class in comparaison to the B class. The absolute minimum is then how low the portion of A can be compared to B in the test set."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "58450ff6",
|
||
"metadata": {},
|
||
"source": [
|
||
"**d) Let’s look back at the PW02 exercise 3 of last week. We have built a knn classification systems for images of digits on the MNIST database.**\n",
|
||
"\n",
|
||
"**How would you build a Bayesian classification for the same task ? Comment on the prior probabilities and on the likelihood estimators. More specifically, what kind of likelihood estimator could we use in this case ?**"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "d2bf1500",
|
||
"metadata": {},
|
||
"source": [
|
||
"The a priori probability is simply the repartition of each class in the dataset.\n",
|
||
"The likelihood is the tricky part, because the system would need to be multivariate (because of all the pixels), which makes it very complex. We could use the Naive Bayes formula witch states that the features (pixels) are completely uncorrelated and then we cound perform the operation pixel per pixel for each image. However, the pixels ARE NOT uncorrelated, because a pixel is spatially positionned. If a pixel is white, there is a strong change that there are white pixels somewhere around as well. The Naive Bayes would technically still work-ish, but with a false presomption.\n",
|
||
"\n",
|
||
"To do it correctly, we would have to use something like the multivariate gaussian distribution"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "a3ca9715",
|
||
"metadata": {},
|
||
"source": [
|
||
"***Optional:* implement it and report performance !**"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 10,
|
||
"id": "4de72736",
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Training data shape: (10000, 28, 28)\n",
|
||
"Training labels shape: (10000,)\n",
|
||
"Test data shape: (10000, 28, 28)\n",
|
||
"Test labels shape: (10000,)\n",
|
||
"(10000, 784) (10000, 784)\n",
|
||
"Accuracy score : 0.5711\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"import numpy as np\n",
|
||
"import matplotlib.pyplot as plt\n",
|
||
"import pandas as pd\n",
|
||
"import os\n",
|
||
"\n",
|
||
"\n",
|
||
"# This is a method to read the MNIST dataset from a ROOT directory\n",
|
||
"def load_MNIST(ROOT):\n",
|
||
" '''load all of mnist\n",
|
||
" training set first'''\n",
|
||
" Xtr = []\n",
|
||
" train = pd.read_csv(os.path.join(ROOT, 'mnist_train.csv'))\n",
|
||
" X = np.array(train.drop('label', axis=1))\n",
|
||
" Ytr = np.array(train['label'])\n",
|
||
" # With this for-loop we give the data a shape of the acctual image (28x28)\n",
|
||
" # instead of the shape in file (1x784)\n",
|
||
" for row in X:\n",
|
||
" Xtr.append(row.reshape(28,28))\n",
|
||
" # load test set second\n",
|
||
" Xte = []\n",
|
||
" test = pd.read_csv(os.path.join(ROOT, 'mnist_test.csv'))\n",
|
||
" X = np.array(test.drop('label', axis=1))\n",
|
||
" Yte = np.array(test['label'])\n",
|
||
" # same reshaping\n",
|
||
" for row in X:\n",
|
||
" Xte.append(row.reshape(28,28))\n",
|
||
" \n",
|
||
" return np.array(Xtr), np.array(Ytr), np.array(Xte), np.array(Yte)\n",
|
||
"\n",
|
||
"# Load the raw MNIST data.\n",
|
||
"mnist_dir = '' \n",
|
||
"X_train, y_train, X_test, y_test = load_MNIST(mnist_dir)\n",
|
||
"\n",
|
||
"# As a sanity check, we print out the size of the training and test data.\n",
|
||
"print('Training data shape: ', X_train.shape)\n",
|
||
"print('Training labels shape: ', y_train.shape)\n",
|
||
"print('Test data shape: ', X_test.shape)\n",
|
||
"print('Test labels shape: ', y_test.shape)\n",
|
||
"X_train = np.reshape(X_train, (X_train.shape[0], -1)) \n",
|
||
"X_test = np.reshape(X_test, (X_test.shape[0], -1)) \n",
|
||
"\n",
|
||
"print(X_train.shape, X_test.shape)\n",
|
||
"def predict_gaussian(X_test, mu, sigma2, priors):\n",
|
||
" n_samples = X_test.shape[0]\n",
|
||
" y_pred = np.zeros(n_samples)\n",
|
||
"\n",
|
||
" K, n_pixels = mu.shape\n",
|
||
" \n",
|
||
" for idx in range(n_samples):\n",
|
||
" x = X_test[idx]\n",
|
||
" proba_classes = np.zeros(K)\n",
|
||
"\n",
|
||
" for c in range(K):\n",
|
||
" log_likelihood = -0.5 * np.log(2 * np.pi * sigma2[c]) - ((x - mu[c])**2) / (2 * sigma2[c])\n",
|
||
" proba_classes[c] = np.log(priors[c]) + np.sum(log_likelihood)\n",
|
||
" \n",
|
||
" y_pred[idx] = np.argmax(proba_classes)\n",
|
||
"\n",
|
||
" return y_pred\n",
|
||
"classes = np.unique(y_train)\n",
|
||
"priors = np.array([np.mean(y_train == c) for c in classes])\n",
|
||
"\n",
|
||
"n_pixels = X_train.shape[1]\n",
|
||
"\n",
|
||
"mu = np.zeros((len(classes), n_pixels))\n",
|
||
"sigma2 = np.zeros((len(classes), n_pixels))\n",
|
||
"\n",
|
||
"for c in classes:\n",
|
||
" X_c = X_train[y_train == c]\n",
|
||
" mu[c, :] = X_c.mean(axis=0)\n",
|
||
" sigma2[c, :] = X_c.var(axis=0) + 1e-5\n",
|
||
" \n",
|
||
"y_pred = predict_gaussian(X_test, mu, sigma2, priors)\n",
|
||
"accuracy = np.mean(y_pred == y_test)\n",
|
||
"\n",
|
||
"print(\"Accuracy score :\", accuracy)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "07cb7aee",
|
||
"metadata": {},
|
||
"source": [
|
||
"The .57 accuracy observed here might prove that the method is not the right one for this type of problems, because if each pixel is a feature, the number of dimensions become way to big. This might also be caused by the fact that pixels are not uncorrelated."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "b812b46f",
|
||
"metadata": {},
|
||
"source": [
|
||
"**e) Read [europe-border-control-ai-lie-detector](https://theintercept.com/2019/07/26/europe-border-control-ai-lie-detector/). The described system is \"a virtual policeman designed to strengthen European borders\". It can be seen as a 2-class problem, either you are a suspicious traveler or you are not. If you are declared as suspicious by the system, you are routed to a human border agent who analyses your case in a more careful way.**\n",
|
||
"\n",
|
||
"1. What kind of errors can the system make ? Explain them in your own words.\n",
|
||
"2. Is one error more critical than the other ? Explain why.\n",
|
||
"3. According to the previous points, which metric would you recommend to tune your MLsystem ?"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "1adf1760",
|
||
"metadata": {},
|
||
"source": [
|
||
"1. The system can make false positives or false negatives. This means it could say that an innocent man is a threat or that a dangerous person is safe to cross the border.\n",
|
||
"2. Yes, a false negative is the most critical one. In the case of a false positive, the only consequence is a lost of time because you have to interrogate the \"suspect\", maybe resulting in a angry customer. On the other hand, a false negative means a real threat has entered the country and has not been detect, wich could have way more concequences than an angry customer.\n",
|
||
"3. In this case, we could use the Area Under the Curve with this system. This would allow to tune the the treshold and impact the decision to tend more to false positives rather than false negatives"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "195a1f73-c0f7-4707-9551-c71bfa379960",
|
||
"metadata": {},
|
||
"source": [
|
||
"**f) When a deep learning architecture is trained using an unbalanced training set, we usually observe a problem of bias, i.e. the system favors one class over another one. Using the Bayes equation, explain what is the origin of the problem.**"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "fa5ffd45-0645-4093-9a1b-0a7aeaeece0e",
|
||
"metadata": {},
|
||
"source": [
|
||
"The bayes equation : P(Ck|x) = (p(x|Ck)*P(Ck))/p(x).\n",
|
||
"\n",
|
||
"The a priori probability (P(Ck)) is what reprensents the unbalance in the training set. This value is the probability to have this class, so it is linked to the number of data in it. This means that a class with 3x more data in it will have a way bigger P(Ck) witch will impact the decision in favor of the biggest P(Ck)."
|
||
]
|
||
}
|
||
],
|
||
"metadata": {
|
||
"kernelspec": {
|
||
"display_name": ".venv",
|
||
"language": "python",
|
||
"name": "python3"
|
||
},
|
||
"language_info": {
|
||
"codemirror_mode": {
|
||
"name": "ipython",
|
||
"version": 3
|
||
},
|
||
"file_extension": ".py",
|
||
"mimetype": "text/x-python",
|
||
"name": "python",
|
||
"nbconvert_exporter": "python",
|
||
"pygments_lexer": "ipython3",
|
||
"version": "3.12.3"
|
||
}
|
||
},
|
||
"nbformat": 4,
|
||
"nbformat_minor": 5
|
||
}
|