Linear regression w/ normal equation & gradient descent

notebooks/machine-learning/23.linear-regression-with-gradient-descent.ipynb

+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "5a51118b-c70f-43b9-b8a4-44021fb2547b",
+   "metadata": {},
+   "source": [
+    "# Une droite de régression avec la méthode de descente de gradient"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "ed5517d4-621a-4ec3-8030-cf68f1ae4749",
+   "metadata": {},
+   "source": [
+    "Dernièrement, nous avons calculé une droite de régression avec deux méthodes :\n",
+    "- les moindres carrés ;\n",
+    "- l’équation normale.\n",
+    "\n",
+    "Une autre méthode, adaptable à des problèmes plus complexes que celui qui nous concerne, consiste à effectuer la recherche des meilleurs paramètres par tâtonnement en commençant par des valeurs aléatoires. Les paramètres s’ajustent au fur et à mesure que la fonction de coût, généralement l’erreur quadratique moyenne (MSE), diminue. On appelle cette méthode la descente de gradient."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "888792c9-c1d0-44a4-bb8d-0d893f718874",
+   "metadata": {},
+   "source": [
+    "## Pré-requis"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "2d4b6850-ee1c-4d02-8976-11823c22df8f",
+   "metadata": {},
+   "source": [
+    "Sans rentrer dans les détails, la descente de gradient nécessite trois paramètres :\n",
+    "- des valeurs aléatoires pour initialiser $\\theta$ ($m$ et $b$) ;\n",
+    "- un nombre d’itérations suffisant pour converger ;\n",
+    "- un taux d’apprentissage, noté $\\eta$ (*eta*), qui ne soit ni trop faible, afin d’éviter de ralentir l’entraînement, ni trop élevé, afin d’éviter le sur-entraînement.\n",
+    "\n",
+    "En plus de ces paramètres, la descente de gradient ne peut se calculer que sur des **données standardisées**. Effectuons tout d’abord une copie des données sur le recensement des manchots en Antarctique :"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "e0d61329-50d8-46e2-8fd0-c25ce3cf26a8",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import pandas as pd\n",
+    "import numpy as np\n",
+    "\n",
+    "\n",
+    "# csv file\n",
+    "df = pd.read_csv(\"./files/penguin-census.csv\")\n",
+    "\n",
+    "# data\n",
+    "coords = df.loc[:,[\"body_mass_g\",\"flipper_length_mm\"]]\n",
+    "coords = coords.dropna()\n",
+    "coords = coords.to_numpy()\n",
+    "\n",
+    "# matrices with coords\n",
+    "X = np.c_[coords[:, 0]]\n",
+    "Y = np.c_[coords[:, 1]]"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "b774c6df-6275-4ff3-9578-9cbe72c91bd8",
+   "metadata": {},
+   "source": [
+    "## Standardisation des données"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "caec2d0c-7d6e-4f38-bb27-7014ae6e9996",
+   "metadata": {},
+   "source": [
+    "Avant tout, utilisons la classe `StandardScaler` pour centrer-réduire nos données :"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "726b112d-69ff-497a-9b48-95dadcc43b77",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from sklearn.preprocessing import StandardScaler\n",
+    "\n",
+    "# scaler: Z score normalization\n",
+    "scaler = StandardScaler()\n",
+    "\n",
+    "# scaling\n",
+    "X_scaled = scaler.fit_transform(X)\n",
+    "Y_scaled = scaler.fit_transform(Y)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "d3decd71-3f97-4ee2-8aec-f1427f70bd08",
+   "metadata": {},
+   "source": [
+    "## Insertion d’une dimension neutre"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "74a8b53a-f77e-4bc1-80cd-20b134cef1d3",
+   "metadata": {},
+   "source": [
+    "Comme lors du calcul de la droite de régression avec l’équation normale, et parce que la descente de gradient implique au final un produit matriciel avec une matrice de dimensions $(2, 1)$ il est nécessaire de rajouter une dimension aux coordonnées sur l’axe des abscisses ($X$) :"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "dd00c6eb-5fb4-434b-a8f1-b8297a954c3c",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# x0 = 1\n",
+    "X_scaled = np.c_[\n",
+    "    np.ones(X.shape),\n",
+    "    X_scaled\n",
+    "]"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "d604f6e2-8f5e-49e6-8cb5-95603290e20c",
+   "metadata": {},
+   "source": [
+    "## Paramètres obligatoires"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "28c5d6a1-4b62-4a67-b24e-6e104f8ee5ce",
+   "metadata": {},
+   "source": [
+    "Fournissons à présent des valeurs aux paramètres obligatoires de la descente de gradient :"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "a888aba4-fd8e-4bed-b651-e67f7588885f",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# learning rate\n",
+    "eta = 0.1\n",
+    "\n",
+    "# number of iterations\n",
+    "n_steps = 1000\n",
+    "\n",
+    "# rows in X\n",
+    "m = len(X_scaled)\n",
+    "\n",
+    "# random values to initialize theta\n",
+    "theta = np.random.randn(2, 1)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "2060fa28-d0eb-4c4d-b43e-6180e8dc9fd0",
+   "metadata": {},
+   "source": [
+    "## Résolution de la formule"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "1abc5ee1-e520-47e7-a4f6-bba96a7ef344",
+   "metadata": {},
+   "source": [
+    "Appliquons maintenant la formule :\n",
+    "\n",
+    "$$\\theta = \\theta - \\eta \\nabla_\\theta \\text{MSE}(\\theta)$$"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "7898e6a0-7948-410f-9a04-50d3e29174f8",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# for each step\n",
+    "for step in range(n_steps):\n",
+    "    # the gradients\n",
+    "    gradients = 2/m * X_scaled.T.dot(X_scaled.dot(theta) - Y_scaled)\n",
+    "    # theta\n",
+    "    theta = theta - eta * gradients"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "fe954153-9c56-424f-80b5-8f76faf66a94",
+   "metadata": {},
+   "source": [
+    "Afin de s’assurer que les valeurs de $\\theta$ sont correctes, comparons-les avec la solution obtenue grâce à l’équation normale :"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "aadedff9-0f3e-4f40-bf53-ba5e1ed15dcc",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "theta_norm = np.linalg.inv(X_scaled.T.dot(X_scaled)).dot(X_scaled.T).dot(Y_scaled)\n",
+    "\n",
+    "print(\n",
+    "    theta[0].round(10) == theta_norm[0].round(10),\n",
+    "    theta[1].round(10) == theta_norm[1].round(10),\n",
+    "    sep=\"\\n\"\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "f43554c1-e312-49ea-9756-0fb9dc77890a",
+   "metadata": {},
+   "source": [
+    "Et pour une vérification graphique :"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "3856b9db-64d8-428b-b18e-a51da24b3d80",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import matplotlib.pyplot as plt\n",
+    "import seaborn as sns\n",
+    "\n",
+    "# slope and intercept\n",
+    "m = theta[1]\n",
+    "b = theta[0]\n",
+    "\n",
+    "# points\n",
+    "X = list(map(float, X_scaled[:, 1]))\n",
+    "Y = list(map(float, Y_scaled))\n",
+    "\n",
+    "# predictions\n",
+    "Y_pred = [ float(m * x + b) for x in X ]\n",
+    "\n",
+    "# vizualisation\n",
+    "ax = plt.subplots()\n",
+    "\n",
+    "ax = sns.lineplot(x=X, y=Y_pred, color=\"fuchsia\")\n",
+    "ax = sns.scatterplot(x=X, y=Y, color=\"seagreen\")\n",
+    "\n",
+    "ax.set(xlabel=\"Body mass (g)\", ylabel=\"Flipper length (mm)\")\n",
+    "\n",
+    "sns.despine()\n",
+    "\n",
+    "plt.show()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "b3c65cc8-48c2-4b0a-8c4e-5f3ba2d6b834",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.6"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}