{
 "cells": [
  {
   "cell_type": "markdown",
   "source": [
    "# CSC 3105 Project"
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "cda961ffb493d00c"
  },
  {
   "cell_type": "markdown",
   "source": [
    "# Data Preprocessing and Analysis"
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "73fe8802e95a784f"
  },
  {
   "cell_type": "code",
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From C:\\Users\\PC\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages\\keras\\src\\losses.py:2976: The name tf.losses.sparse_softmax_cross_entropy is deprecated. Please use tf.compat.v1.losses.sparse_softmax_cross_entropy instead.\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import pywt\n",
    "import tensorflow as tf\n",
    "from skimage import restoration\n",
    "from sklearn.metrics import classification_report\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.preprocessing import LabelEncoder\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from tensorflow.keras import regularizers\n",
    "from tensorflow.keras.callbacks import EarlyStopping\n",
    "from tensorflow.keras.layers import Conv1D, Flatten, Dense, Dropout, BatchNormalization\n",
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.optimizers import Adam\n",
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import Dense, Dropout, BatchNormalization\n",
    "from tensorflow.keras.callbacks import EarlyStopping\n",
    "from tensorflow.keras import regularizers\n",
    "from tensorflow.keras.optimizers import Adam\n",
    "from sklearn.metrics import classification_report, confusion_matrix, roc_curve, auc\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "\n"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2024-03-21T12:05:16.532199Z",
     "start_time": "2024-03-21T12:05:10.492514Z"
    }
   },
   "id": "2aa3c6c09e8645d1",
   "execution_count": 1
  },
  {
   "cell_type": "markdown",
   "source": [
    "# Load the Data\n",
    "\n",
    "This code block defines a function `load_data` to load the dataset from the specified directory. The function uses the `os` and `pandas` libraries to load the data from multiple CSV files into a single DataFrame. The `os` library is used to traverse the directory and find all the CSV files, while the `pandas` library is used to read the CSV files and concatenate them into a single DataFrame. The function then prints the shape of the original data."
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "e7597045d3c34419"
  },
  {
   "cell_type": "code",
   "outputs": [],
   "source": [
    "# Define the directory where the dataset is located\n",
    "DATASET_DIR = './UWB-LOS-NLOS-Data-Set/dataset'\n",
    "\n",
    "\n",
    "def load_data(dataset_dir):\n",
    "    # Load the data\n",
    "    # file_paths = [os.path.join(dirpath, file) for dirpath, _, filenames in os.walk(dataset_dir) for file in filenames if 'uwb_dataset_part7.csv' not in file]\n",
    "    file_paths = [os.path.join(dirpath, file) for dirpath, _, filenames in os.walk(dataset_dir) for file in filenames]\n",
    "    data = pd.concat((pd.read_csv(file_path) for file_path in file_paths))\n",
    "    print(f\"Original data shape: {data.shape}\")\n",
    "    return data\n"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2024-03-21T12:05:35.543184Z",
     "start_time": "2024-03-21T12:05:35.539640Z"
    }
   },
   "id": "7bcd7cfc8dd11cbb",
   "execution_count": 2
  },
  {
   "cell_type": "markdown",
   "source": [
    "# Statistical Analysis and Plots\n",
    "\n",
    "This code block defines a function `stat_analysis_and_plots` to perform statistical analysis and plot boxplots of the first 15 columns of the data. The function uses the `describe` method of the DataFrame to perform statistical analysis and the `boxplot` method of the `matplotlib` library to plot boxplots of the first 15 columns of the data. The boxplot is used to check for outliers in the data."
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "8cdfcf73ad317dd5"
  },
  {
   "cell_type": "code",
   "outputs": [],
   "source": [
    "def stat_analysis_and_plots(data):\n",
    "    # Statistical Analysis\n",
    "    print(\"Statistical Analysis:\")\n",
    "    print(data.describe())\n",
    "\n",
    "    # Plot Boxplot to check for outliers for the first 15 columns\n",
    "    print(\"Boxplot of the first 15 columns:\")\n",
    "    fig, axs = plt.subplots(15, 1, dpi=95, figsize=(7, 17))\n",
    "    for i, col in enumerate(data.columns[:15]):\n",
    "        axs[i].boxplot(data[col], vert=False)\n",
    "        axs[i].set_ylabel(col)\n",
    "    plt.show()\n"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2024-03-21T12:05:39.897831Z",
     "start_time": "2024-03-21T12:05:39.894324Z"
    }
   },
   "id": "9e0b1ed6f23a17cf",
   "execution_count": 3
  },
  {
   "cell_type": "markdown",
   "source": [
    "# Channel Impulse Response (CIR) Graphs\n",
    "\n",
    "This code block defines a function `cir_graphs` to plot the Channel Impulse Response (CIR) for Line of Sight (LOS) and Non-Line of Sight (NLOS) data. The CIR is a sequence of values representing the channel response to a single impulse. It is used to characterize the channel in wireless communication systems."
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "1dd92fe7b6881ea6"
  },
  {
   "cell_type": "code",
   "outputs": [],
   "source": [
    "def cir_graphs(data):\n",
    "    # Separate the data into LOS and NLOS\n",
    "    data_los = data[data['NLOS'] == 0]\n",
    "    data_nlos = data[data['NLOS'] == 1]\n",
    "\n",
    "    # Extract the CIR columns\n",
    "    cir_columns = [col for col in data.columns if 'CIR' in col]\n",
    "    data_los_cir = data_los[cir_columns]\n",
    "    data_nlos_cir = data_nlos[cir_columns]\n",
    "\n",
    "    # Calculate the magnitude and time for each CIR column\n",
    "    time_los = np.arange(len(data_los_cir.columns))\n",
    "    magnitude_los = np.linalg.norm(data_los_cir.values, axis=0)\n",
    "\n",
    "    time_nlos = np.arange(len(data_nlos_cir.columns))\n",
    "    magnitude_nlos = np.linalg.norm(data_nlos_cir.values, axis=0)\n",
    "\n",
    "    # Plot the magnitude vs time for LOS\n",
    "    plt.figure(figsize=(20, 10), dpi=300)  # Increase figure size and DPI\n",
    "    plt.plot(time_los, magnitude_los, linewidth=2)  # Increase line width\n",
    "    plt.title('Magnitude vs Time for LOS')\n",
    "    plt.xlabel('Time (ns)')\n",
    "    plt.ylabel('Magnitude')\n",
    "    plt.xlim([600, max(time_los)])  # Set x-axis limits\n",
    "    # plt.ylim([0, 2e6])\n",
    "    plt.show()\n",
    "\n",
    "    # Plot the magnitude vs time for NLOS\n",
    "    plt.figure(figsize=(20, 10), dpi=300)  # Increase figure size and DPI\n",
    "    plt.plot(time_nlos, magnitude_nlos, linewidth=2)  # Increase line width\n",
    "    plt.title('Magnitude vs Time for NLOS')\n",
    "    plt.xlabel('Time (ns)')\n",
    "    plt.ylabel('Magnitude')\n",
    "    plt.xlim([600, max(time_los)])  # Set x-axis limits\n",
    "    # plt.ylim([0, 2e6])\n",
    "    plt.show()\n"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2024-03-21T12:05:41.658844Z",
     "start_time": "2024-03-21T12:05:41.653601Z"
    }
   },
   "id": "308d64639b199bc7",
   "execution_count": 4
  },
  {
   "cell_type": "markdown",
   "source": [
    "# Calculate Total Distance\n",
    "\n",
    "This code block defines a function `calculate_total_distance` to calculate the total distance for each data point. The total distance is calculated as the sum of the absolute values of the Channel Impulse Response (CIR) columns multiplied by the speed of light in meters per nanosecond.\n",
    "\n",
    "$$\n",
    "\\text{Total Distance} = \\sum_{i=1}^{n} |CIR_i| \\times \\text{speed of light}\n",
    "$$\n",
    "where:\n",
    "- $n$ is the number of CIR columns\n",
    "- $CIR_i$ is the $i$-th CIR column\n",
    "- $\\text{speed of light}$ is the speed of light in meters per nanosecond"
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "531a6ffbad89ba15"
  },
  {
   "cell_type": "code",
   "outputs": [],
   "source": [
    "def calculate_total_distance(data):\n",
    "    # Speed of light in meters per nanosecond\n",
    "    speed_of_light_ns = 0.299792458\n",
    "\n",
    "    # Extract the CIR columns\n",
    "    cir_columns = [col for col in data.columns if 'CIR' in col]\n",
    "\n",
    "    # Calculate the total distance for each row\n",
    "    data['Total_Distance'] = data[cir_columns].abs().sum(axis=1) * speed_of_light_ns\n",
    "\n",
    "    return data"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2024-03-21T12:05:44.069283Z",
     "start_time": "2024-03-21T12:05:44.066364Z"
    }
   },
   "id": "80cfcfac265d9357",
   "execution_count": 5
  },
  {
   "cell_type": "markdown",
   "source": [
    "# Signal to Noise Ratio (SNR) Graph\n",
    "\n",
    "This code block defines a function `snr_graph` to plot the Signal to Noise Ratio (SNR) for Line of Sight (LOS) and Non-Line of Sight (NLOS) data. The SNR is calculated as the ratio of the 'CIR_PWR' to the 'STDEV_NOISE' for each data point."
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "bfd97fbe797a7067"
  },
  {
   "cell_type": "code",
   "outputs": [],
   "source": [
    "\n",
    "def snr_graph(data):\n",
    "    # Separate the data into LOS and NLOS\n",
    "    data_los = data[data['NLOS'] == 0]\n",
    "    data_nlos = data[data['NLOS'] == 1]\n",
    "\n",
    "    # Extract the SNR values\n",
    "    snr_los = data_los['SNR']\n",
    "    snr_nlos = data_nlos['SNR']\n",
    "\n",
    "    # Create a new figure\n",
    "    plt.figure(figsize=(10, 5))\n",
    "\n",
    "    # Plot SNR for LOS\n",
    "    plt.plot(snr_los, label='LOS')\n",
    "\n",
    "    # Plot SNR for NLOS\n",
    "    plt.plot(snr_nlos, color='red', label='NLOS')\n",
    "\n",
    "    # Set title and labels\n",
    "    plt.title('SNR for LOS and NLOS')\n",
    "    plt.xlabel('Index')\n",
    "    plt.ylabel('SNR')\n",
    "\n",
    "    # Add a legend\n",
    "    plt.legend()\n",
    "\n",
    "    # Show the plot\n",
    "    plt.show()\n"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2024-03-21T12:05:46.175335Z",
     "start_time": "2024-03-21T12:05:46.170401Z"
    }
   },
   "id": "4afc8d71b3271351",
   "execution_count": 6
  },
  {
   "cell_type": "markdown",
   "source": [
    "# Plot Histogram\n",
    "\n",
    "This code block defines a function `plot_histogram` to plot a histogram of a given feature in the data for Line of Sight (LOS) and Non-Line of Sight (NLOS) data."
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "74cbef5687382eea"
  },
  {
   "cell_type": "code",
   "outputs": [],
   "source": [
    "\n",
    "\n",
    "def plot_histogram(data, feature):\n",
    "    \"\"\"\n",
    "    Function to plot a histogram of a given feature in the data for 'NLOS' and 'LOS'.\n",
    "\n",
    "    Parameters:\n",
    "    data (pd.DataFrame): The data.\n",
    "    feature (str): The name of the feature to plot.\n",
    "\n",
    "    Returns:\n",
    "    None\n",
    "    \"\"\"\n",
    "    # Check if the feature exists in the data\n",
    "    if feature not in data.columns:\n",
    "        print(f\"The feature '{feature}' does not exist in the data.\")\n",
    "        return\n",
    "\n",
    "    # Separate the data into 'NLOS' and 'LOS'\n",
    "    data_nlos = data[data['NLOS'] == 1]\n",
    "    data_los = data[data['NLOS'] == 0]\n",
    "\n",
    "    # Create a figure with two subplots side by side\n",
    "    fig, axs = plt.subplots(1, 2, figsize=(10, 5))\n",
    "\n",
    "    # Plot the histogram for 'NLOS'\n",
    "    axs[0].hist(data_nlos[feature], bins=30, edgecolor='black')\n",
    "    axs[0].set_title(f'Histogram of {feature} for NLOS')\n",
    "    axs[0].set_xlabel(feature)\n",
    "    axs[0].set_ylabel('Frequency')\n",
    "\n",
    "    # Plot the histogram for 'LOS'\n",
    "    axs[1].hist(data_los[feature], bins=30, edgecolor='black')\n",
    "    axs[1].set_title(f'Histogram of {feature} for LOS')\n",
    "    axs[1].set_xlabel(feature)\n",
    "    axs[1].set_ylabel('Frequency')\n",
    "\n",
    "    # Display the plots\n",
    "    plt.tight_layout()\n",
    "    plt.show()\n",
    "\n",
    "# Usage:\n",
    "# plot_histogram(data, 'First_Path_Power_Level')"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2024-03-21T12:05:48.433880Z",
     "start_time": "2024-03-21T12:05:48.427111Z"
    }
   },
   "id": "22025d6c8281fc09",
   "execution_count": 7
  },
  {
   "cell_type": "code",
   "outputs": [],
   "source": [
    "def plot_features(data, labels, feature1, feature2):\n",
    "    reds = labels == 1\n",
    "    blacks = labels == 0\n",
    "    plt.scatter(data[reds][feature1], data[reds][feature2], c=\"red\", s=20, edgecolor='k')\n",
    "    plt.scatter(data[blacks][feature1], data[blacks][feature2], c=\"yellow\", s=20, edgecolor='k')\n",
    "    plt.xlabel(feature1)\n",
    "    plt.ylabel(feature2)\n",
    "    plt.title(f\"Plot of data: {feature1} versus {feature2}\")\n",
    "    plt.show()"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2024-03-21T12:05:50.542095Z",
     "start_time": "2024-03-21T12:05:50.538349Z"
    }
   },
   "id": "ac4db13fed3f9916",
   "execution_count": 8
  },
  {
   "cell_type": "markdown",
   "source": [
    "## denoise_cir Function\n",
    "\n",
    "The `denoise_cir` function uses the Discrete Wavelet Transform (DWT) to denoise the Channel Impulse Response (CIR) values. The DWT is a linear transformation that operates on a data vector whose length is an integer power of two, transforming it into a numerically different vector of the same length. The DWT of a signal `x` is calculated as follows:\n",
    "\n",
    "1. **Wavelet Decomposition:**\n",
    "\n",
    "    The input signal `x` is passed through two complementary filters and emerges as two signals. The filter outputs are decimated by 2 (down-sampled) to get the approximation coefficients (cA) and detail coefficients (cD).\n",
    "\n",
    "    The approximation coefficients represent the high-scale, low-frequency component of the signal, while the detail coefficients represent the low-scale, high-frequency component.\n",
    "\n",
    "2. **Thresholding:**\n",
    "\n",
    "    The detail coefficients are thresholded to remove noise. The thresholding function `T` applied to the detail coefficients `x` is defined as:\n",
    "$$\n",
    "T(x) = x * I(|x| > \\text{{value}}) \\quad \\text{{for 'hard' thresholding}}\n",
    "$$\n",
    "\n",
    "$$\n",
    "T(x) = \\text{{sign}}(x)(|x| - \\text{{value}})_+ \\quad \\text{{for 'soft' thresholding}}\n",
    "$$\n",
    "\n",
    "where $I$ is the indicator function that is one if the argument is true and zero otherwise, $\\text{{value}}$ is the threshold value, and $(x)_+$ equals $x$ if $x > 0$ and zero otherwise.\n",
    "\n",
    "3. **Wavelet Reconstruction:**\n",
    "\n",
    "    The original signal is reconstructed from the approximation and detail coefficients."
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "69413268ac5b549d"
  },
  {
   "cell_type": "code",
   "outputs": [],
   "source": [
    "def denoise_cir(cir_values, wavelet='db1', level=1):\n",
    "    # Perform wavelet decomposition\n",
    "    coeffs = pywt.wavedec(cir_values, wavelet, level=level)\n",
    "\n",
    "    # Set the detail coefficients to zero\n",
    "    for i in range(1, len(coeffs)):\n",
    "        coeffs[i] = pywt.threshold(coeffs[i], value=0.5, mode='soft')\n",
    "\n",
    "    # Perform wavelet reconstruction\n",
    "    denoised_cir = pywt.waverec(coeffs, wavelet)\n",
    "\n",
    "    return denoised_cir\n"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2024-03-21T12:05:52.476346Z",
     "start_time": "2024-03-21T12:05:52.472568Z"
    }
   },
   "id": "fe3089568e99a58d",
   "execution_count": 9
  },
  {
   "cell_type": "markdown",
   "source": [
    "## deconvolve_cir Function\n",
    "\n",
    "The `deconvolve_cir` function applies the Richardson-Lucy deconvolution algorithm to deconvolve the Channel Impulse Response (CIR) values. \n",
    "\n",
    "In the context of signal processing, deconvolution is the process of reversing the effects of convolution on a signal. Convolution is a mathematical operation that blends two functions together and is often used to describe the effect of a linear time-invariant system on a signal. Deconvolution, therefore, attempts to recover the original signal that was convolved with the system's impulse response to produce the current signal.\n",
    "\n",
    "The Richardson-Lucy algorithm is an iterative method for deconvolution. It is particularly suitable for cases where the impulse response of the system (also known as the Point Spread Function, or PSF) is known, and the noise is Poissonian (such as in astronomical images). \n",
    "\n",
    "The algorithm works by iteratively refining an estimate of the original signal. In each iteration, it performs a prediction step where it convolves the current estimate with the PSF to predict the observed signal, and a correction step where it computes the ratio of the observed signal to the predicted signal, convolves this ratio with the PSF, and multiplies the result with the current estimate to get the next estimate.\n",
    "\n",
    "This process is repeated for a fixed number of iterations, or until the estimate converges to a stable solution. The result is a denoised estimate of the original signal.\n",
    "\n",
    "The mathematical formulas involved in the Richardson-Lucy deconvolution algorithm are as follows:\n",
    "\n",
    "1. **Prediction Step:**\n",
    "\n",
    "    The current estimate of the latent image `x` is convolved with the PSF `h` to predict the observed image `y`. This can be represented as:\n",
    "\n",
    "$$\n",
    "y = h \\ast x\n",
    "$$\n",
    "\n",
    "where $\\ast$ denotes the convolution operation.\n",
    "\n",
    "2. **Correction Step:**\n",
    "\n",
    "The ratio of the observed image $y$ to the predicted image $y'$ is computed, then the PSF $h$ is convolved with this ratio and multiplied with the current estimate $x$ to get the next estimate $x'$. This can be represented as:\n",
    "\n",
    "$$\n",
    "x' = x \\cdot (h \\ast \\left(\\frac{y}{y'}\\right))\n",
    "$$\n",
    "\n",
    "where $\\div$ denotes element-wise division and $\\ast$ denotes the convolution operation.\n"
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "e1edd5ef4f54e752"
  },
  {
   "cell_type": "code",
   "outputs": [],
   "source": [
    "\n",
    "def deconvolve_cir(cir_values, psf=None, iterations=50):\n",
    "    # If no point spread function is provided, create a simple one\n",
    "    if psf is None:\n",
    "        psf = np.ones((5,)) / 5\n",
    "\n",
    "    # Perform Richardson-Lucy deconvolution\n",
    "    deconvolved_cir = restoration.richardson_lucy(cir_values, psf, num_iter=iterations)\n",
    "\n",
    "    return deconvolved_cir"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2024-03-21T12:05:54.971831Z",
     "start_time": "2024-03-21T12:05:54.968502Z"
    }
   },
   "id": "670e8c2cf19126ea",
   "execution_count": 10
  },
  {
   "cell_type": "markdown",
   "source": [
    "# Clean the Data\n",
    "\n",
    "This code block defines a function `clean_data` to clean the dataset. The function performs the following steps:\n",
    "\n",
    "1. **Drop Missing Values**: It drops any rows with missing values in the dataset.\n",
    "2. **Drop Duplicate Rows**: It drops any duplicate rows in the dataset.\n",
    "3. **Convert 'NLOS' Column to Integer**: It converts the 'NLOS' column to integer data type (0 for Line of Sight (LOS), 1 for Non-Line of Sight (NLOS)).\n",
    "4. **Calculate 'RX_Level'**: It calculates the 'RX_Level' feature using the formula.\n",
    "5. **Replace Zero Values in 'RX_Level'**: It replaces zero values in 'RX_Level' with the median of 'RX_Level'.\n",
    "6. **Calculate 'First_Path_Power_Level'**: It calculates the 'First_Path_Power_Level' feature using the formula.\n",
    "7. **Calculate 'SNR'**: It calculates the 'SNR' feature as the ratio of 'CIR_PWR' to 'STDEV_NOISE' for each data point.\n",
    "8. **One-Hot Encode Categorical Features**: It one-hot encodes the categorical features 'CH', 'FRAME_LEN', 'PREAM_LEN', and 'BITRATE'.\n",
    "9. **Denoise 'CIR' Columns**: It denoises the 'CIR' columns using the `denoise_cir` function.\n",
    "10. **Drop Columns with Only One Unique Value**: It drops any columns that have only one unique value.\n",
    "11. **Print the Shape of the Cleaned Data**: It prints the shape of the cleaned data.\n",
    "12. **Return the Cleaned Data**: It returns the cleaned data."
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "b61e454327e5acbf"
  },
  {
   "cell_type": "code",
   "outputs": [],
   "source": [
    "def clean_data(data):\n",
    "    print(\"Starting data cleaning process...\")\n",
    "\n",
    "    # print(\"Before Cleaning\")\n",
    "    # stat_analysis_and_plots(data)\n",
    "\n",
    "    # Calculate total number of missing values in the data\n",
    "    total_missing = data.isnull().sum().sum()\n",
    "    print(f\"Total number of missing values: {total_missing}\")\n",
    "\n",
    "    # Data has no missing values\n",
    "    data = data.dropna()\n",
    "    print(\"Missing values dropped.\")\n",
    "\n",
    "    # Data has no duplicate rows\n",
    "    data = data.drop_duplicates()\n",
    "    print(\"Duplicate rows dropped.\")\n",
    "\n",
    "    # Convert 'NLOS' column to integer data type (0 for LOS, 1 for NLOS)\n",
    "    data['NLOS'] = data['NLOS'].astype(int)\n",
    "    print(\"'NLOS' column converted to integer data type.\")\n",
    "\n",
    "    # Print line where CIR_PWR is 0\n",
    "    print(f\"Line where CIR_PWR is 0: {data[data['CIR_PWR'] == 0]}\")\n",
    "\n",
    "    # Calculate the expression inside the log10 function\n",
    "    expression = (data['CIR_PWR'] * (2 ** 17)) / (data['RXPACC'] ** 2)\n",
    "\n",
    "    # If the expression is 0, set 'RX_Level' to 0\n",
    "    zero_indices = expression == 0\n",
    "    data.loc[zero_indices, 'RX_Level'] = 0\n",
    "\n",
    "    # For the rest of the data where the expression is not 0, calculate 'RX_Level'\n",
    "    # First, update the 'expression' and 'data' to exclude zero_indices\n",
    "    expression = expression.loc[~zero_indices]\n",
    "    data = data.loc[~zero_indices]\n",
    "\n",
    "    # Now, calculate 'RX_Level' for the rest of the data\n",
    "    data['RX_Level'] = 10 * np.log10(expression) - data['PRFR']\n",
    "\n",
    "    # Calculate the median of 'RX_Level'\n",
    "    median = data['RX_Level'].median()\n",
    "\n",
    "    # Create the boolean mask on the same DataFrame 'data'\n",
    "    zero_indices = (data['RX_Level'] == 0)\n",
    "\n",
    "    # Replace zero values in 'RX_Level' with the median\n",
    "    data.loc[zero_indices, 'RX_Level'] = median\n",
    "\n",
    "    print(\"New feature 'RX_Level' created.\")\n",
    "\n",
    "    # Calculate new feature 'First_Path_Power_Level'\n",
    "    data['First_Path_Power_Level'] = (10 * np.log10(\n",
    "        (data['FP_AMP1'] ** 2 + data['FP_AMP2'] ** 2 + data['FP_AMP3'] ** 2) / (data['RXPACC'] ** 2))) - 64\n",
    "    print(\"New feature 'First_Path_Power_Level' calculated.\")\n",
    "    data.drop(['FP_AMP1', 'FP_AMP2', 'FP_AMP3', 'RXPACC', 'PRFR'], axis=1, inplace=True)\n",
    "\n",
    "    # Calculate SNR as the ratio of 'CIR_PWR' to 'STDEV_NOISE' for each data point\n",
    "    data['SNR'] = data['CIR_PWR'] / data['STDEV_NOISE']\n",
    "    print(\"New feature 'SNR' created.\")\n",
    "    data.drop(['CIR_PWR', 'STDEV_NOISE'], axis=1, inplace=True)\n",
    "\n",
    "    plot_histogram(data, 'First_Path_Power_Level')\n",
    "    plot_histogram(data, 'RX_Level')\n",
    "\n",
    "    # One-hot encode categorical features\n",
    "    categorical_features = ['CH', 'FRAME_LEN', 'PREAM_LEN', 'BITRATE']\n",
    "    encoder = LabelEncoder()\n",
    "    for feature in categorical_features:\n",
    "        data[feature] = encoder.fit_transform(data[feature])\n",
    "    print(\"Categorical features one-hot encoded.\")\n",
    "\n",
    "    # Extract the 'CIR' columns\n",
    "    cir_columns = [col for col in data.columns if 'CIR' in col]\n",
    "    cir_data = data[cir_columns]\n",
    "    print(\"'CIR' columns extracted.\")\n",
    "\n",
    "    # Convert 'CIR' columns to float\n",
    "    cir_data = cir_data.astype(float)\n",
    "    print(\"'CIR' columns converted to float.\")\n",
    "\n",
    "    # Denoise 'CIR' columns\n",
    "    denoised_cir_data = cir_data.apply(denoise_cir)\n",
    "    # denoised_cir_data = cir_data.apply(deconvolve_cir)\n",
    "    print(\"'CIR' columns denoised.\")\n",
    "\n",
    "    # Replace original 'CIR' columns with denoised data\n",
    "    data[cir_columns] = denoised_cir_data\n",
    "    print(\"Original 'CIR' columns replaced with denoised data.\")\n",
    "\n",
    "    # List of columns to check for unique values\n",
    "    columns_to_check = ['CH', 'PREAM_LEN', 'BITRATE']\n",
    "\n",
    "    # Iterate over the columns\n",
    "    for column in columns_to_check:\n",
    "        # If the column has only one unique value, drop it\n",
    "        if data[column].nunique() == 1:\n",
    "            data = data.drop(column, axis=1)\n",
    "            print(f\"Column '{column}' dropped due to having only one unique value.\")\n",
    "\n",
    "    # Print the shape of the cleaned data\n",
    "    print(f\"Cleaned data shape: {data.shape}\")\n",
    "\n",
    "    # print(\"After Cleaning\")\n",
    "    # stat_analysis_and_plots(data)\n",
    "\n",
    "    print(\"Data cleaning process completed.\")\n",
    "\n",
    "    # Return the cleaned data\n",
    "    return data"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2024-03-21T12:05:59.248952Z",
     "start_time": "2024-03-21T12:05:59.241423Z"
    }
   },
   "id": "685463c2d6065b08",
   "execution_count": 11
  },
  {
   "cell_type": "markdown",
   "source": [
    "# Load and Clean the Data\n",
    "\n",
    "This code block loads the dataset from the specified directory and cleans the data using the `load_data` and `clean_data` functions. It then prints the first few rows of the cleaned data and the column headers."
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "8323170f6733d41d"
  },
  {
   "cell_type": "code",
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Pickle file not found. Loading and cleaning data...\n",
      "Original data shape: (42000, 1031)\n"
     ]
    },
    {
     "data": {
      "text/plain": "<Figure size 6000x3000 with 1 Axes>",
      "image/png": "[encoded data, 273592 characters]"
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": "<Figure size 6000x3000 with 1 Axes>",
      "image/png": "[encoded data, 287096 characters]"
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Starting data cleaning process...\n",
      "Total number of missing values: 0\n",
      "Missing values dropped.\n",
      "Duplicate rows dropped.\n",
      "'NLOS' column converted to integer data type.\n",
      "Line where CIR_PWR is 0:       NLOS  RANGE  FP_IDX  FP_AMP1  FP_AMP2  FP_AMP3  STDEV_NOISE  CIR_PWR  \\\n",
      "4343     1   7.02   757.0     30.0    214.0    413.0         36.0      0.0   \n",
      "837      1   4.88   739.0    112.0    323.0    227.0         40.0      0.0   \n",
      "1356     1   6.33   747.0    293.0    311.0    187.0         28.0      0.0   \n",
      "\n",
      "      MAX_NOISE  RXPACC  ...  CIR1006  CIR1007  CIR1008  CIR1009  CIR1010  \\\n",
      "4343      412.0   192.0  ...    252.0    271.0    190.0    292.0    271.0   \n",
      "837       322.0   128.0  ...    161.0    219.0    295.0    242.0    279.0   \n",
      "1356      310.0   160.0  ...    197.0     84.0    246.0    353.0    196.0   \n",
      "\n",
      "      CIR1011  CIR1012  CIR1013  CIR1014  CIR1015  \n",
      "4343    239.0    210.0    260.0    223.0    256.0  \n",
      "837      67.0    153.0    177.0    159.0      0.0  \n",
      "1356     38.0    228.0     42.0    173.0      0.0  \n",
      "\n",
      "[3 rows x 1031 columns]\n",
      "New feature 'RX_Level' created.\n",
      "New feature 'First_Path_Power_Level' calculated.\n",
      "New feature 'SNR' created.\n"
     ]
    },
    {
     "data": {
      "text/plain": "<Figure size 1000x500 with 2 Axes>",
      "image/png": "[encoded data, 41660 characters]"
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": "<Figure size 1000x500 with 2 Axes>",
      "image/png": "[encoded data, 37692 characters]"
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Categorical features one-hot encoded.\n",
      "'CIR' columns extracted.\n",
      "'CIR' columns converted to float.\n",
      "'CIR' columns denoised.\n",
      "Original 'CIR' columns replaced with denoised data.\n",
      "Column 'CH' dropped due to having only one unique value.\n",
      "Column 'BITRATE' dropped due to having only one unique value.\n",
      "Cleaned data shape: (41997, 1025)\n",
      "Data cleaning process completed.\n"
     ]
    },
    {
     "data": {
      "text/plain": "<Figure size 640x480 with 1 Axes>",
      "image/png": "[encoded data, 139776 characters]"
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": "<Figure size 1000x500 with 1 Axes>",
      "image/png": "[encoded data, 67724 characters]"
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": "<Figure size 6000x3000 with 1 Axes>",
      "image/png": "[encoded data, 280144 characters]"
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": "<Figure size 6000x3000 with 1 Axes>",
      "image/png": "[encoded data, 282388 characters]"
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\PC\\AppData\\Local\\Temp\\ipykernel_117340\\2998122009.py:9: PerformanceWarning: DataFrame is highly fragmented.  This is usually the result of calling `frame.insert` many times, which has poor performance.  Consider joining all columns at once using pd.concat(axis=1) instead. To get a de-fragmented frame, use `newframe = frame.copy()`\n",
      "  data['Total_Distance'] = data[cir_columns].abs().sum(axis=1) * speed_of_light_ns\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "      NLOS  RANGE  FP_IDX  MAX_NOISE  FRAME_LEN  PREAM_LEN        CIR0  \\\n",
      "0        0   3.90   745.0      967.0          2          0  128.646447   \n",
      "1        0   0.66   749.0     1133.0          0          0  117.353553   \n",
      "2        1   7.86   746.0      894.0          0          0  433.353553   \n",
      "3        1   3.48   750.0     1127.0          2          0  465.646447   \n",
      "4        0   1.19   746.0     1744.0          0          0  259.353553   \n",
      "...    ...    ...     ...        ...        ...        ...         ...   \n",
      "5995     0   2.43   745.0     2379.0          0          0  395.646447   \n",
      "5996     0   1.39   745.0     1810.0          2          0  183.353553   \n",
      "5997     1   5.48   747.0     1092.0          2          0  202.646447   \n",
      "5998     1   3.40   745.0     1077.0          0          0  484.353553   \n",
      "5999     0   2.43   747.0     1224.0          0          0  747.646447   \n",
      "\n",
      "            CIR1        CIR2        CIR3  ...     CIR1010     CIR1011  \\\n",
      "0     311.646447  330.646447  140.646447  ...  197.646447   87.353553   \n",
      "1     163.353553  199.353553  136.353553  ...  187.353553  212.646447   \n",
      "2     240.353553  233.353553   81.353553  ...  124.353553  329.353553   \n",
      "3     246.646447  406.646447  224.646447  ...  433.646447  396.646447   \n",
      "4     239.353553   57.353553   19.353553  ...   87.353553  178.353553   \n",
      "...          ...         ...         ...  ...         ...         ...   \n",
      "5995  277.646447  279.646447  278.353553  ...  193.353553  417.646447   \n",
      "5996  304.646447  247.646447  162.353553  ...  211.353553  144.353553   \n",
      "5997  242.353553  228.353553  188.646447  ...  253.646447  191.646447   \n",
      "5998  210.353553   75.353553  246.353553  ...   86.353553  123.353553   \n",
      "5999  366.646447  744.646447  717.646447  ...  726.646447  366.646447   \n",
      "\n",
      "         CIR1012     CIR1013     CIR1014     CIR1015   RX_Level  \\\n",
      "0     295.646447  504.646447  306.646447    0.000000 -27.806709   \n",
      "1     202.353553   89.353553  103.353553    0.000000 -23.050836   \n",
      "2     207.353553   96.353553  218.353553    0.353553 -28.334789   \n",
      "3     289.646447  154.646447  341.646447  255.646447 -33.611812   \n",
      "4     313.646447  247.353553  292.353553  256.000000 -21.081660   \n",
      "...          ...         ...         ...         ...        ...   \n",
      "5995  242.646447  225.646447  256.353553    0.000000 -21.648732   \n",
      "5996   47.353553  110.353553   82.353553    0.000000 -20.809477   \n",
      "5997  180.646447  445.646447  299.646447    0.000000 -41.462381   \n",
      "5998  522.353553  295.353553  223.353553  256.353553 -41.296123   \n",
      "5999  802.646447  818.646447  466.646447  767.646447 -23.907380   \n",
      "\n",
      "      First_Path_Power_Level         SNR  Total_Distance  \n",
      "0                 -32.018158  185.234375   120371.292417  \n",
      "1                 -34.255986  296.375000   135265.933614  \n",
      "2                 -43.509715  244.983333   153365.885081  \n",
      "3                 -41.706556  115.105263   125802.050979  \n",
      "4                 -23.834943  167.352941   110697.999893  \n",
      "...                      ...         ...             ...  \n",
      "5995              -24.337520  104.031250   110297.192886  \n",
      "5996              -28.059910  165.250000   106982.743632  \n",
      "5997              -43.758684   19.930556   122190.602961  \n",
      "5998              -42.910043   20.708333   132871.634243  \n",
      "5999              -32.863387  162.785714   234026.967141  \n",
      "\n",
      "[41997 rows x 1026 columns]\n",
      "Data loaded and cleaned successfully.\n",
      "Saving cleaned data to pickle file...\n",
      "Cleaned data saved to pickle file successfully.\n",
      "First few rows of the data:\n",
      "   NLOS  RANGE  FP_IDX  MAX_NOISE  FRAME_LEN  PREAM_LEN        CIR0  \\\n",
      "0     0   3.90   745.0      967.0          2          0  128.646447   \n",
      "1     0   0.66   749.0     1133.0          0          0  117.353553   \n",
      "2     1   7.86   746.0      894.0          0          0  433.353553   \n",
      "3     1   3.48   750.0     1127.0          2          0  465.646447   \n",
      "4     0   1.19   746.0     1744.0          0          0  259.353553   \n",
      "\n",
      "         CIR1        CIR2        CIR3  ...     CIR1010     CIR1011  \\\n",
      "0  311.646447  330.646447  140.646447  ...  197.646447   87.353553   \n",
      "1  163.353553  199.353553  136.353553  ...  187.353553  212.646447   \n",
      "2  240.353553  233.353553   81.353553  ...  124.353553  329.353553   \n",
      "3  246.646447  406.646447  224.646447  ...  433.646447  396.646447   \n",
      "4  239.353553   57.353553   19.353553  ...   87.353553  178.353553   \n",
      "\n",
      "      CIR1012     CIR1013     CIR1014     CIR1015   RX_Level  \\\n",
      "0  295.646447  504.646447  306.646447    0.000000 -27.806709   \n",
      "1  202.353553   89.353553  103.353553    0.000000 -23.050836   \n",
      "2  207.353553   96.353553  218.353553    0.353553 -28.334789   \n",
      "3  289.646447  154.646447  341.646447  255.646447 -33.611812   \n",
      "4  313.646447  247.353553  292.353553  256.000000 -21.081660   \n",
      "\n",
      "   First_Path_Power_Level         SNR  Total_Distance  \n",
      "0              -32.018158  185.234375   120371.292417  \n",
      "1              -34.255986  296.375000   135265.933614  \n",
      "2              -43.509715  244.983333   153365.885081  \n",
      "3              -41.706556  115.105263   125802.050979  \n",
      "4              -23.834943  167.352941   110697.999893  \n",
      "\n",
      "[5 rows x 1026 columns]\n",
      "Column headers:\n",
      "Index(['NLOS', 'RANGE', 'FP_IDX', 'MAX_NOISE', 'FRAME_LEN', 'PREAM_LEN',\n",
      "       'CIR0', 'CIR1', 'CIR2', 'CIR3',\n",
      "       ...\n",
      "       'CIR1010', 'CIR1011', 'CIR1012', 'CIR1013', 'CIR1014', 'CIR1015',\n",
      "       'RX_Level', 'First_Path_Power_Level', 'SNR', 'Total_Distance'],\n",
      "      dtype='object', length=1026)\n"
     ]
    }
   ],
   "source": [
    "import pickle\n",
    "\n",
    "# File='data_original.pkl'\n",
    "File = 'data.pkl'\n",
    "\n",
    "# Check if the file exists\n",
    "if os.path.exists(File):\n",
    "    # If the file exists, load it\n",
    "    print(\"Loading data from pickle file...\")\n",
    "    with open(File, 'rb') as f:\n",
    "        data = pickle.load(f)\n",
    "        # plot_features(data, data['NLOS'], \"First_Path_Power_Level\", \"RX_Level\")\n",
    "        # plot_features(data, data['NLOS'], \"SNR\", \"RX_Level\")\n",
    "        # plot_features(data, data['NLOS'], \"SNR\", \"First_Path_Power_Level\")\n",
    "        snr_graph(data)\n",
    "        cir_graphs(data)\n",
    "    print(\"Data loaded successfully.\")\n",
    "else:\n",
    "    # If the file doesn't exist, load and clean the data\n",
    "    print(\"Pickle file not found. Loading and cleaning data...\")\n",
    "    data = load_data(DATASET_DIR)\n",
    "    cir_graphs(data)\n",
    "    data = clean_data(data)\n",
    "    plot_features(data, data['NLOS'], \"First_Path_Power_Level\", \"RX_Level\")\n",
    "    snr_graph(data)\n",
    "    cir_graphs(data)\n",
    "    print(calculate_total_distance(data))\n",
    "    print(\"Data loaded and cleaned successfully.\")\n",
    "    print(\"Saving cleaned data to pickle file...\")\n",
    "    with open(File, 'wb') as f:\n",
    "        pickle.dump(data, f)\n",
    "    print(\"Cleaned data saved to pickle file successfully.\")\n",
    "\n",
    "print(\"First few rows of the data:\")\n",
    "print(data.head())\n",
    "\n",
    "# Print Headers\n",
    "print(\"Column headers:\")\n",
    "print(data.columns)"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2024-03-21T12:06:16.836565Z",
     "start_time": "2024-03-21T12:06:02.934935Z"
    }
   },
   "id": "79c2c23691b26753",
   "execution_count": 12
  },
  {
   "cell_type": "markdown",
   "source": [
    "# Data Preprocessing"
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "20d838f85d847ea0"
  },
  {
   "cell_type": "markdown",
   "source": [
    "# Train and Save Model\n",
    "\n",
    "This code block defines a function `train_and_save_model` to train a classifier on the training data and save the trained model to a file. The function takes the following parameters:\n",
    "\n",
    "- `classifier`: The classifier to train and save.\n",
    "- `X_train`: The training data.\n",
    "- `y_train`: The training labels.\n",
    "- `file_name`: The name of the file to save the trained model to."
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "3e7a8932c36118eb"
  },
  {
   "cell_type": "code",
   "outputs": [],
   "source": [
    "MODEL_DIR = './models'\n",
    "\n",
    "\n",
    "def train_and_save_model(classifier, X_train, y_train, file_name):\n",
    "    if not os.path.exists(MODEL_DIR):\n",
    "        os.makedirs(MODEL_DIR)\n",
    "\n",
    "    file_path = os.path.join(MODEL_DIR, file_name)\n",
    "\n",
    "    # Check if the file exists\n",
    "    if not os.path.exists(file_path):\n",
    "        print(f\"Training the model and saving it to {file_path}\")\n",
    "        # Train the classifier\n",
    "        classifier.fit(X_train, y_train)\n",
    "\n",
    "        # Save the trained model as a pickle string.\n",
    "        saved_model = pickle.dumps(classifier)\n",
    "\n",
    "        # Save the pickled model to a file\n",
    "        with open(file_path, 'wb') as file:\n",
    "            file.write(saved_model)\n",
    "\n",
    "    # Load the pickled model from the file\n",
    "    with open(file_path, 'rb') as file:\n",
    "        loaded_model = pickle.load(file)\n",
    "\n",
    "    return loaded_model"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2024-03-20T12:32:22.766983Z",
     "start_time": "2024-03-20T12:32:22.762778Z"
    }
   },
   "id": "12e16974341e6266",
   "execution_count": 127
  },
  {
   "cell_type": "markdown",
   "source": [
    "# Split the Data\n",
    "\n",
    "This code block splits the data into training and testing sets using the `train_test_split` function from the `sklearn.model_selection` module. The training set contains 80% of the data, and the testing set contains 20% of the data.\n",
    "\n",
    "The target column 'NLOS' is separated from the rest of the dataset, and the input data is scaled using the `StandardScaler` from the `sklearn.preprocessing` module. The `StandardScaler` standardizes features by removing the mean and scaling to unit variance.\n",
    "\n",
    "The random seed is set to 42 for reproducibility."
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "686f0615a70388cb"
  },
  {
   "cell_type": "code",
   "outputs": [],
   "source": [
    "# Set random seed for reproducibility\n",
    "tf.random.set_seed(42)\n",
    "\n",
    "# Drop the target column 'NLOS' from the data and assign the remaining data to X\n",
    "X = data.drop('NLOS', axis=1)\n",
    "# Assign the target column 'NLOS' to y\n",
    "y = data['NLOS']\n",
    "\n",
    "# Split the data into training and testing sets with a 80:20 ratio\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)\n",
    "\n",
    "# Initialize a StandardScaler object\n",
    "scaler = StandardScaler()\n",
    "# Fit the scaler to the training data and transform it\n",
    "X_train = scaler.fit_transform(X_train)\n",
    "# Transform the testing data using the fitted scaler\n",
    "X_test = scaler.transform(X_test)"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2024-03-21T12:06:49.970355Z",
     "start_time": "2024-03-21T12:06:47.694912Z"
    }
   },
   "id": "72c1e647b6c1992",
   "execution_count": 13
  },
  {
   "cell_type": "markdown",
   "source": [
    "# Convolution Neural Network\n",
    "\n",
    "This code block constructs and trains a Convolutional Neural Network (CNN) for a binary classification task using TensorFlow. CNNs are a type of deep learning model primarily used for image analysis, but they can also be applied to other types of data that have a grid-like topology. Here's a step-by-step breakdown of the code:\n",
    "\n",
    "1. **Model Creation**: A Sequential model is created using Keras. This model is composed of the following layers:\n",
    "\n",
    "   - **Conv1D layers**: These are convolutional layers that will convolve the input data with a set of learnable filters, each producing one feature map in the output. The kernel size is set to 3, and the activation function used is ReLU (Rectified Linear Unit).\n",
    "\n",
    "   - **MaxPooling1D layers**: These layers are used to down-sample the input along its spatial dimensions (height and width). The pool size is set to 2.\n",
    "\n",
    "   - **Dense layers**: These are fully connected layers. The first Dense layer has 64 units and uses the ReLU activation function. The second Dense layer has a number of units equal to the number of classes and uses the softmax activation function to output a probability distribution over the classes.\n",
    "\n",
    "2. **Model Compilation**: The model is compiled with the Adam optimizer, categorical cross-entropy loss function, and accuracy as the evaluation metric.\n",
    "\n",
    "3. **Model Training**: The model is trained on the training data for 10 epochs with a batch size of 32. The validation data is set to the testing set.\n",
    "\n",
    "4. **Model Evaluation**: The model's performance is evaluated on the testing set and the accuracy is printed.\n",
    "\n",
    "The mathematical concept behind the Convolutional layer (Conv1D) is the convolution operation, which is a mathematical operation on two functions that produces a third function. In the context of a CNN, the two functions are the input data and the kernel or filter. The convolution operation involves sliding the kernel across the input data and computing the dot product at each position.\n",
    "\n",
    "The mathematical formula for the convolution operation is:  $$ (f * g)(t) = \\int_{-\\infty}^{\\infty} f(\\tau)g(t - \\tau) d\\tau $$  Where:  \n",
    "- $f$ and $g$ are the input data and kernel respectively\n",
    "- $t$ is the position of the kernel\n",
    "- $\\tau$ is a dummy integration variable\n",
    "\n",
    "In the context of a CNN, the integral is replaced by a sum over the discrete spatial dimensions (height and width) of the input data and kernel."
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "862a9b7ee430a667"
  },
  {
   "cell_type": "code",
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/30\n",
      "1050/1050 [==============================] - 26s 23ms/step - loss: 0.7339 - accuracy: 0.6683 - val_loss: 0.4898 - val_accuracy: 0.8121\n",
      "Epoch 2/30\n",
      "1050/1050 [==============================] - 23s 22ms/step - loss: 0.5412 - accuracy: 0.7951 - val_loss: 0.4465 - val_accuracy: 0.8405\n",
      "Epoch 3/30\n",
      "1050/1050 [==============================] - 24s 22ms/step - loss: 0.4912 - accuracy: 0.8236 - val_loss: 0.4425 - val_accuracy: 0.8493\n",
      "Epoch 4/30\n",
      "1050/1050 [==============================] - 23s 22ms/step - loss: 0.4767 - accuracy: 0.8330 - val_loss: 0.4382 - val_accuracy: 0.8488\n",
      "Epoch 5/30\n",
      "1050/1050 [==============================] - 24s 23ms/step - loss: 0.4644 - accuracy: 0.8382 - val_loss: 0.4399 - val_accuracy: 0.8474\n",
      "Epoch 6/30\n",
      "1050/1050 [==============================] - 24s 23ms/step - loss: 0.4524 - accuracy: 0.8436 - val_loss: 0.4366 - val_accuracy: 0.8499\n",
      "Epoch 7/30\n",
      "1050/1050 [==============================] - 23s 22ms/step - loss: 0.4461 - accuracy: 0.8470 - val_loss: 0.4357 - val_accuracy: 0.8504\n",
      "Epoch 8/30\n",
      "1050/1050 [==============================] - 24s 23ms/step - loss: 0.4411 - accuracy: 0.8497 - val_loss: 0.4321 - val_accuracy: 0.8505\n",
      "Epoch 9/30\n",
      "1050/1050 [==============================] - 24s 23ms/step - loss: 0.4353 - accuracy: 0.8497 - val_loss: 0.4296 - val_accuracy: 0.8504\n",
      "Epoch 10/30\n",
      "1050/1050 [==============================] - 24s 22ms/step - loss: 0.4288 - accuracy: 0.8554 - val_loss: 0.4294 - val_accuracy: 0.8519\n",
      "Epoch 11/30\n",
      "1050/1050 [==============================] - 24s 23ms/step - loss: 0.4299 - accuracy: 0.8548 - val_loss: 0.4276 - val_accuracy: 0.8527\n",
      "Epoch 12/30\n",
      "1050/1050 [==============================] - 25s 24ms/step - loss: 0.4219 - accuracy: 0.8566 - val_loss: 0.4262 - val_accuracy: 0.8551\n",
      "Epoch 13/30\n",
      "1050/1050 [==============================] - 24s 23ms/step - loss: 0.4195 - accuracy: 0.8561 - val_loss: 0.4262 - val_accuracy: 0.8532\n",
      "Epoch 14/30\n",
      "1050/1050 [==============================] - 24s 23ms/step - loss: 0.4147 - accuracy: 0.8572 - val_loss: 0.4226 - val_accuracy: 0.8531\n",
      "Epoch 15/30\n",
      "1050/1050 [==============================] - 24s 23ms/step - loss: 0.4117 - accuracy: 0.8578 - val_loss: 0.4219 - val_accuracy: 0.8546\n",
      "Epoch 16/30\n",
      "1050/1050 [==============================] - 24s 23ms/step - loss: 0.4110 - accuracy: 0.8600 - val_loss: 0.4197 - val_accuracy: 0.8555\n",
      "Epoch 17/30\n",
      "1050/1050 [==============================] - 24s 23ms/step - loss: 0.4040 - accuracy: 0.8611 - val_loss: 0.4179 - val_accuracy: 0.8563\n",
      "Epoch 18/30\n",
      "1050/1050 [==============================] - 23s 22ms/step - loss: 0.4010 - accuracy: 0.8623 - val_loss: 0.4164 - val_accuracy: 0.8550\n",
      "Epoch 19/30\n",
      "1050/1050 [==============================] - 24s 22ms/step - loss: 0.4013 - accuracy: 0.8594 - val_loss: 0.4163 - val_accuracy: 0.8542\n",
      "Epoch 20/30\n",
      "1050/1050 [==============================] - 23s 22ms/step - loss: 0.4013 - accuracy: 0.8600 - val_loss: 0.4172 - val_accuracy: 0.8543\n",
      "Epoch 21/30\n",
      "1050/1050 [==============================] - 23s 22ms/step - loss: 0.3975 - accuracy: 0.8633 - val_loss: 0.4108 - val_accuracy: 0.8571\n",
      "Epoch 22/30\n",
      "1050/1050 [==============================] - 23s 22ms/step - loss: 0.3918 - accuracy: 0.8645 - val_loss: 0.4114 - val_accuracy: 0.8573\n",
      "Epoch 23/30\n",
      "1050/1050 [==============================] - 23s 22ms/step - loss: 0.3954 - accuracy: 0.8633 - val_loss: 0.4073 - val_accuracy: 0.8567\n",
      "Epoch 24/30\n",
      "1050/1050 [==============================] - 23s 22ms/step - loss: 0.3944 - accuracy: 0.8642 - val_loss: 0.4096 - val_accuracy: 0.8587\n",
      "Epoch 25/30\n",
      "1050/1050 [==============================] - 23s 22ms/step - loss: 0.3890 - accuracy: 0.8652 - val_loss: 0.4049 - val_accuracy: 0.8615\n",
      "Epoch 26/30\n",
      "1050/1050 [==============================] - 23s 22ms/step - loss: 0.3873 - accuracy: 0.8657 - val_loss: 0.4034 - val_accuracy: 0.8596\n",
      "Epoch 27/30\n",
      "1050/1050 [==============================] - 23s 22ms/step - loss: 0.3852 - accuracy: 0.8683 - val_loss: 0.4070 - val_accuracy: 0.8589\n",
      "Epoch 28/30\n",
      "1050/1050 [==============================] - 23s 22ms/step - loss: 0.3839 - accuracy: 0.8681 - val_loss: 0.4070 - val_accuracy: 0.8605\n",
      "Epoch 29/30\n",
      "1050/1050 [==============================] - 23s 22ms/step - loss: 0.3826 - accuracy: 0.8685 - val_loss: 0.4033 - val_accuracy: 0.8605\n",
      "Epoch 30/30\n",
      "1050/1050 [==============================] - 25s 24ms/step - loss: 0.3812 - accuracy: 0.8691 - val_loss: 0.4035 - val_accuracy: 0.8631\n",
      "263/263 [==============================] - 1s 3ms/step\n"
     ]
    }
   ],
   "source": [
    "# Initialize a Sequential model\n",
    "model = Sequential()\n",
    "\n",
    "# Add a Conv1D layer\n",
    "model.add(Conv1D(filters=64, kernel_size=3, activation='relu', input_shape=(X_train.shape[1], 1),\n",
    "                 kernel_regularizer=regularizers.l2(0.001)))\n",
    "model.add(BatchNormalization())\n",
    "model.add(Dropout(0.5))\n",
    "\n",
    "# Add another Conv1D layer\n",
    "model.add(Conv1D(filters=32, kernel_size=3, activation='relu', kernel_regularizer=regularizers.l2(0.001)))\n",
    "model.add(BatchNormalization())\n",
    "model.add(Dropout(0.5))\n",
    "\n",
    "# Add a Flatten layer\n",
    "model.add(Flatten())\n",
    "\n",
    "# Add a Dense layer\n",
    "model.add(Dense(16, activation='relu', kernel_regularizer=regularizers.l2(0.001)))\n",
    "model.add(BatchNormalization())\n",
    "model.add(Dropout(0.5))\n",
    "\n",
    "# Add the output Dense layer\n",
    "model.add(Dense(1, activation='sigmoid'))\n",
    "\n",
    "# Define early stopping\n",
    "early_stopping = EarlyStopping(monitor='val_loss', patience=10)\n",
    "\n",
    "# Compile the model\n",
    "model.compile(loss='binary_crossentropy', optimizer=Adam(learning_rate=0.0001), metrics=['accuracy'])\n",
    "\n",
    "# Train the model\n",
    "history = model.fit(X_train, y_train, epochs=30, batch_size=32, validation_data=(X_test, y_test),\n",
    "                    callbacks=[early_stopping])\n",
    "\n",
    "# Evaluate the model\n",
    "scores = model.evaluate(X_test, y_test, verbose=0)\n",
    "\n",
    "# Make predictions\n",
    "y_pred = model.predict(X_test)\n",
    "y_pred_classes = (y_pred > 0.5).astype(\"int32\")\n",
    "\n",
    "# Generate a classification report\n",
    "report = classification_report(y_test, y_pred_classes)"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2024-03-20T12:44:18.952408Z",
     "start_time": "2024-03-20T12:32:23.830299Z"
    }
   },
   "id": "1c1dd203ad7db076",
   "execution_count": 129
  },
  {
   "cell_type": "markdown",
   "source": [
    "# Plot the training and validation accuracy over epochs\n",
    "\n",
    "This code block plots the training and validation accuracy over epochs. The training accuracy is plotted in red, and the validation accuracy is plotted in blue. The x-axis represents the number of epochs, and the y-axis represents the accuracy. The plot shows how the accuracy changes over the course of training and validation."
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "b5f3cff03a03da6a"
  },
  {
   "cell_type": "code",
   "outputs": [
    {
     "data": {
      "text/plain": "<Figure size 640x480 with 1 Axes>",
      "image/png": "[encoded data, 49020 characters]"
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": "<Figure size 1200x600 with 1 Axes>",
      "image/png": "[encoded data, 53936 characters]"
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test loss: 0.40349453687667847\n",
      "Test accuracy: 0.863095223903656\n",
      "Classification Report: \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.84      0.90      0.87      4192\n",
      "           1       0.90      0.82      0.86      4208\n",
      "\n",
      "    accuracy                           0.86      8400\n",
      "   macro avg       0.87      0.86      0.86      8400\n",
      "weighted avg       0.87      0.86      0.86      8400\n"
     ]
    }
   ],
   "source": [
    "# Plot the training and validation accuracy over epochs\n",
    "plt.plot(history.history['accuracy'], 'ro-', history.history['val_accuracy'], 'bv--')\n",
    "plt.title('Training and Test Accuracy')\n",
    "plt.xlabel('Epoch')\n",
    "plt.ylabel('Accuracy')\n",
    "plt.legend(['Training Accuracy', 'Test Accuracy'])\n",
    "plt.show()\n",
    "\n",
    "# Plot the training and validation loss over epochs\n",
    "plt.figure(figsize=(12, 6))\n",
    "plt.plot(history.history['loss'], label='Training Loss')\n",
    "plt.plot(history.history['val_loss'], label='Validation Loss')\n",
    "plt.title('Training and Validation Loss Over Time')\n",
    "plt.xlabel('Epoch')\n",
    "plt.ylabel('Loss')\n",
    "plt.legend()\n",
    "plt.show()\n",
    "\n",
    "# Print the testing loss and accuracy\n",
    "print('Test loss:', scores[0])\n",
    "print('Test accuracy:', scores[1])\n",
    "\n",
    "# Print the classification report\n",
    "print('Classification Report: \\n', report)\n"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2024-03-20T12:44:19.380929Z",
     "start_time": "2024-03-20T12:44:18.953819Z"
    }
   },
   "id": "89aa08d7d1866179",
   "execution_count": 130
  },
  {
   "cell_type": "markdown",
   "source": [
    "# Visualize Weights and Biases\n",
    "\n",
    "This code block visualizes the weights and biases of the model. For each Dense layer in the model, the weights and biases are extracted and plotted as histograms. The weights are plotted in the left subplot, and the biases are plotted in the right subplot. The x-axis represents the value of the weights or biases, and the y-axis represents the frequency of occurrence of each value. The histograms show the distribution of the weights and biases in each layer."
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "dba8a77d06dc51f8"
  },
  {
   "cell_type": "code",
   "outputs": [
    {
     "data": {
      "text/plain": "<Figure size 1000x500 with 2 Axes>",
      "image/png": "[encoded data, 30280 characters]"
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": "<Figure size 1000x500 with 2 Axes>",
      "image/png": "[encoded data, 20056 characters]"
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Visualize Weights and Biases\n",
    "for layer in model.layers:\n",
    "    if 'dense' in layer.name:\n",
    "        weights, biases = layer.get_weights()\n",
    "        plt.figure(figsize=(10, 5))\n",
    "        plt.subplot(1, 2, 1)\n",
    "        plt.hist(weights.flatten())\n",
    "        plt.title(f'{layer.name} weights')\n",
    "        plt.subplot(1, 2, 2)\n",
    "        plt.hist(biases.flatten())\n",
    "        plt.title(f'{layer.name} biases')\n",
    "        plt.show()\n"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2024-03-20T12:44:19.992940Z",
     "start_time": "2024-03-20T12:44:19.381949Z"
    }
   },
   "id": "9739272beb4981b6",
   "execution_count": 131
  },
  {
   "cell_type": "markdown",
   "source": [
    "\n",
    "# Confusion Matrix\n",
    "\n",
    "This code block plots the confusion matrix for the model's predictions on the testing set. The confusion matrix is a table that is often used to describe the performance of a classification model on a set of test data for which the true values are known. The confusion matrix shows the number of true positive, true negative, false positive, and false negative predictions made by the model."
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "7f3da1aa73d186e1"
  },
  {
   "cell_type": "code",
   "outputs": [
    {
     "data": {
      "text/plain": "<Figure size 500x500 with 2 Axes>",
      "image/png": "[encoded data, 25676 characters]"
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Confusion Matrix\n",
    "cm = confusion_matrix(y_test, y_pred_classes)\n",
    "plt.figure(figsize=(5, 5))\n",
    "sns.heatmap(cm, annot=True, fmt='d', cmap='Blues')\n",
    "plt.title('Confusion matrix')\n",
    "plt.xlabel('Predicted class')\n",
    "plt.ylabel('True class')\n",
    "plt.show()"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2024-03-20T12:44:20.152983Z",
     "start_time": "2024-03-20T12:44:19.994211Z"
    }
   },
   "id": "92964bc26d5c26cc",
   "execution_count": 132
  },
  {
   "cell_type": "markdown",
   "source": [
    "\n",
    "# ROC Curve\n",
    "\n",
    "This code block plots the Receiver Operating Characteristic (ROC) curve for the model's predictions on the testing set. The ROC curve is a graphical plot that illustrates the diagnostic ability of a binary classifier system as its discrimination threshold is varied. The curve is created by plotting the true positive rate (TPR) against the false positive rate (FPR) at various threshold settings. The area under the ROC curve (AUC) is also calculated and printed."
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "2d90420f6a5cc744"
  },
  {
   "cell_type": "code",
   "outputs": [
    {
     "data": {
      "text/plain": "<Figure size 640x480 with 1 Axes>",
      "image/png": "[encoded data, 47072 characters]"
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# ROC Curve\n",
    "fpr, tpr, _ = roc_curve(y_test, y_pred)\n",
    "roc_auc = auc(fpr, tpr)\n",
    "plt.figure()\n",
    "lw = 2\n",
    "plt.plot(fpr, tpr, color='darkorange', lw=lw, label='ROC curve (area = %0.2f)' % roc_auc)\n",
    "plt.plot([0, 1], [0, 1], color='navy', lw=lw, linestyle='--')\n",
    "plt.xlim([0.0, 1.0])\n",
    "plt.ylim([0.0, 1.05])\n",
    "plt.xlabel('False Positive Rate')\n",
    "plt.ylabel('True Positive Rate')\n",
    "plt.title('Receiver Operating Characteristic')\n",
    "plt.legend(loc=\"lower right\")\n",
    "plt.show()"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2024-03-20T12:44:20.328279Z",
     "start_time": "2024-03-20T12:44:20.154242Z"
    }
   },
   "id": "dd49203934ca9cf6",
   "execution_count": 133
  },
  {
   "cell_type": "markdown",
   "source": [
    "# Plot the model\n",
    "\n",
    "This code block plots the model architecture using the `plot_model` function from Keras. The plot shows the structure of the model, including the input and output shapes, the layers, and the connections between layers."
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "427d26655cf59f4d"
  },
  {
   "cell_type": "code",
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 184380 characters]",
      "text/plain": "<IPython.core.display.Image object>"
     },
     "execution_count": 134,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Plot the model\n",
    "from tensorflow.keras.utils import plot_model\n",
    "\n",
    "# Generate the plot\n",
    "plot_model(model, to_file='model_plot.png', show_shapes=True, show_layer_names=True)\n"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2024-03-20T12:44:20.448825Z",
     "start_time": "2024-03-20T12:44:20.329360Z"
    }
   },
   "id": "81f2d793ada5c410",
   "execution_count": 134
  },
  {
   "cell_type": "markdown",
   "source": [
    "# Save the model"
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "225526f72ffc154c"
  },
  {
   "cell_type": "code",
   "outputs": [],
   "source": [
    "# Save the model\n",
    "model.save('CNN.keras')"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2024-03-20T12:44:20.516718Z",
     "start_time": "2024-03-20T12:44:20.451223Z"
    }
   },
   "id": "6b7329b28452b82a",
   "execution_count": 135
  },
  {
   "cell_type": "markdown",
   "source": [
    "# Multi-Layer Perceptron (MLP)\n",
    "\n",
    "This code block is implementing a Multi-Layer Perceptron (MLP) for a binary classification task using TensorFlow. The MLP is a class of feedforward artificial neural network that consists of at least three layers of nodes: an input layer, a hidden layer, and an output layer. Except for the input nodes, each node is a neuron that uses a nonlinear activation function. MLP utilizes a supervised learning technique called backpropagation for training.\n",
    "\n",
    "Here's a step-by-step breakdown of what the code does:\n",
    "\n",
    "1. **Model Creation**: A Sequential model is created using Keras. This model is composed of the following layers:\n",
    "    - Dense layers: These are fully connected layers. The first Dense layer has 64 units and uses the ReLU activation function. The second and third Dense layers have 32 and 16 units respectively, and also use the ReLU activation function. The final Dense layer has 1 unit and uses the sigmoid activation function for binary classification.\n",
    "    - BatchNormalization layers: These layers are used to normalize the activations of the previous layer, which speeds up learning and provides some regularization, reducing generalization error.\n",
    "    - Dropout layers: These layers are used to prevent overfitting. They randomly set a fraction of input units to 0 at each update during training time.\n",
    "\n",
    "2. **Model Compilation**: The model is compiled with the Adam optimizer, binary cross-entropy loss function, and accuracy as the evaluation metric.\n",
    "\n",
    "3. **Model Training**: The model is trained on the training data for 20 epochs with a batch size of 32. The validation data is set to the testing set. Early stopping is used to stop training when the validation loss has not improved for 10 epochs.\n",
    "\n",
    "4. **Model Evaluation**: The model's performance is evaluated on the testing data and the loss and accuracy are printed. The model also makes predictions on the testing data, converts the predicted probabilities to binary outputs, and generates a classification report.\n",
    "\n",
    "5. **Visualization**: The training and validation accuracy and loss over epochs are plotted.\n",
    "\n",
    "The mathematical concept behind the Dense layer is the dot product operation, which is a mathematical operation that takes two equal-length sequences of numbers and returns a single number. In the context of a MLP, the two sequences are the input data and the weights of the neurons. The dot product operation involves multiplying each pair of input and weight and summing the result.\n",
    "\n",
    "The mathematical formula for the dot product operation is: $$ a \\cdot b = \\sum_{i=1}^{n} a_i b_i $$ Where:\n",
    "- $a$ and $b$ are the input data and weights respectively\n",
    "- $n$ is the number of dimensions (length of the sequences)\n",
    "- $a_i$ and $b_i$ are the $i$-th elements of the input data and weights respectively."
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "42eff9445377f73c"
  },
  {
   "cell_type": "code",
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/20\n",
      "1050/1050 [==============================] - 3s 1ms/step - loss: 1.0799 - accuracy: 0.5144 - val_loss: 0.8551 - val_accuracy: 0.5844\n",
      "Epoch 2/20\n",
      "1050/1050 [==============================] - 1s 1ms/step - loss: 0.9646 - accuracy: 0.5434 - val_loss: 0.8099 - val_accuracy: 0.6630\n",
      "Epoch 3/20\n",
      "1050/1050 [==============================] - 1s 1ms/step - loss: 0.8904 - accuracy: 0.5891 - val_loss: 0.7648 - val_accuracy: 0.7145\n",
      "Epoch 4/20\n",
      "1050/1050 [==============================] - 1s 1ms/step - loss: 0.8270 - accuracy: 0.6339 - val_loss: 0.7140 - val_accuracy: 0.7608\n",
      "Epoch 5/20\n",
      "1050/1050 [==============================] - 1s 1ms/step - loss: 0.7637 - accuracy: 0.6911 - val_loss: 0.6635 - val_accuracy: 0.7911\n",
      "Epoch 6/20\n",
      "1050/1050 [==============================] - 1s 1ms/step - loss: 0.7141 - accuracy: 0.7305 - val_loss: 0.6225 - val_accuracy: 0.8102\n",
      "Epoch 7/20\n",
      "1050/1050 [==============================] - 1s 1ms/step - loss: 0.6718 - accuracy: 0.7602 - val_loss: 0.5870 - val_accuracy: 0.8201\n",
      "Epoch 8/20\n",
      "1050/1050 [==============================] - 1s 1ms/step - loss: 0.6386 - accuracy: 0.7813 - val_loss: 0.5622 - val_accuracy: 0.8275\n",
      "Epoch 9/20\n",
      "1050/1050 [==============================] - 1s 1ms/step - loss: 0.6117 - accuracy: 0.7930 - val_loss: 0.5436 - val_accuracy: 0.8290\n",
      "Epoch 10/20\n",
      "1050/1050 [==============================] - 1s 1ms/step - loss: 0.5868 - accuracy: 0.8076 - val_loss: 0.5300 - val_accuracy: 0.8301\n",
      "Epoch 11/20\n",
      "1050/1050 [==============================] - 1s 1ms/step - loss: 0.5731 - accuracy: 0.8127 - val_loss: 0.5162 - val_accuracy: 0.8329\n",
      "Epoch 12/20\n",
      "1050/1050 [==============================] - 1s 1ms/step - loss: 0.5565 - accuracy: 0.8180 - val_loss: 0.5040 - val_accuracy: 0.8365\n",
      "Epoch 13/20\n",
      "1050/1050 [==============================] - 1s 1ms/step - loss: 0.5428 - accuracy: 0.8225 - val_loss: 0.4976 - val_accuracy: 0.8361\n",
      "Epoch 14/20\n",
      "1050/1050 [==============================] - 1s 1ms/step - loss: 0.5264 - accuracy: 0.8289 - val_loss: 0.4871 - val_accuracy: 0.8383\n",
      "Epoch 15/20\n",
      "1050/1050 [==============================] - 1s 1ms/step - loss: 0.5191 - accuracy: 0.8292 - val_loss: 0.4794 - val_accuracy: 0.8385\n",
      "Epoch 16/20\n",
      "1050/1050 [==============================] - 1s 1ms/step - loss: 0.5026 - accuracy: 0.8335 - val_loss: 0.4738 - val_accuracy: 0.8400\n",
      "Epoch 17/20\n",
      "1050/1050 [==============================] - 1s 1ms/step - loss: 0.4927 - accuracy: 0.8378 - val_loss: 0.4668 - val_accuracy: 0.8387\n",
      "Epoch 18/20\n",
      "1050/1050 [==============================] - 1s 1ms/step - loss: 0.4887 - accuracy: 0.8372 - val_loss: 0.4616 - val_accuracy: 0.8410\n",
      "Epoch 19/20\n",
      "1050/1050 [==============================] - 1s 1ms/step - loss: 0.4771 - accuracy: 0.8408 - val_loss: 0.4589 - val_accuracy: 0.8396\n",
      "Epoch 20/20\n",
      "1050/1050 [==============================] - 1s 1ms/step - loss: 0.4713 - accuracy: 0.8430 - val_loss: 0.4511 - val_accuracy: 0.8417\n",
      "263/263 [==============================] - 0s 531us/step\n"
     ]
    },
    {
     "data": {
      "text/plain": "<Figure size 1200x600 with 1 Axes>",
      "image/png": "[encoded data, 104308 characters]"
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": "<Figure size 1200x600 with 1 Axes>",
      "image/png": "[encoded data, 69776 characters]"
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": "<Figure size 1200x600 with 1 Axes>",
      "image/png": "[encoded data, 69404 characters]"
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": "<Figure size 640x480 with 0 Axes>"
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test loss: 0.4510885179042816\n",
      "Test accuracy: 0.8416666388511658\n",
      "Classification Report: \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.80      0.90      0.85      4192\n",
      "           1       0.89      0.78      0.83      4208\n",
      "\n",
      "    accuracy                           0.84      8400\n",
      "   macro avg       0.85      0.84      0.84      8400\n",
      "weighted avg       0.85      0.84      0.84      8400\n"
     ]
    }
   ],
   "source": [
    "# Initialize a Sequential model\n",
    "model = Sequential()\n",
    "# Add a Dense layer with 64 units, ReLU activation function and L2 regularization\n",
    "model.add(Dense(64, input_dim=X_train.shape[1], activation='relu', kernel_regularizer=regularizers.l2(0.001)))\n",
    "# Add a BatchNormalization layer to normalize the activations of the previous layer\n",
    "model.add(BatchNormalization())\n",
    "# Add a Dropout layer to prevent overfitting\n",
    "model.add(Dropout(0.5))\n",
    "# Add another Dense layer with 32 units, ReLU activation function and L2 regularization\n",
    "model.add(Dense(32, activation='relu', kernel_regularizer=regularizers.l2(0.001)))\n",
    "# Add another BatchNormalization layer\n",
    "model.add(BatchNormalization())\n",
    "# Add another Dropout layer\n",
    "model.add(Dropout(0.5))\n",
    "# Add another Dense layer with 16 units, ReLU activation function and L2 regularization\n",
    "model.add(Dense(16, activation='relu', kernel_regularizer=regularizers.l2(0.001)))\n",
    "# Add another BatchNormalization layer\n",
    "model.add(BatchNormalization())\n",
    "# Add another Dropout layer\n",
    "model.add(Dropout(0.5))\n",
    "# Add the output Dense layer with 1 unit and sigmoid activation function\n",
    "model.add(Dense(1, activation='sigmoid'))\n",
    "\n",
    "# Define early stopping to stop training when the validation loss has not improved for 10 epochs\n",
    "early_stopping = EarlyStopping(monitor='val_loss', patience=10)\n",
    "\n",
    "# Compile the model with Adam optimizer, binary cross-entropy loss function and accuracy as the evaluation metric\n",
    "model.compile(loss='binary_crossentropy', optimizer=Adam(learning_rate=0.0001), metrics=['accuracy'])\n",
    "\n",
    "# Train the model on the training data for 20 epochs with a batch size of 32 and validate on the testing data\n",
    "history = model.fit(X_train, y_train, epochs=20, batch_size=32, validation_data=(X_test, y_test),\n",
    "                    callbacks=[early_stopping])\n",
    "\n",
    "# Evaluate the model on the testing data and store the loss and accuracy in 'scores'\n",
    "scores = model.evaluate(X_test, y_test, verbose=0)\n",
    "\n",
    "# Make predictions on the testing data\n",
    "y_pred = model.predict(X_test)\n",
    "# Convert the predicted probabilities to binary outputs\n",
    "y_pred_classes = (y_pred > 0.5).astype(\"int32\")\n",
    "# Generate a classification report\n",
    "report = classification_report(y_test, y_pred_classes)\n",
    "\n",
    "fig, ax = plt.subplots(figsize=(12, 6))\n",
    "\n",
    "# Plot training & validation accuracy values\n",
    "ax.plot(history.history['accuracy'], 'ro-', label='Training Accuracy')\n",
    "ax.plot(history.history['val_accuracy'], 'bv--', label='Test Accuracy')\n",
    "\n",
    "# Plot training & validation loss values\n",
    "ax.plot(history.history['loss'], 'go-', label='Training Loss')\n",
    "ax.plot(history.history['val_loss'], 'yv--', label='Validation Loss')\n",
    "\n",
    "ax.set_title('Learning Curve')\n",
    "ax.set_ylabel('Values')\n",
    "ax.set_xlabel('Epoch')\n",
    "ax.legend()\n",
    "\n",
    "# Show the figure\n",
    "plt.tight_layout()\n",
    "plt.show()\n",
    "\n",
    "# Create a figure for the accuracy plot\n",
    "fig1, ax1 = plt.subplots(figsize=(12, 6))\n",
    "\n",
    "# Plot training & validation accuracy values\n",
    "ax1.plot(history.history['accuracy'], 'ro-', label='Training Accuracy')\n",
    "ax1.plot(history.history['val_accuracy'], 'bv--', label='Test Accuracy')\n",
    "ax1.set_title('Model Accuracy')\n",
    "ax1.set_ylabel('Accuracy')\n",
    "ax1.set_xlabel('Epoch')\n",
    "ax1.legend()\n",
    "\n",
    "# Show the figure\n",
    "plt.tight_layout()\n",
    "plt.show()\n",
    "\n",
    "# Create a figure for the loss plot\n",
    "fig2, ax2 = plt.subplots(figsize=(12, 6))\n",
    "\n",
    "# Plot training & validation loss values\n",
    "ax2.plot(history.history['loss'], 'ro-', label='Training Loss')\n",
    "ax2.plot(history.history['val_loss'], 'bv--', label='Validation Loss')\n",
    "ax2.set_title('Model Loss')\n",
    "ax2.set_ylabel('Loss')\n",
    "ax2.set_xlabel('Epoch')\n",
    "ax2.legend()\n",
    "\n",
    "# Show the figure\n",
    "plt.tight_layout()\n",
    "plt.show()\n",
    "\n",
    "# Show the figure\n",
    "plt.tight_layout()\n",
    "plt.show()\n",
    "\n",
    "# Print the testing loss and accuracy\n",
    "print('Test loss:', scores[0])\n",
    "print('Test accuracy:', scores[1])\n",
    "# Print the classification report\n",
    "print('Classification Report: \\n', report)"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2024-03-21T12:08:46.922522Z",
     "start_time": "2024-03-21T12:08:16.554423Z"
    }
   },
   "id": "c8745832a585d5ec",
   "execution_count": 15
  },
  {
   "cell_type": "markdown",
   "source": [
    "# Multi-Layer Perceptron (MLP) visualization\n",
    "This code block is used to visualize the performance of a trained Multi-Layer Perceptron (MLP) model. It generates three types of visualizations:\n",
    "\n",
    "1. Weights and Biases Visualization: This visualization is used to understand the distribution of weights and biases in the model's layers. For each layer in the model, if the layer is a dense layer, it retrieves the weights and biases, and plots histograms of their values. The x-axis of the histogram represents the value of the weights/biases and the y-axis represents the frequency of these values.\n",
    "\n",
    "2. Confusion Matrix: A confusion matrix is a table that is often used to describe the performance of a classification model on a set of test data for which the true values are known. It gives a more detailed breakdown of correct and incorrect classifications for each class.\n",
    "\n",
    "3. ROC Curve: The Receiver Operating Characteristic (ROC) curve is a plot that illustrates the diagnostic ability of a binary classifier system as its discrimination threshold is varied. It is created by plotting the true positive rate (TPR) against the false positive rate (FPR).\n",
    "\n",
    "## Weights and Biases Evaluation\n",
    "\n",
    "The weights and biases of the MLP model layers were visualized to understand their distributions. The weights in all layers (dense_20, dense_21, dense_22, and dense_23) are not close to zero, indicating they are likely being updated during training and contributing to the model's learning. The weight distributions show a spread around zero, suggesting the model is capturing complex relationships in the data.\n",
    "\n",
    "The biases in dense_20 and dense_22 introduce a slight positive bias to the activations in subsequent layers, potentially affecting the model's predictions. The biases in dense_21 and dense_23 are centered around zero, with a slight spread towards positive values, introducing a small positive shift in the activations of the next layer.\n",
    "\n",
    "The impact of these biases would depend on the network architecture and data. Overall, the model's weights and biases suggest that it is learning effectively from the training data."
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "4114f5c851874555"
  },
  {
   "cell_type": "code",
   "outputs": [
    {
     "data": {
      "text/plain": "<Figure size 1000x500 with 2 Axes>",
      "image/png": "[encoded data, 30868 characters]"
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": "<Figure size 1000x500 with 2 Axes>",
      "image/png": "[encoded data, 25792 characters]"
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": "<Figure size 1000x500 with 2 Axes>",
      "image/png": "[encoded data, 21636 characters]"
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": "<Figure size 1000x500 with 2 Axes>",
      "image/png": "[encoded data, 25300 characters]"
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 1. Visualize Weights and Biases\n",
    "for layer in model.layers:\n",
    "    if 'dense' in layer.name:\n",
    "        weights, biases = layer.get_weights()\n",
    "        plt.figure(figsize=(10, 5))\n",
    "        plt.subplot(1, 2, 1)\n",
    "        plt.hist(weights.flatten())\n",
    "        plt.title(f'{layer.name} weights')\n",
    "        plt.subplot(1, 2, 2)\n",
    "        plt.hist(biases.flatten())\n",
    "        plt.title(f'{layer.name} biases')\n",
    "        plt.show()"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2024-03-20T12:45:13.997738Z",
     "start_time": "2024-03-20T12:45:12.759536Z"
    }
   },
   "id": "41091791008ff727",
   "execution_count": 137
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Confusion Matrix Evaluation\n",
    "\n",
    "The confusion matrix is a table that is often used to describe the performance of a classification model on a set of test data for which the true values are known. It gives a more detailed breakdown of correct and incorrect classifications for each class.\n",
    "\n",
    "The confusion matrix shows that the model has a good performance in distinguishing between positive and negative instances. The majority of the instances are correctly classified, with a small number of false positives and false negatives."
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "618bc6deb5ea296b"
  },
  {
   "cell_type": "code",
   "outputs": [
    {
     "data": {
      "text/plain": "<Figure size 500x500 with 2 Axes>",
      "image/png": "[encoded data, 25620 characters]"
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 2. Confusion Matrix\n",
    "# Convert the predicted probabilities to binary outputs\n",
    "y_pred_classes = (y_pred > 0.5).astype(\"int32\")\n",
    "# Generate the confusion matrix\n",
    "cm = confusion_matrix(y_test, y_pred_classes)\n",
    "# Plot the confusion matrix\n",
    "plt.figure(figsize=(5, 5))\n",
    "sns.heatmap(cm, annot=True, fmt='d', cmap='Blues')\n",
    "plt.title('Confusion matrix')\n",
    "plt.xlabel('Predicted class')\n",
    "plt.ylabel('True class')\n",
    "plt.show()"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2024-03-21T12:10:13.555575Z",
     "start_time": "2024-03-21T12:10:13.467732Z"
    }
   },
   "id": "6b7d586ea49a858a",
   "execution_count": 16
  },
  {
   "cell_type": "markdown",
   "source": [
    "## ROC Curve Evaluation\n",
    "\n",
    "The Receiver Operating Characteristic (ROC) curve is a graphical representation that illustrates the performance of a binary classification model at all classification thresholds. It is commonly used in machine learning to evaluate the performance of a classifier system for two-class problems.\n",
    "\n",
    "The ROC curve has two axes:\n",
    "- The X-axis represents the False Positive Rate (FPR), which is the proportion of negative instances that are incorrectly classified as positive.\n",
    "- The Y-axis represents the True Positive Rate (TPR), which is the proportion of positive instances that are correctly classified as positive.\n",
    "\n",
    "A perfect classifier would classify all positive instances correctly (TPR = 1) and all negative instances correctly (FPR = 0). This would be represented by a curve that goes straight up the left side of the ROC graph and then along the top to the right corner.\n",
    "\n",
    "The Area Under the Curve (AUC) is a numerical measure of the ROC curve’s performance. A larger AUC indicates a better performance. In our case, the AUC is 0.91, which is considered to be very good.\n",
    "\n",
    "In summary, the ROC curve shows that our binary classification model has a good performance in distinguishing between positive and negative instances."
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "8645f7e159d38f0a"
  },
  {
   "cell_type": "code",
   "outputs": [
    {
     "data": {
      "text/plain": "<Figure size 640x480 with 1 Axes>",
      "image/png": "[encoded data, 46988 characters]"
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 3. ROC Curve\n",
    "# Compute ROC curve and ROC area for each class\n",
    "fpr, tpr, _ = roc_curve(y_test, y_pred)\n",
    "roc_auc = auc(fpr, tpr)\n",
    "# Plot the ROC curve\n",
    "plt.figure()\n",
    "lw = 2\n",
    "plt.plot(fpr, tpr, color='darkorange', lw=lw, label='ROC curve (area = %0.2f)' % roc_auc)\n",
    "plt.plot([0, 1], [0, 1], color='navy', lw=lw, linestyle='--')\n",
    "plt.xlim([0.0, 1.0])\n",
    "plt.ylim([0.0, 1.05])\n",
    "plt.xlabel('False Positive Rate')\n",
    "plt.ylabel('True Positive Rate')\n",
    "plt.title('Receiver Operating Characteristic')\n",
    "plt.legend(loc=\"lower right\")\n",
    "plt.show()"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2024-03-21T12:12:01.664077Z",
     "start_time": "2024-03-21T12:12:01.557243Z"
    }
   },
   "id": "4d080bef0cf9bec4",
   "execution_count": 18
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Learning curve"
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "5d1867cb3af9788d"
  },
  {
   "cell_type": "code",
   "outputs": [
    {
     "data": {
      "text/plain": "<Figure size 1000x600 with 1 Axes>",
      "image/png": "[encoded data, 79496 characters]"
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "def plot_learning_curve(history):\n",
    "    plt.figure(figsize=(10, 6))\n",
    "    plt.plot(history.history['loss'], label='Training Loss')\n",
    "    plt.plot(history.history['val_loss'], label='Validation Loss')\n",
    "    plt.plot(history.history['accuracy'], label='Training Accuracy')\n",
    "    plt.plot(history.history['val_accuracy'], label='Validation Accuracy')\n",
    "    plt.xlabel('Epoch')\n",
    "    plt.ylabel('Loss / Accuracy')\n",
    "    plt.title('Learning Curve')\n",
    "    plt.legend()\n",
    "    plt.show()\n",
    "\n",
    "\n",
    "plot_learning_curve(history)\n"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2024-03-20T12:45:14.573591Z",
     "start_time": "2024-03-20T12:45:14.347420Z"
    }
   },
   "id": "c67bb53e5a864293",
   "execution_count": 140
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
