{
 "cells": [
  {
   "cell_type": "markdown",
   "source": [
    "# New data derivers\n",
    "\n",
    "In this package, a very limited number of derivers are currently provided. A deriver can be used to calculate new features (continuous or categorical) based on existing features, or load images, text, etc. as multimodal data. The source code of the integrated `tabensemb.data.dataderiver.RelativeDeriver` is extended here to demonstrate the implementation procedure.\n"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "outputs": [],
   "source": [
    "from tabensemb.data.dataderiver import AbstractDeriver"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "Data derivers inherit `tabensemb.data.AbstractDervier` and four methods should be implemented:\n",
    "\n",
    "* `_required_cols`: Arguments for columns that must exist in the tabular dataset. The following code means that the arguments `absolute_col` and `relative2_col` should be given in the configuration, such as `\"data_derivers\": [(\"MyRelativeDeriver\", {\"absolute_col\": \"cont_0\", \"relative2_col\": \"cont_1\"})]`\n",
    "\n",
    "```python\n",
    "class MyRelativeDeriver(AbstractDeriver):\n",
    "    def _required_cols(self):\n",
    "        return [\"absolute_col\", \"relative2_col\"]\n",
    "```\n",
    "\n",
    "* `_required_kwargs`: Parameters that must be specified in the configuration. The following code means that the parameter `some_param` should be given in the configuration, such as `\"data_derivers\": [(\"MyRelativeDeriver\", {\"some_param\": 1.5})]`\n",
    "\n",
    "```python\n",
    "    def _required_kwargs(self):\n",
    "        return [\"some_param\"]\n",
    "```\n",
    "\n",
    "**Remark**: \"stacked\", \"intermediate\", \"derived_name\", and \"is_continuous\" are shared necessary kwargs and do not need to be added to `_required_kwargs`.\n",
    "\n",
    "* `_defaults`: Default values of those in `_required_cols`, `_required_kwargs`, and `[\"stacked\", \"intermediate\", \"derived_name\", \"is_continuous\"]`. If default values are given, no error will be raised if the argument is not set in the configuration.\n",
    "\n",
    "```python\n",
    "    def _defaults(self):\n",
    "        return dict(stacked=True, intermediate=False, is_continuous=True)\n",
    "```\n",
    "\n",
    "* `_derive`: The main derivation step. It receives the tabular data (a `DataFrame`) and a `DataModule` and should return an `np.ndarray`. The returned array can not be 1d. Arguments are checked and recorded in `self.kwargs` when initializing.\n",
    "\n",
    "```python\n",
    "    def _derive(self, df, datamodule):\n",
    "        absolute_col = self.kwargs[\"absolute_col\"]\n",
    "        relative2_col = self.kwargs[\"relative2_col\"]\n",
    "        some_param = self.kwargs[\"some_param\"]\n",
    "        stacked = self.kwargs[\"stacked\"]\n",
    "\n",
    "        relative = df[absolute_col] / df[relative2_col]\n",
    "        relative = relative.values.reshape(-1, 1)\n",
    "        return relative\n",
    "```"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "class MyRelativeDeriver(AbstractDeriver):\n",
    "    def _required_cols(self):\n",
    "        return [\"absolute_col\", \"relative2_col\"]\n",
    "\n",
    "    def _required_kwargs(self):\n",
    "        return [\"some_param\"]\n",
    "\n",
    "    def _defaults(self):\n",
    "        return dict(stacked=True, intermediate=False, is_continuous=True)\n",
    "\n",
    "    def _derive(self, df, datamodule):\n",
    "        absolute_col = self.kwargs[\"absolute_col\"]\n",
    "        relative2_col = self.kwargs[\"relative2_col\"]\n",
    "        some_param = self.kwargs[\"some_param\"]\n",
    "        stacked = self.kwargs[\"stacked\"]\n",
    "\n",
    "        relative = df[absolute_col] / df[relative2_col]\n",
    "        relative = relative.values.reshape(-1, 1)\n",
    "        return relative"
   ]
  },
  {
   "cell_type": "markdown",
   "source": [
    "The implemented splitter should be registered as follows to be recognized by `DataModule.set_data_derivers` automatically."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "outputs": [],
   "source": [
    "from tabensemb.data.dataderiver import deriver_mapping\n",
    "deriver_mapping[\"MyRelativeDeriver\"] = MyRelativeDeriver"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The project will be saved to ../../../../output/sample/2023-09-18-18-15-00-0_sample\n"
     ]
    }
   ],
   "source": [
    "from tabensemb.trainer import Trainer\n",
    "import tabensemb\n",
    "\n",
    "prefix = \"../../../../\"\n",
    "tabensemb.setting[\"default_output_path\"] = prefix + \"output\"\n",
    "tabensemb.setting[\"default_config_path\"] = prefix + \"configs\"\n",
    "tabensemb.setting[\"default_data_path\"] = prefix + \"data\"\n",
    "\n",
    "trainer = Trainer(device=\"cpu\")\n",
    "\n",
    "trainer.load_config(\"sample\")"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "If `stacked` is `True`:"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataset size: 153 51 52\n",
      "Data saved to ../../../../output/sample/2023-09-18-18-15-00-0_sample (data.csv and tabular_data.csv).\n",
      "cont_0_relative2_cont_1 in continuous features?: True\n"
     ]
    },
    {
     "data": {
      "text/plain": "       cont_0    cont_1    cont_2    cont_3    cont_4    cont_5    cont_6  \\\n0   -1.306527  0.065895 -0.118164 -0.159573  1.658131 -1.346718 -0.680178   \n1    2.011257  0.117717  0.195070  0.527004 -0.044595  0.616887 -1.781563   \n2   -1.216077  0.065895 -0.743672  0.730184  0.140672  1.272954 -0.159012   \n3    0.559299  0.117717 -0.431096 -0.809627 -1.063696 -0.860153  0.572751   \n4    0.910179 -0.213096  0.786328 -0.042257  0.317218  0.379152 -0.466419   \n..        ...       ...       ...       ...       ...       ...       ...   \n251  0.280442 -0.206904  0.841631  0.880179 -0.993124 -1.570623 -0.249459   \n252 -1.165150 -1.070753  0.465662  1.054452  0.900826 -0.179925 -1.536244   \n253 -0.069856 -0.186691 -1.021913 -1.143641  0.250114  1.040239 -1.150438   \n254 -1.031482 -0.860262 -0.061638  0.328301 -1.429991 -1.048170 -1.432735   \n255 -1.461733  0.960693  0.367545  1.329063 -0.683440 -1.184687  0.190312   \n\n       cont_7    cont_8    cont_9  ...  cat_4       cat_5  cat_6  cat_7  \\\n0   -1.334258  0.666383 -0.460720  ...      2  category_4      3      4   \n1    0.354758 -0.729045  0.196557  ...      3  category_3      3      1   \n2   -0.475175  0.240057  0.100159  ...      4  category_3      4      1   \n3   -0.467441  0.677557  1.307184  ...      1  category_3      4      2   \n4   -0.017020 -0.944446 -0.410050  ...      0  category_2      0      2   \n..        ...       ...       ...  ...    ...         ...    ...    ...   \n251  0.643314  0.049495  0.493837  ...      2  category_2      2      3   \n252  1.178780  1.488252  1.895889  ...      2  category_4      4      2   \n253  0.258798 -0.836111  0.642211  ...      3  category_3      2      2   \n254  0.607112  0.087531  0.938747  ...      0  category_3      4      1   \n255 -0.521580 -0.851729  1.822724  ...      1  category_3      4      1   \n\n     cat_8 cat_9      target  target_binary  target_multi_class  \\\n0        4     3  -71.084217              0                   1   \n1        3     2   13.415675              1                   2   \n2        0     2  -47.492280              0                   2   \n3        0     0  -94.482614              1                   2   \n4        3     0  195.819531              1                   3   \n..     ...   ...         ...            ...                 ...   \n251      0     2 -171.249549              0                   0   \n252      1     1   23.708442              0                   2   \n253      2     2  -33.414215              1                   1   \n254      4     4 -359.199191              0                   4   \n255      1     4 -135.199100              1                   2   \n\n     cont_0_relative2_cont_1  \n0                 -19.827301  \n1                  17.085552  \n2                 -18.454666  \n3                   4.751225  \n4                  -4.271217  \n..                       ...  \n251                -1.355422  \n252                 1.088160  \n253                 0.374183  \n254                 1.199032  \n255                -1.521539  \n\n[256 rows x 24 columns]",
      "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>cont_0</th>\n      <th>cont_1</th>\n      <th>cont_2</th>\n      <th>cont_3</th>\n      <th>cont_4</th>\n      <th>cont_5</th>\n      <th>cont_6</th>\n      <th>cont_7</th>\n      <th>cont_8</th>\n      <th>cont_9</th>\n      <th>...</th>\n      <th>cat_4</th>\n      <th>cat_5</th>\n      <th>cat_6</th>\n      <th>cat_7</th>\n      <th>cat_8</th>\n      <th>cat_9</th>\n      <th>target</th>\n      <th>target_binary</th>\n      <th>target_multi_class</th>\n      <th>cont_0_relative2_cont_1</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>-1.306527</td>\n      <td>0.065895</td>\n      <td>-0.118164</td>\n      <td>-0.159573</td>\n      <td>1.658131</td>\n      <td>-1.346718</td>\n      <td>-0.680178</td>\n      <td>-1.334258</td>\n      <td>0.666383</td>\n      <td>-0.460720</td>\n      <td>...</td>\n      <td>2</td>\n      <td>category_4</td>\n      <td>3</td>\n      <td>4</td>\n      <td>4</td>\n      <td>3</td>\n      <td>-71.084217</td>\n      <td>0</td>\n      <td>1</td>\n      <td>-19.827301</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>2.011257</td>\n      <td>0.117717</td>\n      <td>0.195070</td>\n      <td>0.527004</td>\n      <td>-0.044595</td>\n      <td>0.616887</td>\n      <td>-1.781563</td>\n      <td>0.354758</td>\n      <td>-0.729045</td>\n      <td>0.196557</td>\n      <td>...</td>\n      <td>3</td>\n      <td>category_3</td>\n      <td>3</td>\n      <td>1</td>\n      <td>3</td>\n      <td>2</td>\n      <td>13.415675</td>\n      <td>1</td>\n      <td>2</td>\n      <td>17.085552</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>-1.216077</td>\n      <td>0.065895</td>\n      <td>-0.743672</td>\n      <td>0.730184</td>\n      <td>0.140672</td>\n      <td>1.272954</td>\n      <td>-0.159012</td>\n      <td>-0.475175</td>\n      <td>0.240057</td>\n      <td>0.100159</td>\n      <td>...</td>\n      <td>4</td>\n      <td>category_3</td>\n      <td>4</td>\n      <td>1</td>\n      <td>0</td>\n      <td>2</td>\n      <td>-47.492280</td>\n      <td>0</td>\n      <td>2</td>\n      <td>-18.454666</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>0.559299</td>\n      <td>0.117717</td>\n      <td>-0.431096</td>\n      <td>-0.809627</td>\n      <td>-1.063696</td>\n      <td>-0.860153</td>\n      <td>0.572751</td>\n      <td>-0.467441</td>\n      <td>0.677557</td>\n      <td>1.307184</td>\n      <td>...</td>\n      <td>1</td>\n      <td>category_3</td>\n      <td>4</td>\n      <td>2</td>\n      <td>0</td>\n      <td>0</td>\n      <td>-94.482614</td>\n      <td>1</td>\n      <td>2</td>\n      <td>4.751225</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>0.910179</td>\n      <td>-0.213096</td>\n      <td>0.786328</td>\n      <td>-0.042257</td>\n      <td>0.317218</td>\n      <td>0.379152</td>\n      <td>-0.466419</td>\n      <td>-0.017020</td>\n      <td>-0.944446</td>\n      <td>-0.410050</td>\n      <td>...</td>\n      <td>0</td>\n      <td>category_2</td>\n      <td>0</td>\n      <td>2</td>\n      <td>3</td>\n      <td>0</td>\n      <td>195.819531</td>\n      <td>1</td>\n      <td>3</td>\n      <td>-4.271217</td>\n    </tr>\n    <tr>\n      <th>...</th>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n    </tr>\n    <tr>\n      <th>251</th>\n      <td>0.280442</td>\n      <td>-0.206904</td>\n      <td>0.841631</td>\n      <td>0.880179</td>\n      <td>-0.993124</td>\n      <td>-1.570623</td>\n      <td>-0.249459</td>\n      <td>0.643314</td>\n      <td>0.049495</td>\n      <td>0.493837</td>\n      <td>...</td>\n      <td>2</td>\n      <td>category_2</td>\n      <td>2</td>\n      <td>3</td>\n      <td>0</td>\n      <td>2</td>\n      <td>-171.249549</td>\n      <td>0</td>\n      <td>0</td>\n      <td>-1.355422</td>\n    </tr>\n    <tr>\n      <th>252</th>\n      <td>-1.165150</td>\n      <td>-1.070753</td>\n      <td>0.465662</td>\n      <td>1.054452</td>\n      <td>0.900826</td>\n      <td>-0.179925</td>\n      <td>-1.536244</td>\n      <td>1.178780</td>\n      <td>1.488252</td>\n      <td>1.895889</td>\n      <td>...</td>\n      <td>2</td>\n      <td>category_4</td>\n      <td>4</td>\n      <td>2</td>\n      <td>1</td>\n      <td>1</td>\n      <td>23.708442</td>\n      <td>0</td>\n      <td>2</td>\n      <td>1.088160</td>\n    </tr>\n    <tr>\n      <th>253</th>\n      <td>-0.069856</td>\n      <td>-0.186691</td>\n      <td>-1.021913</td>\n      <td>-1.143641</td>\n      <td>0.250114</td>\n      <td>1.040239</td>\n      <td>-1.150438</td>\n      <td>0.258798</td>\n      <td>-0.836111</td>\n      <td>0.642211</td>\n      <td>...</td>\n      <td>3</td>\n      <td>category_3</td>\n      <td>2</td>\n      <td>2</td>\n      <td>2</td>\n      <td>2</td>\n      <td>-33.414215</td>\n      <td>1</td>\n      <td>1</td>\n      <td>0.374183</td>\n    </tr>\n    <tr>\n      <th>254</th>\n      <td>-1.031482</td>\n      <td>-0.860262</td>\n      <td>-0.061638</td>\n      <td>0.328301</td>\n      <td>-1.429991</td>\n      <td>-1.048170</td>\n      <td>-1.432735</td>\n      <td>0.607112</td>\n      <td>0.087531</td>\n      <td>0.938747</td>\n      <td>...</td>\n      <td>0</td>\n      <td>category_3</td>\n      <td>4</td>\n      <td>1</td>\n      <td>4</td>\n      <td>4</td>\n      <td>-359.199191</td>\n      <td>0</td>\n      <td>4</td>\n      <td>1.199032</td>\n    </tr>\n    <tr>\n      <th>255</th>\n      <td>-1.461733</td>\n      <td>0.960693</td>\n      <td>0.367545</td>\n      <td>1.329063</td>\n      <td>-0.683440</td>\n      <td>-1.184687</td>\n      <td>0.190312</td>\n      <td>-0.521580</td>\n      <td>-0.851729</td>\n      <td>1.822724</td>\n      <td>...</td>\n      <td>1</td>\n      <td>category_3</td>\n      <td>4</td>\n      <td>1</td>\n      <td>1</td>\n      <td>4</td>\n      <td>-135.199100</td>\n      <td>1</td>\n      <td>2</td>\n      <td>-1.521539</td>\n    </tr>\n  </tbody>\n</table>\n<p>256 rows × 24 columns</p>\n</div>"
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trainer.datamodule.set_data_derivers([(\"MyRelativeDeriver\", {\"absolute_col\": \"cont_0\", \"relative2_col\": \"cont_1\", \"derived_name\": \"cont_0_relative2_cont_1\", \"some_param\": 1.0, \"stacked\": True})])\n",
    "trainer.load_data()\n",
    "print(f\"cont_0_relative2_cont_1 in continuous features?: {'cont_0_relative2_cont_1' in trainer.cont_feature_names}\")\n",
    "trainer.df"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "If `stacked` is `True` but `intermediate` is True:"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using previously used data path ../../../../data/sample.csv\n",
      "Dataset size: 153 51 52\n",
      "Data saved to ../../../../output/sample/2023-09-18-18-15-00-0_sample (data.csv and tabular_data.csv).\n",
      "cont_0_relative2_cont_1 in continuous features?: False\n"
     ]
    },
    {
     "data": {
      "text/plain": "       cont_0    cont_1    cont_2    cont_3    cont_4    cont_5    cont_6  \\\n0   -1.306527 -0.409756 -0.118164 -0.159573  1.658131 -1.346718 -0.680178   \n1    2.011257 -0.409756  0.195070  0.527004 -0.044595  0.616887 -1.781563   \n2   -1.216077  0.104704 -0.743672  0.730184  0.140672  1.272954 -0.159012   \n3    0.559299  0.104704 -0.431096 -0.809627 -1.063696 -0.860153  0.572751   \n4    0.910179 -0.409756  0.786328 -0.042257  0.317218  0.379152 -0.466419   \n..        ...       ...       ...       ...       ...       ...       ...   \n251  0.280442 -0.206904  0.841631  0.880179 -0.993124 -1.570623 -0.249459   \n252 -1.165150 -1.070753  0.465662  1.054452  0.900826 -0.179925 -1.536244   \n253 -0.069856 -0.186691 -1.021913 -1.143641  0.250114  1.040239 -1.150438   \n254 -1.031482 -0.860262 -0.061638  0.328301 -1.429991 -1.048170 -1.432735   \n255 -1.461733  0.960693  0.367545  1.329063 -0.683440 -1.184687  0.190312   \n\n       cont_7    cont_8    cont_9  ...  cat_4       cat_5  cat_6  cat_7  \\\n0   -1.334258  0.666383 -0.460720  ...      2  category_4      3      4   \n1    0.354758 -0.729045  0.196557  ...      3  category_3      3      1   \n2   -0.475175  0.240057  0.100159  ...      4  category_3      4      1   \n3   -0.467441  0.677557  1.307184  ...      1  category_3      4      2   \n4   -0.017020 -0.944446 -0.410050  ...      0  category_2      0      2   \n..        ...       ...       ...  ...    ...         ...    ...    ...   \n251  0.643314  0.049495  0.493837  ...      2  category_2      2      3   \n252  1.178780  1.488252  1.895889  ...      2  category_4      4      2   \n253  0.258798 -0.836111  0.642211  ...      3  category_3      2      2   \n254  0.607112  0.087531  0.938747  ...      0  category_3      4      1   \n255 -0.521580 -0.851729  1.822724  ...      1  category_3      4      1   \n\n     cat_8 cat_9      target  target_binary  target_multi_class  \\\n0        4     3  -71.084217              0                   1   \n1        3     2   13.415675              1                   2   \n2        0     2  -47.492280              0                   2   \n3        0     0  -94.482614              1                   2   \n4        3     0  195.819531              1                   3   \n..     ...   ...         ...            ...                 ...   \n251      0     2 -171.249549              0                   0   \n252      1     1   23.708442              0                   2   \n253      2     2  -33.414215              1                   1   \n254      4     4 -359.199191              0                   4   \n255      1     4 -135.199100              1                   2   \n\n     cont_0_relative2_cont_1  \n0                   3.188552  \n1                  -4.908431  \n2                 -11.614467  \n3                   5.341736  \n4                  -2.221273  \n..                       ...  \n251                -1.355422  \n252                 1.088160  \n253                 0.374183  \n254                 1.199032  \n255                -1.521539  \n\n[256 rows x 24 columns]",
      "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>cont_0</th>\n      <th>cont_1</th>\n      <th>cont_2</th>\n      <th>cont_3</th>\n      <th>cont_4</th>\n      <th>cont_5</th>\n      <th>cont_6</th>\n      <th>cont_7</th>\n      <th>cont_8</th>\n      <th>cont_9</th>\n      <th>...</th>\n      <th>cat_4</th>\n      <th>cat_5</th>\n      <th>cat_6</th>\n      <th>cat_7</th>\n      <th>cat_8</th>\n      <th>cat_9</th>\n      <th>target</th>\n      <th>target_binary</th>\n      <th>target_multi_class</th>\n      <th>cont_0_relative2_cont_1</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>-1.306527</td>\n      <td>-0.409756</td>\n      <td>-0.118164</td>\n      <td>-0.159573</td>\n      <td>1.658131</td>\n      <td>-1.346718</td>\n      <td>-0.680178</td>\n      <td>-1.334258</td>\n      <td>0.666383</td>\n      <td>-0.460720</td>\n      <td>...</td>\n      <td>2</td>\n      <td>category_4</td>\n      <td>3</td>\n      <td>4</td>\n      <td>4</td>\n      <td>3</td>\n      <td>-71.084217</td>\n      <td>0</td>\n      <td>1</td>\n      <td>3.188552</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>2.011257</td>\n      <td>-0.409756</td>\n      <td>0.195070</td>\n      <td>0.527004</td>\n      <td>-0.044595</td>\n      <td>0.616887</td>\n      <td>-1.781563</td>\n      <td>0.354758</td>\n      <td>-0.729045</td>\n      <td>0.196557</td>\n      <td>...</td>\n      <td>3</td>\n      <td>category_3</td>\n      <td>3</td>\n      <td>1</td>\n      <td>3</td>\n      <td>2</td>\n      <td>13.415675</td>\n      <td>1</td>\n      <td>2</td>\n      <td>-4.908431</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>-1.216077</td>\n      <td>0.104704</td>\n      <td>-0.743672</td>\n      <td>0.730184</td>\n      <td>0.140672</td>\n      <td>1.272954</td>\n      <td>-0.159012</td>\n      <td>-0.475175</td>\n      <td>0.240057</td>\n      <td>0.100159</td>\n      <td>...</td>\n      <td>4</td>\n      <td>category_3</td>\n      <td>4</td>\n      <td>1</td>\n      <td>0</td>\n      <td>2</td>\n      <td>-47.492280</td>\n      <td>0</td>\n      <td>2</td>\n      <td>-11.614467</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>0.559299</td>\n      <td>0.104704</td>\n      <td>-0.431096</td>\n      <td>-0.809627</td>\n      <td>-1.063696</td>\n      <td>-0.860153</td>\n      <td>0.572751</td>\n      <td>-0.467441</td>\n      <td>0.677557</td>\n      <td>1.307184</td>\n      <td>...</td>\n      <td>1</td>\n      <td>category_3</td>\n      <td>4</td>\n      <td>2</td>\n      <td>0</td>\n      <td>0</td>\n      <td>-94.482614</td>\n      <td>1</td>\n      <td>2</td>\n      <td>5.341736</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>0.910179</td>\n      <td>-0.409756</td>\n      <td>0.786328</td>\n      <td>-0.042257</td>\n      <td>0.317218</td>\n      <td>0.379152</td>\n      <td>-0.466419</td>\n      <td>-0.017020</td>\n      <td>-0.944446</td>\n      <td>-0.410050</td>\n      <td>...</td>\n      <td>0</td>\n      <td>category_2</td>\n      <td>0</td>\n      <td>2</td>\n      <td>3</td>\n      <td>0</td>\n      <td>195.819531</td>\n      <td>1</td>\n      <td>3</td>\n      <td>-2.221273</td>\n    </tr>\n    <tr>\n      <th>...</th>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n    </tr>\n    <tr>\n      <th>251</th>\n      <td>0.280442</td>\n      <td>-0.206904</td>\n      <td>0.841631</td>\n      <td>0.880179</td>\n      <td>-0.993124</td>\n      <td>-1.570623</td>\n      <td>-0.249459</td>\n      <td>0.643314</td>\n      <td>0.049495</td>\n      <td>0.493837</td>\n      <td>...</td>\n      <td>2</td>\n      <td>category_2</td>\n      <td>2</td>\n      <td>3</td>\n      <td>0</td>\n      <td>2</td>\n      <td>-171.249549</td>\n      <td>0</td>\n      <td>0</td>\n      <td>-1.355422</td>\n    </tr>\n    <tr>\n      <th>252</th>\n      <td>-1.165150</td>\n      <td>-1.070753</td>\n      <td>0.465662</td>\n      <td>1.054452</td>\n      <td>0.900826</td>\n      <td>-0.179925</td>\n      <td>-1.536244</td>\n      <td>1.178780</td>\n      <td>1.488252</td>\n      <td>1.895889</td>\n      <td>...</td>\n      <td>2</td>\n      <td>category_4</td>\n      <td>4</td>\n      <td>2</td>\n      <td>1</td>\n      <td>1</td>\n      <td>23.708442</td>\n      <td>0</td>\n      <td>2</td>\n      <td>1.088160</td>\n    </tr>\n    <tr>\n      <th>253</th>\n      <td>-0.069856</td>\n      <td>-0.186691</td>\n      <td>-1.021913</td>\n      <td>-1.143641</td>\n      <td>0.250114</td>\n      <td>1.040239</td>\n      <td>-1.150438</td>\n      <td>0.258798</td>\n      <td>-0.836111</td>\n      <td>0.642211</td>\n      <td>...</td>\n      <td>3</td>\n      <td>category_3</td>\n      <td>2</td>\n      <td>2</td>\n      <td>2</td>\n      <td>2</td>\n      <td>-33.414215</td>\n      <td>1</td>\n      <td>1</td>\n      <td>0.374183</td>\n    </tr>\n    <tr>\n      <th>254</th>\n      <td>-1.031482</td>\n      <td>-0.860262</td>\n      <td>-0.061638</td>\n      <td>0.328301</td>\n      <td>-1.429991</td>\n      <td>-1.048170</td>\n      <td>-1.432735</td>\n      <td>0.607112</td>\n      <td>0.087531</td>\n      <td>0.938747</td>\n      <td>...</td>\n      <td>0</td>\n      <td>category_3</td>\n      <td>4</td>\n      <td>1</td>\n      <td>4</td>\n      <td>4</td>\n      <td>-359.199191</td>\n      <td>0</td>\n      <td>4</td>\n      <td>1.199032</td>\n    </tr>\n    <tr>\n      <th>255</th>\n      <td>-1.461733</td>\n      <td>0.960693</td>\n      <td>0.367545</td>\n      <td>1.329063</td>\n      <td>-0.683440</td>\n      <td>-1.184687</td>\n      <td>0.190312</td>\n      <td>-0.521580</td>\n      <td>-0.851729</td>\n      <td>1.822724</td>\n      <td>...</td>\n      <td>1</td>\n      <td>category_3</td>\n      <td>4</td>\n      <td>1</td>\n      <td>1</td>\n      <td>4</td>\n      <td>-135.199100</td>\n      <td>1</td>\n      <td>2</td>\n      <td>-1.521539</td>\n    </tr>\n  </tbody>\n</table>\n<p>256 rows × 24 columns</p>\n</div>"
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trainer.datamodule.set_data_derivers([(\"MyRelativeDeriver\", {\"absolute_col\": \"cont_0\", \"relative2_col\": \"cont_1\", \"derived_name\": \"cont_0_relative2_cont_1\", \"some_param\": 1.0, \"stacked\": True, \"intermediate\": True})])\n",
    "trainer.load_data()\n",
    "print(f\"cont_0_relative2_cont_1 in continuous features?: {'cont_0_relative2_cont_1' in trainer.cont_feature_names}\")\n",
    "trainer.df"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "If `stacked` is `False`:"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using previously used data path ../../../../data/sample.csv\n",
      "Dataset size: 153 51 52\n",
      "Data saved to ../../../../output/sample/2023-09-18-18-15-00-0_sample (data.csv and tabular_data.csv).\n",
      "cont_0_relative2_cont_1 in continuous features?: False\n"
     ]
    },
    {
     "data": {
      "text/plain": "       cont_0    cont_1    cont_2    cont_3    cont_4    cont_5    cont_6  \\\n0   -1.306527  0.138315 -0.118164 -0.159573  1.658131 -1.346718 -0.680178   \n1    2.011257 -0.006111  0.195070  0.527004 -0.044595  0.616887 -1.781563   \n2   -1.216077  0.138315 -0.743672  0.730184  0.140672  1.272954 -0.159012   \n3    0.559299 -0.006111 -0.431096 -0.809627 -1.063696 -0.860153  0.572751   \n4    0.910179 -0.006111  0.786328 -0.042257  0.317218  0.379152 -0.466419   \n..        ...       ...       ...       ...       ...       ...       ...   \n251  0.280442 -0.206904  0.841631  0.880179 -0.993124 -1.570623 -0.249459   \n252 -1.165150 -1.070753  0.465662  1.054452  0.900826 -0.179925 -1.536244   \n253 -0.069856 -0.186691 -1.021913 -1.143641  0.250114  1.040239 -1.150438   \n254 -1.031482 -0.860262 -0.061638  0.328301 -1.429991 -1.048170 -1.432735   \n255 -1.461733  0.960693  0.367545  1.329063 -0.683440 -1.184687  0.190312   \n\n       cont_7    cont_8    cont_9  ...  cat_3  cat_4       cat_5  cat_6  \\\n0   -1.334258  0.666383 -0.460720  ...      0      2  category_4      3   \n1    0.354758 -0.729045  0.196557  ...      4      3  category_3      3   \n2   -0.475175  0.240057  0.100159  ...      0      4  category_3      4   \n3   -0.467441  0.677557  1.307184  ...      4      1  category_3      4   \n4   -0.017020 -0.944446 -0.410050  ...      1      0  category_2      0   \n..        ...       ...       ...  ...    ...    ...         ...    ...   \n251  0.643314  0.049495  0.493837  ...      1      2  category_2      2   \n252  1.178780  1.488252  1.895889  ...      4      2  category_4      4   \n253  0.258798 -0.836111  0.642211  ...      0      3  category_3      2   \n254  0.607112  0.087531  0.938747  ...      0      0  category_3      4   \n255 -0.521580 -0.851729  1.822724  ...      2      1  category_3      4   \n\n     cat_7 cat_8  cat_9      target  target_binary  target_multi_class  \n0        4     4      3  -71.084217              0                   1  \n1        1     3      2   13.415675              1                   2  \n2        1     0      2  -47.492280              0                   2  \n3        2     0      0  -94.482614              1                   2  \n4        2     3      0  195.819531              1                   3  \n..     ...   ...    ...         ...            ...                 ...  \n251      3     0      2 -171.249549              0                   0  \n252      2     1      1   23.708442              0                   2  \n253      2     2      2  -33.414215              1                   1  \n254      1     4      4 -359.199191              0                   4  \n255      1     1      4 -135.199100              1                   2  \n\n[256 rows x 23 columns]",
      "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>cont_0</th>\n      <th>cont_1</th>\n      <th>cont_2</th>\n      <th>cont_3</th>\n      <th>cont_4</th>\n      <th>cont_5</th>\n      <th>cont_6</th>\n      <th>cont_7</th>\n      <th>cont_8</th>\n      <th>cont_9</th>\n      <th>...</th>\n      <th>cat_3</th>\n      <th>cat_4</th>\n      <th>cat_5</th>\n      <th>cat_6</th>\n      <th>cat_7</th>\n      <th>cat_8</th>\n      <th>cat_9</th>\n      <th>target</th>\n      <th>target_binary</th>\n      <th>target_multi_class</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>-1.306527</td>\n      <td>0.138315</td>\n      <td>-0.118164</td>\n      <td>-0.159573</td>\n      <td>1.658131</td>\n      <td>-1.346718</td>\n      <td>-0.680178</td>\n      <td>-1.334258</td>\n      <td>0.666383</td>\n      <td>-0.460720</td>\n      <td>...</td>\n      <td>0</td>\n      <td>2</td>\n      <td>category_4</td>\n      <td>3</td>\n      <td>4</td>\n      <td>4</td>\n      <td>3</td>\n      <td>-71.084217</td>\n      <td>0</td>\n      <td>1</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>2.011257</td>\n      <td>-0.006111</td>\n      <td>0.195070</td>\n      <td>0.527004</td>\n      <td>-0.044595</td>\n      <td>0.616887</td>\n      <td>-1.781563</td>\n      <td>0.354758</td>\n      <td>-0.729045</td>\n      <td>0.196557</td>\n      <td>...</td>\n      <td>4</td>\n      <td>3</td>\n      <td>category_3</td>\n      <td>3</td>\n      <td>1</td>\n      <td>3</td>\n      <td>2</td>\n      <td>13.415675</td>\n      <td>1</td>\n      <td>2</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>-1.216077</td>\n      <td>0.138315</td>\n      <td>-0.743672</td>\n      <td>0.730184</td>\n      <td>0.140672</td>\n      <td>1.272954</td>\n      <td>-0.159012</td>\n      <td>-0.475175</td>\n      <td>0.240057</td>\n      <td>0.100159</td>\n      <td>...</td>\n      <td>0</td>\n      <td>4</td>\n      <td>category_3</td>\n      <td>4</td>\n      <td>1</td>\n      <td>0</td>\n      <td>2</td>\n      <td>-47.492280</td>\n      <td>0</td>\n      <td>2</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>0.559299</td>\n      <td>-0.006111</td>\n      <td>-0.431096</td>\n      <td>-0.809627</td>\n      <td>-1.063696</td>\n      <td>-0.860153</td>\n      <td>0.572751</td>\n      <td>-0.467441</td>\n      <td>0.677557</td>\n      <td>1.307184</td>\n      <td>...</td>\n      <td>4</td>\n      <td>1</td>\n      <td>category_3</td>\n      <td>4</td>\n      <td>2</td>\n      <td>0</td>\n      <td>0</td>\n      <td>-94.482614</td>\n      <td>1</td>\n      <td>2</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>0.910179</td>\n      <td>-0.006111</td>\n      <td>0.786328</td>\n      <td>-0.042257</td>\n      <td>0.317218</td>\n      <td>0.379152</td>\n      <td>-0.466419</td>\n      <td>-0.017020</td>\n      <td>-0.944446</td>\n      <td>-0.410050</td>\n      <td>...</td>\n      <td>1</td>\n      <td>0</td>\n      <td>category_2</td>\n      <td>0</td>\n      <td>2</td>\n      <td>3</td>\n      <td>0</td>\n      <td>195.819531</td>\n      <td>1</td>\n      <td>3</td>\n    </tr>\n    <tr>\n      <th>...</th>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n    </tr>\n    <tr>\n      <th>251</th>\n      <td>0.280442</td>\n      <td>-0.206904</td>\n      <td>0.841631</td>\n      <td>0.880179</td>\n      <td>-0.993124</td>\n      <td>-1.570623</td>\n      <td>-0.249459</td>\n      <td>0.643314</td>\n      <td>0.049495</td>\n      <td>0.493837</td>\n      <td>...</td>\n      <td>1</td>\n      <td>2</td>\n      <td>category_2</td>\n      <td>2</td>\n      <td>3</td>\n      <td>0</td>\n      <td>2</td>\n      <td>-171.249549</td>\n      <td>0</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>252</th>\n      <td>-1.165150</td>\n      <td>-1.070753</td>\n      <td>0.465662</td>\n      <td>1.054452</td>\n      <td>0.900826</td>\n      <td>-0.179925</td>\n      <td>-1.536244</td>\n      <td>1.178780</td>\n      <td>1.488252</td>\n      <td>1.895889</td>\n      <td>...</td>\n      <td>4</td>\n      <td>2</td>\n      <td>category_4</td>\n      <td>4</td>\n      <td>2</td>\n      <td>1</td>\n      <td>1</td>\n      <td>23.708442</td>\n      <td>0</td>\n      <td>2</td>\n    </tr>\n    <tr>\n      <th>253</th>\n      <td>-0.069856</td>\n      <td>-0.186691</td>\n      <td>-1.021913</td>\n      <td>-1.143641</td>\n      <td>0.250114</td>\n      <td>1.040239</td>\n      <td>-1.150438</td>\n      <td>0.258798</td>\n      <td>-0.836111</td>\n      <td>0.642211</td>\n      <td>...</td>\n      <td>0</td>\n      <td>3</td>\n      <td>category_3</td>\n      <td>2</td>\n      <td>2</td>\n      <td>2</td>\n      <td>2</td>\n      <td>-33.414215</td>\n      <td>1</td>\n      <td>1</td>\n    </tr>\n    <tr>\n      <th>254</th>\n      <td>-1.031482</td>\n      <td>-0.860262</td>\n      <td>-0.061638</td>\n      <td>0.328301</td>\n      <td>-1.429991</td>\n      <td>-1.048170</td>\n      <td>-1.432735</td>\n      <td>0.607112</td>\n      <td>0.087531</td>\n      <td>0.938747</td>\n      <td>...</td>\n      <td>0</td>\n      <td>0</td>\n      <td>category_3</td>\n      <td>4</td>\n      <td>1</td>\n      <td>4</td>\n      <td>4</td>\n      <td>-359.199191</td>\n      <td>0</td>\n      <td>4</td>\n    </tr>\n    <tr>\n      <th>255</th>\n      <td>-1.461733</td>\n      <td>0.960693</td>\n      <td>0.367545</td>\n      <td>1.329063</td>\n      <td>-0.683440</td>\n      <td>-1.184687</td>\n      <td>0.190312</td>\n      <td>-0.521580</td>\n      <td>-0.851729</td>\n      <td>1.822724</td>\n      <td>...</td>\n      <td>2</td>\n      <td>1</td>\n      <td>category_3</td>\n      <td>4</td>\n      <td>1</td>\n      <td>1</td>\n      <td>4</td>\n      <td>-135.199100</td>\n      <td>1</td>\n      <td>2</td>\n    </tr>\n  </tbody>\n</table>\n<p>256 rows × 23 columns</p>\n</div>"
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trainer.datamodule.set_data_derivers([(\"MyRelativeDeriver\", {\"absolute_col\": \"cont_0\", \"relative2_col\": \"cont_1\", \"derived_name\": \"cont_0_relative2_cont_1\", \"some_param\": 1.0, \"stacked\": False})])\n",
    "trainer.load_data()\n",
    "print(f\"cont_0_relative2_cont_1 in continuous features?: {'cont_0_relative2_cont_1' in trainer.cont_feature_names}\")\n",
    "trainer.df"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "outputs": [
    {
     "data": {
      "text/plain": "dict_keys(['cont_0_relative2_cont_1', 'categorical'])"
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trainer.derived_data.keys()"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}