{ "cells": [ { "cell_type": "markdown", "source": [ "# Build your own model upon others\n", "\n", "Models can be built based on other trained models in the current model base or in other model bases. Both `AbstractModel` and `TorchModel` support this feature.\n", "\n", "## For `AbstractModel`" ], "metadata": { "collapsed": false, "pycharm": { "name": "#%% md\n" } } }, { "cell_type": "code", "execution_count": 1, "metadata": { "collapsed": true, "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [ "import tabensemb\n", "import numpy as np\n", "import torch\n", "import os\n", "from tempfile import TemporaryDirectory\n", "from tabensemb.model import WideDeep, AbstractModel\n", "\n", "temp_path = TemporaryDirectory()\n", "tabensemb.setting[\"default_output_path\"] = os.path.join(temp_path.name, \"output\")\n", "tabensemb.setting[\"default_config_path\"] = os.path.join(temp_path.name, \"configs\")\n", "tabensemb.setting[\"default_data_path\"] = os.path.join(temp_path.name, \"data\")\n", "\n", "device = \"cuda\" if torch.cuda.is_available() else \"cpu\"" ] }, { "cell_type": "markdown", "source": [ "Suppose that we want to call TabMlp of WideDeep in another model base `CallTabMlp`\n", "\n", "```python\n", "class CallTabMlp(AbstractModel):\n", " def _get_program_name(self):\n", " return \"CallTabMlp\"\n", "\n", " def _get_model_names(self):\n", " return [\"CalledTabMlp\"]\n", "\n", " def _space(self, model_name):\n", " return []\n", "\n", " def _initial_values(self, model_name):\n", " return {}\n", "```\n", "\n", "Extracting another model can be done by setting `required_models` in a specific format. In the following code, \"EXTERN\" means that the model is from another model base. \"WideDeep\" is the name of the model base which the wanted model is from. \"TabMlp\" is the wanted model in the model base. If the model is from the current model base, only the name of the wanted model is needed (`return [\"TabMlp\"]`). Multiple required models can be specified in the returned list.\n", "\n", "```python\n", " def required_models(self, model_name: str):\n", " return [\"EXTERN_WideDeep_TabMlp\"]\n", "```\n", "\n", "As normal, `_train_data_preprocess`, `_data_preprocess`, `_new_model`, `_train_single_model`, and `_pred_single_model` should be implemented. First, `_train_data_preprocess` is called, and `_get_required_models` is used to extract the external model. In this case, a `WideDeep` instance containing the trained TabMlp model is returned. If the model is from the current model base, calling `self._get_required_models(\"TabMlp\")` is equivalent to calling `self.model[\"TabMlp\"]`.\n", "\n", "Then the `_train_data_preprocess` method from `WideDeep` is directly used to process the dataset to get compatible processed data.\n", "\n", "```python\n", " def _train_data_preprocess(self, model_name):\n", " if not hasattr(self, \"net\"):\n", " self.net = self._get_required_models(\"TabMlp\")[\"EXTERN_WideDeep_TabMlp\"]\n", " self.net.trainer = self.trainer\n", " return self.net._train_data_preprocess(\"TabMlp\")\n", "```\n", "\n", "Also, `_data_preprocess` calls the same method from `WideDeep` instead to get compatible processed data.\n", "\n", "```python\n", " def _data_preprocess(self, df, derived_data, model_name):\n", " return self.net._data_preprocess(df, derived_data, \"TabMlp\")\n", "```\n", "\n", "In `_new_model`, the extracted model is directly returned.\n", "\n", "```python\n", " def _new_model(self, model_name, verbose, **kwargs):\n", " return self.net\n", "```\n", "\n", "`_pred_single_model` calls the same method from `WideDeep` to make predictions based on the extracted model.\n", "\n", "```python\n", " def _pred_single_model(self, model, X_test, verbose, **kwargs):\n", " return model._pred_single_model(model.model[\"TabMlp\"], X_test, verbose, **kwargs)\n", "```\n", "\n", "In this example, we won't do further training on the extracted model, but it is straightforward to do other operations on the predictions from the extracted model obtained by `model._pred_single_model` as shown above.\n", "\n", "```python\n", " def _train_single_model(self, *args, **kwargs):\n", " pass\n", "```" ], "metadata": { "collapsed": false, "pycharm": { "name": "#%% md\n" } } }, { "cell_type": "code", "execution_count": 2, "outputs": [], "source": [ "class CallTabMlp(AbstractModel):\n", " def _get_program_name(self):\n", " return \"CallTabMlp\"\n", "\n", " def _get_model_names(self):\n", " return [\"TabMlp\"]\n", "\n", " def _space(self, model_name):\n", " return []\n", "\n", " def _initial_values(self, model_name):\n", " return {}\n", "\n", " def required_models(self, model_name: str):\n", " return [\"EXTERN_WideDeep_TabMlp\"]\n", "\n", " def _train_data_preprocess(self, model_name):\n", " if not hasattr(self, \"net\"):\n", " self.net = self._get_required_models(\"TabMlp\")[\"EXTERN_WideDeep_TabMlp\"]\n", " self.net.trainer = self.trainer\n", " return self.net._train_data_preprocess(\"TabMlp\")\n", "\n", " def _data_preprocess(self, df, derived_data, model_name):\n", " return self.net._data_preprocess(df, derived_data, \"TabMlp\")\n", "\n", " def _new_model(self, model_name, verbose, **kwargs):\n", " return self.net\n", "\n", " def _train_single_model(self, *args, **kwargs):\n", " pass\n", "\n", " def _pred_single_model(self, model, X_test, verbose, **kwargs):\n", " return model._pred_single_model(model.model[\"TabMlp\"], X_test, verbose, **kwargs)" ], "metadata": { "collapsed": false, "pycharm": { "name": "#%%\n" } } }, { "cell_type": "markdown", "source": [ "## For `TorchModel`\n", "\n", "It is easier to build a model based on others in `TorchModel` because we have already implemented complex dataset-building operations internally.\n", "\n", "Similar to the implementation above, we specify methods except for `_train_data_preprocess` and `_data_preprocess`.\n", "\n", "```python\n", "class CallTabMlpTorch(TorchModel):\n", " def _get_program_name(self):\n", " return \"CallTabMlpTorch\"\n", "\n", " def _get_model_names(self):\n", " return [\"TabMlp\"]\n", "\n", " def required_models(self, model_name: str):\n", " return [\"EXTERN_WideDeep_TabMlp\"]\n", "\n", " def _space(self, model_name):\n", " return []\n", "\n", " def _initial_values(self, model_name):\n", " return {}\n", "```\n", "\n", "We build our model `CallTabMlpNN` on the top of TabMlp from WideDeep. In this tutorial, we will not train anything.\n", "\n", "```python\n", " def _new_model(self, model_name, verbose, **kwargs):\n", " return CallTabMlpNN(datamodule=self.trainer.datamodule, **kwargs)\n", "\n", " def _train_single_model(self, *args, **kwargs):\n", " pass\n", "```\n", "\n", "Now comes `CallTabMlpNN`. A positional argument `required_models` is passed to `__init__` containing all required and extracted models specified in `CallTabMlpTorch.required_models`.\n", "\n", "```python\n", "class CallTabMlpNN(AbstractNN):\n", " def __init__(self, datamodule, required_models, **kwargs):\n", " super(CallTabMlpNN, self).__init__(datamodule, **kwargs)\n", " self.net = required_models[\"EXTERN_WideDeep_TabMlp\"]\n", "```\n", "\n", "To get results from the extracted model, use `self.call_required_model`.\n", "\n", "```python\n", " def _forward(self, x: torch.Tensor, derived_tensors) -> torch.Tensor:\n", " return self.call_required_model(self.net, x, derived_tensors)\n", "```\n", "\n", "**Remark**: Indeed, the output of the model is already calculated when preparing the dataset and is stored in `derived_tensors[\"data_required_models\"][\"MODELNAME_pred\"]`. `self.call_required_model` first tries to find the pre-calculated output. If failed, the output is calculated using the dataset for the model base stored in `derived_tensors[\"data_required_models\"][\"MODELNAME\"]`. Therefore, if you want to actually calculate the output during `forward`, just remove the stored predictions in `derived_tensors`." ], "metadata": { "collapsed": false, "pycharm": { "name": "#%% md\n" } } }, { "cell_type": "code", "execution_count": 3, "outputs": [], "source": [ "from tabensemb.model import TorchModel, AbstractNN\n", "\n", "class CallTabMlpNN(AbstractNN):\n", " def __init__(self, datamodule, required_models, **kwargs):\n", " super(CallTabMlpNN, self).__init__(datamodule, **kwargs)\n", " self.net = required_models[\"EXTERN_WideDeep_TabMlp\"]\n", "\n", " def _forward(self, x: torch.Tensor, derived_tensors) -> torch.Tensor:\n", " return self.call_required_model(self.net, x, derived_tensors)\n", "\n", "class CallTabMlpTorch(TorchModel):\n", " def _new_model(self, model_name, verbose, **kwargs):\n", " return CallTabMlpNN(datamodule=self.trainer.datamodule, **kwargs)\n", "\n", " def _get_program_name(self):\n", " return \"CallTabMlpTorch\"\n", "\n", " def _get_model_names(self):\n", " return [\"TabMlp\"]\n", "\n", " def required_models(self, model_name: str):\n", " return [\"EXTERN_WideDeep_TabMlp\"]\n", "\n", " def _space(self, model_name):\n", " return []\n", "\n", " def _initial_values(self, model_name):\n", " return {}\n", "\n", " def _train_single_model(self, *args, **kwargs):\n", " pass" ], "metadata": { "collapsed": false, "pycharm": { "name": "#%%\n" } } }, { "cell_type": "markdown", "source": [ "We can compare results from the original model and the extracted model. They get exactly the same results." ], "metadata": { "collapsed": false, "pycharm": { "name": "#%% md\n" } } }, { "cell_type": "code", "execution_count": 4, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Downloading https://archive.ics.uci.edu/static/public/9/auto+mpg.zip to /tmp/tmpvlx3s8em/data/Auto MPG.zip\n", "cylinders is Integer and will be treated as a continuous feature.\n", "model_year is Integer and will be treated as a continuous feature.\n", "origin is Integer and will be treated as a continuous feature.\n", "Unknown values are detected in ['horsepower']. They will be treated as np.nan.\n", "The project will be saved to /tmp/tmpvlx3s8em/output/auto-mpg/2023-09-23-20-41-06-0_UserInputConfig\n", "Dataset size: 238 80 80\n", "Data saved to /tmp/tmpvlx3s8em/output/auto-mpg/2023-09-23-20-41-06-0_UserInputConfig (data.csv and tabular_data.csv).\n", "\n", "-------------Run WideDeep-------------\n", "\n", "Training TabMlp\n", "Epoch: 1/300, Train loss: 635.5330, Val loss: 555.4755, Min val loss: 555.4755\n", "Epoch: 21/300, Train loss: 441.6902, Val loss: 375.7337, Min val loss: 375.7337\n", "Epoch: 41/300, Train loss: 145.8623, Val loss: 119.9598, Min val loss: 119.9598\n", "Epoch: 61/300, Train loss: 45.9133, Val loss: 34.0160, Min val loss: 34.0160\n", "Epoch: 81/300, Train loss: 27.6878, Val loss: 24.1525, Min val loss: 24.1525\n", "Epoch: 101/300, Train loss: 23.0877, Val loss: 18.2096, Min val loss: 18.2096\n", "Epoch: 121/300, Train loss: 21.4056, Val loss: 17.2203, Min val loss: 17.1303\n", "Epoch: 141/300, Train loss: 21.2559, Val loss: 16.0746, Min val loss: 16.0746\n", "Epoch: 161/300, Train loss: 19.2337, Val loss: 15.3027, Min val loss: 15.3027\n", "Epoch: 181/300, Train loss: 16.1232, Val loss: 14.5777, Min val loss: 14.5777\n", "Epoch: 201/300, Train loss: 16.7095, Val loss: 14.2274, Min val loss: 14.2274\n", "Epoch: 221/300, Train loss: 15.7366, Val loss: 13.5223, Min val loss: 13.5223\n", "Epoch: 241/300, Train loss: 16.9825, Val loss: 12.9892, Min val loss: 12.9892\n", "Epoch: 261/300, Train loss: 15.3358, Val loss: 12.4278, Min val loss: 12.4278\n", "Epoch: 281/300, Train loss: 13.3989, Val loss: 12.1155, Min val loss: 12.1155\n", "Restoring model weights from the end of the best epoch\n", "Training mse loss: 10.17037\n", "Validation mse loss: 11.66271\n", "Testing mse loss: 6.43856\n", "Trainer saved. To load the trainer, run trainer = load_trainer(path='/tmp/tmpvlx3s8em/output/auto-mpg/2023-09-23-20-41-06-0_UserInputConfig/trainer.pkl')\n", "\n", "-------------WideDeep End-------------\n", "\n", "\n", "-------------Run CallTabMlp-------------\n", "\n", "Training TabMlp\n", "Training mse loss: 10.17037\n", "Validation mse loss: 11.66271\n", "Testing mse loss: 6.43856\n", "Trainer saved. To load the trainer, run trainer = load_trainer(path='/tmp/tmpvlx3s8em/output/auto-mpg/2023-09-23-20-41-06-0_UserInputConfig/trainer.pkl')\n", "\n", "-------------CallTabMlp End-------------\n", "\n", "\n", "-------------Run CallTabMlpTorch-------------\n", "\n", "Training TabMlp\n", "Training mse loss: 10.17037\n", "Validation mse loss: 11.66271\n", "Testing mse loss: 6.43856\n", "Trainer saved. To load the trainer, run trainer = load_trainer(path='/tmp/tmpvlx3s8em/output/auto-mpg/2023-09-23-20-41-06-0_UserInputConfig/trainer.pkl')\n", "\n", "-------------CallTabMlpTorch End-------------\n", "\n", "WideDeep metrics\n", "TabMlp 1/1\n", "CallTabMlp metrics\n", "TabMlp 1/1\n", "CallTabMlpTorch metrics\n", "TabMlp 1/1\n", "Trainer saved. To load the trainer, run trainer = load_trainer(path='/tmp/tmpvlx3s8em/output/auto-mpg/2023-09-23-20-41-06-0_UserInputConfig/trainer.pkl')\n" ] }, { "data": { "text/plain": " Program Model Training RMSE Training MSE Training MAE \\\n0 WideDeep TabMlp 3.189102 10.170372 2.318564 \n1 CallTabMlp TabMlp 3.189102 10.170372 2.318564 \n2 CallTabMlpTorch TabMlp 3.189102 10.170372 2.318564 \n\n Training MAPE Training R2 Training MEDIAN_ABSOLUTE_ERROR \\\n0 0.096454 0.842218 1.669983 \n1 0.096454 0.842218 1.669983 \n2 0.096454 0.842218 1.669983 \n\n Training EXPLAINED_VARIANCE_SCORE Testing RMSE ... Testing R2 \\\n0 0.859805 2.537431 ... 0.88025 \n1 0.859805 2.537431 ... 0.88025 \n2 0.859805 2.537431 ... 0.88025 \n\n Testing MEDIAN_ABSOLUTE_ERROR Testing EXPLAINED_VARIANCE_SCORE \\\n0 1.767459 0.900587 \n1 1.767459 0.900587 \n2 1.767459 0.900587 \n\n Validation RMSE Validation MSE Validation MAE Validation MAPE \\\n0 3.415071 11.662707 2.539188 0.116035 \n1 3.415071 11.662707 2.539188 0.116035 \n2 3.415071 11.662707 2.539188 0.116035 \n\n Validation R2 Validation MEDIAN_ABSOLUTE_ERROR \\\n0 0.791657 1.90416 \n1 0.791657 1.90416 \n2 0.791657 1.90416 \n\n Validation EXPLAINED_VARIANCE_SCORE \n0 0.806152 \n1 0.806152 \n2 0.806152 \n\n[3 rows x 23 columns]", "text/html": "
\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
ProgramModelTraining RMSETraining MSETraining MAETraining MAPETraining R2Training MEDIAN_ABSOLUTE_ERRORTraining EXPLAINED_VARIANCE_SCORETesting RMSE...Testing R2Testing MEDIAN_ABSOLUTE_ERRORTesting EXPLAINED_VARIANCE_SCOREValidation RMSEValidation MSEValidation MAEValidation MAPEValidation R2Validation MEDIAN_ABSOLUTE_ERRORValidation EXPLAINED_VARIANCE_SCORE
0WideDeepTabMlp3.18910210.1703722.3185640.0964540.8422181.6699830.8598052.537431...0.880251.7674590.9005873.41507111.6627072.5391880.1160350.7916571.904160.806152
1CallTabMlpTabMlp3.18910210.1703722.3185640.0964540.8422181.6699830.8598052.537431...0.880251.7674590.9005873.41507111.6627072.5391880.1160350.7916571.904160.806152
2CallTabMlpTorchTabMlp3.18910210.1703722.3185640.0964540.8422181.6699830.8598052.537431...0.880251.7674590.9005873.41507111.6627072.5391880.1160350.7916571.904160.806152
\n

3 rows × 23 columns

\n
" }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from tabensemb.trainer import Trainer\n", "from tabensemb.config import UserConfig\n", "\n", "trainer = Trainer(device=device)\n", "mpg_columns = [\n", " \"mpg\",\n", " \"cylinders\",\n", " \"displacement\",\n", " \"horsepower\",\n", " \"weight\",\n", " \"acceleration\",\n", " \"model_year\",\n", " \"origin\",\n", " \"car_name\",\n", "]\n", "cfg = UserConfig.from_uci(\"Auto MPG\", column_names=mpg_columns, sep=r\"\\s+\")\n", "trainer.load_config(cfg)\n", "trainer.load_data()\n", "trainer.add_modelbases(\n", " [\n", " WideDeep(trainer, model_subset=[\"TabMlp\"]),\n", " CallTabMlp(trainer),\n", " CallTabMlpTorch(trainer),\n", " ]\n", ")\n", "trainer.train(stderr_to_stdout=True)\n", "trainer.get_leaderboard()" ], "metadata": { "collapsed": false, "pycharm": { "name": "#%%\n" } } }, { "cell_type": "markdown", "source": [ "## Extract learned hidden representation from models\n", "\n", "The original correlation among input features and targets can be complex, especially for high dimensional inputs and multimodal inputs, which is why we want deep learning models to extract the internal relations and reduce the dimension. For most deep learning models, no matter what the backbone structure is, the output of the backbone is normally a low dimension tensor (for instance, `(batch_size, 16)`), which contains learned information from the deep learning model, so we name it \"hidden representation\" of the deep learning model. The hidden representation will be projected to the output dimension through a linear layer, an MLP, etc.\n", "\n", "Most models in two model bases, `pytorch_widedeep` (WideDeep) and `pytorch_tabular` (PyTorchTabular), are supported to extract hidden representations in an `AbstractNN`.\n", "\n", "To use this functionality, first, change the name in `required_models`. A postfix \"_WRAP\" is added.\n", "\n", "```python\n", "class CallTabMlpTorchWrapped(CallTabMlpTorch):\n", " def required_models(self, model_name: str):\n", " return [\"EXTERN_WideDeep_TabMlp_WRAP\"]\n", "\n", " def _get_program_name(self):\n", " return \"CallTabMlpTorchWrapped\"\n", "\n", " def _new_model(self, model_name, verbose, **kwargs):\n", " return CallTabMlpNNWrapped(datamodule=self.trainer.datamodule, **kwargs)\n", "```\n", "\n", "Now more operations can be done in the `AbstractNN`. In `__init__`, `_test_required_model` can be used to check the validity of hidden representations and get its dimension to further generate `nn.Module`s like a linear layer or MLP.\n", "\n", "```python\n", "class CallTabMlpNNWrapped(AbstractNN):\n", " def __init__(self, datamodule, required_models, **kwargs):\n", " super(CallTabMlpNNWrapped, self).__init__(datamodule, **kwargs)\n", " print(required_models)\n", " self.net = required_models[\"EXTERN_WideDeep_TabMlp_WRAP\"]\n", " self.use_hidden_rep, hidden_rep_dim = self._test_required_model(\n", " self.n_inputs, self.net\n", " )\n", " print(f\"Does the model support extracting hidden representation?: {self.use_hidden_rep}\")\n", " print(f\"The dimension of the hidden representation: {hidden_rep_dim}\")\n", "```\n", "\n", "When doing forward propagation, the hidden representation can be extracted using `get_hidden_state`.\n", "\n", "```python\n", " def _forward(self, x: torch.Tensor, derived_tensors) -> torch.Tensor:\n", " print(derived_tensors[\"data_required_models\"].keys())\n", " output = self.call_required_model(self.net, x, derived_tensors)\n", " hidden = self.get_hidden_state(self.net, x, derived_tensors)\n", " print(f\"The dimensions of the batched hidden representation: {hidden.shape}\")\n", " return output\n", "```\n", "\n", "**Remark**: If the model does not support extracting hidden representations, `self.use_hidden_rep` shown above will be `False`, and `hidden_rep_dim` will be `self.n_inputs + 1`, which suggests concatenating continuous features and the output of the model.\n", "\n", "**Remark**: Same as the output, the hidden representation is calculated when preparing the dataset and is stored in `derived_tensors[\"data_required_models\"][\"MODELNAME_hidden\"]`." ], "metadata": { "collapsed": false, "pycharm": { "name": "#%% md\n" } } }, { "cell_type": "code", "execution_count": 5, "outputs": [], "source": [ "from tabensemb.model import TorchModel, AbstractNN\n", "\n", "class CallTabMlpNNWrapped(AbstractNN):\n", " def __init__(self, datamodule, required_models, **kwargs):\n", " super(CallTabMlpNNWrapped, self).__init__(datamodule, **kwargs)\n", " print(required_models)\n", " self.net = required_models[\"EXTERN_WideDeep_TabMlp_WRAP\"]\n", " self.use_hidden_rep, hidden_rep_dim = self._test_required_model(\n", " self.n_inputs, self.net\n", " )\n", " print(f\"Does the model support extracting hidden representation?: {self.use_hidden_rep}\")\n", " print(f\"The dimension of the hidden representation: {hidden_rep_dim}\")\n", "\n", " def _forward(self, x: torch.Tensor, derived_tensors) -> torch.Tensor:\n", " print(derived_tensors[\"data_required_models\"].keys())\n", " output = self.call_required_model(self.net, x, derived_tensors)\n", " hidden = self.get_hidden_state(self.net, x, derived_tensors)\n", " print(f\"The dimensions of the batched hidden representation: {hidden.shape}\")\n", " return output\n", "\n", "class CallTabMlpTorchWrapped(CallTabMlpTorch):\n", " def _get_program_name(self):\n", " return \"CallTabMlpTorchWrapped\"\n", "\n", " def _new_model(self, model_name, verbose, **kwargs):\n", " return CallTabMlpNNWrapped(datamodule=self.trainer.datamodule, **kwargs)\n", "\n", " def required_models(self, model_name: str):\n", " return [\"EXTERN_WideDeep_TabMlp_WRAP\"]" ], "metadata": { "collapsed": false, "pycharm": { "name": "#%%\n" } } }, { "cell_type": "markdown", "source": [ "We can show the information of the extracted model, the hidden representation, and the stored data and predictions." ], "metadata": { "collapsed": false, "pycharm": { "name": "#%% md\n" } } }, { "cell_type": "code", "execution_count": 6, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "-------------Run CallTabMlpTorchWrapped-------------\n", "\n", "Training TabMlp\n", "{'EXTERN_WideDeep_TabMlp_WRAP': }\n", "Does the model support extracting hidden representation?: True\n", "The dimension of the hidden representation: 100\n", "dict_keys(['EXTERN_WideDeep_TabMlp', 'EXTERN_WideDeep_TabMlp_pred', 'EXTERN_WideDeep_TabMlp_hidden'])\n", "The dimensions of the batched hidden representation: torch.Size([238, 100])\n", "Training mse loss: 10.17037\n", "dict_keys(['EXTERN_WideDeep_TabMlp', 'EXTERN_WideDeep_TabMlp_pred', 'EXTERN_WideDeep_TabMlp_hidden'])\n", "The dimensions of the batched hidden representation: torch.Size([80, 100])\n", "Validation mse loss: 11.66271\n", "dict_keys(['EXTERN_WideDeep_TabMlp', 'EXTERN_WideDeep_TabMlp_pred', 'EXTERN_WideDeep_TabMlp_hidden'])\n", "The dimensions of the batched hidden representation: torch.Size([80, 100])\n", "Testing mse loss: 6.43856\n", "Trainer saved. To load the trainer, run trainer = load_trainer(path='/tmp/tmpvlx3s8em/output/auto-mpg/2023-09-23-20-41-06-0_UserInputConfig/trainer.pkl')\n", "\n", "-------------CallTabMlpTorchWrapped End-------------\n", "\n", "WideDeep metrics\n", "TabMlp 1/1\n", "CallTabMlp metrics\n", "TabMlp 1/1\n", "CallTabMlpTorch metrics\n", "TabMlp 1/1\n", "CallTabMlpTorchWrapped metrics\n", "TabMlp 1/1\n", "dict_keys(['EXTERN_WideDeep_TabMlp', 'EXTERN_WideDeep_TabMlp_pred', 'EXTERN_WideDeep_TabMlp_hidden'])\n", "The dimensions of the batched hidden representation: torch.Size([238, 100])\n", "dict_keys(['EXTERN_WideDeep_TabMlp', 'EXTERN_WideDeep_TabMlp_pred', 'EXTERN_WideDeep_TabMlp_hidden'])\n", "The dimensions of the batched hidden representation: torch.Size([80, 100])\n", "dict_keys(['EXTERN_WideDeep_TabMlp', 'EXTERN_WideDeep_TabMlp_pred', 'EXTERN_WideDeep_TabMlp_hidden'])\n", "The dimensions of the batched hidden representation: torch.Size([80, 100])\n", "Trainer saved. To load the trainer, run trainer = load_trainer(path='/tmp/tmpvlx3s8em/output/auto-mpg/2023-09-23-20-41-06-0_UserInputConfig/trainer.pkl')\n" ] }, { "data": { "text/plain": " Program Model Training RMSE Training MSE Training MAE \\\n0 WideDeep TabMlp 3.189102 10.170372 2.318564 \n1 CallTabMlp TabMlp 3.189102 10.170372 2.318564 \n2 CallTabMlpTorch TabMlp 3.189102 10.170372 2.318564 \n3 CallTabMlpTorchWrapped TabMlp 3.189102 10.170372 2.318564 \n\n Training MAPE Training R2 Training MEDIAN_ABSOLUTE_ERROR \\\n0 0.096454 0.842218 1.669983 \n1 0.096454 0.842218 1.669983 \n2 0.096454 0.842218 1.669983 \n3 0.096454 0.842218 1.669983 \n\n Training EXPLAINED_VARIANCE_SCORE Testing RMSE ... Testing R2 \\\n0 0.859805 2.537431 ... 0.88025 \n1 0.859805 2.537431 ... 0.88025 \n2 0.859805 2.537431 ... 0.88025 \n3 0.859805 2.537431 ... 0.88025 \n\n Testing MEDIAN_ABSOLUTE_ERROR Testing EXPLAINED_VARIANCE_SCORE \\\n0 1.767459 0.900587 \n1 1.767459 0.900587 \n2 1.767459 0.900587 \n3 1.767459 0.900587 \n\n Validation RMSE Validation MSE Validation MAE Validation MAPE \\\n0 3.415071 11.662707 2.539188 0.116035 \n1 3.415071 11.662707 2.539188 0.116035 \n2 3.415071 11.662707 2.539188 0.116035 \n3 3.415071 11.662707 2.539188 0.116035 \n\n Validation R2 Validation MEDIAN_ABSOLUTE_ERROR \\\n0 0.791657 1.90416 \n1 0.791657 1.90416 \n2 0.791657 1.90416 \n3 0.791657 1.90416 \n\n Validation EXPLAINED_VARIANCE_SCORE \n0 0.806152 \n1 0.806152 \n2 0.806152 \n3 0.806152 \n\n[4 rows x 23 columns]", "text/html": "
\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
ProgramModelTraining RMSETraining MSETraining MAETraining MAPETraining R2Training MEDIAN_ABSOLUTE_ERRORTraining EXPLAINED_VARIANCE_SCORETesting RMSE...Testing R2Testing MEDIAN_ABSOLUTE_ERRORTesting EXPLAINED_VARIANCE_SCOREValidation RMSEValidation MSEValidation MAEValidation MAPEValidation R2Validation MEDIAN_ABSOLUTE_ERRORValidation EXPLAINED_VARIANCE_SCORE
0WideDeepTabMlp3.18910210.1703722.3185640.0964540.8422181.6699830.8598052.537431...0.880251.7674590.9005873.41507111.6627072.5391880.1160350.7916571.904160.806152
1CallTabMlpTabMlp3.18910210.1703722.3185640.0964540.8422181.6699830.8598052.537431...0.880251.7674590.9005873.41507111.6627072.5391880.1160350.7916571.904160.806152
2CallTabMlpTorchTabMlp3.18910210.1703722.3185640.0964540.8422181.6699830.8598052.537431...0.880251.7674590.9005873.41507111.6627072.5391880.1160350.7916571.904160.806152
3CallTabMlpTorchWrappedTabMlp3.18910210.1703722.3185640.0964540.8422181.6699830.8598052.537431...0.880251.7674590.9005873.41507111.6627072.5391880.1160350.7916571.904160.806152
\n

4 rows × 23 columns

\n
" }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "trainer.add_modelbases([CallTabMlpTorchWrapped(trainer)])\n", "trainer.get_modelbase(\"CallTabMlpTorchWrapped\").train(stderr_to_stdout=True)\n", "trainer.get_leaderboard()" ], "metadata": { "collapsed": false, "pycharm": { "name": "#%%\n" } } }, { "cell_type": "markdown", "source": [ "**Remark**: If a model from the same `TorchModel` is required, the `AbstractNN` is extracted and passed as `required_models`. When calling `call_required_model` and `get_hidden_state`, you must pass the `model_name` argument." ], "metadata": { "collapsed": false, "pycharm": { "name": "#%% md\n" } } } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 2 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython2", "version": "2.7.6" } }, "nbformat": 4, "nbformat_minor": 0 }