{ "cells": [ { "cell_type": "markdown", "source": [ "# Cross-validation\n", "\n", "Cross-validation is required to validate the generalization ability of models, avoid the effect of randomization, etc. Randomization may affect the dataset splitting, model initialization, forward propagation (especially convolution operations), and optimization." ], "metadata": { "collapsed": false, "pycharm": { "name": "#%% md\n" } } }, { "cell_type": "code", "execution_count": 1, "metadata": { "collapsed": true, "pycharm": { "name": "#%%\n" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Using cuda device\n", "Downloading https://archive.ics.uci.edu/static/public/9/auto+mpg.zip to /tmp/tmpnpjgki5b/data/Auto MPG.zip\n", "cylinders is Integer and will be treated as a continuous feature.\n", "model_year is Integer and will be treated as a continuous feature.\n", "origin is Integer and will be treated as a continuous feature.\n", "Unknown values are detected in ['horsepower']. They will be treated as np.nan.\n", "The project will be saved to /tmp/tmpnpjgki5b/output/auto-mpg/2023-09-23-20-37-47-0_UserInputConfig\n", "Dataset size: 238 80 80\n", "Data saved to /tmp/tmpnpjgki5b/output/auto-mpg/2023-09-23-20-37-47-0_UserInputConfig (data.csv and tabular_data.csv).\n" ] } ], "source": [ "import torch\n", "from tabensemb.trainer import Trainer\n", "from tabensemb.model import *\n", "from tabensemb.config import UserConfig\n", "import tabensemb\n", "import os\n", "\n", "device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n", "print(\"Using {} device\".format(device))\n", "\n", "from tempfile import TemporaryDirectory\n", "\n", "temp_path = TemporaryDirectory()\n", "tabensemb.setting[\"default_output_path\"] = os.path.join(temp_path.name, \"output\")\n", "tabensemb.setting[\"default_config_path\"] = os.path.join(temp_path.name, \"configs\")\n", "tabensemb.setting[\"default_data_path\"] = os.path.join(temp_path.name, \"data\")\n", "\n", "trainer = Trainer(device=device)\n", "mpg_columns = [\n", " \"mpg\",\n", " \"cylinders\",\n", " \"displacement\",\n", " \"horsepower\",\n", " \"weight\",\n", " \"acceleration\",\n", " \"model_year\",\n", " \"origin\",\n", " \"car_name\",\n", "]\n", "cfg = UserConfig.from_uci(\"Auto MPG\", column_names=mpg_columns, sep=r\"\\s+\")\n", "trainer.load_config(cfg)\n", "trainer.load_data()\n", "models = [\n", " PytorchTabular(trainer, model_subset=[\"Category Embedding\"]),\n", "]\n", "trainer.add_modelbases(models)" ] }, { "cell_type": "markdown", "source": [ "## K-fold cross-validation\n", "\n", "Some of the data splitters (See \"Using data functionalities\") in `tabensemb` support k-fold cross-validation. To activate k-fold CV, pass the argument `split_type=\"cv\"` to `Trainer.get_leaderboard`. In this case, the ratio of training/validation/testing sets is (k-2):1:1. Here we present an example of a 4-fold CV." ], "metadata": { "collapsed": false, "pycharm": { "name": "#%% md\n" } } }, { "cell_type": "code", "execution_count": 2, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "----------------------------1/4 cv----------------------------\n", "Using previously used data path /tmp/tmpnpjgki5b/data/auto-mpg.csv\n", "Dataset size: 199 99 100\n", "Data saved to /tmp/tmpnpjgki5b/output/auto-mpg/2023-09-23-20-37-47-0_UserInputConfig (data.csv and tabular_data.csv).\n", "\n", "-------------Run PytorchTabular-------------\n", "\n", "Training Category Embedding\n", "Global seed set to 42\n", "2023-09-23 20:37:48,600 - {pytorch_tabular.tabular_model:473} - INFO - Preparing the DataLoaders\n", "2023-09-23 20:37:48,600 - {pytorch_tabular.tabular_datamodule:290} - INFO - Setting up the datamodule for regression task\n", "2023-09-23 20:37:48,610 - {pytorch_tabular.tabular_model:521} - INFO - Preparing the Model: CategoryEmbeddingModel\n", "2023-09-23 20:37:48,621 - {pytorch_tabular.tabular_model:268} - INFO - Preparing the Trainer\n", "/home/xlluo/anaconda3/envs/tabular_ensemble/lib/python3.10/site-packages/pytorch_lightning/trainer/connectors/accelerator_connector.py:589: LightningDeprecationWarning: The Trainer argument `auto_select_gpus` has been deprecated in v1.9.0 and will be removed in v2.0.0. Please use the function `pytorch_lightning.accelerators.find_usable_cuda_devices` instead.\n", " rank_zero_deprecation(\n", "Auto select gpus: [0]\n", "GPU available: True (cuda), used: True\n", "TPU available: False, using: 0 TPU cores\n", "IPU available: False, using: 0 IPUs\n", "HPU available: False, using: 0 HPUs\n", "2023-09-23 20:37:49,693 - {pytorch_tabular.tabular_model:582} - INFO - Training Started\n", "You are using a CUDA device ('NVIDIA GeForce RTX 3090') that has Tensor Cores. To properly utilize them, you should set `torch.set_float32_matmul_precision('medium' | 'high')` which will trade-off precision for performance. For more details, read https://pytorch.org/docs/stable/generated/torch.set_float32_matmul_precision.html#torch.set_float32_matmul_precision\n", "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]\n", "\n", " | Name | Type | Params\n", "---------------------------------------------------------------\n", "0 | _backbone | CategoryEmbeddingBackbone | 11.4 K\n", "1 | _embedding_layer | Embedding1dLayer | 14 \n", "2 | head | LinearHead | 33 \n", "3 | loss | MSELoss | 0 \n", "---------------------------------------------------------------\n", "11.4 K Trainable params\n", "0 Non-trainable params\n", "11.4 K Total params\n", "0.046 Total estimated model params size (MB)\n", "Epoch: 1/300, Train loss: 659.3594, Val loss: 628.6083, Min val loss: 628.6083, Epoch time: 0.035s.\n", "Epoch: 20/300, Train loss: 341.1675, Val loss: 317.6812, Min val loss: 317.6812, Epoch time: 0.011s.\n", "Epoch: 40/300, Train loss: 90.1501, Val loss: 70.8835, Min val loss: 70.8835, Epoch time: 0.011s.\n", "Epoch: 60/300, Train loss: 44.8901, Val loss: 35.4742, Min val loss: 35.4742, Epoch time: 0.011s.\n", "Epoch: 80/300, Train loss: 35.4699, Val loss: 30.2076, Min val loss: 30.2076, Epoch time: 0.011s.\n", "Epoch: 100/300, Train loss: 28.3062, Val loss: 27.7918, Min val loss: 27.7918, Epoch time: 0.012s.\n", "Epoch: 120/300, Train loss: 22.4722, Val loss: 25.4099, Min val loss: 25.4099, Epoch time: 0.011s.\n", "Epoch: 140/300, Train loss: 23.3103, Val loss: 24.1631, Min val loss: 24.1631, Epoch time: 0.010s.\n", "Epoch: 160/300, Train loss: 18.9331, Val loss: 23.2310, Min val loss: 22.8011, Epoch time: 0.013s.\n", "Epoch: 180/300, Train loss: 17.4051, Val loss: 22.7842, Min val loss: 22.7842, Epoch time: 0.010s.\n", "Epoch: 200/300, Train loss: 15.6233, Val loss: 21.3642, Min val loss: 21.3436, Epoch time: 0.011s.\n", "Epoch: 220/300, Train loss: 16.8699, Val loss: 20.2019, Min val loss: 20.2019, Epoch time: 0.010s.\n", "Epoch: 240/300, Train loss: 14.3514, Val loss: 19.4510, Min val loss: 19.4510, Epoch time: 0.010s.\n", "Epoch: 260/300, Train loss: 16.2243, Val loss: 19.0539, Min val loss: 19.0495, Epoch time: 0.009s.\n", "Epoch: 280/300, Train loss: 12.8429, Val loss: 18.1077, Min val loss: 18.0788, Epoch time: 0.011s.\n", "Epoch: 300/300, Train loss: 14.4400, Val loss: 17.8435, Min val loss: 17.8435, Epoch time: 0.009s.\n", "`Trainer.fit` stopped: `max_epochs=300` reached.\n", "2023-09-23 20:37:55,840 - {pytorch_tabular.tabular_model:584} - INFO - Training the model completed\n", "2023-09-23 20:37:55,841 - {pytorch_tabular.tabular_model:1258} - INFO - Loading the best model\n", "/home/xlluo/anaconda3/envs/tabular_ensemble/lib/python3.10/site-packages/pytorch_lightning/utilities/cloud_io.py:33: LightningDeprecationWarning: `pytorch_lightning.utilities.cloud_io.get_filesystem` has been deprecated in v1.8.0 and will be removed in v2.0.0. Please use `lightning_fabric.utilities.cloud_io.get_filesystem` instead.\n", " rank_zero_deprecation(\n", "Training mse loss: 10.68487\n", "Validation mse loss: 17.84354\n", "Testing mse loss: 9.95495\n", "Trainer saved. To load the trainer, run trainer = load_trainer(path='/tmp/tmpnpjgki5b/output/auto-mpg/2023-09-23-20-37-47-0_UserInputConfig/trainer.pkl')\n", "\n", "-------------PytorchTabular End-------------\n", "\n", "Category Embedding 1/1\n", "--------------------------End 1/4 cv--------------------------\n", "----------------------------2/4 cv----------------------------\n", "Using previously used data path /tmp/tmpnpjgki5b/data/auto-mpg.csv\n", "Dataset size: 199 99 100\n", "Data saved to /tmp/tmpnpjgki5b/output/auto-mpg/2023-09-23-20-37-47-0_UserInputConfig (data.csv and tabular_data.csv).\n", "\n", "-------------Run PytorchTabular-------------\n", "\n", "Training Category Embedding\n", "Global seed set to 42\n", "2023-09-23 20:37:56,591 - {pytorch_tabular.tabular_model:473} - INFO - Preparing the DataLoaders\n", "2023-09-23 20:37:56,592 - {pytorch_tabular.tabular_datamodule:290} - INFO - Setting up the datamodule for regression task\n", "2023-09-23 20:37:56,601 - {pytorch_tabular.tabular_model:521} - INFO - Preparing the Model: CategoryEmbeddingModel\n", "2023-09-23 20:37:56,613 - {pytorch_tabular.tabular_model:268} - INFO - Preparing the Trainer\n", "/home/xlluo/anaconda3/envs/tabular_ensemble/lib/python3.10/site-packages/pytorch_lightning/trainer/connectors/accelerator_connector.py:589: LightningDeprecationWarning: The Trainer argument `auto_select_gpus` has been deprecated in v1.9.0 and will be removed in v2.0.0. Please use the function `pytorch_lightning.accelerators.find_usable_cuda_devices` instead.\n", " rank_zero_deprecation(\n", "Auto select gpus: [0]\n", "GPU available: True (cuda), used: True\n", "TPU available: False, using: 0 TPU cores\n", "IPU available: False, using: 0 IPUs\n", "HPU available: False, using: 0 HPUs\n", "2023-09-23 20:37:56,627 - {pytorch_tabular.tabular_model:582} - INFO - Training Started\n", "You are using a CUDA device ('NVIDIA GeForce RTX 3090') that has Tensor Cores. To properly utilize them, you should set `torch.set_float32_matmul_precision('medium' | 'high')` which will trade-off precision for performance. For more details, read https://pytorch.org/docs/stable/generated/torch.set_float32_matmul_precision.html#torch.set_float32_matmul_precision\n", "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]\n", "\n", " | Name | Type | Params\n", "---------------------------------------------------------------\n", "0 | _backbone | CategoryEmbeddingBackbone | 11.4 K\n", "1 | _embedding_layer | Embedding1dLayer | 14 \n", "2 | head | LinearHead | 33 \n", "3 | loss | MSELoss | 0 \n", "---------------------------------------------------------------\n", "11.4 K Trainable params\n", "0 Non-trainable params\n", "11.4 K Total params\n", "0.046 Total estimated model params size (MB)\n", "Epoch: 1/300, Train loss: 696.0748, Val loss: 545.8853, Min val loss: 545.8853, Epoch time: 0.012s.\n", "Epoch: 20/300, Train loss: 372.0458, Val loss: 276.3550, Min val loss: 276.3550, Epoch time: 0.014s.\n", "Epoch: 40/300, Train loss: 85.0721, Val loss: 59.3774, Min val loss: 59.3774, Epoch time: 0.013s.\n", "Epoch: 60/300, Train loss: 48.5832, Val loss: 39.0821, Min val loss: 39.0821, Epoch time: 0.017s.\n", "Epoch: 80/300, Train loss: 31.3669, Val loss: 30.6529, Min val loss: 30.6529, Epoch time: 0.015s.\n", "Epoch: 100/300, Train loss: 24.9108, Val loss: 27.0339, Min val loss: 26.9302, Epoch time: 0.009s.\n", "Epoch: 120/300, Train loss: 21.2625, Val loss: 24.2448, Min val loss: 24.2448, Epoch time: 0.012s.\n", "Epoch: 140/300, Train loss: 19.6647, Val loss: 22.5396, Min val loss: 22.5396, Epoch time: 0.009s.\n", "Epoch: 160/300, Train loss: 18.2782, Val loss: 20.3885, Min val loss: 20.3885, Epoch time: 0.014s.\n", "Epoch: 180/300, Train loss: 19.8687, Val loss: 18.9228, Min val loss: 18.9228, Epoch time: 0.013s.\n", "Epoch: 200/300, Train loss: 17.5661, Val loss: 18.1803, Min val loss: 18.1723, Epoch time: 0.010s.\n", "Epoch: 220/300, Train loss: 17.5160, Val loss: 17.2254, Min val loss: 17.2254, Epoch time: 0.018s.\n", "Epoch: 240/300, Train loss: 18.5148, Val loss: 16.5343, Min val loss: 16.5343, Epoch time: 0.013s.\n", "Epoch: 260/300, Train loss: 19.2128, Val loss: 15.7374, Min val loss: 15.7374, Epoch time: 0.011s.\n", "Epoch: 280/300, Train loss: 13.2446, Val loss: 15.1371, Min val loss: 15.1371, Epoch time: 0.011s.\n", "Epoch: 300/300, Train loss: 12.9830, Val loss: 14.6800, Min val loss: 14.4976, Epoch time: 0.018s.\n", "`Trainer.fit` stopped: `max_epochs=300` reached.\n", "2023-09-23 20:38:02,306 - {pytorch_tabular.tabular_model:584} - INFO - Training the model completed\n", "2023-09-23 20:38:02,307 - {pytorch_tabular.tabular_model:1258} - INFO - Loading the best model\n", "/home/xlluo/anaconda3/envs/tabular_ensemble/lib/python3.10/site-packages/pytorch_lightning/utilities/cloud_io.py:33: LightningDeprecationWarning: `pytorch_lightning.utilities.cloud_io.get_filesystem` has been deprecated in v1.8.0 and will be removed in v2.0.0. Please use `lightning_fabric.utilities.cloud_io.get_filesystem` instead.\n", " rank_zero_deprecation(\n", "Training mse loss: 10.70534\n", "Validation mse loss: 14.49761\n", "Testing mse loss: 13.68175\n", "Trainer saved. To load the trainer, run trainer = load_trainer(path='/tmp/tmpnpjgki5b/output/auto-mpg/2023-09-23-20-37-47-0_UserInputConfig/trainer.pkl')\n", "\n", "-------------PytorchTabular End-------------\n", "\n", "Category Embedding 1/1\n", "--------------------------End 2/4 cv--------------------------\n", "----------------------------3/4 cv----------------------------\n", "Using previously used data path /tmp/tmpnpjgki5b/data/auto-mpg.csv\n", "Dataset size: 200 99 99\n", "Data saved to /tmp/tmpnpjgki5b/output/auto-mpg/2023-09-23-20-37-47-0_UserInputConfig (data.csv and tabular_data.csv).\n", "\n", "-------------Run PytorchTabular-------------\n", "\n", "Training Category Embedding\n", "Global seed set to 42\n", "2023-09-23 20:38:02,872 - {pytorch_tabular.tabular_model:473} - INFO - Preparing the DataLoaders\n", "2023-09-23 20:38:02,874 - {pytorch_tabular.tabular_datamodule:290} - INFO - Setting up the datamodule for regression task\n", "2023-09-23 20:38:02,883 - {pytorch_tabular.tabular_model:521} - INFO - Preparing the Model: CategoryEmbeddingModel\n", "2023-09-23 20:38:02,899 - {pytorch_tabular.tabular_model:268} - INFO - Preparing the Trainer\n", "/home/xlluo/anaconda3/envs/tabular_ensemble/lib/python3.10/site-packages/pytorch_lightning/trainer/connectors/accelerator_connector.py:589: LightningDeprecationWarning: The Trainer argument `auto_select_gpus` has been deprecated in v1.9.0 and will be removed in v2.0.0. Please use the function `pytorch_lightning.accelerators.find_usable_cuda_devices` instead.\n", " rank_zero_deprecation(\n", "Auto select gpus: [0]\n", "GPU available: True (cuda), used: True\n", "TPU available: False, using: 0 TPU cores\n", "IPU available: False, using: 0 IPUs\n", "HPU available: False, using: 0 HPUs\n", "2023-09-23 20:38:02,922 - {pytorch_tabular.tabular_model:582} - INFO - Training Started\n", "You are using a CUDA device ('NVIDIA GeForce RTX 3090') that has Tensor Cores. To properly utilize them, you should set `torch.set_float32_matmul_precision('medium' | 'high')` which will trade-off precision for performance. For more details, read https://pytorch.org/docs/stable/generated/torch.set_float32_matmul_precision.html#torch.set_float32_matmul_precision\n", "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]\n", "\n", " | Name | Type | Params\n", "---------------------------------------------------------------\n", "0 | _backbone | CategoryEmbeddingBackbone | 11.4 K\n", "1 | _embedding_layer | Embedding1dLayer | 14 \n", "2 | head | LinearHead | 33 \n", "3 | loss | MSELoss | 0 \n", "---------------------------------------------------------------\n", "11.4 K Trainable params\n", "0 Non-trainable params\n", "11.4 K Total params\n", "0.046 Total estimated model params size (MB)\n", "Epoch: 1/300, Train loss: 645.7930, Val loss: 634.2722, Min val loss: 634.2722, Epoch time: 0.017s.\n", "Epoch: 20/300, Train loss: 326.0877, Val loss: 323.5532, Min val loss: 323.5532, Epoch time: 0.012s.\n", "Epoch: 40/300, Train loss: 79.7064, Val loss: 70.4397, Min val loss: 70.4397, Epoch time: 0.011s.\n", "Epoch: 60/300, Train loss: 46.4769, Val loss: 33.4822, Min val loss: 33.4822, Epoch time: 0.011s.\n", "Epoch: 80/300, Train loss: 33.4557, Val loss: 23.5222, Min val loss: 23.5222, Epoch time: 0.011s.\n", "Epoch: 100/300, Train loss: 27.6559, Val loss: 19.9502, Min val loss: 19.9502, Epoch time: 0.011s.\n", "Epoch: 120/300, Train loss: 20.6489, Val loss: 17.9041, Min val loss: 17.9041, Epoch time: 0.013s.\n", "Epoch: 140/300, Train loss: 19.6383, Val loss: 17.4171, Min val loss: 17.4171, Epoch time: 0.010s.\n", "Epoch: 160/300, Train loss: 16.6578, Val loss: 16.3154, Min val loss: 16.2164, Epoch time: 0.011s.\n", "Epoch: 180/300, Train loss: 19.6480, Val loss: 15.2104, Min val loss: 15.2104, Epoch time: 0.010s.\n", "Epoch: 200/300, Train loss: 15.6033, Val loss: 14.5753, Min val loss: 14.5753, Epoch time: 0.011s.\n", "Epoch: 220/300, Train loss: 13.8016, Val loss: 13.8601, Min val loss: 13.8468, Epoch time: 0.009s.\n", "Epoch: 240/300, Train loss: 15.3606, Val loss: 12.9441, Min val loss: 12.9441, Epoch time: 0.011s.\n", "Epoch: 260/300, Train loss: 14.5253, Val loss: 12.5276, Min val loss: 12.5276, Epoch time: 0.011s.\n", "Epoch: 280/300, Train loss: 15.3802, Val loss: 12.0176, Min val loss: 12.0176, Epoch time: 0.009s.\n", "Epoch: 300/300, Train loss: 14.3208, Val loss: 11.5972, Min val loss: 11.5972, Epoch time: 0.010s.\n", "`Trainer.fit` stopped: `max_epochs=300` reached.\n", "2023-09-23 20:38:07,954 - {pytorch_tabular.tabular_model:584} - INFO - Training the model completed\n", "2023-09-23 20:38:07,954 - {pytorch_tabular.tabular_model:1258} - INFO - Loading the best model\n", "/home/xlluo/anaconda3/envs/tabular_ensemble/lib/python3.10/site-packages/pytorch_lightning/utilities/cloud_io.py:33: LightningDeprecationWarning: `pytorch_lightning.utilities.cloud_io.get_filesystem` has been deprecated in v1.8.0 and will be removed in v2.0.0. Please use `lightning_fabric.utilities.cloud_io.get_filesystem` instead.\n", " rank_zero_deprecation(\n", "Training mse loss: 8.63738\n", "Validation mse loss: 11.59718\n", "Testing mse loss: 16.02349\n", "Trainer saved. To load the trainer, run trainer = load_trainer(path='/tmp/tmpnpjgki5b/output/auto-mpg/2023-09-23-20-37-47-0_UserInputConfig/trainer.pkl')\n", "\n", "-------------PytorchTabular End-------------\n", "\n", "Category Embedding 1/1\n", "--------------------------End 3/4 cv--------------------------\n", "----------------------------4/4 cv----------------------------\n", "Using previously used data path /tmp/tmpnpjgki5b/data/auto-mpg.csv\n", "Dataset size: 200 99 99\n", "Data saved to /tmp/tmpnpjgki5b/output/auto-mpg/2023-09-23-20-37-47-0_UserInputConfig (data.csv and tabular_data.csv).\n", "\n", "-------------Run PytorchTabular-------------\n", "\n", "Training Category Embedding\n", "Global seed set to 42\n", "2023-09-23 20:38:08,438 - {pytorch_tabular.tabular_model:473} - INFO - Preparing the DataLoaders\n", "2023-09-23 20:38:08,439 - {pytorch_tabular.tabular_datamodule:290} - INFO - Setting up the datamodule for regression task\n", "2023-09-23 20:38:08,447 - {pytorch_tabular.tabular_model:521} - INFO - Preparing the Model: CategoryEmbeddingModel\n", "2023-09-23 20:38:08,458 - {pytorch_tabular.tabular_model:268} - INFO - Preparing the Trainer\n", "/home/xlluo/anaconda3/envs/tabular_ensemble/lib/python3.10/site-packages/pytorch_lightning/trainer/connectors/accelerator_connector.py:589: LightningDeprecationWarning: The Trainer argument `auto_select_gpus` has been deprecated in v1.9.0 and will be removed in v2.0.0. Please use the function `pytorch_lightning.accelerators.find_usable_cuda_devices` instead.\n", " rank_zero_deprecation(\n", "Auto select gpus: [0]\n", "GPU available: True (cuda), used: True\n", "TPU available: False, using: 0 TPU cores\n", "IPU available: False, using: 0 IPUs\n", "HPU available: False, using: 0 HPUs\n", "2023-09-23 20:38:08,473 - {pytorch_tabular.tabular_model:582} - INFO - Training Started\n", "You are using a CUDA device ('NVIDIA GeForce RTX 3090') that has Tensor Cores. To properly utilize them, you should set `torch.set_float32_matmul_precision('medium' | 'high')` which will trade-off precision for performance. For more details, read https://pytorch.org/docs/stable/generated/torch.set_float32_matmul_precision.html#torch.set_float32_matmul_precision\n", "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]\n", "\n", " | Name | Type | Params\n", "---------------------------------------------------------------\n", "0 | _backbone | CategoryEmbeddingBackbone | 11.4 K\n", "1 | _embedding_layer | Embedding1dLayer | 14 \n", "2 | head | LinearHead | 33 \n", "3 | loss | MSELoss | 0 \n", "---------------------------------------------------------------\n", "11.4 K Trainable params\n", "0 Non-trainable params\n", "11.4 K Total params\n", "0.046 Total estimated model params size (MB)\n", "Epoch: 1/300, Train loss: 636.0693, Val loss: 673.2845, Min val loss: 673.2845, Epoch time: 0.011s.\n", "Epoch: 20/300, Train loss: 320.2209, Val loss: 352.5726, Min val loss: 352.5726, Epoch time: 0.009s.\n", "Epoch: 40/300, Train loss: 72.3203, Val loss: 74.2078, Min val loss: 74.2078, Epoch time: 0.010s.\n", "Epoch: 60/300, Train loss: 41.2314, Val loss: 47.7063, Min val loss: 47.7063, Epoch time: 0.008s.\n", "Epoch: 80/300, Train loss: 31.4739, Val loss: 34.3773, Min val loss: 34.3773, Epoch time: 0.009s.\n", "Epoch: 100/300, Train loss: 26.7886, Val loss: 28.7385, Min val loss: 28.7385, Epoch time: 0.008s.\n", "Epoch: 120/300, Train loss: 20.0182, Val loss: 24.3363, Min val loss: 24.3363, Epoch time: 0.009s.\n", "Epoch: 140/300, Train loss: 19.3822, Val loss: 22.6521, Min val loss: 22.6521, Epoch time: 0.012s.\n", "Epoch: 160/300, Train loss: 19.0349, Val loss: 21.4546, Min val loss: 21.4546, Epoch time: 0.009s.\n", "Epoch: 180/300, Train loss: 16.7342, Val loss: 19.9216, Min val loss: 19.9216, Epoch time: 0.009s.\n", "Epoch: 200/300, Train loss: 14.9367, Val loss: 18.7599, Min val loss: 18.7599, Epoch time: 0.009s.\n", "Epoch: 220/300, Train loss: 15.6413, Val loss: 18.2248, Min val loss: 18.2248, Epoch time: 0.010s.\n", "Epoch: 240/300, Train loss: 16.4146, Val loss: 17.3294, Min val loss: 17.3294, Epoch time: 0.009s.\n", "Epoch: 260/300, Train loss: 13.5992, Val loss: 16.4097, Min val loss: 16.4097, Epoch time: 0.013s.\n", "Epoch: 280/300, Train loss: 12.5998, Val loss: 15.9688, Min val loss: 15.5961, Epoch time: 0.009s.\n", "Epoch: 300/300, Train loss: 15.1886, Val loss: 14.6322, Min val loss: 14.6322, Epoch time: 0.011s.\n", "`Trainer.fit` stopped: `max_epochs=300` reached.\n", "2023-09-23 20:38:12,907 - {pytorch_tabular.tabular_model:584} - INFO - Training the model completed\n", "2023-09-23 20:38:12,908 - {pytorch_tabular.tabular_model:1258} - INFO - Loading the best model\n", "/home/xlluo/anaconda3/envs/tabular_ensemble/lib/python3.10/site-packages/pytorch_lightning/utilities/cloud_io.py:33: LightningDeprecationWarning: `pytorch_lightning.utilities.cloud_io.get_filesystem` has been deprecated in v1.8.0 and will be removed in v2.0.0. Please use `lightning_fabric.utilities.cloud_io.get_filesystem` instead.\n", " rank_zero_deprecation(\n", "Training mse loss: 8.95723\n", "Validation mse loss: 14.63222\n", "Testing mse loss: 12.50395\n", "Trainer saved. To load the trainer, run trainer = load_trainer(path='/tmp/tmpnpjgki5b/output/auto-mpg/2023-09-23-20-37-47-0_UserInputConfig/trainer.pkl')\n", "\n", "-------------PytorchTabular End-------------\n", "\n", "Category Embedding 1/1\n", "--------------------------End 4/4 cv--------------------------\n", "Trainer saved. To load the trainer, run trainer = load_trainer(path='/tmp/tmpnpjgki5b/output/auto-mpg/2023-09-23-20-37-47-0_UserInputConfig/trainer.pkl')\n" ] }, { "data": { "text/plain": " Program Model Training RMSE Training MSE \\\n0 PytorchTabular Category Embedding 3.121511 9.743828 \n\n Training MAE Training MAPE Training R2 Training MEDIAN_ABSOLUTE_ERROR \\\n0 2.361333 0.100122 0.839443 1.824559 \n\n Training EXPLAINED_VARIANCE_SCORE Testing RMSE ... Testing R2 \\\n0 0.870805 3.610387 ... 0.786089 \n\n Testing MEDIAN_ABSOLUTE_ERROR Testing EXPLAINED_VARIANCE_SCORE \\\n0 2.024105 0.814192 \n\n Validation RMSE Validation MSE Validation MAE Validation MAPE \\\n0 3.82657 14.642641 2.699677 0.119046 \n\n Validation R2 Validation MEDIAN_ABSOLUTE_ERROR \\\n0 0.761043 1.905207 \n\n Validation EXPLAINED_VARIANCE_SCORE \n0 0.785311 \n\n[1 rows x 23 columns]", "text/html": "
\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
ProgramModelTraining RMSETraining MSETraining MAETraining MAPETraining R2Training MEDIAN_ABSOLUTE_ERRORTraining EXPLAINED_VARIANCE_SCORETesting RMSE...Testing R2Testing MEDIAN_ABSOLUTE_ERRORTesting EXPLAINED_VARIANCE_SCOREValidation RMSEValidation MSEValidation MAEValidation MAPEValidation R2Validation MEDIAN_ABSOLUTE_ERRORValidation EXPLAINED_VARIANCE_SCORE
0PytorchTabularCategory Embedding3.1215119.7438282.3613330.1001220.8394431.8245590.8708053.610387...0.7860892.0241050.8141923.8265714.6426412.6996770.1190460.7610431.9052070.785311
\n

1 rows × 23 columns

\n
" }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "trainer.get_leaderboard(cross_validation=4, split_type=\"cv\", stderr_to_stdout=True)" ], "metadata": { "collapsed": false, "pycharm": { "name": "#%%\n" } } }, { "cell_type": "markdown", "source": [ "## Splitting the dataset randomly\n", "\n", "We can simply split the dataset with different random seeds. This is achieved by passing the argument `split_type=\"random\"`. In this case, the ratio of training/validation/testing sets is the one specified in the configuration (or 6:2:2 by default)." ], "metadata": { "collapsed": false, "pycharm": { "name": "#%% md\n" } } }, { "cell_type": "code", "execution_count": 3, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "----------------------------1/4 random----------------------------\n", "Using previously used data path /tmp/tmpnpjgki5b/data/auto-mpg.csv\n", "Dataset size: 238 80 80\n", "Data saved to /tmp/tmpnpjgki5b/output/auto-mpg/2023-09-23-20-37-47-0_UserInputConfig (data.csv and tabular_data.csv).\n", "\n", "-------------Run PytorchTabular-------------\n", "\n", "Training Category Embedding\n", "Global seed set to 42\n", "2023-09-23 20:38:13,428 - {pytorch_tabular.tabular_model:473} - INFO - Preparing the DataLoaders\n", "2023-09-23 20:38:13,428 - {pytorch_tabular.tabular_datamodule:290} - INFO - Setting up the datamodule for regression task\n", "2023-09-23 20:38:13,437 - {pytorch_tabular.tabular_model:521} - INFO - Preparing the Model: CategoryEmbeddingModel\n", "2023-09-23 20:38:13,447 - {pytorch_tabular.tabular_model:268} - INFO - Preparing the Trainer\n", "/home/xlluo/anaconda3/envs/tabular_ensemble/lib/python3.10/site-packages/pytorch_lightning/trainer/connectors/accelerator_connector.py:589: LightningDeprecationWarning: The Trainer argument `auto_select_gpus` has been deprecated in v1.9.0 and will be removed in v2.0.0. Please use the function `pytorch_lightning.accelerators.find_usable_cuda_devices` instead.\n", " rank_zero_deprecation(\n", "Auto select gpus: [0]\n", "GPU available: True (cuda), used: True\n", "TPU available: False, using: 0 TPU cores\n", "IPU available: False, using: 0 IPUs\n", "HPU available: False, using: 0 HPUs\n", "2023-09-23 20:38:13,463 - {pytorch_tabular.tabular_model:582} - INFO - Training Started\n", "You are using a CUDA device ('NVIDIA GeForce RTX 3090') that has Tensor Cores. To properly utilize them, you should set `torch.set_float32_matmul_precision('medium' | 'high')` which will trade-off precision for performance. For more details, read https://pytorch.org/docs/stable/generated/torch.set_float32_matmul_precision.html#torch.set_float32_matmul_precision\n", "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]\n", "\n", " | Name | Type | Params\n", "---------------------------------------------------------------\n", "0 | _backbone | CategoryEmbeddingBackbone | 11.4 K\n", "1 | _embedding_layer | Embedding1dLayer | 14 \n", "2 | head | LinearHead | 33 \n", "3 | loss | MSELoss | 0 \n", "---------------------------------------------------------------\n", "11.4 K Trainable params\n", "0 Non-trainable params\n", "11.4 K Total params\n", "0.046 Total estimated model params size (MB)\n", "Epoch: 1/300, Train loss: 677.8015, Val loss: 582.9557, Min val loss: 582.9557, Epoch time: 0.012s.\n", "Epoch: 20/300, Train loss: 353.7851, Val loss: 302.0203, Min val loss: 302.0203, Epoch time: 0.009s.\n", "Epoch: 40/300, Train loss: 85.0776, Val loss: 62.1153, Min val loss: 62.1153, Epoch time: 0.009s.\n", "Epoch: 60/300, Train loss: 45.2654, Val loss: 34.2778, Min val loss: 34.2691, Epoch time: 0.009s.\n", "Epoch: 80/300, Train loss: 33.9537, Val loss: 26.8622, Min val loss: 26.8622, Epoch time: 0.011s.\n", "Epoch: 100/300, Train loss: 26.9038, Val loss: 23.2417, Min val loss: 23.2372, Epoch time: 0.009s.\n", "Epoch: 120/300, Train loss: 24.9622, Val loss: 20.4360, Min val loss: 20.4360, Epoch time: 0.009s.\n", "Epoch: 140/300, Train loss: 24.1636, Val loss: 19.4010, Min val loss: 19.4010, Epoch time: 0.009s.\n", "Epoch: 160/300, Train loss: 22.9200, Val loss: 18.0232, Min val loss: 17.9749, Epoch time: 0.008s.\n", "Epoch: 180/300, Train loss: 19.7677, Val loss: 16.9469, Min val loss: 16.9469, Epoch time: 0.010s.\n", "Epoch: 200/300, Train loss: 17.9390, Val loss: 16.6545, Min val loss: 16.4093, Epoch time: 0.012s.\n", "Epoch: 220/300, Train loss: 19.4496, Val loss: 15.4451, Min val loss: 15.1788, Epoch time: 0.010s.\n", "Epoch: 240/300, Train loss: 16.0483, Val loss: 14.5508, Min val loss: 14.5508, Epoch time: 0.012s.\n", "Epoch: 260/300, Train loss: 16.4672, Val loss: 13.8354, Min val loss: 13.8354, Epoch time: 0.014s.\n", "Epoch: 280/300, Train loss: 13.6031, Val loss: 12.9315, Min val loss: 12.9315, Epoch time: 0.011s.\n", "Epoch: 300/300, Train loss: 16.5369, Val loss: 12.3673, Min val loss: 12.3673, Epoch time: 0.010s.\n", "`Trainer.fit` stopped: `max_epochs=300` reached.\n", "2023-09-23 20:38:17,915 - {pytorch_tabular.tabular_model:584} - INFO - Training the model completed\n", "2023-09-23 20:38:17,916 - {pytorch_tabular.tabular_model:1258} - INFO - Loading the best model\n", "/home/xlluo/anaconda3/envs/tabular_ensemble/lib/python3.10/site-packages/pytorch_lightning/utilities/cloud_io.py:33: LightningDeprecationWarning: `pytorch_lightning.utilities.cloud_io.get_filesystem` has been deprecated in v1.8.0 and will be removed in v2.0.0. Please use `lightning_fabric.utilities.cloud_io.get_filesystem` instead.\n", " rank_zero_deprecation(\n", "Training mse loss: 11.25175\n", "Validation mse loss: 12.36725\n", "Testing mse loss: 7.83801\n", "Trainer saved. To load the trainer, run trainer = load_trainer(path='/tmp/tmpnpjgki5b/output/auto-mpg/2023-09-23-20-37-47-0_UserInputConfig/trainer.pkl')\n", "\n", "-------------PytorchTabular End-------------\n", "\n", "Category Embedding 1/1\n", "--------------------------End 1/4 random--------------------------\n", "----------------------------2/4 random----------------------------\n", "Using previously used data path /tmp/tmpnpjgki5b/data/auto-mpg.csv\n", "Dataset size: 238 80 80\n", "Data saved to /tmp/tmpnpjgki5b/output/auto-mpg/2023-09-23-20-37-47-0_UserInputConfig (data.csv and tabular_data.csv).\n", "\n", "-------------Run PytorchTabular-------------\n", "\n", "Training Category Embedding\n", "Global seed set to 42\n", "2023-09-23 20:38:18,385 - {pytorch_tabular.tabular_model:473} - INFO - Preparing the DataLoaders\n", "2023-09-23 20:38:18,385 - {pytorch_tabular.tabular_datamodule:290} - INFO - Setting up the datamodule for regression task\n", "2023-09-23 20:38:18,394 - {pytorch_tabular.tabular_model:521} - INFO - Preparing the Model: CategoryEmbeddingModel\n", "2023-09-23 20:38:18,404 - {pytorch_tabular.tabular_model:268} - INFO - Preparing the Trainer\n", "/home/xlluo/anaconda3/envs/tabular_ensemble/lib/python3.10/site-packages/pytorch_lightning/trainer/connectors/accelerator_connector.py:589: LightningDeprecationWarning: The Trainer argument `auto_select_gpus` has been deprecated in v1.9.0 and will be removed in v2.0.0. Please use the function `pytorch_lightning.accelerators.find_usable_cuda_devices` instead.\n", " rank_zero_deprecation(\n", "Auto select gpus: [0]\n", "GPU available: True (cuda), used: True\n", "TPU available: False, using: 0 TPU cores\n", "IPU available: False, using: 0 IPUs\n", "HPU available: False, using: 0 HPUs\n", "2023-09-23 20:38:18,418 - {pytorch_tabular.tabular_model:582} - INFO - Training Started\n", "You are using a CUDA device ('NVIDIA GeForce RTX 3090') that has Tensor Cores. To properly utilize them, you should set `torch.set_float32_matmul_precision('medium' | 'high')` which will trade-off precision for performance. For more details, read https://pytorch.org/docs/stable/generated/torch.set_float32_matmul_precision.html#torch.set_float32_matmul_precision\n", "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]\n", "\n", " | Name | Type | Params\n", "---------------------------------------------------------------\n", "0 | _backbone | CategoryEmbeddingBackbone | 11.4 K\n", "1 | _embedding_layer | Embedding1dLayer | 14 \n", "2 | head | LinearHead | 33 \n", "3 | loss | MSELoss | 0 \n", "---------------------------------------------------------------\n", "11.4 K Trainable params\n", "0 Non-trainable params\n", "11.4 K Total params\n", "0.046 Total estimated model params size (MB)\n", "Epoch: 1/300, Train loss: 632.3258, Val loss: 662.7650, Min val loss: 662.7650, Epoch time: 0.012s.\n", "Epoch: 20/300, Train loss: 318.3275, Val loss: 344.3962, Min val loss: 344.3962, Epoch time: 0.011s.\n", "Epoch: 40/300, Train loss: 69.3076, Val loss: 76.5520, Min val loss: 76.5520, Epoch time: 0.011s.\n", "Epoch: 60/300, Train loss: 42.2416, Val loss: 46.7150, Min val loss: 46.7150, Epoch time: 0.010s.\n", "Epoch: 80/300, Train loss: 29.2576, Val loss: 34.4237, Min val loss: 34.4237, Epoch time: 0.008s.\n", "Epoch: 100/300, Train loss: 26.1472, Val loss: 29.0749, Min val loss: 29.0749, Epoch time: 0.008s.\n", "Epoch: 120/300, Train loss: 24.8970, Val loss: 26.5129, Min val loss: 26.5129, Epoch time: 0.009s.\n", "Epoch: 140/300, Train loss: 17.9044, Val loss: 25.3725, Min val loss: 25.3725, Epoch time: 0.009s.\n", "Epoch: 160/300, Train loss: 17.4626, Val loss: 23.7306, Min val loss: 23.7306, Epoch time: 0.009s.\n", "Epoch: 180/300, Train loss: 16.5023, Val loss: 22.9689, Min val loss: 22.9689, Epoch time: 0.010s.\n", "Epoch: 200/300, Train loss: 15.1315, Val loss: 22.0908, Min val loss: 22.0908, Epoch time: 0.010s.\n", "Epoch: 220/300, Train loss: 14.4462, Val loss: 21.3679, Min val loss: 21.3679, Epoch time: 0.011s.\n", "Epoch: 240/300, Train loss: 15.4215, Val loss: 20.7016, Min val loss: 20.6893, Epoch time: 0.011s.\n", "Epoch: 260/300, Train loss: 13.5062, Val loss: 20.0930, Min val loss: 20.0624, Epoch time: 0.009s.\n", "Epoch: 280/300, Train loss: 14.5558, Val loss: 19.2505, Min val loss: 19.2505, Epoch time: 0.009s.\n", "Epoch: 300/300, Train loss: 14.8346, Val loss: 18.8435, Min val loss: 18.8435, Epoch time: 0.010s.\n", "`Trainer.fit` stopped: `max_epochs=300` reached.\n", "2023-09-23 20:38:23,402 - {pytorch_tabular.tabular_model:584} - INFO - Training the model completed\n", "2023-09-23 20:38:23,403 - {pytorch_tabular.tabular_model:1258} - INFO - Loading the best model\n", "/home/xlluo/anaconda3/envs/tabular_ensemble/lib/python3.10/site-packages/pytorch_lightning/utilities/cloud_io.py:33: LightningDeprecationWarning: `pytorch_lightning.utilities.cloud_io.get_filesystem` has been deprecated in v1.8.0 and will be removed in v2.0.0. Please use `lightning_fabric.utilities.cloud_io.get_filesystem` instead.\n", " rank_zero_deprecation(\n", "Training mse loss: 10.18515\n", "Validation mse loss: 18.84349\n", "Testing mse loss: 9.93399\n", "Trainer saved. To load the trainer, run trainer = load_trainer(path='/tmp/tmpnpjgki5b/output/auto-mpg/2023-09-23-20-37-47-0_UserInputConfig/trainer.pkl')\n", "\n", "-------------PytorchTabular End-------------\n", "\n", "Category Embedding 1/1\n", "--------------------------End 2/4 random--------------------------\n", "----------------------------3/4 random----------------------------\n", "Using previously used data path /tmp/tmpnpjgki5b/data/auto-mpg.csv\n", "Dataset size: 238 80 80\n", "Data saved to /tmp/tmpnpjgki5b/output/auto-mpg/2023-09-23-20-37-47-0_UserInputConfig (data.csv and tabular_data.csv).\n", "\n", "-------------Run PytorchTabular-------------\n", "\n", "Training Category Embedding\n", "Global seed set to 42\n", "2023-09-23 20:38:23,909 - {pytorch_tabular.tabular_model:473} - INFO - Preparing the DataLoaders\n", "2023-09-23 20:38:23,910 - {pytorch_tabular.tabular_datamodule:290} - INFO - Setting up the datamodule for regression task\n", "2023-09-23 20:38:23,919 - {pytorch_tabular.tabular_model:521} - INFO - Preparing the Model: CategoryEmbeddingModel\n", "2023-09-23 20:38:23,929 - {pytorch_tabular.tabular_model:268} - INFO - Preparing the Trainer\n", "/home/xlluo/anaconda3/envs/tabular_ensemble/lib/python3.10/site-packages/pytorch_lightning/trainer/connectors/accelerator_connector.py:589: LightningDeprecationWarning: The Trainer argument `auto_select_gpus` has been deprecated in v1.9.0 and will be removed in v2.0.0. Please use the function `pytorch_lightning.accelerators.find_usable_cuda_devices` instead.\n", " rank_zero_deprecation(\n", "Auto select gpus: [0]\n", "GPU available: True (cuda), used: True\n", "TPU available: False, using: 0 TPU cores\n", "IPU available: False, using: 0 IPUs\n", "HPU available: False, using: 0 HPUs\n", "2023-09-23 20:38:23,943 - {pytorch_tabular.tabular_model:582} - INFO - Training Started\n", "You are using a CUDA device ('NVIDIA GeForce RTX 3090') that has Tensor Cores. To properly utilize them, you should set `torch.set_float32_matmul_precision('medium' | 'high')` which will trade-off precision for performance. For more details, read https://pytorch.org/docs/stable/generated/torch.set_float32_matmul_precision.html#torch.set_float32_matmul_precision\n", "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]\n", "\n", " | Name | Type | Params\n", "---------------------------------------------------------------\n", "0 | _backbone | CategoryEmbeddingBackbone | 11.4 K\n", "1 | _embedding_layer | Embedding1dLayer | 14 \n", "2 | head | LinearHead | 33 \n", "3 | loss | MSELoss | 0 \n", "---------------------------------------------------------------\n", "11.4 K Trainable params\n", "0 Non-trainable params\n", "11.4 K Total params\n", "0.046 Total estimated model params size (MB)\n", "Epoch: 1/300, Train loss: 663.8328, Val loss: 585.2318, Min val loss: 585.2318, Epoch time: 0.009s.\n", "Epoch: 20/300, Train loss: 341.2238, Val loss: 306.3730, Min val loss: 306.3730, Epoch time: 0.015s.\n", "Epoch: 40/300, Train loss: 74.4331, Val loss: 64.6773, Min val loss: 64.6773, Epoch time: 0.010s.\n", "Epoch: 60/300, Train loss: 45.9507, Val loss: 41.3845, Min val loss: 41.3845, Epoch time: 0.011s.\n", "Epoch: 80/300, Train loss: 32.7711, Val loss: 30.7545, Min val loss: 30.7545, Epoch time: 0.009s.\n", "Epoch: 100/300, Train loss: 23.9204, Val loss: 26.2439, Min val loss: 26.2439, Epoch time: 0.011s.\n", "Epoch: 120/300, Train loss: 20.2654, Val loss: 23.0741, Min val loss: 23.0741, Epoch time: 0.010s.\n", "Epoch: 140/300, Train loss: 20.3352, Val loss: 20.7483, Min val loss: 20.7483, Epoch time: 0.008s.\n", "Epoch: 160/300, Train loss: 19.5925, Val loss: 18.9536, Min val loss: 18.9536, Epoch time: 0.011s.\n", "Epoch: 180/300, Train loss: 18.0734, Val loss: 17.7072, Min val loss: 17.7072, Epoch time: 0.012s.\n", "Epoch: 200/300, Train loss: 15.6461, Val loss: 16.8639, Min val loss: 16.8639, Epoch time: 0.011s.\n", "Epoch: 220/300, Train loss: 13.4826, Val loss: 15.9317, Min val loss: 15.9317, Epoch time: 0.009s.\n", "Epoch: 240/300, Train loss: 15.6729, Val loss: 15.3439, Min val loss: 15.3068, Epoch time: 0.009s.\n", "Epoch: 260/300, Train loss: 14.0022, Val loss: 14.3354, Min val loss: 14.3354, Epoch time: 0.009s.\n", "Epoch: 280/300, Train loss: 12.9512, Val loss: 13.6573, Min val loss: 13.6573, Epoch time: 0.011s.\n", "Epoch: 300/300, Train loss: 13.2983, Val loss: 13.2018, Min val loss: 13.2018, Epoch time: 0.009s.\n", "`Trainer.fit` stopped: `max_epochs=300` reached.\n", "2023-09-23 20:38:28,571 - {pytorch_tabular.tabular_model:584} - INFO - Training the model completed\n", "2023-09-23 20:38:28,572 - {pytorch_tabular.tabular_model:1258} - INFO - Loading the best model\n", "/home/xlluo/anaconda3/envs/tabular_ensemble/lib/python3.10/site-packages/pytorch_lightning/utilities/cloud_io.py:33: LightningDeprecationWarning: `pytorch_lightning.utilities.cloud_io.get_filesystem` has been deprecated in v1.8.0 and will be removed in v2.0.0. Please use `lightning_fabric.utilities.cloud_io.get_filesystem` instead.\n", " rank_zero_deprecation(\n", "Training mse loss: 9.30384\n", "Validation mse loss: 13.20180\n", "Testing mse loss: 14.17569\n", "Trainer saved. To load the trainer, run trainer = load_trainer(path='/tmp/tmpnpjgki5b/output/auto-mpg/2023-09-23-20-37-47-0_UserInputConfig/trainer.pkl')\n", "\n", "-------------PytorchTabular End-------------\n", "\n", "Category Embedding 1/1\n", "--------------------------End 3/4 random--------------------------\n", "----------------------------4/4 random----------------------------\n", "Using previously used data path /tmp/tmpnpjgki5b/data/auto-mpg.csv\n", "Dataset size: 238 80 80\n", "Data saved to /tmp/tmpnpjgki5b/output/auto-mpg/2023-09-23-20-37-47-0_UserInputConfig (data.csv and tabular_data.csv).\n", "\n", "-------------Run PytorchTabular-------------\n", "\n", "Training Category Embedding\n", "Global seed set to 42\n", "2023-09-23 20:38:29,037 - {pytorch_tabular.tabular_model:473} - INFO - Preparing the DataLoaders\n", "2023-09-23 20:38:29,038 - {pytorch_tabular.tabular_datamodule:290} - INFO - Setting up the datamodule for regression task\n", "2023-09-23 20:38:29,046 - {pytorch_tabular.tabular_model:521} - INFO - Preparing the Model: CategoryEmbeddingModel\n", "2023-09-23 20:38:29,057 - {pytorch_tabular.tabular_model:268} - INFO - Preparing the Trainer\n", "/home/xlluo/anaconda3/envs/tabular_ensemble/lib/python3.10/site-packages/pytorch_lightning/trainer/connectors/accelerator_connector.py:589: LightningDeprecationWarning: The Trainer argument `auto_select_gpus` has been deprecated in v1.9.0 and will be removed in v2.0.0. Please use the function `pytorch_lightning.accelerators.find_usable_cuda_devices` instead.\n", " rank_zero_deprecation(\n", "Auto select gpus: [0]\n", "GPU available: True (cuda), used: True\n", "TPU available: False, using: 0 TPU cores\n", "IPU available: False, using: 0 IPUs\n", "HPU available: False, using: 0 HPUs\n", "2023-09-23 20:38:29,072 - {pytorch_tabular.tabular_model:582} - INFO - Training Started\n", "You are using a CUDA device ('NVIDIA GeForce RTX 3090') that has Tensor Cores. To properly utilize them, you should set `torch.set_float32_matmul_precision('medium' | 'high')` which will trade-off precision for performance. For more details, read https://pytorch.org/docs/stable/generated/torch.set_float32_matmul_precision.html#torch.set_float32_matmul_precision\n", "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]\n", "\n", " | Name | Type | Params\n", "---------------------------------------------------------------\n", "0 | _backbone | CategoryEmbeddingBackbone | 11.4 K\n", "1 | _embedding_layer | Embedding1dLayer | 14 \n", "2 | head | LinearHead | 33 \n", "3 | loss | MSELoss | 0 \n", "---------------------------------------------------------------\n", "11.4 K Trainable params\n", "0 Non-trainable params\n", "11.4 K Total params\n", "0.046 Total estimated model params size (MB)\n", "Epoch: 1/300, Train loss: 673.4464, Val loss: 604.0322, Min val loss: 604.0322, Epoch time: 0.017s.\n", "Epoch: 20/300, Train loss: 354.8526, Val loss: 304.8196, Min val loss: 304.8196, Epoch time: 0.009s.\n", "Epoch: 40/300, Train loss: 83.0153, Val loss: 65.4248, Min val loss: 65.4248, Epoch time: 0.008s.\n", "Epoch: 60/300, Train loss: 48.7386, Val loss: 33.2992, Min val loss: 33.2992, Epoch time: 0.009s.\n", "Epoch: 80/300, Train loss: 34.5437, Val loss: 25.6234, Min val loss: 25.6234, Epoch time: 0.010s.\n", "Epoch: 100/300, Train loss: 30.8209, Val loss: 21.4857, Min val loss: 21.4857, Epoch time: 0.010s.\n", "Epoch: 120/300, Train loss: 21.8472, Val loss: 18.3320, Min val loss: 18.3320, Epoch time: 0.010s.\n", "Epoch: 140/300, Train loss: 23.5546, Val loss: 15.9204, Min val loss: 15.9204, Epoch time: 0.008s.\n", "Epoch: 160/300, Train loss: 21.0466, Val loss: 14.0073, Min val loss: 14.0073, Epoch time: 0.009s.\n", "Epoch: 180/300, Train loss: 19.9606, Val loss: 12.7407, Min val loss: 12.7407, Epoch time: 0.008s.\n", "Epoch: 200/300, Train loss: 19.3325, Val loss: 11.2465, Min val loss: 11.2465, Epoch time: 0.010s.\n", "Epoch: 220/300, Train loss: 14.9372, Val loss: 10.2214, Min val loss: 10.2214, Epoch time: 0.010s.\n", "Epoch: 240/300, Train loss: 17.1906, Val loss: 9.6617, Min val loss: 9.4835, Epoch time: 0.008s.\n", "Epoch: 260/300, Train loss: 17.4219, Val loss: 8.9945, Min val loss: 8.9184, Epoch time: 0.008s.\n", "Epoch: 280/300, Train loss: 14.6068, Val loss: 8.4856, Min val loss: 8.4856, Epoch time: 0.008s.\n", "Epoch: 300/300, Train loss: 12.2139, Val loss: 8.1110, Min val loss: 8.0176, Epoch time: 0.009s.\n", "`Trainer.fit` stopped: `max_epochs=300` reached.\n", "2023-09-23 20:38:33,382 - {pytorch_tabular.tabular_model:584} - INFO - Training the model completed\n", "2023-09-23 20:38:33,382 - {pytorch_tabular.tabular_model:1258} - INFO - Loading the best model\n", "/home/xlluo/anaconda3/envs/tabular_ensemble/lib/python3.10/site-packages/pytorch_lightning/utilities/cloud_io.py:33: LightningDeprecationWarning: `pytorch_lightning.utilities.cloud_io.get_filesystem` has been deprecated in v1.8.0 and will be removed in v2.0.0. Please use `lightning_fabric.utilities.cloud_io.get_filesystem` instead.\n", " rank_zero_deprecation(\n", "Training mse loss: 10.11050\n", "Validation mse loss: 8.01764\n", "Testing mse loss: 8.99186\n", "Trainer saved. To load the trainer, run trainer = load_trainer(path='/tmp/tmpnpjgki5b/output/auto-mpg/2023-09-23-20-37-47-0_UserInputConfig/trainer.pkl')\n", "\n", "-------------PytorchTabular End-------------\n", "\n", "Category Embedding 1/1\n", "--------------------------End 4/4 random--------------------------\n", "Trainer saved. To load the trainer, run trainer = load_trainer(path='/tmp/tmpnpjgki5b/output/auto-mpg/2023-09-23-20-37-47-0_UserInputConfig/trainer.pkl')\n" ] }, { "data": { "text/plain": " Program Model Training RMSE Training MSE \\\n0 PytorchTabular Category Embedding 3.195748 10.212808 \n\n Training MAE Training MAPE Training R2 Training MEDIAN_ABSOLUTE_ERROR \\\n0 2.387656 0.101595 0.838209 1.844076 \n\n Training EXPLAINED_VARIANCE_SCORE Testing RMSE ... Testing R2 \\\n0 0.865946 3.199201 ... 0.822679 \n\n Testing MEDIAN_ABSOLUTE_ERROR Testing EXPLAINED_VARIANCE_SCORE \\\n0 2.022343 0.856969 \n\n Validation RMSE Validation MSE Validation MAE Validation MAPE \\\n0 3.620434 13.107543 2.541972 0.11167 \n\n Validation R2 Validation MEDIAN_ABSOLUTE_ERROR \\\n0 0.771539 1.739592 \n\n Validation EXPLAINED_VARIANCE_SCORE \n0 0.783368 \n\n[1 rows x 23 columns]", "text/html": "
\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
ProgramModelTraining RMSETraining MSETraining MAETraining MAPETraining R2Training MEDIAN_ABSOLUTE_ERRORTraining EXPLAINED_VARIANCE_SCORETesting RMSE...Testing R2Testing MEDIAN_ABSOLUTE_ERRORTesting EXPLAINED_VARIANCE_SCOREValidation RMSEValidation MSEValidation MAEValidation MAPEValidation R2Validation MEDIAN_ABSOLUTE_ERRORValidation EXPLAINED_VARIANCE_SCORE
0PytorchTabularCategory Embedding3.19574810.2128082.3876560.1015950.8382091.8440760.8659463.199201...0.8226792.0223430.8569693.62043413.1075432.5419720.111670.7715391.7395920.783368
\n

1 rows × 23 columns

\n
" }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "trainer.get_leaderboard(cross_validation=4, split_type=\"random\", stderr_to_stdout=True)" ], "metadata": { "collapsed": false, "pycharm": { "name": "#%%\n" } } }, { "cell_type": "markdown", "source": [ "## Unexpected termination\n", "\n", "It may take quite a long time to cross-validate various models on the large dataset, especially with Bayesian hyperparameter optimization (Yes, Bayesian hyperparameter optimization and cross-validation can both be activated). If the script terminates unexpectedly, you can use a functionality that loads the stored cross-validation state to continue a previous execution.\n", "\n", "First, we assume that the script terminates after the first run finishes." ], "metadata": { "collapsed": false, "pycharm": { "name": "#%% md\n" } } }, { "cell_type": "code", "execution_count": 4, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "----------------------------1/1 random----------------------------\n", "Using previously used data path /tmp/tmpnpjgki5b/data/auto-mpg.csv\n", "Dataset size: 238 80 80\n", "Data saved to /tmp/tmpnpjgki5b/output/auto-mpg/2023-09-23-20-37-47-0_UserInputConfig (data.csv and tabular_data.csv).\n", "\n", "-------------Run PytorchTabular-------------\n", "\n", "Training Category Embedding\n", "Global seed set to 42\n", "2023-09-23 20:38:33,947 - {pytorch_tabular.tabular_model:473} - INFO - Preparing the DataLoaders\n", "2023-09-23 20:38:33,947 - {pytorch_tabular.tabular_datamodule:290} - INFO - Setting up the datamodule for regression task\n", "2023-09-23 20:38:33,960 - {pytorch_tabular.tabular_model:521} - INFO - Preparing the Model: CategoryEmbeddingModel\n", "2023-09-23 20:38:33,977 - {pytorch_tabular.tabular_model:268} - INFO - Preparing the Trainer\n", "/home/xlluo/anaconda3/envs/tabular_ensemble/lib/python3.10/site-packages/pytorch_lightning/trainer/connectors/accelerator_connector.py:589: LightningDeprecationWarning: The Trainer argument `auto_select_gpus` has been deprecated in v1.9.0 and will be removed in v2.0.0. Please use the function `pytorch_lightning.accelerators.find_usable_cuda_devices` instead.\n", " rank_zero_deprecation(\n", "Auto select gpus: [0]\n", "GPU available: True (cuda), used: True\n", "TPU available: False, using: 0 TPU cores\n", "IPU available: False, using: 0 IPUs\n", "HPU available: False, using: 0 HPUs\n", "2023-09-23 20:38:33,993 - {pytorch_tabular.tabular_model:582} - INFO - Training Started\n", "You are using a CUDA device ('NVIDIA GeForce RTX 3090') that has Tensor Cores. To properly utilize them, you should set `torch.set_float32_matmul_precision('medium' | 'high')` which will trade-off precision for performance. For more details, read https://pytorch.org/docs/stable/generated/torch.set_float32_matmul_precision.html#torch.set_float32_matmul_precision\n", "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]\n", "\n", " | Name | Type | Params\n", "---------------------------------------------------------------\n", "0 | _backbone | CategoryEmbeddingBackbone | 11.4 K\n", "1 | _embedding_layer | Embedding1dLayer | 14 \n", "2 | head | LinearHead | 33 \n", "3 | loss | MSELoss | 0 \n", "---------------------------------------------------------------\n", "11.4 K Trainable params\n", "0 Non-trainable params\n", "11.4 K Total params\n", "0.046 Total estimated model params size (MB)\n", "Epoch: 1/300, Train loss: 677.8015, Val loss: 582.9557, Min val loss: 582.9557, Epoch time: 0.010s.\n", "Epoch: 20/300, Train loss: 353.7851, Val loss: 302.0203, Min val loss: 302.0203, Epoch time: 0.009s.\n", "Epoch: 40/300, Train loss: 85.0776, Val loss: 62.1153, Min val loss: 62.1153, Epoch time: 0.009s.\n", "Epoch: 60/300, Train loss: 45.2654, Val loss: 34.2778, Min val loss: 34.2691, Epoch time: 0.009s.\n", "Epoch: 80/300, Train loss: 33.9537, Val loss: 26.8622, Min val loss: 26.8622, Epoch time: 0.008s.\n", "Epoch: 100/300, Train loss: 26.9038, Val loss: 23.2417, Min val loss: 23.2372, Epoch time: 0.011s.\n", "Epoch: 120/300, Train loss: 24.9622, Val loss: 20.4360, Min val loss: 20.4360, Epoch time: 0.008s.\n", "Epoch: 140/300, Train loss: 24.1636, Val loss: 19.4010, Min val loss: 19.4010, Epoch time: 0.010s.\n", "Epoch: 160/300, Train loss: 22.9200, Val loss: 18.0232, Min val loss: 17.9749, Epoch time: 0.009s.\n", "Epoch: 180/300, Train loss: 19.7677, Val loss: 16.9469, Min val loss: 16.9469, Epoch time: 0.011s.\n", "Epoch: 200/300, Train loss: 17.9390, Val loss: 16.6545, Min val loss: 16.4093, Epoch time: 0.011s.\n", "Epoch: 220/300, Train loss: 19.4496, Val loss: 15.4451, Min val loss: 15.1788, Epoch time: 0.008s.\n", "Epoch: 240/300, Train loss: 16.0483, Val loss: 14.5508, Min val loss: 14.5508, Epoch time: 0.012s.\n", "Epoch: 260/300, Train loss: 16.4672, Val loss: 13.8354, Min val loss: 13.8354, Epoch time: 0.009s.\n", "Epoch: 280/300, Train loss: 13.6031, Val loss: 12.9315, Min val loss: 12.9315, Epoch time: 0.009s.\n", "Epoch: 300/300, Train loss: 16.5369, Val loss: 12.3673, Min val loss: 12.3673, Epoch time: 0.010s.\n", "`Trainer.fit` stopped: `max_epochs=300` reached.\n", "2023-09-23 20:38:38,411 - {pytorch_tabular.tabular_model:584} - INFO - Training the model completed\n", "2023-09-23 20:38:38,412 - {pytorch_tabular.tabular_model:1258} - INFO - Loading the best model\n", "/home/xlluo/anaconda3/envs/tabular_ensemble/lib/python3.10/site-packages/pytorch_lightning/utilities/cloud_io.py:33: LightningDeprecationWarning: `pytorch_lightning.utilities.cloud_io.get_filesystem` has been deprecated in v1.8.0 and will be removed in v2.0.0. Please use `lightning_fabric.utilities.cloud_io.get_filesystem` instead.\n", " rank_zero_deprecation(\n", "Training mse loss: 11.25175\n", "Validation mse loss: 12.36725\n", "Testing mse loss: 7.83801\n", "Trainer saved. To load the trainer, run trainer = load_trainer(path='/tmp/tmpnpjgki5b/output/auto-mpg/2023-09-23-20-37-47-0_UserInputConfig/trainer.pkl')\n", "\n", "-------------PytorchTabular End-------------\n", "\n", "Category Embedding 1/1\n", "--------------------------End 1/1 random--------------------------\n", "Trainer saved. To load the trainer, run trainer = load_trainer(path='/tmp/tmpnpjgki5b/output/auto-mpg/2023-09-23-20-37-47-0_UserInputConfig/trainer.pkl')\n" ] } ], "source": [ "_ = trainer.get_leaderboard(cross_validation=1, split_type=\"random\", stderr_to_stdout=True)" ], "metadata": { "collapsed": false, "pycharm": { "name": "#%%\n" } } }, { "cell_type": "markdown", "source": [ "To continue the cross-validation, set the argument `load_from_previous` to `True`" ], "metadata": { "collapsed": false, "pycharm": { "name": "#%% md\n" } } }, { "cell_type": "code", "execution_count": 5, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Previous cross validation state is loaded.\n", "----------------------------2/2 random----------------------------\n", "Using previously used data path /tmp/tmpnpjgki5b/data/auto-mpg.csv\n", "Dataset size: 238 80 80\n", "Data saved to /tmp/tmpnpjgki5b/output/auto-mpg/2023-09-23-20-37-47-0_UserInputConfig (data.csv and tabular_data.csv).\n", "\n", "-------------Run PytorchTabular-------------\n", "\n", "Training Category Embedding\n", "Global seed set to 42\n", "2023-09-23 20:38:38,911 - {pytorch_tabular.tabular_model:473} - INFO - Preparing the DataLoaders\n", "2023-09-23 20:38:38,912 - {pytorch_tabular.tabular_datamodule:290} - INFO - Setting up the datamodule for regression task\n", "2023-09-23 20:38:38,920 - {pytorch_tabular.tabular_model:521} - INFO - Preparing the Model: CategoryEmbeddingModel\n", "2023-09-23 20:38:38,932 - {pytorch_tabular.tabular_model:268} - INFO - Preparing the Trainer\n", "/home/xlluo/anaconda3/envs/tabular_ensemble/lib/python3.10/site-packages/pytorch_lightning/trainer/connectors/accelerator_connector.py:589: LightningDeprecationWarning: The Trainer argument `auto_select_gpus` has been deprecated in v1.9.0 and will be removed in v2.0.0. Please use the function `pytorch_lightning.accelerators.find_usable_cuda_devices` instead.\n", " rank_zero_deprecation(\n", "Auto select gpus: [0]\n", "GPU available: True (cuda), used: True\n", "TPU available: False, using: 0 TPU cores\n", "IPU available: False, using: 0 IPUs\n", "HPU available: False, using: 0 HPUs\n", "2023-09-23 20:38:38,946 - {pytorch_tabular.tabular_model:582} - INFO - Training Started\n", "You are using a CUDA device ('NVIDIA GeForce RTX 3090') that has Tensor Cores. To properly utilize them, you should set `torch.set_float32_matmul_precision('medium' | 'high')` which will trade-off precision for performance. For more details, read https://pytorch.org/docs/stable/generated/torch.set_float32_matmul_precision.html#torch.set_float32_matmul_precision\n", "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]\n", "\n", " | Name | Type | Params\n", "---------------------------------------------------------------\n", "0 | _backbone | CategoryEmbeddingBackbone | 11.4 K\n", "1 | _embedding_layer | Embedding1dLayer | 14 \n", "2 | head | LinearHead | 33 \n", "3 | loss | MSELoss | 0 \n", "---------------------------------------------------------------\n", "11.4 K Trainable params\n", "0 Non-trainable params\n", "11.4 K Total params\n", "0.046 Total estimated model params size (MB)\n", "Epoch: 1/300, Train loss: 632.3258, Val loss: 662.7650, Min val loss: 662.7650, Epoch time: 0.010s.\n", "Epoch: 20/300, Train loss: 318.3275, Val loss: 344.3962, Min val loss: 344.3962, Epoch time: 0.012s.\n", "Epoch: 40/300, Train loss: 69.3076, Val loss: 76.5520, Min val loss: 76.5520, Epoch time: 0.012s.\n", "Epoch: 60/300, Train loss: 42.2416, Val loss: 46.7150, Min val loss: 46.7150, Epoch time: 0.014s.\n", "Epoch: 80/300, Train loss: 29.2576, Val loss: 34.4237, Min val loss: 34.4237, Epoch time: 0.012s.\n", "Epoch: 100/300, Train loss: 26.1472, Val loss: 29.0749, Min val loss: 29.0749, Epoch time: 0.012s.\n", "Epoch: 120/300, Train loss: 24.8970, Val loss: 26.5129, Min val loss: 26.5129, Epoch time: 0.008s.\n", "Epoch: 140/300, Train loss: 17.9044, Val loss: 25.3725, Min val loss: 25.3725, Epoch time: 0.009s.\n", "Epoch: 160/300, Train loss: 17.4626, Val loss: 23.7306, Min val loss: 23.7306, Epoch time: 0.009s.\n", "Epoch: 180/300, Train loss: 16.5023, Val loss: 22.9689, Min val loss: 22.9689, Epoch time: 0.011s.\n", "Epoch: 200/300, Train loss: 15.1315, Val loss: 22.0908, Min val loss: 22.0908, Epoch time: 0.013s.\n", "Epoch: 220/300, Train loss: 14.4462, Val loss: 21.3679, Min val loss: 21.3679, Epoch time: 0.010s.\n", "Epoch: 240/300, Train loss: 15.4215, Val loss: 20.7016, Min val loss: 20.6893, Epoch time: 0.008s.\n", "Epoch: 260/300, Train loss: 13.5062, Val loss: 20.0930, Min val loss: 20.0624, Epoch time: 0.009s.\n", "Epoch: 280/300, Train loss: 14.5558, Val loss: 19.2505, Min val loss: 19.2505, Epoch time: 0.010s.\n", "Epoch: 300/300, Train loss: 14.8346, Val loss: 18.8435, Min val loss: 18.8435, Epoch time: 0.009s.\n", "`Trainer.fit` stopped: `max_epochs=300` reached.\n", "2023-09-23 20:38:43,583 - {pytorch_tabular.tabular_model:584} - INFO - Training the model completed\n", "2023-09-23 20:38:43,584 - {pytorch_tabular.tabular_model:1258} - INFO - Loading the best model\n", "/home/xlluo/anaconda3/envs/tabular_ensemble/lib/python3.10/site-packages/pytorch_lightning/utilities/cloud_io.py:33: LightningDeprecationWarning: `pytorch_lightning.utilities.cloud_io.get_filesystem` has been deprecated in v1.8.0 and will be removed in v2.0.0. Please use `lightning_fabric.utilities.cloud_io.get_filesystem` instead.\n", " rank_zero_deprecation(\n", "Training mse loss: 10.18515\n", "Validation mse loss: 18.84349\n", "Testing mse loss: 9.93399\n", "Trainer saved. To load the trainer, run trainer = load_trainer(path='/tmp/tmpnpjgki5b/output/auto-mpg/2023-09-23-20-37-47-0_UserInputConfig/trainer.pkl')\n", "\n", "-------------PytorchTabular End-------------\n", "\n", "Category Embedding 1/1\n", "--------------------------End 2/2 random--------------------------\n", "Trainer saved. To load the trainer, run trainer = load_trainer(path='/tmp/tmpnpjgki5b/output/auto-mpg/2023-09-23-20-37-47-0_UserInputConfig/trainer.pkl')\n" ] }, { "data": { "text/plain": " Program Model Training RMSE Training MSE \\\n0 PytorchTabular Category Embedding 3.273904 10.718448 \n\n Training MAE Training MAPE Training R2 Training MEDIAN_ABSOLUTE_ERROR \\\n0 2.439552 0.103957 0.827288 1.87891 \n\n Training EXPLAINED_VARIANCE_SCORE Testing RMSE ... Testing R2 \\\n0 0.857783 2.980939 ... 0.844204 \n\n Testing MEDIAN_ABSOLUTE_ERROR Testing EXPLAINED_VARIANCE_SCORE \\\n0 1.972121 0.890054 \n\n Validation RMSE Validation MSE Validation MAE Validation MAPE \\\n0 3.950363 15.605369 2.875634 0.123573 \n\n Validation R2 Validation MEDIAN_ABSOLUTE_ERROR \\\n0 0.745909 2.082971 \n\n Validation EXPLAINED_VARIANCE_SCORE \n0 0.763982 \n\n[1 rows x 23 columns]", "text/html": "
\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
ProgramModelTraining RMSETraining MSETraining MAETraining MAPETraining R2Training MEDIAN_ABSOLUTE_ERRORTraining EXPLAINED_VARIANCE_SCORETesting RMSE...Testing R2Testing MEDIAN_ABSOLUTE_ERRORTesting EXPLAINED_VARIANCE_SCOREValidation RMSEValidation MSEValidation MAEValidation MAPEValidation R2Validation MEDIAN_ABSOLUTE_ERRORValidation EXPLAINED_VARIANCE_SCORE
0PytorchTabularCategory Embedding3.27390410.7184482.4395520.1039570.8272881.878910.8577832.980939...0.8442041.9721210.8900543.95036315.6053692.8756340.1235730.7459092.0829710.763982
\n

1 rows × 23 columns

\n
" }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "l1 = trainer.get_leaderboard(cross_validation=2, split_type=\"random\", stderr_to_stdout=True, load_from_previous=True)\n", "l1" ], "metadata": { "collapsed": false, "pycharm": { "name": "#%%\n" } } }, { "cell_type": "markdown", "source": [ "Let's compare the result without termination." ], "metadata": { "collapsed": false, "pycharm": { "name": "#%% md\n" } } }, { "cell_type": "code", "execution_count": 6, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "----------------------------1/2 random----------------------------\n", "Using previously used data path /tmp/tmpnpjgki5b/data/auto-mpg.csv\n", "Dataset size: 238 80 80\n", "Data saved to /tmp/tmpnpjgki5b/output/auto-mpg/2023-09-23-20-37-47-0_UserInputConfig (data.csv and tabular_data.csv).\n", "\n", "-------------Run PytorchTabular-------------\n", "\n", "Training Category Embedding\n", "Global seed set to 42\n", "2023-09-23 20:38:44,123 - {pytorch_tabular.tabular_model:473} - INFO - Preparing the DataLoaders\n", "2023-09-23 20:38:44,124 - {pytorch_tabular.tabular_datamodule:290} - INFO - Setting up the datamodule for regression task\n", "2023-09-23 20:38:44,137 - {pytorch_tabular.tabular_model:521} - INFO - Preparing the Model: CategoryEmbeddingModel\n", "2023-09-23 20:38:44,148 - {pytorch_tabular.tabular_model:268} - INFO - Preparing the Trainer\n", "/home/xlluo/anaconda3/envs/tabular_ensemble/lib/python3.10/site-packages/pytorch_lightning/trainer/connectors/accelerator_connector.py:589: LightningDeprecationWarning: The Trainer argument `auto_select_gpus` has been deprecated in v1.9.0 and will be removed in v2.0.0. Please use the function `pytorch_lightning.accelerators.find_usable_cuda_devices` instead.\n", " rank_zero_deprecation(\n", "Auto select gpus: [0]\n", "GPU available: True (cuda), used: True\n", "TPU available: False, using: 0 TPU cores\n", "IPU available: False, using: 0 IPUs\n", "HPU available: False, using: 0 HPUs\n", "2023-09-23 20:38:44,162 - {pytorch_tabular.tabular_model:582} - INFO - Training Started\n", "You are using a CUDA device ('NVIDIA GeForce RTX 3090') that has Tensor Cores. To properly utilize them, you should set `torch.set_float32_matmul_precision('medium' | 'high')` which will trade-off precision for performance. For more details, read https://pytorch.org/docs/stable/generated/torch.set_float32_matmul_precision.html#torch.set_float32_matmul_precision\n", "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]\n", "\n", " | Name | Type | Params\n", "---------------------------------------------------------------\n", "0 | _backbone | CategoryEmbeddingBackbone | 11.4 K\n", "1 | _embedding_layer | Embedding1dLayer | 14 \n", "2 | head | LinearHead | 33 \n", "3 | loss | MSELoss | 0 \n", "---------------------------------------------------------------\n", "11.4 K Trainable params\n", "0 Non-trainable params\n", "11.4 K Total params\n", "0.046 Total estimated model params size (MB)\n", "Epoch: 1/300, Train loss: 677.8015, Val loss: 582.9557, Min val loss: 582.9557, Epoch time: 0.011s.\n", "Epoch: 20/300, Train loss: 353.7851, Val loss: 302.0203, Min val loss: 302.0203, Epoch time: 0.008s.\n", "Epoch: 40/300, Train loss: 85.0776, Val loss: 62.1153, Min val loss: 62.1153, Epoch time: 0.008s.\n", "Epoch: 60/300, Train loss: 45.2654, Val loss: 34.2778, Min val loss: 34.2691, Epoch time: 0.009s.\n", "Epoch: 80/300, Train loss: 33.9537, Val loss: 26.8622, Min val loss: 26.8622, Epoch time: 0.009s.\n", "Epoch: 100/300, Train loss: 26.9038, Val loss: 23.2417, Min val loss: 23.2372, Epoch time: 0.008s.\n", "Epoch: 120/300, Train loss: 24.9622, Val loss: 20.4360, Min val loss: 20.4360, Epoch time: 0.009s.\n", "Epoch: 140/300, Train loss: 24.1636, Val loss: 19.4010, Min val loss: 19.4010, Epoch time: 0.010s.\n", "Epoch: 160/300, Train loss: 22.9200, Val loss: 18.0232, Min val loss: 17.9749, Epoch time: 0.009s.\n", "Epoch: 180/300, Train loss: 19.7677, Val loss: 16.9469, Min val loss: 16.9469, Epoch time: 0.013s.\n", "Epoch: 200/300, Train loss: 17.9390, Val loss: 16.6545, Min val loss: 16.4093, Epoch time: 0.008s.\n", "Epoch: 220/300, Train loss: 19.4496, Val loss: 15.4451, Min val loss: 15.1788, Epoch time: 0.008s.\n", "Epoch: 240/300, Train loss: 16.0483, Val loss: 14.5508, Min val loss: 14.5508, Epoch time: 0.009s.\n", "Epoch: 260/300, Train loss: 16.4672, Val loss: 13.8354, Min val loss: 13.8354, Epoch time: 0.009s.\n", "Epoch: 280/300, Train loss: 13.6031, Val loss: 12.9315, Min val loss: 12.9315, Epoch time: 0.009s.\n", "Epoch: 300/300, Train loss: 16.5369, Val loss: 12.3673, Min val loss: 12.3673, Epoch time: 0.012s.\n", "`Trainer.fit` stopped: `max_epochs=300` reached.\n", "2023-09-23 20:38:48,081 - {pytorch_tabular.tabular_model:584} - INFO - Training the model completed\n", "2023-09-23 20:38:48,081 - {pytorch_tabular.tabular_model:1258} - INFO - Loading the best model\n", "/home/xlluo/anaconda3/envs/tabular_ensemble/lib/python3.10/site-packages/pytorch_lightning/utilities/cloud_io.py:33: LightningDeprecationWarning: `pytorch_lightning.utilities.cloud_io.get_filesystem` has been deprecated in v1.8.0 and will be removed in v2.0.0. Please use `lightning_fabric.utilities.cloud_io.get_filesystem` instead.\n", " rank_zero_deprecation(\n", "Training mse loss: 11.25175\n", "Validation mse loss: 12.36725\n", "Testing mse loss: 7.83801\n", "Trainer saved. To load the trainer, run trainer = load_trainer(path='/tmp/tmpnpjgki5b/output/auto-mpg/2023-09-23-20-37-47-0_UserInputConfig/trainer.pkl')\n", "\n", "-------------PytorchTabular End-------------\n", "\n", "Category Embedding 1/1\n", "--------------------------End 1/2 random--------------------------\n", "----------------------------2/2 random----------------------------\n", "Using previously used data path /tmp/tmpnpjgki5b/data/auto-mpg.csv\n", "Dataset size: 238 80 80\n", "Data saved to /tmp/tmpnpjgki5b/output/auto-mpg/2023-09-23-20-37-47-0_UserInputConfig (data.csv and tabular_data.csv).\n", "\n", "-------------Run PytorchTabular-------------\n", "\n", "Training Category Embedding\n", "Global seed set to 42\n", "2023-09-23 20:38:48,582 - {pytorch_tabular.tabular_model:473} - INFO - Preparing the DataLoaders\n", "2023-09-23 20:38:48,583 - {pytorch_tabular.tabular_datamodule:290} - INFO - Setting up the datamodule for regression task\n", "2023-09-23 20:38:48,593 - {pytorch_tabular.tabular_model:521} - INFO - Preparing the Model: CategoryEmbeddingModel\n", "2023-09-23 20:38:48,607 - {pytorch_tabular.tabular_model:268} - INFO - Preparing the Trainer\n", "/home/xlluo/anaconda3/envs/tabular_ensemble/lib/python3.10/site-packages/pytorch_lightning/trainer/connectors/accelerator_connector.py:589: LightningDeprecationWarning: The Trainer argument `auto_select_gpus` has been deprecated in v1.9.0 and will be removed in v2.0.0. Please use the function `pytorch_lightning.accelerators.find_usable_cuda_devices` instead.\n", " rank_zero_deprecation(\n", "Auto select gpus: [0]\n", "GPU available: True (cuda), used: True\n", "TPU available: False, using: 0 TPU cores\n", "IPU available: False, using: 0 IPUs\n", "HPU available: False, using: 0 HPUs\n", "2023-09-23 20:38:48,620 - {pytorch_tabular.tabular_model:582} - INFO - Training Started\n", "You are using a CUDA device ('NVIDIA GeForce RTX 3090') that has Tensor Cores. To properly utilize them, you should set `torch.set_float32_matmul_precision('medium' | 'high')` which will trade-off precision for performance. For more details, read https://pytorch.org/docs/stable/generated/torch.set_float32_matmul_precision.html#torch.set_float32_matmul_precision\n", "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]\n", "\n", " | Name | Type | Params\n", "---------------------------------------------------------------\n", "0 | _backbone | CategoryEmbeddingBackbone | 11.4 K\n", "1 | _embedding_layer | Embedding1dLayer | 14 \n", "2 | head | LinearHead | 33 \n", "3 | loss | MSELoss | 0 \n", "---------------------------------------------------------------\n", "11.4 K Trainable params\n", "0 Non-trainable params\n", "11.4 K Total params\n", "0.046 Total estimated model params size (MB)\n", "Epoch: 1/300, Train loss: 632.3258, Val loss: 662.7650, Min val loss: 662.7650, Epoch time: 0.012s.\n", "Epoch: 20/300, Train loss: 318.3275, Val loss: 344.3962, Min val loss: 344.3962, Epoch time: 0.007s.\n", "Epoch: 40/300, Train loss: 69.3076, Val loss: 76.5520, Min val loss: 76.5520, Epoch time: 0.008s.\n", "Epoch: 60/300, Train loss: 42.2416, Val loss: 46.7150, Min val loss: 46.7150, Epoch time: 0.008s.\n", "Epoch: 80/300, Train loss: 29.2576, Val loss: 34.4237, Min val loss: 34.4237, Epoch time: 0.009s.\n", "Epoch: 100/300, Train loss: 26.1472, Val loss: 29.0749, Min val loss: 29.0749, Epoch time: 0.008s.\n", "Epoch: 120/300, Train loss: 24.8970, Val loss: 26.5129, Min val loss: 26.5129, Epoch time: 0.008s.\n", "Epoch: 140/300, Train loss: 17.9044, Val loss: 25.3725, Min val loss: 25.3725, Epoch time: 0.008s.\n", "Epoch: 160/300, Train loss: 17.4626, Val loss: 23.7306, Min val loss: 23.7306, Epoch time: 0.010s.\n", "Epoch: 180/300, Train loss: 16.5023, Val loss: 22.9689, Min val loss: 22.9689, Epoch time: 0.008s.\n", "Epoch: 200/300, Train loss: 15.1315, Val loss: 22.0908, Min val loss: 22.0908, Epoch time: 0.008s.\n", "Epoch: 220/300, Train loss: 14.4462, Val loss: 21.3679, Min val loss: 21.3679, Epoch time: 0.008s.\n", "Epoch: 240/300, Train loss: 15.4215, Val loss: 20.7016, Min val loss: 20.6893, Epoch time: 0.009s.\n", "Epoch: 260/300, Train loss: 13.5062, Val loss: 20.0930, Min val loss: 20.0624, Epoch time: 0.010s.\n", "Epoch: 280/300, Train loss: 14.5558, Val loss: 19.2505, Min val loss: 19.2505, Epoch time: 0.008s.\n", "Epoch: 300/300, Train loss: 14.8346, Val loss: 18.8435, Min val loss: 18.8435, Epoch time: 0.009s.\n", "`Trainer.fit` stopped: `max_epochs=300` reached.\n", "2023-09-23 20:38:52,437 - {pytorch_tabular.tabular_model:584} - INFO - Training the model completed\n", "2023-09-23 20:38:52,437 - {pytorch_tabular.tabular_model:1258} - INFO - Loading the best model\n", "/home/xlluo/anaconda3/envs/tabular_ensemble/lib/python3.10/site-packages/pytorch_lightning/utilities/cloud_io.py:33: LightningDeprecationWarning: `pytorch_lightning.utilities.cloud_io.get_filesystem` has been deprecated in v1.8.0 and will be removed in v2.0.0. Please use `lightning_fabric.utilities.cloud_io.get_filesystem` instead.\n", " rank_zero_deprecation(\n", "Training mse loss: 10.18515\n", "Validation mse loss: 18.84349\n", "Testing mse loss: 9.93399\n", "Trainer saved. To load the trainer, run trainer = load_trainer(path='/tmp/tmpnpjgki5b/output/auto-mpg/2023-09-23-20-37-47-0_UserInputConfig/trainer.pkl')\n", "\n", "-------------PytorchTabular End-------------\n", "\n", "Category Embedding 1/1\n", "--------------------------End 2/2 random--------------------------\n", "Trainer saved. To load the trainer, run trainer = load_trainer(path='/tmp/tmpnpjgki5b/output/auto-mpg/2023-09-23-20-37-47-0_UserInputConfig/trainer.pkl')\n" ] }, { "data": { "text/plain": " Program Model Training RMSE Training MSE \\\n0 PytorchTabular Category Embedding 3.273904 10.718448 \n\n Training MAE Training MAPE Training R2 Training MEDIAN_ABSOLUTE_ERROR \\\n0 2.439552 0.103957 0.827288 1.87891 \n\n Training EXPLAINED_VARIANCE_SCORE Testing RMSE ... Testing R2 \\\n0 0.857783 2.980939 ... 0.844204 \n\n Testing MEDIAN_ABSOLUTE_ERROR Testing EXPLAINED_VARIANCE_SCORE \\\n0 1.972121 0.890054 \n\n Validation RMSE Validation MSE Validation MAE Validation MAPE \\\n0 3.950363 15.605369 2.875634 0.123573 \n\n Validation R2 Validation MEDIAN_ABSOLUTE_ERROR \\\n0 0.745909 2.082971 \n\n Validation EXPLAINED_VARIANCE_SCORE \n0 0.763982 \n\n[1 rows x 23 columns]", "text/html": "
\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
ProgramModelTraining RMSETraining MSETraining MAETraining MAPETraining R2Training MEDIAN_ABSOLUTE_ERRORTraining EXPLAINED_VARIANCE_SCORETesting RMSE...Testing R2Testing MEDIAN_ABSOLUTE_ERRORTesting EXPLAINED_VARIANCE_SCOREValidation RMSEValidation MSEValidation MAEValidation MAPEValidation R2Validation MEDIAN_ABSOLUTE_ERRORValidation EXPLAINED_VARIANCE_SCORE
0PytorchTabularCategory Embedding3.27390410.7184482.4395520.1039570.8272881.878910.8577832.980939...0.8442041.9721210.8900543.95036315.6053692.8756340.1235730.7459092.0829710.763982
\n

1 rows × 23 columns

\n
" }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "l2 = trainer.get_leaderboard(cross_validation=2, split_type=\"random\", stderr_to_stdout=True)\n", "\n", "import numpy as np\n", "cols = [\"Training RMSE\", \"Testing RMSE\", \"Validation RMSE\"]\n", "assert np.allclose(l1[cols].values.astype(float), l2[cols].values.astype(float))\n", "l2" ], "metadata": { "collapsed": false, "pycharm": { "name": "#%%\n" } } } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 2 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython2", "version": "2.7.6" } }, "nbformat": 4, "nbformat_minor": 0 }