Cross-validation#
Cross-validation is required to validate the generalization ability of models, avoid the effect of randomization, etc. Randomization may affect the dataset splitting, model initialization, forward propagation (especially convolution operations), and optimization.
[1]:
import torch
from tabensemb.trainer import Trainer
from tabensemb.model import *
from tabensemb.config import UserConfig
import tabensemb
import os
device = "cuda" if torch.cuda.is_available() else "cpu"
print("Using {} device".format(device))
from tempfile import TemporaryDirectory
temp_path = TemporaryDirectory()
tabensemb.setting["default_output_path"] = os.path.join(temp_path.name, "output")
tabensemb.setting["default_config_path"] = os.path.join(temp_path.name, "configs")
tabensemb.setting["default_data_path"] = os.path.join(temp_path.name, "data")
trainer = Trainer(device=device)
mpg_columns = [
"mpg",
"cylinders",
"displacement",
"horsepower",
"weight",
"acceleration",
"model_year",
"origin",
"car_name",
]
cfg = UserConfig.from_uci("Auto MPG", column_names=mpg_columns, sep=r"\s+")
trainer.load_config(cfg)
trainer.load_data()
models = [
PytorchTabular(trainer, model_subset=["Category Embedding"]),
]
trainer.add_modelbases(models)
Using cuda device
Downloading https://archive.ics.uci.edu/static/public/9/auto+mpg.zip to /tmp/tmpnpjgki5b/data/Auto MPG.zip
cylinders is Integer and will be treated as a continuous feature.
model_year is Integer and will be treated as a continuous feature.
origin is Integer and will be treated as a continuous feature.
Unknown values are detected in ['horsepower']. They will be treated as np.nan.
The project will be saved to /tmp/tmpnpjgki5b/output/auto-mpg/2023-09-23-20-37-47-0_UserInputConfig
Dataset size: 238 80 80
Data saved to /tmp/tmpnpjgki5b/output/auto-mpg/2023-09-23-20-37-47-0_UserInputConfig (data.csv and tabular_data.csv).
K-fold cross-validation#
Some of the data splitters (See “Using data functionalities”) in tabensemb support k-fold cross-validation. To activate k-fold CV, pass the argument split_type="cv" to Trainer.get_leaderboard. In this case, the ratio of training/validation/testing sets is (k-2):1:1. Here we present an example of a 4-fold CV.
[2]:
trainer.get_leaderboard(cross_validation=4, split_type="cv", stderr_to_stdout=True)
----------------------------1/4 cv----------------------------
Using previously used data path /tmp/tmpnpjgki5b/data/auto-mpg.csv
Dataset size: 199 99 100
Data saved to /tmp/tmpnpjgki5b/output/auto-mpg/2023-09-23-20-37-47-0_UserInputConfig (data.csv and tabular_data.csv).
-------------Run PytorchTabular-------------
Training Category Embedding
Global seed set to 42
2023-09-23 20:37:48,600 - {pytorch_tabular.tabular_model:473} - INFO - Preparing the DataLoaders
2023-09-23 20:37:48,600 - {pytorch_tabular.tabular_datamodule:290} - INFO - Setting up the datamodule for regression task
2023-09-23 20:37:48,610 - {pytorch_tabular.tabular_model:521} - INFO - Preparing the Model: CategoryEmbeddingModel
2023-09-23 20:37:48,621 - {pytorch_tabular.tabular_model:268} - INFO - Preparing the Trainer
/home/xlluo/anaconda3/envs/tabular_ensemble/lib/python3.10/site-packages/pytorch_lightning/trainer/connectors/accelerator_connector.py:589: LightningDeprecationWarning: The Trainer argument `auto_select_gpus` has been deprecated in v1.9.0 and will be removed in v2.0.0. Please use the function `pytorch_lightning.accelerators.find_usable_cuda_devices` instead.
rank_zero_deprecation(
Auto select gpus: [0]
GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
2023-09-23 20:37:49,693 - {pytorch_tabular.tabular_model:582} - INFO - Training Started
You are using a CUDA device ('NVIDIA GeForce RTX 3090') that has Tensor Cores. To properly utilize them, you should set `torch.set_float32_matmul_precision('medium' | 'high')` which will trade-off precision for performance. For more details, read https://pytorch.org/docs/stable/generated/torch.set_float32_matmul_precision.html#torch.set_float32_matmul_precision
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
| Name | Type | Params
---------------------------------------------------------------
0 | _backbone | CategoryEmbeddingBackbone | 11.4 K
1 | _embedding_layer | Embedding1dLayer | 14
2 | head | LinearHead | 33
3 | loss | MSELoss | 0
---------------------------------------------------------------
11.4 K Trainable params
0 Non-trainable params
11.4 K Total params
0.046 Total estimated model params size (MB)
Epoch: 1/300, Train loss: 659.3594, Val loss: 628.6083, Min val loss: 628.6083, Epoch time: 0.035s.
Epoch: 20/300, Train loss: 341.1675, Val loss: 317.6812, Min val loss: 317.6812, Epoch time: 0.011s.
Epoch: 40/300, Train loss: 90.1501, Val loss: 70.8835, Min val loss: 70.8835, Epoch time: 0.011s.
Epoch: 60/300, Train loss: 44.8901, Val loss: 35.4742, Min val loss: 35.4742, Epoch time: 0.011s.
Epoch: 80/300, Train loss: 35.4699, Val loss: 30.2076, Min val loss: 30.2076, Epoch time: 0.011s.
Epoch: 100/300, Train loss: 28.3062, Val loss: 27.7918, Min val loss: 27.7918, Epoch time: 0.012s.
Epoch: 120/300, Train loss: 22.4722, Val loss: 25.4099, Min val loss: 25.4099, Epoch time: 0.011s.
Epoch: 140/300, Train loss: 23.3103, Val loss: 24.1631, Min val loss: 24.1631, Epoch time: 0.010s.
Epoch: 160/300, Train loss: 18.9331, Val loss: 23.2310, Min val loss: 22.8011, Epoch time: 0.013s.
Epoch: 180/300, Train loss: 17.4051, Val loss: 22.7842, Min val loss: 22.7842, Epoch time: 0.010s.
Epoch: 200/300, Train loss: 15.6233, Val loss: 21.3642, Min val loss: 21.3436, Epoch time: 0.011s.
Epoch: 220/300, Train loss: 16.8699, Val loss: 20.2019, Min val loss: 20.2019, Epoch time: 0.010s.
Epoch: 240/300, Train loss: 14.3514, Val loss: 19.4510, Min val loss: 19.4510, Epoch time: 0.010s.
Epoch: 260/300, Train loss: 16.2243, Val loss: 19.0539, Min val loss: 19.0495, Epoch time: 0.009s.
Epoch: 280/300, Train loss: 12.8429, Val loss: 18.1077, Min val loss: 18.0788, Epoch time: 0.011s.
Epoch: 300/300, Train loss: 14.4400, Val loss: 17.8435, Min val loss: 17.8435, Epoch time: 0.009s.
`Trainer.fit` stopped: `max_epochs=300` reached.
2023-09-23 20:37:55,840 - {pytorch_tabular.tabular_model:584} - INFO - Training the model completed
2023-09-23 20:37:55,841 - {pytorch_tabular.tabular_model:1258} - INFO - Loading the best model
/home/xlluo/anaconda3/envs/tabular_ensemble/lib/python3.10/site-packages/pytorch_lightning/utilities/cloud_io.py:33: LightningDeprecationWarning: `pytorch_lightning.utilities.cloud_io.get_filesystem` has been deprecated in v1.8.0 and will be removed in v2.0.0. Please use `lightning_fabric.utilities.cloud_io.get_filesystem` instead.
rank_zero_deprecation(
Training mse loss: 10.68487
Validation mse loss: 17.84354
Testing mse loss: 9.95495
Trainer saved. To load the trainer, run trainer = load_trainer(path='/tmp/tmpnpjgki5b/output/auto-mpg/2023-09-23-20-37-47-0_UserInputConfig/trainer.pkl')
-------------PytorchTabular End-------------
Category Embedding 1/1
--------------------------End 1/4 cv--------------------------
----------------------------2/4 cv----------------------------
Using previously used data path /tmp/tmpnpjgki5b/data/auto-mpg.csv
Dataset size: 199 99 100
Data saved to /tmp/tmpnpjgki5b/output/auto-mpg/2023-09-23-20-37-47-0_UserInputConfig (data.csv and tabular_data.csv).
-------------Run PytorchTabular-------------
Training Category Embedding
Global seed set to 42
2023-09-23 20:37:56,591 - {pytorch_tabular.tabular_model:473} - INFO - Preparing the DataLoaders
2023-09-23 20:37:56,592 - {pytorch_tabular.tabular_datamodule:290} - INFO - Setting up the datamodule for regression task
2023-09-23 20:37:56,601 - {pytorch_tabular.tabular_model:521} - INFO - Preparing the Model: CategoryEmbeddingModel
2023-09-23 20:37:56,613 - {pytorch_tabular.tabular_model:268} - INFO - Preparing the Trainer
/home/xlluo/anaconda3/envs/tabular_ensemble/lib/python3.10/site-packages/pytorch_lightning/trainer/connectors/accelerator_connector.py:589: LightningDeprecationWarning: The Trainer argument `auto_select_gpus` has been deprecated in v1.9.0 and will be removed in v2.0.0. Please use the function `pytorch_lightning.accelerators.find_usable_cuda_devices` instead.
rank_zero_deprecation(
Auto select gpus: [0]
GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
2023-09-23 20:37:56,627 - {pytorch_tabular.tabular_model:582} - INFO - Training Started
You are using a CUDA device ('NVIDIA GeForce RTX 3090') that has Tensor Cores. To properly utilize them, you should set `torch.set_float32_matmul_precision('medium' | 'high')` which will trade-off precision for performance. For more details, read https://pytorch.org/docs/stable/generated/torch.set_float32_matmul_precision.html#torch.set_float32_matmul_precision
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
| Name | Type | Params
---------------------------------------------------------------
0 | _backbone | CategoryEmbeddingBackbone | 11.4 K
1 | _embedding_layer | Embedding1dLayer | 14
2 | head | LinearHead | 33
3 | loss | MSELoss | 0
---------------------------------------------------------------
11.4 K Trainable params
0 Non-trainable params
11.4 K Total params
0.046 Total estimated model params size (MB)
Epoch: 1/300, Train loss: 696.0748, Val loss: 545.8853, Min val loss: 545.8853, Epoch time: 0.012s.
Epoch: 20/300, Train loss: 372.0458, Val loss: 276.3550, Min val loss: 276.3550, Epoch time: 0.014s.
Epoch: 40/300, Train loss: 85.0721, Val loss: 59.3774, Min val loss: 59.3774, Epoch time: 0.013s.
Epoch: 60/300, Train loss: 48.5832, Val loss: 39.0821, Min val loss: 39.0821, Epoch time: 0.017s.
Epoch: 80/300, Train loss: 31.3669, Val loss: 30.6529, Min val loss: 30.6529, Epoch time: 0.015s.
Epoch: 100/300, Train loss: 24.9108, Val loss: 27.0339, Min val loss: 26.9302, Epoch time: 0.009s.
Epoch: 120/300, Train loss: 21.2625, Val loss: 24.2448, Min val loss: 24.2448, Epoch time: 0.012s.
Epoch: 140/300, Train loss: 19.6647, Val loss: 22.5396, Min val loss: 22.5396, Epoch time: 0.009s.
Epoch: 160/300, Train loss: 18.2782, Val loss: 20.3885, Min val loss: 20.3885, Epoch time: 0.014s.
Epoch: 180/300, Train loss: 19.8687, Val loss: 18.9228, Min val loss: 18.9228, Epoch time: 0.013s.
Epoch: 200/300, Train loss: 17.5661, Val loss: 18.1803, Min val loss: 18.1723, Epoch time: 0.010s.
Epoch: 220/300, Train loss: 17.5160, Val loss: 17.2254, Min val loss: 17.2254, Epoch time: 0.018s.
Epoch: 240/300, Train loss: 18.5148, Val loss: 16.5343, Min val loss: 16.5343, Epoch time: 0.013s.
Epoch: 260/300, Train loss: 19.2128, Val loss: 15.7374, Min val loss: 15.7374, Epoch time: 0.011s.
Epoch: 280/300, Train loss: 13.2446, Val loss: 15.1371, Min val loss: 15.1371, Epoch time: 0.011s.
Epoch: 300/300, Train loss: 12.9830, Val loss: 14.6800, Min val loss: 14.4976, Epoch time: 0.018s.
`Trainer.fit` stopped: `max_epochs=300` reached.
2023-09-23 20:38:02,306 - {pytorch_tabular.tabular_model:584} - INFO - Training the model completed
2023-09-23 20:38:02,307 - {pytorch_tabular.tabular_model:1258} - INFO - Loading the best model
/home/xlluo/anaconda3/envs/tabular_ensemble/lib/python3.10/site-packages/pytorch_lightning/utilities/cloud_io.py:33: LightningDeprecationWarning: `pytorch_lightning.utilities.cloud_io.get_filesystem` has been deprecated in v1.8.0 and will be removed in v2.0.0. Please use `lightning_fabric.utilities.cloud_io.get_filesystem` instead.
rank_zero_deprecation(
Training mse loss: 10.70534
Validation mse loss: 14.49761
Testing mse loss: 13.68175
Trainer saved. To load the trainer, run trainer = load_trainer(path='/tmp/tmpnpjgki5b/output/auto-mpg/2023-09-23-20-37-47-0_UserInputConfig/trainer.pkl')
-------------PytorchTabular End-------------
Category Embedding 1/1
--------------------------End 2/4 cv--------------------------
----------------------------3/4 cv----------------------------
Using previously used data path /tmp/tmpnpjgki5b/data/auto-mpg.csv
Dataset size: 200 99 99
Data saved to /tmp/tmpnpjgki5b/output/auto-mpg/2023-09-23-20-37-47-0_UserInputConfig (data.csv and tabular_data.csv).
-------------Run PytorchTabular-------------
Training Category Embedding
Global seed set to 42
2023-09-23 20:38:02,872 - {pytorch_tabular.tabular_model:473} - INFO - Preparing the DataLoaders
2023-09-23 20:38:02,874 - {pytorch_tabular.tabular_datamodule:290} - INFO - Setting up the datamodule for regression task
2023-09-23 20:38:02,883 - {pytorch_tabular.tabular_model:521} - INFO - Preparing the Model: CategoryEmbeddingModel
2023-09-23 20:38:02,899 - {pytorch_tabular.tabular_model:268} - INFO - Preparing the Trainer
/home/xlluo/anaconda3/envs/tabular_ensemble/lib/python3.10/site-packages/pytorch_lightning/trainer/connectors/accelerator_connector.py:589: LightningDeprecationWarning: The Trainer argument `auto_select_gpus` has been deprecated in v1.9.0 and will be removed in v2.0.0. Please use the function `pytorch_lightning.accelerators.find_usable_cuda_devices` instead.
rank_zero_deprecation(
Auto select gpus: [0]
GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
2023-09-23 20:38:02,922 - {pytorch_tabular.tabular_model:582} - INFO - Training Started
You are using a CUDA device ('NVIDIA GeForce RTX 3090') that has Tensor Cores. To properly utilize them, you should set `torch.set_float32_matmul_precision('medium' | 'high')` which will trade-off precision for performance. For more details, read https://pytorch.org/docs/stable/generated/torch.set_float32_matmul_precision.html#torch.set_float32_matmul_precision
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
| Name | Type | Params
---------------------------------------------------------------
0 | _backbone | CategoryEmbeddingBackbone | 11.4 K
1 | _embedding_layer | Embedding1dLayer | 14
2 | head | LinearHead | 33
3 | loss | MSELoss | 0
---------------------------------------------------------------
11.4 K Trainable params
0 Non-trainable params
11.4 K Total params
0.046 Total estimated model params size (MB)
Epoch: 1/300, Train loss: 645.7930, Val loss: 634.2722, Min val loss: 634.2722, Epoch time: 0.017s.
Epoch: 20/300, Train loss: 326.0877, Val loss: 323.5532, Min val loss: 323.5532, Epoch time: 0.012s.
Epoch: 40/300, Train loss: 79.7064, Val loss: 70.4397, Min val loss: 70.4397, Epoch time: 0.011s.
Epoch: 60/300, Train loss: 46.4769, Val loss: 33.4822, Min val loss: 33.4822, Epoch time: 0.011s.
Epoch: 80/300, Train loss: 33.4557, Val loss: 23.5222, Min val loss: 23.5222, Epoch time: 0.011s.
Epoch: 100/300, Train loss: 27.6559, Val loss: 19.9502, Min val loss: 19.9502, Epoch time: 0.011s.
Epoch: 120/300, Train loss: 20.6489, Val loss: 17.9041, Min val loss: 17.9041, Epoch time: 0.013s.
Epoch: 140/300, Train loss: 19.6383, Val loss: 17.4171, Min val loss: 17.4171, Epoch time: 0.010s.
Epoch: 160/300, Train loss: 16.6578, Val loss: 16.3154, Min val loss: 16.2164, Epoch time: 0.011s.
Epoch: 180/300, Train loss: 19.6480, Val loss: 15.2104, Min val loss: 15.2104, Epoch time: 0.010s.
Epoch: 200/300, Train loss: 15.6033, Val loss: 14.5753, Min val loss: 14.5753, Epoch time: 0.011s.
Epoch: 220/300, Train loss: 13.8016, Val loss: 13.8601, Min val loss: 13.8468, Epoch time: 0.009s.
Epoch: 240/300, Train loss: 15.3606, Val loss: 12.9441, Min val loss: 12.9441, Epoch time: 0.011s.
Epoch: 260/300, Train loss: 14.5253, Val loss: 12.5276, Min val loss: 12.5276, Epoch time: 0.011s.
Epoch: 280/300, Train loss: 15.3802, Val loss: 12.0176, Min val loss: 12.0176, Epoch time: 0.009s.
Epoch: 300/300, Train loss: 14.3208, Val loss: 11.5972, Min val loss: 11.5972, Epoch time: 0.010s.
`Trainer.fit` stopped: `max_epochs=300` reached.
2023-09-23 20:38:07,954 - {pytorch_tabular.tabular_model:584} - INFO - Training the model completed
2023-09-23 20:38:07,954 - {pytorch_tabular.tabular_model:1258} - INFO - Loading the best model
/home/xlluo/anaconda3/envs/tabular_ensemble/lib/python3.10/site-packages/pytorch_lightning/utilities/cloud_io.py:33: LightningDeprecationWarning: `pytorch_lightning.utilities.cloud_io.get_filesystem` has been deprecated in v1.8.0 and will be removed in v2.0.0. Please use `lightning_fabric.utilities.cloud_io.get_filesystem` instead.
rank_zero_deprecation(
Training mse loss: 8.63738
Validation mse loss: 11.59718
Testing mse loss: 16.02349
Trainer saved. To load the trainer, run trainer = load_trainer(path='/tmp/tmpnpjgki5b/output/auto-mpg/2023-09-23-20-37-47-0_UserInputConfig/trainer.pkl')
-------------PytorchTabular End-------------
Category Embedding 1/1
--------------------------End 3/4 cv--------------------------
----------------------------4/4 cv----------------------------
Using previously used data path /tmp/tmpnpjgki5b/data/auto-mpg.csv
Dataset size: 200 99 99
Data saved to /tmp/tmpnpjgki5b/output/auto-mpg/2023-09-23-20-37-47-0_UserInputConfig (data.csv and tabular_data.csv).
-------------Run PytorchTabular-------------
Training Category Embedding
Global seed set to 42
2023-09-23 20:38:08,438 - {pytorch_tabular.tabular_model:473} - INFO - Preparing the DataLoaders
2023-09-23 20:38:08,439 - {pytorch_tabular.tabular_datamodule:290} - INFO - Setting up the datamodule for regression task
2023-09-23 20:38:08,447 - {pytorch_tabular.tabular_model:521} - INFO - Preparing the Model: CategoryEmbeddingModel
2023-09-23 20:38:08,458 - {pytorch_tabular.tabular_model:268} - INFO - Preparing the Trainer
/home/xlluo/anaconda3/envs/tabular_ensemble/lib/python3.10/site-packages/pytorch_lightning/trainer/connectors/accelerator_connector.py:589: LightningDeprecationWarning: The Trainer argument `auto_select_gpus` has been deprecated in v1.9.0 and will be removed in v2.0.0. Please use the function `pytorch_lightning.accelerators.find_usable_cuda_devices` instead.
rank_zero_deprecation(
Auto select gpus: [0]
GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
2023-09-23 20:38:08,473 - {pytorch_tabular.tabular_model:582} - INFO - Training Started
You are using a CUDA device ('NVIDIA GeForce RTX 3090') that has Tensor Cores. To properly utilize them, you should set `torch.set_float32_matmul_precision('medium' | 'high')` which will trade-off precision for performance. For more details, read https://pytorch.org/docs/stable/generated/torch.set_float32_matmul_precision.html#torch.set_float32_matmul_precision
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
| Name | Type | Params
---------------------------------------------------------------
0 | _backbone | CategoryEmbeddingBackbone | 11.4 K
1 | _embedding_layer | Embedding1dLayer | 14
2 | head | LinearHead | 33
3 | loss | MSELoss | 0
---------------------------------------------------------------
11.4 K Trainable params
0 Non-trainable params
11.4 K Total params
0.046 Total estimated model params size (MB)
Epoch: 1/300, Train loss: 636.0693, Val loss: 673.2845, Min val loss: 673.2845, Epoch time: 0.011s.
Epoch: 20/300, Train loss: 320.2209, Val loss: 352.5726, Min val loss: 352.5726, Epoch time: 0.009s.
Epoch: 40/300, Train loss: 72.3203, Val loss: 74.2078, Min val loss: 74.2078, Epoch time: 0.010s.
Epoch: 60/300, Train loss: 41.2314, Val loss: 47.7063, Min val loss: 47.7063, Epoch time: 0.008s.
Epoch: 80/300, Train loss: 31.4739, Val loss: 34.3773, Min val loss: 34.3773, Epoch time: 0.009s.
Epoch: 100/300, Train loss: 26.7886, Val loss: 28.7385, Min val loss: 28.7385, Epoch time: 0.008s.
Epoch: 120/300, Train loss: 20.0182, Val loss: 24.3363, Min val loss: 24.3363, Epoch time: 0.009s.
Epoch: 140/300, Train loss: 19.3822, Val loss: 22.6521, Min val loss: 22.6521, Epoch time: 0.012s.
Epoch: 160/300, Train loss: 19.0349, Val loss: 21.4546, Min val loss: 21.4546, Epoch time: 0.009s.
Epoch: 180/300, Train loss: 16.7342, Val loss: 19.9216, Min val loss: 19.9216, Epoch time: 0.009s.
Epoch: 200/300, Train loss: 14.9367, Val loss: 18.7599, Min val loss: 18.7599, Epoch time: 0.009s.
Epoch: 220/300, Train loss: 15.6413, Val loss: 18.2248, Min val loss: 18.2248, Epoch time: 0.010s.
Epoch: 240/300, Train loss: 16.4146, Val loss: 17.3294, Min val loss: 17.3294, Epoch time: 0.009s.
Epoch: 260/300, Train loss: 13.5992, Val loss: 16.4097, Min val loss: 16.4097, Epoch time: 0.013s.
Epoch: 280/300, Train loss: 12.5998, Val loss: 15.9688, Min val loss: 15.5961, Epoch time: 0.009s.
Epoch: 300/300, Train loss: 15.1886, Val loss: 14.6322, Min val loss: 14.6322, Epoch time: 0.011s.
`Trainer.fit` stopped: `max_epochs=300` reached.
2023-09-23 20:38:12,907 - {pytorch_tabular.tabular_model:584} - INFO - Training the model completed
2023-09-23 20:38:12,908 - {pytorch_tabular.tabular_model:1258} - INFO - Loading the best model
/home/xlluo/anaconda3/envs/tabular_ensemble/lib/python3.10/site-packages/pytorch_lightning/utilities/cloud_io.py:33: LightningDeprecationWarning: `pytorch_lightning.utilities.cloud_io.get_filesystem` has been deprecated in v1.8.0 and will be removed in v2.0.0. Please use `lightning_fabric.utilities.cloud_io.get_filesystem` instead.
rank_zero_deprecation(
Training mse loss: 8.95723
Validation mse loss: 14.63222
Testing mse loss: 12.50395
Trainer saved. To load the trainer, run trainer = load_trainer(path='/tmp/tmpnpjgki5b/output/auto-mpg/2023-09-23-20-37-47-0_UserInputConfig/trainer.pkl')
-------------PytorchTabular End-------------
Category Embedding 1/1
--------------------------End 4/4 cv--------------------------
Trainer saved. To load the trainer, run trainer = load_trainer(path='/tmp/tmpnpjgki5b/output/auto-mpg/2023-09-23-20-37-47-0_UserInputConfig/trainer.pkl')
[2]:
| Program | Model | Training RMSE | Training MSE | Training MAE | Training MAPE | Training R2 | Training MEDIAN_ABSOLUTE_ERROR | Training EXPLAINED_VARIANCE_SCORE | Testing RMSE | ... | Testing R2 | Testing MEDIAN_ABSOLUTE_ERROR | Testing EXPLAINED_VARIANCE_SCORE | Validation RMSE | Validation MSE | Validation MAE | Validation MAPE | Validation R2 | Validation MEDIAN_ABSOLUTE_ERROR | Validation EXPLAINED_VARIANCE_SCORE | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | PytorchTabular | Category Embedding | 3.121511 | 9.743828 | 2.361333 | 0.100122 | 0.839443 | 1.824559 | 0.870805 | 3.610387 | ... | 0.786089 | 2.024105 | 0.814192 | 3.82657 | 14.642641 | 2.699677 | 0.119046 | 0.761043 | 1.905207 | 0.785311 |
1 rows × 23 columns
Splitting the dataset randomly#
We can simply split the dataset with different random seeds. This is achieved by passing the argument split_type="random". In this case, the ratio of training/validation/testing sets is the one specified in the configuration (or 6:2:2 by default).
[3]:
trainer.get_leaderboard(cross_validation=4, split_type="random", stderr_to_stdout=True)
----------------------------1/4 random----------------------------
Using previously used data path /tmp/tmpnpjgki5b/data/auto-mpg.csv
Dataset size: 238 80 80
Data saved to /tmp/tmpnpjgki5b/output/auto-mpg/2023-09-23-20-37-47-0_UserInputConfig (data.csv and tabular_data.csv).
-------------Run PytorchTabular-------------
Training Category Embedding
Global seed set to 42
2023-09-23 20:38:13,428 - {pytorch_tabular.tabular_model:473} - INFO - Preparing the DataLoaders
2023-09-23 20:38:13,428 - {pytorch_tabular.tabular_datamodule:290} - INFO - Setting up the datamodule for regression task
2023-09-23 20:38:13,437 - {pytorch_tabular.tabular_model:521} - INFO - Preparing the Model: CategoryEmbeddingModel
2023-09-23 20:38:13,447 - {pytorch_tabular.tabular_model:268} - INFO - Preparing the Trainer
/home/xlluo/anaconda3/envs/tabular_ensemble/lib/python3.10/site-packages/pytorch_lightning/trainer/connectors/accelerator_connector.py:589: LightningDeprecationWarning: The Trainer argument `auto_select_gpus` has been deprecated in v1.9.0 and will be removed in v2.0.0. Please use the function `pytorch_lightning.accelerators.find_usable_cuda_devices` instead.
rank_zero_deprecation(
Auto select gpus: [0]
GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
2023-09-23 20:38:13,463 - {pytorch_tabular.tabular_model:582} - INFO - Training Started
You are using a CUDA device ('NVIDIA GeForce RTX 3090') that has Tensor Cores. To properly utilize them, you should set `torch.set_float32_matmul_precision('medium' | 'high')` which will trade-off precision for performance. For more details, read https://pytorch.org/docs/stable/generated/torch.set_float32_matmul_precision.html#torch.set_float32_matmul_precision
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
| Name | Type | Params
---------------------------------------------------------------
0 | _backbone | CategoryEmbeddingBackbone | 11.4 K
1 | _embedding_layer | Embedding1dLayer | 14
2 | head | LinearHead | 33
3 | loss | MSELoss | 0
---------------------------------------------------------------
11.4 K Trainable params
0 Non-trainable params
11.4 K Total params
0.046 Total estimated model params size (MB)
Epoch: 1/300, Train loss: 677.8015, Val loss: 582.9557, Min val loss: 582.9557, Epoch time: 0.012s.
Epoch: 20/300, Train loss: 353.7851, Val loss: 302.0203, Min val loss: 302.0203, Epoch time: 0.009s.
Epoch: 40/300, Train loss: 85.0776, Val loss: 62.1153, Min val loss: 62.1153, Epoch time: 0.009s.
Epoch: 60/300, Train loss: 45.2654, Val loss: 34.2778, Min val loss: 34.2691, Epoch time: 0.009s.
Epoch: 80/300, Train loss: 33.9537, Val loss: 26.8622, Min val loss: 26.8622, Epoch time: 0.011s.
Epoch: 100/300, Train loss: 26.9038, Val loss: 23.2417, Min val loss: 23.2372, Epoch time: 0.009s.
Epoch: 120/300, Train loss: 24.9622, Val loss: 20.4360, Min val loss: 20.4360, Epoch time: 0.009s.
Epoch: 140/300, Train loss: 24.1636, Val loss: 19.4010, Min val loss: 19.4010, Epoch time: 0.009s.
Epoch: 160/300, Train loss: 22.9200, Val loss: 18.0232, Min val loss: 17.9749, Epoch time: 0.008s.
Epoch: 180/300, Train loss: 19.7677, Val loss: 16.9469, Min val loss: 16.9469, Epoch time: 0.010s.
Epoch: 200/300, Train loss: 17.9390, Val loss: 16.6545, Min val loss: 16.4093, Epoch time: 0.012s.
Epoch: 220/300, Train loss: 19.4496, Val loss: 15.4451, Min val loss: 15.1788, Epoch time: 0.010s.
Epoch: 240/300, Train loss: 16.0483, Val loss: 14.5508, Min val loss: 14.5508, Epoch time: 0.012s.
Epoch: 260/300, Train loss: 16.4672, Val loss: 13.8354, Min val loss: 13.8354, Epoch time: 0.014s.
Epoch: 280/300, Train loss: 13.6031, Val loss: 12.9315, Min val loss: 12.9315, Epoch time: 0.011s.
Epoch: 300/300, Train loss: 16.5369, Val loss: 12.3673, Min val loss: 12.3673, Epoch time: 0.010s.
`Trainer.fit` stopped: `max_epochs=300` reached.
2023-09-23 20:38:17,915 - {pytorch_tabular.tabular_model:584} - INFO - Training the model completed
2023-09-23 20:38:17,916 - {pytorch_tabular.tabular_model:1258} - INFO - Loading the best model
/home/xlluo/anaconda3/envs/tabular_ensemble/lib/python3.10/site-packages/pytorch_lightning/utilities/cloud_io.py:33: LightningDeprecationWarning: `pytorch_lightning.utilities.cloud_io.get_filesystem` has been deprecated in v1.8.0 and will be removed in v2.0.0. Please use `lightning_fabric.utilities.cloud_io.get_filesystem` instead.
rank_zero_deprecation(
Training mse loss: 11.25175
Validation mse loss: 12.36725
Testing mse loss: 7.83801
Trainer saved. To load the trainer, run trainer = load_trainer(path='/tmp/tmpnpjgki5b/output/auto-mpg/2023-09-23-20-37-47-0_UserInputConfig/trainer.pkl')
-------------PytorchTabular End-------------
Category Embedding 1/1
--------------------------End 1/4 random--------------------------
----------------------------2/4 random----------------------------
Using previously used data path /tmp/tmpnpjgki5b/data/auto-mpg.csv
Dataset size: 238 80 80
Data saved to /tmp/tmpnpjgki5b/output/auto-mpg/2023-09-23-20-37-47-0_UserInputConfig (data.csv and tabular_data.csv).
-------------Run PytorchTabular-------------
Training Category Embedding
Global seed set to 42
2023-09-23 20:38:18,385 - {pytorch_tabular.tabular_model:473} - INFO - Preparing the DataLoaders
2023-09-23 20:38:18,385 - {pytorch_tabular.tabular_datamodule:290} - INFO - Setting up the datamodule for regression task
2023-09-23 20:38:18,394 - {pytorch_tabular.tabular_model:521} - INFO - Preparing the Model: CategoryEmbeddingModel
2023-09-23 20:38:18,404 - {pytorch_tabular.tabular_model:268} - INFO - Preparing the Trainer
/home/xlluo/anaconda3/envs/tabular_ensemble/lib/python3.10/site-packages/pytorch_lightning/trainer/connectors/accelerator_connector.py:589: LightningDeprecationWarning: The Trainer argument `auto_select_gpus` has been deprecated in v1.9.0 and will be removed in v2.0.0. Please use the function `pytorch_lightning.accelerators.find_usable_cuda_devices` instead.
rank_zero_deprecation(
Auto select gpus: [0]
GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
2023-09-23 20:38:18,418 - {pytorch_tabular.tabular_model:582} - INFO - Training Started
You are using a CUDA device ('NVIDIA GeForce RTX 3090') that has Tensor Cores. To properly utilize them, you should set `torch.set_float32_matmul_precision('medium' | 'high')` which will trade-off precision for performance. For more details, read https://pytorch.org/docs/stable/generated/torch.set_float32_matmul_precision.html#torch.set_float32_matmul_precision
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
| Name | Type | Params
---------------------------------------------------------------
0 | _backbone | CategoryEmbeddingBackbone | 11.4 K
1 | _embedding_layer | Embedding1dLayer | 14
2 | head | LinearHead | 33
3 | loss | MSELoss | 0
---------------------------------------------------------------
11.4 K Trainable params
0 Non-trainable params
11.4 K Total params
0.046 Total estimated model params size (MB)
Epoch: 1/300, Train loss: 632.3258, Val loss: 662.7650, Min val loss: 662.7650, Epoch time: 0.012s.
Epoch: 20/300, Train loss: 318.3275, Val loss: 344.3962, Min val loss: 344.3962, Epoch time: 0.011s.
Epoch: 40/300, Train loss: 69.3076, Val loss: 76.5520, Min val loss: 76.5520, Epoch time: 0.011s.
Epoch: 60/300, Train loss: 42.2416, Val loss: 46.7150, Min val loss: 46.7150, Epoch time: 0.010s.
Epoch: 80/300, Train loss: 29.2576, Val loss: 34.4237, Min val loss: 34.4237, Epoch time: 0.008s.
Epoch: 100/300, Train loss: 26.1472, Val loss: 29.0749, Min val loss: 29.0749, Epoch time: 0.008s.
Epoch: 120/300, Train loss: 24.8970, Val loss: 26.5129, Min val loss: 26.5129, Epoch time: 0.009s.
Epoch: 140/300, Train loss: 17.9044, Val loss: 25.3725, Min val loss: 25.3725, Epoch time: 0.009s.
Epoch: 160/300, Train loss: 17.4626, Val loss: 23.7306, Min val loss: 23.7306, Epoch time: 0.009s.
Epoch: 180/300, Train loss: 16.5023, Val loss: 22.9689, Min val loss: 22.9689, Epoch time: 0.010s.
Epoch: 200/300, Train loss: 15.1315, Val loss: 22.0908, Min val loss: 22.0908, Epoch time: 0.010s.
Epoch: 220/300, Train loss: 14.4462, Val loss: 21.3679, Min val loss: 21.3679, Epoch time: 0.011s.
Epoch: 240/300, Train loss: 15.4215, Val loss: 20.7016, Min val loss: 20.6893, Epoch time: 0.011s.
Epoch: 260/300, Train loss: 13.5062, Val loss: 20.0930, Min val loss: 20.0624, Epoch time: 0.009s.
Epoch: 280/300, Train loss: 14.5558, Val loss: 19.2505, Min val loss: 19.2505, Epoch time: 0.009s.
Epoch: 300/300, Train loss: 14.8346, Val loss: 18.8435, Min val loss: 18.8435, Epoch time: 0.010s.
`Trainer.fit` stopped: `max_epochs=300` reached.
2023-09-23 20:38:23,402 - {pytorch_tabular.tabular_model:584} - INFO - Training the model completed
2023-09-23 20:38:23,403 - {pytorch_tabular.tabular_model:1258} - INFO - Loading the best model
/home/xlluo/anaconda3/envs/tabular_ensemble/lib/python3.10/site-packages/pytorch_lightning/utilities/cloud_io.py:33: LightningDeprecationWarning: `pytorch_lightning.utilities.cloud_io.get_filesystem` has been deprecated in v1.8.0 and will be removed in v2.0.0. Please use `lightning_fabric.utilities.cloud_io.get_filesystem` instead.
rank_zero_deprecation(
Training mse loss: 10.18515
Validation mse loss: 18.84349
Testing mse loss: 9.93399
Trainer saved. To load the trainer, run trainer = load_trainer(path='/tmp/tmpnpjgki5b/output/auto-mpg/2023-09-23-20-37-47-0_UserInputConfig/trainer.pkl')
-------------PytorchTabular End-------------
Category Embedding 1/1
--------------------------End 2/4 random--------------------------
----------------------------3/4 random----------------------------
Using previously used data path /tmp/tmpnpjgki5b/data/auto-mpg.csv
Dataset size: 238 80 80
Data saved to /tmp/tmpnpjgki5b/output/auto-mpg/2023-09-23-20-37-47-0_UserInputConfig (data.csv and tabular_data.csv).
-------------Run PytorchTabular-------------
Training Category Embedding
Global seed set to 42
2023-09-23 20:38:23,909 - {pytorch_tabular.tabular_model:473} - INFO - Preparing the DataLoaders
2023-09-23 20:38:23,910 - {pytorch_tabular.tabular_datamodule:290} - INFO - Setting up the datamodule for regression task
2023-09-23 20:38:23,919 - {pytorch_tabular.tabular_model:521} - INFO - Preparing the Model: CategoryEmbeddingModel
2023-09-23 20:38:23,929 - {pytorch_tabular.tabular_model:268} - INFO - Preparing the Trainer
/home/xlluo/anaconda3/envs/tabular_ensemble/lib/python3.10/site-packages/pytorch_lightning/trainer/connectors/accelerator_connector.py:589: LightningDeprecationWarning: The Trainer argument `auto_select_gpus` has been deprecated in v1.9.0 and will be removed in v2.0.0. Please use the function `pytorch_lightning.accelerators.find_usable_cuda_devices` instead.
rank_zero_deprecation(
Auto select gpus: [0]
GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
2023-09-23 20:38:23,943 - {pytorch_tabular.tabular_model:582} - INFO - Training Started
You are using a CUDA device ('NVIDIA GeForce RTX 3090') that has Tensor Cores. To properly utilize them, you should set `torch.set_float32_matmul_precision('medium' | 'high')` which will trade-off precision for performance. For more details, read https://pytorch.org/docs/stable/generated/torch.set_float32_matmul_precision.html#torch.set_float32_matmul_precision
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
| Name | Type | Params
---------------------------------------------------------------
0 | _backbone | CategoryEmbeddingBackbone | 11.4 K
1 | _embedding_layer | Embedding1dLayer | 14
2 | head | LinearHead | 33
3 | loss | MSELoss | 0
---------------------------------------------------------------
11.4 K Trainable params
0 Non-trainable params
11.4 K Total params
0.046 Total estimated model params size (MB)
Epoch: 1/300, Train loss: 663.8328, Val loss: 585.2318, Min val loss: 585.2318, Epoch time: 0.009s.
Epoch: 20/300, Train loss: 341.2238, Val loss: 306.3730, Min val loss: 306.3730, Epoch time: 0.015s.
Epoch: 40/300, Train loss: 74.4331, Val loss: 64.6773, Min val loss: 64.6773, Epoch time: 0.010s.
Epoch: 60/300, Train loss: 45.9507, Val loss: 41.3845, Min val loss: 41.3845, Epoch time: 0.011s.
Epoch: 80/300, Train loss: 32.7711, Val loss: 30.7545, Min val loss: 30.7545, Epoch time: 0.009s.
Epoch: 100/300, Train loss: 23.9204, Val loss: 26.2439, Min val loss: 26.2439, Epoch time: 0.011s.
Epoch: 120/300, Train loss: 20.2654, Val loss: 23.0741, Min val loss: 23.0741, Epoch time: 0.010s.
Epoch: 140/300, Train loss: 20.3352, Val loss: 20.7483, Min val loss: 20.7483, Epoch time: 0.008s.
Epoch: 160/300, Train loss: 19.5925, Val loss: 18.9536, Min val loss: 18.9536, Epoch time: 0.011s.
Epoch: 180/300, Train loss: 18.0734, Val loss: 17.7072, Min val loss: 17.7072, Epoch time: 0.012s.
Epoch: 200/300, Train loss: 15.6461, Val loss: 16.8639, Min val loss: 16.8639, Epoch time: 0.011s.
Epoch: 220/300, Train loss: 13.4826, Val loss: 15.9317, Min val loss: 15.9317, Epoch time: 0.009s.
Epoch: 240/300, Train loss: 15.6729, Val loss: 15.3439, Min val loss: 15.3068, Epoch time: 0.009s.
Epoch: 260/300, Train loss: 14.0022, Val loss: 14.3354, Min val loss: 14.3354, Epoch time: 0.009s.
Epoch: 280/300, Train loss: 12.9512, Val loss: 13.6573, Min val loss: 13.6573, Epoch time: 0.011s.
Epoch: 300/300, Train loss: 13.2983, Val loss: 13.2018, Min val loss: 13.2018, Epoch time: 0.009s.
`Trainer.fit` stopped: `max_epochs=300` reached.
2023-09-23 20:38:28,571 - {pytorch_tabular.tabular_model:584} - INFO - Training the model completed
2023-09-23 20:38:28,572 - {pytorch_tabular.tabular_model:1258} - INFO - Loading the best model
/home/xlluo/anaconda3/envs/tabular_ensemble/lib/python3.10/site-packages/pytorch_lightning/utilities/cloud_io.py:33: LightningDeprecationWarning: `pytorch_lightning.utilities.cloud_io.get_filesystem` has been deprecated in v1.8.0 and will be removed in v2.0.0. Please use `lightning_fabric.utilities.cloud_io.get_filesystem` instead.
rank_zero_deprecation(
Training mse loss: 9.30384
Validation mse loss: 13.20180
Testing mse loss: 14.17569
Trainer saved. To load the trainer, run trainer = load_trainer(path='/tmp/tmpnpjgki5b/output/auto-mpg/2023-09-23-20-37-47-0_UserInputConfig/trainer.pkl')
-------------PytorchTabular End-------------
Category Embedding 1/1
--------------------------End 3/4 random--------------------------
----------------------------4/4 random----------------------------
Using previously used data path /tmp/tmpnpjgki5b/data/auto-mpg.csv
Dataset size: 238 80 80
Data saved to /tmp/tmpnpjgki5b/output/auto-mpg/2023-09-23-20-37-47-0_UserInputConfig (data.csv and tabular_data.csv).
-------------Run PytorchTabular-------------
Training Category Embedding
Global seed set to 42
2023-09-23 20:38:29,037 - {pytorch_tabular.tabular_model:473} - INFO - Preparing the DataLoaders
2023-09-23 20:38:29,038 - {pytorch_tabular.tabular_datamodule:290} - INFO - Setting up the datamodule for regression task
2023-09-23 20:38:29,046 - {pytorch_tabular.tabular_model:521} - INFO - Preparing the Model: CategoryEmbeddingModel
2023-09-23 20:38:29,057 - {pytorch_tabular.tabular_model:268} - INFO - Preparing the Trainer
/home/xlluo/anaconda3/envs/tabular_ensemble/lib/python3.10/site-packages/pytorch_lightning/trainer/connectors/accelerator_connector.py:589: LightningDeprecationWarning: The Trainer argument `auto_select_gpus` has been deprecated in v1.9.0 and will be removed in v2.0.0. Please use the function `pytorch_lightning.accelerators.find_usable_cuda_devices` instead.
rank_zero_deprecation(
Auto select gpus: [0]
GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
2023-09-23 20:38:29,072 - {pytorch_tabular.tabular_model:582} - INFO - Training Started
You are using a CUDA device ('NVIDIA GeForce RTX 3090') that has Tensor Cores. To properly utilize them, you should set `torch.set_float32_matmul_precision('medium' | 'high')` which will trade-off precision for performance. For more details, read https://pytorch.org/docs/stable/generated/torch.set_float32_matmul_precision.html#torch.set_float32_matmul_precision
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
| Name | Type | Params
---------------------------------------------------------------
0 | _backbone | CategoryEmbeddingBackbone | 11.4 K
1 | _embedding_layer | Embedding1dLayer | 14
2 | head | LinearHead | 33
3 | loss | MSELoss | 0
---------------------------------------------------------------
11.4 K Trainable params
0 Non-trainable params
11.4 K Total params
0.046 Total estimated model params size (MB)
Epoch: 1/300, Train loss: 673.4464, Val loss: 604.0322, Min val loss: 604.0322, Epoch time: 0.017s.
Epoch: 20/300, Train loss: 354.8526, Val loss: 304.8196, Min val loss: 304.8196, Epoch time: 0.009s.
Epoch: 40/300, Train loss: 83.0153, Val loss: 65.4248, Min val loss: 65.4248, Epoch time: 0.008s.
Epoch: 60/300, Train loss: 48.7386, Val loss: 33.2992, Min val loss: 33.2992, Epoch time: 0.009s.
Epoch: 80/300, Train loss: 34.5437, Val loss: 25.6234, Min val loss: 25.6234, Epoch time: 0.010s.
Epoch: 100/300, Train loss: 30.8209, Val loss: 21.4857, Min val loss: 21.4857, Epoch time: 0.010s.
Epoch: 120/300, Train loss: 21.8472, Val loss: 18.3320, Min val loss: 18.3320, Epoch time: 0.010s.
Epoch: 140/300, Train loss: 23.5546, Val loss: 15.9204, Min val loss: 15.9204, Epoch time: 0.008s.
Epoch: 160/300, Train loss: 21.0466, Val loss: 14.0073, Min val loss: 14.0073, Epoch time: 0.009s.
Epoch: 180/300, Train loss: 19.9606, Val loss: 12.7407, Min val loss: 12.7407, Epoch time: 0.008s.
Epoch: 200/300, Train loss: 19.3325, Val loss: 11.2465, Min val loss: 11.2465, Epoch time: 0.010s.
Epoch: 220/300, Train loss: 14.9372, Val loss: 10.2214, Min val loss: 10.2214, Epoch time: 0.010s.
Epoch: 240/300, Train loss: 17.1906, Val loss: 9.6617, Min val loss: 9.4835, Epoch time: 0.008s.
Epoch: 260/300, Train loss: 17.4219, Val loss: 8.9945, Min val loss: 8.9184, Epoch time: 0.008s.
Epoch: 280/300, Train loss: 14.6068, Val loss: 8.4856, Min val loss: 8.4856, Epoch time: 0.008s.
Epoch: 300/300, Train loss: 12.2139, Val loss: 8.1110, Min val loss: 8.0176, Epoch time: 0.009s.
`Trainer.fit` stopped: `max_epochs=300` reached.
2023-09-23 20:38:33,382 - {pytorch_tabular.tabular_model:584} - INFO - Training the model completed
2023-09-23 20:38:33,382 - {pytorch_tabular.tabular_model:1258} - INFO - Loading the best model
/home/xlluo/anaconda3/envs/tabular_ensemble/lib/python3.10/site-packages/pytorch_lightning/utilities/cloud_io.py:33: LightningDeprecationWarning: `pytorch_lightning.utilities.cloud_io.get_filesystem` has been deprecated in v1.8.0 and will be removed in v2.0.0. Please use `lightning_fabric.utilities.cloud_io.get_filesystem` instead.
rank_zero_deprecation(
Training mse loss: 10.11050
Validation mse loss: 8.01764
Testing mse loss: 8.99186
Trainer saved. To load the trainer, run trainer = load_trainer(path='/tmp/tmpnpjgki5b/output/auto-mpg/2023-09-23-20-37-47-0_UserInputConfig/trainer.pkl')
-------------PytorchTabular End-------------
Category Embedding 1/1
--------------------------End 4/4 random--------------------------
Trainer saved. To load the trainer, run trainer = load_trainer(path='/tmp/tmpnpjgki5b/output/auto-mpg/2023-09-23-20-37-47-0_UserInputConfig/trainer.pkl')
[3]:
| Program | Model | Training RMSE | Training MSE | Training MAE | Training MAPE | Training R2 | Training MEDIAN_ABSOLUTE_ERROR | Training EXPLAINED_VARIANCE_SCORE | Testing RMSE | ... | Testing R2 | Testing MEDIAN_ABSOLUTE_ERROR | Testing EXPLAINED_VARIANCE_SCORE | Validation RMSE | Validation MSE | Validation MAE | Validation MAPE | Validation R2 | Validation MEDIAN_ABSOLUTE_ERROR | Validation EXPLAINED_VARIANCE_SCORE | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | PytorchTabular | Category Embedding | 3.195748 | 10.212808 | 2.387656 | 0.101595 | 0.838209 | 1.844076 | 0.865946 | 3.199201 | ... | 0.822679 | 2.022343 | 0.856969 | 3.620434 | 13.107543 | 2.541972 | 0.11167 | 0.771539 | 1.739592 | 0.783368 |
1 rows × 23 columns
Unexpected termination#
It may take quite a long time to cross-validate various models on the large dataset, especially with Bayesian hyperparameter optimization (Yes, Bayesian hyperparameter optimization and cross-validation can both be activated). If the script terminates unexpectedly, you can use a functionality that loads the stored cross-validation state to continue a previous execution.
First, we assume that the script terminates after the first run finishes.
[4]:
_ = trainer.get_leaderboard(cross_validation=1, split_type="random", stderr_to_stdout=True)
----------------------------1/1 random----------------------------
Using previously used data path /tmp/tmpnpjgki5b/data/auto-mpg.csv
Dataset size: 238 80 80
Data saved to /tmp/tmpnpjgki5b/output/auto-mpg/2023-09-23-20-37-47-0_UserInputConfig (data.csv and tabular_data.csv).
-------------Run PytorchTabular-------------
Training Category Embedding
Global seed set to 42
2023-09-23 20:38:33,947 - {pytorch_tabular.tabular_model:473} - INFO - Preparing the DataLoaders
2023-09-23 20:38:33,947 - {pytorch_tabular.tabular_datamodule:290} - INFO - Setting up the datamodule for regression task
2023-09-23 20:38:33,960 - {pytorch_tabular.tabular_model:521} - INFO - Preparing the Model: CategoryEmbeddingModel
2023-09-23 20:38:33,977 - {pytorch_tabular.tabular_model:268} - INFO - Preparing the Trainer
/home/xlluo/anaconda3/envs/tabular_ensemble/lib/python3.10/site-packages/pytorch_lightning/trainer/connectors/accelerator_connector.py:589: LightningDeprecationWarning: The Trainer argument `auto_select_gpus` has been deprecated in v1.9.0 and will be removed in v2.0.0. Please use the function `pytorch_lightning.accelerators.find_usable_cuda_devices` instead.
rank_zero_deprecation(
Auto select gpus: [0]
GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
2023-09-23 20:38:33,993 - {pytorch_tabular.tabular_model:582} - INFO - Training Started
You are using a CUDA device ('NVIDIA GeForce RTX 3090') that has Tensor Cores. To properly utilize them, you should set `torch.set_float32_matmul_precision('medium' | 'high')` which will trade-off precision for performance. For more details, read https://pytorch.org/docs/stable/generated/torch.set_float32_matmul_precision.html#torch.set_float32_matmul_precision
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
| Name | Type | Params
---------------------------------------------------------------
0 | _backbone | CategoryEmbeddingBackbone | 11.4 K
1 | _embedding_layer | Embedding1dLayer | 14
2 | head | LinearHead | 33
3 | loss | MSELoss | 0
---------------------------------------------------------------
11.4 K Trainable params
0 Non-trainable params
11.4 K Total params
0.046 Total estimated model params size (MB)
Epoch: 1/300, Train loss: 677.8015, Val loss: 582.9557, Min val loss: 582.9557, Epoch time: 0.010s.
Epoch: 20/300, Train loss: 353.7851, Val loss: 302.0203, Min val loss: 302.0203, Epoch time: 0.009s.
Epoch: 40/300, Train loss: 85.0776, Val loss: 62.1153, Min val loss: 62.1153, Epoch time: 0.009s.
Epoch: 60/300, Train loss: 45.2654, Val loss: 34.2778, Min val loss: 34.2691, Epoch time: 0.009s.
Epoch: 80/300, Train loss: 33.9537, Val loss: 26.8622, Min val loss: 26.8622, Epoch time: 0.008s.
Epoch: 100/300, Train loss: 26.9038, Val loss: 23.2417, Min val loss: 23.2372, Epoch time: 0.011s.
Epoch: 120/300, Train loss: 24.9622, Val loss: 20.4360, Min val loss: 20.4360, Epoch time: 0.008s.
Epoch: 140/300, Train loss: 24.1636, Val loss: 19.4010, Min val loss: 19.4010, Epoch time: 0.010s.
Epoch: 160/300, Train loss: 22.9200, Val loss: 18.0232, Min val loss: 17.9749, Epoch time: 0.009s.
Epoch: 180/300, Train loss: 19.7677, Val loss: 16.9469, Min val loss: 16.9469, Epoch time: 0.011s.
Epoch: 200/300, Train loss: 17.9390, Val loss: 16.6545, Min val loss: 16.4093, Epoch time: 0.011s.
Epoch: 220/300, Train loss: 19.4496, Val loss: 15.4451, Min val loss: 15.1788, Epoch time: 0.008s.
Epoch: 240/300, Train loss: 16.0483, Val loss: 14.5508, Min val loss: 14.5508, Epoch time: 0.012s.
Epoch: 260/300, Train loss: 16.4672, Val loss: 13.8354, Min val loss: 13.8354, Epoch time: 0.009s.
Epoch: 280/300, Train loss: 13.6031, Val loss: 12.9315, Min val loss: 12.9315, Epoch time: 0.009s.
Epoch: 300/300, Train loss: 16.5369, Val loss: 12.3673, Min val loss: 12.3673, Epoch time: 0.010s.
`Trainer.fit` stopped: `max_epochs=300` reached.
2023-09-23 20:38:38,411 - {pytorch_tabular.tabular_model:584} - INFO - Training the model completed
2023-09-23 20:38:38,412 - {pytorch_tabular.tabular_model:1258} - INFO - Loading the best model
/home/xlluo/anaconda3/envs/tabular_ensemble/lib/python3.10/site-packages/pytorch_lightning/utilities/cloud_io.py:33: LightningDeprecationWarning: `pytorch_lightning.utilities.cloud_io.get_filesystem` has been deprecated in v1.8.0 and will be removed in v2.0.0. Please use `lightning_fabric.utilities.cloud_io.get_filesystem` instead.
rank_zero_deprecation(
Training mse loss: 11.25175
Validation mse loss: 12.36725
Testing mse loss: 7.83801
Trainer saved. To load the trainer, run trainer = load_trainer(path='/tmp/tmpnpjgki5b/output/auto-mpg/2023-09-23-20-37-47-0_UserInputConfig/trainer.pkl')
-------------PytorchTabular End-------------
Category Embedding 1/1
--------------------------End 1/1 random--------------------------
Trainer saved. To load the trainer, run trainer = load_trainer(path='/tmp/tmpnpjgki5b/output/auto-mpg/2023-09-23-20-37-47-0_UserInputConfig/trainer.pkl')
To continue the cross-validation, set the argument load_from_previous to True
[5]:
l1 = trainer.get_leaderboard(cross_validation=2, split_type="random", stderr_to_stdout=True, load_from_previous=True)
l1
Previous cross validation state is loaded.
----------------------------2/2 random----------------------------
Using previously used data path /tmp/tmpnpjgki5b/data/auto-mpg.csv
Dataset size: 238 80 80
Data saved to /tmp/tmpnpjgki5b/output/auto-mpg/2023-09-23-20-37-47-0_UserInputConfig (data.csv and tabular_data.csv).
-------------Run PytorchTabular-------------
Training Category Embedding
Global seed set to 42
2023-09-23 20:38:38,911 - {pytorch_tabular.tabular_model:473} - INFO - Preparing the DataLoaders
2023-09-23 20:38:38,912 - {pytorch_tabular.tabular_datamodule:290} - INFO - Setting up the datamodule for regression task
2023-09-23 20:38:38,920 - {pytorch_tabular.tabular_model:521} - INFO - Preparing the Model: CategoryEmbeddingModel
2023-09-23 20:38:38,932 - {pytorch_tabular.tabular_model:268} - INFO - Preparing the Trainer
/home/xlluo/anaconda3/envs/tabular_ensemble/lib/python3.10/site-packages/pytorch_lightning/trainer/connectors/accelerator_connector.py:589: LightningDeprecationWarning: The Trainer argument `auto_select_gpus` has been deprecated in v1.9.0 and will be removed in v2.0.0. Please use the function `pytorch_lightning.accelerators.find_usable_cuda_devices` instead.
rank_zero_deprecation(
Auto select gpus: [0]
GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
2023-09-23 20:38:38,946 - {pytorch_tabular.tabular_model:582} - INFO - Training Started
You are using a CUDA device ('NVIDIA GeForce RTX 3090') that has Tensor Cores. To properly utilize them, you should set `torch.set_float32_matmul_precision('medium' | 'high')` which will trade-off precision for performance. For more details, read https://pytorch.org/docs/stable/generated/torch.set_float32_matmul_precision.html#torch.set_float32_matmul_precision
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
| Name | Type | Params
---------------------------------------------------------------
0 | _backbone | CategoryEmbeddingBackbone | 11.4 K
1 | _embedding_layer | Embedding1dLayer | 14
2 | head | LinearHead | 33
3 | loss | MSELoss | 0
---------------------------------------------------------------
11.4 K Trainable params
0 Non-trainable params
11.4 K Total params
0.046 Total estimated model params size (MB)
Epoch: 1/300, Train loss: 632.3258, Val loss: 662.7650, Min val loss: 662.7650, Epoch time: 0.010s.
Epoch: 20/300, Train loss: 318.3275, Val loss: 344.3962, Min val loss: 344.3962, Epoch time: 0.012s.
Epoch: 40/300, Train loss: 69.3076, Val loss: 76.5520, Min val loss: 76.5520, Epoch time: 0.012s.
Epoch: 60/300, Train loss: 42.2416, Val loss: 46.7150, Min val loss: 46.7150, Epoch time: 0.014s.
Epoch: 80/300, Train loss: 29.2576, Val loss: 34.4237, Min val loss: 34.4237, Epoch time: 0.012s.
Epoch: 100/300, Train loss: 26.1472, Val loss: 29.0749, Min val loss: 29.0749, Epoch time: 0.012s.
Epoch: 120/300, Train loss: 24.8970, Val loss: 26.5129, Min val loss: 26.5129, Epoch time: 0.008s.
Epoch: 140/300, Train loss: 17.9044, Val loss: 25.3725, Min val loss: 25.3725, Epoch time: 0.009s.
Epoch: 160/300, Train loss: 17.4626, Val loss: 23.7306, Min val loss: 23.7306, Epoch time: 0.009s.
Epoch: 180/300, Train loss: 16.5023, Val loss: 22.9689, Min val loss: 22.9689, Epoch time: 0.011s.
Epoch: 200/300, Train loss: 15.1315, Val loss: 22.0908, Min val loss: 22.0908, Epoch time: 0.013s.
Epoch: 220/300, Train loss: 14.4462, Val loss: 21.3679, Min val loss: 21.3679, Epoch time: 0.010s.
Epoch: 240/300, Train loss: 15.4215, Val loss: 20.7016, Min val loss: 20.6893, Epoch time: 0.008s.
Epoch: 260/300, Train loss: 13.5062, Val loss: 20.0930, Min val loss: 20.0624, Epoch time: 0.009s.
Epoch: 280/300, Train loss: 14.5558, Val loss: 19.2505, Min val loss: 19.2505, Epoch time: 0.010s.
Epoch: 300/300, Train loss: 14.8346, Val loss: 18.8435, Min val loss: 18.8435, Epoch time: 0.009s.
`Trainer.fit` stopped: `max_epochs=300` reached.
2023-09-23 20:38:43,583 - {pytorch_tabular.tabular_model:584} - INFO - Training the model completed
2023-09-23 20:38:43,584 - {pytorch_tabular.tabular_model:1258} - INFO - Loading the best model
/home/xlluo/anaconda3/envs/tabular_ensemble/lib/python3.10/site-packages/pytorch_lightning/utilities/cloud_io.py:33: LightningDeprecationWarning: `pytorch_lightning.utilities.cloud_io.get_filesystem` has been deprecated in v1.8.0 and will be removed in v2.0.0. Please use `lightning_fabric.utilities.cloud_io.get_filesystem` instead.
rank_zero_deprecation(
Training mse loss: 10.18515
Validation mse loss: 18.84349
Testing mse loss: 9.93399
Trainer saved. To load the trainer, run trainer = load_trainer(path='/tmp/tmpnpjgki5b/output/auto-mpg/2023-09-23-20-37-47-0_UserInputConfig/trainer.pkl')
-------------PytorchTabular End-------------
Category Embedding 1/1
--------------------------End 2/2 random--------------------------
Trainer saved. To load the trainer, run trainer = load_trainer(path='/tmp/tmpnpjgki5b/output/auto-mpg/2023-09-23-20-37-47-0_UserInputConfig/trainer.pkl')
[5]:
| Program | Model | Training RMSE | Training MSE | Training MAE | Training MAPE | Training R2 | Training MEDIAN_ABSOLUTE_ERROR | Training EXPLAINED_VARIANCE_SCORE | Testing RMSE | ... | Testing R2 | Testing MEDIAN_ABSOLUTE_ERROR | Testing EXPLAINED_VARIANCE_SCORE | Validation RMSE | Validation MSE | Validation MAE | Validation MAPE | Validation R2 | Validation MEDIAN_ABSOLUTE_ERROR | Validation EXPLAINED_VARIANCE_SCORE | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | PytorchTabular | Category Embedding | 3.273904 | 10.718448 | 2.439552 | 0.103957 | 0.827288 | 1.87891 | 0.857783 | 2.980939 | ... | 0.844204 | 1.972121 | 0.890054 | 3.950363 | 15.605369 | 2.875634 | 0.123573 | 0.745909 | 2.082971 | 0.763982 |
1 rows × 23 columns
Let’s compare the result without termination.
[6]:
l2 = trainer.get_leaderboard(cross_validation=2, split_type="random", stderr_to_stdout=True)
import numpy as np
cols = ["Training RMSE", "Testing RMSE", "Validation RMSE"]
assert np.allclose(l1[cols].values.astype(float), l2[cols].values.astype(float))
l2
----------------------------1/2 random----------------------------
Using previously used data path /tmp/tmpnpjgki5b/data/auto-mpg.csv
Dataset size: 238 80 80
Data saved to /tmp/tmpnpjgki5b/output/auto-mpg/2023-09-23-20-37-47-0_UserInputConfig (data.csv and tabular_data.csv).
-------------Run PytorchTabular-------------
Training Category Embedding
Global seed set to 42
2023-09-23 20:38:44,123 - {pytorch_tabular.tabular_model:473} - INFO - Preparing the DataLoaders
2023-09-23 20:38:44,124 - {pytorch_tabular.tabular_datamodule:290} - INFO - Setting up the datamodule for regression task
2023-09-23 20:38:44,137 - {pytorch_tabular.tabular_model:521} - INFO - Preparing the Model: CategoryEmbeddingModel
2023-09-23 20:38:44,148 - {pytorch_tabular.tabular_model:268} - INFO - Preparing the Trainer
/home/xlluo/anaconda3/envs/tabular_ensemble/lib/python3.10/site-packages/pytorch_lightning/trainer/connectors/accelerator_connector.py:589: LightningDeprecationWarning: The Trainer argument `auto_select_gpus` has been deprecated in v1.9.0 and will be removed in v2.0.0. Please use the function `pytorch_lightning.accelerators.find_usable_cuda_devices` instead.
rank_zero_deprecation(
Auto select gpus: [0]
GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
2023-09-23 20:38:44,162 - {pytorch_tabular.tabular_model:582} - INFO - Training Started
You are using a CUDA device ('NVIDIA GeForce RTX 3090') that has Tensor Cores. To properly utilize them, you should set `torch.set_float32_matmul_precision('medium' | 'high')` which will trade-off precision for performance. For more details, read https://pytorch.org/docs/stable/generated/torch.set_float32_matmul_precision.html#torch.set_float32_matmul_precision
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
| Name | Type | Params
---------------------------------------------------------------
0 | _backbone | CategoryEmbeddingBackbone | 11.4 K
1 | _embedding_layer | Embedding1dLayer | 14
2 | head | LinearHead | 33
3 | loss | MSELoss | 0
---------------------------------------------------------------
11.4 K Trainable params
0 Non-trainable params
11.4 K Total params
0.046 Total estimated model params size (MB)
Epoch: 1/300, Train loss: 677.8015, Val loss: 582.9557, Min val loss: 582.9557, Epoch time: 0.011s.
Epoch: 20/300, Train loss: 353.7851, Val loss: 302.0203, Min val loss: 302.0203, Epoch time: 0.008s.
Epoch: 40/300, Train loss: 85.0776, Val loss: 62.1153, Min val loss: 62.1153, Epoch time: 0.008s.
Epoch: 60/300, Train loss: 45.2654, Val loss: 34.2778, Min val loss: 34.2691, Epoch time: 0.009s.
Epoch: 80/300, Train loss: 33.9537, Val loss: 26.8622, Min val loss: 26.8622, Epoch time: 0.009s.
Epoch: 100/300, Train loss: 26.9038, Val loss: 23.2417, Min val loss: 23.2372, Epoch time: 0.008s.
Epoch: 120/300, Train loss: 24.9622, Val loss: 20.4360, Min val loss: 20.4360, Epoch time: 0.009s.
Epoch: 140/300, Train loss: 24.1636, Val loss: 19.4010, Min val loss: 19.4010, Epoch time: 0.010s.
Epoch: 160/300, Train loss: 22.9200, Val loss: 18.0232, Min val loss: 17.9749, Epoch time: 0.009s.
Epoch: 180/300, Train loss: 19.7677, Val loss: 16.9469, Min val loss: 16.9469, Epoch time: 0.013s.
Epoch: 200/300, Train loss: 17.9390, Val loss: 16.6545, Min val loss: 16.4093, Epoch time: 0.008s.
Epoch: 220/300, Train loss: 19.4496, Val loss: 15.4451, Min val loss: 15.1788, Epoch time: 0.008s.
Epoch: 240/300, Train loss: 16.0483, Val loss: 14.5508, Min val loss: 14.5508, Epoch time: 0.009s.
Epoch: 260/300, Train loss: 16.4672, Val loss: 13.8354, Min val loss: 13.8354, Epoch time: 0.009s.
Epoch: 280/300, Train loss: 13.6031, Val loss: 12.9315, Min val loss: 12.9315, Epoch time: 0.009s.
Epoch: 300/300, Train loss: 16.5369, Val loss: 12.3673, Min val loss: 12.3673, Epoch time: 0.012s.
`Trainer.fit` stopped: `max_epochs=300` reached.
2023-09-23 20:38:48,081 - {pytorch_tabular.tabular_model:584} - INFO - Training the model completed
2023-09-23 20:38:48,081 - {pytorch_tabular.tabular_model:1258} - INFO - Loading the best model
/home/xlluo/anaconda3/envs/tabular_ensemble/lib/python3.10/site-packages/pytorch_lightning/utilities/cloud_io.py:33: LightningDeprecationWarning: `pytorch_lightning.utilities.cloud_io.get_filesystem` has been deprecated in v1.8.0 and will be removed in v2.0.0. Please use `lightning_fabric.utilities.cloud_io.get_filesystem` instead.
rank_zero_deprecation(
Training mse loss: 11.25175
Validation mse loss: 12.36725
Testing mse loss: 7.83801
Trainer saved. To load the trainer, run trainer = load_trainer(path='/tmp/tmpnpjgki5b/output/auto-mpg/2023-09-23-20-37-47-0_UserInputConfig/trainer.pkl')
-------------PytorchTabular End-------------
Category Embedding 1/1
--------------------------End 1/2 random--------------------------
----------------------------2/2 random----------------------------
Using previously used data path /tmp/tmpnpjgki5b/data/auto-mpg.csv
Dataset size: 238 80 80
Data saved to /tmp/tmpnpjgki5b/output/auto-mpg/2023-09-23-20-37-47-0_UserInputConfig (data.csv and tabular_data.csv).
-------------Run PytorchTabular-------------
Training Category Embedding
Global seed set to 42
2023-09-23 20:38:48,582 - {pytorch_tabular.tabular_model:473} - INFO - Preparing the DataLoaders
2023-09-23 20:38:48,583 - {pytorch_tabular.tabular_datamodule:290} - INFO - Setting up the datamodule for regression task
2023-09-23 20:38:48,593 - {pytorch_tabular.tabular_model:521} - INFO - Preparing the Model: CategoryEmbeddingModel
2023-09-23 20:38:48,607 - {pytorch_tabular.tabular_model:268} - INFO - Preparing the Trainer
/home/xlluo/anaconda3/envs/tabular_ensemble/lib/python3.10/site-packages/pytorch_lightning/trainer/connectors/accelerator_connector.py:589: LightningDeprecationWarning: The Trainer argument `auto_select_gpus` has been deprecated in v1.9.0 and will be removed in v2.0.0. Please use the function `pytorch_lightning.accelerators.find_usable_cuda_devices` instead.
rank_zero_deprecation(
Auto select gpus: [0]
GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
2023-09-23 20:38:48,620 - {pytorch_tabular.tabular_model:582} - INFO - Training Started
You are using a CUDA device ('NVIDIA GeForce RTX 3090') that has Tensor Cores. To properly utilize them, you should set `torch.set_float32_matmul_precision('medium' | 'high')` which will trade-off precision for performance. For more details, read https://pytorch.org/docs/stable/generated/torch.set_float32_matmul_precision.html#torch.set_float32_matmul_precision
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
| Name | Type | Params
---------------------------------------------------------------
0 | _backbone | CategoryEmbeddingBackbone | 11.4 K
1 | _embedding_layer | Embedding1dLayer | 14
2 | head | LinearHead | 33
3 | loss | MSELoss | 0
---------------------------------------------------------------
11.4 K Trainable params
0 Non-trainable params
11.4 K Total params
0.046 Total estimated model params size (MB)
Epoch: 1/300, Train loss: 632.3258, Val loss: 662.7650, Min val loss: 662.7650, Epoch time: 0.012s.
Epoch: 20/300, Train loss: 318.3275, Val loss: 344.3962, Min val loss: 344.3962, Epoch time: 0.007s.
Epoch: 40/300, Train loss: 69.3076, Val loss: 76.5520, Min val loss: 76.5520, Epoch time: 0.008s.
Epoch: 60/300, Train loss: 42.2416, Val loss: 46.7150, Min val loss: 46.7150, Epoch time: 0.008s.
Epoch: 80/300, Train loss: 29.2576, Val loss: 34.4237, Min val loss: 34.4237, Epoch time: 0.009s.
Epoch: 100/300, Train loss: 26.1472, Val loss: 29.0749, Min val loss: 29.0749, Epoch time: 0.008s.
Epoch: 120/300, Train loss: 24.8970, Val loss: 26.5129, Min val loss: 26.5129, Epoch time: 0.008s.
Epoch: 140/300, Train loss: 17.9044, Val loss: 25.3725, Min val loss: 25.3725, Epoch time: 0.008s.
Epoch: 160/300, Train loss: 17.4626, Val loss: 23.7306, Min val loss: 23.7306, Epoch time: 0.010s.
Epoch: 180/300, Train loss: 16.5023, Val loss: 22.9689, Min val loss: 22.9689, Epoch time: 0.008s.
Epoch: 200/300, Train loss: 15.1315, Val loss: 22.0908, Min val loss: 22.0908, Epoch time: 0.008s.
Epoch: 220/300, Train loss: 14.4462, Val loss: 21.3679, Min val loss: 21.3679, Epoch time: 0.008s.
Epoch: 240/300, Train loss: 15.4215, Val loss: 20.7016, Min val loss: 20.6893, Epoch time: 0.009s.
Epoch: 260/300, Train loss: 13.5062, Val loss: 20.0930, Min val loss: 20.0624, Epoch time: 0.010s.
Epoch: 280/300, Train loss: 14.5558, Val loss: 19.2505, Min val loss: 19.2505, Epoch time: 0.008s.
Epoch: 300/300, Train loss: 14.8346, Val loss: 18.8435, Min val loss: 18.8435, Epoch time: 0.009s.
`Trainer.fit` stopped: `max_epochs=300` reached.
2023-09-23 20:38:52,437 - {pytorch_tabular.tabular_model:584} - INFO - Training the model completed
2023-09-23 20:38:52,437 - {pytorch_tabular.tabular_model:1258} - INFO - Loading the best model
/home/xlluo/anaconda3/envs/tabular_ensemble/lib/python3.10/site-packages/pytorch_lightning/utilities/cloud_io.py:33: LightningDeprecationWarning: `pytorch_lightning.utilities.cloud_io.get_filesystem` has been deprecated in v1.8.0 and will be removed in v2.0.0. Please use `lightning_fabric.utilities.cloud_io.get_filesystem` instead.
rank_zero_deprecation(
Training mse loss: 10.18515
Validation mse loss: 18.84349
Testing mse loss: 9.93399
Trainer saved. To load the trainer, run trainer = load_trainer(path='/tmp/tmpnpjgki5b/output/auto-mpg/2023-09-23-20-37-47-0_UserInputConfig/trainer.pkl')
-------------PytorchTabular End-------------
Category Embedding 1/1
--------------------------End 2/2 random--------------------------
Trainer saved. To load the trainer, run trainer = load_trainer(path='/tmp/tmpnpjgki5b/output/auto-mpg/2023-09-23-20-37-47-0_UserInputConfig/trainer.pkl')
[6]:
| Program | Model | Training RMSE | Training MSE | Training MAE | Training MAPE | Training R2 | Training MEDIAN_ABSOLUTE_ERROR | Training EXPLAINED_VARIANCE_SCORE | Testing RMSE | ... | Testing R2 | Testing MEDIAN_ABSOLUTE_ERROR | Testing EXPLAINED_VARIANCE_SCORE | Validation RMSE | Validation MSE | Validation MAE | Validation MAPE | Validation R2 | Validation MEDIAN_ABSOLUTE_ERROR | Validation EXPLAINED_VARIANCE_SCORE | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | PytorchTabular | Category Embedding | 3.273904 | 10.718448 | 2.439552 | 0.103957 | 0.827288 | 1.87891 | 0.857783 | 2.980939 | ... | 0.844204 | 1.972121 | 0.890054 | 3.950363 | 15.605369 | 2.875634 | 0.123573 | 0.745909 | 2.082971 | 0.763982 |
1 rows × 23 columns