{
 "cells": [
  {
   "cell_type": "markdown",
   "source": [
    "# Change optimizers and lr shedulers"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "The default optimizer of `PytorchTabular`, `WideDeep`, and `TorchModel` is `torch.optim.Adam` while the default lr scheduler is `torch.optim.lr_scheduler.StepLR` (but with `gamma=1` so that the learning rate does not change).\n",
    "\n",
    "To set optimizers or lr schedulers for each model in each model base, use the `optimizers` and `lr_schedulers` arguments, such as\n",
    "\n",
    "```python\n",
    "models = [\n",
    "    PytorchTabular(\n",
    "        trainer,\n",
    "        model_subset=[\"Category Embedding\"],\n",
    "        optimizers={\n",
    "            \"Category Embedding\": (\"AdamW\", {\"lr\": None, \"weight_decay\": None})\n",
    "        },\n",
    "        lr_schedulers={\n",
    "            \"Category Embedding\": (\"StepLR\", {\"gamma\": 0.1, \"step_size\": 1})\n",
    "        },\n",
    "    )\n",
    "]\n",
    "```\n",
    "\n",
    "Optimizers and lr schedulers are set by their names in `torch.optim` and `torch.optim.lr_scheduler`, respectively.\n",
    "\n",
    "Note that, if names match, the values of parameters (such as the `None`s shown above) will be updated by those from `AbstractModel._initial_values` or those optimized during the bayesian hyperparameter optimization."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}