{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We presented at IJCNN, 2015 the following paper, which won the [Best Paper Award](http://www.ijcnn.org/assets/docs/ijcnn2015-awards.pdf)\n",
    "\n",
    "  * Anderson, C., Lee, M., and Elliott, D., \"[Faster Reinforcement Learning After Pretraining Deep Networks to Predict State Dynamics](publications/pretrainijcnn15.pdf)\", Proceedings of the IJCNN, 2015, Killarney, Ireland.\n",
    "\n",
    "*Abstract:* Deep learning algorithms have recently appeared that pretrain hidden layers of neural networks in unsupervised ways, leading to state-of-the-art performance on large classification problems. These methods can also pretrain networks used for reinforcement learning. However, this ignores the additional information that exists in a reinforcement learning paradigm via the ongoing sequence of state, action, new state tuples. This paper demonstrates that learning a predictive model of state dynamics can result in a pretrained hidden layer structure that reduces the time needed to solve reinforcement learning problems. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below is a sequence of animations showing the progress of reinforcement learning.  We simulated the dynamics of the cart-pole problem with full 360 pole rotation and collisions of the cart at the ends of the track. Details of this simulation will be provided here soon."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "<video width=\"400\" controls src=\"http://www.cs.colostate.edu/~anderson/wp/wp-content/uploads/2017/06/pole-2015-0min.mp4\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The animation above shows that after a few steps, our RL agent seems to prefer pushes to the right."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "<video controls width=\"400\" src=\"../wp-content/uploads/2017/06/pole-2015-10min.mp4\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After 10 minutes of training, it is starting to move away from the right edge."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "<video controls width=\"400\" src=\"videos/pole-2015-50min.mp4\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After 50 minutes of training, the pole is swinging up, but not being balanced."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "<video controls width=\"400\" src=\"videos/pole-2015-100min.mp4\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After 100 minutes of training, our RL agent occasionally is able to use bumps at end of track to help swing the pole up."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "<video controls width=\"400\" src=\"videos/pole-2015-200min.mp4\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And finally, after 200 minutes, the pole is being balanced!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "<video controls width=\"400\" src=\"videos/pole-2015-test1.mp4\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With $\\epsilon = 0$, so no exploration, we see quick swing up and balance."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "<video controls width=\"400\" src=\"videos/pole-2015-test1-slow.mp4\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again, in slow motion."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "<video controls width=\"400\" src=\"videos/pole-2015-test2-slow.mp4\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is another run with $\\epsilon = 0$, in slow motion."
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}