{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Reinforcement Learning Solution to the Towers of Hanoi Puzzle"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For this assignment, you will use reinforcement learning to solve the [Towers of Hanoi](https://en.wikipedia.org/wiki/Tower_of_Hanoi) puzzle with three pegs and five disks.\n",
    "\n",
    "To accomplish this, you must modify the code discussed in lecture for learning to play Tic-Tac-Toe.  Modify the code  so that it learns to solve the five-disk, three-peg\n",
    "Towers of Hanoi Puzzle.  In some ways, this will be simpler than the\n",
    "Tic-Tac-Toe code.  \n",
    "\n",
    "Steps required to do this include the following:\n",
    "\n",
    "  - Represent the state and move, and use it as a tuple as a key to the Q dictionary.\n",
    "  - Make sure only valid moves are tried from each state.\n",
    "  - Assign reinforcement of $1$ to each move, even for the move that results in the goal state.\n",
    "\n",
    "Make a plot of the number of steps required to reach the goal for each\n",
    "trial.  Each trial starts from the same initial state.  Decay epsilon\n",
    "as in the Tic-Tac-Toe code."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Requirements"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, how should we represent the state of this puzzle?  We need to keep track of which disks are on which pegs. Name the disks 1, 2, 3, 4, and 5, with 1 being the smallest disk and 5 being the largest. The set of disks on a peg can be represented as a list of integers.  Then the state can be a list of three lists.\n",
    "\n",
    "For example, the starting state with all disks being on the left peg would be `[[1, 2, 3, 4, 5], [], []]`.  After moving disk 1 to peg 2, we have `[[2, 3, 4, 5], [1], []]`.\n",
    "\n",
    "To represent that move we just made, we can use a list of two peg numbers, like `[1, 2]`, representing a move of the top disk on peg 1 to peg 2."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now on to some functions. Define at least the following functions. Examples showing required output appear below.\n",
    "\n",
    "   - `print_state(state)`: prints the state in the form shown below\n",
    "   - `get_valid_moves(state)`: returns list of moves that are valid from `state`\n",
    "   - `make_move(state, move)`: returns new (copy of) state after move has been applied.\n",
    "   - `train_Q(n_repetitions, learning_rate, epsilon_decay_factor, get_valid_moves, make_move)`: train the Q function for number of repetitions, decaying epsilon at start of each repetition. Returns Q and list or array of number of steps to reach goal for each repetition.\n",
    "   - `test_Q(Q, max_steps, get_valid_moves, make_move)`: without updating Q, use Q to find greedy action each step until goal is found. Return path of states.\n",
    "\n",
    "A function that you might choose to implement is\n",
    "\n",
    "   - `state_move_tuple(state, move)`: returns tuple of state and move.  \n",
    "    \n",
    "This is useful for converting state and move to a key to be used for the Q dictionary.\n",
    "\n",
    "Show the code and results for testing each function.  Then experiment with various values of `n_repetitions`, `learning_rate`, and `epsilon_decay_factor` to find values that work reasonably well, meaning that eventually the minimum solution path of 31 steps is found consistently.\n",
    "\n",
    "Make a plot of the number of steps in the solution path versus number of repetitions. The plot should clearly show the number of steps in the solution path eventually reaching the minimum of 31 steps, though the decrease will not be monotonic.  Also plot a horizontal, dashed line at a height (value on  y axis) of 31 to show the optimal path length.\n",
    "\n",
    "Use at least a total of 15 sentences to describe the following results:\n",
    "- Add markdown cells in which you describe the Q learning algorithm and your implementation of Q learning as applied to the Towers of Hanoi problem.\n",
    "- Add code cells to examine several Q values from the start state with different moves and discuss if the Q values  make sense. \n",
    "- Also add code cells to examine several Q values from one or two states that are two steps away from the goal and discuss if these Q values make sense."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Examples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1     \n",
      "2     \n",
      "3     \n",
      "4     \n",
      "5     \n",
      "------\n",
      "\n"
     ]
    }
   ],
   "source": [
    "state = [[1, 2, 3, 4, 5], [], []]\n",
    "print_state(state)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(((1, 2, 3, 4, 5), (), ()), (1, 2))"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "move =[1, 2]  # Move top (smallest) disk from first peg to second peg\n",
    "\n",
    "state_move_tuple(state, move)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[2, 3, 4, 5], [1], []]"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_state = make_move(state, move)\n",
    "new_state"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[1, 3], [2, 1], [2, 3]]"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_valid_moves(new_state)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2     \n",
      "3     \n",
      "4     \n",
      "5 1   \n",
      "------\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print_state(new_state)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "Q, steps_to_goal = train_Q(200, 0.5, 0.7, get_valid_moves, make_move)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1302.,  425.,  900., 1919.,  528.,  907.,  982.,  834.,  280.,\n",
       "        314.,  519.,  307.,  325.,  704.,  246.,  618.,  330.,  420.,\n",
       "        144.,  241.,  179.,  199.,  405.,  379.,  556.,  183.,  559.,\n",
       "        167.,  162.,  257.,   89.,  160.,  425.,  262.,  109.,  164.,\n",
       "        322.,  217.,  164.,   96.,  198.,  104.,  113.,  196.,   93.,\n",
       "         94.,  288.,  121.,  372.,   89.,  146.,  343.,  233.,   82.,\n",
       "         63.,   58.,  260.,   90.,   96.,   70.,  342.,  124.,   67.,\n",
       "        115.,   86.,  360.,   93.,  189.,   80.,   67.,   70.,  253.,\n",
       "         60.,   97.,   41.,  362.,   80.,  109.,  174.,  249.,   81.,\n",
       "         63.,   52.,  207.,  275.,   51.,   43.,   92.,   48.,  123.,\n",
       "         57.,  157.,   48.,   42.,   50.,   42.,   55.,  149.,  228.,\n",
       "         60.,   68.,   37.,   36.,   68.,   38.,   50.,  261.,   93.,\n",
       "        126.,   60.,   36.,   50.,   46.,   40.,  126.,   50.,   82.,\n",
       "        108.,   35.,   39.,   33.,  102.,   33.,   45.,   36.,   37.,\n",
       "         49.,  224.,   31.,   39.,   32.,  157.,   39.,   34.,   53.,\n",
       "         31.,   31.,   32.,   31.,   31.,   35.,   31.,  103.,   31.,\n",
       "         79.,   31.,   31.,   31.,   40.,   31.,   31.,   31.,   31.,\n",
       "         31.,   31.,   31.,   31.,   31.,   31.,   31.,   31.,   31.,\n",
       "         31.,   31.,   31.,   31.,   31.,   31.,   31.,   31.,   31.,\n",
       "         31.,   31.,   31.,   31.,   31.,   31.,   31.,   31.,   31.,\n",
       "         31.,   31.,   31.,   31.,   31.,   31.,   31.,   31.,   31.,\n",
       "         31.,   31.,   31.,   31.,   31.,   31.,   31.,   31.,   31.,\n",
       "         31.,   31.])"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "steps_to_goal"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "path = test_Q(Q, 100, get_valid_moves, make_move)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[[1, 2, 3, 4, 5], [], []],\n",
       " [[2, 3, 4, 5], [], [1]],\n",
       " [[3, 4, 5], [2], [1]],\n",
       " [[3, 4, 5], [1, 2], []],\n",
       " [[4, 5], [1, 2], [3]],\n",
       " [[1, 4, 5], [2], [3]],\n",
       " [[1, 4, 5], [], [2, 3]],\n",
       " [[4, 5], [], [1, 2, 3]],\n",
       " [[5], [4], [1, 2, 3]],\n",
       " [[5], [1, 4], [2, 3]],\n",
       " [[2, 5], [1, 4], [3]],\n",
       " [[1, 2, 5], [4], [3]],\n",
       " [[1, 2, 5], [3, 4], []],\n",
       " [[2, 5], [3, 4], [1]],\n",
       " [[5], [2, 3, 4], [1]],\n",
       " [[5], [1, 2, 3, 4], []],\n",
       " [[], [1, 2, 3, 4], [5]],\n",
       " [[1], [2, 3, 4], [5]],\n",
       " [[1], [3, 4], [2, 5]],\n",
       " [[], [3, 4], [1, 2, 5]],\n",
       " [[3], [4], [1, 2, 5]],\n",
       " [[3], [1, 4], [2, 5]],\n",
       " [[2, 3], [1, 4], [5]],\n",
       " [[1, 2, 3], [4], [5]],\n",
       " [[1, 2, 3], [], [4, 5]],\n",
       " [[2, 3], [], [1, 4, 5]],\n",
       " [[3], [2], [1, 4, 5]],\n",
       " [[3], [1, 2], [4, 5]],\n",
       " [[], [1, 2], [3, 4, 5]],\n",
       " [[1], [2], [3, 4, 5]],\n",
       " [[1], [], [2, 3, 4, 5]],\n",
       " [[], [1], [2, 3, 4, 5]],\n",
       " [[], [], [1, 2, 3, 4, 5]]]"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "path"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1     \n",
      "2     \n",
      "3     \n",
      "4     \n",
      "5     \n",
      "------\n",
      "\n",
      "\n",
      "2     \n",
      "3     \n",
      "4     \n",
      "5   1 \n",
      "------\n",
      "\n",
      "\n",
      "3     \n",
      "4     \n",
      "5 2 1 \n",
      "------\n",
      "\n",
      "\n",
      "3     \n",
      "4 1   \n",
      "5 2   \n",
      "------\n",
      "\n",
      "\n",
      "4 1   \n",
      "5 2 3 \n",
      "------\n",
      "\n",
      "\n",
      "1     \n",
      "4     \n",
      "5 2 3 \n",
      "------\n",
      "\n",
      "\n",
      "1     \n",
      "4   2 \n",
      "5   3 \n",
      "------\n",
      "\n",
      "\n",
      "    1 \n",
      "4   2 \n",
      "5   3 \n",
      "------\n",
      "\n",
      "\n",
      "    1 \n",
      "    2 \n",
      "5 4 3 \n",
      "------\n",
      "\n",
      "\n",
      "  1 2 \n",
      "5 4 3 \n",
      "------\n",
      "\n",
      "\n",
      "2 1   \n",
      "5 4 3 \n",
      "------\n",
      "\n",
      "\n",
      "1     \n",
      "2     \n",
      "5 4 3 \n",
      "------\n",
      "\n",
      "\n",
      "1     \n",
      "2 3   \n",
      "5 4   \n",
      "------\n",
      "\n",
      "\n",
      "2 3   \n",
      "5 4 1 \n",
      "------\n",
      "\n",
      "\n",
      "  2   \n",
      "  3   \n",
      "5 4 1 \n",
      "------\n",
      "\n",
      "\n",
      "  1   \n",
      "  2   \n",
      "  3   \n",
      "5 4   \n",
      "------\n",
      "\n",
      "\n",
      "  1   \n",
      "  2   \n",
      "  3   \n",
      "  4 5 \n",
      "------\n",
      "\n",
      "\n",
      "  2   \n",
      "  3   \n",
      "1 4 5 \n",
      "------\n",
      "\n",
      "\n",
      "  3 2 \n",
      "1 4 5 \n",
      "------\n",
      "\n",
      "\n",
      "    1 \n",
      "  3 2 \n",
      "  4 5 \n",
      "------\n",
      "\n",
      "\n",
      "    1 \n",
      "    2 \n",
      "3 4 5 \n",
      "------\n",
      "\n",
      "\n",
      "  1 2 \n",
      "3 4 5 \n",
      "------\n",
      "\n",
      "\n",
      "2 1   \n",
      "3 4 5 \n",
      "------\n",
      "\n",
      "\n",
      "1     \n",
      "2     \n",
      "3 4 5 \n",
      "------\n",
      "\n",
      "\n",
      "1     \n",
      "2   4 \n",
      "3   5 \n",
      "------\n",
      "\n",
      "\n",
      "    1 \n",
      "2   4 \n",
      "3   5 \n",
      "------\n",
      "\n",
      "\n",
      "    1 \n",
      "    4 \n",
      "3 2 5 \n",
      "------\n",
      "\n",
      "\n",
      "  1 4 \n",
      "3 2 5 \n",
      "------\n",
      "\n",
      "\n",
      "    3 \n",
      "  1 4 \n",
      "  2 5 \n",
      "------\n",
      "\n",
      "\n",
      "    3 \n",
      "    4 \n",
      "1 2 5 \n",
      "------\n",
      "\n",
      "\n",
      "    2 \n",
      "    3 \n",
      "    4 \n",
      "1   5 \n",
      "------\n",
      "\n",
      "\n",
      "    2 \n",
      "    3 \n",
      "    4 \n",
      "  1 5 \n",
      "------\n",
      "\n",
      "\n",
      "    1 \n",
      "    2 \n",
      "    3 \n",
      "    4 \n",
      "    5 \n",
      "------\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for s in path:\n",
    "    print_state(s)\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Grading"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Download and extract `A4grader.py` from [A4grader.tar](http://www.cs.colostate.edu/~anderson/cs440/notebooks/A4grader.tar)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "======================= Code Execution =======================\n",
      "\n",
      "['Anderson-A4.ipynb']\n",
      "Extracting python code from notebook named 'Anderson-A4.ipynb' and storing in notebookcode.py\n",
      "Removing all statements that are not function or class defs or import statements.\n",
      "\n",
      "Testing\n",
      "\n",
      "  state = [[1], [2,3], [4, 5]]\n",
      "  moves = get_valid_moves(state)\n",
      "\n",
      "\n",
      "--- 5/5 points. Correctly returned [[1, 2], [1, 3], [2, 3]]\n",
      "\n",
      "Testing\n",
      "\n",
      "    state = [[], [], [1, 2, 3, 4, 5]]\n",
      "    moves = get_valid_moves(state)\n",
      "\n",
      "\n",
      "--- 5/5 points. Correctly returned [[3, 1], [3, 2]]\n",
      "\n",
      "Testing\n",
      "\n",
      "    state = [[], [], [1, 2, 3, 4, 5]]\n",
      "    new_state = make_move(state, [3, 1])\n",
      "\n",
      "\n",
      "--- 5/5 points. Correctly returned [[1], [], [2, 3, 4, 5]]\n",
      "\n",
      "Testing\n",
      "\n",
      "    state = [[1, 2], [3], [4, 5]]\n",
      "    new_state = make_move(state, [1, 3])\n",
      "\n",
      "\n",
      "--- 5/5 points. Correctly returned [[2], [3], [1, 4, 5]]\n",
      "\n",
      "Testing\n",
      "\n",
      "    Q, steps = train_Q(1000, 0.5, 0.7, get_valid_moves, make_move)\n",
      "\n",
      "\n",
      "--- 10/10 points. Correctly returned list of steps that has 1000 elements.\n",
      "\n",
      "--- 10/10 points. Correctly returned a Q dictionary with at least 700 elements.\n",
      "\n",
      "Testing\n",
      "\n",
      "    path = test_Q(Q, 20, get_valid_moves, make_move)\n",
      "\n",
      "\n",
      "--- 20/20 points. Correctly returned path with fewer than 40 states.\n",
      "\n",
      "======================================================================\n",
      "A4 Execution Grade is 60 / 60\n",
      "======================================================================\n",
      "\n",
      "                \n",
      "___ / 10 points. Correct plot of the number of steps in the solution path versus the\n",
      "                 number of repetitions.\n",
      "\n",
      "___ / 10 points. Add markdown cells in which you describe the Q learning algorithm\n",
      "                 and your implementation of Q learning as applied to the \n",
      "                 Towers of Hanoi problem.\n",
      "\n",
      "___ / 10 points. Add code cells to examine several Q values from the start state\n",
      "                 with different moves and discuss if the Q values make sense.\n",
      "\n",
      "___ / 10 points. Also add code cells to examine several Q values from one or two states\n",
      "                 that are two steps away from the goal and discuss if these Q values make sense.\n",
      "\n",
      "\n",
      "======================================================================\n",
      "A4 Additional Grade is __ / 40\n",
      "======================================================================\n",
      "\n",
      "======================================================================\n",
      "A4 FINAL GRADE is  _  / 100\n",
      "======================================================================\n",
      "\n",
	 "Extra Credit: Earn one point of extra credit for your code for solving\n",
	 "the Towers of Hanoi puzzle with four pegs and five disks and the experiments\n",
	 "and discussion of results.\n",
      "\n",
      "A4 EXTRA CREDIT is 0 / 1\n"
     ]
    }
   ],
   "source": [
    "%run -i A4grader.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Extra Credit"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Modify your code to solve the Towers of Hanoi puzzle with four pegs and five disks.  Name your functions\n",
    "\n",
    "    - print_state_4pegs\n",
    "    - get_valid_moves_4pegs\n",
    "    - make_move_4pegs\n",
    "\n",
    "Find values for number of repetitions, learning rate, and epsilon decay factor for which train_Q learns a Q function that test_Q can use to find the shortest solution path.  Include the output from the successful calls to train_Q and test_Q."
   ]
  }
 ],
 "metadata": {
  "jupytext": {
   "formats": "ipynb,py:light"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}