{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Assignment 3: ridge regression\n", "\n", "Due date: Friday 10/5 at 11:59pm\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Preliminaries \n", "\n", "In this assignment you will explore ridge regression applied to the task of predicting wine quality.\n", "You will use the [wine quality](http://archive.ics.uci.edu/ml/datasets/Wine+Quality)\n", "dataset from the UCI machine learning repository, and compare accuracy obtained using ridge regression to the results from a [recent publication](http://www.sciencedirect.com/science/article/pii/S0167923609001377#).\n", "If you have trouble accessing that version of the paper, here's a link to a [preprint](http://www3.dsi.uminho.pt/pcortez/wine5.pdf).\n", "The wine data is composed of two datasets - one for white wines, and one for red wines. In this assignment perform all your analyses on just the red wine data.\n", "\n", "The features for the wine dataset are not standardized, so make sure you do this, especially since we are going to consider the magnitude of the weight vector (recall that standardization entails subtracting the mean and then dividing by the standard deviation for each feature; you can use Numpy/scikit-learn to perform this computation.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Part 1\n", "\n", "Implement ridge regression in a class called RidgeRegression that implements the classifier API, i.e. ``fit`` and ``predict`` methods with the same signature as the classifiers you implemented in the previous assignment. Also implement functions for computing the following measures of error:\n", "\n", " * The Root Mean Square Error (RMSE).\n", " * The Maximum Absolute Deviation (MAD).\n", "\n", "For a hypothesis $h$, they are defined as follows:\n", "\n", "$$RMSE(h) = \\sqrt{\\frac{1}{N}\\sum_{i=1}^N (y_i - h(\\mathbf{x}_i))^2}$$\n", "\n", "and\n", "\n", "$$MAD(h) = \\frac{1}{N}\\sum_{i=1}^N |y_i - h(\\mathbf{x}_i)|.$$\n", "\n", "With the code you just implemented, your next task is to explore the dependence of validation-set error on the value of the regularization parameter, $\\lambda$.\n", "In what follows set aside 30% of the data for validation, and compute the in-sample error, and the validation-set error as a function of the parameter $\\lambda$ on the red wine data. Choose the values of $\\lambda$ on a logarithmic scale with values 0.01, 0.1, 1, 10, 100, 1000 and plot the RMSE.\n", "Repeat the same experiment where instead of using all the training data, choose 20 random examples out of the training set, and train your model using those 20 examples, while evaluating on the same validation set.\n", "\n", "Now answer the following:\n", "\n", " * What is the optimal value of $\\lambda$?\n", " * What observations can you make on the basis of these plots? (The concepts of overfitting/underfitting should be addressed in your answer).\n", " * Finally, compare the results that you are getting with the published results in the paper linked above. In particular, is the performance you have obtained comparable to that observed in the paper? Please note that for a completely fair comparison, the comparison needs to be made on a separate test set, in the same way the experiment was performed in the paper.\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Your answer here" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Part 2: regression error characteristic curves\n", "\n", "Regression Error Characteristic (REC) curves are an interesting way of visualizing regression error as described\n", "in the following [paper](http://machinelearning.wustl.edu/mlpapers/paper_files/icml2003_BiB03.pdf).\n", "Write a function that plots the REC curve of a regression method, and plot the REC curve of the best regressor you found in Part 1 of the assignment (i.e. the one that gave the lowest error on the validation set). Plot the REC curve for both the validation set and the training set.\n", "What can you learn from this curve that you cannot learn from an error measure such as RMSE or MAD?\n", "\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Your answer here" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Part 3: weight vector analysis\n", "\n", "As we discussed in class, the magnitude of the weight vector can be interpreted as a measure of feature importance.\n", "Train a ridge regression classifier on a subset of the dataset that you reserved for training.\n", "We will explore the relationship between the magnitude of weight vector components and their relevance to the classification task in several ways.\n", "Each feature is associated with a component of the weight vector. It can also be associated with the correlation of that feature with the vector of labels.\n", "As we discussed in class, the magnitude of the weight vector can give an indication of feature relevance; another measure of relevance of a feature is its correlation with the labels. To compare the two, \n", "create a scatter plot of weight vector components against the [Pearson correlation coefficient](https://en.wikipedia.org/wiki/Pearson_product-moment_correlation_coefficient) of the corresponding feature with the labels (again, you can use Numpy to compute it).\n", "What can you conclude from this plot?\n", "The paper ranks features according to their importance using a different approach. Compare your results with what they obtain.\n", "\n", "Next, perform the following experiment:\n", "Incrementally remove the feature with the lowest absolute value of the weight vector and retrain the ridge regression classifier.\n", "Plot RMSE as a function of the number of features that remain on the test set which you have set aside and comment on the results.\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Your answer here" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Your Report\n", "\n", "Answer the questions in the cells reserved for that purpose.\n", "\n", "Mathematical equations should be written as LaTex equations; the assignment contains multiple examples of both inline formulas (such as the one exemplifying the notation for the norm of a vector $||\\mathbf{x}||$ and those that appear on separate lines, e.g.:\n", "\n", "$$\n", "||\\mathbf{x}|| = \\sqrt{\\mathbf{x}^T \\mathbf{x}}.\n", "$$\n", "\n", "\n", "\n", "### Submission\n", "\n", "Submit your report as a Jupyter notebook via Canvas. Running the notebook should generate all the plots and results in your notebook.\n", "\n", "\n", "### Grading \n", "\n", "Here is what the grade sheet will look like for this assignment. A few general guidelines for this and future assignments in the course:\n", "\n", " * Your answers should be concise and to the point. We will take off points if that is not the case.\n", " * Always provide a description of the method you used to produce a given result in sufficient detail such that the reader can reproduce your results on the basis of the description. You can use a few lines of python code or pseudo-code.\n", "\n", "\n", "Grading sheet for the assignment:\n", "\n", "```\n", "Part 1: 50 points.\n", "(15 points): Ridge regression is correctly implemented.\n", "(15 points): Plots of RMSE as a function of lambda are generated correctly.\n", "(20 points): Discussion of the results\n", "\n", "Part 2: 25 points.\n", "(15 points): REC curves are generated correctly\n", "(10 points): discussion of REC curves\n", "\n", "Part 3: 25 points.\n", "(20 points): Weight vector analysis\n", "( 5 points): Comparison to the published results\n", "```\n", "\n", "\n", "Grading will be based on the following criteria:\n", "\n", " * Correctness of answers to math problems\n", " * Math is formatted as LaTex equations\n", " * Correct behavior of the required code\n", " * Easy to understand plots \n", " * Overall readability and organization of the notebook\n", " * Effort in making interesting observations where requested.\n", " * Conciseness. Points may be taken off if the notebook is overly \n", " " ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.5.5" } }, "nbformat": 4, "nbformat_minor": 1 }