Due: 10/4 at 4pm.

In this assignment you will explore ridge regression applied to the task of predicting wine quality. You will use the wine quality dataset from the UCI machine learning repository, and compare accuracy obtained using ridge regression to the results from a recent publication recent publication (if you have trouble accessing that version of the paper, here's a link to a preprint.

You will also need to remove or comment out the first line in the dataset which gives the names of the features. The wine data is composed of two datasets - one for white wines, and one for reds. Perform all your analyses on both. The features for this dataset are not standardized, so make sure you do this, especially since we are going to consider the magnitude of the weight vector. You can use PyML for standardization:

>>> from PyML.preproc import preproc
>>> p = preproc.Standardizer()
>>> p.train(data)

Plot the RMSE and MAD of ridge regression on this dataset computed using 5-fold cross-validation over a range of the regularization parameter, lambda (choose values on a logarithmic scale, e.g. 0.01, 0.1, 1, 10, 100, 1000). When you use cross validation, provide the method a seed parameter, which will ensure that the same division in to folds is used each time, as in:

>>> results = classifier.cv(data, seed = 1)
# obtaining the average error:
>>> rmse = results.getRMSE()
>>> mad = results.getMAD()

In addition to RMSE and MAD, plot the Regression Error Characteristic (REC) curves of a representative classifier. REC curves are described in the following paper. What can you learn from this curve that you cannot learn from RMSE or MAD?

Compare the results that you are getting with the published results in the paper.

As we discussed in class, the magnitude of the weight vector can be interpreted as a measure of feature importance. Train a ridge regression classifier on a subset of the dataset that you reserve for training. We will explore the relationship between the magnitude of weight vector components and their relevance to the classification task in several ways. Each feature is associated with a component of the weight vector. It can also be associated with the correlation of that feature with the vector of labels. Create a scatter plot of the weight vector component against the Pearson correlation coefficient of a feature against the labels. What can you conclude from this plot? The paper ranks features according to their importance using a different approach. Compare your results with what they obtain.

Next, perform the following experiment: Incrementally remove the feature with the lowest absolute value of the weight vector and retrain the ridge regression classifier. Plot RMSE and MAD as a function of the number of features that remain on the test set which you have set aside.

Note: I have made a few minor refinements of PyML to help you in this assignment, so you will need the latest version (0.7.13.2). For example, it now handles ; as a delimiter for csv files, and computes the mean absolute deviation (MAD) for a regression object.

Submit your report via RamCT. Python code can be displayed in your report if it is succinct (not more than a page or two at the most) or submitted separately. The latex sample document shows how to display Python code in a latex document. Also, please check-in a text file named README that describes what you found most difficult in completing this assignment (or provide that as a comment on ramct).

Here is what the grade sheet will look like for this assignment. A few general guidelines for this and future assignments in the course:

• Always provide a description of the method you used to produce a given result in sufficient detail such that the reader can reproduce your results on the basis of the description. You can use a few lines of python code or pseudo-code. If your code is more than a few lines, you can include it as an appendix to your report. For example, for the first part of the assignment, provide the protocol you use to evaluate classifier accuracy.
• You can provide results in the form of tables, figures or text - whatever form is most appropriate for a given problem. There are no rules about how much space each answer should take. BUT we will take off points if we have to wade through a lot of redundant data.
• In any machine learning paper there is a discussion of the results. There is a similar expectation from your assignments that you reason about your results. For example, for the learning curve problem, what can you say on the basis of the observed learning curve?

Grading sheet for assignment 2

Part 1:  50 points.
(20 points):  Plots of MAD and RMSE as a function of lambda are generated correctly.
(20 points):  REC curves are correct
( 5 points):  discussion of REC curves
( 5 points):  Discussion of the MAD and RMSE plots and comparison of results to the published ones.

Part 2:  35 points.
(25 points):  Weight vector analysis
(10 points):  Comparison to the published results

Report structure, grammar and spelling:  15 points
( 5 points):  Heading and subheading structure easy to follow and
              clearly divides report into logical sections.
( 5 points):  Code, math, figure captions, and all other aspects of  
              report are well-written and formatted.
( 5 points):  Grammar, spelling, and punctuation.