CS 540, Spring 2008: Assignment 3
Information Retrieval and Data Mining
Due Friday 4/25/08 at beginning of class
For this assignment, you are required to implement a data mining
algorithm that learns to filter spam from positive and negative
examples. As with previous assignments, you have some latitude in your
selection of language and algorithm. For language, if you wish to
choose a language other than C, C++, Java or Lisp, you must obtain
instructor's permission. For algorithm, you should use one of the
algorithms described in class or a different one (provided it isn't
covered in CS545) if you obtain instructor's permission.
Spam Examples
The key to instance learning is to have good features and lots of
examples. In this case, the examples are email messages. The messages
need to have been rated as spam or not spam. The features can be whatever
you want to extract from the email messages.
The baseline example set comes from a course taught by Noah Smith and
David Smith at Johns Hopkins University, which they in turn took from
a study published in:
I. Androutsopoulos et al., "An Evaluation of Naive Bayesian Anti-Spam Filtering". In Proceedings of the Workshop on Machine Learning in the New Information Age, 11th European Conference on Machine Learning, Barcelona, Spain, pp. 9-17, 2000.
The messages are taken from a
linguistics mailing list and are all straight ascii text. The format
is fairly straightforward: a subject line followed by a blank line
followed by some text. The messages have been pre-processed to
expedite parsing. The files containing the messages are available
at:
~cs540/spring08/hw3
on the department file system. There
are two kinds of files: those starting with sp are spam and those
starting with numbers are normal mail. To expedite copying, I have
also placed the original tar file in that directory, called lingspam.tar
Specifics
You should name your program spamfilter. Your program should
take as its first three arguments: training directory, testing
directory and output file. The program should use all the messages in
the training directory to learn what is spam and what is not. After
learning a model, it should read the messages in the testing
directory; for each message in the testing set, it should print in the
output file a line containing the filename, followed by three blank
spaces and then and either "SPAM" or
"MAIL".
A few notes:
- You may add other arguments if you have parameters you wish to
vary.
- The output should be ordered lexicographically as it appears
if you do an "ls".
- Don't count on the file naming convention to hold for your
testing files! I will probably re-name the testing files so that
your program cannot tell which is which...
As before, include a README file that describes what parameter
settings should be used as well as what compiler and settings you
use. Include a MAKEFILE if needed to run your program. In other words,
make this as easy as possible for the instructor to run the code.
In addition to producing your code, everyone needs to contribute to
the collection of examples. I think the examples in this set are
somewhat old at this point. So we will also look at the effect of time
on what is learned. Each student is required to submit 20 files of
spam and 20 files of mail, both in the same format as the current
set. To start it off, I have created a subdirectory in the one with
the examples called "new" and put 40 files taken from recent messages
in there. Name your files using the same convention as I did: spam files are
sp-{your initials}-{some number}.txt regular messages are {some number}-{your initials}.txt
Grading Criteria
Grading criteria will be similar to those used in the other
assignments. The last 10 points will be awarded based on how well your
solution performas relative to the rest of the class. I will use the
additional examples when I test your code.
Although it is
hard to imagine your solution requiring overly much computation time,
as before, each trial will be allotted up to 15 minutes. Your program
will be run on one of the machines in the HP classroom; make sure your
code runs on those machines.
Questions
- Describe your solution. How did you represent your features? What
data mining algorithm did you use? How did you decide on your features
and your solution?
- Try training your program on some of the original training data
and testing on my and maybe your new examples. What happened? How well
did the program do? Looking at some of the test cases, explain whether
you think your solution did a good job of generalizing to the new
examples.
- Try training your program with 50 percent of the data and test on
the other 50 percent. Do the same with 80 percent for training and 20
for testing. How does the performance differ? How accurate were the
results in each case? Why do you think you got the results you did?
Each answer is expected be 1/2-1 page in length.
What to hand in
You can submit everything electronically if you wish, but I would prefer to
receive your question answers in hardcopy.
- 20 files of spam messages and 20 files of mail messages
- Written answers to your questions in ASCII, PDF or PS.
- A file (tarfile or zip) containing the source code. You should submit this via email to howe@cs.colostate.edu by the due date/time for the assignment.
- A README file describing exactly how to compile and run your code and how
to set the parameters, if you have any. The README should include a
line that can be cut-and-pasted into the instructor's script for
running as batch.
- And if appropriate, a makefile for compiling your code and/or a
script file for running it.