Expression Lexing

Objectives
  • Introduce the topic of lexical analysis in a programming language such as Java.

  • Develop a robust lexer that is successful regardless of the whitespace.

    • Should be able to parse both "(6 * a) + (b / 4)" and "(6*a)+(b/4)".

Getting Started

Your directory should look like this:

W9L1/
└── src
    ├── Lexer.java
    └── TestCode.java
Description

Lexical analysis is the first phase of a compiler. It involves taking a series of words and breaking them down into tokens by removing whitespace and comments.

The Lexer has several different versions of a lexing method for identifying tokens within an expression.

  • The first method is called scannerLexer, and uses a Scanner object.

  • The second method is called splitLexer, which uses the method String.split().

  • The third method is call tokenizerLexer and is based on a StringTokenizer object.

Instructions

Use the javadoc to implement the methods is the Lexer class.

HINT: After a token is returned from each of the different lexing methods, call the String.trim() method to remove extra whitespace from the beginning and end of the string. If the token is empty, do not add it to the ArrayList.

Answer the questions at the bottom of Lexer.java

Regular Expressions

Use the following resources to learn more about Regular Expressions and to answer the attached questions.

regular_expressions.png
Submission

To receive credit for this recitation show your TA or helper that your program passes the TestCode and answers the questions in Lexer.java.