Practical Four - Html File Generator and COVID-19

Introduction

COVID-19 has changed our world, and as part of that change, Computer Science and various applications have shined. During isolation, everyone found the value of telecommuting, a concept which CS has used for industry for a while. At CSU, research around COVID-19 has continued throughout the entire pandemic, including the Computer Science department, who has research both in looking at the health data, and the Cyber Security Center is looking at security around data integrity and news. When the pandemic first started, numerous models came out, and the amount of data being collected and shared for all to research. John Hopkins, New York Times, CDC, and The World Health Organization are just a few of many examples of data visualizations of the pandemic.

For this assignment, you will be building an HTML Table generator based on the COVID-19 data provided by WHO. This is fairly common to do in industry, and many HTML pages are actually generated server-side using templates and data. Even this website uses a static site generator called Jekyll, so we can have templates for building the pages. You will only be implementing a single tag, but there is a lot to do for that one special customized tag. The customized tag will be a WHO Table, which connects the data from WHO to the web-page you are making.

What you will learn

Loops
File Reading and Writing
String parsing

Provided Code

As with the previous two practical assignments, you will want to go through canvas to download the code, and add the files to the project. However, the one difference for this assignment, is that you will be adding some files from scratch, so there won’t be templates except for one file. Overall, we are providing the “data layer” for the assignment, and you are building the HTML generator or view layer of the assignment.

Suggested File Structure

After you download the files, we recommend you store the files as follows (from the project root)

src
All .java files go in your src folder, including ones you will create.
tmpl
Storing the .tmpl files in this directory, will make it easier to edit them.
data
You should store th who-data.csv file here.
examples
You should store the .html files here. You will actually be writing .html to the root path, but by putting these files in a separate directory, you won’t accidentally overwrite them. You will want to copy the style.css file twice - into this directory, and your root directory. You can open them in your web-browser, to see what you will be working towards.
You can also see the examples of what you will be building here:

At this point, you should take the Code Tracing Quiz in canvas.

You will also have some questions about HTML to make your life easier for coding. If you haven’t learned HTML, you can go to W3Schools in particular, you will need to learn about the table tag. To save time, don’t learn about all of the HTML world now! You probably don’t need to spend five minutes on it.

We will be making use of the following tags:

<table>
<caption>Table caption</caption>
<thead>
 <tr>
  <th>Table head</th>
 </tr>
</thead>
<tbody>
 <tr> 
  <td>Table element</td>
 </tr>
</tbody>
</table>

Additionally

<h1></h1>
<h2></h2>
<h3></h3>
<h4></h4>

Step 0 - Build Empty Classes

Before you can compile the project as a whole, you’ll need to create a few new classes from scratch. Create the following classes within your src directory. You will fill them out in later steps.

WhoTable.java
Query.java
HtmlFormat.java

Step 1 - HtmlFormat.java

Your first step is to write the HtmlFormat.java class. In Eclipse, you will want to create a new class, stored in the src directory. The classes name should be HtmlFormat with the matching file name being HtmlFormat.java.

You will implement the following methods in the class. They are all public, static, and all return a String.

header(String str, int size)

This method uses String.format to build a heading tag based on the size passed in. For example:

1
return String.format("<h%d>%s</h%d>%n", size, str, size);

Once you have this method written, you will implement the following methods

h1(String str)
Returns an <h1> tag with the value of str between. hint: return header(str, 1);
h2(String str)
returns an <h2> tag with value of str between.
h3(String str)
returns an <h3> tag with value of str between.
h4(String str)
returns an <h4> tag with value of str between.

Testing header and h1-h4

You will notice there are no Unit Tests written. You will want to write your own at this point to make sure the proper HTML is being outputted when the methods are called.

To run the unit tests, run the program with the --tests program argument similar to what you did in practical 3.

table commands

Now you will write the heart of HtmlFormat - the table methods. You will implement the following methods:

table(String str, String caption)
Returns a String in the format of

    <table>
      <caption>caption value</caption>
      str value (it will assume that contains properly formatted tags)
    </table>

thead(String str)
Returns a String in the format of

    <thead>String value</thead>

tbody(String str)
Same as thead, but using the tbody tag.
th(String str)
Same as tbody, but using the th tag.
td(String str) Same as th, but using the td tag.
tr(String str, String style)
Returns a String of format. Don’t forget to escape your quoted values.

    <tr class="style value">Str value</tr>

To make reading the file easier, you can add tab characters \t\t\t<th>str</th>%n. You will notice there is no space between your String value and the tag. That does matter. Any pre and post spacing (the tabs) won’t matter, and it makes looking at your output easier.

Testing table methods

You will really want to write unit tests to make sure you can build a properly formatted table. Make sure to have multiple columns, so you have to think about the order you have to call the methods in to get what you want (hint: table(String, String) is the last method you call, after you build the inside). Making sure you do this will not only help you debug your methods, but also give you an advantage on a later part of this assignment!

Example:

1
2
3
4
5
6
7
8
9
10
11
12
System.out.println(HtmlFormat.table(
                   HtmlFormat.thead(
                    HtmlFormat.tr(
                     HtmlFormat.th("Col Header") +
                     HtmlFormat.td("Col head2") +
                     HtmlFormat.td("Col head3"), "heads"   
                    )) +
                   HtmlFormat.tbody(
                    HtmlFormat.tr(
                      HtmlFormat.th("Row Header") +
                      HtmlFormat.td("Cool") +
                      HtmlFormat.td("Runnings"), "odd")), "My caption"));

You could also set all the ‘inner sections’ to variables for example:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
String tableHead = HtmlFormat.thead(HtmlFormat.tr(
                                     HtmlFormat.th("Col Header") +
                                     HtmlFormat.td("Col head2") +
                                     HtmlFormat.td("Col head3"), ""));
                                     
String row1 =  HtmlFormat.tr(HtmlFormat.th("Row Header") +
                             HtmlFormat.td("Cool") +
                             HtmlFormat.td("Runnings"), "odd");
                             
String row2 =  HtmlFormat.tr(HtmlFormat.th("Row2 header") +
                             HtmlFormat.td("Lost") +
                             HtmlFormat.td("In Space"), "even");

String tableBody  = HtmlFormat.tbody(row1+row2);

String table = HtmlFormat.table(tableHead+tableBody, "This is a cool table");
System.out.println(table);

Should output (depending on your spacing and line returns-which are optional):

<table>
	<caption>This is a cool table</caption>
	<thead>
		<tr class="">
			<th>Col Header</th>
			<td>Col head2</td>
			<td>Col head3</td>
		</tr>
	</thead>
	<tbody>
		<tr class="odd">
			<th>Row Header</th>
			<td>Cool</td>
			<td>Runnings</td>
		</tr>
		<tr class="even">
			<th>Row2 header</th>
			<td>Lost</td>
			<td>In Space</td>
		</tr>
	</tbody>
</table>

And if you are curious, this is what the table would look like in html

This is a cool table
Col Header	Col head2	Col head3
Row Header	Cool	Runnings
Row2 header	Lost	In Space

Pro Tip
You will notice in this assignment, most every method takes in something, and tries to return something. This is called functional programming. It actually allows debugging to be much easier. Imagine if you had to keep track of where you were at in the table trying to build it from the top down. Instead, we build it from the inside out, based on sections, and everything is done without knowledge of the other.

Step 2: Query.java

You have completed HtmlFormat.java, so lets move onto the next class you will create, Query.java. As before, you will want to create a new file called Query.java and with the class name Query (always true for java, file names must match the class names). Query creates an immutable class (meaning once the data is created, it can’t be changed / mutated), that will be used to query the dataset, and also determine what you need to do with that data.

The methods you will implement are as follows:

public String getRegion()
public String getCountry()
public boolean isCumulative()
public Query(String[] tokens)

You will notice they are only accessors and the constructor. When we wrote it, we also added a private method to help the constructor, but that is optional / how you choose to break up your code.

getRegion(), getCountry(), isCumulative()

These three accessor methods are direct and return the values stored in private instance variables, that we called region, country, cumulative. All three of the variables have a default value.

region and country, are Strings and default to “ALL”
cumulative, is a boolean and defaults to true

Query(String[] tokens)

The constructor of Query does all the work, but we did write a helper method to simplify our code (called parse). What query does is go through each (hint: enhanced for loop), value in tokens. You then parse that value. What does parsing entail?

Does the value contain a : ? if it does, continue, if not, skip. Only values split by the colon character are valid.
Splitting on the :
The first part is the variable to modify, the second part is the value.
- country:US
  would take US and store it in the private instance variable country. (US could be anything)
- region:AMRO would take AMRO and store it in the private instance region variable. (AMRO could be anything)
- cumulative:false would take the string value, and convert it to a boolean, and store the answer in cumulative. (could be true or false)

You do not need to error check the second half of the statement, except in the case of Boolean (good use of try-catch, and also toLowerCase(). As such, the overall parse method is a series of if/else if checks, that then have a line to set values. Anything not in that list, you can ignore.

As a reminder, to convert to boolean, the code is as follows:

1
Boolean.parseBoolean(value.toLowerCase());  // depending on where the value is stored

Testing Query

Testing Query.java is extremely important, and if done correctly, will help you understand how it works with the provided code in addition to make sure it is working. You should go to UnitTests.java and write your own tests. One hint, declare a number of String arrays. For example:

1
2
3
4
5
6
7
8
String[] test1 = {"country:US"};
String[] test2 = {"region:AMRO"};
String[] test3 = {"country:US,JP,IT", "cumulative:false"};
String[] test3 = {"should", "skip", "these", "country:US,JP,IT", "cumulative:false"};
Query q1 = new Query(test1);
Query q2 = new Query(test2);
Query q3 = new Query(test3);
Query q4 = new Query(test4);

How would you check your values are getting set properly? Use your accessors and print the results! You can look at who-data.csv to see various options for regions and countries.

Testing Query with WhoData

Often in the testing stage, you want to test your object with other objects you didn’t necessarily write, so you can better understand what they are doing. It is also essential for the next step, so before you move on, you will want to get a better idea of how WhoData works.

To construct a WhoData object, it needs the file to read. The if you stored the file in the data directory, it would look like the following:

1
2
3
4
5
WhoData data = new WhoData(new CSVReader("data/who-data.csv"));  
WhoDataItem[] items = data.queryData(query);
for(WhoDataItem item : items) {
   System.out.println(item);
}

You should try using different queries. Also, you could reduce the lines to be the following:

1
2
3
4
WhoData data = new WhoData(new CSVReader("data/who-data.csv"));  
for(WhoDataItem item : data.queryData(query)) {
   System.out.println(item);
}

It is important you understand how query is working. A couple things to notice, everything is always sorted by country, so if your country changes, you may want to print a line between! This would also be good to try to see if you can do that, and will help later.

After you complete this, you may want to submit for grading, using one of your submission attempts.

Pro Tip
The second way putting the call in the loop itself is actually the better way to write it. That way someone could change the type of underlining structure, and as long as you can iterate through it, you don’t care what it is.

Step 3 - Writing WhoTable.java

The WhoTable class follows the steps needed to build a specialized tag in the Template engine. For example, in the .tmpl files, you will see:

{{who-table region:EURO cumulative:true}}
{{who-table country:ALL region:ALL cumulative:true}}
{{who-table country:US,IT,JP cumulative:false}}

The format of the commands should look familiar based on the Query.java class you just finished! Your ultimate goal is to take in a String[] array, and a WhoData object, and build series of HTML tables to return out of the class as a String.

public WhoTable(WhoData data, String[] queryData)

When creating the class, you will want to build the following constructor. The class needs two instance variables to work:

final WhoData data;
final Query query;

The constructor sets the values of those two variables. The data is passed in from outside the object, so you can set that directly, this.data = data;

The Query object needs to be constructed, but as you can see the String[] queryData array matches the constructor for Query, so that can be passed in directly. Making your constructor look like:

1
2
3
4
public WhoTable(WhoData data ,String[] queryData) {
   this.data = data;
   this.query = new Query(queryData);
}

Step 4 - WhoTable Helper Methods

While the primary method in WhoTable is .render(), there are a number of helper methods we found essential to make sure we are keeping it simple and divide, conquering, and gluing our problem! While these methods are often private, we are going to ask you to specify them as package protected (no scope/default scope), so we can grade them. You are also free to add additional methods beyond these ones.

headerRow() and closeTable()

You can copy the following bit of code directly into WhoTable.java. This is to give you an idea of what you need to do for the other methods.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
  String headerRow() {
    return HtmlFormat.thead(
        HtmlFormat.tr(
            HtmlFormat.th("Country") +
                HtmlFormat.th("Cases") +
                HtmlFormat.th("Deaths"), ""));
  }

  String closeTable(String str, WhoDataItem last) {
    String caption = "";
    if(last != null && !this.query.getRegion().equals("ALL")) {
      caption = String.format("Cases and Deaths by Country <br/>Region: %s", last.getWhoRegion());
    } else if (last != null) {
      caption = "Cases and Deaths by Country";
    }
    return HtmlFormat.table(headerRow() +
        HtmlFormat.tbody(str), caption);
  }

void setTotals(Query query, WhoDataItem data, int[] totals)

Set setTotals uses the fact that Arrays are objects to modify the totals array. It uses both the Query object and the WhoDataItem to modify totals. Most notably, query.isCumulative() and data.getNewCases(), data.getNewDeaths(). The totals array is only two deep, and tracks cases in the 0 index, and deaths in the 1 index. However, the total changes based on the query.

In all the templates provided and in all the tests, query.isCumulative() will always be true. Therefore, separating the functionality based on the boolean is optional and can be viewed as a challenge. Otherwise, you can assume query.isCumualtive() is always true and shorten this method accordingly. An outline for using query.isCumulative() is provided below.

If the query is cumulative, then you add data.getNewCases() and data.getNewDeaths() to the current value in totals.

If the query is not cumulative, then you just reset the value in totals with the values in getNewCases() and getNewDeaths()

As always, you should test this method. You can construct both a Query and WhoDataItem directly without using the data from the CSV file.

String buildRow(WhoDataItem data, int[] totals, boolean isOdd)

The buidRow method takes in the totals array, the data item and whether or not the row is an odd row in the table or not. The output of the method would be as follows:

<tr class="odd">  <!-- could also be even -->
  <th>(Country)</th>   <!-- data.getCountry().toString() -->
  <td>(Cases)</td> <!-- String.valueOf(totals[0])  will help -->
  <td>(Deaths)</td> <!-- String.valueOf(totals[1])  will help -->
</tr>

Needless to say, you are going to make use of the following methods:

HtmlFormat.tr
HtmlFormat.th
HtmlFormat.td

For the ‘style’ you will use the isOdd parameter to set the style to “odd” if isOdd is true, or “even” if isOdd is false.

Testing

Remember, keep it simple. These methods only uses the parameters passed into them! That is it. You can test these methods in RunTests before you move on and should!

Step 5 - WhoTable : render()

It will help to write out what you want to do in .render() before you do it. It is an easy method to get confused if you overthink it.

What you are doing in render is building a very large String, and returning that String. What that String contains is the html code for a row for every country in the query! For example if the query is country:US,IT,JP, it would contain three separate rows, all stored as in the String. If the query is country:ALL, it would create a row for every country!

The problem is you never know exactly how many rows you will need to generate, but you do know every time the country changes, you can add a row as the entries are always in order of coutries.

This means the algorithm was as follows for us:

Declare and initialize a return (often called rtn) String
Have a rowCounter set to 0, so you can keep track of even odd rows
Create an int[] array of two spots (we called our totals)
Set WhoDataItem last to null (You will quickly change this)
Set WhoDataItem current to null (Also will be quickly changed)
Query the data with the query (both stored in instance variables), and start looping through returned results (think about the tests you wrote!)
Check to see if the country is different from last
- if different:
  - call the buildRow() method on the last item add the result to the rtn String
  - reset totals
- otherwise, do nothing different from a regular pass through the loop
Each time through the loop:
- Update totals using setTotals
- Set last to equal current
After my loop, call the buildRow() method on the current item add the result to the rtn String
Return the completed table using the closeTable() method on your rtn String

Testing WhoTable.java

Now is the time you can test it. Query the data, and print out the results from calling .render()! Make sure this is working as expected.

After you complete this, you may want to submit for grading, and use one of your submission attempts, as this is now completes 90% of the practical. The last 10% is the part that glues it all together in this divide-conquer-glue.

Step 6 - Templater.java

The Templater.java focuses on reading the template provided, and outputting valid HTML. It involves both reading in the template file, and writing out the contents of a new file. There are only two methods you have to implement, but we would suggest thinking about how you can break the problem up into smaller methods (we used two private helper methods).

The two required methods are:

public Templater(WhoData data)
Your constructor. It takes WhoData data, and sets it to an instance variable.
public void write(String templateFile)
You will notice in your template, it has “throw new NotImplementedException();”. You must remove this line to avoid an error. It is common in industry to use that exception, as it makes it obvious if you call a method that isn’t written yet. Also, notice the import statement at the top of the class: “import sun.reflect.generics.reflectiveObjects.NotImplementedException; “. This import statement must also be deleted before submitting to ZyBooks.

The heart of the work is in write. A process is as follows:

While reading in from your template, either:
- write the line directly to your outfile (PrintWriter)
- if the line matches the format below:
- use WhoTable to generate HTML table code, and then write the entire result of the WhoTable render method to the your outfile. - make sure to remember to close the files when done.

How do you know when to build a WhoTable as compared to just write the line directly? We have designed a specialized “tag” that your program is looking for.

who-table tag

The format for the who-table tag is:

{{who-table  query:arg  query:arg }} 

Example:
{{who-table region:EURO cumulative:true}}

You can also open one of the .tmpl files to see it in action.

Valid Assumption/Pre-condition
While not common in templating languages, for this class you can assume the tag is the only thing on that line in the template file. As such, you can use the following code (depending a bit on how you set it up)

1
2
3
4
5
6
7
8
9
10
11
12
Scanner fileScanner = new Scanner(new File(templateFile));
// loop
// inside the loop
   String line = fileScanner.getNextLine().trim();
  
   if(line.startsWith("{{")) {
      // parse the tokens by grabbing the substring between {{ and }} (Split)
      int first = line.indexOf("{{")+2;
      String[] tokens = line.substring(first, line.indexOf("}}", first)).trim().split("\\s+");
      // construct a WhoTable, set value of render to lineToWrite
   } 
   //

This may change slightly based on your helper methods. The \s+ is something you should use as it causes the String to split on any white space (tabs or spaces).

Pro Tip
While not learned until a future class \s+ is a regular expression. Regular expressions are powerful ways in which to parse Strings, look for matches in Strings and more! While we don’t suggest you look at them for this class, if you are comfortable with the content, here is a tutorial for later use about them.

Testing Templater.java

You will want to test as you develop, including using ample print statements like Logger.debug. As a reminder, your program args options are as follows:

usage: Main [-v|--verbose] [-d|--debug] [--tests] dataFile templateFiles...
	data file is required to run the program unless running tests
	any file after dataFile that isn't a named argument, will be a template to be processed
	-v|--verbose to enable verbose printing, add -v or --verbose
	-d|--debug to enable debug printing, add -d or --debug
	--tests to enable unit testing
Examples:
	java Main data/who-data.csv template.tmpl
	java Main --tests

At this point, you can also modify your program arguments to read-in the datafile. For example:

--tests -d data/who-data.csv tmpl/test.tmpl
or
--tests -d data/who-data.csv tmpl/test.tmpl tmpl/us.tmpl tmpl/euro.tmpl tmpl/us-it-jp.tmpl tmpl/regions.tmpl

Pro Tip
The {{ tag }} syntax is known as mustache templating. Yes, think of the emote face :{). There are many types of templating markups out there, and the markup used for this website is Liquid, which is based on mustache.

Running the Program

At this point, you should have a fully functional program. As long as your style.css file is in the same directory as your .html files, you should have some nicely formatted web pages. If it isn’t in the same directory, the formatting may look odd, but the content is there. If you open them in Eclipse, it will show just the text (markup) behind the hyper text markup language. However, if you open those files in your web browser, you will see them as web pages.

Here are some examples of running the program without the tests

data/who-data.csv tmpl/test.tmpl
or
data/who-data.csv tmpl/test.tmpl tmpl/us.tmpl tmpl/euro.tmpl tmpl/us-it-jp.tmpl tmpl/regions.tmpl

You can view each of the files below to view how they should look:

Step 7 - Turning in and Reflection

You should now turn in your code, and write the reflection in Canvas. What did you learn? What could you improve upon in the future? How did designing a class from scratch go? The next practical will involve you writing all th code from scratch, but you do have all these classes to help you! Are you ready?

Did working with HTML spark interest in designing your own website? As someone with a course in this department, you already have a web server setup. You can learn about this here. Though many students also setup their git web page as part of their git account.

Going Further
You explored two major areas in computer science with this practical. Data Visualization / Human Centered Computing and Big Data. While admittedly, the COVID-19 data we provided is not big data, the amount of information being generated by COVID-19, even to break it down to that file is big data. Going beyond this course, we have two concentrations in the department you may want to look at. If you enjoyed the idea of how to display and interact with the data, HCC would be something good to look with a focus on stats. If you really like getting and analyzing the data, Systems is important in how you do that. You can learn about the research labs at CSU, and many undergrads are encouraged to get involved in research.