Algorithm Efficiency

Chapter 10 of Prichard and Carrano.

• measuring the time efficiency of algorithms
• approximation with Big-O notation
• examples: sorting algorithms

So far we have thought about

• time to develop, test, debug and modify code.

Now we will consider

• running time of an algorithm.

We won't focus on the speed of particular implementations.

• Anyone can make an implementation run a little faster.
• But, can get amazing gains in speed with a different algorithm!

Should consider space, or memory, efficiency, too. But we won't.

Could write Java implementations of different algorithms and time their execution.

Bad idea, because relative execution times are affected by

• differences in programming styles,
• the choice of computer,
• data used.

Instead, we will count the number of “significant operations” in an algorithm.

Ignore the actual time each operation takes.

Node node = head;
int i = 0;
while (i < n) {
   node = node.next;
   ++i;
}
System.out.println(node.item);

How long does this take to find element $n$?

Node node = head;
int i = 0;
while (i < n) {
   node = node.next;
   ++i;
}
System.out.println(node.item);

How long does this take to find element index?

Well….it depends. Good answer! On what?

• value of $n$
• time to run node = node.next, $x$
• time to compare i < index, $c$
• time to run ++i, $i$

Node node = head;
int i = 0;
while (i < n) {
   node = node.next;
   ++i;
}
System.out.println(node.item);

• value of $n$
• time to run node = node.next, $x$
• time to compare i < index, $c$
• time to run ++i, $i$

So, total time is $n\;x\;c\;i$.

Can reduce total time by trying to speed up each operation.

But, time will always grow as $n$ grows. Can't be avoided!

Node node = head;
int i = 0;
while (i < n) {
   node = node.next;
   ++i;
}
System.out.println(node.item);

But, time will always grow as $n$ grows. Can't be avoided!

Unless, we come up with a new algorithm.

Unless, we come up with a new algorithm, and data structure!

System.out.println(dataArray[n]);

Time no longer depends on $n$!

Which approach would you use if you often need to print the 1,000,000$^{th}$ element?

double x = 1;
for (int i = 0; i < n; i++)
  for (int j = 0; j < i; j++)
    for (int k = 0; k < 5; k++)
      x = x * 2.0;

If x = x * 2.0 takes $m$ time, what is total time for all iterations?

double x = 1;
for (int i = 0; i < n; i++)
  for (int j = 0; j < i; j++)
    for (int k = 0; k < 5; k++)
      x = x * 2.0;

If x = x * 2.0 takes $m$ time, what is total time for all iterations?

Loop for k takes $5\; m$ time.

Loop for j takes $5\; m\; i$.

Now, including the outer loop, the total time is $$5\;m\;(1 + 2 + 3 + \cdots + n)$$

Now, including the outer loop, the total time is $$5\;m\;(1 + 2 + 3 + \cdots + n)$$

Can you simplify this?

Now, including the outer loop, the total time is $$5\;m\;(1 + 2 + 3 + \cdots + n)$$

Can you simplify this?

Sure! $$5\;m\;(1 + 2 + 3 + \cdots + n) = 5 \; m \; n (n+1) / 2$$

$$5\;m\;(1 + 2 + 3 + \cdots + n) = 5 \; m \; n (n+1) / 2$$

But what are the units of this calculation?

• seconds
• milliseconds

Will depend on the computer.

Want to ignore this.

$$5\;m\;(1 + 2 + 3 + \cdots + n) = 5 \; m \; n (n+1) / 2$$

But what are the units of this calculation?

• seconds
• milliseconds

Want to ignore this.

Instead, just focus on how fast the value grows with the size of the problem.

Instead, just focus on how fast the value grows with the size of the problem.

double x = 1;
for (int i = 0; i < n; i++)
  for (int j = 0; j < i; j++)
    for (int k = 0; k < 5; k++)
      x = x * 2.0;

For this code, the size of the problem is $n$.

Our formula was $5 \; m \; n (n+1) / 2$. Ignoring all but $n$ leaves $$n (n+1) = n^2 + n$$

Our formula was $5 \; m \; n (n+1) / 2$. Ignoring all but $n$ leaves $$n (n+1) = n^2 + n$$

The expression $5\;m\;(n^2 + n)$ increases with problem size by a factor of $n^2+n$.

Denoted in Big-O notation by $O(n^2+n)$, or, it is of order $n^2+n$.

Relative rates of growth: $$O(1) < O(\log_2 n) < O(n) < O(n\log_2 n) < O(n^2) < O(n^3) < O(2^n)$$

We really are only interested in efficiency for time-consuming problems, ones with large values of $n$.

When $n$ is large, $n^2$ is much bigger than $n$.

So, let's ignore lower-order terms.

$$O(1) < O(\log_2 n) < O(n) < O(n\log_2 n) < O(n^2) < O(n^3) < O(2^n)$$

$$O(1) < O(\log_2 n) < O(n) < O(n\log_2 n) < O(n^2) < O(n^3) < O(2^n)$$

$$O(1) < O(\log_2 n) < O(n) < O(n\log_2 n) < O(n^2) < O(n^3) < O(2^n)$$

Back to searching for the $n^{th}$ element in a list.

If the list has 1,000,000 elements, the worst case is that it will take 1,000,000 operations to find it.

How long will it take to find the

• first element?
• the 500,000 element?

If all values of $n$ are equally likely, the average case performance will be 500,000.

If we now let $n$ be the size of the list, the

• best case efficiency is $1$
• average case efficiency is $n/2$
• worst case efficiency is $n$

Worst case is usually easier to calculate, and maybe most relevant.

Big-O analysis applies to large problems.

If your application is small, don't worry about Big-O. Use the algorithm that

• is easy to understand
• is easy to implement and debug
• does not use too much memory.

But, if you want to get a raise for that big insurance company, or get a job at that big search engine company,

• know how to analyze the efficiency of an algorithm,
• practice picking or designing more efficient algorithms for large problems.