Asymptotics I Study Guide
Author: Josh Hug


Runtime Minimization. One of the most important properties of a program is the time it takes to execute. One goal as a programmer is to minimize the time (in seconds) that a program takes to complete.

Runtime Measurement. Some natural techniques:

Algorithm Scaling. While the thing that we actually care about is the runtime, often we'll compare two different algorithms based on how they scale. For example, if the runtime R(N) grows quadratically as a function of the input size vs. linearly, then the linear algorithm will be much faster for large N. In practice, this difference in large N performance is the dominant factor (though this is not an obvious), and in almost all cases we'd prefer the linear algorithm.

Simplfying Algebraic Runtime. We utilize several simplifications to make runtime analysis simpler:

As an example, if we have an algorithm that performs 2N + 1 increment operations and 4N^2 + 2N + 6 compares, we'd intutively just saw this algorithm has a runtime proportional to N^2.

Big Theta. To formalize our notation, we introduce Big-Theta notation. We say that a function R(N) belongs to BigTheta(f(N)) if there exists positive constants k1 and k2 such that k1f1(N) <= R(N) <= k2*f2(N). An alternate definition is that R(N) belongs to BigTheta(f(N)) iff the limit of R(N)/f(N) approaches some constant k > 0 as N -> infinity. We did not use this definition in class.

When using BigTheta to capture a function's asymptotic scaling, we avoid unnecessary terms in our BigTheta expression. For example, while 4N^2 + 3N + 6 is BigTheta(4N^2 + 3N), we will usually make the simpler claim that it is BigTheta(N^2).

Order of Growth.* If a function R(N) is BigTheta(f(N)), we saw the the order of growth is f(N). For example 4N^2 + 3N + 6 is BigTheta(N^2), so we say its order of growth is N^2.

Big O. Big O notation is similar to BigTheta. However, instead of bounding from below and above, big O only bounds from above. In other words R(N) belongs to O(f(N)) if there exists a positive constant k2 such that R(N) <= k2*f2(N). Or in terms of limits, R(N) belongs to O(f(N)) if the limit of R(N)/f(N) converges to some constant 0 <= k <= 1 as N -> infinity.

Best Case vs. Always vs. Worst Case. One particularly vexing topic is what happens when a piece of code has a runtime that depends on the nature of its input. For example, consider a containsZero function that checks an array to see if it has zeros. The runtime of this algorithm for arrays seems like it should just be BigTheta(N), eaning that the runtime should grow linearly with the array size.

However, this isn't quite correct. What we've said is only true in the worst case. If we instead were to feed increasingly large arrays that are all zero, then the runtime does not grow with the input size, and we'd say it is BigTheta(1).

Here, the problem isn't Big Theta, but just imprecise problem definitions. For most circumstances, we'll simply ask for the worst case runtime. See lecture 20 for more.

Big Theta in the Worst Case vs. Big O. In the real world, it is very common for people to say Big O when they really mean Big Theta in the Worst Case. For example, the containsZero function above is Theta(N) in the worst case, but most authors would say it is simply O(N) (with no explicit mention of the phrase 'worst case').

While both statements are correct, Theta(N) in the worst case is a slightly stronger statement. The difference is identical to the one between the English sentences "Every member of my family is less than or equal to 100 years old" vs. "The oldest member of my family is 100 years old." In a family where the oldest person is 100 years old, both statements are true, but the former statement is a stronger statement.

This is not particular important, but since we're students at arguably the best CS program in the country, it seems like a thing worth mentioning given the ubiquity of big O notation.

See lec20 instead.