Onegram

Assignment materials

Automated testing suite, solutions, and script for generating datasets from the official (much larger) Google dataset also available upon request by email.

Overview

The Google Ngram database provides ~3 terabytes of information about the frequencies of all observed words and phrases in English (or more precisely all observed kgrams). Google provides the Google Ngram Viewer on the web, allowing users to visualize the relative historical popularity of words and phrases.

In this assignment, students will build their own version of the Google Ngram Viewer for 1grams. They will then build three additional visualization tools to explore other aspects of the dataset that cannot be addressed using the Google Ngram Viewer.

Google n-grams viewer	Tool 1 built by students on this assignment: 1-gram history viewer

The three additional visualization tools include the distribution of letters in the English language, the distribution of all word frequencies in English, and the length of the average word in English since the 1800s. Examples of each are shown below (with a restricted dataset to avoid spoilers).

Tool 2: Distribution of letters using very small example file	Tool 3: Distribution of word frequencies of words starting with Q (Zipf's law)	Tool 4: Average length of words from 1860-1880 Plot for 1800-present not shown (to avoid interesting spoiler)

Assignment Tasks

Part 1

1. Simple Wordfile parser. Build a simple parameterized CSV parser to extract information about a single word from the CSV file and store the results in an associative array mapping year to word count.
2. Simple Totalfile parser. Build a simple CSV parser that reads in the total word counts for each calendar year and store the results in an associative array mapping word to total word count.
3. Tool 1: History visualization tool. Given a list of words, create a graph of normalized word frequencies for each word. This results in a tool much like the Google Ngram Viewer (but without the web interface).

   Part 2

4. Complex Wordfile parser. Build a CSV parser to read in the entire word count database stored in the Wordfile. Data will be stored in an associative array mapping word to an array of (year, count_in_year) pairs. If used in a Python course, this neatly utilizes most of our first class data structures (dictionaries, lists, and tuples).
5. Tool 2: Letter frequency tool. Using the word_data generated by the Wordfile parser, create a bar plot of English letter frequencies.
6. Tool 3: Total count vs. rank. Using the word_data sorted by decreasing order of word frequency, make a log-log plot with the count of each word on the y-axis, and the numerical ranking on the x-axis (i.e. the most common word in the English language would have rank 1, the next would have rank 2, and so forth). Optionally, include the ability to highlight words of interest (e.g. "questions" and "quest" on the plot showed in the summary above).

   Students will observe a roughly linear relationship on the plot. It is a surprising fact that the frequency of words and their rank should obey a power law. This is known as Zipf's law.

7. Tool 4: Average word lengths per year. Using the word_data generated by the Wordfile parser, make a plot of the average length of the English word in all books from 1800 to present. The results may (or may not) surprise you! This is a fairly tricky task, and two helper functions are suggested in the provided course materials).

Metadata