This course provides a broad introduction to the exploration and management of large datasets being generated and used in the modern world. First, practical techniques used in exploratory data analysis and mining are introduced; topics include data preparation, visualization, statistics for understanding data, and grouping and prediction techniques. Second, approaches used to store, retrieve, and manage data in the real world are presented; topics include traditional database systems, query languages, and data integrity and quality. Case studies will examine issues in data capture, organization, storage, retrieval, visualization, and analysis in diverse settings such as urban crime, drug research, census data, social networking, and space exploration. Big data exploration and management projects, a term paper and a presentation are required. Sufficient background in database systems and statistics is recommended.

This course examines how database systems evolved to meet the workloads of modern applications. Limitations of relational databases led to NoSQL systems that are highly scalable and provide flexible data modeling but sacrifice important consistency properties. More recently, “NewSQL” data systems seek to understand and address fundamental scalability bottlenecks while maintaining relational database consistency. This course will describe shortcomings of relational databases for certain data management tasks and the specific challenges addressed by NoSQL and NewSQL database systems. Case studies will investigate both established and state-of-the-art systems. Students will critique and present existing work in the area and complete a research project individually or in teams that explores an outstanding problem in the area.