CSE6242 / CX4242, Fall 2017
Data and Visual Analytics
Georgia Tech, College of Computing
4:35 - 5:45pm, Clough 152, Tue & ThuProf. Duen Horng (Polo) Chau
This course will introduce you to broad classes of techniques and tools for analyzing and visualizing data at scale.
It emphasizes on how to complement computation and visualization to perform effective analysis.
We will cover methods from each side, and hybrid ones that combine the best of both worlds.
Students will work in small teams to complete a project exploring novel approaches for interactive data & visual analytics.
Office Hours
 |
Polo Chau | Tue, 3:30PM-4:00PM + FREE after-class coffee, at Clough Starbucks |
Klaus 1324 |
 |
Kiran Sudhir Head TA |
Thu, 3:05PM-4:05PM | Klaus- Open area next to Polo’s office |
 |
Varun Bezzam | Wed, 12:30PM-1:30PM | Klaus- Open area next to Polo’s office |
 |
Yuyu Zhang | Wed, 12:30PM-1:30PM | Klaus- Open area next to Polo’s office |
 |
Akanksha Bindal | Mon, 11:00AM-12:00PM | Klaus- Open area next to Polo’s office |
 |
Vishal Bhatnagar | Mon, 11:00AM-12:00PM | Klaus- Open area next to Polo’s office |
 |
Vivek Iyer | Thu, 3:05PM-4:05PM | Klaus- Open area next to Polo’s office |
Announcements and Discussion
We use Piazza for announcements and discussion.Everyone must join this class's Piazza, at https://piazza.com/gatech/fall2017/cse6242aqcx4242a/.
Double check that you are joining the right Piazza!When you have questions about class, homework, project, etc., post your questions there. Our teaching staff and your fellow classmates will help answer them quickly. You can also use Piazza to find project teammates.
T-square will only be used for submission of assignments and projects.
While we welcome everyone to share their experiences in tackling issues and helping each other out, but please do not post your answers, as that may affect the learning experience of your fellow classmates.
Class Schedule and Lecture Slides
| Wk | Dates | Topics | Tue | Thu | Events (eastern time; EST) | |
|---|---|---|---|---|---|---|
| 1 | Aug | 22, 24 | * Course introduction * Big data analytics building blocks |
intro | building blocks, data collection | |
| 2 | 29, 31 |
* Data Collection, and simple storage (SQLite) * Data cleaning * Class Project overview; Heilmeier questions * Data Cleaning: Starting with the End in Mind (VantagePoint) by Prof. Alan Porter, Denise Chiavetta, and Stephen Carley |
SQLite, Data cleaning | VantagePoint lecture, project overview | HW1 out | |
| 3 | Sept | 5, 7 |
* Example projects:
(1) Firebird: Predicting Fire Risks in Atlanta, by Shang-Tse Chen (2) PASSAGE: A Travel Safety Assistant, by Nilaksh Das * GT Github; one drive * Data integration: knowledge graph; data reconciliation/de-duplication; similarity functions |
Firebird, PASSAGE | Github, data integration, analytics concepts | |
| 4 | 12, 14 |
* Visualization 101 |
Irma | vis101 | HW1 due (Fri, 9/15,11:55pm) | |
| 5 | 19, 21 |
* Data visualization for the web (D3) * Fixing common visualization issues (* Fixing presentation issues) |
vis fix | D3 | Form project teams by Fri, 9/22; HW2 out |
|
| 6 | 26, 28 |
* Scaling up: Hadoop, Pig, Hive * Data analytics concepts & tasks * Overview of project proposal and presentation |
Hadoop, Pig, Hive | analytics tasks, project proposal and presentation | ||
| 7 | Oct | 3, 5 |
* Scaling up: Spark, Spark SQL * Scaling up: HBase |
spark | hbase | |
| 8 | 10, 12 |
* Classification key concepts, k-NN, cross validation * Clustering: k-means, hierarchical clustering, DBSCAN |
X (student recess) |
classification, clustering | HW2 due (Wed, 10/11, 11:55pm); HW3 out |
|
| 9 | 17, 19 | Project proposal presentations | Show time! | Show time! | Project proposal & slides due (Mon, 10/16, 11:55pm) | |
| 10 | 24, 26 |
* Classification: decision tree, vis (ROC, AUC, confusion matrix) * Clustering vis * Ensemble method, bagging, random forests * Graph analytics
|
classification vis, clustering vis, ensemble, random forests | graph basics | ||
| 11 | Nov | 31, 2 |
* Graph analytics
|
graph centrality & algorithms | mmap | HW3 due (Fri, 11/3, 11:55pm) HW4 out |
| 12 | 7, 9 |
* Text analytics: concepts * Text analytics: algorithms (LSI=SVD) |
text analytics | cont'd | Proj progress report due (Fri, 11/10, 11:55pm) | |
| 13 | 14, 16 |
* Time series: algorithms, visualization, & applications |
time series: basics, linear forecast | time series: non-linear forecast, vis | ||
| 14 | 21, 23 | Thanksgiving | X | X | ||
| 15 | 28, 30 |
* Publication-quality figures * Project poster presentations |
pub-quality figures, lessons learned | Poster presentation. 4:30pm to 6pm-ish. Klaus Atrium. Pizza + drinks served! | HW4 due (Fri, 12/1, 11:55pm) | |
| 16 | Dec | 5 | * Closing words and course review |
lessons learned | X | Proj final report due (Tue, 12/5, 11:55pm) |
Homework (50% of grade)
The fastest way to get help with homework assignments is to post your questions on Piazza.
If you prefer that your question addresses to only our TAs and the instructor, you can use the private post feature (i.e., check the "Individual Students(s) / Instructors(s)" radio box).
While collaboration is allowed for homework assignments, each student must write up their own answers.
All GT students must observe the honor code.
Any suspected plagiarism and academic misconduct will be reported and directly handled by the Office of Student Integrity (OSI).
We have 4 assignments in total.
Project (50% of grade)
See project description. See the schedule table above for deliverable due dates.Late Submissions Policy
- Homework: each student has 4 slip days total. No questions asked.
- Project: each team has 3 slip days total. No questions asked. Slip days may not be used on in-class activities (e.g., proposal presentation, poster presentation, etc.).
- To use slip days, specify the number of days you have used in the textbox on T-Square (when you submit your work).
- Each slip day equals 24 hours. E.g., if a submission is late for 30 hours, that counts as 2 slip days
- After all slip days are used up, 5% deduction for every 24 hours of delay. (e.g., 5 points for a 100-point homework)
- We will not consider late submission of any missing parts of an homework assignment or project deliverable. To make sure you have submitted everything, download your submitted files to double check.
- No penalties for medical reasons or emergencies. And should they arise, you must contact the Dean of Students office. Doctor's notes, medical documentation, explanation of emergencies, etc. should be submitted to the Dean’s office. After their office receives the information, we will notify me on your behalf.
Distance Learning Sections (Q & Q3)
A standard 3-day lag applies to all homework and project deliverables. For project presentation, a group that has DL student member can choose to:- Present in class without 3-day lag; or
- Submit a video presentation with 3-day lag (e.g., screen capture)
Dataset Ideas (may need API, or scraping)
- Google public datasets. Thanks Revant!
- NYC Taxi data for 2013 (FOILed by Chris Wong). 2013 Trip Data (11.0GB). 2013 Fare Data (7.7GB). Visualization for a days trip. Thanks Jitesh.
- Large datasets publicly available. Thanks Gopi!
- Georgia Tech's campus data (has APIs): bus info, directory, building, T-square, room reservation, building facilities usage (e.g., electricity, lights, A/C, etc.), Oscar/course info/registration, etc.
- Yahoo WebScope
- Data.gov: U.S. Government's open data
- IPEDS data: Postsecondary education data from National Centre for Education Statistics
- Bureau of Labor Statistics data
- Uber data: Anonymized data from over 2 billion trips
- Freebase
- Yelp
- Microsoft Academic Graph
- Numerous APIs from Google (e.g., Maps, Freebase, YouTube, etc.)
- Zillow: real estate listing site
- Numerous graph datasets (large and small): SNAP, Konect
- Movies data: IMDB
- List of lists of datasets for recommendations.
Thanks Jon! - Million song dataset by Echo Nest.
It contains not only the basic information of songs (artist, genre, year, length etc), but also some musical features(like tempo, pitch, key, brightness).
Thanks Minwei! - Dataset about soccer games, players, clubs.
No API, but easy to scrape.
For a soccer player: transfer history, performance, nationality, birth date, etc.
For a soccer club: performance, squad, etc.
Thanks Ding! - The Free 'Big Data' Sources Everyone Should Know
-
Quandl - a dataset search engine for time-series data.
Thanks Henry! -
UCI also has a collection of links to various datasets sorted for various tasks (Classification, Regression, etc)
Thanks Vinodh! - Amazon AWS Public Data Sets (Thanks Jonathan!)
- KDD Cup: annual competition in data mining, like Kaggle
- Academic domain: Microsoft Academic Search, DBLP
- Retrosheet: MLB statistics (Game/Play logs)
-
Classification datasets
Thanks Amish! -
Various geophysical datasets for the oceans (magnetism, gravity, seismology, etc).
Thanks Ryan! - Social trends (Thanks Jonathan!)
- Beer data (Thanks Jonathan!). Website offline :( . Older version at web.archive.org
- Academic torrents (terabytes) (Thanks Vaibhav!)
- Article Search API from the New York Times (all the way back to 1851!) (Thanks Guido!)
- (Kayak: flight, hotel, car, etc.)
- Data Science Initiative - Microsoft Research has various datasets and access to tools that can aid in data science research
Reading Materials & Resources
This course does NOT have any required textbook. We recommend the following books and resources.Data science, machine learning, data mining
- Data Science for Business by Foster Provost and Tom Fawcett
- The Elements of Statistical Learning: Data Mining, Inference, and Prediction by Trevor Hastie, Robert Tibshirani, and Jerome Friedman
Visualization
- A nice D3 tutorial
- The Wall Street Journal Guide to Information Graphics: The Dos and Don'ts of Presenting Data, Facts, and Figures by Dona Wong
- Information Dashboard Design: Displaying Data for At-a-Glance Monitoring by Stephen Few
- Show Me the Numbers: Designing Tables and Graphs to Enlighten by Stephen Few
SQL
- Video courses and lectures: (1) Coursera, (2) lynda.gatech.edu and search for SQL courses
- Interactive SQL tutorials
- Books: (1) SQL Cookbook (recipes to solve specific problems), Visual QuickStart Guide (succinct topic-by-topic), SQL Pocket Guide (covers syntax variations of MySQL, Oracle, etc.)
- Introductory tutorial
Probability
- FREE probability book, by Prof. Guy Lebanon. (From Amazon.)
Human Computation
- Human Computation book by Edith Law and Luis von Ahn
Prerequisites & Expectation
For both CSE 6242 (grad) and CX 4242 (undergrad)
Students are expected to complete significant programming assignments (homework, project) that
may involve higher-level languages or scripting (e.g., Java, R, Matlab, Python, C++, etc.).
Some assignments may involve web programming and D3 (e.g., Javascript, CSS).
You are expected to quickly learn many new things. For example, an assignment on Hadoop programming may require you to learn some basic Java and Scala quickly, which should not be too challenging if you already know another high-level language like Python or C++. Please make sure you are comfortable with this.
Please take a look at the assignments (homework and project) of the previous offerings of this course, which will give you some idea about the difficulty level of the assignments.
Basic linear algebra, probability knowledge is expected.
Some assignments may involve web programming and D3 (e.g., Javascript, CSS).
You are expected to quickly learn many new things. For example, an assignment on Hadoop programming may require you to learn some basic Java and Scala quickly, which should not be too challenging if you already know another high-level language like Python or C++. Please make sure you are comfortable with this.
Please take a look at the assignments (homework and project) of the previous offerings of this course, which will give you some idea about the difficulty level of the assignments.
Basic linear algebra, probability knowledge is expected.
Additional formal prerequisites for CSE 6242
None, but you should have taken courses similar to those listed in the next section, at Georgia Tech or at another school.Additional formal prerequisites for CX 4242
(Undergraduate Semester level MATH 2605 Minimum Grade of D orUndergraduate Semester level MATH 2401 Minimum Grade of D or
Undergraduate Semester level MATH 24X1 Minimum Grade of D) or
and
(Undergraduate Semester level MATH 3215 Minimum Grade of D or
Undergraduate Semester level MATH 3225 Minimum Grade of D or
Undergraduate Semester level ECE 3077 Minimum Grade of D or
Undergraduate Semester level ISYE 2027 Minimum Grade of D)
and
(Undergraduate Semester level CS 1371 Minimum Grade of C or
Undergraduate Semester level CS 1372 Minimum Grade of C or
Undergraduate Semester level CX 4010 Minimum Grade of C or
Undergraduate Semester level CX 4240 Minimum Grade of C)
Auditing & Pass/Fail
Due to the class size, I am not offering auditing and pass/fail option this semester.Acknowledgement & Related Classes
We thank Amazon's AWS in Education grant program for providing support for Amazon Web Services.Tableau's data visualization software is provided through the Tableau for Teaching program.
Many thanks to my colleagues for sharing their course materials:
- Prof. John Stasko - Information Visualization - Fall 2012
- Prof. Jeff Heer - Research Topics in Interactive Data Analysis - Spring 2011
- Prof. Christos Faloutsos - Multimedia Databases and Data Mining - Fall 2012