Readings

You are expected to write and submit a paper review of the readings before each class, and answer some questions about the readings. The review should be akin to a conference paper review. The purpose of the readings is to provide an illustrative example of the research area. You are encouraged but not required to read the supplemental readings to better understand the materials.

You can discuss questions and ask for clarifications with your colleagues and/or on piazza. You are expected to formulate your own opinion of the reading(s) and write the review yourself. See for a description of what we expect in paper reviews.

We may select a random review to read and discuss in class. This serves to highlight important characteristics of reading papers and writing good reviews.

Submission

Overview

• Reviews are due 11:59PM EST the night before lecture.
• Late submissions are given a score of 0 without prior approval.
• You may miss submissions for up to 3 classes.
• To submit, go to the class wiki and click on the appropriate topic

Reading Tips

Ask the following questions while readings

• Context
  • What are the actual hypotheses?
  • What was the unmet need or opportunity? Does it make sense?
  • What were existing approaches and why do they work or not work?
  • What is the simplest example that highlights the problem that this approach works best for?
• Approach
  • When does the approach work? Assess the underlying assumptions.
  • How well does the evaluation validate the core hypotheses/claims?
    • Do you believe their results?
    • Are the results presented well?

How to read papers

• How to Read a Paper - S. Keshav
• How to Read a Research Compendium - Nüst et al.

How to review papers

• Review AntiPatterns (written for FSE 14 PC)
• Ethics of Peer Review
• How NOT to review a paper
• Conference Reviewing Considered Harmful: A view on how reviewing works in practice.

Background

Background you should be comfortable with

• Relational algebra and SQL
• Cost-based optimization
• A good video series is the Stanford Database Lectures by Jennifer Widom
• All of the materials for Professor Wu’s 4111 are on https://w4111.github.io
• An advanced video series is the CMU Introduction to Databases by Andy Pavlo, it covers more of the systems side.

Visualization Classics

• Tukey, John W. Exploratory Data Analysis. 1979

Surveys

• Making Database Systems Usable
• Overview of Data Exploration Techniques

The Papers

Intro

Readings

• Optional: Jeanette Wing The Data Lifecycle
• Optional: JCR Licklider Man-Computer Symbiosis
• Optional: Tamara Munzner (Textbook) Visualization Analysis and Design
• Optional: Jeffrey Heer Agency plus automation: Designing artificial intelligence into interactive systems
• Optional: Kristi Morton et al. Support the Data Enthusiast: Challenges for Next-generation Data-analysis Systems

Vis: Tasks

Readings

• Required: Matthew Brehmer, Tamara Munzner A Multi-Level Typology of Abstract Visualization Tasks
  • Her talk Problem-Driven Visualization Through Design Studies contains more examples that motivate the typology.
• Optional: Heer, Shneiderman Interactive Dynamics for Visual Analysis
• Optional: Ben Shneiderman The Eyes Have It

Vis: Languages

Readings

• Required: Chris Stolte et al. Polaris
  • Focus on the layout algebra
• Required: VizQL algebra are in Chapter 4 of Stolte’s thesis
• Optional: Pu and Kay Probabilistic Grammar of Graphics
• Optional: Aidan Slingsby, Jason Dykes, Jo Wood Configuring Hierarchical Layouts to Address Research Questions
  • Website with syntax documentation
• Optional: Wickham and Hofmann Product Plots
• Optional: Arvind Satyanarayan et al. Vega-Lite: A Grammar of Interactive Graphics
• Optional: Hadley Wickham A Layered Grammar of Graphics Sections 1-3. documentation. ggplot2 book

Vis: Interaction Design

Readings

• Required: Michel Beaudouin-Lafon. Instrumental Interaction
• Required: Heer and Shneiderman. Interaction Dynamics for Visual Analytics
• Optional: Arvind Satyanarayan et al. Vega-Lite: A Grammar of Interactive Graphics
• Optional: Dimara and Perin. What is Interaction for Data Visualization?
• Optional: Bahador Saket et al. Investigating Direct Manipulation of Graphical Encodings as a Method for User Interaction (Liger demo video)
• Optional: Ali Sarvghad et al. Embedded Merge & Split: Visual Adjustment of Data Grouping

Vis: Perception

Readings

• Required: Talbot et al. Four Experiments on the Perception of Bar Charts
• Optional: Jeff Heer et al. Crowdsourcing graphical perception: using mechanical turk to assess visualization design
• Optional: Cleveland and Mcgill (the classic 1985 paper) Graphical Perception and Graphical Methods for Analyzing Scientific Data
• Optional: Gordon Kindlmann et al. An Algebraic Process for Visualization Design
• Optional: McColeman et al. Rethinking the Ranks of Visual Channels
• Optional: Burns et al. How to evaluate vis across different levels of understanding
• Optional: Zhicheng Liu and Jeffrey Heer. The Effects of Interactive Latency on Exploratory Visual Analysis
• Optional: Ryan et al. At a glance: pixel approximate entropy as a measure of line chart complexity
• Optional: Jardine et al. Perceptual Proxies of Visual Comparison

Vis: Cognition

Readings

• Required: Kim, YS., Walls, L., Krafft, P., and Hullman, J. A Bayesian Cognition Approach to Improve Data Visualization
• Optional: In Kwon Choi, Taylor Childers, Nirmal R., Swati Mishra, Kyle Harris, Khairi Reda. Concept-driven visual analytics: Hypothesis Based Reasoning
• Optional: Kale et al. Hypothetical Outcome Plots Help Untrained Observers Judge Trends in Ambiguous Data

Vis: Design Recommendation

Readings

• Required: Jock D. Mackinlay et al. Show Me: Automatic Presentation for Visual Analysis
• Optional: referenced in A2
  • Dominik Moritz et al. Formalizing Visualization Design Knowledge as Constraints: Actionable and Extensible Models in Draco
  • Kanit Wongsuphasawat et al. Voyager: Exploratory analysis via faceted browsing of visualization recommendations
  • Lee et al. Lux: Always-on Vis Recommendations
• Optional: Zong et al. Lyra 2: Designing Interactive Visualizations by Demonstration
• Optional: Bahador Saket et al. Beyond Heuristics: Learning Visualization Design
• Optional: Chenglong Wang et al. Visualization By Example
• Optional: Manasi Vartak et al. SeeDB: efficient data-driven visualization recommendations to support visual analysis.
• Optional: Qian et al. Learning to Recommend Visualizations from Data

Multiverse Analysis

Readings

• Jun et al. Tea: Automated Statistical Analysis
  • Watch the talk
• Liu et al. Boba: Authoring and Visualizing Multiverse Analyses
  (builds on Paths Explored, Paths Omitted)
• Dragicevic et al. Increasing the Transparency of Research Papers with Explorable Multiverse Analyses
  • See their interactive demo
• Gelman and Loken Garden of Forking Paths
• Sedlmair et al Visual Parameter Space Analysis

Data Models

Readings

• Required: Mike Stonebraker, Joe Hellerstein What Goes Around Comes Around
• Fotis Psallidas, Eugene Wu Provenance for Interactive Visualizations
• Cohort Query Processing

Data Interfaces from a Data Perspective

Readings

• Required: Wu et al Case for Data Visualization Management System
• Required: Moritz et al Dynamic Client-Server Optimization for Scalable Interactive Visualization on the Web
• Optional: Kyrix-S: Authoring Scalable Scatterplot Visualizations of Big Data
• Optional: Bin-Summarize-Smooth
• Optional: Heer and Agrawala Design considerations for collaborative visual analytics

Performance and Engines

Readings

• Required: Leilani Battle and Carlos Scheidegger A Structured Review of Data Management Technology for Interactive Visualization and Analysis
• Optional: C-Store: A Column-oriented DBMS

Performance: Approximation and Precomputation

Readings

• Required: Sameer Agarwal et al. BlinkDB: Queries with Bounded Errors and Bounded Response Times on Very Large Data
• Required: Zhicheng Liu et al. imMens: real-time visual querying of big data
• Approximation
  • Optional: Dominik Moritz, et al. Trust, but Verify: Optimistic Visualizations of Approximate Queries for Exploring Big Data
  • Optional: Bolin Ding et al. Sample + Seek: Approximating Aggregates with Distribution Precision Guarantee
  • Optional: Alabi, Daniel and Eugene Wu. PFunk-H: approximate query processing using perceptual models
  • Optional: Feifei Li et al. Wander Join: Online Aggregation via Random Walks
  • Optional: VerdictDB: Universalizing Approximate Query Processing
• Precomputation
  • Optional: Niranjan Kamat et al. Distributed and interactive cube exploration
  • Optional: Wenbo Tao et al. Kyrix: Interactive Pan/Zoom Visualizations at Scale

Performance: Columnar Execution

Readings

• Required: Column Store Survey Chapters 1, Sections 3.1-3.2, Sections 4.1, 4.2 (skip 4.2.1-5), Skim 4.4.
• Optional: C-Store: A Column-oriented DBMS
• Optional: Richard Wesley et al. Leveraging Compression in the Tableau Data Engine

Performance: Physical Design

Readings

• Required: Physical Visualization Design (short paper)
• Required: Sanjay Agrawal, Nicolas Bruno, Surajit Chaudhuri, Vivek Narasayya

Optional Readings

• Lineage
  • Fotis Psallidas et al. Smoke: Fine-grained Lineage at Interactive Speed
• Communication
  • Jugel, Uwe et al. M4: A Visualization-Oriented Time Series Data Aggregation
  • Haneen Mohammed, Ziyun Wei, Ravi Netravali, Eugene Wu Continuous Prefetch for Interactive Data Applications
• Multi-query Optimization
  • Georgios Giannikis et al. Shared Workload Optimization
  • Marcin Zukowski et al. Cooperative Scans: Dynamic Bandwidth Sharing in a DBMS
  • Roee Ebenstein, et al. FluxQuery: An Execution Framework for Highly Interactive Query Workloads
• Compression
  • Richard Wesley et al. Leveraging Compression in the Tableau Data Engine
• Effects of poor performance
  • Kelly Mack et al. Characterizing Scalability Issues in Spreadsheet Software Using Online Forums
• Sampling
  • Edward Gan et al. Moment-based Quantile Sketches for Efficient High Cardinality Aggregation Queries

Modalities: Voice and Natural Language

• One of the following is required:
  • Immanuel Trummer Data Vocalization with CiceroDB
  • Narechania et al NL4DV: Toolkit for Generating Analytic Specs for Data Vis from Natural Language Queries
• Optional: Vraj Shah et al. SpeakQL: Towards Speech-driven Multimodal Querying of Structured Data
• Optional: Christopher Baik et al. Bridging the Semantic Gap with SQL Query Logs in Natural Language Interfaces to Databases
• Optional: Jimmy Lin ICDE 21 Keynote: The Attack of the Muppets!

Modalities: Spreadsheets

Readings

• Required: Eirik Bakke et al. Expressive Query Construction through Direct Manipulation of Nested Relational Results
• Optional: Bendre et al. Dataspread: Unifying Databases and Spreadsheets
• Optional: Lyn Bartram, Michael Correll, and Melanie Tory Untidy Data: The Unreasonable Effectiveness of Tables
• Optional: Espalier/Object Spreadsheets

Modalities: Additional Modalities

Optional Reading

• Codi Burley et al. ARQuery: Hallucinating Analytics over Real-World Data using Augmented Reality
• Willett et al. Embedded Data Representations

Modalities: Touch and Gesture

• Required: Arnab Nandi, Lilong Liang, Michael Mandel Gestural Query Specification
• Optional: Kenrick Kin, Björn Hartmann, Tony DeRose, Maneesh Agrawala Proton: Multitouch Gestures as Regular Expressions
• Optional: Beyond Mouse and Keyboard: Expanding Design Considerations for Information Visualization Interactions
• Optional: VizDOM
• Optional: DBTouch

Tasks: Comparison

Readings

• Required: Wu et al. VCA: View Composition Algebra
• Optional: Michael Gleicher Considerations for Visualizing Comparison

Tasks: Data Extraction

Readings

• Required: Sean Kandel et al. Wrangler: Interactive Visual Specification of Data Transformation Scripts
• Required: Maeda Hanafi et al. Texture: A Collaborative Framework for Structure Identification over Print Documents
• Optional: Sumit Gulwani. Automating String Processing in Spreadsheets Using Input-output Examples
• Optional: Maeda F. Hanafi et al. SEER: Auto-Generating Information Extraction Rules from User-Specified Examples
• Optional: Mikael Mayer et al. User Interaction Models for Disambiguation in Programming by Example
• Optional: Eric Horvitz. Principles of Mixed-initiative User Interfaces
• Optional: Jeffrey Heer, et al. Predictive Interaction for Data Transformation

Tasks: Event Analysis

Readings

• Required: Guo et al. Survey on Visual Analysis of Event Sequence Data
• Required (skim): Frequent Pattern Mining
• Optional: Kerr et al. Core Flow
• Optional: Wang et al. Patterns and Sequences: Interactive Exploration of Clickstream Data
• Optional: Wu et al. High Performance Complex Event Processing Over Streams

Tasks: Data Cleaning

Readings

• Optional: Sean Kandel et al. Profiler: Integrated Statistical Analysis and Visualization for Data Quality Assessment
• Optional: Jie Song et al. GeoFlux: Hands-Off Data IntegrationLeveraging Join Key Knowledge
• Optional: Sanjay Krishnan et al. ActiveClean: Interactive Data Cleaning For Statistical Modeling

Tasks: Automation

Readings

• Pablo Estevez et al. 150 successful machine learning models: 6 lessons learned at Booking.com
• Shan Carter, Michael Nielsen Using Artificial Intelligence to Augment Human Intelligence

Tasks: Debugging and Interpretable ML

Readings

• Required: (Skim) Chris Olah et al. The Building Blocks of Interpretability
• Required: Cynthia Rudin Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models Instead

Tasks: Debugging Analytics

Readings

• Required: Eugene Wu, Sam Madden Scorpion: Explaining Away Outliers in Aggregate Queries
• Required: Tableau Explain Data demo video
• Optional: Firas Abuzaid et al. DIFF: A Relational Interface for Large-Scale Data Explanation
• Optional: Wu et al. Complaint-driven Training Data Debugging for Query 2.0

Misc Papers

• Qianrui Zhang et al. Precision Interfaces
• Xiong Zhang and Philip J. Guo. DS.Js: Turn Any Webpage into an Example-Centric Live Programming Environment for Learning Data Science
• Kuang Chen et al. Shreddr: Pipelined Paper Digitization for Low-resource Organizations
• Kuang Chen, Harr Chen, et al. Usher: Improving Data Quality with Dynamic Forms
• Yuze Lou et al. Knowledge Graph Programming with a Human-in-the-Loop: Preliminary Results
• Leo Meyerovich et al. Empirical Analysis of Programming Language Adoption
• Christian Bors et al. A Provenance Task Abstraction Framework
• Eric Ragan et al. Characterizing Provenance in Visualization and Data Analysis: An Organizational Framework of Provenance Types and Purposes

Neat applications

• https://www.remix.com/