Visualizing Billions of Data Points: Doing it Right

Visualizing Billions of Data Points: Doing it Right

1. 1. Visualizing Billions of Data Points: Doing It Right
2. 2. About Us Peter Wang is the CTO and Co-founder of Continuum Analytics and the creator of Bokeh. He has been developing commercial scientific computing and visualization software for over 15 years. As a creator of the PyData conference, he devotes time and energy to growing the Python data community, and advocating and teaching Python at conferences worldwide. Jim Bednar is a Solutions Architect at Continuum Analytics, and an Honorary Fellow at the University of Edinburgh, Scotland. He holds a Ph.D. in Computer Science from the University of Texas, as well as undergraduate degrees in Electrical Engineering and Philosophy. He has published more than 50 papers and books about the visual system and software development, and manages the open-source Python projects HoloViews, ImaGen, Param, and Datashader.
3. 3. Overview • Visualizing big data — what is the problem? • Datashading • Dataset 1: 10 million points of NYC Taxi data • Dataset 2: 3 billion points of OpenStreetMap data • Dataset 3: 300 million points of US Census data • Sneak preview: Data explorer web interface • Q&A
4. 4. Visualizing big data: What is the problem?
5. 5. 5 Big data magnifies small problems • Of course, big data presents storage and computation problems • More importantly, standard plotting tools have problems that are magnified by big data: • Overdrawing/Overplotting • Saturation • Undersaturation • Binning issues • We’ll first explain these problems, and then present a new technique called datashading to address them head-on.
6. 6. 6 Overdrawing • For a scatterplot, the order in which points are drawn is very important • The same distribution can look entirely different depending on plotting order • Last data plotted overplots
7. 7. 7 Overdrawing • Underlying issue is just occlusion • Same problem happens with one category, but less obvious • Can prevent occlusion using transparency
8. 8. 8 Saturation • E.g. for alpha = 0.1, up to 10 points can overlap before saturating the available brightness • Now the order of plotting matters less • After 10 points, first-plotted data still lost • For one category, 10, 20, or 2000 points overlapping will look identical
9. 9. 9 Saturation • Same alpha value, more points: • Now is highly misleading • alpha value depends on size, overlap of dataset • Difficult-to-set parameter, hard to know when data is misrepresented
10. 10. 10 Saturation • Can try to reduce point size to reduce overplotting and saturation • Now points are hard to see, with no guarantee of avoiding problems • Another difficult-to-set parameter • For really big data, scatterplots start to become very inefficient, because there are many datapoints per pixel — may as well be binning by pixel
11. 11. 11 Binning issues • Can use heatmap instead of scatter • Avoids saturation by auto-ranging on bins • Result independent of data size • Here two merged normal distributions look very different at different binning • Another difficult-to-set parameter
12. 12. 12 Plotting big data • When exploring really big data, the visualization is all you have — there’s no way to look at each of the individual data points • Common plotting problems can lead to completely incorrect conclusions based on misleading visualizations • Slow processing makes trial and error approach ineffective When data is large, you don’t know when the viz is lying.
13. 13. Datashading
14. 14. 14 Datashading • Flexible, configurable pipeline for automatic plotting • Provides flexible plugins for viz stages, like in graphics shaders • Completely prevents overplotting, saturation, and undersaturation • Mitigates binning issues by providing fully interactive exploration in web browsers, even of very large datasets on ordinary machines • Statistical transformations of data are a first-class aspect of the visualization • Allows rapid iteration of visual styles & configs, interactive selections and filtering, to support data exploration
15. 15. 15 Datashading Pipeline: Projection Data Project / Synthesize Scene • Stage 1: select variables (columns) to project onto the screen • Data often filtered at this stage
16. 16. 16 Datashading Pipeline: Aggregation Data Project / Synthesize Scene Aggregates Sample / Raster • Stage 2: Aggregate data into a fixed set of bins • Each bin yields one or more scalars (total count, mean, stddev, etc.)
17. 17. 17 Datashading Pipeline: Transfer Data Project / Synthesize Scene Aggregates Sample / Raster Transfer Image • Stage 3: Transform data using one or more transfer functions, culminating in a function that yields a visible image • Each stage can be replaced and configured separately
18. 18. 18 Datashader software architecture • Open-source BSD-licensed library, written in Python • Download from https://github.com/bokeh/datashader • conda  install  -­‐c  bokeh  datashader   • Works well with the Bokeh library, for interactive visualization of large data in the browser, but is fully independent • Supports large parallel machines, but works well on desktops and laptops — all demos shown here will be on a standard MacBook Pro
19. 19. Dataset 1: NYC Taxi data
20. 20. Dataset 1: Overview This demo shows how traditional plotting tools break down for large datasets, and how to use datashading to make even large datasets practical interactively. • Data for 10 million New York City taxi trips • Even 100,000 points gets slow for scatterplot • Parameters usually need adjusting for every zoom • True relationships within data not visible in std plot Datashading automatically reveals the entire dataset, including outliers, hot spots, and missing data
21. 21. Dataset 2: 3 billion OSM points
22. 22. Dataset 2: Overview Because Datashader decouples the data-processing from the visualization, it can handle arbitrarily large data • E.g. Open Street Map data: • About 3 billion GPS coordinates • https://blog.openstreetmap.org/ 2012/04/01/bulk-gps-point-data/. • This image was rendered in one minute on a standard MacBook with 16 GB RAM • Renders in 7 seconds on a 128GB Amazon EC2 instance
23. 23. Out of core operation • OSM database is 40GB, larger than the 16GB RAM on this machine • Fast out-of-core operation powered by: • Numba: generates fast C and GPU code from Python source • Dask: Parallelizes tasks • datashader source code is all Python
24. 24. Dataset 3: 300 million points Census data
25. 25. 25 Categorical data: 2010 US Census • One point per person • 300 million total • Categorized by race • Datashading shows faithful distribution per pixel
26. 26. 26 Categorical data: Illinois Census • Zooming in doesn’t require new parameter settings • All ranges, etc. update automatically
27. 27. 27 Categorical data: Chicago Census • Chicago area is highly racially segregated, on average • Population density differences still visible, automatically
28. 28. 28 Categorical data: Chinatown Census • At neighborhood level, the full racial distribution is clear • Size of dots increases automatically for visibility
29. 29. 29 Categorical data: Chinatown Census • Bokeh supports geographic data tiles • Borders between race- based neighborhoods are related to geographic features
30. 30. Visualizing Big Data: Preview of “Data Explorer”
31. 31. Data Explorer User application in development, to be included with the Anaconda platform
32. 32. 32 Data Explorer features • Automatically visualize and explore datasets • Flexibly select filters, fields, projections, aggregations, transfer fns • Powered by the open-source datashader and Bokeh libraries • Lets you explore your data interactively, regardless of the size • Show data relative to other datasets or geographic data • Easy migration to custom code when necessary, since it is all based on open libraries Get in touch with us to help steer this project!
33. 33. Contact Information and Additional Details • Contact sales@continuum.io for more information about
 Anaconda subscriptions and about becoming an early adopter for Data Explorer — help make sure our product fits your needs! • View documentation and examples at github.com/bokeh/datashader and bokeh.pydata.org • View demo notebooks on Anaconda Cloud notebooks.anaconda.org/jbednar/
34. 34. Thank you Email: sales@continuum.io Twitter: @ContinuumIO Peter Wang Twitter: @pwang Jim Bednar Twitter: @JamesABednar