Data as a Creative Constraint – Eric Socolofsky

>>Hi. So we’re kind of in the doldrums of the afternoon. We just had lunch two full talks ago. So if you would like to stand up and stretch
for a moment, please do. I’ll just prattle for a little bit while you
shake it all out. Okay. Sit down. So I have been wanting to come to OpenVis
Conf for quite a while. This is the first time I have been here, and
I finally figured out how to do it, which is to have Irene pay for it. Thank you for having me. And I was talking to Jim yesterday, and I
understand from talking to Jim that the art talk is kind of an uncommon thing here. So that either means the bar is very low or
it’s very high. And I’m not exactly sure, but we’ll find out. I wanted to talk about this concept of using
data as a creative constraint to an artistic or creative process. And I think that as creative people in the
room, we’re familiar with the constraints in a creative or artistic process. Charles Seems say they depend on constraints. Constraints limit the number of variables
to balance and give a sturdy framework. So to bring data as a constraint into the
process, it will not dictate the form or the output. But offers a raw material that we can use
to chisel down into artistic expression. That’s the concept of bringing data into the
art. That was the founding concept that I was going
to do this talk on. The way I do work, and also goes the other
direction, which is to bring the artistic perspective to your visualization. Cute kid, huh? So data visualization fundamentally is communication. Communication is a fluid medium. Takes on the characteristics of the communicator,
the content you’re working with, and often of the audience itself. So the concept of the medium being the message
is very true for data visualization as well. The techniques that you use to visualize your
data has an outsized impact on the perception Of the data visualization. If you bring thinking into the data visualization
practice, you’re offering new opportunities for avenues of visualization of the data. We are looking at the what we generally refer
to as data art. First and foremost, it’s art. The intention it to generate an artistic work. And the data is used to augment or constrain
the art. And we can go back the other direction, bring
it back into the data visualization work. And this is what Marie said was artful data
viz in applied concepts. And it’s a way to think about thousand produce
work and it’s a research technique. So expose yourself to both processes in the
cycle, I think it will add richness to the work that you present. Even if the work you present is not necessarily
visually complex or, you know, if it’s more a sort of succinct and legible is your focus,
then this can still apply in the way you think about your work. So in terms of art in general, I think the
area of art that is the most applicable to data visualization in my perspective is generative
art. Generative art means a lot of things to a
lot of people. I’m not going to try to define it exactly,
but I’ll try to sort of describe it. Also has a lot of aliases. So known as algorithmic art, creative coding
is a newer that term, procedural art and computer art. Whatever that means. Generative art. And there are a number of characteristics
of generative art. The first is that it’s generative. You generate something. Employees a set of rules in a system in which
the rules bound the transformation of your input into some sort of output. And generative art works generally� not
generally. Sometimes. You have emergent behavior. You feed something into the set of rules,
you don’t necessarily know what’s going to come out. I found some of the most compelling generative
art have emergent behavior. In data visualization, emergence is not something
we want, if we don’t know how the end result is going to look, then how to convey and communicate
to people? Pros and cons depending on how you’re looking
at it. Sorry, another bullet point slide and one
more after this. We’ll get through it. There’s a number of techniques that I think
are applicable from generative art to data visualization. Can think of algorithms, randomness and probability. Particle systems. Recursion and repetition. Motion and color. I’m going to dive into each of these and how
they’re used in generative art and how they can be applied to visualization. We’ll start with algorithms. So algorithms can be used to illustrate behaviors
of complex systems. You can great an algorithm that sort of defines
the way elements sort of appear in your work. You can use them to generate textures to indicate
some quality of the data or to act as a backdrop to the data. The data becomes apparent, a negative space
against this algorithmic texture. And you can use algorithms to generate empathy. Use algorithms in your visualization or artwork,
you can create an empathetic link between the artwork and the audience. This here and are Sol LeWitt pieces. I think a lot of a lot of us are familiar
if the work. Primarily in the 20th century, drew things
on walls and galleries. He would write a set of rules and present
the set of rules to an artist or group of artists who would execute the rules on the
wall often with pencil or some other form of marking. So the rule set for this piece is lines, wall
drawing 69. Not touching, drawn at random, four colors,
uniformly dispersed with maximum density, covering the entire surface of the wall. This is one of many examples how this could
have played out in a gallery context. The point is it’s all about this sort of process
that merges. Or this piece of art that emerges from the
rules that he’s created. Oh, no. Internet. Sorry. Oh, Internet. No Internet. Do we have a wired connection? I’m sorry, I totally forgot about that detail. Bear with me. Oh, have you guys seen this. Can I do it? It doesn’t work in embed. When you don’t have Internet and you load
up a page that shows a little dinosaur in Chrome, you can� if you start messing with
your arrow keys, then the dinosaur comes to life and you can play a game where the dinosaur
hops around. You can’t do it in the embed player, so I
can’t entertain you while we get the Internet working. What’s that?>>AUDIENCE MEMBER: [ Away from microphone
]>>I don’t want to do that. Okay. Let’s try refreshing, maybe. Oh, no. I’m sorry. Totally forgot about this detail and I can’t
see where system preferences is. Is this it? Yes. Network. Where is network? Connected. Sorry? Oh, I shouldn’t have to use WiFi, right? It looks like I’m there. Let’s try one more time. Oh, go back. All right, we got it. Thank you. Okay. Network. Thank you. Sorry. Where’s my mouse, I need it. This is the piece by Casey Reas. Heel follows a similar pattern. Might be familiar as the cocreator with Ben
here in Boston. And he’s similar that he will set up a set
of rules, and instead of having artists execute the rules, he has software execute his rules. And he’ll often display his rule set alongside
his works on the wall in the gallery. So the rule set here� sorry� is a rectangular
surface filled with instances of element three. Which is another rule set. Each with a different size and color. Draw a tiny, transparent circle with the element. Increase the size and opacity, and decrease
it when it’s not. You can see the artwork emerges based on the
set of rules. Here’s an example that is more directly tied
to data visualization. This is a piece by Nervous System, also in
Boston. In which they have an algorithm that generates
a highly control phenokistoscope, zoetrope and praxinoscope� yeah it’s lightweight
and strong, a cellular structure. They adapt the density of that foam per the
foot strike of the runners. The sample individual runners foot strike. So we’ll see in a moment the areas of highest
force from the foot strike are the areas of the highest density in the foam generated
by the algorithm. So the presence of the data manipulates the
algorithmic texture. This is for Stamen, a news organization, shows
the stories getting the most views at a center of a cluster that’s determined by all the
other stories in the section. This is a simple clustering algorithm that
lumps together everything in the same section of the paper. But by using an algorithm, they can remove
the algorithm and allow the particle system to fall back into a more standard layout. Randomness and probability is another technique
that’s used to great effect in generative art. Less frequently in data visualization, but
we can use it to indicate natural variation that’s inherent in the data for which the
quantity is undefined, or not as useful for understanding the patterns. We can use it to look at fuzziness. They’re operations, and not always precise. Data visualizations have a lot of inherent
fuzziness or noise. Using probability to represent probability
itself is something we have to do as data visualizers. This was an early artwork. It’s called Musikalisches Wurfelspiel, sorry
to German speakers. You roll the dice and hear things if the video
plays. Whoa! All right. No� play. It doesn’t sound particularly random because
it’s Mozart. And it’s a bunch of individual pieces that
are stitched together in the order determined by your die rolls. Jumps ahead a bit in time and moving from
sound to visuals, this is the simple random walk algorithm which is used quite often in
generative art. It’s a way of using randomness to move through
space. The algorithms says move forward or backward
or left or right a random amount every step for every cycle. Looking at how that can be applied in an artistic
concept. This is Guido Carrillo, there are triangles
that randomly move around and leave a trail behind them. This is maybe not� yeah. And Plankton Populations, we worked on at
the Exploratorium, shows the plankton in the world’s oceans with colors on the map like
a biome map. And you can see individual plankton swimming
around. And the likelihood of the plankton in the
lens depends on the density of the plankton in the ocean at the time. I’m possibility generating them based on a
simulation that models plankton. And then� sorry. Next. Hi. This one Amanda already showed. So I don’t need to say too much about it. But I think it’s a great example. I personally do� I know not everybody does�
of portraying the uncertainty inherent in this case in polling data. One of the nice points they read is it’s not
completely random. It’s bounded by the 25th and 75th percentile
of the outcomes. Shows the uncertainty without being crazy
and wild about it. And if you want to use randomness in generative
or data art, there’s a series of writeups in that show you how you
can shepherd and bound randomness and shape them into different forms. So particle systems are something we’re very
familiar with here. So I’ll run through it fairly quickly. We can use particle systems to represent components
of a system. Representing the behavior of a system via
the interaction between particles. And show relationships between elements. So, for example, a forcedirected network graph
in which the distance is directionally proportional to the strength of the nodes. And data elements, don’t have to be dots on
a screen. They can show complexity, glyphs, smarts,
and has meta information about the entire system. This is an early example of a particle system
from 1986 by Craig Reynolds. You’re probably familiar with this. Each particle tries to stay approximately
the same distance from the adjacent particles. So what happens is this naturally exhibits
flocks behavior from the rule set. This is a more modern version, Robert Hodgins,
it looks like a starling murmuration. And this piece by Memo Akten and Quayola is
an art piece that shows the movement of Olympic athletes from the 2012 Olympics. They have run video analysis on footage and
used that to determine salient points of their bodies as they move through space and apply
that to the particle systems and particle systems, triangles, lines, bars, use different
ones as it goes on. I would love to the watch the entire thing,
but I can’t. And this one, Facebook flowers, shows a George
Takei post on Facebook as it moves through. And each petal is an individual share. We see the burst, the ten drills, that’s a
viral moment where it gets shared through many people. And we know this page. Fernanda Viegas and Martin Wattenberg. They
set up a flow field from wind data and introduce particles into it. When I put together a talk full of streaming
videos, I thought, there’s no way this is going to happen, right? Sorry?>>AUDIENCE MEMBER: [ Away from microphone
]>>Yeah, hint.FM/wind. It would be great to not sell them short. My apologies, I’m going to move on. Great. So my workloads and theirs doesn’t. So this is a piece that I did at the Exploratorium. Instead of using individual dots, I’m using
low fall from white squares. And instead of only wind data, it’s a model
from NOAA that looks at water content in the air. What we call in the Bay area, fog. This is essentially a fog forecast that’s
projected on to a topographical model at the Exploratorium. There are rolls over the mountains. This is 280 for anyone familiar with the Bay
area. And kind of goes between the peaks of the
mountains. So this is where things get a little strange. [ Laughter ]
Recursion and repetition are tough to find a use case for in data visualization, but
I think that exists. I was trying to rack my brain to find context
in which it would happen is the one I found is contexts or domains in which R and repetition
are important. If you have other ideas, I would love to see
that. This is one of my favorite days from the heady
day of flash, from Yugo Nakamura, no longer online. And splits into clay balls and then two. Whenever there are four adjacent, they crumple
back together. It’s a grid of 16 by 16. Which means when we fill up the screen, we
end up with this� punchline� this singularity. A recursion that we restart completely. And it goes on. This is from Alvin Lucier, 1980, maybe, which
is an audio piece. That might play. In which�
>>Any semblance of my speech with perhaps the exception of rhythm is destroyed. What you will hear, then, are the natural,
resonant frequencies of the room�>>So he actually speaks into this microphone
and then records that recording back in the� so he plays the recording back in the same
room which is then recorded by the microphone and then plays that back into the microphone
and does that over and over again until you end up with the resonant frequencies of the
room itself. It’s a fascinating piece conceptually and
really difficult to sit and watch a performance of. [ Laughter ]
So a more modern example of this is Patrick Liddell’s version by doing the same thing
with YouTube. Records himself talking into the camera. Uploads to YouTube, compressed, and repeats
the process over and over again and you end up with� scary, right? That was about a thousand iterations right
there. So this is an example from the illustration
world. John Franzen, each alone, one breath. One breath, and then draws a single line. And the next breath as close as possible to
the first line. And over and over again. Through this process of repetition, recursion,
it’s muddy, errors are from left to right and it is a texture that feels like fabric
or cloth. I was in a really boring meeting recently,
so I did one myself. It was pretty fun. But I hope you are not doing that right now. And an example in data viz. This is by Tony Chu and Stephanie Yee, R2D3. They demonstrate how a decision tree is finding
all the split points through recursion and feed the data down the tree. Another technique is motion. Motion is great for bestowing personality. This is when animators do. Bring them to life by creating motion. Motion it represent� illustrate change. Can be used to bring attention to individual
elements or the piece itself. That was important to me when I was at the
Exploratorium and there were a lot of distractions. This piece by design I/O, an early augmented
reality experiment. Early� 2009. In which you hold up something� maybe 2011�
the camera sees the marker and gives the feeling of moving up and down through space. And elements on the screen move around, and
they sort of come to life through this motion that you can’t see right now. And this experience of being immersed in this
world that has all of this character and this really sort of, I don’t know, quality of wonder
that comes from the illustration, but especially from the motion of the illustrations. And another work by� with balloons in it�
by Jon Harris and Sep�Kamvar. I want you to want me. They were mining our data profiles from online
dating sites in which they represent each person in the online dating site as an individual
balloon. And they have characteristics determined through
naturalistic processing. And this is algorithmic pairing up two profiles. These balloons are just sort of intertwined
and expressing their love for one another. This is a piece� next. This is a piece that I did with Zana Armstrong
at Stamen called the atlas of emotions, we took the five universal human emotions, you
might be familiar with it from the movie from Pixar. And it’s meant to emulate the emotion of Angular. And this is fear, flicks out its talons. The next one is discuss, which has this kind
of heaving quality to it. And sadness takes a deep breath and a sigh. And then we have enjoyment, which is bouncy
and happy. So we tried to represent the emotions themselves
using motion. And, of course, the canonical example of how
we can use motion to indicate a change in state is Hans Rosling’s 200 countries, 200
years, four minutes.>>We have seen 200 years of remarkable progress. That huge historical gap between the west
and the rest is now�>>And we see the world’s countries lifting
themselves out of poverty, increasing their life spans and their per capita net income. So color. You obviously all know why color is important
to data visualization, so I’m not going to attempt to drill into that too deeply. But worth mentioning it’s good for encoding,
highlighting, legibility and visual appeal. I want to talk about how generative artists
source color. Of course, we can use ColorBrewer, but there
are other techniques as well. This is a piece� that was not meant as a
stab. ColorBrewer is awesome. You should all use it. Eric, a flash artist back in the 2000s, primarily. He would take photos and then he would use
those as a back drop for digital brush strokes. As the brush hits the screen, it is from the
source photo and it represents that. They sort of repaint the photo. Jared Tarbell used a box fitting algorithm
to fill out space and color it with a dominant color under each box. And Mario Klingemann, Quasimodo, did a pie
packing visualization, shows the distribution of the colors underneath each circle. That’s about sampling color from a point. This is an example from Viegas and Wattenberg,
Flickr flow. They take pictures from Boston commons and
sample them throughout the year. The year moves in a circle, and the thickness
is the frequency in the photos they found from Flickr. Upper lefthand side, summer time, and the
bottom, winter time. I hear it snows a lot here. This is a way to look at color sampled across
time instead of from a point in space. And Brendan Dawes did this with cinema redux,
this is Alfred Hitchcock’s vertigo down to one frame per second, reduced to an eight
by six-pixel image. You have a reduction down to the primary colors
of each frame. You can see the stripe down where the protagonist
goes through a total trip and if you have seen the movie� it’s color. And then Kevin Ferguson had western roundup,
averaged the frames together across time and space resourcing his colors. Great. So I showed you a bunch of pretty pictures,
and this is OpenVis Conf, not IO. So more skeptics in the audience than I might
find at IO. While this is an interaction of complexity
in artful techniques, not advocating in all cases. Use them responsibly. Aiming for clarity or the people to understand
the point immediately when they look at your visualizations, a lot of these techniques
are probably not going to work. Be conscientious how you apply depending on
the audience and the dataset. This is bouncing around the visualization
community, how to make it depending on the role. The products that we’re making. This is a diagram that Elijah Meeks put out,
shows on the engineers on the left side, precise, numeric clean work, and the righthand side,
consultants and artists doing design and engaging and complex work. He’s argues this is not the way it should
be, but we can move towards the center and learn from each other. This is especially relevant to me because
I transitioned from Stamen design to Uber. I moved from the right to the left. I want to land in the middle. And this is something to capture in a media
indication, visualizing the field. And please Tweet me for more information,
and I would love to tell you more. [ Applause ]
I know we’re a little bit over. Started a little bit late. Two minutes? I was pretty heartened to hear� the next
few slides I had are a side project and I was happy to hear there are other people in
the room that are considering this as well. Lately I have been feeling I’m not very well
represented by my government. And I have been trying to think about what
I can do about that and why that is. And I think that this is a large part of it. This is about the third time, maybe the fourth,
that gerrymandering has come up in this conference. Which is awesome. And this is a fundamental reason I don’t feel
well represented. It’s broken. We have a D3 team, people from Stamen, current,
former, CARTO, and we have been doing experiments. This was from Micah Migurski, generated what
ifs for Wisconsin. This is six different ways in which the map
could be drawn fairly according to the map for Wisconsin. It all started when U.S. Open data contacted
Stamen to bring the restricting process out of the smoky back rooms into the public. It uses GitHub as a way to display redistricting
proposals to display revisions and for people to comment and potentially suggest their own
alternatives if they’re so inclined. So another call to action, if you’re interested
in trying to fix gerrymandering, then Tweet me and I’ll get you on the Slack and we can
go from there. Thank you. [ Applause ]

Leave a Reply

Your email address will not be published. Required fields are marked *