Building software for artists
How can we build software tools that empower artists to use computation in their own work?
I’m Sonia and I’m a member of the Expressive Computing Lab led by Jennifer Jacobs at the Media, Arts, and Technology Department at the University of California Santa Barbara. We’re trying to learn how to maximize expressive power — diversity in the kinds of outcomes and ideas that can be created using a digital system.
One way that my group studies expressive power is by learning from and working with domain experts, such as visual artists, to make programming systems and software tools for art.
I decided to write this article to share why I want to build systems that support artists in choosing how to use computation after doing a study on how visual artists approach software. There’s important, interdisciplinary work to be done to create more ways for creative practitioners to use code on their own terms in their workflows.
I’ve organized this post around three main discussion points: diversifying ways for people to use computation, answering why would artists want to code, and learning from visual artists about software development. If you’re an artist or software developer, we have an opportunity to build systems that can empower artists to extend software tools to create processes and outcomes beyond what we ourselves can imagine.
#1 Diversifying ways for people to use computation
Software developers can build domain-specific interfaces for more people to apply computational ideas in their work. For anyone like me interested in combining technology and the arts to make expressive systems, we can learn how to build systems that empower artists to use computation in their workflows.
Seymour Papert & Sherry Turkle’s paper Epistemological Pluralism pointed out a pervasive and harmful attitude that computer science is only accessible to a small group of people that learn how to program. To this day, this attitude prevents people from participating in computer science and keeps them from extending software use in diverse ways.
Software tools can diversify how people use computation. Think about how many people you know that use spreadsheets. Spreadsheets are powerful, because they have enabled people (who might not code) to use computation in diverse ways in their work.
Similarly, direct manipulation tools — tools that allow artists to directly select and edit objects (see Direct Manipulation by Ben Schneiderman) — can enable artists to use computational ideas like procedural constraints without explicitly writing code. For instance, Para lets artists use ordered lists to create and edit procedural art.
As a software developer, I can work with artists to design systems for artists that create pathways for them to choose how to apply computation to create artwork. And I believe the right way to build expressive systems is to build software that supports artists in developing one-of-a-kind workflows in order to create singular outcomes.
#2 Why would artists want to code?
Many people don’t think that artists want to use computational thinking in their work. Code is a powerful medium for expression (see Code as A Creative Medium). By building software that enables artists to code, I can create pathways for artists to achieve singular outcomes beyond what I can imagine.
When people in tech ask me “why would artists want to code?” they almost always fall into two camps.
And, before I go there: artists are coders. They build complex systems to express their ideas and are pioneers across every medium. If you don’t believe me, check out work by media artists Vera Molnár or Memo Akten.
Code as using computational thinking
Many people have internalized this idea that just because a person doesn’t work in a technical or scientific field they wouldn’t want to use computation in their work. In this case, being able to code means the same thing as being able to apply powerful computational ideas like using variables to create your own routines or formulas or using loops.
To me, that’s a bit like saying someone wouldn’t care about being able to read or write because they are not training to be writers. Do you see how far off the mark that is?
In undergrad, I tried to build a programming language for choreographers that was inspired by formal notational systems for dance. While I had amazing opportunities to interview a former dancer with the American Ballet Theater, discuss ideas with the head of the musicology department, and learn from an adviser in program analysis, I felt so dissuaded by a respected CS pioneer asking me why “girls, no sorry women, in tutus” would want to code.
A sound bite doesn’t make a person, but that comment goes to show just how pervasive and dismissive this idea is that artists or any other group of people wouldn’t care to code or to use computational ideas in their own way for their work. I wish I had shown him a clip of Martha Graham or Kyle Abraham’s work: artists have creative, singular points of view.
Because of their unique perspectives, which I cannot replicate, artists are best suited to understanding how and when to use computation in their processes. And unlike other domains where I might be able to generalize common process to automate specific workflows, in the arts, personal and individual processes truly matter.
Code as textual programming (writing C++)
I believe that others ask this question — why would artists want to code — in good faith. They see code as textual programming that is out of reach for many because it is challenging to learn and can be tangential to the ways that artists work.
Often, computer scientists who fall into this camp either work on developing complex algorithmic solutions that are black-boxed to make them accessible or work on taking expert, domain-specific knowledge from a discipline in the arts and packaging it in a way that a broader audience can use it.
I think that work is compelling and, at the same time, we need more focus on the rest of the story. How can we, as systems researchers, expand our focus on how software is used and the ways in which it might empower (or disempower) people who use it? And for those focused on building tools for novices, how do we build self-reliance and encourage creative points of view beyond proficiency — how do we support people in articulating an individual perspective?
I am fortunate to be working in human-computer interaction (HCI) — a critical area of computer science to shape how people interact with computational ideas in order to empower people to then, in turn, shape how computation is used — that creates opportunities to focus on these questions and validates their significance.
So far, my biggest and most exciting challenge has been thinking about how to create software that lets artists choose how to use computation in meaningful ways. Understanding how professional artists approach (i.e. use, build, and extend) software is critical to building an expressive software tool for visual art.
#3 Learning about software from visual artists
Last summer, when I had the opportunity to work with Jennifer and a collaborator from Stanford, Jingyi Li, on examining how visual artists use and build software, I jumped at it. We examined a data set of 13 interviews with visual artists from diverse domains like interactive media art and painting. From this data, we identified patterns in areas like artists’ attitudes towards efficiency. These patterns suggest how to build better software for artists, including how to build software tools that enable artists to actively choose how to use computation in their work in meaningful ways.
For starters, artists didn’t necessarily value the kinds of automation that software developers are building in software tools for visual art. Artists rejected automated generative processes (e.g. Lynne Yun, a type designer, rejected an automated process for arranging type around a shape) because it created visually displeasing outcomes. And, while they rejected high-level automation, artists valued “in the loop” automated processes like undo or version management. These types of processes let them maintain precise control over their process and, as a result, the aesthetics of their artwork.
While I don’t think that this rules out building systems with high-levels of automation, it does suggest that automated processes should offer artists fine-grained control at multiple levels. Giving artists choice over how to use computation at every level enables “in-the-loop” automation. And, giving artists precise control over automated processes enables them to articulate a unique point of view and develop self-reliance in applying computation in the way that aligns with their mental model for how it fits into their work. And lastly, enabling more levers (i.e. choices that an artist can make that influence the process of how to make things using a software tool) maximizes the expressive power of software.
Another thing that stood out to me is how much artists valued working by hand, not just to work out problems in their head or get ideas through physical processes, but also at times as the most efficient way to precisely translate an idea in their mind to their canvas. Some artists also alternated between manual and digital processes.
For example, Ben Tritt, a painter, stored digital paint-mix recipes on his tablet to record precise color combinations and worked by hand to build up layers of paint and develop the material and volumetric aspect of an artwork. Michael Salter and Chris Coleman, new media artists, physically ripped apart outputs from a program and recombined and scanned them in as inputs while modifying the code of their program in one of their collaborations. Emily Gobeille, an interactive media artist, sketched out ideas for characters and their movements by hand but then found translating them into digital media to be painstakingly repetitive.
This collection of observations suggests exciting opportunities to create software tools that lets artists organize and develop computational processes for managing their data across custom digital and physical workflows. It also shows how important working by hand is for artists. Software tools that enable artists to use computation can prioritize this by supporting manual mark-making and enabling artists to manually draw or define computational relationships.
Lastly, I learned that artists, in addition to building software for functional applications like making artwork interactive, experiment with software for purely intellectual or creative growth. For example, some artists shared building systems that imposed specific, productive constraints on their workflows.
Fish McGill, a digital illustrator and animator, built a drawing tool where strokes fade automatically over time to emphasize process and form. Eran Hilleli, an animator, built a tool that lets animators loop hand-drawn animations seamlessly, so that instead of working with a linear timeline, artists use circular timelines where the last frame flows immediately back into the first.
To me, these tools built by Eran and Fish suggest opportunity spaces to software tools that artists can extend in order to define ad-hoc productive constraints on their workflows. It also suggests an opportunity space to build software tools that let animators use code to define how time impacts their composition.
This is a small set of our findings but in short they point to the fact that visual artists’ experiences with software are aligned with those of end-user programmers — people who use programming to create, modify, and experiment with ad-hoc tools instead of using an out-of-the-box tool.
This is in keeping with the kinds of observations that others, including systems researchers and artists, have made about how software is used in creative workflows. For example, Golan Levin, a pioneering media artist and mentor to founders of creative coding platforms, once contrasted two types of software use in the arts: (A) popular commercial applications that achieve packaged effects and (B) technological craft where the artist develops software tools as a part of their process like a painter making their own paints.
I’m excited to work with artists to develop expressive software tools that artists can take from A to B. By building software and communities around software tools that empower artists to use computational ideas in their own way, we can create pathways for artists to extend how computation transforms creative practice.
If you want to know more about our methods, findings or how we think systems researchers can use them to build end-user programming tools for artists, you can check out our paper: What We Can Learn from Visual Artists About Software Development.
Thanks for sharing your time with me. I hope you’re excited about interdisciplinary opportunities in HCI to use and build expressive software tools that empower artists to use computation in their work.
Reach out to me at soniapriyahashim@gmail.com.
