Engineering Ethics

The Powerful and the Powerless: Reflections from FAccT 2020

By Zoe Kahn, CTSP Fellow 2019

Images of people soaring through space wearing astronaut helmets adorn the walls of a narrow hallway leading from the street-level entrance to the hotel lobby elevator bank. A life-sized model of an astronaut stands off to the side of the hotel registration desk. It’s safe to say… the conference hotel has an outer space theme. 

The words “What is fairness?” repeat on a large circular digital display at the center of the hotel lobby. I can’t decide if the display belongs in a newsroom, stock trade room, or train station. Perhaps art? Perhaps commentary? 

The word in question changes each day. What is Fairness? What is Accountability? What is Transparency? What questions are we missing? 

Image of digital sign reading "what is fairness?"

It is here—in Barcelona, Spain—that ~600 people gather for the ACM Fairness, Accountability, and Transparency (FAccT) conference to discuss efforts to design, develop, deploy, and evaluate machine learning models in ways that are more fair, accountable, and transparent. 

Despite the conference’s roots in computer science, there were several papers, tutorials, and CRAFT sessions that challenged the community to think beyond technical fixes to broader socio-technical contexts rife with power and complex social dynamics.

Let me begin with a discussion of power. The ACLU of Washington and the Media Mobilizing Project put on a terrific CRAFT session titled, Creating Community-Based Tech Policy: Case Studies, Lessons Learned, and What Technologist and Communities Can Do Together. The two-part session first documented the ACLU of Washington’s efforts, in partnership with academics primarily from the University of Washington (Katell et al., 2020), to include marginalized and vulnerable communities in the development of tech policy surrounding the acquisition and use of surveillance technologies in the City of Seattle. The Media Mobilizing Project then presented their work investigating the use of risk assessment tools in the criminal justice system across the country. And, in particular, their efforts with community groups to combat the use of criminal justice risk assessment tools in Philadelphia. Taken together, the session provided models for meaningfully engaging marginalized and vulnerable populations in algorithmic accountability, forming the foundation for workshop participants to reflect upon their own power.

READ MORE

Backstage Decisions, Front-stage Experts: Non-technical Experiences and the Political Engagement of Scientists

by Santiago Molina and Gordon PherriboCTSP Fellows

This is the second in a series of posts on the project “Democratizing” Technology: Expertise and Innovation in Genetic Engineering.

See the first post in the series: Backstage Decisions, Front-stage Experts: Interviewing Genome-Editing Scientists.

Since 2015, scientists, ethicists, and regulators have attempted address the ethical, moral, and social concerns involving genetic modifications to the human germline. Discourse involving these concerns focused on advancing a culture of responsibility and precaution within the scientific community, rather than the creation of new institutional policies and laws. Confidence in scientist’s ability to self-regulate has become increasingly tenuous with the recent news of the birth of genome-edited twins on November 26th, 2018, despite the scientific consensus that such experiments are medically and ethically unwarranted. In response, journalists, social scientists and critical researchers in the life sciences have posed the question: Who should be involved in deciding how genome-editing technologies should be used and for what aims?

In this post, we complicate the idea that technical expertise, which is usually narrowly defined on the basis of professional experience or knowledge, should be the main criteria for having a seat at the table during scientific decision-making. Drawing from eight interviews with scientists who participated in a small meeting held in Napa Valley in 2015, we highlight the role of non-technical experiences in shaping scientists’ views of decision-making about genome editing.

We identify three experiences that have influenced scientists’ views and deliberations about the application and potential consequences and benefits of genetic engineering technologies: 1) reading and group discussions outside of their academic disciplines, 2) direct engagement with patient communities, and 3) involvement in social movements. To wrap up, we make some modest suggestions for what these might mean in the context of STEM education.

1. Reading Outside of the Discipline and Group Discussions.

During our interviews we asked scientists how they shaped their viewpoints about biotechnology and its relationship to society. Respondents described their exposure to new viewpoints and reflected on the effect this exposure had on their decision-making. One of the sources of these exposures was reading outside of their academic discipline. We were surprised to hear about how the work of philosophers of science and sociologists of science did inform the decision making of one senior scientist at the Napa Valley meeting. This faculty member discussed their interest in finding opportunities to supplement their laboratory training with philosophical discussions about issues tangential to the science they were working on. With other graduate students, they created a small group that met regularly to discuss concepts and theories in philosophy of science, ethics and sociology of science.

We met- I don’t remember whether it was once a month or once every two weeks to discuss issues around the philosophy and societal issues of science. So we would find books, read books, um from you know – from Bertrand Russell, the philosopher, to Jacob Bronowski to Alfred Lord Whitehead, you know books on the philosophy and the applications of science,

The scientist described that this work added additional layers to their understanding of societal issues related to science. Even though this reading group was instrumental in developing his own awareness of the relationship between science and broader social, political and cultural issues, this respondent also lamented how the opportunity to delve into topics outside of a graduate student’s normal routine, “was not encouraged by any of [their] mentors.” This theme came up in several of our interviews, reinforcing the importance of mentors in shaping how scientists make meaning of their discipline in relation to society, and what educational and professional development opportunities graduate students feel comfortable pursuing outside of their formal training.

2. Direct engagement through service.

The most distinctly communicated experiences our interviewees engaged in outside of their formal training were service-related learning experiences that involved direct interaction with communities that would medically benefit from the technology. These experiences appeared to give individuals a greater sense of civic responsibility, and afforded them a more expansive understanding of the relationship between their work and broader communities. For genome-editing researchers, this crucially meant being aware of the social and medical realities of patients that might be research subjects in clinical trials for CRISPR-based therapies.

In our interviews, scientists had direct engagement with people outside of their discipline within national scientific boards, federal organizations, health clinics, and the biotech and pharmaceutical industry. These types of experiences provide an opportunity to collaborate with stakeholders on pressing issues, learn and benefit from industry and market knowledge, and ensure that the outcome of decisions are both relevant and meaningful to community stakeholders outside of the lab.

One of our respondents reflected on how they learned important skills, such as active listening, through professional experiences with indigenous patient communities–which helped this respondent better serve the community’s needs.

I’ve learned a whole lot from the patients I’ve taken care of and the people I’ve met. I certainly learned a great deal from going to the Navajo reservation. I’m – just to be able to sit down in a very different culture and listen and I think it’s very important for doctors to listen to their patients.

This interviewee was additionally committed to modeling the listening behavior of physicians and teaching these listening skills to others. When we further asked “What what do you think was specific about the way that [your mentors] spoke with patients and interacted with them?” the interviewee responded with clarity:        

Sitting back and not speaking and letting them talk about what’s important to them.

The interviewee conveyed that if you listen, people will tell you what is most important to them. They further argued that as decision-makers guiding the usage of far-reaching technologies, it is important to not make assumptions about what a particular community needs.

Similarly, in another interview, a molecular biologist described their experience setting up clinical trials and discussing the risks and benefits of an experimental treatment. This experience not only gave them a more concrete sense of what was at stake in the discussions held at the Napa Meeting, but also helped sensitize them towards the lived experiences of the patient communities that may be affected (for better or worse) by genome editing technology. When asked if experiences during their doctoral program, postdoc or work at a biotech firm, had prepared them for discussing genome editing and its implications, the molecular biologist responded:

Having been involved in therapeutic programs in which you’re discussing the pluses and minuses of therapies that can have side effects can prepare you for that. […] To me that was very helpful because it was a very concrete discussion. That conversation was not a like, “oh, I’m an academic and I wanna write a paper and someone’s going to read it and then enough.” […] [In a therapeutic program] the conversation was like, “we have a molecule, are we going to put it in people?” And if the answer is “yes,” like there is a living person on the other end that is going to take that molecule and [they are] going to have to live with the consequences positive and negative. […] 

            The distinction being drawn here between scientific work with concrete outcomes for people and work with solely academic outcomes, suggests that there are practical experiences that researchers at Universities may only have indirect knowledge of that are important for understanding how the products of science may affect others. As the interviewee further explained, the stakes of being unfamiliar with patient’s experiences are particularly high,

[My work at a biotech firm] has sort of prepared me at least a little bit for some of the discussion around therapeutic editing because different patient populations have wildly different ideas about gene editing. There are certain forms of inherited blindness where people are frankly insulted that you would call it a genetic disease, right? And I think rightly so. That’s their experience. It’s their disease. Why should we call this something that should be quote-unquote “corrected,” right?

In this case, prior experience with clinical trials alerted the researcher towards the heterogeneity of experiences of different patient populations. They further described how, in other interactions with patient advocates through public engagement, they were able to learn a great deal about the uniqueness of each patient group and their different views about genome editing. Here, the researcher additionally conveyed concern over the ableism that is often implicit in medical views of difference. They recounted how listening to perspectives from different patient communities led them to reflect on how procedurally safe genome editing can still cause harm in other ways.

3. Involvement in social movements.

The third non-technical form of expertise came from researchers’ political participation. While the recent fervor against the GOP’s “war on science” may give us ample evidence that politics and science don’t mix well, the role of social movements in the creation of scientific knowledge has been extensively documented by sociologists. For example, post World War II environmental movements changed the content, form and meaning of ecological research (Jamison 2006) and Gay Rights and AIDS activists helped steer the direction of biomedical research (Epstein 1996). What is less emphasized in these studies though, is how participation in social movements by scientists can impact their worldview and decision-making. When asked what personal experiences shaped how they thought of the process of decision-making around new biotech, one interviewee mentioned their engagement with political movements in the late 1960’s during anti-Vietnam War protests :

So I was in Berkeley in the late 60s…This is a time of a lot of social activity. Protests that went on against the Vietnam War in favor of civil rights. There was a lot of protest activity going on and I was involved in that to some extent, you know, I went on marches. I went door-to-door one summer in opposition to the Vietnam War…Um, so I had to you know- I had sort of a social equity outlook on life. All the way from my upbringing from college- and then at Berkeley you really couldn’t avoid being involved in some of these social issues.

This respondent went on to discuss how their commitments towards social equity shaped their decision-making around emerging technologies. In another interview, a respondent described how taking time off of their graduate program to work on a local election campaign motivated them to participate in science policy forums later in their career.

However, these example also suggests that how a scientist chooses to engage with social movements can have lasting effects on how they think of themselves as being a part of a larger community. If scientists participate unreflexively, social movements can fail to challenge individual’s to consider how the network building and activism they are doing affects themselves and may be excluding others from different communities.

To give a contemporary example, the March for Science (MfS) movement in January of 2017 protested against the Trump administration’s anti-science policies and actions. While the issues about science funding were urgent, MfS organizers failed to address language issues in MfS that were dismissive of the experience of marginalized communities in science. Whether or not a participant in MfS chose to critically engage in the movement, will influence how this individual sees the world and whether they intentionally or unintentionally reproduce inequities in science. By asking scientists to think about both their role in society and about the community of science itself, social movements provide a large quantity of knowledge and creativity that scientists can contribute to and use as a resource when making decisions and reflecting on the implications of emerging technologies.

The Value of Non-technical Expertise in Training

Many of the experiences that shaped our interviewees decision-making occurred during their early graduate and professional training. Despite the personal and professional value they found in these experiences, our interviewees noted the lack of support from their graduate mentors in their exploration of non-technical interests and a lack of incentives to participate more broadly in political endeavors during their training. While this may be changing for newer generations of scientists, this raises questions about how scientists in the natural and physical sciences are socialized into the broader scientific community, and the impact of that socialization on what they think of their political responsibilities are.

For example, a consensus study of the National Academies of Sciences, Engineering, and Medicine (2018) found that there is a lack of social and institutional support for activities located outside of the traditional realm of an individual’s discipline and argued for the creation of novel training pathways that could lead to holistic STEM research training. One way of creating more holistic STEM training programs noted by the study that our findings support would be to provide resources and structures to facilitate the connection between graduate training in the life sciences and fields, such as STS, sociology and philosophy. Exposure to these disciplines can help aspiring researchers grapple with the social interactions of their discipline and serve as additional tools for constructive debates around scientific issues. Promoting interdisciplinary collaboration may also help reduce stigma associated with non-traditional pathways to scientific training and provide easier channels to integrate professional development and internship opportunities into the curriculum.

The urgency of this current gap in training is apparent if you look at who is currently at the the decision making table. The committees and meetings for deliberation about the social and ethical issues of genome editing are almost exclusively constituted by senior scientists. These researchers are mainly conscripted into these roles because of their technical expertise and status in disciplinary networks. Historically, the academic institutions these scientists were trained in were not built to prepare scientists for making political decisions or for appreciating the social complexity and nuance that comes with the introduction of emergent technologies into society. In our third blog post we will explore the political stakes of this form of science governance, which are surprisingly high.


References:

Epstein, S. (1996). Impure science: AIDS, activism, and the politics of knowledge (Vol. 7). Univ of California Press.

Jamison, A. (2006). Social movements and science: Cultural appropriations of cognitive praxis. Science as Culture, 15(01), 45-59.

National Academies of Sciences, Engineering, and Medicine (2018) Graduate STEM Education for the 21st Century. Washington, DC: The National Academies Press. doi: https://doi.org/10.17226/25038.

Data Science and Expanding Our Sources of Ethical Inspiration

By Luke Stark & Anna Lauren Hoffmann, CTSP Fellows | Permalink

Steam rising from nuclear reactors

Photo by Mark Goebel

Recent public controversies regarding the collection, analysis, and publication of data sets about sensitive topics—from identity and sexuality to suicide and emotion—have helped push conversations around data ethics to the fore. In popular articles and emerging scholarly work (some of it supported by our backers at CTSP), scholars, practitioners and policymakers have begun to flesh out the longstanding conceptual and practical tensions expressed not only in the notion of “data ethics,” but in related categories such as “data science,” “big data,” and even plain old “data” itself.

Against this uncertain and controversial backdrop, what kind of ethical commitments might bind those who work with data—for example, researchers, analysts, and (of course) data scientists? One impulse might be to claim that the unprecedented size, scope, and attendant possibilities of so-called “big data” sets require a wholly new kind of ethics, one built with digital data’s particular affordances in mind from the start. Another impulse might be to suggest that even though “Big Data” seems new or even revolutionary, its ethical problems are not—after all, we’ve been dealing with issues like digital privacy for quite some time.

READ MORE

A User-Centered Perspective on Algorithmic Personalization

By Rena Coen, Emily Paul, Pavel Vanegas, and G.S. Hans, CTSP Fellows | Permalink

We conducted a survey using experimentally controlled vignettes to measure user attitudes about online personalization and develop an understanding of the factors that contribute to personalization being seen as unfair or discriminatory. Come learn more about these findings and hear from the Center for Democracy & Technology on the policy implications of this work at our event tonight!

What is online personalization?

Some of you may be familiar with a recent story, in which United Artists presented Facebook users with different movie trailers for the film Straight Outta Compton based on their race, or “ethnic affinity group,” which was determined based on users’ activity on the site.

This is just one example of online personalization, where content is tailored to users based on some user attribute. Such personalization can be beneficial to consumers but it can also have negative and discriminatory effects, as in the targeted trailers for Straight Outta Compton or Staples’ differential retail pricing based on zip code. Of course, not all personalization is discriminatory; there are examples of online personalization that many of us see as useful and have even come to expect. One example of this is providing location-based results for generic search terms like “coffee” or “movie showtimes.”

READ MORE

Design Wars: The FBI, Apple and hundreds of millions of phones

By Deirdre K. Mulligan and Nick Doty, UC Berkeley, School of Information | Permalink | Also posted to the Berkeley Blog

After forum-and fact-shopping and charting a course via the closed processes of district courts, the FBI has honed in on the case of the San Bernardino terrorist who killed 14 people, injured 22 and left an encrypted iPhone behind. The agency hopes the highly emotional and political nature of the case will provide a winning formula for establishing a legal precedent to compel electronic device manufacturers to help police by breaking into devices they’ve sold to the public.

The phone’s owner (the San Bernardino County Health Department) has given the government permission to break into the phone; the communications and information at issue belong to a deceased mass murderer; the assistance required, while substantial by Apple’s estimate, is not oppressive; the hack being requested is a software downgrade that enables a brute force attack on the crypto — an attack on the implementation rather than directly disabling encryption altogether and, the act under investigation is heinous.

But let’s not lose sight of the extraordinary nature of the power the government is asking the court to confer.READ MORE

Rough cuts on the incredibly interesting implications of Facebook’s Reactions

By Galen Panger, CTSP | Permalink

How do we express ourselves in social media, and how does that make other people feel? These are two questions at the very heart of social media research including, of course, the ill-fated Facebook experiment. Facebook Reactions are fascinating because they are, even more explicitly than the Facebook experiment, an intervention into our emotional lives.

Let me be clear that I support Facebook’s desire to overcome the emotional stuntedness of the Like button (don’t even get me started on the emotional stuntedness of the Poke button). I support the steps the company has taken to expand the Like button’s emotional repertoire, particularly in light of the company’s obvious desire to maintain its original simplicity. But as a choice about which emotional expressions and reactions to officially reward and sanction on Facebook, they are consequential. They explicitly present the company with the knotty challenge of determining the shape of Facebook’s emotional environment, and they have wide implications for the 1.04 billion of us who visit Facebook each day. Here are a few rough reactions to Facebook Reactions.

READ MORE

The need for interdisciplinary tech policy training

By Nick Doty, CTSP, with Richmond Wong, Anna Lauren Hoffman and Deirdre K. Mulligan | Permalink

Conversations about substantive tech policy issues — privacy-by-design, net neutrality, encryption policy, online consumer protection — frequently evoke questions of education and people. “How can we encourage privacy earlier in the design process?” becomes “How can we train and hire engineers and lawyers who understand both technical and legal aspects of privacy?” Or: “What can the Federal Trade Commission do to protect consumers from online fraud scams?” becomes “Who could we hire into an FTC bureau of technologists?” Over the past month, members of the I School community have participated in several events where these tech policy conversations have occurred:

  • Catalyzing Privacy by Design: fourth in a series of NSF-sponsored workshops, organized with the Computing Community Consortium, to develop a privacy by design research agenda
  • Workshop on Problems in the Public Interest: hosted by the Technology Science Research Collaboration Network at Harvard to generate new research questions
  • PrivacyCon: an event to bridge academic research and policymaking at the Federal Trade Commission

READ MORE

Ethical Pledges for Individuals and Collectives

By Andrew McConachie | Permalink

[Ed. note: As a follow-up to Robyn’s explanation of the proposed I School Pledge, Andrew McConachie provides some challenges regarding the effectiveness of pledges, and individual vs. collective action for ethical behavior in software development. We’re pleased to see this conversation continue and welcome further input; it will also be a topic of discussion in this week’s Catalyzing Privacy-by-Design workshop in Washington, DC. —npd]

I am conflicted about how effective individualized ethics are at creating ethical outcomes, and the extent to which individuals can be held accountable for the actions of a group. The I School Pledge is for individuals to take. It asks individuals to hold themselves accountable. However, most technology/software is produced as part of a team effort, usually in large organizations. Or, in the case of most open source software, it is produced through a collaborative effort with contributors acting both as individuals, and as members of contributing organizations. The structures of these organizations and communities play a fundamental role in what kind of software gets produced (cf. Conway’s Law, which focuses on internal communications structures), and what kinds of ethical outcomes eventuate.

READ MORE

Should Facebook watch out for our well-being?

By Galen Panger, CTSP | Permalink

Last year, when Facebook published the results of its emotional contagion experiment, it triggered a firestorm of criticism in the press and launched a minor cottage industry within academia around the ethical gray areas of Big Data research. What should count as ‘informed consent’ in massive experiments like Facebook’s? What are the obligations of Internet services to seek informed consent when experimentally intervening in the lives, emotions and behaviors of their users? Is there only an obligation when they want to publish in academic journals? These are not easy questions.

Perhaps more importantly, what are the obligations of these Internet services to users and their well-being more broadly?
 

Facebook's Infection

Credit: ‘Facebook’s Infection’ by ksayer1

READ MORE

A Pledge of Ethics for I School Graduates

By Robyn Perry, I School MIMS ’15 | Permalink

When you hear about Volkswagen engineers cheating emissions tests, or face recognition software that can’t “see” Black people, you start to wonder who is in charge here. Or more to the point, who is to blame?

Well, I just graduated from UC Berkeley’s School of Information Master of Information Management and Systems program (MIMS for short). My colleagues and I are the kind of people that are going to be making decisions about this stuff in all sorts of industries.

This post is about one way we might hold ourselves accountable to an ethical standard that we agree to by means of a pledge.

As you might imagine, we spend a significant part of our coursework thinking about how people think about technology, how people use technology, and how what we design can do a better job of not destroying the world.

READ MORE