Hello, this genderless chatbot is not sorry

Jacqueline Feldman is an artificial intelligence user experience designer.
In the world of tech acronyms, that would be the futuristic-sounding “AI UX designer,” but we like her description of the work better: she creates characters through language.
Feldman, who moved to Crown Heights after completing a Fulbright Fellowship in Paris where she was reporting on squatter communities, works at Kasisto, a spinoff of SRI, the Menlo Park-based inventor of Siri. At Kasisto, Feldman designed Kai, a chatbot that assists in managing bank accounts. Her work was recently featured on Engadget.
We spoke with Feldman about how gender stereotypes play into developing bots and the delicate line between designing a bot that’s approachable yet not too human-like.

###

This interview has been condensed and edited for clarity.

How did you get into AI?
I was catching up with old friends because I had just moved back, and I had a friend from childhood who was a programmer at Kasisto. He told me they were looking for someone to design a personality for this bot. He knew about my interest in writing and reporting, and I think, in particular, in creating a character through language. So he encouraged me to apply.
It struck me as a fascinating challenge. I hadn’t had much experience in tech, but I loved the idea of writing for an AI personality and trying to figure out how it should express itself.
How did you go about designing Kai’s personality?
I first designed the personality at a relatively high level, envisioning what charateristics would be most helpful. I also wanted to make it engaging in a way that I thought would be fun to write. I wanted it to be humanlike, but not a human, to express itself in a way that seemed friendly but authoritative.
Quickly, I noticed that a lot of AI assistants were assigned a female gender. That became something I thought about a lot.

Did you consciously design Kai in reaction to gender roles and stereotypes?
I did have gender in mind. I had in mind that it would be called “it,” not “she.” I felt that by avoiding some behaviors that women in our culture are socialized to do by sexism — the things Amy Schumer brings up in her sketches — it would do its job well.
For example, when I got on the job, there were provisional error messages in place that had already been written. They all started off with an apology. But in most of those cases, Kai has not done anything wrong. It’s simply that a person has typed something totally random or misspelled or [asked something] totally off topic in relation to what Kai has been trained to do.
What was your process of writing for the bot like?
I wrote a lot of phrases around Kai’s functionality. I wrote a lot of less functional moments in conversation, too: I created goodbyes, thank-you’s. A lot of people thank Kai, which I think reflects well on it because it’s an inanimate object. I added personality elements: jokes, answers to questions that are specifically bot-like, not necessarily humanlike.
I was really happy to see just this morning someone ask, “Do you play Pokémon?” And Kai answered, “As a text-only entity, I stay away from sports. Your next question?”
How did you ensure that Kai was approachable without trying to make it seem too human-like?
I do think it’s important that Kai demonstrates awareness that it’s a bot and not a human. There are some funny self-aware jokes where if someone puts it on the spot and asks if it’s a real human, it becomes a bit bashful or defensive. Similarly, if you ask Kai’s favorite food or whether it’s a boy or a girl. I had fun with those jokes.
But I think there’s a more serious reason to signal that it’s a bot and not a human. The bot can be so helpful that it might elicit private or tricky questions from a human. There was a study about the appropriate responses to crises such as rape or suicide. In those instances, it’s important that bots are programmed to recommend human help. I equipped Kai to connect to the Gamblers Anonymous help line.
What other projects are you working on now? Will you continue to work in AI?
I am continuing to work with Kai. I’m also working on other projects for our clients, as well as on other writing projects. The writing that I’ve been able to do with Kai has really interested me, so I think will continue in this field for a while.
I hope that there’s a similar curiosity, sensibility and creativity that underlies Kai and my other writing projects, although the process is quite different. I’ve begun to do more and more writing about bots as well.
Read Feldman’s “This Is What a Feminist Looks Like” on Real Life Mag.