Can Machines Help Us Understand Orcas?

How researchers are decoding the culture of killer whales.

The largest member of the dolphin family, orcas are highly vocal marine mammals found in all regions of the ocean. They have rich social lives and frequently communicate within their family group through touch, behavior, and vocalizations. Like bats, orcas use echolocation to navigate and locate prey. They click and whistle and produce discrete, pulsed calls of varying tone and harmony. They may squeak, buzz, squeal, growl, chirp, honk and trumpet. But what are they saying?

Researchers from the Pattern Recognition Lab of the Friedrich-Alexander University in Germany hope to answer this question by applying deep machine learning to orca calls recorded off the western coast of Canada. This month, the Pattern Recognition Lab team will be floating off the coast of Canada for their fourth field trip. This time, they will try to “come up with a first hypotheses” of orca dialects, the lab’s lead researcher, Christian Bergler, said via email recently, while en route to the expedition. Bergler and his team will also be fine-tuning a system that matches orca calls to individuals. The pattern recognition technique, for which the lab is named, analyses images of sound frequencies called spectrograms and is the same artificial intelligence used to decipher words in a foreign human language. But the research will not only bolster our understanding of orcas. It could also help us understand animal culture, and how it differs (and doesn’t) from our own.

Northeast Pacific orcas are some of the most closely studied wild marine mammals in the world. Since the 1970s, researchers have been listening to their vocal repertoires and observing their unique social behavior. Canadian scientist John KB Ford first catalogued the dialects of orca families in British Columbia. Led by a matriarch, orcas spend their whole lives in closely-related family groups, which may contain up to five generations of her descendants. Each family group has a distinct dialect, socially learned, with up to 17 discrete calls that are unique to the group and are passed down through the generations. These vocal traditions are stable over many decades and have been described by Ford as part of orca culture. Ford’s research confirmed that, like humans, orcas have both geographic and social dialects. They lead complex social lives, with multi-generational cultural traditions.

Such extensive studies have produced an abundance of data — including a large bio-acoustic archive spanning 25 years and over 20,000 hours of recordings. Friedrich-Alexander University researchers have trained algorithms to identify and extract millions of calls from the raw audio, sorting a data set that would normally take years, or decades, to manually organize. Additional enhancement processes remove background noise and can separate multiple, overlapping orca calls. These preliminary steps are improving analyses of orca communication, even as researchers and their hydrophones record more data.

A lone orca in the coastal waters of British Columbia, Canada. Researchers are hoping to understand what these marine mammals’ vocalizations mean by applying deep machine learning to orca calls recorded off the western coast of Canada. Photo by Thomas Lipke/Unsplash.

The Pattern Recognition Lab now aims to develop a system that recognizes the dialects of all orca populations found in the Northeast Pacific — transients, offshore, northern and southern residents. To train a machine-learning system to distinguish between any kind of orca and recognize their dialects, researchers need sufficient vocalization and behavioral data from each family group and individual. So, from a machine-learning perspective, although an abundance of photos and underwater sound recordings of Northeast Pacific orcas exists, a data set that associates calls with behavior is missing.

In previous work, the Pattern Recognition Lab sought to address this data gap by teaming up with behavioral biologists to record individual calls and social behavior of orcas in the coastal waters of British Columbia. Over the course of three summers, field expeditions were led both by Bergler, who specializes in speech processing and understanding, and by Elmar Noeth, who is an expert in automatic speech recognition for human languages. After each expedition, the team analyzed video footage of interesting vocalization recordings, along with handwritten notes of orca behavior. To identify linguistic patterns, they mapped reoccurring acoustic sequences to the behavior of the group.

The current challenge is to associate calls with individual orcas. While hydrophones are able to localize acoustic data at the family group level, they are not sophisticated enough to determine which individual orca is vocalizing. And as orcas spend very little time at the surface, where they are easily observable, researchers are working on a better underwater localization system that can collect acoustic data where the vocalizing individual is known.

“The real challenge with orcas and all marine mammals is how much time they spend underwater,” says Monika Wieland Shields, director of the Orca Behavior Institute, in Washington state’s San Juan Islands. “We are getting such a small glimpse.” Shields points out that limited opportunities to observe orcas may lead to an overly simplistic categorization of their behavior. “We can look at broad behavioral contexts,” she says, like “they seem to be traveling” resting, foraging, “but those are super broad categories that are nuanced behaviorally [and] you don’t see what’s going on underwater in terms of their positioning relative to one another [and] what their actual interactions are with each other, and their environment.”

The next big breakthrough in orca linguistic research will come from pairing vocalizations with more nuanced behavioral observations of family groups and individual orcas, she says. This includes research with the use of drones, which is already providing additional behavioral insights.

Some machine-learning tools do offer assistance with the task of associating behavioral data with individual orcas, as each orca from the northern and southern resident groups, as well as the transients, can be identified by their fin shape and skin pigmentation. The Pattern Recognition Lab developed a deep learning photo-ID tool kit to automate identification of individual orcas from large photo and video-data sets. This helps to rapidly determine which individuals within a family group are present during an interesting behavioral or vocal event, which can be especially helpful for the silent-hunting transient orcas, who feed only on marine mammals. This is one of the many machine-learning tools the researchers have developed and continue to use to analyze large orca data-sets.

What this all means is that we are one step closer to understanding orcas, and their discrete calls. This includes sounds that can’t be heard by humans. Shields suspects these small details are important for orca communication, and wonders whether some harmonic structures create a badge of social identity, identifying the orca and their place of origin. Still, she doubts that they are saying the same thing over and over, given how complex they are. “It sort of seems like there must be more going on,” she says, “and we haven’t figured out what angle to think about that yet.”

Get the Journal in your inbox.
Sign up for our biweekly newsletter.

You Make Our Work Possible

You Make Our Work Possible

We don’t have a paywall because, as a nonprofit publication, our mission is to inform, educate and inspire action to protect our living world. Which is why we rely on readers like you for support. If you believe in the work we do, please consider making a tax-deductible year-end donation to our Green Journalism Fund.

Get the Journal in your inbox.
Sign up for our biweekly newsletter.

The Latest

COP27 Was Billed as the First “Food COP.” Did it Deliver?

Big Ag’s influence was evident in a final text that crowded out small-scale farmers’ demands.

Amrita Gupta

Make Ecocide an International Crime and Other Legal Ideas to Help Save the Planet

A list of the five most promising legal steps we can take to help fight climate change.

Steven Donziger The Guardian

Jamaica’s ‘Cockpit Country’ Faces Growing Threats from Mining Interests

The island nation’s government has begun granting mining leases in its rare limestone karst rainforest.

Gladstone Taylor

On Grief and Celestial Bodies

In Review: An Orchid Astronomy​

ER Zarevich

Africa’s ‘Fossil Fallacy’ Will Devastate Climate, Wreck Communities, Says Report

“The ‘dash for gas’ is nothing more than a short-sighted strategy to profit from the energy crisis.”

Christopher Bonasia

Saving the Dragon’s Blood

Facing a range of threats, from droughts and storms to goats and war, these iconic trees are staging a patient comeback.

Jess Craig