See no evil, hear no evil: siloed trust and safety teams

The work of trust and safety teams is anything but straightforward. As newcomers in a nascent industry, it’s no surprise that trust and safety teams aren’t always fully integrated with their firms’ business operations.

Trust and safety teams must navigate the complex internal bureaucracy of large technology firms. Simultaneously, they need to deal with an increasingly complex and limiting regulatory landscape, all while protecting their user bases and platforms from constant emerging and existing threats.

The teams must carry out the juggling act of protecting their platforms while simultaneously ensuring regulatory compliance and optimal business operations. Often this needs to be done while legitimate internal barriers restrict their work.

This leaves trust and safety teams thinly spread, leading to several significant issues:

Team structure

Trust and safety teams are increasingly spanning multiple functions, departments, cultures, and languages, all dispersed globally.

This dispersion and decentralization is important to ensure effective threat protection throughout the platform. However, this same decentralization can be detrimental to the effective use of data throughout an organization’s trust and safety team. Time zone and language differences between teams can also reduce the utilization of data, and in some cases may prevent the use of relevant data in key investigations.

Data persistency, integrity, and accessibility are all key to optimizing trust and safety teams’ productivity. When teams store their own data and reports disparately, it can make it difficult to utilize data effectively. Many trust and safety departments are reliant on actively sharing data and reports from team to team, with less-than-ideal visibility into what other teams are doing. 

This lack of visibility harms productivity and reduces the ability to collaborate against emerging threats.

Additionally, many trust and safety teams are multidisciplinary, integrating qualitative and quantitative specialists as well as subject matter experts. The associated wide range of responsibilities, investigations, and interventions  requires a vast array of data to be used. This, in turn, further reduces the ability of teams in other threat domains to effectively use all of their potential data resources across the organization.

Platform information sharing

Trust and safety teams face many challenges when trying to effectively use their existing data to protect their user bases and platforms.

Internal data, however, is only one part of the puzzle, and its role is becoming less significant. Increasingly capable threat actors will leverage any asset they can, be it online or offline, to achieve their goals to the detriment of user and platform safety. Trust and safety teams must therefore expand their scope and visibility beyond malicious behavior that is happening solely on their platform.

This cooperation, however, is not always handled optimally. Several factors limit the potential viability of cross-platform information sharing. These include the potential business impact of sharing sensitive user information, regulatory and PR risks, and more.

Bureaucracy can also have a negative impact on information sharing. Internal bureaucracy, combined with legacy systems and information compartmentalization, can delay internal and external information sharing. Different formats for data storage and different data points stored can also delay the process, especially when shared information doesn’t match existing data storage formats.

Third-party information sharing

Trust and safety teams often cooperate with investigative firms and vendors, academic organizations, think tanks, journalists, newsrooms, NGOs, and other organizations to identify and remove harmful content and threats.

This cross-functional and interdisciplinary cooperation is crucial to the success of trust and safety teams. Cooperation has proved to be useful in numerous cases, as demonstrated by the quarterly reports of Google’s Threat Analysis Group (TAG). These reports show that tips received from Meta, Mandiant, Graphika, and other sources led to malign network disruption.

Similarly, public-sector organizations such as DFRLab, the Institute for Strategic Dialogue, and others cooperate with social media platforms and trust and safety teams. Their reports are often published publicly and can be useful for trust and safety teams. However, integrating their qualitative and quantitative reporting into existing trust and safety workflows can be difficult without predefined, automated ways to ingest the data.

Cooperating with other external actors is also crucial. These actors can range from academic partners to law enforcement agencies, regulatory bodies, and more. Trust and safety teams must be able to effectively handle and export relevant data in collaboration with trusted external partners, and ideally be able to do so in automated ways.