Time is a flat circle: optimizing digital investigations

Analysts often emphasize that investigations are laborious and time-intensive. And it's true that investigations are highly complex, requiring a great deal of diligence.

Data cleaning and collation

For many analysts, data cleaning, collation, and organization are the greatest time sinks slowing down their investigations. Analysts routinely work with disparate data sets in a wide variety of formats, often lacking the tools they need to effectively centralize their data and then analyze and manipulate it efficiently. 

Analysts still commonly use Excel spreadsheets (CSVs, XLS documents, and so on) to store and aggregate their data. In some ways, these documents are convenient to use as they are almost universally compatible and are surprisingly powerful considering their ostensible simplicity. In addition, analysts often have to investigate a wide variety of files, such as Word and PDF documents, audio files, and more; these are also commonly stored in spreadsheets as links.

However, spreadsheets have several significant downsides. Firstly, they are often unwieldy for even moderately complex tasks, and some degree of knowledge is required to properly clean and organize data when using them. Some newcomers to the market have innovated and supported advanced features like lambda functions and regular expressions to make spreadsheets more effective for cleaning. But even relatively basic features, like duplicate detection and deletion, formatting URLs, and others require analysts to write scripts of varying complexity – often negating any time saved.

These features, while certainly a step in the right direction, don’t completely solve the issue of slow investigation workflows. Many analysts can recount harrowing stories of manually editing data for hours and endlessly converting file formats due to inadequate software.

The requirements of various end customers when it comes to reports and data sets also complicate the jobs of analysts. In some cases, analysts must spend more time adapting their data sets and tweaking commas and periods to their end customers’ requirements than they spend creating the given data sets.

Data integration

The ubiquity of spreadsheets for data storage and cleaning often makes them an analyst’s first stop for integrating various data sources. Analysts often store disparate data sets in different tabs in either the same or different spreadsheets, making specific data points difficult to find and even more difficult to manipulate quickly. Analysts also have to find, access, and juggle siloed and disparate sources of information – ranging from on-prem databases and servers to specific online directories and services.

Moreover, the relevant knowledge is sometimes held by a person or a group of people and must then be released and integrated into the investigation workflow. These inefficiencies in data storage and communication often force analysts to convert data sets into universal formats manually, creating potentially huge time sinks for investigations.

Cross-referencing

One of the key functions of most analysts is cross-checking and cross-referencing information. This can take more time than expected, as checking one piece of information against a sea of others requires meticulous organization in spreadsheets or other data storage formats. Additionally, it requires a strong ability to assess credibility and relevance. Furthermore, having access to the creation date or origin of given information is often key to effectively utilizing it as well. Curating different data, determining its relation to other information, and describing/storing these relationships can all be extremely time-consuming.

Utilizing stored data

Spreadsheets can be useful when working with some data types but not all of them. Data formats including images, videos, documents, and more all have to be effectively stored and utilized. The reports that analysts create based on this data – and any associated conclusions, assumptions, and insights – must also be stored somewhere accessible. However, spreadsheets are inadequate for this purpose. 

Additionally, identifying and exporting data from some systems to external spreadsheets can be laborious and time-consuming. Beyond this, analysts may also need to eventually re-update internal systems with new spreadsheets. This can be a tedious, manual process that can eat into an analyst’s time, and drain their mental energy, adding to the inefficiencies.

In short, both raw data and the insights that analysts draw from it must be easily findable, accessible, and usable – but inefficient and complex data storage methods make this very challenging.

Data visualization

Data visualization is one of the later stages of an investigation and involves turning investigative findings into easy-to-understand charts, graphs, and other representations that can enable end clients to better understand the information and make a decision based on what they see.

We can’t expect all investigative analysts to be data scientists and analysts as well – it’s neither their role nor their area of expertise. Data visualization is key to quality reporting, but often takes a great deal of time — whether investigators are creating graphs in spreadsheets or working with complex external tools. Manually creating graphs and charts and other methods of beautifying reports takes a lot of time, and end customers often are unsatisfied with or less engaged by “ugly” or bare-bones reporting.

So, to satisfy customers, analysts are commonly forced to spend time making visualizations more aesthetically pleasing – something that is far outside their expertise and usual remit.

Collaboration

Collaborating with others – be they analysts or managers – on investigations can be a Sisyphean effort at times. Every element of collaboration is in danger of becoming needlessly difficult, including sharing data and files securely , tracking changes from multiple teams and individuals, delegating parts of an investigation, and more.

Additionally, collaboration vectors are often needlessly diverse; on different occasions, information might be transmitted by emails, phone calls, messages, printed documents sent by courier, or via a host of other methods. Such (often arbitrary) inconsistencies in communication require extra effort, lead to miscommunication, and ultimately waste time.

Report writing

Report writing is one of the most important – if not the most important – elements of an investigation. Despite this importance, inefficient processes often still take up time that could be used more productively. Complex investigations include potentially dozens, hundreds, or even thousands of disparate data points that can’t always be easily placed in pre-made templates. 

To deal with this, analysts must justify their work – sometimes conducted months earlier – in long, highly tailored reports, eating up hours of valuable time and often requiring skills outside of the analysts’ expertise. Similarly, integrating multimedia content into reporting, as well as visual aids and graphs, can further waste time, since simply placing them in a document can be a cumbersome task.