Looking Inside Simulation's Black Box

11 February 2019

At the 2018 Science and Art of Simulation Workshop, philosophers, sociologists, historians, and computer scientists sought to develop a more rigorous framework for evaluating the trustworthiness of computational science.

Science and mathematics have long been celebrated for their ability to shed light on previously unknown features of the natural world. Through logic, experiment, proof, and self-critique within the scientific community, science was seen as a march toward increasing clarity.

In recent years, however, the rise of computer simulation and machine learning has complicated this view of science. The ability to create and apply complex algorithms running on computers has undoubtedly opened up new kinds of opportunities for scientific discovery and technological innovation. Nevertheless, as "black boxes" whose inner workings are partly impossible to observe and understand, even the scientists who use them often do not know exactly how their computer programs generate their results. In such a nontransparent situation, how can we be confident that we can trust the simulations our technologies produce?

Philosopher Paul Humphreys coined the term "epistemic opacity" in 2004 to describe this situation. Since then the concept has spurred much discussion among philosophers, sociologists, and simulation scientists, although developing a consistent, comprehensive, and theoretically grounded understanding of epistemic opacity in all of its forms has been difficult. Doing so would not only be intellectually significant but could also have the practical effect of providing a useful framework for evaluating the results of simulation and machine learning. Such knowledge could also potentially help to improve the design of new applications in these fields.

On November 28-30, 2018, the Philosophy of Science & Technology of Computer Simulation Program at the High-Performance Computing Center Stuttgart (HLRS) hosted an interdisciplinary conference to take steps toward this goal. Bringing together an international community of investigators, the event sought to achieve a more refined understanding of how the simulation sciences differ from more traditional disciplines. The event also sought to formulate a deeper understanding of the many sources and flavors of epistemic opacity.

The meeting spotlighted sources of epistemic opacity that are unique to computer-intensive research. In particular, the combination of advanced mathematics and information technology in high-performance computing constitute a methodological opacity. Here, scientists must rely on results produced by algorithms whose mechanistic operation is impossible to observe directly.

In addition, social characteristics of the scientific communities that make up interdisciplinary teams affect almost all research in science and engineering. In large-scale projects, for example, individual scientists might bring expertise in applying one specific type of analysis, but lack in-depth knowledge about the workings of other parts of the project on which their own work relies.

Because science relies on the comprehensibility of the scientific method, there was a broad consensus at the meeting that epistemic opacity touches upon fundamental questions of what science is and how the scientific method should be understood. The rise computer-intensive science, leading to new sources of opacity, suggests that the scientific method may be undergoing further change.

Proceedings from the 2018 Science and Art of Simulation workshop will be published by Springer in 2019. A book gathering papers from the first SAS workshop is available here.

Christopher Williams