Highlight
Proactive Responses to Safety and Security Risks
Achievement/Results
This IGERT was created to improve responses to emerging technologies. This project integrates four elements to improve proactive responses to risks that may be posed by emerging technologies, through improved design, testing and demonstration for risk mitigation.
- A design and testing component based on formal analytic literature is providing a basis for design and testing under conditions of uncertainty and complexity. This element extends established work on design and testing of systems by Daniel Hastings and Frank Field.
- A testing and demonstration component based on the qualitative policy studies literature is being used to improve the substance and perception of risk management under conditions of uncertainty and controversy. This element extends work by Lawrence McCray and Kenneth Oye.
- Historical case studies are being used to identify and evaluate methods of design, testing and demonstration, enriching conventional analytic methods. Historian Merritt Roe Smith is leading a team examining technologies commonly seen as posing risks, such as the steam boiler, elevator, nuclear reactor, and early recombinant DNA labs.
- Experimental testbeds on current technologies are being used to apply and assess principles of design, testing and demonstration. These include Synthetic Biology Chassis with biological engineers from NSF SynBERC and Future Internet Protocols and Standards with computer scientist David Clark of NSF FIND. If funding materializes, we may add a testbed on the Nuclear Fuel Cycle with physicist Geoffrey Forden.
Methods of design and testing draw on formal analytic literature to characterize uncertainty and develop optimal strategies of testing under varying priors. Our efforts build on Daniel Hastings’ work on methods for characterizing uncertainties that affect complex systems, projecting resulting risks and strategies to mitigate risks; and on methods for the proper design of experiments to test complex systems (McManus and Hastings, 2006). Our research takes on the challenge of incorporating broader social criteria into the design and testing of technical systems. (McManus, Hastings and Warmkessel 2004; McManus, Richards 2009). Non-traditional design criteria, such as flexibility, survivability, and robustness, manifest specific aspects of contextual uncertainty that are of increasing importance in the design of technical systems (Ross, Hastings et al. 2004). Hastings and IGERT trainee Matthew Richards have developed methods for quantifying the effects of design choices on nontraditional criteria including safety, security and sustainability and for presenting tradeoffs across nontraditional and traditional performance indices (Richards 2009).
Methods of Testing and Demonstrating: Technologists have a long record of responding to concerns over risks with claims that redesigned technical systems are safe, secure and sustainable, with societal responses to claims ranging from extreme skepticism to blind acceptance. The political importance of testing and demonstration methods increases as one moves from simple to complex systems, from lower to higher degrees of uncertainty, and from lower to higher degrees of controversy. Our team is evaluating testing and demonstration methods that engage regulators, insurers, NGO and other actors. This testing and demonstration component draws on a qualitative policy studies literature to develop processes suited to the credible and effective assessment of risks. This extends work on politics of risk assessment by Lawrence McCray, including work on credible evaluation of risks (McCray 2004) and on adaptive risk management by McCray and Oye with Arthur Petersen of the Netherlands Environmental Assessment Agency (McCray, Oye, Petersen, 2009). We are now setting up experiments on novel institutional means, including red-teaming and other decision aiding strategies. In our post IGERT work, we plan to extend our analysis to include vetted impact-statements, the role of ombudspersons, independent safety/security audits, rigorous post-marketing assessment, and any other methods we find in high-risk spheres that might be imported into the design process.
Methods of Integrating Design, Testing and Demonstration: We are examining how political context affects redesign efforts, how institutional arrangements affect definition of risks, and how optimal testing theories might inform development of regulatory test standards. In one example, we found that an NRC panel on redesign of the space shuttle booster recommended a rigorous program of testing of redesigned boosters, with systematic attention to how information generated by testing might be used to improve designs. NASA rejected NRC recommendations, fearing that tests that appeared to be failures could lead to termination of the shuttle. These complex interaction effects and responses are being identified through empirical work. The project features retrospective studies on technologies associated with significant perceptions of risks; and prospective testbeds on Biological Chassis and Internet Protocols and Standards to drive research and training methods and to provide proofs of concept in areas of ongoing concern. With support from SynBERC, we have negotiated a post-IGERT partnership with the Woodrow Wilson Center of Smithsonian to set up red teaming exercises with regulators and NGOs on SynBio safety issues. Other testbeds will depend on funding of proposals under review and on the evolution of a large scale DoD project on cybersecurity.
Address Goals
DISCOVERY AND LEARNING: FEDERAL NEEDS AND PROJECTED EFFECTS
This activity addresses explicit demands from NSF and other agencies for the proactive design and demonstration of emerging technologies for safety, security and sustainability. With exponential advances in DNA synthesis, design and assembly, the NSABB and NIH RAC Roundtable on Synthetic Biology asked “What kinds of efforts have been, or are being taken, to engineer containment into synthetic systems/organisms?” and asked engineers to rise to the challenge. DoD and IAEA call for redesign of the nuclear fuel cycle to decrease proliferation vulnerabilities posed by the enrichment programs of nations of concern. NIST and GAO are among many organizations calling for redesign of internet protocols and standards to improve the terms of tradeoffs across security, privacy and sharing. NRC noted that “nanotechnology-related risk research needs to be proactive – identifying possible risks and ways to mitigate risks before the technology has widespread commercial presence” (NRC 2008).
The NSF created a Center for Biological and Environmental Nanotechnology at Rice University to assess and address nanotechnology related environmental, health and safety risks in advance of crises. Historically, demands for redesign followed manifest safety, security or sustainability failures. Today, the challenge is to design technologies for safety, security and sustainability without the clarity and urgency generated by catastrophic failure or a series of accidents.
(1) Why are demands for improved methods of risk assessment and management through design increasing? Technological capabilities are growing at exponential rates in many realms. Extraordinary improvements in the cost and speed of transforming, storing, and transporting information are now matched by revolutionary changes in the speed and cost of sequencing and synthesizing DNA. The exponential increase in speed and exponential decrease in cost of DNA sequencing and DNA synthesis were based on integration of developments in an even broader range of fields, including electrical engineering, molecular biology, and genetic engineering. Change in energy levels in particle accelerators and doubling times of fusion products follow similar curves. (Koh 2006, Kurzweil 2001, Carlson 2003, 2008, ITER 2008.) Methods of risk assessment and management have not kept pace with underlying technological change. Moore’s Law on computing and Carlson’s Curve on DNA sequencing and synthesis do not have analogs in risk assessment and mitigation. Yet the application and acceptance of technologies rests on development of methods of design to contain risks, development of tests to demonstrate credibly that risks have been reduced to acceptable levels, and codification of methods of design and testing into public regulations, industrial protocols, and insurance standards.
(2) How will this project help meet these demands? At a conceptual level, the project integrates insights of systems engineers on formal analytic methods of design, testing and experimentation and insights of social scientists on the design of the governmental and private institutions that assess and manage risks. As noted above, at an empirical level, the project includes retrospective work by historians of technology on 19th century and 20th century cases of risk management through design with prospective testbeds that engage technologists working on safety of synthetic biology, the security and privacy of future internet standards, and the proliferation resistance of nuclear fuel enrichment. Formal methods may be modified to take account of insights gleaned from observed practices in retrospectives (Bazerman et al 1998). Finally, integrated methods of designing, testing and demonstrating based on both formal analytic and institutional work will be refined and evaluated through testbeds on synthetic biology, future internet protocols and standards, and nuclear fuel enrichment.
(3) What larger effects may follow from this work? As noted at the outset, this project responds to demands from government for methods of design, testing and demonstration in emerging technologies. The testbeds on synthetic biology, future internet protocols and standards, and nuclear fuel enrichment directly address demands from the NSABB/RAC, NIST, and IAEA. It is hoped that the methods developed through this project and evaluated using testbeds may applied to risk assessment and mitigation problems in other emerging technologies, including nanotechnologies; and to risk assessment and management problems associated with more traditional technologies where risks have not been systematically evaluated or addressed. These direct effects will be supplemented by effects on education. At the doctoral level, this includes teaching research methods developed in this project to advanced graduate students in ESD, Political Science, and STS. At the undergraduate and masters levels, this includes integrating teaching materials on design, testing and demonstration for safety into the core MIT design class taught by Daniel Frey, a course that serves as a model for design classes nationwide; the core MIT class on science, technology and public policy taught by PI Kenneth Oye, and into a textbook on public policy Oye is writing with IGERT trainee Lindsay; and the core MIT course on biological engineering and synthetic biology for use by participants in the International Genetically Engineered Machine competition (iGEM), an event that drew over 1000 undergraduate and high school participants from over 80 universities competing to design biological parts. Finally, PI Daniel Hastings serves as Dean of Undergraduate Education at MIT and is committed to incorporating integrated approaches to the analysis of technical, social and ethical issues into the curriculum of all MIT students.
