SHARP (Strengthening Human Adaptive Reasoning and Problem Solving): A case study for highlighting the role of independent test and evaluation in government funded research
Dimitrios Donavos1, Alexis Jeannotte2, Amber Sprenger3, Chrissy Thuy-Diem Vu3; 1Booz Allen Hamilton, 2IARPA/ODNI (US Government), 3MITRE Corporation
SHARP is a multi-site and cross-team, collaborative research effort aimed at investigating whether fluid intelligence (Gf) can be enhanced through interventions focused on behavioral, cognitive, and neurophysiological outcome measures in high performing adults funded by Intelligence Advanced Research Projects Activity (IARPA). Programs at IARPA are structured as competitions, where only teams with the strongest results continue throughout the program period. In designing candidate interventions for enhancing Gf on the SHARP program, teams were required to test a sufficiently powered subject cohort and employ research designs that utilized active controls and placebo controlled, double-blinded intervention groups. These interventions were independently verified using a battery of tests developed by a third-party organization (MITRE). This battery is comprised of measures that are available for use by other researchers and were developed to have a number of features, including strong construct validity evidence, parallel versions to enable pretest-posttest comparisons, and appropriate difficulty for a high ability population. As the primary evaluation metric, the battery is used to evaluate whether teams meet predetermined targets for improving Gf in a high-performing cohort, relative to an active control group. Against a backdrop of a broader replication crisis in the scientific community, the value and outcomes of a testing approach that employs a rigorous and common set of test and evaluation methods and metrics will be further discussed.
Topic Area: METHODS: Other