simongoring / ecahm_metrics

A repository for the EarthCube All Hands Meeting Metrics Presentation

EarthCube Metrics: Key Considerations

Goring SJ. 2018. EarthCube Metrics: Key Considerations. EarthCube All-Hands Meeting. Arlington, VA.

Contributions

This work was made possible thanks to contributions from a number of individuals, including members of the EarthCube Engagement Committee and Scott Peckham. This work was funded in part by the National Science Foundation through the EarthCube Program.

This section requires editing and does not currently reflect the full scope of contributions to this project.

About the Repository

This repository contains a presenation, made using reveal.js for the EarthCube All Hands Meeting in Arlington, VA. The presentation can be run using any browser, either by pointing the browser to goring.org, or by cloning the repository and opening the index.html file locally.

Key Concepts

Assesing program success is complicated, particularly when research programs are large and are focused on Transformation as opposed to Production. Goals that assess production can be measured in the short term, for example "How many papers were published". EarthCube seeks to transform the geosciences in the 21st Century, as such it is expected that many of the goals will not produce measurable outcomes until later in the decade, or century. As such few of the commonplace metrics of success can be directly assessed.

Key Findings

By examining NSF award data we show that researcher grant size increases by approximately 33% following the first EarthCube Grant, while research networks remain similar in size (indicating higher value awards, when controlling for the number of collaborators). Researchers continue to apply to the same pool of Directorates within NSF, but more careful analysis is warranted.

Key Conclusions

To better assess metrics at a program level:

• Researchers should be encouraged to use standard reporting for workshops, publications, whitepapers and presentations.
• Individuals working in scientometrics and social sciences dshould be engaged to help provide guidance for assessment of impacts
• NSF annual award reporting focuses on impacts for: * Principal discipline * Other disciplines * Human resource development * Physical infrastructure * Institutional resources * Information resources * Technology transfer * Societal impacts
• We suggest a focus on:
  • Disciplinary Leadership
  • Academic Leadership
  • Policy and Management Outcomes
  • Funding Success
  • Partner Brokering
  • Interdisciplinary Brokering
  • Mentoring and Training
  • Public Outreach
  • Knowledge Generation
  • Data and Product Creation

Additional Project Elements

cql files in the folder ./cql can be used to query the Throughput Database. The database is not currently publically available.

The files in ./resources are used to plot a d3.js map of linked award categories in the NSF or EC schema.