Skip to main content

Supplemental Materials: Research Programs That Have Shaped Sustainability Science

An analysis of influential research programs that have shaped sustainability science. Includes summaries of their substantive focus, analysis of their “footprint” on the larger field of sustainability science, and guidance for keeping up with the individual research programs.

Published onSep 10, 2020
Supplemental Materials: Research Programs That Have Shaped Sustainability Science
·

Cite as: Hurt, Wyatt A., Alicia G. Harley, and William C. Clark. 2020. “Supplemental Materials: Research Programs That Have Shaped Sustainability Science.” In Sustainability Science: A Guide for Researchers, edited by Alicia G. Harley and William C. Clark, 1st ed. Retrieved from https://www.sustainabilityscience.org/pub/kzs257kc


The main purpose of this Supplementary Material is to explain how we arrived at the list of influential research programs listed in Table 1 of this Research Guide (see An Integrative Framework for Sustainability Science). In addition, we provide additional information about those programs: their substantive focus, their “footprint” on sustainability science, and how future contributions from each of them can be tracked by interested researchers. Readers interested in a deeper dive into the bibliometric analyses that underlie the information summarized here are encouraged to consult a longer working paper we have prepared (Hurt, Harley, and Clark 2020). Wyatt Hurt did much of the data collection and analysis for this SI, and is the lead author on the working paper.

We derived the final set of influential research programs provided in Table 1 of the Research Guide from a longer preliminary list of candidate research programs that emerged from our initial discussions with colleagues, existing surveys of the sustainability science literature, and searches in various databases. Two big findings emerged from that preliminary work. First, the argument of Bettencourt and Kaur (2011) that the field of sustainability science had “unified” around the year 2000 may have been somewhat premature: many of the research programs in our candidate list are still relatively isolated from one another and are only now beginning to collaborate and draw from one another’s insights. Second, much scholarship that experienced researchers see as being highly relevant to sustainable development does not describe itself as “sustainability science.” That work is missed by literature searches requiring only that some variant of the word “sustainable” appear in titles, abstracts, or key words. Most published bibliometric studies seeking to identify and analyze scholarly contributions to our understanding of sustainable development have adopted just such expedients and are therefore seriously incomplete (e.g., Bettencourt and Kaur 2011; Buter and Van Raan 2013; Kajikawa, Saito, and Takeuchi 2017; Fang et al. 2018).

The strategy we found most useful to address the sobering findings of our preliminary survey of research on sustainability may be summarized as follows:

  • We began by treating each candidate research program as a potential source of scholarship relevant to sustainability, regardless of whether its publications commonly used the term.

  • Next, we sought to design a set of search terms that would reliably identify the subset of works published in that program that qualified as “sustainability science” as we had defined the field in our Research Guide.1 These search terms, and our strategy more generally, could in principle be applied to any database, including the “everything (books+journals)” catalogs of holdings in particular libraries now being offered by an increasing number of universities.2 As a practical matter, we applied our terms only to the Web of Science (WoS), with all the advantages and limitations that use of this particular database entails.3 In particular, we searched all indexes in the Web of Science Core Collection, restricting results to English-language articles, books, and book chapters only. (Note that the Research Guide itself cites works from a larger range of sources, not all indexed by WoS.)

  • Our tuning of the search terms depended heavily on our own determination of whether the works a given combination of terms identified should qualify as “sustainability science.” We made the “qualification” judgment through our own (admittedly subjective) reading of the title, abstract, and often substantial fractions of the body of the text in question. We performed this judgment for the most recent 100 works identified by a given configuration of search terms.4 Most of the works that we “disqualified” through this close reading were clearly irrelevant to sustainable development. In addition, however, we disqualified a non-negligible number of works because they fell outside of the Pasteur’s Quadrant focus of our Research Guide, i.e., they were plausibly relevant to sustainability, but were too much Bohr’s Quadrant (basic research that may have provide foundations for sustainability studies but did not explicitly build sustainability arguments from those foundations) or too much Edison’s Quadrant (applications to particular situations with little to offer our search for generalizable knowledge).

  • We continued tuning the search terms until we achieved an acceptable balance of Type 1 and Type 2 errors. Our first step was to achieve a result in which Type 1 errors (works identified by the search that we ultimately disqualified) were less than 10% of all the items identified in the search. (Most of our initial search term combinations in fact yielded a Type 1 error rate of less than 5%.) To limit Type 2 errors (works that we judged to be qualified, and thus should have been identified by the search, but were not), we experimented with less restrictive variations of each search. Our goal was to drive the Type 2 error rate down while maintaining a Type 1 error rate of 10% or less. We refer in what follows to the resulting list as “qualified works,” meaning that—in our judgment—they qualify as works centered in Pasteur’s Quadrant and relevant to sustainable development. In short, they qualified as the “sustainability science” that was the focus of the Research Guide.

The results of applying our strategy are summarized in Table SM1. Here are some highlights:

  • What search terms are needed to identify the most important works in sustainability science (column B)? We ultimately designed a search query combination that achieved our desired Type 1/Type 2 balance for each individual research program we investigated. These search query combinations are provided in Column B of Table SM1. This methodology is imperfect. Nevertheless, we think it is a powerful and comprehensive approach to understanding the broad contours of research contributing to sustainability science. Our results confirm the sad fact we noted earlier: any search strategy based on variations of the word “sustainable” alone will both turn up a lot of false positives (e.g., “sustainable sales strategy”) and miss a significant fraction of the all the scholarship relevant to sustainable development. Indeed, about a third of the total set of qualified works returned by the search queries in Table SM1 would have been missed by “sustainability” searches prevalent among bibliometric analyses of sustainability science, underscoring the need for more complex search strategies. To be practical and effective these search strategies must be tailored to the individual research programs listed in Table SM1. We have found it useful to create weekly WoS alerts for each of these search queries as a relatively easy if imperfect way of having our attention called to new publications in each research program on a regular basis. Here is how: https://clarivate.libguides.com/woscc/saving. We hope our readers will find the search queries useful for similar or related purposes of their own. We welcome suggestions for how the search queries can be used, and improved, to support the pursuit of sustainability.

  • How broad is the emerging field of sustainability science (Column C)? The magnitude of each research program’s offerings to sustainability science (specifically, to the Pasteur’s Quadrant part of sustainability science) is shown in Column C: the count of qualified works in WoS identified by the search string in Column B for the period 2000–2019. Note that research programs that were included in our preliminary list, but that achieved very low scores on this metric, are excluded from this table and from the simpler version of it published in the Research Guide. In particular, new research programs that are just getting started may not yet score high enough to be included. Beyond these caveats, for all the reasons noted above, not much weight should be placed on the specific or perhaps even the relative counts listed in Column C: there are many limitations to the WoS database and different search queries would have produced different counts. Nonetheless, we were surprised by some of the results. And we suspect that most readers will find, as we did, that whichever research program(s) they consider to be their own “home” or starting place, there exist other programs that have produced a larger number of relevant offerings and therefore perhaps merit more attention than we are giving them in our respective efforts to shape a synoptic perspective on the emerging field of sustainability science.

  • Trends in number of qualified works in sustainability science (Column D): Trends in the number of qualified sustainability science works for each research program are shown in Column D. (The time period covered by our searches was usually set to cover only the period 2000–2019 because this Research Guide was meant to cover that period, not the broader history of scholarship relevant to sustainable development. For this column only we extended the time period back to 1980 to let us explore which programs started early and which started late.) The general surge in publications over the last decade is clear if not surprising: the overall scientific literature relevant to sustainable development is now doubling every 3–4 years. (Column D shades in blue for each research program the years during which the most recent 50% of all qualified works were published. For the qualified works from all research programs combined—i.e., for what might be called the field of sustainability science—the time trend is shown in Table SM4.) More interesting, perhaps, is the suggestion from the trends of the listed research programs that some have long contributed to sustainability science and continue to do so today, while others are relatively new arrivals and still others may have already peaked in their contributions. We conclude that the field of sustainability science is growing rapidly in numbers of publications but also changing rapidly in terms of which research programs contribute most to those publications.

  • How many journals must a scholar read to keep up with sustainability science (Column E)? The 5 journals that publish the greatest number of qualified works for each research program are listed in Column E. (Where there is a tie in fifth place, we list more than five journals). Note that publications in the research programs are spread across a very large number of journals: the most commonly used journal for each program rarely publishes as much of 10% of all its qualified works. For all research programs combined, only one journal appears in the list of “5 journals with the most qualified publications” for two thirds of the research programs, and only four journals appear for more than a one third (see Table SM2). The situation is only slightly less daunting when we turn to the list of journals cited in the Research Guide (see Table SM5). (These publications report what we judged to be some of the most significant and high-quality work in sustainability science. But to catch just one third of the articles cited in the Research Guide requires keeping up with 5 different journals; to catch one half requires 10 journals; to catch two thirds requires 20 different journals. We conclude that there is no “short list” of journals that will let a diligent reader keep up with the field of sustainability science. This stands on contrast to many scholarly fields in which where keeping up with two to three journals is sufficient to have a good grasp of the most important recent literature in the field.

  • How many organizations must a scholar follow to keep up with sustainability science (Column F)? The 5 organizations whose researchers publish the greatest number of qualified works for each research program are listed in column F of Table SM1. Note that this research relevant to sustainable development is spread across a very large number of organizations, most of which specialize in only a few of the research programs that have contributed to the field. In particular, the most prolific organization houses researchers who publish only 3% of the papers assembled through our search queries across all research programs (Table SM3). And any one organization is a major player in at most half of the research programs listed in Table SM1. We conclude that there is no “short list” of organizations that dominate research production across the multiple research programs contributing to sustainability science.

  • How influential are the various research programs that contribute to sustainability science (Column G)? The findings noted above have been based primarily on the total quantity of research output that our search strategies have identified as “qualifying” as, and contributing to, sustainability science. What we would most like to know, however, is the quality or influence of those programs. Col. G in Table SM1 reports a first step toward assessment of influence other than quantity. It does so by tabulating the number of citations to all of the works identified through our search queries in each of the listed research programs (Column C). The results, sadly, tell us little that we didn’t already know from the raw publication counts of Col. H: research programs that publish most get cited most. We conclude that there is no “short list” of research programs that dominate in the production of cited (and therefore, presumably, quality) research.

  • Contributions to this Research Guide (columns H and I): Columns H and I of Table SM1 are the bridge from the details of bibliometric analysis reported here to the Research Guide itself. Drawing on our close reading of the literature for each research program, we summarize in Col. H the contributions particular to each program that have, in our subjective judgment, done most to influence the course of contemporary sustainability science. As we note in the body of the Research Guide, all the research programs listed in Table SM1 have been converging in recent years. They increasingly share the synoptic view of key questions, elements, and relationships that we have highlighted in the Research Guide. Historically, however, each of the research programs listed here has brought something special to the party. It is those special contributions that we note in Col. H. Column I indicates the review paper(s) that we believe provides the best recent overview of each listed research program.

Table SM1: Research Program Bibliometrics: Publications relevant to sustainability science indexed in the Web of Science

If you would like to annotate or comment on this table, please click here.

If you would like a printer-friendly pdf version of Table SM1, please click here.

Table SM2: Which journals publish the greatest number of works relevant to sustainability science (across all research programs)

This table displays: (1) journal name; (2) the total contribution of each journal as the percent of all qualified works identified in all of our searches (i.e., summed across all research programs, N = 20,813 works); (3) the number (N) of research programs for which each journal is one of those in which authors most frequently publish (i.e. number of appearances in Col. E of Table SM1).

1) Journal

2) Total Contributions

(% out of 20,813 existing records)

3) N Research Programs

(see caption)

Agriculture, Ecosystems and Environment

0.52%

0

Ambio

0.39%

1

Annual Review of Environment and Resources

0.14%

1

Anthropocene Review

0.05%

1

Biodiversity and Conservation

0.24%

1

Current Opinion in Environmental Sustainability

0.29%

1

Ecological Economics

2.42%

9

Ecological Indicators

1.24%

1

Ecology and Society

1.47%

7

Ecosystem Services

1.65%

3

Energy Policy

0.62%

0

Energy Research & Social Science

0.47%

1

Environmental Health Perspectives

0.28%

1

Environmental Innovation and Societal Transition

0.51%

1

Environmental Justice

0.40%

1

Environmental Management

0.51%

0

Environmental Modelling Software

0.26%

3

Environmental and Resource Economics

0.20%

1

Environmental Science & Policy

0.72%

2

Forest Policy and Economics

0.58%

0

Geoforum

0.63%

1

Geografisk Tidsskrift Danish Journal of Geography

0.05%

1

Global Environmental Change

0.74%

3

Innovation: The European Journal of Social Science Research

0.07%

1

International Environmental Agreements: Politics, Law, and Economics

0.10%

1

International Forestry Review

0.29%

1

International Journal of Environmental Research and Public Health

0.66%

1

Journal of Cleaner Production

3.16%

5

Journal of Environmental Management

0.97%

2

Journal of Industrial Ecology

4.42%

3

Journal of Water Resources Planning and Management

0.05%

1

Land Use Policy

1.50%

2

Landscape and Urban Planning

0.70%

0

Local Environment

0.37%

1

Marine Policy

0.53%

1

PLOS ONE

0.70%

0

PNAS

0.55%

3

Regional Environmental Change

0.59%

0

Resources, Conservation & Recycling

0.75%

2

Science of the Total Environment

1.40%

2

Sustainability

4.14%

10

Sustainability Science

0.64%

4

Sustainable Development

0.24%

1

Sustainable Production and Consumption

0.05%

1

Technological Forecasting and Social Change

0.47%

2

Water Resources Research

0.08%

1

World Development

0.49%

1


Table SM3: Which organizations support authors responsible for the greatest number of qualified works (across all research programs)?

This table displays: (1) organization name; (2) the total contribution of each organization as the percent of works (articles, books, and book chapters) identified in our searches (i.e., summed across all research programs, N = 20,813 works); (3) the number (N) of research programs for which each organization is among those hosting researchers responsible for the greatest number of qualified works (i.e. number of appearances in Col. F of Table SM1.

1) Organization

2) Total Contribution

(% out of a total of 20,813 research programs)

3) N Research Programs

(see caption)

Arizona State University

1.94%

5

Australian National University

0.85%

1

Autonomous University of Barcelona

0.29%

2

Beijing Normal University

1.40%

1

Center for International Forestry Research

0.61%

1

Centre National De La Recherche Scientifique

1.30%

3

Chiang Mai University

0.17%

2

Chinese Academy of Sciences

1.07%

5

Clark University

0.36%

1

Colorado State University

0.86%

0

Commonwealth Scientific and Industrial Research Org.

0.06%

2

Cornell University

0.68%

0

Delft University of Technology

0.78%

1

Duke University

0.64%

0

Eindhoven University of Technology

0.28%

1

Erasmus University Rotterdam

0.70%

1

ETH Zurich

0.91%

1

Ghent University

0.62%

0

Harvard University

0.50%

0

Helmholtz Association

1.22%

1

Helmholtz Center for Environmental Research

0.87%

1

Humboldt University of Berlin

0.78%

1

Institute of Geographic Sciences and Natural Resources Research

0.79%

0

James Cook University

0.73%

0

Kobe University

0.05%

1

Kyushu University

0.32%

1

London School of Economics and Political Science

0.21%

1

Lund University

1.14%

1

McGill University

0.96%

0

Michigan State University

1.87%

1

Monash University

0.56%

0

National Autonomous University of Mexico

0.69%

0

National University of Singapore

0.50%

0

North Carolina State University

0.38%

1

Norwegian University of Science and Technology

0.64%

1

Ohio State University

0.88%

0

Oregon State University

0.70%

0

Peking University

0.51%

0

Penn State University

0.61%

0

Princeton University

0.26%

1

Royal Swedish Academy of Sciences

0.05%

1

Stanford University

1.20%

1

State University System of Florida

0.11%

1

Stockholm University

1.06%

2

Swedish University of Agricultural Sciences

0.95%

0

Swiss Federal Institute of Technology

0.68%

0

Texas A&M University

0.55%

0

The World Bank

0.36%

1

Tohoku University

0.18%

1

Tsinghua University

0.55%

0

United States Department of Agriculture

0.08%

1

Universite De Montpellier

0.26%

1

University of Arizona

0.63%

0

University of Berlin

0.04%

0

University of Bonn

0.57%

0

University of British Columbia

1.09%

0

University of California System

4.20%

8

University of Cambridge

0.96%

1

University of Cape Town

0.55%

0

University of Colorado

0.59%

0

University of Copenhagen

1.06%

2

University of East Anglia

0.73%

0

University of Edinburgh

0.66%

0

University of Exeter

0.53%

0

University of Florida

0.89%

0

University of Guelph

0.29%

1

University of Helsinki

0.80%

0

University of Illinois

0.79%

0

University of Leeds

1.48%

0

University of London

0.63%

4

University of Manchester

0.83%

1

University of Maryland

0.84%

0

University of Melbourne

0.89%

0

University of Michigan

1.31%

0

University of Minnesota

1.09%

0

University of North Carolina

0.69%

2

University of Oxford

1.12%

2

University of Pretoria

0.51%

0

University of Queensland

1.18%

0

University of Sao Paulo

0.58%

0

University of Sheffield

0.55%

0

University of Southampton

0.68%

0

University of St. Andrews

0.21%

1

University of Sussex

0.89%

2

University of Sydney

0.53%

0

University of Texas System

0.73%

1

University of the Chinese Academy of Sciences

0.96%

0

University of Tokyo

0.72%

0

University of Toronto

0.54%

0

University of Vermont

0.82%

0

University of Washington

0.66%

0

University of Waterloo

0.59%

1

University of Wisconsin

0.87%

0

University of Witwatersrand

0.51%

0

University System of Maryland

0.92%

1

U.S. Forest Service

0.53%

0

Utrecht University

0.85%

1

Vrije Universiteit Amsterdam

0.79%

1

Wageningen University & Research

2.42%

8

Yale University

0.91%

2

Table SM4: Growth in Total Annual Production of Qualified Works

This plot displays the number of qualified works published each year, summed across all research programs listed in Table SM1 (period covered: 1990 (year 0) to 2019 (year 29)). Columns are shaded blue for the years during which the most recent 50% of all qualified works were published.

Exponential fit curve: y = 9.360e .207(x) + 22.378

Residual standard error: 49.45 on 27 degrees of freedom


Table SM5: Journals Cited in the Research Guide (Clark & Harley 2020)

This table displays: (1) all journals cited by Clark & Harley 2020; (2) the count of times articles in those journals were cited in the Research Guide. The appearance of multiple Annual Review publications in the list reflects in part Clark and Harley’s effort to cite recent reviews of specialized topics relevant to sustainability science. We note that one of the authors of this SM (WCC) is a member of the editorial board of PNAS. We worked hard to assure that this did not bias our selection of which articles to cite in the Guide.

1) Journal

2) Count

Annual Review of Environment and Resources

25

PNAS

25

Ecology and Society

14

Global Environmental Change

11

Science

11

Nature Sustainability

10

Current Opinion in Environmental Sustainability

9

Sustainability

8

Annual Review of Resource Economics

7

World Development

7

Environmental Research Letters

6

Research Policy

6

Energy Research & Social Science

5

Environmental Innovation and Societal Transitions

5

Environmental Science & Policy

5

Journal of Cleaner Production

5

Sustainability Science

5

Annual Review of Political Science

4

Climatic Change

4

Ecological Economics

3

Ambio

2

Annals of the American Association of Geographers

2

Annual Review of Ecology, Evolution, and Systematics

2

Annual Review of Economics

2

Annual Review of Sociology

2

Earth System Governance

2

Ecosystems

2

Energy Policy

2

Environment and Development Economics

2

Environmental and Resource Economics

2

Nature

2

Policy and Society

2

Policy Sciences

2

Technological Forecasting and Social Change

2

Agricultural Systems

1

American University Law Review

1

Annual Review of Anthropology

1

Behavioral Science

1

Bioscience

1

Bulletin of the American Mathematical Society

1

Capitalism Nature Socialism

1

Daedalus: Journal of the American Academy of Arts and Science

1

Development

1

Development and Change

1

Earth System Dynamics

1

Elementa: Science of the Anthropocene

1

Environment, Development and Sustainability

1

Environment: Science and Policy for Sustainable Development

1

Environmental Management

1

Environmental Policy and Governance

1

Environmental Science and Pollution Research

1

European Economic Review

1

Global Environmental Politics

1

Global Governance

1

Global Policy

1

Global Sustainability

1

Globalization and Health

1

Health Research Policy and Systems

1

Hydrogeology Journal

1

Industrial and Corporate Change

1

Intergenerational Equity: Environmental and Cultural Concerns

1

International Journal of the Commons

1

International Journal of Water Resources Development

1

JAMA Internal Medicine

1

Journal of Economic Issues

1

Journal of Economic Literature

1

Journal of Environmental Management

1

Journal of Environmental Planning and Management

1

Journal of Environmental Policy & Planning

1

Journal of Global Security Studies

1

Journal of Human Development and Capabilities

1

Journal of Population Economics

1

Landscape and Urban Planning

1

Nature Climate Change

1

Nature Ecology & Evolution

1

Nature Reviews Earth & Environment

1

Organization Science

1

Oxford Review of Economic Policy

1

PLOS ONE

1

Policy Studies

1

Policy Studies Journal

1

Science, Technology, & Human Values

1

Slate Magazine

1

Social Psychological and Personality Science

1

Strategic Management Journal

1

Sustainable Development

1

Technology Analysis & Strategic Management

1

Territory, Politics, Governance

1

The Economic History Review

1

The Journal of Peasant Studies

1

The Lancet Infectious Diseases

1

The Public Interest

1

Trends in Ecology & Evolution

1

Weather, Climate, and Society

1

References

Bettencourt, Luís M. A., and Jasleen Kaur. 2011. “Evolution and Structure of Sustainability Science.” Proceedings of the National Academy of Sciences 108 (49): 19540–45. https://doi.org/10.1073/pnas.1102712108.

Buter, R. K., and A. F. J. Van Raan. 2013. “Identification and Analysis of the Highly Cited Knowledge Base of Sustainability Science.” Sustainability Science 8 (2): 253–67. https://doi.org/10.1007/s11625-012-0185-1.

Clark, William C., and Alicia G. Harley. 2020. “Sustainability Science: Toward a Synthesis.” Annual Review of Environment and Resources 45 (accepted preprint). http://nrs.harvard.edu/urn-3:HUL.InstRepos:42660129.

Fang, Xuening, Bingbing Zhou, Xingyue Tu, Qun Ma, and Jianguo Wu. 2018. “‘What Kind of a Science Is Sustainability Science?’ An Evidence-Based Reexamination.” Sustainability 10 (5): 1478. https://doi.org/10.3390/su10051478.

Garfield, Eugene. 1955. “Citation Indexes for Science: A New Dimension in Documentation through Association of Ideas.” Science 122 (3159): 108–11. https://doi.org/10.1126/science.122.3159.108.

Gusenbauer, Michael. 2019. “Google Scholar to Overshadow Them All? Comparing the Sizes of 12 Academic Search Engines and Bibliographic Databases.” Scientometrics 118 (1): 177–214. https://doi.org/10.1007/s11192-018-2958-5.

Hurt, Wyatt W., Alicia G. Harley, and William C. Clark. 2020. “Searching for Sustainability Science: Practical Bibliometric and Curatorial Approaches.” 2020–02. Sustainability Science Program Working Paper. Cambridge, Massachusetts: John F. Kennedy School of Government, Harvard University.

Kajikawa, Yuya, Osamu Saito, and Kazuhiko Takeuchi. 2017. “Academic Landscape of 10 Years of Sustainability Science.” Sustainability Science 12 (6): 869–73. https://doi.org/10.1007/s11625-017-0477-6.

Stokes, Donald E. 1997. Pasteur’s Quadrant: Basic Science and Technological Innovation. Washington, DC: Brookings Institution Press.

Comments
0
comment
No comments here
Why not start the discussion?