dhimmel / scopus

User-friendly Scopus and Journal Metrics data

Tidying and mapping Scopus and Journal Metrics data

This repository creates user-friendly (tidy) TSVs of data from Scopus and Journal Metrics and converts data to NLM journal IDs for PubMed integration. Data pulled from Scopus include journal subject areas and open access status. Data pulled from Journal Metrics include journal three measures (CiteScore, SJR, SNIP) of journal prestige and a Scopus–ISSN mapping.

Execution is performed by running notebooks in the following order:

• 1.process-titles.ipynb to process the raw Scopus title list
• 2.process-metrics.ipynb to process the raw Journal Metric download
• 3.pubmed.ipynb to convert Scopus IDs to NLM journal IDs

Scopus titles

The data directory contains the following tidy outputs:

• titles.tsv: IDs and names for titles in Scopus
• title-attributes.tsv: attributes for Scopus titles such as open access status, publisher, and active status (excludes conference proceedings)
• publishers.tsv: number of journals per publisher as well as URL-friendly slugs. Redundant or misspelled publisher names can be manually fixed in publisher-name-patches.tsv.
• title-top-levels.tsv: top-level subject categories for each Scopus title
• asjc-codes.tsv: ASJC (All Science Journal Classification) code definitions
• subject-areas.tsv: ASJC subject areas for each Scopus title

Scopus mappings

The data directory contains the following tidy outputs:

• issn.tsv: a mapping between Scopus titles and ISSNs, including linking ISSNs.
• pubmed-map.tsv: a Scopus–NLM journal mapping

Journal metrics

• metrics.tsv.gz: metrics for Scopus journals
• pubmed-metrics.tsv.gz: metrics for PubMed journals

Source and version info

This repository is built from the following publicly-available inputs in download:

• extlistJuly2021.xlsx: Scopus title list (from "Download Scopus Source List" at source)
• CiteScore 2011-2020 new methodology - May 2021.xlsb: Journal Metrics
• pubmed-journals.tsv: PubMed journal information (source via process-nlm-catalog.ipynb)
• 20210912.ISSN-to-ISSN-L.txt.gz: The "ISSN-L matching table" is extracted and compressed from issnltables.zip which is available upon request from ISSN.

Environment

This repository uses conda to manage its environment as specified in environment.yml. Install the environment with:

conda env create --file=environment.yml

Then use conda activate scopus and conda deactivate to activate or deactivate the environment.

License

All original work in this repository is dedicated to the public domain under CC0 1.0 Universal. Note that this repository incorporates publicly available datasets that were not explicitly released with a public license. The authors of this repository claim no ownership of this content.

scopus's Issues

CiteScore dumps outdated

Example, from metrics:

21206   2016    CiteScore       13.33
21206   2016    SJR     18.13
21206   2016    SNIP    8.039

while https://www.scopus.com/sourceid/21206: for 2016 CiteScore is 49.2.

Looks like the score formula was updated and backfilled?

Update to the January 2017 release of CiteScore

https://journalmetrics.scopus.com/ has been updated to use CiteScore. At this time, the latest data is available at https://journalmetrics.scopus.com/downloads/metrics/CiteScore_Metrics_2011-2105_Download_19_Jan2017.xlsx

Compile journal homepage links for each Scopus source (journal)

The Scopus source download doesn't contain journal homepage links. According to a Twitter conversation, the best way is to use the Scopus API to lookup homepages for ISSNs. For example, PeerJ has a source-id of 21100239256 and an ISSN of 2167-8359. To get the homepage, we can do:

curl --silent \
  https://api.elsevier.com/content/serial/title/issn/2167-8359?httpAccept=application/json \
  | python -m json.tool

Which returns:

{
    "serial-metadata-response": {
        "entry": [
            {
                "@_fa": "true",
                "dc:publisher": "PeerJ",
                "dc:title": "PeerJ",
                "link": [
                    {
                        "@_fa": "true",
                        "@href": "https://www.scopus.com/source/sourceInfo.url?sourceId=21100239256",
                        "@ref": "scopus-source"
                    },
                    {
                        "@_fa": "true",
                        "@href": "http://PeerJ.com",
                        "@ref": "homepage"
                    }
                ],
                "message": "A full list of Scopus titles can be found on http://info.scopus.sciverse.com",
                "openArchiveArticle": null,
                "openaccess": "1",
                "openaccessArticle": true,
                "openaccessStartDate": null,
                "openaccessType": null,
                "prism:issn": "2167-8359",
                "prism:url": "https://api.elsevier.com/content/serial/title/issn/2167-8359"
            }
        ],
        "link": [
            {
                "@_fa": "true",
                "@href": "https://api.elsevier.com/content/serial/title/issn/2167-8359",
                "@ref": "self",
                "@type": "application/json"
            }
        ]
    }
}

Get ready for some API calls!

idea to try Scopus API client

HI @dhimmel - I made a lightweight Scopus API client, maybe it is useful for anything in this repo? Maybe a double-check on information retrieved?

The client can retrieve journal information based on ISSN or electronic ISSN, works with key from the publisher, repo here:
https://gitlab.com/epogrebnyak/scopus

Use ISSN-L to improve journal mapping

Sorry if this is a dumb newbie suggestion, but for matching journals in different datasets via ISSN, I think adding ISSN-Ls to each dataset, then matching on that, might catch more overlap. I've seen cases where WoS and Scopus have chosen different ISSNs for the same journal. The conversion table can be obtained here:
http://www.issn.org/services/online-services/access-to-issn-l-table/
or I suppose could be scraped from individual entries.