This repository contains the datasets and evaluation results of our study. For a detailed overview regarding the provided materials, please refer to README.md.

Context

Spotify for Developers offers a wide range of possibilities to utilize the extensive catalog of Spotify data. One of them are the audio features calculated for each song and made available via the official Spotify Web API.

There are several datasets on Kaggle published in the past containing similar data. However, as of December 2018 this dataset is the most recent one (latest data retrieval was on 3rd December 2018) as well as the biggest one (more than 116k unique songs).

More information about the collection process as well as some additional information can be found in this github repo.

Content

Each song (row) has values for artist name, track name, track id and the audio features itself (for more information about the audio features check out this doc from Spotify).

Additionally, there is also a popularity feature included in this dataset. Please note that Spotify recalculates this value based on the number of plays the track receives so it might not be correct value anymore when you access the data.

Acknowledgements

Credit goes entirely to Spotify for providing this data via their Web API.

DataForSEO Labs API offers three powerful keyword research algorithms and historical keyword data:

• Related Keywords from the “searches related to” element of Google SERP. • Keyword Suggestions that match the specified seed keyword with additional words before, after, or within the seed key phrase. • Keyword Ideas that fall into the same category as specified seed keywords. • Historical Search Volume with current cost-per-click, and competition values.

Based on in-market categories of Google Ads, you can get keyword ideas from the relevant Categories For Domain and discover relevant Keywords For Categories. You can also obtain Top Google Searches with AdWords and Bing Ads metrics, product categories, and Google SERP data.

You will find well-rounded ways to scout the competitors:

• Domain Whois Overview with ranking and traffic info from organic and paid search. • Ranked Keywords that any domain or URL has positions for in SERP. • SERP Competitors and the rankings they hold for the keywords you specify. • Competitors Domain with a full overview of its rankings and traffic from organic and paid search. • Domain Intersection keywords for which both specified domains rank within the same SERPs. • Subdomains for the target domain you specify along with the ranking distribution across organic and paid search. • Relevant Pages of the specified domain with rankings and traffic data. • Domain Rank Overview with ranking and traffic data from organic and paid search. • Historical Rank Overview with historical data on rankings and traffic of the specified domain from organic and paid search. • Page Intersection keywords for which the specified pages rank within the same SERP.

All DataForSEO Labs API endpoints function in the Live mode. This means you will be provided with the results in response right after sending the necessary parameters with a POST request.

The limit is 2000 API calls per minute, however, you can contact our support team if your project requires higher rates.

We offer well-rounded API documentation, GUI for API usage control, comprehensive client libraries for different programming languages, free sandbox API testing, ad hoc integration, and deployment support.

We have a pay-as-you-go pricing model. You simply add funds to your account and use them to get data. The account balance doesn't expire.

This dataset provides data on Metallica's music, retrieved via Spotify's API. It was inspired by Jarred Priester's Metallica dataset (https://www.kaggle.com/datasets/jarredpriester/metallica-spotify-dataset/data), but includes more recent data and additional features.

Data included:

• All Metallica tracks from all albums published on Spotify.
• Metallica studio albums only (Some have two version e. g. remastered and not remastered. I've only included one of those versions for each.)

Metallica is one of the most successful bands in music history. This dataset allows for analysis of various audio features and metadata associated with their tracks. Additionally, it introduces an album popularity metric, which differs from track popularity. Album popularity reflects the overall performance of the album on Spotify, offering a broader view than individual track performance.

Dataset Columns:

• name – Name of the track.
• album – Name of the album the track is from.
• release_date – Release date of the album (YYYY-MM-DD format).
• track_number – Track's position within the album.
• id – Spotify ID for the track.
• uri – Spotify URI for the track.
• acousticness – A confidence measure (0.0 to 1.0) indicating the likelihood that the track is acoustic.
• danceability – A measure (0.0 to 1.0) describing how suitable a track is for dancing, based on factors such as tempo and rhythm stability.
• energy – A measure (0.0 to 1.0) indicating the intensity and activity of the track.
• instrumentalness – A measure (0.0 to 1.0) predicting whether the track contains no vocals.
• liveness – A measure (0.0 to 1.0) detecting the presence of an audience, indicating if the track is likely a live recording.
• loudness – The overall loudness of the track, measured in decibels (dB).
• speechiness – A measure (0.0 to 1.0) detecting the presence of spoken words. Higher values indicate more spoken-word content.
• tempo – Estimated tempo of the track in beats per minute (BPM).
• valence – A measure (0.0 to 1.0) describing the musical positiveness of the track.
• track_popularity – Popularity of the track, from 0 to 100, as determined by Spotify's engagement data.
• album_popularity – Popularity of the entire album, from 0 to 100, calculated based on overall album performance.
• duration_ms – Duration of the track in milliseconds.

Possible Use Cases:

• Audio feature analysis – Compare and analyze various audio features such as danceability, energy, or acousticness across different albums or time periods.
• Album vs. track performance – Explore differences between track popularity and album popularity to understand how individual songs perform relative to their albums.
• Music genre classification – Use audio features to classify tracks into sub-genres of Metallica's music or metal in general.
• Album evolution – Study how Metallica’s musical style has evolved over time by analyzing the changes in audio features and popularity across albums.
• Correlation analysis – Investigate relationships between different audio features and how they correlate with popularity (both track and album).
• Recommendation systems – Build or refine recommendation algorithms based on specific Metallica track attributes.

Spotify API Reference:

For further information about these audio features and how they are calculated, see the official Spotify API documentation.

https://developer.spotify.com/documentation/web-api/reference/get-audio-features

https://developer.spotify.com/documentation/web-api/reference/get-an-album

CIMIS data is available to the public free of charge via a web Application Programming Interface (API). The CIMIS Web API delivers data over the REST protocol from an enterprise production platform. The system provides reference evapotranspiration (ETo) and weather data from the CIMIS Weather Station Network and the Spatial CIMIS System. Spatial CIMIS provides daily maps of ETo and solar radiation (Rs) data at 2-km grid by coupling remotely sensed satellite data with point measurements from the CIMIS weather stations. In summary, the data provided through the CIMIS Web API is comprised by a) Weather and ETo data registered at the CIMIS Weather Station Network (more than 150 stations located throughout the state of California and b) Spatial CIMIS System data that provides statewide ETo and solar radiation (Rs) data as well as aeraged ETo by zip-codes. The RESTful HTTP services reach a broader range of clients; including Wi-Fi aware irrigation smart controllers as well as browser and mobile applications, all while expanding the delivery options by providing data in either JSON or XML formats.

The Statec publishes a significant amount of statistical data on its LUSTAT database. All this data is accessible via this web portal but also via a REST API, which is based in particular on the [SDMX] standard(https://sdmx.org/): Documentation of the REST API This API is complemented by a faceted search API in metadata: Documentation of the faceted search API

Examples:

— the list of tables made available by STATEC on LUSTAT can be obtained via a GET request on this URL: ‘https://lustat.statec.lu/rest/dataflow/LU1/all/all’ — by knowing the identifier of a dataflow, it is possible to recover your data as a CSV file using the cURL tool on the command line: “'Curl -o myfile.csv -X GET -H”Accept: application/vnd.sdmx.data+csv;urn=true;file=true;labels=both“ https://lustat.statec.lu/rest/data/LU1,ID_DATAFLOW/all?dimensionAtObservation=AllDimensions”’ In this command line, it is necessary to replace ID_DATAFLOW with the relevant dataflow identifier, e.g. ‘DF_D7100’

Discover the booming market for online software documentation tools! This in-depth analysis reveals a $5 billion market in 2025, projected to grow at a 15% CAGR through 2033. Learn about key drivers, trends, and top players like Atlassian & Bit.ai. Get insights to improve your documentation strategy.

The global Help Authoring Tools (HAT) market is experiencing robust growth, projected to reach a significant valuation by 2025. Driven by the increasing demand for comprehensive and accessible documentation across diverse industries, the market is anticipated to expand at a Compound Annual Growth Rate (CAGR) of approximately 9-11% between 2025 and 2033. This upward trajectory is fueled by the growing need for efficient content creation, management, and delivery solutions, particularly in the software development, IT services, and manufacturing sectors. The adoption of cloud-based HAT solutions is a dominant trend, offering scalability, flexibility, and cost-effectiveness, which appeals to both large enterprises and small to medium-sized businesses (SMEs). Furthermore, the continuous evolution of digital platforms and the proliferation of online self-service options necessitate sophisticated tools for generating user guides, knowledge bases, and API documentation. The market landscape is characterized by a competitive environment with numerous players offering a wide range of functionalities, from basic text editing to advanced features like single-sourcing, translation management, and integration with content management systems. While the demand for user-friendly and collaborative HAT is on the rise, potential restraints include the initial implementation costs for some advanced solutions and the challenge of training users to effectively leverage the full capabilities of these tools. However, the overwhelming benefits of enhanced customer support, reduced training burdens, and improved product adoption are expected to outweigh these challenges. Regional analysis indicates that North America and Europe are leading markets, owing to early adoption of technology and the presence of major software vendors. The Asia Pacific region is demonstrating significant growth potential, driven by rapid digitalization and the expansion of the technology sector in countries like China and India.

The Reference Data as a Service (RDaaS) API provides a list of codesets, classifications, and concordances that are used within Statistics Canada. These resources are shared to help harmonize data, enabling better interdepartmental data integration and analysis. This dataset provides an updated version of the StatCan RDaaS API specification, originally part of the Government of Canada’s GC API Store, which permanently closed on September 29th, 2023. The archived version of the original API specification can be accessed via the Wayback Machine . The specification has been updated to the OpenAPI 3.0 (Swagger 3) standard, enabling use of current tools and features for API exploration and integration. Key interactive features of the updated specification include: * Try-It-Out Functionality: Allows a user to interact with API endpoints directly from the documentation in their browser, submitting test requests and viewing live responses. * Interactive Parameter Input: Simplifies experimentation with filters and parameters to explore API behavior. * Schema Visualization: Provides clear representations of request and response structures.

Snapshot img

API Management Market Size 2025-2029

The API management market size is forecast to increase by USD 3.75 billion at a CAGR of 12.3% between 2024 and 2029.

The market is experiencing significant growth, driven by the increasing adoption of digital payment solutions and the proliferation of digital wallets. However, challenges persist, including poor internet connectivity in developing countries, which can hinder the adoption and effective implementation of Api Management solutions. Companies must navigate these challenges to capitalize on the market's potential. Strategies such as investing in offline solutions and partnering with local providers can help overcome connectivity issues and expand market reach.
Additionally, focusing on security and scalability will be crucial, as businesses demand reliable and secure Api Management solutions to support their digital initiatives. These trends reflect the digital transformation underway in various industries, as businesses seek to enhance customer experience and streamline operations. Overall, the market presents opportunities for innovation and growth, with companies that address the unique challenges of this dynamic landscape poised to succeed.

What will be the Size of the API Management Market during the forecast period?

Explore in-depth regional segment analysis with market size data - historical 2019-2023 and forecasts 2025-2029 - in the full report.
Request Free Sample

The market is experiencing significant innovation, with a focus on enhancing API Return on Investment (ROI) through multi-cloud API adoption and API-driven development. API maturity is on the rise, driving the need for advanced API logging, performance benchmarking, and usage analytics. API interoperability and standardization are crucial to addressing integration challenges in complex API ecosystems. API observability and developer experience are becoming key differentiators, with the emergence of API documentation generators and debugging tools. API adoption rates continue to grow, fueled by the increasing use of composite and hybrid cloud APIs, serverless functions, and microservices orchestration.

The market is experiencing significant growth, driven by the increasing adoption of digital payment solutions and the proliferation of digital wallets. API platform comparisons and compliance are essential for businesses navigating the diverse landscape of API offerings. API monetization strategies, such as API-led connectivity and edge computing APIs, are gaining traction. API evolution is ongoing, with a shift towards API-first design and headless CMS integration. API usage patterns are evolving, requiring new testing frameworks and security measures to address API performance optimization and vulnerabilities. Ultimately, API governance policies and discovery tools are essential for managing the complexities of API consumption and ensuring compliance in the dynamic API market.

How is this API Management Industry segmented?

The api management industry research report provides comprehensive data (region-wise segment analysis), with forecasts and estimates in 'USD million' for the period 2025-2029, as well as historical data from 2019-2023 for the following segments.

Deployment

  Cloud
  On-premises


Solution

  API gateways
  API lifecycle management
  API security
  API analytics and monitoring
  API developer portals


End-user

  Large enterprises
  SMEs


Geography

  North America

    US
    Canada


  Europe

    France
    Germany
    UK


  Middle East and Africa

    UAE


  APAC

    China
    India
    Japan


  South America

    Brazil


  Rest of World (ROW)

By Deployment Insights

The cloud segment is estimated to witness significant growth during the forecast period. The market is experiencing significant growth, driven by the digital transformation sweeping across industries. Cloud-based API solutions dominate the market, enabling seamless communication and data transfer between applications and the cloud. This segment's dominance is attributed to the proliferation of IoT and Big Data, which enhance application interfaces for superior customer experiences. Additionally, the increasing awareness of security vulnerabilities and the demand for automation have fueled the market's expansion in sectors like BFSI, e-commerce, healthcare and life sciences, education, and retail. Cloud APIs facilitate the integration of various cloud and on-premises applications, simplifying API provisioning, activation, setup, monitoring, and troubleshooting for developers and administrators.

Agile development methodologies, such as DevOps and CI/CD, have further accelerated the adoption of cloud APIs. APIs have become essential components of modern application architectures, including microservices, event-driven, and real-time systems. GraphQL APIs and service meshes have emerged as popu

DataForSEO will land you with accurate data for a SERP monitoring solution. In particular, our SERP API provides data from:

• Google Organic search, Maps, News, and Images tabs in vertical search
• Bing Organic and Local Pack search
• Yahoo, Yandex, Baidu, and Naver search

For each of the search engines, we support all possible locations. You can set any keyword, location, and language, as well as define additional parameters, e.g. time frame, category, number of results.

You can set the device and the OS that you want to obtain SERP results for. We support Android/iOS for mobile and Windows/macOS for desktop.

We can supply you with all organic, paid, and extra Google SERP elements, including featured snippet, answer box, knowledge graph, local pack, map, people also ask, people also search, and more.

We offer well-rounded API documentation, GUI for API usage control, comprehensive client libraries for different programming languages, free sandbox API testing, ad hoc integration, and deployment support.

We have a pay-as-you-go pricing model. You simply add funds to your account and use them to get data. The account balance doesn't expire.

WONDER online databases include county-level Compressed Mortality (death certificates) since 1979; county-level Multiple Cause of Death (death certificates) since 1999; county-level Natality (birth certificates) since 1995; county-level Linked Birth / Death records (linked birth-death certificates) since 1995; state & large metro-level United States Cancer Statistics mortality (death certificates) since 1999; state & large metro-level United States Cancer Statistics incidence (cancer registry cases) since 1999; state and metro-level Online Tuberculosis Information System (TB case reports) since 1993; state-level Sexually Transmitted Disease Morbidity (case reports) since 1984; state-level Vaccine Adverse Event Reporting system (adverse reaction case reports) since 1990; county-level population estimates since 1970. The WONDER web server also hosts the Data2010 system with state-level data for compliance with Healthy People 2010 goals since 1998; the National Notifiable Disease Surveillance System weekly provisional case reports since 1996; the 122 Cities Mortality Reporting System weekly death reports since 1996; the Prevention Guidelines database (book in electronic format) published 1998; the Scientific Data Archives (public use data sets and documentation); and links to other online data sources on the "Topics" page.

This list contains the list of standards and technical specifications analysed in the APIs4DGov study "Web API landscape: relevant general purpose ICT standards, technical specifications and terms". Notice that the list does not include a set of documents which are considered of (i) general purpose for the Web and (ii) consolidated background knowledge of the reader (e.g. HTTP, JSON, XML, URI, SOA, ROA, RDF, etc.).

Each document is classified accordingly to the following rationale:

• Name: extended name (with acronym, if available).

• TS/S: we distinguish the documents into two main categories: (1) “Technical Specification” and (2) “Standard”. The definitions of the two terms are in use in official and technical documents, including the ones of CEN, IEC, ISO and Open Geospatial Consortium (OGC). For the purposes of this dataset, we choose the definitions proposed by the OGC: (1) "Specification" or "Technical Specification" (TS): "a document written by a consortium, vendor, or user that specifies a technological area with a well-defined scope, primarily for use by developers as a guide to implementation. A specification is not necessarily a formal standard"; (2) "Standard" (S): "a document that specifies a technological area with a well-defined scope, usually by a formal standardisation body and process".

• Category: each document is classified by its functional specification (Resource Representation, Protocol), Security (Authentication, Authorisation), Usability (Documentation, Design), Test, Performance, and Licence. See section 2.2 for a description of each category.

• Short Description: a short description of the TS/S.

• Link: URL of online document describing the TS/S.

• API Type: RPC or REST, both if not specified.

• Initial Release: the year when the TS/S was proposed the first time (where not available the most probable year, calculated by additional desk research, was given).

• By: the organisation (i.e. standard body, consortium, vendor) or individual that proposes the standard.

Currently, users can either view this data directly in a web browser by accessing the OGC SensorThings API endpoints, though this can be confusing to users who do not understand the SensorThings API (https://newmexicowaterdata.org/faq/#sensorthingsapi) structure which organizes sensor data through interconnected entities like Things, Locations, Datastreams, and Observations. Users who have some programming knowledge can also query this standardized sensor data with the Python programming language following this tutorial (https://developer.newmexicowaterdata.org/help), or perform CRUD operations using the comprehensive API documentation (https://developers.sensorup.com/docs/). Development is currently underway for applications that more easily allow general users to query and visualize this environmental monitoring data from the New Mexico Bureau of Geology and Mineral Resources without requiring technical knowledge of the underlying API structure.

ProgrammableWeb Dataset

Overview

This dataset contains structured information about APIs, mashups, and categories from ProgrammableWeb, one of the most comprehensive directories of web APIs. The data has been extracted from a MySQL database and converted into CSV format for easy use in data analysis and machine learning applications.

Dataset Contents

The dataset is composed of multiple CSV files, each representing a different aspect of the API ecosystem:

1. Category.csv

Contains information about API categories. - ID: Unique identifier for the category. - Name: Name of the category. - PwURL: ProgrammableWeb URL for the category. - Amount: Count of APIs in this category (approximate).

2. ApiSketch.csv

Stores basic details about APIs before full data retrieval. - Name: API name. - PwURL: API URL on ProgrammableWeb. - Description: Short API description. - CategoryName: Primary category of the API. - CategoryURL: URL of the category. - SubmitDate: Date the API was submitted.

3. ApiBasic.csv

Contains detailed information about APIs. - ID: Unique API identifier. - Name: API name. - PwURL: API URL. - Provider: API provider. - ProviderURL: API provider's website. - PorHomePage: API portal/homepage. - Endpoint: API endpoint. - Version: API version. - Type: API type (1-Browser, 2-Product, 3-Standard, 4-System/Embedded, 5-Web/Internet). - ArchStyle: Architectural style (1-Indirect, 2-Native/Browser, 3-Push/Streaming, 4-REST, 5-RPC). - IsDeviceSpec: Whether the API is device-specific (0-False, 1-True). - Scope: API scope (1-Metaservice API, 2-Microservice API, 3-Single Purpose API). - Description: Detailed API description.

4. ApiAddition.csv

Includes API metadata and support information. - ID: API ID. - DocsHomePage: Documentation URL. - TwitterURL: Twitter support URL. - SupEmail: Support email. - Forum: API forum/message boards. - ConsoleURL: Interactive console URL. - TermURL: Terms of service URL. - DescFileURL: API description file URL. - DescFileType: File type (e.g., Swagger, RAML, WSDL). - IsNonPrptry: Whether the API is non-proprietary (0-False, 1-True). - LiceType: License type. - IsSslSup: SSL support (0-False, 1-True). - AuthModel: Authentication model. - ReqFmt: Supported response formats. - IsHyperApi: Hypermedia API flag (0-False, 1-True). - IsRstctAces: Restricted access (0-False, 1-True). - IsUnofficial: Whether it's an unofficial API (0-False, 1-True).

5. ApiCate.csv

Maps APIs to their respective categories. - ApiID: API ID. - CateID: Category ID. - IsPri: Whether it’s the primary category (0-False, 1-True).

6. MashupSketch.csv

Stores basic details about mashups. - Name: Mashup name. - PwURL: Mashup URL. - Description: Short description. - CategoryName: Primary category. - CategoryURL: Category URL. - SubmitDate: Submission date.

7. Mashup.csv

Detailed information about mashups. - ID: Unique mashup ID. - Name: Mashup name. - PwURL: Mashup URL. - Company: Company associated with the mashup. - URL: Direct link to the mashup. - Description: Detailed description. - Type: Type (1-Web, 2-Mobile, 3-Desktop, 4-Other).

8. MashupCate.csv

Maps mashups to categories. - MashupID: Mashup ID. - CateID: Category ID. - IsPri: Whether it’s the primary category (0-False, 1-True).

9. MashupApi.csv

Maps mashups to the APIs they use. - MashupID: Mashup ID. - ApiID: API ID.

Usage

• This dataset is ideal for research on API usage trends, category distributions, and mashup compositions.
• It can be used to study API popularity, analyze technological trends, or build recommendation systems for developers looking for APIs.

Acknowledgments

Data sourced from ProgrammableWeb.

Technical Reference Model (TRM) is a set of categorized software and non-commodity hardware products that is the starting point for all IT purchases at the Department. It also includes the associated status of each software and hardware such as approved, prohibited, restricted, etc. This is a searchable reference model governed by TRM as a Service (TRMaaS). This service lists only those products that have been govern by the United States Secret Service.

Link to CYC's online application to find out refuse and recycling collections dates by postcode. Also includes the public API and API documentation for retrieving collection data by UPRN.

An open Rest API for querying all Solar System data

From the Web site: The Nursing Home COVID-19 Public File includes data reported by nursing homes to the CDC’s National Healthcare Safety Network (NHSN) system COVID-19 Long Term Care Facility Module, including Resident Impact, Facility Capacity, Staff & Personnel, and Supplies & Personal Protective Equipment, and Ventilator Capacity and Supplies Data Elements.

Site includes access to the dataset, API documentation, a Data Dictionary, archived data and additional resources.

This site is hosted on the Socrata platform

The Repository Analytics and Metrics Portal (RAMP) is a web service that aggregates use and performance use data of institutional repositories. The data are a subset of data from RAMP, the Repository Analytics and Metrics Portal (http://rampanalytics.org), consisting of data from all participating repositories for the calendar year 2018. For a description of the data collection, processing, and output methods, please see the "methods" section below. Note that the RAMP data model changed in August, 2018 and two sets of documentation are provided to describe data collection and processing before and after the change.

Methods

RAMP Data Documentation – January 1, 2017 through August 18, 2018

Data Collection

RAMP data were downloaded for participating IR from Google Search Console (GSC) via the Search Console API. The data consist of aggregated information about IR pages which appeared in search result pages (SERP) within Google properties (including web search and Google Scholar).

Data from January 1, 2017 through August 18, 2018 were downloaded in one dataset per participating IR. The following fields were downloaded for each URL, with one row per URL:

url: This is returned as a 'page' by the GSC API, and is the URL of the page which was included in an SERP for a Google property.
impressions: The number of times the URL appears within the SERP.
clicks: The number of clicks on a URL which took users to a page outside of the SERP.
clickThrough: Calculated as the number of clicks divided by the number of impressions.
position: The position of the URL within the SERP.
country: The country from which the corresponding search originated.
device: The device used for the search.
date: The date of the search.

Following data processing describe below, on ingest into RAMP an additional field, citableContent, is added to the page level data.

Note that no personally identifiable information is downloaded by RAMP. Google does not make such information available.

More information about click-through rates, impressions, and position is available from Google's Search Console API documentation: https://developers.google.com/webmaster-tools/search-console-api-original/v3/searchanalytics/query and https://support.google.com/webmasters/answer/7042828?hl=en

Data Processing

Upon download from GSC, data are processed to identify URLs that point to citable content. Citable content is defined within RAMP as any URL which points to any type of non-HTML content file (PDF, CSV, etc.). As part of the daily download of statistics from Google Search Console (GSC), URLs are analyzed to determine whether they point to HTML pages or actual content files. URLs that point to content files are flagged as "citable content." In addition to the fields downloaded from GSC described above, following this brief analysis one more field, citableContent, is added to the data which records whether each URL in the GSC data points to citable content. Possible values for the citableContent field are "Yes" and "No."

Processed data are then saved in a series of Elasticsearch indices. From January 1, 2017, through August 18, 2018, RAMP stored data in one index per participating IR.

About Citable Content Downloads

Data visualizations and aggregations in RAMP dashboards present information about citable content downloads, or CCD. As a measure of use of institutional repository content, CCD represent click activity on IR content that may correspond to research use.

CCD information is summary data calculated on the fly within the RAMP web application. As noted above, data provided by GSC include whether and how many times a URL was clicked by users. Within RAMP, a "click" is counted as a potential download, so a CCD is calculated as the sum of clicks on pages/URLs that are determined to point to citable content (as defined above).

For any specified date range, the steps to calculate CCD are:

Filter data to only include rows where "citableContent" is set to "Yes."
Sum the value of the "clicks" field on these rows.

Output to CSV

Published RAMP data are exported from the production Elasticsearch instance and converted to CSV format. The CSV data consist of one "row" for each page or URL from a specific IR which appeared in search result pages (SERP) within Google properties as described above.

The data in these CSV files include the following fields:

url: This is returned as a 'page' by the GSC API, and is the URL of the page which was included in an SERP for a Google property.
impressions: The number of times the URL appears within the SERP.
clicks: The number of clicks on a URL which took users to a page outside of the SERP.
clickThrough: Calculated as the number of clicks divided by the number of impressions.
position: The position of the URL within the SERP.
country: The country from which the corresponding search originated.
device: The device used for the search.
date: The date of the search.
citableContent: Whether or not the URL points to a content file (ending with pdf, csv, etc.) rather than HTML wrapper pages. Possible values are Yes or No.
index: The Elasticsearch index corresponding to page click data for a single IR.
repository_id: This is a human readable alias for the index and identifies the participating repository corresponding to each row. As RAMP has undergone platform and version migrations over time, index names as defined for the index field have not remained consistent. That is, a single participating repository may have multiple corresponding Elasticsearch index names over time. The repository_id is a canonical identifier that has been added to the data to provide an identifier that can be used to reference a single participating repository across all datasets. Filtering and aggregation for individual repositories or groups of repositories should be done using this field.

Filenames for files containing these data follow the format 2018-01_RAMP_all.csv. Using this example, the file 2018-01_RAMP_all.csv contains all data for all RAMP participating IR for the month of January, 2018.

Data Collection from August 19, 2018 Onward

RAMP data are downloaded for participating IR from Google Search Console (GSC) via the Search Console API. The data consist of aggregated information about IR pages which appeared in search result pages (SERP) within Google properties (including web search and Google Scholar).

Data are downloaded in two sets per participating IR. The first set includes page level statistics about URLs pointing to IR pages and content files. The following fields are downloaded for each URL, with one row per URL:

url: This is returned as a 'page' by the GSC API, and is the URL of the page which was included in an SERP for a Google property.
impressions: The number of times the URL appears within the SERP.
clicks: The number of clicks on a URL which took users to a page outside of the SERP.
clickThrough: Calculated as the number of clicks divided by the number of impressions.
position: The position of the URL within the SERP.
date: The date of the search.

Following data processing describe below, on ingest into RAMP a additional field, citableContent, is added to the page level data.

The second set includes similar information, but instead of being aggregated at the page level, the data are grouped based on the country from which the user submitted the corresponding search, and the type of device used. The following fields are downloaded for combination of country and device, with one row per country/device combination:

country: The country from which the corresponding search originated.
device: The device used for the search.
impressions: The number of times the URL appears within the SERP.
clicks: The number of clicks on a URL which took users to a page outside of the SERP.
clickThrough: Calculated as the number of clicks divided by the number of impressions.
position: The position of the URL within the SERP.
date: The date of the search.

Note that no personally identifiable information is downloaded by RAMP. Google does not make such information available.

More information about click-through rates, impressions, and position is available from Google's Search Console API documentation: https://developers.google.com/webmaster-tools/search-console-api-original/v3/searchanalytics/query and https://support.google.com/webmasters/answer/7042828?hl=en

Data Processing

Upon download from GSC, the page level data described above are processed to identify URLs that point to citable content. Citable content is defined within RAMP as any URL which points to any type of non-HTML content file (PDF, CSV, etc.). As part of the daily download of page level statistics from Google Search Console (GSC), URLs are analyzed to determine whether they point to HTML pages or actual content files. URLs that point to content files are flagged as "citable content." In addition to the fields downloaded from GSC described above, following this brief analysis one more field, citableContent, is added to the page level data which records whether each page/URL in the GSC data points to citable content. Possible values for the citableContent field are "Yes" and "No."

The data aggregated by the search country of origin and device type do not include URLs. No additional processing is done on these data. Harvested data are passed directly into Elasticsearch.

Processed data are then saved in a series of Elasticsearch indices. Currently, RAMP stores data in two indices per participating IR. One index includes the page level data, the second index includes the country of origin and device type data.

About Citable Content Downloads

Data visualizations and aggregations in RAMP dashboards present information about citable content downloads, or CCD. As a measure of use of institutional repository