The global big data market is forecasted to grow to 103 billion U.S. dollars by 2027, more than double its expected market size in 2018. With a share of 45 percent, the software segment would become the large big data market segment by 2027.

What is Big data?

Big data is a term that refers to the kind of data sets that are too large or too complex for traditional data processing applications. It is defined as having one or some of the following characteristics: high volume, high velocity or high variety. Fast-growing mobile data traffic, cloud computing traffic, as well as the rapid development of technologies such as artificial intelligence (AI) and the Internet of Things (IoT) all contribute to the increasing volume and complexity of data sets.

Big data analytics

Advanced analytics tools, such as predictive analytics and data mining, help to extract value from the data and generate new business insights. The global big data and business analytics market was valued at 169 billion U.S. dollars in 2018 and is expected to grow to 274 billion U.S. dollars in 2022. As of November 2018, 45 percent of professionals in the market research industry reportedly used big data analytics as a research method.

Snapshot img

Big Data Services Market Size 2025-2029

The big data services market size is forecast to increase by USD 604.2 billion, at a CAGR of 54.4% between 2024 and 2029.

The market is experiencing significant growth, driven by the increasing adoption of big data in various industries, particularly in blockchain technology. The ability to process and analyze vast amounts of data in real-time is revolutionizing business operations and decision-making processes. However, this market is not without challenges. One of the most pressing issues is the need to cater to diverse client requirements, each with unique data needs and expectations. This necessitates customized solutions and a deep understanding of various industries and their data requirements. Additionally, ensuring data security and privacy in an increasingly interconnected world poses a significant challenge. Companies must navigate these obstacles while maintaining compliance with regulations and adhering to ethical data handling practices. To capitalize on the opportunities presented by the market, organizations must focus on developing innovative solutions that address these challenges while delivering value to their clients. By staying abreast of industry trends and investing in advanced technologies, they can effectively meet client demands and differentiate themselves in a competitive landscape.

What will be the Size of the Big Data Services Market during the forecast period?

Explore in-depth regional segment analysis with market size data - historical 2019-2023 and forecasts 2025-2029 - in the full report.
Request Free SampleThe market continues to evolve, driven by the ever-increasing volume, velocity, and variety of data being generated across various sectors. Data extraction is a crucial component of this dynamic landscape, enabling entities to derive valuable insights from their data. Human resource management, for instance, benefits from data-driven decision making, operational efficiency, and data enrichment. Batch processing and data integration are essential for data warehousing and data pipeline management. Data governance and data federation ensure data accessibility, quality, and security. Data lineage and data monetization facilitate data sharing and collaboration, while data discovery and data mining uncover hidden patterns and trends. Real-time analytics and risk management provide operational agility and help mitigate potential threats. Machine learning and deep learning algorithms enable predictive analytics, enhancing business intelligence and customer insights. Data visualization and data transformation facilitate data usability and data loading into NoSQL databases. Government analytics, financial services analytics, supply chain optimization, and manufacturing analytics are just a few applications of big data services. Cloud computing and data streaming further expand the market's reach and capabilities. Data literacy and data collaboration are essential for effective data usage and collaboration. Data security and data cleansing are ongoing concerns, with the market continuously evolving to address these challenges. The integration of natural language processing, computer vision, and fraud detection further enhances the value proposition of big data services. The market's continuous dynamism underscores the importance of data cataloging, metadata management, and data modeling for effective data management and optimization.

How is this Big Data Services Industry segmented?

The big data services industry research report provides comprehensive data (region-wise segment analysis), with forecasts and estimates in 'USD billion' for the period 2025-2029, as well as historical data from 2019-2023 for the following segments. ComponentSolutionServicesEnd-userBFSITelecomRetailOthersTypeData storage and managementData analytics and visualizationConsulting servicesImplementation and integration servicesSupport and maintenance servicesSectorLarge enterprisesSmall and medium enterprises (SMEs)GeographyNorth AmericaUSMexicoEuropeFranceGermanyItalyUKMiddle East and AfricaUAEAPACAustraliaChinaIndiaJapanSouth KoreaSouth AmericaBrazilRest of World (ROW).

By Component Insights

The solution segment is estimated to witness significant growth during the forecast period.Big data services have become indispensable for businesses seeking operational efficiency and customer insight. The vast expanse of structured and unstructured data presents an opportunity for organizations to analyze consumer behaviors across multiple channels. Big data solutions facilitate the integration and processing of data from various sources, enabling businesses to gain a deeper understanding of customer sentiment towards their products or services. Data governance ensures data quality and security, while data federation and data lineage provide transparency and traceability. Artificial intelligenc

An Integrated Science Data Analytics Platform is an environment that enables the confluence of resources for scientific investigation. It harmonizes data, tools and computational resources to enable the research community to focus on the investigation rather than spending time on security, data preparation, management, etc. OceanWorks is a NASA technology integration project to establish a cloud-based Integrated Ocean Science Data Analytics Platform for big ocean science at NASA’s Physical Oceanography Distributed Active Archive Center (PO.DAAC) for big ocean science. It focuses on advancement and maturity by bringing together several NASA open-source, big data projects for parallel analytics, anomaly detection, in situ to satellite data matchup, quality-screened data subsetting, search relevancy, and data discovery. Our communities are relying on data available through distributed data centers to conduct their research. In typical investigations, scientists would (1) search for data, (2) evaluate the relevance of that data, (3) download it, and (4) then apply algorithms to identify trends, anomalies, or other attributes of the data. Such a workflow cannot scale if the research involves a massive amount of data or multi-variate measurements. With the upcoming NASA Surface Water and Ocean Topography (SWOT) mission expected to produce over 20PB of observational data during its 3-year nominal mission, the volume of data will challenge all existing Earth Science data archival, distribution and analysis paradigms. This paper discusses how OceanWorks enhances the analysis of physical ocean data where the computation is done on an elastic cloud platform next to the archive to deliver fast, web-accessible services for working with oceanographic measurements.

The USDA Agricultural Research Service (ARS) recently established SCINet , which consists of a shared high performance computing resource, Ceres, and the dedicated high-speed Internet2 network used to access Ceres. Current and potential SCINet users are using and generating very large datasets so SCINet needs to be provisioned with adequate data storage for their active computing. It is not designed to hold data beyond active research phases. At the same time, the National Agricultural Library has been developing the Ag Data Commons, a research data catalog and repository designed for public data release and professional data curation. Ag Data Commons needs to anticipate the size and nature of data it will be tasked with handling. The ARS Web-enabled Databases Working Group, organized under the SCINet initiative, conducted a study to establish baseline data storage needs and practices, and to make projections that could inform future infrastructure design, purchases, and policies. The SCINet Web-enabled Databases Working Group helped develop the survey which is the basis for an internal report. While the report was for internal use, the survey and resulting data may be generally useful and are being released publicly. From October 24 to November 8, 2016 we administered a 17-question survey (Appendix A) by emailing a Survey Monkey link to all ARS Research Leaders, intending to cover data storage needs of all 1,675 SY (Category 1 and Category 4) scientists. We designed the survey to accommodate either individual researcher responses or group responses. Research Leaders could decide, based on their unit's practices or their management preferences, whether to delegate response to a data management expert in their unit, to all members of their unit, or to themselves collate responses from their unit before reporting in the survey.
Larger storage ranges cover vastly different amounts of data so the implications here could be significant depending on whether the true amount is at the lower or higher end of the range. Therefore, we requested more detail from "Big Data users," those 47 respondents who indicated they had more than 10 to 100 TB or over 100 TB total current data (Q5). All other respondents are called "Small Data users." Because not all of these follow-up requests were successful, we used actual follow-up responses to estimate likely responses for those who did not respond. We defined active data as data that would be used within the next six months. All other data would be considered inactive, or archival. To calculate per person storage needs we used the high end of the reported range divided by 1 for an individual response, or by G, the number of individuals in a group response. For Big Data users we used the actual reported values or estimated likely values.

Resources in this dataset:Resource Title: Appendix A: ARS data storage survey questions. File Name: Appendix A.pdfResource Description: The full list of questions asked with the possible responses. The survey was not administered using this PDF but the PDF was generated directly from the administered survey using the Print option under Design Survey. Asterisked questions were required. A list of Research Units and their associated codes was provided in a drop down not shown here. Resource Software Recommended: Adobe Acrobat,url: https://get.adobe.com/reader/ Resource Title: CSV of Responses from ARS Researcher Data Storage Survey. File Name: Machine-readable survey response data.csvResource Description: CSV file includes raw responses from the administered survey, as downloaded unfiltered from Survey Monkey, including incomplete responses. Also includes additional classification and calculations to support analysis. Individual email addresses and IP addresses have been removed. This information is that same data as in the Excel spreadsheet (also provided).Resource Title: Responses from ARS Researcher Data Storage Survey. File Name: Data Storage Survey Data for public release.xlsxResource Description: MS Excel worksheet that Includes raw responses from the administered survey, as downloaded unfiltered from Survey Monkey, including incomplete responses. Also includes additional classification and calculations to support analysis. Individual email addresses and IP addresses have been removed.Resource Software Recommended: Microsoft Excel,url: https://products.office.com/en-us/excel

Systematic reviews are the method of choice to synthesize research evidence. To identify main topics (so-called hot spots) relevant to large corpora of original publications in need of a synthesis, one must address the “three Vs” of big data (volume, velocity, and variety), especially in loosely defined or fragmented disciplines. For this purpose, text mining and predictive modeling are very helpful. Thus, we applied these methods to a compilation of documents related to digitalization in aesthetic, arts, and cultural education, as a prototypical, loosely defined, fragmented discipline, and particularly to quantitative research within it (QRD-ACE). By broadly querying the abstract and citation database Scopus with terms indicative of QRD-ACE, we identified a corpus of N = 55,553 publications for the years 2013–2017. As the result of an iterative approach of text mining, priority screening, and predictive modeling, we identified n = 8,304 potentially relevant publications of which n = 1,666 were included after priority screening. Analysis of the subject distribution of the included publications revealed video games as a first hot spot of QRD-ACE. Topic modeling resulted in aesthetics and cultural activities on social media as a second hot spot, related to 4 of k = 8 identified topics. This way, we were able to identify current hot spots of QRD-ACE by screening less than 15% of the corpus. We discuss implications for harnessing text mining, predictive modeling, and priority screening in future research syntheses and avenues for future original research on QRD-ACE. Dataset for: Christ, A., Penthin, M., & Kröner, S. (2019). Big Data and Digital Aesthetic, Arts, and Cultural Education: Hot Spots of Current Quantitative Research. Social Science Computer Review, 089443931988845. https://doi.org/10.1177/0894439319888455:

*** TYPE OF SURVEY AND METHODS *** The data set includes responses to a survey conducted by professionally trained interviewers of a social and market research company in the form of computer-aided telephone interviews (CATI) from 2017-02 to 2017-04. The target population was inhabitants of Germany aged 18 years and more, who were randomly selected by using the sampling approaches ADM eASYSAMPLe (based on the Gabler-Häder method) for landline connections and eASYMOBILe for mobile connections. The 1,331 completed questionnaires comprise 44.2 percent mobile and 55.8 percent landline phone respondents. Most questions had options to answer with a 5-point rating scale (Likert-like) anchored with ‘Fully agree’ to ‘Do not agree at all’, or ‘Very uncomfortable’ to ‘Very comfortable’, for instance. Responses by the interviewees were weighted to obtain a representation of the entire German population (variable ‘gewicht’ in the data sets). To this end, standard weighting procedures were applied to reduce differences between the sample and the entire population with regard to known rates of response and non-response depending on household size, age, gender, educational level, and place of residence. *** RELATED PUBLICATION AND FURTHER DETAILS *** The questionnaire, analysis and results will be published in the corresponding report (main text in English language, questionnaire in Appendix B in German language of the interviews and English translation). The report will be available as open access publication at KIT Scientific Publishing (https://www.ksp.kit.edu/). Reference: Orwat, Carsten; Schankin, Andrea (2018): Attitudes towards big data practices and the institutional framework of privacy and data protection - A population survey, KIT Scientific Report 7753, Karlsruhe: KIT Scientific Publishing. *** FILE FORMATS *** The data set of responses is saved for the repository KITopen at 2018-11 in the following file formats: comma-separated values (.csv), tapulator-separated values (.dat), Excel (.xlx), Excel 2007 or newer (.xlxs), and SPSS Statistics (.sav). The questionnaire is saved in the following file formats: comma-separated values (.csv), Excel (.xlx), Excel 2007 or newer (.xlxs), and Portable Document Format (.pdf). *** PROJECT AND FUNDING *** The survey is part of the project Assessing Big Data (ABIDA) (from 2015-03 to 2019-02), which receives funding from the Federal Ministry of Education and Research (BMBF), Germany (grant no. 01IS15016A-F). http://www.abida.de *** CONTACT *** Carsten Orwat, Karlsruhe Institute of Technology, Institute for Technology Assessment and Systems Analysis orwat@kit.edu Andrea Schankin, Karlsruhe Institute of Technology, Institute of Telematics andrea.schankin@kit.edu

The total amount of data created, captured, copied, and consumed globally is forecast to increase rapidly, reaching *** zettabytes in 2024. Over the next five years up to 2028, global data creation is projected to grow to more than *** zettabytes. In 2020, the amount of data created and replicated reached a new high. The growth was higher than previously expected, caused by the increased demand due to the COVID-19 pandemic, as more people worked and learned from home and used home entertainment options more often. Storage capacity also growing Only a small percentage of this newly created data is kept though, as just * percent of the data produced and consumed in 2020 was saved and retained into 2021. In line with the strong growth of the data volume, the installed base of storage capacity is forecast to increase, growing at a compound annual growth rate of **** percent over the forecast period from 2020 to 2025. In 2020, the installed base of storage capacity reached *** zettabytes.

These datasets were calculated using the mechanistic modeling approach. See “Big data training data for artificial intelligence-based Li-ion diagnosis and prognosis“ (Journal of Power Sources, Volume 479, 15 December 2020, 228806) and "Analysis of Synthetic Voltage vs. Capacity Datasets for Big Data Li-ion Diagnosis and Prognosis" (Energies, under review) for more details

The V vs. Q dataset (Gr-NCA_Co33_cha_n) was compiled with a resolution of 0.01 for the triplets and C/25 charges. This accounts for more than 5,000 different paths. Each path was simulated with at most 0.85% increases for each. The dataset contains more than 700,000 unique voltage vs. capacity curves. 4 Variables are included, see corresponding read me file for details and example how to use.

The duty cycle dataset is composed of close to 125,000 duty cycles with one voltage curve at most every 200 cycles (one every 10 cycles for cycles 1-200). 15 variables are included in the duty cycle dataset (Gr-NCA_Co33_cha_124652duty). See corresponding read me file for details and example how to use.

Genomics is a Big Data science and is going to get much bigger, very soon, but it is not known whether the needs of genomics will exceed other Big Data domains. Projecting to the year 2025, we compared genomics with three other major generators of Big Data: astronomy, YouTube, and Twitter. Our estimates show that genomics is a “four-headed beast”—it is either on par with or the most demanding of the domains analyzed here in terms of data acquisition, storage, distribution, and analysis. We discuss aspects of new technologies that will need to be developed to rise up and meet the computational challenges that genomics poses for the near future. Now is the time for concerted, community-wide planning for the “genomical” challenges of the next decade.

The Water Conservancy Project Information Management System (WCPIMS) market is experiencing robust growth, driven by increasing government investments in infrastructure development and a rising need for efficient water resource management. The market's expansion is fueled by the urgent need to modernize aging infrastructure, enhance data collection and analysis capabilities, and improve decision-making processes related to flood control, irrigation, and water resource allocation. Technological advancements, such as cloud computing, big data analytics, and the Internet of Things (IoT), are further accelerating market growth by enabling real-time monitoring, predictive modeling, and improved collaboration among stakeholders. While challenges such as data security concerns and the need for skilled professionals in data management exist, the long-term outlook for the WCPIMS market remains positive, particularly in regions facing water scarcity and increasing flood risks. We estimate the 2025 market size to be approximately $2.5 billion USD, based on observed growth in similar sectors and industry reports showing a positive correlation between infrastructure spending and adoption of such management systems. Assuming a conservative CAGR of 12% (a figure reflective of growth in related technology sectors), the market is projected to significantly expand over the forecast period (2025-2033). Competition is primarily concentrated among domestic players, including Four Faith, Beijing Automic, and Wuhan Dexi Technology, showcasing a strong regional emphasis in market development. The segmentation of the WCPIMS market likely encompasses various deployment models (cloud-based vs. on-premise), functionalities (data acquisition, analysis, reporting), and target user groups (government agencies, private contractors). Further expansion is anticipated through partnerships with established players in the broader water management industry, creating integrated solutions for comprehensive water resource optimization. The market's growth trajectory will be significantly impacted by government policies promoting digital transformation in the water sector, advancements in data analytics capabilities, and the availability of skilled workforce for implementation and maintenance of these systems. Successful players will be those demonstrating advanced technological capabilities, robust data security measures, and strong customer service. The historical period (2019-2024) likely saw a slower growth rate, with the base year of 2025 reflecting a significant acceleration driven by accelerated digitalization and infrastructure development projects.

Background: For some time, the VIVO for Weill Cornell Medical College (WCMC) had struggled with both unacceptable page load times and unreliable uptime. With some individual profiles containing upwards of 800 publications, WCMC VIVO has relatively large profiles, but no profile was so large that it could account for this performance. The WCMC VIVO Implementation Team explored a number of options for improving performance including caching, better hardware, query optimization, limiting user access to large pages, using another instance of Tomcat, throttling bots, and blocking IP's issuing too many requests. But none of these avenues were fruitful. Analysis of triple stores: With the 1.7 version, VIVO ships with the Jena SDB triple store, but the SDB version of Jena is no longer supported by its developers. In April, we reviewed various published analyses and benchmarks suggesting there were alternatives to Jena such as Virtuoso that perform better than even Jena's successor, TDB. In particular, the Berlin SPARQL Benchmark v. 3.1[1] showed that Virtuoso had the strongest performance compared to the other data stores measured including BigData, BigOwlim, and Jena TDB. In addition, Virtuoso is used on dbpedia.org which serves up 3 billion triples compared to the only 12 million with WCMC's VIVO site. Whereas Jena SDB stores its triples in a MySQL database, Virtuoso manages its in a binary file. The software is available in open source and commercial editions. Configuration: In late 2014, we installed Virtuoso on a local machine and loaded data from our production VIVO. Some queries completed in about 10% of the time as compared to our production VIVO. However, we noticed that the listview queries invoked whenever profile pages were loaded were still slow. After soliciting feedback from members of both the Virtuoso and VIVO communities, we modified these queries to rely on the OPTIONAL instead of UNION construct. This modification, which wasn't possible in a Jena SDB environment, reduced by eight-fold the number of queries that the application makes of the triple store. About four or five additional steps were required for VIVO and Virtuoso to work optimally with one another; these are documented in the VIVO Duraspace wiki. Results: On March 31, WCMC launched Virtuoso in its production environment. According to our instance of New Relic, VIVO has an average page load of about four seconds and 99% uptime, both of which are dramatic improvements. There are opportunities for further tuning: the four second average includes pages such as the visualizations as well as pages served up to logged in users, which are slower than other types of pages. [1] http://wifo5-03.informatik.uni-mannheim.de/bizer/berlinsparqlbenchmark/results/V7/#comparison

The dataset is comprised of text responses from GPT-4 after reading passages of text. It is scored and organized in folders.

Clinical studies, especially randomized controlled trials, are essential for generating evidence for clinical practice. However, generalizability is a long-standing concern when applying trial results to real-world patients. Generalizability assessment is thus important, nevertheless, not consistently practiced. We performed a systematic scoping review to understand the practice of generalizability assessment. We identified 187 relevant papers and systematically organized these studies in a taxonomy with three dimensions: (1) data availability (i.e., before or after trial [a priori vs a posteriori generalizability]), (2) result outputs (i.e., score vs non-score), and (3) populations of interest. We further reported disease areas, underrepresented subgroups, and types of data used to profile target populations. We observed an increasing trend of generalizability assessments, but less than 30% of studies reported positive generalizability results. As a priori generalizability can be assessed using only study design information (primarily eligibility criteria), it gives investigators a golden opportunity to adjust the study design before the trial starts. Nevertheless, less than 40% of the studies in our review assessed a priori generalizability. With the wide adoption of electronic health records systems, rich real-world patient databases are increasingly available for generalizability assessment; however, informatics tools are lacking to support the adoption of generalizability assessment practice.

Methods We performed the literature search over the following 4 databases: MEDLINE, Cochrane, PychINFO, and CINAHL. Following the Institute of Medicine’s standards for systematic review and Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA), we conducted the scoping review in the following six steps: 1) gaining an initial understanding about clinical trial generalizability assessment, population representativeness, internal validity, and external validity, 2) identifying relevant keywords, 3) formulating four search queries to identify relevant articles in the 4 databases, 4) screening the articles by reviewing titles and abstracts, 5) reviewing articles’ full-text to further filter out irrelevant ones based on inclusion and exclusion criteria, and 6) coding the articles for data extraction.

Study selection and screening process

We used an iterative process to identify and refine the search keywords and search strategies. We identified 5,352 articles as of February 2019 from MEDLINE, CINAHL, PychINFO, and Cochrane. After removing duplicates, 3,569 records were assessed for relevancy by two researchers (ZH and XT) through reviewing the titles and abstracts against the inclusion and exclusion criteria. Conflicts were resolved with a third reviewer (JB). During the screening process, we also iteratively refined the inclusion and exclusion criteria. Out of the 3,569 articles, 3,275 were excluded through the title and abstract screening process. Subsequently, we reviewed the full texts of 294 articles, among which 106 articles were further excluded based on the exclusion criteria. The inter-rater reliability of the full-text review between the two annotators is 0.901 (i.e., Cohen’s kappa, p < .001). 187 articles were included in the final scoping review.

Data extraction and reporting

We coded and extracted data from the 187 eligible articles according to the following aspects: (1) whether the study performed an a priori generalizability assessment or a posteriori generalizability assessment or both; (2) the compared populations and the conclusions of the assessment; (3) the outputs of the results (e.g., generalizability scores, descriptive comparison); (4) whether the study focused on a specific disease. If so, we extracted the disease and disease category; (5) whether the study focused on a particular population subgroup (e.g., elderly). If so, we extracted the specific population subgroup; (6) the type(s) of the real-world patient data used to profile the target population (i.e., trial data, hospital data, regional data, national data, and international data). Note that trial data can also be regional, national, or even international, depending on the scale of the trial. Regardless, we considered them in the category of “trial data” as the study population of a trial is typically small compared to observational cohorts or real-world data. For observational cohorts or real-world data (e.g., EHRs), we extracted the specific scale of the database (i.e., regional, national, and international). For the studies that compared the characteristics of different populations to indicate generalizability issues, we further coded the populations that were compared (e.g., enrolled patients, eligible patients, general population, ineligible patients), and the types of characteristics that were compared (i.e., demographic information, clinical attributes and comorbidities, treatment outcomes, and adverse events). We then used Fisher’s exact test to assess whether there is a difference in the types of characteristics compared between a priori and a posteriori generalizability assessment studies.

Competitive intelligence monitoring goes beyond your sales team. Our CI solutions also bring powerful insights to your production, logistics, operation & marketing departments.

Why should you use our Competitive intelligence data? 1. Increase visibility: Our geolocation approach allows us to “get inside” any facility in the US, providing visibility in places where other solutions do not reach. 2. In-depth 360º analysis: Perform a unique and in-depth analysis of competitors, suppliers and customers. 3. Powerful Insights: We use alternative data and big data methodologies to peel back the layers of any private or public company. 4. Uncover your blind spots against leading competitors: Understand the complete business environment of your competitors, from third-tier suppliers to main investors. 5. Identify business opportunities: Analyze your competitor's strategic shifts and identify unnoticed business opportunities and possible threats or disruptions. 6. Keep track of your competitor´s influence around any specific area: Maintain constant monitoring of your competitors' actions and their impact on specific market areas.

How other companies are using our CI Solution? 1. Enriched Data Intelligence: Our Market Intelligence data bring you key insights from different angles. 2. Due Diligence: Our data provide the required panorama to evaluate a company’s cross-company relations to decide whether or not to proceed with an acquisition. 3. Risk Assessment: Our CI approach allows you to anticipate potential disruptions by understanding behavior in all the supply chain tiers. 4. Supply Chain Analysis: Our advanced Geolocation approach allows you to visualize and map an entire supply chain network. 5. Insights Discovery: Our relationship identifiers algorithms generate data matrix networks that uncover new and unnoticed insights within a specific market, consumer segment, competitors' influence, logistics shifts, and more.

From "digital" to the real field: Most competitive intelligence companies focus their solutions analysis on social shares, review sites, and sales calls. Our competitive intelligence strategy consists on tracking the real behavior of your market on the field, so that you can answer questions like: -What uncovered need does my market have? -How much of a threat is my competition? -How is the market responding to my competitor´s offer? -How my competitors are changing? -Am I losing or winning market?

Please cite the following paper when using this dataset:

N. Thakur, V. Su, M. Shao, K. Patel, H. Jeong, V. Knieling, and A. Bian “A labelled dataset for sentiment analysis of videos on YouTube, TikTok, and other sources about the 2024 outbreak of measles,” Proceedings of the 26th International Conference on Human-Computer Interaction (HCII 2024), Washington, USA, 29 June - 4 July 2024. (Accepted as a Late Breaking Paper, Preprint Available at: https://doi.org/10.48550/arXiv.2406.07693)

Abstract

This dataset contains the data of 4011 videos about the ongoing outbreak of measles published on 264 websites on the internet between January 1, 2024, and May 31, 2024. These websites primarily include YouTube and TikTok, which account for 48.6% and 15.2% of the videos, respectively. The remainder of the websites include Instagram and Facebook as well as the websites of various global and local news organizations. For each of these videos, the URL of the video, title of the post, description of the post, and the date of publication of the video are presented as separate attributes in the dataset. After developing this dataset, sentiment analysis (using VADER), subjectivity analysis (using TextBlob), and fine-grain sentiment analysis (using DistilRoBERTa-base) of the video titles and video descriptions were performed. This included classifying each video title and video description into (i) one of the sentiment classes i.e. positive, negative, or neutral, (ii) one of the subjectivity classes i.e. highly opinionated, neutral opinionated, or least opinionated, and (iii) one of the fine-grain sentiment classes i.e. fear, surprise, joy, sadness, anger, disgust, or neutral. These results are presented as separate attributes in the dataset for the training and testing of machine learning algorithms for performing sentiment analysis or subjectivity analysis in this field as well as for other applications. The paper associated with this dataset (please see the above-mentioned citation) also presents a list of open research questions that may be investigated using this dataset.

Analysis of ‘Current and projected research data storage needs of Agricultural Research Service researchers in 2016’ provided by Analyst-2 (analyst-2.ai), based on source dataset retrieved from https://catalog.data.gov/dataset/e2b7daf0-c8fe-4c68-b62d-891360ba8f96 on 26 January 2022.

--- Dataset description provided by original source is as follows ---

The USDA Agricultural Research Service (ARS) recently established SCINet , which consists of a shared high performance computing resource, Ceres, and the dedicated high-speed Internet2 network used to access Ceres. Current and potential SCINet users are using and generating very large datasets so SCINet needs to be provisioned with adequate data storage for their active computing. It is not designed to hold data beyond active research phases. At the same time, the National Agricultural Library has been developing the Ag Data Commons, a research data catalog and repository designed for public data release and professional data curation. Ag Data Commons needs to anticipate the size and nature of data it will be tasked with handling.

The ARS Web-enabled Databases Working Group, organized under the SCINet initiative, conducted a study to establish baseline data storage needs and practices, and to make projections that could inform future infrastructure design, purchases, and policies. The SCINet Web-enabled Databases Working Group helped develop the survey which is the basis for an internal report. While the report was for internal use, the survey and resulting data may be generally useful and are being released publicly.

From October 24 to November 8, 2016 we administered a 17-question survey (Appendix A) by emailing a Survey Monkey link to all ARS Research Leaders, intending to cover data storage needs of all 1,675 SY (Category 1 and Category 4) scientists. We designed the survey to accommodate either individual researcher responses or group responses. Research Leaders could decide, based on their unit's practices or their management preferences, whether to delegate response to a data management expert in their unit, to all members of their unit, or to themselves collate responses from their unit before reporting in the survey.

Larger storage ranges cover vastly different amounts of data so the implications here could be significant depending on whether the true amount is at the lower or higher end of the range. Therefore, we requested more detail from "Big Data users," those 47 respondents who indicated they had more than 10 to 100 TB or over 100 TB total current data (Q5). All other respondents are called "Small Data users." Because not all of these follow-up requests were successful, we used actual follow-up responses to estimate likely responses for those who did not respond.

We defined active data as data that would be used within the next six months. All other data would be considered inactive, or archival.

To calculate per person storage needs we used the high end of the reported range divided by 1 for an individual response, or by G, the number of individuals in a group response. For Big Data users we used the actual reported values or estimated likely values.

--- Original source retains full ownership of the source dataset ---

The NOAA National Water Model Retrospective dataset contains input and output from multi-decade CONUS retrospective simulations. These simulations used meteorological input fields from meteorological retrospective datasets. The output frequency and fields available in this historical NWM dataset differ from those contained in the real-time operational NWM forecast model. Additionally, note that no streamflow or other data assimilation is performed within any of the NWM retrospective simulations

One application of this dataset is to provide historical context to current near real-time streamflow, soil moisture and snowpack conditions. The retrospective data can be used to infer flow frequencies and perform temporal analyses with hourly streamflow output and 3-hourly land surface output. This dataset can also be used in the development of end user applications which require a long baseline of data for system training or verification purposes.


Details for Each Version of the NWM Retrospective Output

CONUS Domain - CONUS retrospective output is provided by all four versions of the NWM

1. Version 3.0 - A 44-year (February 1979 through January 2023) retrospective simulation using version 3.0 of the National Water Model.
2. Version 2.1 - A 42-year (February 1979 through December 2020) retrospective simulation using version 2.1 of the National Water Model.
3. Version 2.0 - A 26-year (January 1993 through December 2018) retrospective simulation using version 2.0 of the National Water Model.
4. Version 1.2 - A 25-year (January 1993 through December 2017) retrospective simulation using version 1.2 of the National Water Model.
   
   

oCOUNS Domains - Only v3.0 of the NWM Retrospective data set provides coverage of the NWM Alaska, Hawaii and Puerto Rico / US Virgin Island domains.

CONUS Meteorological Forcing

1. Versions 3.0 and 2.1: NWM Retrospective simulations used forcing from the Office of Water Prediction Analysis of Record for Calibration (AORC) dataset. NWM v2.1 used AORC v1.0 for 1979-2006 and AORC v1.1 for 2007-2020, while NWM v3.0 used AORC v1.1 for the full v3.0 (1979-2023 period)

Important Warning - While the metadata tag in the NWM v3.0 forcing files label the files as “v2.1”, the files are in fact v3.0 forcing files. 2) Versions 2.0 and 1.2 - NWM Retrospective simulation uses forcing from the North American Land Data Assimilation System (NLDAS) dataset
oCONUS Meteorological Forcing

1. Version 3.0 - AORC Alaska forcing was used to drive the NWM Alaska simulation, while North American Regional Reanalysis (NARR) data along with precipitation from the Alaska Pacific River Forecast Center (APRFC) was used to drive the Hawaii retrospective simulation. Similarly, the Puerto Rico / US Virgin Island retrospective simulation was driven by NARR data along with precipitation from the Southeast River Forecast Center.
   
   

Formats - NWM Retrospective data is available in two formats, NetCDF and Zarr. The NetCDF files contain the full set of NWM output data, while the Zarr files contain a subset of NWM output fields that vary with model version.

1. NWM V3.0: All model output and forcing input fields are available in the NetCDF format. All model output fields along with the precipitation forcing field are available in the Zarr format.
2. NWM V2.1: All model output and forcing input fields are available in the NetCDF format. Many of the model output fields along with the precipitation forcing field are available in the Zarr format
3. NWM V2.0: All model output fields are available in NetCDF format. Model channel output including streamflow and related fields are available in Zarr format.
4. NWM V1.2: All model output fields are available in NetCDF format.
   
   A table listing the data available within each NetCDF and Zarr file is located in the 'documentation page'. This data includes meteorological NWM forcing inputs along with NWM hydrologic and land surface outputs, and varies by version number.

Dataset created to monitor the Brazilian vegetation combining 4 different systems: (i) an inventory of Brazilian seed plants created to map the country biodiversity; (ii) the Fraction of Absorbed Photosynthetically Active Radiation; (iii) the NASA Power database to include meteorological data; and (iv) the DATASUS system which makes available geographical information from Brazil. The final dataset comprises a large number of attributes including meteorological and vegetational features, which correspond to a total of 8 and 8471 attributes, respectively. Moreover, the dataset contains 20 labels and 865 geographical positions (latitude and longitude) used during the vegetation monitoring. This project makes available raw and preprocessed data, and Machine Learning models (including source codes) adjusted to : i) predict the occurrence of specific type of vegetation in different region without requiring a constant monitoring task; ii) monitor whether or not the prediction accuracy is changing after collecting new data, which provides an important tool to detect how the environment is evolving over time; and iii) study this dataset as an extra data source to better understand and simulate meteorological influences on predicted vegetation types.

More information about the original datasets:

(i) http://www.scielo.br/scielo.php?script=sci_arttext&pid=S2175-78602015000401085 (ii) https://fapar.jrc.ec.europa.eu/Home.php (iii) https://power.larc.nasa.gov/ (iv) http://datasus.saude.gov.br/

Is there a chemically distinct reservoir in the Earth?
Do superplumes overly denser-than-average material?
Can we detect these anomalies with seismic data?
Can we evaluate statistical significance of the features in tomography?

This study presents the strongest evidence to date (ca. 2015) of large-scale thermo-chemical heterogeneities in the lowermost mantle using the full spectrum of seismic data. A large data set of surface-wave phase anomalies, body-wave travel times, normal-mode splitting functions and long-period waveforms is used to investigate the scaling between shear velocity, density and compressional velocity in the Earth's mantle (ϱ=dln ρ/dln vS, ν=dln vS/dln vP). Our preferred joint model consists of denser-than-average anomalies (∼1% peak-to-peak) at the base of the mantle roughly coincident with the low-velocity superplumes. The relative variation of shear velocity, density and compressional velocity in our study disfavors a purely thermal contribution to heterogeneity in the lowermost mantle, with implications for the long-term stability and evolution of superplumes.

Note on Odd Degree Structure:

Since the self-coupled normal-mode splitting observations constrain only even-degree density variations, all inversions strongly disfavored even-degree vS-ρ correlation (R2 ~ –0.46 to –0.25) in the lowermost mantle, which also disfavors a purely thermal contribution to heterogeneity in this region. However, the starting assumptions on positive vS-ρ correlation persisted in the remaining regions and for odd degree variations. In viscosity inversions with the geoid, opposing sign of the correlation of the longest wavelength even-versus odd-degree structure maps into a region of reduced viscosity in the lower mantle (Rudolph et al., 2020, doi:10.1029/2020gc009335). While important for such dynamical implications, odd-degree density variations in the lowermost mantle are poorly constrained in this study and should not be interpreted. We therefore used even-degree variations up to degree 6 for our inferences on thermo-chemical variations in the lowermost mantle (Figure 14), and provide those values in the files below.

Feedback/Questions? Please contact Raj Moulik (rajmoulik.com) at moulik@caa.columbia.edu

Reference:

Please cite the following work if you use this data or software.

• Moulik, P. & Ekström, G., 2016. The relationships between large-scale variations in shear velocity, density and compressional velocity in the Earth's mantle, J. Geophys. Res., 121, doi: 10.1002/2015JB012679. pdf

You can also cite the dataset and software from this Zenodo page (Optional).

Moulik, P. & Ekström, G. (2016). Dataset and Software for The Relationships Between Large-scale Variations in Shear Velocity, Density, and Compressional Velocity in the Earth's Mantle. In J. Geophys. Res. Solid Earth (v1.0, Vol. 121, pp. 2737–2771). Zenodo. doi: 10.5281/zenodo.8356540

Data Products:

• ME16_Figures(.tar.gz or .pdf) - contains all figures from the paper in .png format
• ME16 - Coefficients of the spline basis functions for each parameter. Refer cij in equation 3. This is our preferred global model of anisotropic elastic parameters and density. Density variations are allowed to deviate from a constant scaling with shear-velocity variations in the lowermost mantle, which is required to fit the longest-period normal modes (e.g. 0S2). Radial anisotropy is confined to the uppermost mantle (that is, since the anisotropy is parameterized with only the four uppermost splines, it becomes very small below a depth of 250 km, and vanishes at 410 km). This is an updated version of S362ANI+M (Moulik and Ekström, 2014) which did not solve independently for density and compressional-wave velocity variations and imposed a constant scaling throughout the mantle instead (ϱ=0, ν=1/0.55).
• STW105 - reference model used in ME16. Described in Kustowski et al. (2008)
• setup.cfg - Some configuration metadata relevant to this model for reproducibility.
• epix.tar.gz - Perturbations in horizontally (vsh) and vertically polarized shear velocity (vsv), Voigt-average isotropic shear-wave (vs) and compressional-wave velocity (vp), density (rho). anisotropy (as) and topography of the internal boundaries. This is calculated from the spline coefficients at every 1 by 1 degree cell-centered pixel and at every ~25 km depth region from Moho to the core-mantle boundary and stored in extended pixel format (.epix) ASCII files. Even-degree variations up to degree 6 are provided for density (rho_even6) and isotropic shear-wave velocity (vs_even6), which should be used for density inferences on thermochemical structure (See note above).
• ME16.BOX25km_PIX1X1.avni.nc4 - The perturbations in a standard AVNI format that utilizes the NETCDF4 container format. This file can be read in Python using either xarray or AVNI libraries. For example, to plot even-degree variations up to degree 6 in Voigt-averaged shear velocity perturbations at the bottom of the mantle (2875-2891 km depth)
  • import xarray as xr
  • ds = xr.open_dataset('ME16.BOX25km_PIX1X1.avni.nc4')
  • ds['vs_even6'][-1].plot()
• PROGRAMS.tar.gz - Fortran tools for obtaining model values at specific locations. After creating the executables from source code in the src folder, the readme script generates most of the epix files provided in epix.tar.gz above.
• profilescaling.txt - contains the median scaling ratios as used in Figure 15(a).
• scaling3D_MoulikJGR16.tar.gz - contains the scaling ratios and poisson ratio calculated from the joint model, as used in Figure 15(b).

This dataset contains citation-based impact indicators (a.k.a, "measures") for ~187,8M distinct PIDs (persistent identifiers) that correspond to research products (scientific publications, datasets, etc). In particular, for each PID, we have calculated the following indicators (organized in categories based on the semantics of the impact aspect that they better capture):

Influence indicators (i.e., indicators of the "total" impact of each research product; how established it is in general)

Citation Count: The total number of citations of the product, the most well-known influence indicator.

PageRank score: An influence indicator based on the PageRank [1], a popular network analysis method. PageRank estimates the influence of each product based on its centrality in the whole citation network. It alleviates some issues of the Citation Count indicator (e.g., two products with the same number of citations can have significantly different PageRank scores if the aggregated influence of the products citing them is very different - the product receiving citations from more influential products will get a larger score).

Popularity indicators (i.e., indicators of the "current" impact of each research product; how popular the product is currently)

RAM score: A popularity indicator based on the RAM [2] method. It is essentially a Citation Count where recent citations are considered as more important. This type of "time awareness" alleviates problems of methods like PageRank, which are biased against recently published products (new products need time to receive a number of citations that can be indicative for their impact).

AttRank score: A popularity indicator based on the AttRank [3] method. AttRank alleviates PageRank's bias against recently published products by incorporating an attention-based mechanism, akin to a time-restricted version of preferential attachment, to explicitly capture a researcher's preference to examine products which received a lot of attention recently.

Impulse indicators (i.e., indicators of the initial momentum that the research product received right after its publication)

Incubation Citation Count (3-year CC): This impulse indicator is a time-restricted version of the Citation Count, where the time window length is fixed for all products and the time window depends on the publication date of the product, i.e., only citations 3 years after each product's publication are counted.

More details about the aforementioned impact indicators, the way they are calculated and their interpretation can be found here and in the respective references (e.g., in [5]).

From version 5.1 onward, the impact indicators are calculated in two levels:

• The PID level (assuming that each PID corresponds to a distinct research product).
• The OpenAIRE-id level (leveraging PID synonyms based on OpenAIRE's deduplication algorithm [4] - each distinct article has its own OpenAIRE id).

Previous versions of the dataset only provided the scores at the PID level.

From version 12 onward, two types of PIDs are included in the dataset: DOIs and PMIDs (before that version, only DOIs were included).

Also, from version 7 onward, for each product in our files we also offer an impact class, which informs the user about the percentile into which the product score belongs compared to the impact scores of the rest products in the database. The impact classes are: C1 (in top 0.01%), C2 (in top 0.1%), C3 (in top 1%), C4 (in top 10%), and C5 (in bottom 90%).

Finally, before version 10, the calculation of the impact scores (and classes) was based on a citation network having one node for each product with a distinct PID that we could find in our input data sources. However, from version 10 onward, the nodes are deduplicated using the most recent version of the OpenAIRE article deduplication algorithm. This enabled a correction of the scores (more specifically, we avoid counting citation links multiple times when they are made by multiple versions of the same product). As a result, each node in the citation network we build is a deduplicated product having a distinct OpenAIRE id. We still report the scores at PID level (i.e., we assign a score to each of the versions/instances of the product), however these PID-level scores are just the scores of the respective deduplicated nodes propagated accordingly (i.e., all version of the same deduplicated product will receive the same scores). We have removed a small number of instances (having a PID) that were assigned (by error) to multiple deduplicated records in the OpenAIRE Graph.

For each calculation level (PID / OpenAIRE-id) we provide five (5) compressed CSV files (one for each measure/score provided) where each line follows the format "identifier

From version 9 onward, we also provide topic-specific impact classes for PID-identified products. In particular, we associated those products with 2nd level concepts from OpenAlex; we chose to keep only the three most dominant concepts for each product, based on their confidence score, and only if this score was greater than 0.3. Then, for each product and impact measure, we compute its class within its respective concepts. We provide finally the "topic_based_impact_classes.txt" file where each line follows the format "identifier

The data used to produce the citation network on which we calculated the provided measures have been gathered from the OpenAIRE Graph v7.1.0, including data from (a) OpenCitations' COCI & POCI dataset, (b) MAG [6,7], and (c) Crossref. The union of all distinct citations that could be found in these sources have been considered. In addition, versions later than v.10 leverage the filtering rules described here to remove from the dataset PIDs with problematic metadata.

References:

[1] R. Motwani L. Page, S. Brin and T. Winograd. 1999. The PageRank Citation Ranking: Bringing Order to the Web. Technical Report. Stanford InfoLab.

[2] Rumi Ghosh, Tsung-Ting Kuo, Chun-Nan Hsu, Shou-De Lin, and Kristina Lerman. 2011. Time-Aware Ranking in Dynamic Citation Networks. In Data Mining Workshops (ICDMW). 373–380

[3] I. Kanellos, T. Vergoulis, D. Sacharidis, T. Dalamagas, Y. Vassiliou: Ranking Papers by their Short-Term Scientific Impact. CoRR abs/2006.00951 (2020)

[4] P. Manghi, C. Atzori, M. De Bonis, A. Bardi, Entity deduplication in big data graphs for scholarly communication, Data Technologies and Applications (2020).

[5] I. Kanellos, T. Vergoulis, D. Sacharidis, T. Dalamagas, Y. Vassiliou: Impact-Based Ranking of Scientific Publications: A Survey and Experimental Evaluation. TKDE 2019 (early access)

[6] Arnab Sinha, Zhihong Shen, Yang Song, Hao Ma, Darrin Eide, Bo-June (Paul) Hsu, and Kuansan Wang. 2015. An Overview of Microsoft Academic Service (MA) and Applications. In Proceedings of the 24th International Conference on World Wide Web (WWW '15 Companion). ACM, New York, NY, USA, 243-246. DOI=http://dx.doi.org/10.1145/2740908.2742839

[7] K. Wang et al., "A Review of Microsoft Academic Services for Science of Science Studies", Frontiers in Big Data, 2019, doi: 10.3389/fdata.2019.00045

Find our Academic Search Engine built on top of these data here. Further note, that we also provide all calculated scores through BIP! Finder's API.

Terms of use: These data are provided "as is", without any warranties of any kind. The data are provided under the CC0 license.

More details about BIP! DB can be found in our relevant peer-reviewed publication:

Thanasis Vergoulis, Ilias Kanellos, Claudio Atzori, Andrea Mannocci, Serafeim Chatzopoulos, Sandro La Bruzzo, Natalia Manola, Paolo Manghi: BIP! DB: A Dataset of Impact Measures for Scientific Publications. WWW (Companion Volume) 2021: 456-460

We kindly request that any published research that makes use of BIP! DB cite the above article.