Recently big data and its applications had sharp growth in various fields such as IoT, bioinformatics, eCommerce, and social media. The huge volume of data incurred enormous challenges to the architecture, infrastructure, and computing capacity of IT systems. Therefore, the compelling need of the scientific and industrial community is large-scale and robust computing systems. Since one of the characteristics of big data is value, data should be published for analysts to extract useful patterns from them. However, data publishing may lead to the disclosure of individuals’ private information. Among the modern parallel computing platforms, Apache Spark is a fast and in-memory computing framework for large-scale data processing that provides high scalability by introducing the resilient distributed dataset (RDDs). In terms of performance, Due to in-memory computations, it is 100 times faster than Hadoop. Therefore, Apache Spark is one of the essential frameworks to implement distributed methods for privacy-preserving in big data publishing (PPBDP). This paper uses the RDD programming of Apache Spark to propose an efficient parallel implementation of a new computing model for big data anonymization. This computing model has three-phase of in-memory computations to address the runtime, scalability, and performance of large-scale data anonymization. The model supports partition-based data clustering algorithms to preserve the λ-diversity privacy model by using transformation and actions on RDDs. Therefore, the authors have investigated Spark-based implementation for preserving the λ-diversity privacy model by two designed City block and Pearson distance functions. The results of the paper provide a comprehensive guideline allowing the researchers to apply Apache Spark in their own researches.

The global big data and business analytics (BDA) market was valued at 168.8 billion U.S. dollars in 2018 and is forecast to grow to 215.7 billion U.S. dollars by 2021. In 2021, more than half of BDA spending will go towards services. IT services is projected to make up around 85 billion U.S. dollars, and business services will account for the remainder. Big data High volume, high velocity and high variety: one or more of these characteristics is used to define big data, the kind of data sets that are too large or too complex for traditional data processing applications. Fast-growing mobile data traffic, cloud computing traffic, as well as the rapid development of technologies such as artificial intelligence (AI) and the Internet of Things (IoT) all contribute to the increasing volume and complexity of data sets. For example, connected IoT devices are projected to generate 79.4 ZBs of data in 2025. Business analytics Advanced analytics tools, such as predictive analytics and data mining, help to extract value from the data and generate business insights. The size of the business intelligence and analytics software application market is forecast to reach around 16.5 billion U.S. dollars in 2022. Growth in this market is driven by a focus on digital transformation, a demand for data visualization dashboards, and an increased adoption of cloud.

Global Big Data Processing and Distribution Software market size 2025 was XX Million. Big Data Processing and Distribution Software Industry compound annual growth rate (CAGR) will be XX% from 2025 till 2033.

Snapshot img

Big Data Market Size 2024-2028

The big data market size is forecast to increase by USD 508.73 billion at a CAGR of 21.46% between 2023 and 2028.

The market is experiencing significant growth due to the growth in data generation from various sources, including IoT platforms and digital transformation services. This data deluge presents opportunities for businesses to leverage advanced analytics tools for applications such as fraud detection and prevention, workforce analytics, and business intelligence. However, the increasing adoption of big data implementation also brings challenges, including the need for data security and privacy measures. Quantum computing and blockchain technology are emerging trends In the big data landscape, offering potential solutions to complex data processing and security issues. In healthcare analytics, data protection regulations are driving the need for secure data management and sharing.
Additionally, supply chain optimization is another area where big data can bring significant value, enabling real-time monitoring and predictive analytics. Overall, the market is poised for continued growth, driven by the need to extract valuable insights from the vast amounts of data being generated.

What will be the Size of the Big Data Market During the Forecast Period?

Request Free Sample

The market is experiencing growth as businesses increasingly leverage information from vast datasets to drive strategic decision-making, enhance customer experiences, and improve operational efficiency. The digital revolution has led to an exponential increase in data creation, fueling demand for advanced analytics capabilities, real-time processing, and data protection and privacy solutions. Hardware and software companies offer on-premise and cloud-based systems to accommodate various industry needs, including customer analytics in retail and e-commerce, supply chain analytics in manufacturing, marketing analytics, pricing analytics, spatial analytics, workforce analytics, risk and credit analytics, transportation analytics, healthcare, energy and utilities, and IT and telecom. Big data applications span numerous sectors, enabling organizations to gain valuable insights from their data to optimize operations, mitigate risks, and innovate new products and services.

How is this Big Data Industry segmented and which is the largest segment?

The big data industry research report provides comprehensive data (region-wise segment analysis), with forecasts and estimates in 'USD billion' for the period 2024-2028, as well as historical data from 2018-2022 for the following segments.

Deployment

  On-premises
  Cloud-based
  Hybrid


Type

  Services
  Software


Geography

  North America

    Canada
    US


  Europe

    Germany
    UK


  APAC

    China


  South America



  Middle East and Africa

By Deployment Insights

The on-premises segment is estimated to witness significant growth during the forecast period. On-premises big data software solutions involve the installation of hardware and software by the end-user, granting them complete control over the system. Despite the high upfront costs, on-premises solutions offer advantages such as full ownership and operational efficiency. In contrast, cloud-based solutions require recurring monthly payments and involve data storage on companies' servers, increasing security concerns. Advanced analytics, real-time processing, and integrated analytics are key features driving the market. Data creation from digital transformation, customer experiences, and various industries like retail, healthcare, and finance, fuel the demand for scalable infrastructure and user-friendly interfaces. Technologies such as quantum computing, blockchain, AI-driven analytics platforms, and automation are transforming business intelligence solutions.

Ensuring data protection and privacy, accessibility, and seamless data transactions are crucial in this data-driven era. Key technologies include distributed computing, visualization tools, and social media. Target audiences range from decision-makers to various industries, including transportation, energy, and consumer engagement.

Get a glance at the market report of share of various segments Request Free Sample

The On-premises segment was valued at USD 86.53 billion in 2018 and showed a gradual increase during the forecast period.

Regional Analysis

North America is estimated to contribute 47% to the growth of the global market during the forecast period. Technavio's analysts have elaborately explained the regional trends and drivers that shape the market during the forecast period.

For more insights on the market size of various regions, Request Free Sample

The market in North America is experiencing significant growth due to digital transformation initiatives by enterprises in sectors such as healthcare, retail

Please cite the following paper when using this dataset:N. Thakur, "Twitter Big Data as a Resource for Exoskeleton Research: A Large-Scale Dataset of about 140,000 Tweets from 2017–2022 and 100 Research Questions", Journal of Analytics, Volume 1, Issue 2, 2022, pp. 72-97, DOI: https://doi.org/10.3390/analytics1020007AbstractThe exoskeleton technology has been rapidly advancing in the recent past due to its multitude of applications and diverse use cases in assisted living, military, healthcare, firefighting, and industry 4.0. The exoskeleton market is projected to increase by multiple times its current value within the next two years. Therefore, it is crucial to study the degree and trends of user interest, views, opinions, perspectives, attitudes, acceptance, feedback, engagement, buying behavior, and satisfaction, towards exoskeletons, for which the availability of Big Data of conversations about exoskeletons is necessary. The Internet of Everything style of today’s living, characterized by people spending more time on the internet than ever before, with a specific focus on social media platforms, holds the potential for the development of such a dataset by the mining of relevant social media conversations. Twitter, one such social media platform, is highly popular amongst all age groups, where the topics found in the conversation paradigms include emerging technologies such as exoskeletons. To address this research challenge, this work makes two scientific contributions to this field. First, it presents an open-access dataset of about 140,000 Tweets about exoskeletons that were posted in a 5-year period from 21 May 2017 to 21 May 2022. Second, based on a comprehensive review of the recent works in the fields of Big Data, Natural Language Processing, Information Retrieval, Data Mining, Pattern Recognition, and Artificial Intelligence that may be applied to relevant Twitter data for advancing research, innovation, and discovery in the field of exoskeleton research, a total of 100 Research Questions are presented for researchers to study, analyze, evaluate, ideate, and investigate based on this dataset.

This statistic shows the importance of big data analysis and machine learning technologies worldwide as of 2019. Tensorflow was seen as the most important big data analytics and machine learning technology, with 59 percent of respondents stating that it was important to critial for their organization.

Snapshot img

Big Data as a Service Market Size 2024-2028

The big data as a service market size is forecast to increase by USD 41.20 billion at a CAGR of 28.45% between 2023 and 2028.

The market is experiencing significant growth due to the increasing volume of data and the rising demand for advanced data insights. Machine learning algorithms and artificial intelligence are driving product quality and innovation in this sector. Hybrid cloud solutions are gaining popularity, offering the benefits of both private and public cloud platforms for optimal data storage and scalability. Industry standards for data privacy and security are increasingly important, as large amounts of data pose unique risks. The BDaaS market is expected to continue its expansion, providing valuable data insights to businesses across various industries.

What will be the Big Data as a Service Market Size During the Forecast Period?

Request Free Sample

Big Data as a Service (BDaaS) has emerged as a game-changer in the business world, enabling organizations to harness the power of big data without the need for extensive infrastructure and expertise. This service model offers various components such as data management, analytics, and visualization tools, enabling businesses to derive valuable insights from their data. BDaaS encompasses several key components that drive market growth. These include Business Intelligence (BI), Data Science, Data Quality, and Data Security. BI provides organizations with the ability to analyze data and gain insights to make informed decisions.



Data Science, on the other hand, focuses on extracting meaningful patterns and trends from large datasets using advanced algorithms. Data Quality is a critical component of BDaaS, ensuring that the data being analyzed is accurate, complete, and consistent. Data Security is another essential aspect, safeguarding sensitive data from cybersecurity threats and data breaches. Moreover, BDaaS offers various data pipelines, enabling seamless data integration and data lifecycle management. Network Analysis, Real-time Analytics, and Predictive Analytics are other essential components, providing businesses with actionable insights in real-time and enabling them to anticipate future trends. Data Mining, Machine Learning Algorithms, and Data Visualization Tools are other essential components of BDaaS.

How is this market segmented and which is the largest segment?

The market research report provides comprehensive data (region-wise segment analysis), with forecasts and estimates in 'USD billion' for the period 2024-2028, as well as historical data from 2018-2022 for the following segments.

Type

  Data analytics-as-a-Service
  Hadoop-as-a-service
  Data-as-a-service


Deployment

  Public cloud
  Hybrid cloud
  Private cloud


Geography

  North America

    Canada
    US


  APAC

    China


  Europe

    Germany
    UK


  South America



  Middle East and Africa

By Type Insights

The data analytics-as-a-service segment is estimated to witness significant growth during the forecast period.

Big Data as a Service (BDaaS) is a significant market segment, highlighted by the availability of Hadoop-as-a-Service solutions. These offerings enable businesses to access essential datasets on-demand without the burden of expensive infrastructure. DAaaS solutions facilitate real-time data analysis, empowering organizations to make informed decisions. The DAaaS landscape is expanding rapidly as companies acknowledge its value in enhancing internal data. Integrating DAaaS with big data systems amplifies analytics capabilities, creating a vibrant market landscape. Organizations can leverage diverse datasets to gain a competitive edge, driving the growth of the global BDaaS market. In the context of digital transformation, cloud computing, IoT, and 5G technologies, BDaaS solutions offer optimal resource utilization.

However, regulatory scrutiny poses challenges, necessitating stringent data security measures. Retail and other industries stand to benefit significantly from BDaaS, particularly with distributed computing solutions. DAaaS adoption is a strategic investment for businesses seeking to capitalize on the power of external data for valuable insights.

Get a glance at the market report of share of various segments Request Free Sample

The Data analytics-as-a-Service segment was valued at USD 2.59 billion in 2018 and showed a gradual increase during the forecast period.

Regional Analysis

North America is estimated to contribute 35% to the growth of the global market during the forecast period.

Technavio's analysts have elaborately explained the regional trends and drivers that shape the market during the forecast period.

For more insights on the market share of various regions Request Free Sample

Big Data as a Service Market analysis, North America is experiencing signif

The Data Analytics in Retail Industry is segmented by Application (Merchandising and Supply Chain Analytics, Social Media Analytics, Customer Analytics, Operational Intelligence, Other Applications), by Business Type (Small and Medium Enterprises, Large-scale Organizations), and Geography. The market size and forecasts are provided in terms of value (USD billion) for all the above segments.

The big data security market is projected to be valued at US$ 20,418.4 million in 2023 and is expected to rise to US$ 72,652.6 million by 2033. The sales of big data security are expected to record a significant CAGR of 13.5% during the forecast period.

Scope of the Report

Question Paper Solutions of chapter Hdfs-Hive and Hiveql-Hbase of Big Data Analysis, 8th Semester , Applied Electronics and Instrumentation Engineering

This statistic depicts the revenue generated by the big data services market in the Asia Pacific (excluding Japan) from 2012 to 2014, as well as a forecast of revenue from 2015 to 2017. In 2014, revenues associated with the big data services market in the Asia Pacific amounted to 290 million U.S. dollars. 'Big data' refers to data sets that are too large or too complex for traditional data processing applications. Additionally, the term is often used to refer to the technologies that enable predictive analytics or other methods of extracting value from data.

In 2022, around 77 percent of companies that used big data analysis and related services in South Korea did so with public data. Following this was the analysis of customer information, at around 39 percent.

The open-source big data tools market is experiencing robust growth, driven by the increasing need for scalable, cost-effective data management and analysis solutions across diverse sectors. The market, estimated at $15 billion in 2025, is projected to exhibit a Compound Annual Growth Rate (CAGR) of 18% from 2025 to 2033. This expansion is fueled by several key factors. Firstly, the rising volume and velocity of data generated across industries, from banking and finance to manufacturing and government, necessitate powerful and adaptable tools. Secondly, the cost-effectiveness and flexibility of open-source solutions compared to proprietary alternatives are major drawcards, especially for smaller organizations and startups. The ease of customization and community support further enhance their appeal. Growth is also being propelled by technological advancements such as the development of more sophisticated data analytics tools, improved cloud integration, and increased adoption of containerization technologies like Docker and Kubernetes for deployment and management. The market's segmentation across application (banking, manufacturing, etc.) and tool type (data collection, storage, analysis) reflects the diverse range of uses and specialized tools available. Key restraints to market growth include the complexity associated with implementing and managing open-source solutions, requiring skilled personnel and ongoing maintenance. Security concerns and the need for robust data governance frameworks also pose challenges. However, the growing maturity of the open-source ecosystem, coupled with the emergence of managed services providers offering support and expertise, is mitigating these limitations. The continued advancements in artificial intelligence (AI) and machine learning (ML) are further integrating with open-source big data tools, creating synergistic opportunities for growth in predictive analytics and advanced data processing. This integration, alongside the ever-increasing volume of data needing analysis, will undoubtedly drive continued market expansion over the forecast period.

According to the Cognitive Market Research Report, the Data Processing and Hosting Service market size in 2023 was XX Million and is projected to have a compounded annual growth rate of XX% from 2024 to 2031. The emergence of cloud-based platforms and the growing number of small and medium enterprises are driving the market growth of Data Processing and Hosting Services. This market is further segmented by type, application, and deployment. The shared hosting under product type, public website, and public deployment holds the dominant share in the data processing and hosting service. The market is divided into shared hosting, dedicated hosting, collocated hosting, virtual private server hosting, managed hosting, self-managed hosting, and others. The shared hosting sector leads the market since small and medium-sized businesses choose shared servers over other forms of hosting. The Asia-Pacific region is the most dominant due to its high share of the global internet population and major organizations' and SMEs' quick adoption of cloud services The Data Processing and Hosting Services Market is relatively competitive, with significant companies including GoDaddy Operating Company LLC, Bluehost (Endurance International Group), HostGator.com LLC, Hostinger International, Ltd., and Amazon Web Services Inc. Some players presently have a large market share. However, as hosting solutions for professional services progress, new firms are strengthening their market presence, consequently expanding their corporate footprint into emerging markets.

Market Dynamics of Data Processing And Hosting Service

Key Drivers

Web Hosting is gaining traction due to the emergence of cloud-based platforms.

Web hosting services are gaining pace in response to increased customer demand for web hosting services that are appropriate for their needs. Furthermore, the increased acceptance of cloud services in organizations is opening up new potential for the web hosting market over time. The rise of the cloud has had a massive impact on data management and hosting services. It is a low-cost way for businesses to make use of current technology and design without incurring the high upfront costs of acquiring, installing, and configuring the necessary hardware, software, and infrastructure. Furthermore, major firms were able to swiftly adapt to a developing data-driven economy by leveraging their current resources and competencies to manage it efficiently. Furthermore, SMBs globally are increasingly demanding cloud-based hosting services, which is likely to boost the web hosting sector throughout the projection period. The move to the cloud makes it easier to create programmes that users can use in their browsers rather than downloading on their devices. This greatly accelerates market expansion. Furthermore, with the introduction of web-based applications, app building became so simple that hosting several apps on a single server became straightforward. For instance, Hostinger International Ltd. is a well-known web hosting firm that offers hosting solutions. Hostinger is a trustworthy web hosting company. They offer fast loading speeds and excellent uptime rates to ensure that users may access the site anytime they want. Hostinger also provides knowledgeable and courteous customer service that is available around the clock. (Source: https://www.hostinger.in/about#:~:text=Hostinger%20is%20one%20of%20the,Hostinger%20and%20hustle%20with%20us) Therefore, the emergence of cloud-based platforms has expanded the data processing and hosting service market.

Growing small and medium enterprises and their requirement of increasing internet penetration are driving market growth. 

Small and medium-sized enterprises (SMEs) are critical to the economic prosperity of any country. The existence of SMEs promotes efficient usage of adjacent assets and boosts economies throughout the world. According to Siteefy, there are 1.13 billion websites globally, but only 200,121,724 are regularly accessed and maintained in 2023. Companies have recognised the necessity of an online presence, particularly in the aftermath of the COVID-19 epidemic. This enables them to reach a larger audience while remaining competitive in today's digital world. Consumers are increasingly relying on the internet to make purchases, indicating corporate success. As a result, small businesses have recognised the v...

The need for advanced analytical approaches to provide HPDA solutions is driving the market growth of High Performance Data Analytics (HPDA). According to the analyst from Verified Market Research, The High Performance Data Analytics (HPDA) Market is estimated to reach a valuation of USD 597.06 Billion over the forecast period 2031, by subjugating around USD 113.23 Billion in 2023.

The adoption of an open-source framework for big data analytics is driving market growth. This surge in demand enables the market to grow at a CAGR of 23.1% from 2024 to 2031.

High Performance Data Analytics (HPDA) Market: Definition/ Overview

HPDA refers to big data analytics that uses High-Performance Computing (HPC) techniques. Big data analytics has always relied on high-performance computing (HPC), but as data grows exponentially, new forms of high-performance computing will be required to access previously unimaginable volumes of data. The combination of big data analytics and high-performance computing is called “high-performance data analytics.” High-performance data analytics is the process of quickly finding insights from large data sets by running powerful analytical tools in parallel on high-performance computing systems.

Furthermore, high-performance data analytics infrastructure is a rapidly expanding market for government and commercial organizations that need to combine high-performance computing with data-intensive analysis. For complex modeling and simulations, big data analytics techniques like Hadoop and Spark have long required high-performance computing, which they lack.

The USDA Agricultural Research Service (ARS) recently established SCINet , which consists of a shared high performance computing resource, Ceres, and the dedicated high-speed Internet2 network used to access Ceres. Current and potential SCINet users are using and generating very large datasets so SCINet needs to be provisioned with adequate data storage for their active computing. It is not designed to hold data beyond active research phases. At the same time, the National Agricultural Library has been developing the Ag Data Commons, a research data catalog and repository designed for public data release and professional data curation. Ag Data Commons needs to anticipate the size and nature of data it will be tasked with handling. The ARS Web-enabled Databases Working Group, organized under the SCINet initiative, conducted a study to establish baseline data storage needs and practices, and to make projections that could inform future infrastructure design, purchases, and policies. The SCINet Web-enabled Databases Working Group helped develop the survey which is the basis for an internal report. While the report was for internal use, the survey and resulting data may be generally useful and are being released publicly. From October 24 to November 8, 2016 we administered a 17-question survey (Appendix A) by emailing a Survey Monkey link to all ARS Research Leaders, intending to cover data storage needs of all 1,675 SY (Category 1 and Category 4) scientists. We designed the survey to accommodate either individual researcher responses or group responses. Research Leaders could decide, based on their unit's practices or their management preferences, whether to delegate response to a data management expert in their unit, to all members of their unit, or to themselves collate responses from their unit before reporting in the survey.
Larger storage ranges cover vastly different amounts of data so the implications here could be significant depending on whether the true amount is at the lower or higher end of the range. Therefore, we requested more detail from "Big Data users," those 47 respondents who indicated they had more than 10 to 100 TB or over 100 TB total current data (Q5). All other respondents are called "Small Data users." Because not all of these follow-up requests were successful, we used actual follow-up responses to estimate likely responses for those who did not respond. We defined active data as data that would be used within the next six months. All other data would be considered inactive, or archival. To calculate per person storage needs we used the high end of the reported range divided by 1 for an individual response, or by G, the number of individuals in a group response. For Big Data users we used the actual reported values or estimated likely values.

Resources in this dataset:Resource Title: Appendix A: ARS data storage survey questions. File Name: Appendix A.pdfResource Description: The full list of questions asked with the possible responses. The survey was not administered using this PDF but the PDF was generated directly from the administered survey using the Print option under Design Survey. Asterisked questions were required. A list of Research Units and their associated codes was provided in a drop down not shown here. Resource Software Recommended: Adobe Acrobat,url: https://get.adobe.com/reader/ Resource Title: CSV of Responses from ARS Researcher Data Storage Survey. File Name: Machine-readable survey response data.csvResource Description: CSV file includes raw responses from the administered survey, as downloaded unfiltered from Survey Monkey, including incomplete responses. Also includes additional classification and calculations to support analysis. Individual email addresses and IP addresses have been removed. This information is that same data as in the Excel spreadsheet (also provided).Resource Title: Responses from ARS Researcher Data Storage Survey. File Name: Data Storage Survey Data for public release.xlsxResource Description: MS Excel worksheet that Includes raw responses from the administered survey, as downloaded unfiltered from Survey Monkey, including incomplete responses. Also includes additional classification and calculations to support analysis. Individual email addresses and IP addresses have been removed.Resource Software Recommended: Microsoft Excel,url: https://products.office.com/en-us/excel

The total amount of data created, captured, copied, and consumed globally is forecast to increase rapidly, reaching 149 zettabytes in 2024. Over the next five years up to 2028, global data creation is projected to grow to more than 394 zettabytes. In 2020, the amount of data created and replicated reached a new high. The growth was higher than previously expected, caused by the increased demand due to the COVID-19 pandemic, as more people worked and learned from home and used home entertainment options more often. Storage capacity also growing Only a small percentage of this newly created data is kept though, as just two percent of the data produced and consumed in 2020 was saved and retained into 2021. In line with the strong growth of the data volume, the installed base of storage capacity is forecast to increase, growing at a compound annual growth rate of 19.2 percent over the forecast period from 2020 to 2025. In 2020, the installed base of storage capacity reached 6.7 zettabytes.

To explore the application effect of the deep learning (DL) network model in the Internet of Things (IoT) database query and optimization. This study first analyzes the architecture of IoT database queries, then explores the DL network model, and finally optimizes the DL network model through optimization strategies. The advantages of the optimized model in this study are verified through experiments. Experimental results show that the optimized model has higher efficiency than other models in the model training and parameter optimization stages. Especially when the data volume is 2000, the model training time and parameter optimization time of the optimized model are remarkably lower than that of the traditional model. In terms of resource consumption, the Central Processing Unit and Graphics Processing Unit usage and memory usage of all models have increased as the data volume rises. However, the optimized model exhibits better performance on energy consumption. In throughput analysis, the optimized model can maintain high transaction numbers and data volumes per second when handling large data requests, especially at 4000 data volumes, and its peak time processing capacity exceeds that of other models. Regarding latency, although the latency of all models increases with data volume, the optimized model performs better in database query response time and data processing latency. The results of this study not only reveal the optimized model’s superior performance in processing IoT database queries and their optimization but also provide a valuable reference for IoT data processing and DL model optimization. These findings help to promote the application of DL technology in the IoT field, especially in the need to deal with large-scale data and require efficient processing scenarios, and offer a vital reference for the research and practice in related fields.

This dataset is about book subjects and is filtered where the books is Beginning big data with Power BI and Excel 2013 : big data processing and analysis using Power BI in Excel 2013, featuring 10 columns including authors, average publication date, book publishers, book subject, and books. The preview is ordered by number of books (descending).

Scientific and related management challenges in the water domain require synthesis of data from multiple domains. Many data analysis tasks are difficult because datasets are large and complex; standard formats for data types are not always agreed upon nor mapped to an efficient structure for analysis; water scientists may lack training in methods needed to efficiently tackle large and complex datasets; and available tools can make it difficult to share, collaborate around, and reproduce scientific work. Overcoming these barriers to accessing, organizing, and preparing datasets for analyses will be an enabler for transforming scientific inquiries. Building on the HydroShare repository’s established cyberinfrastructure, we have advanced two packages for the Python language that make data loading, organization, and curation for analysis easier, reducing time spent in choosing appropriate data structures and writing code to ingest data. These packages enable automated retrieval of data from HydroShare and the USGS’s National Water Information System (NWIS), loading of data into performant structures keyed to specific scientific data types and that integrate with existing visualization, analysis, and data science capabilities available in Python, and then writing analysis results back to HydroShare for sharing and eventual publication. These capabilities reduce the technical burden for scientists associated with creating a computational environment for executing analyses by installing and maintaining the packages within CUAHSI’s HydroShare-linked JupyterHub server. HydroShare users can leverage these tools to build, share, and publish more reproducible scientific workflows. The HydroShare Python Client and USGS NWIS Data Retrieval packages can be installed within a Python environment on any computer running Microsoft Windows, Apple MacOS, or Linux from the Python Package Index using the PIP utility. They can also be used online via the CUAHSI JupyterHub server (https://jupyterhub.cuahsi.org/) or other Python notebook environments like Google Collaboratory (https://colab.research.google.com/). Source code, documentation, and examples for the software are freely available in GitHub at https://github.com/hydroshare/hsclient/ and https://github.com/USGS-python/dataretrieval.

This presentation was delivered as part of the Hawai'i Data Science Institute's regular seminar series: https://datascience.hawaii.edu/event/data-science-and-analytics-for-water/