Big data, with N × P dimension where N is extremely large, has created new challenges for data analysis, particularly in the realm of creating meaningful clusters of data. Clustering techniques, such as K-means or hierarchical clustering are popular methods for performing exploratory analysis on large datasets. Unfortunately, these methods are not always possible to apply to big data due to memory or time constraints generated by calculations of order P*N(N−1)2. To circumvent this problem, typically the clustering technique is applied to a random sample drawn from the dataset; however, a weakness is that the structure of the dataset, particularly at the edges, is not necessarily maintained. We propose a new solution through the concept of “data nuggets”, which reduces a large dataset into a small collection of nuggets of data, each containing a center, weight, and scale parameter. The data nuggets are then input into algorithms that compute methods such as principal components analysis and clustering in a more computationally efficient manner. We show the consistency of the data nuggets based covariance estimator and apply the methodology of data nuggets to perform exploratory analysis of a flow cytometry dataset containing over one million observations using PCA and K-means clustering for weighted observations. Supplementary materials for this article are available online.

The Big Data Technology and Services market is experiencing robust growth, projected to reach a market size of $38.28 billion in 2025. While the provided CAGR is missing, considering the rapid adoption of big data solutions across various industries and the continuous innovation in areas like AI and machine learning, a conservative estimate of a 15% CAGR for the forecast period (2025-2033) seems plausible. This would translate to significant market expansion over the next decade. Key drivers include the increasing volume of data generated by businesses and individuals, the need for improved data analytics capabilities for better decision-making, and the growing adoption of cloud-based big data solutions. Furthermore, the rising demand for real-time data processing and insights across sectors like finance, healthcare, and retail fuels market growth. While data security and privacy concerns represent a restraint, the development of robust security protocols and regulatory frameworks is mitigating this risk. The market is segmented across various technologies (e.g., Hadoop, NoSQL databases, data warehousing), services (e.g., data integration, data analytics, consulting), and deployment models (cloud, on-premise). Leading players like IBM, Microsoft, and others are constantly innovating and expanding their offerings, fostering competition and driving market evolution. The market's growth is further propelled by trends such as the increasing adoption of advanced analytics techniques, the integration of big data with IoT (Internet of Things) devices, and the rising demand for specialized big data skills. The diverse applications of big data across various sectors ensure sustained growth, creating opportunities for both established players and emerging startups. The competitive landscape is characterized by a mix of large technology vendors and specialized service providers, with ongoing mergers and acquisitions shaping the market structure. Continued investment in research and development in areas like data visualization and predictive analytics will be crucial for maintaining the market's momentum. Geographical expansion into developing economies presents further growth opportunities. The predicted CAGR and market size reflect a strong growth trajectory, making it an attractive investment opportunity for stakeholders.

Question Paper Solutions of chapter Introduction to Hadoop and Hadoop Architecture of Big Data Analysis, 8th Semester , Applied Electronics and Instrumentation Engineering

Psychology still relies often on questionnaires to gather data. Despite ubiquitous computing and internet, these are often still conducted with paper and clipboard. Where the internet is used, standalone questionnaires from bulk providers like Survey Monkey are the norm. For our studies related to social networks on online disclosure, we have developed a custom site for online questionnaires, designed to engage participants and allow linking of data from one study to the next – PsyQu.com PsyQu is a modern website developed around a database and as such has a ‘schema’. This data structure encapsulates the project, researcher(s) and participant(s) in a manner that allows for participants to link multiple attempts at multiple studies under their single account. This will allow cross-linked and longitudinal studies to be performed. By moving beyond standalone questionnaires, we hope to discover new correlative and predictive patterns between online behavior and other psychological dimensions. At present, the site is in alpha testing mode with only 1 group of researchers and 3 studies: social capital, online self-disclosure and personality. In the social capital study, we used a standard scale for investigating online social capital and social trust, in an attempt to find out differences between various groups. A paper survey was also conducted in order to compare with the online survey since there has been debate on the reliability of online participation. We will present the website, initial results of the social capital study.

The Data Architecture Modernization market is experiencing robust growth, driven by the increasing need for businesses to adapt to the ever-evolving digital landscape. The market's expansion is fueled by several key factors, including the rising adoption of cloud computing, the exponential growth of data volumes, and the imperative for enhanced data security and compliance. Organizations are increasingly recognizing the strategic value of modernizing their data architectures to improve agility, scalability, and overall efficiency. This modernization involves migrating legacy systems to cloud-based platforms, adopting advanced analytics tools, and implementing robust data governance frameworks. The shift towards real-time data processing and the increasing demand for data-driven decision-making are further accelerating market growth. Competition is fierce, with established players like NTT DATA and Rackspace competing alongside specialized analytics firms and cloud providers. The market is segmented based on deployment type (cloud, on-premise, hybrid), organization size (small, medium, large enterprises), and industry vertical (BFSI, healthcare, retail, etc.). While the precise market size is unavailable, reasonable estimates suggest a substantial and rapidly growing market exceeding $10 billion by 2025, exhibiting a Compound Annual Growth Rate (CAGR) of approximately 15% over the forecast period (2025-2033). Despite the rapid growth, challenges remain. These include the complexities of migrating legacy systems, the need for skilled professionals experienced in modern data architectures, and the high initial investment costs associated with modernization projects. Data security and privacy concerns also pose significant hurdles. However, the long-term benefits of improved data management, enhanced operational efficiency, and the ability to gain valuable insights from data are expected to outweigh these challenges, driving continued market growth. The market’s future hinges on ongoing technological innovation, the increasing affordability of cloud-based solutions, and the growing awareness among businesses of the importance of data-driven decision-making.

The research & publication Big Data Analytics in Business for Marketing Research: A Retrospective of Domain and Knowledge Structure, which was indexed by Scopus between 2011 to 2024. The data contains 448 documents data: authors, authors ID Sggggg, title, year, source title, volume, issue, article number in Scopus DOJ, link, affiliation, abstract, index keywords, references, corespondence address, editors, publisher, conference name, conference date, conference code, ISSN. language, document type, access type, and EID

International Journal of Engineering and Advanced Technology Acceptance Rate - ResearchHelpDesk - International Journal of Engineering and Advanced Technology (IJEAT) is having Online-ISSN 2249-8958, bi-monthly international journal, being published in the months of February, April, June, August, October, and December by Blue Eyes Intelligence Engineering & Sciences Publication (BEIESP) Bhopal (M.P.), India since the year 2011. It is academic, online, open access, double-blind, peer-reviewed international journal. It aims to publish original, theoretical and practical advances in Computer Science & Engineering, Information Technology, Electrical and Electronics Engineering, Electronics and Telecommunication, Mechanical Engineering, Civil Engineering, Textile Engineering and all interdisciplinary streams of Engineering Sciences. All submitted papers will be reviewed by the board of committee of IJEAT. Aim of IJEAT Journal disseminate original, scientific, theoretical or applied research in the field of Engineering and allied fields. dispense a platform for publishing results and research with a strong empirical component. aqueduct the significant gap between research and practice by promoting the publication of original, novel, industry-relevant research. seek original and unpublished research papers based on theoretical or experimental works for the publication globally. publish original, theoretical and practical advances in Computer Science & Engineering, Information Technology, Electrical and Electronics Engineering, Electronics and Telecommunication, Mechanical Engineering, Civil Engineering, Textile Engineering and all interdisciplinary streams of Engineering Sciences. impart a platform for publishing results and research with a strong empirical component. create a bridge for a significant gap between research and practice by promoting the publication of original, novel, industry-relevant research. solicit original and unpublished research papers, based on theoretical or experimental works. Scope of IJEAT International Journal of Engineering and Advanced Technology (IJEAT) covers all topics of all engineering branches. Some of them are Computer Science & Engineering, Information Technology, Electronics & Communication, Electrical and Electronics, Electronics and Telecommunication, Civil Engineering, Mechanical Engineering, Textile Engineering and all interdisciplinary streams of Engineering Sciences. The main topic includes but not limited to: 1. Smart Computing and Information Processing Signal and Speech Processing Image Processing and Pattern Recognition WSN Artificial Intelligence and machine learning Data mining and warehousing Data Analytics Deep learning Bioinformatics High Performance computing Advanced Computer networking Cloud Computing IoT Parallel Computing on GPU Human Computer Interactions 2. Recent Trends in Microelectronics and VLSI Design Process & Device Technologies Low-power design Nanometer-scale integrated circuits Application specific ICs (ASICs) FPGAs Nanotechnology Nano electronics and Quantum Computing 3. Challenges of Industry and their Solutions, Communications Advanced Manufacturing Technologies Artificial Intelligence Autonomous Robots Augmented Reality Big Data Analytics and Business Intelligence Cyber Physical Systems (CPS) Digital Clone or Simulation Industrial Internet of Things (IIoT) Manufacturing IOT Plant Cyber security Smart Solutions – Wearable Sensors and Smart Glasses System Integration Small Batch Manufacturing Visual Analytics Virtual Reality 3D Printing 4. Internet of Things (IoT) Internet of Things (IoT) & IoE & Edge Computing Distributed Mobile Applications Utilizing IoT Security, Privacy and Trust in IoT & IoE Standards for IoT Applications Ubiquitous Computing Block Chain-enabled IoT Device and Data Security and Privacy Application of WSN in IoT Cloud Resources Utilization in IoT Wireless Access Technologies for IoT Mobile Applications and Services for IoT Machine/ Deep Learning with IoT & IoE Smart Sensors and Internet of Things for Smart City Logic, Functional programming and Microcontrollers for IoT Sensor Networks, Actuators for Internet of Things Data Visualization using IoT IoT Application and Communication Protocol Big Data Analytics for Social Networking using IoT IoT Applications for Smart Cities Emulation and Simulation Methodologies for IoT IoT Applied for Digital Contents 5. Microwaves and Photonics Microwave filter Micro Strip antenna Microwave Link design Microwave oscillator Frequency selective surface Microwave Antenna Microwave Photonics Radio over fiber Optical communication Optical oscillator Optical Link design Optical phase lock loop Optical devices 6. Computation Intelligence and Analytics Soft Computing Advance Ubiquitous Computing Parallel Computing Distributed Computing Machine Learning Information Retrieval Expert Systems Data Mining Text Mining Data Warehousing Predictive Analysis Data Management Big Data Analytics Big Data Security 7. Energy Harvesting and Wireless Power Transmission Energy harvesting and transfer for wireless sensor networks Economics of energy harvesting communications Waveform optimization for wireless power transfer RF Energy Harvesting Wireless Power Transmission Microstrip Antenna design and application Wearable Textile Antenna Luminescence Rectenna 8. Advance Concept of Networking and Database Computer Network Mobile Adhoc Network Image Security Application Artificial Intelligence and machine learning in the Field of Network and Database Data Analytic High performance computing Pattern Recognition 9. Machine Learning (ML) and Knowledge Mining (KM) Regression and prediction Problem solving and planning Clustering Classification Neural information processing Vision and speech perception Heterogeneous and streaming data Natural language processing Probabilistic Models and Methods Reasoning and inference Marketing and social sciences Data mining Knowledge Discovery Web mining Information retrieval Design and diagnosis Game playing Streaming data Music Modelling and Analysis Robotics and control Multi-agent systems Bioinformatics Social sciences Industrial, financial and scientific applications of all kind 10. Advanced Computer networking Computational Intelligence Data Management, Exploration, and Mining Robotics Artificial Intelligence and Machine Learning Computer Architecture and VLSI Computer Graphics, Simulation, and Modelling Digital System and Logic Design Natural Language Processing and Machine Translation Parallel and Distributed Algorithms Pattern Recognition and Analysis Systems and Software Engineering Nature Inspired Computing Signal and Image Processing Reconfigurable Computing Cloud, Cluster, Grid and P2P Computing Biomedical Computing Advanced Bioinformatics Green Computing Mobile Computing Nano Ubiquitous Computing Context Awareness and Personalization, Autonomic and Trusted Computing Cryptography and Applied Mathematics Security, Trust and Privacy Digital Rights Management Networked-Driven Multicourse Chips Internet Computing Agricultural Informatics and Communication Community Information Systems Computational Economics, Digital Photogrammetric Remote Sensing, GIS and GPS Disaster Management e-governance, e-Commerce, e-business, e-Learning Forest Genomics and Informatics Healthcare Informatics Information Ecology and Knowledge Management Irrigation Informatics Neuro-Informatics Open Source: Challenges and opportunities Web-Based Learning: Innovation and Challenges Soft computing Signal and Speech Processing Natural Language Processing 11. Communications Microstrip Antenna Microwave Radar and Satellite Smart Antenna MIMO Antenna Wireless Communication RFID Network and Applications 5G Communication 6G Communication 12. Algorithms and Complexity Sequential, Parallel And Distributed Algorithms And Data Structures Approximation And Randomized Algorithms Graph Algorithms And Graph Drawing On-Line And Streaming Algorithms Analysis Of Algorithms And Computational Complexity Algorithm Engineering Web Algorithms Exact And Parameterized Computation Algorithmic Game Theory Computational Biology Foundations Of Communication Networks Computational Geometry Discrete Optimization 13. Software Engineering and Knowledge Engineering Software Engineering Methodologies Agent-based software engineering Artificial intelligence approaches to software engineering Component-based software engineering Embedded and ubiquitous software engineering Aspect-based software engineering Empirical software engineering Search-Based Software engineering Automated software design and synthesis Computer-supported cooperative work Automated software specification Reverse engineering Software Engineering Techniques and Production Perspectives Requirements engineering Software analysis, design and modelling Software maintenance and evolution Software engineering tools and environments Software engineering decision support Software design patterns Software product lines Process and workflow management Reflection and metadata approaches Program understanding and system maintenance Software domain modelling and analysis Software economics Multimedia and hypermedia software engineering Software engineering case study and experience reports Enterprise software, middleware, and tools Artificial intelligent methods, models, techniques Artificial life and societies Swarm intelligence Smart Spaces Autonomic computing and agent-based systems Autonomic computing Adaptive Systems Agent architectures, ontologies, languages and protocols Multi-agent systems Agent-based learning and knowledge discovery Interface agents Agent-based auctions and marketplaces Secure mobile and multi-agent systems Mobile agents SOA and Service-Oriented Systems Service-centric software engineering Service oriented requirements engineering Service oriented architectures Middleware for service based systems Service discovery and composition Service level

Our GeoThermalCloud framework is designed to process geothermal datasets using a novel toolbox for unsupervised and physics-informed machine learning called SmartTensors. More information about GeoThermalCloud can be found at the GeoThermalCloud GitHub Repository. More information about SmartTensors can be found at the SmartTensors Github Repository and the SmartTensors page at LANL.gov. Links to these pages are included in this submission. GeoThermalCloud.jl is a repository containing all the data and codes required to demonstrate applications of machine learning methods for geothermal exploration. GeoThermalCloud.jl includes: - site data - simulation scripts - jupyter notebooks - intermediate results - code outputs - summary figures - readme markdown files GeoThermalCloud.jl showcases the machine learning analyses performed for the following geothermal sites: - Brady: geothermal exploration of the Brady geothermal site, Nevada - SWNM: geothermal exploration of the Southwest New Mexico (SWNM) region - GreatBasin: geothermal exploration of the Great Basin region, Nevada Reports, research papers, and presentations summarizing these machine learning analyses are also available and will be posted soon.

[Context] Data-intensive systems, a.k.a. big data systems (BDS), are software systems that handle a large volume of data in the presence of performance quality attributes, such as scalability and availability. Before the advent of big data management systems (e.g. Cassandra) and frameworks (e.g. Spark), organizations had to cope with large data volumes with custom-tailored solutions. In particular, a decade ago, Tecgraf/PUC-Rio developed a system to monitor truck fleet in real-time and proactively detect events from the positioning data received. Over the years, the system evolved into a complex and large obsolescent code base involving a hard maintenance process. [Goal] We report our experience on replacing a legacy BDS with a microservice-based event-driven system. [Method] We applied action research, investigating the reasons that motivate the adoption of a microservice-based event-driven architecture, intervening to define the new architecture, and documenting the challenges and lessons learned. [Results] We perceived that the resulting architecture enabled easier maintenance and fault-isolation. However, the myriad of technologies and the complex data flow were perceived as drawbacks. Based on the challenges faced, we highlight opportunities to improve the design of big data reactive systems. [Conclusions] We believe that our experience provides helpful takeaways for practitioners modernizing systems with data-intensive requirements.

The global Data Preparation Platform market is poised for substantial growth, estimated to reach $15,600 million by the study's end in 2033, up from $6,000 million in the base year of 2025. This trajectory is fueled by a Compound Annual Growth Rate (CAGR) of approximately 12.5% over the forecast period. The proliferation of big data and the increasing need for clean, usable data across all business functions are primary drivers. Organizations are recognizing that effective data preparation is foundational to accurate analytics, informed decision-making, and successful AI/ML initiatives. This has led to a surge in demand for platforms that can automate and streamline the complex, time-consuming process of data cleansing, transformation, and enrichment. The market's expansion is further propelled by the growing adoption of cloud-based solutions, offering scalability, flexibility, and cost-efficiency, particularly for Small & Medium Enterprises (SMEs). Key trends shaping the Data Preparation Platform market include the integration of AI and machine learning for automated data profiling and anomaly detection, enhanced collaboration features to facilitate teamwork among data professionals, and a growing focus on data governance and compliance. While the market exhibits robust growth, certain restraints may temper its pace. These include the complexity of integrating data preparation tools with existing IT infrastructures, the shortage of skilled data professionals capable of leveraging advanced platform features, and concerns around data security and privacy. Despite these challenges, the market is expected to witness continuous innovation and strategic partnerships among leading companies like Microsoft, Tableau, and Alteryx, aiming to provide more comprehensive and user-friendly solutions to meet the evolving demands of a data-driven world. Here's a comprehensive report description on Data Preparation Platforms, incorporating the requested information, values, and structure:

This dataset provides panel data from 82 industrial organizations, each observed consistently over a 5-year period (2019–2023). The dataset is designed to support analysis of how machine learning (ML) and big data technologies are integrated into smart maintenance operations across different industrial sectors. Each organization is uniquely identified and assigned a fixed organizational structure—either centralized, semi-centralized, or decentralized—that remains constant across time.

The dataset includes the following variables:

• maintenance_cost_reduction: annual percentage change in maintenance costs, representing efficiency gains or losses.
• model_prediction_accuracy: the accuracy (%) of predictive ML models in forecasting failures.
• failure_risk_score: a normalized score (0–1) indicating the predicted risk of system failure.
• model_training_frequency: the regularity of model retraining (monthly, quarterly, yearly).
• data_pipeline_latency_hr: average latency (in hours) in processing and transmitting maintenance data through digital pipelines.
• bigdata_storage_utilization_percent: percentage utilization of the organization’s big data storage infrastructure.

The organizations represented in this dataset operate across advanced industrial sectors such as manufacturing, transportation, utilities, energy, and aerospace logistics. Geographically, the entities are based in the United States, Germany, South Korea, Japan, and the Netherlands, countries recognized for their leadership in AI integration, industrial analytics, and data-driven operations.

Data was gathered through structured interviews with IT specialists, plant maintenance managers, and operational analytics teams. The data design reflects realistic organizational behaviors and technological performance patterns, making it well-suited for research on predictive maintenance, digital infrastructure readiness, and performance benchmarking in smart manufacturing.

Additional file 2: Table S1. The list of selected activity types in the PubChem.

NASA has the aim of researching aviation Real-time System-wide Safety Assurance (RSSA) with a focus on the development of prognostic decision support tools as one of its new aeronautics research pillars. The vision of RSSA is to accelerate the discovery of previously unknown safety threats in real time and enable rapid mitigation of safety risks through analysis of massive amounts of aviation data. Our innovation supports this vision by designing a hybrid architecture combining traditional database technology and real-time streaming analytics in a Big Data environment. The innovation includes three major components: a Batch Processing framework, Traditional Databases and Streaming Analytics. It addresses at least three major needs within the aviation safety community. First, the innovation supports the creation of future data-driven safety prognostic decision support tools that must pull data from heterogeneous data sources and seamlessly combine them to be effective for NAS stakeholders. Second, our innovation opens up the possibility to provide real-time NAS performance analytics desired by key aviation stakeholders. Third, our proposed architecture provides a mechanism for safety risk accuracy evaluations. To accomplish this innovation, we have three technical objectives and related work plan efforts. The first objective is the determination of the system and functional requirements. We identify the system and functional requirements from aviation safety stakeholders for a set of use cases by investigating how they would use the system and what data processing functions they need to support their decisions. The second objective is to create a Big Data technology-driven architecture. Here we explore and identify the best technologies for the components in the system including Big Data processing and architectural techniques adapted for aviation data applications. Finally, our third objective is the development and demonstration of a proof-of-concept.

Introduction

The global Data Lake Market is projected to witness substantial growth, reaching approximately USD 90 billion by 2032, marking a significant increase from its 2022 value of USD 16.6 billion. This growth trajectory is expected to unfold steadily, with a Compound Annual Growth Rate (CAGR) of 21.3% from 2023 to 2032.

A Data Lake is a centralized repository designed to store, process, and secure large volumes of structured and unstructured data from multiple sources. It allows for the storage of data in its natural format, without the need to first structure it, making it a flexible option for big data and real-time analytics. Data Lakes support the analysis of data through various methods, including machine learning, predictive analytics, data discovery, and profiling.

The Data Lake market is experiencing rapid growth, driven by the increasing volume of data generated by businesses, the need for advanced analytics to understand customer behavior, and the adoption of cloud computing. Companies are investing in Data Lake solutions to gain insights that can improve decision-making, enhance operational efficiency, and create personalized customer experiences. The market is also seeing innovation in terms of security, data management, and integration capabilities, enabling more robust and scalable data ecosystems. As organizations continue to recognize the value of data-driven strategies, the demand for Data Lake technologies is expected to rise, marking a significant trend in the data management landscape

Presented here is a point cloud produced by the U.S. Geological Survey (USGS) from historical U.S. Air Force vertical aerial imagery, covering the area of the Mud Creek landslide on California State Route 1 (SR1), Mud Creek, Big Sur, California. The point cloud is referenced to previously published lidar data and contains RGB information as well as XYZ. Point cloud coordinates are in NAD83 UTM Zone 10 meters. Imagery was downloaded from USGS Eros Data Center and processed using structure-from-motion photogrammetry with Agisoft PhotoScan version 1.2.8 through 1.3.2. Point clouds were clipped to an AOI using LASTools. The AOI was created from a KMZ in Google Earth and transformed to a shapefile using ArcMap 10.5.

Data Wrangling Market size was valued at USD 1.99 Billion in 2024 and is projected to reach USD 4.07 Billion by 2032, growing at a CAGR of 9.4% during the forecast period 2026-2032.• Big Data Analytics Growth: Organizations are generating massive volumes of unstructured and semi-structured data from diverse sources including social media, IoT devices, and digital transactions. Data wrangling tools become essential for cleaning, transforming, and preparing this complex data for meaningful analytics and business intelligence applications.• Machine Learning and AI Adoption: The rapid expansion of artificial intelligence and machine learning initiatives requires high-quality, properly formatted training datasets. Data wrangling solutions enable data scientists to efficiently prepare, clean, and structure raw data for model training, driving sustained market demand across AI-focused organizations.

This paper provides an abstract analysis of parallel processing strategies for spatial and spatio-temporal data. It isolates aspects such as data locality and computational locality as well as redundancy and locally sequential access as central elements of parallel algorithm design for spatial data. Furthermore, the paper gives some examples from simple and advanced GIS and spatial data analysis highlighting both that big data systems have been around long before the current hype of big data and that they follow some design principles which are inevitable for spatial data including distributed data structures and messaging, which are, however, incompatible with the popular MapReduce paradigm. Throughout this discussion, the need for a replacement or extension of the MapReduce paradigm for spatial data is derived. This paradigm should be able to deal with the imperfect data locality inherent to spatial data hindering full independence of non-trivial computational tasks. We conclude that more research is needed and that spatial big data systems should pick up more concepts like graphs, shortest paths, raster data, events, and streams at the same time instead of solving exactly the set of spatially separable problems such as line simplifications or range queries in manydifferent ways.