The global Big Data in E-commerce market is experiencing robust growth, driven by the escalating volume of consumer data generated through online transactions and interactions. The increasing adoption of personalized marketing strategies, sophisticated fraud detection systems, and advanced supply chain optimization techniques are key factors fueling this expansion. The market is segmented by application (Online Classifieds, Online Education, Online Financials, Online Retail, Online Travel and Leisure) and data type (Structured, Unstructured, Semi-structured). Online retail currently dominates the application segment, benefiting from the immense amount of data generated by customer purchases, browsing behavior, and reviews. However, the online travel and leisure sector is exhibiting high growth potential due to the increasing use of big data analytics to personalize travel recommendations and optimize pricing strategies. The dominance of unstructured data within the data type segment underscores the importance of advanced analytical tools and techniques capable of processing diverse data sources, ranging from social media comments to customer service interactions. While the market faces challenges such as data security concerns and the need for skilled data scientists, technological advancements in cloud computing and artificial intelligence are mitigating these restraints. The increasing affordability and accessibility of big data solutions are further accelerating market expansion. Geographic distribution reveals strong growth across North America and Asia Pacific, driven by high e-commerce penetration rates and substantial investments in technological infrastructure. Europe and other regions are also witnessing significant growth, albeit at a slightly slower pace. The forecast period (2025-2033) anticipates sustained growth, driven by continuous technological innovation and the increasing integration of big data analytics across various e-commerce functions. Key players like Amazon Web Services, Microsoft, and IBM are strategically positioned to benefit from this growth trajectory, offering comprehensive solutions that cater to the diverse needs of e-commerce businesses.

Altosight | AI Custom Web Scraping Data

✦ Altosight provides global web scraping data services with AI-powered technology that bypasses CAPTCHAs, blocking mechanisms, and handles dynamic content.

We extract data from marketplaces like Amazon, aggregators, e-commerce, and real estate websites, ensuring comprehensive and accurate results.

✦ Our solution offers free unlimited data points across any project, with no additional setup costs.

We deliver data through flexible methods such as API, CSV, JSON, and FTP, all at no extra charge.

― Key Use Cases ―

➤ Price Monitoring & Repricing Solutions

🔹 Automatic repricing, AI-driven repricing, and custom repricing rules 🔹 Receive price suggestions via API or CSV to stay competitive 🔹 Track competitors in real-time or at scheduled intervals

➤ E-commerce Optimization

🔹 Extract product prices, reviews, ratings, images, and trends 🔹 Identify trending products and enhance your e-commerce strategy 🔹 Build dropshipping tools or marketplace optimization platforms with our data

➤ Product Assortment Analysis

🔹 Extract the entire product catalog from competitor websites 🔹 Analyze product assortment to refine your own offerings and identify gaps 🔹 Understand competitor strategies and optimize your product lineup

➤ Marketplaces & Aggregators

🔹 Crawl entire product categories and track best-sellers 🔹 Monitor position changes across categories 🔹 Identify which eRetailers sell specific brands and which SKUs for better market analysis

➤ Business Website Data

🔹 Extract detailed company profiles, including financial statements, key personnel, industry reports, and market trends, enabling in-depth competitor and market analysis

🔹 Collect customer reviews and ratings from business websites to analyze brand sentiment and product performance, helping businesses refine their strategies

➤ Domain Name Data

🔹 Access comprehensive data, including domain registration details, ownership information, expiration dates, and contact information. Ideal for market research, brand monitoring, lead generation, and cybersecurity efforts

➤ Real Estate Data

🔹 Access property listings, prices, and availability 🔹 Analyze trends and opportunities for investment or sales strategies

― Data Collection & Quality ―

► Publicly Sourced Data: Altosight collects web scraping data from publicly available websites, online platforms, and industry-specific aggregators

► AI-Powered Scraping: Our technology handles dynamic content, JavaScript-heavy sites, and pagination, ensuring complete data extraction

► High Data Quality: We clean and structure unstructured data, ensuring it is reliable, accurate, and delivered in formats such as API, CSV, JSON, and more

► Industry Coverage: We serve industries including e-commerce, real estate, travel, finance, and more. Our solution supports use cases like market research, competitive analysis, and business intelligence

► Bulk Data Extraction: We support large-scale data extraction from multiple websites, allowing you to gather millions of data points across industries in a single project

► Scalable Infrastructure: Our platform is built to scale with your needs, allowing seamless extraction for projects of any size, from small pilot projects to ongoing, large-scale data extraction

― Why Choose Altosight? ―

✔ Unlimited Data Points: Altosight offers unlimited free attributes, meaning you can extract as many data points from a page as you need without extra charges

✔ Proprietary Anti-Blocking Technology: Altosight utilizes proprietary techniques to bypass blocking mechanisms, including CAPTCHAs, Cloudflare, and other obstacles. This ensures uninterrupted access to data, no matter how complex the target websites are

✔ Flexible Across Industries: Our crawlers easily adapt across industries, including e-commerce, real estate, finance, and more. We offer customized data solutions tailored to specific needs

✔ GDPR & CCPA Compliance: Your data is handled securely and ethically, ensuring compliance with GDPR, CCPA and other regulations

✔ No Setup or Infrastructure Costs: Start scraping without worrying about additional costs. We provide a hassle-free experience with fast project deployment

✔ Free Data Delivery Methods: Receive your data via API, CSV, JSON, or FTP at no extra charge. We ensure seamless integration with your systems

✔ Fast Support: Our team is always available via phone and email, resolving over 90% of support tickets within the same day

― Custom Projects & Real-Time Data ―

✦ Tailored Solutions: Every business has unique needs, which is why Altosight offers custom data projects. Contact us for a feasibility analysis, and we’ll design a solution that fits your goals

✦ Real-Time Data: Whether you need real-time data delivery or scheduled updates, we provide the flexibility to receive data when you need it. Track price changes, monitor product trends, or gather...

Web Scraper Software Market Valuation – 2024-2031

Web Scraper Software Market was valued at USD 568.2 Million in 2024 and is projected to reach USD 1628.6 Million by 2031, growing at a CAGR of 14.1% from 2024 to 2031.

Global Web Scraper Software Market Drivers

Data-Driven Decision Making: Businesses increasingly rely on data-driven insights to make informed decisions. Web scraping tools enable organizations to collect large amounts of structured and unstructured data from various websites, empowering them to analyze market trends, consumer behavior, and competitor activities.

Price Intelligence: E-commerce businesses utilize web scraping to monitor competitor pricing, identify pricing opportunities, and optimize their own pricing strategies.

Market Research and Analysis: Web scraping tools help researchers and analysts gather data on market trends, consumer sentiment, and industry benchmarks. This data is invaluable for conducting in-depth market research and analysis.

Global Web Scraper Software Market Restraints

Ethical and Legal Considerations: Web scraping can raise ethical and legal concerns, particularly when it violates website terms of service or copyright laws. It's crucial to adhere to ethical guidelines and respect website owners' rights.

Technical Challenges: Web scraping can be technically complex, requiring knowledge of programming languages like Python and libraries such as Beautiful Soup and Scrapy. Additionally, websites often implement anti-scraping measures, making data extraction challenging.

Relational Database Software Market Outlook

In 2023, the global market size for relational database software is valued at approximately $61.5 billion, with an anticipated growth to $113.9 billion by 2032, reflecting a robust CAGR of 7.1%. This impressive growth is mainly driven by the increasing volume of data generated across industries and the need for efficient data management solutions. The expanding application of relational database software in various sectors such as BFSI, healthcare, and telecommunications is also a significant contributor to market growth. Furthermore, the transition from legacy systems to modern, scalable database solutions is propelling this market forward.

The proliferation of data from diverse sources, including IoT devices, social media, and enterprise applications, is one of the primary growth factors for the relational database software market. Organizations are increasingly adopting advanced database management systems to handle large volumes of structured and unstructured data efficiently. This necessity aligns with the growing trend of digital transformation, where data plays a crucial role in driving business insights and decision-making processes. Additionally, the rise of big data analytics and artificial intelligence necessitates robust database solutions that can manage and process vast amounts of data in real-time.

Another significant growth driver for this market is the increasing reliance on cloud-based solutions. Cloud computing offers scalable, flexible, and cost-effective database management options, making it an attractive choice for enterprises of all sizes. The adoption of cloud-based relational database software is accelerating as it reduces the need for physical infrastructure, lowers maintenance costs, and provides seamless access to data from any location. Moreover, cloud providers are continually enhancing their offerings with advanced features such as automated backups, disaster recovery, and high availability, further boosting the market demand.

The integration of relational database software with emerging technologies such as blockchain, machine learning, and internet of things (IoT) is also fueling market growth. These integrations enable enhanced data security, improved data analytics capabilities, and efficient data management, which are crucial for modern enterprises. For instance, blockchain technology can provide a secure and transparent way of handling transactions and records within a relational database, while machine learning algorithms can optimize queries and database performance. As these technologies evolve, their synergy with relational database software is expected to create new opportunities and drive further market expansion.

In addition to the growing significance of relational databases, Object-Oriented Databases Software is gaining traction as businesses seek more flexible and efficient ways to manage complex data structures. Unlike traditional relational databases that rely on tables and rows, object-oriented databases store data in objects, similar to how data is organized in object-oriented programming. This approach allows for a more intuitive mapping of real-world entities and relationships, making it particularly beneficial for applications that require complex data representations, such as computer-aided design (CAD), multimedia systems, and telecommunications. As industries continue to evolve and demand more sophisticated data management solutions, the adoption of object-oriented databases is expected to rise, complementing the existing relational database landscape.

Region-wise, North America holds a significant share of the relational database software market, driven by the presence of leading technology companies, high adoption of advanced IT solutions, and substantial investments in research and development. Europe follows closely, with strong growth observed in cloud-based solutions and regulatory frameworks favoring data security and privacy. The Asia Pacific region is projected to exhibit the highest growth rate, attributed to the rapid digitalization of economies, increasing IT expenditures, and expanding tech-savvy population. Conversely, Latin America and the Middle East & Africa regions are also experiencing growth, albeit at a slower pace, due to growing awareness and gradual adoption of database management solutions.

Deployment Mode Analysis

The deployment mode segment of the relational database software market can be bifur

Non-Relational Databases Market Outlook

The global non-relational databases market size was valued at approximately USD 15 billion in 2023 and is expected to reach around USD 45 billion by 2032, growing at a compound annual growth rate (CAGR) of 12.5% during the forecast period. This impressive growth can be attributed to the increasing demand for scalable and flexible database solutions that can handle large volumes of unstructured data. The proliferation of big data, the rise of cloud computing, and the enhanced adoption of advanced technologies across various industries are some of the key factors driving the market's expansion.

One of the primary growth factors for the non-relational databases market is the explosion of big data. With the advent of the Internet of Things (IoT), social media, and e-commerce, the amount of unstructured data generated has skyrocketed. Traditional relational databases struggle to manage such diverse and voluminous datasets, making non-relational databases an attractive alternative. These databases offer superior scalability, flexibility, and performance when dealing with unstructured data, making them indispensable for modern data-driven enterprises.

The rise of cloud computing has also significantly contributed to the growth of the non-relational databases market. As organizations increasingly migrate their operations to cloud environments, the demand for cloud-based database solutions has surged. Non-relational databases, with their inherent ability to scale horizontally and handle distributed data storage and processing, align perfectly with the cloud paradigm. This seamless integration with cloud platforms enables businesses to achieve greater agility, cost efficiency, and faster time-to-market for their applications, thereby driving the widespread adoption of non-relational databases.

Another critical growth factor is the rapid adoption of advanced technologies such as artificial intelligence (AI), machine learning (ML), and data analytics. These technologies heavily rely on massive amounts of unstructured data for training models, generating insights, and making predictions. Non-relational databases, with their ability to store and process such data types efficiently, have become essential tools for organizations looking to leverage AI and ML capabilities. As businesses continue to invest in these technologies to gain a competitive edge, the demand for non-relational databases is expected to grow correspondingly.

The evolution of XML Databases Software has played a pivotal role in the advancement of non-relational databases. XML databases are specifically designed to handle XML data, which is inherently hierarchical and complex. This type of database software allows for the efficient storage, retrieval, and management of XML documents, making it an ideal choice for applications that require the manipulation of structured data with complex relationships. As businesses increasingly rely on XML data for various applications, such as web services and data interchange, the demand for robust XML Databases Software continues to grow. These databases offer significant advantages in terms of flexibility and scalability, enabling organizations to manage large volumes of XML data effectively. As a result, XML Databases Software has become an integral component of the non-relational databases landscape, supporting a wide range of industry applications.

Regionally, the market outlook for non-relational databases is highly promising. North America currently holds the largest market share, driven by the early adoption of advanced technologies and the presence of key players in the region. Europe and the Asia Pacific are also witnessing significant growth, with the latter expected to register the highest CAGR during the forecast period. The growing digital transformation initiatives across emerging economies, coupled with increasing investments in IT infrastructure, are likely to propel the market forward in these regions. Other regions, such as Latin America and the Middle East & Africa, are also poised for steady growth as they gradually embrace digitalization and modern data management solutions.

Type Analysis

The non-relational databases market is segmented by type into document-oriented databases, key-value stores, column-oriented databases, graph databases, and others. Document-oriented databases, such as MongoDB and Couchbase, store data in a flexible, JSON-like format, enabling easy storage and re

WebDS: A Benchmark for Web-based Data Science

WebDS is the first end-to-end benchmark designed for evaluating agents on real-world web-based data science workflows. It contains 870 tasks across 29 containerized websites spanning 10 domains, including economics, health, climate, and scientific research. Agents are tested on:

Multi-hop web navigation Structured and unstructured data processing Tool usage (e.g., Python scripts, visualization tools) Downstream task completion (e.g.… See the full description on the dataset page: https://huggingface.co/datasets/yamhm/WebDS.

Suicide remains a leading cause of preventable death worldwide, despite advances in research and decreases in mental health stigma through government health campaigns. Machine learning (ML), a type of artificial intelligence (AI), is the use of algorithms to simulate and imitate human cognition. Given the lack of improvement in clinician-based suicide prediction over time, advancements in technology have allowed for novel approaches to predicting suicide risk. This systematic review and meta-analysis aimed to synthesize current research regarding data sources in ML prediction of suicide risk, incorporating and comparing outcomes between structured data (human interpretable such as psychometric instruments) and unstructured data (only machine interpretable such as electronic health records). Online databases and gray literature were searched for studies relating to ML and suicide risk prediction. There were 31 eligible studies. The outcome for all studies combined was AUC = 0.860, structured data showed AUC = 0.873, and unstructured data was calculated at AUC = 0.866. There was substantial heterogeneity between the studies, the sources of which were unable to be defined. The studies showed good accuracy levels in the prediction of suicide risk behavior overall. Structured data and unstructured data also showed similar outcome accuracy according to meta-analysis, despite different volumes and types of input data.

Key Value Databases Market Outlook

The global Key Value Databases market size was valued at approximately USD 5.2 billion in 2023 and is anticipated to reach around USD 12.5 billion by 2032, exhibiting a compound annual growth rate (CAGR) of 10.1% during the forecast period. The growth of this market is primarily driven by the rapid digital transformation initiatives across various industries, increasing adoption of NoSQL databases in big data and real-time web applications, and the growing need for high-performance data management solutions.

One of the critical growth factors propelling the Key Value Databases market is the burgeoning volume of unstructured data. Industries ranging from retail to healthcare are increasingly generating significant volumes of unstructured data that traditional relational databases struggle to manage efficiently. Key value databases, with their flexible schema and high performance, offer a robust solution for handling this unstructured data. Additionally, the increasing trend of adopting microservices architecture and distributed systems is encouraging organizations to leverage key value databases to ensure scalability and agility in their applications.

Another significant factor contributing to market growth is the rising demand for real-time data processing capabilities. In the era of digital business, enterprises are focusing on real-time analytics to make swift and informed decisions. Key value databases facilitate rapid data retrieval and low-latency transactions, making them ideal for applications such as fraud detection in BFSI, personalized marketing in retail, and patient monitoring in healthcare. This capability is crucial for businesses seeking competitive advantage through quick and responsive data-driven strategies.

Additionally, the adoption of cloud computing technologies has immensely benefited the key value databases market. Cloud platforms offer scalable infrastructure and services that can dynamically adjust to the demands of the database workloads. As businesses increasingly migrate their operations to the cloud to achieve cost-efficiency, flexibility, and resilience, the deployment of key value databases on cloud platforms has witnessed a significant surge. This shift is further bolstered by advancements in cloud-native technologies and the growing popularity of Database-as-a-Service (DBaaS) offerings.

Document Databases play a crucial role in the modern data landscape, especially as organizations seek more flexible and scalable solutions for managing semi-structured and unstructured data. Unlike traditional relational databases, document databases store data in a format that is more aligned with the way applications naturally handle data, such as JSON or XML. This allows for more intuitive data modeling and easier integration with modern application development frameworks. As businesses increasingly adopt agile methodologies and microservices architectures, the demand for document databases is on the rise, providing a robust foundation for applications that require dynamic schema evolution and rapid development cycles.

Regionally, North America currently holds the largest market share in the key value databases market, driven by the presence of major technology companies and extensive adoption of advanced data management solutions. However, the Asia Pacific region is expected to exhibit the highest growth rate during the forecast period. The rapid digitalization across emerging economies, increasing investments in IT infrastructure, and the growing number of SMEs adopting key value databases are key factors contributing to this growth. Europe, Latin America, and the Middle East & Africa are also witnessing steady adoption of key value databases as organizations in these regions increasingly recognize the benefits of efficient and flexible data management.

Type Analysis

The key value databases market is segmented by type into in-memory and persistent databases. In-memory databases store data directly in the main memory (RAM), which allows for faster data retrieval and processing compared to traditional disk-based storage. The demand for in-memory key value databases is growing rapidly, driven by applications that require high-speed data access and real-time processing capabilities. Industries such as finance, telecommunications, and online gaming are increasingly adopting in-memory databases to meet their performance requirements.

Persistent key value

Non Relational Databases Market Outlook

The global market size for non-relational databases is expected to grow from USD 10.5 billion in 2023 to USD 35.2 billion by 2032, registering a Compound Annual Growth Rate (CAGR) of 14.6% over the forecast period. This substantial growth is primarily driven by increasing demand for scalable, flexible database solutions capable of handling diverse data types and large volumes of data generated across various industries.

One of the significant growth factors for the non-relational databases market is the exponential increase in data generated globally. With the proliferation of Internet of Things (IoT) devices, social media platforms, and digital transactions, the volume of semi-structured and unstructured data is growing at an unprecedented rate. Traditional relational databases often fall short in efficiently managing such data types, making non-relational databases a preferred choice. For example, document-oriented databases like MongoDB allow for the storage of JSON-like documents, offering flexibility in data modeling and retrieval.

Another key driver is the increasing adoption of non-relational databases among enterprises seeking agile and scalable database solutions. The need for high-performance applications that can scale horizontally and handle large volumes of transactions is pushing businesses to shift from traditional relational databases to non-relational databases. This is particularly evident in sectors like e-commerce, where the ability to manage customer data, product catalogs, and transaction histories in real-time is crucial. Additionally, companies in the BFSI (Banking, Financial Services, and Insurance) sector are leveraging non-relational databases for fraud detection, risk management, and customer relationship management.

The advent of cloud computing and the growing trend of digital transformation are also significant contributors to the market growth. Cloud-based non-relational databases offer numerous advantages, including reduced infrastructure costs, scalability, and ease of access. As more organizations migrate their operations to the cloud, the demand for cloud-based non-relational databases is set to rise. Moreover, the availability of Database-as-a-Service (DBaaS) offerings from major cloud providers like Amazon Web Services (AWS), Microsoft Azure, and Google Cloud Platform (GCP) is simplifying the deployment and management of these databases, further driving their adoption.

Regionally, North America holds the largest market share, driven by the early adoption of advanced technologies and the presence of major market players. However, the Asia Pacific region is expected to witness the highest growth rate during the forecast period. The rapid digitalization, growing adoption of cloud services, and increasing investments in IT infrastructure in countries like China and India are propelling the demand for non-relational databases in the region. Additionally, the expanding e-commerce sector and the proliferation of smart devices are further boosting market growth in Asia Pacific.

Type Analysis

The non-relational databases market is segmented into several types, including Document-Oriented Databases, Key-Value Stores, Column-Family Stores, Graph Databases, and Others. Each type offers unique functionalities and caters to specific use cases, making them suitable for different industry requirements. Document-Oriented Databases, such as MongoDB and CouchDB, store data in document format (e.g., JSON or BSON), allowing for flexible schema designs and efficient data retrieval. These databases are widely used in content management systems, e-commerce platforms, and real-time analytics applications due to their ability to handle semi-structured data.

Key-Value Stores, such as Redis and Amazon DynamoDB, store data as key-value pairs, providing extremely fast read and write operations. These databases are ideal for caching, session management, and real-time applications where speed is critical. They offer horizontal scalability and are highly efficient in managing large volumes of data with simple query requirements. The simplicity of the key-value data model and its performance benefits make it a popular choice for high-throughput applications.

Column-Family Stores, such as Apache Cassandra and HBase, store data in columns rather than rows, allowing for efficient storage and retrieval of large datasets. These databases are designed to handle massive amounts of data across distributed systems, making them suitable for use cases involving big data analytics, time-seri

The size of the Storage in Big Data Market was valued at USD 7.90 billion in 2023 and is projected to reach USD 18.89 billion by 2032, with an expected CAGR of 13.26% during the forecast period. The storage segment in the Big Data market is experiencing rapid growth due to the increasing volume of data generated by businesses across various industries. As organizations continue to adopt digital transformation strategies, the demand for efficient, scalable, and cost-effective data storage solutions rises. Traditional data storage methods are being complemented or replaced by cloud-based systems, data lakes, and hybrid models, allowing organizations to manage vast amounts of unstructured and structured data effectively. Cloud storage platforms such as Amazon Web Services (AWS), Microsoft Azure, and Google Cloud are leading the market by offering robust storage solutions that can scale in real-time, providing flexibility and security. The rise of Internet of Things (IoT) devices, artificial intelligence (AI), and machine learning (ML) technologies further fuel the need for advanced data storage options that can handle real-time processing and high throughput. As industries like healthcare, finance, and retail continue to generate massive datasets, the Big Data storage market is expected to evolve, with innovations like edge computing and distributed storage systems shaping the future landscape. Security, data governance, and compliance remain critical factors driving the development of storage technologies in the Big Data market. Key drivers for this market are: Surge in Data Generation and Storage Demand Increasing Adoption of Big Data Analytics Growing Popularity of Cloud-Based Storage Need for Data Security and Compliance. Potential restraints include: Data Privacy and Security Concerns Complexity and Cost of Implementing Big Data Storage Solutions Lack of Skilled Professionals. Notable trends are: Solid-State Drives (SSDs) and NVMe for High-Performance Storage Object Storage for Unstructured Data Management Artificial Intelligence (AI) for Storage Optimization Data Fabric for Storage Consolidation and Management.

NoSQL Database Market size was valued at USD 7.43 Billion in 2024 and is projected to reach USD 60 Billion by 2031, growing at a CAGR of 30% during the forecast period from 2024 to 2031.

Global NoSQL Database Market Drivers

Big Data Management: The exponential growth of unstructured and semi-structured data necessitates flexible and scalable database solutions. Cloud Computing Adoption: The shift towards cloud-based applications and infrastructure is driving demand for NoSQL databases. Real-time Analytics: NoSQL databases excel at handling real-time data processing and analytics, making them suitable for applications like IoT and fraud detection.

Global NoSQL Database Market Restraints

Complexity and Management Challenges: NoSQL databases can be complex to manage and require specialized skills. Lack of Standardization: The absence of a standardized NoSQL query language can hinder data integration and migration.

The Data Extraction Market size was valued at USD 2.45 USD billion in 2023 and is projected to reach USD 9.04 USD billion by 2032, exhibiting a CAGR of 20.5 % during the forecast period. Data extraction is the process of retrieving specific data from diverse sources for further processing or analysis. It involves techniques such as web scraping, database querying, and document parsing. Key features include automated data retrieval, scalability, and the ability to handle various data formats. There are two primary types: structured data extraction, which deals with organized data like databases, and unstructured data extraction, which involves extracting information from sources like text documents or web pages. Applications span across industries, including market research, business intelligence, and data migration, where they aid in transforming raw data into actionable insights for informed decision-making. Key drivers for this market are: Increasing Requirement of Cloud Compliance across Organizations Drives Market Growth . Potential restraints include: Inadequate Integration and Complexity in Deploying the Solutions Restrains Market Growth. Notable trends are: Growing Implementation of Touch-based and Voice-based Infotainment Systems to Increase Adoption of Intelligent Cars.

As the amount of textual information grows explosively in various kinds of business systems, it becomes more and more desirable to analyze both structured data records and unstructured text data simultaneously. Although online analytical processing (OLAP) techniques have been proven very useful for analyzing and mining structured data, they face challenges in handling text data. On the other hand, probabilistic topic models are among the most effective approaches to latent topic analysis and mining on text data. In this paper, we study a new data model called topic cube to combine OLAP with probabilistic topic modeling and enable OLAP on the dimension of text data in a multidimensional text database. Topic cube extends the traditional data cube to cope with a topic hierarchy and stores probabilistic content measures of text documents learned through a probabilistic topic model. To materialize topic cubes efficiently, we propose two heuristic aggregations to speed up the iterative Expectation-Maximization (EM) algorithm for estimating topic models by leveraging the models learned on component data cells to choose a good starting point for iteration. Experimental results show that these heuristic aggregations are much faster than the baseline method of computing each topic cube from scratch. We also discuss some potential uses of topic cube and show sample experimental results.

Abstract The proposal presented in this study seeks to properly represent natural language to ontologies and vice-versa. Therefore, the semi-automatic creation of a lexical database in Brazilian Portuguese containing morphological, syntactic, and semantic information that can be read by machines was proposed, allowing the link between structured and unstructured data and its integration into an information retrieval model to improve precision. The results obtained demonstrated that the methodology can be used in the risco financeiro (financial risk) domain in Portuguese for the construction of an ontology and the lexical-semantic database and the proposal of a semantic information retrieval model. In order to evaluate the performance of the proposed model, documents containing the main definitions of the financial risk domain were selected and indexed with and without semantic annotation. To enable the comparison between the approaches, two databases were created based on the texts with the semantic annotations to represent the semantic search. The first one represents the traditional search and the second contained the index built based on the texts with the semantic annotations to represent the semantic search. The evaluation of the proposal was based on recall and precision. The queries submitted to the model showed that the semantic search outperforms the traditional search and validates the methodology used. Although more complex, the procedure proposed can be used in all kinds of domains.

Collective behaviour such as the flocks of birds and schools of fish is inspired by computer-based systems and is widely used in agents’ formation. The human could easily recognise these behaviours; however, it is hard for a computer system to recognise these behaviours. Since humans could easily recognise these behaviours, ground truth data on human perception of collective behaviour could enable machine learning methods to mimic this human perception. Hence ground truth data has been collected from human perception of collective behaviour recognition by running an online survey. Specific collective motions considered in this online survey include 16 structured and unstructured behaviours. The defined structured collective motions include boids’ movements with an identifiable embedded pattern. Unstructured collective motions consist of random movement of boids with no patterns. The participants are from diverse levels of knowledge, all over the world, and are over 18 years old. Each question contains a short video (around 10 seconds), captured from one of the 16 simulated movements. The videos are shown in a randomized order to the participants. Then they were asked to label each structured motion of boids as ‘flocking’, ‘aligned’, or ‘grouped’ and others as ‘not flocking’, ‘not aligned’, or ‘not grouped’. By averaging human perceptions, three binary labelled datasets of these motions are created. The data could be trained by machine learning methods, which enabled them to automatically recognise collective behaviour.