Fork this notebook to get started on accessing data in the BigQuery dataset by writing SQL queries using the BQhelper module.

Context

Google Patents Public Data, provided by IFI CLAIMS Patent Services, is a worldwide bibliographic and US full-text dataset of patent publications. Patent information accessibility is critical for examining new patents, informing public policy decisions, managing corporate investment in intellectual property, and promoting future scientific innovation. The growing number of available patent data sources means researchers often spend more time downloading, parsing, loading, syncing and managing local databases than conducting analysis. With these new datasets, researchers and companies can access the data they need from multiple sources in one place, thus spending more time on analysis than data preparation.

Content

The Google Patents Public Data dataset contains a collection of publicly accessible, connected database tables for empirical analysis of the international patent system.

Acknowledgements

Data Origin: https://bigquery.cloud.google.com/dataset/patents-public-data:patents

For more info, see the documentation at https://developers.google.com/web/tools/chrome-user-experience-report/

“Google Patents Public Data” by IFI CLAIMS Patent Services and Google is licensed under a Creative Commons Attribution 4.0 International License.

Banner photo by Helloquence on Unsplash

The AlphaFold Protein Structure Database is a collection of protein structure predictions made using the machine learning model AlphaFold. AlphaFold was developed by DeepMind , and this database was created in partnership with EMBL-EBI . For information on how to interpret, download and query the data, as well as on which proteins are included / excluded, and change log, please see our main dataset guide and FAQs . To interactively view individual entries or to download proteomes / Swiss-Prot please visit https://alphafold.ebi.ac.uk/ . The current release aims to cover most of the over 200M sequences in UniProt (a commonly used reference set of annotated proteins). The files provided for each entry include the structure plus two model confidence metrics (pLDDT and PAE). The files can be found in the Google Cloud Storage bucket gs://public-datasets-deepmind-alphafold-v4 with metadata in the BigQuery table bigquery-public-data.deepmind_alphafold.metadata . If you use this data, please cite: Jumper, J et al. Highly accurate protein structure prediction with AlphaFold. Nature (2021) Varadi, M et al. AlphaFold Protein Structure Database: massively expanding the structural coverage of protein-sequence space with high-accuracy models. Nucleic Acids Research (2021) This public dataset is hosted in Google Cloud Storage and is available free to use. Use this quick start guide to quickly learn how to access public datasets on Google Cloud Storage.

Discover the booming Analytical Data Store Tools market! This comprehensive analysis reveals a $50 billion market in 2025, projected to reach $150 billion by 2033 at a 15% CAGR. Learn about key drivers, trends, and top players like Snowflake, Google, and Microsoft, and gain insights into regional market shares.

Context

Stack Overflow is the largest online community for programmers to learn, share their knowledge, and advance their careers.

Content

Updated on a quarterly basis, this BigQuery dataset includes an archive of Stack Overflow content, including posts, votes, tags, and badges. This dataset is updated to mirror the Stack Overflow content on the Internet Archive, and is also available through the Stack Exchange Data Explorer.

Fork this kernel to get started with this dataset.

Acknowledgements

Dataset Source: https://archive.org/download/stackexchange

https://bigquery.cloud.google.com/dataset/bigquery-public-data:stackoverflow

https://cloud.google.com/bigquery/public-data/stackoverflow

Banner Photo by Caspar Rubin from Unplash.

Inspiration

What is the percentage of questions that have been answered over the years?

What is the reputation and badge count of users across different tenures on StackOverflow?

What are 10 of the “easier” gold badges to earn?

Which day of the week has most questions answered within an hour?

You can check the fields description in the documentation: current Keyword database: https://docs.dataforseo.com/v3/databases/google/keywords/?bash; Historical Keyword database: https://docs.dataforseo.com/v3/databases/google/history/keywords/?bash. You don’t have to download fresh data dumps in JSON or CSV – we can deliver data straight to your storage or database. We send terrabytes of data to dozens of customers every month using Amazon S3, Google Cloud Storage, Microsoft Azure Blob, Eleasticsearch, and Google Big Query. Let us know if you’d like to get your data to any other storage or database.

Context

Bitcoin and other cryptocurrencies have captured the imagination of technologists, financiers, and economists. Digital currencies are only one application of the underlying blockchain technology. Like its predecessor, Bitcoin, the Ethereum blockchain can be described as an immutable distributed ledger. However, creator Vitalik Buterin also extended the set of capabilities by including a virtual machine that can execute arbitrary code stored on the blockchain as smart contracts.

Both Bitcoin and Ethereum are essentially OLTP databases, and provide little in the way of OLAP (analytics) functionality. However the Ethereum dataset is notably distinct from the Bitcoin dataset:

• The Ethereum blockchain has as its primary unit of value Ether, while the Bitcoin blockchain has Bitcoin. However, the majority of value transfer on the Ethereum blockchain is composed of so-called tokens. Tokens are created and managed by smart contracts.

• Ether value transfers are precise and direct, resembling accounting ledger debits and credits. This is in contrast to the Bitcoin value transfer mechanism, for which it can be difficult to determine the balance of a given wallet address.

• Addresses can be not only wallets that hold balances, but can also contain smart contract bytecode that allows the programmatic creation of agreements and automatic triggering of their execution. An aggregate of coordinated smart contracts could be used to build a decentralized autonomous organization.

Content

The Ethereum blockchain data are now available for exploration with BigQuery. All historical data are in the ethereum_blockchain dataset, which updates daily.

Our hope is that by making the data on public blockchain systems more readily available it promotes technological innovation and increases societal benefits.

Querying BigQuery tables

You can use the BigQuery Python client library to query tables in this dataset in Kernels. Note that methods available in Kernels are limited to querying data. Tables are at bigquery-public-data.crypto_ethereum.[TABLENAME]. Fork this kernel to get started.

Acknowledgements

Cover photo by Thought Catalog on Unsplash

Inspiration

• What are the most popularly exchanged digital tokens, represented by ERC-721 and ERC-20 smart contracts?
• Compare transaction volume and transaction networks over time
• Compare transaction volume to historical prices by joining with other available data sources like Bitcoin Historical Data

You can check the fields description in the documentation: current Full database: https://docs.dataforseo.com/v3/databases/google/full/?bash; Historical Full database: https://docs.dataforseo.com/v3/databases/google/history/full/?bash.

Full Google Database is a combination of the Advanced Google SERP Database and Google Keyword Database.

Google SERP Database offers millions of SERPs collected in 67 regions with most of Google’s advanced SERP features, including featured snippets, knowledge graphs, people also ask sections, top stories, and more.

Google Keyword Database encompasses billions of search terms enriched with related Google Ads data: search volume trends, CPC, competition, and more.

This database is available in JSON format only.

You don’t have to download fresh data dumps in JSON – we can deliver data straight to your storage or database. We send terrabytes of data to dozens of customers every month using Amazon S3, Google Cloud Storage, Microsoft Azure Blob, Eleasticsearch, and Google Big Query. Let us know if you’d like to get your data to any other storage or database.

Context

ChEMBL is maintained by the European Bioinformatics Institute (EBI), of the European Molecular Biology Laboratory (EMBL), based at the Wellcome Trust Genome Campus, Hinxton, UK.

Content

ChEMBL is a manually curated database of bioactive molecules with drug-like properties used in drug discovery, including information about existing patented drugs.

Schema: http://ftp.ebi.ac.uk/pub/databases/chembl/ChEMBLdb/releases/chembl_23/chembl_23_schema.png

Documentation: http://ftp.ebi.ac.uk/pub/databases/chembl/ChEMBLdb/releases/chembl_23/schema_documentation.html

Fork this notebook to get started on accessing data in the BigQuery dataset using the BQhelper package to write SQL queries.

Acknowledgements

“ChEMBL” by the European Bioinformatics Institute (EMBL-EBI), used under CC BY-SA 3.0. Modifications have been made to add normalized publication numbers.

Data Origin: https://bigquery.cloud.google.com/dataset/patents-public-data:ebi_chembl

Banner photo by rawpixel on Unsplash

You can check the fields description in the documentation: regular SERP: https://docs.dataforseo.com/v3/databases/google/serp_regular/?bash; Advanced SERP: https://docs.dataforseo.com/v3/databases/google/serp_advanced/?bash; Historical SERP: https://docs.dataforseo.com/v3/databases/google/history/serp_advanced/?bash You don’t have to download fresh data dumps in JSON or CSV – we can deliver data straight to your storage or database. We send terrabytes of data to dozens of customers every month using Amazon S3, Google Cloud Storage, Microsoft Azure Blob, Eleasticsearch, and Google Big Query. Let us know if you’d like to get your data to any other storage or database.

This research endeavor applies Design Science Research as its principle research strategy as it focuses on the development of an experimental artifact for a unified query system. The artifact encompasses a set of architectural guidelines and principles when a applying a unified querying mechanism for the four types of NoSQL categories: key-value, document, graph and column store data models. The scope of this study is limit to specific vendor implementations, namely: Redis, MongoDB, Neo4j and Cassandra.Ethical Clearance no: 202028917/2023/20A variety of experiments were conducted to evaluate the prototype’s effectiveness and efficiency. The experiments were actioned by a group of automated participants, each test representing a subset of a particular goal. The culmination of these results indicated the feasibility of the proposed solution. The datasets for this study comprises of metrics such as Apdex, error rate, CPU and memory utilization as well as the respective NoSQL generated queries for each data store. The observed data is indicative of how efficient the prototype consumed resources whilst effectively generating an executable query at runtime.

Context

The USPTO grants US patents to inventors and assignees all over the world. For researchers in particular, PatentsView is intended to encourage the study and understanding of the intellectual property (IP) and innovation system; to serve as a fundamental function of the government in creating “public good” platforms in these data; and to eliminate redundant cleaning, converting and matching of these data by individual researchers, thus freeing up researcher time to do what they do best—study IP, innovation, and technological change.

Content

PatentsView Data is a database that longitudinally links inventors, their organizations, locations, and overall patenting activity. The dataset uses data derived from USPTO bulk data files.

Fork this notebook to get started on accessing data in the BigQuery dataset using the BQhelper package to write SQL queries.

Acknowledgements

“PatentsView” by the USPTO, US Department of Agriculture (USDA), the Center for the Science of Science and Innovation Policy, New York University, the University of California at Berkeley, Twin Arch Technologies, and Periscopic, used under CC BY 4.0.

Data Origin: https://bigquery.cloud.google.com/dataset/patents-public-data:patentsview

Banner photo by rawpixel on Unsplash

Discover the booming Analytics Query Accelerator (AQA) market, projected to reach $50 billion by 2033 with a 15% CAGR. This comprehensive analysis explores market drivers, trends, restraints, and regional insights, providing valuable data for businesses and investors in the data analytics sector. Learn about key players and future growth opportunities in this rapidly evolving market.

The Analytics Query Accelerator (AQA) market is experiencing robust growth, driven by the increasing demand for real-time insights from massive datasets across various industries. The market, estimated at $15 billion in 2025, is projected to achieve a Compound Annual Growth Rate (CAGR) of 20% from 2025 to 2033, reaching an estimated $70 billion by 2033. This expansion is fueled by several key factors. Firstly, the proliferation of big data and the need for rapid data analysis across sectors like finance, healthcare, and e-commerce are creating significant demand. Secondly, advancements in cloud computing and distributed database technologies are enabling faster query processing and improved performance of AQAs. Finally, the rising adoption of advanced analytics techniques such as machine learning and artificial intelligence is further driving the need for efficient query acceleration solutions. Key players like Google, Amazon, Snowflake, Microsoft, Databricks, Teradata, and Cloudera are actively competing in this rapidly evolving landscape, investing heavily in R&D and strategic partnerships to maintain market leadership. The growth trajectory of the AQA market is further shaped by emerging trends such as the increasing adoption of serverless computing and the expansion of edge analytics. However, challenges remain, including the complexity of implementing and managing AQA solutions, the need for skilled professionals, and concerns related to data security and privacy. Despite these restraints, the long-term outlook for the AQA market remains exceptionally positive, fueled by continuous technological innovations and the ever-increasing reliance on data-driven decision-making across all industries. The market segmentation is likely diversified across various deployment models (cloud, on-premise), data types (structured, unstructured), and industry verticals. This diverse landscape presents numerous opportunities for both established players and emerging companies to capture market share.

The global Cloud-Based Time Series Database market is poised for substantial growth, projected to reach an estimated USD 12,500 million by 2025 and expand at a Compound Annual Growth Rate (CAGR) of 22% through 2033. This robust expansion is primarily fueled by the escalating demand for real-time data analytics across diverse industries. Key drivers include the proliferation of IoT devices generating massive volumes of time-stamped data, the increasing adoption of cloud infrastructure for scalability and cost-efficiency, and the critical need for efficient data management and analysis in sectors like BFSI, manufacturing, and telecommunications. The ability of cloud-based time series databases to ingest, store, and query vast amounts of temporal data at high velocity makes them indispensable for applications such as predictive maintenance, anomaly detection, and performance monitoring. The market is further stimulated by advancements in database technologies, offering enhanced query performance, data compression, and integration capabilities with other cloud services. The market landscape is characterized by a dynamic interplay of public, private, and hybrid cloud models, with hybrid cloud solutions gaining traction due to their flexibility and ability to address specific data governance and security requirements. Major players like Amazon (AWS), Microsoft, Google, and IBM are heavily investing in R&D to offer sophisticated, feature-rich time series database solutions, driving innovation and competition. Emerging trends include the integration of AI and machine learning for advanced analytics on time-series data, the development of specialized time series databases optimized for specific workloads, and a growing emphasis on data security and compliance. While the market benefits from strong growth drivers, potential restraints such as data migration complexities, vendor lock-in concerns, and the need for skilled personnel to manage and operate these systems will require strategic consideration by market participants. The Asia Pacific region, led by China and India, is expected to witness the fastest growth, driven by rapid industrialization and digital transformation initiatives. Here is a unique report description on Cloud-Based Time Series Databases, structured as requested:

The Structured Query Language (SQL) server transformation market is experiencing robust growth, projected to reach $15 million in 2025 and maintain a Compound Annual Growth Rate (CAGR) of 9.4% from 2025 to 2033. This expansion is fueled by several key drivers. The increasing adoption of cloud-based solutions and the rise of big data analytics are pushing organizations to adopt more efficient and scalable SQL server solutions. Furthermore, the growing demand for real-time data processing and improved data integration capabilities within large enterprises and SMEs is significantly driving market growth. The market segmentation reveals strong demand across various application areas, with large enterprises leading the way due to their greater need for robust and scalable data management infrastructure. Data integration scripts remain a prominent segment, highlighting the critical need for seamless data flow across diverse systems. The competitive landscape is marked by established players like Oracle, IBM, and Microsoft, alongside emerging innovative companies specializing in cloud-based SQL server technologies. Geographic analysis suggests North America and Europe currently hold the largest market share, but significant growth potential exists in the Asia-Pacific region, driven by rapid digital transformation and economic growth in countries like India and China. The restraints on market growth are primarily related to the complexities involved in migrating existing legacy systems to new SQL server solutions, along with the need for skilled professionals to manage and optimize these systems. However, the ongoing advancements in automation tools and the increased availability of training programs are mitigating these challenges. The future trajectory of the market indicates continued growth, driven by emerging technologies such as AI-powered query optimization, enhanced security features, and the growing adoption of serverless architectures. This will lead to a wider adoption of SQL server transformation across various sectors, including finance, healthcare, and retail, as organizations seek to leverage data to gain competitive advantage and improve operational efficiency. The market is ripe for innovation and consolidation, with opportunities for both established players and new entrants to capitalize on this ongoing transformation.

Columnar Database Market Outlook

According to our latest research, the global Columnar Database market size reached USD 3.2 billion in 2024, reflecting a robust demand for high-performance data management solutions across various industries. The market is expected to grow at a CAGR of 13.1% from 2025 to 2033, reaching a forecasted value of USD 8.6 billion by 2033. This remarkable growth trajectory is primarily driven by the exponential increase in data volume, the surge in business intelligence and analytics applications, and the rapid digital transformation initiatives being adopted by enterprises worldwide.

A significant growth factor for the columnar database market is the escalating need for real-time analytics and high-speed data processing. Organizations are increasingly leveraging big data and complex analytics to gain actionable insights and maintain a competitive edge. Traditional row-based databases often struggle with performance bottlenecks when handling large-scale analytical queries. In contrast, columnar databases excel in such environments by enabling faster data retrieval and optimized storage, making them a preferred choice for enterprises seeking to enhance their decision-making processes. The adoption of advanced analytics, artificial intelligence, and machine learning is further fueling the demand for columnar database solutions, as these technologies require rapid access to vast datasets and efficient query performance.

Another critical driver is the widespread adoption of cloud computing and hybrid IT infrastructures. As businesses migrate their workloads to cloud environments, the flexibility, scalability, and cost-effectiveness of columnar databases become increasingly attractive. Cloud-based columnar database solutions offer seamless integration, real-time scalability, and robust disaster recovery capabilities, which are essential for modern enterprises operating in dynamic markets. Additionally, the proliferation of Software-as-a-Service (SaaS) applications and the growing reliance on data-driven business models are pushing organizations to invest in advanced database architectures that can handle the complexities of multi-tenant environments and massive concurrent queries, further accelerating market expansion.

The surge in regulatory compliance requirements and data governance standards is also shaping the growth of the columnar database market. Industries such as BFSI, healthcare, and government are under increasing pressure to manage, store, and analyze sensitive data securely and efficiently. Columnar databases offer enhanced data compression, encryption, and auditing capabilities, making them ideal for organizations that must adhere to stringent regulatory frameworks like GDPR, HIPAA, and PCI DSS. As data privacy concerns and compliance mandates intensify globally, organizations are prioritizing investments in database technologies that not only deliver high performance but also ensure robust data security and governance, thereby fueling market growth.

From a regional perspective, North America continues to lead the columnar database market, driven by the presence of major technology vendors, early adoption of innovative IT solutions, and the high concentration of data-centric industries. Europe follows closely, with significant investments in digital transformation and regulatory compliance initiatives. The Asia Pacific region is emerging as a high-growth market, propelled by rapid industrialization, expanding digital infrastructure, and increasing adoption of cloud-based services across sectors such as retail, BFSI, and healthcare. Latin America and the Middle East & Africa are also witnessing steady growth, albeit at a relatively slower pace, as enterprises in these regions gradually embrace digital transformation and data-driven business strategies.

Component Analysis

The columnar database market is segmented by component into software and services, each playing a pivotal role in the overall ecosystem. The software segment dominates the market, accounting for the largest revenue share in 2024. This dominance is attributed to the continuous advancements in database technologies, increasing demand for high-performance data processing, and the proliferation of data-intensive applications. Modern columnar database software solutions are designed to deliver exceptional query performance, scalability, and flexibility, enabling organizations to efficiently manage and analyze vast volumes of

Context

This dataset contains information on Chicago crime reported between 2015 and 2020.

Content

This dataset is a subset of the BigQuery public database on Chicago Crime.

Acknowledgements

I appreciate the efforts of BigQuery hosting and allowing access to their public databases and Kaggle for providing a space for the widespread sharing of data and knowledge.

Inspiration

This dataset is a useful learning tool for applying descriptive statistics, analytics, and visualisations. For example, one could look at crime trends over time, identify areas with the lowest amount of crime, calculate the propability that an arrest is made based on crime type or area, and determine days of the week with the highest and lowest crime.

Hacker News corpus, 2007-Nov 2022

  Dataset Description





  Dataset Summary

Dataset Name: Hacker News Full Corpus (2007 - November 2022) Description:

NOTE: I am not affiliated with Y Combinator.

This dataset is a July 2023 snapshot of YCombinator's BigQuery dump of the entire archive of posts and comments made on Hacker News. It contains posts from Hacker News' inception in 2007 through to November 16, 2022, when the BigQuery database was last updated. The dataset… See the full description on the dataset page: https://huggingface.co/datasets/jkeisling/hacker-news-corpus-2007-2022.

The MGnify Protein Database is a comprehensive resource that collects protein sequences predicted from publicly available metagenomic assemblies. By integrating data from a vast array of metagenomic datasets, MGnify Proteins enables researchers to explore and analyze over 2.5 billion protein sequences, all of which have stable MGYP-prefixed accessions. Since its launch in August 2017, the database has expanded from just under 50 million sequences to its current scale, offering a rich resource for studying microbial diversity and function. The database facilitates the systematic identification and exploration of protein sequences across diverse environmental and biological contexts, providing valuable insights into the functional potential of microbial communities. To learn more about MGnify Proteins, read our documentation or contact us . About BigQuery This public dataset is hosted in Google BigQuery and is included in BigQuery's 1TB/mo of free tier processing. This means that each user receives 1TB of free BigQuery processing every month, which can be used to run queries on this public dataset. Watch this short video to learn how to get started quickly using BigQuery to access public datasets. What is BigQuery

This is the US Coronavirus data repository from The New York Times . This data includes COVID-19 cases and deaths reported by state and county. The New York Times compiled this data based on reports from state and local health agencies. More information on the data repository is available here . For additional reporting and data visualizations, see The New York Times’ U.S. coronavirus interactive site . This public dataset is hosted in Google BigQuery and is included in BigQuery's 1TB/mo of free tier processing. This means that each user receives 1TB of free BigQuery processing every month, which can be used to run queries on this public dataset. Watch this short video to learn how to get started quickly using BigQuery to access public datasets. What is BigQuery . This dataset has significant public interest in light of the COVID-19 crisis. All bytes processed in queries against this dataset will be zeroed out, making this part of the query free. Data joined with the dataset will be billed at the normal rate to prevent abuse. After September 15, queries over these datasets will revert to the normal billing rate. Users of The New York Times public-use data files must comply with data use restrictions to ensure that the information will be used solely for noncommercial purposes.