A comprehensive Wikipedia dataset containing 100,000 pages with 28.9 million links, collected using breadth-first search crawling algorithm. This dataset includes complete page metadata, link relationships, and a network graph representation suitable for network analysis, graph algorithms, NLP research, and machine learning applications.

Dataset Overview

• Total Pages: 100,000
• Total Links: 28,855,738 (directed edges)
• Average Words per Page: 3,531
• Language: English (en.wikipedia.org)
• Collection Method: BFS (Breadth-First Search) crawling, depth 5
• Data Quality Score: 99.76/100

Files Description

1. pages_export.csv

Complete page metadata including: - id: Unique page ID - title: Page title - language: Language code (en) - content_length: Content length in characters - word_count: Word count - categories: JSON array of categories - infobox: JSON object of infobox data - created_at: Timestamp - url: Full Wikipedia URL

Size: ~70 MB | Rows: 100,000

2. links_export.csv

Complete link graph with URLs: - id: Unique link ID - source_title: Source page title - target_title: Target page title - language: Language code - position: Link position on page - depth: Crawl depth where link was discovered - created_at: Timestamp - source_url: Full source page URL - target_url: Full target page URL

Size: ~4.5 GB | Rows: 28,855,738

3. graph.json

Network graph in JSON format: - nodes: Array of node objects with id field - edges: Array of edge objects with source and target fields

Size: ~2.1 GB | Edges: 28,855,738

Data Quality

• Content Coverage: 99.99% (99,992 pages have quality content)
• Link Quality: 99.22%
• Uniqueness: 100% (all links are unique)
• Content Quality: 100% (average 3,531 words per page)
• Duplicate Pages: Minimal (cleaned)
• Self-Links: 4,326 (removed)
• Data Validation: ✅ All entries validated and cleaned

Use Cases

1. Network Analysis: Study Wikipedia link structure and page connectivity
2. Graph Algorithms: Test shortest path, centrality, community detection algorithms
3. NLP Research: Analyze Wikipedia content, categories, and relationships
4. Machine Learning: Train models on Wikipedia link prediction
5. Knowledge Graph: Build knowledge graphs from Wikipedia structure
6. PageRank: Implement and test PageRank algorithms
7. Recommendation Systems: Build content recommendation systems

Collection Methodology

1. Seed Selection: Started with 5 Wikipedia pages
2. Crawling: BFS algorithm, depth 5
3. Rate Limiting: Balanced (0.82 pages/second)
4. Parallel Processing: Optimized concurrent workers
5. Caching: HTML content cached for efficiency
6. Validation: All data validated and deduplicated
7. Quality Control: Automated quality checks and cleaning

Technical Details

• Database: SQLite with WAL mode
• Crawl Duration: ~29 hours
• Crawl Rate: 0.82 pages/second
• Checkpoint System: Resume-capable crawling
• Data Cleaning: Automated duplicate removal and quality checks