AI University News

Overview

AI University News is an automated web crawler that monitors press releases and news articles from top universities, national laboratories, and research institutions worldwide, focusing specifically on AI-related research and developments. The system runs daily to discover, analyze, and report the latest AI breakthroughs from academia and research labs.

How the Crawler Works

Phase 1: Discovery & Crawling

The crawler visits official news and press release pages from:

• Peer Institutions: Top-tier research universities (MIT, Stanford, Carnegie Mellon, etc.)
• R1 Universities: All Carnegie R1 research universities across the United States
• HPC & Research Centers: NSF supercomputing centers and DOE computing facilities (TACC, SDSC, NERSC, etc.)
• National Laboratories: DOE labs, federal research labs, and FFRDCs (Argonne, DARPA, MITRE, etc.)
• Global Institutions: Leading international universities and research organizations (Oxford, ETH Zurich, Tsinghua, etc.)

Using the Scrapy framework, the system respectfully crawls these sites following robots.txt rules, implementing politeness delays between requests, and using proper identification.

Phase 2: Content Extraction

When new articles are discovered, the crawler uses Trafilatura (a state-of-the-art content extraction library) to extract clean article text, metadata, publication dates, and author information with 95%+ accuracy.

Phase 3: Deduplication

The system maintains a PostgreSQL database tracking every URL and content hash to ensure:

• No duplicate URLs are processed
• Updated articles are detected through content hash comparison
• Fast O(1) lookups using SHA-256 hashing

Phase 4: AI Analysis

This is where the magic happens. Each new article is analyzed by Claude (Sonnet 4.6), Anthropic's frontier model for deep research understanding. Articles are classified by relevance, key topics are extracted, summaries are generated, and confidence scores are assigned.

Phase 5: Categorization & Organization

Articles are automatically organized into five categories:

• Peer Institutions: Elite research universities with the highest AI research output
• R1 Institutions: All US Carnegie R1 research universities
• HPC & Research Centers: NSF supercomputing centers and DOE computing facilities
• National Laboratories: DOE national labs, federal government research labs, and FFRDCs
• Global Institutions: Leading international universities and research organizations

Phase 6: Publishing

The crawler automatically generates this website with:

• Today's Page: Latest articles from the past 3 days
• Archive: Historical daily reports accessible by date
• Five-Column Layout: Easy browsing by institution category

Results can also be delivered via Slack webhooks and email notifications for real-time updates.

Technical Architecture

Technology Stack

• Language: Python 3.11+
• Crawling: Scrapy 2.11+ with custom spiders
• Content Extraction: Trafilatura 2.0+ with htmldate
• Database: PostgreSQL 15+ for metadata and tracking
• AI APIs: Anthropic Claude (Sonnet 4.6)
• Deployment: Systemd service with daily automated runs

Ethical Crawling

This crawler follows web crawling best practices:

• Always respects robots.txt directives
• Implements per-domain rate limiting (1 request/second default)
• Uses descriptive User-Agent with contact information
• Implements exponential backoff for failed requests
• Never attempts to bypass access controls or paywalls

Cost & Efficiency

The system is designed to be cost-effective:

• Estimated monthly cost: ~$36/month for AI API usage (100 articles/day)
• Optimization: Claude Sonnet 4.6 handles all analysis in a single pass, minimizing redundant API calls
• Caching: All AI responses stored to avoid reprocessing
• Smart limits: Token limits and max articles per run prevent runaway costs

Source Code

This is an open-source project. The complete source code, documentation, and deployment guides are available on GitHub. The system is designed as a standalone Linux application that can be deployed on any server with Python 3.11+ and PostgreSQL.

Complete Source List

This crawler monitors 391 sources across five categories:

Updates & Schedule

The crawler runs automatically once per day (typically early morning UTC) and this website updates immediately after each run completes. The archive preserves all historical daily reports for research and trend analysis.