post/2025-08-24-rss_llm_digest_blog.md
Building Your Own LLM-Powered RSS Digest
In an era of information overload, staying up-to-date with your favorite blogs and news sources can be overwhelming. On top of that, the internet is full of click bait content, that distract us from the real important information. RSS feeds offer a solution, but who has time to read dozens of articles daily? This tutorial shows you how to build an intelligent RSS digest that automatically summarizes content using AI.
We’ll walk through three key steps: parsing RSS feeds, extracting full article content, and generating AI-powered summaries using OpenAI’s API.
Prerequisites
Before we start, you’ll need:
- Python 3.7+
- An OpenAI API key
- Basic familiarity with Python
Install the required packages:
1pip install feedparser requests beautifulsoup4 openai
Step 1: Parsing RSS Feeds with Python
RSS (Really Simple Syndication) feeds are XML files that websites use to publish their latest content. Python’s feedparser library makes it easy to work with these feeds.
1import feedparser
2import requests
3from datetime import datetime, timezone
4
5def parse_rss_feed(feed_url):
6 """Parse a single RSS feed and extract articles"""
7
8 # Fetch the RSS feed
9 response = requests.get(feed_url, timeout=30)
10 response.raise_for_status()
11
12 # Parse the XML content
13 feed = feedparser.parse(response.content)
14
15 articles = []
16 for entry in feed.entries:
17 # Extract publication date
18 pub_date = None
19 if hasattr(entry, 'published_parsed') and entry.published_parsed:
20 pub_date = datetime(*entry.published_parsed[:6], tzinfo=timezone.utc)
21
22 # Get article content (summary from RSS)
23 content = getattr(entry, 'summary', '')
24
25 article = {
26 'title': entry.get('title', 'No Title'),
27 'url': entry.get('link', ''),
28 'content': content,
29 'published': pub_date,
30 'author': entry.get('author', 'Unknown')
31 }
32 articles.append(article)
33
34 return articles
35
36# Example usage
37feed_url = "https://feeds.bbci.co.uk/news/rss.xml"
38articles = parse_rss_feed(feed_url)
39print(f"Found {len(articles)} articles")
The feedparser library handles the complexity of XML parsing and provides a clean interface to access article metadata like titles, URLs, publication dates, and summary content.
Step 2: Extracting Full Content with Beautiful Soup
RSS feeds typically only include article summaries. To get the full content for better AI analysis, we need to fetch and parse the actual web pages.
1from bs4 import BeautifulSoup
2import re
3
4def extract_article_content(url):
5 """Fetch and extract main content from a web page"""
6
7 headers = {
8 'User-Agent': 'Mozilla/5.0 (compatible; RSS Reader/1.0)'
9 }
10
11 try:
12 response = requests.get(url, headers=headers, timeout=30)
13 response.raise_for_status()
14
15 # Parse HTML with Beautiful Soup
16 soup = BeautifulSoup(response.text, 'html.parser')
17
18 # Remove unwanted elements
19 for element in soup(['script', 'style', 'nav', 'footer', 'header', 'aside']):
20 element.decompose()
21
22 # Extract text content
23 content = soup.get_text(separator=' ', strip=True)
24
25 # Clean up whitespace
26 content = ' '.join(content.split())
27
28 # Limit content length for LLM processing
29 max_chars = 8000 # Roughly 2000 tokens
30 if len(content) > max_chars:
31 content = content[:max_chars] + "..."
32
33 return content if len(content) > 100 else None
34
35 except Exception as e:
36 print(f"Error extracting content from {url}: {e}")
37 return None
38
39# Enhance articles with full content
40def enhance_articles(articles):
41 """Add full content to articles"""
42
43 for article in articles:
44 if article['url']:
45 full_content = extract_article_content(article['url'])
46 if full_content and len(full_content) > len(article['content']):
47 article['content'] = full_content
48 print(f"Enhanced: {article['title']}")
49
50 return articles
Beautiful Soup excels at parsing HTML and extracting clean text content. We remove navigation elements, scripts, and other non-content parts to focus on the article text.
Step 3: Generating AI Summaries with OpenAI
Now comes the magic: using OpenAI’s API to generate intelligent summaries of our collected articles.
1import openai
2
3class DigestGenerator:
4 def __init__(self, api_key):
5 self.client = openai.OpenAI(api_key=api_key)
6
7 def create_digest(self, articles):
8 """Generate an AI-powered digest of articles"""
9
10 # Prepare articles for the prompt
11 article_summaries = []
12 for i, article in enumerate(articles[:10], 1): # Limit to 10 articles
13 summary = {
14 'title': article['title'],
15 'content': article['content'][:1500], # Truncate for token limits
16 'url': article['url'],
17 'published': article['published'].strftime('%Y-%m-%d') if article['published'] else 'Unknown'
18 }
19 article_summaries.append(summary)
20
21 # Create the prompt
22 prompt = self._build_digest_prompt(article_summaries)
23
24 try:
25 response = self.client.chat.completions.create(
26 model="gpt-3.5-turbo",
27 messages=[{"role": "user", "content": prompt}],
28 temperature=0.3,
29 max_tokens=2000
30 )
31
32 return response.choices[0].message.content
33
34 except Exception as e:
35 print(f"Error generating digest: {e}")
36 return self._fallback_digest(article_summaries)
37
38 def _build_digest_prompt(self, articles):
39 """Build the prompt for AI digest generation"""
40
41 prompt = f"""Create a comprehensive daily digest from these {len(articles)} articles.
42
43Instructions:
44- Summarize the key themes and trends
45- Group related topics together
46- Highlight the most important developments
47- Keep it engaging and informative
48- Use markdown formatting
49
50Articles:
51
52"""
53
54 for i, article in enumerate(articles, 1):
55 prompt += f"""
56## Article {i}: {article['title']}
57**URL:** {article['url']}
58**Published:** {article['published']}
59
60{article['content']}
61
62---
63"""
64
65 return prompt
66
67 def _fallback_digest(self, articles):
68 """Simple fallback if AI fails"""
69 digest = f"# Daily Digest - {datetime.now().strftime('%Y-%m-%d')}\n\n"
70
71 for article in articles:
72 digest += f"## {article['title']}\n"
73 digest += f"**Published:** {article['published']}\n"
74 digest += f"**Link:** {article['url']}\n\n"
75 digest += f"{article['content'][:200]}...\n\n---\n\n"
76
77 return digest
The key to good AI summaries is crafting effective prompts. We give the AI clear instructions and provide structured article data for analysis.
Putting It All Together
Here’s the main function putting all three components together:
1import os
2
3def main():
4 RSS_FEEDS = [
5 "https://feeds.bbci.co.uk/news/rss.xml",
6 "http://rss.cnn.com/rss/cnn_latest.rss/"
7 ]
8
9 print("Fetching articles from RSS feeds...")
10 all_articles = []
11
12 # Step 1: Parse RSS feeds
13 for feed_url in RSS_FEEDS:
14 try:
15 articles = parse_rss_feed(feed_url)
16 all_articles.extend(articles)
17 print(f"Fetched {len(articles)} articles from {feed_url}")
18 except Exception as e:
19 print(f"Error processing {feed_url}: {e}")
20
21 if not all_articles:
22 print("No articles found!")
23 return
24
25 # Sort by publication date (newest first)
26 all_articles.sort(key=lambda x: x['published'] or datetime.min, reverse=True)
27
28 # Step 2: Enhance with full content
29 print("Extracting full article content...")
30 enhanced_articles = enhance_articles(all_articles[:15]) # Process top 15
31
32 # Step 3: Generate AI digest
33 print("Generating AI digest...")
34
35 openai_api_key = os.getenv("OPENAI_API_KEY")
36 generator = DigestGenerator(openai_api_key)
37 digest = generator.create_digest(enhanced_articles)
38
39 # Save and display results
40 timestamp = datetime.now().strftime('%Y%m%d_%H%M%S')
41 filename = f"digest_{timestamp}.md"
42
43 with open(filename, 'w', encoding='utf-8') as f:
44 f.write(digest)
45
46 print(f"\n✅ Digest saved to {filename}")
47 print("\n" + "="*60)
48 print(digest)
49 print("="*60)
50
51if __name__ == "__main__":
52 main()
The full program is available here
Key Considerations
When building your own RSS digest system, keep these points in mind:
Rate Limiting: Be respectful to websites by adding delays between requests and using proper User-Agent headers.
Error Handling: RSS feeds can be unreliable. Always include try-catch blocks and graceful fallbacks.
Content Filtering: Consider adding keyword filtering to focus on topics that interest you most.
Token Limits: OpenAI models have token limits. Truncate content appropriately and batch large requests.
Caching: Store processed articles in a database to avoid re-processing and respect API usage limits. Sqlite3 is a good candidate
Conclusion
With a couple hundred lines of Python code we were able to build a digest system that scrapes information from different feeds, and summarize those using an LLM. I hope this inspired you to byukd your own feed. If the topic is interesting to you, check out Colino, an configurable open source digest system.
In an era of information overload, staying up-to-date with your favorite blogs and news sources can be overwhelming.
On top of that, the internet is full of click bait content, that distract us from the real important information.
RSS feeds offer a solution, but who has time to read dozens of articles daily? This tutorial shows you how to build an intelligent RSS digest that automatically summarizes content using AI.
We'll walk through three key steps: parsing RSS feeds, extracting full article content, and generating AI-powered summaries using OpenAI's API.
## Prerequisites
Before we start, you'll need:
- Python 3.7+
- An OpenAI API key
- Basic familiarity with Python
Install the required packages:
```bash
pip install feedparser requests beautifulsoup4 openai
```
## Step 1: Parsing RSS Feeds with Python
RSS (Really Simple Syndication) feeds are XML files that websites use to publish their latest content. Python's `feedparser` library makes it easy to work with these feeds.
```python
import feedparser
import requests
from datetime import datetime, timezone
def parse_rss_feed(feed_url):
"""Parse a single RSS feed and extract articles"""
# Fetch the RSS feed
response = requests.get(feed_url, timeout=30)
response.raise_for_status()
# Parse the XML content
feed = feedparser.parse(response.content)
articles = []
for entry in feed.entries:
# Extract publication date
pub_date = None
if hasattr(entry, 'published_parsed') and entry.published_parsed:
pub_date = datetime(*entry.published_parsed[:6], tzinfo=timezone.utc)
# Get article content (summary from RSS)
content = getattr(entry, 'summary', '')
article = {
'title': entry.get('title', 'No Title'),
'url': entry.get('link', ''),
'content': content,
'published': pub_date,
'author': entry.get('author', 'Unknown')
}
articles.append(article)
return articles
# Example usage
feed_url = "https://feeds.bbci.co.uk/news/rss.xml"
articles = parse_rss_feed(feed_url)
print(f"Found {len(articles)} articles")
```
The `feedparser` library handles the complexity of XML parsing and provides a clean interface to access article metadata like titles, URLs, publication dates, and summary content.
## Step 2: Extracting Full Content with Beautiful Soup
RSS feeds typically only include article summaries. To get the full content for better AI analysis, we need to fetch and parse the actual web pages.
```python
from bs4 import BeautifulSoup
import re
def extract_article_content(url):
"""Fetch and extract main content from a web page"""
headers = {
'User-Agent': 'Mozilla/5.0 (compatible; RSS Reader/1.0)'
}
try:
response = requests.get(url, headers=headers, timeout=30)
response.raise_for_status()
# Parse HTML with Beautiful Soup
soup = BeautifulSoup(response.text, 'html.parser')
# Remove unwanted elements
for element in soup(['script', 'style', 'nav', 'footer', 'header', 'aside']):
element.decompose()
# Extract text content
content = soup.get_text(separator=' ', strip=True)
# Clean up whitespace
content = ' '.join(content.split())
# Limit content length for LLM processing
max_chars = 8000 # Roughly 2000 tokens
if len(content) > max_chars:
content = content[:max_chars] + "..."
return content if len(content) > 100 else None
except Exception as e:
print(f"Error extracting content from {url}: {e}")
return None
# Enhance articles with full content
def enhance_articles(articles):
"""Add full content to articles"""
for article in articles:
if article['url']:
full_content = extract_article_content(article['url'])
if full_content and len(full_content) > len(article['content']):
article['content'] = full_content
print(f"Enhanced: {article['title']}")
return articles
```
Beautiful Soup excels at parsing HTML and extracting clean text content. We remove navigation elements, scripts, and other non-content parts to focus on the article text.
## Step 3: Generating AI Summaries with OpenAI
Now comes the magic: using OpenAI's API to generate intelligent summaries of our collected articles.
```python
import openai
class DigestGenerator:
def __init__(self, api_key):
self.client = openai.OpenAI(api_key=api_key)
def create_digest(self, articles):
"""Generate an AI-powered digest of articles"""
# Prepare articles for the prompt
article_summaries = []
for i, article in enumerate(articles[:10], 1): # Limit to 10 articles
summary = {
'title': article['title'],
'content': article['content'][:1500], # Truncate for token limits
'url': article['url'],
'published': article['published'].strftime('%Y-%m-%d') if article['published'] else 'Unknown'
}
article_summaries.append(summary)
# Create the prompt
prompt = self._build_digest_prompt(article_summaries)
try:
response = self.client.chat.completions.create(
model="gpt-3.5-turbo",
messages=[{"role": "user", "content": prompt}],
temperature=0.3,
max_tokens=2000
)
return response.choices[0].message.content
except Exception as e:
print(f"Error generating digest: {e}")
return self._fallback_digest(article_summaries)
def _build_digest_prompt(self, articles):
"""Build the prompt for AI digest generation"""
prompt = f"""Create a comprehensive daily digest from these {len(articles)} articles.
Instructions:
- Summarize the key themes and trends
- Group related topics together
- Highlight the most important developments
- Keep it engaging and informative
- Use markdown formatting
Articles:
"""
for i, article in enumerate(articles, 1):
prompt += f"""
## Article {i}: {article['title']}
**URL:** {article['url']}
**Published:** {article['published']}
{article['content']}
---
"""
return prompt
def _fallback_digest(self, articles):
"""Simple fallback if AI fails"""
digest = f"# Daily Digest - {datetime.now().strftime('%Y-%m-%d')}\n\n"
for article in articles:
digest += f"## {article['title']}\n"
digest += f"**Published:** {article['published']}\n"
digest += f"**Link:** {article['url']}\n\n"
digest += f"{article['content'][:200]}...\n\n---\n\n"
return digest
```
The key to good AI summaries is crafting effective prompts. We give the AI clear instructions and provide structured article data for analysis.
## Putting It All Together
Here's the main function putting all three components together:
```python
import os
def main():
RSS_FEEDS = [
"https://feeds.bbci.co.uk/news/rss.xml",
"http://rss.cnn.com/rss/cnn_latest.rss/"
]
print("Fetching articles from RSS feeds...")
all_articles = []
# Step 1: Parse RSS feeds
for feed_url in RSS_FEEDS:
try:
articles = parse_rss_feed(feed_url)
all_articles.extend(articles)
print(f"Fetched {len(articles)} articles from {feed_url}")
except Exception as e:
print(f"Error processing {feed_url}: {e}")
if not all_articles:
print("No articles found!")
return
# Sort by publication date (newest first)
all_articles.sort(key=lambda x: x['published'] or datetime.min, reverse=True)
# Step 2: Enhance with full content
print("Extracting full article content...")
enhanced_articles = enhance_articles(all_articles[:15]) # Process top 15
# Step 3: Generate AI digest
print("Generating AI digest...")
openai_api_key = os.getenv("OPENAI_API_KEY")
generator = DigestGenerator(openai_api_key)
digest = generator.create_digest(enhanced_articles)
# Save and display results
timestamp = datetime.now().strftime('%Y%m%d_%H%M%S')
filename = f"digest_{timestamp}.md"
with open(filename, 'w', encoding='utf-8') as f:
f.write(digest)
print(f"\n✅ Digest saved to {filename}")
print("\n" + "="*60)
print(digest)
print("="*60)
if __name__ == "__main__":
main()
```
The full program is available [here](https://gist.github.com/dovidio/a36ad141cfd6f33966a8bbfc2ffcb92f)
## Key Considerations
When building your own RSS digest system, keep these points in mind:
**Rate Limiting**: Be respectful to websites by adding delays between requests and using proper User-Agent headers.
**Error Handling**: RSS feeds can be unreliable. Always include try-catch blocks and graceful fallbacks.
**Content Filtering**: Consider adding keyword filtering to focus on topics that interest you most.
**Token Limits**: OpenAI models have token limits. Truncate content appropriately and batch large requests.
**Caching**: Store processed articles in a database to avoid re-processing and respect API usage limits. Sqlite3 is a good candidate
## Conclusion
With a couple hundred lines of Python code we were able to build a digest system that scrapes information from different feeds, and summarize those using an LLM.
I hope this inspired you to byukd your own feed. If the topic is interesting to you, check out [Colino](https://github.com/dovidio/colino), an configurable open source digest system.