BM25 & TF-IDF Algorithms For SEO

9 min read

Updated 1 day ago

BM25 & TF-IDF Algorithms

Overview

The SEO system leverages two powerful information retrieval algorithms - BM25 (Best Matching 25) and TF-IDF (Term Frequency-Inverse Document Frequency) - to provide advanced content analysis, keyword optimization, and relevance scoring. These algorithms form the mathematical foundation for content optimization recommendations, keyword density analysis, and search relevance scoring throughout the system.

Accessing Algorithm Features

Navigation Path: Dashboard > SEO System > Various modules (integrated throughout)

Required Permissions:

seo.view - View SEO analysis results
seo.analyze - Run SEO analysis
blog.view - Access blog SEO integration

Algorithm Overview

TF-IDF (Term Frequency-Inverse Document Frequency)

TF-IDF is a numerical statistic that reflects how important a word is to a document in a collection of documents. It's used throughout the SEO system for:

Keyword extraction from content
Keyword density analysis and optimization
Content similarity calculations
Document ranking and comparison

Mathematical Formula:

TF-IDF = TF(term, document) × IDF(term, corpus)

Where:
- TF = (Number of times term appears) / (Total terms in document)
- IDF = log(Total documents / Documents containing term)

BM25 (Best Matching 25)

BM25 is a more sophisticated ranking function that improves upon TF-IDF by addressing its limitations. It's used for:

Content relevance scoring against target keywords
Search result ranking and optimization
Content optimization recommendations
Similarity analysis between documents

Mathematical Formula:

BM25 = Σ IDF(qi) × (f(qi,D) × (k1 + 1)) / (f(qi,D) + k1 × (1 - b + b × |D| / avgdl))

Where:
- qi = query term
- f(qi,D) = frequency of qi in document D
- |D| = length of document D
- avgdl = average document length in corpus
- k1 = term frequency saturation parameter (default: 1.5)
- b = length normalization parameter (default: 0.75)

Key Features

TF-IDF Analysis Features

Keyword Extraction: Automatically identify the most important terms in content
Density Analysis: Calculate optimal keyword density percentages
Content Similarity: Find similar content based on term frequency patterns
Stop Word Filtering: Intelligent filtering of common words
Multi-language Support: Tokenization and analysis for different languages

BM25 Scoring Features

Relevance Scoring: Advanced scoring of content relevance to target keywords
Length Normalization: Accounts for document length in scoring
Saturation Handling: Prevents over-optimization penalties
Query Analysis: Multi-term query processing and scoring
Optimization Suggestions: AI-powered recommendations based on BM25 analysis

Algorithm Integration Points

1. Blog Post Editor Integration

Location: Blog Management > Create/Edit Post > SEO Analysis Panel

The algorithms provide real-time analysis while writing:

// Real-time TF-IDF keyword extraction
$keywords = $tfidfService->extractKeywords($content, $corpus, 10);

// BM25 relevance scoring for focus keyword
$relevance = $bm25Service->analyzeRelevance($content, $focusKeyword, $relatedKeywords, $corpus);

Features:

Live keyword density calculation
Real-time relevance scoring
Optimization suggestions as you type
Keyword prominence analysis

2. SEO Dashboard Analytics

Location: SEO System > Dashboard > Bulk Analysis

Bulk content analysis using both algorithms:

// Analyze multiple posts for keyword optimization
foreach ($posts as $post) {
    $tfidfScores = $tfidfService->calculate($post->content, $corpus);
    $bm25Score = $bm25Service->score($targetKeyword, $post->content, $corpus);
}

Features:

Batch processing of content
Comparative analysis across posts
Performance trending
Optimization priority ranking

3. Keyword Research Tools

Location: SEO System > Keywords > Analysis Tools

Advanced keyword analysis and suggestions:

// Find similar content using TF-IDF
$similarPosts = $tfidfService->findSimilarDocuments($content, $corpus, 5);

// Get BM25-based optimization suggestions
$suggestions = $bm25Service->getOptimizationSuggestions($content, $keyword, $corpus);

Features:

Semantic keyword discovery
Content gap analysis
Competitive keyword analysis
Optimization opportunity identification

4. Content Optimization Engine

Location: SEO System > Content Optimization > Analysis Results

Comprehensive content analysis and recommendations:

// Comprehensive content analysis
$analysis = [
    'tfidf_keywords' => $tfidfService->extractKeywords($content, $corpus),
    'keyword_density' => $tfidfService->analyzeKeywordDensity($content, $keywords),
    'bm25_relevance' => $bm25Service->analyzeRelevance($content, $focusKeyword),
    'optimization_suggestions' => $bm25Service->getOptimizationSuggestions($content, $focusKeyword)
];

Algorithm Configuration

TF-IDF Parameters

Configurable Settings:

Stop Words List
```
$tfidfService->setStopWords(['custom', 'stop', 'words']);
```
- Default: Common English stop words
- Customizable per language
- Industry-specific exclusions
Minimum Word Length
```
$tfidfService->setMinWordLength(3);
```
- Default: 3 characters
- Filters very short words
- Improves analysis quality

Cache Settings

$options = [
    'use_cache' => true,
    'cache_key' => 'post_' . $postId,
];

1-hour default TTL
Improves performance
Automatic cache invalidation

BM25 Parameters

Tunable Parameters:

k1 Parameter (Term Frequency Saturation)
```
$bm25Service->setParameters($k1 = 1.5, $b = 0.75);
```
- Default: 1.5
- Range: 1.2 - 2.0
- Higher values = less saturation
b Parameter (Length Normalization)
- Default: 0.75
- Range: 0.0 - 1.0
- Higher values = more length normalization

Custom Corpus

$corpus = [
    ['id' => 1, 'content' => 'Document 1 content...'],
    ['id' => 2, 'content' => 'Document 2 content...'],
];

Performance Optimization

Caching Strategy

Both algorithms implement intelligent caching:

// TF-IDF caching
Cache::put("tfidf:{$cacheKey}", $results, 3600);

// BM25 caching
Cache::put("bm25:{$cacheKey}", $scores, 3600);

Cache Benefits:

Reduces computation time by 90%+
Automatic cache invalidation
Memory-efficient storage
Configurable TTL

Large Content Sets

Optimization Techniques:

Batch Processing
- Process content in chunks
- Prevent memory exhaustion
- Progress tracking
Selective Analysis
- Analyze only changed content
- Skip unchanged documents
- Incremental updates
Background Processing
- Queue heavy computations
- Non-blocking user interface
- Progress notifications

Database Optimization

Performance Considerations:

-- Optimized queries for corpus building
SELECT id, title, content, created_at 
FROM blog_posts 
WHERE status = 'published' 
ORDER BY created_at DESC 
LIMIT 1000;

Indexing Strategy:

Index on content length
Full-text search indexes
Composite indexes for filtering

Algorithm Integration with Search

Search Functionality

The algorithms power advanced search features:

// BM25-powered search ranking
$results = $bm25Service->rank($query, $corpus, $limit = 10);

// TF-IDF similarity search
$similar = $tfidfService->findSimilarDocuments($document, $corpus, $limit = 5);

Search Features:

Relevance-based ranking
Semantic similarity matching
Multi-term query processing
Result scoring and explanation

Content Recommendations

Related Content Discovery:

// Find related posts using TF-IDF
$relatedPosts = $tfidfService->findSimilarDocuments(
    $currentPost->content,
    $allPosts,
    5
);

// BM25-based content suggestions
$suggestions = $bm25Service->findSimilar(
    $currentPost->content,
    $corpus,
    3
);

Common Tasks

Task 1: Analyze Content Relevance

Navigate to SEO System > Content Optimization
Select the content to analyze
Enter your target keyword
Click "Analyze Relevance"
Review BM25 relevance score and recommendations
Apply suggested optimizations

Task 2: Extract Important Keywords

Go to SEO System > Keywords > Analysis
Paste or select content to analyze
Set the number of keywords to extract (default: 10)
Click "Extract Keywords"
Review TF-IDF scores for each keyword
Use keywords for content optimization

Task 3: Find Similar Content

Navigate to Blog Management > Posts
Select a post to find similar content for
Click "Find Similar" in the SEO panel
Review similarity scores and suggestions
Use insights for content planning

Task 4: Optimize Keyword Density

Open the Blog Post Editor
Enter your target keyword in SEO settings
Write or edit your content
Monitor real-time density analysis
Adjust content based on TF-IDF recommendations
Aim for optimal density range (0.5% - 2.5%)

Algorithm Results Interpretation

TF-IDF Scores

Score Ranges:

0.0 - 0.1: Low importance terms
0.1 - 0.3: Medium importance terms
0.3 - 0.5: High importance terms
0.5+: Very high importance terms

Interpretation:

Higher scores indicate more unique and important terms
Use high-scoring terms as primary keywords
Balance with search volume and competition

BM25 Relevance Scores

Score Interpretation:

0-2: Poor relevance
2-4: Fair relevance
4-6: Good relevance
6-8: Very good relevance
8+: Excellent relevance

Optimization Targets:

Aim for scores above 4.0 for target keywords
Scores above 6.0 indicate well-optimized content
Monitor score changes after optimization

Keyword Density Analysis

Optimal Ranges:

Primary Keyword: 0.5% - 2.5%
Secondary Keywords: 0.3% - 1.5%
Long-tail Keywords: 0.1% - 1.0%

Status Indicators:

Low: Below optimal range, increase usage
Optimal: Within recommended range
High: Above optimal range, reduce usage

Troubleshooting

Problem: Low TF-IDF Scores for Important Keywords

Symptoms:

Important keywords showing low TF-IDF scores
Poor keyword extraction results
Unexpected keyword rankings

Solutions:

Check Stop Words List
- Verify important terms aren't in stop words
- Customize stop words for your industry
- Remove domain-specific terms from exclusions
Adjust Minimum Word Length
- Lower minimum length for short important terms
- Consider abbreviations and acronyms
- Review tokenization results
Expand Corpus
- Include more relevant documents in corpus
- Ensure corpus represents your content domain
- Update corpus regularly

Problem: BM25 Scores Not Improving

Symptoms:

Optimization changes don't improve BM25 scores
Scores remain consistently low
Unexpected score fluctuations

Solutions:

Review BM25 Parameters

// Try different parameter values
$bm25Service->setParameters($k1 = 1.2, $b = 0.5);

Check Document Length
- Very short or very long documents may score poorly
- Aim for average document length in your corpus
- Consider content structure and formatting
Analyze Keyword Placement
- Ensure keywords appear early in content
- Use keywords in headings and important sections
- Avoid keyword stuffing

Problem: Performance Issues with Large Content Sets

Symptoms:

Slow analysis processing
Memory exhaustion errors
Timeout issues

Solutions:

Enable Caching

$options = ['use_cache' => true, 'cache_key' => $uniqueKey];

Implement Batch Processing
- Process content in smaller chunks
- Use background job queues
- Implement progress tracking
Optimize Corpus Size
- Limit corpus to most relevant documents
- Use sampling for very large datasets
- Implement corpus rotation

Problem: Inconsistent Results

Symptoms:

Different results for same content
Scores varying between analyses
Unexpected algorithm behavior

Solutions:

Check Cache Consistency

// Clear cache for fresh analysis
$tfidfService->invalidateCache($cacheKey);
$bm25Service->invalidateCache($cacheKey);

Verify Input Data
- Ensure consistent text encoding
- Check for hidden characters
- Validate content preprocessing
Review Configuration
- Verify algorithm parameters
- Check stop words configuration
- Ensure consistent corpus usage

API Integration

TF-IDF API Endpoints

// Extract keywords via API
POST /api/seo/tfidf/extract-keywords
{
    "content": "Your content here...",
    "limit": 10,
    "corpus_id": "blog_posts"
}

// Calculate similarity
POST /api/seo/tfidf/similarity
{
    "document1": "First document...",
    "document2": "Second document...",
    "corpus_id": "blog_posts"
}

BM25 API Endpoints

// Analyze relevance
POST /api/seo/bm25/analyze-relevance
{
    "content": "Your content here...",
    "focus_keyword": "target keyword",
    "related_keywords": ["keyword1", "keyword2"],
    "corpus_id": "blog_posts"
}

// Get optimization suggestions
POST /api/seo/bm25/optimize
{
    "content": "Your content here...",
    "target_keyword": "optimization target",
    "corpus_id": "blog_posts"
}