Pattern Overview

Scheduled analysis runs recurring analysis on a fixed schedule to track changes over time.

The Scheduled Analysis Flow

┌─────────────────────────────────────────────────────────────┐
│           SCHEDULED ANALYSIS PATTERN                        │
│                                                             │
│  1. Run on Schedule                                        │
│     • Daily (market reports)                               │
│     • Weekly (competitive analysis)                        │
│     • Monthly (trend reports)                              │
│     • Quarterly (business reviews)                         │
│                                                             │
│            ▼                                                │
│  2. Gather Information                                     │
│     • Web scraping                                         │
│     • API calls                                            │
│     • Screenshots                                          │
│     • Database queries                                     │
│                                                             │
│            ▼                                                │
│  3. Detect Changes                                         │
│     • Compare with previous                                │
│     • Calculate diffs                                      │
│     • Score significance                                   │
│     • Filter noise                                         │
│                                                             │
│            ▼                                                │
│  4. Analyze Trends                                         │
│     • Pattern recognition                                  │
│     • Impact assessment                                    │
│     • Insight extraction                                   │
│     • Recommendations                                      │
│                                                             │
│            ▼                                                │
│  5. Report Findings                                        │
│     • Generate report                                      │
│     • Distribute to team                                   │
│     • Archive results                                      │
│     • Alert on critical changes                            │
└─────────────────────────────────────────────────────────────┘

Common Use Cases

Competitive Intelligence:

Monitor competitor websites → Detect changes → Analyze impact
Track pricing → Identify trends → Pricing recommendations
Watch product launches → Assess threat → Strategic response

Market Analysis:

Industry news → Trend identification → Market report
Social sentiment → Brand perception → Marketing insights
Job postings → Hiring trends → Talent strategy

Performance Monitoring:

Website performance → Compare competitors → Optimization recommendations
SEO rankings → Track movement → SEO strategy
App store rankings → Competitive position → Product insights

Regulatory Monitoring:

Policy changes → Impact analysis → Compliance report
Legal updates → Risk assessment → Legal briefing
Industry standards → Gap analysis → Compliance roadmap

Example: Competitive Research Agent

The Problem

Your product team needs to track 5 competitors across:

Website changes (features, pricing, messaging)
Product updates (new features, announcements)
Marketing campaigns (ads, content, positioning)
Job postings (team growth, new roles)

Current process: Product manager manually checks competitor sites weekly, takes notes, shares in Slack (3-4 hours/week).

What we'll build: An AI agent that monitors competitors daily, detects significant changes, analyzes impact, and delivers a weekly competitive intelligence report.

Architecture

┌─────────────────────────────────────────────────────────────┐
│      COMPETITIVE RESEARCH AGENT ARCHITECTURE                │
│                                                             │
│  Daily Cron (9am)                                          │
│       │                                                     │
│       ▼                                                     │
│  ┌──────────────────┐                                      │
│  │ Data Collectors  │                                      │
│  │                  │                                      │
│  │ • Web scraper    │                                      │
│  │ • Screenshot     │                                      │
│  │ • LinkedIn API   │                                      │
│  │ • Social APIs    │                                      │
│  └────────┬─────────┘                                      │
│           │                                                 │
│           ▼                                                 │
│  ┌─────────────────────────┐                              │
│  │ Change Detection         │                              │
│  │                          │                              │
│  │ • Compare with baseline  │                              │
│  │ • Calculate diffs        │                              │
│  │ • Score significance     │                              │
│  │ • Filter noise           │                              │
│  └────────┬────────────────┘                               │
│           │                                                 │
│           ▼                                                 │
│  ┌─────────────────────────┐                              │
│  │  Analysis Agent          │                              │
│  │                          │                              │
│  │ • Categorize changes     │                              │
│  │ • Assess impact          │                              │
│  │ • Identify trends        │                              │
│  │ • Generate insights      │                              │
│  └────────┬────────────────┘                               │
│           │                                                 │
│           ▼                                                 │
│  ┌─────────────────────────┐                              │
│  │  Weekly Reporter         │                              │
│  │  (Sundays at 6pm)        │                              │
│  │                          │                              │
│  │ • Aggregate week's data  │                              │
│  │ • Generate report        │                              │
│  │ • Send to Slack + Email  │                              │
│  └──────────────────────────┘                              │
└─────────────────────────────────────────────────────────────┘

Setup

Dependencies


npm install @mastra/core @ai-sdk/openai zod
npm install cheerio playwright
npm install sharp # Screenshot processing
npm install diff # Text diffing
npm install prisma @prisma/client

Configuration


// competitors.config.ts
export const competitors = [
  {
    id: 'competitor-a',
    name: 'Competitor A',
    website: 'https://competitora.com',
    pricingPage: 'https://competitora.com/pricing',
    blogRss: 'https://competitora.com/blog/feed',
    linkedin: 'https://linkedin.com/company/competitora',
    priority: 'high',
  },
  {
    id: 'competitor-b',
    name: 'Competitor B',
    website: 'https://competitorb.com',
    pricingPage: 'https://competitorb.com/pricing',
    priority: 'medium',
  },
];

export const trackingConfig = {
  pages: [
    { type: 'homepage', selector: 'main' },
    { type: 'pricing', selector: '.pricing-table' },
    { type: 'features', selector: '.features-list' },
  ],
  schedule: {
    daily: '0 9 * * *', // 9am daily
    weekly: '0 18 * * 0', // 6pm Sunday
  },
  changeThreshold: 0.05, // 5% change to be significant
};

Database Schema


model Competitor {
  id        String   @id
  name      String
  website   String
  priority  String
  snapshots Snapshot[]
  changes   Change[]
}

model Snapshot {
  id           String   @id @default(cuid())
  competitorId String
  competitor   Competitor @relation(fields: [competitorId], references: [id])
  pageType     String
  url          String
  content      String   @db.Text
  screenshot   String?  // S3 URL
  metadata     Json
  capturedAt   DateTime @default(now())

  @@index([competitorId, pageType, capturedAt])
}

model Change {
  id           String   @id @default(cuid())
  competitorId String
  competitor   Competitor @relation(fields: [competitorId], references: [id])
  pageType     String
  changeType   String   // 'content', 'pricing', 'feature', 'design'
  description  String
  significance Float    // 0-1 score
  before       String   @db.Text
  after        String   @db.Text
  detectedAt   DateTime @default(now())
  analyzed     Boolean  @default(false)

  @@index([competitorId, detectedAt])
  @@index([analyzed])
}

model Report {
  id          String   @id @default(cuid())
  weekStart   DateTime
  weekEnd     DateTime
  content     Json
  generatedAt DateTime @default(now())
}

Scheduled Trigger

Vercel Cron Setup


// vercel.json
{
  "crons": [
    {
      "path": "/api/cron/daily-competitive-scan",
      "schedule": "0 9 * * *"
    },
    {
      "path": "/api/cron/weekly-competitive-report",
      "schedule": "0 18 * * 0"
    }
  ]
}

Daily Scan Endpoint


// app/api/cron/daily-competitive-scan/route.ts
import { NextRequest, NextResponse } from 'next/server';
import { competitors } from '@/competitors.config';
import { DataCollector } from '@/lib/data-collector';
import { ChangeDetector } from '@/lib/change-detector';
import { analysisAgent } from '@/mastra/agents/analysis';

export async function GET(request: NextRequest) {
  if (request.headers.get('authorization') !== `Bearer ${process.env.CRON_SECRET}`) {
    return NextResponse.json({ error: 'Unauthorized' }, { status: 401 });
  }

  const collector = new DataCollector();
  const detector = new ChangeDetector();

  let totalChanges = 0;

  for (const competitor of competitors) {
    try {
      console.log(`Scanning ${competitor.name}...`);

      // Collect current data
      const snapshots = await collector.collectAll(competitor);

      // Detect changes
      const changes = await detector.detectChanges(competitor.id, snapshots);

      if (changes.length > 0) {
        console.log(`${competitor.name}: ${changes.length} changes detected`);
        totalChanges += changes.length;

        // Analyze significant changes immediately
        for (const change of changes) {
          if (change.significance > 0.7) {
            // Alert on critical changes
            await alertCriticalChange(competitor, change);
          }
        }
      }
    } catch (error) {
      console.error(`Failed to scan ${competitor.name}:`, error);
    }
  }

  return NextResponse.json({
    success: true,
    competitors: competitors.length,
    changes: totalChanges,
  });
}

Data Collection

Web Scraper


// lib/data-collector.ts
import * as cheerio from 'cheerio';
import { chromium } from 'playwright';

export class DataCollector {
  async collectAll(competitor: Competitor): Promise<Snapshot[]> {
    const snapshots: Snapshot[] = [];

    // Homepage
    const homepage = await this.scrapeHTML(
      competitor.website,
      'homepage',
      'main'
    );
    snapshots.push(homepage);

    // Pricing page
    if (competitor.pricingPage) {
      const pricing = await this.scrapeHTML(
        competitor.pricingPage,
        'pricing',
        '.pricing-table, .pricing-section, main'
      );
      snapshots.push(pricing);
    }

    // Screenshots
    const screenshot = await this.captureScreenshot(competitor.website);
    snapshots[0].screenshot = screenshot;

    return snapshots;
  }

  private async scrapeHTML(
    url: string,
    pageType: string,
    selector: string
  ): Promise<Snapshot> {
    const response = await fetch(url);
    const html = await response.text();

    const $ = cheerio.load(html);
    
    // Extract relevant content
    const content = $(selector).text().trim();
    
    // Extract metadata
    const metadata = {
      title: $('title').text(),
      description: $('meta[name="description"]').attr('content'),
      h1: $('h1').first().text(),
      links: $('a').map((_, el) => $(el).attr('href')).get(),
    };

    return {
      id: crypto.randomUUID(),
      competitorId: '',
      pageType,
      url,
      content: this.normalizeText(content),
      metadata,
      capturedAt: new Date(),
    };
  }

  private async captureScreenshot(url: string): Promise<string> {
    const browser = await chromium.launch();
    const page = await browser.newPage();

    await page.goto(url, { waitUntil: 'networkidle' });
    
    const screenshot = await page.screenshot({
      fullPage: true,
      type: 'png',
    });

    await browser.close();

    // Upload to S3 or similar
    const s3Url = await this.uploadToS3(screenshot, `screenshot-${Date.now()}.png`);
    
    return s3Url;
  }

  private normalizeText(text: string): string {
    return text
      .replace(/\s+/g, ' ')
      .replace(/\n+/g, '\n')
      .trim();
  }

  private async uploadToS3(buffer: Buffer, key: string): Promise<string> {
    // S3 upload implementation
    return `https://s3.amazonaws.com/bucket/${key}`;
  }
}

LinkedIn Job Scraper


// lib/linkedin-scraper.ts
export class LinkedInScraper {
  async getJobPostings(companyUrl: string): Promise<JobPosting[]> {
    const browser = await chromium.launch();
    const page = await browser.newPage();

    await page.goto(`${companyUrl}/jobs`);
    await page.waitForSelector('.job-card');

    const jobs = await page.$$eval('.job-card', (elements) =>
      elements.map((el) => ({
        title: el.querySelector('.job-title')?.textContent || '',
        location: el.querySelector('.job-location')?.textContent || '',
        postedDate: el.querySelector('.posted-date')?.textContent || '',
        link: el.querySelector('a')?.href || '',
      }))
    );

    await browser.close();

    return jobs;
  }

  async detectHiringTrends(
    competitorId: string,
    currentJobs: JobPosting[]
  ): Promise<HiringTrend> {
    const previousJobs = await db.jobPosting.findMany({
      where: { competitorId },
      orderBy: { scrapedAt: 'desc' },
      take: 100,
    });

    const newRoles = currentJobs.filter(
      (job) => !previousJobs.some((prev) => prev.title === job.title)
    );

    const departments = this.categorizeByDepartment(newRoles);

    return {
      totalOpenings: currentJobs.length,
      newRoles: newRoles.length,
      topDepartments: departments,
      growthSignal: newRoles.length > 10 ? 'high' : 'low',
    };
  }
}

Change Detection

Diff Calculator


// lib/change-detector.ts
import { diffWords, diffLines } from 'diff';
import { db } from './db';

export class ChangeDetector {
  async detectChanges(
    competitorId: string,
    newSnapshots: Snapshot[]
  ): Promise<Change[]> {
    const changes: Change[] = [];

    for (const snapshot of newSnapshots) {
      // Get previous snapshot of same page type
      const previous = await db.snapshot.findFirst({
        where: {
          competitorId,
          pageType: snapshot.pageType,
        },
        orderBy: { capturedAt: 'desc' },
      });

      if (!previous) {
        // First snapshot - save and continue
        await this.saveSnapshot(competitorId, snapshot);
        continue;
      }

      // Calculate diff
      const diff = this.calculateDiff(previous.content, snapshot.content);

      if (diff.significance > 0.05) {
        const change = await this.createChange(
          competitorId,
          snapshot.pageType,
          previous.content,
          snapshot.content,
          diff
        );

        changes.push(change);
      }

      // Save new snapshot
      await this.saveSnapshot(competitorId, snapshot);
    }

    return changes;
  }

  private calculateDiff(before: string, after: string): DiffResult {
    const wordDiff = diffWords(before, after);

    let addedWords = 0;
    let removedWords = 0;
    let totalWords = 0;

    for (const part of wordDiff) {
      const words = part.value.split(/\s+/).length;
      totalWords += words;

      if (part.added) {
        addedWords += words;
      } else if (part.removed) {
        removedWords += words;
      }
    }

    const changedWords = addedWords + removedWords;
    const significance = totalWords > 0 ? changedWords / totalWords : 0;

    // Extract notable changes
    const notableChanges = wordDiff
      .filter((part) => part.added || part.removed)
      .map((part) => ({
        type: part.added ? 'added' : 'removed',
        text: part.value.trim(),
      }));

    return {
      significance,
      addedWords,
      removedWords,
      notableChanges,
      diff: wordDiff,
    };
  }

  private async createChange(
    competitorId: string,
    pageType: string,
    before: string,
    after: string,
    diff: DiffResult
  ): Promise<Change> {
    // Classify change type
    const changeType = this.classifyChange(pageType, diff);

    // Generate description
    const description = this.generateDescription(changeType, diff);

    const change = await db.change.create({
      data: {
        competitorId,
        pageType,
        changeType,
        description,
        significance: diff.significance,
        before: before.substring(0, 5000),
        after: after.substring(0, 5000),
      },
    });

    return change;
  }

  private classifyChange(pageType: string, diff: DiffResult): string {
    const text = diff.notableChanges.map((c) => c.text).join(' ').toLowerCase();

    if (pageType === 'pricing') {
      if (text.includes('$') || text.includes('price') || text.includes('plan')) {
        return 'pricing';
      }
    }

    if (text.includes('feature') || text.includes('new') || text.includes('launch')) {
      return 'feature';
    }

    if (text.includes('design') || diff.significance > 0.3) {
      return 'design';
    }

    return 'content';
  }

  private generateDescription(changeType: string, diff: DiffResult): string {
    const { addedWords, removedWords, notableChanges } = diff;

    const additions = notableChanges
      .filter((c) => c.type === 'added')
      .slice(0, 3)
      .map((c) => c.text);

    const removals = notableChanges
      .filter((c) => c.type === 'removed')
      .slice(0, 3)
      .map((c) => c.text);

    let description = `${changeType} change detected. `;

    if (additions.length > 0) {
      description += `Added: ${additions.join(', ')}. `;
    }

    if (removals.length > 0) {
      description += `Removed: ${removals.join(', ')}. `;
    }

    description += `(${addedWords} words added, ${removedWords} removed)`;

    return description;
  }
}

Analysis

Analysis Agent


// src/mastra/agents/analysis.ts
import { Agent } from '@mastra/core';
import { z } from 'zod';

const AnalysisSchema = z.object({
  summary: z.string().describe('2-3 sentence summary of the change'),
  impact: z.enum(['critical', 'high', 'medium', 'low']),
  category: z.enum([
    'pricing',
    'product',
    'marketing',
    'design',
    'team',
    'other',
  ]),
  insights: z.array(z.string()).describe('Key insights (3-5 points)'),
  implications: z.array(z.string()).describe('What this means for us'),
  recommendations: z.array(z.string()).describe('Suggested actions'),
  sentiment: z.enum(['positive', 'neutral', 'negative']),
});

export const analysisAgent = new Agent({
  name: 'Competitive Analysis Agent',
  instructions: `You are a competitive intelligence analyst.

# YOUR ROLE
Analyze competitor changes and extract strategic insights.

# WHAT TO ANALYZE
- **Impact**: How significant is this change?
- **Category**: What area does this affect?
- **Insights**: What does this tell us about their strategy?
- **Implications**: What does this mean for our business?
- **Recommendations**: What should we do in response?

# GUIDELINES
- Be objective and data-driven
- Focus on actionable insights
- Consider strategic implications
- Highlight opportunities and threats
- Suggest concrete next steps`,
  model: {
    provider: 'OPEN_AI',
    name: 'gpt-4o',
  },
});

export async function analyzeChange(
  competitor: Competitor,
  change: Change
): Promise<Analysis> {
  const prompt = `Analyze this competitive change:

**Competitor**: ${competitor.name}
**Page**: ${change.pageType}
**Change Type**: ${change.changeType}
**Significance**: ${(change.significance * 100).toFixed(1)}%

**Description**: ${change.description}

**Before**:
${change.before}

**After**:
${change.after}

Provide strategic analysis.`;

  const result = await analysisAgent.generate(prompt, {
    output: AnalysisSchema,
  });

  // Save analysis
  await db.change.update({
    where: { id: change.id },
    data: { analyzed: true },
  });

  return result.object;
}

Trend Identification


// lib/trend-identifier.ts
export class TrendIdentifier {
  async identifyTrends(
    competitorId: string,
    timeframe: 'week' | 'month' | 'quarter'
  ): Promise<Trend[]> {
    const since = this.getTimeframeStart(timeframe);

    const changes = await db.change.findMany({
      where: {
        competitorId,
        detectedAt: { gte: since },
      },
      orderBy: { detectedAt: 'desc' },
    });

    const trends: Trend[] = [];

    // Group by category
    const byCategory = this.groupBy(changes, 'changeType');

    for (const [category, categoryChanges] of Object.entries(byCategory)) {
      if (categoryChanges.length >= 3) {
        trends.push({
          type: 'frequency',
          category,
          description: `High activity in ${category}`,
          occurrences: categoryChanges.length,
          significance: 'medium',
        });
      }
    }

    // Detect pricing trends
    const pricingChanges = changes.filter((c) => c.changeType === 'pricing');
    if (pricingChanges.length > 0) {
      trends.push({
        type: 'pricing',
        category: 'pricing',
        description: 'Pricing strategy shift detected',
        occurrences: pricingChanges.length,
        significance: 'high',
      });
    }

    return trends;
  }

  private groupBy<T>(array: T[], key: keyof T): Record<string, T[]> {
    return array.reduce((acc, item) => {
      const group = String(item[key]);
      if (!acc[group]) acc[group] = [];
      acc[group].push(item);
      return acc;
    }, {} as Record<string, T[]>);
  }
}

Reporting

Weekly Report Generator


// lib/report-generator.ts
import { analysisAgent } from '@/mastra/agents/analysis';

export class ReportGenerator {
  async generateWeeklyReport(): Promise<Report> {
    const weekStart = this.getWeekStart();
    const weekEnd = new Date();

    // Gather all changes from the week
    const changes = await db.change.findMany({
      where: {
        detectedAt: {
          gte: weekStart,
          lte: weekEnd,
        },
      },
      include: { competitor: true },
    });

    // Analyze each change
    const analyses = await Promise.all(
      changes.map((change) => analyzeChange(change.competitor, change))
    );

    // Group by competitor
    const byCompetitor = this.groupByCompetitor(changes, analyses);

    // Identify trends
    const trendIdentifier = new TrendIdentifier();
    const trends = await Promise.all(
      competitors.map((c) => trendIdentifier.identifyTrends(c.id, 'week'))
    );

    // Generate executive summary
    const summary = await this.generateExecutiveSummary(
      changes,
      analyses,
      trends.flat()
    );

    const report = {
      weekStart,
      weekEnd,
      summary,
      competitors: byCompetitor,
      trends: trends.flat(),
      totalChanges: changes.length,
      criticalChanges: analyses.filter((a) => a.impact === 'critical').length,
    };

    // Save report
    await db.report.create({
      data: {
        weekStart,
        weekEnd,
        content: report as any,
      },
    });

    return report;
  }

  private async generateExecutiveSummary(
    changes: Change[],
    analyses: Analysis[],
    trends: Trend[]
  ): Promise<string> {
    const prompt = `Generate an executive summary for this week's competitive intelligence:

**Total Changes**: ${changes.length}
**Critical Changes**: ${analyses.filter((a) => a.impact === 'critical').length}
**High Impact**: ${analyses.filter((a) => a.impact === 'high').length}

**Key Changes**:
${analyses.slice(0, 10).map((a) => `- ${a.summary}`).join('\n')}

**Trends**:
${trends.map((t) => `- ${t.description}`).join('\n')}

Provide a 2-3 paragraph executive summary highlighting the most important developments and strategic implications.`;

    const result = await analysisAgent.generate(prompt);

    return result.text;
  }
}

Slack Distribution


// lib/distribute-report.ts
import { WebClient } from '@slack/web-api';

const slack = new WebClient(process.env.SLACK_BOT_TOKEN);

export async function distributeWeeklyReport(report: Report): Promise<void> {
  const blocks = [
    {
      type: 'header',
      text: {
        type: 'plain_text',
        text: `📊 Weekly Competitive Intelligence Report`,
      },
    },
    {
      type: 'context',
      elements: [{
        type: 'mrkdwn',
        text: `${report.weekStart.toLocaleDateString()} - ${report.weekEnd.toLocaleDateString()}`,
      }],
    },
    { type: 'divider' },
    {
      type: 'section',
      text: {
        type: 'mrkdwn',
        text: `*Executive Summary*\n${report.summary}`,
      },
    },
    {
      type: 'section',
      fields: [
        {
          type: 'mrkdwn',
          text: `*Total Changes*\n${report.totalChanges}`,
        },
        {
          type: 'mrkdwn',
          text: `*Critical*\n${report.criticalChanges}`,
        },
      ],
    },
    { type: 'divider' },
    {
      type: 'section',
      text: {
        type: 'mrkdwn',
        text: `*Top Trends*\n${report.trends
          .slice(0, 5)
          .map((t) => `• ${t.description}`)
          .join('\n')}`,
      },
    },
  ];

  // Add competitor sections
  for (const [competitorId, data] of Object.entries(report.competitors)) {
    blocks.push({
      type: 'section',
      text: {
        type: 'mrkdwn',
        text: `*${data.name}*\n${data.changes.length} changes detected`,
      },
    });
  }

  await slack.chat.postMessage({
    channel: process.env.SLACK_CHANNEL_ID!,
    text: 'Weekly Competitive Intelligence Report',
    blocks: blocks as any,
  });
}

Key Takeaways

Scheduled analysis - Run recurring analysis on fixed schedule
Multi-source collection - Web scraping, screenshots, APIs
Change detection - Diff calculation, significance scoring
AI analysis - Impact assessment, trend identification
Automated reporting - Weekly summaries, Slack distribution

Pattern 4 complete! All four core patterns now covered.

Get chapter updates & code samples

We’ll email diagrams, code snippets, and additions.

Pattern 4: Scheduled Analysis