Automated Competitor Pricing Intelligence and Scraping Pipeline with Python, Playwright, and n8n

In fast-moving e-commerce markets, SaaS industries, and digital marketplaces, product pricing is intensely dynamic. Competitors frequently test promotional flash sales, adjust shipping tiers, and offer localized discounts. Companies that monitor competitor pricing manually or once a quarter operate at an extreme commercial disadvantage, losing margin and market share without realizing why sales slowed down.

In 2026, leading growth and revenue operations teams build autonomous competitor pricing intelligence pipelines using Python, Playwright, and self-hosted n8n. By deploying stealth headless browser scrapers that bypass modern bot detection systems, storing historical price changes in PostgreSQL, and orchestrating instant Slack price-drop alerts, businesses maintain real-time market pricing advantage. This technical guide outlines the full production setup.

Why Playwright Outperforms Legacy Scrapers (Selenium / BeautifulSoup)

Modern websites render products dynamically using client-side React/Vue frameworks and protect pricing behind anti-scraping firewalls (Cloudflare Turnstile, DataDome). BeautifulSoup only parses static HTML, and Selenium is slow and easily fingerprinted:

Step 1: The Stealth Playwright Pricing Scraper (`price_scraper.py`)

Configure Playwright with anti-detection plugins and randomized user-agent rotation:

# scraper/price_scraper.py
import asyncio
from playwright.async_api import async_playwright
import json
import sys

async def scrape_competitor_product(target_url: str):
    async with async_playwright() as p:
        # Launch Chromium with anti-bot arguments
        browser = await p.chromium.launch(
            headless=True,
            args=[
                "--disable-blink-features=AutomationControlled",
                "--no-sandbox"
            ]
        )
        
        context = await browser.new_context(
            user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36",
            viewport={"width": 1920, "height": 1080}
        )

        page = await context.new_page()

        try:
            await page.goto(target_url, wait_until="networkidle", timeout=20000)
            
            # Extract structured product information
            title = await page.locator("h1.product-title").text_content()
            price_raw = await page.locator("span.current-price").text_content()
            availability = await page.locator("div.stock-status").text_content()

            # Clean and sanitize numerical price
            clean_price = float(price_raw.replace("$", "").replace(",", "").strip())

            result = {
                "url": target_url,
                "title": title.strip() if title else "Unknown",
                "price": clean_price,
                "in_stock": "in stock" in availability.lower() if availability else True,
                "timestamp": asyncio.get_event_loop().time()
            }
            print(json.dumps(result))

        except Exception as e:
            print(json.dumps({"error": str(e), "url": target_url}), file=sys.stderr)
        finally:
            await browser.close()

if __name__ == "__main__":
    url = sys.argv[1] if len(sys.argv) > 1 else "https://example-competitor.com/product/123"
    asyncio.run(scrape_competitor_product(url))

Step 2: Designing the Historical Price Tracking Schema

Store price changes over time in PostgreSQL to detect promotional discount patterns:

-- Competitor pricing surveillance table
CREATE TABLE competitor_price_history (
    id BIGSERIAL PRIMARY KEY,
    competitor_name VARCHAR(100) NOT NULL,
    product_sku VARCHAR(64) NOT NULL,
    scraped_price DECIMAL(10, 2) NOT NULL,
    in_stock BOOLEAN DEFAULT TRUE,
    recorded_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP
);

CREATE INDEX idx_comp_pricing ON competitor_price_history (product_sku, recorded_at DESC);

Step 3: Orchestrating Autonomous Alerts in n8n

The n8n workflow executes on a twice-daily schedule:

1. Cron Trigger Node: Runs at 09:00 AM and 03:00 PM.
2. Execute Command Node: Runs the Python Playwright scraper against target URLs.
3. PostgreSQL Query Node: Retrieves the previous recorded price and calculates percentage price delta: ((previous_price - current_price) / previous_price) * 100.
4. IF Condition Node: Evaluates if price_drop_percent >= 10%.
5. Slack Alert Node: Posts an immediate competitive intelligence alert:

{
  "blocks": [
    {
      "type": "header",
      "text": { "type": "plain_text", "text": "📉 Competitor Price Drop Alert!" }
    },
    {
      "type": "section",
      "fields": [
        { "type": "mrkdwn", "text": "*Product:* " + $json.title },
        { "type": "mrkdwn", "text": "*Competitor:* Enterprise Rival Ltd" },
        { "type": "mrkdwn", "text": "*Previous Price:* $" + $json.previous_price },
        { "type": "mrkdwn", "text": "*New Price:* *$" + $json.price + "* (-" + $json.price_drop_percent + "%)" }
      ]
    },
    {
      "type": "actions",
      "elements": [
        {
          "type": "button",
          "text": { "type": "plain_text", "text": "🔍 Inspect Competitor Page" },
          "url": $json.url,
          "style": "primary"
        }
      ]
    }
  ]
}

Conclusion

Automated market surveillance eliminates pricing blind spots. By orchestrating stealth Playwright scrapers with self-hosted n8n workflows and historical PostgreSQL analytics, digital businesses detect competitor promotions instantly, optimize profit margins, and defend their market position with automated intelligence.