Contact Me

Scraping SPAs: Leveraging Selenium/Playwright for Dynamic Content and Network Request Interception

By Alex Nechyporenko Published on Category: Web Scraping
Scraping SPAs: Leveraging Selenium/Playwright for Dynamic Content and Network Request Interception

As a seasoned web scraping expert, you've likely encountered the wall that traditional HTTP request-based scrapers hit when facing modern web applications. These are the Single Page Applications (SPAs), built with frameworks like React, Angular, and Vue.js, which dynamically load content using JavaScript after the initial page load. Standard requests and BeautifulSoup can only see the initial HTML, often leaving you with an empty <body> tag where the data should be.

This is where browser automation tools like Selenium and Playwright become indispensable (Selenium, 2025; Playwright, 2025). They launch a real browser (headless or not) and allow you to interact with the page just like a human user, rendering all the JavaScript-driven content. But we won't stop there. We'll also explore a powerful advanced technique: intercepting network requests to bypass browser rendering altogether and directly tap into the data APIs.

The Challenge of Scraping SPAs

Traditional scrapers fetch the HTML content of a URL directly. For static websites, this is sufficient. However, SPAs load most of their content asynchronously. When you fetch the initial HTML of an SPA, you'll often see something like this:

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <title>My SPA</title>
</head>
<body>
    <div id="root"></div>
    <script src="/bundle.js"></script>
</body>
</html>

The actual data (e.g., product listings, articles, user comments) is fetched via AJAX requests (XHR/Fetch) and injected into the DOM by JavaScript after the browser has rendered the initial empty HTML (MDN Web Docs: Fetch API, 2025; MDN Web Docs: XMLHttpRequest, 2025). This is why a simple requests.get() will often return an almost empty <body> day.

Solution 1: Browser Automation with Selenium and Playwright

Both Selenium and Playwright provide a robust way to interact with a full browser, mimicking user behavior. This allows them to "see" and interact with dynamically loaded content, overcoming the limitations of traditional HTTP request-based scrapers.

Key Capabilities:

  • Page Loading and Waiting: These tools can wait for elements to appear, for network requests to finish, or for specific conditions to be met before attempting to interact with the page. This is crucial for SPAs where content loads asynchronously.
  • Element Interaction: You can simulate almost any user action: clicking buttons, filling forms, scrolling, hovering over elements, and even dragging and dropping.
  • JavaScript Execution: Run custom JavaScript code directly on the page to manipulate elements or extract data that's hard to get otherwise.
  • Screenshotting: Capture visual proof of the page's state at any point, which is great for debugging.
  • Headless Mode: Run browsers without a visible user interface. This is perfect for server-side scraping environments, as it consumes fewer resources and is generally faster.

Selenium Example: Scraping Dynamic Content

Let's use a simple example: scraping quotes from a page that loads its content dynamically using JavaScript. (Quotes to Scrape, 2025).

Prerequisites:

  • Install Selenium: pip install selenium
  • Download and place a WebDriver executable (e.g., chromedriver for Chrome) in your system's PATH or specify its path in your script.

selenium_scraper.py:

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

# For simplicity, using a common setup for headless Chrome
# In a real scenario, you might specify the exact path to your chromedriver:
# service = Service(executable_path='/path/to/chromedriver')
# driver = webdriver.Chrome(service=service)

options = webdriver.ChromeOptions()
options.add_argument('--headless') # Run Chrome in headless mode (no GUI)
options.add_argument('--no-sandbox') # Required for some environments (e.g., Docker)
options.add_argument('--disable-dev-shm-usage') # Overcome limited resource problems

driver = webdriver.Chrome(options=options)

try:
    url = "[https://quotes.toscrape.com/js/](https://quotes.toscrape.com/js/)" # Example site that loads content dynamically
    print(f"Loading page: {url}")
    driver.get(url)

    # Use explicit waits for robustness instead of time.sleep()
    # Wait until an element with class 'quote' is present on the page
    print("Waiting for quotes to load...")
    WebDriverWait(driver, 10).until(
        EC.presence_of_element_located((By.CLASS_NAME, "quote"))
    )
    print("Quotes loaded.")

    # Now that content is loaded, we can extract it
    quotes = driver.find_elements(By.CLASS_NAME, "quote")
    for i, quote in enumerate(quotes[:3]): # Just scrape the first 3 quotes for brevity
        text = quote.find_element(By.CLASS_NAME, "text").text
        author = quote.find_element(By.CLASS_NAME, "author").text
        print(f"Quote {i+1}:")
        print(f"  Text: {text}")
        print(f"  Author: {author}\n")

except Exception as e:
    print(f"An error occurred: {e}")
finally:
    print("Closing browser.")
    driver.quit() # Always close the browser to free up resources

Playwright Example: Similar Dynamic Scraping

Playwright is a newer, often faster, and more robust alternative to Selenium, supporting Chrome, Firefox, and WebKit (Safari's engine).

Prerequisites:

  • Install Playwright: pip install playwright
  • Download browser binaries: playwright install (Run this command in your terminal)

playwright_scraper.py: 

from playwright.sync_api import sync_playwright

def scrape_dynamic_content(url: str):
    with sync_playwright() as p:
        browser = p.chromium.launch(headless=True) # Or .firefox.launch(), .webkit.launch()
        page = browser.new_page()

        print(f"Loading page: {url}")
        page.goto(url)

        # Wait for the specific element to appear and be visible
        print("Waiting for quotes to load...")
        page.wait_for_selector(".quote", state="visible", timeout=10000) # Timeout in milliseconds
        print("Quotes loaded.")

        quotes = page.query_selector_all(".quote")
        for i, quote in enumerate(quotes[:3]): # Scrape the first 3 quotes
            text_element = quote.query_selector(".text")
            author_element = quote.query_selector(".author")

            text = text_element.inner_text() if text_element else "N/A"
            author = author_element.inner_text() if author_element else "N/A"

            print(f"Quote {i+1}:")
            print(f"  Text: {text}")
            print(f"  Author: {author}\n")

        browser.close()
        print("Browser closed.")

if __name__ == "__main__":
    target_url = "[https://quotes.toscrape.com/js/](https://quotes.toscrape.com/js/)"
    scrape_dynamic_content(target_url)

Both Selenium and Playwright effectively render the page and wait for content, allowing you to extract data that would be invisible to requests.

Solution 2: Intercepting Network Requests (The Advanced Technique)

Often, the data displayed in an SPA is fetched from a backend API using XHR (XMLHttpRequest) or Fetch API requests. Instead of waiting for the browser to render the data, you can intercept these network requests and extract the data directly from the API responses, usually in JSON format. This is often faster and more efficient than full browser rendering because you avoid the overhead of rendering and parsing HTML.

When to Use Network Interception:

  • When you need large volumes of data.
  • When the data is clearly coming from a well-structured API (e.g., JSON, XML).
  • To bypass complex frontend rendering or JavaScript logic that makes HTML parsing difficult.
  • To minimize resource usage compared to full browser rendering.

How to Find the API Calls:

  1. Open your browser's Developer Tools (F12 or Ctrl+Shift+I).
  2. Go to the "Network" tab.
  3. Refresh the page.
  4. Filter by "XHR" or "Fetch/XHR" to see the AJAX requests.
  5. Look for requests that carry the data you need. Examine their headers, payloads, and responses. You might need to click around the site to trigger data loading.

Playwright Example: Intercepting a JSON API Call

Let's assume quotes.toscrape.com/js/ eventually makes an internal API call to fetch its quotes. (Note: The actual quotes.toscrape.com/js/ does not use a direct JSON API for its quotes; it renders directly. For demonstration purposes, I'll show how the concept of interception works with a placeholder API. In a real scenario, you'd identify the actual API endpoint.)

from playwright.sync_api import sync_playwright

def intercept_network_requests(url: str, api_substring: str, block_images: bool = False):
    data_found = []
    with sync_playwright() as p:
        browser = p.chromium.launch(headless=True)
        page = browser.new_page()

        # Enable request interception
        # Define a handler for requests
        def handle_request(route):
            request = route.request

            # 1. Blocking unnecessary requests (e.g., images, CSS, fonts)
            if block_images and request.resource_type == "image":
                print(f"Blocking image: {request.url}")
                route.abort()
                return

            # 2. Modifying requests (e.g., adding authentication headers)
            # This is a hypothetical example where an API might need a token
            if "/api/protected_data" in request.url:
                headers = request.headers.copy()
                # In a real scenario, YOUR_AUTH_TOKEN would be extracted from a previous login
                headers["Authorization"] = "Bearer YOUR_AUTH_TOKEN_HERE" 
                print(f"Modifying request headers for {request.url}")
                route.continue_(headers=headers)
                return

            # Allow all other requests to continue by default
            route.continue_()

        page.route("**/*", handle_request) # Apply the request handler

        # Intercept and process responses
        def handle_response(response):
            if api_substring in response.url and response.status == 200:
                try:
                    # Check if response is JSON (often API responses are)
                    if 'application/json' in response.headers.get('content-type', ''):
                        json_data = response.json()
                        print(f"Intercepted JSON data from {response.url}:")
                        # Process your JSON data here
                        # For demonstration, let's assume it has a 'quotes' key
                        if 'quotes' in json_data:
                            print(f"  Found {len(json_data['quotes'])} quotes.")
                            data_found.extend(json_data['quotes'])
                        else:
                            print(f"  JSON structure: {json_data.keys()}")

                except Exception as e:
                    print(f"Could not parse JSON or process response from {response.url}: {e}")
            elif api_substring in response.url:
                print(f"Intercepted non-200 response from {response.url} (Status: {response.status})")

        page.on("response", handle_response)

        print(f"Navigating to {url} to trigger network requests...")
        page.goto(url, wait_until="networkidle") # Wait until no network requests for 500ms

        # You can add more interactions here if the API call is triggered by a click etc.
        # Example: page.click("button#load-more")

        browser.close()
        print("Browser closed.")

    if data_found:
        print("\n--- Collected Data (Example) ---")
        for item in data_found[:2]: # Print first 2 collected items
            print(item)
    else:
        print("\nNo relevant data collected from intercepted requests.")

if __name__ == "__main__":
    # This is a hypothetical URL and API endpoint for demonstration.
    # You would replace these with the actual values found in DevTools.
    target_url = "[http://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html](http://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html)"
    api_endpoint_substring = "api" # Look for responses from any URL containing 'api'

    print("Note: This example uses a hypothetical API endpoint for interception.")
    print("You need to find the actual API calls using browser developer tools for real sites.")
    print("-" * 60)

    # To show a working interception, you'd need a site with a clear JSON API.
    # The [quotes.toscrape.com/js/](https://quotes.toscrape.com/js/) example does not make a direct JSON API call for quotes,
    # so this specific example will likely not find data from that site.
    # It serves to illustrate the INTERCEPTION MECHANISM.

    intercept_network_requests(
        target_url,
        api_endpoint_substring,
        block_images=True # Demonstrate blocking images
    )

    # Example of a URL that actually provides JSON data (for testing the interception logic)
    print("\n--- Testing with a known JSON API (jsonplaceholder.typicode.com) ---")
    intercept_network_requests(
        "[https://jsonplaceholder.typicode.com/posts/1](https://jsonplaceholder.typicode.com/posts/1)", # Direct API call
        "posts", # Look for responses from URLs containing 'posts'
        block_images=False # No images to block here
    )

Important Notes on Network Interception:

  • Finding the Right Request: This is the most critical step. You need to identify the exact API endpoint that provides the data you're looking for. It might involve pagination, authentication tokens, or specific query parameters.
  • Request Types: Look for XHR or Fetch requests. They usually carry data in JSON or XML format.
  • Headers: Pay attention to headers, especially Authorization headers or Content-Type headers, as they might be necessary to replicate the request directly later.
  • Payloads: If the API call is a POST request, inspect the request payload to understand what data is being sent.
  • Error Handling: API calls can fail. Implement robust error handling for different HTTP status codes.
  • Complexity: Sometimes SPAs use complex client-side logic to construct API requests. Intercepting might reveal the endpoint, but replicating the request outside of a browser can still be challenging.
  • Authentication & Sessions: For authenticated content, you'll often need to first perform a login using browser automation (Selenium/Playwright) to obtain necessary cookies or authentication tokens. These can then be used in subsequent intercepted requests or when making direct API calls. Playwright's page.context.storage_state() can save and load authentication states for re-use.

Conclusion: Choosing Your Scraping Strategy

When facing SPAs, you have a powerful toolkit:

Browser Automation (Selenium/Playwright):

  • Pros: Simulates human interaction perfectly, handles all JavaScript rendering, good for complex UIs, reliable for dynamic content.
  • Cons: Slower, more resource-intensive, higher overhead.
  • Best for: Websites with very complex JavaScript logic, forms, multi-step processes, or when you need to interact heavily with the UI.

Network Request Interception (with Playwright, or by reverse-engineering):

  • Pros: Much faster, less resource-intensive, gets data directly in structured format (often JSON), bypasses rendering overhead.
  • Cons: Requires more investigation (Developer Tools expertise), might break if API changes, difficult if requests rely on complex client-side computations (e.g., token generation).
  • Best for: High-volume data extraction from sites that clearly load data via structured APIs.

As an expert, your best strategy often involves a combination:

  1. Use browser automation (Selenium/Playwright) to initially load the page, login, or trigger specific events (like clicking a "Load More" button).
  2. Then, intercept the resulting API calls to efficiently extract large volumes of data.
  3. If direct API replication is too complex, fall back to parsing the rendered HTML with BeautifulSoup on the page.content() from Playwright/Selenium.

Mastering these techniques will elevate your web scraping capabilities, allowing you to tackle even the most challenging modern web applications with confidence and efficiency. Happy scraping!

Used Sources:

Tags: #Python #Selenium #Playwright #Dynamic Content #Automation #Network Requests
Back to Blog