Understanding Test Failures

Test failures are a natural part of the testing process. In many cases, they’re doing exactly what they’re supposed to, alerting you to an issue with your WordPress website. However, not every failure points to a genuine problem with your site. Some failures may stem from changes in your site, configuration issues, errors in the test design itself, or even bugs on our end.

Why Do Tests Fail?

  1. A Real Issue with Your WordPress Site
    • What It Means: The test has identified a legitimate problem, such as:
      • A form not submitting correctly.
      • Missing or broken elements.
      • JavaScript errors breaking interactivity.
      • PHP erros breaking functionality.
      • A failed API response.
    • What to Do:
      • Review the test recording, each test step and related logs, and the Network Log (HAR) to identify where the failure occurred.
      • Validate the issue by testing manually on your site.
  2. Changes to Your Website
    • What It Means: Updates to your site’s design, plugins, or functionality may have altered key elements or workflows, causing the test to fail.
    • Examples:
      • A form field’s CSS ID or class was updated.
      • A page structure change caused a selector mismatch.
      • A plugin update introduced new or modified elements.
    • What to Do:
      • Review the failed step and adjust selectors or test logic to reflect the changes.
      • Use fallback selectors in cases where structure or IDs may change frequently.
  3. Test Configuration Issues
    • What It Means:
      • The test itself may not be properly set up to handle your site’s unique behavior or environment.
    • Examples:
      • The test moves too quickly for your site to load elements, leading to timeouts.
      • The wrong type of assertion or selector is used.
      • Test steps were created for an outdated version of your workflow.
    • What to Do:
      • Adjust Step Timing in advanced settings (e.g., increase Element Timeout or add a Pause step for slower elements).
      • Update selectors to target the correct elements.
      • Revisit your test flow to ensure it reflects your site’s current setup and add / remove any test steps necessary.
  4. Blocked by Security Settings
    • What It Means: Your website or hosting provider may have security settings that block CheckView’s bot.
    • Examples:
      • Firewalls blocking CheckView’s IPs.
      • Plugins like PerfMatters or WordFence disabling the REST API.
    • What to Do:
      • Whitelist CheckView’s IP addresses.
      • Enable the WP REST API if it’s disabled within the associated plugin or firewall.
      • Work with your hosting provider to ensure bots like CheckView are not blocked.
  5. Security Plugin or Firewall Block
    • What It Means: CheckView automatically detects when security plugins (Wordfence, Cloudflare, Sucuri, etc.) are blocking test execution. You’ll see a red error card identifying the blocker with a direct link to the resolution guide.
    • What to Do: Follow the resolution link shown in the error card to whitelist CheckView in the detected service. See Blocker Detection for details.
  6. HTTP Error Responses
    • What It Means: Tests that navigate to pages returning HTTP 403 (access denied), 404 (page not found), or 5xx (server error) will now fail immediately with a clear error code instead of timing out.
    • What to Do: Check that the target URL is correct, the page exists, and your server is not returning errors. Review the error code in the test results for the specific HTTP status encountered.
  7. Outdated or Unsupported Plugins
    • What It Means: Your test may rely on features or workflows from plugins that have changed or are not fully supported.
    • Examples:
      • A plugin update modified the DOM structure.
      • Unsupported plugins don’t expose the necessary selectors for CheckView to interact with.
    • What to Do:
      • Use the Custom Test Flow type for unsupported plugins.
      • Stay up-to-date with plugin compatibility and updates from CheckView.

AI-Powered Failure Analysis

Click ‘Get detailed analysis’ on any failed test for an AI-generated explanation of what went wrong, including observations, root cause, and suggested fixes. See AI Test Failure Summaries for more information.

Selector Information in Error Messages

Error messages now include the specific CSS selector that was being targeted, making it easier to identify which element caused the failure. This is especially helpful when debugging selector mismatches after site updates.

Tips for Fixing Test Failures

  1. Start with the Test Recording:
    • Watch the step-by-step playback to see exactly where and why the test failed.
    • Look for missing elements, unexpected redirects, error messages or incorrect behavior.
  2. Analyze the Test Logs:
    • Logs provide detailed information about requests, responses, and interactions.
    • Use these to identify timeouts, API failures, or misconfigurations with your developers.
  3. Use Network Log (HAR)s for Debugging:
    • Download the Network Log (HAR) for the test and analyze it with a tool like HAR Analyzer.
    • Look for failed network requests or blocked resources.
  4. Check Test Timing:
    • If elements are loading too slowly, adjust the Step Delay or Element Timeout in advanced settings.
    • Add a Pause Step for dynamic elements like AJAX forms or slow-loading pages.
  5. Verify Your Selectors:
    • Ensure your selectors are targeting the correct elements. Use Chrome DevTools or extensions like SelectorsHub to refine them.
    • Include fallback selectors for elements prone to changes.
  6. Stay Proactive with Updates:
    • Regularly review your tests after major site updates or plugin changes.
    • Use the Test Flow History Page to analyze trends in test performance over time.

Troubleshooting Tools

If you’re still unable to identify or resolve a test failure, CheckView offers tools and resources to help:

  • Specific Troubleshooting Errors: Click here for detailed troubleshooting steps.
  • Contact Support: Share your organization ID, Test ID, Network Log (HAR), and a description of the issue with our support team for further assistance.

Test failures are a critical part of ensuring your site’s workflows are functioning correctly. By understanding the root causes, whether genuine site issues, configuration problems, or design changes, you can troubleshoot efficiently and maintain reliable test flows.

Related Articles

Docs