Understanding Test Failures

Test failures are a natural part of the testing process. In many cases, they’re doing exactly what they’re supposed to, alerting you to an issue with your WordPress website. However, not every failure points to a genuine problem with your site. Some failures may stem from changes in your site, configuration issues, errors in the test design itself, or even bugs on our end.

Why Do Tests Fail?

  1. A Real Issue with Your WordPress Site
    • What It Means: The test has identified a legitimate problem, such as:
      • A form not submitting correctly.
      • Missing or broken elements.
      • JavaScript errors breaking interactivity.
      • PHP erros breaking functionality.
      • A failed API response.
    • What to Do:
      • Review the test recording, each test step and related logs, and the HAR file to identify where the failure occurred.
      • Validate the issue by testing manually on your site.
  2. Changes to Your Website
    • What It Means: Updates to your site’s design, plugins, or functionality may have altered key elements or workflows, causing the test to fail.
    • Examples:
      • A form field’s CSS ID or class was updated.
      • A page structure change caused a selector mismatch.
      • A plugin update introduced new or modified elements.
    • What to Do:
      • Review the failed step and adjust selectors or test logic to reflect the changes.
      • Use fallback selectors in cases where structure or IDs may change frequently.
  3. Test Configuration Issues
    • What It Means:
      • The test itself may not be properly set up to handle your site’s unique behavior or environment.
    • Examples:
      • The test moves too quickly for your site to load elements, leading to timeouts.
      • The wrong type of assertion or selector is used.
      • Test steps were created for an outdated version of your workflow.
    • What to Do:
      • Adjust Step Timing in advanced settings (e.g., increase Element Timeout or add a Pause step for slower elements).
      • Update selectors to target the correct elements.
      • Revisit your test flow to ensure it reflects your site’s current setup and add / remove any test steps necessary.
  4. Blocked by Security Settings
    • What It Means: Your website or hosting provider may have security settings that block CheckView’s bot.
    • Examples:
      • Firewalls blocking CheckView’s IPs.
      • Plugins like PerfMatters or WordFence disabling the REST API.
    • What to Do:
      • Whitelist CheckView’s IP addresses.
      • Enable the WP REST API if it’s disabled within the associated plugin or firewall.
      • Work with your hosting provider to ensure bots like CheckView are not blocked.
  5. Outdated or Unsupported Plugins
    • What It Means: Your test may rely on features or workflows from plugins that have changed or are not fully supported.
    • Examples:
      • A plugin update modified the DOM structure.
      • Unsupported plugins don’t expose the necessary selectors for CheckView to interact with.
    • What to Do:
      • Use the Custom Test Flow type for unsupported plugins.
      • Stay up-to-date with plugin compatibility and updates from CheckView.

Tips for Fixing Test Failures

  1. Start with the Test Recording:
    • Watch the step-by-step playback to see exactly where and why the test failed.
    • Look for missing elements, unexpected redirects, error messages or incorrect behavior.
  2. Analyze the Test Logs:
    • Logs provide detailed information about requests, responses, and interactions.
    • Use these to identify timeouts, API failures, or misconfigurations with your developers.
  3. Use HAR Files for Debugging:
    • Download the HAR file for the test and analyze it with a tool like HAR Analyzer.
    • Look for failed network requests or blocked resources.
  4. Check Test Timing:
    • If elements are loading too slowly, adjust the Step Delay or Element Timeout in advanced settings.
    • Add a Pause Step for dynamic elements like AJAX forms or slow-loading pages.
  5. Verify Your Selectors:
    • Ensure your selectors are targeting the correct elements. Use Chrome DevTools or extensions like SelectorsHub to refine them.
    • Include fallback selectors for elements prone to changes.
  6. Stay Proactive with Updates:
    • Regularly review your tests after major site updates or plugin changes.
    • Use the Test Flow History Page to analyze trends in test performance over time.

Troubleshooting Tools

If you’re still unable to identify or resolve a test failure, CheckView offers tools and resources to help:

Test failures are a critical part of ensuring your site’s workflows are functioning correctly. By understanding the root causes, whether genuine site issues, configuration problems, or design changes, you can troubleshoot efficiently and maintain reliable test flows.

Docs