Understanding Test Statuses

CheckView categorizes test outcomes into two primary statuses (Passed and Failed) to ensure clarity and reliability when analyzing results. Below is a breakdown of each status, along with key insights on how to interpret them and optimize your testing process.

Test Statuses

  1. Passed
    • Definition: The test successfully completed all required steps without errors.
    • What It Means: Each test step executed as expected and all assertions (e.g., element visibility, text validation) were met.
    • Pro Tip: A passed test indicates your workflow is functioning as intended, but periodic retesting is recommended to catch any potential issues caused by updates or changes to your site.
  2. Failed
    • Definition: The test encountered an error because one or more steps did not complete as expected.
    • What It Means: A failure typically halts the test immediately at the step where the issue occurred. The failed step is highlighted in the test results for review. In some cases, the root cause may not be directly tied to the failed step but could stem from a previous action or environmental issue so it is key to review the video recording: CheckView provides a video of the test execution which is critical for identifying the exact point of failure and understanding the context. Further, analyze the logs which may have additional details about what the bot encountered during the test flow.
  3. Pending
    • Definition: The test has been created but has not yet been executed. It is typically waiting for its scheduled run time or is in the process of running, but a final status has not been assigned.
    • What It Means: A pending test indicates that CheckView has generated the test flow, but it has not yet started or completed. Pending tests are common for scheduled test runs or tests that are queued due to system workload.
    • Pro Tip: If a test remains in a pending state for an extended period, verify that your test schedule is correctly configured and that there are no issues with system resources. Contact our support if it stays in pending past the scheduled date and time.
  4. Aborted
    • Definition: The test was intentionally stopped before completing all steps. This will result from a user clicking the ‘Abort’ button while the test was running.
    • What It Means: The test did not reach a pass or fail conclusion based on its defined steps. No further steps were executed after the abort event besides packaging up the steps ran and the corresponding video.
    • Pro Tip: Use the Abort button if you realize the test is misconfigured, targeting the wrong environment, or is taking longer than expected due to unexpected page behavior. This helps you avoid skewing your test history with unnecessary failures and speeds up iteration during test building.

Why CheckView Doesn’t Use Partial Passes

CheckView is designed to provide consistent and reliable statuses. To maintain this standard, a test is all or nothing. A test is only marked as “Passed” if all required steps are completed successfully. Otherwise, it is marked as “Failed.” Optional steps, such as dismissing a modal or interacting with non-critical elements, are not tested and do not affect the overall test status. This ensures optional actions do not introduce unnecessary complexity or false failures.

Pro Tip: If you use optional steps, be mindful that their execution is not guaranteed. For instance, if a modal doesn’t appear in a given test scenario, the associated step will be skipped rather than marked as failed.

Additional Tips for Understanding Test Statuses

  1. Interpreting Failures: A failed test doesn’t always indicate a major issue. For example, failures can occur due to:
    • Temporary server slowdowns or timeouts.
    • Missing elements due to page load delays.
    • Incorrect or outdated selectors in the test flow.
    • Regular maintenance of selectors and test flows can minimize false failures.
  2. Using Video and Logs Effectively: Always watch the video recording to understand the bot’s behavior leading up to the failure. Also review the video the first time a test passes to ensure it is working as expected. Use the test logs to cross-reference actions and identify specific points of failure.
  3. Testing Changes in Real Time: If a test fails, you can edit the test steps, selectors, or timing and rerun the test immediately to verify fixes.
  4. Consistent Testing Environment: Ensure the testing environment remains stable. For example:
  5. Plan for Resilience: Build redundancy into your test flows by using multiple selectors for critical steps (e.g., fallback CSS or XPath selectors). Use Pause steps or assertions to handle delays in dynamic elements.

For any questions or advanced troubleshooting tips, contact support for personalized assistance.

Docs