Why Don’t More WordPress Agencies and Plugin Developers Use E2E Testing?

In the relatively fast world of WordPress development, delivering reliable WordPress products is quite crucial for maintaining customer satisfaction and trust. This is especially true for WordPress plugin developers who need to ensure their products are stable and functional and can be used across a variety of site setups and configurations with other third party plugins and themes. Automated end-to-end (E2E) testing tools like CheckView, Ghost Inspector, BugBug and Testim offer robust solutions, yet their adoption among WordPress developers remains relatively low compared to unit testing. It also remains quite low compared to the use of end to end testing in other software communities. Let’s explore the reasons behind this trend and why E2E testing is an important part of WP development as the community continues to evolve, regardless of the current trend.

Complexity and Learning Curve

1. Technical Expertise: E2E testing tools like Selenium and Cypress often require a higher level of technical expertise and familiarity with testing frameworks, which can be daunting for developers who are more accustomed to simpler unit tests. Saas-based automated testing tools do cut down on the expertise needed, but still require some knowledge of software quality assurance and testing which isn’t something traditional considered in WordPress development.
2. Setup and Maintenance: Setting up and maintaining end-to-end tests can be complex and time-consuming, depending on the functionality. This involves not just writing the tests (whether by-hand with open source tools or a cloud-based automated testing solution), but also setting up test environments, handling test data, and ensuring tests run reliably across different browsers and devices and are truly effective, takes additional configuration.

Resource Constraints

1. Time and Cost: Many agencies and plugin developers operate under tight deadlines and budget constraints. E2E testing can be seen as an additional overhead that consumes valuable time and resources, especially if they do not have dedicated QA teams. This makes automated testing a cost-driver, not a revenue center at many small and medium WordPress agencies and plugin shops. But these companies must consider the cost when something fails, especially on eCommerce sites running WooCommerce.
2. Resource Allocation: Smaller agencies and independent developers might not have the resources to invest in extensive testing infrastructure or the expertise to maintain it effectively, especially early on with the bootstrapped mentality that often comes with early WordPress plugin development.

Perceived Value and ROI

1. Immediate vs. Long-Term Benefits: The benefits of automated testing and even smoke tests, are often long-term, focusing on reducing bugs and improving product quality over time. In contrast, unit testing can provide more immediate feedback and is easier to integrate into the development process right off the bat.
2. Client Expectations: Agency clients especially might not always understand or value the importance of comprehensive testing, leading agencies to prioritize features and visible improvements that drive revenue or sales over robust testing that will ultimately provide a client with a more stable digital platform.

Testing Philosophy

1. Testing Culture: The culture within the WP community has traditionally leaned more towards unit and integration testing rather than automated end-to-end testing. This makes sense based on the small startup environment of plugin developers and agencies where such testing was outside of their available resources and budget. That is beginning to change as the industry matures though and cloud-based tools have made E2E much easier. Still, E2E testing is often seen as a more advanced technique that may not align with the simpler testing practices common in many WordPress projects. In fact, E2E testing and modern devops pipelines just weren’t a major thing when WordPress as a project began in the early 2000s.
2. Focus on Speed: With the rapid development cycles common in WordPress projects, the focus is often on speed and agility. That is one of WordPress’ major strengths which doesn’t always align with setting up proper tests. E2E tests, which can be slower and more resource-intensive to run, might be seen as a hindrance to many agencies and plugin companies.

Tools and Ecosystem

1. Tool Availability and Compatibility: While there are many tools available for end to end testing, many of them don’t integrate easily with WordPress and don’t fit within WordPress devop pipelines. Developers might face challenges in finding the right tools that fit well within their existing workflows without having to rebuild their devops and development processes. It is worth noting CheckView is one of the only tools built specifically for WordPress, making the transitioning of adding E2E testing easier for existing WP development teams.
2. Reliability of Tools: E2E testing tools can sometimes be less reliable, leading to flaky tests that fail intermittently. This can undermine confidence in the testing process and lead to frustration. Of course, many tools have ways to mitigate such as CheckView and BugBug’s test auto-healing features.

Perceived Redundancy

1. Overlapping Tests: Developers might feel that unit and integration tests already cover the essential aspects of their code, seeing E2E tests as redundant or unnecessary. This is unfortunate since it ultimately does a disservice to agency clients or agencies / freelancers using specific plugins or themes where they are forced to fend for themselves on testing their websites once other common plugins / themes have been introduced. This could be often avoided if E2E was baked into the process by the initial development team.
2. Focus on Critical Paths: E2E testing is often used to test critical user paths and workflows. Developers might prioritize other forms of testing for less critical components, relying on manual testing for the most crucial parts. This works up to an extent, but is not sustainable as WordPress client sites and applications get more complex as manual testing cannot scale well.

Why End-to-End Testing Should Be Used in WordPress

Despite the challenges noted, end-to-end WordPress testing offers significant benefits that make it a valuable addition to the development workflow.

1. Comprehensive Coverage: E2E testing covers the entire WordPress website workflow, ensuring that all themes, plugins and code work together as expected. This comprehensive approach can catch issues that unit and integration tests can miss. This is a huge win as anybody who has worked on a WordPress site knows that the open-source and modular-style of WordPress is both its greatest strength and largest flaw: conflicting third party code can wreak havoc on a website’s reliability and E2E testing can catch this before users do.
2. Improved User Experience: Speaking of users, by simulating real user interactions with headless browsers, automated tests and E2E smoke tests can help ensure that the user experience remains smooth and error-free, which is crucial for maintaining trust with users and providing the best UX possible. Not to mention, a reliable UX ensures higher revenue in terms of website orders such as on WooCommerce or with leads using form plugins such as Gravity Forms, WS Form, etc.
3. Early Bug Detection: End to end testing can catch bugs early in the development process, reducing the likelihood of issues making it to production or having to re-do a bunch of code due to an early logic flaw. This proactive approach can often save time and resources in the long run, not to mention contribute to the bottom line in terms of orders or leads.
4. Confidence in Releases: Automated E2E tests provide a safety net that allows developers to release updates with confidence, knowing that critical user flows have been thoroughly tested. This can lead to more frequent and reliable releases that can long-term bring more features and quality of life improvements to end users and clients.
5. Competitive Advantage: In a crowded market, delivering a high-quality, bug-free product can set your WP plugin apart from the competition. Automated testing such as E2E tests helps ensure that your WordPress product, such as a plugin or theme, meets the highest standards of quality and can lead to larger agencies and enterprises using the tool since they know proper quality testing is in place.

Conclusion

While automated E2E testing with tools like CheckView, Lambda Test, BrowserStack, and Ghost Inspector can provide significant benefits in terms of ensuring product quality and reducing bugs, various factors such as complexity, resource constraints, perceived value, and the existing testing culture within the WordPress community contribute to its relatively lower adoption among WordPress plugin developers and WordPress agency devs. However, as the WordPress ecosystem continues to mature and the demand for higher-quality products grows, more WordPress developers should start to see the value in incorporating comprehensive automated testing into their workflows. Embracing end-to-end testing can lead to improved user experiences, early bug detection, and greater confidence in WordPress software releases, ultimately providing a competitive edge in the crowed WordPress marketplace.