WooCommerce 3.2.2 and Testing

WooCommerce 3.2.2 is now out!

WooCommerce 3.2 was billed as “a minor release … fully backwards compatible with 3.1” and was patched only 2 days later by 3.2.1 with 43 commits (Oct 11, Oct 13) and now 3.2.2 with 81 commits on October 31.

So what we can conclude?

• great velocity in development, testing and release
• 3.2 release was not as fully backwards compatible as promised

The compatibility assertion was made because of the negative impact of version 3.0 (rapidly fixed by an additional 7 releases in the following 6 weeks).

So what went wrong?

The 3.2 release had:

• automated testing and
• long beta release period

Actually these factors lead to a false sense of security:

• a high number of automated tests does not mean that test coverage is high: a full test pack would require the number of tests to be an order of magnitude higher.
• a long beta/pre-release period gives the false impression that downstream projects have had time to test.   This is not the case:  when resources are limited testing effort could be optimised by ignoring the beta release and delaying testing until, for example, the second patch release.  Since resources are always limited (even more so on unfunded open source projects), this optimisation almost always applies, so the long beta period is simply inefficient.

WooCommerce does now have a new standard for components to indicate to which version they are tested against.  A dev test highlighted 11 untested components including the Stripe payment gateway from the WooCommerce team themselves, as well as extensions for Coupons, translation/localization, Invoices, Wishlists, multi-currency, gift vouchers and product bundles amongst others:  as releases continue to appear more than once per month, [probability of component being fully tested with latest release]^[number of components] yields virtually nil probability of a fully tested solution.

So what to do in this situation?

• Automatic upgrades – necessary for security fixes but for new releases, will break some site functionality.
• Test and upgrade – for SMEs this is too expensive to do for each release.

So the most practical options are:

• Outsourcing  – an outsourced IT team can provide more cost-effective support than a dedicated internal team, providing a managed solution to multiple companies:  this applies whether the solution is based on Shopify (itself a fully managed service) or WooCommerce.
• Prioritised Testing – test the most important functionality, according to the capacity available.  Automate where possible according to priorities but include a manual test of critical functionality, both failure and success paths, including live checkout test:  it only takes a small settings change or javascript or CSS incompatiblity to cause a checkout failure or to cause a validation or failure message to fail to become visible, resulting in confused and lost clients.