WooCommerce 3.2.2 and Testing

WooCommerce 3.2.2 is now out!

WooCommerce 3.2 was billed as “a minor release … fully backwards compatible with 3.1” and was patched only 2 days later by 3.2.1 with 43 commits (Oct 11, Oct 13) and now 3.2.2 with 81 commits on October 31.

So what we can conclude?

  • great velocity in development, testing and release
  • 3.2 release was not as fully backwards compatible as promised

The compatibility assertion was made because of the negative impact of version 3.0 (rapidly fixed by an additional 7 releases in the following 6 weeks).

So what went wrong?

The 3.2 release had:

Actually these factors lead to a false sense of security:

  • a high number of automated tests does not mean that test coverage is high: a full test pack would require the number of tests to be an order of magnitude higher.
  • a long beta/pre-release period gives the false impression that downstream projects have had time to test.   This is not the case:  when resources are limited testing effort could be optimised by ignoring the beta release and delaying testing until, for example, the second patch release.  Since resources are always limited (even more so on unfunded open source projects), this optimisation almost always applies, so the long beta period is simply inefficient.

WooCommerce does now have a new standard for components to indicate to which version they are tested against.  A dev test highlighted 11 untested components including the Stripe payment gateway from the WooCommerce team themselves, as well as extensions for Coupons, translation/localization, Invoices, Wishlists, multi-currency, gift vouchers and product bundles amongst others:  as releases continue to appear more than once per month, [probability of component being fully tested with latest release]^[number of components] yields virtually nil probability of a fully tested solution.

So what to do in this situation?

  • Automatic upgrades – necessary for security fixes but for new releases, will break some site functionality.
  • Test and upgrade – for SMEs this is too expensive to do for each release.

So the most practical options are:

  • Outsourcing  – an outsourced IT team can provide more cost-effective support than a dedicated internal team, providing a managed solution to multiple companies:  this applies whether the solution is based on Shopify (itself a fully managed service) or WooCommerce.
  • Prioritised Testing – test the most important functionality, according to the capacity available.  Automate where possible according to priorities but include a manual test of critical functionality, both failure and success paths, including live checkout test:  it only takes a small settings change or javascript or CSS incompatiblity to cause a checkout failure or to cause a validation or failure message to fail to become visible, resulting in confused and lost clients.
Advertisements

6 thoughts on “WooCommerce 3.2.2 and Testing

    1. WooCommerce 3.2.4 is released today with another 78 commits, many appearing to relate to changes introduced in 3.2.
      https://woocommerce.wordpress.com/2017/11/16/woocommerce-3-2-4-security-fix-release-notes/

      after 215 commits in 4 patch releases in 5 weeks, it is to be hoped that 3.2 is now ready for use.

      3.2 involved refactoring of code affecting shopping cart and coupons calculations which rings immediate alarm bells over the need for careful testing, and serious questions over the claim “3.2 is a minor update. All changes should be backwards compatible with 3.0 and 3.1 sites, “

      Like

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s