Automated Testing for the Indie Web?

(This post is mostly for Indie Web geeks. Feel free to ignore)

The problem

• We spend too much time debugging why things didn’t work as expected
• Not so much Gen 1 people running their own, custom-developed software, but later-gen people who try to, say, make WordPress work with a previously unseen combination of plugins and themes

Proposed solution: systematic, preferably automated, testing

(Note: the remainder focuses on microformats only. Of course there’s much more to test, but we got to start somewhere…)

Example test case 1

• Given a new installation of WordPress, with N installed themes.
• Given the URL to the “installation succeeded” default WordPress post that comes with
  all new WordPress installations (lets call it WP_POST_URL).
• For each of the installed themes:
  • The microformat content of the page at URL WP_POST_URL, as parsed by mf2py,
    must be the same.

(This may be a bit more complicated — e.g. some theme might show only one post, while
others have references to other posts etc — but as I put it, it’s good enough to get a
discussion going)

Example test case 2

• WordPress has a new version, but no other changes.
• Verify that the microformats on my blog are still interpreted the same way after the
  upgrade.

Considerations for possible implementations

• QA is very time consuming, and unless you get paid (and often not even then), nobody
  likes to do it on an ongoing basis.
• I’m biased because I use a self-written utility called webapptest to test that web
  applications on UBOS still run after various OS/app/middleware upgrades. It could be
  adapted to our purpose here. Here’s how it would work:
  • The script instantiates a new WordPress site temporarily in a Linux container,
    with a configurable set of plugins and themes.
  • The script restores previously saved WordPress site data (database, uploaded files)
  • The script runs mf2py on a set of URLs pointing to various pages of the WordPress
    install
  • The script compares the output with a previously saved baseline.
  • If there is a discrepancy, emit an error
  • The Linux container is torn down.
• The work involved would be:
  • Create one/several WordPress sites with “interesting” content, and save their
    databases/uploaded files.
  • Identify “interesting” URLs for each
  • Run mf2py against the interesting URLs, and save the output
  • Augment webapptest utility to load WordPress backups once it has installed
    the sites temporarily in a container (it knows how to do that today), and run
    mf2py and some kind of JSON diff instead of the default webapptest scripts.
  • Create an overall script that iterates through the test cases.
  • … and then run this on a regular basis.

This would apply similarly to tools other than WordPress, of course.

Thoughts?