Automated Testing for the Indie Web?

(This post is mostly for Indie Web geeks. Feel free to ignore)

The problem

  • We spend too much time debugging why things didn’t work as expected
  • Not so much Gen 1 people running their own, custom-developed software, but later-gen people who try to, say, make WordPress work with a previously unseen combination of plugins and themes

Proposed solution: systematic, preferably automated, testing

(Note: the remainder focuses on microformats only. Of course there’s much more to test, but we got to start somewhere…)

Example test case 1

  • Given a new installation of WordPress, with N installed themes.
  • Given the URL to the “installation succeeded” default WordPress post that comes with
    all new WordPress installations (lets call it WP_POST_URL).
  • For each of the installed themes:
    • The microformat content of the page at URL WP_POST_URL, as parsed by mf2py,
      must be the same.

(This may be a bit more complicated — e.g. some theme might show only one post, while
others have references to other posts etc — but as I put it, it’s good enough to get a
discussion going)

Example test case 2
  • WordPress has a new version, but no other changes.
  • Verify that the microformats on my blog are still interpreted the same way after the
    upgrade.

Considerations for possible implementations

  • QA is very time consuming, and unless you get paid (and often not even then), nobody
    likes to do it on an ongoing basis.
  • I’m biased because I use a self-written utility called webapptest to test that web
    applications on UBOS still run after various OS/app/middleware upgrades. It could be
    adapted to our purpose here. Here’s how it would work:

    • The script instantiates a new WordPress site temporarily in a Linux container,
      with a configurable set of plugins and themes.
    • The script restores previously saved WordPress site data (database, uploaded files)
    • The script runs mf2py on a set of URLs pointing to various pages of the WordPress
      install
    • The script compares the output with a previously saved baseline.
    • If there is a discrepancy, emit an error
    • The Linux container is torn down.
  • The work involved would be:
    • Create one/several WordPress sites with “interesting” content, and save their
      databases/uploaded files.
    • Identify “interesting” URLs for each
    • Run mf2py against the interesting URLs, and save the output
    • Augment webapptest utility to load WordPress backups once it has installed
      the sites temporarily in a container (it knows how to do that today), and run
      mf2py and some kind of JSON diff instead of the default webapptest scripts.
    • Create an overall script that iterates through the test cases.
    • … and then run this on a regular basis.

This would apply similarly to tools other than WordPress, of course.

Thoughts?

Comments are closed.