Qubyte Codes

Dispatching Webmentions with a Netlify build plugin

Published

Last updated

Until recently I relied on a glitch to act as a webhook dispatcher. When a post was published, an on-success hook would be sent from Netlify (which hosts this site) to the glitch to trigger it. The glitch then checked a sitemap against a cached sitemap to determine new URLs and dispatch webmentions for them. While this works, it had some issues:

  • The glitch is a server so that it can receive webhook requests.
  • The server hashes the body content and checks a webook signature header.
  • Glitch can take a while to wake up an app.
  • Difficult to know when there was a problem. You have to remember to visit the glitch and hope the logs have been kept.
  • Code is kept in a different place to the static site generator.
  • Difficult to add automated tests and linting.

Netlify now has a build plugin feature, which allows custom code to be run at particular times in the build lifecycle. In particular, the onSuccess hook means that checks for mentions (which are subject to requests to linked-to sites and have lots of potential for flakiness) can happen after the site is deployed, as it did for the webhook. This avoids delaying or failing the build.

I wrote build plugin inside the repo for this site which makes a request for the atom feed of my site before the build is run, and compares it to the generated atom feed after. A collection of new URLs is derived from the before and after atom feeds, and new mentions determined and dispatched. The plugin resolves all the above issues:

  • No server code needed.
  • No webhook used so no signature checking needed.
  • No wake-up delay.
  • Logs are kept with other deploy logs in Netlify.
  • Code can be kept in the same repo as the static site generator and content.
  • It can use the same testing and linting setup as the rest of the repo.
  • Fewer hard coded things like URLs. Netlify provides things as context and through the environment.

The real thing is a bit more complex, but the gist of it is:

const getOldFeedUrls = require('./get-old-feed-urls');
const readNewFeedUrls = require('./read-new-feed-urls');
const dispatchMentionsForUrl = require('./dispatch-mentions');

let oldFeed;

// Get a copy of the atom feed before the build is run.
exports.onPreBuild = async () => {
  // Skip unless this is a production build.
  if (process.env.CONTEXT !== 'production') {
    return;
  }

  // A set derived from an HTTPS request for the atom feed
  // for this site.
  oldFeed = await getOldFeedUrls();
}

// Compare the newly generated atom feed with the earlier
// feed to get a list of new URLs, check each for outbound
// links, and dispatch mentions.
exports.onSuccess = async () => {
  // Skip unless this is a production build.
  if (process.env.CONTEXT !== 'production') {
    return;
  }

  // A Set of URLs from a feed loaded from the build files.
  const newFeed = await readNewFeedUrls();

  // Dispatch mentions for each new URL in sequence.
  for (const url of newUrls) {
    if (!oldUrls.has(url)) {
      try {
        await dispatchWebmentionsForUrl(url);
      } catch (e) {
        console.error(e);
      }
    }
  }
}

For the real thing I enourage you to check out the source. There remain some limitations. Only new articles are checked for mentions. When an article is updated it'll be ignored (new links may be added or old ones removed). One way to resolve this would be to collect new and updated URLs when comparing the atom feeds. A database to associate a URL with dispatched mentions could be used to know when links have been added or removed. That database could be stored as a file in the netlify cache. From the documentation it seems like this cache will store a file indefinitely.

This is just the beginning! I plan to port other capabilities over to Netlify build plugins in time.