Converting Bandcamp Email Updates to an RSS Feed

I love music. I like to closely track bands and labels to learn about upcoming releases. Unfortunately, the type of music I listen to tends to stay more underground, so there isn’t any mainstream coverage of it, which just means I get to do all the work myself.

Fortunately, a majority of the bands and labels I am interested in are on bandcamp. Bandcamp makes it pretty easy to subscribe to artists and labels and get updates from them directly. Unfortunately, this all comes through email. While it is easy enough to filter to avoid cluttering my inbox, email isn’t my desired place for this information.

I run a private freshrss instance and curate several hundred RSS feeds. It syncs nicely between my various devices, and is just a great way to consume. My goal is to get all my music updates from bandcamp here, so that’d they be in the same place as where I get music reviews and follow some other underground music blogs.

Bandcamp doesn’t provide any rss feeds. There are some options, like using RSSHub and its bandcamp addon. However, many of the updates that come through email are private, and don’t show up in an artists’ or labels’ feed.

I decided, I’d go a different route, and just convert the emails I was getting into an RSS feed directly.

Step 1 – imapfilter

I already use imapfilter heavily for filtering and tagging my emails. I decided I could use it to pipe the bandcamp emails to a custom script that would convert it into an rss entry. Here is the relevant section:

function filter_bandcamp(messages)
   -- Grab anything from Bandcamp where the subject
   --  starts with "New". This indicates a message
   --  or updated rather than a receipt.
   -- I am taking these "mailing list" bandcamps
   --  messages and converting them to an rss feed.
   results = messages:contain_from('noreply@bandcamp.com') *
             messages:contain_subject('New')

   for _, mesg in ipairs(results) do
      mbox, uid = table.unpack(mesg)
      text = mbox[uid]:fetch_message()
      pipe_to('/opt/email2rss/email2rss', text)
   end

   -- delete the bandcamp messages
   results:delete_messages()

   return messages;
end

You’ll notice I don’t want all messages from “noreply@bandcamp.com”, since that would also include things like purchases and system notifications.

Step 2 – email2rss script

The email2rss script is a python program. I am using feedgen to generate the feed, and built in python libraries to parse the emails.

One issue you’ll notice immediately, is that this script runs once for every email message. For RSS, we want to create a continuous feed with all the entries. This means we have to insert the new entry into an existing file and have some persistence. The quickest/dirtiest method was to use python’s builtin pickle to serialize and deserialize the whole state. That way, I can quickly load the previous state, create a new entry, write out the rss file, then serialize and save the state to disk.

Here is the program in its entirety:

#!/usr/bin/env python3
#

import sys
import email.parser
import datetime
import pickle
import re
import os
from feedgen.feed import FeedGenerator

DATADIR=os.path.join('/', 'opt', 'email2rss', 'data')

def default_feed():
  fg = FeedGenerator()
  fg.id('https://feeds.line72.net/')
  fg.title('Bandcamp Updates')
  fg.description('Bandcamp Updates')
  fg.link(href = 'https://feeds.line72.net')
  fg.language('en')

  return fg

def get_feedgenerator():
  try:
    with open(os.path.join(DATADIR, 'feed.obj'), 'rb') as f:
      return pickle.load(f)
  except IOError:
    return default_feed()

def save_feedgenerator(fg):
  with open(os.path.join(DATADIR, 'feed.obj'), 'wb') as f:
    pickle.dump(fg, f)

def add_item(fg, msg, content):
  msg_id_header = msg.get('Message-ID')
  msg_id = re.match(r'^\<(.*?)@.*$', msg_id_header).group(1)
  sender = msg.get('From')
  subject = msg.get('Subject')

  fe = fg.add_entry()
  fe.id(f'https://feeds.line72.net/{msg_id}')
  fe.title(subject)
  fe.author(name = 'Bandcamp', email = 'noreply@bandcamp.com')
  fe.pubDate(datetime.datetime.utcnow().astimezone())
  fe.description(subject)
  fe.content(content, type = 'CDATA')

def go():
  fg = get_feedgenerator()

  parser = email.parser.Parser()
  msg = parser.parse(sys.stdin)
  for part in msg.walk():
    if part.get_content_type() == 'text/html':
      add_item(fg, msg, part.get_payload())
      break

  fg.rss_file(os.path.join(DATADIR, 'feed.rss'), pretty = True)

  save_feedgenerator(fg)

if __name__ == '__main__':
  go()

Step 3 – Host the Feed

I just have a simple nginx server running that hosts the feed.rss. Then I simply add this new feed to my freshrss instance.

Future Work

There are still some improvements that could be made to this. The history is going to grow out of control at some point, so I really should probably go through and delete old entries, maybe to keep a history of 100 or 500 entries.

The other possible issue (I haven’t run into it yet), is that the email2rss could be run simultaneously. If that is the case, then one entry will likely be lost. I really should probably have a lock around the feed.obj to keep a second instance from doing anything until the first was written out the new state.

Fixing the Audio on a Pinebook Pro after resume

I recently purchased a Pinebook Pro, which is a Linux laptop based on the Arm processor. It is a great travel laptop, and surprisingly, I have found this to now be my primary personal device.

Everything has worked well, except one annoying bug: after resuming from sleep, the audio stops working. After digging around, I found that a simple command to write some data to the device to unbind it, then rebind it resolves it. I incorporated this into a systemd script that it is automatically run upon resume.

Create a new file called: /usr/lib/systemd/system-sleep/audio with the following (sudo vim /usr/lib/systemd/system-sleep/audio)

#!/bin/sh
#
# mwd - 20200726
# This is a script to reload the audio module
#  upon resume.
case $1 in
    post|resume)
        tee /sys/bus/i2c/drivers/es8316/{un,}bind <<< 1-0011
        ;;
esac

Save this file and make sure it is executable:

sudo chmod +x /usr/lib/systemd/system-sleep/audio

New Bus Icons in Go Transit App

I just released version 1.6.1 of Montclair for all the Go Transit cities. This includes a minor visual change of a new icon for the bus.

Previously, I was using Availtec’s Icon Factory, which lets you specify a color and heading, and it generates a .png image of that color with an arrow indicating the direction the bus is going. It has worked well, and looks nice enough, but I don’t like depending upon a 3rd party service, especially for cities that don’t use Availtec.

After some deliberation, I decided I wanted to do everything client side. I would have a single SVG icon that the client would load in javascript and would manipulate the stroke/fill colors and rotate the arrow to the heading. This works quite well, and the result is a nice clear image that we only have to load once!

This code works by loading the SVG through the DOMParser, then modifying attributes, reserializing it, then returning it as a data: url that is base64 encoded.

// load the svg
let xml = new DOMParser().parseFromString(svg.data, 'image/svg+xml');
// update the attributes //

// 1. the gradient
// stop1
let stop1 = xml.querySelector('#stop958');
stop1.style.stopColor = '#' + this.color;

...

// 5. The bearing, set its rotation
let bearing = xml.querySelector('#bearing');
bearing.setAttribute('transform', `rotate(${this.heading}, 250, 190)`);

// 6. Serialize and generate a data url with base64 encoded data
let serialized = new XMLSerializer().serializeToString(xml);
const url = 'data:image/svg+xml;base64,' + btoa(serialized);

Here’s the result:

New Bus Icons using custom Icon Factory

Go Transit App Updates

I have released a new version of the Montclair software, 1.6.0, that includes some major improvements to seeing estimated arrivals of buses at stops.

When selecting a stop, the app now goes into a full screen split mode with the estimated arrivals and the map. Clicking on one of the estimated arrivals will show you where that specific bus relative to your stop and will track the bus until it arrives. This makes seeing your next bus super simple!

This version will automatically roll out to your favorite “Go Transit” city app!

Go Memphis and Go Nashville

I have added two more cities to the Go Transit Apps series, Go Memphis and Go Nashville. Both are available through the web, or in the Android Play Store and Apple App Store.

Go Memphis

‎Go Memphis
‎Go Memphis
Developer: Marcus Dillavou
Price: Free
Go Memphis
Go Memphis
Developer:
Price: Free

Go Nashville

‎Go Nashville
‎Go Nashville
Developer: Marcus Dillavou
Price: Free
Go Nashville
Go Nashville
Developer:
Price: Free

For a full list of cities and features, see the Go Transit App website.

More cities supported by Go Transit

I have been working on adding support for more cities. The following new cities now have Go Transit Apps:

Go Akron, OH

‎Go Akron
‎Go Akron
Developer: Marcus Dillavou
Price: Free
Go Akron
Go Akron
Developer:
Price: Free

Go Athens, GA

‎Go Athens, GA
‎Go Athens, GA
Developer: Marcus Dillavou
Price: Free
Go Athens, GA
Go Athens, GA
Developer:
Price: Free

Go Grand Rapids, MI

‎Go Grand Rapids
‎Go Grand Rapids
Developer: Marcus Dillavou
Price: Free
Go Grand Rapids
Go Grand Rapids
Developer:
Price: Free

Go Huntsville, Alabama

‎Go Huntsville
‎Go Huntsville
Developer: Marcus Dillavou
Price: Free
Go Huntsville
Go Huntsville
Developer:
Price: Free

Go Montgomery, AL

‎Go Montgomery, AL
‎Go Montgomery, AL
Developer: Marcus Dillavou
Price: Free
Go Montgomery, AL
Go Montgomery, AL
Developer:
Price: Free

You can also find all of these in the Google Play Store or the Apple App Store.

For a full list of cities and features, see the Go Transit App website.

See Stops and Arrival Estimates in Go Transit Apps

I have been making several improvements to the montclair code base, which is the software that powers all the Go Transit apps.

Go Transit now shows the stops of the selected route
See the stops for the Highland #12

First, for the selected routes, you can now see where all the stops are. This helps people who are unfamiliar with the system know where to pick up a bus that is near by.

See the arrival times of buses for the selected stop
Stop Arrival Times

Selecting a stop on the map will show you the estimated arrival time of buses for that stop! This is a live view that is continuously updated as the arrival estimates change.

If multiple buses use this stop, view all their arrival times
Arrival Times of a Stop with Multiple Buses

If multiple buses go through the same stop, then the app will show the arrival estimates for all the buses. This is especially useful in cases where routes overlap and you have multiple options.

All of these changes have already been pushed out, and any apps you have should automatically update!

Announcing the Go Transit Apps

During a recent visit to Steamboat Springs, Colorado, I quickly became frustrated with the app for the free bus system. I thought to myself that it would be really nice to have my Montclair app available for other cities. With that, the Go Transit series of apps were born.

I started off building out github actions for the montclair, montclair-pwa-android, and montclair-pwa-ios projects. The github actions will automatically generate a compiled build when the repository is tagged.

The next step was to build a transmogrifier. The transmogrifier takes a configuration, then pulls down a specific version of montclair, montclair-pwa-android, and montclair-pwa-ios, and replaces all the necessary names, strings, icons, and other information. The new versions are then pushed to their own repositories and tagged, which automatically kicks off the white labeled builds.

All that is left it to create a custom configuration (see Steamboat for an example). The assets projects all have github actions too, and each time a commit is pushed, it will then run the transmogrifier.

When all the builds are done, they can be uploaded to my website hosting provider, the Google Play Store, and the Apple App Store.

So far I have created white label builds for:

Go Birmingham, AL

‎Go Birmingham
‎Go Birmingham
Developer: Marcus Dillavou
Price: Free
Go Birmingham
Go Birmingham
Developer:
Price: Free

Go Mobile, AL

‎Go Mobile, AL
‎Go Mobile, AL
Developer: Marcus Dillavou
Price: Free
Go Mobile
Go Mobile
Developer:
Price: Free

Go Steamboat Springs, CO

‎Go Steamboat
‎Go Steamboat
Developer: Marcus Dillavou
Price: Free
Go Steamboat
Go Steamboat
Developer:
Price: Free

Go Indianapolis, IN

‎Go Indianapolis
‎Go Indianapolis
Developer: Marcus Dillavou
Price: Free
Go Indianapolis
Go Indianapolis
Developer:
Price: Free

Go Raleigh, NC

‎Go Raleigh
‎Go Raleigh
Developer: Marcus Dillavou
Price: Free
Go Raleigh
Go Raleigh
Developer:
Price: Free

For more information, see the Go Transit App website.