Each “block” is a bus trip throughout the day. The dark black line represents the actual time the bus arrived at this stop. The dark blue line shows when Availtec estimated the bus would arrive at that specific time.
Ideally, the dark blue line would be as close as possible to the dark black line.
From the image above, we can see that the 4th trip, from 10:44:17 to 10:46:17 had the best estimate. From 10:00am to 10:50am, Availtec estimated the bus would arrive ±30 seconds from when it actually did.
We can also see that the trip after it, had the worst estimate. From the time the trip started, just before 10:50, until around 11:20, Availtec was estimating the bus would arrive at 11:32, when it actually came at 11:38. It wasn’t until about 11:25, that the system suddenly corrected and started estimating a more realistic time.
This project works with any bus system that utilizes Availtec for real time tracking. You can leave the capturer program running in the background for days, then generate graphs based on that historical data. The graphing script allows you to specify which stops, routes, and days to plot.
I recently set up a raspberry pi to use as a smart dashboard on my TV. The goal is to turn on my TV and have weather, my calendar, todo list, and other useful widgets readily available. Instead of building my own, I decided to use the Smashing Dashboard, a fork of the original Dashing developed by Shopify. It is a nice looking, simple dashboard and can easily be extended with lots of widgets.
One widget that I really needed is the ability to see when the next bus arrives at the stop down the street from me. Birmingham, Alabama’s bus system uses Availtec, which has an API for getting the real time location of the bus along with the estimated arrival times at a stop. Using this, I wrote a new widget for smashing called smashing-availtec. It should work with any transit system powered by Availtec. Here is what it looks like:
Since there are multiple stops I want to monitor, my widget allows creating multiple widgets, and having each widget filter for a specific stop and route (in case multiple routes go through the same stop). It will give an estimated time of arrival for the next two buses and will also how a live map of where the next bus is.
Instructions for getting the source code and configuring it are available on my github project page. Enjoy!
This came about because I have been unhappy about how iOS handles progressive web apps. Every time the app is brought to the foreground, iOS does a full reload, which can be slow and annoying. The native app handles this much nicer, and doesn’t ever have to reload the state.
To build the native apps, I used this handy tool: Progress Web App Builder. It takes your website, wraps it using Cordova, and generates both an iOS and an Android project. With only a few tweaks, I was able to get them submitted to the app store.
Birmingham, Alabama’s transit agency, the BJCTA (Birmingham-Jefferson County Transit Authority) has a website for tracking the buses in real time. This website along with their mobile app are ok, but have some things that constantly annoy me. My biggest annoyance is that every time the App is brought to the foreground, it refreshes and resets you back to the main page, losing the bus you were trying to track. The other problem is that you can only view one route at a time (at least on the mobile app). I have 3 routes that I can choose between every day. When I open the app, I want those three routes selected by default so I can quickly see which route to pick.
With the recent arrival of gnome 3.28, and with it, the Epiphany Web browser 3.28, saving a web page as a webapp has dramatically improved. Saving a page as a webapp has several advantages:
Cookies are separate for each app. This helps limit tracking, but can also make managing multiple accounts much easier, like separating out your person gmail account with a work gmail account.
A clear separation of tasks. I spend a lot of time in Jira, confluence, and the AWS console. Keeping them as separate applications instead of digging through 50 tabs in firefox, is very useful.
They are treated as completely separate apps. This means that they have their own launcher with their own applications, can be pinned to the dock, and show up as separate applications in the window switcher
Creating a web app in Epiphany is as simple as browsing to the website you want, clicking on the Menu button (三), and choose ‘Install Site as Web Application…’
iOS and Android support the concept of Deep Linking, also sometimes known as Universal Links. The idea, is that if you click on a link for a specific website, say Reddit, the operating system can decide to open that link with the installed Reddit App instead of the web browser.
I have found this to be a nice concept. When I get an email or slack message with a link to a Jira ticket, I want it to open up with my Jira web app, not my default web browser. After all, that was part of the point of having created these separate Web Apps. Unfortunately, there is no native support for this feature, but, since Linux is an open and configurable system, there is always a way!
Creating a new Default Web Browser
Gnome only allows registering applications based on the MIMEType. Unfortunately, all websites share the same scheme: ‘x-scheme-handler/http’. By default, gnome opens up your default browser and passes it the URL. We will create a new simple shell script and make it our default web browser. This simple shell script can do some basic parsing of URLs, then decide to open up a specific web app or to pass everything on to firefox.
A .desktop file is required to register our new App. Create a new file called:
This declares that it can handle the necessary MimeTypes for URLs. You will also notice that it executes /home/username/.local/bin/http-handler. This is the script we are going to create next, however, please change the username to be your username.
Building the Web App Script
I am just using a simple bash script to parse the URL and decide whether to open a specific web app or to open the default web browser. You can modify this or write it in any language that you want. This is just mean to be a simple example.
Create a new file named `~/.local/bin/http-handler` with the following:
My script looks to see if a URL matches my Jira url, and if so, opens that webapp. Otherwise, it checks to see if it matches AWS console, and opens my AWS webapp. Otherwise, it runs firefox and opens the URL in a new window.
While there are many ways to parse the URL, this one uses gawk so we can test some regular expressions. The Jira URL was interesting because I want everything that starts with https://company.atlassian.net/* except https://company.atlassian.net/wiki/* since that is confluence, a separate app.
One thing about Epiphany Web Apps, is that there isn’t an easy way to run them directly. They must be executed through their .desktop file. This is so that they set the correct WM class so that the desktop environment knows to treat them as a separate application. Fortunately, we can use the gtk-launch program which takes a .desktop file (minus the .desktop) as a parameter, and launches it.
Fixing the .desktop files
When you install a Web App, epiphany creates a new desktop file in `~/.local/share/applications/` with a name like `epiphany-website-a8d40c5d8d04433783264b93849886b867fa11c4.desktop`. These desktop files have the website hard coded into the Exec line. This means that we can’t pass it a URL, so it always opens up to the default page, which isn’t what we want. I haven’t yet found a way to have a default parameter that can be overridden. The trick is to create a copy of the .desktop file that does take a URL as a parameter. This means we will end up with two launchers, but we’ll set the NoDisplay parameter in our new one, so that it doesn’t show up in the launcher screen.
Copy the webapp .desktop file to a new name, something like:
My previous blog post (Visualizing the On Time Performance of Birmingham’s Bus System) looked at creating visualizations based on the real time data provided for Birmingham, Alabama’s public transit system. I have now created a simple, public API that can be used to query the historical data captured from the real time system.
Using my dv8 scripts, I am capturing a snapshot of the real time data every 30 seconds. I am then storing this data in a sql database, so that we will have a historical database. This allows for generating reports and visualizations over time. As of writing this, I have about 2.5 months of data captured.
I am making all of this data public, along with a simple API for making queries. The API is located at:
However, if you visit that site, you will notice that you get a ‘Page Not Found’ error. That is because it is only a backend API, and you will need to construct URLS to make queries. Let’s look at a few sample queries (All data is returned in JSON format. I have found that making these queries in Firefox is preferred over Chrome, since Firefox will nicely format the results):
This returned a list of all the known routes and their properties. Each route also has a set of “Trips”. A trip is a unique instance of a bus running along a route. For example, if everyday at 4.00pm a bus leaves Central Station along the route 44 and arrives at its final destination at 5.00pm, this would be a Trip. We can get a list of all the trips for a specific route.
Query for all the trips on the Route 44
Based on the previous query, I have found that the “44 Montclair” has an id of 21.
This returned all 61 trips associated with the route 44. Unfortunately, this doesn’t give us that much information about the individual trip. For instance, we have no idea based on the returned results what time of day a trip runs. For every trip, we can get a series of “Waypoints”. Waypoints are all the known information about a bus on a trip at a given time. They are essentially a snapshot, showing the location, passenger count, time deviation, and various other details. We can pick a specific trip and query it.
Query all Waypoints for a single trip on the Route 44
I am going to select a trip at random from the previous query. I will use 699, and will query its waypoints:
This returned A LOT of data. This trip runs every weekday, and we have samples at every 30 seconds. The API returns a maximum of 1000 waypoints, so only the first part of the data was returned. Typically, we want to filter this data, and only get a specific range. For example, let’s only get the waypoints on a specific date. We can use the start_date and end_date parameters to put limits:
On this date, this trip was an Outbound (0) trip. It left Central Station at 11:45:00 AM and arrived at Eastwood Mall on time at 12:21:00 PM.
We have samples every 30 seconds of this trip. Using that, we could calculated the average on time performance or visualize the number of passengers on the bus. You can take this json data and convert it to CSV using a simple converter tool. You can then import your CSV into a spreadsheet program, like Excel or LibreOffice Calc and create visualizations.
A few months ago, Birmingham, Alabama’s bus system (BJCTA Max Transit) retrofitted all of their buses with a GPS system. Along with this upgrade, they released a website allowing riders and the public to track the locations of buses in real time. The website is backed with a public RESTful API, so I decided I would poll the API to retrieve live information on all the buses throughout the day and store them in a local database.
With this information, it is possible to run some analysis on the system. We could look at passenger count, travel time, travel deviation, areas where buses tend to slow down, performance at different times of the day, and a host of other analytics.
The first analysis I have done is looking at the time deviation. This is how far ahead or behind schedule a bus currently is. A negative number means it is ahead of schedule and a positive number means it is behind schedule. The general rule is that ± 5 minutes is “on time”.
The graph below goes through an entire day looking at all the buses on each route. Each route is separated into its own section, so you can clearly see how a single route performs throughout the day. Along a route, multiple buses will simultaneously be making individual trips along that route. Each color represents a different bus making a different trip. Since there is overlap between buses sometimes making it difficult to see where a trip starts and ends, I have placed bars below the graph. Each of these bars mirrors a single bus making a single trip, allowing you to easily see how many active buses there are and when they started and stopped their trip.
If we zoom into a specific route, we can get more detail (Click on the image to see the full day). This is line 17 – Eastwood Mall. It is one of the busier routes with several buses making simultaneous trips. As you can see, most trips start and end on time, however, there is a bit of a peak about halfway through the trips, where the bus is behind schedule by up to 15 minutes.
After looking through the data, I have few interesting conclusions:
There is quite a bit of deviation along most routes. However, the most important thing to look at is the start and end of each trip, and for most trips, those have a tendency to converge on 0 deviation. This means the overall trip length is as expected, but buses are being held up in the middle of trips due to traffic and other unforeseeable circumstances.
I don’t know what happened to the 14 (Idlewild Palisades) around 5pm. I’m thinking the bus broke down and a new bus (purple) had to be sent out to replace it.
Some routes are interlined. For example, the 12 and 18. The bus starts off as the 12, does its trip, but once it pulls into central station, that same bus becomes the 18 and does a trip along route 18. When it returns to central station, it once again becomes the 12. It appears as though the 18 is constantly taking longer than the expected time which makes the 12 start late. Even though the 12 makes up time, it can’t seem to ever get on schedule.
The 280 runs down a major suburban corridor that has the worst traffic in the region. Even during rush hour at 5.00pm, the route is expected to have the same trip time as 10am. Clearly it doesn’t.
The most heavily used routes: 17, 1, 3, 44, 45, 6, 28, 8, 1 Ex all have minor deviation and run on time.
The following are instructions for setting up an L2TP VPN server on a Raspberry Pi running Raspbian Jessie. This allows you to connect your iPhone or other device using L2TP VPN to your home network, to securely access resources on it. This set up uses a Raspberry Pi sitting behind your normal router.
These instructions are based on an older forum post on the Raspberry Pi forums.
All of the following commands will need to be run as root. Use sudo to become the root user.
$ sudo su -
Configuring a Static IP Address
Since your Raspberry Pi is running a server, it will be important to give it a consistent IP address so that we can forward the necessary ports to it. The IP address you choose depends on your local network setup. My network uses the 192.168.1.XXX range, so I have decided to use 192.168.1.16 for my Raspberry Pi. Here are the full settings for my setup:
IP Address: 192.168.1.16
DNS Server(s): 192.168.1.254
With the release of Raspbian Jessie, the method for configuring IP addresses has changed. Raspbian now uses dhcpcd as the default, so it is no longer recommended that you directly modify /etc/network/interfaces. Instead, we will modify dhcpcd’s configuration.
Edit /etc/dhcpcd.conf and add the following to the end. You will need to modify some of these values based on your setup.
IPSec is the encryption layer for your VPN tunnel. We are using the openswan implementation.
Replace your /etc/ipsec.conf with the following. Again, you will need to replace any values depending on your network setup.
# /etc/ipsec.conf - Openswan IPsec configuration file
# This file: /usr/share/doc/openswan/ipsec.conf-sample
# Manual: ipsec.conf.5
version 2.0 # conforms to second version of ipsec.conf specification
# basic configuration
# Do not set debug options to debug configuration issues!
# plutodebug / klipsdebug = "all", "none" or a combation from below:
# "raw crypt parsing emitting control klips pfkey natt x509 dpd private"
# plutodebug="control parsing"
# Again: only enable plutodebug or klipsdebug when asked by a developer
# enable to get logs per-peer
# Enable core dumps (might require system changes, like ulimit -C)
# This is required for abrtd to work properly
# Note: incorrect SElinux policies might prevent pluto writing the core
# NAT-TRAVERSAL support, see README.NAT-Traversal
# exclude networks used on server side by adding %v4:!a.b.c.0/24
# It seems that T-Mobile in the US and Rogers/Fido in Canada are
# using 25/8 as "private" address space on their 3G network.
# This range has not been announced via BGP (at least upto 2010-12-21)
# OE is now off by default. Uncomment and change to on, to enable.
# which IPsec stack to use. auto will try netkey, then klips then mast
# Use this to log to a file, or disable logging on embedded systems (like openwrt)
# Add connections here
# sample VPN connection
# for more examples, see /etc/ipsec.d/examples/
# # Left security gateway, subnet behind it, nexthop toward right.
# # Right security gateway, subnet behind it, nexthop toward left.
# # To authorize this connection, but not actually start it,
# # at startup, uncomment this.
# !mwd - disabling this fixed stuff
# we cannot rekey for %any, let client rekey
# Apple iOS doesn't send delete notify so we need dead peer detection
# to detect vanishing clients
# Set ikelifetime and keylife to same defaults windows has
# l2tp-over-ipsec is transport mode
# For updated Windows 2000/XP clients,
# to support old clients as well, use leftprotoport=17/%any
# The remote user.
# Using the magic port of "%any" means "any one single port". This is
# a work around required for Apple OSX clients that use a randomly
# high port.
#force all to be nat'ed. because of ios
# Normally, KLIPS drops all plaintext traffic from IP's it has a crypted
# connection with. With L2TP clients behind NAT, that's not really what
# you want. The connection below allows both l2tp/ipsec and plaintext
# connections from behind the same NAT router.
# The l2tpd use a leftprotoport, so they are more specific and will be used
# first. Then, packets for the host on different ports and protocols (eg ssh)
# will match this passthrough conn.
Configuring your Secret Key
The secret key is a shared key that all of your users will use. Edit /etc/ipsec.secrets
# This file holds shared secrets or RSA private keys for inter-Pluto
# authentication. See ipsec_pluto(8) manpage, and HTML documentation.
# RSA private key for this host, authenticating it to any other host
# which knows the public part. Suitable public keys, for ipsec.conf, DNS,
# or configuration of other implementations, can be extracted conveniently
# with "ipsec showhostkey".
# this file is managed with debconf and will contain the automatically created RSA keys
192.168.1.16 %any: PSK "MYSECRET"
Configuring your Users
You can create as many vpn users as you want. These users are separate from any linux user accounts on your Raspberry Pi. Edit /etc/ppp/chap-secrets
# Secrets for authentication using CHAP
# client server secret IP addresses
username * password *
Modifying iptables and System Services
We need to make some changes to the routing table and system configuration. First we will set some values and add them to our /etc/sysctl.conf which will be loaded each time the system starts up:
$ echo "net.ipv4.ip_forward = 1" | tee -a /etc/sysctl.conf
$ echo "net.ipv4.conf.all.accept_redirects = 0" | tee -a /etc/sysctl.conf
$ echo "net.ipv4.conf.all.send_redirects = 0" | tee -a /etc/sysctl.conf
$ sysctl -p
The iptables and /proc settings won’t survive a reboot. We’ll add these commands to the end of our /etc/rc.local to make sure they are executed on start up:
for vpn in /proc/sys/net/ipv4/conf/*; do echo 0 > $vpn/accept_redirects; echo 0 > $vpn/send_redirects; done
iptables --table nat --append POSTROUTING --jump MASQUERADE
Finally, let’s make sure our xl2tpd and ipsec services will be started on boot:
At this point, you should restart your Raspberry Pi to make sure all settings have taken effect and is configured correctly.
Configuring your Router Port Forwarding
This section depends on your router. Most consumer routers/wifi have a web admin interface at either 192.168.1.1 or 192.168.1.254. Once you are logged in, you’ll need to find the port forwarding or NAT/Gaming section. You will need to have the following ports forwarded to your Raspberry Pi IP Address, which in my case is 192.168.1.16. Please be aware that these ports are UDP not TCP.
Port 4500 UDP
Port 500 UDP
Connecting an iPhone
On your iPhone, go to Setting > General > VPN
Choose “Add VPN Configuration”. Select “L2TP” as the Type.
Description: Home VPN
Server: Your public IP address (this is NOT your 192.168.1.16 address. You can get this from your router or from http://whatismyip.com)
Account: The username you configured in /etc/ppp/chap-secrets
RSA SecureID: Disabled
Password: The password you configured in /etc/ppp/chap-secrets
Secret: The Shared secret you configured in /etc/ipsec.secrets
Send All Traffic: If enabled, then ALL your internet traffic will be routed through your home network. If you disable this, then normal internet traffic won’t go through your home network. The VPN will only be used to access devices on your home network
Git is an incredibly powerful tool, but often times very obtuse. There are many different types of work flows you can use with git, but one that I use often is having a master branch where all new development happens, and having a maintenance branch, where certain fixes get backported for a new stable release. I have found that if you don’t do things in a certain way, backporting changes can be a very tedious task.
For my work flow, anytime new work is going to begin on a feature or a bug, it is always done in a new feature branch created from master. In my work flow, that branch is always tied to a ticket (that could be a github issue or a jira ticket) that explains the details of the fix. Each branch fixes or adds one and only one thing. Doing this will make backporting specific fixes or features so much easier. Once this feature branch is completed, it is then merged into master. I ALWAYS merge using the –no-commit and –no-ff options. This is for several reasons:
$ git merge --no-commit --no-ff issue-23
I always want a single commit that represents the merge of the entire branch. When backporting, I will be able to specifying this single commit and it will bring in all the associated commits with it
It keeps my top level git log clean. Running git log –first-parent will only show my merge commits, not every single commit that was ever made.
It gives me a chance to update the ChangeLog appropriately and to write a standard commit message, like “Merged Issue 23 – Added new feature blah”
Let’s look at an example. At some point, I decided master was ready for a 1.0.0 release. I tagged this version, and then created a new maintenance branch from master called 1.0.x
$ git checkout -b 1.0.x master
Development will continue on in the master branch, but any critical bug fixes will get back ported from master into the 1.0.x branch, where I will then make new stable releases of 1.0.1, 1.0.2, …
If we take a look at the following log of master, you will see there have been several top level commits into master merging in various feature branches. Each of the commits has lots of subcommits that made up that feature branch.
If we look at the following, the very bottom commit shows where master and the 1.0.x branch diverged. Looking at master, we can see we have 4 merge commits. Each of these branches that were merged in contain subcommits:
HPM-1258 – invitation_sent_at timestamp
HPM-1248 – Update validation script to check for duplicates in invitations and contact_requests
HPM-1232 – Script to fix users with no enterprise
HPM-1234 – Script to fix users with no pods
Let’s say that HPM-1234 is the change that we want to merge in. This should bring in every single commit along the red line. This would be very tedious if we had to backport all six individual commits in that branch. That is why we merged in this branch into master using the –no-ff –no-commit options, so we could guarantee to create a single merge commit representing all of them.
Normally, in git, if we wanted to merge in HPM-1234, we would do a git merge 04145ec where 04145ec is the sha1 change. But in git, this will merge every change up to and including HPM-1234. So this will also merge in HPM-1232, HPM-1248, and HPM-1258, which is not what we want!
Git does have a special command for just picking out a single change: cherry-pick. Cherry picking allows us to just pull out a specific commit and merge it into a separate branch. Unfortunately, cherry-pick doesn’t handle scenarios well where a subcommit is a merge. If you look in our example, we can see along our red graph, where master was merged back into our branch. This is going to confuse the cherry-pick command. For the cherry-pick command to work, we will have to manually specific every single commit in the red branch one at a time, instead of just being able to specify the merge commit. And we will either have to skip over the merge commit, or handle it specially using the -m option (depending on what was merged in)
Instead of cherry-pick, we can instead generate a series of patches, then reapply them. We will do this through the use of two commands: format-patch and am.
There is an issue with doing this. Git loses some of the history, so the relationship of this merge between master and the 1.0.x branch is lost. However, for our case of backporting, this is acceptable.
Each one of the commits in the red graph will come into our maintenance branch as a separate commit. This is not exactly what we want, I would rather have a single top level merge commit in my 1.0.x. Therefore, I will create a new staging branch from the 1.0.x, backport the changes, then merge the staging branch into my 1.0.x branch. Here are the steps:
# create a new staging branch from
# the 1.0.x branch
$ git checkout -b backport-hpm-1234 1.0.x
# We will now run format-patch then pipe
# the output directly into am. This will
# create the patches and apply them in a
# single step. We will tell git to just
# use our top level commit
$ git format-patch -n --stdout 04145ec^..04145ec | git am
Applying: fixed the users that does not have a pod
Now we our staging branch looks like the following:
Now we can go back to our maintenance 1.0.x branch, and merge in our staging branch using the –no-commit and –no-ff options so that we can create a custom merge commit:
$ git checkout 1.0.x
$ git merge --no-commit --no-ff backport-hpm-1234
# We can update our ChangeLog or do anything else,
# then commit
$ git commit -m "Backported HPM-1234 - Script to fix users with no pods"
Our 1.0.x tree now looks like:
You can see we now have a single top level commit in our 1.0.x branch representing the HPM-1234 branch that was backported in.
As an update to a project I was working on last year, I have been updating and redesigning some wayfinding signs. These signs are created to fulfill several purposes:
(1) Provide awareness about the bus system, (2) give bus riders and potential bus riders easy information about their options at their current location, (3) and to provide pedestrians with orientation and landmark information
Since each wayfinding sign is specific to its location, I have been choosing sites that specifically have a lot of pedestrian access and potential choice bus riders. So far I have created wayfinding signs for areas in Highland Park for the route #12 and #44, and also in Avondale for the #17. Here’s an example:
I have been using all open source software to create these. All the routes and maps have been created using QGIS.Inkscape is used for creating vector designs, and Scribus for the final page layout and designs.
I have included all source material in the following downloads. If you would like to add additional locations our new routes, I would welcome any contributions.
If you would like to help hang signs on bus stops, Birmingham has over 3,000. I have found that laminating the signs keeps them protected from weather and that they can easily be hung with simple mounting brackets and industrial zip ties.