088: TransitMap from a live XML feed with python

Note: an updated version of this project has been added to the Palette > Techniques section starting TouchDesigner088 build 50000

map.jpg
After looking for a source of real-time data on the web, I came across Open Data – City of Toronto where one of the datafeeds they have is the fleet location of the local bus and subway transit authority TTC. After looking at the document which describes the datafeed (nextbus.com/xmlFeedDocs/NextBusXMLFeed.pdf) I found that it not only provides real-time updates on fleet location but also the full definition of the transit system, enabling me to create a full map.

The component as it is right now lets you choose a Transit Agency among 20+ cities, select a Route by either clicking on the route on the map or selecting a route from the list on the left side of the screen. Once a route is selected it will show and update the location of any transit vehicles on the route in 10 second intervals.

When selecting a new Transit Agency and building a map please note that:

[b]- you need an Internet connection and

  • rebuilding a map takes a while as there is a large amount of data to be processed[/b](2 to 3 minutes - perhaps by moving the SOP generation to another process would let you keep a more interactive feel of the application)

To do all this I created a python function collection called getXML located in /project1/local/modules.
The class can fetch XML from a source and parse it for:

  • a list of Agencies (the Transit Authorities participating in this project)
  • a list of Routes of a selected Agency
  • a detailed description of the Routes including stops and pathpoints
  • a list of vehicles on a specific route

For parsing the XML feed I choose to use the default minidom python set of functions and the urllib class to fetch data from a URL:

import urllib.request
from xml.dom import minidom

to retrieve data from a URL and use it as a minidom parsed object use:

myUrl = 'http://some.url/toaxmlfeed' 
dom = minidom.parse(urllib.request.urlopen(myUrl))

The received XML might look like this:

<body>
	<route tag="1" title="1 - California" shortTitle="1-Calif/>
	<route tag="3" title="3 - Jackson" shortTitle="3-Jacksn"/>
	<route tag="4" title="4 - Sutter" shortTitle="4-Sutter"/>
	<route tag="5" title="5 - Fulton" shortTitle="5-Fulton"/>
	<route tag="6" title="6 - Parnassus" shortTitle="6-Parnas"/>
	<route tag="7" title="7 - Haight" shortTitle="7-Haight"/>
	<route tag="14" title="14 - Mission" shortTitle="14-Missn"/>
	<route tag="21" title="21 - Hayes" shortTitle="21-Hayes"/>
</body>

Next I loop through the XML structure to retrieve data from a tag by a certain name and add them to a dictionary:

route = []
for node in dom.getElementsByTagName('route'):
	route.append({
		'title' : node.getAttribute('title'),
		'shorttitle' : node.getAttribute('shortTitle'),
		'tag' : node.getAttribute('tag')
	})

As the getXML is saved in a local/modules folder, any script below the local Base COMPs location can find getXML and I can call a function from a different DAT in such a way:

import getXML
a = getXML.getAgencyList()

This call would get me all the agencies available from the service provider. Analogue I would call:

import getXML
tag = op('selected_agency')[1,'tag'].val
a = getXML.getRouteList(str(tag))

to retrieve all routes for a specific agency.

All in all, it is a pretty straightforward process.
All the data for the routes are added to a Table DAT which is then used as a Template Table for a Replicator Component (/project1/routeDetail) which automatically creates Geometry Components for each route, containing Table DATs to save Path description, Stop details and Vehicle locations.

To convert the Table containing the Path definition to a actual geometry I’m using the new Script SOP which proves to be a very convenient way to create 3D geometry from any kind of source. The Script SOP lets you add polygons and points via python. In my example I essentially loop over the table with all the route’s path points and generate an open polygon with it:
scriptSOP.jpg

#pointList is a list of valid points derived from a Table DAT
numVertices = len(pointList)
a = scriptOP.appendPoly(numVertices,closed=False)
	for v in range(numVertices):
		a[v].point.x = pointList[v][1]
		a[v].point.y = pointList[v][0]

The geometry is then rendered in /project1/map, and I added a Render Pick DAT setup to select a Route which vehicles (if any) are then shown and updated on that Route.

If you have a look in /project1/map/container2/table1 which is the left hand size route selection list, you can also find some Script DATs which, similar to Script SOPs utilize python to generate data. Where before very long expressions or references to macros had to be used in Evaluate DATs, the Script DAT allows for much more complex data manipulation and/or generation.

Note: This project requires TouchDesigner088 beta

Snaut,

Thank you so much for posting this example. It’s an excellent educational tool for learning how to use TouchDesigner + Python to parse through XML files and use the information for data visualization. I’ll be working through this over the next few days. :slight_smile:

-L05

yes, that is an incredible source of inspiration , techniques, tricks…

  • the generative design examples might keep me busy for a while… :smiley:

Many thanks Snaut.

Xavier

I would really like to look at this project, unfortunately it seems like I am getting errors in all three of the /project1/ COMPS.

Do you have any ideas on how to possibly fix these?