New Relic delivered a SaaS product built as a Ruby on Rails monolith which was later converted to cooperating microservices. The "Treasure Map" was to provide an inventory with graphical maps generated from metadata provided by the teams that built the service that held the "treasure" we might want to locate.
We've seen three distinct approaches that differ mostly in the data format that holds the map together. First YAML files in GitHub, then property graphs in Neo4j, and ultimately SVG images with JSON annotations.
# YAML
The first map was a Rails app that read cleverly hierarchical YAML files to which teams would have to write pull-requests whenever anything changed. This proved to be onerous for teams and the architects that merged the changes.
Teams eventually insisted that their YAML metadata would be maintained in their own repos and updated by development and deployment workflows each team designed for themselves. The architects cleverly constructed an automated pull-request authoring system so that the final responsibility for reliable information remained within their organization.
# Neo4j
El Dorado was created to make better use of Treasure Map data but soon showed the value of integrating more data sources and parameterizing saved queries served from a web app. Soon we were expanding the tabular results from Graphviz dot fragments saved with the queries and linking to next logical query when particular nodes were clicked.
El Dorado's diagrams were useful and new variations could be authored in moments and shared as urls on Slack or any other web medium. In spite of this versatility end-users rarely composed new queries. Worse still, the diagrams answered specific questions rather than painting pictures that showed "what a system does?"
Here we collect various mentions of the work we've done observing software through the metadata produced throughout its creation and operation. github
El Dorado is a company wide database that incorporates forty distinct information sources to build and maintain a graph database with a half-million relations. It could map its own contents with a stored query that joined 150 other reusable queries. Called "Schema with Sources" it is most representative of our drill-down-to-queries mechanisms.
# SVG
Teams were often asked to present "deep dives" into how their work was done. This lead to talking tours of a system diagram prepared for the purpose and shared widely afterwords. Although various drawing tools were used the diagrams followed conventions like information flowing left to right and control paths above data paths below.
We were similarly importing many drawings of many kinds but were particularly attracted to SVG format that could be "enriched" with click-handling information that let them participate in wiki in a style similar to El Dorado.
A new web app called Open Treasure Map was born to collect, enrich, browse and search these as a lasting resource. A public version focused on navigating wiki while a private, New Relic specific, version favored GitHub repositories and live system monitoring infrastructure.
A collection of diagrams describe the base system as we collectively understand it. We navigate these through metadata annotations collected with a web tool. demo
This interactive editor dynamically loads diagrams as images and metadata about each diagram as json. This requires some coordination to get the right files at the right time. github
Three concentric circles grow and fade and then wrap back to radius zero to repeat the process. I saw a similar animation artfully employed to advance scenes. site github
# Future
In our long career we have made a habit of writing programs to understand what things mean and how things work. Smalltalk and it's constructionist roots inspired many of our experimental methods supported with end-user programming one way or another.
We used PEG grammars to mine documents including all of Wikipedia with curiosity driven, quick turn-around experiments. See AboutUs Getting Started blog post announcing the open-sourcing of this technology. post demo
Four distinct concerns surface when designing federated wiki based information communities. We identify each and suggest forces that must be resolved before creative collaboration will take place.
We have created 30 experiments building and using property graphs on wiki pages. We're beginning to see complementary workflows building on the work so far.
We are using this page to remember those who we are collaborating within our creative noosphere. We form weekly activity into clusters and then merge those as recommended aspect of our collaborations.