How El Dorado Query Worked

I'm looking around in the open-source version of El Dorado, the service I developed for New Relic that made good use of the Neo4j graph database. matrix

This is the ruby statement I used to run a query: `@results = neo4j.query(cypher(@query), @params)`

The conversation we had at the end of today's call concerned getting @results into wiki. The code I have in El Dorado does this with graphviz dot format as the desired target. We'd like to do the some but not all of this. github

This is done with two subroutines. The first, dotify, formats data extracted from @results in dot format. We would rather just convert this directly into Graph objects, the Neo4j inspired javascript implementation of property graph we use inside our wiki code.

The dotify calls the second subroutine, interpolate_kv, which converts the @results format into something the dotify ruby code can use directly.

I suspect the Neo4j javascript driver has something like the ruby driver's @result format. We would need to understand that, and then write a replacement for interpret_kv. With this we would write a replacement for dotify, perhaps called graphify, that yields our Graph objects instead of dot strings. graph

# Details

If we are talking about rows and fields within rows we still have to recognize nodes and relations within this data. Maybe interpolate_kv doesn't do the whole job. This is coming back to me. I would write El Dorado queries in two parts: cypher query template and a dot rendering template.

For example in cypher I would output a list of variables, maybe something like: `return employee.name as engineer, employee.supervisor.name as boss`

Then I would turn this into dot by substituting these words into a dot edge: `engineer -> boss [label="reports"]`

This is an important feature of El Dorado queries: they knew how to do two things. Both were expressed as template strings with substitutable parameters. For cypher templates the parameters came from the url that was used to access El Dorado. For dot templates the parameters came from the cypher query result. The author of an El Dorado introduced temporary words, like boss and engineer that connect the two templates together.

# More

There was a third layer of specification that made the graphviz nodes clickable. There was a way of saying that when an engineer is clicked run the query for module dependency graph surrounding modules owned by the named engineer, an employee.

Alternatively, should one click the boss node it would run the query for budget responsibilities of the named supervisor, also an employee, but one for which we would expect to find budget responsibilities in the large Neo4j graph.

I've mentioned that our catalog of queries grew to 150 objects that had lots more going on than in cypher on its own. Each El Dorado exposed an endpoint to the web, accepted parameters to narrow the result of the query to specifics, composed and ran a query from a template interpolated with url parameters, expanded the result for display via graphviz dot format templates interpolated with Neo4j query chosen outputs, and then enabled click handing on nodes with new El Dorado web requests with specifics, such as employee names, for a known to be useful follow on query.

El Dorado queries were stored in YAML that could be composed and revised in the web interface. Here is the most interesting query I used to recall what Neo4j held for me with enough source information that I could trust the results I got. I'll explain the various parts. This is online here: github

First the cypher. This doesn't take any parameters because it sought to show the whole database.

match (n)-[r]->(m) with distinct '.*' as Regex, count(r) as Count, r.source as Source, type(r) as Type, head(labels(n)) as From, head(labels(m)) as To return * order by Source

This is the dot for a complete diagram. Lines are repeated for each field which appears multiple times.

strict digraph { rankdir=LR node [shape=box style=filled] "{Source}" [shape=plaintext] "{From}" [URL="..."] "{To}" [URL="..."] "{From}{Type}{To}" [shape=plaintext] "{From}" -> "{From}{Type}{To}" "{From}{Type}{To}" -> "{To}" "{Source}" -> "{From}{Type}{To}" }

Now I am thinking how this could be mapped from YAML to wiki pages with Mech to perform all steps expected of an El Dorado query. There is lots more going on here than I was remembering. But it is within reach.