This release includes support for macros pulling panel results in any panel type (not just panels that support DM_getPanel functions), support for querying Airtable and AWS Athena, support for graphing with unique lines for each unique element, and more!

This release also has the largest contributions made by someone other than myself, support for loading Apache ORC files.

Macros

Before this release, only code panels and MySQL/PostgreSQL/SQLite panels could refer to other panel results using DM_getPanel() calls. With this release though all panel contents are run through a Jinja-like macro expander (pongo2, specifically) that allows you to reference other panel results.

For example, you can now upload a panel result to an HTTP server by entering {{ DM_getPanel("0") | json }} as an HTTP panel's content.

Note: unlike the non-macro version, the argument to DM_getPanel must be a string. The value can still be the panel's position or the panel's name.

Another way macros can be used is for passing data between completely different kinds of data sources. For example, let's say you want to filter ElasticSearch logs based on a certain kind of customer. Your SQL panel and query might be something like SELECT id FROM customers WHERE type ='x'. Then your ElasticSearch panel filter might be req.params.customer_id IN ({% for row in DM_getPanel("0") %}{{ row.id }}{% if not forloop.Last %}, {% endif %}{% endfor %}).

Macros can be used in every kind of panel, including code panels and database panels that already supported DM_getPanel() calls without macros. In those cases the existing non-macro DM_getPanel() is more efficient.

Airtable and AWS Athena

You can now fetch data from Airtable with or without filters. And you can query AWS Athena. A tutorial for interacting with Airtable in DataStation is available here. And here is a tutorial for querying AWS Athena.

Graphing unique entities

Before this release it was possible to graph multiple lines but where the multiple lines corresponded to entire columns. In this release it's now possible to generate graphs with multiple lines based on a unique values in the column being graphed.

You might use this feature, for example, if you are trying to show traffic by unique IP over time and you want to graph each unique IP traffic over time, one line for each unique IP.

Apache ORC

This release adds support for loading Apache ORC files. This feature was implemented by a first-time OSS contributor!

SQL queries on nested arrays

Before this release, you couldn't run DM_getPanel() in a database panel if the panel you wanted to reference was not an array of objects. With this release you can pass a second parameter which is a "path" to a nested array within the panel's result.

For example if a panel returns {"data": [{"a": 1}, {"b": 2}]}, you can now run a SQL query on that: SELECT * FROM DM_getPanel('0', 'data') and it will return [{"a": 1, "b": null}, {"b": 2, "a": null}].

Automatic visual panel evaluation

In this release, visual panels are re-run automatically when their settings changed *and* when their dependent panel changes. In normal circumstances you should not need to run a graph or table panel manually any more.

Custom CA certificates

Now inside of settings you can add a custom CA certificate. This certificate is automatically tried in all HTTPS requests.

Install or upgrade

Get the 0.8.0 release now!

How you can help

If you are a developer or engineering manager, install DataStation and start using it at work! Report bugs and usability issues (there are surely many). Join the Discord and subscribe to updates.

If you are an investor, get in touch and subscribe to updates.

Share

DataStation 0.8.0 is released! This release features the first major feature contributed by an external developer; includes support for Airtable and AWS Athena; includes support for text macros for importing data in any kind of panel; and much more!https://t.co/bpi5s1kF6j pic.twitter.com/37NAUoAmPg

— Multiprocess Labs (@multiprocessio) March 15, 2022