Experiments in crossfiltering and linked selections over Zarr, with DataFusion
I ran an experiment to see how Zarr would fit into a query-coordinated linked visual analytics system, using DataFusion running in WASM as the query engine behind it. Disclaimer: this is far from an attempt to build a Zarr SQL engine, but more of a cross-filtering and linked selections experiment.
For a dataset, I picked ECMWF’s ensemble forecast. This weather model runs around fifty times with tiny tweaks to the starting conditions each time. So instead of one predicted temperature per grid cell, you get fifty and the spread between them tells you how confident (or not confident) the forecast really is.
That data is streamed from dynamical.org’s public Zarr store using zarrita. I’m only pulling a central Europe crop, centered on Switzerland. It’s queried with DataFusion and rendered with @developmentseed/deck.gl-zarr, with Mosaic handling the crossfilter across the charts and the map. I wanted to be able to visually explore things like “where are the ‘warm’ cells at lead N” and get an answer on the map instead of scrolling through many raster frames by eye. Dragging the lead slider and clicking the “warm” category do just that and hovering the map does the reverse by narrowing the charts down to the cells under the cursor (radius).
Final result: https://mosaic-datafusion-zarr-deckgl.netlify.app/
Loading the grid
The grid is read as Arrow IPC (via flechette) and registered as a table with id, time_index and temperature columns. The full grid would be far too much for a browser tab so I only load the central Europe crop.
This step needed 2 patches to the published DataFusion WASM bindings, which I forked at dzole0311/datafusion-wasm-bindings. The published bindings only return query results as printed text or JSON so I added execute_ipc, returning an Arrow IPC stream the charts can consume directly. Also, CREATE TABLE ... AS crashed a few times with a missing Tokio runtime error so I added register_ipc and materialize_table, which collect the result on the current thread and register it as an in-memory table.
From there, I derive two more tables. One unpacks the row id into x/y, lon/lat, cell area and a temperature category. And the other (queried by every chart in the video above), is rematerialized on each slider move by joining the requested lead (how many hours ahead the forecast is) onto the first table. Maybe a view would be more natural fit, but each interaction fires several queries so materializing means the join runs once per lead change instead of once per query.
Connecting all this to Mosaic
Looking into Mosaic’s code, all it needs from a connector is basically a query({ type, sql }) method so the DataFusion wrapper doubles as that connector. The crossfilter Selection is shared across the chart brushes, the category toggle and the map’s hover radius.
Mosaic’s DuckDB assumptions show up in a few places. Schema checks expect DuckDB type names, so the connector translates DataFusion’s answer (Int32 to INTEGER, Utf8 to VARCHAR etc). I also turned off pre-aggregation since it cached on SQL text when the underlying tables get rematerialized underneath it, which would serve stale rows after a chunk or lead change. It also builds its cache tables with CREATE TABLE ... AS, the same statement that crashed the WASM build. Intercepting that call in the connector and routing it through materialize_table is on my list to try out but I think longer term this belongs in Mosaic itself.
Selections to pixels
Every selection change runs one query:
SELECT id FROM cells_current_lead WHERE ...The returned ids fill a mask uploaded as a GPU texture and a deck.gl shader module discards the unselected raster pixels so the map updates. The same predicate also filters the other charts and updates the mean temperature and selected area numbers in the sidebar. Brushing the map works the same way but in reverse, so the hover radius builds a lon/lat predicate and publishes it into that same shared selection.
And also, everything streams! So the first chunk shows up on screen after one 3 MB fetch instead of the whole crop. Each later chunk re-registers the table, rematerializes the derived views and requeries the Mosaic clients so the charts fill in as more and more data arrives.
Worth noting that none of this is really WASM-specific and people usually run DataFusion server-side, not in a browser. There is no official WASM support bundle, but there is a contributor-maintained playground and the bindings I forked, both under the datafusion-contrib org. But with a decent SQL dialect translation layer, the same connector could sit in front of a server-side deployment instead.