Exploring multidimensional Zarr datasets with a 3D forecast cube and SQL

I’ve been working on a 3D forecast cube that shows spatial and temporal dimensions together in the same view with data loaded directly from a Zarr store into the browser. There are many approaches, with lots of literature and open source libraries for visualizing multidimensional data. But I wanted to try a combination that I’ve seen less of, one that uses deck.gl to render the volume and SQL to select cells across its dimensions. The selections should stay active as I slice through the cube and inspect point forecasts, all within one visual analytics workflow. Also, using deck.gl leaves room to add other geospatial layers and views later.

The result has few example datasets, including ECMWF’s AIFS forecast from dynamical.org, with forecast time stacked vertically.

Explore the cube: Zarr-SQL-Views

Github repo: zarr-sql-views

Getting Zarr into the browser

The experiment uses Zarrita in a web worker to fetch and then decode the Zarr chunks that cover the selected region. The worker also builds the values into a volume ordered by time and latitude/longitude.

The selected region doesn’t tell me how much data I’ll download in the browser. Zarr reads happen at the chunk boundaries so even a small crop can require fetching much more data than I need. There is room to optimize both the source chunk layout and how the browser fetches those chunks.

Once the volume is loaded, I pass the values as Arrow batches to DuckDB-WASM. To query them by location and time, I create a SQL view that exposes things like latitude/longitude and forecast hour. Then its just a simple SQL query to select the cells I want:

SELECT cell_id
FROM loaded_forecast
WHERE value > 28
  AND forecast_hour BETWEEN 24 AND 72

This selects cells above 28°C and between forecast hours 24 and 72, within the loaded region. The SQL runs over the values that are already stored in browser memory.

Filtering the cube with SQL

When through the interaction I change the threshold, the forecast window or I run a custom SQL, DuckDB-WASM returns the cell_ids that match. I turn these ids into a mask with one entry per cell and pass it to the custom deck.gl volume layer.

The layer has the values in a 3D texture on the GPU and the selection mask in a separate texture. When there’s no filter applied, it draws the cube’s outer surfaces and the surfaces exposed by slicing. With a filter applied, it draws the exposed faces of matching cells so we can see where the selected values occur inside the 3D volume.

From here, any interaction is connected and each selection stays active. For example, I can apply a SQL query, then slice through the selected cells and also click a location to get time series without clearing the selection.

One caveat remains since I’m loading a bounded region at native resolution. To explore larger areas as I zoom out, I’d need coarser overviews for the example datasets and a fetching strategy that loads finer detail as I zoom in, to avoid the browser running out of memory.