Introducing the Rerun Blueprint APIs

Written by Jeremy Leibs 6 months ago

Developers in fields like robotics, spatial computing, and finance use Rerun to understand their systems by visualizing the multimodal data they process and produce over time. The details of how that data is displayed are just as important as the data itself, but have mainly been controllable through the UI until now.

In Rerun, the specification of which parts of the data to show, where to show them, and how, is called Blueprint. With the new Blueprint APIs in Rerun 0.15 you can define the layout and contents of views directly from code. You can also save and load Blueprints to and from (.rbl) files. This unlocks new workflows and makes it easier to share repeatable visualizations.

For example, this video shows live coding of how the output of a face tracking algorithm is shown, as the algorithm is running.

The ability to seamlessly control the blueprint both from the UI and from code has been in the works for close to a year, and was a motivating factor behind the API overhaul we did back in 0.9. Over the last 6 releases, we have been replacing the core of the visualizer with structured blueprint types to make this release possible.

If you want to start playing around with controlling blueprints from code right away, check out the blueprint tutorial. If you want to read up a bit more let’s dive in.

Rerun Viewer = Data recording x Blueprint rerun-viewer--data-recording-x-blueprint

In Rerun there are two separate pieces that combine to produce what you see: the "recording" and the "blueprint". The recording is constructed by logging multimodal data with rr.log(), and provides the actual data you are visualizing. The blueprint is the configuration that determines how the data from the recording is displayed and defines things like the grouping and layout of containers, and what data is included in which view.

Until now, the default blueprint for any recording has been created by a heuristics engine that looks at the data in the recording and tries to guess a good blueprint. If you wanted anything other than that, you would have to configure the viewer to your liking in the UI manually. There was no way to share blueprints, and making edits by hand was a lengthy and error-prone process.

The declarative Python blueprint APIs the-declarative-python-blueprint-apis

The 0.15 release introduces blueprint APIs to the Python SDK. Let's dive into some code to see how it works.

Defining a blueprint is as simple as instantiating a collection of containers and space-views.

import rerun.blueprint as rrb

my_blueprint = rrb.Blueprint(
    rrb.Horizontal(
        rrb.Spatial3DView(origin="reconstruction"),
        rrb.Spatial2DView(origin="video")
    ),
    collapse_panels=True,
)

Above, rrb.Spatial3DView(origin="reconstruction") defines a 3D view with its spatial origin at the entity path reconstruction. What's important but hidden here is the default argument content=$origin/**, which means the view will query for any visualizable data under its origin. These queries can become much more specific and fine grained so you can control exactly what data is shown where.

rrb.Horizontal unsurprisingly defines a container that lays out its contents horizontally, and rrb.Blueprint is the base blueprint, that e.g. allows you to indicate that you want all panels collapsed.

You can use this blueprint to spawn a new Rerun viewer with it as the default blueprint instead of one generated by the heuristics engine.

my_blueprint.spawn("my_application")

You can also send the blueprint directly to a viewer you have connected to.

rr.send_blueprint(my_blueprint)

Programmatic layout of blueprints programmatic-layout-of-blueprints

Because blueprints can be constructed directly via the API, it’s possible to construct blueprints dynamically to better suit the needs of your application.

This example creates a grid of views by iterating over dates and stock symbols in a dataset:

blueprint = rrb.Blueprint(
    rrb.Vertical(
        contents=[
            rrb.Horizontal(
                contents=[
                    rrb.TextDocumentView(
                        name=f"{symbol}",
                        origin=f"/stocks/{symbol}/info",
                    ),
                ]
                + [
                    rrb.TimeSeriesView(
                        name=f"{day}",
                        origin=f"/stocks/{symbol}/{day}",
                    )
                    for day in dates
                ],
                name=symbol,
            )
            for symbol in symbols
        ]
    ),
)
rr.send_blueprint(blueprint)

If you want to learn more about how to use the new APIs, check out this guide.

Blueprints can be saved and shared blueprints-can-be-saved-and-shared

Writing Python code isn’t the only way to work with blueprints. If you have created a layout that’s valuable to you or your team, you can now save it through the Rerun file menu as an .rbl file.

Now when you open it again whilst running your recording it restores your layout.

You can swap out either blueprint or recording you-can-swap-out-either-blueprint-or-recording

The blueprint and the recording are loosely coupled. This means that either can be changed independently of the other as long as they share the same application_id. Keeping the blueprint constant while changing the recording will allow you to compare different datasets using a consistent set of views. On the other hand, changing the blueprint while keeping a recording constant will allow you to view the same data in different ways.

More coming soon more-coming-soon

This is just the beginning of our expanded support for blueprints.

Although we started with Python to prove out the concepts and get users access to these features quickly, we plan to implement comparable APIs in C++ and Rust.

Additionally, there are still many aspects of space-view configuration such as visible-time-range configuration, styling, and property overrides that are not yet exposed via the blueprint APIs. In future releases we will be working to ensure that any change you can make through the viewer UI can also be defined through a corresponding API.

We’re extremely excited for all the new use-cases this functionality will unlock and can’t wait to see what you will build with it! Join us on Github or Discord and let us know what you have built or hope to see from us next!