Full-stack observability in Grafana Cloud: How to investigate issues across services and infrastructure

A quick jump into the database’s logs reveals the root cause. The database has too many simultaneous connections and is refusing new ones, which is causing some related services to break. From there, you can begin to troubleshoot, whether that’s increasing resources or horizontally scaling additional instances.

You can also create and share shortened URLs that bring teammates directly to the same view, making it easier to collaborate during investigations.

You might notice that when you open one of the Drilldown views, some filters are already applied to surface only the most relevant data. This is because the knowledge graph is configurable and can be tailored to fit your needs. Let’s take a closer look at how these configurations work and how you can customize them for your environment.

Have it your way: customizing configurations

As shown in the example above, Drilldown is a powerful tool that helps you understand your data without learning an entirely new query language. However, pinpointing the fields and labels that are most useful across your environments can be a challenge, as every system and team may follow different conventions for structuring and emitting telemetry.

There are default configurations that control how the knowledge graph filters, narrows down, and correlates your observability data with the entities in your environment. These configurations cover common scenarios by mapping labels such as pod, namespace, and cluster, along with standard OpenTelemetry fields like service.name and service.namespace for logs. 

There are many potential setups: different labeling strategies, OpenTelemetry or non-OpenTelemetry, internal conventions, and more. This results in an almost endless number of possible scenarios.

Instead of trying to support every configuration out of the box, we empower users to customize their own experience within the knowledge graph.

Creating and editing a configuration

You can create configurations for specific environments, apply them only to certain entity types, or define matchers based on entity properties. You can even configure them to query any base data source you choose. 

Take the following example (shown in the GIF below), which shows a new configuration being created to map the entity property deployment.environment to the log label service_namespace, and the entity property service to the log label service_name. Furthermore, filters ensure this configuration is applied only to entities whose deployment environment starts with prod. This could represent a real scenario in which your production metrics use deployment_environment, while your logs only include service_namespace.