Deepgram deepens Amazon SageMaker AI observability with Enhanced Metrics

Self-hosted speech AI has historically carried an observability trade-off. The service can tell you an endpoint is up and how many requests it served. The questions that actually drive capacity planning and cost management stay locked inside the vendor’s container: what you are billed for, which features your traffic uses, and what the inference engine is doing on each GPU.

If you run Deepgram’s speech-to-text (STT) and text-to-speech (TTS) models on SageMaker AI, audio and transcripts stay inside your own AWS account. This can help support your data residency and compliance efforts without giving up a managed control plane for deployment, scaling, and monitoring. Your specific obligations depend on your own controls and assessments, so consult your compliance team and review the AWS shared responsibility model.

Deepgram is closing the gap on billing, feature usage, and engine behavior with the following two innovations, available today on Deepgram SageMaker AI deployments.

In this post, we walk through both capabilities: how they work, what they show you that you couldn’t see before, and how to start using them on a Deepgram SageMaker AI endpoint.

Deepgram on Amazon SageMaker AI

Deepgram’s speech models are available as model packages on AWS Marketplace and deploy as SageMaker AI real-time endpoints in your account. With SageMaker AI, you get baseline observability on your endpoints automatically: invocation metrics such as ConcurrentRequestsPerModel and FirstChunkLatency in CloudWatch, container logs in Amazon CloudWatch Logs, and alarm-driven automatic scaling.

AWS Marketplace model packages run with network isolation. The container cannot make outbound connections, which is why security-conscious customers choose this deployment model. The same isolation, however, is what normally makes vendor-level telemetry hard. The container can’t push metrics to an external collector, and you can’t see inside the container. Both capabilities in this post work within that constraint: neither requires the container to open a network path, and both land the data in your own CloudWatch account.

A note on terminology: SageMaker AI has its own endpoint feature called enhanced metrics (EnableEnhancedMetrics in MetricsConfig), which adds per-instance and per-GPU utilization dimensions to the standard endpoint metrics. The SageMaker AI feature is distinct from and complementary to Deepgram Enhanced Metrics, the Deepgram-published billing and usage metrics described in the following sections. This post covers the Deepgram capability. For the SageMaker AI capability, see Deepgram’s SageMaker observability guide.

Innovation 1: Billing and usage transparency with Deepgram Enhanced Metrics

Deepgram Enhanced Metrics answer two questions no standard endpoint metric can: exactly what you’re being billed for, and how your traffic actually uses Deepgram’s features.

The Deepgram container writes CloudWatch Embedded Metric Format (EMF) records to container stdout. SageMaker AI already forwards container output to the endpoint’s CloudWatch log group, and CloudWatch Logs extracts EMF records into metrics automatically. The EMF path gives you the following benefits.

All dimensions are low-cardinality and contain no personally identifiable information (PII): no transcripts, TTS input, or per-request identifiers.

Deepgram Enhanced Metrics aggregate across the Deepgram endpoints in your AWS account and Region. The dimensions cover category, model, and transport, but not endpoint name or instance ID, so you cannot filter this stream to a single endpoint or instance. For per-endpoint, per-instance, or per-GPU breakdowns, use the Prometheus and OpenTelemetry metrics in the next section.

The billing namespace emits one record per completed request, covering each streaming session, pre-recorded request, and TTS request. The ConsumedUnits metric carries the same billable-unit values that drive AWS Marketplace metered billing. The following table lists the metrics this namespace publishes.

Dimensions are published at three granularities ([Category], [Category, Model], and [Category, Model, Transport]), so you can answer what did streaming STT cost this month and how much of that was nova-3 with the same namespace. For example, total consumed units per hour for streaming speech-to-text:

SampleCount is the number of billed requests. Because these are ordinary CloudWatch metrics, you can build a finance-facing billing dashboard or set a budget alarm on consumed units.

A second stream, emitted by the Deepgram API server itself, breaks raw usage down by method, model tier, and enabled feature, independent of billing. It answers product-level questions: how much traffic is streaming versus pre-recorded, which model tiers process the most audio, and which features (diarization, smart formatting, redaction, keyterm prompting) your applications actually enable. The following table lists the metrics in this namespace and the dimensions each one carries.

For example, how much audio ran with diarization enabled in the last 24 hours:

The usage stream is on by default and can be disabled with a single environment variable override in the endpoint configuration (DEEPGRAM_API_01: emf.enabled=false). The billing stream cannot be disabled. It is part of the metering pipeline.

Innovation 2: Engine-level and per-GPU visibility with Prometheus and OpenTelemetry

Deepgram containers serve a Prometheus metrics endpoint, and SageMaker AI detailed observability runs an AWS managed OpenTelemetry Collector on each instance backing the endpoint. The collector scrapes the container and exports the results to CloudWatch, and it exports per-GPU and host-level metrics.

With detailed observability enabled, three metric sources publish to the CloudWatch OTel-compatible metric store. The following table describes each source and why it matters.

engine_estimated_stream_capacity is the Deepgram engine’s own estimate of how many concurrent streams the instance can sustain. Comparing it against engine_active_requests gives you an engine-reported headroom signal for scaling decisions.

Every series carries SageMaker resource labels, including aws.sagemaker.endpoint.name, the variant name, and the instance ID. You can filter to a single endpoint, isolate one instance in a scaled-out fleet, or compare GPUs within an instance. Because the collector runs on the host, outside the model container, this works under AWS Marketplace network isolation.

Detailed observability is on by default for newly created endpoints, publishing every 60 seconds. To set it explicitly, for example on an endpoint created before the feature launched, or to change the publish frequency, use MetricsConfig on the endpoint configuration:

For an endpoint already serving traffic, create a new endpoint configuration and run update-endpoint. The update is a blue/green deployment, and the endpoint stays in service.

The metrics land in the CloudWatch OTel metric store and are queried with PromQL. In the CloudWatch console’s PromQL editor, filter per-GPU utilization to one endpoint:

CloudWatch also exposes a standard Prometheus-compatible HTTP API for these metrics, authenticated with SigV4:

If you’ve standardized on Grafana or another Prometheus-native observability stack, you can point your existing tooling at your SageMaker AI fleet: no parallel monitoring stack, no export pipeline.

Choosing the right metric stream

The two innovations complement the built-in metrics of SageMaker AI rather than replacing them. The following table maps common questions to the metric stream that answers them.

Deepgram Enhanced Metrics require no setup: they flow into your CloudWatch account from the moment a Deepgram SageMaker AI endpoint serves traffic. Detailed observability is on by default for new endpoints and takes one endpoint-configuration change to add to existing ones. To get started, use the following resources.

Self-hosting speech AI no longer leaves a gap between the service’s request counters and the vendor’s invoice. Deepgram Enhanced Metrics land the exact units that drive AWS Marketplace billing in your own CloudWatch account, with zero infrastructure. The feature-level usage behind those units lands there too. With Prometheus and OpenTelemetry support through SageMaker AI detailed observability, the Deepgram engine’s own capacity and load metrics and per-GPU utilization are one PromQL query away in the tools your team already uses.

These capabilities are Deepgram’s latest investment in SageMaker AI as a preferred deployment option for self-hosted speech AI, following IAM Temporary Delegation for support access. They continue the commitment of making the day-two experience match the day-one experience. Deepgram models currently listed on SageMaker AI, including Nova, Flux, and Aura-2, come with a 14-day trial at no additional cost. Your team can stand up a real deployment in your own AWS account before committing. Running Deepgram models on SageMaker AI incurs charges for endpoint hosting, including GPU instances. It also incurs charges for Amazon CloudWatch logs and metrics and associated networking resources. While the Deepgram trial is offered at no additional cost, AWS infrastructure costs apply from the start of deployment. Review the SageMaker AI pricing and use AWS Cost Explorer to monitor your spending.