Responsibilities:
Own the end-to-end observability architecture, including edge collection, transport, ingestion, storage, querying, visualization, and alerting across metrics, logs, and traces.
Design and operate the edge telemetry layer, including OpenTelemetry Collector agents running alongside brokers at thousands of sites, managing their configuration, buffering, rollout processes, and version lifecycle.
Build and operate the centralized backend on Azure Kubernetes Service (AKS), including time-series storage, log storage, trace storage, and a Grafana-based (or equivalent) visualization and alerting stack.
Evaluate, select, and justify the metrics backend (e.g., VictoriaMetrics, Grafana Mimir, or Azure Monitor Managed Service for Prometheus) based on trade-offs involving cost, scalability, retention, and operational complexity.
Design the telemetry ingestion and networking architecture for fleet-scale deployments using private connectivity, ensuring the ingestion layer remains healthy under aggregated load.
Treat telemetry cardinality and cost optimization as core engineering concerns by developing relabeling, sampling, retention, and rollup strategies that keep the platform performant and cost-efficient as the fleet grows.
Define everything as code using Terraform for infrastructure, Helm and GitOps for application and stack delivery, and GitHub Actions-based CI/CD pipelines for safe and repeatable deployments.
Build the alerting and SLO framework with particular focus on signal-to-noise ratio at fleet scale—consolidating scenarios where many identical sites degrade into actionable alerts instead of generating thousands of notifications.
Implement observability for the observability platform itself, detecting silent failures where a site stops reporting and continuously verifying pipeline health end-to-end.
Establish retention policies, upgrade procedures, runbooks, and on-call practices for the platform, while participating in the on-call rotation.
Collaborate closely with Messaging, Networking, Security, and Application teams, mentoring engineers and raising the overall standard of reliability, automation, and operational excellence.
Technical Environment
Cloud & Infrastructure:
Microsoft Azure
Azure Kubernetes Service (AKS)
Site-to-Site VPN
ExpressRoute
Azure Networking
Azure Storage solutions
Microsoft Entra ID and Managed Identities
Azure Key Vault
Observability Stack:
Prometheus ecosystem and PromQL
Grafana (including Azure Managed Grafana)
Metrics time-series databases such as:VictoriaMetrics
Grafana Mimir
Thanos
Azure Monitor Managed Prometheus
Loki for logs
Tempo (or Jaeger) for distributed tracing
Telemetry Pipeline:
OpenTelemetry Collector
Remote Write architecture
Metrics, logs, and traces collection at the edge
Containers & Delivery:
Kubernetes
Helm
GitOps (Argo CD or Flux)
Terraform
CI/CD pipelines
Programming & Automation:
Python (preferred)
Bash
Domain Context:
Event-driven messaging systems
Solace PubSub+
Kafka
RabbitMQ
Comparable message broker technologies
