About the job
We are looking for a Staff Engineer to be the detection authority for Crusoe’s Command Center platform—the person who owns what “something is wrong” means. You will define the heuristics, thresholds, and rules that power our alerting and anomaly detection systems, translating raw infrastructure telemetry into precise, actionable signals. This is a full software engineering role. You will ship customer-facing features, build systems from 0 to 1, and scale existing services alongside a team of strong generalist engineers. What sets you apart is deep expertise in anomaly detection, heuristics, and machine/reinforcement learning—applied to real infrastructure at global scale. Beyond your technical contributions, you will be a cultural bar-raiser and force multiplier: someone who elevates the entire team’s craft.
Responsibilities
- Detection & Intelligence Ownership: Own the full detection stack — heuristics, threshold calibration, precision/recall tuning, and the rule systems that define what "something is wrong" means for the platform.
- Anomaly Detection Pipelines: Design and maintain detection systems including straggler node detection, GPU health signals, and fleet-level behavioral baselines.
- Signal Calibration: Drive detection fidelity by reducing false positives, increasing signal coverage, and building feedback loops that keep thresholds accurate as the fleet grows.
- ML/RL Integration: Evaluate and integrate machine learning and reinforcement learning techniques where they outperform rule-based approaches — and know when not to reach for a model.
- Product Engineering: Ship customer-facing features end-to-end across the CCIA stack — alert rule engine, control plane APIs, automated action systems, and insights delivery surfaces.
- 0-to-1 & Scale: Build new systems from scratch and scale existing ones to support Crusoe's rapidly growing global fleet.
- Cross-Functional Collaboration: Work closely with product counterparts to shape requirements early and partner with the data science team to develop and validate detection models.
- System Design: Participate in design discussions across teams, contribute architectural perspective, and help evaluate technical trade-offs.
- Technical Mentorship: Mentor engineers at all levels through code review, design feedback, and direct coaching, and contribute to hiring by helping define what great looks like.
Qualifications
Minimum
- Anomaly Detection & Heuristics Expertise: Deep experience building anomaly detection systems, heuristics-based rule engines, or ML/RL systems for infrastructure or data-intensive domains.
- Threshold & Signal Calibration: Demonstrated ability to reason about precision/recall trade-offs and build feedback loops that keep detection systems accurate over time.
- Distributed Systems Fundamentals: Strong foundations in the building blocks of reliable, scalable backend systems—you can hold your own in any system design conversation.
- Full Software Engineering Craft: 5+ years shipping production software; experience with modern compiled or systems languages (Go, Rust, C++, Java, or similar).
- Data & Observability Fluency: Comfortable with time-series data, telemetry pipelines, and observability primitives—you understand how raw metrics become actionable insights.
Preferred
- Experience with GPU profiling tools (Nsight, NCCL Inspector) or hardware-level infrastructure diagnostics.
- Background in observability platforms or products.
- Experience with reinforcement learning applied to operational or infrastructure problems.
- Familiarity with large-scale fleet management or cloud infrastructure.
- Passion for building team culture and engineering quality of life.