AI Research Engineer - Datadog AI Research (DAIR)

The role

This role supports research scientists at Datadog AI Research by turning experimental ideas into functioning systems. The work centers on two tracks: multimodal models of distributed system behavior and post-trained agents that act on infrastructure incidents. The position is based in Paris and follows a hybrid arrangement.

What you would do

• Create and maintain multimodal data pipelines, evaluation harnesses, benchmarks, and internal tooling
• Run large-scale experiments and tune models for stability, speed, and compute cost
• Construct simulation and replay environments used to train and assess agents
• Coordinate distributed training and reinforcement learning jobs on Ray, covering scheduling and recovery from failures
• Define automated regression tests for forecasting quality, agent behavior, and simulator accuracy
• Work with scientists, product, and engineering teams to ship research results into products and production services
• Publish research and produce maintainable code, documentation, and open-source releases

What they asked for

• Solid background in distributed systems, reinforcement learning infrastructure, or large-scale ML training and serving
• Strong Python skills plus working knowledge of a lower-level language such as Rust, C++, or Go
• Hands-on experience running ML training or inference stacks in PyTorch or JAX, including containers, orchestration, and GPU use
• Practical exposure to large model fine-tuning and post-training methods such as SFT, RLVR, or RLHF
• Ability to communicate design and performance tradeoffs to both technical and non-technical listeners
• Prior involvement in research papers or academic contributions

Nice to have

• Experience with Ray, Slurm, or comparable schedulers
• Familiarity with training frameworks such as Megatron-LM, DeepSpeed, or TorchTitan
• Background in observability, SRE, or security software
• Track record moving research prototypes into customer-facing products
• CUDA or GPU kernel optimization experience
• Production data pipeline development
• Experience building simulators or sandboxes for agent training