Staff Software Engineer, AI Inference Gateway

Our mission is to democratize cloud computing

by utilizing latent consumer resources to power a sustainable, affordable, environmentally friendly cloud for everyone.

salad.com/about

Staff Software Engineer, AI Inference Gateway

Salad Technologies · Remote · Full-time

The role

SaladCloud runs inference on thousands of geodistributed workstation and consumer GPUs — NVIDIA and AMD — that nobody else can use. Sitting in front of that fleet is a unique AI Gateway written in Rust, built on Pingora and WireGuard, that serves real-time and batch inference and embeddings to customers as a single, reliable API. You'll own it.

This is an infrastructure role with real autonomy. You'll design and ship the systems that decide how requests are fulfilled, when to add or drop capacity, and how to bill for every token — and you'll be the person who knows whether a new model, quantization, or GPU is actually worth running.

What you'll do

• Own the AI Gateway system end to end: request routing, streaming, batch/async job handling, and horizontal scaling across multiple gateway servers • Build and tune the fleet-efficiency algorithms — autoscaling nodes, scoring performance, and evicting underperformers so the network converges on peak throughput per dollar • Operate and scale the underlying inference nodes running vLLM, llama.cpp, and similar servers • Design and maintain OpenTelemetry-based observability across the gateway and the fleet; pay-per-token and subscription billing already…