One forward pass running a Candle classifier inside a Rust service, batched

Load the weights once, check shape and dtype at the door, and score a batch in one forward pass.

The problem

Your Rust API receives support tickets, and each one has to land in one of three queues. Someone on the team has already trained a small model that does this: four numbers go in (features computed from the ticket) and three scores come out, one per queue. The question is where that model runs.

The usual answer is a separate Python service. That adds a network hop to every request, a second thing to deploy and monitor, and a second language runtime. For a model this small, the service around the model can end up costing more than the model.

Running the model inside your Rust process removes all three costs. The catch is that a naive in-process version wastes most of the CPU time on overhead. One naive version reloads the weights file on every request. Another runs the model once per row in a loop. Each call has a fixed setup cost, and when the arithmetic is as small as four numbers times three outputs, that fixed cost dominates. The lab below measures how much: scoring 4096 rows one at a time takes about 75 times longer than scoring them together, with identical results.

The version below loads the weights once and batches the rows, using Hugging Face Candle 0.11.0 on CPU.

The idea, in plain words

A trained model is a set of named arrays of numbers, called tensors, saved to a file. Running the model on new data (inference) is arithmetic on those arrays. Candle is a Rust crate that gives you a tensor type and ready-made layers, and its README describes it as a "Minimalist ML framework for Rust". So the service reads the arrays from the file once at startup, keeps them in memory, and multiplies each request's numbers through them. Batching changes the shape of that arithmetic. Many tiny multiplications become one large one, so the fixed cost of a call is paid once for the whole batch instead of once per row.

Concept A: tensors, with shape, dtype and device

A tensor is an n-dimensional array. It has three properties you will see in every Candle error message:

• shape (also called dims): the size along each dimension. A 3-by-2 grid of numbers has shape [3, 2].
• dtype: the element type, such as F32 (32-bit float) or F64 (64-bit float).
• device: where the memory lives. Device::Cpu is ordinary RAM. A GPU is available only when Candle is built with the cuda or metal feature, and those are optional features of candle-core.

You need to know all three because Candle checks them for you instead of guessing. Shape tells it whether two tensors can be multiplied. Dtype tells it what arithmetic to run. Device tells it where the bytes are. When two tensors disagree on any of these, the operation returns an error you can read, as the common mistake at the end of this section shows.

The first lab test builds a small tensor and prints those properties:

#[test]
fn t1_tensor_has_shape_dtype_device() {
    let t = Tensor::new(&[[1f32, 2.0], [3.0, 4.0], [5.0, 6.0]], &Device::Cpu).unwrap();
    println!("shape={:?} dtype={:?} device={:?}", t.shape(), t.dtype(), t.device());
    assert_eq!(t.dims(), &[3, 2]);
    assert_eq!(t.dtype(), DType::F32);
}

The nested array has three rows of two numbers, and the f32 suffix on the first literal makes the whole thing F32. The last argument to Tensor::new is the device, here the CPU. The lab printed shape=[3, 2] dtype=F32 device=Cpu, and the two assertions confirm dims() is [3, 2] and the dtype is F32. You can read dims() as a plain slice of usize, which is handy for validating input sizes in a handler.

Two more facts help in a server. First, the Candle source documents that tensors are reference counted with Arc, so cloning them is cheap. Passing a tensor around or storing a copy in a struct does not copy the numbers. Second, choosing a device is one call. The lab binary uses Device::cuda_if_available(0), whose source in candle-core 0.11.0 is:

pub fn cuda_if_available(ordinal: usize) -> Result<Self> {
    if crate::utils::cuda_is_available() {
        Self::new_cuda(ordinal)
    } else {
        Ok(Self::Cpu)
    }
}

When CUDA is not available, it returns Device::Cpu instead of failing. The lab crate was built without any GPU feature (none of cuda, metal, accelerate or mkl was enabled), and the binary printed device = Cpu. GPU paths were not run. All timings in this lesson are CPU timings.

Common mistake: mixing dtypes. Symptom: a matrix multiply fails with dtype mismatch in matmul, lhs: F32, rhs: F64. Cause: Candle does not silently cast between element types, and a Rust float literal such as 1.0 inside an f64 array makes an F64 tensor, so one operand was F32 and the other F64. Fix: convert one side with to_dtype(DType::F32), or build both tensors from f32 data. The lab test shows both halves:

#[test]
fn t7_dtype_mismatch_is_an_error_not_a_cast() {
    let a = Tensor::new(&[[1f32, 2.0]], &Device::Cpu).unwrap();
    let b = Tensor::new(&[[1f64], [2.0]], &Device::Cpu).unwrap();
    let err = a.matmul(&b).unwrap_err();
    println!("err={err}");
    let ok = a.matmul(&b.to_dtype(DType::F32).unwrap()).unwrap();
    println!("after to_dtype: {:?}", ok.to_vec2::<f32>().unwrap());
}

Tensor a has shape [1, 2] and dtype F32. Tensor b has shape [2, 1] and dtype F64, because its first element is written as 1f64. The first matmul returns an error and the test prints the message above. After b.to_dtype(DType::F32) the shapes and dtypes agree, and the product is 1*1 + 2*2 = 5. The lab printed after to_dtype: [[5.0]]. Expect this error whenever data comes from two different sources, for example request features held as f64 in your handler and weights loaded as f32.

Concept B: a Linear layer loaded from a safetensors file

A Linear layer is the simplest building block of a neural network. The candle-nn module documentation says it applies a linear transformation y = [email protected]() + b: multiply the input x by the transposed weight matrix w, then add the bias b. The weight has shape (out_features, in_features). An input of shape (batch, in) produces an output of shape (batch, out). For our classifier that means a weight of shape (3, 4), a bias of length 3, and a batch of N tickets of shape (N, 4) becoming (N, 3) scores.

A safetensors file is a file format that stores named tensors. The model's weights are saved under names such as classifier.weight, and code that loads the file asks for tensors by those names. The lab has no real trained model, so it writes a small file by hand with candle_core::safetensors::save, which takes a HashMap of name to Tensor:

pub fn write_weights(path: &Path, dev: &Device) -> Result<()> {
    // Linear stores weight as (out_features, in_features).
    let w = Tensor::new(
        &[
            [0.5f32, -0.25, 0.0, 1.0],
            [0.1, 0.2, 0.3, 0.4],
            [-1.0, 0.0, 0.5, 0.25],
        ],
        dev,
    )?;
    let b = Tensor::new(&[0.0f32, 0.1, -0.1], dev)?;
    let mut map = HashMap::new();
    map.insert("classifier.weight".to_string(), w);
    map.insert("classifier.bias".to_string(), b);
    candle_core::safetensors::save(&map, path)
}

The weight is a 3-by-4 grid, one row of four numbers per output class, so its shape is (out, in) = (3, 4). The bias has one number per class. The two map keys are the names the loader will look for later, and save writes both tensors to path. In a real deployment, a training script would produce this file and your service would only read it. The point of the lab version is that the names and shapes in the file are the contract between training and serving.

Loading is the part that runs once at startup:

pub struct Scorer {
    head: Linear,
    device: Device,
}

impl Scorer {
    /// Load once at startup; reuse for every request.
    pub fn load(path: &Path, prefix: &str, device: &Device) -> Result<Self> {
        let vb = unsafe { VarBuilder::from_mmaped_safetensors(&[path], DType::F32, device)? };
        let head = candle_nn::linear(IN, OUT, vb.pp(prefix))?;
        Ok(Self { head, device: device.clone() })
    }
}

VarBuilder::from_mmaped_safetensors memory-maps the file, which means the operating system maps the file's bytes into the process instead of the code reading them through a copy loop. It is an unsafe fn; the Candle documentation says "The unsafe is inherited from memmap2::MmapOptions", so treat the weights file as read-only while the service runs (general memory-map practice; the lab did not test a file changing under the map). The second argument, DType::F32, is the element type the tensors are loaded as, and the third is the device. vb.pp("classifier") returns a builder that adds the prefix classifier to every name it looks up. Then candle_nn::linear(IN, OUT, vb) asks that builder for "classifier.weight" with shape (out_dim, in_dim) and "classifier.bias" with shape out_dim. With a file-backed builder the random-initialisation hints in that function's source are not used, and the names and shapes are what get looked up. The returned Scorer owns the layer and is built once, then shared for every request.

To see what the layer computes, take the lab's test t2 with input [1, 2, 3, 4]. Each output is a row of the weight dotted with the input, plus that class's bias:

• Row 0, [0.5, -0.25, 0, 1]: 0.5 - 0.5 + 0 + 4 = 4.0, plus bias 0.0 gives 4.0.
• Row 1, [0.1, 0.2, 0.3, 0.4]: 0.1 + 0.4 + 0.9 + 1.6 = 3.0, plus bias 0.1 gives 3.1.
• Row 2, [-1, 0, 0.5, 0.25]: -1 + 0 + 1.5 + 1 = 1.5, plus bias -0.1 gives 1.4.

The lab printed logits=[4.0, 3.1, 1.4], which matches. These raw scores are called logits. They can be any size, positive or negative, and they do not sum to anything useful.

Softmax turns a row of logits into probabilities that sum to 1, so the largest score gets the largest probability. We want that because a router wants "67% queue 0" more than "4.0". The candle-nn source is short:

pub fn softmax<D: candle::shape::Dim>(xs: &Tensor, dim: D) -> Result<Tensor> {
    let dim = dim.to_index(xs.shape(), "softmax")?;
    let max = xs.max_keepdim(dim)?;
    let diff = xs.broadcast_sub(&max)?;
    let num = diff.exp()?;
    let den = num.sum_keepdim(dim)?;
    num.broadcast_div(&den)
}

It finds the largest value along the chosen dimension (max_keepdim keeps that dimension so shapes still line up), subtracts it from every element, applies exp, sums the results, and divides each by the sum. Subtracting the max does not change the final answer, because the same factor cancels in the division. It does keep exp from overflowing when scores are large, since the biggest value after subtraction is 0. In the lab code the dimension is D::Minus1, meaning the last one, so each row is normalised on its own. Test t3 printed probs=[0.67529094, 0.2745528, 0.05015628] sum=1: the 4.0 logit gets about 68%, and the three values add up to 1.

Common mistake: wrong name prefix. Symptom: startup fails with cannot find tensor head.weight. Cause: the prefix passed to vb.pp does not match the names in the file, so the loader asked for head.weight while the file holds classifier.weight. Fix: use the names the file was saved with, and inspect them before wiring the service. This failure is good news in one respect. Test t6 shows it happens inside Scorer::load, once, before the service takes any traffic, and not on the first user request.

A second, shorter diagnostic. Symptom: a request with 5 features instead of 4 fails with shape mismatch in matmul, lhs: [1, 5], rhs: [4, 3]. Cause: the input row is [1, 5] but the transposed weight expects 4 inputs. Fix: check the feature count at the API boundary and return a 400, so malformed input never reaches the model.

Concept C: the hot path, one forward pass over a batch

The forward pass is one run of the model's arithmetic on an input. The hot path is the code that runs for every request, so its cost is multiplied by your traffic. Here are the two ways the lab scores rows:

/// One row per forward pass: the slow shape.
pub fn score_one(&self, features: &[f32]) -> Result<Vec<f32>> {
    let x = Tensor::from_slice(features, (1, features.len()), &self.device)?;
    let logits = self.head.forward(&x)?;
    let probs = candle_nn::ops::softmax(&logits, D::Minus1)?;
    probs.squeeze(0)?.to_vec1::<f32>()
}

/// The whole batch in one forward pass.
pub fn score_batch(&self, rows: &[f32], n: usize) -> Result<Vec<Vec<f32>>> {
    let x = Tensor::from_slice(rows, (n, IN), &self.device)?;
    let logits = self.head.forward(&x)?;
    candle_nn::ops::softmax(&logits, D::Minus1)?.to_vec2::<f32>()
}

Both functions do the same steps: build an input tensor from a flat &[f32], run the Linear layer, apply softmax, and copy the result back into ordinary Rust vectors. The difference is the shape. score_one makes a (1, 4) tensor, so a loop over 4096 rows repeats every step 4096 times. score_batch takes all rows laid out end to end in one slice and makes a single (n, 4) tensor, so every step runs once.

The reason the difference is so large comes from the Linear source. For a 2-D input, Linear::forward takes the default branch: it transposes the weight, calls x.matmul(&w), and then calls broadcast_add to add the bias to every row. A (4096, 4) input is therefore one matrix multiply and one add. The 4096 separate calls instead each run a matmul, an add, and a softmax made of max, subtract, exp, sum and divide, plus an input allocation and a copy back to a Vec. Every one of those steps is its own operation with its own setup cost. With only 12 multiplications per row, the setup dominates the arithmetic, and the measurements below fit that reading. Batching pays that setup once.

The same source shows Candle avoiding a slow path for you. When the input has three dimensions, such as (batch, sequence, features), and is contiguous in memory, forward reshapes it to two dimensions, runs one standard matmul, and reshapes the result back. A comment there gives the reason: "When possible, we avoid using a broadcasted matmul as it is much slower than the standard matmul for the cuda and cpu backends." The library makes the same trade you are making in the handler, one large ordinary multiply instead of many small ones.

The lab measured this in a release build with 4096 rows. The binary first runs one untimed batch as a warm-up, then times the per-row loop and the single batch call. Three runs gave identical probabilities each time and these timings:

The last column is the largest absolute difference between the two methods' probabilities, and it was exactly zero. Test t4 checks the same equivalence on two rows: score_batch returned [[0.67529094, 0.2745528, 0.05015628], [0.3777576, 0.23492181, 0.3873206]], and calling score_one on each row separately agreed within 1e-6.

Read these numbers carefully. They come from one CPU run of a toy model with 4 inputs. A bigger model does more arithmetic per row, so the fixed cost is a smaller share and the ratio will differ. Measure your own model on your own machine. Also run timings with cargo run --release, because a debug build measures unoptimized code rather than what you ship.

In a service, the practical rule is to collect the rows that belong together and score them in one call. That means all the tickets in one request, or a short window of requests gathered into one batch. The handler builds one flat Vec<f32> and calls score_batch(&rows, n) once.

Common mistake: a forward call in a loop. Symptom: the model endpoint uses far more CPU than the model size suggests. Cause: a loop calls forward once per item, so per-call overhead is paid per item. Fix: build one (n, features) tensor and call forward once.

Putting it together

Each lab test proved one thing the service depends on:

In an HTTP handler setup, you load the Scorer once at startup and keep it in shared state, such as an Arc held in your framework's application state. Each handler checks that every row has 4 features and returns a 400 otherwise, then packs the rows and calls score_batch. As general guidance, which the lab did not run: inference is CPU-bound work, so in an async server it should not block the async runtime's worker threads for long.

The limits of this lab, all of them not run:

• No GPU. The cuda and metal features were not built, so every number here is CPU.
• No real trained model. The weights were written by hand to make the arithmetic checkable.
• No quantization or half precision. Everything was F32.
• No comparison with Burn. Burn is another Rust ML framework, and it was not tried.

How to run it

Tested with rustc 1.99.0, candle-core 0.11.0, candle-nn 0.11.0, CPU. Add these two dependency lines to Cargo.toml:

candle-core = "=0.11.0"
candle-nn = "=0.11.0"

Run the tests, then the timing binary in release mode:

cargo test --test hot_path -- --nocapture --test-threads=1
cargo run --release

The test run ends with this line:

test result: ok. 7 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

Check yourself

1. What three properties does every Candle tensor carry, and which one does to_dtype change? *Answer: shape, dtype and device. to_dtype changes the dtype, for example F64 to F32.*

1. Why does the matmul of an F32 tensor and an F64 tensor fail instead of working? Answer: Candle does not silently cast between element types, so you convert one side or build both from the same type.

1. For a Linear layer with 4 inputs and 3 outputs, what shape is the weight, and what shape is the output for a batch of 10 rows? *Answer: the weight is (3, 4), which is (out_features, in_features). The output is (10, 3).*

1. A service fails at startup with cannot find tensor head.weight. What is the likely cause, and why is it better than failing on a request? *Answer: the vb.pp prefix does not match the names saved in the file. It fails once at load, before traffic, not on the first request.*

1. Why was one forward pass over 4096 rows about 75 times faster than 4096 single-row passes, and what caveat applies to that number? Answer: the batch pays the per-call setup cost once, as one matrix multiply, instead of once per row. The caveat is that it is a toy 4-input model on one CPU, so a larger model will show a different ratio and you should measure your own.

Sources

• Candle repository and README, first line "Minimalist ML framework for Rust": https://github.com/huggingface/candle
• crates.io, candle-core 0.11.0 (package source: src/tensor.rs doc on Tensor, src/device.rs cuda_if_available, src/safetensors.rs save, Cargo.toml features): https://crates.io/crates/candle-core/0.11.0
• crates.io, candle-nn 0.11.0 (package source: src/linear.rs module doc, Linear::forward and fn linear; src/var_builder.rs from_mmaped_safetensors; src/ops.rs softmax): https://crates.io/crates/candle-nn/0.11.0