Building Fast AI CLI Tools with Rust and Ollama on Ubuntu

Building developer command-line tools with Python is convenient, but 300ms interpreter startup delays and virtual environment requirements can frustrate terminal users. Rust offers instant startup, static binaries, and memory safety without a garbage collector. Pairing Rust with a local Ollama daemon gives you blazing fast AI utilities that run completely offline.

In this guide, we construct a real CLI tool in Rust called ai-explain that accepts piped terminal output or git diffs, sends them to local Ollama, and streams the explanation directly to stdout.

Direct Answer: Why Rust for Local AI CLI Tools?

Rust compiles down to a single self-contained binary with a cold startup time under 8 milliseconds. For interactive shell workflows (such as shell aliases, git hooks, or pipe filters), Rust provides instantaneous responsiveness compared to the 250ms to 450ms required by Python virtual environments.

Prerequisites and Toolchain Setup

Ensure you have Rust and Cargo installed via rustup on your Ubuntu system:

# Install Rust toolchain
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
source "$HOME/.cargo/env"

# Verify compiler version
rustc --version
cargo --version

Initializing the Cargo Project

Create a new Cargo binary project and configure the required HTTP and streaming dependencies:

cargo new ai-explain --bin
cd ai-explain

Add tokio, reqwest, and serde to your Cargo.toml:

[package]
name = "ai-explain"
version = "0.1.0"
edition = "2021"

[dependencies]
tokio = { version = "1.38", features = ["full"] }
reqwest = { version = "0.12", features = ["json", "stream"] }
serde = { version = "1.0", features = ["derive"] }
serde_json = "1.0"
futures-util = "0.3"

Implementing the Streaming Client

Open src/main.rs and add the following asynchronous implementation:

use futures_util::StreamExt;
use reqwest::Client;
use serde::{Deserialize, Serialize};
use std::io::{self, Read, Write};

#[derive(Serialize)]
struct OllamaRequest {
    model: String,
    prompt: String,
    stream: bool,
}

#[derive(Deserialize)]
struct OllamaResponse {
    response: String,
    done: bool,
}

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let mut input = String::new();
    io::stdin().read_to_string(&mut input)?;

    if input.trim().is_empty() {
        eprintln!("Usage: cat log.txt | ai-explain");
        std::process::exit(1);
    }

    let client = Client::new();
    let prompt = format!("Explain this error or diff concisely in 3 bullet points:\n\n{}", input);

    let req_body = OllamaRequest {
        model: "llama3.2:3b".to_string(),
        prompt,
        stream: true,
    };

    let mut response = client
        .post("http://localhost:11434/api/generate")
        .json(&req_body)
        .send()
        .await?
        .bytes_stream();

    let mut stdout = io::stdout();

    while let Some(chunk) = response.next().await {
        let bytes = chunk?;
        if let Ok(item) = serde_json::from_slice::<OllamaResponse>(&bytes) {
            print!("{}", item.response);
            stdout.flush()?;
            if item.done {
                break;
            }
        }
    }

    println!();
    Ok(())
}

Compiling for Release and Installing Locally

# Build optimized release binary
cargo build --release

# Copy binary to local user path
cp target/release/ai-explain ~/.local/bin/

# Test piping terminal output directly
git diff | ai-explain

The resulting binary is only 8.4 MB, requires zero external runtime libraries, and streams tokens immediately from your local Ollama daemon.

For more local AI development workflows, check our guide on Setting Up Continue.dev with Ollama in VS Code on Linux and System Prompt Templates for Generating Safe Linux Bash Scripts.