Petco Simple AI Inference Server - Nutanix GPU Cluster Deployment

Custom Ubuntu ISO with Petco SimpleAI and Systemd Service

This project creates a custom Ubuntu ISO that includes the Petco Simple AI inference server pre-installed, along with a systemd service to manage it. The ISO is specifically designed for rapid deployment in a Nutanix GPU cluster environment, enabling quick setup of AI inference capabilities across your infrastructure. The SimpleAI service is configured to start automatically on boot, and it can be easily managed and customized via systemd or cloud-init.

Table of Contents

• Introduction
• Prerequisites
• Building the ISO
• Deploying the ISO
• Configuring Petco SimpleAI
• Managing the Service
• Troubleshooting

Introduction

This custom Ubuntu ISO is built to streamline the deployment of Petco SimpleAI, an AI inference server, on virtual machines (VMs). It includes:

• A pre-configured Petco SimpleAI binary.
• A systemd service to manage the Petco SimpleAI application.
• Default configuration files for easy setup.
• Support for GPU acceleration (e.g., NVIDIA L40S).

The ISO automates the installation of dependencies and configures the system to run SimpleAI on first boot, making it ideal for rapid deployment across Nutanix GPU cluster nodes.

Prerequisites

Before building or deploying the ISO, ensure you have:

• A Linux system (e.g., Ubuntu or Manjaro) with Vagrant and VirtualBox installed.
• The SimpleAI project directory tarred as `petcoaiserver.tar.gz`.
• Access to the internet for downloading the Ubuntu ISO and dependencies.
• (Optional) NVIDIA GPUs for AI inference tasks in your Nutanix cluster.

Building the ISO

The build process is automated using a Makefile and runs inside a Vagrant VM to isolate the build environment.

Steps:

1. Set up the Vagrant VM: ```bash vagrant up vagrant ssh ```

2. Run the Makefile: ```bash make ISO_NAME=ubuntu-22.04.4-live-server-amd64.iso ```

This will:

• Download the specified Ubuntu ISO (or use a local copy).
• Set up a chroot environment.
• Install necessary packages (e.g., Go, CMake, GRPC).
• Copy the SimpleAI binary and configuration files.
• Set up a systemd service for SimpleAI.
• Repackage the ISO as `custom-ubuntu-22.04.4-live-server-amd64.iso`.

Deploying the ISO

1. Upload the custom ISO to your Nutanix Prism Central or any VM management interface.
2. Create a new VM and attach the ISO as a bootable image.
3. Start the VM. On first boot, the system will:
   • Install NVIDIA drivers (if GPUs are present).
   • Compile and install the Petco SimpleAI binary.
   • Start the Petco SimpleAI service using the default configuration.

Configuring Petco SimpleAI

Petco SimpleAI is configured via:

• Default Config File: `/etc/petcoai/config.yaml` (YAML format).
• Environment Variables: Set in `/etc/petcoai/env`.

Example `config.yaml`:

```yaml context_size: 50000 parallel_requests: true enable_watchdog_idle: true watchdog_idle_timeout: "15m" enable_watchdog_busy: true watchdog_busy_timeout: "30m" debug: true log_level: trace cors: true ```

Overriding Configuration

• Via systemctl: ```bash sudo systemctl set-environment EXTRA_ARGS="--context-size=60000 --log-level=info" sudo systemctl restart petcoai ```
• Via cloud-init: Modify `/etc/petcoai/env` or set `EXTRA_ARGS` during VM provisioning.

Managing the Service

The Petco SimpleAI service is managed by systemd. Common commands:

• Start: `sudo systemctl start petcoai`
• Stop: `sudo systemctl stop petcoai`
• Restart: `sudo systemctl restart petcoai`
• Status: `sudo systemctl status petcoai`
• Enable on boot: `sudo systemctl enable petcoai`
• Disable on boot: `sudo systemctl disable petcoai`

Troubleshooting

• Service fails to start: Check logs with `journalctl -u petcoai`.
• Configuration issues: Verify `/etc/petcoai/config.yaml` and `/etc/petcoai/env`.
• GPU not detected: Ensure NVIDIA drivers are installed and the VM has GPU passthrough configured in your Nutanix cluster.

Requirements

Based on your query, here's what we need:

1. Custom Ubuntu ISO: Based on Ubuntu (e.g., 22.04), with the Petco SimpleAI binary and a systemd service included.
2. Petco SimpleAI Configuration:
   • Default config file at `/etc/petcoai/config.yaml`.
   • systemd service that reads this config and supports overrides via command-line arguments or cloud-init.
3. Flexibility: Ability to stop and restart the service with different configurations easily.
4. Build Automation: Use a Makefile and scripts, taking an ISO image name or URL as input.
5. Isolation: Ensure the build process doesn't affect your Manjaro host system.

To achieve this, we'll use Vagrant to create an isolated Ubuntu VM for the build, avoiding any impact on your host. The Makefile will orchestrate the process, with separate scripts handling each step.

File Structure

Here's the directory structure for your project:

``` project/ ├── Makefile # Orchestrates the build process ├── scripts/ # Contains all build scripts │ ├── download_iso.sh # Downloads the Ubuntu ISO │ ├── setup_chroot.sh # Sets up the chroot environment │ ├── install_packages.sh # Installs dependencies in the ISO │ ├── configure_system.sh # Configures Petco SimpleAI and systemd │ └── repackage_iso.sh # Repackages the ISO ├── downloads/ # Stores downloaded ISO images │ └── ubuntu-22.04.4-live-server-amd64.iso ├── config/ # Configuration files for Petco SimpleAI │ └── petcoai/ │ └── config.yaml # Default config for Petco SimpleAI └── Vagrantfile # Defines the VM for building the ISO ```

Step 1: Set Up the Build Environment

To isolate the build process from your Manjaro host, we'll use Vagrant with VirtualBox to create an Ubuntu VM.

Vagrant and virtualbox are installed on this manjaro arch host.

Vagrantfile

Create a `Vagrantfile` in the project root to define the VM: ```ruby Vagrant.configure("2") do |config| config.vm.box = "ubuntu/jammy64" # Ubuntu 22.04 config.vm.provider "virtualbox" do |vb| vb.memory = "4096" # 4GB RAM vb.cpus = 2 # 2 CPUs end config.vm.provision "shell", path: "provision.sh" end ```

Provision Script (`provision.sh`)

Create this script in the project root to install necessary tools in the VM: ```bash #!/bin/bash sudo apt update sudo apt install -y xorriso squashfs-tools libarchive-tools ```

Start the VM: ```bash vagrant up vagrant ssh # Log into the VM to run the build ```

Step 2: Makefile and Scripts

The Makefile will manage the build process, with each step in a separate script. The ISO image name can be provided as input, and the download script will fetch it if needed.

Makefile

Place this in the project root: ```makefile ISO_NAME ?= ubuntu-22.04.4-live-server-amd64.iso ISO_URL ?= https://releases.ubuntu.com/22.04/$(ISO_NAME) WORK_DIR := /tmp/iso_build SQUASHFS_DIR := $(WORK_DIR)/squashfs ISO_MOUNT := $(WORK_DIR)/iso_mount ISO_NEW := $(WORK_DIR)/iso_new CUSTOM_ISO_NAME := custom-$(ISO_NAME)

all: download setup_chroot install_packages configure_system repackage clean

download: @bash scripts/download_iso.sh "$(ISO_URL)" downloads/$(ISO_NAME)

setup_chroot: @bash scripts/setup_chroot.sh downloads/$(ISO_NAME) $(WORK_DIR)

install_packages: @bash scripts/install_packages.sh $(SQUASHFS_DIR)

configure_system: @bash scripts/configure_system.sh $(SQUASHFS_DIR)

repackage: @bash scripts/repackage_iso.sh $(WORK_DIR) $(CUSTOM_ISO_NAME)

clean: @sudo umount $(SQUASHFS_DIR)/dev $(SQUASHFS_DIR)/proc $(SQUASHFS_DIR)/sys || true @sudo rm -rf $(WORK_DIR) ```

You can run the build with: ```bash make ISO_NAME=ubuntu-22.04.4-live-server-amd64.iso ```

Scripts

Place these in the `scripts/` directory and make them executable (`chmod +x scripts/*.sh`).

1. `download_iso.sh`
   Downloads the ISO if it's not already in `downloads/`: ```bash #!/bin/bash URL=$1 OUTPUT=$2 if [ ! -f "$OUTPUT" ]; then echo "Downloading $URL to $OUTPUT..." curl -o "$OUTPUT" "$URL" else echo "$OUTPUT already exists, skipping download." fi ```

2. `setup_chroot.sh`
   Sets up the chroot environment by unpacking the ISO: ```bash #!/bin/bash ISO=$1 WORK_DIR=$2 mkdir -p $WORK_DIR $ISO_MOUNT $ISO_NEW $SQUASHFS_DIR sudo mount -o loop $ISO $ISO_MOUNT sudo cp -a $ISO_MOUNT/. $ISO_NEW/ sudo umount $ISO_MOUNT sudo unsquashfs -f -d $SQUASHFS_DIR $ISO_NEW/casper/filesystem.squashfs sudo mount --bind /dev $SQUASHFS_DIR/dev sudo mount --bind /proc $SQUASHFS_DIR/proc sudo mount --bind /sys $SQUASHFS_DIR/sys sudo chroot $SQUASHFS_DIR /bin/bash -c "export DEBIAN_FRONTEND=noninteractive; apt update" ```

   Note: We use `chroot` here because it's a standard way to modify an ISO's filesystem. Running this inside a Vagrant VM ensures your host system remains unaffected.

3. `install_packages.sh`
   Installs dependencies required by SimpleAI: ```bash #!/bin/bash SQUASHFS_DIR=$1 sudo chroot $SQUASHFS_DIR /bin/bash -c "apt install -y build-essential curl git python3-pip"

   Add more packages as needed for SimpleAI dependencies

   ```

4. `configure_system.sh`
   Configures the SimpleAI binary, config file, and systemd service: ```bash #!/bin/bash SQUASHFS_DIR=$1

   Copy SimpleAI binary (replace /path/to/simpleai with the actual path)

   sudo cp /path/to/simpleai $SQUASHFS_DIR/usr/local/bin/ sudo chmod +x $SQUASHFS_DIR/usr/local/bin/simpleai

   Set up config directory and file

   sudo mkdir -p $SQUASHFS_DIR/etc/petcoai sudo cp config/petcoai/config.yaml $SQUASHFS_DIR/etc/petcoai/

   Create systemd service

   sudo bash -c "cat << 'EOF' > $SQUASHFS_DIR/etc/systemd/system/petcoai.service [Unit] Description=Petco SimpleAI Service After=network.target

   [Service] EnvironmentFile=/etc/petcoai/env ExecStart=/usr/local/bin/simpleai run --config /etc/petcoai/config.yaml $EXTRA_ARGS Restart=always User=root

   [Install] WantedBy=multi-user.target EOF"

   Enable the service

   sudo chroot $SQUASHFS_DIR /bin/bash -c "systemctl enable petcoai.service" ```

5. `repackage_iso.sh`
   Repackages the modified filesystem into a new ISO: ```bash #!/bin/bash WORK_DIR=$1 CUSTOM_ISO_NAME=$2 sudo mksquashfs $SQUASHFS_DIR $ISO_NEW/casper/filesystem.squashfs -comp xz -noappend sudo du -sx --block-size=1 $SQUASHFS_DIR | cut -f1 > $ISO_NEW/casper/filesystem.size cd $ISO_NEW sudo xorriso -as mkisofs -r -V "Custom Ubuntu 22.04" -o ../$CUSTOM_ISO_NAME \ -J -joliet-long -l -iso-level 3 \ -b boot/grub/i386-pc/eltorito.img -c boot.catalog \ -no-emul-boot -boot-load-size 4 -boot-info-table \ -eltorito-alt-config -eltorito-boot isolinux/isolinux.bin -no-emul-boot \ -eltorito-catalog boot.catalog . cd .. ```

Step 3: Configuration Flexibility

To make the SimpleAI service flexible:

Default Config File

Create `config/petcoai/config.yaml` with your default settings. For example: ```yaml context_size: 50000 parallel_requests: true enable_watchdog_idle: true watchdog_idle_timeout: "15m" enable_watchdog_busy: true watchdog_busy_timeout: "30m" debug: true log_level: trace cors: true ```

Environment File

Create `config/petcoai/env` (copied to `/etc/petcoai/env` in `configure_system.sh`) for additional variables: ``` VLLM_LOGGING_LEVEL=INFO PYTHON_GRPC_MAX_WORKERS=1 LLAMACPP_PARALLEL=4 THREADS=8 ```

The systemd service uses `EnvironmentFile` to load these variables, and `ExecStart` appends `$EXTRA_ARGS` for overrides.

Overriding Configuration

• Via systemctl: Set custom arguments and restart the service: ```bash sudo systemctl set-environment EXTRA_ARGS="--context-size=60000 --log-level=info" sudo systemctl restart petcoai ```

• Via cloud-init: Use cloud-init to modify `/etc/petcoai/env` or set `EXTRA_ARGS` during VM provisioning. Example cloud-init snippet: ```yaml write_files:

  • path: /etc/petcoai/env content: | VLLM_LOGGING_LEVEL=DEBUG THREADS=12 runcmd:
  • systemctl set-environment EXTRA_ARGS="--context-size=70000"
  • systemctl restart petcoai ```

Running the Build

1. Start the Vagrant VM: ```bash vagrant up vagrant ssh ```

2. Inside the VM, navigate to the project directory (e.g., `/vagrant`) and run: ```bash make ISO_NAME=ubuntu-22.04.4-live-server-amd64.iso ```

3. The custom ISO will be output as `custom-ubuntu-22.04.4-live-server-amd64.iso`.

Why This Approach?

• Isolation: Vagrant ensures the build runs in a VM, protecting your Manjaro host.
• Chroot: Used within the VM to modify the ISO's filesystem safely and efficiently.
• Modularity: Separate scripts make the process easy to debug and extend.
• Flexibility: The systemd service supports default configs and overrides via `systemctl` or cloud-init.
• Nutanix Optimization: Built specifically for rapid deployment across Nutanix GPU cluster nodes.

This solution enables Petco to quickly deploy the SimpleAI inference server across their Nutanix GPU cluster infrastructure, providing a streamlined path from development to production deployment. The automated build process ensures consistency and reduces manual configuration overhead when scaling AI inference capabilities across multiple cluster nodes.

If you'd prefer a different isolation method (e.g., Docker), let me know, but Vagrant provides a clean, VM-based solution that aligns with your needs. Let me know if you need further adjustments!