LLM — Steps to Build a Custom NIM Using Strong Base Container

Version: SGLang v0.5.9

Supported NVIDIA hardware, CUDA, OS, and CUDA driver

Supported NVIDIA Hardware	CUDA version	Supported OS	CUDA Compute Capability	CUDA Driver Version	Platform
* Blackwell * Hopper * Ampere * Turing	13.0 and newer, see enhanced CUDA compatibility guide	* Ubuntu 24.04 * Ubuntu 22.04 * Ubuntu 20.04 * and other manylinux2_28 compatible	SM 7.5 and later	r580 or later	linux x86_64
* Blackwell * Hopper * Ampere * Turing * Thor	13.0 and newer, see enhanced CUDA compatibility guide	* Ubuntu 24.04 * Ubuntu 22.04 * Ubuntu 20.04 * and other manylinux2_28 compatible	SM 7.5 and later	r580 or later	linux aarch64 SBSA

refernece link: https://docs.nvidia.com/deeplearning/dali/user-guide/docs/support_matrix.html

Major Enhancements

No open source license issue
No critical and high CVE issue
Profile selection and management system
Model artifacts auto download
Artifacts and profiles checking
OTEL
Hardware and inference metrics
Tuned parameter injection system

Minor Features

Fix response api tool calling

Build Steps

1. Test whether the specific model can be launched by 0.5.9 SGLang

2. Build manifest file

Decide scope, including SKUs, precision, parallel strategy combination
A profile means single combination of SKU, precision and parallel strategy
Fill each profile with content. Best practice is, user can maintain two types of profiles:

Normal profile (verified, SKU specific)
Fallback SKU (unverified, used to cover border SKUs) Here is what profile.yaml looks like for these two types:
Normal profile example
Fallback profile example

schema_version: "1.0" # Do not change
model: "minimaxai_minimax-m2_5" # model name
release: "1.7.0-variant" # model release version
other_llm:
  llm_engine: sglang # Do not change if it is a sglang base container
  profiles:
    - model_version: "hf-3040beaf-nim"
      tags:
        # dp: data parallel degree, default is 1
        # vram: minimal vram required for the profile, need sum across launch GPUs
        # nim_custom_startup_script: add-on custom script, advanced users only
        feat_lora: "false"
        pp: "1"
        tp: "8"
        ep: "8"
        gpu: H20-3e
        precision: "fp8"
        profile: throughput
        gpu_device: 232c:10de # necessary for nimtools to identify the GPU device
      ignore_precision_in_config: true # enable it to avoid nimtools from overwriting precision config

- model_version: "hf-3040beaf-nim"
tags:
  feat_lora: "false"
  pp: "1"
  tp: ["2", "4"]
  ep: "1"
  gpu: B200
  precision: "fp8"
  profile: throughput
  gpu_device: 2901:10de
ignore_precision_in_config: true

For more details about profile creation, please refer to: https://developer.nvidia.com/docs/nim-tools/latest/templates/nimlib/nim-generate-manifest.html

Use nim_generate_manifest to generate model_manifest.yaml

3. Performance config injection

Find optimal raw SGLang server configs — performance tuning tools ongoing, but AIC might help decide parallel strategy.
Config overwrite order: config file > ENV > tags > default

Config file is dynamic: For example

# ENV: Environment variables. After render, inference.py sets these in os.environ
# inside the function; they are not included in the return value.
# Literal key-value pairs must be strictly respected; use {{ var }} for dynamic
# values, consumed by heuristic plugins.
ENV:
  # Literal example
  NIM_LOG_LEVEL: "INFO"
  # Dynamic example (uses default when no handler provides a value)
  CUSTOM_PREFIX: "{{ custom_prefix | default('minimax') }}"

# sglang_config section: after render, returned as a dict from render_config and merged with engine_extra_args.
# Literal parameters are strictly respected; use {{ var }} for dynamic values, provided by registered handlers.
attention_backend: flashinfer
trust_remote_code: true
mem_fraction_static: 0.9
max_running_requests: {{ concurrency | default(8) }}
kv_cache_dtype: fp8_e4m3
quantization: "{{ quantization | default('fp8') }}"
reasoning_parser: minimax
tool_call_parser: minimax-m2

User can choose to create a YAML config for static performance configuration at the profile level, or create a yaml.j2 file + plugin system for more intelligent performance tuning.

Check the actual running configuration is expected.

4. ENV list

Reference: https://developer.nvidia.com/docs/nim-tools/latest/templates/supported-image-labels-and-environment-variables.html

Environment Variable	Description
`NIM_FORCE_DETERMINISTIC`	Need adaptation
`TOOL_CALL_PARSER`	Boolean to on/off this feature
`REASONING_PARSER`	Boolean to on/off this feature
`NIM_ENABLE_INFERENCE_METRICS`	Enable inference metrics
`NIM_COLLECT_TOKENS_HISTOGRAM`	Collect tokens histogram
`NIM_CONFIG_FILE`	Overwrite config file; otherwise auto looks at `{gpu}_{profile}_{precision}_{tp}.j2` (or `.yaml`)

5. Dockerfile sample

ARG BASE_IMAGE
FROM ${BASE_IMAGE}

USER root

# Basic settings - mandatory
ARG NIM_NAME
ARG NIM_MODEL_NAME
ARG MANIFEST_FILE
ENV NIM_NAME=${NIM_NAME}
ENV NIM_MODEL_NAME=${NIM_MODEL_NAME}
COPY ./LICENSE ./VERSION $NIM_DIR_PATH/
COPY ${MANIFEST_FILE} /opt/nim/etc/default/model_manifest.yaml

# adjust per model - unified across profiles - mandatory
ENV NIM_TOOL_CALL_PARSER_NAME=""
ENV NIM_REASONING_PARSER_NAME="deepseek-r1"
ENV NIM_MODEL_LOADER_EXTRA_CONFIG='{"enable_multithread_load": "true","num_threads": 32}'

# Advanced settings - optional
ENV NIM_HTTP_API_PORT="8000"                                    # default 8000
ENV NIM_HEURISTIC_PLUGIN_DIR="/opt/nim/heuristic_plugin/"       # default /opt/nim/heuristic_plugin
# default is optimized config file path per profile
# file_name structure is {gpu}_{profile}_{precision}_{tp}.yaml (optional yaml.j2)
ENV NIM_CONFIG_FILE=""
# Others stay same with:
# https://developer.nvidia.com/docs/nim-tools/latest/templates/supported-image-labels-and-environment-variables.html
# Notice it is a sglang backend, so some variables are not applicable.

# Performance tunings - optional
# caution: tuning_configs should be registered in profile tag or follow the auto search naming convention
COPY ./tuning_configs/* /opt/nim/
# to enable this feature, tuning configs must be j2 format and plugin can catch the var name in j2
COPY ./heuristic_plugins/* ${NIM_HEURISTIC_PLUGIN_DIR}

USER nvs:1000

Known Issues

Version publish_v1 is cu12.9.1 for best compatibility of the broader install base, but currently it can't launch VLM model due to cuDNN and PyTorch version issue. We are actively fixing it. We will release two versions shortly: cu130 and updated cu129 version to fix the issue.

Supported GPUs:

Generic

NIM-BaseContainer-Spark 容器镜像 公开资源