Skip to main content

Qwen3.5 122B A10B On Shadeform B300

Private Stack Exposure​

Remote benchmark workers must reach the Metrum AI Bench control plane over a public URL. When you run a private or local stack, expose PostgREST before launch and use the public HTTPS URL for METRUM_PUBLIC_API_URL, INSIGHTS_API_URL, and INSIGHTS_CONTROL_PLANE_URL:

cloudflared tunnel --url http://localhost:${POSTGREST_PORT:-3000}
export METRUM_PUBLIC_API_URL="https://<cloudflared-host>.trycloudflare.com"
export INSIGHTS_API_URL="$METRUM_PUBLIC_API_URL"
export INSIGHTS_CONTROL_PLANE_URL="$METRUM_PUBLIC_API_URL"
curl -fsS "$METRUM_PUBLIC_API_URL/" >/dev/null

Do not give a remote worker localhost, 127.0.0.1, host.docker.internal, or a private-only DNS name. Keep the tunnel alive through onboarding, polling, result upload, and teardown; expose any private package/WebUI endpoint through a public URL too when the worker downloads artifacts from that stack.

This example shows the API shape for a metrumbench-llm run using:

  • provider: Shadeform
  • hardware: 1x B300, Ubuntu 24.04
  • model: Qwen/Qwen3.5-122B-A10B-FP8
  • framework: vLLM 0.20.1
  • API authentication: POST /rpc/mint_postgrest_write_jwt
  • required artifacts: Restic repo/auth plus the complete tagged Insights package set for the selected version tag

Use the full operator runbook when executing the run: docs/runbooks/gpu/b300/manual-shadeform-b300-qwen35-122b-a10b-curl-runbook.md.

Local Control Plane Exposure​

For a local stack, expose PostgREST before provisioning the Shadeform instance. The remote worker cannot onboard against localhost.

cloudflared tunnel --url http://localhost:23040

Use the generated HTTPS URL as the public PostgREST URL for the provisioner and worker bootstrap path.

Mint The JWT​

Use the database JWT minting RPC. Do not use local Python snippets or CLI token minting for this example.

curl -fsS -X POST "$METRUM_API_URL/rpc/mint_postgrest_write_jwt" \
  -H "Content-Type: application/json" \
  -d '{
    "p_jwt_secret":"<PGRST_JWT_SECRET>",
    "p_account_id":"<ACCOUNT_ID>",
    "p_email":"<ACCOUNT_EMAIL>",
    "p_expires_in_seconds":43200
  }' | jq -er 'if type=="string" then . else .token end'

Use the returned bare JWT as Authorization: Bearer <METRUM_JWT_TOKEN> on subsequent PostgREST calls. The 12-hour TTL is intentional for large-model pulls, model load, and multi-scenario matrices.

Verify Worker Artifacts​

Before provisioning paid capacity, verify Restic and the tagged package set. Do not use untagged, latest, partial, or ad hoc package contents.

RESTIC_REPOSITORY=<resolved, do not print> \
RESTIC_PASSWORD=<resolved, do not print> \
restic snapshots --tag "$INSIGHTS_PACKAGE_TAG" --json >/tmp/restic.snapshots.check.json

Expose the tagged package server to the worker when running from a local stack:

cloudflared tunnel --url http://localhost:25041

Verify the package surface before launch:

curl -fsS "https://<public-package-host-or-dns>/api/scripts/worker-startup.sh"
curl -fsS "https://<public-package-host-or-dns>/api/scripts/restic-restore.sh"
curl -fsS "https://<public-package-host-or-dns>/api/packages/"
curl -fsS "https://<public-package-host-or-dns>/api/packages/?tag=<INSIGHTS_PACKAGE_TAG>"
curl -fsS "https://<public-package-host-or-dns>/downloads/metrumbench-llm" \
  -o /tmp/metrumbench-llm.b300.check

Runtime Contract​

The model-server launch should resolve to the Qwen3.5 B300 vLLM recipe:

curl -fsS -X POST "$METRUM_API_URL/rpc/resolve_model_server_runtime_contract" \
  -H "Authorization: Bearer <METRUM_JWT_TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{
    "p_framework_code":"vllm",
    "p_framework_version":"0.20.1",
    "p_model_identifier":"Qwen/Qwen3.5-122B-A10B-FP8",
    "p_engine_args_set_code":"vllm-qwen35-122b-a10b-b300-fp8",
    "p_bind_host":"0.0.0.0",
    "p_port":8000,
    "p_launch_adapter":"direct_process",
    "p_container_image_override":""
  }'

The resolved command must include the required FP8 and chunked-prefill args. For vLLM 0.20.1, --enable-chunked-prefill is a bare flag.

Required runtime environment values should be applied through the supported template/runtime contract, such as framework_command_templates.runtime_env:

VLLM_ATTENTION_BACKEND=FLASHINFER
VLLM_USE_DEEP_GEMM=1
VLLM_MULTI_STREAM_GEMM_TOKEN_THRESHOLD=256
VLLM_TORCH_COMPILE_LEVEL=O2

Submit The Cloud Execution​

Create the project, workload, and scenario matrix using the same APIs described in Project Lifecycle and metrumbench-llm Workload And Scenario, then submit managed cloud executions. The intended matrix crosses ISL/OSL pairs 128/128, 128/1024, 1024/128, and 1024/1024 with concurrency 32, 128, 512, and 1024. Use scenario codes shaped as c<CONCURRENCY>-isl<ISL>-osl<OSL>.

Submit one reusable worker execution with auto_terminate=false, then dispatch the remaining scenario matrix to the same ready worker:

curl -fsS -X POST "$METRUM_API_URL/rpc/create_cloud_execution_request" \
  -H "Authorization: Bearer <METRUM_JWT_TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{
    "p_owner_account_id":"<ACCOUNT_ID>",
    "p_org_id":"<ORG_ID>",
    "p_project_name":"Qwen3.5 122B A10B B300 metrumbench-llm",
    "p_workload_code":"qwen35-122b-a10b-b300-vllm0201",
    "p_config_code":"shadeform-b300",
    "p_scenario_code":"<SCENARIO_CODE>",
    "p_auto_terminate":false,
    "p_hf_token_id":<HF_TOKEN_ID_OR_NULL>
  }'

Patch startup_extra_args_cli before the provisioner picks up the request if you need a launch flag that is not represented in the engine args templates:

curl -fsS -X PATCH "$METRUM_API_URL/cloud_executions?execution_id=eq.<EXECUTION_ID>" \
  -H "Authorization: Bearer <METRUM_JWT_TOKEN>" \
  -H "Content-Type: application/json" \
  -H "Prefer: return=representation" \
  -d '{
    "startup_extra_args_cli":"--enable-chunked-prefill"
  }'

After readiness, dispatch additional scenarios with the run scenario execution API used by the current control plane, keeping the same server_hostname.

Monitor​

curl -fsS "$METRUM_API_URL/cloud_provisioning_requests?request_id=eq.<REQUEST_ID>&select=request_id,status_code,error_message,updated_at" \
  -H "Authorization: Bearer <METRUM_JWT_TOKEN>"
curl -fsS "$METRUM_API_URL/server_config_instances?hostname=eq.<HOSTNAME>&select=server_id,hostname,created_at" \
  -H "Authorization: Bearer <METRUM_JWT_TOKEN>"
curl -fsS "$METRUM_API_URL/v_metrumbench_llm_outputs?job_id=eq.<JOB_ID>&limit=50" \
  -H "Authorization: Bearer <METRUM_JWT_TOKEN>"

Stop Conditions​

  • Stop if the public API URL points at localhost or 127.0.0.1.
  • Stop if Restic repo/auth cannot be resolved and queried.
  • Stop if the selected tagged Insights package set is incomplete.
  • Stop if no Shadeform B300 capacity is available.
  • Stop if server readiness fails after provisioning.
  • Stop if the resolved vLLM command or runtime environment does not match the required B300 Qwen FP8 contract.