Qwen2.5 72B Instruct (AWQ)

A 70B-class model split across both GPUs of a 2× RTX 5090 machine (64 GB VRAM) with tensor parallelism.

Overview

Qwen2.5 72B Instruct is one of the strongest open models in the 70B class. The official AWQ 4-bit weights (~41 GB) don't fit on one 32 GB card, so this deploys on a 2× instance and vLLM shards the model over both GPUs (--tensor-parallel-size 2), leaving room for a 16K context.

Key features

• ✓
  70B-class quality
• ✓
  Runs on 2× RTX 5090 (64 GB)
• ✓
  Tensor parallel over both GPUs

Use cases

• →
  GPT-4-class private chat
• →
  Long-document analysis
• →
  Distilling data for smaller models

On this template

Runs on the vLLM (OpenAI-compatible API) template, exposing its OpenAI-compatible API at a private HTTPS address. You also get root SSH and JupyterLab.

MODEL_ID	Qwen/Qwen2.5-72B-Instruct-AWQ
VLLM_ARGS	--tensor-parallel-size 2 --max-model-len 16384 --gpu-memory-utilization 0.92

Quick start

1. Click Deploy and pick an available 2× GPU offer — only offers with enough VRAM are shown.
2. Wait for the status to turn running. The first start downloads the model, which can take a few minutes.
3. Use the connection details on your rental page:

curl https://r12-abcd1234-app.gpu.clearlayer.ca/v1/chat/completions \
  -H "Authorization: Bearer <app-token>" \
  -H "Content-Type: application/json" \
  -d '{"model": "Qwen/Qwen2.5-72B-Instruct-AWQ", "messages": [{"role": "user", "content": "Hello!"}]}'

from openai import OpenAI

client = OpenAI(base_url="https://r12-abcd1234-app.gpu.clearlayer.ca/v1", api_key="<app-token>")
resp = client.chat.completions.create(
    model="Qwen/Qwen2.5-72B-Instruct-AWQ",
    messages=[{"role": "user", "content": "Hello!"}],
)
print(resp.choices[0].message.content)