Spheron Network

Same Llama 3.1 70B model, same batch size, four chips. The decode speed ceiling moves entirely with memory bandwidth.

- H100 SXM5 (HBM3, 3.35 TB/s): about 24 tokens/sec
- H200 (HBM3e, 4.8 TB/s): about 34 tokens/sec
- B200 (8.0 TB/s): about 57 tokens/sec
- B300 (10.0 TB/s): about 71 tokens/sec

Vera Rubin's projected HBM4 (up to 22 TB/s) points to roughly 157 tokens/sec, about 6.5x the H100 ceiling. That figure is derived from NVIDIA's system-level spec, not a confirmed benchmark, and pricing isn't published yet.

NVIDIA markets 2:4 structured sparsity as 2x the math throughput of dense tensor cores.

Real numbers land lower. A100 running TensorRT 8.0 at high batch hits about 1.2x, not 2x. A40 running ResNet-34 INT8 moves from 1.21x at batch 1 to 1.66x only at a large 3x4096x2048 input. On an L4 running Qwen3-4B-Instruct across concurrency 1-128, the gain is 1.0x: no speedup at all.

The accuracy side holds up fine, Red Hat's Sparse Llama fine-tuning recovers 98.4%-101.1% of dense baseline accuracy. It's the throughput claim worth checking before you plan a deployment around it.

A 128K-token system prompt takes 11.2s to hit the first output token when every node reads it cold. Share that memory across nodes with LMCache: 1.5s.

LMCache shares the KV cache across vLLM nodes instead of each rebuilding it. SGLang works too. TensorRT-LLM is coming.

Setup:
- Run vLLM with LMCache on every node
- Point all nodes at one shared cache backend
- Serve requests with repeated long prompts, not one-offs
- Let new nodes reuse the cache instead of recomputing

At 200 concurrent requests, 80% prompt reuse, a 4x H100 cluster went from 1.4 req/s cold to 21.3 with LMCache: 15.2x. Lower reuse or shorter prompts narrow that gap, and the shared backend adds memory and upkeep cost.

Long, repeated system prompts: use it. Mostly one-off traffic: skip it, nothing to cache.

You should not have to commit to a year of GPU spend just to get a rate worth having.

Spheron bills per minute, not per month, so a training run that finishes in 40 minutes does not get charged for an hour it did not use. Spin up on spot pricing for a non-critical job, switch to on-demand for a deadline, or commit to a reserved rate once you actually know your usage pattern.

Payment runs in USDC or USDT alongside normal billing too, for teams that would rather not run a GPU bill through a card processor at all. Three billing models, one marketplace, no lock-in either way.

You're renting a GPU in a data center you've never seen. One question matters more than price: can the people running that rack read your data?

NVIDIA TEE encrypts the GPU's memory so the operator running that server can't read your model weights or inputs while the job runs. But TEE can't vet the building. That's a separate guarantee: Tier 3+ certification.

Before trusting a TEE-enabled node:

- Confirm TEE is enabled on the node, not just available on the chip
- Confirm the facility is Tier 3+ certified
- Confirm the node came from a certified partner, not a reseller
- Confirm you can audit both before data leaves your network

We've sourced exactly this: confidential inference on H200, NVIDIA TEE enabled, from a Tier 3+ certified partner.

Running inference you can't expose to the host? Request a quote for TEE-enabled nodes.