GPU Cost Governance for AI Workloads (2026)

Quick answer

GPU cost governance fails on three things that have nothing to do with negotiating a better rate: you are measuring utilization with a metric that reports 99% when the real number is 40%, you are buying capacity through mechanisms your commitment discounts explicitly do not cover, and you are paying up to 1.79x more than the next provider for identical silicon. Rate negotiation is the last lever, not the first. Fix the measurement, then fix the procurement, then shop.

Our GPU cost optimization for AI teams playbook covers the five operational levers - scheduling, spot, reservations, idle detection, right-sizing. This post is the layer underneath it: what a GPU-hour actually costs in September 2026, and the measurement and procurement traps that make the levers misfire.

What a GPU-hour costs right now

All figures are published list prices read from vendor pricing feeds on 6 September 2026. Hyperscaler rows are 8-GPU nodes, normalized to per-GPU so they compare honestly.

Sources: AWS EC2 price list, Google accelerator-optimized pricing, Azure Retail Prices API.

The specialist providers sit in a different band entirely, quoted per GPU-hour on demand: RunPod Secure $3.49, Nebius $3.85, Crusoe $3.90, Lambda $3.99, Together AI $3.99, DigitalOcean $4.41, CoreWeave $6.16, Oracle $10.00.

Three things worth saying out loud about that table.

Google prices H200 below H100. The a3-ultragpu-8g (H200) lists at $84.81 against a3-highgpu-8g (H100) at $88.49. That is what the page says. Newer is not automatically dearer, and nobody re-shops because they assume it is.

The unit is where people get fooled. CoreWeave headlines H100 at $49.24, which is a full 8-GPU node and works out to $6.16 per GPU - more than RunPod Secure at $3.49, not less. Any “cheapest H100” claim without a unit attached is marketing.

The spread is the finding. Same generation, same nominal silicon: 1.79x between the cheapest and dearest hyperscaler, about 2x between AWS and RunPod Secure, and 3.5x between Azure and RunPod Secure. Before optimizing utilization by 10%, check whether you are on the wrong invoice entirely.

The utilization metric that lies to you

This is the single most expensive misunderstanding in AI infrastructure, and it is a measurement bug rather than an engineering failure.

DCGM_FI_DEV_GPU_UTIL, the metric almost everyone dashboards, comes from NVML and is defined as the percent of time over the past sample period during which one or more kernels was executing on the GPU. Read that definition carefully. One small kernel occupying a single streaming multiprocessor for the whole window reports 100%. It measures whether the GPU is busy, not whether it is working.

The gap is documented at scale. The Acme study (NSDI 2024) traced 4,704 A100s across six months and reported median GPU utilization of 97% and 99% in its two clusters while median SM activity in the same clusters was approximately 40%. The paper notes directly that GPU utilization “may sometimes be a weak utilization indicator”. Meta’s Llama 3 work on a 16,000-GPU H100 cluster reports 38% to 43% model FLOPs utilization, which is a strong result for a well-engineered training run and still nowhere near what a 99% dashboard implies.

The fix is a config change. NVIDIA’s own guidance on DCGM_FI_PROF_SM_ACTIVE gives you thresholds you can turn into a policy: a value of 0.8 or greater is necessary but not sufficient for effective GPU use, and a value below 0.5 likely indicates ineffective GPU usage. The catch is that DCGM_FI_PROF_SM_ACTIVE and DCGM_FI_PROF_SM_OCCUPANCY ship commented out in the dcgm-exporter default counters file. A stock deployment does not export the two metrics that would tell you the truth.

So the first control in any GPU governance program is embarrassingly cheap: uncomment two lines, then re-baseline every utilization claim your team has ever made. Expect the number to fall by half. That is not a regression, it is the first honest measurement you have taken.

A related note for anyone quoting industry averages in a business case: the widely circulated “average GPU utilization is only 30%” claim is very difficult to trace to a primary source, and the survey it is usually credited to contains no such figure. Use the Acme and Meta numbers instead. They are peer-reviewed, they are specific, and they will survive a CFO asking where the number came from.

The commitment traps that void your discount

Every provider advertises deep discounts. What they advertise and what applies to the way AI teams actually buy GPUs are different things.

AWS Spot is not covered by Savings Plans. AWS states it plainly: spot spend does not apply the commitments in your Compute Savings Plans. If your training fleet is mostly spot and you bought a Savings Plan to cover it, you bought coverage for spend that cannot consume it.

Capacity Blocks are excluded from both Savings Plans and Reserved Instances. Capacity Blocks for ML are how most teams get guaranteed H100 and B200 capacity: reserve for 1 to 14 days or multiples of 7 up to 182 days, up to 8 weeks ahead. They are charged fully up front, cannot be cancelled, and priced dynamically with no published discount percentage. That is a prepaid, non-refundable, unhedgeable commitment sitting outside your entire discount structure.

Google excludes A3 and A4 from flexible committed use discounts. Only the G2 and G4 accelerator series qualify. For A3 and A4 - which is to say for H100, H200, and B200 - you need resource-based commitments with reservations attached. Teams that budgeted a flexible CUD against their Blackwell roadmap have budgeted a discount they cannot buy.

Azure reservations do not reserve anything. Microsoft’s wording: reserved VM instances “provide a billing discount only and do not reserve or guarantee compute capacity”. If your capacity plan assumes an RI holds GPUs for you, it does not.

Spot economics remain genuinely strong where the workload can checkpoint. Observed Azure spot for ND96isr_H100_v5 in East US was $18.17 against $98.32 on demand, roughly 81% off. Google advertises up to 91%, AWS and Azure up to 90%. But note the exception that proves why you verify per region and per SKU: Azure’s ND H200 v5 in westus2 and westus3 shows spot priced identically to on demand. A spot strategy that assumes a discount without checking gets you interruption risk for free.

Sharing a GPU costs you your cost data

Time-slicing looks like the obvious answer for dev and notebook workloads, and it has a FinOps consequence people discover late. NVIDIA documents that with time-slicing there is no memory or fault isolation between replicas, and that requesting several time-sliced GPUs does not guarantee proportional compute. Worse for our purposes: DCGM-Exporter does not support associating metrics to containers when GPU time-slicing is enabled. The moment you share that way, per-team and per-pod GPU cost attribution stops working.

MIG is the answer when the hardware allows it, because it partitions in hardware with real memory and fault isolation. H100 80GB supports 1g.10gb (up to seven instances), 1g.20gb, 2g.20gb, 3g.40gb, 4g.40gb, and 7g.80gb. H200 141GB and B200 180GB have equivalent ladders. A single H100 carved into seven 1g.10gb slices turns one $6.88 GPU-hour into seven independently attributable tenancies, which is both a utilization win and an allocation win.

The governance checklist

Six controls, in the order that pays:

1. Export SM activity. Uncomment DCGM_FI_PROF_SM_ACTIVE and re-baseline. Everything downstream depends on measuring the right thing.
2. Set a utilization floor as policy. Below 0.5 sustained SM activity triggers a review, not an email nobody reads.
3. Map every GPU dollar to a discount mechanism and mark the ones that are structurally ineligible. Spot and Capacity Block spend should never appear in Savings Plan coverage forecasts.
4. Re-shop annually against per-GPU-hour normalized prices. Node pricing hides a factor of eight.
5. Choose MIG over time-slicing wherever attribution matters, and accept the attribution loss consciously where it does not.
6. Tag and reconcile. Namespace labels and resource tags on every GPU workload, reconciled against the invoice.

Those last two are the same discipline we apply to token spend in LLM API cost optimization, and both roll up into the tooling and allocation domains scored in the FinOps maturity assessment framework. If your dashboard has been reporting 99% utilization for a year and nobody questioned it, that is the argument for treating FinOps as a quality problem rather than a finance problem: the data was wrong, and nobody tested it.

Start with the measurement

Do not open a procurement negotiation on the back of a 99% utilization dashboard. You will be told, correctly, that your fleet is saturated and you need more of it. Fix the metric first, find out what the fleet is actually delivering, and the conversation changes from “buy more” to “use what we have and re-shop the rest”.

Our AI & GPU Cost Governance QA engagement does exactly this pass: instruments real utilization, maps your spend against the discount mechanisms that actually apply to it, and hands back a prioritized roadmap with monthly savings per fix. For the full three-layer picture across tokens, inference, and training, start with our AI cost audit methodology, or get in touch for a free 30-minute review.