# Instrument resolution

Display increments, effective discrimination, quantization uncertainty and practical limits on selecting measurement equipment.

Reviewed: 2026-11-01

Tags: inspection, metrology, resolution, uncertainty

## Definition and the distinction that matters

The VIM defines resolution as the smallest change in the measured quantity that produces a perceptible indication change. Noise, friction and the quantity value can affect it. [JCGM VIM3, 4.14](https://jcgm.bipm.org/vim/en/4.14.html)

Display resolution describes the distinction between displayed indications. Sensor response can differ from a digital increment or dial graduation. [JCGM VIM3, 4.15](https://jcgm.bipm.org/vim/en/4.15.html)

For an indication rounded to 0.01 mm, the illustrative rectangular error model has half-width 0.005 mm and standard uncertainty **0.01/√12 ≈ 0.002887 mm**. Use this contribution only when the rounding model applies and the effect is not already captured elsewhere in the budget.

NIST recommends checking performance under actual measurement conditions. Compare artifacts with known differences to see when the system ceases to distinguish them. [NIST/SEMATECH, resolution](https://itl.nist.gov/div898/handbook/mpc/section4/mpc451.htm)

## Related specifications answer different questions

| Term or specification | What to establish |
| --- | --- |
| Display increment | The numerical step shown or recorded in the selected range/mode. |
| Effective resolution | Whether relevant input changes are distinguishable under working conditions. |
| Repeatability | Variation when the measurement procedure is repeated under specified conditions. |
| Specified error limit | The manufacturer's permitted indication error under its stated conditions. |
| Measurement uncertainty | Dispersion assigned to the measurement result after considering relevant contributions. |

Do not substitute one row for another. A stable indication can contain bias, and a fine readout can fluctuate because of noise. Calibration establishes relevant instrument behaviour; it does not make the display increment the uncertainty of every subsequent use.

NPL's **Good Practice Guide 11, Issue 2 with amendments March 2001**, explains that uncertainty can originate in the instrument, measured item, environment and operator. A useful equipment choice therefore considers the complete measurement task, including contact geometry, force, temperature, alignment, fixturing and the measured surface where relevant. [NPL, beginner's guide to uncertainty](https://www.npl.co.uk/getattachment/63c00201-a38d-44e5-8470-2440284c3dfe/gpg11_uncertainty.pdf?lang=en-GB)

## Digital quantization and a worked example

For a digital indication rounded to increments **q**, a commonly used model assigns the unresolved input within **±q/2** a uniform, or rectangular, distribution. Its standard uncertainty is:

**u₍resolution₎ = q/√12**.

This is a model for one contribution, not a universal expression for total instrument uncertainty. The GUM discusses this treatment and explains why identical repeated indications do not establish zero uncertainty: a range of inputs can produce the same displayed number. [JCGM GUM, Annex F.2.2.1](https://www.iso.org/sites/JCGM/GUM/JCGM100/C045315e-html/C045315e_FILES/MAIN_C045315e/AF_e.html)

**Original arithmetic example:** assume a length instrument rounds to **0.01 mm** and the rectangular model is appropriate. The quantization half-width is **0.005 mm**, while the standard uncertainty contribution is:

**0.01/√12 = 0.002887 mm**, approximately **2.9 µm**.

The half-width and standard uncertainty are different quantities. Under this idealized rounding model, inputs of 12.341 and 12.344 mm can both display **12.34 mm**. Ten copies of that indication do not recover the lost digits or establish zero variation in the underlying length.

Suppose, solely for illustration, other independently evaluated standard-uncertainty contributions combine to **0.006 mm**. Combining that amount with the resolution term by root-sum-of-squares gives:

**uᶜ = √(0.006² + 0.002887²) = 0.006658 mm**.

If a coverage factor **k = 2** is justified for that measurement model, the corresponding expanded uncertainty is approximately **0.0133 mm**. Improving only the display increment to **0.001 mm**, leaving everything else unchanged, would give uᶜ ≈ **0.006007 mm** and U ≈ **0.0120 mm**. The improvement is limited because the other assumed contributions dominate. These are invented inputs, not a calibration or acceptance claim for a named instrument. [NPL, combining standard uncertainties and coverage factors](https://www.npl.co.uk/getattachment/63c00201-a38d-44e5-8470-2440284c3dfe/gpg11_uncertainty.pdf?lang=en-GB)

## State the model rather than applying a shortcut

Type B uncertainty evaluation uses available information and scientific judgement, including specifications, calibration data and assumed distributions. NIST gives different divisors for rectangular and other distributions. Establish the bounds and distribution appropriate to the instrument and measurement before converting a specification into standard uncertainty. [NIST, Type B evaluation](https://physics.nist.gov/cuu/Uncertainty/typeb.html)

The ±q/2 rounding model should not be applied blindly to truncating software, hysteretic indications, filtered signals or an input that is already noisy. Similarly, do not automatically add every nominal resolution term to a repeatability estimate if the same effect is already represented. Document the budget's treatment and dependencies. A numerical formula cannot decide whether two contributions describe independent effects.

Averaging can improve an estimate when the observations and noise support the model. It does not guarantee arbitrarily fine discrimination. Preserve the acquisition procedure and avoid treating rounded identical readings as independent evidence that systematic effects have vanished.

## Range, speed and settings can change performance

Resolution must be stated with units and operating mode. A digit count or ADC bit count does not identify the engineering-unit increment without the selected range and conversion. Autoranging, filtering, integration time and data export may also make a displayed result differ from a raw reading.

As a bounded instrumentation example, Keysight's **BenchVue DMM documentation** describes integration time in power-line cycles or seconds for DC voltage/current, resistance and temperature measurements. Longer integration improves resolution, while shorter integration permits faster readings. It also identifies integral power-line-cycle settings for line-frequency rejection. These relationships apply to the supported DMM functions and settings, not every sensor or meter. [Keysight, aperture time and resolution](https://helpfiles.keysight.com/BenchVueSoftware_HDML5HelpFiles/DMMApp/English/Content/GUI/Instrument%20Settings/Measurement/Aperture%20Time%20and%20Resolution.htm)

A setting suitable for a steady laboratory value may conceal short-duration changes. Match the acquisition bandwidth and timing to the measurand as well as the desired numerical detail. Record range, integration, filtering and relevant update rates so another operator can reproduce the result.

## Selecting and reporting a fit-for-purpose system

Start with the characteristic, tolerance or required change detection, measurement conditions and decision rule. Determine the uncertainty and discrimination needed for that task, then check the candidate system in representative conditions. A ratio such as ten display increments across a tolerance can be a screening convention, but it is not by itself proof of measurement capability or a universal acceptance rule.

Poor discrimination can also hide process variation. NIST warns that a gauge with inadequate resolution relative to the measurement need cannot provide useful study information. Consider that limitation before calculating process-capability statistics from heavily quantized data. [NIST/SEMATECH, consequences of poor resolution](https://itl.nist.gov/div898/handbook/mpc/section4/mpc451.htm)

Retain the instrument/configuration, range, displayed and exported increments, calibration basis, environmental conditions, repeatability evidence and uncertainty treatment. Report results with appropriate rounding and their uncertainty where required. Keep original observations for analysis; adding zeros during unit conversion or formatting creates no new measurement information.

## Related reference

[Inspection and quality documents](../../sourcing/documents/index.md)