ZFP

ZFP ^[1] is a high-throughput transform-based compressor for n-dimensional floating-point and integer data, which compresses the data in independent $4 \times \ldots \times 4$ blocks. The values inside a block are rescaled, decorrelated, and transformed into a bit stream that can be truncated at any point to select the desired quality.

ZFP supports several compression modes, including (1) targeting a specific compression ratio, (2) bounding the pointwise absolute error, or (3) losslessly compressing the data, and more. You can find out more about the other modes at https://zfp.readthedocs.io/en/release1.0.1/modes.html.

We use ZFP v1.0.1 with the ZFP_ROUNDING_MODE=ZFP_ROUND_FIRST and ZFP_WITH_TIGHT_ERROR=ON settings, also known as ZFP-ROUND ^[2]^[3], which produces unbiased compression errors and higher compression ratios.

from pathlib import Path

import netCDF4
import numpy as np
import xarray as xr

data = Path("data")

import earthkit.plots

from quickplot import quickplot

Importing the `Zfp` compressor¶

from numcodecs_wasm_zfp import Zfp

?Zfp

Init signature:
Zfp(
    mode,
    _version='0.2.0',
    max_bits=None,
    max_prec=None,
    min_bits=None,
    min_exp=None,
    non_finite='deny',
    precision=None,
    rate=None,
    tolerance=None,
)
Docstring:     
Codec providing compression using ZFP

Parameters
----------
mode : ...
     - "expert": The most general mode, which can describe all four other modes
    
     - "fixed-rate": In fixed-rate mode, each d-dimensional compressed block of `$4^d$`
        values is stored using a fixed number of bits. This number of
        compressed bits per block is amortized over the `$4^d$` values to give
        a rate of `$rate = \frac{maxbits}{4^d}$` in bits per value.
    
     - "fixed-precision": In fixed-precision mode, the number of bits used to encode a block may
        vary, but the number of bit planes (the precision) encoded for the
        transform coefficients is fixed.
    
     - "fixed-accuracy": In fixed-accuracy mode, all transform coefficient bit planes up to a
        minimum bit plane number are encoded. The smallest absolute bit plane
        number is chosen such that
        `$minexp = \text{floor}(\log_{2}(tolerance))$`.
    
     - "reversible": Lossless per-block compression that preserves integer and floating point
        bit patterns.
_version : ..., optional, default = "0.2.0"
    The codec's encoding format version. Do not provide this parameter explicitly.
max_bits : ..., optional
    Maximum number of bits used to represent a block
max_prec : ..., optional
    Maximum number of bit planes encoded
min_bits : ..., optional
    Minimum number of compressed bits used to represent a block
min_exp : ..., optional
    Smallest absolute bit plane number encoded.
    
    This parameter applies to floating-point data only and is ignored
    for integer data.
non_finite : ..., optional, default = "deny"
    ZFP non-finite values mode
precision : ..., optional
    Number of bit planes encoded
rate : ..., optional
    Rate in bits per value
tolerance : ..., optional
    Absolute error tolerance
File:           ~/egu26-compression-sc2.5/.venv/lib/python3.13/site-packages/numcodecs_wasm_zfp/__init__.py
Type:           ABCMeta
Subclasses:

Targeting a specific compression ratio¶

ZFP can target a specific compression ratio using:

cr = 10  # x10 compression

# dtype = da.dtype
dtype = np.dtype(np.float64)  # for example

Zfp(mode="fixed-rate", rate=dtype.itemsize * 8 / cr)

Zfp(mode='fixed-rate', rate=6.4, non_finite='deny', _version='0.2.0')

Bounding the pointwise absolute error¶

ZFP can bound the absolute error using:

eb_abs = 0.1

Zfp(mode="fixed-accuracy", tolerance=eb_abs)

Zfp(mode='fixed-accuracy', tolerance=0.1, non_finite='deny', _version='0.2.0')

It is worth noting that ZFP often produces errors that are significantly below the error bound.

Bounding the pointwise relative error¶

ZFP can, with some caveats, be configured in export mode to bound the pointwise relative error ^[4]. The easiest way to bound the pointwise relative error with ZFP is to transform the relative error bound into an absolute error bound ^[5] using a metacompressor such as the pw_rel_compressor_plugin in LibPressio ^[6] or the numcodecs_pw_ratio.PointwiseRatioErrorBoundedCodec port:

from numcodecs_pw_ratio import PointwiseRatioErrorBoundedCodec
from numcodecs_wasm_zstd import Zstd

eb_rel = 0.01

PointwiseRatioErrorBoundedCodec(
    # transform pointwise relative error bound into pointwise ratio error bound
    eb_ratio=1 + eb_rel,
    # mark how the absolute error is configured
    eb_abs_marker="$eb_abs",
    # lossy compressor that will use an absolute error bound
    log_codec={
        **Zfp(mode="fixed-accuracy", tolerance=-1).get_config(),
        "tolerance": "$eb_abs",
    },
    # lossless compressor for compressing the data signs
    sign_codec=Zstd(level=3),
)

PointwiseRatioErrorBoundedCodec(eb_ratio=1.01, eb_abs_marker='$eb_abs', log_codec={'id': 'zfp.rs', 'mode': 'fixed-accuracy', 'tolerance': '$eb_abs', 'non_finite': 'deny', '_version': '0.2.0'}, sign_codec=Zstd(level=3, _version='0.1.0'))

Preserving NaN Missing Values¶

ZFP does not yet support preserving infinite and NaN values during lossy compression, though work on supporting special values is ongoing ^[7]. By default, the non_finite="deny" setting will raise an error when these values are encountered. It is also possible to pass non_finite="allow-unsafe" to encode data with these values, though this may cause undefined behaviour and produce incorrect values in the $4 \times \ldots \times 4$ blocks that contain non-finite values.

Example¶

# Load the data
ds = xr.open_dataset(
    data / "hplp" / "hplp_sfc_regridded_t_025deg_levels_steps_204_216_228_240.nc",
    engine="netcdf4",
    decode_timedelta=True,
)
da = ds["2t"]

eb_abs = 0.1  # 0.1 K

codec = Zfp(mode="fixed-accuracy", tolerance=eb_abs)

# encode and decode the data
da_enc = codec.encode(da.values)
da_dec = da.copy(data=codec.decode(da_enc))

# plot a comparison figure
fig = earthkit.plots.Figure(
    size=(15, 4),
    rows=1,
    columns=3,
)

quickplot(da, fig.add_map(0, 0), title="Original {default_title}")
quickplot(
    da_dec, fig.add_map(0, 1), title="ZFP", cr=da.nbytes / np.array(da_enc).nbytes
)
quickplot(da_dec - da, fig.add_map(0, 2), error=True, title="Compression Error")

fig.show()

Footnotes¶

Lindstrom, P. (2014). Fixed-Rate Compressed Floating-Point arrays. IEEE Transactions on Visualization and Computer Graphics, 20(12), 2674–2683. Available from: Lindstrom (2014).
↩
Hammerling, D. M., Baker, A. H., Pinard, A., & Lindstrom, P. (2019). A Collaborative Effort to Improve Lossy Compression Methods for Climate Data. 2019 IEEE/ACM 5th International Workshop on Data Analysis and Reduction for Big Scientific Data (DRBSD-5), 16–22. Available from: Hammerling et al. (2019).
↩
Fox, A., & Lindstrom, P. (2026). Enhancing ZFP: a statistical approach to understanding and reducing error bias in a lossy Floating-Point compression algorithm. SIAM Journal on Scientific Computing, 48(1), B1–B30. Available from: Fox & Lindstrom (2026).
↩
https://zfp.readthedocs.io/en/release1.0.1/faq.html#q-relerr
↩
Liang, X., Di, S., Tao, D., Chen, Z., & Cappello, F. (2018). An Efficient Transformation Scheme for Lossy Data Compression with Point-Wise Relative Error Bound. 2018 IEEE International Conference on Cluster Computing (CLUSTER), 179–189. Available from: Liang et al. (2018).
↩
Underwood, R., Malvoso, V., Calhoun, J. C., Di, S., & Cappello, F. (2021). Productive and Performant Generic Lossy Data Compression with LibPressio. 2021 7th International Workshop on Data Analysis and Reduction for Big Scientific Data (DRBSD-7), 1–10. Available from: Underwood et al. (2021).
↩
Lindstrom, P. (2025). Supporting special values in ZFP. Available from: Lindstrom (2025).
↩

References¶

Lindstrom, P. (2014). Fixed-Rate Compressed Floating-Point Arrays. IEEE Transactions on Visualization and Computer Graphics, 20(12), 2674–2683. 10.1109/tvcg.2014.2346458
Hammerling, D. M., Baker, A. H., Pinard, A., & Lindstrom, P. (2019). A Collaborative Effort to Improve Lossy Compression Methods for Climate Data. 2019 IEEE/ACM 5th International Workshop on Data Analysis and Reduction for Big Scientific Data (DRBSD-5), 16–22. 10.1109/drbsd-549595.2019.00008
Fox, A., & Lindstrom, P. (2026). Enhancing ZFP: A Statistical Approach to Understanding and Reducing Error Bias in a Lossy Floating-Point Compression Algorithm. SIAM Journal on Scientific Computing, 48(1), B1–B30. 10.1137/24m1679586
Liang, X., Di, S., Tao, D., Chen, Z., & Cappello, F. (2018). An Efficient Transformation Scheme for Lossy Data Compression with Point-Wise Relative Error Bound. 2018 IEEE International Conference on Cluster Computing (CLUSTER), 179–189. 10.1109/cluster.2018.00036
Underwood, R., Malvoso, V., Calhoun, J. C., Di, S., & Cappello, F. (2021). Productive and Performant Generic Lossy Data Compression with LibPressio. 2021 7th International Workshop on Data Analysis and Reduction for Big Scientific Data (DRBSD-7), 1–10. 10.1109/drbsd754563.2021.00005
Lindstrom, P. (2025). Supporting Special Values in ZFP. Office of Scientific. 10.2172/2998448

Importing the Zfp compressor¶