Easy Compression with numcodecs

In this short course, we use the simple numcodecs ^[1] API for compression. In numcodecs, compressors are called Codecs and only have two important methods: codec.encode(data) and codec.decode(encoded). numcodecs works great for the simple case of compressing and decompressing in-memory numpy arrays in one go, though it can also be used with chunked data.

This product includes software produced by UChicago Argonne, LLC under Contract No. DE-AC02-06CH11357 with the Department of Energy.

Configuration¶

Each codec is a class that inherits from numcodecs.abc.Codec. Codecs are configured upon instantiation using keyword arguments. For example, here we create an instance of the lossless Zlib compressor:

We can obtain the full configuration of the codec in JSON format using codec.get_config():

And then use numcodecs.get_codec(config) to recreate the codec from its configuration:

Compression¶

We use the codec.encode(data) function to encode the data:

Decompression¶

We use the codec.decode(encoded) function to decode the data:

Some codecs, like Zlib, need to know the type and shape of the original data to correctly decompress it. For these cases, it is also possible to decode directly into an existing array using codec.decode(encoded, out=decoded):

Building up more complex compressors¶

It is often useful to combine several codecs, e.g. to use one to transform the data and one to compress it. Meta-compressors are such flexible combinators, and some are provided in the numcodecs-combinators ^[2] package:

CodecStack(a, b, c) combines codecs a, b, and c into a linear stack. During encoding, the data is first encoded by a, then b, then c. During decoding, the encoded data is first decoded by c, then by b, then a. The CodecStack supports an arbitrary number of sub-codecs.

The CodecStack also provides two useful helper methods:

• CodecStack.encode_decode(data) encodes and then decodes the data array in one go.

• CodecStack.encode_decode_data_array(da) encodes and then decodes an xarray.DataArray and preserves all metadata. If the data is chunked, the sub-codecs are applied independently to each data chunk.

The FramedCodecStack is a specialised CodecStack that remembers the type and shape of the data during encoding, encodes it alongside the encoded data, and then uses this information during decoding to provide the out parameter to each codec.decode. The FramedCodecStack is thus useful when trying to stack codecs that require this information to decode correctly, or to ensure that any codec produces its encoded data as a byte string.

Finally, the PickBestCodec(a, b, c) combinator tries to encode the data with codecs a, b, or c, chooses the one with the greatest reduction in byte size, and outputs its encoded data alongside the index of the codec that was picked. During decoding, the picked sub-codec is used to decode the data. The PickBestCodec supports an arbitrary number of sub-codecs.

Portable scientific compressors with numcodecs-wasm¶

In this short course, we utilize the numcodecs-wasm ^[3] project, which provides several scientific compressors in a numcodecs-compatible form. The compressors are compiled to WebAssembly to ensure that compression and decompression can be performed reproducibly across any CPU machine ^[4]. Each compressor is published as its own Python package on PyPi, e.g. numcodecs-wasm-sz3 for the SZ3 compressor.

The complete list of currently supported compressors is:

• numcodecs_wasm_asinh: $\text{asinh}(x)$

• numcodecs_wasm_bit_round: bit rounding

• numcodecs_wasm_ebcc: EBCC

• numcodecs_wasm_fixed_offset_scale: $\frac{x - o}{s}$

• numcodecs_wasm_fourier_network: fourier feature neural network

• numcodecs_wasm_identity: $f(x) = x$

• numcodecs_wasm_jpeg2000: JPEG 2000

• numcodecs_wasm_lc: LC

• numcodecs_wasm_linear_quantize: linear quantization

• numcodecs_wasm_log: $\ln(x)$

• numcodecs_wasm_pco: pcodec

• numcodecs_wasm_pressio: Pressio (work-in-progress)

• numcodecs_wasm_qpet_sperr: QPET-SPERR

• numcodecs_wasm_random_projection: random projection

• numcodecs_wasm_reinterpret: binary reinterpret

• numcodecs_wasm_round: rounding

• numcodecs_wasm_sperr: SPERR

• numcodecs_wasm_stochastic_rounding: stochastic rounding

• numcodecs_wasm_swizzle_reshape: array axis swizzle and reshape

• numcodecs_wasm_sz3: SZ3

• numcodecs_wasm_tthresh: Tthresh

• numcodecs_wasm_uniform_noise: add uniform noise

• numcodecs_wasm_zfp: ZFP-ROUND

• numcodecs_wasm_zfp_classic: ZFP

• numcodecs_wasm_zlib: Zlib

• numcodecs_wasm_zstd: Zstdandard

While numcodecs-wasm provides reproducibility and makes it easy to quickly test out different compressors, it has reduced performance as compressors are run single-threaded in a WebAssembly engine and not as native code. Furthermore, the codecs run inside a sandboxed 32 bit memory space and can typically only compress arrays up to 1-2 GB in size. To compress larger data with numcodecs-wasm, please chunk the data using xarray and use CodecStack.encode_decode_data_array(da).