======================
What's new in v2.12?
======================

C API Improvements
------------------

New functions for safely retrieving string data of unknown length using a
two-call pattern (first call with NULL buffer to query length, then call
again with allocated buffer to retrieve data):

- ``adios2_get_string()`` - for string variables
- ``adios2_attribute_string_data()`` - for single-value string attributes
- ``adios2_attribute_string_data_array()`` - for string array attributes

This addresses a gap in the C API where string lengths could not be queried
before reading.

Selection API for Get() Operations
-----------------------------------

A new ``Selection`` class provides an explicit, self-contained way to specify
what data to read in ``Get()`` calls. Instead of mutating the ``Variable``
object with ``SetSelection()``, ``SetBlockSelection()``, ``SetStepSelection()``,
etc., all selection parameters are bundled into a ``Selection`` object that is
passed directly to ``Get()``. This avoids relying on potentially stale variable
state and makes the intent of each ``Get()`` call clear at the call site.

A selection has two independent aspects: a spatial selection —
``Selection::All()`` (the default — reads the entire variable) or
``Selection::BoundingBox(start, count)`` (hyperslab) — and an optional block
selection via ``Selection::Block(blockID)`` or ``.WithBlock(blockID)``. These
are orthogonal: a block ID can be combined with a bounding box to read a
sub-region of a specific write block. Additional parameters (steps, memory
layout, accuracy) can be added via fluent ``With*()`` methods or mutable
``Set*()`` methods. A ``ToString()`` method is provided for debugging.

.. code-block:: c++

   // Select everything
   engine.Get(var, data, adios2::Selection::All());

   // Fluent style — bounding box with step range
   auto sel = adios2::Selection::BoundingBox({0, 0}, {10, 20}).WithSteps(0, 5);
   engine.Get(var, data, sel);

   // Mutable style — reuse across iterations
   adios2::Selection sel;
   sel.SetBoundingBox({0, 0}, {10, 20});
   for (size_t step = 0; step < nsteps; ++step)
   {
       sel.SetSteps(step, 1);
       engine.Get(var, buffers[step], sel);
   }

   // Debug output
   std::cout << sel.ToString() << std::endl;

Existing ``SetSelection()``-based APIs remain available and are not deprecated.
See :ref:`Selection` for full documentation.

Plugin Interface v2 (Breaking Change)
--------------------------------------

Both the engine and operator plugin interfaces have been redesigned as
standalone classes that no longer inherit from internal core types
(``core::Engine`` and ``core::Operator``). This is a **breaking change** for
existing plugins written against the v1 interface (ADIOS2 2.11 and earlier).

**What changed:**

- **Engine plugins** now inherit from ``PluginEngineInterface`` (in
  ``adios2/plugin/PluginEngineInterface.h``) which uses public C++ API types
  (``adios2::IO``, ``adios2::Variable<T>``) instead of internal core types.
  The ``EngineCreate()`` signature now takes ``adios2::IO`` and ``adios2::Mode``
  instead of ``core::IO &`` and ``helper::Comm``.
- **Operator plugins** now inherit from ``PluginOperatorInterface`` (in
  ``adios2/plugin/PluginOperatorInterface.h``) which is fully self-contained
  with no dependencies on ADIOS internals.
- Plugin shared libraries now only need to link against ``adios2::cxx``
  (not ``adios2::core``).

**Migrating from v1:** Update your plugin class to inherit from the new
interface, replace any ``core::IO`` / ``core::Variable<T>`` usage with the
public ``adios2::IO`` / ``adios2::Variable<T>`` types, and update the
``EngineCreate()`` / ``EngineDestroy()`` signatures accordingly. See the
examples in ``examples/plugins/`` for reference implementations.

**New engine plugin capabilities:** The v2 engine interface has also been
extended with optional methods for more advanced functionality:

- **Selection-based Get**: ``DoGetSync()`` and ``DoGetDeferred()`` overloads
  that accept a ``Selection`` parameter, enabling sub-array and block reads.
- **Span-based Put**: ``DoPut()`` with a ``Span`` parameter for zero-copy
  writes of primitive types.
- **Step queries**: ``Steps()`` to report the total number of available steps.
- **Block introspection**: ``MinBlocksInfo()`` to return block decomposition
  metadata for a given variable and step.

All new methods have empty default implementations, so they are optional.
See :ref:`Plugins` for full documentation.

XRootD Server Resource Management
-----------------------------------

The XRootD server plugin used for remote campaign access now includes
resource-aware cache management:

- **FD and metadata eviction**: The file pool tracks per-file file descriptor
  costs and metadata memory usage, evicting idle entries when configurable
  limits are approached. Limits can be set via environment variables
  (``ADIOS_POOL_FD_LIMIT``, ``ADIOS_POOL_METADATA_LIMIT``) or at runtime
  through the admin HTTP interface.

- **Shared tar FD cache**: When multiple datasets are packed into a single
  tar file (common in campaign archives), a process-wide singleton shares
  one file descriptor across all engines reading from the same tar,
  reducing FD consumption from N to 1 per tar file.

- **Batched data requests**: The remote reader now batches multiple
  variable reads into a single network round-trip, reducing latency
  for workloads that read many variables per step.

Cross-Endian Interoperability
-----------------------------

The ``ADIOS2_USE_Endian_Reverse`` CMake option has been removed. Cross-endian
file interoperability (reading files written on a machine with different byte
order) is now always enabled. This means files written on big-endian systems
can be read on little-endian systems and vice versa without any special build
configuration.

S3 Object Storage for BP5 Data
------------------------------

BP5 can now write data files to S3-compatible object storage (Amazon S3, MinIO,
Ceph, etc.) while keeping metadata on the local filesystem. This hybrid model
provides cheap bulk storage for data with fast local metadata access.

Set ``DataFileTransport=awssdk``, ``S3Endpoint``, and ``S3Bucket`` as engine
parameters. Credentials are accepted as parameters or via standard AWS
environment variables (``AWS_ACCESS_KEY_ID``, ``AWS_SECRET_ACCESS_KEY``).

Key features:

- **Multi-object mode** (default): each data segment is a separate S3 object,
  providing automatic crash recovery
- **Async write**: automatically enabled for S3 transports so uploads overlap
  with computation
- **Append and AppendAfterSteps**: supported via segment discovery and
  selective object deletion
- **Stale object cleanup**: previous data objects are deleted on ``Open(Write)``
- **s3.json sidecar**: written locally so readers auto-detect the S3 location

See :ref:`BP5 S3 Object Storage<S3 Object Storage>` for configuration details.

C++17 Minimum Requirement
--------------------------

The minimum C++ standard required to build ADIOS2 has been raised from C++14
to C++17.

Python Bindings: Nanobind Migration
-------------------------------------

The Python bindings have been migrated from pybind11 to nanobind. Nanobind
produces smaller, faster extension modules and enables stable-ABI (``abi3``)
wheels for Python 3.12 and later, so a single wheel binary works across
multiple Python versions without recompilation.

Asymmetric Encryption Operator
--------------------------------

A new ``Encryption`` operator supports asymmetric encryption of variable data
using libsodium's sealed-box construction (X25519 key exchange +
XSalsa20-Poly1305). Writers encrypt data using the recipient's public key;
only the holder of the corresponding private key can decrypt. No writer
keypair is required, so the data producer does not need to be trusted with
decryption credentials.

SZ3 Compression Operator
--------------------------

A new ``SZ3`` operator is available for lossy data compression using the SZ3
library, extending the available compression options alongside SZ, ZFP, MGARD,
and BigWhoop.

RefactorProDM Operator
-----------------------

A new ``RefactorProDM`` operator integrates ADIOS2 with ProDM, a data
decomposition library by Dr. Xin Liang. When ProDM is available at build time,
this operator provides decomposition-based data reduction as an alternative to
traditional compressors.

Python Free-Threading Support
------------------------------

The Python bindings are now compatible with free-threaded Python (PEP 703,
``python3.13t``), allowing ADIOS2 to operate without the GIL in environments
that support it.

HTTPS Transport for Remote Access
-----------------------------------

An HTTPS-based remote transport (``XrootdHttpsRemote``) using libcurl has been
added as an alternative to the native XRootD protocol. This enables reading
ADIOS2 datasets over standard HTTPS URLs from XRootD SSI services, useful in
environments where native XRootD connectivity is unavailable.

Campaign Reader Improvements
------------------------------

The campaign reader received several improvements: host aliases in
``~/.config/hpc-campaign/hosts.yaml`` allow symbolic machine names to be
defined once and reused across dataset entries; XRootD connections can now be
declared as host entries; in-memory metadata caching reduces repeated file
access overhead; and the reader now strictly accepts ACA version 0.7, rejecting
unsupported versions with a clear error. For large archives the SQLite query was
rewritten as a single join across the dataset, replica, file, and resolution
tables, reducing open time from ~27 seconds to under one second for collections
of thousands of images.

TAR File Reading
-----------------

The BP5 reader now supports datasets packaged inside TAR archives. When a TAR
index is provided, BP5 datasets, text files, and images can be read from a
local TAR, an HTTPS-hosted TAR, or a TAR accessible via the remote server. This
is primarily used by the campaign reader to access archived datasets.

Python Stream: minmax()
------------------------

``adios2.Stream`` gains a new ``minmax(name, [step], [block_info_list])`` method
that returns the minimum and maximum values of a variable. In read mode or when
a step is specified it returns per-step statistics; in random-access mode
without a step argument it returns the global min/max.

Inline Engine Buffering
------------------------

The inline engine now supports basic output buffering, a step toward full
feature parity with file-based engines for in-situ workflows.

Windows and macOS pip Wheel Support
-------------------------------------

Automated integration tests for pip ``sdist`` and wheel builds on Windows and
macOS have been added to CI, verifying correct library loading and packaging
under ``pip install`` workflows.

REUSE License Compliance
-------------------------

ADIOS2 now complies with the REUSE specification, ensuring every source file
carries a machine-readable SPDX license and copyright notice.


===================
What's new in 2.11?
===================

This is a major release with new features and lots of bug fixes. The main
highlights include enhanced derived variables, remote data access, GPU
improvements, campaign management overhaul, and advanced compression operators.

Derived Variables
-----------------

Support for reader-side computed variables with expression evaluation. This
release adds comprehensive functionality including new mathematical operations
(trigonometric functions, multiplication, division, power) and the
ability to apply derived variables to aggregated arrays.

Remote Data Access Infrastructure
----------------------------------

Implementation of XRootD and remote server capabilities enabling distributed
file access. This includes support for remote streaming, metadata caching, and
optimized query handling across networked systems.

GPU and Memory Space Improvements
----------------------------------

Enhanced GPU backend support with memory selection adjustments, Kokkos
integration improvements, and bindings for cuPy pointers and torch tensors.
Addresses layout mismatches between user code and device memory.

Campaign Management System Overhaul
------------------------------------

Significant updates to campaign tracking using SQLite databases replacing JSON
files, configuration file support, and attribute handling. Includes multi-host
support and improved metadata organization.

Compression and Data Format Advances
-------------------------------------

New operators including BigWhoop compression and Complex MGARD support. ZFP now
supports 4D arrays, and improved MGARD efficiency through zstd compression of
lossless data portions.

OpenSSF Best Practices
-----------------------

ADIOS2 now adopts OpenSSF (Open Source Security Foundation) best practices
including enhanced security measures, automated vulnerability scanning, and
adherence to industry-standard development practices.


===================
What's new in 2.10?
===================

This is a major release with new features and lots of bug fixes. The main new feature is the new Python API. 

Python
------
Before, ADIOS had two separate APIs for Python. The low-level ("Full") API was written with Pybind11 and directly mimicked the C++ API. The high-level API was another, smaller, and more pythonesque API that allowed for easier scripting with Python. The main problems with these two were that they were independent, and that the high-level API was not complete. Once a developer needed a feature only available in the full API, they had to start from scratch writing a script with the full API. 

In 2.10, there is officially one Python API, written in Python, which in turn uses the old Pybind11 classes. The new API combines the high-level features of the old high-level API -- hopefully in a more consistent and likeable way, -- and the full feature set of the low-level bindings. 

.. note::

   Old scripts that used the full API can still run without almost any modification, just change the import line from ``import adios2`` to ``import adios2.bindings as adios2``

   Old scripts that used the high-level API must be modified to make them work with the new API, see :ref:`Transition from old API to new API`
u


See :ref:`Python APIs`


New/updated features
--------------------

 - BP5 is supported on Windows now 
 - SST and DataMan staging engines are GPU-Aware now
 - SYCL support added for Intel GPUs (besides CUDA and HIP for NVidia and AMD GPUs)
 - the SST/libfabric data transport now works on Frontier (besides the MPI data transport)


Packaging
----------

  - adios2 package is now on `PyPi <https://pypi.org/project/adios2/>`_


==================
What's new in 2.9?
==================

This is a major release with new features and lots of bug fixes.

General
-------

- GPU-Aware I/O enabled by using Kokkos. Device pointers can be passed to Put()/Get() calls directly. Kokkos 3.7.x required for this release. Works with CUDA, HIP and Kokkos applications.  https://adios2.readthedocs.io/en/latest/advanced/gpu_aware.html#gpu-aware-i-o
- GPU-compression. MGARD and ZFP operators can compress data on GPU if they are built for GPU. MGARD operator can be fed with host/device pointers and will move data automaticaly. ZFP operator requires matching data and compressor location.
- Joined Array concept (besides Global Array and Local Array), which lets writers dump Local Arrays (no offsets no global shape) that are put together into a Global Array by the reader. One dimension of the arrays is selected for this join operation, while other dimensions must be the same for all writers. https://adios2.readthedocs.io/en/latest/components/components.html?highlight=Joined#shapes 

File I/O
--------

- Default File engine is now BP5. If for some reason this causes problems, manually specify using "BP4" for your application.
- BP5 engine supports multithreaded reading to accelerate read performance for low-core counts.
- BP5 Two level metadata aggregation and reduction reduced memory impact of collecting metadata and therefore is more scalable in terms of numbers of variables and writers than BP4.
- Uses Blosc-2 instead of Blosc for lossless compression. The new compression operator is backward compatible with old files compressed with blosc. The name of the operator remains "blosc".

Staging
-------

- UCX dataplane added for SST staging engine to support networks under the UCX consortium
- MPI dataplane added for  SST staging engine. It relies on MPI intercommunicators to connect multiple independent MPI applications for staging purposes. Applications must enable multithreaded MPI for this dataplane.

Experimental features
---------------------

- Preliminary support for data structs. A struct can have single variables of basic types, and 1D fixed size arrays of basic types. Supported by BP5, SST and SSC engines. 
