Homomorphic Encryption (HE) enables computations on encrypted data without revealing the underlying information. The Cheon-Kim–Kim–Song (CKKS) scheme is specifically designed for approximate arithmetic, enabling evaluation of machine learning inference, statistical analysis, and other numerical workloads directly on encrypted data. However, its computational cost is still a major obstacle to large-scale adoption, making performance evaluation and optimization a key research topic.
To support reproducible performance evaluation of CKKS workloads, the University of Pisa developed CKKSBenchmark as part of the AERO Project. It is an open-source benchmarking framework designed to evaluate the performance of the CKKS scheme implemented in Microsoft SEAL across x86_64 and AArch64 platforms. The benchmark provides a flexible and reproducible environment for comparing processor architectures, compiler optimizations, and SIMD-based optimizations under consistent experimental conditions.
CKKSBenchmark Architecture
CKKSBenchmark orchestrates the execution of configurable CKKS workloads using Microsoft SEAL and collects performance measurements through different backends, including wall-clock time and CPU cycles. Execution time is suitable for application-level evaluation, while CPU cycles provide a frequency-independent metric for comparing architectures and low-level optimizations. The framework exports the collected results in a CSV structured format, enabling reproducible performance evaluation and fair comparison across different platforms and optimization techniques. Figure 1 shows the high-level architecture of CKKSBenchmark.
Supported Operations
As shown in Table 1, CKKSBenchmark supports benchmarking both individual CKKS operations and complete end-to-end workflows. In workflow mode, the framework executes a realistic sequence of CKKS operations while collecting performance metrics for each operation individually. Before reporting execution time or CPU cycle counts, the framework automatically verifies the correctness of the computed results against a reference output.
| Category | Operations |
| Setup | Key generation |
| Plaintext Operations | Encode, Decode |
| Ciphertext Operations | Encrypt, Decrypt |
| Homomorphic Arithmetic | Add, Multiply, Square |
| Ciphertext Management | Relinearization, Rescale |
| Slot Operations | Rotate Vector |
| End-to-end Workflow | Complete CKKS evaluation pipeline |
In particular, the end-to-end workflow models a representative CKKS application by evaluating the expression −(A 2 +B⋅C+coeff d ⋅D+coeff u ) followed by a one-position forward and backward rotation of the resulting ciphertext. The workflow operates on randomly generated input data and can be configured to execute an arbitrary number of repetitions, enabling the collection of statistically meaningful performance measurements. To avoid redundant measurements, performance metrics are collected only for the first invocation of each CKKS primitive within the workflow (light blue node).
Configuration and metrics
A key feature of CKKSBenchmark is its high degree of configurability. The framework allows users to customize both the cryptographic parameters and the benchmark execution settings, including:
- Polynomial modulus degree
- Coefficient modulus chain
- Scaling factor
- Security level
- Benchmark mode (isolated operation or end-to-end workflow).
- Number of repetitions and warm-up iterations.
- Performance metric (wall-clock time or CPU cycles).
CKKSBenchmark supports both wall-clock time and Linux perf CPU cycle measurements, combining configurable repeated executions with warm-up exclusion and IQR-based outlier removal to ensure reliable and reproducible performance evaluation.
Open Source
CKKSBenchmark is released as an open-source project through the AERO Project GitHub organization, promoting reproducible research and facilitating the evaluation of new optimization techniques for Microsoft SEAL. The repository includes the benchmark source code, comprehensive documentation, example configurations, and build instructions for both x86-64 and Arm/AArch64 platforms.
Additional information available at https://github.com/AERO-Project-EU/CKKSBenchmark