Our library is made up of multiple components that are partitioned into GitHub repositories by language and dependencies. The dependencies of the core components are kept to a bare minimum to enable flexible integration into many different environments. The Platform Adaptor components will have major dependencies on the respective platform envionments.
If you have a specific issue or bug report that impacts only one of these components please open an issue on the respective component. If you are a developer and wish to submit a PR, please choose the appropriate repository.
If you like what you see, give us a Star on these sites!
The key sketches of the Apache DataSketches libraries are available in five (soon six) programming languages. By design, a sketch that is available in one language that is also available in a different language will be “functionally compatible” via serialization. For example, when serialized into its compact form, a sketch created by the DataSketches C++ library, can be read by the DataSketches Java library and visa versa.
Because of differences inherent in the languages, there will be some differences in the APIs, but we try to make the same basic functionality available across all the languages.
| Repository | Distribution | Comments |
|---|---|---|
| Java Core | Downloads | This is the original and the most comprehensive collection of sketch algorithms. It has a dependency on the Memory component |
| C++ Core | Downloads | C++ was our second core language library and provides most of the major algorithms available in Java as well as a few sketches unique to C++. |
| Python Core | Downloads, PyPI | Python was our third core language library and contains most of the major sketch families that are in Java and C++. All the Python sketches are backed by the C++ library via Pybind. |
| Go Core | Downloads | Go is our forth core language and is still evolving. |
| Rust Core | Downloads | Rust is our fifth core language and is still evolving. |
| CUDA Core | Not Formally Released. | CUDA will be our sixth core language and is still evolving. |
Adapters integrate the core library components into the aggregation APIs of specific data processing platforms. Some of these adapters are available as an Apache DataSketches distribution, other adapters are directly integrated into the target platform.
| Repository | Distribution | Comments |
|---|---|---|
| Google BigQuery Adaptor | Downloads | Depends on C++ Core, Integrations |
| Apache Hive Adaptor | Downloads | Depends on Java Core, Integrations |
| Apache Pig Adaptor | Downloads | Depends on Java Core, Integrations |
| PostgreSQL Adaptor | Downloads, pgxn.org | Depends on C++ Core, Integrations |
| Apache Druid Adaptor | Apache Druid Release | Depends on Java Core, Integrations |
| DuckDB Adaptor | DuckDB community extension | Community-maintained by Query.Farm; depends on C++ Core. Documentation |
| Repository | Distribution | Comments |
|---|---|---|
| Memory (supports Java Core) | Downloads | Provides high-performance access to off-heap memory. This is no longer used starting with datasketches-java version 9.0.0. |
| Characterization | Not Formally Released | Used for long-running studies of accuracy and speed performance over many different parameters. |
| Website | Not Formally Released | Public website |
| Vector | Not Formally Released | This component implements the Frequent Directions Algorithm [GLP16]. It is still experimental in that the theoretical work has not yet supplied a suitable measure of error for production work. It can be used as is, but it will not go through a formal Apache Release until we can find a way to provide better error properties. It dependends on the Memory component. |
| Server | Not Formally Released | Under development |
