Skip to content

Commit 5121ed2

Browse files
committed
[DOC] Callout the aggregation result may be approximate
Signed-off-by: Lantao Jin <ltjin@amazon.com>
1 parent 9a6f2b0 commit 5121ed2

3 files changed

Lines changed: 35 additions & 2 deletions

File tree

docs/user/ppl/cmd/stats.rst

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -68,6 +68,39 @@ The stats command supports the following aggregation functions:
6868

6969
For detailed documentation of each function, see `Aggregation Functions <../functions/aggregations.rst>`_.
7070

71+
Limitations:
72+
============
73+
74+
The bucket aggregation result may be approximate in large dataset
75+
-----------------------------------------------------------------
76+
77+
In OpenSearch, ``doc_count`` values for a terms bucket aggregation may be approximate. As a result, any aggregations (such as ``sum`` and ``avg``) on the terms bucket aggregation may also be approximate.
78+
For example, the following PPL query (find the top 10 URLs) may return an approximate result if the cardinality of ``URL`` is high.
79+
80+
PPL query::
81+
82+
> source=hits
83+
| stats bucket_nullable=false count() as c by URL
84+
| sort - c
85+
| head 10
86+
87+
This query is pushed down to a terms bucket aggregation DSL query with ``"order": { "_count": "desc" }``. In OpenSearch, this terms aggregation may throw away some buckets.
88+
89+
90+
Sorting by ascending doc_count may produce inaccurate results
91+
-------------------------------------------------------------
92+
93+
Similar to above PPL query, the following query (find the rare 10 URLs) often produces inaccurate results.
94+
95+
PPL query::
96+
97+
> source=hits
98+
| stats bucket_nullable=false count() as c by URL
99+
| sort + c
100+
| head 10
101+
102+
A term that is globally infrequent might not appear as infrequent on every individual shard or might be entirely absent from the least frequent results returned by some shards. Conversely, a term that appears infrequently on one shard might be common on another. In both scenarios, rare terms can be missed during shard-level aggregation, resulting in incorrect overall results.
103+
71104
Example 1: Calculate the count of events
72105
========================================
73106

docs/user/ppl/functions/aggregations.rst

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@ Aggregation Functions
1111

1212
Description
1313
============
14-
| Aggregation functions perform calculations across multiple rows to return a single result value. These functions are used with ``stats`` and ``eventstats`` commands to analyze and summarize data.
14+
| Aggregation functions perform calculations across multiple rows to return a single result value. These functions are used with ``stats``, ``eventstats`` and ``streamstats`` commands to analyze and summarize data.
1515
1616
| The following table shows how NULL/MISSING values are handled by aggregation functions:
1717

docs/user/ppl/index.rst

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -102,7 +102,7 @@ The query start with search command and then flowing a set of command delimited
102102

103103
* **Functions**
104104

105-
- `Aggregation Functions <functions/aggregation.rst>`_
105+
- `Aggregation Functions <functions/aggregations.rst>`_
106106

107107
- `Collection Functions <functions/collection.rst>`_
108108

0 commit comments

Comments
 (0)