INTPYTHON-792 Add support for window functions

[timgraham]

Claude did the heavy lifting. I did a little refactoring, made some cosmetic edits, and removed unneeded code.

I added additional test coverage in Django's test suite: django/django#21515

[NoahStapp]

According to the comment, "window stages must precede the wrapping pipeline", but here we add the window stages to the end of the pipeline?

[timgraham]

The wrapping pipeline is added in the line below the added code:
subquery.aggregation_pipeline.extend(wrapping_result_pipeline).

[NoahStapp]

A subquery with __in will cause a misordering here:

  top_per_dept = (
      Employee.objects.annotate(r=Window(Rank(), order_by=F("salary").desc()))
      .filter(r=1)
      .values("name")
  )

result = list(
    Employee.objects.filter(name__in=Subquery(top_per_dept))
    .values_list("name", flat=True)
)

# causes this DatabaseError
pymongo.errors.OperationFailure: Executor error during aggregate command on namespace: test_djangotests_windowbugs.window_bugs__employee :: caused by :: $in requires an array as a second argument, found: missing, full error: {'ok': 0.0, 'errmsg': 'Executor error during aggregate command on namespace: test_djangotests_windowbugs.window_bugs__employee :: caused by :: $in requires an array as a second argument, found: missing', 'code': 40081, 'codeName': 'Location40081'}

[timgraham]

Added test_window_qualify_filter_in_subquery.

[NoahStapp]

Two windows with different order_by expressions in the same partition will both have the same field names and cause a collision. For example:

qs = Employee.objects.annotate(
            a=Window(
                Sum("salary"),
                partition_by="department",
                order_by=(F("age") + F("salary")).asc(),
            ),
            b=Window(
                Sum("salary"),
                partition_by="department",
                order_by=(F("salary") + F("hire_date")).asc(),
            ),
        ).order_by("name")

Both will use __wsort1.

[timgraham]

Added test_multiple_windows_different_complex_order_by in django/django#21515.

[NoahStapp]

Window(StdDev), Window(Variance), and Window(Count)` all produce invalid MQL:

Window(StdDev) -> {alias: {"$stddev_samp/stddev_pop": ... # not valid MQL operators
Window(StdDev) -> {alias: {"$var_samp/var_pop": ... # not valid MQL operators
Window(Count) -> {alias: {"$count": ... # must have a value of {}

[timgraham]

Added test_window_count, test_window_stddev, test_window_variance.

[NoahStapp]

Window(Sum('salary', filter=Q(department='ENG'))) drops the filter here since only expr[0] is read.

[timgraham]

Added test_aggregate_filter, test_stddev_filter, test_variance_filter.

[NoahStapp]

SQL ntile puts extras in the first buckets, here we put extras in the last buckets. Is that difference significant and if so do should we document or fix it?

[timgraham]

Fixed. Added test_ntile_uneven_distribution.

[NoahStapp]

Hardcoded window means any user-passed Window.frame is ignored here.

[timgraham]

Added test_nthvalue_full_partition_frame.

[NoahStapp]

A subquery with __in will cause a misordering here:

  top_per_dept = (
      Employee.objects.annotate(r=Window(Rank(), order_by=F("salary").desc()))
      .filter(r=1)
      .values("name")
  )

result = list(
    Employee.objects.filter(name__in=Subquery(top_per_dept))
    .values_list("name", flat=True)
)

# causes this DatabaseError
pymongo.errors.OperationFailure: Executor error during aggregate command on namespace: test_djangotests_windowbugs.window_bugs__employee :: caused by :: $in requires an array as a second argument, found: missing, full error: {'ok': 0.0, 'errmsg': 'Executor error during aggregate command on namespace: test_djangotests_windowbugs.window_bugs__employee :: caused by :: $in requires an array as a second argument, found: missing', 'code': 40081, 'codeName': 'Location40081'}

[NoahStapp]

nulls_first and nulls_last aren't respected here:

qs = (
                Employee.objects.annotate(
                    r=Window(Rank(), order_by=F("hire_date").asc(nulls_last=True))
                )
                .filter(name="nullhire")
                .values_list("r", flat=True)
            )
# nulls are first

[timgraham]

Added test_window_order_by_nulls_last and test_window_order_by_nulls_first.

[NoahStapp]

This fallback doesn't catch other operators that also require a sortBy field:

list(Employee.objects.annotate(r=Window(Rank()))))

# Produces this error
pymongo.errors.OperationFailure: $rank must be specified with a top level sortBy expression with exactly one element, full error: {'ok': 0.0, 'errmsg': '$rank must be specified with a top level sortBy expression with exactly one element', 'code': 5371602, 'codeName': 'Location5371602'}

[timgraham]

Added test_rank_no_order_by, test_dense_rank_no_order_by.