Skip to content

Commit a11fc5d

Browse files
Merge pull request #211 from commercetools/210-fix-fetch-all
210 fix fetch all
2 parents 2bfc4f7 + 18328df commit a11fc5d

7 files changed

Lines changed: 223 additions & 192 deletions

File tree

README.md

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@
33
[![Build Status](https://travis-ci.org/commercetools/commercetools-sync-java.svg?branch=master)](https://travis-ci.org/commercetools/commercetools-sync-java)
44
[![codecov](https://codecov.io/gh/commercetools/commercetools-sync-java/branch/master/graph/badge.svg)](https://codecov.io/gh/commercetools/commercetools-sync-java)
55
[![Download](https://api.bintray.com/packages/commercetools/maven/commercetools-sync-java/images/download.svg) ](https://bintray.com/commercetools/maven/commercetools-sync-java/_latestVersion)
6-
[![Javadoc](http://javadoc-badge.appspot.com/com.commercetools/commercetools-sync-java.svg?label=Javadoc)](https://commercetools.github.io/commercetools-sync-java/v/v1.0.0-M6/)
6+
[![Javadoc](http://javadoc-badge.appspot.com/com.commercetools/commercetools-sync-java.svg?label=Javadoc)](https://commercetools.github.io/commercetools-sync-java/v/v1.0.0-M7/)
77
[![Known Vulnerabilities](https://snyk.io/test/github/commercetools/commercetools-sync-java/4b2e26113d591bda158217c5dc1cf80a88665646/badge.svg)](https://snyk.io/test/github/commercetools/commercetools-sync-java/4b2e26113d591bda158217c5dc1cf80a88665646)
88

99
Java API which exposes utilities for building update actions and automatic syncing of CTP data from external sources
@@ -21,7 +21,7 @@ Java API which exposes utilities for building update actions and automatic synci
2121
- [Ivy](#ivy)
2222
- [Roadmap](#roadmap)
2323
- [Release Notes](/docs/RELEASE_NOTES.md)
24-
- [Javadoc](https://commercetools.github.io/commercetools-sync-java/v/v1.0.0-M6/)
24+
- [Javadoc](https://commercetools.github.io/commercetools-sync-java/v/v1.0.0-M7/)
2525

2626
<!-- END doctoc generated TOC please keep comment here to allow auto update -->
2727
## Usage
@@ -56,20 +56,20 @@ most popular ones:
5656
<dependency>
5757
<groupId>com.commercetools</groupId>
5858
<artifactId>commercetools-sync-java</artifactId>
59-
<version>v1.0.0-M6</version>
59+
<version>v1.0.0-M7</version>
6060
</dependency>
6161
````
6262
#### Gradle
6363
````groovy
64-
implementation 'com.commercetools:commercetools-sync-java:v1.0.0-M6'
64+
implementation 'com.commercetools:commercetools-sync-java:v1.0.0-M7'
6565
````
6666
#### SBT
6767
````
68-
libraryDependencies += "com.commercetools" % "commercetools-sync-java" % "v1.0.0-M6"
68+
libraryDependencies += "com.commercetools" % "commercetools-sync-java" % "v1.0.0-M7"
6969
````
7070
#### Ivy
7171
````xml
72-
<dependency org="com.commercetools" name="commercetools-sync-java" rev="v1.0.0-M6"/>
72+
<dependency org="com.commercetools" name="commercetools-sync-java" rev="v1.0.0-M7"/>
7373
````
7474

7575

docs/RELEASE_NOTES.md

Lines changed: 14 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,7 @@
3333
<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
3434

3535

36+
- [v1.0.0-M7 - Dec 15, 2017](#v100-m7----dec-15-2017)
3637
- [v1.0.0-M6 - Dec 5, 2017](#v100-m6----dec-5-2017)
3738
- [v1.0.0-M5 - Nov 16, 2017](#v100-m5----nov-16-2017)
3839
- [v1.0.0-M4 - Nov 7, 2017](#v100-m4----nov-7-2017)
@@ -44,12 +45,24 @@
4445
<!-- END doctoc generated TOC please keep comment here to allow auto update -->
4546

4647
<!--
47-
### v1.0.0-M7 - Dec 19, 2017
48+
### v1.0.0-M8 - Dec 28, 2017
4849
[Commits](https://github.com/commercetools/commercetools-sync-java/compare/v1.0.0-M6...v1.0.0-M7) |
4950
[Javadoc](https://commercetools.github.io/commercetools-sync-java/v/v1.0.0-M7/) |
5051
[Jar](https://bintray.com/commercetools/maven/commercetools-sync-java/v1.0.0-M7)
5152
-->
5253

54+
55+
56+
### v1.0.0-M7 - Dec 15, 2017
57+
[Commits](https://github.com/commercetools/commercetools-sync-java/compare/v1.0.0-M6...v1.0.0-M7) |
58+
[Javadoc](https://commercetools.github.io/commercetools-sync-java/v/v1.0.0-M7/) |
59+
[Jar](https://bintray.com/commercetools/maven/commercetools-sync-java/v1.0.0-M7)
60+
61+
**Bug Fixes** (1)
62+
- **Commons** - Changed offset-based pagination of querying all elements to a limit-based with sorted ids approach
63+
to mitigate problems of previous approach. [#210](https://github.com/commercetools/commercetools-sync-java/issues/210)
64+
65+
5366
### v1.0.0-M6 - Dec 5, 2017
5467
[Commits](https://github.com/commercetools/commercetools-sync-java/compare/v1.0.0-M5...v1.0.0-M6) |
5568
[Javadoc](https://commercetools.github.io/commercetools-sync-java/v/v1.0.0-M6/) |

src/integration-test/java/com/commercetools/sync/integration/commons/CtpQueryUtilsIT.java

Lines changed: 1 addition & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -104,9 +104,7 @@ public void queryAll_WithCategoryCollectorCallbackWithUniformPageSplitting_Shoul
104104
final Set<String> pageCategoryKeys = new HashSet<>();
105105
categoryPages.forEach(page -> {
106106
assertThat(page.size()).isEqualTo(2);
107-
page.forEach(category -> {
108-
pageCategoryKeys.add(category.getKey());
109-
});
107+
page.forEach(category -> pageCategoryKeys.add(category.getKey()));
110108
});
111109
assertThat(pageCategoryKeys).hasSize(numberOfCategories);
112110
}

src/integration-test/java/com/commercetools/sync/integration/commons/utils/ITUtils.java

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@
66
import com.fasterxml.jackson.databind.ObjectMapper;
77
import io.sphere.sdk.client.SphereClient;
88
import io.sphere.sdk.client.SphereRequest;
9+
import io.sphere.sdk.models.Resource;
910
import io.sphere.sdk.queries.QueryDsl;
1011
import io.sphere.sdk.types.commands.TypeDeleteCommand;
1112
import io.sphere.sdk.types.queries.TypeQuery;
@@ -64,7 +65,7 @@ public static String getStatisticsAsJSONString(@Nonnull final BaseSyncStatistics
6465
* @param resourceMapper defines a mapper function that should be applied on each resource in the fetched page
6566
* from the query on the specified CTP project.
6667
*/
67-
public static <T, C extends QueryDsl<T, C>> void queryAndApply(
68+
public static <T extends Resource, C extends QueryDsl<T, C>> void queryAndApply(
6869
@Nonnull final SphereClient ctpClient,
6970
@Nonnull final Supplier<QueryDsl<T, C>> queryRequestSupplier,
7071
@Nonnull final Function<T, SphereRequest<T>> resourceMapper) {
Lines changed: 73 additions & 52 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
package com.commercetools.sync.commons.utils;
22

33
import io.sphere.sdk.client.SphereClient;
4+
import io.sphere.sdk.models.Resource;
45
import io.sphere.sdk.queries.QueryDsl;
56

67
import javax.annotation.Nonnull;
@@ -17,82 +18,102 @@ private CtpQueryUtils() {
1718
}
1819

1920
/**
20-
* Queries all elements matching a query by using an offset based pagination with page size 500.
21-
* The method takes a callback {@link Function} that returns a result of type {@code <S>} that is returned on every
22-
* page of elements queried. Eventually, the method returns a {@link CompletionStage} that contains a list of all
23-
* the results of the callbacks returned from every page.
21+
* Queries all elements matching a query by using a limit based pagination with a combination of id sorting and a
22+
* page size 500. More on the algorithm can be found here: http://dev.commercetools.com/http-api.html#offset.
2423
*
25-
* @param client commercetools client
26-
* @param query query containing predicates and expansion paths
27-
* @param callBack callback function that is called on every page queried.
28-
* @param <T> type of one query result element
29-
* @param <C> type of the query
30-
* @param <S> type of the returned result of the callback function on every page.
31-
* @return elements
24+
* <p>The method takes a callback {@link Function} that returns a result of type {@code <S>} that is returned on
25+
* every page of elements queried. Eventually, the method returns a {@link CompletionStage} that contains a list of
26+
* all the results of the callbacks returned from every page.
27+
*
28+
* <p>NOTE: This method fetches all paged results sequentially as opposed to fetching the pages in parallel.
29+
*
30+
* @param client commercetools client
31+
* @param query query containing predicates and expansion paths
32+
* @param pageMapper callback function that is called on every page queried
33+
* @param <T> type of one query result element
34+
* @param <C> type of the query
35+
* @param <S> type of the returned result of the callback function on every page.
36+
* @return a completion stage containing a list of mapped pages as a result.
3237
*/
3338
@Nonnull
34-
public static <T, C extends QueryDsl<T, C>, S> CompletionStage<List<S>>
39+
public static <T extends Resource, C extends QueryDsl<T, C>, S> CompletionStage<List<S>>
3540
queryAll(@Nonnull final SphereClient client, @Nonnull final QueryDsl<T, C> query,
36-
@Nonnull final Function<List<T>, S> callBack) {
37-
return queryAll(client, query, callBack, DEFAULT_PAGE_SIZE);
41+
@Nonnull final Function<List<T>, S> pageMapper) {
42+
return queryAll(client, query, pageMapper, DEFAULT_PAGE_SIZE);
3843
}
3944

4045
/**
41-
* Queries all elements matching a query by using an offset based pagination with page size 500. The method takes a
42-
* consumer {@link Consumer} that is applied on on every page of elements queried.
46+
* Queries all elements matching a query by using a limit based pagination with a combination of id sorting and a
47+
* page size 500. More on the algorithm can be found here: http://dev.commercetools.com/http-api.html#offset
48+
*
49+
* <p>The method takes a consumer {@link Consumer} that is applied on on every page of elements queried.
50+
*
51+
* <p>NOTE: This method fetches all paged results sequentially as opposed to fetching the pages in parallel.
4352
*
44-
* @param client commercetools client
45-
* @param query query containing predicates and expansion paths
46-
* @param consumer that is applied on every page queried.
47-
* @param <T> type of one query result element
48-
* @param <C> type of the query
49-
* @return elements
53+
* @param client commercetools client
54+
* @param query query containing predicates and expansion paths
55+
* @param pageConsumer consumer applied on every page queried
56+
* @param <T> type of one query result element
57+
* @param <C> type of the query
58+
* @return a completion stage containing void as a result after the consumer was applied on all pages.
5059
*/
5160
@Nonnull
52-
public static <T, C extends QueryDsl<T, C>> CompletionStage<Void>
61+
public static <T extends Resource, C extends QueryDsl<T, C>> CompletionStage<Void>
5362
queryAll(@Nonnull final SphereClient client, @Nonnull final QueryDsl<T, C> query,
54-
@Nonnull final Consumer<List<T>> consumer) {
55-
return queryAll(client, query, consumer, DEFAULT_PAGE_SIZE);
63+
@Nonnull final Consumer<List<T>> pageConsumer) {
64+
return queryAll(client, query, pageConsumer, DEFAULT_PAGE_SIZE);
5665
}
5766

5867
/**
59-
* Queries all elements matching a query by using an offset based pagination. The method takes a callback
60-
* {@link Function} that returns a result of type {@code <S>} that is returned on every page of elements queried.
61-
* Eventually, the method returns a {@link CompletionStage} that contains a list of all the results of the
62-
* callbacks returned from every page.
68+
* Queries all elements matching a query by using a limit based pagination with a combination of id sorting and the
69+
* supplied {@code pageSize}.
70+
* More on the algorithm can be found here: http://dev.commercetools.com/http-api.html#offset.
6371
*
64-
* @param client commercetools client
65-
* @param query query containing predicates and expansion paths
66-
* @param callback callback function that is called on every page queried.
67-
* @param <T> type of one query result element
68-
* @param <C> type of the query
69-
* @param <S> type of the returned result of the callback function on every page.
70-
* @param pageSize the page size.
71-
* @return elements
72+
* <p>The method takes a callback {@link Function} that returns a result of type {@code <S>} that is returned on
73+
* every page of elements queried. Eventually, the method returns a {@link CompletionStage} that contains a list of
74+
* all the results of the callbacks returned from every page.
75+
*
76+
* <p>NOTE: This method fetches all paged results sequentially as opposed to fetching the pages in parallel.
77+
*
78+
* @param client commercetools client
79+
* @param query query containing predicates and expansion paths
80+
* @param pageMapper callback function that is called on every page queried
81+
* @param <T> type of one query result element
82+
* @param <C> type of the query
83+
* @param <S> type of the returned result of the callback function on every page.
84+
* @param pageSize the page size.
85+
* @return a completion stage containing a list of mapped pages as a result.
7286
*/
7387
@Nonnull
74-
public static <T, C extends QueryDsl<T, C>, S> CompletionStage<List<S>>
88+
public static <T extends Resource, C extends QueryDsl<T, C>, S> CompletionStage<List<S>>
7589
queryAll(@Nonnull final SphereClient client, @Nonnull final QueryDsl<T, C> query,
76-
@Nonnull final Function<List<T>, S> callback, final int pageSize) {
77-
return QueryAll.of(query, pageSize).run(client, callback);
90+
@Nonnull final Function<List<T>, S> pageMapper, final int pageSize) {
91+
final QueryAll<T, C, S> queryAll = QueryAll.of(client, query, pageSize);
92+
return queryAll.run(pageMapper);
7893
}
7994

8095
/**
81-
* Queries all elements matching a query by using an offset based pagination. The method takes a consumer
82-
* {@link Consumer} that is applied on on every page of elements queried.
96+
* Queries all elements matching a query by using a limit based pagination with a combination of id sorting and the
97+
* supplied {@code pageSize}.
98+
* More on the algorithm can be found here: http://dev.commercetools.com/http-api.html#offset
99+
*
100+
* <p>The method takes a consumer {@link Consumer} that is applied on on every page of elements queried.
101+
*
102+
* <p>NOTE: This method fetches all paged results sequentially as opposed to fetching the pages in parallel.
83103
*
84-
* @param client commercetools client
85-
* @param query query containing predicates and expansion paths
86-
* @param consumer that is applied on every page queried.
87-
* @param <T> type of one query result element
88-
* @param <C> type of the query
89-
* @param pageSize the page size.
90-
* @return elements
104+
* @param client commercetools client
105+
* @param query query containing predicates and expansion paths
106+
* @param pageConsumer consumer applied on every page queried
107+
* @param <T> type of one query result element
108+
* @param <C> type of the query
109+
* @param pageSize the page size
110+
* @return a completion stage containing void as a result after the consumer was applied on all pages.
91111
*/
92112
@Nonnull
93-
public static <T, C extends QueryDsl<T, C>> CompletionStage<Void>
113+
public static <T extends Resource, C extends QueryDsl<T, C>> CompletionStage<Void>
94114
queryAll(@Nonnull final SphereClient client, @Nonnull final QueryDsl<T, C> query,
95-
@Nonnull final Consumer<List<T>> consumer, final int pageSize) {
96-
return QueryAll.of(query, pageSize).run(client, consumer);
115+
@Nonnull final Consumer<List<T>> pageConsumer, final int pageSize) {
116+
final QueryAll<T, C, Void> queryAll = QueryAll.of(client, query, pageSize);
117+
return queryAll.run(pageConsumer);
97118
}
98119
}

0 commit comments

Comments
 (0)