Skip to content

Commit c00138a

Browse files
Merge pull request #333 from commercetools/300-refactor-type-and-product-type-together
#300 - Type Sync Feature.
2 parents 11b9b19 + 5377ce7 commit c00138a

75 files changed

Lines changed: 6429 additions & 743 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

README.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,7 @@ Currently this library supports synchronising the following entities in commerce
1717
- [Products](/docs/usage/PRODUCT_SYNC.md)
1818
- [InventoryEntries](/docs/usage/INVENTORY_SYNC.md)
1919
- [ProductTypes](/docs/usage/PRODUCT_TYPE_SYNC.md)
20+
- [Types](/docs/usage/TYPE_SYNC.md)
2021

2122
![commercetools-java-sync-final 001](https://user-images.githubusercontent.com/9512131/31230702-0f2255a6-a9e5-11e7-9412-04ed52641dde.png)
2223
<!-- START doctoc generated TOC please keep comment here to allow auto update -->

config/checkstyle/checkstyle.xml

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -70,7 +70,10 @@
7070
</module>
7171
<module name="AvoidStarImport">
7272
<property name="excludes" value="com.commercetools.sync.products.utils.productvariantupdateactionutils.prices.PriceDraftFixtures,
73-
com.commercetools.sync.products.utils.productvariantupdateactionutils.prices.PriceFixtures"/>
73+
com.commercetools.sync.products.utils.productvariantupdateactionutils.prices.PriceFixtures,
74+
com.commercetools.sync.commons.utils.LocalizedEnumValueFixtures,
75+
com.commercetools.sync.commons.utils.PlainEnumValueFixtures,
76+
com.commercetools.sync.types.utils.FieldDefinitionFixtures"/>
7477
<property name="allowClassImports" value="false"/>
7578
<property name="allowStaticMemberImports" value="false"/>
7679
</module>
@@ -206,4 +209,4 @@
206209
<property name="ignorePrimitiveTypes" value="true"/>
207210
</module>
208211
</module>
209-
</module>
212+
</module>

docs/README.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,7 @@ Currently this library supports synchronising the following entities in commerce
2929
- [Products](usage/PRODUCT_SYNC.md)
3030
- [InventoryEntries](usage/INVENTORY_SYNC.md)
3131
- [ProductTypes](usage/PRODUCT_TYPE_SYNC.md)
32+
- [Types](usage/TYPE_SYNC.md)
3233

3334
![commercetools-java-sync-final 001](https://user-images.githubusercontent.com/9512131/31230702-0f2255a6-a9e5-11e7-9412-04ed52641dde.png)
3435

docs/RELEASE_NOTES.md

Lines changed: 8 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -34,25 +34,28 @@
3434
[Javadoc](https://commercetools.github.io/commercetools-sync-java/v/v1.0.0-M15/) |
3535
[Jar](https://bintray.com/commercetools/maven/commercetools-sync-java/v1.0.0-M15)
3636
37-
- 🎉 **New Features** (1)
37+
- 🎉 **New Features** (9)
3838
- **Commons** - Added `OptionalUtils#filterEmptyOptionals` which are utility methods that filter out the
3939
empty optionals in a supplied list (with a varargs variation) returning a list of the contents of the non-empty
4040
optionals. [#255](https://github.com/commercetools/commercetools-sync-java/issues/255)
41+
- **Type Sync** - Support for syncing types. [#300](https://github.com/commercetools/commercetools-sync-java/issues/300) For more info how to use it please refer to [Type usage doc](/docs/usage/TYPE_SYNC.md).
42+
- **Type Sync** - Exposed `TypeSyncUtils#buildActions` which calculates all needed update actions after comparing a `Type` and a `TypeDraft`. [#300](https://github.com/commercetools/commercetools-sync-java/issues/300)
43+
- **Type Sync** - Exposed `TypeUpdateActionUtils` which contains utils for calculating needed update actions after comparing individual fields of a `Type` and a `TypeDraft`. [#300](https://github.com/commercetools/commercetools-sync-java/issues/300)
4144
4245
- 📋 **Documentation** (4)
4346
- **Commons** - Added the documentation github pages. https://commercetools.github.io/commercetools-sync-java
4447
- **Commons** - Added a [Quick Start Guide](/docs/usage/QUICK_START.md) for a convinient entry into the library.
4548
- **Commons** - Moved documentation of sync options to a separate [doc](/docs/usage/SYNC_OPTIONS.md).
4649
- **Commons** - Added a the earliest compatible version of the commercetools-jvm-sdk](https://github.com/commercetools/commercetools-jvm-sdk) as a prerequisite for using the library.
4750
48-
- 🛠️ **Enhancements** (16)
51+
- 🛠️ **Enhancements** (17)
4952
- **ProductType Sync** - Added concurrency modification exception handling. [#325](https://github.com/commercetools/commercetools-sync-java/issues/325)
5053
- **Commons** - `ProductSyncUtils#buildActions`, `CategorySyncUtils#buildActions`, `InventorySyncUtils#buildActions` and `ProductTypeSyncUtils#buildActions` now don't apply the `beforeUpdateCallback` implicitly.
5154
If you want, you can apply it explicitly on the result of the `..#buildActions` method. [#302](https://github.com/commercetools/commercetools-sync-java/issues/302)
5255
- **Product Sync** - Reference keys are not validated if they are in UUID format anymore. [#166](https://github.com/commercetools/commercetools-sync-java/issues/166)
5356
- **Category Sync** - Reference keys are not validated if they are in UUID format anymore. [#166](https://github.com/commercetools/commercetools-sync-java/issues/166)
5457
- **Inventory Sync** - Reference keys are not validated if they are in UUID format anymore. [#166](https://github.com/commercetools/commercetools-sync-java/issues/166)
55-
- **ProductType Sync** - Added `ProductTypeSyncBenchmark` to benchmark the product type sync, to be able to compare the performance of the sync with the future releases. [#301](https://github.com/commercetools/commercetools-sync-java/issues/301)
58+
- **ProductType Sync** - Added benchmarks for the `productType` sync to be able to compare the performance of the sync with the future releases. [#301](https://github.com/commercetools/commercetools-sync-java/issues/301)
5659
- **Commons** - Bumped commercetools-jvm-sdk to version [1.37.0](http://commercetools.github.io/commercetools-jvm-sdk/apidocs/io/sphere/sdk/meta/ReleaseNotes.html#v1_37_0).
5760
- **Commons** - Bumped `mockito` to 2.23.4.
5861
- **Commons** - Bumped `com.adarshr.test-logger` to 1.6.0.
@@ -63,7 +66,8 @@
6366
- **Commons** - Bumped `com.adarshr.test-logger` to 1.6.0.
6467
- **Commons** - Bumped `org.ajoberstar.grgit` to 3.0.0.
6568
- **Commons** - Bumped gradle to version [gradle-5.0](https://docs.gradle.org/5.0/release-notes.html)
66-
69+
- **Type Sync** - Added benchmarks for the `type` sync to be able to compare the performance of the sync with the future releases. [#300](https://github.com/commercetools/commercetools-sync-java/issues/300)
70+
6771
- 🚧 **Breaking Changes** (11)
6872
- **Product Sync** - `allowUuid` option is now removed. [#166](https://github.com/commercetools/commercetools-sync-java/issues/166)
6973
- **Category Sync** - `allowUuid` option is now removed. [#166](https://github.com/commercetools/commercetools-sync-java/issues/166)

docs/usage/PRODUCT_TYPE_SYNC.md

Lines changed: 16 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,8 @@ against a [ProductTypeDraft](https://docs.commercetools.com/http-api-projects-pr
1212
- [Sync list of product type drafts](#sync-list-of-product-type-drafts)
1313
- [Prerequisites](#prerequisites)
1414
- [Running the sync](#running-the-sync)
15+
- [Important to Note](#important-to-note)
16+
- [More examples of how to use the sync](#more-examples-of-how-to-use-the-sync)
1517
- [Build all update actions](#build-all-update-actions)
1618
- [Build particular update action(s)](#build-particular-update-actions)
1719
- [Caveats](#caveats)
@@ -26,8 +28,8 @@ against a [ProductTypeDraft](https://docs.commercetools.com/http-api-projects-pr
2628

2729
#### Prerequisites
2830

29-
1. The sync expects a list of `ProductTypeDrafts`s that have their `key` fields set to be matched with
30-
product types in the target CTP project. Also, the product types in the target project are expected to have the `key`
31+
1. The sync expects a list of `ProductTypeDraft`s that have their `key` fields set to be matched with
32+
product types in the target CTP project. Also, the product types in the target project are expected to have the `key`
3133
fields set, otherwise they won't be matched.
3234

3335
2. Create a `sphereClient` [as described here](IMPORTANT_USAGE_TIPS.md#sphereclient-creation).
@@ -64,8 +66,18 @@ __Note__ The statistics object contains the processing time of the last batch on
6466

6567
1. The sync processing time should not take into account the time between supplying batches to the sync.
6668
2. It is not known by the sync which batch is going to be the last one supplied.
69+
70+
#### Important to Note
71+
72+
1. If two matching `attributeDefinition`s (old and new) on the matching `productType`s (old and new) have a different `AttributeType`, the sync will
73+
**remove** the existing `attributeDefinition` and then **add** a new `attributeDefinition` with the new `AttributeType`.
6774

68-
More examples of how to use the sync can be found [here](https://github.com/commercetools/commercetools-sync-java/tree/master/src/integration-test/java/com/commercetools/sync/integration/producttypes/ProductTypeSyncIT.java).
75+
2. The `attributeDefinition` for which the `AttributeType` is not defined (`null`) will not be synced.
76+
77+
#### More examples of how to use the sync
78+
79+
1. [Sync from another CTP project as a source](https://github.com/commercetools/commercetools-sync-java/tree/master/src/integration-test/java/com/commercetools/sync/integration/ctpprojectsource/producttypes/ProductTypeSyncIT.java).
80+
2. [Sync from an external source](https://github.com/commercetools/commercetools-sync-java/tree/master/src/integration-test/java/com/commercetools/sync/integration/externalsource/producttypes/ProductTypeSyncIT.java).
6981

7082
*Make sure to read the [Important Usage Tips](IMPORTANT_USAGE_TIPS.md) for optimal performance.*
7183

@@ -88,3 +100,4 @@ More examples of those utils for different fields can be found [here](https://gi
88100

89101
## Caveats
90102
1. Syncing product types with an attribute of type [NestedType](https://docs.commercetools.com/http-api-projects-productTypes.html#nestedtype) is not supported yet.
103+
2. Currently the sync would handle changes to Enum or LocalizedEnum Types but not [Set](https://docs.commercetools.com/http-api-projects-types.html#settype) of either. [#313](https://github.com/commercetools/commercetools-sync-java/issues/313)

docs/usage/QUICK_START.md

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -83,4 +83,5 @@ implementation 'com.commercetools:commercetools-sync-java:v1.0.0-M14'
8383

8484
#### More Details
8585
*[Product Sync](PRODUCT_SYNC.md), [ProductType Sync](PRODUCT_TYPE_SYNC.md),
86-
[Category Sync](CATEGORY_SYNC.md), [Inventory Sync](INVENTORY_SYNC.md)*
86+
[Category Sync](CATEGORY_SYNC.md), [Inventory Sync](INVENTORY_SYNC.md),
87+
[Type Sync](TYPE_SYNC.md)*

docs/usage/TYPE_SYNC.md

Lines changed: 101 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,101 @@
1+
# Type Sync
2+
3+
Module used for importing/syncing Types into a commercetools project.
4+
It also provides utilities for generating update actions based on the comparison of a [Type](https://docs.commercetools.com/http-api-projects-types.html#type)
5+
against a [TypeDraft](https://docs.commercetools.com/http-api-projects-types.html#typedraft).
6+
7+
<!-- START doctoc generated TOC please keep comment here to allow auto update -->
8+
<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
9+
10+
11+
- [Usage](#usage)
12+
- [Sync list of type drafts](#sync-list-of-type-drafts)
13+
- [Prerequisites](#prerequisites)
14+
- [Running the sync](#running-the-sync)
15+
- [Important to Note](#important-to-note)
16+
- [More examples of how to use the sync](#more-examples-of-how-to-use-the-sync)
17+
- [Build all update actions](#build-all-update-actions)
18+
- [Build particular update action(s)](#build-particular-update-actions)
19+
- [Caveats](#caveats)
20+
21+
<!-- END doctoc generated TOC please keep comment here to allow auto update -->
22+
23+
## Usage
24+
25+
### Sync list of type drafts
26+
27+
#### Prerequisites
28+
1. The sync expects a list of `TypeDraft`s that have their `key` fields set to be matched with
29+
types in the target CTP project. Also, the types in the target project are expected to have the `key`
30+
fields set, otherwise they won't be matched.
31+
32+
2. Create a `sphereClient` [as described here](IMPORTANT_USAGE_TIPS.md#sphereclient-creation).
33+
34+
3. After the `sphereClient` is set up, a `TypeSyncOptions` should be be built as follows:
35+
````java
36+
// instantiating a TypeSyncOptions
37+
final TypeSyncOptions typeSyncOptions = TypeSyncOptionsBuilder.of(sphereClient).build();
38+
````
39+
40+
[More information about Sync Options](SYNC_OPTIONS.md).
41+
42+
#### Running the sync
43+
After all the aforementioned points in the previous section have been fulfilled, to run the sync:
44+
````java
45+
// instantiating a type sync
46+
final TypeSync typeSync = new TypeSync(typeSyncOptions);
47+
48+
// execute the sync on your list of types
49+
CompletionStage<TypeSyncStatistics> syncStatisticsStage = typeSync.sync(typeDrafts);
50+
````
51+
The result of the completing the `syncStatisticsStage` in the previous code snippet contains a `TypeSyncStatistics`
52+
which contains all the stats of the sync process; which includes a report message, the total number of updated, created,
53+
failed, processed types and the processing time of the last sync batch in different time units and in a
54+
human-readable format.
55+
56+
````java
57+
final TypeSyncStatistics stats = syncStatisticsStage.toCompletebleFuture().join();
58+
stats.getReportMessage();
59+
/*"Summary: 2000 types were processed in total (1000 created, 995 updated, 5 failed to sync)."*/
60+
````
61+
62+
__Note__ The statistics object contains the processing time of the last batch only. This is due to two reasons:
63+
64+
1. The sync processing time should not take into account the time between supplying batches to the sync.
65+
2. It is not known by the sync which batch is going to be the last one supplied.
66+
67+
#### Important to Note
68+
1. If two matching `fieldDefinition`s (old and new) on the matching `type`s (old and new) have a different `FieldType`, the sync will
69+
**remove** the existing `fieldDefinition` and then **add** a new `fieldDefinition` with the new `FieldType`.
70+
71+
2. The `fieldDefinition` for which the `fieldType` is not defined (`null`) will not be synced.
72+
73+
#### More examples of how to use the sync
74+
75+
1. [Sync from another CTP project as a source](https://github.com/commercetools/commercetools-sync-java/tree/master/src/integration-test/java/com/commercetools/sync/integration/ctpprojectsource/types/TypeSyncIT.java).
76+
2. [Sync from an external source](https://github.com/commercetools/commercetools-sync-java/tree/master/src/integration-test/java/com/commercetools/sync/integration/externalsource/types/TypeSyncIT.java).
77+
78+
*Make sure to read the [Important Usage Tips](IMPORTANT_USAGE_TIPS.md) for optimal performance.*
79+
80+
### Build all update actions
81+
82+
A utility method provided by the library to compare a `Type` with a new `TypeDraft` and results in a list of type update actions.
83+
```java
84+
List<UpdateAction<Type>> updateActions = TypeSyncUtils.buildActions(type, typeDraft, typeSyncOptions);
85+
```
86+
87+
### Build particular update action(s)
88+
89+
Utility methods provided by the library to compare the specific fields of a `Type` and a new `TypeDraft`, and in turn builds
90+
the update action. One example is the `buildChangeNameUpdateAction` which compares names:
91+
````java
92+
Optional<UpdateAction<Type>> updateAction = TypeUpdateActionUtils.buildChangeNameAction(oldType, typeDraft);
93+
````
94+
More examples of those utils for different types can be found [here](https://github.com/commercetools/commercetools-sync-java/tree/master/src/test/java/com/commercetools/sync/types/utils/TypeUpdateActionUtilsTest.java).
95+
96+
## Caveats
97+
98+
1. Updating the label of enum values and localized enum values of field definition is not supported yet. [#339](https://github.com/commercetools/commercetools-sync-java/issues/339)
99+
2. Removing the enum values and localized enum values from the field definition is not supported yet. [#339](https://github.com/commercetools/commercetools-sync-java/issues/339)
100+
3. Updating the input hint of a field definition is not supported yet. [#339](https://github.com/commercetools/commercetools-sync-java/issues/339)
101+
4. Currently the sync would handle changes to Enum or LocalizedEnum Types but not [Set](https://docs.commercetools.com/http-api-projects-types.html#settype) of either. [#313](https://github.com/commercetools/commercetools-sync-java/issues/313)

mkdocs.yml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -49,6 +49,7 @@ nav:
4949
- Quick Start: usage/QUICK_START.md
5050
- Product Sync: usage/PRODUCT_SYNC.md
5151
- ProductType Sync: usage/PRODUCT_TYPE_SYNC.md
52+
- Type Sync: usage/TYPE_SYNC.md
5253
- Category Sync: usage/CATEGORY_SYNC.md
5354
- InventoryEntry Sync: usage/INVENTORY_SYNC.md
5455
- Advanced:

src/benchmark/java/com/commercetools/sync/benchmark/BenchmarkUtils.java

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,7 @@ public class BenchmarkUtils {
3535
static final String PRODUCT_SYNC = "productSync";
3636
static final String INVENTORY_SYNC = "inventorySync";
3737
static final String CATEGORY_SYNC = "categorySync";
38+
static final String TYPE_SYNC = "typeSync";
3839
static final String PRODUCT_TYPE_SYNC = "productTypeSync";
3940
static final String CREATES_ONLY = "createsOnly";
4041
static final String UPDATES_ONLY = "updatesOnly";
@@ -107,9 +108,9 @@ private static double calculateAvg(@Nonnull final List<JsonNode> results) {
107108
}
108109

109110
private static ObjectNode createVersionNode(@Nonnull final ObjectNode rootNode, @Nonnull final String version) {
110-
final ObjectNode newVersionNode = createSyncNode(
111+
final ObjectNode newVersionNode = createSyncNode(createSyncNode(createSyncNode(
111112
createSyncNode(createSyncNode(JsonNodeFactory.instance.objectNode(),
112-
PRODUCT_SYNC), INVENTORY_SYNC), CATEGORY_SYNC);
113+
PRODUCT_SYNC), INVENTORY_SYNC), CATEGORY_SYNC), PRODUCT_TYPE_SYNC), TYPE_SYNC);
113114
return (ObjectNode) rootNode.set(version, newVersionNode);
114115
}
115116

src/benchmark/java/com/commercetools/sync/benchmark/ProductSyncBenchmark.java

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -158,7 +158,7 @@ public void sync_ExistingProducts_ShouldUpdateProducts() throws IOException {
158158

159159

160160
// Calculate time taken for benchmark and assert it lies within threshold
161-
final double diff = calculateDiff(SyncSolutionInfo.LIB_VERSION, PRODUCT_SYNC, CREATES_ONLY, totalTime);
161+
final double diff = calculateDiff(SyncSolutionInfo.LIB_VERSION, PRODUCT_SYNC, UPDATES_ONLY, totalTime);
162162
assertThat(diff)
163163
.withFailMessage(format("Diff of benchmark '%e' is longer than expected threshold of '%d'.", diff,
164164
THRESHOLD))
@@ -222,7 +222,7 @@ public void sync_WithSomeExistingProducts_ShouldSyncProducts() throws IOExceptio
222222

223223

224224
// Calculate time taken for benchmark and assert it lies within threshold
225-
final double diff = calculateDiff(SyncSolutionInfo.LIB_VERSION, PRODUCT_SYNC, CREATES_ONLY, totalTime);
225+
final double diff = calculateDiff(SyncSolutionInfo.LIB_VERSION, PRODUCT_SYNC, CREATES_AND_UPDATES, totalTime);
226226
assertThat(diff)
227227
.withFailMessage(format("Diff of benchmark '%e' is longer than expected threshold of '%d'.", diff,
228228
THRESHOLD))

0 commit comments

Comments
 (0)