Skip to content

Commit 34175af

Browse files
author
Anna Mikhaylova
committed
[DOP-27545] fix files from branch develop
1 parent 31c350a commit 34175af

8 files changed

Lines changed: 287 additions & 0 deletions

File tree

‎docs/changelog/0.2.5.rst‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
0.2.5 (2025-10-10)
2+
==================
3+
4+
Features
5+
--------
6+
7+
- Implement Keycloak login page on frontend. (`#128 <https://github.com/MTSWebServices/syncmaster-ui/pull/128>`_)
8+
- Implement ``GET /v1/auth/logout`` endpoint for ``KeycloakAuthProvider``. (:issue:`275`)
9+
10+
Improvements
11+
------------
12+
13+
- Improved full-text search for technical fields such as hostnames, table names, and directory paths. (:issue:`255`)
14+
- Replace 307 redirect to Keycloak auth page with 401 response, due to browser restrictions for redirect + CORS + localhost. (:issue:`274`)
15+
16+
Bug Fixes
17+
---------
18+
19+
- Replace sync methods of Keycloak client with async ones. (:issue:`177`)
20+
21+
Previously interaction with Keycloak could block asyncio event loop.

‎docs/changelog/0.3.0.rst‎

Lines changed: 105 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,105 @@
1+
0.3.0 (2025-12-18)
2+
==================
3+
4+
Release of Data.SyncMaster 0.3.0 brings up support for Iceberg, Spark-on-K8s and Spark-on-Yarn.
5+
6+
.. note::
7+
8+
Currently Spark-on-K8s and Spark-on-Yarn do not support FTP, FTPS, SFTP, Samba and WebDAV.
9+
10+
Breaking Changes
11+
----------------
12+
13+
- Worker container command should be changed from ``--queues 123-myqueue`` to ``worker --queues 123-myqueue`` (:issue:`295`).
14+
15+
- Application should be configured via ``config.yml`` file (:issue:`289`).
16+
17+
It's still possible to use environment variables instead. But it is not recommended for security reasons, as docker/k8s envs can be read by other users.
18+
19+
Other notable changes:
20+
* Environment variable ``SYNCMASTER__ENTRYPOINT__SUPERUSERS`` is renamed to ``SYNCMASTER__SUPERUSERS``.
21+
* Logging format is configured explicitly via ``config.yml`` instead of having few predefined configuration files.
22+
23+
- Moved ``server.session`` middleware settings to ``auth`` block (:issue:`304`).
24+
Also rename some fields in ``auth.keycloak`` settings block.
25+
26+
.. dropdown:: Before vs after
27+
28+
Before:
29+
30+
.. code:: yaml
31+
32+
auth:
33+
provider: ...
34+
keycloak:
35+
server_url: ...
36+
redirect_url: ...
37+
38+
server:
39+
session:
40+
enabled: true
41+
secret_key: ...
42+
43+
Now:
44+
45+
.. code:: yaml
46+
47+
auth:
48+
provider:
49+
keycloak:
50+
api_url: ...
51+
ui_callback_url: ...
52+
cookie:
53+
secret_key: ...
54+
55+
56+
Features
57+
--------
58+
59+
- Added Iceberg support (:issue:`282`, :issue:`284`, :issue:`294`, :issue:`297`).
60+
61+
Iceberg connection currently supports only Iceberg REST Catalog with S3 warehouse.
62+
63+
- Allow using SyncMaster worker image as ``spark.kubernetes.container.image``. (:issue:`295`)
64+
65+
- Allow passing default Spark session config via worker settings (:issue:`291`):
66+
67+
.. dropdown:: Example
68+
69+
.. code-block:: yaml
70+
:caption: config.yml
71+
72+
worker:
73+
spark_session_default_config:
74+
spark.master: local
75+
spark.driver.host: 127.0.0.1
76+
spark.driver.bindAddress: 0.0.0.0
77+
spark.sql.pyspark.jvmStacktrace.enabled: true
78+
spark.ui.enabled: false
79+
80+
- Added OAuth2GatewayProvider (:issue:`283`).
81+
82+
This allows using Data.SyncMaster under OAuth2 Gateway. Implementation is similar to DummyAuthProvider.
83+
84+
- Allow disabling ``SessionMiddleware``, as it only required by ``KeycloakAuthProvider``.
85+
86+
- Add hooks support to worker classes (TransferController, Handler) (:issue:`279`).
87+
- Pass transfer name and group name to Handlers (:issue:`308`).
88+
89+
Improvements
90+
------------
91+
92+
- Make S3 connection ``region`` a mandatory option, to prevent possible errors.
93+
- Hide ``database_name`` from Clickhouse and MySQL connection pages.
94+
- Frontend: add placeholders to connection params, like host, port and so on.
95+
- Sync frontend and backend checks for some field patterns, e.g. table name should be in format ``schema.table``.
96+
- Improve OpenAPI schema fields description.
97+
98+
Bug Fixes
99+
---------
100+
101+
Fix some file format options were ignored by SyncMaster worker:
102+
* XML: ``root_tag``, ``row_tag``
103+
* Excel ``start_cell``, ``include_header``
104+
* CSV ``include_header``, ``line_sep``
105+
* JSON, JSONLine: ``line_sep``

‎docs/changelog/0.3.1.rst‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
0.3.1 (2026-01-15)
2+
==================
3+
4+
Improvements
5+
------------
6+
7+
- Added table name validation - must be in format ``schema.table`` or ``namespace1.namespace2.table``.
8+
- Cut 400MB of worker images size by removing copies of ``.jar`` files.
9+
- Explicitly log transfer exception instead of ``raised unexpected: UnpickleableExceptionWrapper`` by celery.
10+
11+
Misc
12+
----
13+
14+
- Switch from ``poetry`` to ``uv`` for dependency management.
15+
- Switch from ``black`` to ``ruff format``.
16+
- Switch from ``flake8+wemake-python-styleguide`` to ``ruff check``, and fix found code smells.
17+
- Dependency updates.

‎docs/changelog/0.3.2.rst‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
0.3.2 (2026-03-05)
2+
==================
3+
4+
Features
5+
--------
6+
7+
Implement SQL transformations for transfer (:issue:`327`, :issue:`330`)

‎mddocs/docs/en/changelog/0.2.5.md‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
# 0.2.5 (2025-10-10) { #0.2.5 }
2+
3+
## Features
4+
5+
- Implement Keycloak login page on frontend. ([#128](https://github.com/MTSWebServices/syncmaster-ui/pull/128))
6+
- Implement ``GET /v1/auth/logout`` endpoint for ``KeycloakAuthProvider``. ([#275](https://github.com/MobileTeleSystems/syncmaster/issues/255))
7+
8+
## Improvements
9+
10+
- Improved full-text search for technical fields such as hostnames, table names, and directory paths. ([#255](https://github.com/MobileTeleSystems/syncmaster/issues/255))
11+
- Replace 307 redirect to Keycloak auth page with 401 response, due to browser restrictions for redirect + CORS + localhost. ([#274](https://github.com/MobileTeleSystems/syncmaster/issues/274))
12+
13+
## Bug Fixes
14+
15+
- Replace sync methods of Keycloak client with async ones. ([#177](https://github.com/MobileTeleSystems/syncmaster/issues/177))
16+
17+
Previously interaction with Keycloak could block asyncio event loop.

‎mddocs/docs/en/changelog/0.3.0.md‎

Lines changed: 101 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,101 @@
1+
# 0.3.0 (2025-12-18) { #0.3.0 }
2+
3+
Release of Data.SyncMaster 0.3.0 brings up support for Iceberg, Spark-on-K8s and Spark-on-Yarn.
4+
5+
!!! note
6+
Currently Spark-on-K8s and Spark-on-Yarn do not support FTP, FTPS, SFTP, Samba and WebDAV.
7+
8+
## Breaking Changes
9+
10+
- Worker container command should be changed from `--queues 123-myqueue` to `worker --queues 123-myqueue` ([#295](https://github.com/MobileTeleSystems/syncmaster/issues/295)).
11+
12+
- Application should be configured via `config.yml` file ([#289](https://github.com/MobileTeleSystems/syncmaster/issues/289)).
13+
14+
It's still possible to use environment variables instead. But it is not recommended for security reasons, as docker/k8s envs can be read by other users.
15+
16+
Other notable changes:
17+
18+
- Environment variable `SYNCMASTER__ENTRYPOINT__SUPERUSERS` is renamed to `SYNCMASTER__SUPERUSERS`.
19+
- Logging format is configured explicitly via `config.yml` instead of having few predefined configuration files.
20+
21+
- Moved `server.session` middleware settings to `auth` block ([#304](https://github.com/MobileTeleSystems/syncmaster/issues/304)).
22+
Also rename some fields in `auth.keycloak` settings block.
23+
24+
Before vs after
25+
26+
Before:
27+
28+
```yaml
29+
auth:
30+
provider: ...
31+
keycloak:
32+
server_url: ...
33+
redirect_url: ...
34+
35+
server:
36+
session:
37+
enabled: true
38+
secret_key: ...
39+
```
40+
41+
Now:
42+
43+
```yaml
44+
auth:
45+
provider:
46+
keycloak:
47+
api_url: ...
48+
ui_callback_url: ...
49+
cookie:
50+
secret_key: ...
51+
```
52+
53+
## Features
54+
55+
- Added Iceberg support (([#282](https://github.com/MobileTeleSystems/syncmaster/issues/282)), ([#284](https://github.com/MobileTeleSystems/syncmaster/issues/284)), ([#294](https://github.com/MobileTeleSystems/syncmaster/issues/294)), ([#297](https://github.com/MobileTeleSystems/syncmaster/issues/297))).
56+
57+
Iceberg connection currently supports only Iceberg REST Catalog with S3 warehouse.
58+
59+
- Allow using SyncMaster worker image as `spark.kubernetes.container.image`. ([#295](https://github.com/MobileTeleSystems/syncmaster/issues/295))
60+
61+
- Allow passing default Spark session config via worker settings ([#291](https://github.com/MobileTeleSystems/syncmaster/issues/291)):
62+
63+
### Example config.yml
64+
65+
```yaml
66+
config.yml:
67+
worker:
68+
spark_session_default_config:
69+
spark.master: local
70+
spark.driver.host: 127.0.0.1
71+
spark.driver.bindAddress: 0.0.0.0
72+
spark.sql.pyspark.jvmStacktrace.enabled: true
73+
spark.ui.enabled: false
74+
```
75+
76+
- Added OAuth2GatewayProvider ([#283](https://github.com/MobileTeleSystems/syncmaster/issues/283)).
77+
78+
This allows using Data.SyncMaster under OAuth2 Gateway. Implementation is similar to DummyAuthProvider.
79+
80+
- Allow disabling `SessionMiddleware`, as it only required by `KeycloakAuthProvider`.
81+
82+
- Add hooks support to worker classes (TransferController, Handler) ([#279](https://github.com/MobileTeleSystems/syncmaster/issues/279)).
83+
84+
- Pass transfer name and group name to Handlers ([#308](https://github.com/MobileTeleSystems/syncmaster/issues/308)).
85+
86+
## Improvements
87+
88+
- Make S3 connection `region` a mandatory option, to prevent possible errors.
89+
- Hide `database_name` from Clickhouse and MySQL connection pages.
90+
- Frontend: add placeholders to connection params, like host, port and so on.
91+
- Sync frontend and backend checks for some field patterns, e.g. table name should be in format `schema.table`.
92+
- Improve OpenAPI schema fields description.
93+
94+
## Bug Fixes
95+
96+
Fix some file format options were ignored by SyncMaster worker:
97+
98+
- XML: `root_tag`, `row_tag`
99+
- Excel `start_cell`, `include_header`
100+
- CSV `include_header`, `line_sep`
101+
- JSON, JSONLine: `line_sep`

‎mddocs/docs/en/changelog/0.3.1.md‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
# 0.3.1 (2026-01-15) { #0.3.1 }
2+
3+
## Improvements
4+
5+
- Added table name validation - must be in format `schema.table` or `namespace1.namespace2.table`.
6+
- Cut 400MB of worker images size by removing copies of `.jar` files.
7+
- Explicitly log transfer exception instead of `raised unexpected: UnpickleableExceptionWrapper` by celery.
8+
9+
## Misc
10+
11+
- Switch from `poetry` to `uv` for dependency management.
12+
- Switch from `black` to `ruff format`.
13+
- Switch from `flake8+wemake-python-styleguide` to `ruff check`, and fix found code smells.
14+
- Dependency updates.

‎mddocs/docs/en/changelog/0.3.2.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
# 0.3.2 (2026-03-05) { #0.3.2 }
2+
3+
## Features
4+
5+
Implement SQL transformations for transfer (([#327](https://github.com/MobileTeleSystems/syncmaster/issues/327)), ([#330](https://github.com/MobileTeleSystems/syncmaster/issues/255)))

0 commit comments

Comments
 (0)