SKU/molecule#1234
Draft
sergiimk wants to merge 107 commits into
Draft
Conversation
…es w/o content_length (#1235)
…arlier than expected (#1237)
* Kicked off webhooks domain: described webhook subscription aggregate * Added 'WebhookEvent' simple aggregate * Sketched subscription event store methods * Sketched WebhookDelivery (fact) and it's repository * Couple more methods in delivery repository * Improved ID types. Introduced TaskAttemptId type * Introduced concept of `TaskAttempt` in task, and reshuffled state structure around it without changing event store * Drafter retrying logic and retry policies at task aggregate level. Not integrated or materialized yet. * Drafted retries in task snapshots * Working sqlite migration * Stabilized existing tests * Unit tests added for retry policies * Task scheduler config for retry policy + service-layer tests for retries * Implemented repository for webhook events (3 incarnations, no tests yet) * Unit tests implemented for WebhookEvent repositories. Detecting duplicate ID when creating events. * SQL for webhook subscriptions and deliveries * Drafted in-memory repo for webhook subscriptions * Sketched in-memory webhook delivery repository * Drafted Postgres/SQlite implementations fo webhook delivery repository * Drafted Postgres webhook subscription event store implementation * Implemented SQlite version of webhook subscription store * Tests for webhook delivery repository * Basic coverage for webhook subscription event store * Merge corrections * Finished tests for repository layer * Test corrections * Drafted webhook event bridge service * Sketched webhooks service layer, including: - webhook logical plan in Task System, propagation through planner/runner - webhook sender: populates delivery object, generates headers, sends webhook, updates the delivery object with webhook response - webhook signer: implementation of RFC9421 * udeps cleaned * Wrote very naive webhook signing and sender tests * Removed "ref" field from webhook event + linter fixed * WebhookSender => WebhookDeliveryWorker * Separated webhook outbox bride and event builder * Sending more universal DATASET.REF.UPDATED event with "blockRef" field * Separated WebhookSender from WebhookDelivery worker (to increase testability) * WebhookOutboxBridge => WebhookDeliveryScheduler * Drafted webhook secret generator * Added outbox consumer that removes subscriptions of a removed dataset * MInor correction in sending test * Sketched webhook subscriptions GQL api (no tests yet, incomplete) * Self-review question * Merge corrections * merge corrections * Removed task retries. Binding webhook delivery 1-to-1 to task * First test for subscriptions API * Extended test coverage for webhook subscription create GQL entry. Extracted creation use case * Reorganized GQL api handlers to use use cases more * Reorganized webhook subscriptions API to be more dataset bound to avoid separate security checking * Reshuffled update operations in GQL for webhook subscriptions * Idempotent subscription reactions * More subscription GQL API tests * Webhook subscription use case tests * Tests for WebhookDatasetRemovalHandler * Tests for webhook delivery scheduler service * Tests for webhook delivery worker * Merge corrections * Will lint/codegen pass with project toolchain? * try fixing lint/codegen * Removed obsolete comment * Recovered original formatting of GQL schema
* Custom workflow for SQLX migrations dev branch * GQL: Account Deletion API (#1242) * GQL: AccountMut::delete_account(): scaffolding * validate_password(): use .len() * DidSecretKeyRowModel: use type fullpath for DidEntityType * schema.gql: update * AccountMut::delete_account_by_name(): use DeleteAccountUseCase * AccountMut: use AccountName scalar instead of String * AccountMut: use Email scalar instead of String * AccountService::delete_account_by_name(): add * make sqlx-local-setup: fix * did_secret_keys: drop creator_id * CreateAccountUseCaseImpl::execute(): update fallback email generation * AccountService::delete_account_by_name(): add [2] * AccountRepository::delete_account_by_name(): implement * account_messages.rs -> account_lifecycle_message.rs * DeleteAccountUseCaseImpl: send AccountLifecycleMessage::deleted() * AccountRepository::delete_account_by_name(): return the removed account not only id * AccountLifecycleMessageDeleted::display_name: add * DatasetAccountDeletionHandler: introduce * DatasetRegistry::all_dataset_handles_by_owner_id(): implement * DatasetRegistry::all_dataset_handles_by_owner() -> all_dataset_handles_by_owner_name() * DatasetAccountDeletionHandler: introduce [2] * DidSecretKeyRepository::delete_did_secret_key(): add * kamu-account-services: group message producers and consumers * messaging_outbox::prelude: introduce * messaging_outbox::prelude: introduce * DidSecretService: handle DatasetLifecycleMessage * access_tokens: add ON DELETE CASCADE for account_id * accounts_passwords: add account_id * Linter fixes * CreateAccountUseCaseImpl::generate_email(): extract * PasswordHashRepository::save_password_hash(): add "account_id" * Makefile: resort vertically db crates * AccountLifecycleMessageDeleted: add "email" field * DidSecretKeyRepository::get_did_secret_key(): add * test_insert_and_locate_did_secret_keys(): update * test_create_account(): fixed * test_create_dataset_from_snapshot_creates_did_secret_key(): fixed * test_update_email_bad_email(): fixed * Makefile: remove db crate duplicates * sqlx: add cached queries * AccountsMut::create_account(): absorb * AccountMut::delete(): rethink * AccountMut: access checks * AccountMut: update tracings * DeleteAccountUseCaseImpl: only admins * GQL: AccountMut::modify_password(): re-think * DeleteAccountUseCaseImpl: allow self-deletion * Self-review * CHANGELOG.md: update * DeleteAccountUseCaseImpl::authenticated(): renamed from unauthenticated() * Integration fixes * SelfDeletionIsForbidden: remove unused struct * AccountRepository::delete_account_by_name(): do not return deleted account * OsoDatasetAuthorizer: move dill macros to struct declaration * DeleteAccountUseCase::execute(): take &Account as argument * DeleteAccountUseCaseImpl: use utils::AccountAuthorizationHelper * DatasetRegistry::all_dataset_handles_by_owner_id(): return odfOwnedDatasetHandleStream * Revert "DatasetRegistry::all_dataset_handles_by_owner_id(): return odfOwnedDatasetHandleStream" This reverts commit 8ac0da8. * DatasetAccountDeletionHandler::handle_account_lifecycle_deleted_message(): add a PERF note * CI: fix codegen action * CHANGELOG.md: update * test_delete_account_use_case_impl(): implement * GQL: test_accounts: add tests * CHANGELOG.md: update * Tests fixes * sqlx: update cached queries * Remove password logic from account service level (#1243) * Refactor password logic * Update changelog * GQL: Collection API, `extra_data` validation (#1246) * CollectionMut: take dataset by a ref * VersionedFileMut: take dataset by a ref * GQL: Add ExtraData scalar * GQL: use ExtraData scalar * Typo fixes * Fix tests * GQL: ExtraData scalar: add tests * CHANGELOG.md: update * Allow MIT-0 license usage * kamu-adapter-graphql: correct feature gate * Release (minor): 0.238.0 (#1248) --------- Co-authored-by: Sergei Zaychenko <szaychenko@kamu.dev> Co-authored-by: Roman Boiko <roman.bv20@gmail.com>
* Upgrade to new rustc and 2024 edition (#1254) * Search by account name (#1253) * Search filters also by account name * Update changelog * Fix review comments. Iter 1 * Replace format by to_string() * Fix fmt * Wallet based authentication: Phase 1 (#1239) * EvmWalletAuthenticationProvider: scaffolding * kamu-datasets: remove extra dep (itertools) * kamu-adapter-auth-web3: implement Web3WalletAuthenticationProvider (w/o nonce checking) * kamu --show-error-stack-trace * APIServerRunCommand: get token after HTTP server initialization * kamu-cli: activate Web3WalletAuthenticationProvider * kamu-adapter-graphql: extract auth_mut/ * kamu-adapter-auth-web3: ChecksumWalletAddress -> ChecksumEvmWalletAddress * GQL: AuthWeb3Mut::nonce(): implement * kamu-web3: introduce Web3AuthNonceRepository * kamu-web3-services: introduce Web3NonceServiceImpl * kamu-cli: register kamu-web3-services * kamu-auth-web3: update EIP_4361_EXPECTED_STATEMENT text * schema.gql: update * Web3NonceServiceImpl: impl InitOnStartup * kamu-web -> kamu-auth-web * kamu-auth-web3-inmem: implement * kamu-auth-web3-repo-tests: implement * kamu-auth-web3-inmem: tests * AuthWeb3Mut: nonce() -> eip4361_auth_nonce() * kamu-auth-web3-postgres: scaffolding * kamu-auth-web3-postgres: implement * Web3AuthenticationNonceEntity: expired_at -> expires_at * kamu-auth-web3-sqlite: implement * sqlx: update cached queries * Web3AuthenticationNonce -> Web3AuthenticationEip4361Nonce * Web3AuthenticationNonceEntity -> Web3AuthenticationEip4361NonceEntity * Web3AuthenticationEip4361NonceEntity -> Web3AuthEip4361NonceEntity * Web3AuthNonceRepository -> Web3AuthEip4361NonceRepository * Web3NonceService -> Web3AuthEip4361NonceService * Web3NonceServiceImpl -> Web3AuthEip4361NonceServiceImpl * Add auth_eip4361 to filenames * EvmWalletAddressConvertor: tests * Web3AuthEip4361NonceRepository::consume_nonce(): implement * Web3WalletAuthenticationProvider: verify nonce * PostgresWeb3AuthNonceRepository -> PostgresWeb3AuthEip4361NonceRepository * InMemoryWeb3AuthNonceRepository -> InMemoryWeb3AuthEip4361NonceRepository * SqliteWeb3AuthNonceRepository -> SqliteWeb3AuthEip4361NonceRepository * kamu-cli: register db repos * kamu-cli: move kamu_adapter_auth_web3::register_dependencies() to configure_server_catalog() * Changes after merging * test_login_enabled_methods(): fix * Self-review * Fix typos * clippy fixes * GQL: Account::account_type(): add * Account::prepare_account_name_for_storage(): save checksummed wallet address caseness * odf::AccountID: the type as an enum (initial migration) * AccountID: as_did() -> as_did_odf() * DidPkh: implement * odf::AccountID: tests * Web3WalletAuthenticationProvider::login(): generate did:pkh: account ID * Fixes after merging * test_read_shapefile_geom(): fix test * Web3WalletAuthenticationProvider: tests * Self-review * odf::AccountID::to_stack_string(): implement * odf::AccountID::as_stack_string(): implement * ToStackString: implement * odf::AccountID::as_id_without_did_prefix(): implement * Self-review * Self-review [2] * AccountProvider: introduce * lazy_static: remove dep * GQL: Account::account_provider(): return AccountProvider enum * GQL: Eip4361AuthNonce: scalar * Web3WalletAuthenticationProviderHarness: add signature generation notes * test_signature_verified(): add * kamu-cli: register kamu_adapter_auth_web3 with kamu_adapter_oauth * AccountService: create_account() -> create_password_account() * LoginPasswordAuthProvider::login(): update comment * Web3AuthenticationEip4361Nonce: use regex * opendatafabric-metadata: add "did-pkh" feature * kamu-accounts-repo-tests: remove unused dep * kamu-cli: fix "web-ui" build * GQL: AuthMut::login(): use AccountProvider for "login_method" argument * Unittests fixes * CHANGELOG.md: update * Release (minor): 0.240.0 (#1255) * Update `sqlx` to `0.8.6`, vol.2 (#1222) * sqlx: 0.8.5 * images: sqlx-cli@0.8.5 * CHANGELOG.md: update * sqlx: 0.8.6 * sqlx: 0.8.6 [2] * images/sqlx-cli: read versions from the repo * Hotfix: `web3-wallet` authorization provider: interactive login use case support (Device Flow) (#1257) * Web3WalletAuthenticationProvider: fix device flow * Release (patch): 0.240.1 * Post-merge changes --------- Co-authored-by: Sergii Mikhtoniuk <mikhtoniuk@gmail.com> Co-authored-by: Roman Boiko <roman.bv20@gmail.com>
When creating datasets on user's behalf, ensure naming scheme for the users matches parent account name.
* Simplified common fields generation via flags (is_banned, security, embeddings). Semantic embeddings in Molecule data room and announcement indices.. Support fuill and incremental embeddings indexing from Molecule data room entries (description, content text) and announcements (headline, body) * Implemented caching embeddings in the database and tracking stats for entity chunks and prompts. * Enabled hybrid search in Molecule use cases. Corrections: - dealing with dummy encoder not returning anything for prompts: degrading to textual search - corrected embedings cache queries - fixed error with announcements index sorting: HEADLINE needs a nested keyword field - hybrid search also needs secondary sort criterias for textual part, passing event_time DESC for Molecule case * Telemetry improvements around hybrid search * Disabled test dependencies for search cache crates * Scoring and explanations as hidden fields in Molecule search GQL API. * Improved RRF custom explanation
…ring "ipnft_uid" column
* molecule_data_room_mut_v2: helpers trivial updates * MoleculeActivityMessage::WriteRequested: added * MoleculeActivitySearchUpdater: "handle" WriteRequested message * MoleculeAppendGlobalDataRoomActivityUseCaseImpl: only request not write * MoleculeAsyncGlobalActivityWriter: initial implementation * DEVELOPER.md: typo fixes * Regen di.puml * MoleculeAppendGlobalDataRoomActivityUseCaseImpl: add "global" to the file name * MoleculeActivityMessagePublished: add "system_time" field * MoleculeAsyncGlobalActivityWriter::handle_write_requested_message(): fallback for "source_event_time" * MoleculeViewGlobalActivitiesUseCaseImpl::global_activities_from_search(): sort by "event_time" & add "total_count" fallback * MoleculeViewProjectActivitiesUseCaseImpl::global_activities_from_search(): sort by "event_time" & add "total_count" fallback * MoleculeAppendGlobalDataRoomActivityUseCaseImpl::ensure_within_quota(): add * Regen di.puml * Remove TODOs related to asynchronous writings * test_gql_custom_molecule_v2: provide es & src test variants * clippy fixes * test_molecule_v2_search: stabilize * Fix yanked crate: bytes * Fix yanked crate: ssi-core * test_competitive_writing_of_global_activities_src_multi_thread() * MoleculeAsyncGlobalActivityWriter::handle_write_requested_message(): add note re NOT concurent writes * MoleculeDataRoomActivityPayloadRecord::roughly_estimated_size_in_bytes(): extract the method * Fix yanked crate: time * Ingest: introduce "ignore_quota_check" flag to bypass quota check * Ingest: introduce "skip_quota_check" flag to bypass quota check (renamed)
* Backport: DataFusion `to_table()` * SessionContextBuilder: register ToTableUdtf * Fix DI circle dep issues * test_to_table_udtf_loaded: added * Self-review * ToTableUdtf: use "on_resolve_dataset_callback" * ToTableUdtf: replace Fut with AsyncFn * QueryServiceImpl::session_context(): add a context note * Update Cargo.lock after rebasing * Finalization: unit-tests
…be used across different networks (#1603) * Web3WalletAuthenticationProvider::login(): guarantee of uniqueness * Add migrations
…be used across different networks -- vol.2 (#1604) * DidPkhAccountIdentity::from_did_pkh(): extract & use * Fix yanked crate: lz4_flex (0.12.1) * DidPkhAccountIdentity::from_did_pkh(): return Result<Self, InternalError> * DidPkhAccountIdentity::from_did_pkh(): add a naive unit-test * test_create_wallet_accounts(): update
…be used across different networks -- vol.3 (#1608) * Add a note re the image patch version * DidPkhAccountIdentity::from_did_pkh(): cut email user prefix if needed * DidPkhAccountIdentity::from_did_pkh(): use UUIDv5 for email generation * DidPkhAccountIdentity::from_did_pkh(): update the note * kamu-accounts: remove unused dep secrecy
…ement was created (#1616) * test_molecule_v2_activity: compare file versions * test_gql_custom_molecule_v2: use into_json_data() * test_molecule_v2_activity: correct test * test_gql_custom_molecule_v2: fix other tests * test_gql_custom_molecule_v2: sort for asserts * FindCollectionEntriesUseCaseImpl::execute_find_multi_by_refs(): add "before_event_time" arg * Update tar crate version * MoleculeAnnouncementEntry::attachments(): clarify asVersionedFile usage * Address review comments * Update deny.toml
(cherry picked from commit 30cd9ff)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
2nd delivery includes:
moleculeGQL API group