Skip to content

Cumulative MR #1 Overall Feature Aggregation for Transformer and Provider Configuration and Command Management

Rostislav Dublin (EPAM) requested to merge 547-reset-config-for-provider into master

Type of change

  • Feature

Linked Issues:

Does this introduce a change in the core logic?

  • YES

Does this introduce a change in the cloud provider implementation, if so which cloud?

  • AWS
  • Azure
  • GCP
  • IBM

Does this introduce a breaking change?

  • NO

What is the current behavior?

  1. Transformer app lacks live configuration updates, and configuration is only possible via deployment-time properties files.
  2. There is no API for managing TransformerLoadStatusCache.
  3. Validation of Kind properties lacks depth.
  4. No API exists for consuming logs with flexible date ranges and pagination.
  5. Cache management, cron expression validation, and the description field in Kind are missing.
  6. There is no capability to reset configurations to their initial state or apply new configurations dynamically.

What is the new/expected behavior?

This merge request aggregates several significant features into one comprehensive update across both Transformer and Provider services:

1. Live Configuration Updates for Transformer (#465 (closed), #519)

  • Configuration externalized: Transformer now supports live configuration updates without downtime.
  • Ambassador API: CRUD operations for KindCacheConfiguration and real-time configuration updates for caches.
  • Scalability: Support for multiple replicas with task load-balancing and zero-outage scaling.
  • Persistence: Optional persistent mode, even for local profiles, to test post-restart recovery.

2. TransformerLoadStatusCache API (#520)

  • Introduced the TransformerLoadStatusController to manage TransformerLoadStatusCache with GET, PUT, and PATCH operations.
  • Refactored TransformerLoadStatusCache into an interface and service layer to manage data access more effectively.

3. Enhanced Kind Validation and Testing (#521)

  • Externalized validation logic for Kind structure and entity type identifiers into KindUtil and TransformerConfigurationCacheService.
  • Added robust exception handling and comprehensive unit tests for ValidKindConstraintValidator.

4. Log Consumption API with Pagination (#535 (closed))

  • Introduced /gcz/transformer/admin/logs API for retrieving logs with support for:
    • Date range filtering using flexible date formats (ISO and simple).
    • Pagination and keyword-based filtering.
    • Validation of log stream identifiers via LogStream enum.
  • Swagger and OpenAPI documentation enhanced with usage examples.

5. Cache Update API, Cron Expression Validator, and Kind Description (#536 (closed), #537 (closed), #538 (closed), #540 (closed))

  • Cache Update API: Supports updating a single cache or a list of caches via new API endpoints.
  • Cron Expression Validation: Added CronValidator compatible with Spring’s CronTrigger and validated via unit tests.
  • Description Field: Introduced a description field in the Kind class for better contextual information.
  • Config File Download: Support for downloading Transformer (YAML) and Provider (JSON) configuration files.

6. Reset Configuration for Transformer and Provider (#539 (closed))

  • Transformer: /admin/transformerConfig/reset API added for resetting configurations, either to the initial state or applying a new custom configuration (JSON, YAML, or properties).
  • Provider: /admin/providerConfig/reset API allows resetting provider configurations similarly, ensuring transactional consistency with LayerDefinition entities.

7. Provider Command Cache Management (#547 (closed))

  • Implemented ProviderCommandCacheService to manage provider commands, with an auto-incremented ID system and command persistence.
  • Refactored the polling mechanism to process only the latest unread command per type, with lastProcessedId.txt persisted for continuity after restarts.
  • Koop.js Lifecycle: Refactored Koop.js server lifecycle to manage start, stop, and reload processes upon command detection.

8. Support for "Disabled" Property in Transformer and Provider (#548 (closed), #549 (closed))

  • Added disabled property to KindCacheConfig for Transformer and LayerDefinition for Provider.
  • This property allows disabling specific caches and layers, respectively, making the configuration more flexible.
  • These properties are configurable through the respective APIs and are supported by validation and persistence mechanisms in both services.

Testing and Validation

  1. Configuration Updates:

    • Test live configuration updates by using the Ambassador API for KindCache CRUD operations.
    • Test scaling with multiple replicas, ensuring no conflicts or duplicated task executions.

  2. Log Consumption API:

    • Test log retrieval with date range filters, pagination, and keyword-based searches.
    • Validate correct log stream handling using the LogStream enum.

  3. Cache Updates:

    • Test the new cache update API for both single and group updates, verifying success and failure responses.

  4. Reset Configuration:

    • Test resetting both Transformer and Provider configurations to their initial state or applying new configurations using the API.

  5. Provider Command Cache:

    • Test the polling mechanism and verify proper command processing and logging in both Provider and Transformer services.

  6. "Disabled" Property:

    • Test disabling specific KindCacheConfig in Transformer and LayerDefinition in Provider using the API.
    • Ensure that the disabled caches and layers do not process updates and are skipped during operations.

Swagger Documentation and API Endpoints

  • All new API endpoints are documented and include examples for log consumption, cache management, and configuration reset operations.
  • Swagger doc examples are provided for each API method in /gcz/transformer/admin/swagger-ui.

Have you added/updated Unit Tests and Integration Tests?

  • YES: Unit tests have been added and updated across multiple services, including cache management, configuration resets, command handling, and log consumption.
Edited by Rostislav Dublin (EPAM)

Merge request reports