GCP Reworked notification logic (GONRG-5700)

Type of change

• Bug Fix
• Feature

Does this introduce a change in the core logic?

• [NO]

Does this introduce a change in the cloud provider implementation, if so which cloud?

• AWS
• Azure
• Google Cloud
• IBM

Does this introduce a breaking change?

• [YES]

What is the new/expected behavior?

1. Third-party subscriber subscribes/unsubscribes for specific OSDU events notifications via the Register service REST API
2. Register service publishes registration/unregistration info via the register-subscriber-control topic/exchange
3. Notification service instances process single (statically defined) subscription/queue to obtain registration info changes
4. Notification service instances persists/reads registration info in the shared database/cache.
5. OSDU services (Storage, Schema etc.) publish their events through corresponding topics/exchanges (records-changed, schema-changed etc.)
6. Notification service instances concurrently process single subscription/queue, statically defined per each original topic/exchange
7. Notification service publishes N (N is the number of third-party subscribtions) outgoing messages via the corresponding *-notification topic/exchange (example: records-changed-notification for records changed events). Each outgoing event is the original event enriched with the destination information (subscriber id). The original event is properly acknowledged upon end of processing.
8. RabbitMQ: *-notification exchange should be a delayed exchange (type x-delayed-message), which requires special RabbitMQ rabbitmq_delayed_message_exchange plugin installed
9. RabbitMQ: the outgoing event contains no x-delay header to be processed w/o time delay
10. Notification service instances concurrently process single subscription/queue, statically defined per each outgoing topic/exchange
11. Notification service collects corresponding subscription info from the DB (4) or distributed cache and sends notification (HTTP call to the endpoint) to the third-party subscriber. In case of successful call the outgoing message is properly acknowledged. In case of notification failure (third-party endpoint not available for instance) the following should be performed to provide notification delivery retry logic:
12. GCP PubSub: the outgoing message is NACK-ed and then re-delivered after configured back-off time in accordance with Retry policy
13. RabbitMQ: the outgoing message is ACK-ed and re-published (7) in the the corresponding *-notification exchange with x-delay and x-retries headers to be re-processed with time delay. If x-retries already present in the processed message header it's value should be incremented. If x-retries value equals to configured limit of retries, the message is not re-published. In further implementations it may be routed to dead letter queue (out of scope for now)