Skip to content

GitLab

There is a security vulnerability in SSH key-generation using GitKraken < v8.0.1. If you used this tool to create SSH keys, please update GitKraken and regenerate. If you need help with this, contact forum-support@opengroup.org

Last edited by ethiraj krishnamanaidu
Page history

OSDU Platform Events

  • Introduction
  • OSDU Platform Events
  • Event Producers
  • Event Consumers
  • Registration and Notification
    • Design Option 1
    • Design Option 2
  • Security
  • ADR Decision: We had multiple discussions with CSPs, there is no good or bad design. Based on the current state of registration and notification and voting/preference it was agreed to go with Option 1.

More details : home#58

Introduction

All CSPs implemented notification and register service but there is no clear guidelines on when and how to use notification and register service.

Notification is something that happens in the backend, there is no API Spec to enforce the consistency implementation across the CSPs. This is document will provide guidelines and design options.

OSDU Platform Events

OSDU Platform events can be used to connect internal and external services/apps. The Notification platform provides an event-driven message architecture to enable services to communicate internal and external to the OSDU Platform.

Event Producers

  • Storage Service
  • Schema Service
  • Legal Service
  • Entitlement Service

Event Consumers

Record event consumers

  • Indexer – Trigger Elastic indexing
  • WKS - Trigger WKS transformation
  • Ingestion/Ingestion workflow - Trigger CSV, Manifest, OpenZGY, OpenVDS ingestion..etc
  • External consumers

Schema event consumers

  • Indexer
  • WKS
  • External consumers

Registration and Notification

The Register service allows consumers to register for event notification. You can find more details in the docs

The Notification service push events to all registered subscribers. You can find more details in the docs

A typical workflow using notification is:

  • Consumer finds a "topic" that they want to keep up to date with any changes in OSDU.
  • Consumer creates Push endpoint, that is used to receive notifications on the interested topic.
  • Consumer creates a Subscription in OSDU Notification System, and proves the ownership of the Push endpoint.
  • Consumer starts to receive notifications for that topic and processes the message to synchronize with the OSDU state.
  • Consumer periodically rotates the "secret" used for subscription.

Design 1 - Internal vs External

In this design, we are separating the internal and external flow.

  • External consumers is any consumer/app/service not part of the OSDU Data platform.

  • Tight coupling between Internal consumers and cloud technologies

  • Infrastructure configuration changes to add new consumer.

  • Pull or Push for Internal consumers.

  • Push Domain verification.

  • Better performance and less scaling requirement on notification service.

image

Design 2

In this design, every consumers have to go through the notification service.

  • External and Internal consumers go through same flow.

  • Loose coupling between consumers and cloud technologies.

  • All consumers will be registered via Register service.

  • Only push is supported.

  • High load on Notification service. Ex: 1 Million msg per day with 3 Consumer = 3 Million msg

  • Notification service have to support auto scaling on demand and highly available.

image

Security

Register service supports two types of subscription

  1. Hash-based message Authentication code(HMAC) using secret string.
  2. Service Account subscription using audience and service account key. This methods need to be reviewed and updated to support CSPs requirement.

Event payload doesn’t have record, it has only basic info(Event Id, Record Id, Schema Id, etc.)