Upgrade re-index & clear-index API for x-collaboration support
re-index & clear-index respect x-collaboration header
[Any technical details, guides, documentation, wikis, and many more that could be useful to complete this story.]
Implement service methods to request lists of WIP-resources for the CP from Search service. It should be triggered by the API: #72
See discussion in the https://gitlab.opengroup.org/osdu/subcommittees/data-def/work-products/schema/-/issues/644 issue
Service methods are implemented so when /wip-resources API is consumed they are invoked and requesting Search for the records in the CP's namespace.
Today's Search logic is namespace-aware. It doesn't respect the x-collaboration header and can't perform a search in a custom namespace. We need to implement an accordant logic for it.
See ADR - Project & Workflow Services - Core Services Integration - Search Service Support
When Indexer receives a message from "Record Changed V2" topic, the message contains x-collaboration property that points on record in a custom NS. The indexing logic for records in custom namespace should be implemented in compliance with the recuirements in the following ADR: ADR - Project & Workflow Services - Core Services Integration - Search Service Support
When Storage service changes a record in a custom namespace, it is sending a message into a new "Record Changed V2" topic instead of the legacy "record Changed V1" topic. The listener for this new topic needs to be implemented in the Indexer service.
See ADR - Project & Workflow Services - Core Services Integration - Search Service Support
When /projects/{id}/publish API is used to publish a project, the accordant service methods should handle the operation. They should consume Storage servicew record copy API and copy all record references from the CP's custom namespace to the SOR.
See ADR - Project & Workflow Services - ADR Summary
/projects/{id}/publish API implementation
See ADR - Project & Workflow Services - ADR Summary
When a record is changed in the custom namespace, a message should be sent to the "Record Changed V2" topic on message broker instead of the usual "Record Changed V1" topic. The requirement is mentioned in the following ADRs:
It has already been implemented for Azure CSP, see MR !553
[Any technical details, guides, documentation, wikis, and many more that could be useful to complete this story.]
Implement copying records between namespaces in OSDU Storage service. See ADR - Project & Workflow Services - Core Services Integration - Copy Record references between namespaces for details
Storage service has API and service implemented for copying record references between namespaces
getMultipleRecords
createUpdateRecords
with collab context provided, which should be taken from the request body.getMultipleRecords
with the provided Collaboration Context, should be taken from header.createUpdateRecords
with collab context provided, which should be taken from the request body.getMultipleRecords
with the provided Collaboration Context, should be taken from header.createUpdateRecords
without collab context provided, adjust record ID if necessary.3.In order to comply with the requirement from ADR, We need to update the hard deletion method in :RecordServiceImpl
, but that might require adding a dependency on the PWS service since we do not have any other means to verify that the record exists in WIP.
The hard delete API in storage service needs to add extra validation that the data blob being deleted from storage is not referenced in a different context.
Messaging for collaboration context is not implemented in the AWS module. We may use Azure MessageBusImpl
and ServiceBusPublisher
as a samples. From a high-level perspective, it appears that Azure does not utilize separate topics; instead, it enhances event messages with collaboration context if it is available.
Add new test cases to the IT core module https://community.opengroup.org/osdu/platform/system/storage/-/tree/master/testing/storage-test-core/src/main/java/org/opengroup/osdu/storage/records
I should be able to copy existing in the SOR record to the custom WIP Context
I should be able to copy existing in the WIP Context record to the SOR
I should be able to copy existing in the WIP Context record to another WIP Context
I shouldn't be able to copy existing in the SOR record to the custom WIP Context if it exists in WIP
I shouldn't be able to copy not existing records
Rostislav Dublin (EPAM) (a9c18abb) at 11 Mar 15:38
execises
@valeriiaLipchenko, I had no chance to review the MRs code changes, but I performed a quick e2e test of the POST /formationpressuretests method on EPMOSDU instance using an intentionally wrong payload (a manifest of the Welllog kind).
The test failed: the service successfully passed the payload and created a Welllog record with the FTP endpoint. It is wrong. In such a situation, the service MUST return an error similar to what POST /wellboretrajectories method returns when you try to request it with a wrong kind payload:
{
"detail": "Record is not an OSDU trajectory"
}
The found issue shows that the validator service for the /formationpressuretests endpoint is not implemented properly. Please correct the found issue. After that I can continue my review and testing.
This is an umbrella spike investigation. We need to understand what core OSDU services (at least Storage, Indexer, Search, and Notification) must be upgraded to allow us to implement what is included in the P&WS MVP1 requirements for P&WS service.
As a short definition, what we require from those services is the so-named "x-collaboration header" (or in other words "custom namespaces") support.
It is needed, in particular, for the CPs' WIP resources inventory feature (see the Story #72 for "/projects/{id}/wip-resources API endpoint" implementation).
Here is what support is needed from each of the OSDU services:
Storage. It must support storing records in custom namespaces. We know that some part of the work is already done - the Storage service API (at least its Swagger) is upgraded for understanding the "x-collaboration" request header. However, we doubt if the header information is already respected by the CSPs' service/persistence layers. We need to test at least for AWS CSP and figure out the technical debt.
Indexer. When the Storage svc ingests a record it sends a "record_changed" event into the accordant OSDU's Pub/Sub topic. Then the Indexer picks up the message and performs indexing of the record into an accordant ES index. We need to check if the Indexer is already capable of distinguishing SOR and custom namespaces and indexing them into different indexes.
Search. When we send a query to the Search service we want it to distinguish requests for records in SOR or custom namespaces. For that, it must understand and respect "x-collaboration" headers.
Notification. This service subscribes to OSDU Pub/Sub topics and works as a proxy for outer services and applications that are its subscribers. We need to check if it already allows its subscribers to subscribe to only SOR or custom namespaces' records updates.
For each point of the investigation, a separate ticket should be created and assigned to Dev team members. The current task should be closed only when:
Will be defined in the children tickets.
See #72 for understanding API that requires the service
Also see the ADR "#108 Core Services Integration - Search Service Support"
Implement the Service
data.Namespace
The current version of the OSDU Search service is not capable of running queries in a custom namespace context. So, the service can be fed with the mock data. And put TODO for improving it in the future.
Again, see the ADR "#108 Core Services Integration - Search Service Support"
WIP resources are work-in-progress resources that the CP creates and modifies during the CP lifecycle. They are contained in the CP dedicated namespace, so they are not visible in the OSDU SOR namespace until published at the end of the CP. OSDU users create WIP resources in OSDU using direct requests to the OSDU Storage and DDMSs API. The only way to get a list of the WIP resources for the CP is to query the OSDU Search service API.
So, there is only need for a single API method:
GET /projects/{id}/wip-resources
That returns a JSON array of WIP resource record IDs
#66 adds /projects/{id}/resources API endpoint. Need to provide corresponding service layer for all its CRUD methods.
All new API methods are provisioned with corresponding Service layer methods. When the API requested, pointed resource data is validated,
Logic:
check if CP record has data.TrustedResourcesID not empty 1.1. if empty - we create a record of CollaborationProjectResources kind and set it to CP's data.TrustedResourcesID 1.2. if not empty - we get existing CollaborationProjectResources record
changes are applied (add/delete) to the CollaborationProjectResources data.ResourcesIDs[]
list.
PS: inform users if they try to delete non-existing resources references
When the project is Created, it contains a list of SOR resources in the data.Resources
node.
Some resources may be added during the project initial POST
operation.
But resources can also CRUDed after the project creation by the following requests.
Need to implement and document /projects/{id}/resources
API endpoint for list of resources CRUD
The following API methods are implemented and documented: