OSDU Software issueshttps://community.opengroup.org/groups/osdu/-/issues2021-06-16T22:20:36Zhttps://community.opengroup.org/osdu/platform/system/file/-/issues/14File Integration tests are failing.2021-06-16T22:20:36ZGokul NagareFile Integration tests are failing.Below two test are failing when we run integration tests
1. first_getLocation_then_shouldReturnFileList_sameFileId
2. shouldReturnNotEmptyFileIdAndSignedURL_whenGetLocationWithoutBody
After analysing, we found that test case are expect...Below two test are failing when we run integration tests
1. first_getLocation_then_shouldReturnFileList_sameFileId
2. shouldReturnNotEmptyFileIdAndSignedURL_whenGetLocationWithoutBody
After analysing, we found that test case are expecting that when we request for /getLocation without body(without FileID) it expects a random FileID to be generated and return Location and FileId as response but in file-core code we see that when request is without FileID it gives BAD_REQUEST
Also there is code for generating random ID but that method is never gets called as we get BAD_REQUEST before this.
Note - we see there is disabled test case "getLocationShouldReturnForbidden_whenGivenNoDataAccess()" which expects 403 as response but we have checked in code and other services it return 401.ethiraj krishnamanaiduethiraj krishnamanaiduhttps://community.opengroup.org/osdu/platform/system/schema-service/-/issues/22[Schema Service] Schema Integration tests use "mvn verify" instead of "mvn test"2020-10-09T14:37:51ZRucha Deshpande[Schema Service] Schema Integration tests use "mvn verify" instead of "mvn test"
The schema service currently uses "mvn verify" to run integration tests. For consistency, "mvn test" should be used instead so that integration tests always run as part of the test phase.
*Search service uses cucumber for integration t...
The schema service currently uses "mvn verify" to run integration tests. For consistency, "mvn test" should be used instead so that integration tests always run as part of the test phase.
*Search service uses cucumber for integration tests and uses "mvn test" as well.M1 - Release 0.1ethiraj krishnamanaiduJoeRucha DeshpandeMatt Wiseethiraj krishnamanaiduhttps://community.opengroup.org/osdu/platform/system/file/-/issues/15Update naming conventions2020-09-17T17:18:18ZNicholas KarskyUpdate naming conventionsFile deliverable currently comes as a .jar file for provider/file-azure. Update so that the deliverable is a spring-boot.jar like other services. @ethiraj @dkodeihFile deliverable currently comes as a .jar file for provider/file-azure. Update so that the deliverable is a spring-boot.jar like other services. @ethiraj @dkodeihDania Kodeih (Microsoft)Dania Kodeih (Microsoft)https://community.opengroup.org/osdu/platform/system/register/-/issues/14[Register Service] TopicsRepository in register-core retrieves topics from a ...2021-06-16T22:18:17ZRucha Deshpande[Register Service] TopicsRepository in register-core retrieves topics from a json fileIn it's current state the `GET api/register/v1/topics` API retrieves topics from a json file which it reads from the classpath.
Code can be found here:
[TopicsRepository](https://community.opengroup.org/osdu/platform/system/register/-/...In it's current state the `GET api/register/v1/topics` API retrieves topics from a json file which it reads from the classpath.
Code can be found here:
[TopicsRepository](https://community.opengroup.org/osdu/platform/system/register/-/blob/master/register-core/src/main/java/org/opengroup/osdu/register/subscriber/persistence/TopicsRepository.java)
TopicsRepository should be moved outside core code.
We need some clarity on who would be creating the topics and in that case would we need to add PUT and DELETE APIs for TopicsRepository?M1 - Release 0.1ethiraj krishnamanaiduJoeRucha DeshpandeMatt Wiseethiraj krishnamanaiduhttps://community.opengroup.org/osdu/platform/system/schema-service/-/issues/23[Schem-Service] Schema service Integraion tests are failing in azure environment2020-12-02T09:52:42ZAman Verma[Schem-Service] Schema service Integraion tests are failing in azure environmentThe integration tests for schema service are failing with: `Error code: 500- Internal server error`. This failure is happening due to CosmosDB containers not being in correct shape. The current cosmos db containers for schema service, i....The integration tests for schema service are failing with: `Error code: 500- Internal server error`. This failure is happening due to CosmosDB containers not being in correct shape. The current cosmos db containers for schema service, i.e. `Source`, `Authority`, `EntityType` and `SchemaInfoStore` are created with partition key as `id`, while it should be `dataPartitionId`
WIP for fixing this.
cc: @danielscholl , @mmahathiAman VermaAman Vermahttps://community.opengroup.org/osdu/platform/system/home/-/issues/54Crs reference to fault data is not present in search results or manifest file2021-06-16T22:20:07ZMichaelCrs reference to fault data is not present in search results or manifest fileThere is no reference to the crs for fault data in the search results, manifest file, or fault data itself.
The only crs that is referenced is the projected ED50 crs that is used for the outline.
[faultDataManifest.txt](/uploads/7a83c28...There is no reference to the crs for fault data in the search results, manifest file, or fault data itself.
The only crs that is referenced is the projected ED50 crs that is used for the outline.
[faultDataManifest.txt](/uploads/7a83c28dc349ccb84b74e190759cf9f5/faultDataManifest.txt)
[searchResultsForFaultData.txt](/uploads/45787f7e80cd42b185d0917433f657c6/searchResultsForFaultData.txt)https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/issues/7Configuration Approach for New Microservices2020-10-28T22:24:28ZRostislav Dublin (EPAM)Configuration Approach for New Microservices## Change Type:
- [ ] Feature
- [ ] Bugfix
- [x] Refactoring
## Context and Scope
Currently used (legacy) microservices configuration approach is error-pron, not flexible and poorly scalable:
- The @Value annotation is commonly used. It...## Change Type:
- [ ] Feature
- [ ] Bugfix
- [x] Refactoring
## Context and Scope
Currently used (legacy) microservices configuration approach is error-pron, not flexible and poorly scalable:
- The @Value annotation is commonly used. It is redundant, hard to find and change, not type-safe (error-pron);
- Configuration properties (in .property files and @Value) are almost always mapped on capitalized names of environment variables, which reduces the set of possible means of parameter passing and possibility of hierarchic properties definition;
- The unsuitable @Component annotation is commonly used for ostensibly "configuration properties set" classes;
It does not take advantage of the modern configuration methods offered by Spring Boot.
This makes it difficult to transition to a "configuration server/clients" approach usage.
## Decision
### 1. Stop using obsolete/legacy approach:
- @Value annotations;
- @Component annotation for "configuration properties set" classes;
- Stop mapping application properties on environment variables names;
### 2. Start using recommended Spring Boot configuration/externalization approach:
Start following [Spring Boot 2 Externalized Configuration guide](https://docs.spring.io/spring-boot/docs/current/reference/html/spring-boot-features.html#boot-features-external-config) tightly;
- [Type-safe @ConfigurationProperties annotated configuration properties beans](https://docs.spring.io/spring-boot/docs/current/reference/html/spring-boot-features.html#boot-features-external-config-typesafe-configuration-properties) should be used:
- each service modules (root, core and providers') introducing at least one configuration property SHOULD have at least one such class in ```org.opengroup.osdu.<service>.<module>.config``` package, eg. ```org.opengroup.osdu.backup.provider.gcp.BackupGcpProperties```.
- More when one can be used where convenient;
- to simplify properties beans definition, Lombok @Getter and @Setter annotations should be used;
- properties beans should be [enabled with @ConfigurationPropertiesScan](https://docs.spring.io/spring-boot/docs/current/reference/html/spring-boot-features.html#boot-features-external-config-enabling)
- properties beans should be [injected into interested classes (using constructor injection)](https://docs.spring.io/spring-boot/docs/current/reference/html/spring-boot-features.html#boot-features-external-config-using)
- certain properties should be consumed by getters, eg. ```propertiesBeanVariable.getSomeProperty()```
## Rational
Current configuration approach is obsolete, error-prone, and hard for understanding maintaining.
By establishing a better configuration standard we improve, simplify and speed-up our own development process
and make it easier for all new community contributors to join and become involved faster. Plus, we wish to make our services potentially configurable with MSA "configuration server/client" model.
## Consequences
- Providers should plan refactoring their code as soon as they slots;
- No urgent activity required. Providers shouldn't sync their changes with others;
- When properties re-defined properly, nothing should break in currently defined CI/CD etcDmitriy RudkoRostislav Dublin (EPAM)Dmitriy Rudko2020-10-27https://community.opengroup.org/osdu/platform/deployment-and-operations/multi-region-deployment/multi-region/-/issues/2[Multi-Region Deployment] : Horizon 1 : AWS Configuration2022-08-23T10:47:25ZStephen Whitley (Invited Expert)[Multi-Region Deployment] : Horizon 1 : AWS ConfigurationSee [Design Review](https://community.opengroup.org/osdu/platform/deployment-and-operations/home/-/wikis/multi-region/Design/R3-Horizon1) for additional Details
**System Configuration**
- [ ] able to provision and stand-up multiple reg...See [Design Review](https://community.opengroup.org/osdu/platform/deployment-and-operations/home/-/wikis/multi-region/Design/R3-Horizon1) for additional Details
**System Configuration**
- [ ] able to provision and stand-up multiple region deployments for an OSDU instance
- [ ] repeatable deployment capability for all OSDU core services to each regional deployment
- [ ] complete set of OSDU services deployed to all regions
- [ ] region registry in place to manage which regions deployments make up an OSDU federated instance
- [ ] routing in place to direct end-user / application calls to the nearest service instances
- [ ] automation tests in place to report pass/fail for fully functional OSDU services in each region
- [ ] demonstrate observability (health and status) for these deployments
- [ ] dashboard or CLI to query service availability and notify operators on failureM1 - Release 0.1Stephen Whitley (Invited Expert)Brian KirklandGregStephen Whitley (Invited Expert)2020-11-30https://community.opengroup.org/osdu/platform/deployment-and-operations/multi-region-deployment/multi-region/-/issues/3[Multi-Region Deployment] : Horizon 1 : Azure Configuration2021-06-16T22:18:56ZStephen Whitley (Invited Expert)[Multi-Region Deployment] : Horizon 1 : Azure ConfigurationSee [Design Review](https://community.opengroup.org/osdu/platform/deployment-and-operations/home/-/wikis/multi-region/Design/R3-Horizon1) for additional Details
**System Configuration**
- [ ] able to provision and stand-up multiple reg...See [Design Review](https://community.opengroup.org/osdu/platform/deployment-and-operations/home/-/wikis/multi-region/Design/R3-Horizon1) for additional Details
**System Configuration**
- [ ] able to provision and stand-up multiple region deployments for an OSDU instance
- [ ] repeatable deployment capability for all OSDU core services to each regional deployment
- [ ] complete set of OSDU services deployed to all regions
- [ ] region registry in place to manage which regions deployments make up an OSDU federated instance
- [ ] routing in place to direct end-user / application calls to the nearest service instances
- [ ] automation tests in place to report pass/fail for fully functional OSDU services in each region
- [ ] demonstrate observability (health and status) for these deployments
- [ ] dashboard or CLI to query service availability and notify operators on failureM1 - Release 0.1Stephen Whitley (Invited Expert)Dania Kodeih (Microsoft)Brian KirklandStephen Whitley (Invited Expert)2021-02-26https://community.opengroup.org/osdu/platform/deployment-and-operations/multi-region-deployment/multi-region/-/issues/4[Multi-Region Deployment] : Horizon 1 : Open Shift Configuration2021-06-16T22:18:55ZStephen Whitley (Invited Expert)[Multi-Region Deployment] : Horizon 1 : Open Shift ConfigurationSee [Design Review](https://community.opengroup.org/osdu/platform/deployment-and-operations/home/-/wikis/multi-region/Design/R3-Horizon1) for additional Details
**System Configuration**
- [ ] stand-up multiple region deployments for an...See [Design Review](https://community.opengroup.org/osdu/platform/deployment-and-operations/home/-/wikis/multi-region/Design/R3-Horizon1) for additional Details
**System Configuration**
- [ ] stand-up multiple region deployments for an OSDU instance
- [ ] compatible deployment of core services to these deployments
- [ ] demonstrate observability (health and status) for these deploymentsM1 - Release 0.1Stephen Whitley (Invited Expert)Wladmir FrazaoBrian KirklandStephen Whitley (Invited Expert)https://community.opengroup.org/osdu/platform/deployment-and-operations/multi-region-deployment/multi-region/-/issues/5[Multi-Region Deployment] : Horizon 1 : GCP Configuration2021-06-16T22:18:54ZStephen Whitley (Invited Expert)[Multi-Region Deployment] : Horizon 1 : GCP ConfigurationSee [Design Review](https://community.opengroup.org/osdu/platform/deployment-and-operations/home/-/wikis/multi-region/Design/R3-Horizon1) for additional Details
**System Configuration**
- [ ] stand-up multiple region deployments for an...See [Design Review](https://community.opengroup.org/osdu/platform/deployment-and-operations/home/-/wikis/multi-region/Design/R3-Horizon1) for additional Details
**System Configuration**
- [ ] stand-up multiple region deployments for an OSDU instance
- [ ] compatible deployment of core services to these deployments
- [ ] demonstrate observability (health and status) for these deploymentsM1 - Release 0.1Stephen Whitley (Invited Expert)Ferris ArgyleDmitriy RudkoBrian KirklandStephen Whitley (Invited Expert)https://community.opengroup.org/osdu/platform/deployment-and-operations/multi-region-deployment/multi-region/-/issues/6[Multi-Region Deployment] : Horizon 1 : Simple Data Replication Scenarios2021-06-16T22:18:53ZStephen Whitley (Invited Expert)[Multi-Region Deployment] : Horizon 1 : Simple Data Replication ScenariosSee [Design Review](https://community.opengroup.org/osdu/platform/deployment-and-operations/home/-/wikis/multi-region/Design/R3-Horizon1) for additional Details
**Simple Scenarios**
Within all regions in an operator's OSDU instance, be...See [Design Review](https://community.opengroup.org/osdu/platform/deployment-and-operations/home/-/wikis/multi-region/Design/R3-Horizon1) for additional Details
**Simple Scenarios**
Within all regions in an operator's OSDU instance, be able to:
- [ ] replicate everything in the storage service (metadata + entitlements) between the regions and
- [ ] on user's search, leverage the delivery service to serve the requested dataset remotely
- [ ] share identity, entitlements (for the same end user), and legality tags among the regions
Acceptable functional limitations:
- no support for embargoed data
- no support for enforcing data residency restrictions when replicating metadata/dataM1 - Release 0.1Brian KirklandGregBrian Kirklandhttps://community.opengroup.org/osdu/platform/deployment-and-operations/multi-region-deployment/multi-region/-/issues/7[Multi-Region Deployment] : Horizon 1 : Azure Simple Scenarios2021-06-16T22:18:52ZStephen Whitley (Invited Expert)[Multi-Region Deployment] : Horizon 1 : Azure Simple ScenariosSee [Design Review](https://community.opengroup.org/osdu/platform/deployment-and-operations/home/-/wikis/multi-region/Design/R3-Horizon1) for additional Details
**Simple Scenarios**
- [ ] replicate everything in the storage service (me...See [Design Review](https://community.opengroup.org/osdu/platform/deployment-and-operations/home/-/wikis/multi-region/Design/R3-Horizon1) for additional Details
**Simple Scenarios**
- [ ] replicate everything in the storage service (metadata + entitlements) between them and
- [ ] on user's search, leverage the delivery service to serve the requested record remotely
- [ ] share identity and entitlements (for the same end user) among the regionsM1 - Release 0.1Dania Kodeih (Microsoft)Brian KirklandDania Kodeih (Microsoft)https://community.opengroup.org/osdu/platform/deployment-and-operations/multi-region-deployment/multi-region/-/issues/8[Multi-Region Deployment] : Horizon 1 : Open Shift Simple Scenarios2021-06-16T22:18:51ZStephen Whitley (Invited Expert)[Multi-Region Deployment] : Horizon 1 : Open Shift Simple ScenariosSee [Design Review](https://community.opengroup.org/osdu/platform/deployment-and-operations/home/-/wikis/multi-region/Design/R3-Horizon1) for additional Details
**Simple Scenarios**
- [ ] replicate everything in the storage service (me...See [Design Review](https://community.opengroup.org/osdu/platform/deployment-and-operations/home/-/wikis/multi-region/Design/R3-Horizon1) for additional Details
**Simple Scenarios**
- [ ] replicate everything in the storage service (metadata + entitlements) between them and
- [ ] on user's search, leverage the delivery service to serve the requested record remotely
- [ ] share identity and entitlements (for the same end user) among the regionsM1 - Release 0.1Wladmir FrazaoBrian KirklandWladmir Frazaohttps://community.opengroup.org/osdu/platform/deployment-and-operations/multi-region-deployment/multi-region/-/issues/9[Multi-Region Deployment] : Horizon 1 : GCP Simple Scenarios2021-06-16T22:18:51ZStephen Whitley (Invited Expert)[Multi-Region Deployment] : Horizon 1 : GCP Simple ScenariosSee [Design Review](https://community.opengroup.org/osdu/platform/deployment-and-operations/home/-/wikis/multi-region/Design/R3-Horizon1) for additional Details
**Simple Scenarios**
- [ ] replicate everything in the storage service (me...See [Design Review](https://community.opengroup.org/osdu/platform/deployment-and-operations/home/-/wikis/multi-region/Design/R3-Horizon1) for additional Details
**Simple Scenarios**
- [ ] replicate everything in the storage service (metadata + entitlements) between them and
- [ ] on user's search, leverage the delivery service to serve the requested record remotely
- [ ] share identity and entitlements (for the same end user) among the regionsM1 - Release 0.1Ferris ArgyleDmitriy RudkoBrian KirklandFerris Argylehttps://community.opengroup.org/osdu/platform/deployment-and-operations/multi-region-deployment/multi-region/-/issues/10[Multi-Region Deployment] : Horizon 1 : Design Review2020-10-29T20:24:41ZStephen Whitley (Invited Expert)[Multi-Region Deployment] : Horizon 1 : Design ReviewSee [Design Review](https://community.opengroup.org/osdu/platform/deployment-and-operations/home/-/wikis/multi-region/Design/R3-Horizon1) for additional Details
Hold design review to define strategy for multi-region deployment - focus o...See [Design Review](https://community.opengroup.org/osdu/platform/deployment-and-operations/home/-/wikis/multi-region/Design/R3-Horizon1) for additional Details
Hold design review to define strategy for multi-region deployment - focus on simple scenarios for horizon 1M1 - Release 0.1Ferris ArgyleDania Kodeih (Microsoft)Wladmir FrazaoChris ZhangDmitriy RudkoBrian KirklandGregFerris Argylehttps://community.opengroup.org/osdu/platform/deployment-and-operations/multi-region-deployment/multi-region/-/issues/11[Multi-Region Deployment] : Horizon 2 : Design Review2021-06-16T22:18:50ZStephen Whitley (Invited Expert)[Multi-Region Deployment] : Horizon 2 : Design ReviewSee [Design Review](https://community.opengroup.org/osdu/platform/deployment-and-operations/home/-/wikis/multi-region/Design/R3-Horizon1) for additional Details
Hold design review with all CSPs to define strategy for multi-region deploy...See [Design Review](https://community.opengroup.org/osdu/platform/deployment-and-operations/home/-/wikis/multi-region/Design/R3-Horizon1) for additional Details
Hold design review with all CSPs to define strategy for multi-region deployment - Review the lessons from H1 and address some of the gaps
**Horizon**
- Design review prior to start (Target: 1st week of December)
- Proposed Scope:
- implement replication of data within CSP-specific high availability zones.
- declaration and assignment of replication policy for data types / replication sets
- implement caching based on the replication policy & the record data size for low latency subsequent retrievals.M1 - Release 0.1Ferris ArgyleDania Kodeih (Microsoft)Wladmir FrazaoChris ZhangDmitriy RudkoBrian KirklandGregFerris Argylehttps://community.opengroup.org/osdu/platform/system/schema-service/-/issues/24[Schema service] : Vendor/AWS pom missing2021-06-16T22:18:40ZNeha Sarda[Schema service] : Vendor/AWS pom missingIn schema test core pom, aws pom is missing and schema test core pom has the reference of aws core lib. Hence, unable to run test cases as it is not getting compiled.In schema test core pom, aws pom is missing and schema test core pom has the reference of aws core lib. Hence, unable to run test cases as it is not getting compiled.ethiraj krishnamanaiduethiraj krishnamanaiduhttps://community.opengroup.org/osdu/platform/system/schema-service/-/issues/25[Schema service] : Azure dependency is nor resolved because of version mismatch2020-10-06T14:51:47ZNeha Sarda[Schema service] : Azure dependency is nor resolved because of version mismatchAzure dependency is nor resolved because of version mismatch, hence unable to run test cases as it's not getting compiled.Azure dependency is nor resolved because of version mismatch, hence unable to run test cases as it's not getting compiled.ethiraj krishnamanaiduethiraj krishnamanaiduhttps://community.opengroup.org/osdu/platform/system/register/-/issues/15[Register Service] Timestamp class used from com.google.cloud package in core...2021-06-16T22:18:16ZRucha Deshpande[Register Service] Timestamp class used from com.google.cloud package in core code[Ddms.java](https://community.opengroup.org/osdu/platform/system/register/-/blob/master/register-core/src/main/java/org/opengroup/osdu/register/ddms/model/Ddms.java)
and
[Subscription.java](https://community.opengroup.org/osdu/platform...[Ddms.java](https://community.opengroup.org/osdu/platform/system/register/-/blob/master/register-core/src/main/java/org/opengroup/osdu/register/ddms/model/Ddms.java)
and
[Subscription.java](https://community.opengroup.org/osdu/platform/system/register/-/blob/master/register-core/src/main/java/org/opengroup/osdu/register/subscriber/model/Subscription.java)
both import the `Timestamp` class from `com.google.cloud` package.
The standard [java.sql.Timestamp](https://docs.oracle.com/javase/8/docs/api/java/sql/Timestamp.html) should be used instead.M1 - Release 0.1ethiraj krishnamanaiduJoeRucha DeshpandeMatt Wiseethiraj krishnamanaidu2021-01-15