CRS Catalog issueshttps://community.opengroup.org/osdu/platform/system/reference/crs-catalog-service/-/issues2022-05-25T13:42:37Zhttps://community.opengroup.org/osdu/platform/system/reference/crs-catalog-service/-/issues/21Fix failing pipeline in release version of catalog2022-05-25T13:42:37ZRostislav Vatolinvatolinrp@gmail.comFix failing pipeline in release version of catalogFix failing pipeline:
https://community.opengroup.org/osdu/platform/system/reference/crs-catalog-service/-/pipelines/93466Fix failing pipeline:
https://community.opengroup.org/osdu/platform/system/reference/crs-catalog-service/-/pipelines/93466M10 - Release 0.13Rostislav Vatolinvatolinrp@gmail.comRostislav Vatolinvatolinrp@gmail.comhttps://community.opengroup.org/osdu/platform/system/reference/crs-catalog-service/-/issues/20Use actuator endpoint to check readiness of the service2022-04-05T16:52:47ZRostislav Vatolinvatolinrp@gmail.comUse actuator endpoint to check readiness of the serviceUsing actuator endpoint to check health of the service brings benefits:
1. no need to use custom or swagger API
2. no need to flood the system with useless logs
This practice is already used in such services as entitlements, legal, par...Using actuator endpoint to check health of the service brings benefits:
1. no need to use custom or swagger API
2. no need to flood the system with useless logs
This practice is already used in such services as entitlements, legal, partitionM11 - Release 0.14Rostislav Vatolinvatolinrp@gmail.comRostislav Vatolinvatolinrp@gmail.comhttps://community.opengroup.org/osdu/platform/system/reference/crs-catalog-service/-/issues/19Incorrect Info endpoint [GONRG-4529]2022-03-18T17:55:20ZDenis Karpenok (EPAM)Incorrect Info endpoint [GONRG-4529]`GET 'https://preship.gcp.gnrg-osdu.projects.epam.com/api/crs/catalog/v2/info' - 401, "Unauthorized"`
Expected:
Works well`GET 'https://preship.gcp.gnrg-osdu.projects.epam.com/api/crs/catalog/v2/info' - 401, "Unauthorized"`
Expected:
Works wellM11 - Release 0.14Chris ZhangDzmitry Malkevich (EPAM)Chris Zhanghttps://community.opengroup.org/osdu/platform/system/reference/crs-catalog-service/-/issues/6CRS Catalog APIs need to retrieve values from Reference data CoordinateRefere...2022-04-18T11:52:04ZDebasis ChatterjeeCRS Catalog APIs need to retrieve values from Reference data CoordinateReferenceSystem and CoordinateTransformationThe openAPI CRS Catalog APIs currently retrieve information from a catalog file, for example to obtain a list of Coordinate Reference Systems.
This is a legacy solution and now incorrect, i.e., the geodetic information in OSDU is stored ...The openAPI CRS Catalog APIs currently retrieve information from a catalog file, for example to obtain a list of Coordinate Reference Systems.
This is a legacy solution and now incorrect, i.e., the geodetic information in OSDU is stored as reference data CoordinateReferenceSystem and CoordinateTransformation.
Developers could use the Search API to retrieve any information but because the dedicated CRS Catalog API is there the EA and PMC desire is to keep this platform service (CRS Catalog APIs). Hence the required change is that these APIs either directly or indirectly retrieve the information from the reference data such that the same source is used.
1. Search for CRS entries in Reference data -
POST {{search_api_url}}/query
Body
```
{
"kind": "{{data-partition-id}}:wks:reference-data--CoordinateReferenceSystem:1.0.0",
"query": "id:\"odesprod:reference-data--CoordinateReferenceSystem:Geographic2D:EPSG::4267\"",
"limit": 20
}
```
2. Leverage CRS Catalog service
GET {{CRSCatalog_HOST}}/crs?limit=2300M11 - Release 0.14https://community.opengroup.org/osdu/platform/system/reference/crs-catalog-service/-/issues/27CRS Catalog release/0.15 build Failure2022-08-23T15:16:56ZShrikant GargCRS Catalog release/0.15 build FailureIndexer release/0.15 build Failure start failing as it is referring to core-common 0.15.0-SNAPSHOT which is cleaned up. Ideally SNAPSHOT versions should not be referenced from release branch/tag. So upgrading it to latest version is reco...Indexer release/0.15 build Failure start failing as it is referring to core-common 0.15.0-SNAPSHOT which is cleaned up. Ideally SNAPSHOT versions should not be referenced from release branch/tag. So upgrading it to latest version is recommended.M12 - Release 0.15Shrikant GargShrikant Garghttps://community.opengroup.org/osdu/platform/system/reference/crs-catalog-service/-/issues/31CRS Catalog does not return all records but max 10002023-10-03T15:12:58ZBert KampesCRS Catalog does not return all records but max 1000@fhoueto.amz @gehrmann @debasisc @josh.townsend @sigridmatthes @MonicaJohns
- [x] discuss issue and agree option to fix
- [ ] fix/demonstrate coordinate-reference-system works
- [ ] fix/demonstrate coordinate-transformation works
- [ ...@fhoueto.amz @gehrmann @debasisc @josh.townsend @sigridmatthes @MonicaJohns
- [x] discuss issue and agree option to fix
- [ ] fix/demonstrate coordinate-reference-system works
- [ ] fix/demonstrate coordinate-transformation works
- [ ] describe clearly in swagger and tutorial what these endpoints do (i.e., paginate to get all results.
Also check it clearly says the invalid data are excluded automatically)
- [ ] closeout
The [CRS Catalog](https://community.opengroup.org/osdu/platform/system/reference/crs-catalog-service/-/blob/master/docs/api_spec/crs-catalog-openapi-v3.yaml)
was specified and intents to make life easier for developers not used to working with CoordinateReferenceSystem. In general use cases applications would want to fetch a list of all CRS names and then display/sort/filter them to present to a user to associate with a project or dataset.
Therefore it by default is specified to return all records, as described on swagger.
However this is not what happens. Consider following queries as examples
```
{{osduonaws_base_url}}/api/crs/catalog/v3/coordinate-reference-system
{
"codeSpace": "EPSG"
}
```
```
{{osduonaws_base_url}}/api/crs/catalog/v3/coordinate-reference-system
{
"limit": 99999
}
```
```
{{osduonaws_base_url}}/api/crs/catalog/v3/coordinate-reference-system
{
"returnBoundProjectedAndProjectedBasedOnWgs84": true
}
```
All state at the end of response body it says "totalCount": >1000 for these cases, but when counting the number of actually returned records it is 1000.
I.e., the API is supposed to return approximately 1203 records for the latter case with the standard OSDU loaded CoordinateReferenceSystems.
The reason for this issue is that the Search service query API returns max 1000 records no matter what limit is specified, and the CRS catalog is wrapping to the search query. It is likely the current query set "limit": 99999.
The solutions to fix this problem could be:
1. OSDU to increase the max. limit to 10000 or 100000. For CRS data, there are a few thousand records so it would really solve the issue if this limit was removed or increased to 10000. Please see the attached email for some notes on this (this would also help people who use ordinary search).
Attachment: [https___osdu-community_ideas_aha_io_ideas_IDEA-I-75.msg](/uploads/7db44a491970344033630544b0698b95/https___osdu-community_ideas_aha_io_ideas_IDEA-I-75.msg)
2. Let catalog/v3/coordinate-reference-system API make the first call as currently is done with a query. But check the TotalCount returned (e.g., TotalCount=1203) against the number of records actually retrieved (e.g., CurrentCount=1000). Then as long as the CurrentCount is less than TotalCount, keep fetching more data using the offset parameter. When done, return all.
3. Similar to 2, but instead of query, use query_by_cursor. I don't see any benefit or difference between this and the offset method described above.
To get this resolved, clearly item 2 can be done and controlled and will work as quick fix. But it does appear in the bigger picture perhaps that a platform solution (1) is better. A decision/advice is needed from PMC/CSPs what approach should be implemented.
In the meantime, a workaround is obviously if caller monitors and checks retrieved records vs. reported totalCount. However, the purpose of the CRS Service is that developers do not have to do this.
To explain solution 2 above, I imagine currently the code does like
```
CRS_list = search/query { some_query }
return CRS_list
```
And bug fix would be like following I imagine to keep fetching records until are retrieved:
```
CRS_list = search/query { some_query }
MyCount = len(number of fetched records)
while MyCount<totalCount
CRS_list += search/query { some_query, offset=MyCount} // Maybe offset=MyCount +1 or -1 depending if the first record is 0 or 1 (I assume it is zero based in which case in first call you would get records 0 to 999, and offset by MyCount=1000 would give records 1000 to 1999 in second call, etc. but one should not rely on the current max limit that is set but simply count what gets returned vs the total number of records).
MyCount += len(number of fetched records) // Append the new records
endwhile
return CRS_list
```M18 - Release 0.21Okoun-Ola Fabien HouetoGustavo UrdanetaKIRAN ALLAMSETYOkoun-Ola Fabien Houetohttps://community.opengroup.org/osdu/platform/system/reference/crs-catalog-service/-/issues/35CRS Catalog query_with_cursor needed2023-12-05T11:19:36ZBert KampesCRS Catalog query_with_cursor needed@puneet.bhardwaj @kiranallamsety @nanigopal.roy - this issue is to fix the identified bug that query returns duplicates when using limit and offset. CRS Catalog services return all records (more than 1000 if there are more than 1000). Re...@puneet.bhardwaj @kiranallamsety @nanigopal.roy - this issue is to fix the identified bug that query returns duplicates when using limit and offset. CRS Catalog services return all records (more than 1000 if there are more than 1000). Relates to #31.
- [ ] change query to query_with_cursor and check that APIs `coordinate-reference-system` and `coordinate-transformation` do not return duplicates.
@gehrmann @chad - We found a confusing thing working on EpsgManifestPublisher. We get duplicates in the responses when Search POST query is called with limit 1000 and offset 0, then offset 1000, then offset 2000, etc. until we get TotalCount. That seems to defeat the purpose of the limit/offset parameter. We resolved it in EpsgPublisher by switching code to use query_with_cursor which does not return duplicates (using the returned cursor in the subsequent calls). I asked for an example and then I might log an issue in the Search repo, assuming this is unexpected, or is this expected behavior? To reproduce, basically call
```json
{{osduonaws_base_url}}/api/search/v2/query
{
"kind": "osdu:wks:reference-data--CoordinateReferenceSystem:1.0.0",
"returnedFields": ["id"],
"limit": 1000,
"offset": 0
}
{
"kind": "osdu:wks:reference-data--CoordinateReferenceSystem:1.0.0",
"returnedFields": ["id"],
"limit": 1000,
"offset": 1000
}
{
"kind": "osdu:wks:reference-data--CoordinateReferenceSystem:1.0.0",
"returnedFields": ["id"],
"limit": 1000,
"offset": 2000
}
```
Once I have run this I will add the responses.M21 - Release 0.24Puneet BhardwajNanigopal RoyPuneet Bhardwajhttps://community.opengroup.org/osdu/platform/system/reference/crs-catalog-service/-/issues/34CRS Catalog points-in-aou enable terminating colon for CRS record id2024-03-17T00:54:59ZBert KampesCRS Catalog points-in-aou enable terminating colon for CRS record idThe CRS conversion APIs all require a terminating colon when referencing a CRS record id in the request.
The CRS Catalog APIs for some endpoints accept a terminating colon, and in some cases do not.
This issue is intended to make the Ca...The CRS conversion APIs all require a terminating colon when referencing a CRS record id in the request.
The CRS Catalog APIs for some endpoints accept a terminating colon, and in some cases do not.
This issue is intended to make the Catalog APIs optionally accept a colon if they currently do not. This way it is backward compatible, and the various CRS APIs in Convert and Catalog are all consistent with each other and with the OSDU platform standards.
_(Side note: the Core Services Search and Storage APIs may not all be complying, but those will be ignored and the understanding is those will not be changed anyway. This issue is only about the CRS Catalog APIs)._
It is low priority, but presumed to be easy change of input parsing. Wherever currently an `id string` is passed in the request like `osdu:reference-data\--CoordinateReferenceSystem:Geographic2D:EPSG::4158` (without a terminating colon), then check that passing `osdu:reference-data\--CoordinateReferenceSystem:Geographic2D:EPSG::4158:` (with a terminating colon) does not work.
If it fails with a terminating colon, then make it work (both ways).
The suggested code fix is as follows. For API that work without a terminating colon, but not with. Then when handling the input id string, remove the terminating colon if the string has it. Then pass it on as before with colon removed. This should fix the issue, i.e., this will allow input with and without terminating colon, and the response will be unchanged (i.e., it will execute the logic as if the request was as before, without a colon).
- [ ] check [swagger](https://community.opengroup.org/osdu/platform/system/reference/crs-catalog-service/-/blob/master/docs/api_spec/crs-catalog-openapi-v3.yaml) and update as needed.
- [ ] update [tutorial](https://community.opengroup.org/osdu/platform/system/reference/crs-catalog-service/-/blob/master/docs/tutorial/CRS_Catalog_Service.md#2-crs-catalog-endpoints) with text and examples.
- NOTE: **the updated tutorial should have footnotes or remarks, i.e., the examples can consistently show usage of the colon, but then a footnote or note that prior to M21 the syntax did not accept the colon at the end of the id.**
- This should be done in issue !297 if that is still open and not in this branch. Actually !297 branch could be used to do the code changes rather than creating a branch here (or at least merge 297 before creating a branch here to avoid conflicts in tutorial updates).
- [ ] find for all APIs any `record id` (optionally) used in request body (or url parameter). This started by a report that `points-in-aou` was inconsistent with other APIs, but lets check swagger and tutorial for all APIs. An initial check shows the following:
1. `{{osduonaws_base_url}}/api/crs/catalog/v3/coordinate-reference-system?recordId=osdu:reference-data\--CoordinateReferenceSystem:Geographic2D:EPSG::4158`
- This GET currently works without a terminating colon for recordId, and I believe **it fails if a colon is added** (TBC). We can leave it that way, but ideally we fix it and change it to accept either with or without terminating colon. if this is easy to do then fix it (I mean to say, the priority is points-in-aou and we can live with this if need be).
- NOTE: the dataId= should never have a terminating colon.
2. `{{osduonaws_base_url}}/api/crs/catalog/v3/coordinate-transformation?recordId=osdu:reference-data\--CoordinateTransformation:EPSG::15851`
- This GET currently works without a terminating colon for recordId, and I believe i**t fails if a colon is added** (TBC). (same as above).
- NOTE: the dataId= should never have a terminating colon.
3. `{{osduonaws_base_url}}/api/crs/catalog` POST /v3/coordinate-reference-system`
- [This example in tutorial](https://community.opengroup.org/osdu/platform/system/reference/crs-catalog-service/-/blob/master/docs/tutorial/CRS_Catalog_Service.md#46-fetch-all-projected-crss-based-on-wgs-84-that-use-the-unit-meter) shows `BaseCRS` without a colon. **This should be made to work with and without colon**.
- NOTE: the above example in tutorial uses "opendes". This has been changed to "osdu" of course.
- NOTE: [tutorial alternatives using Search or Storage Services](https://community.opengroup.org/osdu/platform/system/reference/crs-catalog-service/-/blob/master/docs/tutorial/CRS_Catalog_Service.md#321-alternative-1-retrieve-a-single-crsct-record-using-the-search-service) shows the query does not have a terminating colon and that may be why AWS did not implement the CRS Service with a colon (because it wraps to search). This does not have to (should not be) addressed! I.e., This issue is only about CRS Catalog, very simple, and not about the platform or other Core Services.
4. `{{osduonaws_base_url}}/api/crs/catalog` POST /v3/coordinate-transformation`
- [This example](https://community.opengroup.org/osdu/platform/system/reference/crs-catalog-service/-/blob/master/docs/tutorial/CRS_Catalog_Service.md#53-find-all-cts-between-two-horizontal-crss) shows `SourceCRS` and `TargetCrs` already work with terminating colon. Please confirm in a test, and if it works with terminating colon then there is no action.
5. `{{osduonaws_base_url}}/api/crs/catalog` POST /v3/points-in-aou`
- [This example](https://community.opengroup.org/osdu/platform/system/reference/crs-catalog-service/-/blob/master/docs/tutorial/CRS_Catalog_Service.md#62-examples) shows that `"recordId": "osdu:reference-data--CoordinateTransformation:EPSG::15851"` does **not have a terminating colon**. Also we have tested a terminating colon is not accepted. To fix this, it is suggested to not change the code, except for the input parsing of the `recordId string`, i.e., if the request has a terminating colon then remove that and pass on the request further as in the current version. That way it should allow both a terminating colon and not.
<details><summary>Example - Click to expand</summary>
CRS Catalog Service references CRS and CT records by id without a closing colon (error if terminating colon is used). CRS Conversion Service requires a terminating colon (error if colon is omitted). What is way forward for this (is it provider dependent? What is the OSDU standard? Does it need to be fixed? I just added it here as placeholder and have a comment in the system. If the way forward is to fix this then a new issue should be opened for it. Also see attached email.
[RE__CRS_Catalog_Service_should_use_a_terminating_colon_in_requests_using_a_record_id_.msg](/uploads/679962df21862012b1b3ee65a839b15f/RE__CRS_Catalog_Service_should_use_a_terminating_colon_in_requests_using_a_record_id_.msg)
1. The standard is: record `id` come without trailing colon/without version as the version number is explicitly represented in the system property `version`.
2. Relationships, e.g. as in `data.BaseCRSID` _can_ refer to a specific version (`{id}:{version}`) or to implicitly the latest version by omitting the version number (`{id}:`), this is the trailing colon variant as used in the majority of the cases.
No change required for 2.
Ad 1).
The [documentation of EpsgManifestGenerator](https://gitlab.opengroup.org/osdu/subcommittees/data-def/work-products/schema/-/tree/master/ReferenceValues/Resources/IOGP#support-spatial-discovery) mentions the Multipolygon.
The current generated manifest does not. I did not check (yet) if perhaps somehow it is related to an option running Generator, or for special case with Operator that has multiple usages. The first thing would be to check the Generator with the standard -crs 3851 option to epsg.org to see what that returns.
**Test that fails:**
```json
curl --location 'https://osdu.com/api/crs/catalog/v3/points-in-aou' \
--header 'data-partition-id: osdu' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer eyJhbGciOiJm8lCA' \
--header 'Cookie: PF=lOekY5BExFoeE8n9DvELw2' \
--data '{
"recordId": "osdu:reference-data--CoordinateReferenceSystem:Projected:EPSG::3851",
"points": [
{
"latitude": -40,
"longitude": 175
},
{
"latitude": -40,
"longitude": -175
},
{
"latitude": 25,
"longitude": -90
},
{
"latitude": 45,
"longitude": -92
}
]
}
'
```
**Gives a (not so helpful) error. Expected result (Response):**
</details>M21 - Release 0.24Puneet BhardwajKIRAN ALLAMSETYPuneet Bhardwajhttps://community.opengroup.org/osdu/platform/system/reference/crs-catalog-service/-/issues/83Integrations tests fail if there are more than 1000 `osdu:wks:reference-data-...2024-02-07T13:18:13ZGuillaume CailletIntegrations tests fail if there are more than 1000 `osdu:wks:reference-data--Coordinate*:1.1.0` in the systemHi,
While checking the [AWS failures](https://community.opengroup.org/osdu/platform/system/reference/crs-catalog-service/-/jobs/2568748) I noticed that the `test_crs_catalog_v3.py` is using a basic "search" query to validate that all th...Hi,
While checking the [AWS failures](https://community.opengroup.org/osdu/platform/system/reference/crs-catalog-service/-/jobs/2568748) I noticed that the `test_crs_catalog_v3.py` is using a basic "search" query to validate that all the records are found.
But the `search` service has an [hard limit of 1000](https://community.opengroup.org/osdu/platform/system/search-service/-/blob/master/docs/tutorial/SearchService.md?ref_type=heads#parameters) records.
So if you have more than 1000 records matching this `osdu:wks:reference-data--Coordinate*:1.1.0` kind in the system the `while not found` loop checking these will never exit successfully.
A fix is to use the `query_by_cursor` API instead.M23 - Release 0.26Guillaume CailletGuillaume Caillethttps://community.opengroup.org/osdu/platform/system/reference/crs-catalog-service/-/issues/33Update CRS Catalog tutorial2023-09-11T15:40:55ZBert KampesUpdate CRS Catalog tutorialAddress feedback:
* [x] in section 1.1 , in the Projected example you are missing a colon between Projected and EPSG.
* [x] And in section 6.2, it would help adding the API call “{{osduonaws_base_url}}/api/crs/catalog/v3/points-in-aou” ...Address feedback:
* [x] in section 1.1 , in the Projected example you are missing a colon between Projected and EPSG.
* [x] And in section 6.2, it would help adding the API call “{{osduonaws_base_url}}/api/crs/catalog/v3/points-in-aou” prior to showing the payload, so users like myself understand what call is being executed (I only knew because I’ve been looking at this case).Bert KampesBert Kampeshttps://community.opengroup.org/osdu/platform/system/reference/crs-catalog-service/-/issues/30Get request can not find transformation between ED50 and WGS84 for areas sout...2022-11-22T06:42:26ZPer Helge Aarnespeaar@equinor.comGet request can not find transformation between ED50 and WGS84 for areas south of 62°N (EPSG:1613) (Azure)By running this request:
https://osdu-ship.msft-osdu-test.org/api/crs/catalog/v3/coordinate-transformation?dataId=EPSG::1613
The response is:
{
"searchResults": {
"results": [],
"aggregations": [],
"totalCoun...By running this request:
https://osdu-ship.msft-osdu-test.org/api/crs/catalog/v3/coordinate-transformation?dataId=EPSG::1613
The response is:
{
"searchResults": {
"results": [],
"aggregations": [],
"totalCount": 0
},
"query": "data.ID: \"EPSG::1613\""
}
It does respond for the 1612 transformation (north of 62°N).https://community.opengroup.org/osdu/platform/system/reference/crs-catalog-service/-/issues/29SEARCH /api/crs/catalog/v3/coordinate-transformation2022-08-31T01:14:56ZBert KampesSEARCH /api/crs/catalog/v3/coordinate-transformation_This issue should be moved to CRS Catalog; not Convert service._
Placeholder; not tested in any detail.
less important than SEARCH CRS (issue crs-conversion-service#44 ). See also issue crs-conversion-service#39 with use cases to be ..._This issue should be moved to CRS Catalog; not Convert service._
Placeholder; not tested in any detail.
less important than SEARCH CRS (issue crs-conversion-service#44 ). See also issue crs-conversion-service#39 with use cases to be satisfied by Catalog endpoint.
* [x] return more properties (see spec doc)
* [x] Demonstrate how support find all vertical
* [x] find all deprecated
* [x] find all CTs from or to specific record-id for geog2D
* [x] check if codeSpace is mandatory as in CRS (should not be)Spencer Suttonsuttonsp@amazon.comSpencer Suttonsuttonsp@amazon.comhttps://community.opengroup.org/osdu/platform/system/reference/crs-catalog-service/-/issues/25CRS Catalog API doc is missing healthcheck endpoint2022-07-11T21:34:00ZBert KampesCRS Catalog API doc is missing healthcheck endpoint@fhoueto.amz
cc - @debasisc
In AWS collection /CRS v3/CRS Catalog (and Convert services) there is an endpoint {{osduonaws_base_url}}/api/crs/catalog/actuator/health.
this endpoint seems undocumented in the API swagger page. Request ...@fhoueto.amz
cc - @debasisc
In AWS collection /CRS v3/CRS Catalog (and Convert services) there is an endpoint {{osduonaws_base_url}}/api/crs/catalog/actuator/health.
this endpoint seems undocumented in the API swagger page. Request is
* [ ] add endpoint to doc
* [ ] it would be opportune to ensure all endpoints are covered. I did not do a 1 by 1.
thanks - bertOkoun-Ola Fabien HouetoOkoun-Ola Fabien Houetohttps://community.opengroup.org/osdu/platform/system/reference/crs-catalog-service/-/issues/24The CRS Catalog service V3 in AWS platform validation environment R3 M12 is n...2022-12-09T13:32:50ZKamlesh TodaiThe CRS Catalog service V3 in AWS platform validation environment R3 M12 is not able to find the existing coordinateTransformation reference dataOn AWS platform validation environment R3 M12, when trying to use the CRS Catalog Service V3, the request to get the coordinatetransformation record using the recordid is not returning any record.
curl --location --request **GET 'https:/...On AWS platform validation environment R3 M12, when trying to use the CRS Catalog Service V3, the request to get the coordinatetransformation record using the recordid is not returning any record.
curl --location --request **GET 'https://sp221718.forumtesting.osdu.aws/api/crs/catalog/v3/coordinate-transformation?recordId**=osdu:reference-data--CoordinateTransformation:EPSG::1111' \
--header 'data-partition-id: osdu' \
--header 'Authorization: Bearer eyJraWQiOiJRV3RKSUxzZHhhMndr...IrBfULlZ3T3jfQ'
Response: 200 OK
{
"searchResults": {
"results": [],
"aggregations": [],
**"totalCount": 0**
},
"query": "id: \"osdu:reference-data--CoordinateTransformation:EPSG::1111\""
}
On other hand using the recordid and the SEARCH API, one is able to find the same coordinatetransformation record using the recordid
curl --location --request POST 'https://sp221718.forumtesting.osdu.aws/api/search/v2/query' \
--header 'Content-Type: application/json' \
--header 'data-partition-id: osdu' \
--header 'Authorization: Bearer eyJraWQiOiJRV3RKSUxzZHhhMndr...IrBfULlZ3T3jfQ' \
--data-raw '{
"kind": "osdu:wks:reference-data--CoordinateTransformation:1.0.0",
"query": "id: \"osdu:reference-data--CoordinateTransformation:EPSG::1111\""
}
Response 200 OK
{
"results": [
{
"data": {
"PreferredUsage.Extent.Description": null,
"PersistableReference": "{\"authCode\":{\"auth\":\"EPSG\",\"code\":\"1111\"},\"name\":\"Ain_El_Abd_To_WGS_1984_2\",\"type\":\"ST\",\"ver\":\"PE_10_9_1\",\"wkt\":\"GEOGTRAN[\\\"Ain_El_Abd_To_WGS_1984_2\\\",GEOGCS[\\\"GCS_Ain_el_Abd_1970\\\",DATUM[\\\"D_Ain_el_Abd_1970\\\",SPHEROID[\\\"International_1924\\\",6378388.0,297.0]],PRIMEM[\\\"Greenwich\\\",0.0],UNIT[\\\"Degree\\\",0.0174532925199433]],GEOGCS[\\\"GCS_WGS_1984\\\",DATUM[\\\"D_WGS_1984\\\",SPHEROID[\\\"WGS_1984\\\",6378137.0,298.257223563]],PRIMEM[\\\"Greenwich\\\",0.0],UNIT[\\\"Degree\\\",0.0174532925199433]],METHOD[\\\"Geocentric_Translation\\\"],PARAMETER[\\\"X_Axis_Translation\\\",-143.0],PARAMETER[\\\"Y_Axis_Translation\\\",-236.0],PARAMETER[\\\"Z_Axis_Translation\\\",7.0],OPERATIONACCURACY[18.0],AUTHORITY[\\\"EPSG\\\",1111]]\"}",
"CoordinateTransformationType": null,
"PreferredUsage.AuthorityCode.Code": 8032,
"ResourceCurationStatus": null,
"Name": "Ain el Abd to WGS 84 (2)",
"CoordTfmVersion": "DMA-Sau",
"PreferredUsage.Scope.AuthorityCode.Authority": "EPSG",
"PreferredUsage.Extent.BoundingBoxSouthBoundLatitude": 16.37,
"PreferredUsage.Extent.BoundingBoxWestBoundLongitude": 34.51,
"PreferredUsage.Name": "Saudi Arabia - onshore",
"Kind": null,
"ResourceSecurityClassification": null,
"PreferredUsage.Extent.AuthorityCode.Authority": "EPSG",
"ID": "EPSG::1111",
"ExistenceKind": null,
"Usages": [
{
"Scope": {
"AuthorityCode": {
"Authority": "EPSG",
"Code": 1160
},
"Name": "Military survey."
},
"Extent": {
"BoundingBoxWestBoundLongitude": 34.51,
"AuthorityCode": {
"Authority": "EPSG",
"Code": 3303
},
"BoundingBoxNorthBoundLatitude": 32.16,
"BoundingBoxEastBoundLongitude": 55.67,
"BoundingBoxSouthBoundLatitude": 16.37,
"Name": "Saudi Arabia - onshore"
},
"AuthorityCode": {
"Authority": "EPSG",
"Code": 8032
},
"Name": "Saudi Arabia - onshore"
}
],
"Method.Name": "Geocentric translations (geog2D domain)",
"PreferredUsage.Scope.AuthorityCode.Code": 1160,
"Code": "1111",
"TargetCRS.AuthorityCode.Code": 4326,
"Method.AuthorityCode.Code": 9603,
"AttributionAuthority": null,
"Variant": 2,
"AttributionRevision": null,
"TargetCRS.AuthorityCode.Authority": "EPSG",
"TargetCRS.Name": "WGS 84",
"Description": null,
"PreferredUsage.Extent.BoundingBoxEastBoundLongitude": 55.67,
"SourceCRS.SourceCRSID": "osdu:reference-data--CoordinateReferenceSystem:Geographic2D:EPSG::4204:",
"ResourceLifecycleStatus": null,
"TechnicalAssuranceID": null,
"TargetCRS.TargetCRSID": "osdu:reference-data--CoordinateReferenceSystem:Geographic2D:EPSG::4326:",
"Source": "Workbook Resources/IOGP/Manifests/reference-data/CoordinateTransformation.1.0.0.json; commit SHA 9ed9f2cc.",
"SourceCRS.AuthorityCode.Authority": "EPSG",
"Accuracy": 18.0,
"PreferredUsage.Extent.AuthorityCode.Code": 3303,
"PreferredUsage.Extent.BoundingBoxNorthBoundLatitude": 32.16,
"Method.AuthorityCode.Authority": "EPSG",
"CommitDate": "2021-12-13T13:08:21+0100",
"AttributionPublication": null,
"InactiveIndicator": null,
"CodeSpace": "EPSG",
"CodeAsNumber": 1111,
"SourceCRS.AuthorityCode.Code": 4204,
"InformationSource": null,
"ResourceHomeRegionID": null,
"RevisionDate": "2020-03-14T00:00:00+0000",
"SourceCRS.Name": "Ain el Abd",
"PreferredUsage.Extent.Name": "Saudi Arabia - onshore",
"PreferredUsage.AuthorityCode.Authority": "EPSG",
"PreferredUsage.Scope.Name": "Military survey."
},
"kind": "osdu:wks:reference-data--CoordinateTransformation:1.0.0",
"source": "wks",
"acl": {
"viewers": [
"data.default.owners@osdu.example.com"
],
"owners": [
"data.default.viewers@osdu.example.com"
]
},
"type": "reference-data--CoordinateTransformation",
"version": 1652798924649780,
"tags": null,
"modifyUser": "serviceprincipal@testing.com",
"modifyTime": "2022-05-17T14:48:44.798Z",
"createTime": "2022-05-17T14:48:29.723Z",
"authority": "osdu",
"namespace": "osdu:wks",
"legal": {
"legaltags": [
"osdu-public-usa-dataset"
],
"otherRelevantDataCountries": [
"US"
],
"status": "compliant"
},
"createUser": "serviceprincipal@testing.com",
"id": "osdu:reference-data--CoordinateTransformation:EPSG::1111"
}
],
"aggregations": null,
"totalCount": 1
}
The collection that I am using was provided by AWS team member Nehru, Santhos.Okoun-Ola Fabien HouetoOkoun-Ola Fabien Houetohttps://community.opengroup.org/osdu/platform/system/reference/crs-catalog-service/-/issues/22ADR - Deprecate current set of APIs and create new set which works from (dyna...2023-01-26T01:33:01ZDebasis ChatterjeeADR - Deprecate current set of APIs and create new set which works from (dynamic) source of Reference data - per Geomatics team's recommendation## Status
- [x] Proposed
- [x] Trialing
- [x] Under review
- [x] Approved
- [ ] Retired
## Context
Recent discussions involving Geomatics team, PMC and EA members.
One of them being on 9-Mar-2022 (see attachment, slide-3).
[OSDU_Geoma...## Status
- [x] Proposed
- [x] Trialing
- [x] Under review
- [x] Approved
- [ ] Retired
## Context
Recent discussions involving Geomatics team, PMC and EA members.
One of them being on 9-Mar-2022 (see attachment, slide-3).
[OSDU_Geomatics_for_meeting_23-09-21_with_EA_team__002_.pdf](/uploads/fda9b9caf91defbda43b441592fc2616/OSDU_Geomatics_for_meeting_23-09-21_with_EA_team__002_.pdf)
There are two parts in this ADR :
1. Deprecate (over time) currently available set of "CRS Catalog" APIs
2. Design specification and implement new set of APIs
Do not expect to find detailed design specification of new APIs here as they will be created separately.
This issue is simply to initiate discussion for decision so that real work can start.
Task-2 will be done by Geomatics SMEs (@bert.kampes , @sigridmatthes , Phil (phil.summerfield@exxonmobil.com), Jin Zhu (jin.zhu@chevron.com) , @josh.townsend et al) and @gehrmann of "Data Definition" team in collaboration with Developer resources (assigned from AWS).
AWS contact is @fhoueto.amz
You can find historical context in the following issues:
- APIs need to retrieve values from Reference data CoordinateReferenceSystem and CoordinateTransformation (https://community.opengroup.org/osdu/platform/system/reference/crs-catalog-service/-/issues/6)
- Need simple mechanism to perform regional search, with further filtering (https://community.opengroup.org/osdu/platform/system/reference/crs-catalog-service/-/issues/9)
- more will be added here
## Scope
Provide enough contextual information here for the Decision makers to agree (or disagree) about deprecating current set of APIs and building a new set of APIs to effectively perform "CRS Catalog" functions from dynamic source.
## Decision
EA team may review "pros and cons", risks and provide decision (with approval or otherwise) -
1. Deprecate (over time - perhaps 6-9 months to play safe) currently available set of "CRS Catalog" APIs
2. Design specification and implement new set of APIs
## Rationale
**Why do we need to deprecate** current set of APIs and not attempt to fix?
Current set of "CRS Catalog" APIs are sourcing information from a static file and are completely disconnected from the true/dynamic source (Reference data).
The catalog is using a different data model that we don’t use with the legacy data. That is why it is a dead end.
**Why do we need to build new** set of APis?
Instead of taking the band-aid approach of "making the old set of APis work off new dynamic source", Geomatics team SMEs thought it is better approach to build new end-points that make more sense at the same time of switching source (from static to dynamic).
**Why we cannot simply get by with complex/advanced search?**
One reason being this will be "tall" ask for end users to figure out very complex search syntax in Lucen for Elasticsearch.
The envisioned end points are not just things that can be accomplished by a single complex call the ordinary search. It may be it takes 2 calls for example to accomplish fetching a list of CRS to associated with ingested data (i.e., fetch all boundCRSs and then augment that with a fetched list of WGS 84 based CRSs (because they are not considered of type bound in OSDU).
Another example is dealing with the antimeridian for area search may not be possible with direct search but the Catalog service will enable this convenient way for programmers to use when developing applications using this service.
## Consequences
- Alert community about upcoming deprecation of existing API services (in case someone is using these in their applications).
- Make suitable documentation and socialize widely new set of APIshttps://community.opengroup.org/osdu/platform/system/reference/crs-catalog-service/-/issues/17Upgrade to Log4J 2.172021-12-21T01:50:29ZDavid Diederichd.diederich@opengroup.orgUpgrade to Log4J 2.17The Apache Foundation released another Log4j2 update, version 2.17, which address a denial of service vulnerability.
This issue tracks progress to upgrade this dependency for this project.The Apache Foundation released another Log4j2 update, version 2.17, which address a denial of service vulnerability.
This issue tracks progress to upgrade this dependency for this project.https://community.opengroup.org/osdu/platform/system/reference/crs-catalog-service/-/issues/16Log4J Expedient Updates and Patches2021-12-16T02:39:57ZDavid Diederichd.diederich@opengroup.orgLog4J Expedient Updates and PatchesThis issue associates MRs that were applied to this project quickly to get a patched version ready as soon as possible. The intent is to provide a reference point for later, more thoughtful, analysis.This issue associates MRs that were applied to this project quickly to get a patched version ready as soon as possible. The intent is to provide a reference point for later, more thoughtful, analysis.David Diederichd.diederich@opengroup.orgDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/system/reference/crs-catalog-service/-/issues/15Use HPA for kubernetes service2021-10-12T13:57:53ZRostislav Vatolinvatolinrp@gmail.comUse HPA for kubernetes serviceImplement practices described here: https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/Implement practices described here: https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/https://community.opengroup.org/osdu/platform/system/reference/crs-catalog-service/-/issues/14[BUG] CORS configuration missing allowed header2021-10-20T14:33:44ZYifan Ye[BUG] CORS configuration missing allowed headerThe allowed header is missing: Content-Type causing such error:
Access to fetch has been blocked by CORS policy: Request header field content-type is not allowed by Access-Control-Allow-Headers in preflight response.The allowed header is missing: Content-Type causing such error:
Access to fetch has been blocked by CORS policy: Request header field content-type is not allowed by Access-Control-Allow-Headers in preflight response.Yifan YeYifan Yehttps://community.opengroup.org/osdu/platform/system/reference/crs-catalog-service/-/issues/13Add memory limits2021-10-20T14:33:52ZYifan YeAdd memory limitsAdding memory limits:
AKS node autoscaler uses memory limits to add nodes to cluster when HPA(Horizontal Pod Autoscaler) needs more capacity. Values were experimentally determined. Given implementation allows managing the list of envs w...Adding memory limits:
AKS node autoscaler uses memory limits to add nodes to cluster when HPA(Horizontal Pod Autoscaler) needs more capacity. Values were experimentally determined. Given implementation allows managing the list of envs where limits are enabled.Yifan YeYifan Ye