Frame of Reference and Reference Normalization
Table of Contents
Introduction
For all the existing records in Data Ecosystem, we need a way for the user to retrieve the data so that it is rationalized to common standard allowing for comparison from multiple data sources. For this, we currently support a few ways of supplying the frame of reference(FoR). The FoR provides the context in which the numbers are to be interpreted. With FoR properly supplied, we can then use the FoR to convert to a standard format, which is units in SI, crs in wgs84, elevation in msl, azimuth in true north, dates in utc.
Data records in Data Ecosystem collect their FoR in the meta section. As for now, it contains the full, self-contained description of Units, Measurements and Coordinate Reference Systems used in the record/document. We currently support conversion for only units and crs, other three types(dates, elevation and azimuth) will be available later. In Addition, only single coordinates are supported and the coordinate property names must follow a specific naming convention. This is later explained int the Scenarios we support section.
Meta Section
What is in meta section
Meta section is an array of elements collecting FoR information in records. The arrayed elements are called MetaItem.
What is MetaItem
A MetaItem is a member of record's meta section and includes following properties:
- kind: a string(enumeration) identifying the kind of reference. Currently we only support the conversion for kind "Unit" and "CRS".
- name: a string containing a human readable name/symbol for the reference.
- persistableReference: a string containing the self-contained description of the unit or CRS.
- propertyNames: an array of strings. The elements are property names, for which the reference is fixed.*CRS Catalog Serviceand Unit Catalog services are suppliers of persistableReference strings.
CRS example persistable reference strings
A non-trivial projected CRS:
"name": "NAD27 * OGP-Usa Conus / Louisiana South [26782,15851]"
"persistableReference": "{\"lateBoundCRS\":{\"wkt\":\"PROJCS[\\\"NAD_1927_StatePlane_Louisiana_South_FIPS_1702\\\",GEOGCS[\\\"GCS_North_American_1927\\\",DATUM[\\\"D_North_American_1927\\\",SPHEROID[\\\"Clarke_1866\\\",6378206.4,294.9786982]],PRIMEM[\\\"Greenwich\\\",0.0],UNIT[\\\"Degree\\\",0.0174532925199433]],PROJECTION[\\\"Lambert_Conformal_Conic\\\"],PARAMETER[\\\"False_Easting\\\",2000000.0],PARAMETER[\\\"False_Northing\\\",0.0],PARAMETER[\\\"Central_Meridian\\\",-91.3333333333333],PARAMETER[\\\"Standard_Parallel_1\\\",29.3],PARAMETER[\\\"Standard_Parallel_2\\\",30.7],PARAMETER[\\\"Latitude_Of_Origin\\\",28.6666666666667],UNIT[\\\"Foot_US\\\",0.304800609601219],AUTHORITY[\\\"EPSG\\\",26782]]\",\"ver\":\"PE_10_3_1\",\"name\":\"NAD_1927_StatePlane_Louisiana_South_FIPS_1702\",\"authCode\":{\"auth\":\"EPSG\",\"code\":\"26782\"},\"type\":\"LBC\"},\"singleCT\":{\"wkt\":\"GEOGTRAN[\\\"NAD_1927_To_WGS_1984_79_CONUS\\\",GEOGCS[\\\"GCS_North_American_1927\\\",DATUM[\\\"D_North_American_1927\\\",SPHEROID[\\\"Clarke_1866\\\",6378206.4,294.9786982]],PRIMEM[\\\"Greenwich\\\",0.0],UNIT[\\\"Degree\\\",0.0174532925199433]],GEOGCS[\\\"GCS_WGS_1984\\\",DATUM[\\\"D_WGS_1984\\\",SPHEROID[\\\"WGS_1984\\\",6378137.0,298.257223563]],PRIMEM[\\\"Greenwich\\\",0.0],UNIT[\\\"Degree\\\",0.0174532925199433]],METHOD[\\\"NADCON\\\"],PARAMETER[\\\"Dataset_conus\\\",0.0],AUTHORITY[\\\"EPSG\\\",15851]]\",\"ver\":\"PE_10_3_1\",\"name\":\"NAD_1927_To_WGS_1984_79_CONUS\",\"authCode\":{\"auth\":\"EPSG\",\"code\":\"15851\"},\"type\":\"ST\"},\"ver\":\"PE_10_3_1\",\"name\":\"NAD27 * OGP-Usa Conus / Louisiana South [26782,15851]\",\"authCode\":{\"auth\":\"SLB\",\"code\":\"26782079\"},\"type\":\"EBC\"}"A simple UTM projection
"name": "WGS_1984_UTM_Zone_31N"
"persistableReference": "{\"wkt\":\"PROJCS[\\\"WGS_1984_UTM_Zone_31N\\\",GEOGCS[\\\"GCS_WGS_1984\\\",DATUM[\\\"D_WGS_1984\\\",SPHEROID[\\\"WGS_1984\\\",6378137.0,298.257223563]],PRIMEM[\\\"Greenwich\\\",0.0],UNIT[\\\"Degree\\\",0.0174532925199433]],PROJECTION[\\\"Transverse_Mercator\\\"],PARAMETER[\\\"False_Easting\\\",500000.0],PARAMETER[\\\"False_Northing\\\",0.0],PARAMETER[\\\"Central_Meridian\\\",3.0],PARAMETER[\\\"Scale_Factor\\\",0.9996],PARAMETER[\\\"Latitude_Of_Origin\\\",0.0],UNIT[\\\"Meter\\\",1.0],AUTHORITY[\\\"EPSG\\\",32631]]\",\"ver\":\"PE_10_3_1\",\"name\":\"WGS_1984_UTM_Zone_31N\",\"authCode\":{\"auth\":\"EPSG\",\"code\":\"32631\"},\"type\":\"LBC\"}"A geographic CRS
"name": "NAD27 * OGP-Usa Conus [4267,15851]"
"persistableReference": "{\"lateBoundCRS\":{\"wkt\":\"GEOGCS[\\\"GCS_North_American_1927\\\",DATUM[\\\"D_North_American_1927\\\",SPHEROID[\\\"Clarke_1866\\\",6378206.4,294.9786982]],PRIMEM[\\\"Greenwich\\\",0.0],UNIT[\\\"Degree\\\",0.0174532925199433],AUTHORITY[\\\"EPSG\\\",4267]]\",\"ver\":\"PE_10_3_1\",\"name\":\"GCS_North_American_1927\",\"authCode\":{\"auth\":\"EPSG\",\"code\":\"4267\"},\"type\":\"LBC\"},\"singleCT\":{\"wkt\":\"GEOGTRAN[\\\"NAD_1927_To_WGS_1984_79_CONUS\\\",GEOGCS[\\\"GCS_North_American_1927\\\",DATUM[\\\"D_North_American_1927\\\",SPHEROID[\\\"Clarke_1866\\\",6378206.4,294.9786982]],PRIMEM[\\\"Greenwich\\\",0.0],UNIT[\\\"Degree\\\",0.0174532925199433]],GEOGCS[\\\"GCS_WGS_1984\\\",DATUM[\\\"D_WGS_1984\\\",SPHEROID[\\\"WGS_1984\\\",6378137.0,298.257223563]],PRIMEM[\\\"Greenwich\\\",0.0],UNIT[\\\"Degree\\\",0.0174532925199433]],METHOD[\\\"NADCON\\\"],PARAMETER[\\\"Dataset_conus\\\",0.0],AUTHORITY[\\\"EPSG\\\",15851]]\",\"ver\":\"PE_10_3_1\",\"name\":\"NAD_1927_To_WGS_1984_79_CONUS\",\"authCode\":{\"auth\":\"EPSG\",\"code\":\"15851\"},\"type\":\"ST\"},\"ver\":\"PE_10_3_1\",\"name\":\"NAD27 * OGP-Usa Conus [4267,15851]\",\"authCode\":{\"auth\":\"SLB\",\"code\":\"4267079\"},\"type\":\"EBC\"}"WGS84
"name": "GCS_WGS_1984"
"persistableReference": "{\"wkt\":\"GEOGCS[\\\"GCS_WGS_1984\\\",DATUM[\\\"D_WGS_1984\\\",SPHEROID[\\\"WGS_1984\\\",6378137.0,298.257223563]],PRIMEM[\\\"Greenwich\\\",0.0],UNIT[\\\"Degree\\\",0.0174532925199433],AUTHORITY[\\\"EPSG\\\",4326]]\",\"ver\":\"PE_10_3_1\",\"name\":\"GCS_WGS_1984\",\"authCode\":{\"auth\":\"EPSG\",\"code\":\"4326\"},\"type\":\"LBC\"}"Unit example persistable reference strings
"name": "ftUS"
"persistableReference": "{\"scaleOffset\":{\"scale\":0.3048006096012192,\"offset\":0.0},\"symbol\":\"ftUS\",\"baseMeasurement\":{\"ancestry\":\"Length\",\"type\":\"UM\"},\"type\":\"USO\"}"
"name": "ft"
"persistableReference": "{\"scaleOffset\":{\"scale\":0.3048,\"offset\":0.0},\"symbol\":\"ft\",\"baseMeasurement\":{\"ancestry\":\"Length\",\"type\":\"UM\"},\"type\":\"USO\"}"
"name": "m"
"persistableReference": "{\"scaleOffset\":{\"scale\":1.0,\"offset\":0.0},\"symbol\":\"m\",\"baseMeasurement\":{\"ancestry\":\"Length\",\"type\":\"UM\"},\"type\":\"USO\"}"MetaItem Examples
Following are some examples of metaItems. Note: "persistable reference String" is not a valid value for persistable reference, only used for demonstration purpose.
- Everything is properly provided, need further check on data values to apply conversion:
{
"kind": "CRS",
"persistableReference": "persistable reference String",
"propertyNames": [
"X",
"Y",
"Z"
],
"name": "GCS_WGS_1984"
}- Missing value for kind, no conversion could be applied:
{
"kind": "",
"persistableReference": "persistable reference String",
"propertyNames": [
"X",
"Y",
"Z"
],
"name": "GCS_WGS_1984"
}- Missing value for persistable reference, no conversion could be applied:
{
"kind": "CRS",
"persistableReference": "",
"propertyNames": [
"X",
"Y",
"Z"
],
"name": "GCS_WGS_1984"
}- Missing value for property names, no conversion could be applied:
{
"kind": "CRS",
"persistableReference": "persistable reference String",
"propertyNames": [],
"name": "GCS_WGS_1984"
}How we use Meta Section
In one of our storage API(which will be introduced later in this documentation), we get meta section of records, extract FoR information and properties need to be converted, do the conversion and put back the converted value into record.
Storage Fetch Record API
As mentioned in the Introduction part, we need a way for the user to retrieve the data so that it is rationalized to common standard allowing for comparison from multiple data sources, we now have an API in Storage service to fetch multiple records at once, and it allows user to request data being converted to common standard by using custom header {slb-frame-of-reference}.
This API can fetch at most 20 records at once, and if conversion is requested by the user, meta section of each record would be checked, and each metaItem in the meta section would be looped over to get information of its kind, property names and persistableReference. If any error happens during the conversion, detailed error messages would be noted down and returned to user together with records the user is asking for.
What do we support
There are 5 types of attributes included in common standard: Unit, CRS, Dates, Elevation and Azimuth. As for now, we only support the conversion for Unit and CRS, Dates, Elevation and Azimuth will be available later.
If record itself is missing anything necessary for the conversion, for example, field "kind" in metaItem, or missing the properties which are listed in metaItem, we will provide explicit error messages indicating what are missing, what is the result(in these cases, no conversion applied) back to the user. Although data quality itself is out of this API's scope, with those explicit error messages, data owner knows what needs to be fixed before their data can be successfully converted.
How to use this API
To use this API, users need specify the {slb-data-partition-id} and {slb-frame-of-reference} in the header, Currently only "none" and "units=SI;crs=wgs84;elevation=msl;azimuth=true north;dates=utc;" are valid values for the header {slb-frame-of-reference}.
In addition, in request body, users should provide no more than 20 record ids to fetch record with or without the conversion.
Returned records could be either original value or converted(units=SI;crs=wgs84) value depending on users' requests and conversion status, original value will be returned when users not request the conversion or the conversion is requested but failed. In addition to records user requests, if conversion is requested, a list of conversion status of each record would be included in the response, indicating whether the conversion was successful or not, it not, what were the errors happened.
POST /api/storage/v2/query/records:batchcurl
curl --request POST \
--url '/api/storage/v2/query/records:batch' \
--header 'Authorization: Bearer <JWT>' \
--header 'Content-Type: application/json' \
--header 'Slb-Data-Partition-Id: common' \
--header 'slb-frame-of-reference: units=SI;crs=wgs84;elevation=msl;azimuth=true north;dates=utc;' \
--data '{
"records": [
"common:well:123456789",
"common:wellTop:abc789456",
"common:wellLog:4531wega22"
]
}Examples
Following are some examples you can refer to when using this API.
- Data all good, conversion applied, converted value would replace the original value, no error message in this case.
Original record:
{
"id": "common:test:fetchtest-1",
"kind": "common:test:fetchtest:1.0.0",
"acl": {
"viewers": [
"data.default.viewers@instance.osdu.opengroup.org"
],
"owners": [
"data.default.owners@instance.osdu.opengroup.org"
]
},
"legal": {
"legaltags": [
"common-test-fetchtest-1"
],
"otherRelevantDataCountries": [
"US"
]
},
"data": {
"msg": "testing record 1 for new fetch api",
"X": -61.04340628871454,
"Y": 10.673103179456877,
"Z": 0
},
"meta": [
{
"kind": "CRS",
"persistableReference": "persistable reference String",
"propertyNames": [
"X",
"Y",
"Z"
],
"name": "GCS_WGS_1984"
}
]
}- Missing MetaBlock in the record, if this record is requested, original record would be returned together with an error message indicating meta block is missing in the record.
Original record:
{
"id": "common:test:fetchtest-1",
"kind": "common:test:fetchtest:1.0.0",
"acl": {
"viewers": [
"data.default.viewers@instance.osdu.opengroup.org"
],
"owners": [
"data.default.owners@instance.osdu.opengroup.org"
]
},
"legal": {
"legaltags": [
"common-test-fetchtest-1"
],
"otherRelevantDataCountries": [
"US"
]
},
"data": {
"msg": "testing record 1 for new fetch api",
"X": -61.04340628871454,
"Y": 10.673103179456877,
"Z": 0
}
}- MetaItem is invalid(in this case, missing kind in metaItem) as described in MetaItem part, original record would be returned together with error message indicating something is invalid from metaItem.
Original record:
{
"id": "common:test:fetchtest-1",
"kind": "common:test:fetchtest:1.0.0",
"acl": {
"viewers": [
"data.default.viewers@instance.osdu.opengroup.org"
],
"owners": [
"data.default.owners@instance.osdu.opengroup.org"
]
},
"legal": {
"legaltags": [
"common-test-fetchtest-1"
],
"otherRelevantDataCountries": [
"US"
]
},
"data": {
"msg": "testing record 1 for new fetch api",
"X": -61.04340628871454,
"Y": 10.673103179456877,
"Z": 0
},
"meta": [
{
"kind": "",
"persistableReference": "persistable reference String",
"propertyNames": [
"X",
"Y",
"Z"
],
"name": "GCS_WGS_1984"
}
]
}- Missing data in dataBlock(in this case, missing property Y), if the properties listed in metaItem does not exist in dataBlcok of this record, original record would be returned together with error message indicating data is missing from datablock.
Original record:
{
"id": "common:test:fetchtest-1",
"kind": "common:test:fetchtest:1.0.0",
"acl": {
"viewers": [
"data.default.viewers@instance.osdu.opengroup.org"
],
"owners": [
"data.default.owners@instance.osdu.opengroup.org"
]
},
"legal": {
"legaltags": [
"common-test-fetchtest-1"
],
"otherRelevantDataCountries": [
"US"
]
},
"data": {
"msg": "testing record 1 for new fetch api",
"X": -61.04340628871454,
"Z": 0
},
"meta": [
{
"kind": "CRS",
"persistableReference": "persistable reference String",
"propertyNames": [
"X",
"Y",
"Z"
],
"name": "GCS_WGS_1984"
}
]
}There will be other errors happening with real data, for example, data value should be a double type but shows up as a string, the value of persistable reference is incorrect, etc. We will include as detailed error message as we can in response body so that users will have an idea why conversion cannot be applied and what to do to fix the data with data owners.
Scenarios we support
- Scenarios we support for CRS:
{
"kind": "CRS",
"persistableReference": "persistable reference String",
"propertyNames": [
"X",
"Y",
"Z"
],
"name": "NAD27 * OGP-Usa Conus / Louisiana South [26782,15851]"
}
{
"kind": "CRS",
"persistableReference": "persistable reference String",
"propertyNames": [
"X",
"Y"
],
"name": "NAD27 * OGP-Usa Conus / Louisiana South [26782,15851]"
}
{
"kind": "CRS",
"persistableReference": "persistable reference String",
"propertyNames": [
"LAT",
"LON"
],
"name": "NAD27 * OGP-Usa Conus [4267,15851]"
}
{
"kind": "CRS",
"persistableReference": "persistable reference String",
"propertyNames": [
"Longitude",
"Latitude"
],
"name": "NAD27 * OGP-Usa Conus [4267,15851]"
}
{
"kind": "CRS",
"persistableReference": "persistable reference String",
"propertyNames": [
"wlbEwUtm",
"wlbNsUtm"
],
"name": "WGS_1984_UTM_Zone_31N"
}
{
"kind": "CRS",
"persistableReference": "persistable reference String",
"propertyNames": [
"wlbEwDesDeg",
"wlbNsDecDeg"
],
"name": "GCS_WGS_1984"
}
{
"kind": "CRS",
"persistableReference": "persistable reference String",
"propertyNames": [
"TOPHOLEXNG",
"TOPHOLEYNG"
],
"name": "GCS_WGS_1984"
}
{
"kind": "CRS",
"persistableReference": "persistable reference String",
"propertyNames": [
"TOPHOLEXDD",
"TOPHOLEYDD"
],
"name": "GCS_WGS_1984"
}
{
"kind": "CRS",
"persistableReference": "persistable reference String",
"propertyNames": [
"BHLongitude",
"BHLatitude"
],
"name": "GCS_WGS_1984"
}
{
"kind": "CRS",
"persistableReference": "persistable reference String",
"propertyNames": [
"NativeGeographicCrs"
],
"name": "NAD27 * OGP-Usa Conus [4267,15851]"
}
{
"kind": "CRS",
"persistableReference": "persistable reference String",
"propertyNames": [
"NativeProjectedCrs"
],
"name": "NAD27 * OGP-Usa Conus / Louisiana South [26782,15851]"
}Error Messages And Why
Error messages are made up of 3 parts:
- Which part of conversion is throwing the error: CRS conversion or Unit conversion for now, there will be Dates conversion soon.
- The exact reason why the error is thrown: most possible reason are some value are missing or invalid, those part could be as detailed as possible, and please let me know if you feel there is anything we can do to help you understand the error message better, we can have a further discussion on that.
- The Result of this error: For now, it’s always no conversion applied. But as the enhancement kicks in in the Unit conversion part(skip the empty property), we will likely to have more.
| Error Meesage Contents | WHen & Why would be thrown |
|---|---|
| CRS/Unit Conversion: DataBlock is missing or empty in this record, no conversion applied. | The record is missing its data part, basically we can’t get any data from the record |
| CRS/Unit conversion: Required property 'kind' in meta block is missing or empty, no conversion applied. | The field ‘kind’ in the metaItem is missing, usually it points out the type of the conversion: CRS/Unit/DateTime |
| CRS/Unit conversion: 'propertyNames' in meta block is missing or empty, no conversion applied | The field ‘propertyNames’ in the metaItem is missing, usually it lists all the properties which share the same persistableReference in dataBlock, without this, we don’t know which data properties we need to convert |
| CRS conversion: Inappropriate number of properties for point conversion, unsupported property set, no conversion applied. | For CRS conversion, we need exactly 1 nested data property(which contains a list of Points) or 2-3 properties(lon, lat, elevation) to construct the structure called Point which is needed for crs conversion. If more than 3 properties is listed in the field ‘propertyNames’, we can’t support the crs conversion as we can’t tell the mapping between those 4 or more properties with lon, lat and elevation in Point. |
| CRS/Unit conversion: 'propertyNames' illegal, no conversion applied. | This is an error for bad format in record, the field ‘propertyNames’ needs to be an JsonArray, otherwise there will be IllegalStatteException thrown when we try to get it from the record itself. |
| CRS/Unit conversion: 'persistableReference' missing, no conversion applied. | The field ‘ persistableReference’ in the metaItem is missing, without knowing this and passing it to the conversion service, no conversion can be applied. |
| CRS/Unit conversion: property '%s' is missing in datablock, (no conversion applied)/(skip). | In CRS conversion, no property listed in the ‘propertyNames’ field in metaItem can be empty or non-existing in data block of the record, as we need all of them to construct the structure Point which is needed in crs conversion. So any non-existence or empty value will cause this error to be thrown. For Unit conversion, we reject the conversion for now, but will support the function to skip empty properties with messages provided but continue with the conversion of other properties very soon. |
| CRS/Unit conversion: cannot cast the value of property '%s' to double, error message: %s, no conversion applied. | Usually, properties listed for CRS/Unit conversion is double type, and if the value is not double type, there will be ClassCastException thrown out, we put the property’s name and the error message we got from the exception in our error message so that users understand which property need a fix before any conversion can be applied. |
| CRS/Unit conversion: illegal value for property '%s', error message: %s, no conversion applied | Similar to above one, when the value of the property is not double(in this case, more likely it’s a String type), when we get a NumberFormatException or IllegalStateException |
| CRS conversion: required properties for point conversion not sufficient, no conversion applied. | Sometimes, the properties listed in the metaItem all represent longitude or latitude so we cannot construct the Point even if there are 2 or 3 properties provided. This is the final check that before sending the data to crs converter to make sure the conversion can be applied. This error means either latitude or longitude of the point is missing. |
| CRS conversion: invalid nested property name: '%s', no conversion applied. | Currently only a fixed list of property names are supported. The conversion must be able to understand which properties represent latitude/longitude/easting/northing. For suppoerted property names, please see Supported Property Names section. |
| CRS conversion: missing the property 'points' in nested property, no conversion applied. | If a nested property is listed in the metaItem, then it needs to contain a field named ‘points’ which holds a list of points. If this field is missing or empty, there is nothing we can convert. |
| CRS conversion: bad request from crs converter, illegal persistable reference, no conversion applied. | We call CRS converter to do the CRS conversion, if CRS converter is returning 400 bad request, it means the json format of the persistable reference is not correct. |
| Response from CRS converter: %s | Other than 400, sometimes we see other 5xx response from CRS converter, and for these cases, we can’t handle from this API. We would give the user whatever response we have from the CRS converter, and let them decide whether to ignore this case or retry or contact for further information. |
| CRS conversion: illegal value in nested property '%s', error message: %s, no conversion applied. | The field ‘points’ in nested property needs to be a JsonArray, otherwise a ClassCastException or IllegalStateException would be thrown when we try to get it. If we see these exceptions, we put the property’s name, and the error message from exception into our error message, so that users understand how to fix it. |
Suported Property Names
Supported property names for Longitude:
| Property Name |
|---|
| X |
| LON |
| Longitude |
| wlbEwUtm |
| wlbEwDesDeg |
| TOPHOLEXNG |
| TOPHOLEXDD |
| BHLongitude |
| Utm_X |
Supported property names for Latitude:
| Property Name |
|---|
| Y |
| LAT |
| Latitude |
| wlbNsUtm |
| wlbNsDecDeg |
| TOPHOLEYNG |
| TOPHOLEYDD |
| BHLatitude |
| Utm_Y |
Supported nested property names:
| Property Name |
|---|
| projectOutlineLocalGeographic |
| projectOutlineProjected |