Skip to content

ADR - Change of CRS Catalog and Convert Services to dynamically read CRS/CT reference data using Record Id

We are often left to address the gaps from architectural principles (which stay at a pretty high and abstract level) to the actual implementation detail. Here is an attempt to bridge that gap by providing a set of Lightweight Architecture Decision Records (LADRs) which are simple to follow and can be implemented in a given team/project by the developers

Decision Title

Dynamic CRS

Status

  • Proposed
  • Trialing
  • Under review
  • Approved
  • Retired

Context & Scope

This ADR is a continuation of a previous one deprecating CRS Catalog: osdu/platform/system/reference/crs-catalog-service#22 (closed)

Problems with the current CRS services:

  • CRS Catalog uses a hardcoded catalog of CRS and CT data
  • Applications using CRS conversion have to construct complex escaped json for request body, likely storing CRS and CT data outside of OSDU

This proposal centers around the idea of leveraging manifests for CRS and CT data. CRS conversion service would change to pull CRS and CT data from Storage service while CRS catalog service would change to interact with Search service as a domain helper.

Example reference data can be found here: https://community.opengroup.org/osdu/data/data-definitions/-/blob/master/E-R/reference-data/CoordinateReferenceSystem.1.0.0.md And here: https://community.opengroup.org/osdu/data/data-definitions/-/blob/master/E-R/reference-data/CoordinateTransformation.1.0.0.md Actual populated reference data can be found here: https://community.opengroup.org/osdu/data/data-definitions/-/tree/master/ReferenceValues/Manifests/reference-data/LOCAL

High level design change:

Screen_Shot_2022-04-05_at_9.34.41_AM

CRS Conversion Service Changes

For CRS Conversion service, we would change any request body against CRS conversion service that contains long, stringified json to use reference data record ids that contain this information instead. Persistable reference data (the long stringified json shown below) will still work as a means to provide backwards compatibility for workloads such as storage service’s normalization.

This would be a new versioned endpoint (v3)

CRS conversion service would look up the record using Storage service, afterwards using the needed information to do conversions.

Here is an example of a current request body for the point conversion API:

{
  "fromCRS": "{\"lateBoundCRS\":{\"wkt\":\"GEOGCS[\\\"GCS_Provisional_S_American_1956\\\",DATUM[\\\"D_Provisional_S_American_1956\\\",SPHEROID[\\\"International_1924\\\",6378388.0,297.0]],PRIMEM[\\\"Greenwich\\\",0.0],UNIT[\\\"Degree\\\",0.0174532925199433],AUTHORITY[\\\"EPSG\\\",4248]]\",\"ver\":\"PE_10_3_1\",\"name\":\"GCS_Provisional_S_American_1956\",\"authCode\":{\"auth\":\"EPSG\",\"code\":\"4248\"},\"type\":\"LBC\"},\"singleCT\":{\"wkt\":\"GEOGTRAN[\\\"PSAD_1956_To_WGS_1984_9\\\",GEOGCS[\\\"GCS_Provisional_S_American_1956\\\",DATUM[\\\"D_Provisional_S_American_1956\\\",SPHEROID[\\\"International_1924\\\",6378388.0,297.0]],PRIMEM[\\\"Greenwich\\\",0.0],UNIT[\\\"Degree\\\",0.0174532925199433]],GEOGCS[\\\"GCS_WGS_1984\\\",DATUM[\\\"D_WGS_1984\\\",SPHEROID[\\\"WGS_1984\\\",6378137.0,298.257223563]],PRIMEM[\\\"Greenwich\\\",0.0],UNIT[\\\"Degree\\\",0.0174532925199433]],METHOD[\\\"Geocentric_Translation\\\"],PARAMETER[\\\"X_Axis_Translation\\\",-295.0],PARAMETER[\\\"Y_Axis_Translation\\\",173.0],PARAMETER[\\\"Z_Axis_Translation\\\",-371.0],AUTHORITY[\\\"EPSG\\\",1209]]\",\"ver\":\"PE_10_3_1\",\"name\":\"PSAD_1956_To_WGS_1984_9\",\"authCode\":{\"auth\":\"EPSG\",\"code\":\"1209\"},\"type\":\"ST\"},\"ver\":\"PE_10_3_1\",\"name\":\"PSAD56 * DMA-Ven [4248,1209]\",\"authCode\":{\"auth\":\"SLB\",\"code\":\"4248009\"},\"type\":\"EBC\"}",
  "toCRS": "{\"lateBoundCRS\":{\"wkt\":\"PROJCS[\\\"Trinidad_1903_Trinidad_Grid\\\",GEOGCS[\\\"GCS_Trinidad_1903\\\",DATUM[\\\"D_Trinidad_1903\\\",SPHEROID[\\\"Clarke_1858\\\",6378293.64520876,294.260676369]],PRIMEM[\\\"Greenwich\\\",0.0],UNIT[\\\"Degree\\\",0.0174532925199433]],PROJECTION[\\\"Cassini\\\"],PARAMETER[\\\"False_Easting\\\",430000.0],PARAMETER[\\\"False_Northing\\\",325000.0],PARAMETER[\\\"Central_Meridian\\\",-61.3333333333333],PARAMETER[\\\"Scale_Factor\\\",1.0],PARAMETER[\\\"Latitude_Of_Origin\\\",10.4416666666667],UNIT[\\\"Link_Clarke\\\",0.201166195164],AUTHORITY[\\\"EPSG\\\",30200]]\",\"ver\":\"PE_10_3_1\",\"name\":\"Trinidad_1903_Trinidad_Grid\",\"authCode\":{\"auth\":\"EPSG\",\"code\":\"30200\"},\"type\":\"LBC\"},\"singleCT\":{\"wkt\":\"GEOGTRAN[\\\"Trinidad_1903_To_WGS_1984_2\\\",GEOGCS[\\\"GCS_Trinidad_1903\\\",DATUM[\\\"D_Trinidad_1903\\\",SPHEROID[\\\"Clarke_1858\\\",6378293.64520876,294.260676369]],PRIMEM[\\\"Greenwich\\\",0.0],UNIT[\\\"Degree\\\",0.0174532925199433]],GEOGCS[\\\"GCS_WGS_1984\\\",DATUM[\\\"D_WGS_1984\\\",SPHEROID[\\\"WGS_1984\\\",6378137.0,298.257223563]],PRIMEM[\\\"Greenwich\\\",0.0],UNIT[\\\"Degree\\\",0.0174532925199433]],METHOD[\\\"Geocentric_Translation\\\"],PARAMETER[\\\"X_Axis_Translation\\\",-61.0],PARAMETER[\\\"Y_Axis_Translation\\\",285.2],PARAMETER[\\\"Z_Axis_Translation\\\",471.6],AUTHORITY[\\\"EPSG\\\",10085]]\",\"ver\":\"PE_10_3_1\",\"name\":\"Trinidad_1903_To_WGS_1984_2\",\"authCode\":{\"auth\":\"EPSG\",\"code\":\"10085\"},\"type\":\"ST\"},\"ver\":\"PE_10_3_1\",\"name\":\"Trinidad 1903 * EOG-Tto Trin / Trinidad Grid [30200,10085]\",\"authCode\":{\"auth\":\"SLB\",\"code\":\"30200002\"},\"type\":\"EBC\"}",
  "points": [
    {
      "x": -61.04340628871454,
      "y": 10.673103179456877,
      "z": 0
    }
  ]
}

Here is the new request body for point conversion API (but still accepting old way for same attribute names):

{
  "fromCRS": "osdu:reference-data--CoordinateReferenceSystem:BoundGeographic2D:EPSG::4168_EPSG::1569",
  "toCRS": "osdu:reference-data--CoordinateReferenceSystem:BoundGeographic2D:EPSG::4201_EPSG::1100",
  "points": [
    {
      "x": -61.04340628871454,
      "y": 10.673103179456877,
      "z": 0
    }
  ]
}

Similarly, here is the GeoJson Conversion API spec as it is currently:

{
  "toCRS": "{\"lateBoundCRS\":{\"wkt\":\"PROJCS[\\\"Trinidad_1903_Trinidad_Grid\\\",GEOGCS[\\\"GCS_Trinidad_1903\\\",DATUM[\\\"D_Trinidad_1903\\\",SPHEROID[\\\"Clarke_1858\\\",6378293.64520876,294.260676369]],PRIMEM[\\\"Greenwich\\\",0.0],UNIT[\\\"Degree\\\",0.0174532925199433]],PROJECTION[\\\"Cassini\\\"],PARAMETER[\\\"False_Easting\\\",430000.0],PARAMETER[\\\"False_Northing\\\",325000.0],PARAMETER[\\\"Central_Meridian\\\",-61.3333333333333],PARAMETER[\\\"Scale_Factor\\\",1.0],PARAMETER[\\\"Latitude_Of_Origin\\\",10.4416666666667],UNIT[\\\"Link_Clarke\\\",0.201166195164],AUTHORITY[\\\"EPSG\\\",30200]]\",\"ver\":\"PE_10_3_1\",\"name\":\"Trinidad_1903_Trinidad_Grid\",\"authCode\":{\"auth\":\"EPSG\",\"code\":\"30200\"},\"type\":\"LBC\"},\"singleCT\":{\"wkt\":\"GEOGTRAN[\\\"Trinidad_1903_To_WGS_1984_2\\\",GEOGCS[\\\"GCS_Trinidad_1903\\\",DATUM[\\\"D_Trinidad_1903\\\",SPHEROID[\\\"Clarke_1858\\\",6378293.64520876,294.260676369]],PRIMEM[\\\"Greenwich\\\",0.0],UNIT[\\\"Degree\\\",0.0174532925199433]],GEOGCS[\\\"GCS_WGS_1984\\\",DATUM[\\\"D_WGS_1984\\\",SPHEROID[\\\"WGS_1984\\\",6378137.0,298.257223563]],PRIMEM[\\\"Greenwich\\\",0.0],UNIT[\\\"Degree\\\",0.0174532925199433]],METHOD[\\\"Geocentric_Translation\\\"],PARAMETER[\\\"X_Axis_Translation\\\",-61.0],PARAMETER[\\\"Y_Axis_Translation\\\",285.2],PARAMETER[\\\"Z_Axis_Translation\\\",471.6],AUTHORITY[\\\"EPSG\\\",10085]]\",\"ver\":\"PE_10_3_1\",\"name\":\"Trinidad_1903_To_WGS_1984_2\",\"authCode\":{\"auth\":\"EPSG\",\"code\":\"10085\"},\"type\":\"ST\"},\"ver\":\"PE_10_3_1\",\"name\":\"Trinidad 1903 * EOG-Tto Trin / Trinidad Grid [30200,10085]\",\"authCode\":{\"auth\":\"SLB\",\"code\":\"30200002\"},\"type\":\"EBC\"}",
  "toUnitZ": "{\"scaleOffset\":{\"scale\":1.0,\"offset\":0.0},\"symbol\":\"m\",\"baseMeasurement\":{\"ancestry\":\"Length\",\"type\":\"UM\"},\"type\":\"USO\"}",
  "featureCollection": {
    "features": [
      {
        "geometry": {
          "coordinates": [
            313405.9477893702,
            6544797.620047403,
            6.561679790026246
          ],
          "bbox": null,
          "type": "AnyCrsPoint"
        },
        "bbox": null,
        "properties": {},
        "type": "AnyCrsFeature"
      }
    ],
    "bbox": null,
    "properties": {},
    "persistableReferenceCrs": "{\"lateBoundCRS\":{\"wkt\":\"PROJCS[\\\"ED_1950_UTM_Zone_32N\\\",GEOGCS[\\\"GCS_European_1950\\\",DATUM[\\\"D_European_1950\\\",SPHEROID[\\\"International_1924\\\",6378388.0,297.0]],PRIMEM[\\\"Greenwich\\\",0.0],UNIT[\\\"Degree\\\",0.0174532925199433]],PROJECTION[\\\"Transverse_Mercator\\\"],PARAMETER[\\\"False_Easting\\\",500000.0],PARAMETER[\\\"False_Northing\\\",0.0],PARAMETER[\\\"Central_Meridian\\\",9.0],PARAMETER[\\\"Scale_Factor\\\",0.9996],PARAMETER[\\\"Latitude_Of_Origin\\\",0.0],UNIT[\\\"Meter\\\",1.0],AUTHORITY[\\\"EPSG\\\",23032]]\",\"ver\":\"PE_10_3_1\",\"name\":\"ED_1950_UTM_Zone_32N\",\"authCode\":{\"auth\":\"EPSG\",\"code\":\"23032\"},\"type\":\"LBC\"},\"singleCT\":{\"wkt\":\"GEOGTRAN[\\\"ED_1950_To_WGS_1984_23\\\",GEOGCS[\\\"GCS_European_1950\\\",DATUM[\\\"D_European_1950\\\",SPHEROID[\\\"International_1924\\\",6378388.0,297.0]],PRIMEM[\\\"Greenwich\\\",0.0],UNIT[\\\"Degree\\\",0.0174532925199433]],GEOGCS[\\\"GCS_WGS_1984\\\",DATUM[\\\"D_WGS_1984\\\",SPHEROID[\\\"WGS_1984\\\",6378137.0,298.257223563]],PRIMEM[\\\"Greenwich\\\",0.0],UNIT[\\\"Degree\\\",0.0174532925199433]],METHOD[\\\"Position_Vector\\\"],PARAMETER[\\\"X_Axis_Translation\\\",-116.641],PARAMETER[\\\"Y_Axis_Translation\\\",-56.931],PARAMETER[\\\"Z_Axis_Translation\\\",-110.559],PARAMETER[\\\"X_Axis_Rotation\\\",0.893],PARAMETER[\\\"Y_Axis_Rotation\\\",0.921],PARAMETER[\\\"Z_Axis_Rotation\\\",-0.917],PARAMETER[\\\"Scale_Difference\\\",-3.52],AUTHORITY[\\\"EPSG\\\",1612]]\",\"ver\":\"PE_10_3_1\",\"name\":\"ED_1950_To_WGS_1984_23\",\"authCode\":{\"auth\":\"EPSG\",\"code\":\"1612\"},\"type\":\"ST\"},\"ver\":\"PE_10_3_1\",\"name\":\"ED50 * EPSG-Nor N62 2001 / UTM zone 32N [23032,1612]\",\"authCode\":{\"auth\":\"SLB\",\"code\":\"23032023\"},\"type\":\"EBC\"}",
    "persistableReferenceUnitZ": "{\"scaleOffset\":{\"scale\":1.0,\"offset\":0.0},\"symbol\":\"m\",\"baseMeasurement\":{\"ancestry\":\"Length\",\"type\":\"UM\"},\"type\":\"USO\"}",
    "type": "AnyCrsFeatureCollection"
  }
}

And this is what it would become:

{
  "toCRS": "osdu:reference-data--CoordinateReferenceSystem:BoundGeographic2D:EPSG::4201_EPSG::1100",
  "featureCollection": {
    "features": [
      {
        "geometry": {
          "coordinates": [
            313405.9477893702,
            6544797.620047403,
            6.561679790026246
          ],
          "bbox": null,
          "type": "AnyCrsPoint"
        },
        "bbox": null,
        "properties": {},
        "type": "AnyCrsFeature"
      }
    ],
    "bbox": null,
    "properties": {},
    "CoordinateReferenceSystemID": "osdu:reference-data--CoordinateReferenceSystem:BoundGeographic2D:EPSG::4168_EPSG::1569",
    "persistableReferenceUnitZ": "{\"scaleOffset\":{\"scale\":1.0,\"offset\":0.0},\"symbol\":\"m\",\"baseMeasurement\":{\"ancestry\":\"Length\",\"type\":\"UM\"},\"type\":\"USO\"}",
    "VerticalUnitID": "osdu:reference-data--UnitOfMeasure:m:",
    "type": "AnyCrsFeatureCollection"
  }
}

Again, with the trajectory conversion API it originally looks like this:

{
  "azimuthReference": "TN",
  "interpolate": true,
  "referencePoint": {
    "y": 6500000,
    "x": 400000,
    "z": 0
  },
  "unitZ": "{\"scaleOffset\":{\"scale\":1.0,\"offset\":0.0},\"symbol\":\"m\",\"baseMeasurement\":{\"ancestry\":\"Length\",\"type\":\"UM\"},\"type\":\"USO\"}",
  "inputStations": [
    {
      "md": 1000,
      "azimuth": 100,
      "inclination": 10
    },
    {
      "md": 2000,
      "azimuth": 200,
      "inclination": 200
    }
  ],
  "trajectoryCRS": "{\"wkt\":\"PROJCS[\\\"WGS_1984_UTM_Zone_31N\\\",GEOGCS[\\\"GCS_WGS_1984\\\",DATUM[\\\"D_WGS_1984\\\",SPHEROID[\\\"WGS_1984\\\",6378137.0,298.257223563]],PRIMEM[\\\"Greenwich\\\",0.0],UNIT[\\\"Degree\\\",0.0174532925199433]],PROJECTION[\\\"Transverse_Mercator\\\"],PARAMETER[\\\"False_Easting\\\",500000.0],PARAMETER[\\\"False_Northing\\\",0.0],PARAMETER[\\\"Central_Meridian\\\",3.0],PARAMETER[\\\"Scale_Factor\\\",0.9996],PARAMETER[\\\"Latitude_Of_Origin\\\",0.0],UNIT[\\\"Meter\\\",1.0],AUTHORITY[\\\"EPSG\\\",32631]]\",\"ver\":\"PE_10_3_1\",\"name\":\"WGS_1984_UTM_Zone_31N\",\"authCode\":{\"auth\":\"EPSG\",\"code\":\"32631\"},\"type\":\"LBC\"}",
  "inputKind": "MD_Incl_Azim",
  "unitXY": "{\"scaleOffset\":{\"scale\":1.0,\"offset\":0.0},\"symbol\":\"m\",\"baseMeasurement\":{\"ancestry\":\"Length\",\"type\":\"UM\"},\"type\":\"USO\"}",
  "method": "AzimuthalEquidistant"
}

And it would change to this

{
  "azimuthReference": "TN",
  "interpolate": true,
  "referencePoint": {
    "y": 6500000,
    "x": 400000,
    "z": 0
  },
  "unitZ": "(record id for unit reference data)",
  "inputStations": [
    {
      "md": 1000,
      "azimuth": 100,
      "inclination": 10
    },
    {
      "md": 2000,
      "azimuth": 200,
      "inclination": 200
    }
  ],
  "trajectoryCRS": "osdu:reference-data--CoordinateReferenceSystem:Projected:EPSG::32631",
  "inputKind": "MD_Incl_Azim",
  "unitXY": "(record id for unit reference data)",
  "method": "AzimuthalEquidistant"
}

Note the change above also with unit fields being referenced by record ids as well.

Appropriate server-side caching will be implemented with configured TTL and a memory limit to help improve performance and reduce load on Storage service. This cache will be based on record-id.

New CRS Catalog Service Endpoints

Since the CRS data will be stored as manifests via Storage service, there isn’t a need for the current CRS Catalog Service as it exists other than to be there for backwards compatibility. New endpoints will be introduced as “domain helpers” where they will interact with Search service, simplifying common use cases. 5 endpoints are to be added:

  • Get /CoordinateTransformation-by-id: Get one Coordinate Transformation (CT). Retrieve metadata for a Coordinate Transformation (CT) specified by a “record id” or code. CT may be simple, concatenated, or vertical.
  • Get /CoordinateTransformation: Get a list of Coordinate Transformations that satisfy search criteria. Returns a list of CTs with limited metadata such as CT name, CodeSpace, Code, Method, SourceCRS, targetCRS, and Extent Name. Use /CoordinateTransformation-by-id to get more properties. Search by keywords and/or location. CTs are assumed to be reversible. Hence the data are searched for both SourceCRS and TargetCRS if a CRS is given as search parameter. Returns totalCount.
  • Get /CoordinateReferenceSystem-by-id: Get one Coordinate Reference System (CRS). Retrieve metadata for a Coordinate Reference System (CRS) specified by a “record id” or code. CRS may be of Kind Geographic2D, Projected, BoundGeographic2D, BoundProjected, or Vertical.
  • Get /CoordinateReferenceSystem: Get a list of Coordinate Reference Systems that satisfy search criteria. Returns a list of CRSs with limited metadata such as CRS name, CodeSpace, Code, HorizontalAxisUnitID, VerticalAxisUnitID, and Extent Name. Use /CoordinateReferenceSystem-by-id to get more properties. Search by keywords and/or location. Returns totalCount.
  • Post /check-points-inside-aou: Check if one or more points are inside the Extent BBOX of a CRS or CT. Return approximate distance in km if a point is outside the entity Extent BBOX, 0 otherwise. Normal usage is to provide a single point, or the lower left and upper right corner of a data extent (i.e., the min and max latitude and longitude of a dataset).

Descriptions of each endpoint were written by Bert Kampes from the Geomatics team.

Naming for endpoints is not final but will end up conforming to normal REST specifications.

IMPORTANT NOTE affecting Search service: New CRS catalog functionality requires that Search service spatial filter supports intersectional queries i.e. passing coordinates making a polygon and getting documents with polygons that intersect with that polygon. This is the proposed new Spatial filter request for intersectional queries:

{
    "kind": "osdu:wks:reference-data--CoordinateTransformation:1.1.0",
    "query": "data.ID:\"EPSG::1078\"",
    "spatialFilter": {
        "field": "data.SpatialLocation.Wgs84Coordinates",
        "byIntersection": {
            "polygons": [
                {
                    "points": [
                        {
                            "latitude": 10.75, 
                            "longitude": -8.61
                        }
                    ]
                }
            ]
        }
    }
}

Also, new CRS catalog functionality requires that Search service spatial filter supports searching against documents' polygons to see if point(s) reside within the polygon on the document. This is another proposed new Spatial filter request for checking if points are within a polygon on the indexed documents:

{
    "kind": "osdu:wks:reference-data--CoordinateTransformation:1.1.0",
    "query": "data.ID:\"EPSG::1078\"",
    "spatialFilter": {
        "field": "data.SpatialLocation.Wgs84Coordinates",
        "byWithinPolygon": {
            "points": [
                        {
                            "latitude": 10.75, 
                            "longitude": -8.60
                        }
                    ]
        }
    },
    "limit": 1
}

Decision

  • Accept design change to leverage reference data records for CRS information in addition to working with CRS information as stringified JSON in the body.
  • Accept new endpoints for CRS conversion service
  • Accept new endpoints for CRS catalog service to interact with CRS and CT data via search service as a domain helper
  • Accept new spatial filter for querying by intersecting polygons.

Rationale

  • Enable better data management processes on CRS data
  • Clear up verbose request structure
  • Enable better integration of OSDU with CRS data for potential, more complex future workflows

Consequences

A new set of endpoints on v3 path will be introduced for CRS Conversion service. A new service will be introduced to help interact with CRS and CT reference data. All existing CRS endpoints and functionality for CRS Conversion service and CRS Catalog service will stay the same.

For CSPs to support new changes in both services they won't need to do anything other than deploy latest M12 code.

For CSPs to support the new Search service changes, new spatial filter code may need to be copied into each respective CSP provider folder in search service. This is because Search service leaves a lot of logic up to each CSP and as the AWS dev, I wouldn't want to change other CSP provider code. But it can be easily copied in if desired by other CSPs.

When to revisit

Design change needs to happen ASAP.

Tradeoff Analysis - Input to decision

See information posted in "Context & Scope". Essentially, the main tradeoff is that CRS conversion data will need to be stored as reference data before being used in CRS conversion workflows. This makes it a little more difficult to use directly as a user but much easier to use for applications long-term.

Alternatives and implications

Decision criteria and tradeoffs

Decision timeline

Decision ready to be made

Edited by Debasis Chatterjee
To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information