Manifest-based Ingestion - add persistable reference information by using content from Reference data

Current approach - specify persistable reference inside load manifest (JSON file). We propose that Manifest Ingestion process can get the information from Reference data and add that row "on the fly" just before using Storage Service to create new record. This way, Data Loader does not need to worry about adding this information by himself or herself.

cc - @Keith_Wall , @ChrisZhang , @epeysson , @Kateryna_Kurach, @chad , @blasscoc (for information)

            "meta": [
                {
                    "kind": "Unit",
                    "name": "ms",
                    "persistableReference": "{\"abcd\":{\"a\":0.0,\"b\":0.001,\"c\":1.0,\"d\":0.0},\"symbol\":\"ms\",\"baseMeasurement\":{\"ancestry\":\"T\",\"type\":\"UM\"},\"type\":\"UAD\"}",
                    "unitOfMeasureID": "{{data-partition-id}}:reference-data--UnitOfMeasure:ms:",
                    "propertyNames": [
                        "RecordLength"
                    ]
                },

For example, see this retrieval query. Body

{
    "kind": "{{data-partition-id}}:wks:reference-data--UnitOfMeasure:1.0.0",
    "limit":5000,
    "query": "id: \"{{data-partition-id}}:reference-data--UnitOfMeasure:ms\"",
    "returnedFields": ["*"]
}

Response

{
    "results": [
        {
            "data": {
                "AttributionPublication": "Energistics Unit of Measure Dictionary V1.0",
                "PersistableReference": "{\"abcd\":{\"a\":0.0,\"b\":0.001,\"c\":1.0,\"d\":0.0},\"symbol\":\"ms\",\"baseMeasurement\":{\"ancestry\":\"T\",\"type\":\"UM\"},\"type\":\"UAD\"}",
                "InactiveIndicator": false,
                "UnitDimensionCode": "T",
                "UnitQuantityID": "odesprod:reference-data--UnitQuantity:T:",
                "Code": "ms",
                "Source": "Workbook Published/UnitOfMeasure.1.0.0.xlsx; commit SHA c1d72417.",
                "Name": "millisecond",
                "AttributionAuthority": "Energistics",
                "IsBaseUnit": false,
                "AttributionRevision": "1.0",
                "ID": "ms",
                "CoefficientC": 1.0,
                "UnitDimensionName": "time",
                "CoefficientD": 0.0,
                "CoefficientA": 0.0,
                "CoefficientB": 0.001
            },
            "kind": "odesprod:wks:reference-data--UnitOfMeasure:1.0.0",
            "source": "wks",
            "acl": {
                "viewers": [
                    "data.default.owners@odesprod.osdu-gcp.go3-nrg.projects.epam.com"
                ],
                "owners": [
                    "data.default.owners@odesprod.osdu-gcp.go3-nrg.projects.epam.com"
                ]
            },
            "type": "reference-data--UnitOfMeasure",
            "version": 1617286097841123,
            "createTime": "2021-04-01T14:08:17.885Z",
            "authority": "odesprod",
            "namespace": "odesprod:wks",
            "legal": {
                "legaltags": [
                    "odesprod-demo-legaltag"
                ],
                "otherRelevantDataCountries": [
                    "US"
                ],
                "status": "compliant"
            },
            "createUser": "osdu-sa-airflow-composer@osdu-service-prod.iam.gserviceaccount.com",
            "id": "odesprod:reference-data--UnitOfMeasure:ms"
        }
    ],
    "totalCount": 1
}

Enclosed is end-to-end example of Unit Conversion (Frame of Reference) in GCP, R3M8 by using manifest-based Ingestion process.

OSDU_PTP_M8_TeamA_GCP-Manifest-Ingestion-unit-convert-Debasis-Naufal.txt