Transformer: Enrich Field Flattening to broaden the depth and clarity of data ingestion
Enrich the current logic for Field/Attribute Flattening to accommodate varied presentations of data and provide context for where the data was originally retrieved.
Examples:
1. Arrays of objects, with each object having identical properties:
2. Capture context of object property when flattening
Example JSON
{
"VerticalMeasurements": [
{
"WellboreTVDTrajectoryID": null,
"VerticalCRSID": null,
"VerticalReferenceID": null,
"VerticalMeasurementSourceID": null,
"VerticalMeasurementID": "KB",
"VerticalMeasurementPathID": "opendes:reference-data--VerticalMeasurementPath:ELEV:",
"VerticalMeasurement": 120.92,
"VerticalMeasurementTypeID": "opendes:reference-data--VerticalMeasurementType:KB:",
"VerticalMeasurementDescription": null,
"VerticalMeasurementUnitOfMeasureID": "opendes:reference-data--UnitOfMeasure:m:",
"VerticalReferenceEntityID": null
},
{
"WellboreTVDTrajectoryID": null,
"VerticalCRSID": null,
"VerticalReferenceID": null,
"VerticalMeasurementSourceID": null,
"VerticalMeasurementID": "Total Depth MD",
"VerticalMeasurementPathID": "opendes:reference-data--VerticalMeasurementPath:MD:",
"VerticalMeasurement": 2431,
"VerticalMeasurementTypeID": null,
"VerticalMeasurementDescription": null,
"VerticalMeasurementUnitOfMeasureID": "opendes:reference-data--UnitOfMeasure:m:",
"VerticalReferenceEntityID": null
},
{
"WellboreTVDTrajectoryID": null,
"VerticalCRSID": null,
"VerticalReferenceID": null,
"VerticalMeasurementSourceID": null,
"VerticalMeasurementID": "Well Head Elevation",
"VerticalMeasurementPathID": "opendes:reference-data--VerticalMeasurementPath:ELEV:",
"VerticalMeasurement": 120.92,
"VerticalMeasurementTypeID": null,
"VerticalMeasurementDescription": null,
"VerticalMeasurementUnitOfMeasureID": "opendes:reference-data--UnitOfMeasure:m:",
"VerticalReferenceEntityID": null
}
]
}
Proposed Solution:
- Treat first element in array like so:
VerticalMeasurements_1_WellboreTVDTrajectoryID
VerticalMeasurements_1_VerticalCRSID
- ...
- Treat subsequent elements in array like so:
VerticalMeasurements_2_WellboreTVDTrajectoryID
VerticalMeasurements_2_VerticalCRSID
- ...
3. Determine whether Legal tags should be ingested and exposed alongside other metadata
4. Refactor Trajectory logic's StationProperty parsing, after steps 1/2 have been implemented into GCZ's ingestion.
Edited by Levi Remington