Rajesh Bollineni (35856ad7) at 18 Aug 06:24
Scheduling endpoints inclusion
#121 as part of incremental fetch implementation to accommodate LastSuccessfulRunDateUTC update on successful retrieving of data from external data source in order to retrieve next cycle data from external data sources based on LastSuccessfulRunDateUTC timestamp
Rajesh Bollineni (2cea42d3) at 13 Aug 04:30
Update src_dags_fetch_and_ingest.py
Rajesh Bollineni (011618e6) at 04 Aug 12:37
Rajesh Bollineni (011618e6) at 03 Aug 16:41
Update secretmanager.py
Rajesh Bollineni (8593ee84) at 16 Jul 11:46
Rajesh Bollineni (b7711d89) at 15 Jul 20:02
@Siarhei_Khaletski As it is short fix and will replace with long time fix soon. Will make sure comments are added. I see still changes not merged into master but approved by you. Could you please help in merging the changes being today last day for checking the code.
Please let me know in case of any other suggestions
{
'runId': '1626330063',
'executionContext': {
'acl': {
'owners': ['data.default.owners@osdu.example.com'],
'viewers': ['data.default.viewers@osdu.example.com']
},
'legal': {
'legaltags': ['osdu-demo-legaltag'],
'otherRelevantDataCountries': ['US']
},
'Payload': {
'AppKey': 'test-app',
'data-partition-id': 'osdu'
},
'manifest': {
'kind': 'osdu:wks:Manifest:1.0.0',
'Data': {
'WorkProductComponents': [{
'kind': 'osdu:wks:work-product-component--SeismicTraceData:1.0.0',
'id': 'osdu:work-product-component--SeismicTraceData:Katalyst-katalyst-19330323',
'version': 1625033500000,
'data': {
'ExtensionProperties': {
'kdmLoadDate': '2018-09-07T00:00:00.000Z'
},
'TraceLengthUOM': '',
'PrincipalAcquisitionProjectID': 'osdu:master-data--SeismicAcquisitionSurvey:katalyst:50004425138:',
'SeismicTraceDataDimensionalityTypeID': 'osdu:reference-data--SeismicTraceDataDimensionalityType:katalyst:3D:',
'Description': 'FIELD',
'TraceCount': 94250,
'TraceLength': 3288.0,
'SampleInterval': 2.0,
'TraceDomainUOM': '',
'SeismicDomainTypeID': 'osdu:reference-data--SeismicDomainType:katalyst:TIME:',
'Datasets': ['osdu:dataset--ConnectedSource.Generic:Katalyst-katalyst-19330323:']
},
'createUser': 'gerrick',
'createTime': '2015-10-07T13:59:15.000Z',
'modifyTime': '2021-06-30T01:11:40.000Z',
'modifyUser': 'kdm_demo_us',
'acl': {
'owners': ['data.default.owners@osdu.example.com'],
'viewers': ['data.default.viewers@osdu.example.com']
},
'legal': {
'legaltags': ['osdu-demo-legaltag'],
'otherRelevantDataCountries': ['US']
}
}, {
'kind': 'osdu:wks:work-product-component--SeismicTraceData:1.0.0',
'id': 'osdu:work-product-component--SeismicTraceData:Katalyst-katalyst-7334713',
'version': 1625033563000,
'data': {
'ExtensionProperties': {
'kdmLoadDate': '2018-09-07T00:00:00.000Z'
},
'TraceLengthUOM': 'osdu:reference-data--UnitOfMeasure:katalyst:ms:',
'PrincipalAcquisitionProjectID': 'osdu:master-data--SeismicAcquisitionSurvey:katalyst:50000594760:',
'SeismicTraceDataDimensionalityTypeID': 'osdu:reference-data--SeismicTraceDataDimensionalityType:katalyst:3D:',
'Description': 'STACK',
'TraceLength': 3000.0,
'SampleInterval': 2.0,
'TraceDomainUOM': 'osdu:reference-data--UnitOfMeasure:katalyst:ms:',
'SeismicDomainTypeID': 'osdu:reference-data--SeismicDomainType:katalyst:TIME:',
'StackAngleRangeMax': 45.0,
'inLineMin': 1.0,
'inLineMax': 339.0,
'crossLineMin': 1.0,
'crossLineMax': 315.0,
'Datasets': ['osdu:dataset--ConnectedSource.Generic:Katalyst-katalyst-7334713:']
},
'createUser': 'DARLON',
'createTime': '2011-02-18T00:00:00.000Z',
'modifyTime': '2021-06-30T01:12:43.000Z',
'modifyUser': 'kdm_demo_us',
'acl': {
'owners': ['data.default.owners@osdu.example.com'],
'viewers': ['data.default.viewers@osdu.example.com']
},
'legal': {
'legaltags': ['osdu-demo-legaltag'],
'otherRelevantDataCountries': ['US']
}
}, {
'kind': 'osdu:wks:work-product-component--SeismicTraceData:1.0.0',
'id': 'osdu:work-product-component--SeismicTraceData:Katalyst-katalyst-7334714',
'version': 1625033563000,
'data': {
'ExtensionProperties': {
'kdmLoadDate': '2018-09-07T00:00:00.000Z'
},
'TraceLengthUOM': 'osdu:reference-data--UnitOfMeasure:katalyst:ms:',
'PrincipalAcquisitionProjectID': 'osdu:master-data--SeismicAcquisitionSurvey:katalyst:50000594760:',
'SeismicTraceDataDimensionalityTypeID': 'osdu:reference-data--SeismicTraceDataDimensionalityType:katalyst:3D:',
'Description': 'STACK',
'TraceLength': 3000.0,
'SampleInterval': 2.0,
'TraceDomainUOM': 'osdu:reference-data--UnitOfMeasure:katalyst:ms:',
'SeismicDomainTypeID': 'osdu:reference-data--SeismicDomainType:katalyst:TIME:',
'StackAngleRangeMax': 45.0,
'inLineMin': 1.0,
'inLineMax': 339.0,
'crossLineMin': 1.0,
'crossLineMax': 315.0,
'Datasets': ['osdu:dataset--ConnectedSource.Generic:Katalyst-katalyst-7334714:']
},
'createUser': 'DARLON',
'createTime': '2011-02-18T00:00:00.000Z',
'modifyTime': '2021-06-30T01:12:43.000Z',
'modifyUser': 'kdm_demo_us',
'acl': {
'owners': ['data.default.owners@osdu.example.com'],
'viewers': ['data.default.viewers@osdu.example.com']
},
'legal': {
'legaltags': ['osdu-demo-legaltag'],
'otherRelevantDataCountries': ['US']
}
}, {
'kind': 'osdu:wks:work-product-component--SeismicTraceData:1.0.0',
'id': 'osdu:work-product-component--SeismicTraceData:Katalyst-katalyst-7334715',
'version': 1625033563000,
'data': {
'ExtensionProperties': {
'kdmLoadDate': '2018-09-07T00:00:00.000Z'
},
'TraceLengthUOM': 'osdu:reference-data--UnitOfMeasure:katalyst:ms:',
'PrincipalAcquisitionProjectID': 'osdu:master-data--SeismicAcquisitionSurvey:katalyst:50000594760:',
'SeismicTraceDataDimensionalityTypeID': 'osdu:reference-data--SeismicTraceDataDimensionalityType:katalyst:3D:',
'Description': 'STACK',
'TraceLength': 3000.0,
'SampleInterval': 2.0,
'TraceDomainUOM': 'osdu:reference-data--UnitOfMeasure:katalyst:ms:',
'SeismicDomainTypeID': 'osdu:reference-data--SeismicDomainType:katalyst:TIME:',
'StackAngleRangeMax': 45.0,
'inLineMin': 1.0,
'inLineMax': 339.0,
'crossLineMin': 1.0,
'crossLineMax': 315.0,
'Datasets': ['osdu:dataset--ConnectedSource.Generic:Katalyst-katalyst-7334715:']
},
'createUser': 'DARLON',
'createTime': '2011-02-18T00:00:00.000Z',
'modifyTime': '2021-06-30T01:12:43.000Z',
'modifyUser': 'kdm_demo_us',
'acl': {
'owners': ['data.default.owners@osdu.example.com'],
'viewers': ['data.default.viewers@osdu.example.com']
},
'legal': {
'legaltags': ['osdu-demo-legaltag'],
'otherRelevantDataCountries': ['US']
}
}, {
'kind': 'osdu:wks:work-product-component--SeismicTraceData:1.0.0',
'id': 'osdu:work-product-component--SeismicTraceData:Katalyst-katalyst-7334716',
'version': 1625033563000,
'data': {
'ExtensionProperties': {
'kdmLoadDate': '2018-09-07T00:00:00.000Z'
},
'TraceLengthUOM': 'osdu:reference-data--UnitOfMeasure:katalyst:ms:',
'PrincipalAcquisitionProjectID': 'osdu:master-data--SeismicAcquisitionSurvey:katalyst:50000594760:',
'SeismicTraceDataDimensionalityTypeID': 'osdu:reference-data--SeismicTraceDataDimensionalityType:katalyst:3D:',
'Description': 'STACK',
'TraceLength': 3000.0,
'SampleInterval': 2.0,
'TraceDomainUOM': 'osdu:reference-data--UnitOfMeasure:katalyst:ms:',
'SeismicDomainTypeID': 'osdu:reference-data--SeismicDomainType:katalyst:TIME:',
'StackAngleRangeMax': 45.0,
'inLineMin': 1.0,
'inLineMax': 339.0,
'crossLineMin': 1.0,
'crossLineMax': 315.0,
'Datasets': ['osdu:dataset--ConnectedSource.Generic:Katalyst-katalyst-7334716:']
},
'createUser': 'DARLON',
'createTime': '2011-02-18T00:00:00.000Z',
'modifyTime': '2021-06-30T01:12:43.000Z',
'modifyUser': 'kdm_demo_us',
'acl': {
'owners': ['data.default.owners@osdu.example.com'],
'viewers': ['data.default.viewers@osdu.example.com']
},
'legal': {
'legaltags': ['osdu-demo-legaltag'],
'otherRelevantDataCountries': ['US']
}
}, {
'kind': 'osdu:wks:work-product-component--SeismicTraceData:1.0.0',
'id': 'osdu:work-product-component--SeismicTraceData:Katalyst-katalyst-7334731',
'version': 1625033563000,
'data': {
'ExtensionProperties': {
'kdmLoadDate': '2018-09-07T00:00:00.000Z'
},
'TraceLengthUOM': 'osdu:reference-data--UnitOfMeasure:katalyst:ms:',
'PrincipalAcquisitionProjectID': 'osdu:master-data--SeismicAcquisitionSurvey:katalyst:50000594769:',
'SeismicTraceDataDimensionalityTypeID': 'osdu:reference-data--SeismicTraceDataDimensionalityType:katalyst:3D:',
'Description': 'MIGRATION',
'TraceCount': 1190444,
'TraceLength': 5000.0,
'SampleInterval': 2.0,
'TraceDomainUOM': 'osdu:reference-data--UnitOfMeasure:katalyst:ms:',
'SeismicDomainTypeID': 'osdu:reference-data--SeismicDomainType:katalyst:TIME:',
'StackAngleRangeMax': 45.0,
'inLineMin': 1.0,
'inLineMax': 1451.0,
'crossLineMin': 1.0,
'crossLineMax': 1063.0,
'Datasets': ['osdu:dataset--ConnectedSource.Generic:Katalyst-katalyst-7334731:']
},
'createUser': 'thum:SSIndex',
'createTime': '2011-09-14T14:54:14.000Z',
'modifyTime': '2021-06-30T01:12:43.000Z',
'modifyUser': 'kdm_demo_us',
'acl': {
'owners': ['data.default.owners@osdu.example.com'],
'viewers': ['data.default.viewers@osdu.example.com']
},
'legal': {
'legaltags': ['osdu-demo-legaltag'],
'otherRelevantDataCountries': ['US']
}
}],
'Datasets': [{
'data': {
'DatasetProperties': {
'ConnectedSourceRegistryEntryID': 'osdu:master-data--ConnectedSourceRegistryEntry:a9d70013-e645-4d7a-a721-89f88c32807f:',
'ConnectedSourceDataJobID': 'osdu:master-data--ConnectedSourceDataJob:a9d70013-e645-4d7a-a721-89f88c32807f:',
'SourceDataPartitionID': 'katalyst',
'SourceRecordID': 'katalyst:dataset--File.Generic:19330323'
}
},
'kind': 'osdu:wks:dataset--ConnectedSource.Generic:0.2.0',
'legal': {
'legaltags': ['osdu-demo-legaltag'],
'otherRelevantDataCountries': ['US']
},
'acl': {
'owners': ['data.default.owners@osdu.example.com'],
'viewers': ['data.default.viewers@osdu.example.com']
},
'id': 'osdu:dataset--ConnectedSource.Generic:Katalyst-katalyst-19330323'
}, {
'data': {
'DatasetProperties': {
'ConnectedSourceRegistryEntryID': 'osdu:master-data--ConnectedSourceRegistryEntry:a9d70013-e645-4d7a-a721-89f88c32807f:',
'ConnectedSourceDataJobID': 'osdu:master-data--ConnectedSourceDataJob:a9d70013-e645-4d7a-a721-89f88c32807f:',
'SourceDataPartitionID': 'katalyst',
'SourceRecordID': 'katalyst:dataset--File.Generic:7334713'
}
},
'kind': 'osdu:wks:dataset--ConnectedSource.Generic:0.2.0',
'legal': {
'legaltags': ['osdu-demo-legaltag'],
'otherRelevantDataCountries': ['US']
},
'acl': {
'owners': ['data.default.owners@osdu.example.com'],
'viewers': ['data.default.viewers@osdu.example.com']
},
'id': 'osdu:dataset--ConnectedSource.Generic:Katalyst-katalyst-7334713'
}, {
'data': {
'DatasetProperties': {
'ConnectedSourceRegistryEntryID': 'osdu:master-data--ConnectedSourceRegistryEntry:a9d70013-e645-4d7a-a721-89f88c32807f:',
'ConnectedSourceDataJobID': 'osdu:master-data--ConnectedSourceDataJob:a9d70013-e645-4d7a-a721-89f88c32807f:',
'SourceDataPartitionID': 'katalyst',
'SourceRecordID': 'katalyst:dataset--File.Generic:7334714'
}
},
'kind': 'osdu:wks:dataset--ConnectedSource.Generic:0.2.0',
'legal': {
'legaltags': ['osdu-demo-legaltag'],
'otherRelevantDataCountries': ['US']
},
'acl': {
'owners': ['data.default.owners@osdu.example.com'],
'viewers': ['data.default.viewers@osdu.example.com']
},
'id': 'osdu:dataset--ConnectedSource.Generic:Katalyst-katalyst-7334714'
}, {
'data': {
'DatasetProperties': {
'ConnectedSourceRegistryEntryID': 'osdu:master-data--ConnectedSourceRegistryEntry:a9d70013-e645-4d7a-a721-89f88c32807f:',
'ConnectedSourceDataJobID': 'osdu:master-data--ConnectedSourceDataJob:a9d70013-e645-4d7a-a721-89f88c32807f:',
'SourceDataPartitionID': 'katalyst',
'SourceRecordID': 'katalyst:dataset--File.Generic:7334715'
}
},
'kind': 'osdu:wks:dataset--ConnectedSource.Generic:0.2.0',
'legal': {
'legaltags': ['osdu-demo-legaltag'],
'otherRelevantDataCountries': ['US']
},
'acl': {
'owners': ['data.default.owners@osdu.example.com'],
'viewers': ['data.default.viewers@osdu.example.com']
},
'id': 'osdu:dataset--ConnectedSource.Generic:Katalyst-katalyst-7334715'
}, {
'data': {
'DatasetProperties': {
'ConnectedSourceRegistryEntryID': 'osdu:master-data--ConnectedSourceRegistryEntry:a9d70013-e645-4d7a-a721-89f88c32807f:',
'ConnectedSourceDataJobID': 'osdu:master-data--ConnectedSourceDataJob:a9d70013-e645-4d7a-a721-89f88c32807f:',
'SourceDataPartitionID': 'katalyst',
'SourceRecordID': 'katalyst:dataset--File.Generic:7334716'
}
},
'kind': 'osdu:wks:dataset--ConnectedSource.Generic:0.2.0',
'legal': {
'legaltags': ['osdu-demo-legaltag'],
'otherRelevantDataCountries': ['US']
},
'acl': {
'owners': ['data.default.owners@osdu.example.com'],
'viewers': ['data.default.viewers@osdu.example.com']
},
'id': 'osdu:dataset--ConnectedSource.Generic:Katalyst-katalyst-7334716'
}, {
'data': {
'DatasetProperties': {
'ConnectedSourceRegistryEntryID': 'osdu:master-data--ConnectedSourceRegistryEntry:a9d70013-e645-4d7a-a721-89f88c32807f:',
'ConnectedSourceDataJobID': 'osdu:master-data--ConnectedSourceDataJob:a9d70013-e645-4d7a-a721-89f88c32807f:',
'SourceDataPartitionID': 'katalyst',
'SourceRecordID': 'katalyst:dataset--File.Generic:7334731'
}
},
'kind': 'osdu:wks:dataset--ConnectedSource.Generic:0.2.0',
'legal': {
'legaltags': ['osdu-demo-legaltag'],
'otherRelevantDataCountries': ['US']
},
'acl': {
'owners': ['data.default.owners@osdu.example.com'],
'viewers': ['data.default.viewers@osdu.example.com']
},
'id': 'osdu:dataset--ConnectedSource.Generic:Katalyst-katalyst-7334731'
}]
}
}
}
}
This is done
Below is the merge raised with resolution reported:
Changes made for MVP1
Dataset record ids being received from external vendor are with same Dataset format which is blocking records to processed with out Dataset on OSDU platform. Added changes to bypass the SourceRecordID being validate whether record exists.