wks merge requestshttps://community.opengroup.org/osdu/platform/data-flow/enrichment/wks/-/merge_requests2021-04-27T05:08:30Zhttps://community.opengroup.org/osdu/platform/data-flow/enrichment/wks/-/merge_requests/49revert wks kind fix2021-04-27T05:08:30Zdevesh bajpairevert wks kind fixIT were failing merged by accidentIT were failing merged by accidenthttps://community.opengroup.org/osdu/platform/data-flow/enrichment/wks/-/merge_requests/48Get wks kind from mapping2021-05-05T06:10:51Zdevesh bajpaiGet wks kind from mappingThis MR fix WKS service to get kind for generated WKS from mapping.This MR fix WKS service to get kind for generated WKS from mapping.https://community.opengroup.org/osdu/platform/data-flow/enrichment/wks/-/merge_requests/47Pick latest version of mapping file2021-06-03T16:39:59Zdevesh bajpaiPick latest version of mapping fileScenario: If multiple version of same mappingId is present then pick latest version only.
This scenario can occur when mapping is updated.Scenario: If multiple version of same mappingId is present then pick latest version only.
This scenario can occur when mapping is updated.https://community.opengroup.org/osdu/platform/data-flow/enrichment/wks/-/merge_requests/46Increasing memory allocated to AKS container2021-05-24T11:42:06Zharshit aggarwalIncreasing memory allocated to AKS containerhttps://community.opengroup.org/osdu/platform/data-flow/enrichment/wks/-/merge_requests/45Corrected ReadMe file2021-04-08T11:02:52ZRatneshCorrected ReadMe filehttps://community.opengroup.org/osdu/platform/data-flow/enrichment/wks/-/merge_requests/44Added mapping info files and mapping files into deployment folder for bootstr...2021-04-16T05:32:07ZNeha SardaAdded mapping info files and mapping files into deployment folder for bootstrappinghttps://community.opengroup.org/osdu/platform/data-flow/enrichment/wks/-/merge_requests/43It id fixes2021-03-31T12:08:04ZNeha SardaIt id fixesHave added new relationship block test cases for WKS and changes in step definition along with addition of payloads for new test casesHave added new relationship block test cases for WKS and changes in step definition along with addition of payloads for new test caseshttps://community.opengroup.org/osdu/platform/data-flow/enrichment/wks/-/merge_requests/42Core Library Upgrades2021-04-07T04:03:42ZAbhishek ChowdhryCore Library Upgrades## Type of change
- [x] Bug Fix
- [ ] Feature
**Please provide link to gitlab issue or ADR(Architecture Decision Record)**
Just Upgrading Core Libraries to the latest stable versions
## Does this introduce a change in the core logic...## Type of change
- [x] Bug Fix
- [ ] Feature
**Please provide link to gitlab issue or ADR(Architecture Decision Record)**
Just Upgrading Core Libraries to the latest stable versions
## Does this introduce a change in the core logic?
- [Yes]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [X] Azure
- [ ] GCP
- [ ] IBM
## Does this introduce a breaking change?
- [NO]
## What is the current behavior?
Current we are referring to the older versions of the core-lib-azure and core-common in WKS service while we have had new stable versions for these libraries for some time now.
## What is the new/expected behavior?
Upgraded the versions of core-lib-azure and core-common Libraries to latest stable versions(0.7.0 for both) in the entire project
## Have you added/updated Unit Tests and Integration Tests?
NA
## Any other useful informationAbhishek ChowdhryAbhishek Chowdhryhttps://community.opengroup.org/osdu/platform/data-flow/enrichment/wks/-/merge_requests/41Wks logging changes2021-04-28T19:08:45ZPramesh PatilWks logging changesPreviously slf4j logger and lombok logger used in WKS service and data partition and slb correlation details not present in those log.
In this MR, we are using central library that provide support for app log and also add data partition ...Previously slf4j logger and lombok logger used in WKS service and data partition and slb correlation details not present in those log.
In this MR, we are using central library that provide support for app log and also add data partition and correlation id in logs.Pramesh PatilPramesh Patilhttps://community.opengroup.org/osdu/platform/data-flow/enrichment/wks/-/merge_requests/40Enhance package structure for WKS2021-05-13T20:09:48Zdevesh bajpaiEnhance package structure for WKSEnhance package structureEnhance package structurehttps://community.opengroup.org/osdu/platform/data-flow/enrichment/wks/-/merge_requests/39Fix mapping info datatype2021-03-02T21:00:08Zdevesh bajpaiFix mapping info datatypeMapping Info is having String as datatype for attributes which are actually int or long.
This MR aim to correct that behavior.Mapping Info is having String as datatype for attributes which are actually int or long.
This MR aim to correct that behavior.https://community.opengroup.org/osdu/platform/data-flow/enrichment/wks/-/merge_requests/38Draft: Master2021-02-26T08:29:57ZNeha SardaDraft: Masterhttps://community.opengroup.org/osdu/platform/data-flow/enrichment/wks/-/merge_requests/37Slb update sli logs2021-05-13T20:09:48Zsachin GuptaSlb update sli logsAdd SLI logs to create dashboard for availabilityAdd SLI logs to create dashboard for availabilityhttps://community.opengroup.org/osdu/platform/data-flow/enrichment/wks/-/merge_requests/36Mapping structure enhancements2021-02-26T10:29:16Zdevesh bajpaiMapping structure enhancementsAt present Mappings used by WKS service support only one operation and that is copy a attribute value from a raw record to its WKS record. User needs WKS mappings to support more complex operations such as
- Copy a JSON object from raw ...At present Mappings used by WKS service support only one operation and that is copy a attribute value from a raw record to its WKS record. User needs WKS mappings to support more complex operations such as
- Copy a JSON object from raw to WKS
- Copy value in to Json object or Json array in WKS
- Aggregate raw values
- Prioritize which raw value to copy in WKS etc...
To support such operations in future mappings structure is modified and changes to WKS service is made to make it extensible for future operations.
Some Concepts related to Mapping enhancements
1. **Mapping Definition** - Definitions that contain mapping identity information and list of mapped attributes from the original schema to the target schema.
2. **Source Schema** - The Source schema refers to the original structural representation of the data that is ingested.
3. **Target Schema** - Target schema is the WKS schema to which the original data will be transformed.
4. **Operation** - A mapping operation declares how and from where the value for a given target property name will be populated during WKS transformation. Mapping operation type defines the type of operation. In current version of the mapping service only two mapping operation types are supported: COPY and COPY SUBSTRUCTURE.https://community.opengroup.org/osdu/platform/data-flow/enrichment/wks/-/merge_requests/35Fixing broken Integration Tests2021-02-05T05:32:34Zharshit aggarwalFixing broken Integration TestsThe MR is fixing broken Integration Tests as result of this [MR ](https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/131)in storage
Changes Include:
Fixing WKS ID Generation Logic
Fixing Test PayloadsThe MR is fixing broken Integration Tests as result of this [MR ](https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/131)in storage
Changes Include:
Fixing WKS ID Generation Logic
Fixing Test Payloadsharshit aggarwalharshit aggarwalhttps://community.opengroup.org/osdu/platform/data-flow/enrichment/wks/-/merge_requests/34Bump up version for libs which are highlighted in white source scan2021-05-17T06:42:16Zdevesh bajpaiBump up version for libs which are highlighted in white source scanBump up version for libs which are highlighted in white source scanBump up version for libs which are highlighted in white source scanhttps://community.opengroup.org/osdu/platform/data-flow/enrichment/wks/-/merge_requests/33enable tags attribute to be copied from raw to wks2021-02-22T14:03:22Zdevesh bajpaienable tags attribute to be copied from raw to wksThis MR intends to copy "tags" attribute from raw record if present to generated WKS records as a default copy operation.This MR intends to copy "tags" attribute from raw record if present to generated WKS records as a default copy operation.https://community.opengroup.org/osdu/platform/data-flow/enrichment/wks/-/merge_requests/32[Azure][Core] - Configuration & Logging Changes2021-02-16T07:53:26Zharshit aggarwal[Azure][Core] - Configuration & Logging ChangesThis MR is introducing bunch of small changes
- Making changes in AKS pod resource configuration and increasing replica counts
- Introducing config classes for Rest Client and Host URL Settings
- DEV and DEMO pipeline caching and chan...This MR is introducing bunch of small changes
- Making changes in AKS pod resource configuration and increasing replica counts
- Introducing config classes for Rest Client and Host URL Settings
- DEV and DEMO pipeline caching and changing maven build option to -pl
- Increasing relevant logs wherever missingharshit aggarwalharshit aggarwalhttps://community.opengroup.org/osdu/platform/data-flow/enrichment/wks/-/merge_requests/31Using SLF4J Logger in core2021-02-04T03:42:05Zharshit aggarwalUsing SLF4J Logger in core Changes in this MR
- Logging changes in Core (Using SLF4J Logger instead of Lombok logger) Relates with Issue https://community.opengroup.org/osdu/platform/data-flow/enrichment/wks/-/issues/5
- Small performance improvement by not cal... Changes in this MR
- Logging changes in Core (Using SLF4J Logger instead of Lombok logger) Relates with Issue https://community.opengroup.org/osdu/platform/data-flow/enrichment/wks/-/issues/5
- Small performance improvement by not calling schema service on every record transformation but to get them while fetching mappings from Blob storage based on kind information
- Minor change of adding correlation id in request headersharshit aggarwalharshit aggarwalhttps://community.opengroup.org/osdu/platform/data-flow/enrichment/wks/-/merge_requests/30Azure: Updating helm config yaml2021-01-05T04:56:18Zharshit aggarwalAzure: Updating helm config yamlThis MR is related with this MR here https://community.opengroup.org/osdu/platform/data-flow/enrichment/wks/-/merge_requests/24
Helm-config.yaml needs to be updated with new values along with values.yaml
ADO pipelines are failing witho...This MR is related with this MR here https://community.opengroup.org/osdu/platform/data-flow/enrichment/wks/-/merge_requests/24
Helm-config.yaml needs to be updated with new values along with values.yaml
ADO pipelines are failing without this changesharshit aggarwalharshit aggarwal