OSDU Software issueshttps://community.opengroup.org/groups/osdu/-/issues2020-09-14T21:47:39Zhttps://community.opengroup.org/osdu/platform/system/reference/unit-service/-/issues/13POMs have goals to display dependency updates which hurts CI performance2020-09-14T21:47:39ZMatt WisePOMs have goals to display dependency updates which hurts CI performanceThe POMs in this project, unlike other projects display all dependency updates during the compile phase.
This adds a ton of console writes during build and slows the CI process down significantly.
Removing these goals from compile shows...The POMs in this project, unlike other projects display all dependency updates during the compile phase.
This adds a ton of console writes during build and slows the CI process down significantly.
Removing these goals from compile shows up to 20x build performance increase:
* <goal>display-dependency-updates</goal>
* <goal>display-plugin-updates</goal>
Full Snippet:
```xml
<plugins>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>versions-maven-plugin</artifactId>
<version>2.7</version>
<executions>
<execution>
<phase>compile</phase>
<goals>
<goal>display-dependency-updates</goal>
<goal>display-plugin-updates</goal>
</goals>
</execution>
</executions>
</plugin>
...
```M1 - Release 0.1David Diederichd.diederich@opengroup.orgethiraj krishnamanaiduJoeDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/system/home/-/issues/48OSDU system services support data partition2021-06-16T22:20:09ZChris ZhangOSDU system services support data partitionWith the [Partition Service ADR](https://community.opengroup.org/osdu/platform/system/home/-/issues/42) approved, this issue is to track the work needed for OSDU system services to support the concept of data partition.With the [Partition Service ADR](https://community.opengroup.org/osdu/platform/system/home/-/issues/42) approved, this issue is to track the work needed for OSDU system services to support the concept of data partition.https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/home/-/issues/12Seismic DMS to support data partition2021-06-15T21:17:55ZChris ZhangSeismic DMS to support data partitionWith the [Partition Service ADR](https://community.opengroup.org/osdu/platform/system/home/-/issues/42) approved, this issue is to track the work needed for Seismic DMS to support the concept of data partition.With the [Partition Service ADR](https://community.opengroup.org/osdu/platform/system/home/-/issues/42) approved, this issue is to track the work needed for Seismic DMS to support the concept of data partition.https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/issues/9Wellbore DMS to support data partition2021-05-20T13:58:12ZChris ZhangWellbore DMS to support data partitionWith the [Partition Service ADR](https://community.opengroup.org/osdu/platform/system/home/-/issues/42) approved, this issue is to track the work needed for Wellbore DMS to support the concept of data partition.With the [Partition Service ADR](https://community.opengroup.org/osdu/platform/system/home/-/issues/42) approved, this issue is to track the work needed for Wellbore DMS to support the concept of data partition.https://community.opengroup.org/osdu/platform/security-and-compliance/home/-/issues/42OSDU Security and compliance services to support data partition2021-06-16T22:20:26ZChris ZhangOSDU Security and compliance services to support data partitionWith the [Partition Service ADR](https://community.opengroup.org/osdu/platform/system/home/-/issues/42) approved, this issue is to track the work needed for OSDU Security and compliance services to support the concept of data partition.With the [Partition Service ADR](https://community.opengroup.org/osdu/platform/system/home/-/issues/42) approved, this issue is to track the work needed for OSDU Security and compliance services to support the concept of data partition.https://community.opengroup.org/osdu/platform/data-flow/ingestion/home/-/issues/39OSDU data flow services to support data partition2021-06-16T22:20:24ZChris ZhangOSDU data flow services to support data partitionWith the [Partition Service ADR](https://community.opengroup.org/osdu/platform/system/home/-/issues/42) approved, this issue is to track the work needed for OSDU data flow services to support the concept of data partition.With the [Partition Service ADR](https://community.opengroup.org/osdu/platform/system/home/-/issues/42) approved, this issue is to track the work needed for OSDU data flow services to support the concept of data partition.https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/issues/55[Parsers] Add support of libraries not copied to Airflow DAG repository2021-06-16T22:17:42ZKateryna Kurach (EPAM)[Parsers] Add support of libraries not copied to Airflow DAG repositoryIt is possible that some of the parsers that will be integrated with OSDU will be commercial solutions and owners of these parsers will not donate their code to OSDU.
Parser developers should follow an established standard and project st...It is possible that some of the parsers that will be integrated with OSDU will be commercial solutions and owners of these parsers will not donate their code to OSDU.
Parser developers should follow an established standard and project structure and do parser development in their own reporsitory. Ingestion team should document this process and project structure.
The same should be done for Airflow DAGs.https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/issues/56[Validation] Support of Access Control Lists and Legal Tags in Manifest2021-06-16T22:17:41ZKateryna Kurach (EPAM)[Validation] Support of Access Control Lists and Legal Tags in ManifestThis issue is related to integration with policy-based Entitlements Service.
We need to include the following functionality:
- Validation of ACL from Manifest
- Validation of Legal Tags
- Validation of Document Author, ACL value from the...This issue is related to integration with policy-based Entitlements Service.
We need to include the following functionality:
- Validation of ACL from Manifest
- Validation of Legal Tags
- Validation of Document Author, ACL value from the Manifest against Legal Tags that are present in the manifest (e.g. that Document Author has permissions to put a specific ACL value into the Manifest; that Doc Author has a right to put specific Legal Tags, etc) - detailed reqs have to be developed here.https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/issues/57[Non-functional Requirements] Ability to parallelize data ingestion activities2021-06-16T22:17:40ZKateryna Kurach (EPAM)[Non-functional Requirements] Ability to parallelize data ingestion activitiesThis feature has to be added to more effectively ingest big volumes of data.This feature has to be added to more effectively ingest big volumes of data.https://community.opengroup.org/osdu/platform/system/home/-/issues/49Change daily meeting format2020-09-27T19:38:15ZChris ZhangChange daily meeting formatThere has been very close collaboration among all teams for R3 development. One of the main sync up point is the current daily standup meeting, and weekly grooming meeting. The current setup is (Houston Time) for teams in Houston:
* Mon...There has been very close collaboration among all teams for R3 development. One of the main sync up point is the current daily standup meeting, and weekly grooming meeting. The current setup is (Houston Time) for teams in Houston:
* Mon - Wed, 3 - 4pm (1 hour): daily meeting to sync up status and discuss any blocker issues.
* Thu, 2 - 3:30pm (1.5 hour): Weekly backlog grooming session.
Meanwhile, considering the global time zones involved in ingestion project, there is another weekly meeting on Thur:
* Thu, 7 - 8am (1 hour): Ingestion project weekly meeting
In order to optimize the efficiency of those meetings, and ensure we have dedicated technical review discussions weekly, based on the feedback from meeting attendees, we propose to change our meetings to below format for teams in Houston:
* Mon, 3 - 3:30pm (0.5 hour): Status update and discuss any blocker issues.
* Tue, 3 - 4pm (1 hour): Dedicated technical discussions and ADR reviews.
* Wed, 3 - 3:30pm (0.5 hour): Status update and discuss any blocker issues. Same format as Monday.
* Thu, 2 - 3:30pm (1.5 hours): Weekly backlog grooming session, starting with blocker issue discussion.
Note, in everyday's meeting, we will cover any blocker issues to ensure teams' development velocity. We are not changing the start time of the meeting, but changing the duration and focus of the meetings.
Meanwhile, in order to have closer collaboration and faster feedback on ingestion projects, we propose to add another weekly meeting on Monday that involves different time zones. So, the new meeting format for ingestion project is:
* Mon, 8 - 9am (1 hour): Ingestion project weekly meeting
* Thu, 7 - 8am (1 hour): Ingestion project weekly meeting (No change)
Please provide your feedback on this change proposal.https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/issues/58[Validation] [Master and Reference Data] Manifest Validation - check Referenc...2022-06-28T19:45:51ZKateryna Kurach (EPAM)[Validation] [Master and Reference Data] Manifest Validation - check Reference Data recordsValidation that Reference data values (SRNs) point to existing records.Validation that Reference data values (SRNs) point to existing records.https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/issues/59[Validation] [Master and Reference Data] Manifest Validation - check Master D...2021-01-29T21:00:37ZKateryna Kurach (EPAM)[Validation] [Master and Reference Data] Manifest Validation - check Master Data recordsValidation that Master data values (SRNs) point to existing records.Validation that Master data values (SRNs) point to existing records.https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/issues/60[Validation] Manifest Validation - check WPC Id for uniqueness2021-06-16T22:17:38ZKateryna Kurach (EPAM)[Validation] Manifest Validation - check WPC Id for uniquenessWPC Id should be unique.
2 cases should be covered:
- If a user provided an Id, it should be unique
- If user didnt provide an Id - the system should generate a unique IdWPC Id should be unique.
2 cases should be covered:
- If a user provided an Id, it should be unique
- If user didnt provide an Id - the system should generate a unique Idhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/issues/61[Parsers] Pluggable logic of duplicated files detection - HASH calculation2021-06-16T22:17:37ZKateryna Kurach (EPAM)[Parsers] Pluggable logic of duplicated files detection - HASH calculationAccording to OSDU Reference Architecture, duplicated files / data should not be saved in OSDU.
HASH code should be calculated for each ingested file / WPC and this value should be stored in the schema.
It seems that it makes sense to imp...According to OSDU Reference Architecture, duplicated files / data should not be saved in OSDU.
HASH code should be calculated for each ingested file / WPC and this value should be stored in the schema.
It seems that it makes sense to implement this logic on the parser side: each parser developer will calculate HASH code for the ingested WPC.https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/issues/62[Parsers] Integrate developed WITSML Parser into Ingestion Framework2021-06-16T22:17:36ZKateryna Kurach (EPAM)[Parsers] Integrate developed WITSML Parser into Ingestion FrameworkWITSML Parser has been developed by Energistics. This user story is to track an effort needed to spend on integration this WITSML Parser into the Ingestion Framework.
Energistics will support 4 WITSML data types: Well Log, Directional Su...WITSML Parser has been developed by Energistics. This user story is to track an effort needed to spend on integration this WITSML Parser into the Ingestion Framework.
Energistics will support 4 WITSML data types: Well Log, Directional Survey, Tubulars, Well Markers. Is it 1 parser or 4 parsers?https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/issues/63[Parsers] Integrate developed RESQML Parser into Ingestion Framework2021-06-23T17:33:05ZKateryna Kurach (EPAM)[Parsers] Integrate developed RESQML Parser into Ingestion FrameworkRESQML Parser is being developed by Energistics.
This user story is to track an effort needed to support integration of RESQML Parser into Ingestion Framework.RESQML Parser is being developed by Energistics.
This user story is to track an effort needed to support integration of RESQML Parser into Ingestion Framework.https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/issues/64[Parsers] Integrate developed SEG-Y - Seismic Parser into Ingestion Framework2021-06-16T22:17:34ZKateryna Kurach (EPAM)[Parsers] Integrate developed SEG-Y - Seismic Parser into Ingestion FrameworkThis user story is create to track an effort needed to integrate developed SEG-Y Seismic Parser (developed by CGI team or some other party) into the Ingestion Framework.This user story is create to track an effort needed to integrate developed SEG-Y Seismic Parser (developed by CGI team or some other party) into the Ingestion Framework.https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/issues/65[Master and Reference Data] Master Data Ingestion process2020-10-22T18:19:05ZKateryna Kurach (EPAM)[Master and Reference Data] Master Data Ingestion processMaster Data Manifest-based ingestionMaster Data Manifest-based ingestionKateryna Kurach (EPAM)Kateryna Kurach (EPAM)2020-11-10https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/issues/66[Master and Reference Data] Reference Data Ingestion process2020-10-22T18:18:21ZKateryna Kurach (EPAM)[Master and Reference Data] Reference Data Ingestion processReference Data manifest-based ingestionReference Data manifest-based ingestionKateryna Kurach (EPAM)Kateryna Kurach (EPAM)2020-11-10https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/issues/67[Master and Reference Data] Frame of Reference Handling2021-06-16T22:17:33ZKateryna Kurach (EPAM)[Master and Reference Data] Frame of Reference HandlingReqs for Frame of Reference handling will be added laterReqs for Frame of Reference handling will be added later