ADR: Option to retain source systems audit info and override audit fields during migration
Status
-
Proposed -
Trialing -
Under review -
Approved -
Retired
Background
When a record is created, the 'createUser'/'modifyUser' field automatically captures the username from the token and sets the 'createTime'/'modifyTime' to the current timestamp. These fields play a crucial role in providing audit information to identify who created or modified and when. While this mechanism works seamlessly for new records created through the OSDU APIs, it may lead to confusion when dealing with migrated data.
The source systems maintain their own set of audit fields, which should be preserved in their original state during migration. Preserving this audit trail is vital to upholding data integrity and regulatory compliance.
Refer Aha ticket IDEA-I-130
Context & Scope
The audit information captured in 'createUser', 'createTime', 'modifyUser' and 'modifyTime' fields can be stored in extendedProperties. However, the limitations of extendedProperties, such as the inability to index values, hinder the efficient filtering and retrieval of records.
To address this issue, either source system’s audit information such as createUser, creatTime, modifyUser and modifyTime should be set in new attribute, or the storage service should allow to override existing attribute values.
Proposed solution
Option 1: Introduce an 'Audit' object attribute into the common schema, integrating it as a standard attribute of all data type schemas. This approach ensures consistent and comprehensive auditing capabilities across different data types.
Option 2: Implement a new user or role with specialized permissions to override audit attributes, including the createUser, createDate, modifyUser and modifyTime fields. This designated user or role is specifically designated for managing data migration processes. For instance, when initiating the ingestion API using this designated user, the Platform verifies its migration status. In such instances, the user's email and creation time will be sourced from Manifest values rather than the token or current timestamp.
Consequences
Option 1: The implementation of Option 1 may entail a time-consuming process and could potentially have a significant impact on existing records. Integrating the 'Audit' object attribute into the common schema may require thorough planning and careful consideration to mitigate disruptions to the system.
Option 2: While Option 2 eliminates the need for introducing new attributes, it necessitates modifications to the Storage Service logic. Adapting the system to accommodate a new user or role with override permissions may require adjustments to the existing logic and workflows within the Storage Service.