ADR: Provide search capability in legal tags
Problem Statement
Is it possible to add the search capability for legal tags based on the legaltag attributes including the ones in extensionProperties? There might be hundreds of thousands of Legaltags. Need to count on the number of leagltags while designing the solutions. For more details please check #36
Features that need to be supported
- Structure of ExtensionProperties attributes has no definitive format. It will be different for different companies. Shell might follow a structure which might be different for other companies. So extensionproperties need to be pretty much flexible
- The formatting of all LegalTags are pretty much standardized. We are referring to the structure outside of the extensionproperties. Currently, it’s having Name, Description, properties, isValid attributes other that id and dataPartitionId
- Search should support all attribute search in legaltags
- Expected new API to have POST request with body which will contain the query. Query can be pretty complex.
- Mostly the feature will be available as a separate API under legal services.
- Multi- attribute search along with complex queries
- Sort and Search together need to be supported along with limit and offset
- The response should return the entire legalTag with all attributes.
- DSL preferred to Lucene syntax – need to revisit this implementationwise.
Potential Solution Approach
We have brainstormed on the solution that we need to adopt in order to provide the Search capability in the Legal Tag.
In any of these implementations what remains common is the introduction of attribute "kind". This attribute can be defaulted so as to support backward compatibility. Next step is to define a Schema on the Legal Tags. Any change in any of the attributes in the Legal tags will call for creating different Schema version on the Legal tags
|
Use the current Indexer Service to Index Legal tags in same format with Storage. |
Replicate part of Indexer in Legal to Index the LegalTag documents. |
Without using Indexer and creating Legaltag as record. (updated design) |
|
|---|---|---|---|
|
Design |
|
|
Same as current design flow as Records
|
| Solution implementation |
As depicted in the above diagram any PUT/POST request on the Legal tags will need to interact with Indexer to Index the Legal Tag documents. Additionally Legal Tag document need to conform to the Legal Tag schema. The current Indexer service with modifications can be leveraged to achieve this. Legal needs to send change event notification for Indexer to pick up changes. Indexer needs to subscribe to change event notification on Legal accordingly and index the legal tags in ElasticSearch(ES). For Search we need to exclude the Legal tag documents from normal search endpoints. We have to introduce separate endpoints for Legal tag search. Exclude policy evaluation (or acls) when searching for legal tag. different CSPs need to be involved for the implementation of the core services. For example, Search and Legal implementation code handled at the CSP layer for the implementation. The underlying datastore and infrastructure used is dependent on the CSP implementation. Need to account for this |
As illustrated above, the heavy lifting operation of indexing the legal tags in ES is being handled by the Legal Services. Some part of Indexer code can be reused for implementation purpose. For Search we need to exclude the Legal tag documents from normal search endpoints, which can already be done by existing functionality -system-meta-data-legal. No need to have separate endpoints for Legal tag search. Exclude policy evaluation (or acls) when searching for legal tag. |
Create storage record for each legal tag (that contains optional schema) with service.legal.owner as the owner in acl. The viewer should come from PUT /legaltag so access can be controlled as needed. This of course could be defaulted. Additionally access to these legal tag records could be controlled by Policy using existing functionality. For Search, by using kind -system-meta-data-*, these will be excluded from normal search endpoints. |
| Development and Testing Effort | High development and testing effort as it involves touchpoint of multiple Core services. This is design, development ad testing heavy approach. | Development, testing effort more as touching multiple core services | Much lighter approach compared to other two. |
| Services impacted | Indexer, Notification, Legal, Search. | Legal, Indexer, Search | Search |
| Risks / feature robustness | Cleanest and most preferred. |
Code duplication which could lead to inconsistency (e.g., ES upgrades, new schema features, etc.) Expand surface of ES (currently only indexer and search have access to ES) |
Defining/using legal tag on legal tag Storage records seems like anti-pattern. |
Proposed Legal Legal Schema
Proposed Legal Query API spec legal POST /query, no params. Payload would include { "kind": {}, "limit": 0, "query": "string", "sort": { "field": [ "string" ], "order": [ "ASC" ] }, "offset": 0 }