File issueshttps://community.opengroup.org/osdu/platform/system/file/-/issues2022-01-18T04:04:46Zhttps://community.opengroup.org/osdu/platform/system/file/-/issues/43File service is not ingesting additional metadata2022-01-18T04:04:46ZPramesh PatilFile service is not ingesting additional metadataOld Implementation - we have created fixed set of schema for extension properties block and user not able to send additional metadata while posting metadata thought POST endpoint.
Changes - Updated model that accept addition metadata th...Old Implementation - we have created fixed set of schema for extension properties block and user not able to send additional metadata while posting metadata thought POST endpoint.
Changes - Updated model that accept addition metadata through extensionProperties.
example - (NewKey and NewValue added as addition metadata)
```
"ExtensionProperties": {
"Classification": "Raw File",
"newKey" :"newValue",
"Description": "An text further describing this file example.",
"ExternalIds": [
"string"
],
"FileDateCreated": {},
"FileDateModified": {},
}
```
MR - https://community.opengroup.org/osdu/platform/system/file/-/merge_requests/157M9 - Release 0.12https://community.opengroup.org/osdu/platform/system/file/-/issues/42Implement status publishing method in IBM2022-07-22T09:15:22ZParesh BehedeImplement status publishing method in IBMWe have recently implemented code for publishing several status of metadata api endpoint as per the decided in approved ADR https://community.opengroup.org/osdu/platform/system/home/-/issues/80
changes done in file core can be found her...We have recently implemented code for publishing several status of metadata api endpoint as per the decided in approved ADR https://community.opengroup.org/osdu/platform/system/home/-/issues/80
changes done in file core can be found here along with its publish method implementation for Azure provider https://community.opengroup.org/osdu/platform/system/file/-/merge_requests/124
This issue has been created for IBM team to implement that publish method to publish the status events in message queue.Anuj GuptaShaonjingdong sunAnuj Guptahttps://community.opengroup.org/osdu/platform/system/file/-/issues/41Implement status publishing method in AWS2022-09-27T11:24:19ZParesh BehedeImplement status publishing method in AWSWe have recently implemented code for publishing several status of metadata api endpoint as per the decided in approved ADR https://community.opengroup.org/osdu/platform/system/home/-/issues/80
changes done in file core can be found her...We have recently implemented code for publishing several status of metadata api endpoint as per the decided in approved ADR https://community.opengroup.org/osdu/platform/system/home/-/issues/80
changes done in file core can be found here along with its publish method implementation for Azure provider https://community.opengroup.org/osdu/platform/system/file/-/merge_requests/124
This issue has been created for AWS team to implement that publish method to publish the status events in message queue.GregGreghttps://community.opengroup.org/osdu/platform/system/file/-/issues/40Implement status publishing method in GCP2022-09-27T11:24:30ZParesh BehedeImplement status publishing method in GCPWe have recently implemented code for publishing several status of metadata api endpoint as per the decided in approved ADR
changes done in file core can be found here along with its publish method implementation for Azure provider http...We have recently implemented code for publishing several status of metadata api endpoint as per the decided in approved ADR
changes done in file core can be found here along with its publish method implementation for Azure provider https://community.opengroup.org/osdu/platform/system/file/-/merge_requests/124
This issue has been created for GCP team to implement that publish method to publish the status events in message queue.Kateryna Kurach (EPAM)Kateryna Kurach (EPAM)https://community.opengroup.org/osdu/platform/system/file/-/issues/39Implement CSP specific changes for File DMS APIs support2021-10-19T19:22:38ZKrishna Nikhil VedurumudiImplement CSP specific changes for File DMS APIs supportFollow up for https://community.opengroup.org/osdu/platform/system/file/-/issues/29 where the Core and Azure changes for File DMS changes were completed.
Implementation Updates:
* Core Common changes with skeleton interfaces and model...Follow up for https://community.opengroup.org/osdu/platform/system/file/-/issues/29 where the Core and Azure changes for File DMS changes were completed.
Implementation Updates:
* Core Common changes with skeleton interfaces and model objects - https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/merge_requests/99
* File service changes with Core and Azure implementation for File DMS APIs - https://community.opengroup.org/osdu/platform/system/file/-/merge_requests/128
* Dataset Service changes consuming the APIs - https://community.opengroup.org/osdu/platform/system/dataset/-/merge_requests/109
* Entitlements changes - New role for dataset - https://community.opengroup.org/osdu/platform/security-and-compliance/entitlements/-/merge_requests/90
Work for Other CSPs
* Using the feature flag in Dataset Service, use the latest DMS API endpoints.
* Create new role in Entitlements -> https://community.opengroup.org/osdu/platform/security-and-compliance/entitlements/-/merge_requests/90
* File Service changes - Implement `final IStorageService storageService;` for interactions with CSP Specific Blob store providers.M9 - Release 0.12https://community.opengroup.org/osdu/platform/system/file/-/issues/38GET /DownloadURL api to support generating signedURL that can be accessible f...2022-09-29T13:41:08ZOrsu AkhilGET /DownloadURL api to support generating signedURL that can be accessible from select IP address.The /DownloadURL api would return signedURL to access file stored in persistent location based on File Metadata id .
Having the feature wherein User can specify the range of IP address , only through which when signedURL is used would re...The /DownloadURL api would return signedURL to access file stored in persistent location based on File Metadata id .
Having the feature wherein User can specify the range of IP address , only through which when signedURL is used would result in access to the file in persistent location would be good. This IP address or range of IP address can be passed by User via GET query Parameter in the api specification.
Supporting this sort of customization in generated signed url is possible in Azure Cloud and would like to know about this in other cloud providers , if is feasible in others as well then could come up with implementation supporting this feature.Orsu AkhilParesh BehedeOrsu Akhilhttps://community.opengroup.org/osdu/platform/system/file/-/issues/37GET /DownloadURL api to support generating signedURL with custom expiry time2021-08-27T21:29:39ZOrsu AkhilGET /DownloadURL api to support generating signedURL with custom expiry timeCurrently in cloud provider implementation of File Application for Azure , GCP the /DownloadURL api would return signedURL to access file stored in persistent location and the expiry time of this URL is by default set to 7 Days. This may...Currently in cloud provider implementation of File Application for Azure , GCP the /DownloadURL api would return signedURL to access file stored in persistent location and the expiry time of this URL is by default set to 7 Days. This mayn't be desirable and giving flexibility to User to determine when the URL should expire would be a good feature add-on for this api. This information can be passed by User via GET query Parameter in the api specification.M8 - Release 0.11Orsu AkhilParesh BehedeOrsu Akhilhttps://community.opengroup.org/osdu/platform/system/file/-/issues/36Request for HTTP header Content-Disposition (optionally) for file download2022-09-27T11:33:23ZSteven ReynoldsRequest for HTTP header Content-Disposition (optionally) for file downloadWe implemented this Download feature in our application, that will return a download URL that points to the OSDU download API, but the download behaviour in browsers is not consistent between csv and las files for instance. Some browsers...We implemented this Download feature in our application, that will return a download URL that points to the OSDU download API, but the download behaviour in browsers is not consistent between csv and las files for instance. Some browsers will actually trigger a download, others will open in a new tab. We would like for the user experience to be consistent regardless of the file type.
We had a look at it and it's up to the the platform to add the following http headers to these downloads http response, so that the browser knows it must initiate a file download :
header("Content-Disposition", "attachment; filename=myfilename.myextension");
Maybe we could also think about an extra parameter to the request to let the server know it must add this header.https://community.opengroup.org/osdu/platform/system/file/-/issues/35File should get download with actual filename and extension after hitting dow...2023-04-13T10:39:15Zsachin GuptaFile should get download with actual filename and extension after hitting downloadUrl### Problem Statement
Currently, the file service stores file in the persistent location with some random name and without extension. When we download the file using download SignedURL, it's download without extension and to see the co...### Problem Statement
Currently, the file service stores file in the persistent location with some random name and without extension. When we download the file using download SignedURL, it's download without extension and to see the content of the file we need to explicitly give an extension.
### Solution
We can overcome this problem by adding Content-Disposition and Content-Type header at the time of download signed URL creation. We can get file name from file metadata payload and based on file extension we can get content type of the file, both name and content type we can use to create download URL.
Note: the name field in the metadata payload is optional and if it is not present in the payload, then the above solution won't work for that and follow the current implementation of download URL creation.M9 - Release 0.12Paresh BehedeParesh Behedehttps://community.opengroup.org/osdu/platform/system/file/-/issues/34Upgrade Core Azure Dependency2022-02-11T22:00:53ZDavid Diederichd.diederich@opengroup.orgUpgrade Core Azure Dependencyhttps://community.opengroup.org/osdu/platform/system/file/-/issues/33Upgrade Core IBM Dependency2022-02-11T22:01:07ZDavid Diederichd.diederich@opengroup.orgUpgrade Core IBM Dependencyhttps://community.opengroup.org/osdu/platform/system/file/-/issues/32Upgrade and Fix Core Common Dependency2021-07-21T04:38:29ZDavid Diederichd.diederich@opengroup.orgUpgrade and Fix Core Common DependencyDavid Diederichd.diederich@opengroup.orgDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/system/file/-/issues/31Soft DELETE functionality for Files and Metadata associated with it2022-09-29T13:41:07ZParesh BehedeSoft DELETE functionality for Files and Metadata associated with itCurrently DELETE endpoint in File service hard deletes the files from file store, we must have ability to do soft delete as well so that if user deletes by mistake then we have ability to restore it back.Currently DELETE endpoint in File service hard deletes the files from file store, we must have ability to do soft delete as well so that if user deletes by mistake then we have ability to restore it back.https://community.opengroup.org/osdu/platform/system/file/-/issues/30Need for DELETE endpoint2021-06-29T09:41:44ZParesh BehedeNeed for DELETE endpointCurrently there is no way to delete already uploaded file by user from data platform, in case user uploads wrong file by mistake that file can not be deleted by user.
We must give ability to delete metadata record and file associated wi...Currently there is no way to delete already uploaded file by user from data platform, in case user uploads wrong file by mistake that file can not be deleted by user.
We must give ability to delete metadata record and file associated with that metadata record to user, so that user can delete file uploaded by him/her when ever its necessary.
New endpoint in File Service could be DELETE /v2/files/{id}/metadataM7 - Release 0.10Paresh BehedeParesh Behedehttps://community.opengroup.org/osdu/platform/system/file/-/issues/29ADR: Rationalizing File and File DMS services.2023-08-08T11:23:16ZKrishna Nikhil VedurumudiADR: Rationalizing File and File DMS services.# Decision Title
Rationalizing File and File DMS services.
## Status
* [x] Proposed
* [ ] Trialing
* [ ] Under review
* [x] Approved
* [ ] Retired
## Context & Scope
The Dataset service in its implementation of the DMS APIs (getStor...# Decision Title
Rationalizing File and File DMS services.
## Status
* [x] Proposed
* [ ] Trialing
* [ ] Under review
* [x] Approved
* [ ] Retired
## Context & Scope
The Dataset service in its implementation of the DMS APIs (getStorageInstructions and getRetrievalInstructions) delegates the call to specific DMS's implementation of the `get*Instructons` API. That lead to creation of new service called `File-DMS` which supports the APIs that Dataset Service requires.
> Note: The File-DMS does not have an approved ADR, and is not an approved by the community yet.
On the other hand, we already have a File Service that owns the responsibility for the management of files. Also, the design and APIs of the File service are approved by a thorough process of ADR.
So, instead of having two services who own the similar responsibility of managing files (File and File-DMS) we should rationalize both the services and move the APIs that Dataset Service requires to File Service itself. It would reduce the additional maintenance overhead of managing another service.
## Decision
The decision is to merge File and File DMS services and consolidate all the File management APIs in **File Service** keeping inline with the File Service ADR that was approved [File Service ADR](https://community.opengroup.org/osdu/platform/system/home/-/issues/47).
## Rationale
Having multiple services with similar functionalities and responsibilities is an additional overhead w.r.t maintenance. Since there is already one approved service for supporting Files, **the Additional APIs on Files required to support DMS Functionalities** should be hosted by File Service.
## Consequences
> The **DMS APIs** **getStorageInstructions** and **getRetrievalInstructions** for Files and File-Collections will be moved to File Service.
| Functionality | API to be Added in File Service | Existing API in File-DMS Service | Status | Capability |
|---------------------|-------------------------|-------------------------|--------|--------------------------------------------------|
| File Get-Storage Instructions | POST /files/storageInstructions | GET /file/getStorageInstructions | Moved from File-DMS Service to File Service | Generates the Signed URLs / temporary access tokens required to upload Files |
| File Get-Retrieval Instructions | POST /files/retrievalInstructions | POST /file/getRetrievalInstructions | Moved from File-DMS Service to File Service | Generates the Signed URLs / temporary access tokens required to download Files |
| File Collection Get-Storage Instructions | POST /file-collections/storageInstructions | GET /file-collection/getStorageInstructions | Moved from File-DMS Service to File Service | Generates the Signed URLs / temporary access tokens required to upload File Collections |
| File Collection Get-Retrieval Instructions | POST /file-collections/retrievalInstructions | POST /file-collection/getRetrievalInstructions | Moved from File-DMS Service to File Service | Generates the Signed URLs / temporary access tokens required to download File Collections |
### Pros of merging File DMS APIs into File Service
- Existing File Service clients will not be impacted because the existing APIs will continue to stay.
- The Core-Logic of handling File uploads and downloads already exist in File Service and are well tested. So, the DMS APIs that are required for [Dataset ADR](https://community.opengroup.org/osdu/platform/system/home/-/issues/57) will be able to directly hook into it reducing the development time.
- Features such as "Uploading to staging container" will be available out of the box. This reduces gaps between Dataset Architecture and existing File Service, which in-turn enables easier migration for clients.JoeMadhur Tanwani [Microsoft]Joehttps://community.opengroup.org/osdu/platform/system/file/-/issues/28Integration tests || Delivery API || Get Signed URL2021-05-18T07:39:23ZKrishna Nikhil VedurumudiIntegration tests || Delivery API || Get Signed URLThe integration tests in Delivery API are disabled [in this file](https://community.opengroup.org/osdu/platform/system/file/-/blob/master/testing/file-test-core/src/main/java/org/opengroup/osdu/file/apitest/Delivery.java)
Was there any...The integration tests in Delivery API are disabled [in this file](https://community.opengroup.org/osdu/platform/system/file/-/blob/master/testing/file-test-core/src/main/java/org/opengroup/osdu/file/apitest/Delivery.java)
Was there any prior context on why the tests were disabled? Seems like they were disabled from the initial commit.
**More context from our investigation**
For R3 schemas, the field which is used to identify record `data.ResourceID` does not exist. The R3 schemas use `id` field for identifying records. More context here - https://community.opengroup.org/osdu/platform/system/dataset/-/issues/15
The delivery's Get Signed URL API, to invoke Search API uses data.ResourceID (this part is hardcoded in the code) `"query":"data.ResourceID: (\"opendes:dataset--File.Generic:db44de91-d893-47ac-8d89-ce436c1794b2\")"` Because the R3 schemas does not have this field, the search query always returns empty result.
This means that the GetSignedURL API is broken for R3 schemas.https://community.opengroup.org/osdu/platform/system/file/-/issues/27File Service should support ingestion of multiple files using 1 request2022-09-27T11:37:18ZKateryna Kurach (EPAM)File Service should support ingestion of multiple files using 1 request(This issue came from R3 pre-shipping testing)
Currently the process to upload a file is the following:
- get {{file_api_url}}/v1/files/uploadURL
(user gets SignedURL)
- put {{upload_signed_url}}
(user uploads a file to the OSDU staging...(This issue came from R3 pre-shipping testing)
Currently the process to upload a file is the following:
- get {{file_api_url}}/v1/files/uploadURL
(user gets SignedURL)
- put {{upload_signed_url}}
(user uploads a file to the OSDU staging area)
- post {{file_api_url}}/v1/files/metadata
(file is moved from the OSDU staging area to OSDU persistent area and metadata record is created)
There is a business need to upload multiple different (not in the same file collection) files using the same SignedURL. That may simplify and speed up the ingestion process.https://community.opengroup.org/osdu/platform/system/file/-/issues/26API Documentation missing indication that file is moved from staging area to ...2023-05-15T10:32:38ZAlan HensonAPI Documentation missing indication that file is moved from staging area to persistent areaIn reviewing the [OpenAPI Spec](https://community.opengroup.org/osdu/platform/system/file/-/blob/master/docs/file-service_openapi.yaml) for File Service, members of the meeting noticed that the OpenAPI Spec documentation for the `/v1/fil...In reviewing the [OpenAPI Spec](https://community.opengroup.org/osdu/platform/system/file/-/blob/master/docs/file-service_openapi.yaml) for File Service, members of the meeting noticed that the OpenAPI Spec documentation for the `/v1/files/metadata` does not mention that the endpoint moves the file from a staging area to a persistent area. The request is as follows:
- Update the OpenAPI Spec documentation linked above to mention the file is moved from a staging area to a persistent area
- The code that performs that operation within the above endpoint is found [here](https://community.opengroup.org/osdu/platform/system/file/-/blob/master/file-core/src/main/java/org/opengroup/osdu/file/service/FileMetadataService.java#L66)https://community.opengroup.org/osdu/platform/system/file/-/issues/25File service does not check for invalid auth token2022-09-27T11:50:40ZRatneshFile service does not check for invalid auth tokenScenario: User hits File service's getFileLocation and getFileList API with invalid/expired auth token.
Expected response code: 401 (Unauthorized)
Actual response code: 400 (Bad request)Scenario: User hits File service's getFileLocation and getFileList API with invalid/expired auth token.
Expected response code: 401 (Unauthorized)
Actual response code: 400 (Bad request)https://community.opengroup.org/osdu/platform/system/file/-/issues/24File service download metadata end point issue2022-09-27T11:53:25ZRatneshFile service download metadata end point issueScenario: User hits File service GET metadata signed API with an Invalid Id (Say opendes:file:21107053-17bf-405e-9dd3-bb0a1e9b8e0aTEST).
Sample Request URL: https://os-file-attcrcktoa-uc.a.run.app/v1/files/opendes:file:21107053-17bf-405...Scenario: User hits File service GET metadata signed API with an Invalid Id (Say opendes:file:21107053-17bf-405e-9dd3-bb0a1e9b8e0aTEST).
Sample Request URL: https://os-file-attcrcktoa-uc.a.run.app/v1/files/opendes:file:21107053-17bf-405e-9dd3-bb0a1e9b8e0aTEST/metadata
Expected:-
{
"error": {
"code": 404,
"message": "Record Not Found",
"errors": [
{
"domain": "global",
"reason": "notFound",
"message": "Record Not Found"
}
]
}
}
Actual:-
{
"error": {
"code": 500,
"message": "Failed to find record for the given file id.",
"errors": [
{
"domain": "global",
"reason": "internalError",
"message": "Failed to find record for the given file id."
}
]
}
}