File issueshttps://community.opengroup.org/osdu/platform/system/file/-/issues2021-10-19T19:22:38Zhttps://community.opengroup.org/osdu/platform/system/file/-/issues/39Implement CSP specific changes for File DMS APIs support2021-10-19T19:22:38ZKrishna Nikhil VedurumudiImplement CSP specific changes for File DMS APIs supportFollow up for https://community.opengroup.org/osdu/platform/system/file/-/issues/29 where the Core and Azure changes for File DMS changes were completed.
Implementation Updates:
* Core Common changes with skeleton interfaces and model...Follow up for https://community.opengroup.org/osdu/platform/system/file/-/issues/29 where the Core and Azure changes for File DMS changes were completed.
Implementation Updates:
* Core Common changes with skeleton interfaces and model objects - https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/merge_requests/99
* File service changes with Core and Azure implementation for File DMS APIs - https://community.opengroup.org/osdu/platform/system/file/-/merge_requests/128
* Dataset Service changes consuming the APIs - https://community.opengroup.org/osdu/platform/system/dataset/-/merge_requests/109
* Entitlements changes - New role for dataset - https://community.opengroup.org/osdu/platform/security-and-compliance/entitlements/-/merge_requests/90
Work for Other CSPs
* Using the feature flag in Dataset Service, use the latest DMS API endpoints.
* Create new role in Entitlements -> https://community.opengroup.org/osdu/platform/security-and-compliance/entitlements/-/merge_requests/90
* File Service changes - Implement `final IStorageService storageService;` for interactions with CSP Specific Blob store providers.M9 - Release 0.12https://community.opengroup.org/osdu/platform/system/file/-/issues/38GET /DownloadURL api to support generating signedURL that can be accessible f...2022-09-29T13:41:08ZOrsu AkhilGET /DownloadURL api to support generating signedURL that can be accessible from select IP address.The /DownloadURL api would return signedURL to access file stored in persistent location based on File Metadata id .
Having the feature wherein User can specify the range of IP address , only through which when signedURL is used would re...The /DownloadURL api would return signedURL to access file stored in persistent location based on File Metadata id .
Having the feature wherein User can specify the range of IP address , only through which when signedURL is used would result in access to the file in persistent location would be good. This IP address or range of IP address can be passed by User via GET query Parameter in the api specification.
Supporting this sort of customization in generated signed url is possible in Azure Cloud and would like to know about this in other cloud providers , if is feasible in others as well then could come up with implementation supporting this feature.Orsu AkhilParesh BehedeOrsu Akhilhttps://community.opengroup.org/osdu/platform/system/file/-/issues/37GET /DownloadURL api to support generating signedURL with custom expiry time2021-08-27T21:29:39ZOrsu AkhilGET /DownloadURL api to support generating signedURL with custom expiry timeCurrently in cloud provider implementation of File Application for Azure , GCP the /DownloadURL api would return signedURL to access file stored in persistent location and the expiry time of this URL is by default set to 7 Days. This may...Currently in cloud provider implementation of File Application for Azure , GCP the /DownloadURL api would return signedURL to access file stored in persistent location and the expiry time of this URL is by default set to 7 Days. This mayn't be desirable and giving flexibility to User to determine when the URL should expire would be a good feature add-on for this api. This information can be passed by User via GET query Parameter in the api specification.M8 - Release 0.11Orsu AkhilParesh BehedeOrsu Akhilhttps://community.opengroup.org/osdu/platform/system/file/-/issues/36Request for HTTP header Content-Disposition (optionally) for file download2022-09-27T11:33:23ZSteven ReynoldsRequest for HTTP header Content-Disposition (optionally) for file downloadWe implemented this Download feature in our application, that will return a download URL that points to the OSDU download API, but the download behaviour in browsers is not consistent between csv and las files for instance. Some browsers...We implemented this Download feature in our application, that will return a download URL that points to the OSDU download API, but the download behaviour in browsers is not consistent between csv and las files for instance. Some browsers will actually trigger a download, others will open in a new tab. We would like for the user experience to be consistent regardless of the file type.
We had a look at it and it's up to the the platform to add the following http headers to these downloads http response, so that the browser knows it must initiate a file download :
header("Content-Disposition", "attachment; filename=myfilename.myextension");
Maybe we could also think about an extra parameter to the request to let the server know it must add this header.https://community.opengroup.org/osdu/platform/system/file/-/issues/35File should get download with actual filename and extension after hitting dow...2023-04-13T10:39:15Zsachin GuptaFile should get download with actual filename and extension after hitting downloadUrl### Problem Statement
Currently, the file service stores file in the persistent location with some random name and without extension. When we download the file using download SignedURL, it's download without extension and to see the co...### Problem Statement
Currently, the file service stores file in the persistent location with some random name and without extension. When we download the file using download SignedURL, it's download without extension and to see the content of the file we need to explicitly give an extension.
### Solution
We can overcome this problem by adding Content-Disposition and Content-Type header at the time of download signed URL creation. We can get file name from file metadata payload and based on file extension we can get content type of the file, both name and content type we can use to create download URL.
Note: the name field in the metadata payload is optional and if it is not present in the payload, then the above solution won't work for that and follow the current implementation of download URL creation.M9 - Release 0.12Paresh BehedeParesh Behedehttps://community.opengroup.org/osdu/platform/system/file/-/issues/34Upgrade Core Azure Dependency2022-02-11T22:00:53ZDavid Diederichd.diederich@opengroup.orgUpgrade Core Azure Dependencyhttps://community.opengroup.org/osdu/platform/system/file/-/issues/33Upgrade Core IBM Dependency2022-02-11T22:01:07ZDavid Diederichd.diederich@opengroup.orgUpgrade Core IBM Dependencyhttps://community.opengroup.org/osdu/platform/system/file/-/issues/32Upgrade and Fix Core Common Dependency2021-07-21T04:38:29ZDavid Diederichd.diederich@opengroup.orgUpgrade and Fix Core Common DependencyDavid Diederichd.diederich@opengroup.orgDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/system/file/-/issues/31Soft DELETE functionality for Files and Metadata associated with it2022-09-29T13:41:07ZParesh BehedeSoft DELETE functionality for Files and Metadata associated with itCurrently DELETE endpoint in File service hard deletes the files from file store, we must have ability to do soft delete as well so that if user deletes by mistake then we have ability to restore it back.Currently DELETE endpoint in File service hard deletes the files from file store, we must have ability to do soft delete as well so that if user deletes by mistake then we have ability to restore it back.https://community.opengroup.org/osdu/platform/system/file/-/issues/30Need for DELETE endpoint2021-06-29T09:41:44ZParesh BehedeNeed for DELETE endpointCurrently there is no way to delete already uploaded file by user from data platform, in case user uploads wrong file by mistake that file can not be deleted by user.
We must give ability to delete metadata record and file associated wi...Currently there is no way to delete already uploaded file by user from data platform, in case user uploads wrong file by mistake that file can not be deleted by user.
We must give ability to delete metadata record and file associated with that metadata record to user, so that user can delete file uploaded by him/her when ever its necessary.
New endpoint in File Service could be DELETE /v2/files/{id}/metadataM7 - Release 0.10Paresh BehedeParesh Behedehttps://community.opengroup.org/osdu/platform/system/file/-/issues/28Integration tests || Delivery API || Get Signed URL2021-05-18T07:39:23ZKrishna Nikhil VedurumudiIntegration tests || Delivery API || Get Signed URLThe integration tests in Delivery API are disabled [in this file](https://community.opengroup.org/osdu/platform/system/file/-/blob/master/testing/file-test-core/src/main/java/org/opengroup/osdu/file/apitest/Delivery.java)
Was there any...The integration tests in Delivery API are disabled [in this file](https://community.opengroup.org/osdu/platform/system/file/-/blob/master/testing/file-test-core/src/main/java/org/opengroup/osdu/file/apitest/Delivery.java)
Was there any prior context on why the tests were disabled? Seems like they were disabled from the initial commit.
**More context from our investigation**
For R3 schemas, the field which is used to identify record `data.ResourceID` does not exist. The R3 schemas use `id` field for identifying records. More context here - https://community.opengroup.org/osdu/platform/system/dataset/-/issues/15
The delivery's Get Signed URL API, to invoke Search API uses data.ResourceID (this part is hardcoded in the code) `"query":"data.ResourceID: (\"opendes:dataset--File.Generic:db44de91-d893-47ac-8d89-ce436c1794b2\")"` Because the R3 schemas does not have this field, the search query always returns empty result.
This means that the GetSignedURL API is broken for R3 schemas.https://community.opengroup.org/osdu/platform/system/file/-/issues/27File Service should support ingestion of multiple files using 1 request2022-09-27T11:37:18ZKateryna Kurach (EPAM)File Service should support ingestion of multiple files using 1 request(This issue came from R3 pre-shipping testing)
Currently the process to upload a file is the following:
- get {{file_api_url}}/v1/files/uploadURL
(user gets SignedURL)
- put {{upload_signed_url}}
(user uploads a file to the OSDU staging...(This issue came from R3 pre-shipping testing)
Currently the process to upload a file is the following:
- get {{file_api_url}}/v1/files/uploadURL
(user gets SignedURL)
- put {{upload_signed_url}}
(user uploads a file to the OSDU staging area)
- post {{file_api_url}}/v1/files/metadata
(file is moved from the OSDU staging area to OSDU persistent area and metadata record is created)
There is a business need to upload multiple different (not in the same file collection) files using the same SignedURL. That may simplify and speed up the ingestion process.https://community.opengroup.org/osdu/platform/system/file/-/issues/26API Documentation missing indication that file is moved from staging area to ...2023-05-15T10:32:38ZAlan HensonAPI Documentation missing indication that file is moved from staging area to persistent areaIn reviewing the [OpenAPI Spec](https://community.opengroup.org/osdu/platform/system/file/-/blob/master/docs/file-service_openapi.yaml) for File Service, members of the meeting noticed that the OpenAPI Spec documentation for the `/v1/fil...In reviewing the [OpenAPI Spec](https://community.opengroup.org/osdu/platform/system/file/-/blob/master/docs/file-service_openapi.yaml) for File Service, members of the meeting noticed that the OpenAPI Spec documentation for the `/v1/files/metadata` does not mention that the endpoint moves the file from a staging area to a persistent area. The request is as follows:
- Update the OpenAPI Spec documentation linked above to mention the file is moved from a staging area to a persistent area
- The code that performs that operation within the above endpoint is found [here](https://community.opengroup.org/osdu/platform/system/file/-/blob/master/file-core/src/main/java/org/opengroup/osdu/file/service/FileMetadataService.java#L66)https://community.opengroup.org/osdu/platform/system/file/-/issues/25File service does not check for invalid auth token2022-09-27T11:50:40ZRatneshFile service does not check for invalid auth tokenScenario: User hits File service's getFileLocation and getFileList API with invalid/expired auth token.
Expected response code: 401 (Unauthorized)
Actual response code: 400 (Bad request)Scenario: User hits File service's getFileLocation and getFileList API with invalid/expired auth token.
Expected response code: 401 (Unauthorized)
Actual response code: 400 (Bad request)https://community.opengroup.org/osdu/platform/system/file/-/issues/24File service download metadata end point issue2022-09-27T11:53:25ZRatneshFile service download metadata end point issueScenario: User hits File service GET metadata signed API with an Invalid Id (Say opendes:file:21107053-17bf-405e-9dd3-bb0a1e9b8e0aTEST).
Sample Request URL: https://os-file-attcrcktoa-uc.a.run.app/v1/files/opendes:file:21107053-17bf-405...Scenario: User hits File service GET metadata signed API with an Invalid Id (Say opendes:file:21107053-17bf-405e-9dd3-bb0a1e9b8e0aTEST).
Sample Request URL: https://os-file-attcrcktoa-uc.a.run.app/v1/files/opendes:file:21107053-17bf-405e-9dd3-bb0a1e9b8e0aTEST/metadata
Expected:-
{
"error": {
"code": 404,
"message": "Record Not Found",
"errors": [
{
"domain": "global",
"reason": "notFound",
"message": "Record Not Found"
}
]
}
}
Actual:-
{
"error": {
"code": 500,
"message": "Failed to find record for the given file id.",
"errors": [
{
"domain": "global",
"reason": "internalError",
"message": "Failed to find record for the given file id."
}
]
}
}https://community.opengroup.org/osdu/platform/system/file/-/issues/22File service record id generation2022-09-30T11:54:43Zethiraj krishnamanaiduFile service record id generationWe have logic in File service to validate and generate the RecordId, this duplicate logic that is already in storage service. We should let storage service generated RecordId and validation.We have logic in File service to validate and generate the RecordId, this duplicate logic that is already in storage service. We should let storage service generated RecordId and validation.https://community.opengroup.org/osdu/platform/system/file/-/issues/21Inconsistent URL versioning2021-03-05T09:56:56ZKishore BattulaInconsistent URL versioning# Context & Scope
The following endpoints have both v1/v2 in their path which is not a clear API versioning
- /api/file/v2/v1/files/{id}/downloadURL
- /api/file/v2/v1/files/uploadURL
- /api/file/v2/v1/files/metadata
- /api/file/v2/v1/fi...# Context & Scope
The following endpoints have both v1/v2 in their path which is not a clear API versioning
- /api/file/v2/v1/files/{id}/downloadURL
- /api/file/v2/v1/files/uploadURL
- /api/file/v2/v1/files/metadata
- /api/file/v2/v1/files/{id}/metadata
# Decision
Change the Above APIs respectively
- /api/file/v1/files/{id}/downloadURL
- /api/file/v1/files/uploadURL
- /api/file/v1/files/metadata
- /api/file/v1/files/{id}/metadata
# Rational
Consistent and clear API versioning.
# Consequences
Users of these APIs need to change the paths for above APIs
# Impact
The changes would be in core module and no needed for CSPs to change any codehttps://community.opengroup.org/osdu/platform/system/file/-/issues/20Metadata request payload schema not aligned to API spec2022-09-28T03:11:27Zdevesh bajpaiMetadata request payload schema not aligned to API specMetadata request payload is not aligned to api spec
https://community.opengroup.org/osdu/platform/system/file/-/blob/master/docs/file-service_openapi.yamlMetadata request payload is not aligned to api spec
https://community.opengroup.org/osdu/platform/system/file/-/blob/master/docs/file-service_openapi.yamlhttps://community.opengroup.org/osdu/platform/system/file/-/issues/19Master build failing on local machine2021-06-16T22:20:35ZRatneshMaster build failing on local machineI am getting compilation error while trying to build master branch code on my local. The issue is in referring to file 'ReplaceCamelCase'. But it is being tried to be referenced from jar of 'file-test-core' module, which is not even buil...I am getting compilation error while trying to build master branch code on my local. The issue is in referring to file 'ReplaceCamelCase'. But it is being tried to be referenced from jar of 'file-test-core' module, which is not even built as part of build process.
To mitigate this I added this file under each provider module and was then able to build the project. Have created a branch 'BuildIssueFix_OnTopOfBranch_UnifiedIT' with the fix.
@akumar290 @devesh @amavermaethiraj krishnamanaiduethiraj krishnamanaiduhttps://community.opengroup.org/osdu/platform/system/file/-/issues/18If partitonId == null, a endpoint will return 403 status instead of 401.2020-11-13T12:25:12ZRiabokon Stanislav(EPAM)[GCP]If partitonId == null, a endpoint will return 403 status instead of 401.This problem has been identified when this MR https://community.opengroup.org/osdu/platform/system/file/-/commit/4ddd2e7228252edef7474ecad0c86d31c3c3598e had been merged into master
commit: https://community.opengroup.org/osdu/platform/...This problem has been identified when this MR https://community.opengroup.org/osdu/platform/system/file/-/commit/4ddd2e7228252edef7474ecad0c86d31c3c3598e had been merged into master
commit: https://community.opengroup.org/osdu/platform/system/file/-/commit/4ddd2e7228252edef7474ecad0c86d31c3c3598e#7e5fc265ef88187dd3731e9bcceee01ff99f7569_44_44
A new behavior "If partitonId == null, a endpoint will return 403 status instead of 401"
I am not sure that it is a correct response from an architecture perspective.
cc: @Dmitriy_Rudko @rostislav.dublinRiabokon Stanislav(EPAM)[GCP]Riabokon Stanislav(EPAM)[GCP]