File issueshttps://community.opengroup.org/osdu/platform/system/file/-/issues2023-07-28T09:39:01Zhttps://community.opengroup.org/osdu/platform/system/file/-/issues/90Fixing sonar quality issues2023-07-28T09:39:01ZGauri ChitaleFixing sonar quality issuesSonarqube analysis has reported multiple code smells in the file service code.Sonarqube analysis has reported multiple code smells in the file service code.https://community.opengroup.org/osdu/platform/system/file/-/issues/89Unable to download the file2023-08-08T16:30:43ZSachin JaiswalUnable to download the file**Problem**
File download fails if file name contains the special character like ","
**Steps to reproduce**
- Generate a signedurl to upload a file (tesing,copy).
- Upload a file
- create a file metadata and use same filename (tesing,...**Problem**
File download fails if file name contains the special character like ","
**Steps to reproduce**
- Generate a signedurl to upload a file (tesing,copy).
- Upload a file
- create a file metadata and use same filename (tesing,copy)
- Generate a signedurl to download a file
- Copy and paste the signedurl in browser
**Error notice on bowser** - `<storage-url> sent an invalid response. ERR_RESPONSE_HEADERS_MULTIPLE_CONTENT_DISPOSITION `
**Proposed Solutions**
- File service - wrap quotes to file name before passing it to OS Core Lib Azure **OR**
- OS Core Lib Azure - Below changes in [BlobStore class](https://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/blob/master/src/main/java/org/opengroup/osdu/azure/blobstorage/BlobStore.java#:~:text=581-%2cblobServiceSasSignatureValues.setContentDisposition%28%22attachment%3b%20filename%3D%20%22%20%2B%20fileName%29%3b%2c-582) of OS Core Lib Azure repo.
`blobServiceSasSignatureValues.setContentDisposition("attachment; filename=\"" + fileName + "\"")`;https://community.opengroup.org/osdu/platform/system/file/-/issues/88[ADR] Dataset service security enhancments2023-07-10T10:43:58ZOm Prakash Gupta[ADR] Dataset service security enhancments# Decision Title
Security Enhancements for Dataset Service's Signed URL APIs
## Status
- [X] Proposed
- [ ] Trialing
- [ ] Under review
- [ ] Approved
- [ ] Retired
## Context & Scope
A customer has voiced a security concern about Fi...# Decision Title
Security Enhancements for Dataset Service's Signed URL APIs
## Status
- [X] Proposed
- [ ] Trialing
- [ ] Under review
- [ ] Approved
- [ ] Retired
## Context & Scope
A customer has voiced a security concern about File Service's `POST GetStorageInstructions` and `POST GetReterievalInstructions` APIs under the scenario of a malicious user getting hold of the generated signed URLs and using them to access files from storage. When Private Link is not a desired option to mitigate these concerns for the customer due to policy and deployment complexity reasons, the following enhancements are proposed to the two existing APIs and introducing a new API to alleviate the customer's security concerns.
## Decision
### Proposed Changes
1. For `POSTS GetStorageInstructions` API: Change default TTL from 7 days to 1 hour and make TTL configurable through a query Paramater `expiryTime` in Time Units Minutes,Hours,Days. The expiry time is capped at 7 Days if the time provided by the User exceeds the capped value. In absence of this parameter, the Signed URL would be valid for 1 Hour by default.
2. For `POST GetReterievalInstructions` API: Change default TTL from 7 days to 1 hour. and make TTL configurable through a query Paramater `expiryTime` in Time Units Minutes,Hours,Days. The expiry time is capped at 7 Days if the time provided by the User exceeds the capped value.
These two changes make the two APIs behave consistently also.
3. New API to revoke all Signed URLs generated for a specified storage account. Storage account is specified through a query parameter `storageAccount`. User can grab the storageAccount from the `GetReterievalInstructions` or `GetStorageInstructions` response.
POST api/Dateset/v1/revokeURLs
This API will use the `StorageAccountRevokeUserDelegationKeys` to revoke all the User Delegation Keys for the storage account and that will revoke all the User Delegation SAS tokens and thus invalidate all the Signed URLs.
4. Start using user-defined delegation keys for storage accounts rather than using storage account keys.
## Rationale
Shortened TTL for the Signed URLs decreases the Window of opportunities for a malicious user to use the Sighed URLs to access any sensitive information; Additional Revoking API provides customers a capability to mitigate the risk at the earliest moment if Signed URL leaking is detected.
## Consequences
**Caution**: SAS token in a Signed URL cannot be individually revoked. This API will revoke all SAS tokens generated and invalidate all signed URLs for that storage account. A user needs to send `GET uploadURL` and `GET downloadURL` requests again to generate new URLs. It should only be used when the customer knows for sure a signed URL has been compromised.
**Caution**: User Delegation Keys are cached by Azure Storage, so there may be a delay between when the user initiates the process of revocation and when an existing user delegation SAS becomes invalid. So after calling `POST revokeURLs`, wait for sometime and verify the compromised URL no longer works before sending `GET uploadURL` and `GET downloadURL` requests again.
These cautions need to be included in the file service open API spec and be communicated to customers clearly.
## Backward Compatibility
This is NOT a breaking change.https://community.opengroup.org/osdu/platform/system/file/-/issues/87Apply role-based access to File V2 endpoints.2023-08-07T11:13:22ZRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comApply role-based access to File V2 endpoints.File V2/DMS API doesn't use Authorization filters (@PreAuthorize), and doesn't evaluate the roles of the requester, which could lead to data leaks.
Also, it was marked as Hidden but this rule was not applied to the Infra level automatic...File V2/DMS API doesn't use Authorization filters (@PreAuthorize), and doesn't evaluate the roles of the requester, which could lead to data leaks.
Also, it was marked as Hidden but this rule was not applied to the Infra level automatically.
https://community.opengroup.org/osdu/platform/system/file/-/blob/master/file-core/src/main/java/org/opengroup/osdu/file/api/FileDmsApi.java#L57
Potential issues:
- If not closed from Istio, data leaks are possible.
- Even if closed from the outside, authorization of internal requests will not be evaluated.M19 - Release 0.22Oleksandr Kosse (EPAM)Riabokon Stanislav(EPAM)[GCP]Andrei Dalhikh [EPAM/GC]Oleksandr Kosse (EPAM)https://community.opengroup.org/osdu/platform/system/file/-/issues/86/api/file/v2/files bugs2023-09-26T14:22:03ZShane Hutchins/api/file/v2/files bugsReceived a response with 5xx status code: 500. I expected a 404 from these APIs but got 500 Internal Server Error.
Run this curl command to reproduce this failure:
GET /api/file/v2/files/%0A/metadata:
curl -X GET -H 'Authorization...Received a response with 5xx status code: 500. I expected a 404 from these APIs but got 500 Internal Server Error.
Run this curl command to reproduce this failure:
GET /api/file/v2/files/%0A/metadata:
curl -X GET -H 'Authorization: Bearer TOKEN' -H 'data-partition-id: osdu' https://osdu.r3m18.preshiptesting.osdu.aws/api/file/v2/files/%0A/metadata
DELETE /api/file/v2/files/%3B/metadata:
curl -X DELETE -H 'Authorization: Bearer TOKEN' -H 'data-partition-id: osdu' https://osdu.r3m18.preshiptesting.osdu.aws/api/file/v2/files/%3B/metadata
GET /api/file/v2/files/%3B/downloadURL:
curl -X GET -H 'Authorization: Bearer TOKEN' -H 'data-partition-id: osdu' https://osdu.r3m18.preshiptesting.osdu.aws/api/file/v2/files/%3B/downloadURL
Confirmed this issue on AWS and Azure.
curl -X GET -H 'Authorization: Bearer TOKEN' -H 'Cookie: JSESSIONID=SESSIONIDHERE' -H 'data-partition-id: opendes' https://osdu-ship.msft-osdu-test.org/api/file/v2/files/%0A/metadatahttps://community.opengroup.org/osdu/platform/system/file/-/issues/85Duplicate PreloadFilePath value on ingestion of File.Generics2023-06-16T13:17:04ZGorm-Erik AarsheimDuplicate PreloadFilePath value on ingestion of File.GenericsWhen ingesting File.Generics into OSDU, we are using the {OSDU_BASE_URL}/file/v2/files/metadata endpoint to create the File.Generic. In the body we are setting the PreloadFilePath value on data.DatasetProperties.FileSourceInfo.PreloadFil...When ingesting File.Generics into OSDU, we are using the {OSDU_BASE_URL}/file/v2/files/metadata endpoint to create the File.Generic. In the body we are setting the PreloadFilePath value on data.DatasetProperties.FileSourceInfo.PreloadFilePath as described in the schema. The file is created successfully but when we read out the record from Storage API there is a duplicate value for PreloadFilePath where it has both 'PreloadFilePath' with a capital 'P', and 'preloadFilePath' with a non-capital 'P'. We've tried to ingest a file with a non-capital 'preloadFilePath' as well, but it still comes out with both values when reading it back. We have also tried generating the File.Generic raw json object from our code and using this to create it directly from the metadata endpoint in Postman to disclose that our code is somehow meddeling with the object. This also comes out with the same result. It's also worth mentioning that we are running OSDU through Microsoft ADME.Om Prakash GuptaOm Prakash Guptahttps://community.opengroup.org/osdu/platform/system/file/-/issues/84File Service rejects Record ID from dataset--File.Generic manifest and genera...2023-06-09T15:27:51ZSamiullah GhousudeenFile Service rejects Record ID from dataset--File.Generic manifest and generate always GUID as record id**File Service rejects Record ID from dataset--File.Generic manifest and generate always GUID as record id**
Please check below CURL request to register metadata record for file dataset, I have included ID in the manifest, however all t...**File Service rejects Record ID from dataset--File.Generic manifest and generate always GUID as record id**
Please check below CURL request to register metadata record for file dataset, I have included ID in the manifest, however all time it rejects this id and create system generated GUID for this registered record.
**{+ Create File metadata - Request +}**
curl --location 'https://osdu-ship.msft-osdu-test.org/api/file/v2/files/metadata' \
--header 'Data-Partition-Id: opendes' \
--header 'Authorization: Bearer ...' \
--header 'Content-Type: application/json' \
--header 'Cookie: JSESSIONID=F4452A7D9F8752E8A82DC6E354D29B26' \
--data-raw '{
"kind": "osdu:wks:dataset--File.Generic:1.0.0",
{+ "id":"opendes:dataset--File.Generic:sami-test1",+}
"acl": {
"viewers": [
"data.default.viewers@opendes.contoso.com"
],
"owners": [
"data.default.viewers@opendes.contoso.com"
]
},
"legal": {
"legaltags": [
"opendes-Test-Legal-Tag-4766549"
],
"otherRelevantDataCountries": [
"US"
],
"status": "compliant"
},
"data": {
"Endian": "BIG",
"DatasetProperties": {
"FileSourceInfo": {
"FileSource": "/osdu-user/1686225883215-2023-06-08-12-04-43-215/4a62ec123d43427e93af2a4a1c515a6b"
}
}
}
}'
**{+ Create File metadata - Response +}**
{
{+"id": "opendes:dataset--File.Generic:d5c226d6-3eb2-4825-8b9b-e0834d0464cb"+}
}
cc- @todaiks @debasisc @chadhttps://community.opengroup.org/osdu/platform/system/file/-/issues/83Checksum values do not match up - value prior to upload, value as auto-popula...2023-07-10T08:37:54ZDebasis ChatterjeeChecksum values do not match up - value prior to upload, value as auto-populated by File service and persisted in Dataset recordPlease see my recent test in Azure/M17/Preship.
https://community.opengroup.org/osdu/platform/pre-shipping/-/blob/main/R3-M17/Test_Plan_Results_M17/Core_Services/M17-Azuere-Core-File-and-Dataset-steps-Debasis.zip
Prior to uploading the ...Please see my recent test in Azure/M17/Preship.
https://community.opengroup.org/osdu/platform/pre-shipping/-/blob/main/R3-M17/Test_Plan_Results_M17/Core_Services/M17-Azuere-Core-File-and-Dataset-steps-Debasis.zip
Prior to uploading the file, I found checksum value from Linux Operating system.
After the file is uploaded and Dataset record is created, I try to compare with the value as auto-populated by File Service.
The values do not match.
Please check this.M19 - Release 0.22Chad LeongChad Leonghttps://community.opengroup.org/osdu/platform/system/file/-/issues/82File Services Context Path2023-03-28T13:03:00ZThulasi Dass SubramanianFile Services Context PathCurrent File service settings
- context path: `server.servlet.contextPath=/api/file/`
- Endpoints: eg for downloadURL operation: `/v2/files/{id}/downloadURL
1. All API Endpoints have **/v2/** prefixed to the **RequestMapping** Path (_Sc...Current File service settings
- context path: `server.servlet.contextPath=/api/file/`
- Endpoints: eg for downloadURL operation: `/v2/files/{id}/downloadURL
1. All API Endpoints have **/v2/** prefixed to the **RequestMapping** Path (_Screenshot attached below_)
1. To be consistent for **swagger ui** and **api-docs** path customization and also to make easy versioning of the API,
_Can we add '**v2**' to the context path and remove it from all endpoints_?
- context path: `server.servlet.contextPath=/api/file/v2`
- Endpoints: eg for downloadURL operation: `/files/{id}/downloadURL`
**Consequences:**
- There will not be any changes w.r.t to Consumers of the API
- Its internal refactoring of the maintenance of the Base Path and Version
CSP can provide their inputs if we see any breaking changes or any settings need to be updated.
![image](/uploads/9878bb676457816652e6760868d73002/image.png)M17 - Release 0.20Thulasi Dass SubramanianThulasi Dass Subramanianhttps://community.opengroup.org/osdu/platform/system/file/-/issues/81Test cases commented FileFlowTest.java - Core module2022-11-22T14:31:27ZPramesh PatilTest cases commented FileFlowTest.java - Core moduleall test cases of FileFlowTest.java got commented and recent M14 someone added annotation to suppress this i.e. `
@SuppressWarnings("java:S2187") // there is no test cases in this class at present
Any specific reason for comment in th...all test cases of FileFlowTest.java got commented and recent M14 someone added annotation to suppress this i.e. `
@SuppressWarnings("java:S2187") // there is no test cases in this class at present
Any specific reason for comment in the FileFlowTest class.Pramesh PatilPramesh Patilhttps://community.opengroup.org/osdu/platform/system/file/-/issues/80ADR: Leverage File Service for Storage Operations2023-07-05T09:43:17ZElizabeth HalperADR: Leverage File Service for Storage Operations# Introduction
## Status
- [x] Initiated
- [x] Proposed
- [x] Under Review
- [ ] Approved
- [ ] Rejected
## Decision
All DMS's will leverage the File Service as a layer between the DMS and storage. Additionally, the File Service will...# Introduction
## Status
- [x] Initiated
- [x] Proposed
- [x] Under Review
- [ ] Approved
- [ ] Rejected
## Decision
All DMS's will leverage the File Service as a layer between the DMS and storage. Additionally, the File Service will provide methods through the exposed service interface to move data on different storage tiers for all DMS's in the most cost-effective manner based on how it is being used. File Service will provide an abstraction over all storage actions, including calls to the partition service. Therefore, this will not need to be implemented in services that don't have that functionality yet. All DDMS's will use File Service for the storage of binary data, and other services will be able to leverage File Service as well.
This decision is a proposed solution to the rejection of [this ADR](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-service/-/issues/39)
![DataStorageFlow](/uploads/92dbaeae4a51fa893e59e8ff2cb7934c/DataStorageFlow.png)
### Add File Service Endpoints
Provide a new utility endpoint to retrieve the list of supported storage tiers. We will need to discuss how implement an endpoint that will return the list of supported storage tiers. Additionally, other functionality, such as finer-grained access which is required for SDMS storage procedures, will need to be included in File Service.
### Refactor DMS Dataset Requests
Instead of directly loading in the Cosmos client library to each DMS, we will send the REST requests above to the file service to add datasets to the database.
## Rationale
We want all capabilities regarding storage to be available for all DMS's with the smallest amount of variation possible. Additionally, by implementing these features once in one service (File Service), the community will save a lot of time because other services will not need to change when storage features change. The example we use above is storage tiers. Although this requires an initial investment in refactoring services to leverage File Service, we will ultimately be able to implement storage tiers for all DMS's without much change to the services themselves.
## Consequences
We will need to:
- Add this additional functionality to the File Service
- "Onboard" services to using the File Service for all their storage actions
- Refactor all services to make REST requests to File Service as opposed to directly using the library
- We would need to enforce uniformity of requests given different services will be adding storage tier to their models
These tasks take a lot of time and effort as well as collaboration across many parties. We will need all CSV's and ISV's to support this motion and contribute to ensuring all services are compliant with this decision.Elizabeth HalperElizabeth Halperhttps://community.opengroup.org/osdu/platform/system/file/-/issues/79File ci cd pipelines do not use file-test-core-bdd with vital test cases.2022-11-04T10:15:34ZRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comFile ci cd pipelines do not use file-test-core-bdd with vital test cases.There is a BDD tests defined in the File testing module: <br/>
https://community.opengroup.org/osdu/platform/system/file/-/tree/master/testing/file-test-core-bdd <br/>
They get test case updates with new feature introductions, for exampl...There is a BDD tests defined in the File testing module: <br/>
https://community.opengroup.org/osdu/platform/system/file/-/tree/master/testing/file-test-core-bdd <br/>
They get test case updates with new feature introductions, for example: <br/>
https://community.opengroup.org/osdu/platform/system/file/-/merge_requests/138/diffs#d67c53013c6814c8d874d0daf0cffc9179ad1d00 <br/>
But they are not used in cicd pipelines, which left those features not cowered. <br/>
And it looks like because of ignoring them for a long time, tests have some compatibility issues which leads to runtime errors like
~~~
java.lang.NoClassDefFoundError: Could not initialize class io.restassured.RestAssured
at org.opengroup.osdu.file.util.test.RestAssuredClient.<init>(RestAssuredClient.java:30)
at org.opengroup.osdu.file.util.test.HttpClientFactory.getInstance(HttpClientFactory.java:8)
at org.opengroup.osdu.file.stepdefs.FileStepDef_GET.lambda$new$1(FileStepDef_GET.java:76)
~~~
Keeping them ignored may cause issues with feature introduction and verification. <br/>
There are several possible solutions: <br/>
- Fix and enable file-test-core-bdd tests in the integration step
- Copy missing tests from to file-test-CSP_PROVIDER_MODULERustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comhttps://community.opengroup.org/osdu/platform/system/file/-/issues/78ADR: Security Enhancements for File Service's Signed URL APIs2023-11-29T09:58:29ZLucy LiuADR: Security Enhancements for File Service's Signed URL APIs
# Decision Title
Security Enhancements for File Service's Signed URL APIs
## Status
- [X] Proposed
- [x] Trialing
- [x] Under review
- [x] Approved
- [ ] Retired
## Context & Scope
A customer has voiced a security concern about File...
# Decision Title
Security Enhancements for File Service's Signed URL APIs
## Status
- [X] Proposed
- [x] Trialing
- [x] Under review
- [x] Approved
- [ ] Retired
## Context & Scope
A customer has voiced a security concern about File Service's `GET uploadURL` and `GET downloadURL` APIs under the scenario of a malicious user getting hold of the generated signed URLs and using them to access files from storage. When Private Link is not a desired option to mitigate these concerns for the customer due to policy and deployment complexity reasons, the following enhancements are proposed to the two existing APIs and introducing a new API to alleviate the customer's security concerns.
## Decision
### Proposed Changes
1. For `GET uploadURL` API: Change default TTL from 7 days to 1 hour and make TTL configurable through a query Paramater `expiryTime` in Time Units Minutes,Hours,Days. The expiry time is capped at 7 Days if the time provided by the User exceeds the capped value. In absence of this parameter, the Signed URL would be valid for 1 Hour by default.
2. For `GET downloadURL` API: Change default TTL from 7 days to 1 hour. TTL is already configurable through the query Paramater `expiryTime`.
These two changes make the two APIs behave consistently also.
3. New API to revoke all Signed URLs generated for a specified storage account. Storage account is specified through a query parameter `storageAccount`. User can grab the storageAccount from the `GET uploadURL` or `GET downloadURL` response.
POST api/file/v2/files/revokeURLs
This API will use the `StorageAccountRevokeUserDelegationKeys` to revoke all the User Delegation Keys for the storage account and that will revoke all the User Delegation SAS tokens and thus invalidate all the Signed URLs.
## Rationale
Shortened TTL for the Signed URLs decreases the Window of opportunities for a malicious user to use the Sighed URLs to access any sensitive information; Additional Revoking API provides customers a capability to mitigate the risk at the earliest moment if Signed URL leaking is detected.
## Consequences
**Caution**: SAS token in a Signed URL cannot be individually revoked. This API will revoke all SAS tokens generated and invalidate all signed URLs for that storage account. A user needs to send `GET uploadURL` and `GET downloadURL` requests again to generate new URLs. It should only be used when the customer knows for sure a signed URL has been compromised.
**Caution**: User Delegation Keys are cached by Azure Storage, so there may be a delay between when the user initiates the process of revocation and when an existing user delegation SAS becomes invalid. So after calling `POST revokeURLs`, wait for sometime and verify the compromised URL no longer works before sending `GET uploadURL` and `GET downloadURL` requests again.
These cautions need to be included in the file service open API spec and be communicated to customers clearly.
## Backward Compatibility
This is NOT a breaking change.M18 - Release 0.21Om Prakash GuptaOm Prakash Guptahttps://community.opengroup.org/osdu/platform/system/file/-/issues/77Concurrent file metadata create calls with same file source path results in 5002022-10-03T15:49:23Zharshit aggarwalConcurrent file metadata create calls with same file source path results in 500During file metadata create flow if concurrent requests with same file source path are made it results in 500 errors, reason being the file gets cleaned up from staging location hence exceptions thrown at cloudStorageOperation.deleteFile...During file metadata create flow if concurrent requests with same file source path are made it results in 500 errors, reason being the file gets cleaned up from staging location hence exceptions thrown at cloudStorageOperation.deleteFile(stagingLocation) are handled as generic internal server errors which is not the correct response code
CSPs can handle exceptions in their implementation to throw 4xx exceptionshttps://community.opengroup.org/osdu/platform/system/file/-/issues/76POST files/metadata Re-try Failure due to Staging File being Deleted Pre-matu...2023-03-09T15:24:22ZLucy LiuPOST files/metadata Re-try Failure due to Staging File being Deleted Pre-maturelyAn issue was observed in POST files/metadata retries during MSFT use of this API in M12: retries performed for a failed POST files/metadata API are likely to result in 400s errors no matter how many times retry is performed. Further inve...An issue was observed in POST files/metadata retries during MSFT use of this API in M12: retries performed for a failed POST files/metadata API are likely to result in 400s errors no matter how many times retry is performed. Further investigation shows the root cause is that when metadata creation failed, staging file is also deleted. Thus subsequent retries with the same source file ID that mapped to the deleted staging file will result in failure. The staging file should not be deleted pre-maturely if metadata creation failed.
Current workaround is to perform the extra two steps to upload the file to staging again and then retry POST files/metadata:
1. Get a signed URL by calling File location API
2. Upload File to blob storage using signed url
3. Create the metadata using POST Metadata API
Suggested fix:
1. In FileMetadataService::saveMetadata, move the deleteStagingFile step to the last step right before successful return. So that staging file will only be deleted when everything succeeds.
2. Check staging file existence before deleting. Catch and ignore exceptions thrown from staging file delete. Staging file deletion failure is very rare but could happen under special concurrency situations: simultaneous calls for Metadata create with same payload results to one of the delete failure because file already deleted by the other caller. Failed staging file deletion should not invalidate successful metadata creation.M17 - Release 0.20Chad LeongChad Leonghttps://community.opengroup.org/osdu/platform/system/file/-/issues/75Need to bypass checksum generation for file size more than 5 gbs2022-12-12T15:21:28ZParesh BehedeNeed to bypass checksum generation for file size more than 5 gbsWhile user makes a call to POST /metadata api endpoint for registering file on data platform, before saving that as record, file service generates checksum of file provided in request to help duplicate detection for further downstream wo...While user makes a call to POST /metadata api endpoint for registering file on data platform, before saving that as record, file service generates checksum of file provided in request to help duplicate detection for further downstream workflows.
As this is HTTP blocking call checksum calculation takes quite a long if file size is huge (like more than 3-5 gbs) and HTTP post call gets hang and never respond.
**We have tested checksum generation and metadata registration takes about 2 mins for file size of 5 GB.**
We have experienced this when one of the user tried uploading file size of 14 GBs.
Though percentage of such a huge file being uploaded is quite low we still need to allow them to register metadata and to enable that we must bypass checksum generation logic for such a huge file sizes.
By doing this we still enable duplicate detection ability (by calculating and saving file checksum in storage record) for majority of files uploaded like for 95% of the files and we ignore that for 5% of the request.
Also to enable this for rest of the 5% of file requests we can think of async way to calculate checksum and update the storage record later.M15 - Release 0.18https://community.opengroup.org/osdu/platform/system/file/-/issues/73Storage and retrieval instructions for file collection in AWS file service mi...2022-07-01T15:23:32ZMorris EstepaStorage and retrieval instructions for file collection in AWS file service missing critical fieldsThe storage and retrieval instructions for file collections returned by AWS file service currently only returns a pre-signed URL. However, pre-signed URLs only work with single S3 objects. Consumers of file collection instructions need t...The storage and retrieval instructions for file collections returned by AWS file service currently only returns a pre-signed URL. However, pre-signed URLs only work with single S3 objects. Consumers of file collection instructions need the following information in order to store and retrieve objects in S3:
* Unsigned URL
* temporary credentials
* region
AWS file service needs to change to return the required information above.M12 - Release 0.15Morris EstepaMorris Estepahttps://community.opengroup.org/osdu/platform/system/file/-/issues/72File service - Compute and store properties such as filesize and checksum at ...2022-08-23T15:21:13ZDebasis ChatterjeeFile service - Compute and store properties such as filesize and checksum at the time of uploading fileCurrently, Data Loader is expected to provide this information manually. This can be error prone and can be missed.
cc - @Keith_Wall and @krveduru for informationCurrently, Data Loader is expected to provide this information manually. This can be error prone and can be missed.
cc - @Keith_Wall and @krveduru for informationhttps://community.opengroup.org/osdu/platform/system/file/-/issues/71File core module Junits are not getting executed2022-06-17T11:26:17ZAbhishek Kumar (SLB)File core module Junits are not getting executedAbhishek Kumar (SLB)Abhishek Kumar (SLB)https://community.opengroup.org/osdu/platform/system/file/-/issues/70Already generated pre-signed url can be use to upload a file multiple time2022-09-27T11:18:15ZPiyush TalwalkarAlready generated pre-signed url can be use to upload a file multiple timeSteps to reproduce an issue:
- Generate Pre-Signed url
- Upload file using Pre-Signed url generated in step1
- Now post metadata for uploaded file and verify that file is moved to persistent location
- Now again try to upload another fil...Steps to reproduce an issue:
- Generate Pre-Signed url
- Upload file using Pre-Signed url generated in step1
- Now post metadata for uploaded file and verify that file is moved to persistent location
- Now again try to upload another file using same pre-signed url from step1
Expected Result:
User should able to Pre-Signed url only once for one file
Actual Result:
Already generated pre-signed url can be use to upload a file multiple time