File issueshttps://community.opengroup.org/osdu/platform/system/file/-/issues2022-08-23T21:03:43Zhttps://community.opengroup.org/osdu/platform/system/file/-/issues/69File Service: Requests to POST metadata are taking long time2022-08-23T21:03:43ZSachin JaiswalFile Service: Requests to POST metadata are taking long time### Problem Statement
Request to post metadata takes long time when try to calculate the checksum for larger files.
### Solution
We can overcome this problem by reading bytes from the input stream and storing them into the buffer array.### Problem Statement
Request to post metadata takes long time when try to calculate the checksum for larger files.
### Solution
We can overcome this problem by reading bytes from the input stream and storing them into the buffer array.https://community.opengroup.org/osdu/platform/system/file/-/issues/68File service: File.GeoJSON kind2024-01-25T06:54:39ZMarton NagyFile service: File.GeoJSON kindFile service currently accepts only `dataset--File.Generic` for kind. We would set it to `dataset--File.GeoJSON` to match the real content of uploaded files, but that currently leads to
> 400 (BadRequest) - {\"error\":{\"code\":400,\"mes...File service currently accepts only `dataset--File.Generic` for kind. We would set it to `dataset--File.GeoJSON` to match the real content of uploaded files, but that currently leads to
> 400 (BadRequest) - {\"error\":{\"code\":400,\"message\":\"Invalid entity in kind\",\"errors\":[{\"domain\":\"global\",\"reason\":\"badRequest\",\"message\":\"Invalid entity in kind\"}]}.
Even swagger doc says: `Kind of data being ingested. Must follow the naming convention:{schema-authority}:wks:dataset--File.Generic:{version}.`
Please allow different kinds like `dataset--File.XxxXxx` to be posted there.
The rule may be extended to accept any kind matching [AbstractFile](https://community.opengroup.org/osdu/data/data-definitions/-/blob/master/E-R/abstract/AbstractFile.1.0.0.md), like most in [dataset](https://community.opengroup.org/osdu/data/data-definitions/-/tree/master/E-R/dataset) group.
cc @gehrmannhttps://community.opengroup.org/osdu/platform/system/file/-/issues/65The postman collection that executes successfully on Azure, GCP and IBM but f...2022-08-24T14:38:35ZKamlesh TodaiThe postman collection that executes successfully on Azure, GCP and IBM but fails on AWSThe following collection works successfully in GCP and IBM environments in the Platform Validation project. But it fails to run in Azure and AWS environments.
[FileAPI_UploadDownload_CI-CD_v2.0.postman_collection.json](/uploads/9e3038f1...The following collection works successfully in GCP and IBM environments in the Platform Validation project. But it fails to run in Azure and AWS environments.
[FileAPI_UploadDownload_CI-CD_v2.0.postman_collection.json](/uploads/9e3038f1bd76c63807ce29ff53fae533/FileAPI_UploadDownload_CI-CD_v2.0.postman_collection.json)
In Azure environment, @krveduru @ankurrawat
Request 2.UploadFile by SignedURL fails with response code 400 An HTTP header that's mandatory for this request is not specified
The API doc is not clear as to what needs to be specified and the same request works in GCP and IBM environment
Reponse:
<?xml version="1.0" encoding="utf-8"?>
<Error>
<Code>MissingRequiredHeader</Code>
<Message>An HTTP header that's mandatory for this request is not specified.
RequestId:3d3902d8-501e-0092-3544-50f68a000000
Time:2022-04-14T21:15:09.4287412Z</Message>
<HeaderName>x-ms-blob-type</HeaderName>
</Error>
In AWS environment, @fhoueto.amz
Request 3. Create File Metadata R3 fails with response code 500 Internal Server Error
Reponse:
{
"error": {
"code": 500,
"message": "Internal server error",
"errors": [
{
"domain": "global",
"reason": "internalError",
"message": "Internal server error"
}
]
}
}
The file being uploaded is a las file (7004_a1501_1978_comp.las) from TNO data.
FYI - @dzmitry_malkevich @anujgupta @debasisc @ChrisZhangM11 - Release 0.14Okoun-Ola Fabien HouetoOkoun-Ola Fabien Houetohttps://community.opengroup.org/osdu/platform/system/file/-/issues/64ADR: Calculate Checksum before saving metadata2023-07-05T09:41:49ZParesh BehedeADR: Calculate Checksum before saving metadata# Decision Title
Calculate checksum of uploaded file before creating its metadata
## Status
- [X] Proposed
- [X] Trialing
- [X] Under review
- [X] Approved
- [ ] Retired
## Context & Scope
We support dataset--File.Generic entity record...# Decision Title
Calculate checksum of uploaded file before creating its metadata
## Status
- [X] Proposed
- [X] Trialing
- [X] Under review
- [X] Approved
- [ ] Retired
## Context & Scope
We support dataset--File.Generic entity record to be created in data platform while user hits /metadata endpoint of File Service. this schema has couple of useful attribute which we don't use as of now which is checksum and checksum algorigthm.
These attributes would be super useful to detect any duplicate file uploads in data platform.
## Mechanism for calculating checksum
I propose to implement new method in core module (lets say generateChecksum()) which can be implemented by every CSPs in provider module before we make call to storage service for saving metadata of file.
Now this method can be implemented in various ways and algorithms as per CSPs choice, for e.g., in Azure, we really don't need to generate checksum explicitly as its been calculated by blob store automatically, so implementation of generateChecksum() will be to just fetch the blob's metadata and they are done. similarly it can be implemented by other providers if there storage solution also supports calculating checksum while storing blob.
## Decision
We should generate checksum of single file before creating its metadata in data platform, so that we can provide that checksum value in metadata record (instance of dataset--File.Generic entity)M12 - Release 0.15Paresh BehedeParesh Behedehttps://community.opengroup.org/osdu/platform/system/file/-/issues/63Preloadfilepath & ExtensionProperties removed from file Metadata API2022-11-28T14:10:32Zivar SoerheimPreloadfilepath & ExtensionProperties removed from file Metadata APIDuring ingestion of file metadata under /files/metadata using POST command the Preloadfilepath and ExtensionProperties are not persisted when returning the record post ingest.
This seems like strange behaviour to me. I would like to ei...During ingestion of file metadata under /files/metadata using POST command the Preloadfilepath and ExtensionProperties are not persisted when returning the record post ingest.
This seems like strange behaviour to me. I would like to either understand why this happens, or extend the file metadata api so these properties are not removed.
This is the workflow:
1. Get Signed URL for upload
2. Upload file using signed URL
3. Upload file metadata using file api (this returns ID of created record and can be searched in storage)
4. Refer to this ID when creating well log record or any other record
The problem with this workflow is that:
- PreloadFilePath and ExtensionProperties are removed from the record during metadata uploadhttps://community.opengroup.org/osdu/platform/system/file/-/issues/62POST /metadata endpoint returns 500 error when Blob Not Found2022-09-27T11:52:47ZTsvetelina IvanovaPOST /metadata endpoint returns 500 error when Blob Not FoundPOST /metadata endpoint returns a BlobStorageExcepiton with message "Error occurred while creating file metadata Status code 404, "<?xml version="1.0" encoding="utf-8"?><Error><Code>BlobNotFound</Code><Message>The specified blob does no...POST /metadata endpoint returns a BlobStorageExcepiton with message "Error occurred while creating file metadata Status code 404, "<?xml version="1.0" encoding="utf-8"?><Error><Code>BlobNotFound</Code><Message>The specified blob does not exist._RequestId:f60ebd44-b01e-0012-532e-3236bf000000_Time:2022-03-07T14:23:07.9454432Z</Message></Error>" Status code 404, "<?xml version="1.0" encoding="utf-8"?><Error><Code>BlobNotFound</Code><Message>The specified blob does not exist.
RequestId:f60ebd44-b01e-0012-532e-3236bf000000
Time:2022-03-07T14:23:07.9454432Z</Message></Error>"" as a 500 Internal Server Error when calling deleteFile method to delete file from the blob storage.
The file is copied successfully from staging to the persistent area, but the delete of the file from staging fails with "The specified blob does not exist".
For exception logs see the attached file:
[File_Service_Azure_Logs_-_BlobStorageException.csv](/uploads/2cfc5a0baf2616fb456791ce8412babb/File_Service_Azure_Logs_-_BlobStorageException.csv)https://community.opengroup.org/osdu/platform/system/file/-/issues/61Swagger shows data.TotalSize is mandatory but it isn't in actual2022-08-26T10:17:33ZParesh BehedeSwagger shows data.TotalSize is mandatory but it isn't in actualThe data returned from /v2/files/{Id}/metadata does not include data.TotalSize. The documentation at https://community.opengroup.org/osdu/platform/system/file/-/blob/master/docs/metadataPayload.json suggests that TotalSize is not option...The data returned from /v2/files/{Id}/metadata does not include data.TotalSize. The documentation at https://community.opengroup.org/osdu/platform/system/file/-/blob/master/docs/metadataPayload.json suggests that TotalSize is not optional
We must remove mandatory flag from swagger for that field to remain in synchttps://community.opengroup.org/osdu/platform/system/file/-/issues/60Discrepancy in API endpoint downloadURL2022-09-07T09:18:09ZParesh BehedeDiscrepancy in API endpoint downloadURLSwagger and document available in gitlab shows it as /DownloadURL but actual endpoint implemented is /downloadUrl. we must fix the documentation and swagger to reflect the current state.Swagger and document available in gitlab shows it as /DownloadURL but actual endpoint implemented is /downloadUrl. we must fix the documentation and swagger to reflect the current state.https://community.opengroup.org/osdu/platform/system/file/-/issues/59Using Publisher Facade to publish status messages2022-02-24T08:50:26ZTsvetelina IvanovaUsing Publisher Facade to publish status messagesAzure core lib introduces a publisher facade which can be used across services in order to publish messages to message brokers(Service Bus/ Event Grid).It will help to manage and update at a single source instead of each service doing it...Azure core lib introduces a publisher facade which can be used across services in order to publish messages to message brokers(Service Bus/ Event Grid).It will help to manage and update at a single source instead of each service doing it individually.The pub sub configuration can be used to configure publishing for Event grid and Service bus.
Link of related issue:
https://community.opengroup.org/osdu/platform/system/notification/-/issues/25Tsvetelina IvanovaTsvetelina Ivanovahttps://community.opengroup.org/osdu/platform/system/file/-/issues/58File SignedURL lifespan2022-09-29T13:41:08ZJan MortensenFile SignedURL lifespanWhen uploading a file there is the concept of a SignedURL that gives full access to the file in question for the lifespan of 7 days. This makes the file available for anyone how happens to have access to this URL.
Now there is the conc...When uploading a file there is the concept of a SignedURL that gives full access to the file in question for the lifespan of 7 days. This makes the file available for anyone how happens to have access to this URL.
Now there is the concept of a staging-area and a persistent-area. When uploading the file it first resides in the staging-area until a metadata has been created and posted to the OSDU-instance; then it will be moved by the system from staging to persistent. BUT this is a manual process and if the user somehow forgets or fails to the this second part the file will stay in the staging-area.
So even if it is the user or the process "fault" the way OSDU has designed this might make the file potentially "open" for "anyone" (with the link).
Is this something that can/should be mitigated?
See also #38 for a mitigating action on the signed url.https://community.opengroup.org/osdu/platform/system/file/-/issues/55Update Swagger documentation for end point - File uploadURL2022-09-29T13:30:08ZDebasis ChatterjeeUpdate Swagger documentation for end point - File uploadURLSee this below-
![API-File-service](/uploads/fb1508499115056f6ccd8abbb10c9a01/API-File-service.PNG)
Should be lowercase "**uploadURL**"
{{FILE_HOST}}/files/uploadURLSee this below-
![API-File-service](/uploads/fb1508499115056f6ccd8abbb10c9a01/API-File-service.PNG)
Should be lowercase "**uploadURL**"
{{FILE_HOST}}/files/uploadURLM14 - Release 0.17Shrikant GargShrikant Garghttps://community.opengroup.org/osdu/platform/system/file/-/issues/54Upgrade to Log4J 2.172021-12-21T03:09:43ZDavid Diederichd.diederich@opengroup.orgUpgrade to Log4J 2.17The Apache Foundation released another Log4j2 update, version 2.17, which address a denial of service vulnerability.
This issue tracks progress to upgrade this dependency for this project.The Apache Foundation released another Log4j2 update, version 2.17, which address a denial of service vulnerability.
This issue tracks progress to upgrade this dependency for this project.https://community.opengroup.org/osdu/platform/system/file/-/issues/53Log4J Expedient Updates and Patches2021-12-17T11:10:27ZDavid Diederichd.diederich@opengroup.orgLog4J Expedient Updates and PatchesThis issue associates MRs that were applied to this project quickly to get a patched version ready as soon as possible. The intent is to provide a reference point for later, more thoughtful, analysis.This issue associates MRs that were applied to this project quickly to get a patched version ready as soon as possible. The intent is to provide a reference point for later, more thoughtful, analysis.David Diederichd.diederich@opengroup.orgDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/system/file/-/issues/52Log4J CVE-2021-442282021-12-17T06:39:56ZTsvetelina IvanovaLog4J CVE-2021-44228Apache Log4j2 <=2.14.1 JNDI features used in configuration, log messages, and parameters do not protect against attacker controlled LDAP and other JNDI related endpoints. An attacker who can control log messages or log message parameters...Apache Log4j2 <=2.14.1 JNDI features used in configuration, log messages, and parameters do not protect against attacker controlled LDAP and other JNDI related endpoints. An attacker who can control log messages or log message parameters can execute arbitrary code loaded from LDAP servers when message lookup substitution is enabled. From log4j 2.15.0, this behavior has been disabled by default. In previous releases (>2.10) this behavior can be mitigated by setting system property "log4j2.formatMsgNoLookups" to “true” or it can be mitigated in prior releases (<2.10) by removing the JndiLookup class from the classpath (example: zip -q -d log4j-core-\*.jar org/apache/logging/log4j/core/lookup/JndiLookup.class).Tsvetelina IvanovaTsvetelina Ivanovahttps://community.opengroup.org/osdu/platform/system/file/-/issues/51Update swagger documentation for end point "downloadURL"2022-09-29T13:29:51ZDebasis ChatterjeeUpdate swagger documentation for end point "downloadURL"Please check here -
https://community.opengroup.org/osdu/platform/system/file/-/blob/master/docs/file-service_openapi.yaml
GET {{FILE_HOST}}/files/{{FILE_ID}}/**downloadURL**
Swagger documentation shows **DownloadURL**
cc - @harshit2...Please check here -
https://community.opengroup.org/osdu/platform/system/file/-/blob/master/docs/file-service_openapi.yaml
GET {{FILE_HOST}}/files/{{FILE_ID}}/**downloadURL**
Swagger documentation shows **DownloadURL**
cc - @harshit283 for informationM14 - Release 0.17Shrikant GargShrikant Garghttps://community.opengroup.org/osdu/platform/system/file/-/issues/49Compilation failure in Master Branch2021-11-18T12:19:28ZRiabokon Stanislav(EPAM)[GCP]Compilation failure in Master Branchhttps://community.opengroup.org/osdu/platform/system/file/-/pipelines/77475
`ERROR: Uploading artifacts as "archive" to coordinator... too large archive id=723684 responseStatus=413 Request Entity Too Large status=413 token=6k_YiZaq
FA...https://community.opengroup.org/osdu/platform/system/file/-/pipelines/77475
`ERROR: Uploading artifacts as "archive" to coordinator... too large archive id=723684 responseStatus=413 Request Entity Too Large status=413 token=6k_YiZaq
FATAL: too large `
Could you increase a limit?M10 - Release 0.13David Diederichd.diederich@opengroup.orgDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/system/file/-/issues/48ADR: Upgrade swagger version from swagger 2.0 to swagger 3.0 (OpenAPI 3)2022-05-12T06:27:04ZAman VermaADR: Upgrade swagger version from swagger 2.0 to swagger 3.0 (OpenAPI 3)## Status
- [X] Proposed
- [x] Under review
- [x] Approved
- [ ] Retired
## Context & Scope
OpenAPI Specification (formerly Swagger Specification) is an API description format for REST APIs. An OpenAPI file allows you to describe your e...## Status
- [X] Proposed
- [x] Under review
- [x] Approved
- [ ] Retired
## Context & Scope
OpenAPI Specification (formerly Swagger Specification) is an API description format for REST APIs. An OpenAPI file allows you to describe your entire API, including: endpoints, optional parameters, authentication methods and contact/ license information.
The latest swagger version has been released with name OpenAPI 3.0
### Why is the upgrade required
The OpenAPI 3.0 offers several new features such as [servers](https://swagger.io/docs/specification/api-host-and-base-path/) field, which enables us to write automation on top of it. Let's say if you have a tool which scans the REST endpoint exposed by your service for various reasons, it can consume the openapi.json directly and skim through all the end points in that service
### Scope
All the services
## Trade-off Analysis
- **Are there any breaking changes in REST API doc while upgrading from swagger 2.0 to swagger 3.0**: NO
- **Are there any breaking changes in UI**: There are no breaking changes in UI per say. If anything, it has become more fluid and lightweight.
- **How involved are the code changes**: For most of the services, it's a matter of upgrading the maven package with minor changes here and there. For few services the changes can be more involved as some the the tags have changes between 2.0 and 3.0, like `@Api` has becom `@Tag`. More details are documented here: https://springdoc.org/#migrating-from-springfox
- **Would the URL change**: YES. The swagger home page for 2.0 is `swagger-ui.html`, while for 3.0, it is `swagger-ui/index.html`. However, we also have an explicit endpoint `/swagger` (or equivalent) which can be leveraged to abstract the underlying swagger URL. Users can continue visiting the swagger homepage by hitting `/swagger` endpoint.
## Decision
cc: @kibattul @madhurtanwaniAman VermaAman Vermahttps://community.opengroup.org/osdu/platform/system/file/-/issues/47KeyVault Quota Issue2021-11-23T08:35:00ZTsvetelina IvanovaKeyVault Quota IssueFile Service makes a large number of calls to Key Vault to retrieve storage account name and key.
When connecting to storage the core-lib-azure library should be used to create Blob clients and use the implemented cache for keyvault sec...File Service makes a large number of calls to Key Vault to retrieve storage account name and key.
When connecting to storage the core-lib-azure library should be used to create Blob clients and use the implemented cache for keyvault secrets.Tsvetelina IvanovaTsvetelina Ivanovahttps://community.opengroup.org/osdu/platform/system/file/-/issues/45File integration test are failing2021-10-18T04:53:46Zsachin GuptaFile integration test are failingIT for DMS API is getting failed in our pipeline. please see the attached image.
![image](/uploads/d998012b6f862c595d56593f0d312dba/image.png)IT for DMS API is getting failed in our pipeline. please see the attached image.
![image](/uploads/d998012b6f862c595d56593f0d312dba/image.png)Riabokon Stanislav(EPAM)[GCP]Riabokon Stanislav(EPAM)[GCP]https://community.opengroup.org/osdu/platform/system/file/-/issues/44Compilation failure in Master2021-09-29T12:53:40ZAbhishek Kumar (SLB)Compilation failure in MasterThere is a compilation issue in the master branch.
@ethiraj : Please assign it to the right person.
Job [#619104](https://community.opengroup.org/osdu/platform/system/file/-/jobs/619104) failed for 384101a9fe1c781d5e2783fce313803344c62...There is a compilation issue in the master branch.
@ethiraj : Please assign it to the right person.
Job [#619104](https://community.opengroup.org/osdu/platform/system/file/-/jobs/619104) failed for 384101a9fe1c781d5e2783fce313803344c6276a:ethiraj krishnamanaiduRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comRiabokon Stanislav(EPAM)[GCP]ethiraj krishnamanaidu