Seismic issueshttps://community.opengroup.org/groups/osdu/platform/domain-data-mgmt-services/seismic/-/issues2024-01-05T10:29:26Zhttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-service/-/issues/107[ADR] Hierarchical deletion of datasets2024-01-05T10:29:26ZMaggie Salak[ADR] Hierarchical deletion of datasets# Introduction
We need a way to delete millions of datasets (including metadata and files in blob storage) in Seismic DMS. A single delete operation can include up to 50 million datasets.
The purpose of this ADR is to define the approa...# Introduction
We need a way to delete millions of datasets (including metadata and files in blob storage) in Seismic DMS. A single delete operation can include up to 50 million datasets.
The purpose of this ADR is to define the approach to implementing a hierarchical delete feature in SDMS.
# Status
* [x] Initiated
* [x] Proposed
* [x] Under Review
* [ ] Approved
* [ ] Rejected
# Problem statement
SDMS API currently exposes the following endpoints for deleting datasets:
- `DELETE /dataset/tenant/{tenantid}/subproject/{subprojectid}/dataset/{datasetid}`
Deletes a single dataset.
- `DELETE /subproject/tenant/{tenantid}/subproject/{subprojectid}`
Deletes a subproject.
The endpoint deleting a subproject currently does not scale to the required number of datasets. The current implementation also leaves a possibility of an inconsistent state between the metadata and files in blob storage - in case some of the files fail to be deleted, the deletion of metadata associated with these datasets is not reverted.
SDMS currently does not have the functionality of deleting only selected datasets in a subproject, filtered by path, tags, labels, etc.
# Proposed Solution
In short:
- Create new API endpoints to support starting and tracking progress of the asynchronous deletion operation.
- Deploy a new service on k8s that would asynchronously delete datasets.
## Overview
We will introduce the bulk-delete feature as follows:
1. Implement and deploy a separate application to the same K8s cluster: the _deletion service_.
This service will accept the bulk deletion requests from SDMS API, perform the deletion and keep track of the progress of this long-running operation.
2. Add the new endpoint to SDMS API to delete all datasets in a specified path:
`PUT /operations/bulk-delete?sdpath={sdpath}`
Status: 202 Accepted
`sdpath` in the format `sd://tenant/subproject/path`
Response schema:
```json
{
"operationId": "{string}"
}
```
3. Add the new endpoint to SDMS API to view the status and progress of the delete operation:
`GET /operations/bulk-delete/status/{operationid}`
Status: 200 OK
Response schema:
```json
{
"OperationId": "{string}",
"CreatedAt": "{string}",
"CreatedBy": "{string}",
"LastUpdatedAt": "{string}",
"Status": "{string}",
"DatasetsCnt": "{int}",
"DeletedCnt": "{int}",
"FailedCnt": "{int}"
}
```
Headers will contain `data-partition-id` information to check if the user is registered in the partition before retrieving the operation status.
## Details
### Initiating a delete operation
- The new `PUT` endpoint will support the following cases for the dataset path, provided in the `sdpath` parameter:
- `path = /<path>/` - all datasets under the specified path should be deleted.
- path not specified - all datasets in the subproject should be deleted.
If the deletion of the subproject (metadata and container) is desired as well, the clients should call the delete subproject endpoint after the datasets bulk delete operation completes to ensure non-blocking deletion of the subproject in case it is composed by many datasets.
- The endpoint triggers the deletion job and returns the ID of the initiated operation.
- The delete operation is initiated in SDMS by pushing a message onto a queue (Azure Storage queue in case of Azure implementation; a different queuing mechanism can be used by other CSPs); the message contains the `operationId` and the parameters from the original request (tenant, subproject, path).
### Deletion service
Deletion service is a separate component from SDMS API, deployed to the same K8s cluster. The implementation details of the service can be decided by the individual CSPs. This section describes the proposed implementation for Azure.
The source code of the new component will be contributed to the Sidecar solution in the `seismic-store-service` repository.
The logic of the deletion service will work as follows:
- The service consumes the message from the Azure Storage queue and initiates the deletion process.
- All items (dataset IDs and `gcsurl` which determines the location in blob storage) matching the provided subproject and path are retrieved from Cosmos database.
- For each dataset, the deletion service checks if it is locked.
- If yes, the item is discarded from the delete operation.
- If not, the deletion service locks the dataset. The lock value in this case will contain a string indicating that the dataset is locked for deletion (e.g. WDELETE). This will allow another delete operation to delete the dataset if the deletion failed previously. However, it will prevent deletion of datasets locked with a regular write lock which would indicate that it is being actively used.
- The retrieved items are added to storage which allows the deletion service to keep track of the datasets to delete. In the first version of the implementation, the deletion service will store the retrieved datasets in memory.
In a later phase we are planning to use a persistent storage (e.g. Service Bus queue) to store the items to be deleted. This will allow the service to resume deletion after a restart as well as retry deletion for the datasets where it failed.
- The deletion service leverages existing Redis queues to keep track of the overall deletion operation status and progress.
- The deletion service retrieves and deletes the datasets by checking the store containing items to be deleted. In the first version of the implementation it simply iterates over items stored in memory.
- The datasets are processed in batches; for each batch we retrieve the associated blobs from the storage account using the `gcsurl` property of the metadata.
- The blobs from the current batch are deleted.
- We then delete the metadata documents from Cosmos DB, leaving the ones for which the blob deletion was unsuccessful. We consider that the deletion was successful if the blobs were not found as we assume they were deleted earlier.
- The deletion status is updated in Redis after processing every dataset.
- At the end, the status of a completed operation (with errors or without) is saved in Redis.
- The deletion status should not be deleted at this point so that users can query the operation status after completion.
### Sequence diagram for the deletion operation
![deletion_diagram_osdu](/uploads/b097c46896644e19a7374df96560aabd/deletion_diagram_updated.png)
### Deletion status
The status of delete operations will be saved in Redis.
It will be written by the deletion service (updated with the current progress) and read by SDMS API
(when users request the deletion status).
SDMS API and the deletion service will agree on the naming convention for the key in Redis,
e.g. `deletequeue:status:{operationId}`.
The new `GET` endpoint allowing users to query the status of a delete operation will return the following information:
- **`OperationId`** - ID of the delete operation.
- **`Status`** - Current status of the delete operation; possible values are: 'Not started', 'Started', 'In progress', 'Completed', 'Completed with errors'.
- `CreatedAt` - Timestamp of the creation of the delete operation.
- `CreatedBy` - Entity initiating the delete operation.
- `LastUpdatedAt` - Timestamp of the last status update of the delete operation.
- `DatasetsCnt` - Total number of datasets to be deleted; initially not set, until the enumeration of datasets for deletion is completed.
- `DeletedCnt` - Number of deleted datasets; updated after each dataset processed by the deletion service, after both blobs and metadata are deleted.
- `FailedCnt` - Number of datasets for which the deletion failed; updated after each dataset processed by the deletion service if a failure occurs.
_(only the fileds in **bold** are required)_
_(dataset counts will be empty if the status is "not started")_
### Sequence diagram for the deletion status
![deletion_status_diagram](/uploads/52b27cfb56a9942cf7628e81aeb41eec/deletion_status_diagram.png)
# Out of scope / limitations
- Detailed statistics about datasets which failed to be deleted. In the first phase of implementation the deletion status endpoint will provide aggregated statistics as mentioned in the `Deletion status` section. Users will need to refer to logs to find out which datasets failed to be deleted.
- The bulk-delete feature does not guarantee the operation can continue after a restart of the deletion service. It will be up to the different CSPs to determine if there is retry logic for failed datasets or recovery support built into the service.
- Deleting 'orphan' blobs with missing metadata. Files without metadata containing a matching `gcsurl` will not be deleted as part of the delete operation as metadata is the source of truth for which blobs need to be deleted.
- Identifying blobs belonging to a different dataset but located in the same virtual folder as files of another dataset. Since `gcsurl` carries information about the location of files to be deleted, the delete operation will not be able to detect 'unrelated' files erroneously uploaded with the same virtual folder.
# Consequences
The same bulk deletion API endpoints can be implemented by any CSPs besides Azure.
The status endpoint is not CSP-specific. As long as the bulk delete implementation saves
the job status with the same schema to Redis, the status endpoint will work for any other CSP out of the box.M22 - Release 0.25Diego MolteniMark YanMaggie SalakSneha PoddarDiego Moltenihttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/issues/190Add OSDU manifest generation capability2023-06-02T14:03:18ZMorten OfstadAdd OSDU manifest generation capabilityVDSInfo (or maybe a new vds utility) should be able to generate OSDU compliant JSON manifest for ingesting VDS. This was suggested by Juliana Fernandes (@fernandes_jfa) and would make life a lot easier when interacting with the OSDU inge...VDSInfo (or maybe a new vds utility) should be able to generate OSDU compliant JSON manifest for ingesting VDS. This was suggested by Juliana Fernandes (@fernandes_jfa) and would make life a lot easier when interacting with the OSDU ingestion service.https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/issues/123Segmentation fault on VDSCopy2023-01-25T12:13:20ZMichał MurawskiSegmentation fault on VDSCopyI was trying to copy data from the local directory to the seismic store using the SD protocol. I was using OpenVDS in version 2.3.7
I executed the following command:
`
VDSCopy ./data.vds sd://osdu/testproject1/test-27 -d "SdAuthorityUr...I was trying to copy data from the local directory to the seismic store using the SD protocol. I was using OpenVDS in version 2.3.7
I executed the following command:
`
VDSCopy ./data.vds sd://osdu/testproject1/test-27 -d "SdAuthorityUrl=https://****/seismic-store/v3;SdApiKey=xxx;AuthTokenUrl=https://*****/token.oauth2;SdToken=***;ClientId=***;ClientSecret=***;RefreshToken=***;Scopes=offline_access;LogLevel=Trace" --tolerance 1 --compression-method None
`
In the end when the counter reached 100% percent it resulted in the following error
`
qemu: uncaught target signal 11 (Segmentation fault) - core dumped
Segmentation fault
`
The same command works fine for 2.2.0https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-sdutil/-/issues/10Subproject name cannot contain "-" , it will result in error 404 even if uplo...2023-05-22T11:56:50ZChad LeongSubproject name cannot contain "-" , it will result in error 404 even if upload shows succesfulI created a subproject with "-" in the naming (chad-test) successfully using both python sdutil mk or directly through postman.
When I try to upload a file, the upload proceeds and when its done, it says successful but the return respon...I created a subproject with "-" in the naming (chad-test) successfully using both python sdutil mk or directly through postman.
When I try to upload a file, the upload proceeds and when its done, it says successful but the return response was error [404] [seismic-store-service] xxxx does not exist. No file was uploaded when I've try to list the directory.
```
File [ST10010ZC11_PZ_PSDM_KIRCH_FULL_D.MIG_FIN.POST_STACK.3D.JS-017536.segy] uploaded successfully
[404] [seismic-store-service] The dataset sd://osdu/chad-test/full_d.segy does not exist
```M17 - Release 0.20https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/issues/212VDSCopy hanging when uploading to Seismic DDMS2023-10-23T11:39:11Zvinicius Vicente Silva RosaVDSCopy hanging when uploading to Seismic DDMSI am attempting to upload a local VDS file (1.5TB) to a SD Path, and after approximately an hour, there is no visible progress in the file upload, creating the impression that the process is stalled. No error messages are being displayed...I am attempting to upload a local VDS file (1.5TB) to a SD Path, and after approximately an hour, there is no visible progress in the file upload, creating the impression that the process is stalled. No error messages are being displayed. I suspect it may be related to the token refresh.
We are using the command line bellow:
OSDU/ADME M16
lIB: VDSCopy - OpenVDS+ 3.3.0 installed on Linux
```bash
VDSCopy -a 01 -a 02 -a 12 --tolerance=1.0 --compression-method=Wavelet -d 'sdAuthorityUrl=
https://{HOST}.energy.azure.com/seistore-svc/api/v3;authTokenUrl=https://login.microsoftonline.com/{TENANT}/oauth2/v2.0/token/;client_id={APP_ID};client_secret={APP_SECRET};scopes={APP_ID}/.default;'
'/local_disk0/vds/FILE.vds' 'sd://{TENANT}/{SUBPROJECT}/dataset_name.vds'
```
The intention of the command above is to authenticate using ClientID and ClientSecrect.
The upload completes successfully when the file is processed within an hour or less.https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-service/-/issues/108[ADR] Hierarchical data distribution statistics based on path - API endpoint2023-12-21T12:00:55ZIzabela Kulakowska[ADR] Hierarchical data distribution statistics based on path - API endpoint# Introduction
We need a solution for retrieving dataset statistics currently consisting of only dataset sizes.
The purpose of this ADR is to define the approach for retrieving the hierarchical data distribution statistics based on a p...# Introduction
We need a solution for retrieving dataset statistics currently consisting of only dataset sizes.
The purpose of this ADR is to define the approach for retrieving the hierarchical data distribution statistics based on a path.
# Status
* [x] Initiated
* [x] Proposed
* [x] Under Review
* [ ] Approved
* [ ] Rejected
# Problem statement
The SDMS API currently exposes the following endpoints for managing the datasets sizes:
- `POST /dataset/tenant/{tenantid}/subproject/{subprojectid}/dataset/{datasetid}/size` - computes the actual dataset size and updates the dataset metadata `computed_size` field.
- (deprecated) `GET /dataset/tenant/{tenantid}/subproject/{subprojectid}/sizes` - fetches the sizes of the datasets based on the metadata field `filemetadata.size`.
# Proposed solution
Create new API endpoint for retrieving the total size value for a dataset, a subfolder and a subproject. The new endpoint would require _viewer_ or _admin_ roles.
## Overview
```bash
GET /dataset/tenant/{tenant}/subproject/{subproject}/size?path={path}&datasetid={datasetname}
```
Path parameters:
- **tenant** - tenant
- **subproject** - subproject
Query parameters:
- **path** - folder path for which the analytics are going to be retrieved [mandatory if query parameter `{datasetid}` is specified]
- **datasetid** - dataset name for which the analytics are going to be retrieved
Response:
HTTP 200
```json
{
"dataset_count": 9999,
"size_bytes": 1024
}
```
- **dataset_count** - number of datasets under a specific subproject/folder
- **size_bytes** - sum of sizes [B] of all datasets under a specific subproject/folder or for a specific dataset
### Examples:
- `GET /dataset/tenant/tenant1/subproject/subproject1/size` - fetch and sum sizes of all datasets in the `subproject1`
- `GET /dataset/tenant/tenant1/subproject/subproject1/size&path=folderA/folderB` - fetch and sum sizes of all datasets under the folder path `folderA/folderB` in subproject `subproject1`
- `GET /dataset/tenant/tenant1/subproject/subproject1/size&path=folderA/folderB&datasetid=file.txt` - fetch the size of a dataset with a name `file.txt` that resides under the folder path `folderA/folderB` in subproject `subproject1`
## Details
Currently, two fields in the dataset metadata record can store information about the dataset size: `filemetadata.size` and `computed_size`. `filemetadata.size` is being used by the SDK on the client side, `computed_size` is intended to be computed and ingested on the server side.
To make sure the chosen field can be a reliable source of truth, the API endpoint implementation will calculate the sum of dataset sizes based on `compute_size` field.
# Out of scope / limitations
A challenge with using `computed_size` field as a source of truth is that some datasets may not have this property calculated, as currently the only way to update this value is by manually calling the `Compute Size` POST endpoint.
The solution to ensure the reliability of the value of the `computed_size` field will be the subject of a separate ADR.M22 - Release 0.25Izabela KulakowskaIzabela Kulakowskahttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-service/-/issues/100SDMS commit "feat: GONRG-6259: Add osdu google" causes import error when running2023-07-12T02:21:00ZRashaad GraySDMS commit "feat: GONRG-6259: Add osdu google" causes import error when runningWhile testing a problem, at run time the service gets confused because of the "Auth" import that was include in your commit. [feat: GONRG-6259: Add osdu google](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seis...While testing a problem, at run time the service gets confused because of the "Auth" import that was include in your commit. [feat: GONRG-6259: Add osdu google](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-service/-/commit/43ea83caa7ff307e6014022244dfbaaa1734d1b2)
The file in question: [gc/index.ts](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-service/-/blob/43ea83caa7ff307e6014022244dfbaaa1734d1b2/app/sdms/src/cloud/providers/gc/index.ts)
When our [Dataset Parser](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-service/-/blob/master/app/sdms/src/services/dataset/parser.ts#L52) uses the Auth.isImpersonationToken() method it goes to your file instead of its proper location [Auth.ts](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-service/-/blob/master/app/sdms/src/auth/auth.ts) class.
This is causing an issue. While using Azure it goes to your method and returns false for all tokens that it checks even if it is an actual impersonation token.
Because it is NOT properly identifying the token as an impersonation token, incorrect information is passed into a new Dataset's metadata.
Please take some time to remedy this, as it affects more than just your added filesYan Sushchynski (EPAM)Yan Sushchynski (EPAM)https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-sdutil/-/issues/21DDMS to third party non-AWS S3 endpoints2023-03-30T16:44:43ZBrian PruittDDMS to third party non-AWS S3 endpointsUnable to connect DDMS to 3rd party non-AWS S3 endpoints. Requesting endpoint parameter input for “get_s3_client” function.Unable to connect DDMS to 3rd party non-AWS S3 endpoints. Requesting endpoint parameter input for “get_s3_client” function.https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/issues/135Refresh Token Flow failing with SDapi 3.162023-01-24T13:01:46ZFilip BrzękRefresh Token Flow failing with SDapi 3.16Hi,
during our testing, we've observed that the same connection string to SD store seems to work with versions of openvds+ linked to sdapi 3.14, but does not work with sdapi 3.16 (default for ~2.3.0 and ~2.4.0 openvds+ versions. Traceb...Hi,
during our testing, we've observed that the same connection string to SD store seems to work with versions of openvds+ linked to sdapi 3.14, but does not work with sdapi 3.16 (default for ~2.3.0 and ~2.4.0 openvds+ versions. Traceback is somewhat cryptic but related to response type.
```
'sd_authority_url=<<REDACTED>>/api/seismic-store/v3;sd_api_key=xxx;auth_token_url=<<REDACTED>>/token.oauth2;sdtoken=<<REDACTED>>;client_id=<<REDACTED>>;client_secret=<<REDACTED>>;refresh_token=<<REDACTED>>;scopes=offline_access'
ERROR:<<REDACTED>>:sdapi 3.16.0 - CallbackAuthProvider::getServiceAuthTokenImpl: Failed converting text to json format
text:
null
Traceback (most recent call last):
File "<<REDACTED>>", line 211, in _grab_vds_base_volume
return openvds.open(input, options)
RuntimeError: sdapi 3.16.0 - CallbackAuthProvider::getServiceAuthTokenImpl: Failed converting text to json format
text:
null
```
As said, exact same connection string does work with sdapi 3.14, so something breaking must have happened between 3.14 and 3.16; but I'm not skilled enough to dissect it further.
Seismic store details:
* OSDU M11 release,
* AWS flavor,
* Custom Identity Provider Oauth2.0 compatible (not the default cognito-idp)
Let me know, what's desired next steps are, both platform access and accessed data are restricted, so I don't think I can provide more details in public.
Best,
Filiphttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/issues/132openvds+ linked sdapi version vs release tag2023-01-11T21:09:07ZFilip Brzękopenvds+ linked sdapi version vs release tagI would like to understand, what versions of seismic-store/sdapi are used to build openvds+ binaries, and if possible the reasoning behind it.
From quick glance at `CMake/Fetch3rdPartyInBuild.cmake` across 2.X openvds versions/tags:
- 2...I would like to understand, what versions of seismic-store/sdapi are used to build openvds+ binaries, and if possible the reasoning behind it.
From quick glance at `CMake/Fetch3rdPartyInBuild.cmake` across 2.X openvds versions/tags:
- 2.0.X - dms - 1e933303
- 2.1.X - dms - 98d59b27b5
- 2.2.X - dms - 98d59b27b5
- 2.3.X - dms - 3633f2030
- 2.4.X - dms - 3633f2030
- `release/0.14` - dms - 98d59b27b5
- `release/0.15` - dms - 3633f2030
whereas sdapi release tags, have the following sha commits:
- `release/0.14` - d96f1e9b9806486e523ac4d9ea74a124af7ee68d
- `release/0.15` - 04d68a061c3311c041d0ace4c222880032172065
it seems openVDS version tagged as `release/0.14` is missing 35 commits (`git rev-list d96f1e9b9 ^98d59b27b5 --pretty=oneline | wc -l`), from the same release tag `release/0.14` on sdapi; and openVDS version tagged as `release/0.15` is missing 5 commits on the corresponding `release/0.15` (`git rev-list 04d68a061c3311c041d0ace4c222880032172065 ^3633f2030 --pretty=oneline`).
I understand that those might not be "feature commits" irrelevant for the functionalities, but can we have some clarity on which seismic-store/sdapi version is supported in a given openVDS release?
Lastly, what is the desired flow of reporting issues emerging at sdapi level, when using openVDS SDK with a given OSDU DP release? Should the tickets be created for openVDS, or directly in [sd-api repository](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-cpp-lib) with reference to openVDS version used?https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/issues/109Issue with two parallel processes writing to same VDS source.2022-02-07T10:52:20ZMichał MurawskiIssue with two parallel processes writing to same VDS source.
I was trying to implement workflow of multiple parallel processes which are writing to separate chunks of the same VDS source. Unfortunately I encountered some issues in the moment of reading modified VDS source.
Here is code example...
I was trying to implement workflow of multiple parallel processes which are writing to separate chunks of the same VDS source. Unfortunately I encountered some issues in the moment of reading modified VDS source.
Here is code example that I am running
```
from multiprocessing import Process
import openvds
import numpy as np
def unlock_dataset(sd_path):
"Disclosed implementation"
return
def write_zero_pages(accessor):
chunks_count = accessor.getChunkCount()
for c in range(chunks_count):
page = accessor.createPage(c)
buf = np.array(page.getWritableBuffer(), copy=False)
buf[:, :, :] = np.zeros(buf.shape, dtype=float)
page.release()
accessor.commit()
def create_vds(
path,
connection_string,
shape=None,
databrick_size=openvds.VolumeDataLayoutDescriptor.BrickSize.BrickSize_128,
access_mode=openvds.IVolumeDataAccessManager.AccessMode.AccessMode_Create,
components=openvds.VolumeDataChannelDescriptor.Components.Components_1,
format=openvds.VolumeDataChannelDescriptor.Format.Format_R32,
create_and_write_pages=True,
):
layout_descriptor = openvds.VolumeDataLayoutDescriptor(
brickSize=databrick_size,
lodLevels=openvds.VolumeDataLayoutDescriptor.LODLevels.LODLevels_1,
brickSize2DMultiplier=4,
options=openvds.VolumeDataLayoutDescriptor.Options.Options_None,
negativeMargin=0,
positiveMargin=0,
fullResolutionDimension=0,
)
metadata_container = openvds.MetadataContainer()
axis_descriptors = []
for i, size in enumerate(shape):
axis_descriptors.append(
openvds.VolumeDataAxisDescriptor(
size,
f"X{i}",
"unitless",
-1000.0,
1000.0,
)
)
channel_descriptors = [
openvds.VolumeDataChannelDescriptor(
format=format,
components=components,
name=f"Channel0",
unit="unitless",
valueRangeMin=0.0,
valueRangeMax=1000.0,
)
]
vds = openvds.create(
path,
connection_string,
layout_descriptor,
axis_descriptors,
channel_descriptors,
metadata_container,
)
access_manager = openvds.getAccessManager(vds)
accessor = access_manager.createVolumeDataPageAccessor(
dimensionsND=openvds.DimensionsND.Dimensions_012,
accessMode=access_mode,
lod=0,
channel=0,
maxPages=8,
chunkMetadataPageSize=1024,
)
chunks_count = accessor.getChunkCount()
if create_and_write_pages:
write_zero_pages(accessor)
openvds.close(vds)
return chunks_count
def writing_process(path, connection_string, chunks_range, number):
vds = openvds.open(path, connection_string)
manager = openvds.getAccessManager(vds)
accessor = manager.createVolumeDataPageAccessor(
dimensionsND=openvds.DimensionsND.Dimensions_012,
lod=0,
channel=0,
maxPages=8,
accessMode=openvds.IVolumeDataAccessManager.AccessMode.AccessMode_ReadWrite,
chunkMetadataPageSize=1024,
)
for c in range(chunks_range[0], chunks_range[1]):
page = accessor.createPage(c)
buf = np.array(page.getWritableBuffer(), copy=False)
buf[:, :, :] = np.reshape(np.array([float(number)] * buf.size), buf.shape)
page.release()
accessor.commit()
# openvds.close(vds)
def get_data(path, connection_string):
with openvds.open(path, connection_string) as vds_source:
layout = openvds.getLayout(vds_source)
axis_descriptors = [
layout.getAxisDescriptor(dim) for dim in range(layout.getDimensionality())
]
begin_slice = [0, 0, 0, 0, 0, 0]
end_slice = (
int(axis_descriptors[0].numSamples),
int(axis_descriptors[1].numSamples),
int(axis_descriptors[2].numSamples),
1,
1,
1,
)
accessManager = openvds.VolumeDataAccessManager(vds_source)
req = accessManager.requestVolumeSubset(
begin_slice, # start slice
end_slice, # end slice
format=openvds.VolumeDataChannelDescriptor.Format.Format_R32,
lod=0,
replacementNoValue=0.0,
channel=0,
)
if req.data is None:
err_code, err_msg = accessManager.getCurrentDownloadError()
print(err_code)
print(err_msg)
raise RuntimeError("requestVolumeSubset failed!")
dims = (
end_slice[2] - begin_slice[2],
end_slice[1] - begin_slice[1],
end_slice[0] - begin_slice[0],
)
return req.data.reshape(*dims)
if __name__ == "__main__":
path = "sd://osdu/example/dataset-4"
connection_string = "sd_authority_url=https://example.com/api/seismic-store/v3;sd_api_key=xxx;auth_token_url=https://example.com/oauth2/token;sdtoken=SDTOKEN;client_id=CLIENTID;refresh_token=REFRESH_TOKEN;scopes=openid email;LogLevel=100"
numer_of_processes = 2
processes = []
chunks_ranges = []
chunks_count = create_vds(path, connection_string, shape=(512, 512, 512))
a = chunks_count // numer_of_processes
r = chunks_count % numer_of_processes
for i in range(numer_of_processes):
if i == numer_of_processes - 1:
chunks_ranges.append((i * a, (i + 1) * a + r))
else:
chunks_ranges.append((i * a, (i + 1) * a))
print(chunks_ranges)
unlock_dataset(path)
for i, chunks_range in enumerate(chunks_ranges):
p = Process(
target=writing_process,
args=(
path,
connection_string,
chunks_range,
i,
),
)
processes.append(p)
p.start()
for p in processes:
p.join()
print("finished")
vds = openvds.open(path, connection_string)
openvds.close(vds)
data = get_data(path, connection_string)
```
[code.py](/uploads/14209e46c690704b192278aecac37b1c/code.py)
Here is output
```
-- sdapi 3.14.0 - Fri Feb 4 13:40:40 2022 -- Write Block Dimensions_012LOD0/ChunkMetadata/0 --- 0.750 s
-- sdapi 3.14.0 - Fri Feb 4 13:40:41 2022 -- Write Block Dimensions_012LOD0/ChunkMetadata/0 --- 0.716 s
-- sdapi 3.14.0 - Fri Feb 4 13:40:41 2022 -- Write Block LayerStatus --- 0.730 s
-- sdapi 3.14.0 - Fri Feb 4 13:40:41 2022 -- Write Block LayerStatus --- 0.737 s
finished
-- sdapi 3.14.0 - Fri Feb 4 13:40:43 2022 -- Open Dataset sd://osdu/mergingtests4/dataset-4 in ReadOnly mode --- 1.722 s
-- sdapi 3.14.0 - Fri Feb 4 13:40:45 2022 -- Get Block Size --- 1.320 s
-- sdapi 3.14.0 - Fri Feb 4 13:40:45 2022 -- Read Block VolumeDataLayout --- 0.599 s
-- sdapi 3.14.0 - Fri Feb 4 13:40:46 2022 -- Get Block Size --- 0.590 s
-- sdapi 3.14.0 - Fri Feb 4 13:40:46 2022 -- Read Block LayerStatus --- 0.596 s
-- sdapi 3.14.0 - Fri Feb 4 13:40:47 2022 -- Close Dataset sd://osdu/mergingtests4/dataset-4 --- 0.412 s
-- sdapi 3.14.0 - Fri Feb 4 13:40:48 2022 -- Open Dataset sd://osdu/mergingtests4/dataset-4 in ReadOnly mode --- 1.272 s
-- sdapi 3.14.0 - Fri Feb 4 13:40:49 2022 -- Get Block Size --- 0.592 s
-- sdapi 3.14.0 - Fri Feb 4 13:40:49 2022 -- Read Block VolumeDataLayout --- 0.575 s
-- sdapi 3.14.0 - Fri Feb 4 13:40:50 2022 -- Get Block Size --- 0.643 s
-- sdapi 3.14.0 - Fri Feb 4 13:40:50 2022 -- Read Block LayerStatus --- 0.579 s
-- sdapi 3.14.0 - Fri Feb 4 13:40:51 2022 -- Get Block Size --- 0.582 s
-- sdapi 3.14.0 - Fri Feb 4 13:40:52 2022 -- Read Block Dimensions_012LOD0/ChunkMetadata/0 --- 0.600 s
-1
Missing data for chunk: Dimensions_012LOD0/0
-- sdapi 3.14.0 - Fri Feb 4 13:40:52 2022 -- Close Dataset sd://osdu/mergingtests4/dataset-4 --- 0.355 s
Traceback (most recent call last):
File "simple.py", line 208, in <module>
data = get_data(path, connection_string)
File "simple.py", line 159, in get_data
raise RuntimeError("requestVolumeSubset failed!")
RuntimeError: requestVolumeSubset failed!
```
[logs_from_sd_path.log](/uploads/316c5d6bce889df37d29a44f45124f67/logs_from_sd_path.log)
I executed code using seismic store and s3 with the same result. Looks like data are getting corupted when multiple processes writing to the same VDS source.
Can I receive some guidance on this problem?
Is my implementation bad or there is something wrong inside OpenVDS?
I am using python lib openvds 2.2.0 and sd api 3.14.0https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-sdutil/-/issues/7GCS: Object name ends with "/0"2021-07-16T11:36:31ZYan Sushchynski (EPAM)GCS: Object name ends with "/0"As I can see, the object's name ends with "0".
Does it somehow matter?
What if we change this "0" to `dataset.name`.
I stumbled across a problem when OpenVDS Converter tried to download the file using `<dataset.gcsurl> + "/" + <data...As I can see, the object's name ends with "0".
Does it somehow matter?
What if we change this "0" to `dataset.name`.
I stumbled across a problem when OpenVDS Converter tried to download the file using `<dataset.gcsurl> + "/" + <dataset.name>` path. And if the file was uploaded to the bucket with the name "0", the parser couldn't find it.
https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-sdutil/-/blob/master/sdlib/api/providers/google/storage_service.py#L231
Example of how I resolved this issue:
https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-sdutil/-/commit/9d6ff0c0e2fd310bb26cb495d0653795b63c3807https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-service/-/issues/32[ADR] Domain API2023-10-23T10:46:12ZSacha Brants[ADR] Domain API# Introduction
In order to natively support seismic datasets as defined by the OSDU authority, avoid duplicating the logic in applications to convert seismic data from one schema version to another, and potentially implement different l...# Introduction
In order to natively support seismic datasets as defined by the OSDU authority, avoid duplicating the logic in applications to convert seismic data from one schema version to another, and potentially implement different logic, Seismic DMS should provide APIs to support and manage seismic datasets by validating their schema model and return them to the latest version of the schema.
## Status
- [X] Proposed
- [x] Trialing
- [x] Under review
- [x] Approved
- [ ] Retired
## Context & Scope
### (1) OSDU SCHEMAS ORGANIZATION
In the [OSDU schemas organization](https://community.opengroup.org/osdu/data/data-definitions/-/tree/master/E-R) schemas are organized into different categories. A “dataset” schema provides a piece of bulk data information along with its logical representation while the seismic record of other categories requires to be linked with an existing (pre-ingested) dataset.
![image](/uploads/4f34b082ffc589ca6cc0a549994c1f0a/image.png)
SDMS will provide a set of domain-specific API to support these schema format:
- FileCollection Datasets:
- [FileCollection.Generic.1.0.0](https://community.opengroup.org/osdu/data/data-definitions/-/blob/master/E-R/dataset/FileCollection.Generic.1.0.0.md)
- [FileCollection.SEGY.1.0.0](https://community.opengroup.org/osdu/data/data-definitions/-/blob/master/E-R/dataset/FileCollection.SEGY.1.0.0.md)
- [FileCollection.Slb.OpenZGY.1.0.0](https://community.opengroup.org/osdu/data/data-definitions/-/blob/master/E-R/dataset/FileCollection.Slb.OpenZGY.1.0.0.md)
- [FileCollection.Bluware.OpenVDS.1.0.0](https://community.opengroup.org/osdu/data/data-definitions/-/blob/master/E-R/dataset/FileCollection.Bluware.OpenVDS.1.0.0.md)
- Work Product Components:
- [SeismicTraceData.1.1.0](https://community.opengroup.org/osdu/data/data-definitions/-/blob/master/E-R/work-product-component/SeismicTraceData.1.1.0.md)
- [SeismicBinGrid.1.0.0](https://community.opengroup.org/osdu/data/data-definitions/-/blob/master/E-R/work-product-component/SeismicBinGrid.1.0.0.md)
- [SeismicLineGeometry.1.0.0.md](https://community.opengroup.org/osdu/data/data-definitions/-/blob/master/E-R/work-product-component/SeismicLineGeometry.1.0.0.md)
- Master Data:
- [SeismicAcquisitionSurvey.1.2.0.md](https://community.opengroup.org/osdu/data/data-definitions/-/blob/master/E-R/master-data/SeismicAcquisitionSurvey.1.2.0.md)
- [SeismicProcessingProject.1.1.0.md](https://community.opengroup.org/osdu/data/data-definitions/-/blob/master/E-R/master-data/SeismicProcessingProject.1.1.0.md)
References and Naming Convention:
- SCHEMA: The seismic schema model, for example, SeismicTraceData / SesmicBinGrid / FileCollection.SEGY /...
- SCHEMA VERSION: The schema model versions, for example, SeismicTraceData 1.0.0 / SeismicTraceData 1.6.2 /...
- RECORD: The seismic object schema recorded in the DE Storage Service
- RECORD-ID: the unique record ID, for example, ABC1234
- RECORD-VERSION: The record versions, for example, ABC1234 V1 / ABC1234 V2 /...
### (2) SDMS DOMAIN SPECIFIC APIs
SDMS will provide domain-specific APIs to handle the ingestion, schema validation, and underline bulk management for seismic datasets and components SCHEMA as defined by the OSDU authority.
For each supported SCHEMA, we will document the model with examples and provide APIs to manage both RECORD and their VERSIONS
![image](/uploads/b9b6d2e5f2371c680bf4ffebc7f741cb/image.png)
- An endpoint to ingest the seismic dataset:
- When an object is ingested using this endpoint, a new RECORD will be created if the RECORD-ID is not specified with the request model. A new RECORD-ID and the RECORD-VERSION will be generated and returned. In addition, for FileCollection dataset schema only, a storage resource will be created to host bulk.
- When an object is ingested using this endpoint, a RECORD will be updated if the RECORD-ID is specified in the request with the request model. A new RECORD-VERSIOn will be generated and returned.
- An endpoint to list all datasets of a specific kind
- This endpoint will support query paginated.
- An endpoint to retrieve the last version of the RECORD-ID
- An endpoint to retrieve a specific version of the RECORD-ID
- An endpoint to retrieve all versions for a RECORD-ID
- An endpoint to delete the RECORD with all associated version
- This endpoint will perform and hard delete by removing all RECORD-VERSIONS athe nd associated bulk.
- ~~An endpoint to reindex dataset ingested with the V3 version of SDMS into the V4~~
The SDMS service will provide support to the highest Patch.Minor version of each Major and automatic conversion between versions. For example, if the client calls the v1 endpoint that supports the schema version 1.1.0 to request a record that was ingested with a schema version 1.0.0, SDMS will automatically convert the required record, from the ingested version 1.0.0 to the supported 1.1.0. In addition, we will support conversion between Major versions if conversion rules have been correctly specified (or an error will be thrown).
For each SCHEMA VERSION, the schema model will be documented (in the shared swagger) and examples will also be provided:
**Schema Definition**
![image](/uploads/fdd3145394948c65a09882281d9f91bc/image.png)
**Schema Example**
![image](/uploads/e9a9332bd5699d52ce113a19b3597213/image.png)
### (3) STORAGE ORGANIZATION AND CONNECTION STRINGS
Each time a FileCollection SCHEMA is ingested in SDMS, a new storage container is created in the CSP storage service. The container name will be automatically generated by SDMS by hashing the dataset name information specified in the request schema with the generated ID to guarantee the unicity of the storage resource in each partition. SDMS will provide specific endpoints to generate connection strings for a RECORD-ID and/or RECORD-VERSION to let the caller independently ingest the associated bulk. These storage resources are protected and the connection strings are released only after the caller has been authorized by the service via shared Entitlement Service (ACL-check).
These are the endpoints SDMS will expose for generating upload or download connection strings:
![image](/uploads/f63a7470d79b673e06a58c401471bb0e/image.png)
### (4) DATASET UPLOAD EXAMPLE WORKFLOW
**Dataset Registration**
![image](/uploads/e4dd914cbad85087991611aa6a4b651c/image.png)
**Dataset Ingestion**
![image](/uploads/6ff4caa1beb7699a908694370d0f6ee4/image.png)
### (5) DATASET DOWNLOAD EXAMPLE WORKFLOW
**Dataset Registration**
![image](/uploads/e4dd914cbad85087991611aa6a4b651c/image.png)
**Dataset Consumption**
![image](/uploads/8cf6d9f3d2ffeb87376202c03dc21090/image.png)
### (6) Implementation
Check [Merge Requests](#related-merge-requests) associated to this issue and [OpenAPI](app/sdms-v4/docs/openapi.yaml) definition for details.
## Decision
## Rationale
## Consequences
## When to revisitDiego MolteniDiego Moltenihttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/issues/33Parallel write support and example2024-03-07T09:17:38ZMorten OfstadParallel write support and exampleFor R3, the HPC group is interested in developing an example of how to do parallel write (e.g. using MPI). This probably requires adding some support functions to make it easier to send metadata to a central coordinator and write the chu...For R3, the HPC group is interested in developing an example of how to do parallel write (e.g. using MPI). This probably requires adding some support functions to make it easier to send metadata to a central coordinator and write the chunk-metadata pages separately.Morten OfstadMorten Ofstadhttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/issues/12Create a custom LOD generation example2023-01-13T07:37:43ZMorten OfstadCreate a custom LOD generation exampleAdd a program to the examples that reads a lower LOD, downsamples and creates a higher LOD. This example will have some value since it 1) shows how to write LODs and 2) can be run in the cloud (e.g. on AWS) after uploading the LOD 0 of t...Add a program to the examples that reads a lower LOD, downsamples and creates a higher LOD. This example will have some value since it 1) shows how to write LODs and 2) can be run in the cloud (e.g. on AWS) after uploading the LOD 0 of the data as a trigger or part of the ingestion service.Morten OfstadMorten Ofstadhttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/issues/8File backend2020-08-07T12:37:29ZMorten OfstadFile backendA file backend based on the huebds library for manipulating data-store files should be added. This needs to have a compatibility layer so it can read files written with the commercial library (translating the serialized objects into JSON...A file backend based on the huebds library for manipulating data-store files should be added. This needs to have a compatibility layer so it can read files written with the commercial library (translating the serialized objects into JSON that is similar to the objects found in an object store version of VDS). The handling of chunk-metadata and metadata-pages is done quite differently for the file format, so there is some refactoring work to be done to make this backend possible.Version 2.0https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/issues/233CRS - Problem when the data is displayed in different UTM zones at the same p...2024-03-07T09:15:19ZJuliana Fernandesjuliana.fernandes@iesbrazil.com.brCRS - Problem when the data is displayed in different UTM zones at the same project.Hello,
IesBrazil team is testing OpenVDS+ with CRS and one of the steps were to QC the data using Headwave from Bluware.
The team noticed a problem and we did a documentation on the tests that I will present below:
**Goal of the Test...Hello,
IesBrazil team is testing OpenVDS+ with CRS and one of the steps were to QC the data using Headwave from Bluware.
The team noticed a problem and we did a documentation on the tests that I will present below:
**Goal of the Tests:** Check if OpenVDS+ is adding correctly the CRS into the VDS file,<br>
**Methodology:** Convert SEGY to VDS using OpenVDS+/Headwave and QC the data using Headwave,<br>
**Data used:** Volve and Brazilian data (Volve doesn't present any problem). From Brazil we used 4 files from Solimões Basin and 1 file from Amazonas Basin, provided by ANP. The data can find [HERE](https://reate.cprm.gov.br/anp/TERRESTRE), below you can download directly all the files used in this test (from Brazil, that is where we identified the problem):
* [0233_LESTE_URUCU.3D.MIG_FIN.1.sgy](https://reate.cprm.gov.br/arquivos/index.php/s/DKD0oj9FsZAU8tI/download?path=%2FSISMICA_3D%2F0233_LESTE_URUCU%2FTEMPO%2FSISMICA&files=0233_LESTE_URUCU.3D.MIG_FIN.1.sgy) - Solimões Basin, SAD69/UTM 20S, EPSG:29190
* [0237_AEROPORTO.3D.MIG_FIN.1.sgy](https://reate.cprm.gov.br/arquivos/index.php/s/DKD0oj9FsZAU8tI/download?path=%2FSISMICA_3D%2F0237_AEROPORTO%2FTEMPO%2FSISMICA&files=0237_AEROPORTO.3D.MIG_FIN.1.sgy) - Solimões Basin, SAD69/UTM 20S, EPSG:29190
* [0237_IGARAPE_MARTA.3D.MIG_FIN.2.sgy](https://reate.cprm.gov.br/arquivos/index.php/s/DKD0oj9FsZAU8tI/download?path=%2FSISMICA_3D%2F0237_IGARAPE_MARTA%2FTEMPO%2FSISMICA&files=0237_IGARAPE_MARTA.3D.MIG_FIN.2.sgy) - Solimões Basin, SAD69/UTM 20S, EPSG:29190
* [R0300_3D_CHIBATA_PSTM.3D.PSTM.1.sgy](https://reate.cprm.gov.br/arquivos/index.php/s/DKD0oj9FsZAU8tI/download?path=%2FSISMICA_3D%2FR0300_3D_CHIBATA%2FTEMPO%2FSISMICA&files=R0300_3D_CHIBATA_PSTM.3D.PSTM.1.sgy) - Solimões Basin, SIRGAS 2000/UTM 20S, EPSG:31980
* [R0300_2D_AM_URUCARA.3D.PSTM.1.sgy](https://reate.cprm.gov.br/arquivos/index.php/s/IPNA8z7hO1vHsxI/download?path=%2FSISMICA_3D%2FR0300_3D_AM_URUCARA%2FTEMPO%2FSISMICA&files=R0300_3D_AM_URUCARA.3D.PSTM.1.sgy) - Amazonas Basin, SAD69/UTM 21S, EPSG:29191<br>
**Shapefile:** Georeferenced polygons of exploratory blocks in geographic coordinates and datum SAD69, available [HERE](https://geomaps.anp.gov.br/geoanp/),<br>
**Problem:** When the project has a different zone from the data (e.g: the project is located at SAD69/ UTM 20S and the data is located at SAD69/ UTM 21S), the file is wrongly spatially positioned (We used a VDS converted by Headwave and the original SEGY to compare),<br>
**OpenVDS+ Version:** 3.3.0,<br>
**Comparative Scenario:**
* SEGY with Original CRS
* SEGY with WGS84 CRS
* VDS from HW with Original CRS
* VDS from HW with WGS84 CRS
* VDS from OpenVDS+ with WGS84 CRS (only option available)
### First Scenario - SEGY with Original CRS
The project is under the coordinate reference system CRS SAD69/ UTM 20S (EPSG:29190) that includes most of the data of the test.<br>
In this test the team uploaded all the segy files, listed in the "Data used" topic, under the Original CRS (also informed with the data list) and displayed. The polygon in red at the superior right corner in the image is the shapefile for the block R0300_3D_AM_URUCARA.<br>
**Result: All the data are at the expected spatial position.**
![SEGY_with_Original_CRS](/uploads/285ab921116f92658bbf43007924bdbb/SEGY_with_Original_CRS.png)
### Second Scenario - SEGY with WGS84 CRS
The project is under the coordinate reference system CRS SAD69/ UTM 20S (EPSG:29190) that includes most of the data of the test.<br>
In this test the team uploaded all the segy files, listed in the "Data used" topic, under the CRS WGS84 and displayed. The polygon in red at the superior right corner in the image is the shapefile for the block R0300_3D_AM_URUCARA.<br>
**Result: All the data are at the expected spatial position.**
![SEGY_with_Original_CRS](/uploads/285ab921116f92658bbf43007924bdbb/SEGY_with_Original_CRS.png)
### Third Scenario - VDS from HW with Original CRS
The project is under the coordinate reference system CRS SAD69/ UTM 20S (EPSG:29190) that includes most of the data of the test.<br>
In this test the team converted to vds, using Headwave, all the segy files, listed in the "Data used" topic, under the Original CRS (also informed with the data list) and displayed. The polygon in red at the superior right corner in the image is the shapefile for the block R0300_3D_AM_URUCARA.<br>
**Result: All the data are at the expected spatial position.**
![SEGY_with_Original_CRS](/uploads/285ab921116f92658bbf43007924bdbb/SEGY_with_Original_CRS.png)
### Fourth Scenario - VDS from HW with WGS84 CRS
The project is under the coordinate reference system CRS SAD69/ UTM 20S (EPSG:29190) that includes most of the data of the test.<br>
In this test the team converted to vds, using Headwave, all the segy files, listed in the "Data used" topic, under the CRS WGS84 and displayed. The polygon in red at the superior right corner in the image is the shapefile for the block R0300_3D_AM_URUCARA.<br>
**Result: All the data are at the expected spatial position.**
![SEGY_with_Original_CRS](/uploads/285ab921116f92658bbf43007924bdbb/SEGY_with_Original_CRS.png)
### Fifth Scenario - VDS from OpenVDS+ with WGS84 CRS (only option available)
The project is under the coordinate reference system CRS SAD69/ UTM 20S (EPSG:29190) that includes most of the data of the test.<br>
In this test the team converted to vds, using OpenVDS+, all the segy files, listed in the "Data used" topic, under the CRS WGS84 and displayed. The polygon in red at the superior right corner in the image is the shapefile for the block R0300_3D_AM_URUCARA.<br>
**Result: The file R0300_3D_AM_URUCARA that is located under a different zone from the project (21S) is wrongly spatially positioned (Should be at the same position that the red polygon is).**
![VDS_with_WGS84_Open](/uploads/fc6638d20fffea098cd2fde1ee44dcac/VDS_with_WGS84_Open.png)
We are available for any additional information needed.
Regards,
Julianahttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/issues/220Add CRS database lookup to SEGYImport tool2023-11-16T12:21:10ZMorten OfstadAdd CRS database lookup to SEGYImport toolIt would be so much easier if it was possible to specify CRS by using the EPSG ID or UTM zone to look up the CRSWkt automatically, e.g. like this: --crs EPSG:23031 or --crs UTM-31N to get the WKT
```
PROJCS["ED50 / UTM zone 31N",GEOGCS["...It would be so much easier if it was possible to specify CRS by using the EPSG ID or UTM zone to look up the CRSWkt automatically, e.g. like this: --crs EPSG:23031 or --crs UTM-31N to get the WKT
```
PROJCS["ED50 / UTM zone 31N",GEOGCS["ED50",DATUM["European_Datum_1950",SPHEROID["International 1924",6378388,297],TOWGS84[-87,-98,-121,0,0,0,0]],PRIMEM["Greenwich",0,AUTHORITY["EPSG","8901"]],UNIT["degree",0.0174532925199433,AUTHORITY["EPSG","9122"]],AUTHORITY["EPSG","4230"]],PROJECTION["Transverse_Mercator"],PARAMETER["latitude_of_origin",0],PARAMETER["central_meridian",3],PARAMETER["scale_factor",0.9996],PARAMETER["false_easting",500000],PARAMETER["false_northing",0],UNIT["metre",1,AUTHORITY["EPSG","9001"]],AXIS["Easting",EAST],AXIS["Northing",NORTH],AUTHORITY["EPSG","23031"]]
```
I looked up this at [https://epsg.io/23031](https://epsg.io/23031)https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/issues/209Adding CRS to a VDS generated by Openvds+2023-11-16T12:21:10ZJuliana Fernandesjuliana.fernandes@iesbrazil.com.brAdding CRS to a VDS generated by Openvds+Hello,
I was taking a look into the doccumentation in order to add CRS to the VDS I'm generating with Openvds+.
In the doccumentation I saw the command "–crs-wkt <string>". The WKT is a Well-known Text and seems to be a geographical c...Hello,
I was taking a look into the doccumentation in order to add CRS to the VDS I'm generating with Openvds+.
In the doccumentation I saw the command "–crs-wkt <string>". The WKT is a Well-known Text and seems to be a geographical coordinate. There is a way to add a UTM coordinate to the data?
Regards,
Julianahttps://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds/-/issues/205fail to copy file from cloud server2023-09-19T10:20:18Znanting liufail to copy file from cloud serveri try to read vds from minio(a open-source, S3 compatible object store)
at first step,i try to read a vds file with _OpenVDS.open()_,but failed,
exception is "Error on downloading VolumeDataLayout object: Http error response: 404 -> http...i try to read vds from minio(a open-source, S3 compatible object store)
at first step,i try to read a vds file with _OpenVDS.open()_,but failed,
exception is "Error on downloading VolumeDataLayout object: Http error response: 404 -> https://endpoint/bucket-name/test.vds/VolumeDataLayout: The specified key does not exist.".
then,i realized that _open()_ can not read a vds file directly, cause the file uploaded manually.
and the second step,l try to use VDSCopy to copy the VDS file to the cloud environment,still fail! with error "Error on uploading VolumeDataLayout object: unexpected AWS signing failure",here is my command `VDSCopy.exe E:\PPCoef.vds s3://endpoint/bucket-name/testVDS -d "Region=us-west-rack-2;SecretKey=xxx;SecretAccessKey=xxx"`,my SecretKey&SecretAccessKey is correct,but l dont know why print this...
Could you please help me figure out how to deal with this situation?