Skip to content
Snippets Groups Projects
Commit 7aca7574 authored by Sviatoslav Nekhaienko's avatar Sviatoslav Nekhaienko
Browse files

Merge branch 'master' of... 

Merge branch 'master' of https://community.opengroup.org/osdu/platform/system/indexer-service into fix-integer-parsing

 Conflicts:
	testing/indexer-test-core/src/main/java/org/opengroup/osdu/util/ElasticUtils.java
	testing/indexer-test-core/src/main/resources/features/indexrecord/indexRecord-schema-service.feature
parents 2d51f522 acc92cfb
No related branches found
No related tags found
1 merge request!77fix unexpected conversion from int to double
Pipeline #32642 failed
Stage: review
Hide whitespace changes
Inline Side-by-side
Showing
with 329 additions and 56 deletions
.fossa.yml
+ 4
− 0
View file @ 7aca7574
......@@ -32,3 +32,7 @@ analyze:
type: mvn
target: provider/indexer-ibm/pom.xml
path: .
- name: indexer-reference
type: mvn
target: provider/indexer-reference/pom.xml
path: .
.gitignore
+ 2
− 0
View file @ 7aca7574
......@@ -41,3 +41,5 @@ provider/indexer-gcp/bin/*
# Environment configuration
*.env
.envrc
/.gradle/checksums/checksums.lock
/.gradle/6.7/fileHashes/fileHashes.lock
NOTICE
+ 31
− 0
View file @ 7aca7574
......@@ -210,6 +210,7 @@ The following software have components provided under the terms of this license:
- AutoValue (from )
- Azure AD Spring Security Integration Spring Boot Starter (from https://github.com/Microsoft/azure-spring-boot)
- Azure Metrics Spring Boot Starter (from https://github.com/Microsoft/azure-spring-boot)
- BSON (from http://bsonspec.org)
- Bean Validation API (from http://beanvalidation.org)
- Byte Buddy (without dependencies) (from )
- Byte Buddy Java agent (from )
......@@ -236,18 +237,24 @@ The following software have components provided under the terms of this license:
- Elasticsearch: 5.0.0-alpha5 (from https://github.com/elastic/elasticsearch)
- FindBugs-jsr305 (from http://findbugs.sourceforge.net/)
- Google APIs Client Library for Java (from )
- Google APIs Client Library for Java (from )
- Google App Engine extensions to the Google HTTP Client Library for Java. (from )
- Google App Engine extensions to the Google HTTP Client Library for Java. (from )
- Google Cloud Core (from https://github.com/GoogleCloudPlatform/google-cloud-java/tree/master/google-cloud-core)
- Google Cloud Core HTTP (from https://github.com/GoogleCloudPlatform/google-cloud-java/tree/master/google-cloud-core-http)
- Google Cloud Core HTTP (from https://github.com/GoogleCloudPlatform/google-cloud-java/tree/master/google-cloud-core-http)
- Google Cloud Core gRPC (from https://github.com/GoogleCloudPlatform/google-cloud-java/tree/master/google-cloud-core-grpc)
- Google Cloud Datastore (from https://github.com/GoogleCloudPlatform/google-cloud-java/tree/master/google-cloud-datastore)
- Google Cloud Datastore (from https://github.com/GoogleCloudPlatform/google-cloud-java/tree/master/google-cloud-datastore)
- Google Cloud Key Management Service (KMS) API v1-rev22-1.23.0 (from )
- Google Cloud Logging (from https://github.com/GoogleCloudPlatform/google-cloud-java/tree/master/google-cloud-logging)
- Google Cloud Pub/Sub (from https://github.com/GoogleCloudPlatform/google-cloud-java/tree/master/google-cloud-pubsub)
- Google Cloud Pub/Sub (from https://github.com/GoogleCloudPlatform/google-cloud-java/tree/master/google-cloud-pubsub)
- Google Cloud Storage (from https://github.com/GoogleCloudPlatform/google-cloud-java/tree/master/google-cloud-storage)
- Google HTTP Client Library for Java (from https://github.com/google/google-http-java-client.git)
- Google HTTP Client Library for Java (from https://github.com/google/google-http-java-client.git)
- Google OAuth Client Library for Java (from )
- Google OAuth Client Library for Java (from )
- Gson (from https://github.com/google/gson)
- Gson (from https://github.com/google/gson)
- Guava InternalFutureFailureAccess and InternalFutures (from )
......@@ -277,6 +284,7 @@ The following software have components provided under the terms of this license:
- Jackson (from http://jackson.codehaus.org)
- Jackson (from http://jackson.codehaus.org)
- Jackson 2 extensions to the Google HTTP Client Library for Java. (from https://github.com/google/google-http-java-client.git/google-http-client-jackson2)
- Jackson 2 extensions to the Google HTTP Client Library for Java. (from https://github.com/google/google-http-java-client.git/google-http-client-jackson2)
- Jackson dataformat: CBOR (from http://github.com/FasterXML/jackson-dataformats-binary)
- Jackson dataformat: CBOR (from http://github.com/FasterXML/jackson-dataformats-binary)
- Jackson datatype: JSR310 (from http://wiki.fasterxml.com/JacksonModuleJSR310)
......@@ -375,6 +383,8 @@ The following software have components provided under the terms of this license:
- Mockito (from http://www.mockito.org)
- Mockito (from http://mockito.org)
- Mojo's Maven plugin for Cobertura (from http://mojo.codehaus.org/cobertura-maven-plugin/)
- MongoDB Driver (from http://www.mongodb.org)
- MongoDB Java Driver Core (from http://www.mongodb.org)
- Netty Reactive Streams HTTP support (from )
- Netty Reactive Streams Implementation (from )
- Netty/Buffer (from http://netty.io/)
......@@ -420,20 +430,24 @@ The following software have components provided under the terms of this license:
- PowerMock (from http://www.powermock.org)
- Protocol Buffer extensions to the Google HTTP Client Library for Java. (from )
- QpidJMS Client (from )
- RabbitMQ Java Client (from http://www.rabbitmq.com)
- Reactive Streams Netty driver (from https://github.com/reactor/reactor-netty)
- Retrofit (from )
- Servlet Specification 2.5 API (from )
- Simple XML (from http://simple.sourceforge.net)
- SnakeYAML (from http://www.snakeyaml.org)
- Spatial4J (from http://www.locationtech.org/projects/locationtech.spatial4j)
- Spring AMQP Core (from https://projects.spring.io/spring-amqp)
- Spring AOP (from https://github.com/spring-projects/spring-framework)
- Spring Beans (from https://github.com/spring-projects/spring-framework)
- Spring Boot (from http://projects.spring.io/spring-boot/)
- Spring Boot AMQP Starter (from http://projects.spring.io/spring-boot/)
- Spring Boot Actuator (from http://projects.spring.io/spring-boot/)
- Spring Boot Actuator AutoConfigure (from https://projects.spring.io/spring-boot/#/spring-boot-parent/spring-boot-actuator-autoconfigure)
- Spring Boot Actuator Starter (from http://projects.spring.io/spring-boot/)
- Spring Boot AutoConfigure (from http://projects.spring.io/spring-boot/)
- Spring Boot Configuration Processor (from http://projects.spring.io/spring-boot/)
- Spring Boot Data MongoDB Starter (from http://projects.spring.io/spring-boot/)
- Spring Boot Dependencies (from http://projects.spring.io/spring-boot/)
- Spring Boot Jersey Starter (from http://projects.spring.io/spring-boot/)
- Spring Boot Json Starter (from https://projects.spring.io/spring-boot/#/spring-boot-parent/spring-boot-starters/spring-boot-starter-json)
......@@ -452,11 +466,14 @@ The following software have components provided under the terms of this license:
- Spring Core (from https://github.com/spring-projects/spring-framework)
- Spring Data Core (from )
- Spring Data Core (from )
- Spring Data MongoDB - Core (from )
- Spring Expression Language (SpEL) (from https://github.com/spring-projects/spring-framework)
- Spring JMS (from https://github.com/spring-projects/spring-framework)
- Spring Messaging (from https://github.com/spring-projects/spring-framework)
- Spring Plugin - Core (from )
- Spring Plugin - Metadata Extension (from )
- Spring RabbitMQ Support (from https://projects.spring.io/spring-amqp)
- Spring Retry (from http://www.springsource.org)
- Spring Security JWT Library (from http://github.com/spring-projects/spring-security-oauth)
- Spring Security JWT Library (from http://github.com/spring-projects/spring-security-oauth)
- Spring TestContext Framework (from https://github.com/spring-projects/spring-framework)
......@@ -475,8 +492,10 @@ The following software have components provided under the terms of this license:
- cli (from https://github.com/elastic/elasticsearch)
- cli (from https://github.com/elastic/elasticsearch)
- com.google.api.grpc:grpc-google-cloud-pubsub-v1 (from https://github.com/googleapis/googleapis)
- com.google.api.grpc:grpc-google-cloud-pubsub-v1 (from https://github.com/googleapis/googleapis)
- com.google.api.grpc:proto-google-cloud-logging-v2 (from https://github.com/googleapis/googleapis)
- com.google.api.grpc:proto-google-cloud-pubsub-v1 (from https://github.com/googleapis/googleapis)
- com.google.api.grpc:proto-google-cloud-pubsub-v1 (from https://github.com/googleapis/googleapis)
- com.google.api.grpc:proto-google-common-protos (from https://github.com/googleapis/googleapis)
- com.google.api.grpc:proto-google-iam-v1 (from https://github.com/googleapis/googleapis)
- commons-collections (from )
......@@ -534,6 +553,7 @@ The following software have components provided under the terms of this license:
- powermock-module-junit4-common (from )
- powermock-reflect (from )
- proto-google-cloud-datastore-v1 (from https://github.com/googleapis/api-client-staging)
- proto-google-cloud-datastore-v1 (from https://github.com/googleapis/api-client-staging)
- proton-j (from )
- rank-eval (from https://github.com/elastic/elasticsearch)
- rank-eval (from https://github.com/elastic/elasticsearch)
......@@ -579,6 +599,7 @@ The following software have components provided under the terms of this license:
- GAX (Google Api eXtensions) (from https://github.com/googleapis)
- GAX (Google Api eXtensions) (from https://github.com/googleapis)
- GAX (Google Api eXtensions) (from https://github.com/googleapis)
- GAX (Google Api eXtensions) (from https://github.com/googleapis)
- Hamcrest Core (from http://hamcrest.org/)
- Lucene Common Analyzers (from )
- Lucene Common Analyzers (from )
......@@ -607,6 +628,8 @@ The following software have components provided under the terms of this license:
- GAX (Google Api eXtensions) (from https://github.com/googleapis)
- GAX (Google Api eXtensions) (from https://github.com/googleapis)
- GAX (Google Api eXtensions) (from https://github.com/googleapis)
- GAX (Google Api eXtensions) (from https://github.com/googleapis)
- Google APIs Client Library for Java (from )
- Google APIs Client Library for Java (from )
- Google Auth Library for Java - Credentials (from )
- Google Auth Library for Java - OAuth2 HTTP (from )
......@@ -653,6 +676,7 @@ The following software have components provided under the terms of this license:
- Checker Qual (from https://checkerframework.org)
- FindBugs-jsr305 (from http://findbugs.sourceforge.net/)
- MongoDB Java Driver Core (from http://www.mongodb.org)
========================================================================
CC-BY-4.0
......@@ -767,6 +791,7 @@ The following software have components provided under the terms of this license:
- JavaMail API (from )
- Mojo's Maven plugin for Cobertura (from http://mojo.codehaus.org/cobertura-maven-plugin/)
- OSGi resource locator (from )
- RabbitMQ Java Client (from http://www.rabbitmq.com)
- Run Level Service (from )
- ServiceLocator Default Implementation (from git://java.net/hk2~git/hk2-locator)
- aopalliance-repackaged (from )
......@@ -806,6 +831,7 @@ The following software have components provided under the terms of this license:
- JavaBeans Activation Framework API jar (from )
- JavaMail API (from )
- OSGi resource locator (from )
- RabbitMQ Java Client (from http://www.rabbitmq.com)
- ServiceLocator Default Implementation (from git://java.net/hk2~git/hk2-locator)
- aopalliance-repackaged (from )
- javax.annotation-api (from http://jcp.org/en/jsr/detail?id=250)
......@@ -872,6 +898,7 @@ The following software have components provided under the terms of this license:
- Apache Log4j API (from )
- Apache Log4j Core (from )
- RabbitMQ Java Client (from http://www.rabbitmq.com)
========================================================================
MIT
......@@ -938,6 +965,7 @@ The following software have components provided under the terms of this license:
- Cobertura code coverage (from http://cobertura.sourceforge.net)
- Javassist (from http://www.javassist.org/)
- Javassist (from http://www.javassist.org/)
- RabbitMQ Java Client (from http://www.rabbitmq.com)
========================================================================
MPL-2.0
......@@ -946,6 +974,7 @@ The following software have components provided under the terms of this license:
- Javassist (from http://www.javassist.org/)
- Javassist (from http://www.javassist.org/)
- RabbitMQ Java Client (from http://www.rabbitmq.com)
========================================================================
PHP-3.01
......@@ -1022,6 +1051,7 @@ The following software have components provided under the terms of this license:
- Microsoft Azure SDK for SQL API of Azure Cosmos DB Service (from https://github.com/Azure/azure-sdk-for-java)
- Microsoft Azure client library for Blob Storage (from https://github.com/Azure/azure-sdk-for-java)
- Project Lombok (from https://projectlombok.org)
- RabbitMQ Java Client (from http://www.rabbitmq.com)
- Spring Security JWT Library (from http://github.com/spring-projects/spring-security-oauth)
- Spring Security JWT Library (from http://github.com/spring-projects/spring-security-oauth)
- Spring Web (from https://github.com/spring-projects/spring-framework)
......@@ -1042,6 +1072,7 @@ The following software have components provided under the terms of this license:
- Byte Buddy (without dependencies) (from )
- JUnit (from http://junit.org)
- JavaBeans Activation Framework API jar (from )
- RabbitMQ Java Client (from http://www.rabbitmq.com)
- Servlet Specification 2.5 API (from )
- Spongy Castle (from http://rtyley.github.io/spongycastle/)
- System Rules (from http://stefanbirkner.github.io/system-rules/)
......
devops/azure/chart/templates/deployment.yaml
+ 3
− 3
View file @ 7aca7574
......@@ -56,7 +56,7 @@ spec:
readOnly: true
env:
- name: spring_application_name
value: indexer
value: {{ .Chart.Name }}
- name: server.servlet.contextPath
value: /api/indexer/v2/
- name: server_port
......@@ -88,7 +88,7 @@ spec:
- name: servicebus_topic_name
value: indexing-progress
- name: entitlements_service_endpoint
value: http://entitlements-azure/entitlements/v1
value: http://entitlements/api/entitlements/v2
- name: entitlements_service_api_key
value: "OBSOLETE"
- name: schema_service_url
......@@ -100,7 +100,7 @@ spec:
- name: STORAGE_QUERY_RECORD_FOR_CONVERSION_HOST
value: http://storage/api/storage/v2/query/records:batch
- name: STORAGE_QUERY_RECORD_HOST
value: http://storage/api/storage/v2/records
value: http://storage/api/storage/v2/query/records
- name: partition_service_endpoint
value: http://partition/api/partition/v1
- name: azure_istioauth_enabled
......
devops/azure/chart/templates/hpa.yaml
+ 1
− 1
View file @ 7aca7574
......@@ -24,7 +24,7 @@ spec:
kind: Deployment
name: {{ .Chart.Name }}
minReplicas: {{ .Values.global.replicaCount }}
maxReplicas: 5
maxReplicas: 10
metrics:
- type: Resource
resource:
......
devops/azure/release.yaml
+ 1
− 1
View file @ 7aca7574
......@@ -161,4 +161,4 @@ spec:
- name: STORAGE_QUERY_RECORD_FOR_CONVERSION_HOST
value: http://storage/api/storage/v2/query/records:batch
- name: STORAGE_QUERY_RECORD_HOST
value: http://storage/api/storage/v2/records
value: http://storage/api/storage/v2/query/records
indexer-core/pom.xml
+ 4
− 2
View file @ 7aca7574
......@@ -4,18 +4,19 @@
<parent>
<groupId>org.opengroup.osdu.indexer</groupId>
<artifactId>indexer-service</artifactId>
<version>0.6.0-SNAPSHOT</version>
<version>0.8.0-SNAPSHOT</version>
<relativePath>../pom.xml</relativePath>
</parent>
<artifactId>indexer-core</artifactId>
<version>0.6.0-SNAPSHOT</version>
<version>0.8.0-SNAPSHOT</version>
<name>indexer-core</name>
<description>Indexer Service Core</description>
<packaging>jar</packaging>
<properties>
<commons-beanutils.version>1.9.4</commons-beanutils.version>
<osdu.oscorecommon.version>0.6.5</osdu.oscorecommon.version>
</properties>
<dependencies>
......@@ -33,6 +34,7 @@
<dependency>
<groupId>org.opengroup.osdu</groupId>
<artifactId>os-core-common</artifactId>
<version>${osdu.oscorecommon.version}</version>
</dependency>
<!-- spring boot dependencies -->
......
indexer-core/src/main/java/org/opengroup/osdu/indexer/error/GlobalExceptionMapperCore.java 0 → 100644
+ 88
− 0
View file @ 7aca7574
// Copyright © Microsoft Corporation
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package org.opengroup.osdu.indexer.error;
import javax.validation.ValidationException;
import javassist.NotFoundException;
import org.opengroup.osdu.core.common.logging.JaxRsDpsLog;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.core.Ordered;
import org.springframework.core.annotation.Order;
import org.springframework.http.HttpStatus;
import org.springframework.http.ResponseEntity;
import org.springframework.security.access.AccessDeniedException;
import org.springframework.web.bind.annotation.ControllerAdvice;
import org.springframework.web.bind.annotation.ExceptionHandler;
import org.springframework.web.servlet.mvc.method.annotation.ResponseEntityExceptionHandler;
import org.opengroup.osdu.core.common.model.http.AppException;
@Order(Ordered.HIGHEST_PRECEDENCE - 1)
@ControllerAdvice
public class GlobalExceptionMapperCore extends ResponseEntityExceptionHandler {
@Autowired
private JaxRsDpsLog logger;
@ExceptionHandler(AppException.class)
protected ResponseEntity<Object> handleAppException(AppException e) {
return this.getErrorResponse(e);
}
@ExceptionHandler(ValidationException.class)
protected ResponseEntity<Object> handleValidationException(ValidationException e) {
return this.getErrorResponse(
new AppException(HttpStatus.BAD_REQUEST.value(), "Validation error.", e.getMessage(), e));
}
@ExceptionHandler(NotFoundException.class)
protected ResponseEntity<Object> handleNotFoundException(NotFoundException e) {
return this.getErrorResponse(
new AppException(HttpStatus.NOT_FOUND.value(), "Resource not found.", e.getMessage(), e));
}
@ExceptionHandler(AccessDeniedException.class)
protected ResponseEntity<Object> handleAccessDeniedException(AccessDeniedException e) {
return this.getErrorResponse(
new AppException(HttpStatus.FORBIDDEN.value(), "Access denied", e.getMessage(), e));
}
@ExceptionHandler(Exception.class)
protected ResponseEntity<Object> handleGeneralException(Exception e) {
return this.getErrorResponse(
new AppException(HttpStatus.INTERNAL_SERVER_ERROR.value(), "Server error.",
"An unknown error has occurred.", e));
}
private ResponseEntity<Object> getErrorResponse(AppException e) {
String exceptionMsg = e.getOriginalException() != null
? e.getOriginalException().getMessage()
: e.getError().getMessage();
if( e.getCause() instanceof Exception) {
Exception original = (Exception) e.getCause();
this.logger.error(original.getMessage(), original);
}
if (e.getError().getCode() > 499) {
this.logger.error(exceptionMsg, e);
} else {
this.logger.warning(exceptionMsg, e);
}
return new ResponseEntity<Object>(e.getError(), HttpStatus.resolve(e.getError().getCode()));
}
}
\ No newline at end of file
indexer-core/src/main/java/org/opengroup/osdu/indexer/schema/converter/PropertiesProcessor.java
+ 59
− 8
View file @ 7aca7574
......@@ -56,8 +56,10 @@ public class PropertiesProcessor {
public Stream<Map<String, Object>> processItem(AllOfItem allOfItem) {
Preconditions.checkNotNull(allOfItem, "allOfItem cannot be null");
if (Objects.nonNull(allOfItem.getAllOf())) {
return allOfItem.getAllOf().stream().flatMap(this::processItem);
Stream<Map<String, Object>> ofItems = processOfItems(allOfItem.getAllOf(), allOfItem.getAnyOf(), allOfItem.getOneOf());
if (Objects.nonNull(ofItems)) {
return ofItems;
}
String ref = allOfItem.getRef();
......@@ -89,13 +91,34 @@ public class PropertiesProcessor {
new AppException(HttpStatus.SC_NOT_FOUND, "Failed to find definition:" + definitionSubRef,
"Unknown definition:" + definitionSubRef));
if (Objects.nonNull(definition.getAllOf())) {
return definition.getAllOf().stream().flatMap(this::processItem);
Stream<Map<String, Object>> ofItems =
processOfItems(definition.getAllOf(), definition.getAnyOf(), definition.getOneOf());
if (Objects.nonNull(ofItems)) {
return ofItems;
}
return processProperties(definition.getProperties());
}
private Stream<Map<String, Object>> processOfItems(List<AllOfItem> allOf, List<AllOfItem> anyOf, List<AllOfItem> oneOf) {
Stream<Map<String, Object>> ofItems = null;
if (Objects.nonNull(allOf)) {
ofItems = allOf.stream().flatMap(this::processItem);
}
if (Objects.nonNull(anyOf)) {
ofItems = Stream.concat(Optional.ofNullable(ofItems).orElseGet(Stream::empty), anyOf.stream().flatMap(this::processItem));
}
if (Objects.nonNull(oneOf)) {
ofItems = Stream.concat(Optional.ofNullable(ofItems).orElseGet(Stream::empty), oneOf.stream().flatMap(this::processItem));
}
return ofItems;
}
public Stream<Map<String, Object>> processProperties(Map<String, TypeProperty> properties){
return properties.entrySet().stream().flatMap(this::processPropertyEntry);
}
......@@ -118,11 +141,10 @@ public class PropertiesProcessor {
return Stream.empty();
}
if (Objects.nonNull(entry.getValue().getAllOf())) {
PropertiesProcessor propertiesProcessor = new PropertiesProcessor(definitions, pathPrefixWithDot + entry.getKey()
, log, new SchemaConverterPropertiesConfig());
Stream<Map<String, Object>> ofItems = processOfItems(entry);
return entry.getValue().getAllOf().stream().flatMap(propertiesProcessor::processItem);
if (Objects.nonNull(ofItems)) {
return ofItems;
}
if (Objects.nonNull(entry.getValue().getProperties())) {
......@@ -139,6 +161,35 @@ public class PropertiesProcessor {
return storageSchemaEntry(getTypeByDefinitionProperty(entry.getValue()), pathPrefixWithDot + entry.getKey());
}
private Stream<Map<String, Object>> processOfItems(Map.Entry<String, TypeProperty> entry) {
Stream<Map<String, Object>> ofItems = null;
if (Objects.nonNull(entry.getValue().getAllOf())) {
PropertiesProcessor propertiesProcessor = new PropertiesProcessor(definitions, pathPrefixWithDot + entry.getKey()
, log, new SchemaConverterPropertiesConfig());
ofItems = entry.getValue().getAllOf().stream().flatMap(propertiesProcessor::processItem);
}
if (Objects.nonNull(entry.getValue().getAnyOf())) {
PropertiesProcessor propertiesProcessor = new PropertiesProcessor(definitions, pathPrefixWithDot + entry.getKey()
, log, new SchemaConverterPropertiesConfig());
ofItems = Stream.concat(Optional.ofNullable(ofItems).orElseGet(Stream::empty),
entry.getValue().getAnyOf().stream().flatMap(propertiesProcessor::processItem));
}
if (Objects.nonNull(entry.getValue().getOneOf())) {
PropertiesProcessor propertiesProcessor = new PropertiesProcessor(definitions, pathPrefixWithDot + entry.getKey()
, log, new SchemaConverterPropertiesConfig());
ofItems = Stream.concat(Optional.ofNullable(ofItems).orElseGet(Stream::empty),
entry.getValue().getOneOf().stream().flatMap(propertiesProcessor::processItem));
}
return ofItems;
}
private Stream<Map<String, Object>> storageSchemaEntry(String kind, String path) {
Preconditions.checkNotNullOrEmpty(kind, "kind cannot be null or empty");
Preconditions.checkNotNullOrEmpty(path, "path cannot be null or empty");
......
indexer-core/src/main/java/org/opengroup/osdu/indexer/schema/converter/SchemaToStorageFormatImpl.java
+ 12
− 0
View file @ 7aca7574
......@@ -97,6 +97,18 @@ public class SchemaToStorageFormatImpl implements SchemaToStorageFormat {
.collect(Collectors.toList()));
}
if (schemaData.getAnyOf() != null) {
storageSchemaItems.addAll(schemaServiceSchema.getProperties().getData().getAnyOf().stream()
.flatMap(propertiesProcessor::processItem)
.collect(Collectors.toList()));
}
if (schemaData.getOneOf() != null) {
storageSchemaItems.addAll(schemaServiceSchema.getProperties().getData().getOneOf().stream()
.flatMap(propertiesProcessor::processItem)
.collect(Collectors.toList()));
}
if (schemaData.getRef() != null) {
storageSchemaItems.addAll(propertiesProcessor.processRef(schemaData.getRef())
.collect(Collectors.toList()));
......
indexer-core/src/main/java/org/opengroup/osdu/indexer/schema/converter/config/SchemaConverterPropertiesConfig.java
+ 1
− 1
View file @ 7aca7574
......@@ -24,7 +24,7 @@ public class SchemaConverterPropertiesConfig implements SchemaConverterConfig {
}
private Set<String> getDefaultSupportedArrayTypes() {
return new HashSet<>(Arrays.asList("boolean", "integer", "number", "string"));
return new HashSet<>(Arrays.asList("boolean", "integer", "number", "string", "object"));
}
private Map<String, String> getDefaultSpecialDefinitionsMap() {
......
indexer-core/src/main/java/org/opengroup/osdu/indexer/schema/converter/readme.md
+ 64
− 1
View file @ 7aca7574
......@@ -299,4 +299,67 @@ Ignored for now (array of references)
```
\"kind\": \"long\"
\ No newline at end of file
\"kind\": \"long\"
Processing specifics
----------------------------------------------------------------------------
allOf, anyOf and oneOf tags are processed at the same way. All internal data(properties) are included into converted schema.
For instance
```json
{
"definitions": {
"wellboreData1": {
"properties": {
"prop1": {
"type": "string"
}
}
},
"wellboreData2": {
"properties": {
"prop2": {
"type": "string"
}
}
}
},
"properties": {
"data": {
"allOf": [
{
"anyOf": [
{
"$ref": "#/definitions/wellboreData1"
} ],
"oneOf": [
{
"$ref": "#/definitions/wellboreData2"
}
]
}
]
}
}
}
```
is converted to
```json
{
"kind": "KIND_VAL",
"schema": [
{
"kind": "string",
"path": "prop1"
},
{
"kind": "string",
"path": "prop2"
}
]
}
```
indexer-core/src/main/java/org/opengroup/osdu/indexer/schema/converter/tags/AllOfItem.java
+ 2
− 0
View file @ 7aca7574
......@@ -27,4 +27,6 @@ public class AllOfItem {
private String type;
private Map<String, TypeProperty> properties;
private List<AllOfItem> allOf;
private List<AllOfItem> oneOf;
private List<AllOfItem> anyOf;
}
\ No newline at end of file
indexer-core/src/main/java/org/opengroup/osdu/indexer/schema/converter/tags/Definition.java
+ 2
− 0
View file @ 7aca7574
......@@ -23,4 +23,6 @@ import java.util.Map;
public class Definition {
private Map<String, TypeProperty> properties;
private List<AllOfItem> allOf;
private List<AllOfItem> oneOf;
private List<AllOfItem> anyOf;
}
indexer-core/src/main/java/org/opengroup/osdu/indexer/schema/converter/tags/PropertiesData.java
+ 2
− 0
View file @ 7aca7574
......@@ -23,6 +23,8 @@ import java.util.Map;
@Data
public class PropertiesData {
private List<AllOfItem> allOf;
private List<AllOfItem> oneOf;
private List<AllOfItem> anyOf;
@JsonProperty("$ref")
private String ref;
private Map<String, TypeProperty> properties;
......
indexer-core/src/main/java/org/opengroup/osdu/indexer/schema/converter/tags/TypeProperty.java
+ 2
− 0
View file @ 7aca7574
......@@ -30,4 +30,6 @@ public class TypeProperty {
private Items items;
private Map<String, TypeProperty> properties;
private List<AllOfItem> allOf;
private List<AllOfItem> oneOf;
private List<AllOfItem> anyOf;
}
indexer-core/src/main/java/org/opengroup/osdu/indexer/service/AttributeParsingServiceImpl.java
+ 10
− 0
View file @ 7aca7574
......@@ -208,6 +208,16 @@ public class AttributeParsingServiceImpl implements IAttributeParsingService {
}
}
@Override
public void tryParseNested(String recordId, String name, Object value, Map<String, Object> dataMap) {
dataMap.put(name,value);
}
@Override
public void tryParseObject(String recordId, String name, Object value, Map<String, Object> dataMap) {
dataMap.put(name,value);
}
private List<String> isArrayType(Object attributeVal) {
try {
......
indexer-core/src/main/java/org/opengroup/osdu/indexer/service/IAttributeParsingService.java
+ 4
− 2
View file @ 7aca7574
package org.opengroup.osdu.indexer.service;
import org.opengroup.osdu.core.common.model.indexer.IndexSchema;
import java.util.Map;
public interface IAttributeParsingService {
......@@ -26,4 +24,8 @@ public interface IAttributeParsingService {
void tryParseGeopoint(String recordId, String attributeName, Map<String, Object> storageRecordData, Map<String, Object> dataMap);
void tryParseGeojson(String recordId, String attributeName, Object attributeVal, Map<String, Object> dataMap);
void tryParseNested(String recordId, String name, Object value, Map<String, Object> dataMap);
void tryParseObject(String recordId, String name, Object value, Map<String, Object> dataMap);
}
indexer-core/src/main/java/org/opengroup/osdu/indexer/service/IndexSchemaServiceImpl.java
+ 14
− 0
View file @ 7aca7574
......@@ -23,6 +23,7 @@ import org.elasticsearch.client.RestHighLevelClient;
import org.opengroup.osdu.core.common.logging.JaxRsDpsLog;
import org.opengroup.osdu.core.common.model.http.AppException;
import org.opengroup.osdu.core.common.model.http.RequestStatus;
import org.opengroup.osdu.core.common.model.indexer.ElasticType;
import org.opengroup.osdu.core.common.model.indexer.IndexSchema;
import org.opengroup.osdu.core.common.model.indexer.OperationType;
import org.opengroup.osdu.core.common.model.indexer.StorageType;
......@@ -47,6 +48,10 @@ import java.util.Map;
public class IndexSchemaServiceImpl implements IndexSchemaService {
private static final String FLATTENED_SCHEMA = "_flattened";
private static final String WELLBORE_MARKER_SET = "WellboreMarkerSet";
private static final String MARKERS = "Markers";
private static final String WELL_LOG = "WellLog";
private static final String CURVES = "Curves";
private final Gson gson = new Gson();
......@@ -221,6 +226,7 @@ public class IndexSchemaServiceImpl implements IndexSchemaService {
meta.put(RecordMetaAttribute.TYPE.getValue(), TypeMapper.getIndexerType(RecordMetaAttribute.TYPE));
meta.put(RecordMetaAttribute.ACL.getValue(), TypeMapper.getIndexerType(RecordMetaAttribute.ACL));
meta.put(RecordMetaAttribute.X_ACL.getValue(), TypeMapper.getIndexerType(RecordMetaAttribute.X_ACL));
meta.put(RecordMetaAttribute.TAGS.getValue(), TypeMapper.getIndexerType(RecordMetaAttribute.TAGS));
meta.put(RecordMetaAttribute.LEGAL.getValue(), TypeMapper.getIndexerType(RecordMetaAttribute.LEGAL));
meta.put(RecordMetaAttribute.ANCESTRY.getValue(), TypeMapper.getIndexerType(RecordMetaAttribute.ANCESTRY));
meta.put(RecordMetaAttribute.INDEX_STATUS.getValue(), TypeMapper.getIndexerType(RecordMetaAttribute.INDEX_STATUS));
......@@ -228,6 +234,14 @@ public class IndexSchemaServiceImpl implements IndexSchemaService {
String kind = schemaObj.getKind();
String type = kind.split(":")[2];
//TODO temporary fix for https://community.opengroup.org/osdu/platform/system/indexer-service/-/issues/1
if(data.get(MARKERS) != null){
data.put(MARKERS, ElasticType.NESTED.getValue());
}
if(data.get(CURVES) != null){
data.put(CURVES, ElasticType.NESTED.getValue());
}
return IndexSchema.builder().dataSchema(data).metaSchema(meta).kind(kind).type(type).build();
} catch (Exception e) {
......
indexer-core/src/main/java/org/opengroup/osdu/indexer/service/IndexerMappingServiceImpl.java
+ 23
− 37
View file @ 7aca7574
......@@ -15,12 +15,6 @@
package org.opengroup.osdu.indexer.service;
import com.google.gson.Gson;
import java.io.IOException;
import java.util.Arrays;
import java.util.HashMap;
import java.util.Map;
import java.util.Set;
import javax.inject.Inject;
import org.apache.http.HttpStatus;
import org.elasticsearch.ElasticsearchException;
import org.elasticsearch.action.support.master.AcknowledgedResponse;
......@@ -38,28 +32,29 @@ import org.elasticsearch.index.reindex.UpdateByQueryRequest;
import org.opengroup.osdu.core.common.Constants;
import org.opengroup.osdu.core.common.logging.JaxRsDpsLog;
import org.opengroup.osdu.core.common.model.http.AppException;
import org.opengroup.osdu.core.common.model.indexer.DEAnalyzerType;
import org.opengroup.osdu.core.common.model.indexer.ElasticType;
import org.opengroup.osdu.core.common.model.indexer.IndexSchema;
import org.opengroup.osdu.core.common.model.indexer.Records;
import org.opengroup.osdu.core.common.model.search.RecordMetaAttribute;
import org.opengroup.osdu.core.common.search.Preconditions;
import org.opengroup.osdu.indexer.config.IndexerConfigurationProperties;
import org.opengroup.osdu.indexer.util.ElasticClientHandler;
import org.opengroup.osdu.indexer.util.TypeMapper;
import org.springframework.stereotype.Service;
import javax.inject.Inject;
import java.io.IOException;
import java.util.Arrays;
import java.util.HashMap;
import java.util.Map;
import java.util.Set;
@Service
public class IndexerMappingServiceImpl extends MappingServiceImpl implements IndexerMappingService {
@Inject
private IndexerConfigurationProperties configurationProperties;
@Inject
private JaxRsDpsLog log;
@Inject
private ElasticClientHandler elasticClientHandler;
private TimeValue REQUEST_TIMEOUT = TimeValue.timeValueMinutes(1);
/**
* Create a new type in Elasticsearch
*
......@@ -81,7 +76,7 @@ public class IndexerMappingServiceImpl extends MappingServiceImpl implements Ind
*
* @param schema Index schema
* @param type Mapping type
* @return String JSON represnetation of type and elastic type
* @return String JSON representation of type and elastic type
*
* sample index mapping:
* "properties": {
......@@ -111,25 +106,14 @@ public class IndexerMappingServiceImpl extends MappingServiceImpl implements Ind
// meta attribute
Map<String, Object> metaMapping = new HashMap<>();
for (Map.Entry<String, Object> entry : schema.getMetaSchema().entrySet()) {
String key = entry.getKey();
if (key.equals(RecordMetaAttribute.ACL.getValue()) || key.equals(RecordMetaAttribute.LEGAL.getValue()) || key.equals(RecordMetaAttribute.ANCESTRY.getValue()) || key.equals(RecordMetaAttribute.INDEX_STATUS.getValue())) {
metaMapping.put(key, entry.getValue());
} else {
metaMapping.put(key, Records.Type.builder().type(entry.getValue().toString()).build());
}
metaMapping.put(entry.getKey(), TypeMapper.getMetaAttributeIndexerMapping(entry.getKey()));
}
// data-source attributes
Map<String, Object> dataMapping = new HashMap<>();
if (schema.getDataSchema() != null) {
for (Map.Entry<String, String> entry : schema.getDataSchema().entrySet()) {
// Apply de_indexer_analyzer and de_search_analyzer to TEXT field
if (configurationProperties.isPreDemo() && ElasticType.TEXT.getValue().equalsIgnoreCase(entry.getValue())) {
log.info(String.format("indexing %s with custom analyzer", entry.getKey()));
dataMapping.put(entry.getKey(), Records.Analyzer.builder().type(entry.getValue()).analyzer(DEAnalyzerType.INDEXER_ANALYZER.getValue()).search_analyzer(DEAnalyzerType.SEARCH_ANALYZER.getValue()).build());
} else {
dataMapping.put(entry.getKey(), Records.Type.builder().type(entry.getValue()).build());
}
dataMapping.put(entry.getKey(), TypeMapper.getDataAttributeIndexerMapping(entry.getValue()));
}
// inner properties.data.properties block
......@@ -159,7 +143,7 @@ public class IndexerMappingServiceImpl extends MappingServiceImpl implements Ind
}
}
}
private boolean updateMappingToEnableKeywordIndexingForField(RestHighLevelClient client, Set<String> indicesSet, String fieldName) throws IOException {
String[] indices = indicesSet.toArray(new String[indicesSet.size()]);
Map<String, Map<String, Map<String, GetFieldMappingsResponse.FieldMappingMetadata>>> indexMappingMap = getIndexFieldMap(new String[]{"data."+fieldName}, client, indices);
......@@ -190,7 +174,7 @@ public class IndexerMappingServiceImpl extends MappingServiceImpl implements Ind
}
}
}
return indexMappingMap;
} catch (ElasticsearchException e) {
log.error(String.format("Failed to get indices: %s. | Error: %s", Arrays.toString(indices), e));
......@@ -198,7 +182,9 @@ public class IndexerMappingServiceImpl extends MappingServiceImpl implements Ind
}
}
private boolean updateMappingForAllIndicesOfSameTypeToEnableKeywordIndexingForField(RestHighLevelClient client, String index, Map<String, Map<String, GetFieldMappingsResponse.FieldMappingMetadata>> indexMapping, String fieldName) throws IOException {
private boolean updateMappingForAllIndicesOfSameTypeToEnableKeywordIndexingForField(
RestHighLevelClient client, String index, Map<String, Map<String, GetFieldMappingsResponse.FieldMappingMetadata>> indexMapping, String fieldName) throws IOException {
PutMappingRequest request = new PutMappingRequest(index);
String type = indexMapping.keySet().iterator().next();
if(type.isEmpty()) {
......@@ -219,7 +205,7 @@ public class IndexerMappingServiceImpl extends MappingServiceImpl implements Ind
log.error(String.format("Could not find field: %s in the mapping of index: %s.", fieldName, index));
return false;
}
//Index the field with additional keyword type
Map<String, Object> keywordMap = new HashMap<>();
keywordMap.put(Constants.TYPE, "keyword");
......@@ -235,22 +221,22 @@ public class IndexerMappingServiceImpl extends MappingServiceImpl implements Ind
data.put(Constants.DATA,mapping);
Map<String, Object> properties = new HashMap<>();
properties.put(Constants.PROPERTIES, data);
request.source(new Gson().toJson(properties), XContentType.JSON);
try {
AcknowledgedResponse response = client.indices().putMapping(request, RequestOptions.DEFAULT);
boolean isIndicesUpdated = updateIndices(client, index);
return response.isAcknowledged() && isIndicesUpdated;
} catch (Exception e) {
log.error(String.format("Could not update mapping of index: %s. | Error: %s", index, e));
return false;
}
}
private boolean updateIndices(RestHighLevelClient client, String index) throws IOException {
UpdateByQueryRequest request = new UpdateByQueryRequest(index);
UpdateByQueryRequest request = new UpdateByQueryRequest(index);
request.setConflicts("proceed");
BulkByScrollResponse response = client.updateByQuery(request, RequestOptions.DEFAULT);
if(!response.getBulkFailures().isEmpty()) {
......@@ -259,7 +245,7 @@ public class IndexerMappingServiceImpl extends MappingServiceImpl implements Ind
}
return true;
}
/**
* Create a new type in Elasticsearch
*
......
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment