README.md 3.31 KB
Newer Older
1
# Streaming Administration Service 
GitLab's avatar
GitLab committed
2

3
4
## Summary
This service provides APIs to register, start and stop streams in OSDU data platform. It currently allows registration and manipulation of 3 types of streams:
GitLab's avatar
GitLab committed
5

6
7
8
* **source** - typically a Kafka producer that connects to external system in order to receive streaming data and ingest it into OSDU data platform (e.g. WITSML/ETP producer).
* **processor** - typically a Kafka Streams application that reads streaming data ingested into OSDU data platform, process it and route it to prepared topics for downstream applications consumption (e.g. aggregate or filter ingested data).
* **sink** - typically a Kafka consumer application that reads streaming data from a prepared topic and consumes it (e.g. visualize or persist in a database).
GitLab's avatar
GitLab committed
9

10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
## More details

### API
[OpenAPI Specification](/docs/api/streaming.openapi.yaml)

### Schema

[Stream Dataset schemas](/docs/data/schemas)

[Stream.Generic.1.0.0.json - record example](/docs/data/examples/Stream.Generic.1.0.0.json)

## Configure
Before you can run and test the service locally there are a few things that need to be configured:

1. Verify that the _local_ profile is configured as active for spring boot. Check that [application.properties](/src/main/resources/application.properties) has the following setting:

```
# Profile
spring.profiles.active=local
```
2. Create a new file in the same folder and name it _application-local.properties_ with the following settings:

```
# OSDU
osdu.storage.api=https://<service-host>/api/storage/v2
```
>Note: Do not check in your _application-local.properties_ files (or any other sensitive information) to gitlab!

>Note: replace _service-host_ with the proper value of the storage service that you are working with.

>Note: Stream.Generic.1.0.0 schema must be registered with the Schema service before you can start working with Streams
>> TODO: Implement schema registration helper!

## Build and Run
Once you are in the project root directory. Execute the following commands:

To build the executable jar file.
GitLab's avatar
GitLab committed
47

48
49
50
 ``` bash
    mvn clean package
```
51
Executing the above command generates the controllers interfaces based on the streaming api spec(open api format). The spec is part of the doc folder under the source tree.
GitLab's avatar
GitLab committed
52

53
54
55
56
To run the executable as a spring boot application

 ``` bash
    mvn spring-boot:run
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
```
## Test
When you launch Spring Boot application it typically provides a few lines of the start up information. Make sure the following lines are there:

```
2021-09-30 00:06:31.339 DEBUG 2067 --- [  restartedMain] o.o.o.s.StreamingAdminApplication        : Running with Spring Boot v2.5.5, Spring v5.3.10
2021-09-30 00:06:31.339  INFO 2067 --- [  restartedMain] o.o.o.s.StreamingAdminApplication        : The following profiles are active: local
```
This indicates that your _local_ profile is activated.

```
2021-09-30 00:06:41.227  INFO 2067 --- [  restartedMain] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat started on port(s): 8080 (http) with context path '/api/streaming/v1'
```
This indicates that your Tomcat has started and is available on the following link:

http://localhost:8080/api/streaming/v1

The embedded Swagger API documentation is also available on the following link:

http://localhost:8080/api/streaming/v1/swagger-ui/

Now you can use either Postman or [stream.helpers](/scripts) to test the API.