Create Pluggable approach for Mappers and Drivers authored by Rustam Lotsmanenko (EPAM)'s avatar Rustam Lotsmanenko (EPAM)
## Solution description, key components, and use cases.
### 1. Introduction
This documentation outlines the process of building Drivers into Uber jars using the Maven shade plugin and subsequently utilizing these Uber jars as Spring plugins in OSDU applications.
### 2. Solution Overview
The solution is built upon two primary pillars: the extensive use of the Spring framework's Dependency Injection in the core platform services written in Java and the capability to create Uber Jars using the Maven Shade plugin.
Key points:
- **Creating a Standalone Driver Jar**: The solution allows the Driver jar to function as a standalone artifact with all the necessary components for proper operation. The Driver is responsible for assembling data sources, maintaining them, and providing their unique dependencies.
- **Dynamic Dependency Wiring**: This solution enables dynamic dependency wiring at runtime, allowing for the swapping of dependencies as needed. The OSDU service relies on the Driver's self-sufficiency, which involves autonomous component assembly within the Driver. This approach fosters a loose coupling that allows for dependency swaps without requiring changes in the service.
- **Plugin usage is not mandatory**. Provider implementations could use community Drivers and their own Drivers as direct dependencies of their service implementation. The compile step configured for community Drivers is not mandatory for provider Drivers. Furthermore, the community Drivers will have 2 artifacts, the original jar which could be used as a regular dependency, and the Uber jar or plugin jar.
### 3. Use Cases
#### Common Binaries and Custom Drivers:
- The OSDU platform can be readily used with common binaries and custom Drivers.
- This eases the effort required to reuse community implementations while accommodating provider-specific sets of Drivers.
- Reusing the community OSDU platform implementation becomes more accessible with reduced effort.
- Community service implementations serve as tested binaries, maintained by the community, and open to updates and contributions.
- To implement the platform on a custom set of technologies, the primary requirement is to develop the necessary Drivers while reusing tested binaries from the platform.
#### Hybrid Platform:
- The current state of the platform doesn't allow selective swapping of technologies.
- The introduction of a plugin-based platform provides the capability to selectively swap underlying resources.
#### Maven Shade and Plugin-Based Approaches Not Mandatory:
- The plugin-based approach does not impose mandatory usage of Maven Shade and Uber jars.
- Providers have the option to introduce their drivers as they are and use them in the provider module as direct dependencies.
- Additionally, the community's south decision point or core-plus module won't require providers to remove already bundled community dependencies since they are not present.
### 4. Prerequisites
**For Local Development Build Environment**:
- JDK 17
- Maven
**For OSDU Service to Use a Pluggable Driver**:
- Java 17
- Spring Boot framework version 2.7.10 or higher
- Spring Boot loader dependency version 2.7.10 or higher https://mvnrepository.com/artifact/org.springframework.boot/spring-boot-loader
- Configured Spring Boot component scanning for "org.opengroup.osdu."
- Direct dependencies: OSM-core, OQM-core, or OBM-core
- Spring context configuration, expecting core Driver interface-based beans to be configured
**For Driver to be discoverable by OSDU service**:
- Java 17
- Spring context version 5.3.30 or higher
- Spring context configuration annotations
- "org.opengroup.osdu." group id and base src package
- Maven shade plugin (or any other build tool that allows building Uber jars)
**For Container Image Build Environment**:
- JDK 17
- Maven
- Docker
### 5. Building Uber Jars with Maven Shade Plugin
#### Maven shade plugin in Driver pom build configuration:
~~~
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>3.5.1</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
...
</plugin>
~~~
#### Maven shade configuration key parts:
1. **Avoid Creating a Dependency-Reduced Pom**:
- By default, the Maven Shade Plugin produces a dependency-reduced POM file for the Uber jar and deploys it to the Artifactory.
- The Uber jar already contains all the required dependencies, and this step is not necessary when the Uber jar is not used as a project dependency.
- To disable this feature, add the following configuration:
~~~
<createDependencyReducedPom>false</createDependencyReducedPom>
~~~
2. **Deploy the Uber Jar with the Original Jar**:
- To keep both the Uber jar and the original jar.
~~~
<shadedArtifactAttached>true</shadedArtifactAttached>
~~~
3. **Add a Classifier**:
- It's a common practice to use classifiers to distinguish jars. T
- To classify the plugin jar, use the shadedClassifierName property.
- https://www.baeldung.com/maven-artifact-classifiers
- https://maven.apache.org/plugins/maven-jar-plugin/examples/attached-jar.html
~~~
<shadedClassifierName>plugin</shadedClassifierName>
~~~
4. **Remove Signatures from Dependencies**:
- Some dependencies of a Driver might have bundled signatures for compiled binaries. Since we are reassembling them, it's essential to remove these signatures. Use the following configuration.
~~~
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
</excludes>
</filter>
</filters>
~~~
5. **Specify Included Dependencies**:
- Ideally, the Driver should contain a minimal set of dependencies that need to be included in the Uber jar. Point out these dependencies directly in the configuration.
~~~
<artifactSet>
<includes>
<include>org.postgresql:postgresql</include>
<include>com.zaxxer:HikariCP</include>
</includes>
</artifactSet>
~~~
6. **Relocate Included Dependencies**:
- To avoid potential classpath conflicts, relocate included dependencies. This is particularly important because you may not have control over which dependencies will be present in the consumer service.
~~~
<relocations>
<relocation>
<pattern>org.postgresql</pattern>
<shadedPattern>shaded.org.postgresql</shadedPattern>
</relocation>
<relocation>
<pattern>com.zaxxer</pattern>
<shadedPattern>shaded.org.zaxxer</shadedPattern>
</relocation>
</relocations>
~~~
### 6. Using Uber Jars as Plugins
1. **Download them with Maven**
~~~
mvn dependency:copy -DrepoUrl=OSM_PACKAGE_REGISTRY_URL -Dartifact="org.opengroup.osdu:os-osm-postgres:OSM_VERSION:jar:CLASSIFIER" -Dtransitive=false -DoutputDirectory="./tmp"
~~~
2. **Configure for running service locally**.
- Spring boot loader dependency:
~~~
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-loader</artifactId>
<version>2.7.15</version>
</dependency>
~~~
- Configured main class should be from spring-boot-loader:
~~~
<build>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
<version>2.7.4</version>
<executions>
<execution>
<goals>
<goal>repackage</goal>
</goals>
<configuration>
<classifier>spring-boot</classifier>
<mainClass>
org.springframework.boot.loader.PropertiesLauncher
</mainClass>
~~~
- Compile the service, and run with the following args:
~~~
CMD java -Dloader.path=tmp/ \ <-- Path to downloaded jars.
-Dloader.main=org.opengroup.osdu.legal.LegalApplication \ <-- Service main class.
-jar /app/legal-${PROVIDER_NAME}.jar
~~~
### 7. Caveats and Best Practices
1. **Potential Challenges**:
- Maven won't help resolve conflicts between a service and a Driver compiled as an Uber jar or plugin jar.
- To properly debug a service using an IDE and work locally, additional configurations for the IDE are necessary.
2. **Best Practices**:
To deliver a Driver that can be easily plugged in without causing conflicts, those who implement it should:
- Handle bundled dependencies responsibly. Aim to minimize them.
- Relocate dependencies, and consider renaming them.
### Appendices
1. Bundling the Driver with an HTTP Server for Use in Non-Java Services
While not necessarily a part of the Plugin approach, it's worth considering the possibility of bundling Drivers with an HTTP server, which would enable communication with them through sets of domain-agnostic APIs. Drivers in the form of a plugin would enforce encapsulation, and self-assembly, and potentially facilitate delivery as a standalone server in the future.
\ No newline at end of file