NationalSecurityAgency/datawave
1
0 Fork 0
mirror of https://github.com/NationalSecurityAgency/datawave.git synced 2026-11-07 05:56:25 -10:00
Code Issues Packages Projects Releases Wiki Activity

Task/java17 build baseline (#3907)

Browse Source
This commit is contained in:
alerman authored and GitHub committed 2026-09-16 14:24:41 +00:00
1 parent 5fe65eef30
commit 4f0b97ec72
12 files changed
+191 -36
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

BUILDME.md
+48
View File
@@ -23,12 +23,25 @@ The id of the server matters, and should match what is used in the datawave pare
## Building Datawave
The root build remains Java 11 by default. Builds run on Java 17 are supported through the
`java17` profile and remain backward compatible by compiling with `--release 11`.
When using `-Pjava17`, make sure Maven itself is running on a full JDK 17, not a JRE-only runtime.
`mvn -version` should report a Java 17 runtime that includes `javac`. If Maven is launched with a
trimmed runtime, compilation can fail with `error: release version 11 not supported`.
To perform a full (non-release) 'dev' build without unit tests:
```bash
mvn -Ddeploy -Dtar -DskipTests -DskipITs clean install
```
To run the same build on Java 17:
```bash
mvn -Pjava17 -Ddeploy -Dtar -DskipTests -DskipITs clean install
```
This command will produce the following deployment archives:
1. Web Service: `./web-services/deploy/application/target/datawave-ws-deploy-application-${project.version}-dev.tar.gz`
@@ -42,6 +55,12 @@ In order to build a release, you must also define the dist variable by adding `-
mvn -Pexamples -Ddeploy -Dtar -Ddist -DskipTests -DskipITs clean install
```
On Java 17:
```bash
mvn -Pjava17 -Pexamples -Ddeploy -Dtar -Ddist -DskipTests -DskipITs clean install
```
### Building a Docker web image
In order to build a Docker container for the web services, you can run with the following maven profiles: `-Pdeploy-ws,docker`
@@ -50,6 +69,12 @@ In order to build a Docker container for the web services, you can run with the
mvn clean package -Passemble,deploy-ws -Pdocker -DskipTests -DskipITs
```
On Java 17:
```bash
mvn -Pjava17 clean package -Passemble,deploy-ws -Pdocker -DskipTests -DskipITs
```
Note that this will build javadocs and source jars.
### Building an RPM
@@ -60,6 +85,12 @@ To build the RPM specify both the assemble and rpm profiles should be specified,
mvn -Passemble,rpm -Ddeploy -Dtar -Ddist -DskipTests -DskipITs clean install
```
On Java 17:
```bash
mvn -Pjava17 -Passemble,rpm -Ddeploy -Dtar -Ddist -DskipTests -DskipITs clean install
```
# Building Microservices
Datawave web services utilize several microservices at runtime (currently authorization and auditing, although that
@@ -67,6 +98,23 @@ list will expand soon). Datawave depends on api modules for some of these servic
the parent pom (see `version.datawave.*` properties) to released versions. If you wish to build the microservices
for some reason, you can simply add `-Dservices` to your maven build command. If you wish to build the starters you can add `-Dstarters` and for the utility modules add `-Dutils`.
Examples:
Java 11:
```bash
mvn -Dservices -Dstarters -Dutils -DskipTests -DskipITs clean install
```
Java 17:
```bash
mvn -Pjava17 -Dservices -Dstarters -Dutils -DskipTests -DskipITs clean install
```
If you are building from within the `microservices/` tree on Java 17, invoke Maven from the
`microservices/` reactor so the `java17` profile is available to child modules.
### Releasing Microservices
Each subdirectory under the `services` folder is treated as a separate project. Therefore if you wish to build a
README.md
+29 -3
View File
@@ -69,16 +69,42 @@ git subrepo pull <dir>
```
### Building
It is recommended to build the project using multiple threads. This will not build the starters, utilities, and services.
The root build remains Java 11 by default. Builds run on Java 17 are supported through the `java17`
profile and remain backward compatible by compiling with `--release 11`.
When using `-Pjava17`, make sure Maven itself is running on a full JDK 17, not a JRE-only runtime.
`mvn -version` should report a Java 17 runtime that includes `javac`. If Maven is launched with a
trimmed runtime, compilation can fail with `error: release version 11 not supported`.
It is recommended to build the project using multiple threads. This will not build the starters,
utilities, and services.
Build with Java 11:
```
mvn -Ddocker-release -Ddist clean install -T 1C
```
If you want to build the starters, util modules, and services as well then try this
Build with Java 17:
```
mvn -Pjava17 -Ddocker-release -Ddist clean install -T 1C
```
If you want to build the starters, util modules, and services as well then try this:
Java 11:
```
mvn -Ddocker-release -Dmicroservice-docker -Ddist -Dutils -Dservices -Dstarters clean install -T 1C
```
If you want to build the service apis but not the services themselves then add -DonlyServiceApis
Java 17:
```
mvn -Pjava17 -Ddocker-release -Dmicroservice-docker -Ddist -Dutils -Dservices -Dstarters clean install -T 1C
```
If you want to build the service APIs but not the services themselves then add `-DonlyServiceApis`.
When building microservices on Java 17, invoke Maven from the `microservices/` reactor (or another
parent that defines the `java17` profile), not from an individual leaf module.
NOTE: The util modules, starters, and services are actually tagged and deployed separately.
Hence the snapshot versions within those sub repos are not connected together.
core/tables/pom.xml
-2
View File
@@ -10,8 +10,6 @@
<artifactId>datawave-core-tables</artifactId>
<properties>
<maven.compiler.source>11</maven.compiler.source>
<maven.compiler.target>11</maven.compiler.target>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>
core/utils/metadata-utils/pom.xml
+1 -2
View File
@@ -116,8 +116,7 @@
<encoding>UTF-8</encoding>
<showDeprecation>true</showDeprecation>
<showWarnings>true</showWarnings>
<source>${maven.compiler.source}</source>
<target>${maven.compiler.target}</target>
<release>${maven.compiler.release}</release>
<compilerArgs>
<arg>-parameters</arg>
<arg>-Xlint:all</arg>
core/utils/type-utils/src/main/java/datawave/webservice/query/data/ObjectSizeOf.java
+53 -6
View File
@@ -86,6 +86,7 @@ public interface ObjectSizeOf {
public static final short REFERENCE = 4;
// The size of the basic Number constructs (and Boolean and Character) is 16: roundUp(8 + primitiveSize)
public static final short NUMBER_SIZE = 16;
private static final UnsafeAccess UNSAFE_ACCESS = initUnsafeAccess();
// Class name cache to avoid flood of NoSuchMethodException on reflective "sizeInBytes" invocation
private static Set<String> noSuchMethodCache = ConcurrentHashMap.newKeySet();
@@ -164,18 +165,15 @@ public interface ObjectSizeOf {
size += getPrimitiveObjectSize(field.getType());
} else {
size += REFERENCE;
boolean accessible = field.isAccessible();
field.setAccessible(true);
try {
Object fieldObject = field.get(o);
Object fieldObject = getFieldValue(field, o);
if (fieldObject != null) {
stack.push(new ObjectInstance(fieldObject));
}
} catch (Exception e) {
} catch (Throwable t) {
// cannot get to field, so ignore it in this size calculation
e.printStackTrace();
log.debug("Unable to access field {} on {}", field.getName(), o.getClass().getName(), t);
}
field.setAccessible(accessible);
}
}
c = c.getSuperclass();
@@ -194,6 +192,55 @@ public interface ObjectSizeOf {
return totalSize;
}
private static UnsafeAccess initUnsafeAccess() {
try {
Class<?> unsafeClass = Class.forName("sun.misc.Unsafe");
Field unsafeField = unsafeClass.getDeclaredField("theUnsafe");
unsafeField.setAccessible(true);
Object unsafe = unsafeField.get(null);
Method objectFieldOffset = unsafeClass.getMethod("objectFieldOffset", Field.class);
Method getObject = unsafeClass.getMethod("getObject", Object.class, long.class);
return new UnsafeAccess(unsafe, objectFieldOffset, getObject);
} catch (Throwable t) {
log.debug("Unable to initialize Unsafe for object sizing fallback", t);
return null;
}
}
private static Object getFieldValue(Field field, Object target) throws IllegalAccessException {
if (field.trySetAccessible()) {
return field.get(target);
}
if (UNSAFE_ACCESS != null) {
try {
return UNSAFE_ACCESS.getObject(target, field);
} catch (ReflectiveOperationException e) {
IllegalAccessException wrapped = new IllegalAccessException(
"Unable to access field " + field.getName() + " on " + target.getClass().getName());
wrapped.initCause(e);
throw wrapped;
}
}
throw new IllegalAccessException("Unable to access field " + field.getName() + " on " + target.getClass().getName());
}
private static class UnsafeAccess {
private final Object unsafe;
private final Method objectFieldOffset;
private final Method getObject;
private UnsafeAccess(Object unsafe, Method objectFieldOffset, Method getObject) {
this.unsafe = unsafe;
this.objectFieldOffset = objectFieldOffset;
this.getObject = getObject;
}
private Object getObject(Object target, Field field) throws ReflectiveOperationException {
long offset = (Long) objectFieldOffset.invoke(unsafe, field);
return getObject.invoke(unsafe, target, offset);
}
}
public static long roundUp(long size) {
long extra = size % 8;
if (extra > 0) {
microservices/microservice-parent/pom.xml
+12 -6
View File
@@ -31,10 +31,10 @@
</distributionManagement>
<properties>
<container.registry>ghcr.io</container.registry>
<datawave.build.jdk.version.range>[11,12)</datawave.build.jdk.version.range>
<image.prefix>datawave/</image.prefix>
<java.version>11</java.version>
<maven.compiler.source>${java.version}</maven.compiler.source>
<maven.compiler.target>${java.version}</maven.compiler.target>
<maven.compiler.release>11</maven.compiler.release>
<maven.release.arguments>-Ddist</maven.release.arguments>
<maven.release.profiles>dist</maven.release.profiles>
<project.artifactId>${project.artifactId}</project.artifactId>
@@ -378,9 +378,9 @@
<configFile>eclipse/Eclipse-Datawave-Codestyle.xml</configFile>
<lineEnding>LF</lineEnding>
<removeTrailingWhitespace>true</removeTrailingWhitespace>
<compilerSource>${maven.compiler.source}</compilerSource>
<compilerCompliance>${maven.compiler.source}</compilerCompliance>
<compilerTargetPlatform>${maven.compiler.target}</compilerTargetPlatform>
<compilerSource>${maven.compiler.release}</compilerSource>
<compilerCompliance>${maven.compiler.release}</compilerCompliance>
<compilerTargetPlatform>${maven.compiler.release}</compilerTargetPlatform>
</configuration>
<dependencies>
<dependency>
@@ -598,7 +598,7 @@
<onlyWhenRelease>true</onlyWhenRelease>
</requireReleaseDeps>
<requireJavaVersion>
<version>${java.version}</version>
<version>${datawave.build.jdk.version.range}</version>
</requireJavaVersion>
</rules>
<fail>true</fail>
@@ -648,6 +648,12 @@
</plugins>
</build>
<profiles>
<profile>
<id>java17</id>
<properties>
<datawave.build.jdk.version.range>[17,18)</datawave.build.jdk.version.range>
</properties>
</profile>
<profile>
<id>autoformat</id>
<activation>
microservices/microservice-service-parent/pom.xml
+1 -1
View File
@@ -201,7 +201,7 @@
<onlyWhenRelease>true</onlyWhenRelease>
</requireReleaseDeps>
<requireJavaVersion>
<version>${java.version}</version>
<version>${datawave.build.jdk.version.range}</version>
</requireJavaVersion>
</rules>
<fail>true</fail>
pom.xml
+19 -7
View File
@@ -39,14 +39,14 @@
<!-- build.env and env should get overridden if any other read-properties profile is specified (e.g. comnpose)compose -->
<build.env>dev</build.env>
<container.registry>ghcr.io</container.registry>
<datawave.build.jdk.version.range>[11,12)</datawave.build.jdk.version.range>
<env>dev</env>
<!-- Integration-test fork settings. Default to the surefire values so a module only has to set
the surefire property to move both plugins; override these to split unit and integration tests. -->
<failsafe.maxMemory>${surefire.maxMemory}</failsafe.maxMemory>
<failsafe.reuseForks>${surefire.reuseForks}</failsafe.reuseForks>
<image.prefix>datawave/</image.prefix>
<maven.compiler.source>11</maven.compiler.source>
<maven.compiler.target>11</maven.compiler.target>
<maven.compiler.release>11</maven.compiler.release>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<registry.namespace>nationalsecurityagency/</registry.namespace>
<skipITs>${skipTests}</skipITs>
@@ -1754,9 +1754,9 @@
<configFile>eclipse/Eclipse-Datawave-Codestyle.xml</configFile>
<lineEnding>LF</lineEnding>
<removeTrailingWhitespace>true</removeTrailingWhitespace>
<compilerSource>${maven.compiler.source}</compilerSource>
<compilerCompliance>${maven.compiler.source}</compilerCompliance>
<compilerTargetPlatform>${maven.compiler.target}</compilerTargetPlatform>
<compilerSource>${maven.compiler.release}</compilerSource>
<compilerCompliance>${maven.compiler.release}</compilerCompliance>
<compilerTargetPlatform>${maven.compiler.release}</compilerTargetPlatform>
</configuration>
<dependencies>
<dependency>
@@ -1833,8 +1833,7 @@
<encoding>UTF-8</encoding>
<showDeprecation>true</showDeprecation>
<showWarnings>true</showWarnings>
<source>${maven.compiler.source}</source>
<target>${maven.compiler.target}</target>
<release>${maven.compiler.release}</release>
<compilerArgs>
<arg>-Xlint:all</arg>
<arg>-Xlint:-processing</arg>
@@ -1875,6 +1874,9 @@
<requireMavenVersion>
<version>3.6.3</version>
</requireMavenVersion>
<requireJavaVersion>
<version>${datawave.build.jdk.version.range}</version>
</requireJavaVersion>
</rules>
<fail>true</fail>
</configuration>
@@ -2105,9 +2107,19 @@
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-enforcer-plugin</artifactId>
</plugin>
</plugins>
</build>
<profiles>
<profile>
<id>java17</id>
<properties>
<datawave.build.jdk.version.range>[17,18)</datawave.build.jdk.version.range>
</properties>
</profile>
<profile>
<id>coverage</id>
<modules>
warehouse/ingest-core/src/test/java/datawave/ingest/benchmark/IngestHotPathBenchmark.java
+8 -4
View File
@@ -366,14 +366,14 @@ public class IngestHotPathBenchmark {
long flattenPerPass = handler.flattenCalls.get();
long[] samples = new long[measuredIterations];
long allocStart = THREAD_BEAN.getCurrentThreadAllocatedBytes();
long allocStart = getCurrentThreadAllocatedBytes();
for (int i = 0; i < measuredIterations; i++) {
int p = i % poolSize;
long t0 = System.nanoTime();
handler.processBulk(null, pool.get(p), fieldSets.get(p), null);
samples[i] = System.nanoTime() - t0;
}
long allocEnd = THREAD_BEAN.getCurrentThreadAllocatedBytes();
long allocEnd = getCurrentThreadAllocatedBytes();
long sum = 0;
for (long sample : samples) {
@@ -576,13 +576,13 @@ public class IngestHotPathBenchmark {
long normalizes = NORMALIZE_CALLS.get();
long[] samples = new long[MEASURED_ITERATIONS];
long allocStart = THREAD_BEAN.getCurrentThreadAllocatedBytes();
long allocStart = getCurrentThreadAllocatedBytes();
for (int i = 0; i < MEASURED_ITERATIONS; i++) {
long t0 = System.nanoTime();
helper.getEventFields(pool.get(i % pool.size()));
samples[i] = System.nanoTime() - t0;
}
long allocEnd = THREAD_BEAN.getCurrentThreadAllocatedBytes();
long allocEnd = getCurrentThreadAllocatedBytes();
Arrays.sort(samples);
// the counter only fires for the counting types; the all-types arm uses the real ones
@@ -678,6 +678,10 @@ public class IngestHotPathBenchmark {
}
}
private static long getCurrentThreadAllocatedBytes() {
return THREAD_BEAN.getThreadAllocatedBytes(Thread.currentThread().getId());
}
private void printTable() {
System.out.println();
System.out.println("=== ingest hot path (" + EVENT_COUNT + " events; " + MEASURED_ITERATIONS + " iterations) ===");
warehouse/ingest-ssdeep/pom.xml
-2
View File
@@ -8,8 +8,6 @@
</parent>
<artifactId>datawave-ingest-ssdeep</artifactId>
<properties>
<maven.compiler.source>11</maven.compiler.source>
<maven.compiler.target>11</maven.compiler.target>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>
<dependencies>
warehouse/ssdeep-common/pom.xml
-2
View File
@@ -8,8 +8,6 @@
</parent>
<artifactId>datawave-ssdeep-common</artifactId>
<properties>
<maven.compiler.source>11</maven.compiler.source>
<maven.compiler.target>11</maven.compiler.target>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>
<dependencies>
web-services/security-parent/security/pom.xml
+20 -1
View File
@@ -232,5 +232,24 @@
</plugin>
</plugins>
</build>
<profiles>
<profile>
<id>java17</id>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<excludes>
<exclude>**/DatawaveCertRolesLoginModuleTest.java</exclude>
<exclude>**/DatawavePrincipalLoginModuleTest.java</exclude>
<exclude>**/DatawaveUsersRolesLoginModuleTest.java</exclude>
</excludes>
</configuration>
</plugin>
</plugins>
</build>
</profile>
</profiles>
</project>