Commit Graph
14 Commits
Author SHA1 Message Date
alerman 4f0b97ec72 Task/java17 build baseline (#3907) 2026-09-16 14:24:41 +00:00
alerman 8d46ad32b3 Change readme badge location (#3605) 2026-06-30 18:30:16 +00:00
Joseph AlphonsoandJoe Alphonso cd63211874 Quickstart improvements (#3063)
assertCreateDir bugfix in Quickstart  
align hadoop, zookeeper, accumulo, wildfly versions across the repo

---------

Co-authored-by: Joe Alphonso <[email protected]>
2025-07-28 18:43:21 +00:00
Joseph AlphonsoandJoe Alphonso 0bf2040ed3 Github CI changes to support pom-updates (#2898)
* Github CI changes to support pom-updates

* fix microservice image push

* remove no-op, reorder, add comment

* MR feedback and split microservice container builds out

* use verbose maven args for readability

* split deploy workflow into two jobs, clean up commented code, add job deps

* tweaks

* remove .github workflows from former submodules

* try 2nd task in same job instead

* rename job

* remove dupcliate dep declaration

* disable build-cache since it is not being saved

* fix rebase issues

* speed up test retries

* add skipITs since skipTests is deprecated for the failsafe plugin

* add skip.unit.tests property

* Update tests.yml

* Update tests.yml

* allow integration tests to run separately from unit tests

---------

Co-authored-by: Joe Alphonso <[email protected]>
2025-05-16 15:19:18 +00:00
Ivan Bella 246642ff6f Merge submodules (#2654)
* git subrepo clone [email protected]:NationalSecurityAgency/datawave-type-utils.git core/utils/type-utils

subrepo:
  subdir:   "core/utils/type-utils"
  merged:   "867b89be18"
upstream:
  origin:   "[email protected]:NationalSecurityAgency/datawave-type-utils.git"
  branch:   "main"
  commit:   "867b89be18"
git-subrepo:
  version:  "0.4.9"
  origin:   "https://github.com/ingydotnet/git-subrepo"
  commit:   "cce3d93"

* git subrepo clone [email protected]:NationalSecurityAgency/datawave-utils.git contrib/datawave-utils

subrepo:
  subdir:   "contrib/datawave-utils"
  merged:   "4348fc36a3"
upstream:
  origin:   "[email protected]:NationalSecurityAgency/datawave-utils.git"
  branch:   "main"
  commit:   "4348fc36a3"
git-subrepo:
  version:  "0.4.9"
  origin:   "https://github.com/ingydotnet/git-subrepo"
  commit:   "cce3d93"

* git subrepo clone [email protected]:NationalSecurityAgency/datawave-base-rest-responses.git core/base-rest-responses

subrepo:
  subdir:   "core/base-rest-responses"
  merged:   "c5b77e8a35"
upstream:
  origin:   "[email protected]:NationalSecurityAgency/datawave-base-rest-responses.git"
  branch:   "main"
  commit:   "c5b77e8a35"
git-subrepo:
  version:  "0.4.9"
  origin:   "https://github.com/ingydotnet/git-subrepo"
  commit:   "cce3d93"

* git subrepo clone [email protected]:NationalSecurityAgency/datawave-in-memory-accumulo.git core/in-memory-accumulo

subrepo:
  subdir:   "core/in-memory-accumulo"
  merged:   "8a9d2f46d2"
upstream:
  origin:   "[email protected]:NationalSecurityAgency/datawave-in-memory-accumulo.git"
  branch:   "main"
  commit:   "8a9d2f46d2"
git-subrepo:
  version:  "0.4.9"
  origin:   "https://github.com/ingydotnet/git-subrepo"
  commit:   "cce3d93"

* git subrepo clone [email protected]:NationalSecurityAgency/datawave-metrics-reporter.git core/metrics-reporter

subrepo:
  subdir:   "core/metrics-reporter"
  merged:   "992378d629"
upstream:
  origin:   "[email protected]:NationalSecurityAgency/datawave-metrics-reporter.git"
  branch:   "main"
  commit:   "992378d629"
git-subrepo:
  version:  "0.4.9"
  origin:   "https://github.com/ingydotnet/git-subrepo"
  commit:   "cce3d93"

* git subrepo clone [email protected]:NationalSecurityAgency/datawave-accumulo-utils.git core/utils/accumulo-utils

subrepo:
  subdir:   "core/utils/accumulo-utils"
  merged:   "638b3eda97"
upstream:
  origin:   "[email protected]:NationalSecurityAgency/datawave-accumulo-utils.git"
  branch:   "main"
  commit:   "638b3eda97"
git-subrepo:
  version:  "0.4.9"
  origin:   "https://github.com/ingydotnet/git-subrepo"
  commit:   "cce3d93"

* git subrepo clone (merge) [email protected]:NationalSecurityAgency/datawave-common-utils.git core/utils/common-utils

subrepo:
  subdir:   "core/utils/common-utils"
  merged:   "c96ed21342"
upstream:
  origin:   "[email protected]:NationalSecurityAgency/datawave-common-utils.git"
  branch:   "main"
  commit:   "c96ed21342"
git-subrepo:
  version:  "0.4.9"
  origin:   "https://github.com/ingydotnet/git-subrepo"
  commit:   "cce3d93"

* git subrepo clone --branch=release/version4.0.9 [email protected]:NationalSecurityAgency/datawave-metadata-utils.git core/utils/metadata-utils

subrepo:
  subdir:   "core/utils/metadata-utils"
  merged:   "0d0872a05b"
upstream:
  origin:   "[email protected]:NationalSecurityAgency/datawave-metadata-utils.git"
  branch:   "release/version4.0.9"
  commit:   "0d0872a05b"
git-subrepo:
  version:  "0.4.9"
  origin:   "https://github.com/ingydotnet/git-subrepo"
  commit:   "cce3d93"

* git subrepo clone [email protected]:NationalSecurityAgency/datawave-parent.git microservices/microservice-parent

subrepo:
  subdir:   "microservices/microservice-parent"
  merged:   "6207c9d657"
upstream:
  origin:   "[email protected]:NationalSecurityAgency/datawave-parent.git"
  branch:   "main"
  commit:   "6207c9d657"
git-subrepo:
  version:  "0.4.9"
  origin:   "https://github.com/ingydotnet/git-subrepo"
  commit:   "cce3d93"

* git subrepo clone [email protected]:NationalSecurityAgency/datawave-service-parent.git microservices/microservice-service-parent

subrepo:
  subdir:   "microservices/microservice-service-parent"
  merged:   "8064d20ccf"
upstream:
  origin:   "[email protected]:NationalSecurityAgency/datawave-service-parent.git"
  branch:   "main"
  commit:   "8064d20ccf"
git-subrepo:
  version:  "0.4.9"
  origin:   "https://github.com/ingydotnet/git-subrepo"
  commit:   "cce3d93"

* git subrepo clone [email protected]:NationalSecurityAgency/datawave-accumulo-service.git microservices/services/accumulo

subrepo:
  subdir:   "microservices/services/accumulo"
  merged:   "6d3839d289"
upstream:
  origin:   "[email protected]:NationalSecurityAgency/datawave-accumulo-service.git"
  branch:   "main"
  commit:   "6d3839d289"
git-subrepo:
  version:  "0.4.9"
  origin:   "https://github.com/ingydotnet/git-subrepo"
  commit:   "cce3d93"

* git subrepo clone (merge) [email protected]:NationalSecurityAgency/datawave-audit-service.git microservices/services/audit

subrepo:
  subdir:   "microservices/services/audit"
  merged:   "2714c97b19"
upstream:
  origin:   "[email protected]:NationalSecurityAgency/datawave-audit-service.git"
  branch:   "main"
  commit:   "2714c97b19"
git-subrepo:
  version:  "0.4.9"
  origin:   "https://github.com/ingydotnet/git-subrepo"
  commit:   "cce3d93"

* git subrepo clone (merge) [email protected]:NationalSecurityAgency/datawave-authorization-service.git microservices/services/authorization

subrepo:
  subdir:   "microservices/services/authorization"
  merged:   "292be85b63"
upstream:
  origin:   "[email protected]:NationalSecurityAgency/datawave-authorization-service.git"
  branch:   "main"
  commit:   "292be85b63"
git-subrepo:
  version:  "0.4.9"
  origin:   "https://github.com/ingydotnet/git-subrepo"
  commit:   "cce3d93"

* git subrepo clone (merge) [email protected]:NationalSecurityAgency/datawave-config-service.git microservices/services/config

subrepo:
  subdir:   "microservices/services/config"
  merged:   "fc187c0e4b"
upstream:
  origin:   "[email protected]:NationalSecurityAgency/datawave-config-service.git"
  branch:   "main"
  commit:   "fc187c0e4b"
git-subrepo:
  version:  "0.4.9"
  origin:   "https://github.com/ingydotnet/git-subrepo"
  commit:   "cce3d93"

* git subrepo clone (merge) [email protected]:NationalSecurityAgency/datawave-dictionary-service.git microservices/services/dictionary

subrepo:
  subdir:   "microservices/services/dictionary"
  merged:   "709957100d"
upstream:
  origin:   "[email protected]:NationalSecurityAgency/datawave-dictionary-service.git"
  branch:   "main"
  commit:   "709957100d"
git-subrepo:
  version:  "0.4.9"
  origin:   "https://github.com/ingydotnet/git-subrepo"
  commit:   "cce3d93"

* git subrepo clone [email protected]:NationalSecurityAgency/datawave-file-provider-service.git microservices/services/file-provider

subrepo:
  subdir:   "microservices/services/file-provider"
  merged:   "27080f3b94"
upstream:
  origin:   "[email protected]:NationalSecurityAgency/datawave-file-provider-service.git"
  branch:   "main"
  commit:   "27080f3b94"
git-subrepo:
  version:  "0.4.9"
  origin:   "https://github.com/ingydotnet/git-subrepo"
  commit:   "cce3d93"

* git subrepo clone (merge) [email protected]:NationalSecurityAgency/datawave-hazelcast-service.git microservices/services/hazelcast

subrepo:
  subdir:   "microservices/services/hazelcast"
  merged:   "8abc2ef7e9"
upstream:
  origin:   "[email protected]:NationalSecurityAgency/datawave-hazelcast-service.git"
  branch:   "main"
  commit:   "8abc2ef7e9"
git-subrepo:
  version:  "0.4.9"
  origin:   "https://github.com/ingydotnet/git-subrepo"
  commit:   "cce3d93"

* git subrepo clone [email protected]:NationalSecurityAgency/datawave-map-service.git microservices/services/map

subrepo:
  subdir:   "microservices/services/map"
  merged:   "473ec43708"
upstream:
  origin:   "[email protected]:NationalSecurityAgency/datawave-map-service.git"
  branch:   "main"
  commit:   "473ec43708"
git-subrepo:
  version:  "0.4.9"
  origin:   "https://github.com/ingydotnet/git-subrepo"
  commit:   "cce3d93"

* git subrepo clone (merge) [email protected]:NationalSecurityAgency/datawave-mapreduce-query-service.git microservices/services/mapreduce-query

subrepo:
  subdir:   "microservices/services/mapreduce-query"
  merged:   "22af8f93c8"
upstream:
  origin:   "[email protected]:NationalSecurityAgency/datawave-mapreduce-query-service.git"
  branch:   "main"
  commit:   "22af8f93c8"
git-subrepo:
  version:  "0.4.9"
  origin:   "https://github.com/ingydotnet/git-subrepo"
  commit:   "cce3d93"

* git subrepo clone (merge) [email protected]:NationalSecurityAgency/datawave-modification-service.git microservices/services/modification

subrepo:
  subdir:   "microservices/services/modification"
  merged:   "768007f319"
upstream:
  origin:   "[email protected]:NationalSecurityAgency/datawave-modification-service.git"
  branch:   "main"
  commit:   "768007f319"
git-subrepo:
  version:  "0.4.9"
  origin:   "https://github.com/ingydotnet/git-subrepo"
  commit:   "cce3d93"

* git subrepo clone [email protected]:NationalSecurityAgency/datawave-query-service.git microservices/services/query

subrepo:
  subdir:   "microservices/services/query"
  merged:   "14c65febd3"
upstream:
  origin:   "[email protected]:NationalSecurityAgency/datawave-query-service.git"
  branch:   "main"
  commit:   "14c65febd3"
git-subrepo:
  version:  "0.4.9"
  origin:   "https://github.com/ingydotnet/git-subrepo"
  commit:   "cce3d93"

* git subrepo clone (merge) [email protected]:NationalSecurityAgency/datawave-query-executor-service.git microservices/services/query-executor

subrepo:
  subdir:   "microservices/services/query-executor"
  merged:   "4517c4c29e"
upstream:
  origin:   "[email protected]:NationalSecurityAgency/datawave-query-executor-service.git"
  branch:   "main"
  commit:   "4517c4c29e"
git-subrepo:
  version:  "0.4.9"
  origin:   "https://github.com/ingydotnet/git-subrepo"
  commit:   "cce3d93"

* git subrepo clone [email protected]:NationalSecurityAgency/datawave-query-metric-service.git microservices/services/query-metric

subrepo:
  subdir:   "microservices/services/query-metric"
  merged:   "9123d49528"
upstream:
  origin:   "[email protected]:NationalSecurityAgency/datawave-query-metric-service.git"
  branch:   "main"
  commit:   "9123d49528"
git-subrepo:
  version:  "0.4.9"
  origin:   "https://github.com/ingydotnet/git-subrepo"
  commit:   "cce3d93"

* git subrepo clone (merge) [email protected]:NationalSecurityAgency/datawave-spring-boot-starter-audit.git microservices/starters/audit

subrepo:
  subdir:   "microservices/starters/audit"
  merged:   "ef18c9e652"
upstream:
  origin:   "[email protected]:NationalSecurityAgency/datawave-spring-boot-starter-audit.git"
  branch:   "main"
  commit:   "ef18c9e652"
git-subrepo:
  version:  "0.4.9"
  origin:   "https://github.com/ingydotnet/git-subrepo"
  commit:   "cce3d93"

* git subrepo clone [email protected]:NationalSecurityAgency/datawave-spring-boot-starter-cache.git microservices/starters/cache

subrepo:
  subdir:   "microservices/starters/cache"
  merged:   "3019600791"
upstream:
  origin:   "[email protected]:NationalSecurityAgency/datawave-spring-boot-starter-cache.git"
  branch:   "main"
  commit:   "3019600791"
git-subrepo:
  version:  "0.4.9"
  origin:   "https://github.com/ingydotnet/git-subrepo"
  commit:   "cce3d93"

* git subrepo clone (merge) [email protected]:NationalSecurityAgency/datawave-spring-boot-starter-cached-results.git microservices/starters/cached-results

subrepo:
  subdir:   "microservices/starters/cached-results"
  merged:   "b22a3d6a17"
upstream:
  origin:   "[email protected]:NationalSecurityAgency/datawave-spring-boot-starter-cached-results.git"
  branch:   "main"
  commit:   "b22a3d6a17"
git-subrepo:
  version:  "0.4.9"
  origin:   "https://github.com/ingydotnet/git-subrepo"
  commit:   "cce3d93"

* git subrepo clone (merge) [email protected]:NationalSecurityAgency/datawave-spring-boot-starter.git microservices/starters/datawave

subrepo:
  subdir:   "microservices/starters/datawave"
  merged:   "081e0a028a"
upstream:
  origin:   "[email protected]:NationalSecurityAgency/datawave-spring-boot-starter.git"
  branch:   "main"
  commit:   "081e0a028a"
git-subrepo:
  version:  "0.4.9"
  origin:   "https://github.com/ingydotnet/git-subrepo"
  commit:   "cce3d93"

* git subrepo clone (merge) [email protected]:NationalSecurityAgency/datawave-spring-boot-starter-metadata.git microservices/starters/metadata

subrepo:
  subdir:   "microservices/starters/metadata"
  merged:   "28337e6c23"
upstream:
  origin:   "[email protected]:NationalSecurityAgency/datawave-spring-boot-starter-metadata.git"
  branch:   "main"
  commit:   "28337e6c23"
git-subrepo:
  version:  "0.4.9"
  origin:   "https://github.com/ingydotnet/git-subrepo"
  commit:   "cce3d93"

* git subrepo clone (merge) [email protected]:NationalSecurityAgency/datawave-spring-boot-starter-query.git microservices/starters/query

subrepo:
  subdir:   "microservices/starters/query"
  merged:   "ae730eab06"
upstream:
  origin:   "[email protected]:NationalSecurityAgency/datawave-spring-boot-starter-query.git"
  branch:   "main"
  commit:   "ae730eab06"
git-subrepo:
  version:  "0.4.9"
  origin:   "https://github.com/ingydotnet/git-subrepo"
  commit:   "cce3d93"

* git subrepo clone (merge) [email protected]:NationalSecurityAgency/datawave-spring-boot-starter-query-metric.git microservices/starters/query-metric

subrepo:
  subdir:   "microservices/starters/query-metric"
  merged:   "fad3ed7859"
upstream:
  origin:   "[email protected]:NationalSecurityAgency/datawave-spring-boot-starter-query-metric.git"
  branch:   "main"
  commit:   "fad3ed7859"
git-subrepo:
  version:  "0.4.9"
  origin:   "https://github.com/ingydotnet/git-subrepo"
  commit:   "cce3d93"

* Updated profiles to optionally add former submodules into build/test
Updated test workflow to add all former submodules into build/.test
Changed README.md instructions from submodules to subrepo
Updated build readme
* Avoid periodic failures on a bunch of tests (force serial execution etc).
2025-02-23 13:28:07 -05:00
9dadf7b2c6 Feature/query microservices (#2192)
* Modified a couple of things and fixed a few tests with the integration merge.

* pulling in latest audit changes

* pulled in latest audit updates

* Fixed metrics version in docker compose

* Updated accumulo utils and datawave starter versions.

* Updated the audit api and audit starter versions

* Updated authorization api version

* Updated accumulo api version.

* Updated config service with latest changes

* updated dictionary and accumulo utils version

* Updated hazelcast version

* bumped query metric api version

* Updated pki for query microservices docker deployment.

* Fixing the docker compose bootstrap to avoid duplicate extra hosts issue.

* adding latest datawave changes

* updated metadata utils

* updated docker compose to use the latest version of each service.

* Added a cached results spring boot starter and updated associated classes accordingly.

* fixing spotbugs error

* fixed install-ingest.sh for datawave.

* Updates to get bulk results map reduce query working again.

* formatting

* Updates per a race condition discovered in the kafka variant of the
query executor tests.

* Update README.md

* accumulo service merge from main

* fixed a test that broke after merging integration

* Added logic from Datawave Principal Login Module to the datawave spring boot starter.

* Updated datawave to 5.4.0-SNAPSHOT

* Updated ProxiedUserDetails to use dynamic type for newInstance method.

* updated audit service to stop intermittent test failures.

* Updated the query results publisher and setup the kafka manager to create topics manually.

* Added github actions for some submodules

* Updated github actions to avoid running out of disk space.

* Updated import sorting for microservices using impsort plugin.

* fixing merge conflicts

* updated metadata utils

* Moved Query, QueryImpl to query-api

* Removed usage of zookeeper stringutils

* added package-info.java for Query, QueryImpl

* updated modification service logic to use ProxiedUserDetails.

* Removed unnecessary jackson annotations to address enunciate issues

* Updated profile ids for submodules

* changes to get the github actions working again

* attempting to fix github actions

* typo

* udpated kafka config

* removed banned dependency

* bumped release version

* formatting

* fixing docker compose kafka setup

* updated deploy poms to work with the latest version of maven

* bumped version for some modules

* bumped microservice service parent version

* Updated with latest changes from main/integration

* fixed the composite query logic for the query microservices

* updated submodules

* Added v1 to the model path

* For this branch we need to checkout all submodules when building tests.

* Added alerman's run cache settings for docker

* Updated github actions to build query microservices correctly

* fixed build cache config typo

* Updated query service to use pool header, and allow for admin override via parameter

* Updated github ci to make sure it runs against the queryMicroservices branch

* Updated lookup uuid to accept multiple terms per field

* Updated confirm ack configuration for query metric service, and added configuration to make rest metrics updates possible

* Added support for long running queries

* Added configcheck command line utility

* updated next call logic for long running queries

* updated query metric service and starter submodules

* Added some plugins to the config check service

* Updated configcheck tests to trim whitespace

* fixing unit tests for config check

* Changed from max long running query timeouts to long running query
timeout

* Updated the query starter submodule

* updated submodules

* fixed composite query logic test

* Refactored query scripts to use common code

* Updated quickstart docker-maven-plugin to not cleanup old tagged releases

* updated common-utils submodule

* Updated LookupService to behave the same as the webservice

* Fixed TypedValue (de)serialization, and added a result post processor to handle things like counts

* reset docker-compose quickstart to just be accumulo, not web

* Changed how the result postprocessor runs to support result aggregation for long running queries

* Renamed aggregateResults to reduceResults

* Added junit-jupiter engine to datawave core pom

* Updated query logic factory to add logic map

* Updated EdgeQueryLogicFactory configuration

* Removed unused RemoteQueryLogic and cleaned up RemoteEventQueryLogic class definition:

* Implemented authorization and query federation for the query microservices

* Fixed copy constructor parameter assignment in QueryData

* Formatting job fix

* Fixing build/formatter issues for latest changes pulled in from integration

* Fixed SSDeepIngestQueryTest

* Updated usage of remote user operations for query microservices

* Fixed QueryPlan constructor

* Moved filtering and delegating query logic functionality into datawave-core-query so it can be used by the microservices

* formatting

* Updated submodules

* Updated package names for commons.lang3 classes due to type-utils fix

* Moved the AuthorizationsPredicate class to authorization-api

* Removed hadoop dependencies from pom

* Updated edge configuration to reset param variables before loading new values

* Excluded log4j from our maven plugins

* Updated in-memory-accumulo javadoc plugin to exclude log4j

* Updated microservice-parent maven-dependency-plugin to exclude log4j

* Removed infinispan from query starter

* Remove validation call from LoadModel

* fixed formatting

* Updated datawave ingest jar libs

* QueryMetricQueryLogic must copy runAsMetricsAdministrator setting in copy constructor

* Revert query metric query logic back to it's original state

* pr feedback

* PR feedback

* PR Feedback

* PR feedback

* PR Feedback

* PR feedback

* PR feedback

* PR Feedback

* PR Feedback

* PR feedback

* PR feedback

* PR feedback

* pr feedback

* PR feedback

* pr feedback

* pr feedback

* pr feedback

* pr feedback

* pr feedback

* pr feedback

* PR feedback

* PR feedback

* pr feedback

* pr feedback

* Updated in-memory-accumulo commit

* in-memory-accumulo

* formatting

* query starter

* pr feedback

* pr feedback

* pr feedback

* pr feedback

* pr feedback

* pr feedback

* pr feedback

* Remoted CachedQueryStatus mechanism

* pr feedback

* pr feedback

* pr feedback

* pr feedback

* pr updates

* pr feedback

* pr review: found duplicate properties between configurations and the
super GenericQueryConfiguration class.  Removed redundant copies in
ShardQueryConfiguration checkpoint copy constructor

* review comments

* review comments

* Reverted path changes in dictionary service

* pr review updates

* updated executor to use QueryStatusUpdateUtil

* updated query microservice submodule

* make github action tests match integration

---------

Co-authored-by: Ivan Bella <[email protected]>
Co-authored-by: GitHub Actions <[email protected]>
Co-authored-by: Bill Oley <[email protected]>
2024-05-20 16:33:34 +00:00
Whitney O'Meara a8039a5b60 Fixed the readme workflow status badge 2024-01-26 21:05:01 +00:00
Whitney O'Meara 1bb6d1eb8e Updated submodule instructions for readme 2023-10-13 13:28:32 +00:00
Whitney O'Meara 11d0b5c29b Added the utils, microservices, and starters as submodules to make it easier to build the various projects 2023-03-13 16:43:54 -04:00
Brian Loss a57f7db066 Add GitHub action workflow to run tests. 2019-10-22 10:25:32 -04:00
keith-ratcliffe c8a8662192 README update to reflect quickstart documentation move 2018-09-12 19:34:29 -04:00
keith-ratcliffe 31af946a3b re #104 - updating docs link based on new site redirects 2018-06-15 15:44:15 -04:00
Brian Loss 93f467112f Use relative link to datawave image. 2018-02-07 08:38:59 -05:00
keith-ratcliffe 2f028f3f3f Initial commit 2018-02-01 10:39:40 -05:00