The whole sharded schema sat in one tablet, so no number of tablet
servers, scan servers or compactors made any difference to a query.
Splits now come from a checked-in file, the deployment moves to three
shards per day, and verifySplits.sh guards that in CI.
Also fixes the num shards cache, which scanned a metadata table name no
deployment that renames its tables actually uses. The missing table was
swallowed, so a day expanded into no shards and the query silently
returned nothing.
---------
Co-authored-by: Claude Opus 5 <[email protected]>
…s from NationalSecurityAgency/datawave-stack-docker-images
---------
Co-authored-by: Moriarty <[email protected]>
Co-authored-by: Claude Opus 5 <[email protected]>
run a dedicated rabbitmq for the query results
Co-authored-by: Christopher Gross cogross <[email protected]>
Co-authored-by: Ivan Bella <[email protected]>
Use correct slf4j impl in wildfly and services, update versions of
type-utils, metadata-utils, common-utils, accumulo-utils,
base-rest-responses in microservice starters, apis, services to track
datawave version
The use of these dependencies under microservices (starters, services,
api) should track the datawave version instead of the version number
from when they were in separate repositories
Because the prior versions used a different groupId, we have the case
where multiple versions of the same artifact are being included in some
microservices.
For instance, this caused a serialization conflict once the
serializationId in BaseType (type-utils) changed in datawave version
7.30.0. When both versions are present on both sides of a serialization,
whether a serialization exception occurs is indeterminate.
gov.nsa.datawave.core:accumulo-utils
gov.nsa.datawave.core:type-utils
gov.nsa.datawave.core:metadata-utils
gov.nsa.datawave.core:common-utils
gov.nsa.datawave.core:base-rest-responses
After this PR is merged, the following should be done to make the
starters, apis, modules, and services depend on released version of each
other.
Release datawave-microservice-service-parent:5.0.8
In all modules with parent datawave-microservice-service-parent, update
to datawave-microservice-service-parent:5.0.8
Release authorization-api:4.0.2
In starter-datawave update to authorization-api:4.0.2
Release starter-datawave:4.0.6
Release query-metric-api:4.1.3
Release query-api:1.0.1
Release accumulo-api:4.0.1
Release modification-api:1.0.1
Release dictionary-api:4.0.6
In hazelcast-common update to starter-datawave:4.0.6
Release hazelcast-common:4.0.3
In hazelcast-client update to hazelcast-common:4.0.3
In starter-audit update to starter-datawave:4.0.6,
authorization-api:4.0.2
In starter-metadata update to starter-datawave:4.0.6
In starter-query-metric update to starter-datawave:4.0.6,
query-metric-api:4.1.3, authorization-api:4.0.2
Release hazelcast-client:4.0.3
Release starter-audit:4.0.4
Release starter-metadata:3.0.3
Release starter-query-metric:3.0.4
In starter-query update to starter-datawave:4.0.6, query-api:1.0.1,
hazelcast-client:4.0.3, starter-metadata:3.0.3,
starter-query-metric:3.0.4
Release starter-query:1.0.9
In starter-cached-query update to starter-datawave:4.0.6,
hazelcast-client:4.0.3, starter-audit:4.0.4, starter-query:1.0.9
Release starter-cached-query:1.0.11
In mapreduce-query-core-job update to starter-query:1.0.9
Release mapreduce-query-core-job:1.0.5
In query-executor-api update to starter-datawave:4.0.6,
starter-query:1.0.9
Release query-executor-api:1.0.8
In accumulo-service update to starter-datawave:4.0.6, accumulo-api:4.0.1
In audit-service update to starter-datawave:4.0.6,
hazelcast-client:4.0.3
In authorization-service update to starter-datawave:4.0.6,
hazelcast-client:4.0.3, authorization-api:4.0.2
In config-service update to starter-datawave:4.0.6
In dictionary-service update to starter-datawave:4.0.6,
dictionary-api:4.0.6, starter-metadata:3.0.3
In file-provider-service update to starter-datawave:4.0.6
In hazelcast-service update to hazelcast-common:4.0.3
In mapreduce-query-service update to mapreduce-query-core-job:1.0.5,
starter-query:1.0.9
In modification-service update to starter-metadata:3.0.3,
query-api:1.0.1, modification-api:1.0.1
In query-service update to starter-datawave:4.0.6, query-api:1.0.1,
starter-cached-results:1.0.11, starter-query:1.0.9
In query-executor-service update to starter-query:1.0.9,
query-executor-api:1.0.8
In query-metric-service update to starter-datawave:4.0.6,
starter-metadata:3.0.3, hazelcast-common:4.0.3
Release accumulo-service:4.0.3
Release audit-service:4.0.3
Release authorization-service:4.0.5
Release config-service:4.0.2
Release dictionary-service:4.0.10
Release file-provider-service:1.0.1
Release hazelcast-service:4.0.3
Release mapreduce-query-service:1.0.5
Release modification-service:1.0.3
Release query-service:1.0.12
Release query-executor-service:1.0.10
Release query-metric-service:4.1.9
When attempting to bring up the docker compose environment the
quickstart container sometimes is unable to bring up the web service.
Waiting for the dependent services to come up first seems to resolve
this.
Co-authored-by: Joe Alphonso <[email protected]>
added zookeeper and tserver exposed ports
added efficient accumulo and zookeeper healthcheck for quickstart
container in compose file
added quickstart as a dependent service to the metrics service and
adjusted its healthcheck configuration
updated CI compose-tests: simplified logic for faster more reliable
startup
fixed issue with quickstart image builds when using maven build cache
---------
Co-authored-by: Joe Alphonso <[email protected]>
Added system environment requirements for RHEL and Docker versions.
Also fixed the github workflow checks to bypass `**/*.md`
---------
Co-authored-by: Joe Alphonso <[email protected]>
if building images locally for docker compose use add
`-Djkube.container-image.tags.1=latest` arg
this is no longer the default as we don't want to publish the latest tag
to the docker registry
Co-authored-by: Joe Alphonso <[email protected]>
renamed `docker.image.prefix` to `docker.registry` to make purpose clear
resolves image push failures
---------
Co-authored-by: Joe Alphonso <[email protected]>
* cleanup quickstart and jkube docker configuration
set compose prefix
stop boostrapping maven twice
add default back for microservices since they do not load the properties files
add latest tag to compose tests
# Conflicts:
# contrib/datawave-quickstart/docker/docker-build.sh
# contrib/datawave-quickstart/docker/pom.xml
# Conflicts:
# microservices/microservice-parent/pom.xml
* remove cdn download from pom and set file urls to local paths
* fix file paths
* testing
# Conflicts:
# microservices/microservice-service-parent/pom.xml
* cleanup and add quickstart web tests back to CI
* update quickstart ci test
* testing fixes
* fix bug
* address compose warning
* bugfix
* dev profile clean up and docker-entrypoint.sh fix
* bootstrap only when install is called
* kill process when there is a fatal error
* update maven to a version we actually have
* install maven via os pkg mgr to start
* move quickstart jdk check to the install.sh scripts
* copy maven from latest maven-3 image at container build time
* remove maven download and bootstrap code
* fix returns and exits so we terminate quickstart build upon first fatal error
* remove quickstart ci job
* remove broken maven download from inside quickstart
* remove obsolete arg
* bump microservice-parent version in child poms
* bump microservice-service-parent version in child poms
* use mvn --settings arg
* rename vars and add to pom for warehouse docker image
* add web tests to CI
* reenable compose snapshot ci job
* add missed mvn --settings arg
* testing
* fix bug
* remove testing changes
* fix web tests
* fixes so users can use scripts outside of the container
---------
Co-authored-by: Joe Alphonso <[email protected]>
* Github CI changes to support pom-updates
* fix microservice image push
* remove no-op, reorder, add comment
* MR feedback and split microservice container builds out
* use verbose maven args for readability
* split deploy workflow into two jobs, clean up commented code, add job deps
* tweaks
* remove .github workflows from former submodules
* try 2nd task in same job instead
* rename job
* remove dupcliate dep declaration
* disable build-cache since it is not being saved
* fix rebase issues
* speed up test retries
* add skipITs since skipTests is deprecated for the failsafe plugin
* add skip.unit.tests property
* Update tests.yml
* Update tests.yml
* allow integration tests to run separately from unit tests
---------
Co-authored-by: Joe Alphonso <[email protected]>
* Add abilty to yield in Ivarators, AndIterator, OrIterator and return metrics before yield (#704)
* Improvement to debug and trace logging in QueryIerator.seek() (#704)
* Feedback from MR comments (#704)
* Add collectTimingDetails and yieldThresholdMs to spring-boot-starter-datawave-query
* Allow yieldThresholdMs < 50ms to work and check for values <= 0
* MR fixes for toStringImpl, toStringNoQueryId in Ivarator and subslasses; add some documentation
* Fix spotbugs issues
* Add yieldthresholdMs to application-query.yml for docker-compose configuration
* Add collectTimingDetails to application-query.yml
* adds system name to dictionary pages
* removes commented code
* change dictionary api version to build
* adds system name to dictionary docker config
* adds default name for system name in Model and ModelList
* changes pom versions to apply differences
* removes commented out code
* changes docker-compose to be back to accumulo
* removes pom changes
* reverses original pom changes
* changes dictionary api version in dictionary service pom
---------
Co-authored-by: foster33 <[email protected]>
* updates to build datawave quickstart with a given repository location.
* updates to build datawave quickstart with a given repository location.
* MR suggestions
* Fix github actions build
* Updating tests
* Added in args to be able to sed docker build environment variables:
* Fixes to get maven-based dependency downloads working for quickstart
* debug prints
* more testing
* more testing
* more testing
* potential fix
* potential fix
* potential fix
* potential fix
* another test
* another test
* another test
* fixed?
* added instructions for downloading tarballs from maven to the docker readme
* Delete docker/rabbitmq-query-config directory
* Update env.sh
---------
Co-authored-by: Whitney O'Meara <[email protected]>
Co-authored-by: Whitney O'Meara <[email protected]>
Drew Farris