The whole sharded schema sat in one tablet, so no number of tablet
servers, scan servers or compactors made any difference to a query.
Splits now come from a checked-in file, the deployment moves to three
shards per day, and verifySplits.sh guards that in CI.
Also fixes the num shards cache, which scanned a metadata table name no
deployment that renames its tables actually uses. The missing table was
swallowed, so a day expanded into no shards and the query silently
returned nothing.
---------
Co-authored-by: Claude Opus 5 <[email protected]>
…s from NationalSecurityAgency/datawave-stack-docker-images
---------
Co-authored-by: Moriarty <[email protected]>
Co-authored-by: Claude Opus 5 <[email protected]>
Added system environment requirements for RHEL and Docker versions.
Also fixed the github workflow checks to bypass `**/*.md`
---------
Co-authored-by: Joe Alphonso <[email protected]>
if building images locally for docker compose use add
`-Djkube.container-image.tags.1=latest` arg
this is no longer the default as we don't want to publish the latest tag
to the docker registry
Co-authored-by: Joe Alphonso <[email protected]>
* cleanup quickstart and jkube docker configuration
set compose prefix
stop boostrapping maven twice
add default back for microservices since they do not load the properties files
add latest tag to compose tests
# Conflicts:
# contrib/datawave-quickstart/docker/docker-build.sh
# contrib/datawave-quickstart/docker/pom.xml
# Conflicts:
# microservices/microservice-parent/pom.xml
* remove cdn download from pom and set file urls to local paths
* fix file paths
* testing
# Conflicts:
# microservices/microservice-service-parent/pom.xml
* cleanup and add quickstart web tests back to CI
* update quickstart ci test
* testing fixes
* fix bug
* address compose warning
* bugfix
* dev profile clean up and docker-entrypoint.sh fix
* bootstrap only when install is called
* kill process when there is a fatal error
* update maven to a version we actually have
* install maven via os pkg mgr to start
* move quickstart jdk check to the install.sh scripts
* copy maven from latest maven-3 image at container build time
* remove maven download and bootstrap code
* fix returns and exits so we terminate quickstart build upon first fatal error
* remove quickstart ci job
* remove broken maven download from inside quickstart
* remove obsolete arg
* bump microservice-parent version in child poms
* bump microservice-service-parent version in child poms
* use mvn --settings arg
* rename vars and add to pom for warehouse docker image
* add web tests to CI
* reenable compose snapshot ci job
* add missed mvn --settings arg
* testing
* fix bug
* remove testing changes
* fix web tests
* fixes so users can use scripts outside of the container
---------
Co-authored-by: Joe Alphonso <[email protected]>
* Github CI changes to support pom-updates
* fix microservice image push
* remove no-op, reorder, add comment
* MR feedback and split microservice container builds out
* use verbose maven args for readability
* split deploy workflow into two jobs, clean up commented code, add job deps
* tweaks
* remove .github workflows from former submodules
* try 2nd task in same job instead
* rename job
* remove dupcliate dep declaration
* disable build-cache since it is not being saved
* fix rebase issues
* speed up test retries
* add skipITs since skipTests is deprecated for the failsafe plugin
* add skip.unit.tests property
* Update tests.yml
* Update tests.yml
* allow integration tests to run separately from unit tests
---------
Co-authored-by: Joe Alphonso <[email protected]>
* updates to build datawave quickstart with a given repository location.
* updates to build datawave quickstart with a given repository location.
* MR suggestions
* Fix github actions build
* Updating tests
* Added in args to be able to sed docker build environment variables:
* Fixes to get maven-based dependency downloads working for quickstart
* debug prints
* more testing
* more testing
* more testing
* potential fix
* potential fix
* potential fix
* potential fix
* another test
* another test
* another test
* fixed?
* added instructions for downloading tarballs from maven to the docker readme
* Delete docker/rabbitmq-query-config directory
* Update env.sh
---------
Co-authored-by: Whitney O'Meara <[email protected]>
Co-authored-by: Whitney O'Meara <[email protected]>
* Modified a couple of things and fixed a few tests with the integration merge.
* pulling in latest audit changes
* pulled in latest audit updates
* Fixed metrics version in docker compose
* Updated accumulo utils and datawave starter versions.
* Updated the audit api and audit starter versions
* Updated authorization api version
* Updated accumulo api version.
* Updated config service with latest changes
* updated dictionary and accumulo utils version
* Updated hazelcast version
* bumped query metric api version
* Updated pki for query microservices docker deployment.
* Fixing the docker compose bootstrap to avoid duplicate extra hosts issue.
* adding latest datawave changes
* updated metadata utils
* updated docker compose to use the latest version of each service.
* Added a cached results spring boot starter and updated associated classes accordingly.
* fixing spotbugs error
* fixed install-ingest.sh for datawave.
* Updates to get bulk results map reduce query working again.
* formatting
* Updates per a race condition discovered in the kafka variant of the
query executor tests.
* Update README.md
* accumulo service merge from main
* fixed a test that broke after merging integration
* Added logic from Datawave Principal Login Module to the datawave spring boot starter.
* Updated datawave to 5.4.0-SNAPSHOT
* Updated ProxiedUserDetails to use dynamic type for newInstance method.
* updated audit service to stop intermittent test failures.
* Updated the query results publisher and setup the kafka manager to create topics manually.
* Added github actions for some submodules
* Updated github actions to avoid running out of disk space.
* Updated import sorting for microservices using impsort plugin.
* fixing merge conflicts
* updated metadata utils
* Moved Query, QueryImpl to query-api
* Removed usage of zookeeper stringutils
* added package-info.java for Query, QueryImpl
* updated modification service logic to use ProxiedUserDetails.
* Removed unnecessary jackson annotations to address enunciate issues
* Updated profile ids for submodules
* changes to get the github actions working again
* attempting to fix github actions
* typo
* udpated kafka config
* removed banned dependency
* bumped release version
* formatting
* fixing docker compose kafka setup
* updated deploy poms to work with the latest version of maven
* bumped version for some modules
* bumped microservice service parent version
* Updated with latest changes from main/integration
* fixed the composite query logic for the query microservices
* updated submodules
* Added v1 to the model path
* For this branch we need to checkout all submodules when building tests.
* Added alerman's run cache settings for docker
* Updated github actions to build query microservices correctly
* fixed build cache config typo
* Updated query service to use pool header, and allow for admin override via parameter
* Updated github ci to make sure it runs against the queryMicroservices branch
* Updated lookup uuid to accept multiple terms per field
* Updated confirm ack configuration for query metric service, and added configuration to make rest metrics updates possible
* Added support for long running queries
* Added configcheck command line utility
* updated next call logic for long running queries
* updated query metric service and starter submodules
* Added some plugins to the config check service
* Updated configcheck tests to trim whitespace
* fixing unit tests for config check
* Changed from max long running query timeouts to long running query
timeout
* Updated the query starter submodule
* updated submodules
* fixed composite query logic test
* Refactored query scripts to use common code
* Updated quickstart docker-maven-plugin to not cleanup old tagged releases
* updated common-utils submodule
* Updated LookupService to behave the same as the webservice
* Fixed TypedValue (de)serialization, and added a result post processor to handle things like counts
* reset docker-compose quickstart to just be accumulo, not web
* Changed how the result postprocessor runs to support result aggregation for long running queries
* Renamed aggregateResults to reduceResults
* Added junit-jupiter engine to datawave core pom
* Updated query logic factory to add logic map
* Updated EdgeQueryLogicFactory configuration
* Removed unused RemoteQueryLogic and cleaned up RemoteEventQueryLogic class definition:
* Implemented authorization and query federation for the query microservices
* Fixed copy constructor parameter assignment in QueryData
* Formatting job fix
* Fixing build/formatter issues for latest changes pulled in from integration
* Fixed SSDeepIngestQueryTest
* Updated usage of remote user operations for query microservices
* Fixed QueryPlan constructor
* Moved filtering and delegating query logic functionality into datawave-core-query so it can be used by the microservices
* formatting
* Updated submodules
* Updated package names for commons.lang3 classes due to type-utils fix
* Moved the AuthorizationsPredicate class to authorization-api
* Removed hadoop dependencies from pom
* Updated edge configuration to reset param variables before loading new values
* Excluded log4j from our maven plugins
* Updated in-memory-accumulo javadoc plugin to exclude log4j
* Updated microservice-parent maven-dependency-plugin to exclude log4j
* Removed infinispan from query starter
* Remove validation call from LoadModel
* fixed formatting
* Updated datawave ingest jar libs
* QueryMetricQueryLogic must copy runAsMetricsAdministrator setting in copy constructor
* Revert query metric query logic back to it's original state
* pr feedback
* PR feedback
* PR Feedback
* PR feedback
* PR Feedback
* PR feedback
* PR feedback
* PR Feedback
* PR Feedback
* PR feedback
* PR feedback
* PR feedback
* pr feedback
* PR feedback
* pr feedback
* pr feedback
* pr feedback
* pr feedback
* pr feedback
* pr feedback
* PR feedback
* PR feedback
* pr feedback
* pr feedback
* Updated in-memory-accumulo commit
* in-memory-accumulo
* formatting
* query starter
* pr feedback
* pr feedback
* pr feedback
* pr feedback
* pr feedback
* pr feedback
* pr feedback
* Remoted CachedQueryStatus mechanism
* pr feedback
* pr feedback
* pr feedback
* pr feedback
* pr updates
* pr feedback
* pr review: found duplicate properties between configurations and the
super GenericQueryConfiguration class. Removed redundant copies in
ShardQueryConfiguration checkpoint copy constructor
* review comments
* review comments
* Reverted path changes in dictionary service
* pr review updates
* updated executor to use QueryStatusUpdateUtil
* updated query microservice submodule
* make github action tests match integration
---------
Co-authored-by: Ivan Bella <[email protected]>
Co-authored-by: GitHub Actions <[email protected]>
Co-authored-by: Bill Oley <[email protected]>
* initial commit
* Updated query metric service/starter
* cleaning up a bit
* Fixed some bash syntax errors and added to default properties
* Updated query metric service version for docker compose
Drew Farris