The whole sharded schema sat in one tablet, so no number of tablet
servers, scan servers or compactors made any difference to a query.
Splits now come from a checked-in file, the deployment moves to three
shards per day, and verifySplits.sh guards that in CI.
Also fixes the num shards cache, which scanned a metadata table name no
deployment that renames its tables actually uses. The missing table was
swallowed, so a day expanded into no shards and the query silently
returned nothing.
---------
Co-authored-by: Claude Opus 5 <[email protected]>
…s from NationalSecurityAgency/datawave-stack-docker-images
---------
Co-authored-by: Moriarty <[email protected]>
Co-authored-by: Claude Opus 5 <[email protected]>
Dan was having github issues so I pushed this for him in case anyone has
questions about this PR
---------
Co-authored-by: d-hwang <[email protected]>
Co-authored-by: Joseph Alphonso <[email protected]>
Better handle case where IvaratorRunnable gets stuck during yield /
suspend
If an IvaratorRunnable is being suspended on a yield and an Accumulo
scan prevents it from checking suspendRequested for greater than 60
seconds, then the suspend request will return, the yield will be
processed, and
the IvaratorRunnable will likely get an exception when using the
Accumulo source and print an ERROR log
To limit unnecessary ERRORs in the log, we should log at INFO if the
suspendRequested AtomicBoolean is set and ERROR otherwise
While some code is in place to make maxYields configurable, it is not
coded throuth to the query logic configuration the same way that the
yield threshold is. We should fix this and sen the maxYields to 20 which
would then be > 60 minutes at which point the query will time out in the
webservice.
renamed `docker.image.prefix` to `docker.registry` to make purpose clear
resolves image push failures
---------
Co-authored-by: Joe Alphonso <[email protected]>
* cleanup quickstart and jkube docker configuration
set compose prefix
stop boostrapping maven twice
add default back for microservices since they do not load the properties files
add latest tag to compose tests
# Conflicts:
# contrib/datawave-quickstart/docker/docker-build.sh
# contrib/datawave-quickstart/docker/pom.xml
# Conflicts:
# microservices/microservice-parent/pom.xml
* remove cdn download from pom and set file urls to local paths
* fix file paths
* testing
# Conflicts:
# microservices/microservice-service-parent/pom.xml
* cleanup and add quickstart web tests back to CI
* update quickstart ci test
* testing fixes
* fix bug
* address compose warning
* bugfix
* dev profile clean up and docker-entrypoint.sh fix
* bootstrap only when install is called
* kill process when there is a fatal error
* update maven to a version we actually have
* install maven via os pkg mgr to start
* move quickstart jdk check to the install.sh scripts
* copy maven from latest maven-3 image at container build time
* remove maven download and bootstrap code
* fix returns and exits so we terminate quickstart build upon first fatal error
* remove quickstart ci job
* remove broken maven download from inside quickstart
* remove obsolete arg
* bump microservice-parent version in child poms
* bump microservice-service-parent version in child poms
* use mvn --settings arg
* rename vars and add to pom for warehouse docker image
* add web tests to CI
* reenable compose snapshot ci job
* add missed mvn --settings arg
* testing
* fix bug
* remove testing changes
* fix web tests
* fixes so users can use scripts outside of the container
---------
Co-authored-by: Joe Alphonso <[email protected]>
* Add query field rule
* Add QueryPlanningStage (missed in first commit)
* updates to allow rules to access node lineage
* Make parseRules public
* Delete warehouse/query-core/src/main/java/datawave/query/planner/QueryPlanningStage.java
remove unused class
* Delete warehouse/query-core/src/main/java/datawave/query/jexl/visitors/QueryFieldsRuleVisitor.java
remove unused class
* move members to impl
---------
Co-authored-by: hlgp <[email protected]>
Co-authored-by: Moriarty <[email protected]>
* Add abilty to yield in Ivarators, AndIterator, OrIterator and return metrics before yield (#704)
* Improvement to debug and trace logging in QueryIerator.seek() (#704)
* Feedback from MR comments (#704)
* Add collectTimingDetails and yieldThresholdMs to spring-boot-starter-datawave-query
* Allow yieldThresholdMs < 50ms to work and check for values <= 0
* MR fixes for toStringImpl, toStringNoQueryId in Ivarator and subslasses; add some documentation
* Fix spotbugs issues
* Add yieldthresholdMs to application-query.yml for docker-compose configuration
* Add collectTimingDetails to application-query.yml
* Modified a couple of things and fixed a few tests with the integration merge.
* pulling in latest audit changes
* pulled in latest audit updates
* Fixed metrics version in docker compose
* Updated accumulo utils and datawave starter versions.
* Updated the audit api and audit starter versions
* Updated authorization api version
* Updated accumulo api version.
* Updated config service with latest changes
* updated dictionary and accumulo utils version
* Updated hazelcast version
* bumped query metric api version
* Updated pki for query microservices docker deployment.
* Fixing the docker compose bootstrap to avoid duplicate extra hosts issue.
* adding latest datawave changes
* updated metadata utils
* updated docker compose to use the latest version of each service.
* Added a cached results spring boot starter and updated associated classes accordingly.
* fixing spotbugs error
* fixed install-ingest.sh for datawave.
* Updates to get bulk results map reduce query working again.
* formatting
* Updates per a race condition discovered in the kafka variant of the
query executor tests.
* Update README.md
* accumulo service merge from main
* fixed a test that broke after merging integration
* Added logic from Datawave Principal Login Module to the datawave spring boot starter.
* Updated datawave to 5.4.0-SNAPSHOT
* Updated ProxiedUserDetails to use dynamic type for newInstance method.
* updated audit service to stop intermittent test failures.
* Updated the query results publisher and setup the kafka manager to create topics manually.
* Added github actions for some submodules
* Updated github actions to avoid running out of disk space.
* Updated import sorting for microservices using impsort plugin.
* fixing merge conflicts
* updated metadata utils
* Moved Query, QueryImpl to query-api
* Removed usage of zookeeper stringutils
* added package-info.java for Query, QueryImpl
* updated modification service logic to use ProxiedUserDetails.
* Removed unnecessary jackson annotations to address enunciate issues
* Updated profile ids for submodules
* changes to get the github actions working again
* attempting to fix github actions
* typo
* udpated kafka config
* removed banned dependency
* bumped release version
* formatting
* fixing docker compose kafka setup
* updated deploy poms to work with the latest version of maven
* bumped version for some modules
* bumped microservice service parent version
* Updated with latest changes from main/integration
* fixed the composite query logic for the query microservices
* updated submodules
* Added v1 to the model path
* For this branch we need to checkout all submodules when building tests.
* Added alerman's run cache settings for docker
* Updated github actions to build query microservices correctly
* fixed build cache config typo
* Updated query service to use pool header, and allow for admin override via parameter
* Updated github ci to make sure it runs against the queryMicroservices branch
* Updated lookup uuid to accept multiple terms per field
* Updated confirm ack configuration for query metric service, and added configuration to make rest metrics updates possible
* Added support for long running queries
* Added configcheck command line utility
* updated next call logic for long running queries
* updated query metric service and starter submodules
* Added some plugins to the config check service
* Updated configcheck tests to trim whitespace
* fixing unit tests for config check
* Changed from max long running query timeouts to long running query
timeout
* Updated the query starter submodule
* updated submodules
* fixed composite query logic test
* Refactored query scripts to use common code
* Updated quickstart docker-maven-plugin to not cleanup old tagged releases
* updated common-utils submodule
* Updated LookupService to behave the same as the webservice
* Fixed TypedValue (de)serialization, and added a result post processor to handle things like counts
* reset docker-compose quickstart to just be accumulo, not web
* Changed how the result postprocessor runs to support result aggregation for long running queries
* Renamed aggregateResults to reduceResults
* Added junit-jupiter engine to datawave core pom
* Updated query logic factory to add logic map
* Updated EdgeQueryLogicFactory configuration
* Removed unused RemoteQueryLogic and cleaned up RemoteEventQueryLogic class definition:
* Implemented authorization and query federation for the query microservices
* Fixed copy constructor parameter assignment in QueryData
* Formatting job fix
* Fixing build/formatter issues for latest changes pulled in from integration
* Fixed SSDeepIngestQueryTest
* Updated usage of remote user operations for query microservices
* Fixed QueryPlan constructor
* Moved filtering and delegating query logic functionality into datawave-core-query so it can be used by the microservices
* formatting
* Updated submodules
* Updated package names for commons.lang3 classes due to type-utils fix
* Moved the AuthorizationsPredicate class to authorization-api
* Removed hadoop dependencies from pom
* Updated edge configuration to reset param variables before loading new values
* Excluded log4j from our maven plugins
* Updated in-memory-accumulo javadoc plugin to exclude log4j
* Updated microservice-parent maven-dependency-plugin to exclude log4j
* Removed infinispan from query starter
* Remove validation call from LoadModel
* fixed formatting
* Updated datawave ingest jar libs
* QueryMetricQueryLogic must copy runAsMetricsAdministrator setting in copy constructor
* Revert query metric query logic back to it's original state
* pr feedback
* PR feedback
* PR Feedback
* PR feedback
* PR Feedback
* PR feedback
* PR feedback
* PR Feedback
* PR Feedback
* PR feedback
* PR feedback
* PR feedback
* pr feedback
* PR feedback
* pr feedback
* pr feedback
* pr feedback
* pr feedback
* pr feedback
* pr feedback
* PR feedback
* PR feedback
* pr feedback
* pr feedback
* Updated in-memory-accumulo commit
* in-memory-accumulo
* formatting
* query starter
* pr feedback
* pr feedback
* pr feedback
* pr feedback
* pr feedback
* pr feedback
* pr feedback
* Remoted CachedQueryStatus mechanism
* pr feedback
* pr feedback
* pr feedback
* pr feedback
* pr updates
* pr feedback
* pr review: found duplicate properties between configurations and the
super GenericQueryConfiguration class. Removed redundant copies in
ShardQueryConfiguration checkpoint copy constructor
* review comments
* review comments
* Reverted path changes in dictionary service
* pr review updates
* updated executor to use QueryStatusUpdateUtil
* updated query microservice submodule
* make github action tests match integration
---------
Co-authored-by: Ivan Bella <[email protected]>
Co-authored-by: GitHub Actions <[email protected]>
Co-authored-by: Bill Oley <[email protected]>
* Created an event predicate for discarding events during ingest processing
* Created a discard interval and future discard interval predicates
* Updated tests to test hit and miss on the interval predicates
* Add default for DISCARD_FUTURE_INTERVAL event
Co-authored-by: foster33 <[email protected]>
Co-authored-by: Keith Ratcliffe <[email protected]>
Co-authored-by: hlgp <[email protected]>
Co-authored-by: palindrome <[email protected]>
Co-authored-by: matthpeterson <[email protected]>
* initial commit
* Updated query metric service/starter
* cleaning up a bit
* Fixed some bash syntax errors and added to default properties
* Updated query metric service version for docker compose
- Updates for Accumulo 2.x bulk api
- Improved junit verification of bulk load via MiniAccumuloCluster integration
- Minor bug fixes to start-loader.sh script
- Quickstart updates to configure bulk ingest
* optionally disable creating uids for the global index
* relocated uid flag to shard-ingest-config.xml, rework create uid logic
* remove return statement
* use shardIndexCreateUids directly
* re #1821: Added a remote user operations implementation
* Updated the authorization mechanisms to validate against the overall
user (which may include the remote authorizations) and then the reduce
by the query authorizations. This allows the auths returned by the list
effective authorizations which include remote auths to be used for local
query logics as well.
* Added optional flag to the listEffectiveAuthorizations avoid federation
* Updated Query to now know about a remote user operations
* logics will use the remote user operation if set for query auths
* Composite Query Logic will recalculate query auths for each logic
* Added test cases for new classes and integration tests
* Updated test cases to handle new query logic getUserOperations call
* Updated to set the response object factory via spring for non-CDI instantiated object (i.e. query logics)
* Updated to fix responseObjectFactory usage various classes
Queries marked as "allowLongRunningQuery" can exceed the 55 minute timeout to return results. This ticket is explicitely for groupBy queries, but it was implemented in a way that can be extending to other types of queries as well. Key changes:
* If a query that's allowed to be long running exceeds 50 minutes to return (so there's no risk of hitting the 55 minute timeout) , an empty ResultsPage is returned with a status of "PARTIAL", and the query continues to execute on the tserver.
* When the query does complete, all results are returned at once with a status of "COMPLETE" .
* The call to getTransformer() in the query logic now returns a singleton instance so that transformers in the list can be updated with the query start time necessary for the logic of this ticket
* The client side and tserver side of the grouping transform were broken out into two separate instances for clarity and maintainability (GroupingTransform and GroupingIterator respectively)
* LongRunningQueryTest tests this logic
