Better handle case where IvaratorRunnable gets stuck during yield /
suspend
If an IvaratorRunnable is being suspended on a yield and an Accumulo
scan prevents it from checking suspendRequested for greater than 60
seconds, then the suspend request will return, the yield will be
processed, and
the IvaratorRunnable will likely get an exception when using the
Accumulo source and print an ERROR log
To limit unnecessary ERRORs in the log, we should log at INFO if the
suspendRequested AtomicBoolean is set and ERROR otherwise
While some code is in place to make maxYields configurable, it is not
coded throuth to the query logic configuration the same way that the
yield threshold is. We should fix this and sen the maxYields to 20 which
would then be > 60 minutes at which point the query will time out in the
webservice.
* Add query field rule
* Add QueryPlanningStage (missed in first commit)
* updates to allow rules to access node lineage
* Make parseRules public
* Delete warehouse/query-core/src/main/java/datawave/query/planner/QueryPlanningStage.java
remove unused class
* Delete warehouse/query-core/src/main/java/datawave/query/jexl/visitors/QueryFieldsRuleVisitor.java
remove unused class
* move members to impl
---------
Co-authored-by: hlgp <[email protected]>
Co-authored-by: Moriarty <[email protected]>
* Add abilty to yield in Ivarators, AndIterator, OrIterator and return metrics before yield (#704)
* Improvement to debug and trace logging in QueryIerator.seek() (#704)
* Feedback from MR comments (#704)
* Add collectTimingDetails and yieldThresholdMs to spring-boot-starter-datawave-query
* Allow yieldThresholdMs < 50ms to work and check for values <= 0
* MR fixes for toStringImpl, toStringNoQueryId in Ivarator and subslasses; add some documentation
* Fix spotbugs issues
* Add yieldthresholdMs to application-query.yml for docker-compose configuration
* Add collectTimingDetails to application-query.yml
* Modified a couple of things and fixed a few tests with the integration merge.
* pulling in latest audit changes
* pulled in latest audit updates
* Fixed metrics version in docker compose
* Updated accumulo utils and datawave starter versions.
* Updated the audit api and audit starter versions
* Updated authorization api version
* Updated accumulo api version.
* Updated config service with latest changes
* updated dictionary and accumulo utils version
* Updated hazelcast version
* bumped query metric api version
* Updated pki for query microservices docker deployment.
* Fixing the docker compose bootstrap to avoid duplicate extra hosts issue.
* adding latest datawave changes
* updated metadata utils
* updated docker compose to use the latest version of each service.
* Added a cached results spring boot starter and updated associated classes accordingly.
* fixing spotbugs error
* fixed install-ingest.sh for datawave.
* Updates to get bulk results map reduce query working again.
* formatting
* Updates per a race condition discovered in the kafka variant of the
query executor tests.
* Update README.md
* accumulo service merge from main
* fixed a test that broke after merging integration
* Added logic from Datawave Principal Login Module to the datawave spring boot starter.
* Updated datawave to 5.4.0-SNAPSHOT
* Updated ProxiedUserDetails to use dynamic type for newInstance method.
* updated audit service to stop intermittent test failures.
* Updated the query results publisher and setup the kafka manager to create topics manually.
* Added github actions for some submodules
* Updated github actions to avoid running out of disk space.
* Updated import sorting for microservices using impsort plugin.
* fixing merge conflicts
* updated metadata utils
* Moved Query, QueryImpl to query-api
* Removed usage of zookeeper stringutils
* added package-info.java for Query, QueryImpl
* updated modification service logic to use ProxiedUserDetails.
* Removed unnecessary jackson annotations to address enunciate issues
* Updated profile ids for submodules
* changes to get the github actions working again
* attempting to fix github actions
* typo
* udpated kafka config
* removed banned dependency
* bumped release version
* formatting
* fixing docker compose kafka setup
* updated deploy poms to work with the latest version of maven
* bumped version for some modules
* bumped microservice service parent version
* Updated with latest changes from main/integration
* fixed the composite query logic for the query microservices
* updated submodules
* Added v1 to the model path
* For this branch we need to checkout all submodules when building tests.
* Added alerman's run cache settings for docker
* Updated github actions to build query microservices correctly
* fixed build cache config typo
* Updated query service to use pool header, and allow for admin override via parameter
* Updated github ci to make sure it runs against the queryMicroservices branch
* Updated lookup uuid to accept multiple terms per field
* Updated confirm ack configuration for query metric service, and added configuration to make rest metrics updates possible
* Added support for long running queries
* Added configcheck command line utility
* updated next call logic for long running queries
* updated query metric service and starter submodules
* Added some plugins to the config check service
* Updated configcheck tests to trim whitespace
* fixing unit tests for config check
* Changed from max long running query timeouts to long running query
timeout
* Updated the query starter submodule
* updated submodules
* fixed composite query logic test
* Refactored query scripts to use common code
* Updated quickstart docker-maven-plugin to not cleanup old tagged releases
* updated common-utils submodule
* Updated LookupService to behave the same as the webservice
* Fixed TypedValue (de)serialization, and added a result post processor to handle things like counts
* reset docker-compose quickstart to just be accumulo, not web
* Changed how the result postprocessor runs to support result aggregation for long running queries
* Renamed aggregateResults to reduceResults
* Added junit-jupiter engine to datawave core pom
* Updated query logic factory to add logic map
* Updated EdgeQueryLogicFactory configuration
* Removed unused RemoteQueryLogic and cleaned up RemoteEventQueryLogic class definition:
* Implemented authorization and query federation for the query microservices
* Fixed copy constructor parameter assignment in QueryData
* Formatting job fix
* Fixing build/formatter issues for latest changes pulled in from integration
* Fixed SSDeepIngestQueryTest
* Updated usage of remote user operations for query microservices
* Fixed QueryPlan constructor
* Moved filtering and delegating query logic functionality into datawave-core-query so it can be used by the microservices
* formatting
* Updated submodules
* Updated package names for commons.lang3 classes due to type-utils fix
* Moved the AuthorizationsPredicate class to authorization-api
* Removed hadoop dependencies from pom
* Updated edge configuration to reset param variables before loading new values
* Excluded log4j from our maven plugins
* Updated in-memory-accumulo javadoc plugin to exclude log4j
* Updated microservice-parent maven-dependency-plugin to exclude log4j
* Removed infinispan from query starter
* Remove validation call from LoadModel
* fixed formatting
* Updated datawave ingest jar libs
* QueryMetricQueryLogic must copy runAsMetricsAdministrator setting in copy constructor
* Revert query metric query logic back to it's original state
* pr feedback
* PR feedback
* PR Feedback
* PR feedback
* PR Feedback
* PR feedback
* PR feedback
* PR Feedback
* PR Feedback
* PR feedback
* PR feedback
* PR feedback
* pr feedback
* PR feedback
* pr feedback
* pr feedback
* pr feedback
* pr feedback
* pr feedback
* pr feedback
* PR feedback
* PR feedback
* pr feedback
* pr feedback
* Updated in-memory-accumulo commit
* in-memory-accumulo
* formatting
* query starter
* pr feedback
* pr feedback
* pr feedback
* pr feedback
* pr feedback
* pr feedback
* pr feedback
* Remoted CachedQueryStatus mechanism
* pr feedback
* pr feedback
* pr feedback
* pr feedback
* pr updates
* pr feedback
* pr review: found duplicate properties between configurations and the
super GenericQueryConfiguration class. Removed redundant copies in
ShardQueryConfiguration checkpoint copy constructor
* review comments
* review comments
* Reverted path changes in dictionary service
* pr review updates
* updated executor to use QueryStatusUpdateUtil
* updated query microservice submodule
* make github action tests match integration
---------
Co-authored-by: Ivan Bella <[email protected]>
Co-authored-by: GitHub Actions <[email protected]>
Co-authored-by: Bill Oley <[email protected]>
* initial commit
* Updated query metric service/starter
* cleaning up a bit
* Fixed some bash syntax errors and added to default properties
* Updated query metric service version for docker compose
* optionally disable creating uids for the global index
* relocated uid flag to shard-ingest-config.xml, rework create uid logic
* remove return statement
* use shardIndexCreateUids directly
* re #1821: Added a remote user operations implementation
* Updated the authorization mechanisms to validate against the overall
user (which may include the remote authorizations) and then the reduce
by the query authorizations. This allows the auths returned by the list
effective authorizations which include remote auths to be used for local
query logics as well.
* Added optional flag to the listEffectiveAuthorizations avoid federation
* Updated Query to now know about a remote user operations
* logics will use the remote user operation if set for query auths
* Composite Query Logic will recalculate query auths for each logic
* Added test cases for new classes and integration tests
* Updated test cases to handle new query logic getUserOperations call
* Updated to set the response object factory via spring for non-CDI instantiated object (i.e. query logics)
* Updated to fix responseObjectFactory usage various classes
Queries marked as "allowLongRunningQuery" can exceed the 55 minute timeout to return results. This ticket is explicitely for groupBy queries, but it was implemented in a way that can be extending to other types of queries as well. Key changes:
* If a query that's allowed to be long running exceeds 50 minutes to return (so there's no risk of hitting the 55 minute timeout) , an empty ResultsPage is returned with a status of "PARTIAL", and the query continues to execute on the tserver.
* When the query does complete, all results are returned at once with a status of "COMPLETE" .
* The call to getTransformer() in the query logic now returns a singleton instance so that transformers in the list can be updated with the query start time necessary for the logic of this ticket
* The client side and tserver side of the grouping transform were broken out into two separate instances for clarity and maintainability (GroupingTransform and GroupingIterator respectively)
* LongRunningQueryTest tests this logic
* closes#1225: add uid aggregator that will keep countOnly keys when 0 or below
* update javadoc to explain count only
* dropped soon-to-be-defunct test case, fixed test names, added test cases
Co-authored-by: Jeffrey Schmidt <[email protected]>
Co-authored-by: Ivan Bella <[email protected]>
Co-authored-by: hgklohr <[email protected]>
* Fix ingest and flagmaker metrics. Also enable -ingestMetricsDisabled flag to be overridden for BulkLoader process.
* Fix FlagMetrics formatting
Co-authored-by: Elizabeth Gilbert <[email protected]>
* Faceted Ingest & Search for Datawave (#793)
* `FacetHandler` generates facets for data at ingest time.
* Adds 3 tables: datawave.facets, datawave.facetMetadata, datawave.facetHashes.
* Adds dependency on `com.clearspring.analytics:stream` for summarization and cardinality estimation.
* Ingest configurations that add the FacetHandler for tvmaze/myjson data
* Adds FacetedQuery query logic and FacetedQueryPlanner
* Override hardcoded references to facet metadata in `FacetedQueryPlanner`, `FacetCheck`, `FacetQueryPlanVisitor`.
* Facet Table test related code added to `AccumuloSetupHelper`.
* Added build properties for facet tables in `QueryLogicFactory.xml`.
* `FacetedQueryLogicTest` now validates results,needs more work needed for the dynamic computation case.
* Added field name to facet values so they can be distinguished when we're faceting multiple fields.
* `DynamicFacetIterator` rebuilds fixed by having cardinality summaries include a max key.
* Improvements to table value handing in `PrintUtility` test utility.
* Fixes for clarity in `RebuildingScannerTestHelper`.
* Resolved concurrency issue in MergedReadAhead to fix intermittent test failures
* Added simple unit test to demonstrate properties of probabilistic data structure test.
* Created a StatsJob which will sample data and generate the selectivity of terms. These stats will eventually be incorporated into the query planning.
* Added the ability to combine ranges against the same field under an or node into a single ivarator.
* Updated travis config to build microservices.
* Updated DatawaveInterpreter and Arithmetic to support evaluation of ExceededOrThresholdMarkerJexlNodes.
* General code cleanup in preparation for RangeStream refactor.
* Added property to control collapsing uid requests that exceede a threshold.
* Refactored RangeStream for cleaner instantiation of limited scanners, added test for CreateUidsIterator, added comments with example use case.
* Added tests for CondensedUidIterator, RangeStream, and RangeCoalescingVisitor.
in favor of 'table.shard.numShardsPerDay'
* HDFS paths uppdated so that everything can be rooted under /datawave
* Quickstart changes:
- Fixes misconfiguration of Accumulo VFS context
- Updates to allow for easier customization
