The standalone install adds it in setup-wildfly.sh, but the image ran no
equivalent, so in-VM messaging connections failed authentication and
retried unthrottled, risking an OutOfMemoryError.
Co-authored-by: Claude Opus 5 <[email protected]>
## Configuration
Add the ability to enforce concurrent query limits across a group of
webservers. Zookeeper is used to track active queries and the following
data:
- The query ID
- The user who submitted the query
- The system the query was submitted on
- The query logic the query originated from
Query limit enforcement is done through the `QueryLimiter` class. Given
a user, system, and query logic, it can determine if any of the
following limits have been exceeded:
- The max allowed concurrent queries for the user.
- The max allowed concurrent queries of the query logic for the user.
- The max allowed concurrent queries for the system.
- The max allowed concurrent queries of the query logic for the system.
Limits may be defined and customized on a per-user and per-system basis.
They may also be defined for groups of query logics. The classes
`UserLimitProvider`, `SystemLimitProvider`, and
`QueryLogicGroupLimitProvider` are respectively responsible for
identifying the best limits to enforce for a user, system, and query
logic. They will be initialized in the `QueryLimiter` after providing a
`QueryLimitConfiguration` instance. The following can be configured:
On a system-wide basis:
- The default concurrent user query limit. This applies to the total
number of queries a user may run across all systems. May be overridden
per user.
- The default concurrent system query limit. Primarily to avoid a system
getting overloaded. May be overridden per system.
On a per-system basis:
- The system name/ids the configuration targets. Regex matching is
supported. Pattern uniqueness is enforced.
- The concurrent system query limit. Overrides the system-wide value.
- Whether queries submitted to the system count towards a user's
concurrent query total. This is always true unless specified otherwise.
- The concurrent system query limit for different query logic groups.
Regex matching against group names is supported. Pattern uniqueness is
enforced.
On a per-user basis:
- The user DN.
- The user's concurrent query limit. Overrides the system-wide
configuration.
- The user's concurrent query limit for different query logic groups.
Regex matching against group names is supported.
On a per-query-logic-group basis:
- The group name.
- The query logics included in the group. Regex matching is supported.
Pattern uniqueness is enforced.
- The default concurrent user query limit. This applies to the total
concurrent queries a user may run that originate from a query logic in
the group across all systems.
Given the possibilities for exact matches, partial regex matches, and
wildcard regex matches, the determination of the best limit to use for
any particular system, query logic, or query logic group is done by
sorting matches into the following 'matching buckets' (in best-match
priority):
1. Exact match: We attempt to find an exact match first and use the
associated limit.
2. Partial regex (non-wildcard-only): If we cannot find an exact match,
then we attempt to find all partial matches, and see if any of their
limits are met.
3. Wildcard-only regex: In the case of no exact or partial matches, we
use the wildcard match with the lowest limit.
## Implementation
Checking limits and marking as active/inactive is done through the
`QueryLimiter` class. The three main methods to know are:
- `QueryLimiter.checkForLimits()`
- `QueryLimiter.countQueryTowardsLimits()`
- `QueryLimiter.stopCountingQueryTowardsLimits()`
When a query is marked as active via
`QueryLimiter.countQueryTowardsLimits()`, it will delegate to the
ActiveQueryTracker, which will in turn create nodes in Zookeeper under
the `ActiveQueries` namespace. When `ActiveQueryTracker.trackQuery()` is
called, the following nodes are created:
```
# Container nodes
/users/<userDn>/<queryLogic> # Only for queries on systems that count towards user limit
/systems/<system>/<queryLogic>
/distinctQueryLogics/<queryLogic> [Only created if it does not exist.]
# Ephemeral nodes. These will auto-delete themselves if their associated Zookeeper connection ever goes down.
/users/<userDn>/<queryLogic>/<queryId> # Only for queries on systems that count towards user limit
/systems/<system>/<queryLogic>/<queryId>
```
`ActiveQueryTracker.trackQuery()` will return a `QueryHeartbeat` that
contain a list of `PersistentNode` (provided by the Apache Curator
library) wrappers around the ephemeral nodes listed above. The
`QueryHeartbeat` will maintain the connection to Zookeeper and attempt
to keep the ephemeral nodes present in Zookeeper until
`QueryHeartbeat.stop()` is called. If `QueryHeartbeat.stop()` is called,
or the webserver crashes, the ephemeral nodes will automatically be
deleted by Zookeeper.
The following error codes have been added:
```
412-20 - Concurrent query limit exceeded
500-164 - Error checking concurrent query limits
```
Closes#3100
---------
Co-authored-by: Bill Oley <[email protected]>
Co-authored-by: foster33 <[email protected]>
Co-authored-by: Ivan Bella <[email protected]>
Use correct slf4j impl in wildfly and services, update versions of
type-utils, metadata-utils, common-utils, accumulo-utils,
base-rest-responses in microservice starters, apis, services to track
datawave version
The use of these dependencies under microservices (starters, services,
api) should track the datawave version instead of the version number
from when they were in separate repositories
Because the prior versions used a different groupId, we have the case
where multiple versions of the same artifact are being included in some
microservices.
For instance, this caused a serialization conflict once the
serializationId in BaseType (type-utils) changed in datawave version
7.30.0. When both versions are present on both sides of a serialization,
whether a serialization exception occurs is indeterminate.
gov.nsa.datawave.core:accumulo-utils
gov.nsa.datawave.core:type-utils
gov.nsa.datawave.core:metadata-utils
gov.nsa.datawave.core:common-utils
gov.nsa.datawave.core:base-rest-responses
After this PR is merged, the following should be done to make the
starters, apis, modules, and services depend on released version of each
other.
Release datawave-microservice-service-parent:5.0.8
In all modules with parent datawave-microservice-service-parent, update
to datawave-microservice-service-parent:5.0.8
Release authorization-api:4.0.2
In starter-datawave update to authorization-api:4.0.2
Release starter-datawave:4.0.6
Release query-metric-api:4.1.3
Release query-api:1.0.1
Release accumulo-api:4.0.1
Release modification-api:1.0.1
Release dictionary-api:4.0.6
In hazelcast-common update to starter-datawave:4.0.6
Release hazelcast-common:4.0.3
In hazelcast-client update to hazelcast-common:4.0.3
In starter-audit update to starter-datawave:4.0.6,
authorization-api:4.0.2
In starter-metadata update to starter-datawave:4.0.6
In starter-query-metric update to starter-datawave:4.0.6,
query-metric-api:4.1.3, authorization-api:4.0.2
Release hazelcast-client:4.0.3
Release starter-audit:4.0.4
Release starter-metadata:3.0.3
Release starter-query-metric:3.0.4
In starter-query update to starter-datawave:4.0.6, query-api:1.0.1,
hazelcast-client:4.0.3, starter-metadata:3.0.3,
starter-query-metric:3.0.4
Release starter-query:1.0.9
In starter-cached-query update to starter-datawave:4.0.6,
hazelcast-client:4.0.3, starter-audit:4.0.4, starter-query:1.0.9
Release starter-cached-query:1.0.11
In mapreduce-query-core-job update to starter-query:1.0.9
Release mapreduce-query-core-job:1.0.5
In query-executor-api update to starter-datawave:4.0.6,
starter-query:1.0.9
Release query-executor-api:1.0.8
In accumulo-service update to starter-datawave:4.0.6, accumulo-api:4.0.1
In audit-service update to starter-datawave:4.0.6,
hazelcast-client:4.0.3
In authorization-service update to starter-datawave:4.0.6,
hazelcast-client:4.0.3, authorization-api:4.0.2
In config-service update to starter-datawave:4.0.6
In dictionary-service update to starter-datawave:4.0.6,
dictionary-api:4.0.6, starter-metadata:3.0.3
In file-provider-service update to starter-datawave:4.0.6
In hazelcast-service update to hazelcast-common:4.0.3
In mapreduce-query-service update to mapreduce-query-core-job:1.0.5,
starter-query:1.0.9
In modification-service update to starter-metadata:3.0.3,
query-api:1.0.1, modification-api:1.0.1
In query-service update to starter-datawave:4.0.6, query-api:1.0.1,
starter-cached-results:1.0.11, starter-query:1.0.9
In query-executor-service update to starter-query:1.0.9,
query-executor-api:1.0.8
In query-metric-service update to starter-datawave:4.0.6,
starter-metadata:3.0.3, hazelcast-common:4.0.3
Release accumulo-service:4.0.3
Release audit-service:4.0.3
Release authorization-service:4.0.5
Release config-service:4.0.2
Release dictionary-service:4.0.10
Release file-provider-service:1.0.1
Release hazelcast-service:4.0.3
Release mapreduce-query-service:1.0.5
Release modification-service:1.0.3
Release query-service:1.0.12
Release query-executor-service:1.0.10
Release query-metric-service:4.1.9
This adds a LazyLoadingRangesIterator, which calculates a certain number
of ranges at a time. The amount of ranges calculated at a time is
configurable. This avoids calculating all ranges at the same time and
storing them all in memory.
Better handle case where IvaratorRunnable gets stuck during yield /
suspend
If an IvaratorRunnable is being suspended on a yield and an Accumulo
scan prevents it from checking suspendRequested for greater than 60
seconds, then the suspend request will return, the yield will be
processed, and
the IvaratorRunnable will likely get an exception when using the
Accumulo source and print an ERROR log
To limit unnecessary ERRORs in the log, we should log at INFO if the
suspendRequested AtomicBoolean is set and ERROR otherwise
While some code is in place to make maxYields configurable, it is not
coded throuth to the query logic configuration the same way that the
yield threshold is. We should fix this and sen the maxYields to 20 which
would then be > 60 minutes at which point the query will time out in the
webservice.
Kevin Toth