## Configuration
Add the ability to enforce concurrent query limits across a group of
webservers. Zookeeper is used to track active queries and the following
data:
- The query ID
- The user who submitted the query
- The system the query was submitted on
- The query logic the query originated from
Query limit enforcement is done through the `QueryLimiter` class. Given
a user, system, and query logic, it can determine if any of the
following limits have been exceeded:
- The max allowed concurrent queries for the user.
- The max allowed concurrent queries of the query logic for the user.
- The max allowed concurrent queries for the system.
- The max allowed concurrent queries of the query logic for the system.
Limits may be defined and customized on a per-user and per-system basis.
They may also be defined for groups of query logics. The classes
`UserLimitProvider`, `SystemLimitProvider`, and
`QueryLogicGroupLimitProvider` are respectively responsible for
identifying the best limits to enforce for a user, system, and query
logic. They will be initialized in the `QueryLimiter` after providing a
`QueryLimitConfiguration` instance. The following can be configured:
On a system-wide basis:
- The default concurrent user query limit. This applies to the total
number of queries a user may run across all systems. May be overridden
per user.
- The default concurrent system query limit. Primarily to avoid a system
getting overloaded. May be overridden per system.
On a per-system basis:
- The system name/ids the configuration targets. Regex matching is
supported. Pattern uniqueness is enforced.
- The concurrent system query limit. Overrides the system-wide value.
- Whether queries submitted to the system count towards a user's
concurrent query total. This is always true unless specified otherwise.
- The concurrent system query limit for different query logic groups.
Regex matching against group names is supported. Pattern uniqueness is
enforced.
On a per-user basis:
- The user DN.
- The user's concurrent query limit. Overrides the system-wide
configuration.
- The user's concurrent query limit for different query logic groups.
Regex matching against group names is supported.
On a per-query-logic-group basis:
- The group name.
- The query logics included in the group. Regex matching is supported.
Pattern uniqueness is enforced.
- The default concurrent user query limit. This applies to the total
concurrent queries a user may run that originate from a query logic in
the group across all systems.
Given the possibilities for exact matches, partial regex matches, and
wildcard regex matches, the determination of the best limit to use for
any particular system, query logic, or query logic group is done by
sorting matches into the following 'matching buckets' (in best-match
priority):
1. Exact match: We attempt to find an exact match first and use the
associated limit.
2. Partial regex (non-wildcard-only): If we cannot find an exact match,
then we attempt to find all partial matches, and see if any of their
limits are met.
3. Wildcard-only regex: In the case of no exact or partial matches, we
use the wildcard match with the lowest limit.
## Implementation
Checking limits and marking as active/inactive is done through the
`QueryLimiter` class. The three main methods to know are:
- `QueryLimiter.checkForLimits()`
- `QueryLimiter.countQueryTowardsLimits()`
- `QueryLimiter.stopCountingQueryTowardsLimits()`
When a query is marked as active via
`QueryLimiter.countQueryTowardsLimits()`, it will delegate to the
ActiveQueryTracker, which will in turn create nodes in Zookeeper under
the `ActiveQueries` namespace. When `ActiveQueryTracker.trackQuery()` is
called, the following nodes are created:
```
# Container nodes
/users/<userDn>/<queryLogic> # Only for queries on systems that count towards user limit
/systems/<system>/<queryLogic>
/distinctQueryLogics/<queryLogic> [Only created if it does not exist.]
# Ephemeral nodes. These will auto-delete themselves if their associated Zookeeper connection ever goes down.
/users/<userDn>/<queryLogic>/<queryId> # Only for queries on systems that count towards user limit
/systems/<system>/<queryLogic>/<queryId>
```
`ActiveQueryTracker.trackQuery()` will return a `QueryHeartbeat` that
contain a list of `PersistentNode` (provided by the Apache Curator
library) wrappers around the ephemeral nodes listed above. The
`QueryHeartbeat` will maintain the connection to Zookeeper and attempt
to keep the ephemeral nodes present in Zookeeper until
`QueryHeartbeat.stop()` is called. If `QueryHeartbeat.stop()` is called,
or the webserver crashes, the ephemeral nodes will automatically be
deleted by Zookeeper.
The following error codes have been added:
```
412-20 - Concurrent query limit exceeded
500-164 - Error checking concurrent query limits
```
Closes#3100
---------
Co-authored-by: Bill Oley <[email protected]>
Co-authored-by: foster33 <[email protected]>
Co-authored-by: Ivan Bella <[email protected]>
bumps the microservice parent versions on the related modules that
reference them
re-enables formatting in CI now that all modules have the same plugin
configuration
verified there are no other commits in the microservice-parent and
microservice-service-parent besides the spotless plugin changes since
the last release
Co-authored-by: Joe Alphonso <[email protected]>
bumps the microservice parent versions on the related modules that
reference them
re-enables formatting in CI now that all modules have the same plugin
configuration
verified there are no other commits in the microservice-parent and
microservice-service-parent besides the spotless plugin change since the
last release
Co-authored-by: Joe Alphonso <[email protected]>
Use correct slf4j impl in wildfly and services, update versions of
type-utils, metadata-utils, common-utils, accumulo-utils,
base-rest-responses in microservice starters, apis, services to track
datawave version
The use of these dependencies under microservices (starters, services,
api) should track the datawave version instead of the version number
from when they were in separate repositories
Because the prior versions used a different groupId, we have the case
where multiple versions of the same artifact are being included in some
microservices.
For instance, this caused a serialization conflict once the
serializationId in BaseType (type-utils) changed in datawave version
7.30.0. When both versions are present on both sides of a serialization,
whether a serialization exception occurs is indeterminate.
gov.nsa.datawave.core:accumulo-utils
gov.nsa.datawave.core:type-utils
gov.nsa.datawave.core:metadata-utils
gov.nsa.datawave.core:common-utils
gov.nsa.datawave.core:base-rest-responses
After this PR is merged, the following should be done to make the
starters, apis, modules, and services depend on released version of each
other.
Release datawave-microservice-service-parent:5.0.8
In all modules with parent datawave-microservice-service-parent, update
to datawave-microservice-service-parent:5.0.8
Release authorization-api:4.0.2
In starter-datawave update to authorization-api:4.0.2
Release starter-datawave:4.0.6
Release query-metric-api:4.1.3
Release query-api:1.0.1
Release accumulo-api:4.0.1
Release modification-api:1.0.1
Release dictionary-api:4.0.6
In hazelcast-common update to starter-datawave:4.0.6
Release hazelcast-common:4.0.3
In hazelcast-client update to hazelcast-common:4.0.3
In starter-audit update to starter-datawave:4.0.6,
authorization-api:4.0.2
In starter-metadata update to starter-datawave:4.0.6
In starter-query-metric update to starter-datawave:4.0.6,
query-metric-api:4.1.3, authorization-api:4.0.2
Release hazelcast-client:4.0.3
Release starter-audit:4.0.4
Release starter-metadata:3.0.3
Release starter-query-metric:3.0.4
In starter-query update to starter-datawave:4.0.6, query-api:1.0.1,
hazelcast-client:4.0.3, starter-metadata:3.0.3,
starter-query-metric:3.0.4
Release starter-query:1.0.9
In starter-cached-query update to starter-datawave:4.0.6,
hazelcast-client:4.0.3, starter-audit:4.0.4, starter-query:1.0.9
Release starter-cached-query:1.0.11
In mapreduce-query-core-job update to starter-query:1.0.9
Release mapreduce-query-core-job:1.0.5
In query-executor-api update to starter-datawave:4.0.6,
starter-query:1.0.9
Release query-executor-api:1.0.8
In accumulo-service update to starter-datawave:4.0.6, accumulo-api:4.0.1
In audit-service update to starter-datawave:4.0.6,
hazelcast-client:4.0.3
In authorization-service update to starter-datawave:4.0.6,
hazelcast-client:4.0.3, authorization-api:4.0.2
In config-service update to starter-datawave:4.0.6
In dictionary-service update to starter-datawave:4.0.6,
dictionary-api:4.0.6, starter-metadata:3.0.3
In file-provider-service update to starter-datawave:4.0.6
In hazelcast-service update to hazelcast-common:4.0.3
In mapreduce-query-service update to mapreduce-query-core-job:1.0.5,
starter-query:1.0.9
In modification-service update to starter-metadata:3.0.3,
query-api:1.0.1, modification-api:1.0.1
In query-service update to starter-datawave:4.0.6, query-api:1.0.1,
starter-cached-results:1.0.11, starter-query:1.0.9
In query-executor-service update to starter-query:1.0.9,
query-executor-api:1.0.8
In query-metric-service update to starter-datawave:4.0.6,
starter-metadata:3.0.3, hazelcast-common:4.0.3
Release accumulo-service:4.0.3
Release audit-service:4.0.3
Release authorization-service:4.0.5
Release config-service:4.0.2
Release dictionary-service:4.0.10
Release file-provider-service:1.0.1
Release hazelcast-service:4.0.3
Release mapreduce-query-service:1.0.5
Release modification-service:1.0.3
Release query-service:1.0.12
Release query-executor-service:1.0.10
Release query-metric-service:4.1.9
renamed `docker.image.prefix` to `docker.registry` to make purpose clear
resolves image push failures
---------
Co-authored-by: Joe Alphonso <[email protected]>
* cleanup quickstart and jkube docker configuration
set compose prefix
stop boostrapping maven twice
add default back for microservices since they do not load the properties files
add latest tag to compose tests
# Conflicts:
# contrib/datawave-quickstart/docker/docker-build.sh
# contrib/datawave-quickstart/docker/pom.xml
# Conflicts:
# microservices/microservice-parent/pom.xml
* remove cdn download from pom and set file urls to local paths
* fix file paths
* testing
# Conflicts:
# microservices/microservice-service-parent/pom.xml
* cleanup and add quickstart web tests back to CI
* update quickstart ci test
* testing fixes
* fix bug
* address compose warning
* bugfix
* dev profile clean up and docker-entrypoint.sh fix
* bootstrap only when install is called
* kill process when there is a fatal error
* update maven to a version we actually have
* install maven via os pkg mgr to start
* move quickstart jdk check to the install.sh scripts
* copy maven from latest maven-3 image at container build time
* remove maven download and bootstrap code
* fix returns and exits so we terminate quickstart build upon first fatal error
* remove quickstart ci job
* remove broken maven download from inside quickstart
* remove obsolete arg
* bump microservice-parent version in child poms
* bump microservice-service-parent version in child poms
* use mvn --settings arg
* rename vars and add to pom for warehouse docker image
* add web tests to CI
* reenable compose snapshot ci job
* add missed mvn --settings arg
* testing
* fix bug
* remove testing changes
* fix web tests
* fixes so users can use scripts outside of the container
---------
Co-authored-by: Joe Alphonso <[email protected]>
* Do not create new branch
* [maven-release-plugin] prepare release microservices/microservice-parent-4.0.4
* [maven-release-plugin] prepare for next development iteration
* inclide settings file in args
* Update all dependent ms to use the new parent
* [maven-release-plugin] prepare release microservices/microservice-service-parent-5.0.4-RC1
* [maven-release-plugin] prepare for next development iteration
* [maven-release-plugin] prepare release microservices/microservice-service-parent-5.0.4
* [maven-release-plugin] prepare for next development iteration
* Update all services to use new parent
* Use tokens with write access
* Revert "Use tokens with write access"
This reverts commit eea89165aa.
* Use tokens with write access
* [maven-release-plugin] prepare release microservices/microservice-parent-4.0.5-CITEST-4
* [maven-release-plugin] prepare for next development iteration
---------
Co-authored-by: alerman <[email protected]>
* Formatting diffs
* formatter and import cleanup
* Pulled utils into main versioning scheme
* Changed the package from gov.nsa.datawave.micreoservice to gov.nsa.datawave.core
* Pulled in the following modules:
1. type-utils
2. accumulo-utils
3. metadata-utils
4. common-utils
5. base-rest-responses
* Update version
* consistent accumulo-utils version
* Changed core packaged to gov.nsa.datawave.core and to the latest
versions
* GitHub Actions: Fix Formatting
* pom formatting
* Spotbugs fix
* Updated re spotbugs
* fixed package names for util packages
* Updated package
* Explicit metadata-utils version
* GitHub Actions: Fix Formatting
* formatting
* Updated everything to the same protobuf version that we deploy.
Regenerated protobufs with the same version as well.
* Advance pom version of dictionary service to next snapshot
* a little test debug
* GitHub Actions: Fix Formatting
* Merge conflicts
* Remove invalid javadoc elements
* Updated to next snapshot version
* merge conflict fix
---------
Co-authored-by: GitHub Actions <[email protected]>
Co-authored-by: Dylan <[email protected]>
* make scm uniform so maven-release-plugin can be used
* add ghcr repo back so modules can download a released parent
---------
Co-authored-by: Joe Alphonso <[email protected]>
* 1st round
* 2nd round
* update mvn extensions.xml
* more plugin cleanup
* more plugin version bumps
* microservices use parent snapshot for development
* exclude log4j from yarn
* update contrib poms
* exclusions and plugin updates
* remove duplicate entry from dist.xml
* updates
* formatter version bump for in-memory-accumulo
* fix formatter config
* typo
* make formatter config consistent
* junit var and set removeTrailingWhitespace to true
* replace spotify docker plugin with fabric8
* fix web-services deploy
* remove several plugin warnings
* junit fixes
* contrib fixes
* revert datawave-code-style version
* remove several contrib warnings
* use relocated and updated git-commit-id-maven-plugin
* remove javadocVersion property to address warning
* add maven-install-plugin to microservice parent
* use jkube for docker builds on warehouse and web
* use spring-boot-maven-plugin native oci image build
* clean up plugins in in-memory-accumulo
* replace relative paths in quickstart with build-in var
* remove unused exec plugin
* Revert "use spring-boot-maven-plugin native oci image build"
This reverts commit b7b76587f52ee6893669771098bfad7226167e3a.
* fix banned exclusions
* remove deprecated javadoc conf option
* fix junit after rebase
* migrate a few test cases to JUnit5 and fix warnings
* use maven.multiModuleProjectDirectory instead of datawave.root
* revert accumulo version bump
* fix quickstart
* skuzzle plugin version bump
* remove duplicate dep
* remove duplicate repository declarations
* resolve spotbugs findings
* get rid of warning when using build-cache
* cleanup and fix web-service application pom
* revert accumulo version
* remove unused property
* formatting
* make docker rpm include less restrictive
---------
Co-authored-by: Joe Alphonso <[email protected]>
Ivan Bella