Pick up in-memory-accumulo 4.0.6, which is the first release built
against Accumulo 2.1.5.
2.1.5 registers an UpgradeUtil KeywordExecutable that every Accumulo
process service-loads at startup. It uses JCommander's IUsageFormatter,
added in 1.78, so with jcommander 1.72 on the classpath
MiniAccumuloCluster fails to launch ZooKeeper and the failure surfaces
as a misleading ZooKeeperBindException.
Co-authored-by: Claude Opus 5 <[email protected]>
Updates the git-commit-id-maven-plugin version to 9.2.0 and switches it
from using JGit to native git.
This reduces the git.properties generation step time from ~65 seconds to
~20 seconds.
Prefix the checkstyle.xml path with the property
`${maven.multiModuleProjectDirectory}`. This will prevent child modules
from looking inside their own sub-folders for checkstyle.xml. This will
allow us to build child modules from within their root folders without
maven complaining about not being able to find the checkstyle.xml.
the banned dependencies in web-service/pom.xml were a subset of those in
pom.xml and were overriding them
enforce trhe currently specified log4j version using banned dependencies
(of prior version)
specify version of log4j dependency in spotbugs and spring-boot plugins
in configcheck, parent poms
these plugins otherwise specify their own versions of log4j which get
downloaded when they are executed
update accumulo-utils to use a POJO for storing markings and then
updated the rest of the codebase that referenced it.
replaced some of the ColumnVisibility usage with AccessExpression to
improve performance.
---------
Co-authored-by: Joe Alphonso <[email protected]>
## Configuration
Add the ability to enforce concurrent query limits across a group of
webservers. Zookeeper is used to track active queries and the following
data:
- The query ID
- The user who submitted the query
- The system the query was submitted on
- The query logic the query originated from
Query limit enforcement is done through the `QueryLimiter` class. Given
a user, system, and query logic, it can determine if any of the
following limits have been exceeded:
- The max allowed concurrent queries for the user.
- The max allowed concurrent queries of the query logic for the user.
- The max allowed concurrent queries for the system.
- The max allowed concurrent queries of the query logic for the system.
Limits may be defined and customized on a per-user and per-system basis.
They may also be defined for groups of query logics. The classes
`UserLimitProvider`, `SystemLimitProvider`, and
`QueryLogicGroupLimitProvider` are respectively responsible for
identifying the best limits to enforce for a user, system, and query
logic. They will be initialized in the `QueryLimiter` after providing a
`QueryLimitConfiguration` instance. The following can be configured:
On a system-wide basis:
- The default concurrent user query limit. This applies to the total
number of queries a user may run across all systems. May be overridden
per user.
- The default concurrent system query limit. Primarily to avoid a system
getting overloaded. May be overridden per system.
On a per-system basis:
- The system name/ids the configuration targets. Regex matching is
supported. Pattern uniqueness is enforced.
- The concurrent system query limit. Overrides the system-wide value.
- Whether queries submitted to the system count towards a user's
concurrent query total. This is always true unless specified otherwise.
- The concurrent system query limit for different query logic groups.
Regex matching against group names is supported. Pattern uniqueness is
enforced.
On a per-user basis:
- The user DN.
- The user's concurrent query limit. Overrides the system-wide
configuration.
- The user's concurrent query limit for different query logic groups.
Regex matching against group names is supported.
On a per-query-logic-group basis:
- The group name.
- The query logics included in the group. Regex matching is supported.
Pattern uniqueness is enforced.
- The default concurrent user query limit. This applies to the total
concurrent queries a user may run that originate from a query logic in
the group across all systems.
Given the possibilities for exact matches, partial regex matches, and
wildcard regex matches, the determination of the best limit to use for
any particular system, query logic, or query logic group is done by
sorting matches into the following 'matching buckets' (in best-match
priority):
1. Exact match: We attempt to find an exact match first and use the
associated limit.
2. Partial regex (non-wildcard-only): If we cannot find an exact match,
then we attempt to find all partial matches, and see if any of their
limits are met.
3. Wildcard-only regex: In the case of no exact or partial matches, we
use the wildcard match with the lowest limit.
## Implementation
Checking limits and marking as active/inactive is done through the
`QueryLimiter` class. The three main methods to know are:
- `QueryLimiter.checkForLimits()`
- `QueryLimiter.countQueryTowardsLimits()`
- `QueryLimiter.stopCountingQueryTowardsLimits()`
When a query is marked as active via
`QueryLimiter.countQueryTowardsLimits()`, it will delegate to the
ActiveQueryTracker, which will in turn create nodes in Zookeeper under
the `ActiveQueries` namespace. When `ActiveQueryTracker.trackQuery()` is
called, the following nodes are created:
```
# Container nodes
/users/<userDn>/<queryLogic> # Only for queries on systems that count towards user limit
/systems/<system>/<queryLogic>
/distinctQueryLogics/<queryLogic> [Only created if it does not exist.]
# Ephemeral nodes. These will auto-delete themselves if their associated Zookeeper connection ever goes down.
/users/<userDn>/<queryLogic>/<queryId> # Only for queries on systems that count towards user limit
/systems/<system>/<queryLogic>/<queryId>
```
`ActiveQueryTracker.trackQuery()` will return a `QueryHeartbeat` that
contain a list of `PersistentNode` (provided by the Apache Curator
library) wrappers around the ephemeral nodes listed above. The
`QueryHeartbeat` will maintain the connection to Zookeeper and attempt
to keep the ephemeral nodes present in Zookeeper until
`QueryHeartbeat.stop()` is called. If `QueryHeartbeat.stop()` is called,
or the webserver crashes, the ephemeral nodes will automatically be
deleted by Zookeeper.
The following error codes have been added:
```
412-20 - Concurrent query limit exceeded
500-164 - Error checking concurrent query limits
```
Closes#3100
---------
Co-authored-by: Bill Oley <[email protected]>
Co-authored-by: foster33 <[email protected]>
Co-authored-by: Ivan Bella <[email protected]>
replace impsort with spotless maven plugin to automatically cleanup
unused imports
formatting is fast and occurs early in the maven lifecycle
prevents checkstyle from failing later for unused imports
---------
Co-authored-by: Joe Alphonso <[email protected]>
Co-authored-by: alerman <[email protected]>
Accumulo changes from 2.1.4-5792fed3-bulkv2 (pre-2.1.4 release) to 2.1.4
reuire datawave code changes.
accumulo class ColumnFamily changed fields columnFamily,
columnQualifier, etc from public to private. Wes houldn't have been
using them anyway.
IteratorConfigUtil.loadIterators() declares throws
ReflectiveOperationException even though that exception is caught
internally and is thrown as a RuntimeException. Also, this declaration
is removed in accumulo 3.0.0.
The microservices can be upgraded to accumulo 2.1.4 after there is a
version of datawave that won't break at runtime while using accumulo
2.1.4. (illegal access)
Kevin Toth