toString() built its mini-maps into instance fields, but the metadata
helper cache hands the same TypeMetadata to every concurrent query with
matching auths, so the first serializations after a cache refresh raced
on unsynchronized TreeMaps. They are now built locally and published
once.
Where type metadata is missing or malformed, fail instead of degrading:
folding away an absent ingest type, or parsing a corrupt mini-map, would
normalize values against the wrong types and silently drop valid
results. An empty instance also serialized to an unterminated string the
tablet servers could not parse.
Planning waits could only iterate on a null result and then spun on a
completed future forever, so they now fail fast, and the iterator
setting poll blocks on its future rather than busy-waiting on isDone().
---------
Co-authored-by: Claude Opus 5 <[email protected]>
Make the following changes to improve the thread-safety of
QueryUncaughtExceptionHandler, and add test coverage.
- Generate a sequence for each call to uncaughtException() to help
ensure the first call with a non-null Throwable wins the race,
regardless of which thread's CAS operation completes first.
- In uncaughtException(), atomically update the sequence number, thread,
and throwable at the same time.
- Use an efficient, thread-safe collection for storing and retrieving
messages.
---------
Co-authored-by: Moriarty <[email protected]>
The test compiled roughly a quarter million patterns from an unseeded
source, so a failure could not be reproduced. A fixed seed, overridable
with datawave.test.seed, now covers ten thousand cases
deterministically.
Co-authored-by: Claude Opus 5 <[email protected]>
datawave.data.ColumnFamilyConstants is deprecated for removal; switch
all consumers to datawave.table.constants.MetadataColumnFamilyConstants
and add datawave-core-tables as a dependency where it wasn't already on
the classpath.
---------
Co-authored-by: Claude Sonnet 5 <[email protected]>
Migrate most instances of Scanner to using a ScannerBuilder. Another
brick in the road to native scan server support.
---------
Co-authored-by: Claude Sonnet 4.6 <[email protected]>
Co-authored-by: Laura Schanno <[email protected]>
Co-authored-by: alerman <[email protected]>
## Summary
- `toString()`/`fromString()` dominated profiler traces
(`HashMap.put`/`calculateHash`) from rebuilding index-to-name lookup
maps per field/value pair; fixes that hot path.
- Adds native Kryo support: `TypeMetadata` implements
`KryoSerializable`, and `TypeMetadataSerializer` wraps it into a Base64
string round trip, reusing its `Kryo`/buffer per instance (matching the
`CandidateResultSerializer` convention) since it's called many times per
thread.
- The Kryo form uses the same mini-map convention as `toString()`: an
ingest type dictionary, a normalizer type dictionary, then one entry per
field name holding its `ingestTypeIndex:normalizerTypeIndex` pairs as
variable length ints. Each name is written exactly once, and `read`
holds both dictionaries as arrays so resolving a pair is an array index
rather than a map rebuild, keeping deserialization O(1) per entry.
- Wires it through a new `kryoTypeMetadata` flag on
`ShardQueryConfiguration`: `DefaultQueryPlanner` serializes via Kryo
when enabled, marking the iterator option so `QueryOptions` decodes it
correctly. The flag is skipped during per-shard type metadata reduction,
since `VisitorFunction` re-serializes via `toString()` afterward and
doesn't understand the Kryo format.
- A parameterized performance test (1/5/20 datatypes x 25/250/2500
fields) shows the Kryo round trip is consistently 1.7-3.0x faster.
## Payload size
The mini-map convention is what keeps the payload smaller than the
string form it replaces. An earlier revision of this branch wrote the
full ingest type and normalizer class name for every field/normalizer
pair, which made it **larger** than `toString()` — measured against the
same fixture, with fully qualified normalizer names:
| datatypes x fields | `toString()` | Kryo, repeated names | Kryo,
mini-map |
| --- | --- | --- | --- |
| 1 x 25 | 526 | 982 (1.87x) | 401 (0.76x) |
| 1 x 2500 | 41,551 | 101,402 (2.44x) | 31,527 (0.76x) |
| 5 x 250 | 8,099 | 49,526 (6.12x) | 5,063 (0.63x) |
| 5 x 2500 | 81,599 | 507,006 (6.21x) | 51,563 (0.63x) |
| 20 x 250 | 25,799 | 198,111 (7.68x) | 12,708 (0.49x) |
| 20 x 2500 | 256,799 | 2,028,031 (7.90x) | 126,708 (0.49x) |
The Kryo form is now 0.49-0.76x of the string form, i.e. 2.5-16x smaller
than writing the names repeatedly.
Two caveats worth noting for review:
- Building the dictionaries and inverting to field-major order is
write-side work the repeated-name layout skipped, so the round trip
speedup drops from 3.1-5.5x to 1.7-3.0x. Both figures are measured on
the same fixture, so this is a real trade: roughly half the speed
advantage in exchange for a payload that is 4-16x smaller and no longer
larger than what it replaces.
- `TypeMetadataSerializer` Base64-encodes on the way out, which adds a
third back. The transported string is therefore at rough parity with
`toString()` for a single ingest type (1.00x) and only pulls ahead from
two ingest types on (0.66-0.94x). The raw Kryo form is smaller in every
combination; Base64 is what consumes the margin at the low end. Both
properties are asserted separately rather than averaged together.
## Test plan
- [x] `metadata-utils` module test suite (379 tests) passes, via
`install` so the enforcer rules run
- [x] Targeted `query-core` suites (`QueryOptionsTest`,
`DefaultQueryPlannerTest`, `DocumentIteratorOptionsTest`,
`VisitorFunctionTest`, `TypeMetadataTest`) pass
- [x] `ColorsTest` exercises the Kryo path end to end through the
planner and iterator
- [x] `TypeMetadataSerializerTest` covers round trip correctness,
instance reuse across differently-sized inputs, and the two payload size
properties above
- [x] `TypeMetadataSerializationPerformanceTest` reports throughput and
payload size across the matrix
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Sonnet 5 <[email protected]>
Covers the unsynchronized read and broken double-checked lock in
QueryUncaughtExceptionHandler, the handler that ScannerSession shares
across every session built from one Query, and the null-executor
handling in BatchScannerSession. The assertions capture current behavior
so the fix has to update them.
Co-authored-by: Claude Opus 4.8 <[email protected]>
- RangeStreamScanners now created via builders
- RangeStream constructors that uses scanner factor are deprecated and
marked for removal, including sub classes
- ScannerFactory methods for creating RangeStreamScanners deprecated and
marked for removal.
- DefaultQueryPlanner no longer uses deprecated RangeStream constructor
---------
Co-authored-by: Mr. Patrickus Agbius <[email protected]>
update accumulo-utils to use a POJO for storing markings and then
updated the rest of the codebase that referenced it.
replaced some of the ColumnVisibility usage with AccessExpression to
improve performance.
---------
Co-authored-by: Joe Alphonso <[email protected]>
- ScanBuilders now support a set of authorizations,
- ScannerHelper is used to set multiple visibility filters
- delegate scanners are used to enforce visibility filters
- ModelKeyParser uses Clock.millis() instead of System.currentTimeMillis
- ModelKeyParserTest migrated from JUnit4 to JUnit5
- ModelKeyParserTest migrated from PowerMock andEasyMock to Mockito
- ModelKeyParser uses Clock.millis() instead of System.currentTimeMillis
- ModelKeyParserTest migrated from JUnit4 to JUnit5
- ModelKeyParserTest migrated from PowerMock andEasyMock to Mockito
## Configuration
Add the ability to enforce concurrent query limits across a group of
webservers. Zookeeper is used to track active queries and the following
data:
- The query ID
- The user who submitted the query
- The system the query was submitted on
- The query logic the query originated from
Query limit enforcement is done through the `QueryLimiter` class. Given
a user, system, and query logic, it can determine if any of the
following limits have been exceeded:
- The max allowed concurrent queries for the user.
- The max allowed concurrent queries of the query logic for the user.
- The max allowed concurrent queries for the system.
- The max allowed concurrent queries of the query logic for the system.
Limits may be defined and customized on a per-user and per-system basis.
They may also be defined for groups of query logics. The classes
`UserLimitProvider`, `SystemLimitProvider`, and
`QueryLogicGroupLimitProvider` are respectively responsible for
identifying the best limits to enforce for a user, system, and query
logic. They will be initialized in the `QueryLimiter` after providing a
`QueryLimitConfiguration` instance. The following can be configured:
On a system-wide basis:
- The default concurrent user query limit. This applies to the total
number of queries a user may run across all systems. May be overridden
per user.
- The default concurrent system query limit. Primarily to avoid a system
getting overloaded. May be overridden per system.
On a per-system basis:
- The system name/ids the configuration targets. Regex matching is
supported. Pattern uniqueness is enforced.
- The concurrent system query limit. Overrides the system-wide value.
- Whether queries submitted to the system count towards a user's
concurrent query total. This is always true unless specified otherwise.
- The concurrent system query limit for different query logic groups.
Regex matching against group names is supported. Pattern uniqueness is
enforced.
On a per-user basis:
- The user DN.
- The user's concurrent query limit. Overrides the system-wide
configuration.
- The user's concurrent query limit for different query logic groups.
Regex matching against group names is supported.
On a per-query-logic-group basis:
- The group name.
- The query logics included in the group. Regex matching is supported.
Pattern uniqueness is enforced.
- The default concurrent user query limit. This applies to the total
concurrent queries a user may run that originate from a query logic in
the group across all systems.
Given the possibilities for exact matches, partial regex matches, and
wildcard regex matches, the determination of the best limit to use for
any particular system, query logic, or query logic group is done by
sorting matches into the following 'matching buckets' (in best-match
priority):
1. Exact match: We attempt to find an exact match first and use the
associated limit.
2. Partial regex (non-wildcard-only): If we cannot find an exact match,
then we attempt to find all partial matches, and see if any of their
limits are met.
3. Wildcard-only regex: In the case of no exact or partial matches, we
use the wildcard match with the lowest limit.
## Implementation
Checking limits and marking as active/inactive is done through the
`QueryLimiter` class. The three main methods to know are:
- `QueryLimiter.checkForLimits()`
- `QueryLimiter.countQueryTowardsLimits()`
- `QueryLimiter.stopCountingQueryTowardsLimits()`
When a query is marked as active via
`QueryLimiter.countQueryTowardsLimits()`, it will delegate to the
ActiveQueryTracker, which will in turn create nodes in Zookeeper under
the `ActiveQueries` namespace. When `ActiveQueryTracker.trackQuery()` is
called, the following nodes are created:
```
# Container nodes
/users/<userDn>/<queryLogic> # Only for queries on systems that count towards user limit
/systems/<system>/<queryLogic>
/distinctQueryLogics/<queryLogic> [Only created if it does not exist.]
# Ephemeral nodes. These will auto-delete themselves if their associated Zookeeper connection ever goes down.
/users/<userDn>/<queryLogic>/<queryId> # Only for queries on systems that count towards user limit
/systems/<system>/<queryLogic>/<queryId>
```
`ActiveQueryTracker.trackQuery()` will return a `QueryHeartbeat` that
contain a list of `PersistentNode` (provided by the Apache Curator
library) wrappers around the ephemeral nodes listed above. The
`QueryHeartbeat` will maintain the connection to Zookeeper and attempt
to keep the ephemeral nodes present in Zookeeper until
`QueryHeartbeat.stop()` is called. If `QueryHeartbeat.stop()` is called,
or the webserver crashes, the ephemeral nodes will automatically be
deleted by Zookeeper.
The following error codes have been added:
```
412-20 - Concurrent query limit exceeded
500-164 - Error checking concurrent query limits
```
Closes#3100
---------
Co-authored-by: Bill Oley <[email protected]>
Co-authored-by: foster33 <[email protected]>
Co-authored-by: Ivan Bella <[email protected]>
replace impsort with spotless maven plugin to automatically cleanup
unused imports
formatting is fast and occurs early in the maven lifecycle
prevents checkstyle from failing later for unused imports
---------
Co-authored-by: Joe Alphonso <[email protected]>
Co-authored-by: alerman <[email protected]>
Updates FieldExistenceRule to also verify that there is data in a field
for the given date range.
In the case of an OR node, ALL fields must be missing from the date
range for a failure message to be returned.
---------
Co-authored-by: Laura Schanno <[email protected]>
Kevin Toth