Make the following changes to improve the thread-safety of
QueryUncaughtExceptionHandler, and add test coverage.
- Generate a sequence for each call to uncaughtException() to help
ensure the first call with a non-null Throwable wins the race,
regardless of which thread's CAS operation completes first.
- In uncaughtException(), atomically update the sequence number, thread,
and throwable at the same time.
- Use an efficient, thread-safe collection for storing and retrieving
messages.
---------
Co-authored-by: Moriarty <[email protected]>
update accumulo-utils to use a POJO for storing markings and then
updated the rest of the codebase that referenced it.
replaced some of the ColumnVisibility usage with AccessExpression to
improve performance.
---------
Co-authored-by: Joe Alphonso <[email protected]>
- remove UserAuthorizationsTest from web-services/client module. This
was the only test using PowerMock.
- the test was duplicated from core/base-rest-responses. The test in
base-rest-responses uses proper reflection without PowerMock.
## Summary
Replace deprecated `Date(String)` constructor with `Date(long)`.
The `Date(String)` constructor has been deprecated since JDK 1.1.
## Changes
```java
// Before
@SuppressWarnings("deprecation")
private Date parseDateCheck = new Date("Sun Aug 7 00:00:00 GMT 2016");
// After
// Aug 7, 2016 00:00:00 GMT (same as accumuloDate)
private Date parseDateCheck = new Date(1470528000000L);
```
## Files Changed
- `QueryParametersTest.java` - 1 change
Fixes#3300
Part of #2443
* Adds KeywordUUIDQuery that pairs id lookup with keyword extraction.
* Creates a chained query logic that first does lookup uuid, followed by document keyword extraction.
* Adds the ability to generate a word cloud across all keyword results, currently stored in messages portion of the response.
* Adds Datawave TagCloudResponse and supporting classes.
* Adds two endpoints to /DataWave/Query - /extractKeywords and /generateTagCloud - each take a POST request and return keywords from individual documents or consolidated tag coluds, respectively.
* Adds response type for /generateTagCloud, the TagCloudResponseBase and accompanying data classes.
* Updates chained query and improves serialization of objects returned from iterators.
* Addressed formatting, review comments, and completed bugfix.
* Formatting diffs
* formatter and import cleanup
* Pulled utils into main versioning scheme
* Changed the package from gov.nsa.datawave.micreoservice to gov.nsa.datawave.core
* Pulled in the following modules:
1. type-utils
2. accumulo-utils
3. metadata-utils
4. common-utils
5. base-rest-responses
* Update version
* consistent accumulo-utils version
* Changed core packaged to gov.nsa.datawave.core and to the latest
versions
* GitHub Actions: Fix Formatting
* pom formatting
* Spotbugs fix
* Updated re spotbugs
* fixed package names for util packages
* Updated package
* Explicit metadata-utils version
* GitHub Actions: Fix Formatting
* formatting
* Updated everything to the same protobuf version that we deploy.
Regenerated protobufs with the same version as well.
* Advance pom version of dictionary service to next snapshot
* a little test debug
* GitHub Actions: Fix Formatting
* Merge conflicts
* Remove invalid javadoc elements
* Updated to next snapshot version
* merge conflict fix
---------
Co-authored-by: GitHub Actions <[email protected]>
Co-authored-by: Dylan <[email protected]>
* Converting strs to chars on lastIndexOf
* Removing test annotation
* Disabled for Junit5 has replaced ignore
* Formatting for MetadataHelper
---------
Co-authored-by: Adam Lerman <[email protected]>
* Implement query validation endpoint
Implement the query/{logicName}/validate endpoint. This feature supports
the ability to configure validation rules that will validate LUCENE and
JEXL queries against a number of criteria and provide meaningful
feedback to customers.
Closes#2585
---------
Co-authored-by: Seth Smucker <[email protected]>
Co-authored-by: Ivan Bella <[email protected]>
* Modified a couple of things and fixed a few tests with the integration merge.
* pulling in latest audit changes
* pulled in latest audit updates
* Fixed metrics version in docker compose
* Updated accumulo utils and datawave starter versions.
* Updated the audit api and audit starter versions
* Updated authorization api version
* Updated accumulo api version.
* Updated config service with latest changes
* updated dictionary and accumulo utils version
* Updated hazelcast version
* bumped query metric api version
* Updated pki for query microservices docker deployment.
* Fixing the docker compose bootstrap to avoid duplicate extra hosts issue.
* adding latest datawave changes
* updated metadata utils
* updated docker compose to use the latest version of each service.
* Added a cached results spring boot starter and updated associated classes accordingly.
* fixing spotbugs error
* fixed install-ingest.sh for datawave.
* Updates to get bulk results map reduce query working again.
* formatting
* Updates per a race condition discovered in the kafka variant of the
query executor tests.
* Update README.md
* accumulo service merge from main
* fixed a test that broke after merging integration
* Added logic from Datawave Principal Login Module to the datawave spring boot starter.
* Updated datawave to 5.4.0-SNAPSHOT
* Updated ProxiedUserDetails to use dynamic type for newInstance method.
* updated audit service to stop intermittent test failures.
* Updated the query results publisher and setup the kafka manager to create topics manually.
* Added github actions for some submodules
* Updated github actions to avoid running out of disk space.
* Updated import sorting for microservices using impsort plugin.
* fixing merge conflicts
* updated metadata utils
* Moved Query, QueryImpl to query-api
* Removed usage of zookeeper stringutils
* added package-info.java for Query, QueryImpl
* updated modification service logic to use ProxiedUserDetails.
* Removed unnecessary jackson annotations to address enunciate issues
* Updated profile ids for submodules
* changes to get the github actions working again
* attempting to fix github actions
* typo
* udpated kafka config
* removed banned dependency
* bumped release version
* formatting
* fixing docker compose kafka setup
* updated deploy poms to work with the latest version of maven
* bumped version for some modules
* bumped microservice service parent version
* Updated with latest changes from main/integration
* fixed the composite query logic for the query microservices
* updated submodules
* Added v1 to the model path
* For this branch we need to checkout all submodules when building tests.
* Added alerman's run cache settings for docker
* Updated github actions to build query microservices correctly
* fixed build cache config typo
* Updated query service to use pool header, and allow for admin override via parameter
* Updated github ci to make sure it runs against the queryMicroservices branch
* Updated lookup uuid to accept multiple terms per field
* Updated confirm ack configuration for query metric service, and added configuration to make rest metrics updates possible
* Added support for long running queries
* Added configcheck command line utility
* updated next call logic for long running queries
* updated query metric service and starter submodules
* Added some plugins to the config check service
* Updated configcheck tests to trim whitespace
* fixing unit tests for config check
* Changed from max long running query timeouts to long running query
timeout
* Updated the query starter submodule
* updated submodules
* fixed composite query logic test
* Refactored query scripts to use common code
* Updated quickstart docker-maven-plugin to not cleanup old tagged releases
* updated common-utils submodule
* Updated LookupService to behave the same as the webservice
* Fixed TypedValue (de)serialization, and added a result post processor to handle things like counts
* reset docker-compose quickstart to just be accumulo, not web
* Changed how the result postprocessor runs to support result aggregation for long running queries
* Renamed aggregateResults to reduceResults
* Added junit-jupiter engine to datawave core pom
* Updated query logic factory to add logic map
* Updated EdgeQueryLogicFactory configuration
* Removed unused RemoteQueryLogic and cleaned up RemoteEventQueryLogic class definition:
* Implemented authorization and query federation for the query microservices
* Fixed copy constructor parameter assignment in QueryData
* Formatting job fix
* Fixing build/formatter issues for latest changes pulled in from integration
* Fixed SSDeepIngestQueryTest
* Updated usage of remote user operations for query microservices
* Fixed QueryPlan constructor
* Moved filtering and delegating query logic functionality into datawave-core-query so it can be used by the microservices
* formatting
* Updated submodules
* Updated package names for commons.lang3 classes due to type-utils fix
* Moved the AuthorizationsPredicate class to authorization-api
* Removed hadoop dependencies from pom
* Updated edge configuration to reset param variables before loading new values
* Excluded log4j from our maven plugins
* Updated in-memory-accumulo javadoc plugin to exclude log4j
* Updated microservice-parent maven-dependency-plugin to exclude log4j
* Removed infinispan from query starter
* Remove validation call from LoadModel
* fixed formatting
* Updated datawave ingest jar libs
* QueryMetricQueryLogic must copy runAsMetricsAdministrator setting in copy constructor
* Revert query metric query logic back to it's original state
* pr feedback
* PR feedback
* PR Feedback
* PR feedback
* PR Feedback
* PR feedback
* PR feedback
* PR Feedback
* PR Feedback
* PR feedback
* PR feedback
* PR feedback
* pr feedback
* PR feedback
* pr feedback
* pr feedback
* pr feedback
* pr feedback
* pr feedback
* pr feedback
* PR feedback
* PR feedback
* pr feedback
* pr feedback
* Updated in-memory-accumulo commit
* in-memory-accumulo
* formatting
* query starter
* pr feedback
* pr feedback
* pr feedback
* pr feedback
* pr feedback
* pr feedback
* pr feedback
* Remoted CachedQueryStatus mechanism
* pr feedback
* pr feedback
* pr feedback
* pr feedback
* pr updates
* pr feedback
* pr review: found duplicate properties between configurations and the
super GenericQueryConfiguration class. Removed redundant copies in
ShardQueryConfiguration checkpoint copy constructor
* review comments
* review comments
* Reverted path changes in dictionary service
* pr review updates
* updated executor to use QueryStatusUpdateUtil
* updated query microservice submodule
* make github action tests match integration
---------
Co-authored-by: Ivan Bella <[email protected]>
Co-authored-by: GitHub Actions <[email protected]>
Co-authored-by: Bill Oley <[email protected]>
* lenient is now the default for fields
* added a strict query property marker
* Refactor JexlNodeFactory.createNodeTreeFromPairs to live in QueryModelVisitor
* If a subset of the model forward mappings cannot be normalized
(regex only), then they will be marked as ignored.
* Removed some redundant calls in the query model visitor
* Fixed some bad logic relative to applying model to methods
and ensuring noExpansion fields are appropriately applied.
* Added IgnoredExpression property marker
* Updated to use the lenient model mappings in the Query Model
* When expanding query model, mappings that were derived from a
lenient model mapping are marked with a LenientExpression marker
* Refacted a few classes into the metadata-utils project
* Added additional functions, and worked through no expansion
test cases (fixes required due to refactoring)
* added lenient and scrict query parameters
* updated query model visitor to look at configured lenient/scrict
* updated normalization expansion visitor as well
* Updated to apply lenient and strict modes to all normalizations
(not just regex)
* Renamed IgnoredExpression to DroppedExpression
(overloaded "ignore" concept with RangeStream)
* Added a reason assignment to the DroppedExpression
* Updated to drop lenient markers after expanding normalizers
* Handle the case where all fields fail all normalizations
* Updated intepreter to not evaluation assignments
* Updated query model visitor to avoid dropping methods/funcs
* Updated tests to appropriately evaluation results.
* Updating formatter version and formatting all code
* Re: #1986
- restores alerman's original javadoc formatting
- captures additional files only included by certain build profiles
* Updating to 2.18.0
* Updating formatting after conflict fixing
---------
Co-authored-by: keith-ratcliffe <[email protected]>
* re #1821: Added a remote user operations implementation
* Updated the authorization mechanisms to validate against the overall
user (which may include the remote authorizations) and then the reduce
by the query authorizations. This allows the auths returned by the list
effective authorizations which include remote auths to be used for local
query logics as well.
* Added optional flag to the listEffectiveAuthorizations avoid federation
* Updated Query to now know about a remote user operations
* logics will use the remote user operation if set for query auths
* Composite Query Logic will recalculate query auths for each logic
* Added test cases for new classes and integration tests
* Updated test cases to handle new query logic getUserOperations call
* First draft of javadoc annotation updates
* More docs updated
* Fixing issue with param types, formatting
* Fixing error with abstract builder docs
* Moving through a few more annotations
* Updating more notes, fixing some existing
* Fixing more annotations
* More work on docs
* More docs more docs
* Adjusting some annotations in config/hash/normalizer packages
* More docs, more docs
* Adding docs for tokenize, reader
* Adjusting formatting, adding some updates
* Updating more docs, in ingest-core
* Fixing a rogue return
* Fixing docs
* Working through extended content classes
* Fixing docs in IngestJob, DelegatingPartitioner
* Finishing up IngestJob
Co-authored-by: pcagbu <[email protected]>
Co-authored-by: Ivan Bella <[email protected]>
Co-authored-by: Moriarty <[email protected]>
Drew Farris