Query Overview

The web service allows users to query the warehouse in several different ways. The Query api presents a single set of methods for starting, using, and closing a query. The web service hosts different types of query logic each with their own response type. Each query has the same lifecycle:

  1. User defines the query be calling either the Query/create or Query/define method. This returns a query id.
  2. User iterates over the results by repetitive calls to Query/{queryId}/next. Each call to next will return a page of results. The page size was set in step 1, or can be changed with a call to the update operation. The type of response is dependent on the query logic, and is listed on the query logic page.
  3. When the user receives a HTTP 204 response code (No Content), then there are no more results. The user should release the resources associated with their query by calling Query/{queryId}/close

JEXL Query Syntax

Query logic classes will use different types of syntax for the query string. Most of the queries that retrieve raw events will accept JEXL (default) or our modified LUCENE syntax. We support a subset of the language elements in the Apache Commons JEXL syntax and we have implemented some of our own custom functions. We support the following JEXL operators:

We have created the following functions, see the examples section below to see their usage:

Since JEXL is an expression language and not a text query language, we also added support for unfielded queries. For most of the query logic types that query raw events, we have provided the capability to search on any indexed field. To achieve this, the user should submit their query with the field name: _ANYFIELD_. See the examples below for its usage.

LUCENE Query Syntax

We have modified the LUCENE syntax such that the NOT operator is not unary, AND's are not fuzzy, and the implicit operator is the AND not the OR. Our LUCENE syntax has the following form: 

		ModClause ::= DisjQuery [NOT DisjQuery]*
		DisjQuery ::= ConjQuery [ OR ConjQuery ]*
		ConjQuery ::= Query [ AND Query ]*
		Query ::= Clause [ Clause ]*
		Clause ::= Term | [ ModClause ]
		Term ::=
			field:selector |
			field:selec* |
			field:selec*or |
			field:*lector |
			field:selec?or |
			selector | (can use wildcards)
			field:[begin TO end] |
			field:{begin TO end} |
			"quick brown dog" |
			"quick brown dog"~20 |
			#FUNCTION(ARG1, ARG2)

Note that to search for punctuation characters within a term, you need to escape it with a backslash.

We have also modified the LUCENE syntax to provide support for the JEXL syntax that LUCENE does not support. The table below maps the JEXL operators to the supported LUCENE syntax:
JEXL OperatorLUCENE Operator
filter:includeRegex(regex)#INCLUDE(regex)
filter:includeRegex(field, regex)#INCLUDE(field, regex)
filter:includeRegex(field1, regex1) <op> filter:includeRegex(field2, regex2) ...#INCLUDE(op, field1, regex1, field2, regex2 ...) where op is 'or' or 'and'
filter:excludeRegex(regex)#EXCLUDE(regex)
filter:excludeRegex(field, regex)#EXCLUDE(field, regex)
filter:excludeRegex(field1, regex1) <op> filter:excludeRegex(field2, regex2) ...#EXCLUDE(op, field1, regex1, field2, regex2 ...) where op is 'or' or 'and' (note: the op provided in the LUCENE query will be reversed for the JEXL intersection of the same query)
filter:isNull(field)#ISNULL(field)
not(filter:isNull(field))#ISNOTNULL(field)
filter:occurrence(field,operator,count))#OCCURRENCE(field,operator,count)
filter:timeFunction(field1,field2,operator,equivalence,goal)#TIME_FUNCTION(field1,field2,operator,equivalence,goal)
filter:text(field)#TEXT(field)
filter:text(field, value)#TEXT(field, value)
filter:text(field1, value1) <op> filter:text(field2, value2) ...#TEXT(op, field1, value1, field2, value2 ...) where op is 'or' or 'and'
filter:compare(field1, op, mode, field2)#COMPARE(field1, op, mode, field2) where op is '>', '>=', '<', '<=', '==', '=', or '!=' and mode is 'ALL' or 'ANY'

Notes:

  1. None of these filter functions can be applied against index-only fields except for filter:text (#TEXT).
  2. No indicies will be used for the INCLUDE and EXCLUDE functions, however they will be used for the TEXT function. All of these will be matched against the original unnormalized value.
  3. The occurrence function is used to count the number of instances of a field in the event. Valid operators are '==' (or '='),'>','>=','<','<=', and '!='.

Some geo functions are supplied as well that may prove useful although the within_bounding_box function may be done with a simple range comparison (i.e. LAT_LON_USER <= <lat1>_<lon1> and LAT_LON_USER >= <lat2>_<lon2>.

JEXL OperatorLUCENE Operator
geo:within_bounding_box(latLonField, lowerLeft, upperRight)#GEO(bounding_box, latLonField, 'lowerLeft', 'upperRight')
geo:within_bounding_box(lonField, latField, minLon, minLat, maxLon, maxLat)#GEO(bounding_box, lonField, latField, minLon, minLat, maxLon, maxLat)
geo:within_circle(latLonField, center, radius)#GEO(circle, latLonField, center, radius)

Notes:

  1. All lat and lon values are in decimal.
  2. The lowerLeft, upperRight, and center are of the form <lat>_<lon> and must be surrounded by single quotes.
  3. The radius is in decimal degrees as well.

New GeoWave geo functions are available as follows:

JEXL OperatorLUCENE Operator
geowave:intersects_bounding_box(geometryField, westLon, eastLon, southLat, northLat)#INTERSECTS_BOUNDING_BOX(geometryField, westLon, eastLon, southLat, northLat)
geowave:intersects_radius_km(geometryField, centerLon, centerLat, radiusKm)#INTERSECTS_RADIUS_KM(geometryField, centerLon, centerLat, radiusKm)
geowave:contains(geometryField, Well-Known Text)#CONTAINS(geometryField, centerLon, centerLat, radiusDegrees)
geowave:covers(geometryField, Well-Known Text)#COVERS(geometryField, Well-Known Text)
geowave:coveredBy(geometryField, Well-Known Text)#COVERED_BY(geometryField, Well-Known Text)
geowave:crosses(geometryField, Well-Known Text)#CROSSES(geometryField, Well-Known Text)
geowave:intersects(geometryField, Well-Known Text)#INTERSECTS(geometryField, Well-Known Text)
geowave:overlaps(geometryField, Well-Known Text)#OVERLAPS(geometryField, Well-Known Text)
geowave:within(geometryField, Well-Known Text)#WITHIN(geometryField, Well-Known Text)

Notes:

  1. All lat and lon values are in decimal degrees.
  2. The lowerLeft, upperRight, and center are of the form <lat>_<lon> and must be surrounded by single quotes.
  3. Geometry is represented according to the Open Geospatial Consortium standard for Well-Known Text. It is in decimal degrees longitude for x, amd latitude for y. For example, a point at New York can be represented as 'POINT (-74.01 40.71)' and a box at New York can be repesented as 'POLYGON(( -74.1 40.75, -74.1 40.69, -73.9 40.69, -73.9 40.75, -74.1 40.75));

An #EVALUATION_ONLY function has recently been introduced which is intended to be used to defer execution of part of a query to the evaluation phase. This could be used to defer a range to the evaluation phase, or possibly to defer GeoWave function evaluation to the evaluation phase, avoiding the need to generate geo indices when additional indexed fields are present (i.e. indexed fields other than the geo field). For example:

JEXL Example
INDEXED_FIELD == 'some value' && ((ASTEvaluationOnly = true) && geowave:intersects(geometryField, Well-Known Text))
LUCENE Example
INDEXED_FIELD: 'some value' AND #EVALUATION_ONLY('#INTERSECTS(geometryField, Well-Known Text)')

There are some additional functions that are supplied to handle dates more smoothly. It is intended that the need for these functions may go away in future versions (bolded parameters are literal, other parameters are substituted with appropriate values):
JEXL OperatorLUCENE Operator
filter:betweenDates(field, start date, end date)#DATE(field, start date, end date) or #DATE(field, between, start date, end date)
filter:betweenDates(field, start date, end date, start/end date format)#DATE(field, start date, end date, start/end date format) or #DATE(field, between, start date, end date, start/end date format)
filter:betweenDates(field, field date format, start date, end date, start/end date format)#DATE(field, field date format, start date, end date, start/end date format) or #DATE(field, between, field date format, start date, end date, start/end date format)
filter:afterDate(field, date)#DATE(field, after, date)
filter:afterDate(field, date, date format)#DATE(field, after, date, date format)
filter:afterDate(field, field date format, date, date format)#DATE(field, after, field date format, date, date format)
filter:beforeDate(field, date)#DATE(field, before, date)
filter:beforeDate(field, date, date format)#DATE(field, before, date, date format)
filter:beforeDate(field, field date format, date, date format)#DATE(field, before, field date format, date, date format)
filter:betweenLoadDates(LOAD_DATE, start date, end date)#LOADED(start date, end date) or #LOADED(between, start date, end date)
filter:betweenLoadDates(LOAD_DATE, start date, end date, start/end date format)#LOADED(start date, end date, start/end date format) or #LOADED(between, start date, end date, start/end date format)
filter:afterLoadDate(LOAD_DATE, date)#LOADED(after, date)
filter:afterLoadDate(LOAD_DATE, date, date format)#LOADED(after, date, date format)
filter:beforeLoadDate(LOAD_DATE, date)#LOADED(before, date)
filter:beforeLoadDate(LOAD_DATE, date, date format)#LOADED(before, date, date format)
filter:timeFunction(DOWNTIME, UPTIME, '-', '>', 2522880000000L)#TIME_FUNCTION(DOWNTIME, UPTIME, '-', '>', '2522880000000L')

Notes:

  1. None of these filter functions can be applied against index-only fields.
  2. Between functions are inclusive, and the other functions are exclusive of the entered dates.
  3. Date formats must be entered in the Java SimpleDateFormat object format.
  4. If the entered date format is not specified, then the following list of date formats will be tried:
  5. A special date format of 'e' can be supplied to mean milliseconds since epoch.

Unique Functions

In queries, unique functions are available to find unique results for fields that are specified, to the granularity of YEAR, MONTH, DAY, HOUR, MINUTE, SECOND, TENTH or MILLISECOND. The field passed to the function needs to be a dateTime field if using a time granularity, otherwise the original field will be returned as a unique result. Multiple granularity levels can be provided to the same field in multiple ways, and granularities will be merged across multiple functions if necessary. Potential function options that the user may use are listed below.

As referenced above, multiple granularity levels can be specified for the same field in different ways. The following queries are equivalent, and will find each unique result for field, to the hour and to the minute specified:

Excerpts Function

In queries where hits are found against phrases using the content:phrase, content:adjacent, and content:within functions, expanded versions of the phrases may be retrieved using the excerpts feature, either via the #excerpt_field query function, or the excerpt.fields query parameter. The field passed to the function must correspond to a field in which a phrase was targeted, and the offset indicates the number of additional words the phrase will be expanded by at the beginning and end of the phrase. The expanded phrases will be available within the HIT_EXCERPT attribute of returned documents.