Compare commits

..
Author SHA1 Message Date
Bill Oley 16d73a1845 Release version 2.1.5 2024-04-15 10:54:17 -04:00
Bill Oley 42b06cf60c Prevent maven plugins from using log4j:log4j 2024-04-15 10:54:11 -04:00
Bill Oley e772728159 Correctly handle load, dwquery, hadoop, accumulo2, metrics in CollectDPluginParent 2024-04-10 16:25:45 -04:00
Bill Oley ebce93684a Save FileSystem instead of constantly re-creating, ensure CsvReader closed 2024-04-10 16:25:44 -04:00
Bill Oley ff7a75c7c4 Fix extraction of native libraries in scripts, make indentation consistent 2024-04-10 16:25:38 -04:00
Bill Oley ad61f2cc85 Set version to 2.1.5-SNAPSHOT 2024-03-29 16:52:06 -04:00
Bill Oley e8beb77cb9 Release version 2.1.4 2024-03-29 16:50:04 -04:00
Bill Oley d6839fbc6c Set client to null on exception, don't try to return it to pool 2024-03-29 16:46:21 -04:00
Bill Oley d53079a996 Allow users to specify an interval to calculate rates over, default downsample interval if set or 1 second otherwise 2024-03-29 13:23:03 -04:00
Bill Oley 2e7489afef Modifications to CollectDPluginParent to support dwquery metrics and to correctly identify rack for a host 2024-03-29 12:24:12 -04:00
Bill Oley ee71a2c74f Improve scripts to extract native libraries only when they are not present 2024-03-29 12:23:23 -04:00
Bill Oley 89a374f9a2 Fix util scripts and use BIN_DIR instead of THIS_DIR 2024-03-21 10:25:21 -04:00
Bill Oley f145d0d1a4 Set version to 2.1.4-SNAPSHOT 2024-03-13 15:41:00 -04:00
Bill Oley 387140722c Release version 2.1.3 2024-03-13 15:39:21 -04:00
Bill Oley b3617820f7 Modifications to CollectDPluginParent to support period delimited metric and tag.value pairs of micrometer metrics ETSY format 2024-03-13 15:32:57 -04:00
Bill Oley ac2d9463dc Set version to 2.1.3-SNAPSHOT 2024-03-07 13:14:13 -05:00
Bill Oley c32867685f Release version 2.1.2 2024-03-07 13:14:13 -05:00
Bill Oley 08057e7d67 Add jcl-over-slf4j dependency for collectd-nsq-plugin to supply logging via commons-logging and slf4j 2024-03-07 13:14:12 -05:00
Bill Oley a058e5a816 Use only netty dependencies that are necessary, ensure extraction of netty-tcnative-boringssl-static and netty-transport-native-epoll (for dynamic linking) for linux OS 2024-03-06 17:49:40 -05:00
Bill Oley 99b81986a4 Remove requiredAuths, requiredRoles from balancer ymland env (not used) and log information about the RemoteAuthenticationProvider when created 2024-03-06 17:06:51 -05:00
Bill Oley d7ecbd345f Maintain auths and roles in sorted order in TimelyUser 2024-03-06 17:04:29 -05:00
Bill Oley 0f9059b6b6 Default ciphers to null in SslClientProperties 2024-03-06 17:03:46 -05:00
Bill Oley fcbdd9296b Set version to 2.1.2-SNAPSHOT 2024-03-01 12:21:26 -05:00
Bill Oley df3ade7929 Release version 2.1.1 2024-03-01 12:19:42 -05:00
Bill Oley 823a6fa08f Ensure that WebClient.Builder has client certificate before creating RemoteAuthenticationProvider, add logging 2024-03-01 12:15:18 -05:00
Bill Oley 89ab2635fd Log format change, async logger setting, remove create-timely-profiles.sh 2024-02-29 14:59:07 -05:00
Bill Oley 13d8db5ab7 Set version to 2.1.1-SNAPSHOT 2024-02-27 14:29:46 -05:00
Bill Oley 8db5358b50 Release version 2.1.0 2024-02-27 14:29:45 -05:00
Bill Oley 3e376e106b Update dependencies spring-boot:2.7.18, spring-cloud:3.1.8, spring:5.3.31, netty:4.1.106.Final, google-flatbuffers:23.5.26, classgraph:4.8.165, spotbugs:4.8.3.0, and others 2024-02-27 14:29:44 -05:00
Bill Oley 18eecfd7b0 Refactor services to better use spring boot configuration, service, tests, use Java 11, set version to 2.1.0-SNAPSHOT 2024-02-27 14:29:32 -05:00
Bill Oley 065d8a9a9f Use either deprecated or newer property for context classloader in TieredCompactionStrategy 2024-02-02 16:44:41 -05:00
Bill Oley 09c2ef7e11 Bump version to 2.0.3-SNAPSHOT 2024-01-31 12:04:08 -05:00
Bill Oley cb875819ff Release version 2.0.2 2024-01-31 11:49:57 -05:00
Bill Oley fcc5f0f716 Fix OneWaySSL tests by creating a PKCS12 file from SelfSignedCertificate 2024-01-29 17:33:13 -05:00
Bill Oley c87543f7d7 Code formatting via autoformat profile 2024-01-29 16:28:06 -05:00
Bill Oley d7f5a3b20b Add codestyle config, Pom changes to support code formatting, import sort, pom sort 2024-01-29 16:24:24 -05:00
Bill Oley 9601cdc927 curator version to 5.5.0 2024-01-29 16:23:27 -05:00
Bill Oley 60c2588c79 Bump version to 2.0.2-SNAPSHOT 2024-01-26 13:12:18 -05:00
Bill Oley f3275c6a34 Release version 2.0.1 2024-01-26 13:10:20 -05:00
Bill Oley 7486b6b761 Add RemoteUserDetailsService, fix MiniAccumulo test, simplify SSL configuration and allow more types of SSL keyStores and trustStores 2024-01-26 13:07:26 -05:00
Bill Oley b1a968590c Bump version to 2.0.1-SNAPSHOT 2024-01-17 10:42:09 -05:00
Bill Oley 8f86b20c02 Release version 2.0.0 2024-01-17 10:42:09 -05:00
Bill Oley a3620a3f24 Fixes for intermittent test failures, version to 2.0.0-SNAPSHOT 2024-01-17 10:42:09 -05:00
Bill Oley 9c99e74958 Use InMemoryAccumulo instead of MiniAccumulo, test fixes 2024-01-17 10:42:09 -05:00
Bill Oley 91ec36e819 Changes to support accumulo2 2024-01-17 10:42:08 -05:00
Bill Oley 9ad1b18b74 Delete grafana-auth module 2024-01-17 10:42:08 -05:00
Bill Oley 82c45d4dd4 Bump version to 1.3.7-SNAPSHOT 2023-10-13 10:28:19 -04:00
Bill Oley a0e47d0b48 Release version 1.3.6 2023-10-13 10:25:43 -04:00
dependabot[bot] d3dc324d84 Bump golang.org/x/net from 0.7.0 to 0.17.0 in /grafana/timely-app (#246)
Bumps [golang.org/x/net](https://github.com/golang/net) from 0.7.0 to 0.17.0.
- [Commits](https://github.com/golang/net/compare/v0.7.0...v0.17.0)

---
updated-dependencies:
- dependency-name: golang.org/x/net
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-10-13 10:18:12 -04:00
Bill Oley 4871b3a4a9 Update dependencies, protect against NPE in AuthCache, AuthorizationsUtil 2023-10-13 10:02:33 -04:00
dependabot[bot] 8d5d5eeca8 Bump terser from 4.8.0 to 4.8.1 in /grafana/timely-app (#235)
Bumps [terser](https://github.com/terser/terser) from 4.8.0 to 4.8.1.
- [Release notes](https://github.com/terser/terser/releases)
- [Changelog](https://github.com/terser/terser/blob/master/CHANGELOG.md)
- [Commits](https://github.com/terser/terser/commits)

---
updated-dependencies:
- dependency-name: terser
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-10-11 14:26:51 -04:00
dependabot[bot] a006a44372 Bump decode-uri-component from 0.2.0 to 0.2.2 in /grafana/timely-app (#240)
Bumps [decode-uri-component](https://github.com/SamVerschueren/decode-uri-component) from 0.2.0 to 0.2.2.
- [Release notes](https://github.com/SamVerschueren/decode-uri-component/releases)
- [Commits](https://github.com/SamVerschueren/decode-uri-component/compare/v0.2.0...v0.2.2)

---
updated-dependencies:
- dependency-name: decode-uri-component
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-10-11 14:26:07 -04:00
dependabot[bot] 6cfcdc1030 Bump golang.org/x/net in /grafana/timely-app (#241)
Bumps [golang.org/x/net](https://github.com/golang/net) from 0.0.0-20210614182718-04defd469f4e to 0.7.0.
- [Release notes](https://github.com/golang/net/releases)
- [Commits](https://github.com/golang/net/commits/v0.7.0)

---
updated-dependencies:
- dependency-name: golang.org/x/net
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-10-11 14:25:46 -04:00
dependabot[bot] 4eb7ceff8c Bump semver from 5.7.1 to 5.7.2 in /grafana/timely-app (#244)
Bumps [semver](https://github.com/npm/node-semver) from 5.7.1 to 5.7.2.
- [Release notes](https://github.com/npm/node-semver/releases)
- [Changelog](https://github.com/npm/node-semver/blob/v5.7.2/CHANGELOG.md)
- [Commits](https://github.com/npm/node-semver/compare/v5.7.1...v5.7.2)

---
updated-dependencies:
- dependency-name: semver
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-10-11 14:25:13 -04:00
dependabot[bot] 6ad8da2b99 Bump word-wrap from 1.2.3 to 1.2.4 in /grafana/timely-app (#245)
Bumps [word-wrap](https://github.com/jonschlinkert/word-wrap) from 1.2.3 to 1.2.4.
- [Release notes](https://github.com/jonschlinkert/word-wrap/releases)
- [Commits](https://github.com/jonschlinkert/word-wrap/compare/1.2.3...1.2.4)

---
updated-dependencies:
- dependency-name: word-wrap
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-10-11 14:24:12 -04:00
dependabot[bot] 7f56a36906 Bump guava from 29.0-jre to 32.0.0-jre (#243)
Bumps [guava](https://github.com/google/guava) from 29.0-jre to 32.0.0-jre.
- [Release notes](https://github.com/google/guava/releases)
- [Commits](https://github.com/google/guava/commits)

---
updated-dependencies:
- dependency-name: com.google.guava:guava
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-06-15 12:53:29 -04:00
Christopher Tubbs 1443fdceb9 Bump formatter and impsort plugins for newer Maven 2023-06-07 11:01:29 -04:00
Christopher Tubbs 3cff14eb03 Bump NPM/Yarn deps
* This avoids warnings about CVEs from GitHub in the referenced
  dependencies
* This required relaxing the node version constraint, since I didn't
  have version 12, only 16
2022-04-11 15:49:13 -04:00
dependabot[bot] 8fa5b56e62 Bump hadoop-common from 2.7.7 to 2.10.1 (#230)
Bumps hadoop-common from 2.7.7 to 2.10.1.

---
updated-dependencies:
- dependency-name: org.apache.hadoop:hadoop-common
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-04-11 14:22:59 -04:00
dependabot[bot] b2f2cef834 Bump follow-redirects from 1.14.1 to 1.14.8 in /grafana/timely-app (#232)
Bumps [follow-redirects](https://github.com/follow-redirects/follow-redirects) from 1.14.1 to 1.14.8.
- [Release notes](https://github.com/follow-redirects/follow-redirects/releases)
- [Commits](https://github.com/follow-redirects/follow-redirects/compare/v1.14.1...v1.14.8)

---
updated-dependencies:
- dependency-name: follow-redirects
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-04-11 14:22:18 -04:00
dependabot[bot] c6a2988083 Bump accumulo-core from 1.9.3 to 1.10.1 (#231)
Bumps [accumulo-core](https://github.com/apache/accumulo) from 1.9.3 to 1.10.1.
- [Release notes](https://github.com/apache/accumulo/releases)
- [Commits](https://github.com/apache/accumulo/compare/rel/1.9.3...rel/1.10.1)

---
updated-dependencies:
- dependency-name: org.apache.accumulo:accumulo-core
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-04-11 14:21:50 -04:00
Bill Oley d54817cc9a Bump version to 1.3.6-SNAPSHOT 2022-01-07 14:43:04 -05:00
Bill Oley c4ee035584 Release version 1.3.5 2022-01-07 14:40:06 -05:00
Chromico Rek d4bfc1a3aa Update log4j to 2.17.1 to address CVE-2021-44832 (#227) 2021-12-30 13:50:34 -05:00
Chromico Rek fb315148a1 Update log4j to 2.17 to address CVE-2021-45105. (#226) 2021-12-20 10:44:47 -05:00
dependabot[bot] e8ae575086 Bump @grafana/data from 8.2.2 to 8.2.3 in /grafana/timely-app (#222)
Bumps [@grafana/data](https://github.com/grafana/grafana/tree/HEAD/packages/grafana-data) from 8.2.2 to 8.2.3.
- [Release notes](https://github.com/grafana/grafana/releases)
- [Changelog](https://github.com/grafana/grafana/blob/main/CHANGELOG.md)
- [Commits](https://github.com/grafana/grafana/commits/v8.2.3/packages/grafana-data)

---
updated-dependencies:
- dependency-name: "@grafana/data"
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-12-15 16:59:06 -05:00
Christopher Tubbs da0146c741 Bump log4j and other cleanup
* Update log4j2 to 2.16.0 to address CVEs
* Update impsort and sortpom plugins
* Update spotbugs plugin to build on newer JDKs and suppress common
  minor warnings from that
* Fix a few minor warnings from spotbugs that are easy: remove hamcrest
  matcher, fix null checks that are always or never null
* Suppress module-info.class from shaded jars
2021-12-15 16:53:24 -05:00
Bill Oley 882b4e2587 Bump version to 1.3.5-SNAPSHOT 2021-11-01 10:17:49 -04:00
Bill Oley 8048862f2c Release version 1.3.4 2021-11-01 10:15:01 -04:00
Bill Oley c2f00c3d19 Convert line endings in pom files from dos->unix 2021-11-01 10:11:06 -04:00
Bill Oley 61b1a245e2 Bump version to 1.3.4-SNAPSHOT 2021-10-26 16:07:30 -04:00
Bill Oley a3c33115c4 Release version 1.3.3 2021-10-26 15:57:25 -04:00
Bill Oley bed1ceb9ee Remove yarn-error.log and add to .gitignore 2021-10-26 15:56:12 -04:00
Bill Oley 06e2699a44 Update to [email protected], [email protected], [email protected] 2021-10-26 14:59:25 -04:00
Bill Oley d09faaa087 Correct log message formatting 2021-10-22 14:21:49 -04:00
dependabot[bot] 088f1a5cab Bump jsoup from 1.11.3 to 1.14.2 (#216)
Bumps [jsoup](https://github.com/jhy/jsoup) from 1.11.3 to 1.14.2.
- [Release notes](https://github.com/jhy/jsoup/releases)
- [Changelog](https://github.com/jhy/jsoup/blob/master/CHANGES)
- [Commits](https://github.com/jhy/jsoup/compare/jsoup-1.11.3...jsoup-1.14.2)

---
updated-dependencies:
- dependency-name: org.jsoup:jsoup
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-10-22 13:47:11 -04:00
dependabot[bot] cf6d4ada3e Bump guava from 27.1-jre to 29.0-jre (#214)
Bumps [guava](https://github.com/google/guava) from 27.1-jre to 29.0-jre.
- [Release notes](https://github.com/google/guava/releases)
- [Commits](https://github.com/google/guava/commits)

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-10-22 13:46:44 -04:00
Bill Oley e464a8c9e0 fixes #217 - Improve ageoff iterator remove/add logic in DataStoreImpl (#218)
* fixes #217 - Improve ageoff iterator remove/add logic in DataStoreImpl

* fixes #217 - Use curator LeaderLatch to elect a leader server which will remove/add the age off iterators
2021-10-22 13:13:00 -04:00
Bill Oley 8e4e61c373 fixes #219 Create scripts to help with running and testing multiple instances of server and balancer 2021-10-22 13:11:41 -04:00
dependabot[bot] 729dca23ba Bump httpclient from 4.5.6 to 4.5.13 (#215)
Bumps httpclient from 4.5.6 to 4.5.13.

---
updated-dependencies:
- dependency-name: org.apache.httpcomponents:httpclient
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-06-03 23:49:36 -04:00
Bill Oley 104d20ad1b Bump version to 1.3.3-SNAPSHOT 2020-11-06 13:30:04 -05:00
Bill Oley f0f005f8fa Release version 1.3.2 2020-11-06 13:23:36 -05:00
Bill Oley 8593218b89 Logging fixes 2020-11-06 12:24:22 -05:00
Bill Oley cbfe6a439f closes #213 Release reference counted netty objects to avoid memory errors 2020-11-06 11:50:24 -05:00
Bill Oley 455616d568 closes #213 Release reference counted netty objects to avoid memory errors 2020-11-05 14:17:01 -05:00
Bill Oley 9d3172cc75 Close objects when TcpClient has problem connecting to Timely 2020-11-04 16:06:08 -05:00
Bill Oley 7aa7b0da2a Remove entries from metricMap when metric is no longer cached on a particular server 2020-11-04 16:04:52 -05:00
Bill Oley 6c61861d83 Fix NPE in BalancedMetricResolver.rebalance where a metric exists in metricMap but not in metricToHostMap 2020-10-29 16:49:55 -04:00
Bill Oley 91882ba363 closes #212 Use writeLock for DataStoreCache.ageOffGorillaStores to avoid ConcurrentModificationException 2020-10-28 15:58:33 -04:00
Bill Oley ed61b5f305 Fix css inconsistent rendering in grafana datasource QueryEditor, remove overridden/unusd css 2020-10-27 14:10:58 -04:00
Bill Oley a083c2d632 Fix oauth issues - intersect TimelyUser auths with Accumulo auths when creating from oauthToken, allow X.509 (PEM) jwt signature check key, increase size of httpHeader max size 2020-10-23 14:41:46 -04:00
Jonathan SteinandJonathan Stein 2891794174 Fixing Nondeterministic Tests (#211)
* Fix MetricAdapterTest and QueryResponseTest flakiness

* change MetricAdapterTest and QueryResponseTest flakiness fix by defining Json key order

* Removed .swp file

* Fix SearchLookupResponse testResponse1

* Fixed flaky AuthCacheTest

Co-authored-by: Jonathan Stein <[email protected]>
2020-10-19 14:01:17 -04:00
Bill Oley c99b2471cf Bump version to 1.3.2-SNAPSHOT 2020-10-14 14:58:07 -04:00
Bill Oley bb479a58d7 Release version 1.3.1 2020-10-14 14:54:59 -04:00
Bill Oley ff37d40801 datasource logging fix, combine datasource golang code for readability 2020-10-14 14:46:09 -04:00
Bill Oley a7d980975e closes #210 Only reload cert and CA when changed
Only reload cert and CA when changed
2020-10-14 13:12:11 -04:00
dependabot[bot] 30f5a0c1b2 Bump junit from 4.12 to 4.13.1 (#209)
Bumps [junit](https://github.com/junit-team/junit4) from 4.12 to 4.13.1.
- [Release notes](https://github.com/junit-team/junit4/releases)
- [Changelog](https://github.com/junit-team/junit4/blob/main/doc/ReleaseNotes4.12.md)
- [Commits](https://github.com/junit-team/junit4/compare/r4.12...r4.13.1)

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2020-10-13 00:54:58 -04:00
Bill Oley 2f9e6ca2d7 Bump version to 1.3.1-SNAPSHOT 2020-09-25 13:04:54 -04:00
Bill Oley 954528a460 Release version 1.3.0 2020-09-25 12:39:33 -04:00
Bill Oley 0f7b514b30 Update authentication configuration and error handling, Grafana depenancies to 7.2.0
Grafana depenancies to 7.2.0
2020-09-25 12:19:08 -04:00
Bill Oley a6ca46e1c1 Rewrite grafana plugin for grafana 7 using typescript, react. Use new backend plugin feature to enable alerts
More work - has errors

more work

more work

Add addOauthPassThru setting

CS fixes, OAuthPassThru setting, Alias fix, Auto Focus on new tag

Logging cleanup and make aliases work

Delete old version of plugin

remove extra files

Accept JWT authentication and test fixes

Accept JWT authentication

Fix assembly

Add JWT settings to sample configs
2020-09-15 16:26:29 -04:00
Bill Oley 487d1d4a5c Use MetaCache for suggest api and implement tagk and tagv types of suggest 2020-09-15 16:26:18 -04:00
Christopher Tubbs e3c26f815c Merge pull request #206 from NationalSecurityAgency/dependabot/maven/commons-beanutils-commons-beanutils-1.9.4
Bump commons-beanutils from 1.9.3 to 1.9.4
2020-07-02 17:16:58 -04:00
Christopher Tubbs bbc1946204 Update plugins for newer JDKs
* Bump formatter and impsort to latest to stay current
* Bump spotbugs to support up to at least JDK 14
2020-07-02 17:09:06 -04:00
dependabot[bot] 991412f99b Bump commons-beanutils from 1.9.3 to 1.9.4
Bumps commons-beanutils from 1.9.3 to 1.9.4.

Signed-off-by: dependabot[bot] <[email protected]>
2020-07-02 20:44:34 +00:00
Christopher Tubbs 5428e93cb5 Merge pull request #205 from NationalSecurityAgency/dependabot/maven/version.log4j-2.13.3
Bump version.log4j from 2.11.2 to 2.13.3
2020-07-02 16:44:06 -04:00
dependabot[bot] c18aae5a89 Bump version.log4j from 2.11.2 to 2.13.3
Bumps `version.log4j` from 2.11.2 to 2.13.3.

Updates `log4j-api` from 2.11.2 to 2.13.3

Updates `log4j-core` from 2.11.2 to 2.13.3

Updates `log4j-slf4j-impl` from 2.11.2 to 2.13.3

Signed-off-by: dependabot[bot] <[email protected]>
2020-07-01 18:20:00 +00:00
Bill Oley 6972c46a1a Merge pull request #204 from matthpeterson/issue-203
Closes issue #203 - avoid calling sizeInBytes if it's not expected to exist
2020-04-17 10:42:14 -04:00
matthpeterson 4d7bee7566 Closes issue #203 - Stop calling sizeInBytes if it's not expected to exist 2020-04-16 15:17:08 +00:00
Bill Oley 4b11b3d50f closes #201 Create balancer assignment file if it doesn't exist 2020-04-10 10:27:18 -04:00
Bill Oley 84b9856ec7 closes #199 Fix issues when using a generated (self-signed) server certificate 2020-04-10 10:20:25 -04:00
Bill Oley c9548a9b9a closes #198 Cleaner shutdown of resources for Timely server, balancer, grafana-auth 2020-04-10 10:01:47 -04:00
Bill Oley aec7bd4c38 closes #197 Fix scan range in MetaCacheImpl 2020-04-09 11:58:30 -04:00
Bill Oley e2b8b643f2 closes #200 Initialize non-cached metrics stored in zookeeper 2020-04-09 11:32:03 -04:00
Bill Oley c19e1d8b2f Set version to 1.2.8-SNAPSHOT 2020-02-21 10:20:48 -05:00
Bill Oley cb73e8e75c Tag version 1.2.7 2020-02-21 10:02:31 -05:00
Bill Oley 4c508c1a7d Merge pull request #195 from dtspence/TIMELY-194
Fixed #194, corrected datasource.js to look at wildcard tag value
2020-02-21 08:55:37 -05:00
dtspence c6c2c0d41e Fixed #194, corrected datasource.js to look at wildcard tag value
The alignment process to match tag query results with the metric definition
looks at the tag k/v pairs and makes a special consideration for a wildcard tag value.
The change adjusts datasource.js so that the '.*' notation will match a query result
and metric definition.
2020-02-05 19:56:03 +00:00
Christopher Tubbs a24cef78b3 Update jackson (multiple CVEs) (#193) 2019-11-07 23:05:38 -05:00
Bill Oley 08f99f6540 Set version to 1.2.7-SNAPSHOT 2019-10-30 12:33:49 -04:00
Bill Oley be11da7d15 Tag version 1.2.6 2019-10-30 12:30:33 -04:00
Bill Oley 4bc5753328 fixes #192 Thread deadlock in BalancedMetricResolver 2019-10-29 12:24:10 -04:00
Bill Oley 2ccb89a06d re #191 Correct typos 2019-10-18 09:16:03 -04:00
Bill Oley dfc3840ad4 re #191 Delete cached metrics when they are no longer being updated 2019-10-17 17:49:24 -04:00
Bill Oley 6235034d8b BalancedMetricResolver in the non-leader balancer should read assignments at intervals 2019-10-17 17:49:24 -04:00
Bill Oley d8de6ac06f Make zookeeper timeout configurable 2019-10-17 17:49:24 -04:00
Christopher Tubbs f381946abe Merge pull request #190 from NationalSecurityAgency/dependabot/maven/io.netty-netty-all-4.1.42.Final
Bump netty-all from 4.1.35.Final to 4.1.42.Final
2019-10-17 09:35:13 -07:00
dependabot[bot] 48bbe6a561 Bump netty-all from 4.1.35.Final to 4.1.42.Final
Bumps [netty-all](https://github.com/netty/netty) from 4.1.35.Final to 4.1.42.Final.
- [Release notes](https://github.com/netty/netty/releases)
- [Commits](https://github.com/netty/netty/compare/netty-4.1.35.Final...netty-4.1.42.Final)

Signed-off-by: dependabot[bot] <[email protected]>
2019-10-11 18:58:57 +00:00
Christopher Tubbs 0f6110ea59 Update jackson to 2.9.10
* Fix CVE-2019-16335
* Fix CVE-2019-14540
2019-09-26 23:40:08 -04:00
Bill Oley 461b5e7e78 Pom version to 1.2.6-SNAPSHOT 2019-08-22 12:17:45 -04:00
Bill Oley 4ec7d94c28 Tag version 1.2.5 2019-08-22 12:12:49 -04:00
Bill Oley 77e910ae06 Use BufferedOutputWriter instead of PrintWriter, pass Exceptions instead of having to check error condition 2019-08-22 10:09:27 -04:00
Bill Oley a4998af887 Move jackson-databind version to the properties section 2019-08-08 13:50:08 -04:00
Bill Oley 2858549ede Merge pull request #188 from ctubbsii/jdk11-build-support
Bump versions / permit build with JDK11
2019-08-08 13:35:54 -04:00
Christopher Tubbs 57f0e1408f Bump versions / permit build with JDK11
Support builds on newer JDKs
* Update tycho-compiler-jdt in analytics/pom.xml
* Ignore spotbugs RCN_REDUNDANT_NULLCHECK_WOULD_HAVE_BEEN_A_NPE for JDK11
* Bump several plugins to latest (maven-compiler-plugin, etc.)
* Add profile for using -release flag in Eclipse and with JDK9 and later

Fix recently added warnings
* Add try-with-resources for TreeCache objects to remove recent warnings
* Suppress generics warning in MetricCompactionStrategy
* Update TabletMetadataConsole to avoid deprecated Accumulo
  ClientConfiguration API

Other
* Bump jackson-databind for CVE-2019-14439, CVE-2019-14379
* Bump hadoop 2.7 to latest in series (2.7.3->2.7.7) for build
2019-08-06 16:14:13 -04:00
Bill Oley 151adc025a Merge pull request #185 from dtspence/TIMELY-184
Fixes #184 Metric compaction strategy and summary implementation
2019-08-06 15:15:45 -04:00
Bill Oley 5abd9d7200 closes #187 Prevent deadlock in balancer distributed read/write lock 2019-08-06 12:00:57 -04:00
Bill Oley fa2e211c72 closes #186 Make balancer websocket incomingBufferSize its own setting 2019-08-06 11:47:36 -04:00
dtspence a21438976e TIMELY-184: Metric compaction strategy and summary implementation. 2019-08-01 14:05:35 +00:00
Christopher Tubbs ba37047f3a Update jackson-databind (CVE-2019-12814) 2019-07-23 20:38:46 -04:00
Bill Oley 9305470c69 re #183 Check tagKey not null or blank in Meta.toKeys 2019-06-19 09:08:20 -04:00
Bill Oley 200233542e re #183 Add timestamps to meta tags to allow proper age off 2019-06-17 15:36:51 -04:00
Bill Oley 3cd78dc624 Merge branch 'drewfarris-config-fixup' 2019-05-30 12:55:35 -04:00
Bill Oley 9027cac6e3 Merge branch 'config-fixup' of https://github.com/drewfarris/timely into drewfarris-config-fixup
Conflicts:
	server/conf/timely-docker.yml
	server/conf/timely-standalone.yml
	server/conf/timely.yml
2019-05-30 12:50:31 -04:00
Bill Oley 3663b4e6d5 Pom version to 1.2.5-SNAPSHOT 2019-05-30 12:38:42 -04:00
Bill Oley 64973d0674 Merge pull request #182 from ctubbsii/fix-jackson
Fix most compiler warnings and many other issues
2019-05-30 12:25:00 -04:00
Christopher Tubbs 5512306f1e Fix most compiler warnings and other issues
Build:
* Fix netty-tcnative-boringssl-static version specs in pom.xml
* Bump the version of accumulo to latest release (1.9.3)
* Update jackson to 2.9.9 (CVE-2019-12086)

Warnings:
* Remove all use of deprecated netty stuffs and use their replacements
  in the netty APIs
* Add missing generics parameters everywhere
* Remove unused code (unused private fields, such as loggers)
* Remove standard javadoc tags that were missing descriptions (these do
  not add value to the javadocs, since default `@param`, `@throws`, and
  `@return` tags are provided by javadoc)
* Add missing serialVersionUid fields to Serializable classes
* Fix a resource leak warning about not closing ScannerOptionsHelper
  (NOTE: 3 other resource leak warnings exist, but did not attempt to
  fix those, since they were non-trivial)
* Remove deprecated annotations from hibernate and replace with their
  non-deprecated counterparts from javax.validation.constraints
* Remove use of deprecated Accumulo ClientConfiguration constructor

Other:
* Use lowercase header names, for HTTP/2 compatibility (and to stop
  incorrectly assuming case-sensitivity for HTTP/1)
* Narrow the scope of necessary `@SuppressWarnings("unchecked")`
  occurrences to a variable instead of suppressing for an entire method or
  an entire class
* IDE added some missing overrides annotations
* (style consistency) Replace tabs with spaces
* Set log4j2 root logger level for tests to WARN instead of TRACE
2019-05-24 12:12:00 -04:00
Bill Oley d35ae08188 Tag version 1.2.4 2019-05-22 17:14:42 -04:00
Bill Oley e448657afb Correction in websocket pipeline from Netty API change 2019-05-22 16:49:11 -04:00
Bill Oley 1dcd893711 Add age off iterator for meta table
Add age off iterator for meta table
2019-05-22 16:49:08 -04:00
Bill Oley 409f5375d1 Change pom version to 1.2.4-SNAPSHOT 2019-05-22 16:48:09 -04:00
Bill Oley c5df8139ca Correct typos 2019-05-22 16:46:55 -04:00
Bill Oley dcd23d4fc8 fixes #181 Use netty-tcnative-boringssl-static 2019-04-18 11:56:54 -04:00
Bill Oley cc7326fc19 Tag version 1.2.2 2019-04-12 11:24:53 -04:00
Bill Oley 0d2b6e61c8 Correct typos 2019-04-12 10:55:04 -04:00
Bill Oley b7cd8765c5 Correct typos and minor changed to MetaCache 2019-04-10 15:31:53 -04:00
Bill Oley a9d27d5b1b re #179 Allow AuthCache to periodically refresh credentials, cache meta table up to a configurable number of tags 2019-04-08 10:50:14 -04:00
Bill Oley 7d67eb218d Update netty-tcnative dependency, remove x509 principal extractor from AuthenticationService 2019-04-03 15:52:19 -04:00
Bill Oley 5fb7e3dd2b Bump pom versions to 1.2.2-SNAPSHOT 2019-04-02 12:57:22 -04:00
Bill Oley 5a89861cec Tag version 1.2.1 2019-04-02 12:53:04 -04:00
Bill Oley a3bc9a01ee Update dependency versions
Update dependencies

Update dependency versions
2019-04-02 12:43:36 -04:00
Drew Farris e2bbe4e212 Minor fix-ups to default configuration files.
. Replace ssl with server-ssl in timely-standalone.yml and timely-docker.yml
  . Add a value for cache.enabled in timely.yml
2019-01-19 22:07:07 -05:00
682 changed files with 188107 additions and 48039 deletions

No files matched your search

+1
View File
@@ -7,4 +7,5 @@ target/
logs/
server/pki/
client/src/main/python/setup.py
grafana/timely-app/yarn-error.log
+2 -2
View File
@@ -1,7 +1,7 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
httpProperties://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
@@ -193,7 +193,7 @@
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
httpProperties://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
+1 -1
View File
@@ -2,7 +2,7 @@
[![Apache License][li]][ll]
Timely is a time series database application that provides secure access to time series data. Timely is written in Java and designed to work with [Apache Accumulo](https://accumulo.apache.org/) and [Grafana](https://www.grafana.com). Documentation is located [here](https://nationalsecurityagency.github.io/timely/).
Timely is a time series database application that provides secure access to time series data. Timely is written in Java and designed to work with [Apache Accumulo](https://accumuloProperties.apache.org/) and [Grafana](https://www.grafana.com). Documentation is located [here](https://nationalsecurityagency.github.io/timely/).
[li]: https://img.shields.io/badge/license-ASL-blue.svg
[ll]: https://www.apache.org/licenses/LICENSE-2.0
+21 -7
View File
@@ -4,7 +4,7 @@
<parent>
<groupId>timely</groupId>
<artifactId>timely</artifactId>
<version>1.2.1-SNAPSHOT</version>
<version>2.1.5</version>
</parent>
<artifactId>timely-analytics</artifactId>
<!--
@@ -53,14 +53,13 @@
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>timely</groupId>
<artifactId>timely-common</artifactId>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<scope>test</scope>
</dependency>
</dependencies>
@@ -68,6 +67,7 @@
<pluginManagement>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<configuration>
<compilerId>jdt</compilerId>
@@ -76,7 +76,7 @@
<dependency>
<groupId>org.eclipse.tycho</groupId>
<artifactId>tycho-compiler-jdt</artifactId>
<version>0.21.0</version>
<version>1.4.0</version>
</dependency>
</dependencies>
</plugin>
@@ -168,6 +168,12 @@
</excludes>
</artifactSet>
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>**/module-info.class</exclude>
</excludes>
</filter>
<filter>
<artifact>org.apache.flink:*</artifact>
<excludes>
@@ -245,6 +251,14 @@
<artifactSet>
<excludes combine.self="override" />
</artifactSet>
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>**/module-info.class</exclude>
</excludes>
</filter>
</filters>
</configuration>
</execution>
</executions>
@@ -7,6 +7,7 @@ import java.util.List;
import java.util.stream.Collectors;
import org.apache.flink.streaming.util.serialization.SerializationSchema;
import timely.model.Tag;
public class MetricHistogram implements Serializable, SerializationSchema<MetricHistogram> {
@@ -1,13 +1,13 @@
package timely.analytics.flink;
import org.apache.flink.api.java.functions.KeySelector;
import timely.api.response.MetricResponse;
/**
* Uniquely defines a time series in a Flink stream. A unique time series is the
* metric name and all of its tags, to include visibility if it exists.
* Uniquely defines a time series in a Flink stream. A unique time series is the metric name and all of its tags, to include visibility if it exists.
*/
public class MetricKeySelector implements KeySelector<MetricResponse, String> {
public class MetricKeySelector implements KeySelector<MetricResponse,String> {
private static final long serialVersionUID = 1L;
@@ -9,11 +9,11 @@ import org.apache.flink.api.common.accumulators.Accumulator;
* Custom accumulator that retains a sorted map of strings and their counts
*
*/
public class SortedStringAccumulator implements Accumulator<String, ConcurrentSkipListMap<String, MutableLong>> {
public class SortedStringAccumulator implements Accumulator<String,ConcurrentSkipListMap<String,MutableLong>> {
private static final long serialVersionUID = 1L;
private ConcurrentSkipListMap<String, MutableLong> values = new ConcurrentSkipListMap<>();
private ConcurrentSkipListMap<String,MutableLong> values = new ConcurrentSkipListMap<>();
@Override
public void add(String value) {
@@ -25,7 +25,7 @@ public class SortedStringAccumulator implements Accumulator<String, ConcurrentSk
}
@Override
public ConcurrentSkipListMap<String, MutableLong> getLocalValue() {
public ConcurrentSkipListMap<String,MutableLong> getLocalValue() {
return values;
}
@@ -35,7 +35,7 @@ public class SortedStringAccumulator implements Accumulator<String, ConcurrentSk
}
@Override
public void merge(Accumulator<String, ConcurrentSkipListMap<String, MutableLong>> other) {
public void merge(Accumulator<String,ConcurrentSkipListMap<String,MutableLong>> other) {
other.getLocalValue().forEach((k, v) -> {
if (values.containsKey(k)) {
values.get(k).add(v.longValue());
@@ -46,7 +46,7 @@ public class SortedStringAccumulator implements Accumulator<String, ConcurrentSk
}
@Override
public Accumulator<String, ConcurrentSkipListMap<String, MutableLong>> clone() {
public Accumulator<String,ConcurrentSkipListMap<String,MutableLong>> clone() {
SortedStringAccumulator a = new SortedStringAccumulator();
a.getLocalValue().putAll(this.values);
return a;
@@ -22,6 +22,7 @@ import org.apache.flink.streaming.api.functions.source.RichSourceFunction;
import org.apache.flink.streaming.api.watermark.Watermark;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import timely.api.response.MetricResponse;
import timely.api.response.MetricResponses;
import timely.client.websocket.ClientHandler;
@@ -41,7 +42,7 @@ public class SubscriptionSource extends RichSourceFunction<MetricResponse> imple
}
}
private static final Logger LOG = LoggerFactory.getLogger(SubscriptionSource.class);
private static final Logger log = LoggerFactory.getLogger(SubscriptionSource.class);
private static final long serialVersionUID = 1L;
private transient WebSocketSubscriptionClient client = null;
@@ -63,12 +64,11 @@ public class SubscriptionSource extends RichSourceFunction<MetricResponse> imple
@Override
public void open(Configuration parameters) throws Exception {
LOG.info("Opening summarization job.");
log.info("Opening summarization job.");
super.open(parameters);
client = new WebSocketSubscriptionClient(jp.getTimelyHostname(), jp.getTimelyHttpsPort(), jp.getTimelyWssPort(),
jp.isClientAuth(), jp.isDoLogin(), jp.getTimelyUsername(), jp.getTimelyPassword(), jp.getKeyStoreFile(),
jp.getKeyStoreType(), jp.getKeyStorePass(), jp.getTrustStoreFile(), jp.getTrustStoreType(),
jp.getTrustStorePass(), jp.isHostVerificationEnabled(), jp.getBufferSize());
client = new WebSocketSubscriptionClient(jp.getTimelyHostname(), jp.getTimelyHttpsPort(), jp.getTimelyWssPort(), jp.isClientAuth(), jp.isDoLogin(),
jp.getKeyStoreFile(), jp.getKeyStoreType(), jp.getKeyStorePass(), jp.getTrustStoreFile(), jp.getTrustStoreType(),
jp.getTrustStorePass(), jp.isHostVerificationEnabled(), jp.getBufferSize());
}
@Override
@@ -81,7 +81,7 @@ public class SubscriptionSource extends RichSourceFunction<MetricResponse> imple
@Override
public void run(SourceContext<MetricResponse> ctx) throws Exception {
LOG.info("Running summarization job");
log.info("Running summarization job");
final SortedStringAccumulator dateTimeAccumulator = new SortedStringAccumulator();
this.getRuntimeContext().addAccumulator("source hourly count", dateTimeAccumulator);
final LongCounter sourceInputs = new LongCounter();
@@ -103,22 +103,20 @@ public class SubscriptionSource extends RichSourceFunction<MetricResponse> imple
@Override
public void onMessage(final String message) {
LOG.info("Message received on Websocket session {}, length: {}", session.getId(),
message.length());
log.info("Message received on Websocket session {}, length: {}", session.getId(), message.length());
try {
// Deserialize in this thread
final MetricResponses responses = JsonSerializer.getObjectMapper().readValue(message,
MetricResponses.class);
final MetricResponses responses = JsonSerializer.getObjectMapper().readValue(message, MetricResponses.class);
// Process in the other
final Runnable r = () -> {
responses.getResponses().forEach(response -> {
if (response.isComplete()) {
LOG.info("Received last message.");
log.info("Received last message.");
ctx.emitWatermark(new Watermark(Long.MAX_VALUE));
subscriptionsRemaining.getAndDecrement();
return;
}
LOG.trace("Received metric: {}", response);
log.trace("Received metric: {}", response);
long time = response.getTimestamp();
ctx.collectWithTimestamp(response, time);
dateTimeAccumulator.add(formatter.format(new Date(response.getTimestamp())));
@@ -141,11 +139,11 @@ public class SubscriptionSource extends RichSourceFunction<MetricResponse> imple
inputProcessedInMainThread.add(1);
}
} catch (IOException e) {
LOG.error("Error deserializing metric response, closing", e);
log.error("Error deserializing metric response, closing", e);
try {
close();
} catch (Exception e1) {
LOG.error("Error closing client", e1);
log.error("Error closing client", e1);
}
}
}
@@ -158,7 +156,7 @@ public class SubscriptionSource extends RichSourceFunction<MetricResponse> imple
try {
close();
} catch (Exception e1) {
LOG.error("Error closing client", e1);
log.error("Error closing client", e1);
}
// Signal done sending data
ctx.emitWatermark(new Watermark(Long.MAX_VALUE));
@@ -170,7 +168,7 @@ public class SubscriptionSource extends RichSourceFunction<MetricResponse> imple
try {
close();
} catch (Exception e1) {
LOG.error("Error closing client", e1);
log.error("Error closing client", e1);
}
}
@@ -178,14 +176,14 @@ public class SubscriptionSource extends RichSourceFunction<MetricResponse> imple
try {
client.open(handler);
for (String m : metrics) {
LOG.info("Adding subscription for {}", m);
log.info("Adding subscription for {}", m);
client.addSubscription(m, null, start, end, 5000);
}
while (!client.isClosed() && subscriptionsRemaining.get() > 0) {
try {
Thread.sleep(5000);
} catch (InterruptedException e) {
LOG.error("Sleep interrupted.");
log.error("Sleep interrupted.");
}
}
} finally {
@@ -195,21 +193,21 @@ public class SubscriptionSource extends RichSourceFunction<MetricResponse> imple
@Override
public void cancel() {
LOG.info("Cancelling subscription source.");
log.info("Cancelling subscription source.");
try {
close();
} catch (Exception e) {
LOG.error("Error closing web socket client");
log.error("Error closing web socket client");
}
}
@Override
public void stop() {
LOG.info("Stopping subscription source.");
log.info("Stopping subscription source.");
try {
close();
} catch (Exception e) {
LOG.error("Error closing web socket client");
log.error("Error closing web socket client");
}
}
@@ -13,11 +13,12 @@ import org.apache.flink.streaming.api.functions.sink.SocketClientSink;
import org.apache.flink.streaming.api.windowing.assigners.TumblingEventTimeWindows;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import timely.api.response.MetricResponse;
public class SummarizationJob {
private static final Logger LOG = LoggerFactory.getLogger(SummarizationJob.class);
private static final Logger log = LoggerFactory.getLogger(SummarizationJob.class);
public static void main(String[] args) throws Exception {
@@ -26,11 +27,11 @@ public class SummarizationJob {
System.exit(1);
}
LOG.info("Loading properties from: {}", args[0]);
log.info("Loading properties from: {}", args[0]);
ParameterTool params = ParameterTool.fromPropertiesFile(args[0]);
final SummarizationJobParameters jp = new SummarizationJobParameters(params);
LOG.info("Properties: {}", jp);
log.info("Properties: {}", jp);
final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
// @formatter:off
@@ -42,7 +43,7 @@ public class SummarizationJob {
.window(TumblingEventTimeWindows.of(jp.getSummarizationInterval()))
.fold(new MetricHistogram(), new FoldFunction<MetricResponse, MetricHistogram>() {
private final Logger LOG = LoggerFactory
private final Logger log = LoggerFactory
.getLogger("timely.analytics.SummarizationJob.FoldFunction@");
private static final long serialVersionUID = 1L;
@@ -58,7 +59,7 @@ public class SummarizationJob {
throw new RuntimeException("Tags don't not match - partitioning problem");
}
}
LOG.trace("Updating histogram for {} with value {} at {} for metric {}", histo.getMetric()
log.trace("Updating histogram for {} with value {} at {} for metric {}", histo.getMetric()
+ histo.getTags(), metric.getValue(), metric.getTimestamp(), metric.getMetric()
+ metric.getTags());
histo.update(metric.getValue(), metric.getTimestamp());
@@ -67,10 +68,10 @@ public class SummarizationJob {
})
.name("Metric Windows")
.addSink(
new SocketClientSink<MetricHistogram>(jp.getTimelyHostname(), jp.getTimelyTcpPort(),
new SocketClientSink<MetricHistogram>(jp.getTimelyHostname(), jp.getTimelyTcpPort(),
new MetricHistogram(), 2, true) {
private final Logger LOG = LoggerFactory
private final Logger log = LoggerFactory
.getLogger("timely.analytics.SummarizationJob.SocketClientSink");
private static final long serialVersionUID = 1L;
private final LongCounter sinkCounter = new LongCounter();
@@ -82,7 +83,7 @@ public class SummarizationJob {
super.open(parameters);
this.getRuntimeContext().addAccumulator("sink counter", this.sinkCounter);
this.getRuntimeContext().addAccumulator("sink hourly count", dateTimeAccumulator);
LOG.info("Sink opened");
log.info("Sink opened");
}
@Override
@@ -96,7 +97,7 @@ public class SummarizationJob {
@Override
public void close() throws Exception {
super.close();
LOG.info("Sink closed");
log.info("Sink closed");
}
}).name("Timely Metric Sink");
@@ -184,8 +184,8 @@ public class SummarizationJobParameters extends GlobalJobParameters implements S
}
@Override
public Map<String, String> toMap() {
Map<String, String> m = new HashMap<>();
public Map<String,String> toMap() {
Map<String,String> m = new HashMap<>();
m.put("timelyHostname", this.timelyHostname);
m.put("timelyHttpsPort", Integer.toString(this.timelyHttpsPort));
m.put("timelyWssPort", Integer.toString(this.timelyWssPort));
+8
View File
@@ -15,4 +15,12 @@
<Method name="getMetrics" />
<Bug pattern="EI_EXPOSE_REP" />
</Match>
<Match>
<!-- these are probably avoidable, but needs further triage to fix -->
<Bug pattern="EI_EXPOSE_REP" />
</Match>
<Match>
<!-- these are probably avoidable, but needs further triage to fix -->
<Bug pattern="EI_EXPOSE_REP2" />
</Match>
</FindBugsFilter>
@@ -6,6 +6,7 @@ import java.util.List;
import org.junit.Assert;
import org.junit.Before;
import org.junit.Test;
import timely.model.Metric;
import timely.model.Tag;
import timely.model.parse.MetricParser;
@@ -96,9 +97,8 @@ public class MetricHistogramTest {
Metric metric = metricParser.parse(put);
Assert.assertEquals("sys.cpu.user_summarized", metric.getName());
metric.getTags().forEach(t -> {
Assert.assertTrue(
t.equals(t1) || t.equals(t2) || t.equals(avg) || t.equals(min) || t.equals(max) || t.equals(sum)
|| t.equals(count) || t.equals(p50) || t.equals(p75) || t.equals(p90) || t.equals(p99));
Assert.assertTrue(t.equals(t1) || t.equals(t2) || t.equals(avg) || t.equals(min) || t.equals(max) || t.equals(sum) || t.equals(count)
|| t.equals(p50) || t.equals(p75) || t.equals(p90) || t.equals(p99));
});
}
}
+18
View File
@@ -0,0 +1,18 @@
#!/bin/bash
if [[ `uname` == "Darwin" ]]; then
THIS_SCRIPT=`python -c 'import os,sys; print os.path.realpath(sys.argv[1])' $0`
else
THIS_SCRIPT=`readlink -f $0`
fi
THIS_DIR="${THIS_SCRIPT%/*}"
BASE_DIR=$(cd $THIS_DIR/.. && pwd)
BIN_DIR=${BASE_DIR}/bin
. ${BIN_DIR}/timely-balancer-env.sh
for i in $(seq 1 $NUM_INSTANCES); do
echo ${BIN_DIR}/timely-balancer.sh $i
${BIN_DIR}/timely-balancer.sh $i
done
+21
View File
@@ -0,0 +1,21 @@
#!/bin/bash
if [[ `uname` == "Darwin" ]]; then
THIS_SCRIPT=`python -c 'import os,sys; print os.path.realpath(sys.argv[1])' $0`
else
THIS_SCRIPT=`readlink -f $0`
fi
THIS_DIR="${THIS_SCRIPT%/*}"
BASE_DIR=$(cd $THIS_DIR/.. && pwd)
BIN_DIR=${BASE_DIR}/bin
. ${BIN_DIR}/timely-balancer-env.sh
for i in $(seq 1 $NUM_INSTANCES); do
PID=`jps -m | grep -E 'timely-balancer-.*-exec.jar' | grep "instance=${i}" | awk '{print $1}'`
if [ "$PID" != "" ]; then
echo "stopping timely-balancer instance ${i} [${PID}]"
kill -9 $PID
fi
done
File diff suppressed because it is too large. Load diff
-11
View File
@@ -1,11 +0,0 @@
#!/bin/bash
export TRUSTSTORE_FILE=
export TRUSTSTORE_TYPE="jks";
export TRUSTSTORE_PASS=
export KEYSTORE_FILE=
export KEYSTORE_TYPE="pkcs12";
export KEYSTORE_PASS=
export NUM_QUERY_THREADS=4
export NUM_INGEST_THREADS=4
-21
View File
@@ -1,21 +0,0 @@
#!/bin/bash
if [[ `uname` == "Darwin" ]]; then
THIS_SCRIPT=`python -c 'import os,sys; print os.path.realpath(sys.argv[1])' $0`
else
THIS_SCRIPT=`readlink -f $0`
fi
THIS_DIR="${THIS_SCRIPT%/*}"
BASE_DIR=${THIS_DIR}/../..
TMP_DIR="${BASE_DIR}/tmp"
CONF_DIR="${BASE_DIR}/conf"
LIB_DIR="${BASE_DIR}/lib"
. ${THIS_DIR}/load-test-env.sh
export CLASSPATH="${CONF_DIR}:${THIS_DIR}:${LIB_DIR}/*"
JVM_ARGS="-Xmx1G -Xms1G"
echo "$JAVA_HOME/bin/java ${JVM_ARGS} timely.testing.MetricPutLoadTest $*"
$JAVA_HOME/bin/java ${JVM_ARGS} timely.testing.MetricPutLoadTest $*
-21
View File
@@ -1,21 +0,0 @@
#!/bin/bash
if [[ `uname` == "Darwin" ]]; then
THIS_SCRIPT=`python -c 'import os,sys; print os.path.realpath(sys.argv[1])' $0`
else
THIS_SCRIPT=`readlink -f $0`
fi
THIS_DIR="${THIS_SCRIPT%/*}"
BASE_DIR=${THIS_DIR}/../..
TMP_DIR="${BASE_DIR}/tmp"
CONF_DIR="${BASE_DIR}/conf"
LIB_DIR="${BASE_DIR}/lib"
. ${THIS_DIR}/load-test-env.sh
export CLASSPATH="${CONF_DIR}:${THIS_DIR}:${LIB_DIR}/*"
JVM_ARGS="-Xmx1G -Xms1G"
echo "$JAVA_HOME/bin/java ${JVM_ARGS} timely.testing.MetricQueryLoadTest $*"
$JAVA_HOME/bin/java ${JVM_ARGS} timely.testing.MetricQueryLoadTest $*
+27
View File
@@ -0,0 +1,27 @@
#!/bin/bash
NUM_INSTANCES=1
ACCUMULO_ZOOKEEPERS=localhost:2181
HADOOP_CONF_DIR=${HADOOP_CONF_DIR}
HADOOP_DEFAULT_FS=hdfs://localhost:8020
TCP_BASE_PORT=4242
HTTP_BASE_PORT=4243
WEBSOCKET_BASE_PORT=4244
UDP_BASE_PORT=4245
HOST_IP=0.0.0.0
HOST=$(hostname -s)
HOST_VERIFICATION_ENABLED=true
GENERATE_KEYS=true
TRUST_STORE_TYPE=
TRUST_STORE_FILE=
TRUST_STORE_PASSWORD=
KEY_STORE_TYPE=
KEY_STORE_FILE=
KEY_STORE_PASSWORD=
AUTHORIZATION_URL=
AUTHORIZATION_CERTIFICATE_TYPE=
AUTHORIZATION_CERTIFICATE_FILE=
ANONYMOUS_HTTP=true
ANONYMOUS_WEBSOCKET=true
+56 -35
View File
@@ -1,43 +1,64 @@
#!/bin/bash
if [[ `uname` == "Darwin" ]]; then
THIS_SCRIPT=`python -c 'import os,sys; print os.path.realpath(sys.argv[1])' $0`
TCNATIVE_SUFFIX="jnilib"
if [[ $(uname) == "Darwin" ]]; then
THIS_SCRIPT=$(python -c 'import os,sys; print os.path.realpath(sys.argv[1])' "$0")
else
THIS_SCRIPT=`readlink -f $0`
TCNATIVE_SUFFIX="so"
THIS_SCRIPT=$(readlink -f "$0")
HW=`uname -m`
OS_PATH=linux-${HW}
fi
THIS_DIR="${THIS_SCRIPT%/*}"
NATIVE_DIR="${THIS_DIR}/META-INF/native"
BASE_DIR=${THIS_DIR}/..
TMP_DIR="${BASE_DIR}/tmp"
CONF_DIR="${BASE_DIR}/conf"
LIB_DIR="${BASE_DIR}/lib"
NUM_SERVER_THREADS=4
if [[ -e ${TMP_DIR} ]]; then
rm -rf ${TMP_DIR}
if [ -z "$1" ]; then
export INSTANCE="1"
else
export INSTANCE="$1"
fi
mkdir ${TMP_DIR}
if [[ -e ${NATIVE_DIR} ]]; then
rm -rf ${NATIVE_DIR}
PID=$(jps -m | grep -E 'timely-balancer-.*-exec.jar' | grep "instance=${INSTANCE}" | awk '{print $1}')
if [ "$PID" == "" ]; then
THIS_DIR="${THIS_SCRIPT%/*}"
BASE_DIR=$(cd "$THIS_DIR"/.. && pwd)
LIB_DIR="${BASE_DIR}/lib"
BIN_DIR="${BASE_DIR}/bin"
export CONF_DIR="${BASE_DIR}/conf"
NATIVE_DIR="${BIN_DIR}/META-INF/native"
set -a
. "${BIN_DIR}/timely-balancer-env.sh"
set +a
# use either a value from timely-balancer-env.sh or the default
export LOG_DIR="${TIMELY_LOG_DIR:-${BASE_DIR}/logs}"
if [[ ! -e ${NATIVE_DIR} ]]; then
mkdir -p "${NATIVE_DIR}"
fi
if [[ ! -e ${LOG_DIR} ]]; then
mkdir "${LOG_DIR}"
fi
JVM_ARGS="-Xmx2G -Xms2G -XX:NewSize=1G -XX:MaxNewSize=1G"
JVM_ARGS="${JVM_ARGS} -Dlogging.config=${CONF_DIR}/log4j2.yml"
JVM_ARGS="${JVM_ARGS} -Dlog4j2.contextSelector=org.apache.logging.log4j.core.async.AsyncLoggerContextSelector"
JVM_ARGS="${JVM_ARGS} -XX:+UseG1GC -XX:+UseStringDeduplication"
JVM_ARGS="${JVM_ARGS} -Djava.net.preferIPv4Stack=true"
JVM_ARGS="${JVM_ARGS} -XX:+UseNUMA"
# for linux varieties of OS, extract native libraries if not already present
if [[ -n "$OS_PATH" ]]; then
if [[ ! -d ${NATIVE_DIR} || -z "$(ls -A ${NATIVE_DIR})" ]]; then
pushd "${BASE_DIR}/bin" || exit
"${JAVA_HOME}"/bin/jar xf "${LIB_DIR}"/netty-tcnative-boringssl-static-*.Final-"${OS_PATH}".jar META-INF/native/libnetty_tcnative_linux_"${HW}".so
"${JAVA_HOME}"/bin/jar xf "${LIB_DIR}"/netty-transport-native-epoll-*-"${OS_PATH}".jar META-INF/native/libnetty_transport_native_epoll_"${HW}".so
popd || exit
fi
JVM_ARGS="${JVM_ARGS} -Djava.library.path=${NATIVE_DIR}"
fi
echo "${JAVA_HOME}/bin/java ${JVM_ARGS} -jar ${BIN_DIR}/timely-balancer-*-exec.jar --instance=${INSTANCE}"
nohup "${JAVA_HOME}"/bin/java ${JVM_ARGS} -jar "${BIN_DIR}"/timely-balancer-*-exec.jar --instance="${INSTANCE}" >> "${LOG_DIR}/timely-balancer${INSTANCE}.out" 2>&1 &
else
echo "timely-balancer --instance=${INSTANCE} already running with pid ${PID}"
fi
mkdir -p ${NATIVE_DIR}
pushd ${BASE_DIR}/bin
$JAVA_HOME/bin/jar xf ${LIB_DIR}/netty-tcnative*.jar META-INF/native/libnetty_tcnative.${TCNATIVE_SUFFIX}
$JAVA_HOME/bin/jar xf ${LIB_DIR}/netty-transport-native-epoll*.jar META-INF/native/libnetty_transport_native_epoll_x86_64.${TCNATIVE_SUFFIX}
popd
export CLASSPATH="${CONF_DIR}:${LIB_DIR}/*:${HADOOP_CONF_DIR}"
JVM_ARGS="-Xmx1G -Xms1G -Dio.netty.eventLoopThreads=${NUM_SERVER_THREADS}"
JVM_ARGS="${JVM_ARGS} -DLog4jContextSelector=org.apache.logging.log4j.core.async.AsyncLoggerContextSelector"
JVM_ARGS="${JVM_ARGS} -Djava.library.path=${NATIVE_DIR}"
JVM_ARGS="${JVM_ARGS} -Djava.library.path=${NATIVE_DIR}/libnetty-${TCNATIVE_SUFFIX}"
#JVM_ARGS="${JVM_ARGS} -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005"
echo "$JAVA_HOME/bin/java ${JVM_ARGS} timely.balancer.Balancer --spring.config.name=timely --spring.profiles.active=balancer"
$JAVA_HOME/bin/java ${JVM_ARGS} timely.balancer.Balancer --spring.config.name=timely --spring.profiles.active=balancer
-13
View File
@@ -1,13 +0,0 @@
<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="warn" monitorInterval="60">
<Appenders>
<Console name="Console" target="SYSTEM_OUT">
<PatternLayout pattern="%d{HH:mm:ss.SSS} [%t] %-5level %logger{36} - %msg%n"/>
</Console>
</Appenders>
<Loggers>
<Root level="error" includeLocation="false">
<AppenderRef ref="Console" level="error" />
</Root>
</Loggers>
</Configuration>
-29
View File
@@ -1,29 +0,0 @@
<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="warn" monitorInterval="60">
<Properties>
<Property name="PID">????</Property>
<Property name="LOG_PATTERN">%clr{%d{yyyy-MM-dd HH:mm:ss.SSS}}{faint} %clr{%5p} %clr{${sys:PID}}{magenta} %clr{---}{faint} %clr{[%15.15t]}{faint} %clr{%-40.40c{1.}}{cyan} %clr{:}{faint} %m%n%wEx</Property>
</Properties>
<Appenders>
<RollingRandomAccessFile name="Rolling" fileName="/srv/logs/timely/timely-balancer.log"
filePattern="/srv/logs/timely/$${date:yyyy-MM}/timely-balancer-%d{MM-dd-yyyy}-%i.log.gz">
<PatternLayout pattern="${LOG_PATTERN}"/>
<Policies>
<SizeBasedTriggeringPolicy size="1024 MB" />
</Policies>
<DefaultRolloverStrategy max="10" />
</RollingRandomAccessFile>
<Console name="Console" target="SYSTEM_OUT">
<PatternLayout pattern="${LOG_PATTERN}"/>
</Console>
</Appenders>
<Loggers>
<Root level="trace" includeLocation="false">
<AppenderRef ref="Console" level="warn" />
<AppenderRef ref="Rolling" level="trace" />
</Root>
<Logger name="timely" additivity="false">
<AppenderRef ref="Console" level="debug" />
</Logger>
</Loggers>
</Configuration>
+36
View File
@@ -0,0 +1,36 @@
Configuration:
status: warn
monitorInterval: 60
Properties:
Property:
- name: logDir
value: ${env:LOG_DIR}
- name: PID
value: "????"
- name: LOG_PATTERN
value: "%d{ISO8601} %-5p [%-40.40t] [%-40.40c{1.}] %m%n%wEx"
Appenders:
RollingFile:
- name: File
fileName: "${sys:logDir}/timely-balancer${env:INSTANCE}.log"
filePattern: "${sys:logDir}/timely-balancer${env:INSTANCE}.log.%d{yyyy-MM-dd}-%i.gz"
append: true
bufferedIO: true
bufferSize: 8192
Policies:
TimeBasedTriggeringPolicy:
interval: 1
SizeBasedTriggeringPolicy:
size: 250MB
DefaultRolloverStrategy:
max: 10
PatternLayout:
pattern: "${LOG_PATTERN}"
Loggers:
Root:
level: info
AppenderRef:
- ref: File
-71
View File
@@ -1,71 +0,0 @@
<beans:beans xmlns="http://www.springframework.org/schema/security"
xmlns:beans="http://www.springframework.org/schema/beans"
xmlns:util="http://www.springframework.org/schema/util"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans-3.0.xsd
http://www.springframework.org/schema/security
http://www.springframework.org/schema/security/spring-security.xsd
http://www.springframework.org/schema/util
http://www.springframework.org/schema/util/spring-util.xsd">
<beans:bean id="x509PrincipalExtractor" class="org.springframework.security.web.authentication.preauth.x509.SubjectDnX509PrincipalExtractor">
<beans:property name="subjectDnRegex" value="CN=(.*?)(?:,|$)"/>
</beans:bean>
<beans:bean id="timelyUserDetailsService" class="timely.auth.UserDetailsService">
<beans:property name="users">
<util:list>
<beans:bean class="timely.auth.TimelyUser">
<beans:constructor-arg name="subjectDn">
<beans:value>CN=example.com</beans:value>
</beans:constructor-arg>
<beans:constructor-arg name="auths">
<util:list>
<beans:value>D</beans:value>
<beans:value>E</beans:value>
<beans:value>F</beans:value>
</util:list>
</beans:constructor-arg>
<beans:constructor-arg name="roles">
<util:list>
<beans:value>G</beans:value>
<beans:value>H</beans:value>
<beans:value>I</beans:value>
</util:list>
</beans:constructor-arg>
</beans:bean>
</util:list>
</beans:property>
</beans:bean>
<beans:bean id="x509AuthorizationProvider" class="org.springframework.security.web.authentication.preauth.PreAuthenticatedAuthenticationProvider">
<beans:property name="preAuthenticatedUserDetailsService" ref="timelyUserDetailsService" />
</beans:bean>
<user-service id="usernamePasswordService">
<user name="test" password="test1" authorities="A,B,C" />
</user-service>
<beans:bean id="noOpPasswordEncoder" class="org.springframework.security.crypto.password.NoOpPasswordEncoder"/>
<beans:bean id="UserNamePasswordAuthenticationProvider" class="org.springframework.security.authentication.dao.DaoAuthenticationProvider">
<beans:property name="userDetailsService" ref="usernamePasswordService"/>
<beans:property name="passwordEncoder" ref="noOpPasswordEncoder"/>
</beans:bean>
<authentication-manager alias="authenticationManager">
<!-- X509 PreAuthenticated authorization provider -->
<authentication-provider ref="x509AuthorizationProvider"/>
<!-- Basic authorization provider, validates provided username and password -->
<authentication-provider ref="UserNamePasswordAuthenticationProvider" />
</authentication-manager>
<util:list id="requiredRoles" value-type="java.lang.String">
<beans:value></beans:value>
</util:list>
<util:list id="requiredAuths" value-type="java.lang.String">
<beans:value></beans:value>
</util:list>
</beans:beans>
+142 -121
View File
@@ -1,128 +1,149 @@
timely-balancer:
zooKeeper:
servers: localhost:2181
timely:
instance: ${INSTANCE}
zookeeper:
servers: ${ACCUMULO_ZOOKEEPERS}
timeout: 30s
checkServerHealthInterval: 10000
serverFailuresBeforeDown: 6
serverSuccessesBeforeUp: 1
assignmentFile: /timely/balancer/assignments.csv
security:
session-max-age: 86400
allow-anonymous-http-access: true
allow-anonymous-ws-access: true
serverSsl:
certificate-file:
key-file:
#key-password:
use-generated-keypair: true
trust-store-file:
use-openssl: true
#use-ciphers:
clientSsl:
keyFile:
keyType: pkcs12
#keyPassword:
trustStoreFile:
trustStoreType: jks
#trustStorePassword:
cache-expiration-minutes: 60
cache-refresh-minutes: 5
authorization-url: ${AUTHORIZATION_URL}
allow-anonymous-http-access: ${ANONYMOUS_HTTP:true}
allow-anonymous-ws-access: ${ANONYMOUS_WEBSOCKET:true}
server-ssl:
key-store-type: ${KEY_STORE_TYPE}
key-store-file: ${KEY_STORE_FILE}
key-store-password: ${KEY_STORE_PASSWORD}
use-generated-keypair: ${GENERATE_KEYS}
ignore-ssl-handshake-errors: true
trust-store-type: ${TRUST_STORE_TYPE}
trust-store-file: ${TRUST_STORE_FILE}
trust-store-password: ${TRUST_STORE_PASSWORD}
client-ssl:
key-store-type: ${KEY_STORE_TYPE}
key-store-file: ${KEY_STORE_FILE}
key-store-password: ${KEY_STORE_PASSWORD}
trust-store-type: ${TRUST_STORE_TYPE}
trust-store-file: ${TRUST_STORE_FILE}
trust-store-password: ${TRUST_STORE_PASSWORD}
hostVerificationEnabled: ${HOST_VERIFICATION_ENABLED:true}
jwt-ssl:
key-store-type: ${AUTHORIZATION_CERTIFICATE_TYPE}
key-store-file: ${AUTHORIZATION_CERTIFICATE_FILE}
key-store-password: ${AUTHORIZATION_CERTIFICATE_PASSWORD}
cache:
enabled: false
metric-age-off-hours:
default: 26
nonCachedMetrics:
server:
ip: 127.0.0.1
tcpPort: 4242
numTcpPools: 1
tcpBufferSize: 100
tcpClientPool:
minIdlePerKey: 0
maxIdlePerKey: 40
maxTotalPerKey: 40
maxTotal: -1
lifo: true
fairness: false
maxWaitMillis: -1
minEvictableIdleTimeMillis: 1800000
evictorShutdownTimeoutMillis: 10000
softMinEvictableIdleTimeMillis: -1
numTestsPerEvictionRun: 3
timeBetweenEvictionRunsMillis: -1
blockWhenExhausted: true
jmxEnabled: true
testOnBorrow: false
testOnReturn: false
udpPort: 4245
udpClientPool:
minIdlePerKey: 0
maxIdlePerKey: 40
maxTotalPerKey: 40
maxTotal: -1
lifo: true
fairness: false
maxWaitMillis: -1
minEvictableIdleTimeMillis: 1800000
evictorShutdownTimeoutMillis: 10000
softMinEvictableIdleTimeMillis: -1
numTestsPerEvictionRun: 3
timeBetweenEvictionRunsMillis: -1
blockWhenExhausted: true
jmxEnabled: true
testOnBorrow: false
testOnReturn: false
http:
host: localhost
ip: 127.0.0.1
port: 4243
redirect-path: /secure-me
strict-transport-max-age: 608400
cors:
allow-any-origin: false
allow-null-origin: false
allow-credentials: true
allowed-origins:
allowed-methods: DELETE,GET,HEAD,OPTIONS,PUT,POST
allowed-headers: content-type
httpClientPool:
minIdlePerKey: 0
maxIdlePerKey: 40
maxTotalPerKey: 40
maxTotal: -1
lifo: true
fairness: false
maxWaitMillis: -1
minEvictableIdleTimeMillis: 1800000
evictorShutdownTimeoutMillis: 10000
softMinEvictableIdleTimeMillis: -1
numTestsPerEvictionRun: 3
timeBetweenEvictionRunsMillis: -1
blockWhenExhausted: true
jmxEnabled: true
testOnBorrow: false
testOnReturn: false
websocket:
ip: 127.0.0.1
port: 5244
timeout: 60
subscription-lag: 120
scanner-batch-size: 10000
flush-interval-seconds: 30
scanner-read-ahead: 1
subscription-batch-size: 5000
wsClientPool:
minIdlePerKey: 0
maxIdlePerKey: 40
maxTotalPerKey: 40
maxTotal: -1
lifo: true
fairness: false
maxWaitMillis: -1
minEvictableIdleTimeMillis: 1800000
evictorShutdownTimeoutMillis: 10000
softMinEvictableIdleTimeMillis: -1
numTestsPerEvictionRun: 3
timeBetweenEvictionRunsMillis: -1
blockWhenExhausted: true
jmxEnabled: true
testOnBorrow: false
testOnReturn: false
non-cached-metrics:
balancer:
check-server-health-interval: 10000
server-failures-before-down: 6
server-successes-before-up: 1
default-fs: ${HADOOP_DEFAULT_FS}
fs-config-resources:
- ${HADOOP_CONF_DIR}/core-site.xml
- ${HADOOP_CONF_DIR}/hdfs-site.xml
assignment-file: hdfs:///timely/balancer/assignments.csv
server:
ip: ${HOST_IP}
tcp-base-port: ${TCP_BASE_PORT}
num-tcp-pools: 1
tcp-buffer-size: 100
tcp-client-pool:
min-idle-per-key: 0
max-idle-per-key: 40
max-total-per-key: 40
max-total: -1
lifo: true
fairness: false
max-wait-millis: -1
min-evictable-idle-time-millis: 1800000
evictor-shutdown-timeout-millis: 10000
soft-min-evictable-idle-time-millis: -1
num-tests-per-eviction-run: 3
time-between-eviction-runs-millis: -1
block-when-exhausted: true
jmx-enabled: true
test-on-borrow: false
test-on-return: false
udp-base-port: ${UDP_BASE_PORT}
udp-client-pool:
min-idle-per-key: 0
max-idle-per-key: 40
max-total-per-key: 40
max-total: -1
lifo: true
fairness: false
max-wait-millis: -1
min-evictable-idle-time-millis: 1800000
evictor-shutdown-timeout-millis: 10000
soft-min-evictable-idle-time-millis: -1
num-tests-per-eviction-run: 3
time-between-eviction-runs-millis: -1
block-when-exhausted: true
jmx-enabled: true
test-on-borrow: false
test-on-return: false
http:
host: ${HOST}
ip: ${HOST_IP}
base-port: ${HTTP_BASE_PORT}
redirect-path: /secure-me
strict-transport-max-age: 604800
cors:
allow-any-origin: false
allow-null-origin: false
allow-credentials: true
allowed-origins:
allowed-methods: DELETE,GET,HEAD,OPTIONS,PUT,POST
allowed-headers: content-type
httpClientPool:
min-idle-per-key: 0
max-idle-per-key: 40
max-total-per-key: 40
max-total: -1
lifo: true
fairness: false
max-wait-millis: -1
min-evictable-idle-time-millis: 1800000
evictor-shutdown-timeout-millis: 10000
soft-min-evictable-idle-time-millis: -1
num-tests-per-eviction-run: 3
time-between-eviction-runs-millis: -1
block-when-exhausted: true
jmx-enabled: true
test-on-borrow: false
test-on-return: false
websocket:
ip: ${HOST_IP}
base-port: ${WEBSOCKET_BASE_PORT}
timeout: 60
subscription-lag: 120
scanner-batch-size: 10000
flush-interval-seconds: 30
scanner-read-ahead: 1
subscription-batch-size: 5000
incoming-buffer-size: 50000000
ws-client-pool:
min-idle-per-key: 0
max-idle-per-key: 40
max-total-per-key: 40
max-total: -1
lifo: true
fairness: false
max-wait-millis: -1
min-evictable-idle-time-millis: 1800000
evictor-shutdown-timeout-millis: 10000
soft-min-evictable-idle-time-millis: -1
num-tests-per-eviction-run: 3
time-between-eviction-runs-millis: -1
block-when-exhausted: true
jmx-enabled: true
test-on-borrow: false
test-on-return: false
logging:
level:
ROOT: info
timely: debug
+93 -68
View File
@@ -4,9 +4,13 @@
<parent>
<groupId>timely</groupId>
<artifactId>timely</artifactId>
<version>1.2.1-SNAPSHOT</version>
<version>2.1.5</version>
</parent>
<artifactId>timely-balancer</artifactId>
<description>Timely Balancer</description>
<properties>
<start-class>timely.balancer.TimelyBalancer</start-class>
</properties>
<dependencies>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
@@ -37,12 +41,8 @@
<artifactId>commons-beanutils</artifactId>
</dependency>
<dependency>
<groupId>io.github.lukehutch</groupId>
<artifactId>fast-classpath-scanner</artifactId>
</dependency>
<dependency>
<groupId>io.netty</groupId>
<artifactId>netty-all</artifactId>
<groupId>io.github.classgraph</groupId>
<artifactId>classgraph</artifactId>
</dependency>
<dependency>
<groupId>net.sourceforge.javacsv</groupId>
@@ -51,16 +51,6 @@
<dependency>
<groupId>org.apache.accumulo</groupId>
<artifactId>accumulo-core</artifactId>
<exclusions>
<exclusion>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.apache.accumulo</groupId>
<artifactId>accumulo-minicluster</artifactId>
</dependency>
<dependency>
<groupId>org.apache.commons</groupId>
@@ -97,21 +87,19 @@
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<exclusions>
<exclusion>
<groupId>jdk.tools</groupId>
<artifactId>jdk.tools</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-minicluster</artifactId>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-1.2-api</artifactId>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-webflux</artifactId>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-websocket</artifactId>
@@ -134,6 +122,18 @@
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-validation</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-context</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-bootstrap</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.security</groupId>
<artifactId>spring-security-config</artifactId>
@@ -150,10 +150,6 @@
<groupId>timely</groupId>
<artifactId>timely-common</artifactId>
</dependency>
<dependency>
<groupId>timely</groupId>
<artifactId>timely-server</artifactId>
</dependency>
<dependency>
<groupId>com.lmax</groupId>
<artifactId>disruptor</artifactId>
@@ -161,9 +157,31 @@
</dependency>
<dependency>
<groupId>io.netty</groupId>
<artifactId>netty-tcnative</artifactId>
<version>${version.netty-tcnative}</version>
<classifier>${os.detected.classifier}</classifier>
<artifactId>netty-tcnative-boringssl-static</artifactId>
<classifier>linux-aarch_64</classifier>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>io.netty</groupId>
<artifactId>netty-tcnative-boringssl-static</artifactId>
<classifier>linux-x86_64</classifier>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>io.netty</groupId>
<artifactId>netty-transport-native-epoll</artifactId>
<classifier>linux-aarch_64</classifier>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>io.netty</groupId>
<artifactId>netty-transport-native-epoll</artifactId>
<classifier>linux-x86_64</classifier>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>javax.activation</groupId>
<artifactId>activation</artifactId>
<scope>runtime</scope>
</dependency>
<dependency>
@@ -176,11 +194,6 @@
<artifactId>log4j-core</artifactId>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-slf4j-impl</artifactId>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>org.javassist</groupId>
<artifactId>javassist</artifactId>
@@ -191,19 +204,38 @@
<artifactId>junit</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-minicluster</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.jsoup</groupId>
<artifactId>jsoup</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-test-autoconfigure</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>timely</groupId>
<artifactId>timely-test</artifactId>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-assembly-plugin</artifactId>
@@ -247,9 +279,8 @@
<artifactItems>
<artifactItem>
<groupId>io.netty</groupId>
<artifactId>netty-tcnative</artifactId>
<artifactId>netty-tcnative-boringssl-static</artifactId>
<version>${version.netty-tcnative}</version>
<classifier>${os.detected.classifier}</classifier>
<type>jar</type>
<overWrite>false</overWrite>
<outputDirectory>${project.build.directory}/native</outputDirectory>
@@ -264,6 +295,20 @@
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-failsafe-plugin</artifactId>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
<version>3.1.2</version>
<configuration>
<archive>
<manifest>
<!--suppress UnresolvedMavenProperty, MavenModelInspection -->
<mainClass>${start-class}</mainClass>
<addDefaultImplementationEntries>true</addDefaultImplementationEntries>
</manifest>
</archive>
</configuration>
</plugin>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>build-helper-maven-plugin</artifactId>
@@ -293,35 +338,15 @@
</build>
<profiles>
<profile>
<id>native-epoll-linux</id>
<activation>
<os>
<family>!mac</family>
</os>
</activation>
<dependencies>
<dependency>
<groupId>io.netty</groupId>
<artifactId>netty-transport-native-epoll</artifactId>
<version>${version.netty}</version>
<classifier>linux-x86_64</classifier>
</dependency>
</dependencies>
</profile>
<profile>
<id>native-epoll</id>
<activation>
<os>
<family>mac</family>
</os>
</activation>
<dependencies>
<dependency>
<groupId>io.netty</groupId>
<artifactId>netty-transport-native-epoll</artifactId>
<version>${version.netty}</version>
</dependency>
</dependencies>
<id>docker</id>
<build>
<plugins>
<plugin>
<groupId>com.spotify</groupId>
<artifactId>docker-maven-plugin</artifactId>
</plugin>
</plugins>
</build>
</profile>
</profiles>
</project>
+14 -11
View File
@@ -1,4 +1,4 @@
<assembly xmlns="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.3"
<assembly xmlns="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.3"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.3 http://maven.apache.org/xsd/assembly-1.1.3.xsd">
<id>dist</id>
@@ -15,20 +15,23 @@
<include>NOTICE*</include>
</includes>
</fileSet>
<fileSet>
<directory>${project.build.directory}</directory>
<outputDirectory>lib</outputDirectory>
<includes>
<include>timely*.jar</include>
</includes>
</fileSet>
<fileSet>
<directory>${project.build.directory}/lib</directory>
<outputDirectory>lib</outputDirectory>
<includes>
<include>*.jar</include>
<include>netty-tcnative-boringssl-static*linux-x86_64.jar</include>
<include>netty-tcnative-boringssl-static*linux-aarch_64.jar</include>
<include>netty-transport-native-epoll*linux-x86_64.jar</include>
<include>netty-transport-native-epoll*linux-aarch_64.jar</include>
</includes>
</fileSet>
</fileSet>
<fileSet>
<directory>${project.build.directory}</directory>
<outputDirectory>bin</outputDirectory>
<includes>
<include>timely-balancer-*-exec.jar</include>
</includes>
</fileSet>
<fileSet>
<directory>${project.basedir}/bin</directory>
<outputDirectory>bin</outputDirectory>
@@ -51,4 +54,4 @@
</includes>
</fileSet>
</fileSets>
</assembly>
</assembly>
+11
View File
@@ -0,0 +1,11 @@
FROM azul/zulu-openjdk-alpine:11
LABEL version=${project.version} \
run="docker run ${docker.image.prefix}${project.artifactId}:latest" \
description="${project.description}"
ADD ${project.build.finalName}-exec.jar /app.jar
RUN apk add libc6-compat curl
EXPOSE 54322 54323 54324 54325
ENTRYPOINT ["java","-jar","app.jar"]
@@ -1,7 +1,7 @@
package timely.balancer;
import java.util.Timer;
import java.util.TimerTask;
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicLong;
import com.google.common.util.concurrent.AtomicDouble;
@@ -20,38 +20,26 @@ public class ArrivalRate {
private AtomicDouble longTermRate = new AtomicDouble(0);
private AtomicLong longTermLastReset = new AtomicLong(0);
public ArrivalRate(Timer timer) {
public ArrivalRate(ScheduledExecutorService arrivalRateExecutor) {
long now = System.currentTimeMillis();
shortTermLastReset.set(now);
mediumTermLastReset.set(now);
longTermLastReset.set(now);
// 2 minute
timer.schedule(new TimerTask() {
@Override
public void run() {
resetShort();
}
}, 120000, 120000);
arrivalRateExecutor.scheduleAtFixedRate(() -> {
resetShort();
}, 2, 2, TimeUnit.MINUTES);
// 5 minute
timer.schedule(new TimerTask() {
@Override
public void run() {
resetMedium();
}
}, 300000, 300000);
arrivalRateExecutor.scheduleAtFixedRate(() -> {
resetMedium();
}, 5, 5, TimeUnit.MINUTES);
// 15 minute
timer.schedule(new TimerTask() {
@Override
public void run() {
resetLong();
}
}, 900000, 900000);
arrivalRateExecutor.scheduleAtFixedRate(() -> {
resetLong();
}, 15, 15, TimeUnit.MINUTES);
}
private void resetShort() {
@@ -1,21 +1,34 @@
package timely.balancer;
import static timely.Server.SERVICE_DISCOVERY_PATH;
import static timely.store.cache.DataStoreCache.NON_CACHED_METRICS;
import static timely.store.cache.DataStoreCache.NON_CACHED_METRICS_LOCK_PATH;
import static org.apache.accumulo.core.conf.ConfigurationTypeHelper.getTimeInMillis;
import static timely.Constants.NON_CACHED_METRICS;
import static timely.Constants.NON_CACHED_METRICS_LOCK_PATH;
import static timely.Constants.SERVICE_DISCOVERY_PATH;
import java.io.File;
import java.net.InetSocketAddress;
import java.nio.charset.StandardCharsets;
import java.util.ArrayList;
import java.util.Date;
import java.util.Arrays;
import java.util.List;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.Future;
import java.util.concurrent.TimeUnit;
import java.util.stream.Collectors;
import javax.net.ssl.SSLContext;
import org.apache.commons.lang3.StringUtils;
import org.apache.curator.RetryPolicy;
import org.apache.curator.framework.CuratorFramework;
import org.apache.curator.framework.CuratorFrameworkFactory;
import org.apache.curator.retry.RetryForever;
import org.apache.zookeeper.CreateMode;
import org.apache.zookeeper.data.Stat;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.boot.SpringApplication;
import org.springframework.context.ApplicationContext;
import io.netty.bootstrap.AbstractBootstrap;
import io.netty.bootstrap.Bootstrap;
import io.netty.bootstrap.ServerBootstrap;
import io.netty.buffer.PooledByteBufAllocator;
@@ -45,45 +58,31 @@ import io.netty.handler.codec.http.HttpResponseEncoder;
import io.netty.handler.codec.http.HttpServerCodec;
import io.netty.handler.codec.http.websocketx.WebSocketServerProtocolHandler;
import io.netty.handler.logging.LoggingHandler;
import io.netty.handler.ssl.ClientAuth;
import io.netty.handler.ssl.OpenSslServerContext;
import io.netty.handler.ssl.OpenSslServerSessionContext;
import io.netty.handler.ssl.SslContext;
import io.netty.handler.ssl.SslContextBuilder;
import io.netty.handler.ssl.SslProvider;
import io.netty.handler.ssl.util.SelfSignedCertificate;
import io.netty.handler.stream.ChunkedWriteHandler;
import io.netty.handler.timeout.IdleStateHandler;
import io.netty.util.NettyRuntime;
import io.netty.util.internal.SystemPropertyUtil;
import org.apache.curator.RetryPolicy;
import org.apache.curator.framework.CuratorFramework;
import org.apache.curator.framework.CuratorFrameworkFactory;
import org.apache.curator.retry.RetryForever;
import org.apache.zookeeper.CreateMode;
import org.apache.zookeeper.data.Stat;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.boot.WebApplicationType;
import org.springframework.boot.builder.SpringApplicationBuilder;
import org.springframework.context.ConfigurableApplicationContext;
import timely.auth.AuthCache;
import timely.balancer.configuration.BalancerConfiguration;
import timely.balancer.configuration.ClientSsl;
import timely.balancer.configuration.SpringBootstrap;
import timely.auth.JWTTokenHandler;
import timely.balancer.configuration.BalancerHttpProperties;
import timely.balancer.configuration.BalancerProperties;
import timely.balancer.configuration.BalancerServerProperties;
import timely.balancer.configuration.BalancerWebsocketProperties;
import timely.balancer.connection.http.HttpClientPool;
import timely.balancer.connection.tcp.TcpClientPool;
import timely.balancer.connection.udp.UdpClientPool;
import timely.balancer.connection.ws.WsClientPool;
import timely.balancer.healthcheck.HealthChecker;
import timely.balancer.netty.http.HttpRelayHandler;
import timely.balancer.netty.tcp.TcpRelayHandler;
import timely.balancer.netty.udp.UdpRelayHandler;
import timely.balancer.netty.ws.WsRelayHandler;
import timely.balancer.resolver.BalancedMetricResolver;
import timely.balancer.resolver.MetricResolver;
import timely.client.http.HttpClient;
import timely.configuration.ServerSsl;
import timely.common.component.AuthenticationService;
import timely.common.configuration.SecurityProperties;
import timely.common.configuration.SslClientProperties;
import timely.common.configuration.SslServerProperties;
import timely.common.configuration.ZookeeperProperties;
import timely.netty.http.HttpStaticFileServerHandler;
import timely.netty.http.NonSslRedirectHandler;
import timely.netty.http.TimelyExceptionHandler;
@@ -103,17 +102,25 @@ public class Balancer {
final public static String ASSIGNMENTS_LOCK_PATH = "/timely/balancer/assignments/lock";
final public static String ASSIGNMENTS_LAST_UPDATED_PATH = "/timely/balancer/assignments/lastUpdated";
private static final Logger LOG = LoggerFactory.getLogger(Balancer.class);
private static final int EPOLL_MIN_MAJOR_VERSION = 2;
private static final int EPOLL_MIN_MINOR_VERSION = 6;
private static final int EPOLL_MIN_PATCH_VERSION = 32;
private static final Logger log = LoggerFactory.getLogger(Balancer.class);
private static final String EPOLL_MIN_VERSION = "2.7";
private static final String OS_NAME = "os.name";
private static final String OS_VERSION = "os.version";
private static final String WS_PATH = "/websocket";
protected static final CountDownLatch LATCH = new CountDownLatch(1);
static ConfigurableApplicationContext applicationContext;
private final BalancerConfiguration balancerConfig;
private BalancerProperties balancerProperties;
private SecurityProperties securityProperties;
private SslServerProperties sslServerProperties;
private SslClientProperties sslClientProperties;
private ZookeeperProperties zookeeperProperties;
private BalancerServerProperties balancerServerProperties;
private BalancerHttpProperties balancerHttpProperties;
private BalancerWebsocketProperties balancerWebsocketProperties;
private ApplicationContext applicationContext;
private AuthenticationService authenticationService;
private CuratorFramework curatorFramework;
private int quietPeriod;
private SslContext serverSslContext;
private SSLContext clientSSLContext;
private EventLoopGroup tcpWorkerGroup = null;
private EventLoopGroup tcpBossGroup = null;
private EventLoopGroup httpWorkerGroup = null;
@@ -127,57 +134,200 @@ public class Balancer {
protected Channel httpChannelHandle = null;
protected Channel wsChannelHandle = null;
protected List<Channel> udpChannelHandleList = new ArrayList<>();
private CuratorFramework curatorFramework;
protected volatile boolean shutdown = false;
private static final int DEFAULT_EVENT_LOOP_THREADS;
protected int DEFAULT_EVENT_LOOP_THREADS;
private String[] zkPaths = new String[] { LEADER_LATCH_PATH, ASSIGNMENTS_LAST_UPDATED_PATH, ASSIGNMENTS_LOCK_PATH,
SERVICE_DISCOVERY_PATH, NON_CACHED_METRICS, NON_CACHED_METRICS_LOCK_PATH };
private String[] zkPaths = new String[] {LEADER_LATCH_PATH, ASSIGNMENTS_LAST_UPDATED_PATH, ASSIGNMENTS_LOCK_PATH, SERVICE_DISCOVERY_PATH,
NON_CACHED_METRICS, NON_CACHED_METRICS_LOCK_PATH};
static {
DEFAULT_EVENT_LOOP_THREADS = Math.max(1,
SystemPropertyUtil.getInt("io.netty.eventLoopThreads", NettyRuntime.availableProcessors() * 2));
}
public Balancer(BalancerConfiguration balancerConf) throws Exception {
this.balancerConfig = balancerConf;
}
protected static ConfigurableApplicationContext initializeConfiguration(String[] args) {
return new SpringApplicationBuilder(SpringBootstrap.class).web(WebApplicationType.NONE).run(args);
}
private void shutdownHook() {
final Runnable shutdownRunner = () -> {
if (!shutdown) {
shutdown();
private static boolean useEpoll() {
final String osName = SystemPropertyUtil.get(OS_NAME).toLowerCase().trim();
final String osVersion = SystemPropertyUtil.get(OS_VERSION).toLowerCase();
// split at periods, keep the first two, join with a period
final String majMinVers = StringUtils.join(Arrays.stream(osVersion.split("\\.")).limit(2).collect(Collectors.toList()), ".");
if (osName.startsWith("linux")) {
try {
Runtime.Version currentVersion = Runtime.Version.parse(majMinVers);
Runtime.Version epollMinVersion = Runtime.Version.parse(EPOLL_MIN_VERSION);
return currentVersion.compareTo(epollMinVersion) > 0;
} catch (Exception e) {
log.error(e.getMessage(), e);
}
};
final Thread hook = new Thread(shutdownRunner, "shutdown-hook-thread");
Runtime.getRuntime().addShutdownHook(hook);
}
return false;
}
public Balancer(ApplicationContext applicationContext, SslContext serverSslContext, SSLContext clientSSLContext,
AuthenticationService authenticationService, MetricResolver metricResolver, BalancerProperties balancerProperties,
ZookeeperProperties zookeeperProperties, SecurityProperties securityProperties, SslServerProperties sslServerProperties,
SslClientProperties sslClientProperties, BalancerServerProperties balancerServerProperties, BalancerHttpProperties balancerHttpProperties,
BalancerWebsocketProperties balancerWebsocketProperties) {
this.serverSslContext = serverSslContext;
this.clientSSLContext = clientSSLContext;
this.quietPeriod = balancerServerProperties.getShutdownQuietPeriod();
this.applicationContext = applicationContext;
this.authenticationService = authenticationService;
this.metricResolver = metricResolver;
this.balancerProperties = balancerProperties;
this.zookeeperProperties = zookeeperProperties;
this.securityProperties = securityProperties;
this.sslServerProperties = sslServerProperties;
this.sslClientProperties = sslClientProperties;
this.balancerServerProperties = balancerServerProperties;
this.balancerHttpProperties = balancerHttpProperties;
this.balancerWebsocketProperties = balancerWebsocketProperties;
DEFAULT_EVENT_LOOP_THREADS = Math.max(1, SystemPropertyUtil.getInt("io.netty.eventLoopThreads", NettyRuntime.availableProcessors() * 2));
}
public void start() {
log.info("Starting {}", this.getClass().getSimpleName());
try {
// start curator framework
RetryPolicy retryPolicy = new RetryForever(1000);
int timeout = Long.valueOf(getTimeInMillis(zookeeperProperties.getTimeout())).intValue();
curatorFramework = CuratorFrameworkFactory.newClient(zookeeperProperties.getServers(), timeout, 10000, retryPolicy);
curatorFramework.start();
ensureZkPaths(curatorFramework, zkPaths);
JWTTokenHandler.init(securityProperties, null);
final boolean useEpoll = useEpoll();
Class<? extends ServerSocketChannel> channelClass;
Class<? extends Channel> datagramChannelClass;
if (useEpoll) {
tcpWorkerGroup = new EpollEventLoopGroup();
tcpBossGroup = new EpollEventLoopGroup();
httpWorkerGroup = new EpollEventLoopGroup();
httpBossGroup = new EpollEventLoopGroup();
wsWorkerGroup = new EpollEventLoopGroup();
wsBossGroup = new EpollEventLoopGroup();
udpWorkerGroup = new EpollEventLoopGroup();
udpBossGroup = new EpollEventLoopGroup();
channelClass = EpollServerSocketChannel.class;
datagramChannelClass = EpollDatagramChannel.class;
} else {
tcpWorkerGroup = new NioEventLoopGroup();
tcpBossGroup = new NioEventLoopGroup();
httpWorkerGroup = new NioEventLoopGroup();
httpBossGroup = new NioEventLoopGroup();
wsWorkerGroup = new NioEventLoopGroup();
wsBossGroup = new NioEventLoopGroup();
udpWorkerGroup = new NioEventLoopGroup();
udpBossGroup = new NioEventLoopGroup();
channelClass = NioServerSocketChannel.class;
datagramChannelClass = NioDatagramChannel.class;
}
log.info("Using channel class {}", channelClass.getSimpleName());
List<TcpClientPool> tcpClientPools = new ArrayList<>();
for (int x = 0; x < balancerServerProperties.getNumTcpPools(); x++) {
tcpClientPools.add(new TcpClientPool(balancerServerProperties));
}
final ServerBootstrap tcpServer = new ServerBootstrap();
tcpServer.group(tcpBossGroup, tcpWorkerGroup);
tcpServer.channel(channelClass);
tcpServer.handler(new LoggingHandler());
tcpServer.childHandler(setupTcpChannel(metricResolver, tcpClientPools));
tcpServer.option(ChannelOption.ALLOCATOR, PooledByteBufAllocator.DEFAULT);
tcpServer.option(ChannelOption.SO_BACKLOG, 128);
tcpServer.childOption(ChannelOption.SO_KEEPALIVE, true);
final int tcpPort = balancerServerProperties.getTcpPort();
final String tcpIp = balancerServerProperties.getIp();
tcpChannelHandle = bind(tcpServer, tcpIp, tcpPort);
final String tcpAddress = ((InetSocketAddress) tcpChannelHandle.localAddress()).getAddress().getHostAddress();
final int httpPort = balancerHttpProperties.getPort();
final String httpIp = balancerHttpProperties.getIp();
if (serverSslContext instanceof OpenSslServerContext) {
OpenSslServerContext openssl = (OpenSslServerContext) serverSslContext;
String application = "Timely_" + httpPort;
OpenSslServerSessionContext opensslCtx = openssl.sessionContext();
opensslCtx.setSessionCacheEnabled(true);
opensslCtx.setSessionCacheSize(128);
opensslCtx.setSessionIdContext(application.getBytes(StandardCharsets.UTF_8));
opensslCtx.setSessionTimeout(securityProperties.getSessionMaxAge());
}
final ServerBootstrap httpServer = new ServerBootstrap();
httpServer.group(httpBossGroup, httpWorkerGroup);
httpServer.channel(channelClass);
httpServer.handler(new LoggingHandler());
HttpClientPool httpClientPool = new HttpClientPool(balancerHttpProperties, clientSSLContext, sslClientProperties);
httpServer.childHandler(setupHttpChannel(balancerHttpProperties, sslServerProperties, serverSslContext, metricResolver, httpClientPool));
httpServer.option(ChannelOption.ALLOCATOR, PooledByteBufAllocator.DEFAULT);
httpServer.option(ChannelOption.SO_BACKLOG, 128);
httpServer.childOption(ChannelOption.SO_KEEPALIVE, true);
httpChannelHandle = bind(httpServer, httpIp, httpPort);
final String httpAddress = ((InetSocketAddress) httpChannelHandle.localAddress()).getAddress().getHostAddress();
final int wsPort = balancerWebsocketProperties.getPort();
final String wsIp = balancerWebsocketProperties.getIp();
final ServerBootstrap wsServer = new ServerBootstrap();
wsServer.group(wsBossGroup, wsWorkerGroup);
wsServer.channel(channelClass);
wsServer.handler(new LoggingHandler());
WsClientPool wsClientPool = new WsClientPool(balancerProperties, balancerWebsocketProperties, sslClientProperties, clientSSLContext);
wsServer.childHandler(setupWSChannel(balancerWebsocketProperties, securityProperties, serverSslContext, metricResolver, wsClientPool));
wsServer.option(ChannelOption.ALLOCATOR, PooledByteBufAllocator.DEFAULT);
wsServer.option(ChannelOption.SO_BACKLOG, 128);
wsServer.childOption(ChannelOption.SO_KEEPALIVE, true);
wsChannelHandle = bind(wsServer, wsIp, wsPort);
final String wsAddress = ((InetSocketAddress) wsChannelHandle.localAddress()).getAddress().getHostAddress();
final int udpPort = balancerServerProperties.getUdpPort();
final String udpIp = balancerServerProperties.getIp();
UdpClientPool udpClientPool = new UdpClientPool(balancerServerProperties);
for (int n = 0; n < DEFAULT_EVENT_LOOP_THREADS; n++) {
final Bootstrap udpServer = new Bootstrap();
udpServer.group(udpBossGroup);
udpServer.channel(datagramChannelClass);
udpServer.handler(setupUdpChannel(metricResolver, udpClientPool));
udpServer.option(ChannelOption.ALLOCATOR, PooledByteBufAllocator.DEFAULT);
udpServer.option(EpollChannelOption.SO_REUSEADDR, true);
udpServer.option(EpollChannelOption.SO_REUSEPORT, true);
udpChannelHandleList.add(bind(udpServer, udpIp, udpPort));
}
log.info("Balancer started. Listening on {}:{} for TCP traffic, {}:{} for HTTP traffic, {}:{} for WebSocket traffic, and {}:{} for UDP traffic",
tcpAddress, tcpPort, httpAddress, httpPort, wsAddress, wsPort, wsAddress, udpPort);
} catch (Exception e) {
log.error(e.getMessage(), e);
SpringApplication.exit(applicationContext, () -> 0);
}
}
protected Channel bind(AbstractBootstrap server, String ip, int port) {
Channel channel = null;
try {
channel = server.bind(ip, port).sync().channel();
} catch (InterruptedException e) {
log.error("Failed to bind to {}:{} -- {}", ip, port, e.getMessage());
}
if (channel == null) {
throw new IllegalStateException("Failed to bind to port:" + ip + ":" + port);
}
return channel;
}
public void shutdown() {
List<ChannelFuture> channelFutures = new ArrayList<>();
if (tcpChannelHandle != null) {
LOG.info("Closing tcpChannelHandle");
log.info("Closing tcpChannelHandle");
channelFutures.add(tcpChannelHandle.close());
}
if (httpChannelHandle != null) {
LOG.info("Closing httpChannelHandle");
log.info("Closing httpChannelHandle");
channelFutures.add(httpChannelHandle.close());
}
if (wsChannelHandle != null) {
LOG.info("Closing wsChannelHandle");
log.info("Closing wsChannelHandle");
channelFutures.add(wsChannelHandle.close());
}
int udpChannel = 1;
for (Channel c : udpChannelHandleList) {
LOG.info("Closing udpChannelHandle");
log.info("Closing udpChannelHandle #" + udpChannel++);
channelFutures.add(c.close());
}
@@ -186,51 +336,49 @@ public class Balancer {
try {
f.get();
} catch (final Exception e) {
LOG.error("Error while shutting down channel: {}", f.channel().config());
LOG.error("{}", e.getMessage());
e.printStackTrace();
log.error("Channel:" + f.channel().config() + " -> " + e.getMessage(), e);
}
});
Integer quietPeriod = balancerConfig.getServer().getShutdownQuietPeriod();
List<Future<?>> groupFutures = new ArrayList<>();
if (tcpBossGroup != null) {
LOG.info("Shutting down tcpBossGroup");
log.info("Shutting down tcpBossGroup");
groupFutures.add(tcpBossGroup.shutdownGracefully(quietPeriod, 10, TimeUnit.SECONDS));
}
if (tcpWorkerGroup != null) {
LOG.info("Shutting down tcpWorkerGroup");
log.info("Shutting down tcpWorkerGroup");
groupFutures.add(tcpWorkerGroup.shutdownGracefully(quietPeriod, 10, TimeUnit.SECONDS));
}
if (httpBossGroup != null) {
LOG.info("Shutting down httpBossGroup");
log.info("Shutting down httpBossGroup");
groupFutures.add(httpBossGroup.shutdownGracefully(quietPeriod, 10, TimeUnit.SECONDS));
}
if (httpWorkerGroup != null) {
LOG.info("Shutting down httpWorkerGroup");
log.info("Shutting down httpWorkerGroup");
groupFutures.add(httpWorkerGroup.shutdownGracefully(quietPeriod, 10, TimeUnit.SECONDS));
}
if (wsBossGroup != null) {
LOG.info("Shutting down wsBossGroup");
log.info("Shutting down wsBossGroup");
groupFutures.add(wsBossGroup.shutdownGracefully(quietPeriod, 10, TimeUnit.SECONDS));
}
if (wsWorkerGroup != null) {
LOG.info("Shutting down wsWorkerGroup");
log.info("Shutting down wsWorkerGroup");
groupFutures.add(wsWorkerGroup.shutdownGracefully(quietPeriod, 10, TimeUnit.SECONDS));
}
if (udpBossGroup != null) {
LOG.info("Shutting down udpBossGroup");
log.info("Shutting down udpBossGroup");
groupFutures.add(udpBossGroup.shutdownGracefully(quietPeriod, 10, TimeUnit.SECONDS));
}
if (udpWorkerGroup != null) {
LOG.info("Shutting down udpWorkerGroup");
log.info("Shutting down udpWorkerGroup");
groupFutures.add(udpWorkerGroup.shutdownGracefully(quietPeriod, 10, TimeUnit.SECONDS));
}
@@ -238,106 +386,17 @@ public class Balancer {
try {
f.get();
} catch (final Exception e) {
LOG.error("Error while shutting down group: {}", f.toString());
LOG.error("{}", e.getMessage());
e.printStackTrace();
log.error("Group:" + f.toString() + " -> " + e.getMessage(), e);
}
});
try {
LOG.info("Closing WebSocketRequestDecoder");
log.info("Closing WebSocketRequestDecoder");
WebSocketRequestDecoder.close();
} catch (Exception e) {
LOG.error("Error closing WebSocketRequestDecoder during shutdown", e);
log.error("Error closing WebSocketRequestDecoder during shutdown", e);
}
if (applicationContext != null) {
LOG.info("Closing applicationContext");
applicationContext.close();
}
if (curatorFramework != null) {
curatorFramework.close();
}
this.shutdown = true;
LOG.info("Server shut down.");
}
private static boolean useEpoll() {
// Should we just return true if this is Linux and if we get an error
// during Epoll
// setup handle it there?
final String os = SystemPropertyUtil.get(OS_NAME).toLowerCase().trim();
final String[] version = SystemPropertyUtil.get(OS_VERSION).toLowerCase().trim().split("\\.");
if (os.startsWith("linux") && version.length >= 3) {
final int major = Integer.parseInt(version[0]);
if (major > EPOLL_MIN_MAJOR_VERSION) {
return true;
} else if (major == EPOLL_MIN_MAJOR_VERSION) {
final int minor = Integer.parseInt(version[1]);
if (minor > EPOLL_MIN_MINOR_VERSION) {
return true;
} else if (minor == EPOLL_MIN_MINOR_VERSION) {
final int patch = Integer.parseInt(version[2].substring(0, 2));
return patch >= EPOLL_MIN_PATCH_VERSION;
} else {
return false;
}
} else {
return false;
}
} else {
return false;
}
}
protected SslContext createSSLContext(BalancerConfiguration config) throws Exception {
ServerSsl sslCfg = config.getSecurity().getServerSsl();
Boolean generate = sslCfg.isUseGeneratedKeypair();
SslContextBuilder ssl;
if (generate) {
LOG.warn("Using generated self signed server certificate");
Date begin = new Date();
Date end = new Date(begin.getTime() + 86400000);
SelfSignedCertificate ssc = new SelfSignedCertificate("localhost", begin, end);
ssl = SslContextBuilder.forServer(ssc.certificate(), ssc.privateKey());
} else {
String cert = sslCfg.getCertificateFile();
String key = sslCfg.getKeyFile();
String keyPass = sslCfg.getKeyPassword();
if (null == cert || null == key) {
throw new IllegalArgumentException("Check your SSL properties, something is wrong.");
}
ssl = SslContextBuilder.forServer(new File(cert), new File(key), keyPass);
}
ssl.ciphers(sslCfg.getUseCiphers());
// Can't set to REQUIRE because the CORS pre-flight requests will fail.
ssl.clientAuth(ClientAuth.OPTIONAL);
Boolean useOpenSSL = sslCfg.isUseOpenssl();
if (useOpenSSL) {
ssl.sslProvider(SslProvider.OPENSSL);
} else {
ssl.sslProvider(SslProvider.JDK);
}
String trustStore = sslCfg.getTrustStoreFile();
if (null != trustStore) {
if (!trustStore.isEmpty()) {
ssl.trustManager(new File(trustStore));
}
}
return ssl.build();
}
protected SSLContext createSSLClientContext(BalancerConfiguration config) throws Exception {
ClientSsl ssl = config.getSecurity().getClientSsl();
return HttpClient.getSSLContext(ssl.getTrustStoreFile(), ssl.getTrustStoreType(), ssl.getTrustStorePassword(),
ssl.getKeyFile(), ssl.getKeyType(), ssl.getKeyPassword());
log.info("{} shut down.", this.getClass().getSimpleName());
}
private void ensureZkPaths(CuratorFramework curatorFramework, String[] paths) {
@@ -345,181 +404,37 @@ public class Balancer {
try {
Stat stat = curatorFramework.checkExists().forPath(s);
if (stat == null) {
curatorFramework.create().creatingParentContainersIfNeeded().withMode(CreateMode.PERSISTENT)
.forPath(s);
curatorFramework.create().creatingParentContainersIfNeeded().withMode(CreateMode.PERSISTENT).forPath(s);
}
} catch (Exception e) {
LOG.info(e.getMessage());
log.info(e.getMessage());
}
}
}
public void run() throws Exception {
// start curator framework
RetryPolicy retryPolicy = new RetryForever(1000);
curatorFramework = CuratorFrameworkFactory.newClient(balancerConfig.getZooKeeper().getServers(), 30000, 1000,
retryPolicy);
curatorFramework.start();
ensureZkPaths(curatorFramework, zkPaths);
AuthCache.setSessionMaxAge(balancerConfig.getSecurity());
final boolean useEpoll = useEpoll();
Class<? extends ServerSocketChannel> channelClass;
Class<? extends Channel> datagramChannelClass;
if (useEpoll) {
tcpWorkerGroup = new EpollEventLoopGroup();
tcpBossGroup = new EpollEventLoopGroup();
httpWorkerGroup = new EpollEventLoopGroup();
httpBossGroup = new EpollEventLoopGroup();
wsWorkerGroup = new EpollEventLoopGroup();
wsBossGroup = new EpollEventLoopGroup();
udpWorkerGroup = new EpollEventLoopGroup();
udpBossGroup = new EpollEventLoopGroup();
channelClass = EpollServerSocketChannel.class;
datagramChannelClass = EpollDatagramChannel.class;
} else {
tcpWorkerGroup = new NioEventLoopGroup();
tcpBossGroup = new NioEventLoopGroup();
httpWorkerGroup = new NioEventLoopGroup();
httpBossGroup = new NioEventLoopGroup();
wsWorkerGroup = new NioEventLoopGroup();
wsBossGroup = new NioEventLoopGroup();
udpWorkerGroup = new NioEventLoopGroup();
udpBossGroup = new NioEventLoopGroup();
channelClass = NioServerSocketChannel.class;
datagramChannelClass = NioDatagramChannel.class;
}
LOG.info("Using channel class {}", channelClass.getSimpleName());
List<TcpClientPool> tcpClientPools = new ArrayList<>();
for (int x = 0; x < balancerConfig.getServer().getNumTcpPools(); x++) {
tcpClientPools.add(new TcpClientPool(this.balancerConfig));
}
HealthChecker healthChecker = new HealthChecker(this.balancerConfig, tcpClientPools.get(0));
this.metricResolver = new BalancedMetricResolver(curatorFramework, this.balancerConfig, healthChecker);
final ServerBootstrap tcpServer = new ServerBootstrap();
tcpServer.group(tcpBossGroup, tcpWorkerGroup);
tcpServer.channel(channelClass);
tcpServer.handler(new LoggingHandler());
tcpServer.childHandler(setupTcpChannel(metricResolver, tcpClientPools));
tcpServer.option(ChannelOption.ALLOCATOR, PooledByteBufAllocator.DEFAULT);
tcpServer.option(ChannelOption.SO_BACKLOG, 128);
tcpServer.option(ChannelOption.SO_KEEPALIVE, true);
final int tcpPort = balancerConfig.getServer().getTcpPort();
final String tcpIp = balancerConfig.getServer().getIp();
tcpChannelHandle = tcpServer.bind(tcpIp, tcpPort).sync().channel();
final String tcpAddress = ((InetSocketAddress) tcpChannelHandle.localAddress()).getAddress().getHostAddress();
final int httpPort = balancerConfig.getHttp().getPort();
final String httpIp = balancerConfig.getHttp().getIp();
SslContext sslCtx = createSSLContext(balancerConfig);
if (sslCtx instanceof OpenSslServerContext) {
OpenSslServerContext openssl = (OpenSslServerContext) sslCtx;
String application = "Timely_" + httpPort;
OpenSslServerSessionContext opensslCtx = openssl.sessionContext();
opensslCtx.setSessionCacheEnabled(true);
opensslCtx.setSessionCacheSize(128);
opensslCtx.setSessionIdContext(application.getBytes(StandardCharsets.UTF_8));
opensslCtx.setSessionTimeout(balancerConfig.getSecurity().getSessionMaxAge());
}
SSLContext clientSSLContext = createSSLClientContext(balancerConfig);
final ServerBootstrap httpServer = new ServerBootstrap();
httpServer.group(httpBossGroup, httpWorkerGroup);
httpServer.channel(channelClass);
httpServer.handler(new LoggingHandler());
HttpClientPool httpClientPool = new HttpClientPool(balancerConfig.getSecurity(), balancerConfig.getHttp(),
clientSSLContext);
httpServer.childHandler(setupHttpChannel(balancerConfig, sslCtx, metricResolver, httpClientPool));
httpServer.option(ChannelOption.ALLOCATOR, PooledByteBufAllocator.DEFAULT);
httpServer.option(ChannelOption.SO_BACKLOG, 128);
httpServer.option(ChannelOption.SO_KEEPALIVE, true);
httpChannelHandle = httpServer.bind(httpIp, httpPort).sync().channel();
final String httpAddress = ((InetSocketAddress) httpChannelHandle.localAddress()).getAddress().getHostAddress();
final int wsPort = balancerConfig.getWebsocket().getPort();
final String wsIp = balancerConfig.getWebsocket().getIp();
final ServerBootstrap wsServer = new ServerBootstrap();
wsServer.group(wsBossGroup, wsWorkerGroup);
wsServer.channel(channelClass);
wsServer.handler(new LoggingHandler());
WsClientPool wsClientPool = new WsClientPool(balancerConfig, clientSSLContext);
wsServer.childHandler(setupWSChannel(balancerConfig, sslCtx, metricResolver, wsClientPool));
wsServer.option(ChannelOption.ALLOCATOR, PooledByteBufAllocator.DEFAULT);
wsServer.option(ChannelOption.SO_BACKLOG, 128);
wsServer.option(ChannelOption.SO_KEEPALIVE, true);
/* Not sure if next three lines are necessary */
wsServer.option(ChannelOption.SO_SNDBUF, 1048576);
wsServer.option(ChannelOption.WRITE_BUFFER_HIGH_WATER_MARK, 838860);
wsServer.option(ChannelOption.WRITE_BUFFER_LOW_WATER_MARK, 620145);
wsChannelHandle = wsServer.bind(wsIp, wsPort).sync().channel();
final String wsAddress = ((InetSocketAddress) wsChannelHandle.localAddress()).getAddress().getHostAddress();
final int udpPort = balancerConfig.getServer().getUdpPort();
final String udpIp = balancerConfig.getServer().getIp();
UdpClientPool udpClientPool = new UdpClientPool(balancerConfig);
for (int n = 1; n < DEFAULT_EVENT_LOOP_THREADS; n++) {
final Bootstrap udpServer = new Bootstrap();
udpServer.group(udpBossGroup);
udpServer.channel(datagramChannelClass);
udpServer.handler(setupUdpChannel(metricResolver, udpClientPool));
udpServer.option(ChannelOption.ALLOCATOR, PooledByteBufAllocator.DEFAULT);
udpServer.option(EpollChannelOption.SO_REUSEADDR, true);
udpServer.option(EpollChannelOption.SO_REUSEPORT, true);
udpChannelHandleList.add(udpServer.bind(udpIp, udpPort).sync().channel());
}
shutdownHook();
LOG.info(
"Server started. Listening on {}:{} for TCP traffic, {}:{} for HTTP traffic, {}:{} for WebSocket traffic, and {}:{} for UDP traffic",
tcpAddress, tcpPort, httpAddress, httpPort, wsAddress, wsPort, wsAddress, udpPort);
}
public static void main(String[] args) throws Exception {
Balancer.applicationContext = Balancer.initializeConfiguration(args);
BalancerConfiguration balancerConf = applicationContext.getBean(BalancerConfiguration.class);
Balancer b = new Balancer(balancerConf);
try {
b.run();
LATCH.await();
} catch (final InterruptedException e) {
LOG.info("Server shutting down.");
} catch (Exception e) {
LOG.error("Error running server.", e);
} finally {
try {
b.shutdown();
} catch (Exception e) {
System.exit(1);
}
}
}
protected ChannelHandler setupHttpChannel(BalancerConfiguration balancerConfig, SslContext sslCtx,
MetricResolver metricResolver, HttpClientPool httpClientPool) {
protected ChannelHandler setupHttpChannel(BalancerHttpProperties balancerHttpProperties, SslServerProperties sslServerProperties, SslContext sslCtx,
MetricResolver metricResolver, HttpClientPool httpClientPool) {
return new ChannelInitializer<SocketChannel>() {
@Override
protected void initChannel(SocketChannel ch) throws Exception {
protected void initChannel(SocketChannel ch) {
ch.pipeline().addLast("ssl", new NonSslRedirectHandler(balancerConfig.getHttp(), sslCtx));
ch.pipeline().addLast("ssl", new NonSslRedirectHandler(balancerHttpProperties, sslCtx));
ch.pipeline().addLast("encoder", new HttpResponseEncoder());
ch.pipeline().addLast("decoder", new HttpRequestDecoder());
ch.pipeline().addLast("decoder", new HttpRequestDecoder(4096, 32768, 8192, true, 128));
ch.pipeline().addLast("compressor", new HttpContentCompressor());
ch.pipeline().addLast("decompressor", new HttpContentDecompressor());
ch.pipeline().addLast("aggregator", new HttpObjectAggregator(65536));
ch.pipeline().addLast("chunker", new ChunkedWriteHandler());
ch.pipeline().addLast("queryDecoder", new timely.netty.http.HttpRequestDecoder(
balancerConfig.getSecurity(), balancerConfig.getHttp()));
ch.pipeline().addLast("fileServer", new HttpStaticFileServerHandler());
ch.pipeline().addLast("login",
new X509LoginRequestHandler(balancerConfig.getSecurity(), balancerConfig.getHttp()));
ch.pipeline().addLast("queryDecoder",
new timely.netty.http.HttpRequestDecoder(authenticationService, securityProperties, balancerHttpProperties));
ch.pipeline().addLast("fileServer",
new HttpStaticFileServerHandler().setIgnoreSslHandshakeErrors(sslServerProperties.isIgnoreSslHandshakeErrors()));
ch.pipeline().addLast("login", new X509LoginRequestHandler(authenticationService, securityProperties, balancerHttpProperties));
ch.pipeline().addLast("httpRelay", new HttpRelayHandler(metricResolver, httpClientPool));
ch.pipeline().addLast("error", new TimelyExceptionHandler());
ch.pipeline().addLast("error", new TimelyExceptionHandler().setIgnoreSslHandshakeErrors(sslServerProperties.isIgnoreSslHandshakeErrors()));
}
};
}
@@ -553,8 +468,8 @@ public class Balancer {
};
}
protected ChannelHandler setupWSChannel(BalancerConfiguration balancerConfig, SslContext sslCtx,
MetricResolver metricResolver, WsClientPool wsClientPool) {
protected ChannelHandler setupWSChannel(BalancerWebsocketProperties balancerWebsocketProperties, SecurityProperties securityProperties, SslContext sslCtx,
MetricResolver metricResolver, WsClientPool wsClientPool) {
return new ChannelInitializer<SocketChannel>() {
@Override
@@ -562,14 +477,12 @@ public class Balancer {
ch.pipeline().addLast("ssl", sslCtx.newHandler(ch.alloc()));
ch.pipeline().addLast("httpServer", new HttpServerCodec());
ch.pipeline().addLast("aggregator", new HttpObjectAggregator(65536));
ch.pipeline().addLast("sessionExtractor", new WebSocketFullRequestHandler());
ch.pipeline().addLast("sessionExtractor", new WebSocketFullRequestHandler(authenticationService));
ch.pipeline().addLast("idle-handler",
new IdleStateHandler(balancerConfig.getWebsocket().getTimeout(), 0, 0));
ch.pipeline().addLast("ws-protocol",
new WebSocketServerProtocolHandler(WS_PATH, null, true, 65536, true));
ch.pipeline().addLast("wsDecoder", new WebSocketRequestDecoder(balancerConfig.getSecurity()));
ch.pipeline().addLast("httpRelay", new WsRelayHandler(balancerConfig, metricResolver, wsClientPool));
ch.pipeline().addLast("idle-handler", new IdleStateHandler(balancerWebsocketProperties.getTimeout(), 0, 0));
ch.pipeline().addLast("ws-protocol", new WebSocketServerProtocolHandler(WS_PATH, null, true, 65536, false, true));
ch.pipeline().addLast("wsDecoder", new WebSocketRequestDecoder(authenticationService, securityProperties));
ch.pipeline().addLast("httpRelay", new WsRelayHandler(balancerWebsocketProperties, metricResolver, wsClientPool));
ch.pipeline().addLast("error", new WSTimelyExceptionHandler());
}
};
@@ -0,0 +1,72 @@
package timely.balancer;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.Executors;
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.TimeUnit;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import timely.balancer.configuration.BalancerProperties;
import timely.balancer.configuration.BalancerServerProperties;
import timely.balancer.connection.TimelyBalancedHost;
import timely.balancer.connection.tcp.TcpClientPool;
import timely.balancer.healthcheck.TCPServerHealthCheck;
public class HealthChecker {
private static final Logger log = LoggerFactory.getLogger(HealthChecker.class);
private ScheduledExecutorService executorService = Executors.newScheduledThreadPool(1);
private TCPServerHealthCheck tcpServerHealthCheck;
private BalancerProperties balancerProperties;
private BalancerServerProperties balancerServerProperties;
protected List<TimelyBalancedHost> timelyHosts = new ArrayList<>();
public HealthChecker(BalancerProperties balancerProperties, BalancerServerProperties balancerServerProperties) {
this.balancerProperties = balancerProperties;
this.balancerServerProperties = balancerServerProperties;
}
public void start() {
TcpClientPool tcpClientPool = new TcpClientPool(balancerServerProperties);
this.tcpServerHealthCheck = new TCPServerHealthCheck(tcpClientPool);
this.executorService.scheduleAtFixedRate(() -> {
synchronized (timelyHosts) {
for (TimelyBalancedHost h : timelyHosts) {
try {
if (tcpServerHealthCheck.isServerHealthy(h)) {
h.reportSuccess();
} else {
h.reportFailure();
}
} catch (Exception e) {
log.error(e.getMessage(), e);
h.reportFailure();
}
}
}
}, 0, balancerProperties.getCheckServerHealthInterval(), TimeUnit.MILLISECONDS);
}
public void shutdown() {
this.executorService.shutdown();
try {
this.executorService.awaitTermination(5, TimeUnit.SECONDS);
} catch (InterruptedException e) {
} finally {
if (!this.executorService.isTerminated()) {
this.executorService.shutdownNow();
}
}
}
public void setTimelyHosts(List<TimelyBalancedHost> timelyHosts) {
synchronized (timelyHosts) {
this.timelyHosts.clear();
this.timelyHosts.addAll(timelyHosts);
}
}
}
@@ -0,0 +1,14 @@
package timely.balancer;
import org.springframework.boot.WebApplicationType;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.boot.autoconfigure.web.servlet.error.ErrorMvcAutoConfiguration;
import org.springframework.boot.builder.SpringApplicationBuilder;
@SpringBootApplication(scanBasePackages = {"timely.balancer", "timely.common"}, exclude = {ErrorMvcAutoConfiguration.class})
public class TimelyBalancer {
public static void main(String[] args) {
new SpringApplicationBuilder(TimelyBalancer.class).web(WebApplicationType.NONE).run(args);
}
}
@@ -1,123 +1,58 @@
package timely.balancer.configuration;
import javax.validation.Valid;
import javax.net.ssl.SSLContext;
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.boot.context.properties.NestedConfigurationProperty;
import org.springframework.stereotype.Component;
import org.springframework.validation.annotation.Validated;
import timely.configuration.Cache;
import org.apache.curator.framework.CuratorFramework;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.boot.autoconfigure.condition.ConditionalOnMissingBean;
import org.springframework.boot.context.properties.EnableConfigurationProperties;
import org.springframework.context.ApplicationContext;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
@Validated
@Component
@ConfigurationProperties(prefix = "timely-balancer")
import io.netty.handler.ssl.SslContext;
import timely.balancer.Balancer;
import timely.balancer.HealthChecker;
import timely.balancer.MetricResolver;
import timely.common.component.AuthenticationService;
import timely.common.configuration.CacheProperties;
import timely.common.configuration.SecurityProperties;
import timely.common.configuration.SslClientProperties;
import timely.common.configuration.SslServerProperties;
import timely.common.configuration.ZookeeperProperties;
@Configuration
@EnableConfigurationProperties({BalancerProperties.class, BalancerServerProperties.class, BalancerWebsocketProperties.class, BalancerHttpProperties.class})
public class BalancerConfiguration {
@Valid
@NestedConfigurationProperty
private BalancerServer server = new BalancerServer();
@Valid
@NestedConfigurationProperty
private BalancerHttp http = new BalancerHttp();
@Valid
@NestedConfigurationProperty
private BalancerWebsocket websocket = new BalancerWebsocket();
@Valid
@NestedConfigurationProperty
private ZooKeeper zooKeeper = new ZooKeeper();
@Valid
@NestedConfigurationProperty
private BalancerSecurity security = new BalancerSecurity();
@Valid
@NestedConfigurationProperty
private Cache cache = new Cache();
private boolean loginRequired = false;
private String assignmentFile;
private Double controlBandPercentage = 0.05;
private long checkServerHealthInterval = 10000;
private int serverFailuresBeforeDown = 3;
private int serverSuccessesBeforeUp = 3;
public ZooKeeper getZooKeeper() {
return zooKeeper;
@Bean
@ConditionalOnMissingBean
public Balancer balancer(ApplicationContext applicationContext, @Qualifier("nettySslContext") SslContext serverSslContext,
@Qualifier("outboundJDKSslContext") SSLContext clientSSLContext, AuthenticationService authenticationService, MetricResolver metricResolver,
BalancerProperties balancerProperties, ZookeeperProperties zookeeperProperties, SecurityProperties securityProperties,
SslServerProperties sslServerProperties, SslClientProperties sslClientProperties, BalancerServerProperties balancerServerProperties,
BalancerHttpProperties balancerHttpProperties, BalancerWebsocketProperties balancerWebsocketProperties) {
Balancer balancer = new Balancer(applicationContext, serverSslContext, clientSSLContext, authenticationService, metricResolver, balancerProperties,
zookeeperProperties, securityProperties, sslServerProperties, sslClientProperties, balancerServerProperties, balancerHttpProperties,
balancerWebsocketProperties);
balancer.start();
return balancer;
}
public void setLoginRequired(boolean loginRequired) {
this.loginRequired = loginRequired;
@Bean
@ConditionalOnMissingBean
public HealthChecker healthChecker(BalancerProperties balancerProperties, BalancerServerProperties balancerServerProperties) {
HealthChecker healthChecker = new HealthChecker(balancerProperties, balancerServerProperties);
healthChecker.start();
return healthChecker;
}
public boolean isLoginRequired() {
return loginRequired;
}
public String getAssignmentFile() {
return assignmentFile;
}
public void setAssignmentFile(String assignmentFile) {
this.assignmentFile = assignmentFile;
}
public void setControlBandPercentage(Double controlBandPercentage) {
this.controlBandPercentage = controlBandPercentage;
}
public Double getControlBandPercentage() {
return controlBandPercentage;
}
public BalancerSecurity getSecurity() {
return security;
}
public BalancerServer getServer() {
return server;
}
public BalancerHttp getHttp() {
return http;
}
public BalancerWebsocket getWebsocket() {
return websocket;
}
public Cache getCache() {
return cache;
}
public void setCheckServerHealthInterval(long checkServerHealthInterval) {
this.checkServerHealthInterval = checkServerHealthInterval;
}
public long getCheckServerHealthInterval() {
return checkServerHealthInterval;
}
public int getServerFailuresBeforeDown() {
return serverFailuresBeforeDown;
}
public void setServerFailuresBeforeDown(int serverFailuresBeforeDown) {
this.serverFailuresBeforeDown = serverFailuresBeforeDown;
}
public int getServerSuccessesBeforeUp() {
return serverSuccessesBeforeUp;
}
public void setServerSuccessesBeforeUp(int serverSuccessesBeforeUp) {
this.serverSuccessesBeforeUp = serverSuccessesBeforeUp;
@Bean
@ConditionalOnMissingBean
public MetricResolver metricResolver(CuratorFramework curatorFramework, BalancerProperties balancerProperties, CacheProperties cacheProperties,
HealthChecker healthChecker) throws Exception {
MetricResolver metricResolver = new MetricResolver(curatorFramework, balancerProperties, cacheProperties, healthChecker);
metricResolver.start();
return metricResolver;
}
}
@@ -1,18 +0,0 @@
package timely.balancer.configuration;
import javax.validation.Valid;
import org.apache.commons.pool2.impl.GenericKeyedObjectPoolConfig;
import org.springframework.boot.context.properties.NestedConfigurationProperty;
import timely.configuration.Http;
public class BalancerHttp extends Http {
@Valid
@NestedConfigurationProperty
private GenericKeyedObjectPoolConfig httpClientPool = new GenericKeyedObjectPoolConfig();
public GenericKeyedObjectPoolConfig getHttpClientPool() {
return httpClientPool;
}
}
@@ -0,0 +1,24 @@
package timely.balancer.configuration;
import javax.validation.Valid;
import org.apache.commons.pool2.impl.GenericKeyedObjectPoolConfig;
import org.apache.http.impl.client.CloseableHttpClient;
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.boot.context.properties.NestedConfigurationProperty;
import org.springframework.cloud.context.config.annotation.RefreshScope;
import timely.common.configuration.HttpProperties;
@RefreshScope
@ConfigurationProperties(prefix = "timely.balancer.http")
public class BalancerHttpProperties extends HttpProperties {
@Valid
@NestedConfigurationProperty
private GenericKeyedObjectPoolConfig<CloseableHttpClient> httpClientPool = new GenericKeyedObjectPoolConfig<>();
public GenericKeyedObjectPoolConfig<CloseableHttpClient> getHttpClientPool() {
return httpClientPool;
}
}
@@ -0,0 +1,93 @@
package timely.balancer.configuration;
import java.util.List;
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.stereotype.Component;
import org.springframework.validation.annotation.Validated;
@Validated
@Component
@ConfigurationProperties(prefix = "timely.balancer")
public class BalancerProperties {
private boolean loginRequired = false;
private String assignmentFile;
private String defaultFs;
private List<String> fsConfigResources;
private Double controlBandPercentage = 0.05;
private long checkServerHealthInterval = 10000;
private int serverFailuresBeforeDown = 3;
private int serverSuccessesBeforeUp = 3;
public void setLoginRequired(boolean loginRequired) {
this.loginRequired = loginRequired;
}
public boolean isLoginRequired() {
return loginRequired;
}
public String getAssignmentFile() {
return assignmentFile;
}
public void setAssignmentFile(String assignmentFile) {
this.assignmentFile = assignmentFile;
}
public String getDefaultFs() {
return defaultFs;
}
public void setDefaultFs(String defaultFs) {
this.defaultFs = defaultFs;
}
public List<String> getFsConfigResources() {
return fsConfigResources;
}
public void setFsConfigResources(List<String> fsConfigResources) {
this.fsConfigResources = fsConfigResources;
}
public void setControlBandPercentage(Double controlBandPercentage) {
this.controlBandPercentage = controlBandPercentage;
}
public Double getControlBandPercentage() {
return controlBandPercentage;
}
public void setCheckServerHealthInterval(long checkServerHealthInterval) {
this.checkServerHealthInterval = checkServerHealthInterval;
}
public long getCheckServerHealthInterval() {
return checkServerHealthInterval;
}
public int getServerFailuresBeforeDown() {
return serverFailuresBeforeDown;
}
public void setServerFailuresBeforeDown(int serverFailuresBeforeDown) {
this.serverFailuresBeforeDown = serverFailuresBeforeDown;
}
public int getServerSuccessesBeforeUp() {
return serverSuccessesBeforeUp;
}
public void setServerSuccessesBeforeUp(int serverSuccessesBeforeUp) {
this.serverSuccessesBeforeUp = serverSuccessesBeforeUp;
}
}
@@ -1,17 +0,0 @@
package timely.balancer.configuration;
import javax.validation.Valid;
import org.springframework.boot.context.properties.NestedConfigurationProperty;
import timely.configuration.Security;
public class BalancerSecurity extends Security {
@Valid
@NestedConfigurationProperty
private ClientSsl clientSsl = new ClientSsl();
public ClientSsl getClientSsl() {
return clientSsl;
}
}
@@ -3,28 +3,35 @@ package timely.balancer.configuration;
import javax.validation.Valid;
import org.apache.commons.pool2.impl.GenericKeyedObjectPoolConfig;
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.boot.context.properties.NestedConfigurationProperty;
import timely.configuration.Server;
import org.springframework.cloud.context.config.annotation.RefreshScope;
public class BalancerServer extends Server {
import timely.client.tcp.TcpClient;
import timely.client.udp.UdpClient;
import timely.common.configuration.ServerProperties;
@RefreshScope
@ConfigurationProperties(prefix = "timely.balancer.server")
public class BalancerServerProperties extends ServerProperties {
@Valid
@NestedConfigurationProperty
private GenericKeyedObjectPoolConfig udpClientPool = new GenericKeyedObjectPoolConfig();
private GenericKeyedObjectPoolConfig<UdpClient> udpClientPool = new GenericKeyedObjectPoolConfig<>();
@Valid
@NestedConfigurationProperty
private GenericKeyedObjectPoolConfig tcpClientPool = new GenericKeyedObjectPoolConfig();
private GenericKeyedObjectPoolConfig<TcpClient> tcpClientPool = new GenericKeyedObjectPoolConfig<>();
private int numTcpPools = 1;
private int tcpBufferSize = -1;
public GenericKeyedObjectPoolConfig getUdpClientPool() {
public GenericKeyedObjectPoolConfig<UdpClient> getUdpClientPool() {
return udpClientPool;
}
public GenericKeyedObjectPoolConfig getTcpClientPool() {
public GenericKeyedObjectPoolConfig<TcpClient> getTcpClientPool() {
return tcpClientPool;
}
@@ -1,18 +0,0 @@
package timely.balancer.configuration;
import javax.validation.Valid;
import org.apache.commons.pool2.impl.GenericKeyedObjectPoolConfig;
import org.springframework.boot.context.properties.NestedConfigurationProperty;
import timely.configuration.Websocket;
public class BalancerWebsocket extends Websocket {
@Valid
@NestedConfigurationProperty
private GenericKeyedObjectPoolConfig wsClientPool = new GenericKeyedObjectPoolConfig();
public GenericKeyedObjectPoolConfig getWsClientPool() {
return wsClientPool;
}
}
@@ -0,0 +1,34 @@
package timely.balancer.configuration;
import javax.validation.Valid;
import org.apache.commons.pool2.impl.GenericKeyedObjectPoolConfig;
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.boot.context.properties.NestedConfigurationProperty;
import org.springframework.cloud.context.config.annotation.RefreshScope;
import timely.client.websocket.subscription.WebSocketSubscriptionClient;
import timely.common.configuration.WebsocketProperties;
@RefreshScope
@ConfigurationProperties(prefix = "timely.balancer.websocket")
public class BalancerWebsocketProperties extends WebsocketProperties {
@Valid
@NestedConfigurationProperty
private GenericKeyedObjectPoolConfig<WebSocketSubscriptionClient> wsClientPool = new GenericKeyedObjectPoolConfig<>();
public GenericKeyedObjectPoolConfig<WebSocketSubscriptionClient> getWsClientPool() {
return wsClientPool;
}
private int incomingBufferSize = 50000000;
public int getIncomingBufferSize() {
return incomingBufferSize;
}
public void setIncomingBufferSize(int incomingBufferSize) {
this.incomingBufferSize = incomingBufferSize;
}
}
@@ -1,77 +0,0 @@
package timely.balancer.configuration;
public class ClientSsl {
private String keyFile;
private String keyType;
private String keyPassword;
private String trustStoreFile;
private String trustStoreType;
private String trustStorePassword;
private boolean useClientCert = true;
private boolean hostVerificationEnabled = true;
public String getKeyFile() {
return keyFile;
}
public void setKeyFile(String keyFile) {
this.keyFile = keyFile;
}
public String getKeyType() {
return keyType;
}
public void setKeyType(String keyType) {
this.keyType = keyType;
}
public String getKeyPassword() {
return keyPassword;
}
public void setKeyPassword(String keyPassword) {
this.keyPassword = keyPassword;
}
public String getTrustStoreFile() {
return trustStoreFile;
}
public void setTrustStoreFile(String trustStoreFile) {
this.trustStoreFile = trustStoreFile;
}
public String getTrustStoreType() {
return trustStoreType;
}
public void setTrustStoreType(String trustStoreType) {
this.trustStoreType = trustStoreType;
}
public String getTrustStorePassword() {
return trustStorePassword;
}
public void setTrustStorePassword(String trustStorePassword) {
this.trustStorePassword = trustStorePassword;
}
public void setUseClientCert(boolean useClientCert) {
this.useClientCert = useClientCert;
}
public boolean isUseClientCert() {
return useClientCert;
}
public void setHostVerificationEnabled(boolean hostVerificationEnabled) {
this.hostVerificationEnabled = hostVerificationEnabled;
}
public boolean isHostVerificationEnabled() {
return hostVerificationEnabled;
}
}
@@ -1,7 +0,0 @@
package timely.balancer.configuration;
import org.springframework.boot.autoconfigure.SpringBootApplication;
@SpringBootApplication(scanBasePackageClasses = { SpringBootstrap.class })
public class SpringBootstrap {
}
@@ -1,14 +0,0 @@
package timely.balancer.configuration;
public class ZooKeeper {
private String servers;
public String getServers() {
return servers;
}
public void setServers(String servers) {
this.servers = servers;
}
}
@@ -1,19 +1,17 @@
package timely.balancer.connection;
import java.util.Timer;
import org.apache.commons.lang3.builder.EqualsBuilder;
import org.apache.commons.lang3.builder.HashCodeBuilder;
import org.apache.commons.lang3.builder.ToStringBuilder;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import timely.balancer.ArrivalRate;
import timely.balancer.configuration.BalancerConfiguration;
import timely.balancer.configuration.BalancerProperties;
public class TimelyBalancedHost {
private static final Logger LOG = LoggerFactory.getLogger(TimelyBalancedHost.class);
static private Timer arrivalRateTimer = new Timer("ArrivalRateTimerTBH", true);
private static final Logger log = LoggerFactory.getLogger(TimelyBalancedHost.class);
private String host;
private int tcpPort;
@@ -21,41 +19,30 @@ public class TimelyBalancedHost {
private int wsPort;
private int udpPort;
private boolean isUp = true;
private BalancerConfiguration balancerConfig;
private BalancerProperties balancerProperties;
private int failures = 0;
private int successes = 0;
private ArrivalRate arrivalRate = new ArrivalRate(arrivalRateTimer);
private ArrivalRate arrivalRate = null;
public TimelyBalancedHost() {
private TimelyBalancedHost() {
}
public TimelyBalancedHost(String host, int tcpPort, int httpPort, int wsPort, int udpPort) {
private TimelyBalancedHost(String host, int tcpPort, int httpPort, int wsPort, int udpPort, ArrivalRate arrivalRate) {
this.host = host;
this.tcpPort = tcpPort;
this.httpPort = httpPort;
this.wsPort = wsPort;
this.udpPort = udpPort;
this.arrivalRate = arrivalRate;
}
public TimelyBalancedHost(String host, int basePort) {
this.host = host;
this.tcpPort = basePort;
this.httpPort = basePort + 1;
this.wsPort = basePort + 2;
this.udpPort = basePort + 3;
static public TimelyBalancedHost of(String host, int tcpPort, int httpPort, int wsPort, int udpPort, ArrivalRate arrivalRate) {
return new TimelyBalancedHost(host, tcpPort, httpPort, wsPort, udpPort, arrivalRate);
}
static public TimelyBalancedHost of(String host, int tcpPort, int httpPort, int wsPort, int udpPort) {
return new TimelyBalancedHost(host, tcpPort, httpPort, wsPort, udpPort);
}
static public TimelyBalancedHost of(String host, int basePort) {
return new TimelyBalancedHost(host, basePort);
}
public void setBalancerConfig(BalancerConfiguration balancerConfig) {
this.balancerConfig = balancerConfig;
public void setBalancerProperties(BalancerProperties balancerProperties) {
this.balancerProperties = balancerProperties;
}
synchronized public boolean isUp() {
@@ -63,38 +50,38 @@ public class TimelyBalancedHost {
}
public void reportSuccess() {
int serverSuccessesBeforeUp = balancerConfig.getServerSuccessesBeforeUp();
int serverSuccessesBeforeUp = balancerProperties.getServerSuccessesBeforeUp();
String h = host;
int p = tcpPort;
synchronized (this) {
if (LOG.isTraceEnabled()) {
LOG.trace("success reported host:{} port:{} isUp:{} successes:{} serverSuccessesBeforeUp:{}", h, p,
isUp, (successes + 1), serverSuccessesBeforeUp);
if (log.isTraceEnabled()) {
log.trace("success reported host:{} port:{} isUp:{} successes:{} serverSuccessesBeforeUp:{}", h, p, isUp, (successes + 1),
serverSuccessesBeforeUp);
}
if (!isUp) {
if (++successes >= serverSuccessesBeforeUp) {
isUp = true;
successes = 0;
LOG.info("host up host:{} port:{}", h, p);
log.info("host up host:{} port:{}", h, p);
}
}
}
}
public void reportFailure() {
int serverFailuresBeforeDown = balancerConfig.getServerFailuresBeforeDown();
int serverFailuresBeforeDown = balancerProperties.getServerFailuresBeforeDown();
String h = host;
int p = tcpPort;
synchronized (this) {
if (LOG.isTraceEnabled()) {
LOG.trace("failure reported host:{} port:{} isUp:{} failures:{} serverFailuresBeforeUp:{}", h, p, isUp,
(failures + 1), serverFailuresBeforeDown);
if (log.isTraceEnabled()) {
log.trace("failure reported host:{} port:{} isUp:{} failures:{} serverFailuresBeforeUp:{}", h, p, isUp, (failures + 1),
serverFailuresBeforeDown);
}
if (isUp) {
if (++failures >= serverFailuresBeforeDown) {
isUp = false;
failures = 0;
LOG.info("host down host:{} port:{}", h, p);
log.info("host down host:{} port:{}", h, p);
}
}
}
@@ -141,11 +128,17 @@ public class TimelyBalancedHost {
}
public void arrived() {
arrivalRate.arrived();
if (this.arrivalRate != null) {
this.arrivalRate.arrived();
}
}
public double getArrivalRate() {
return arrivalRate.getRate();
if (this.arrivalRate == null) {
return 0;
} else {
return this.arrivalRate.getRate();
}
}
@Override
@@ -6,24 +6,26 @@ import org.apache.commons.pool2.KeyedPooledObjectFactory;
import org.apache.commons.pool2.PooledObject;
import org.apache.commons.pool2.impl.DefaultPooledObject;
import org.apache.http.impl.client.CloseableHttpClient;
import timely.balancer.configuration.BalancerSecurity;
import timely.balancer.connection.TimelyBalancedHost;
public class HttpClientFactory implements KeyedPooledObjectFactory<TimelyBalancedHost, CloseableHttpClient> {
import timely.balancer.connection.TimelyBalancedHost;
import timely.client.http.HttpClient;
import timely.common.configuration.SslClientProperties;
public class HttpClientFactory implements KeyedPooledObjectFactory<TimelyBalancedHost,CloseableHttpClient> {
private final SSLContext sslContext;
private final BalancerSecurity security;
private SslClientProperties sslClientProperties;
public HttpClientFactory(BalancerSecurity security, SSLContext sslContext) {
this.security = security;
public HttpClientFactory(SSLContext sslContext, SslClientProperties sslClientProperties) {
this.sslContext = sslContext;
this.sslClientProperties = sslClientProperties;
}
@Override
public PooledObject<CloseableHttpClient> makeObject(TimelyBalancedHost k) throws Exception {
// disable cookie management because we are sharing connections
return new DefaultPooledObject<>(timely.client.http.HttpClient.get(this.sslContext, null,
security.getClientSsl().isHostVerificationEnabled(), security.getClientSsl().isUseClientCert()));
return new DefaultPooledObject<>(
HttpClient.get(this.sslContext, null, sslClientProperties.isHostVerificationEnabled(), sslClientProperties.isUseClientCert()));
}
@Override
@@ -4,17 +4,14 @@ import javax.net.ssl.SSLContext;
import org.apache.commons.pool2.impl.GenericKeyedObjectPool;
import org.apache.http.impl.client.CloseableHttpClient;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import timely.balancer.configuration.BalancerHttp;
import timely.balancer.configuration.BalancerSecurity;
import timely.balancer.configuration.BalancerHttpProperties;
import timely.balancer.connection.TimelyBalancedHost;
import timely.common.configuration.SslClientProperties;
public class HttpClientPool extends GenericKeyedObjectPool<TimelyBalancedHost, CloseableHttpClient> {
public class HttpClientPool extends GenericKeyedObjectPool<TimelyBalancedHost,CloseableHttpClient> {
private static final Logger LOG = LoggerFactory.getLogger(HttpClientPool.class);
public HttpClientPool(BalancerSecurity balancerSecurity, BalancerHttp http, SSLContext sslContext) {
super(new HttpClientFactory(balancerSecurity, sslContext), http.getHttpClientPool());
public HttpClientPool(BalancerHttpProperties http, SSLContext sslContext, SslClientProperties sslClientProperties) {
super(new HttpClientFactory(sslContext, sslClientProperties), http.getHttpClientPool());
}
}
@@ -5,10 +5,11 @@ import java.io.IOException;
import org.apache.commons.pool2.KeyedPooledObjectFactory;
import org.apache.commons.pool2.PooledObject;
import org.apache.commons.pool2.impl.DefaultPooledObject;
import timely.balancer.connection.TimelyBalancedHost;
import timely.client.tcp.TcpClient;
public class TcpClientFactory implements KeyedPooledObjectFactory<TimelyBalancedHost, TcpClient> {
public class TcpClientFactory implements KeyedPooledObjectFactory<TimelyBalancedHost,TcpClient> {
private int bufferSize;
@@ -1,13 +1,14 @@
package timely.balancer.connection.tcp;
import org.apache.commons.pool2.impl.GenericKeyedObjectPool;
import timely.balancer.configuration.BalancerConfiguration;
import timely.balancer.configuration.BalancerServerProperties;
import timely.balancer.connection.TimelyBalancedHost;
import timely.client.tcp.TcpClient;
public class TcpClientPool extends GenericKeyedObjectPool<TimelyBalancedHost, TcpClient> {
public class TcpClientPool extends GenericKeyedObjectPool<TimelyBalancedHost,TcpClient> {
public TcpClientPool(BalancerConfiguration config) {
super(new TcpClientFactory(config.getServer().getTcpBufferSize()), config.getServer().getTcpClientPool());
public TcpClientPool(BalancerServerProperties balancerServerProperties) {
super(new TcpClientFactory(balancerServerProperties.getTcpBufferSize()), balancerServerProperties.getTcpClientPool());
}
}
}
@@ -5,13 +5,13 @@ import java.io.IOException;
import org.apache.commons.pool2.KeyedPooledObjectFactory;
import org.apache.commons.pool2.PooledObject;
import org.apache.commons.pool2.impl.DefaultPooledObject;
import timely.balancer.connection.TimelyBalancedHost;
import timely.client.udp.UdpClient;
public class UdpClientFactory implements KeyedPooledObjectFactory<TimelyBalancedHost, UdpClient> {
public class UdpClientFactory implements KeyedPooledObjectFactory<TimelyBalancedHost,UdpClient> {
public UdpClientFactory() {
}
public UdpClientFactory() {}
@Override
public PooledObject<UdpClient> makeObject(TimelyBalancedHost k) throws Exception {
@@ -1,13 +1,14 @@
package timely.balancer.connection.udp;
import org.apache.commons.pool2.impl.GenericKeyedObjectPool;
import timely.balancer.configuration.BalancerConfiguration;
import timely.balancer.configuration.BalancerServerProperties;
import timely.balancer.connection.TimelyBalancedHost;
import timely.client.udp.UdpClient;
public class UdpClientPool extends GenericKeyedObjectPool<TimelyBalancedHost, UdpClient> {
public class UdpClientPool extends GenericKeyedObjectPool<TimelyBalancedHost,UdpClient> {
public UdpClientPool(BalancerConfiguration config) {
super(new UdpClientFactory(), config.getServer().getUdpClientPool());
public UdpClientPool(BalancerServerProperties serverProperties) {
super(new UdpClientFactory(), serverProperties.getUdpClientPool());
}
}
@@ -5,28 +5,35 @@ import javax.net.ssl.SSLContext;
import org.apache.commons.pool2.KeyedPooledObjectFactory;
import org.apache.commons.pool2.PooledObject;
import org.apache.commons.pool2.impl.DefaultPooledObject;
import timely.balancer.configuration.BalancerConfiguration;
import timely.balancer.configuration.BalancerProperties;
import timely.balancer.configuration.BalancerWebsocketProperties;
import timely.balancer.connection.TimelyBalancedHost;
import timely.client.websocket.subscription.WebSocketSubscriptionClient;
import timely.common.configuration.SslClientProperties;
public class WsClientFactory implements KeyedPooledObjectFactory<TimelyBalancedHost, WebSocketSubscriptionClient> {
public class WsClientFactory implements KeyedPooledObjectFactory<TimelyBalancedHost,WebSocketSubscriptionClient> {
private final SSLContext sslContext;
private BalancerConfiguration balancerConfig;
private BalancerProperties balancerProperties;
private BalancerWebsocketProperties balancerWebsocketProperties;
private SslClientProperties sslClientProperties;
public WsClientFactory(BalancerConfiguration balancerConfig, SSLContext sslContext) {
this.balancerConfig = balancerConfig;
public WsClientFactory(BalancerProperties balancerProperties, BalancerWebsocketProperties balancerWebsocketProperties,
SslClientProperties sslClientProperties, SSLContext sslContext) {
this.balancerProperties = balancerProperties;
this.balancerWebsocketProperties = balancerWebsocketProperties;
this.sslClientProperties = sslClientProperties;
this.sslContext = sslContext;
}
@Override
public PooledObject<WebSocketSubscriptionClient> makeObject(TimelyBalancedHost k) throws Exception {
int bufferSize = balancerConfig.getWebsocket().getSubscriptionBatchSize() * 500;
WebSocketSubscriptionClient client = new WebSocketSubscriptionClient(sslContext, k.getHost(), k.getHttpPort(),
k.getWsPort(), balancerConfig.getSecurity().getClientSsl().isUseClientCert(),
balancerConfig.isLoginRequired(), "", "",
balancerConfig.getSecurity().getClientSsl().isHostVerificationEnabled(), bufferSize);
int bufferSize = balancerWebsocketProperties.getIncomingBufferSize();
WebSocketSubscriptionClient client = new WebSocketSubscriptionClient(sslContext, k.getHost(), k.getHttpPort(), k.getWsPort(),
sslClientProperties.isUseClientCert(), balancerProperties.isLoginRequired(), sslClientProperties.isHostVerificationEnabled(),
bufferSize);
return new DefaultPooledObject<>(client);
}
@@ -3,13 +3,18 @@ package timely.balancer.connection.ws;
import javax.net.ssl.SSLContext;
import org.apache.commons.pool2.impl.GenericKeyedObjectPool;
import timely.balancer.configuration.BalancerConfiguration;
import timely.balancer.configuration.BalancerProperties;
import timely.balancer.configuration.BalancerWebsocketProperties;
import timely.balancer.connection.TimelyBalancedHost;
import timely.client.websocket.subscription.WebSocketSubscriptionClient;
import timely.common.configuration.SslClientProperties;
public class WsClientPool extends GenericKeyedObjectPool<TimelyBalancedHost, WebSocketSubscriptionClient> {
public class WsClientPool extends GenericKeyedObjectPool<TimelyBalancedHost,WebSocketSubscriptionClient> {
public WsClientPool(BalancerConfiguration balancerConfig, SSLContext sslContext) {
super(new WsClientFactory(balancerConfig, sslContext), balancerConfig.getWebsocket().getWsClientPool());
public WsClientPool(BalancerProperties balancerProperties, BalancerWebsocketProperties balancerWebsocketProperties, SslClientProperties sslClientProperties,
SSLContext sslContext) {
super(new WsClientFactory(balancerProperties, balancerWebsocketProperties, sslClientProperties, sslContext),
balancerWebsocketProperties.getWsClientPool());
}
}
@@ -1,53 +0,0 @@
package timely.balancer.healthcheck;
import java.util.ArrayList;
import java.util.List;
import java.util.Timer;
import java.util.TimerTask;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import timely.balancer.configuration.BalancerConfiguration;
import timely.balancer.connection.TimelyBalancedHost;
import timely.balancer.connection.tcp.TcpClientPool;
public class HealthChecker {
private static final Logger LOG = LoggerFactory.getLogger(HealthChecker.class);
private List<TimelyBalancedHost> timelyHosts = new ArrayList<>();
private Timer timer;
final private TCPServerHealthCheck tcpServerHealthCheck;
private TimerTask check = new TimerTask() {
@Override
public void run() {
synchronized (timelyHosts) {
for (TimelyBalancedHost h : timelyHosts) {
try {
if (tcpServerHealthCheck.isServerHealthy(h)) {
h.reportSuccess();
} else {
h.reportFailure();
}
} catch (Exception e) {
LOG.error(e.getMessage(), e);
h.reportFailure();
}
}
}
}
};
public HealthChecker(BalancerConfiguration config, TcpClientPool tcpClientPool) {
this.tcpServerHealthCheck = new TCPServerHealthCheck(tcpClientPool);
this.timer = new Timer("HealthCheckerThread", true);
this.timer.scheduleAtFixedRate(check, 0, config.getCheckServerHealthInterval());
}
public void setTimelyHosts(List<TimelyBalancedHost> timelyHosts) {
synchronized (timelyHosts) {
this.timelyHosts.clear();
this.timelyHosts.addAll(timelyHosts);
}
}
}
@@ -1,6 +1,6 @@
package timely.balancer.netty.http;
import static io.netty.handler.codec.http.HttpHeaders.Names.CONTENT_LENGTH;
import static io.netty.handler.codec.http.HttpHeaderNames.CONTENT_LENGTH;
import java.io.ByteArrayOutputStream;
import java.net.URLEncoder;
@@ -11,17 +11,6 @@ import java.util.Iterator;
import java.util.List;
import java.util.Map;
import com.google.common.collect.Multimap;
import io.netty.buffer.Unpooled;
import io.netty.channel.ChannelHandlerContext;
import io.netty.channel.SimpleChannelInboundHandler;
import io.netty.handler.codec.http.DefaultFullHttpResponse;
import io.netty.handler.codec.http.FullHttpRequest;
import io.netty.handler.codec.http.FullHttpResponse;
import io.netty.handler.codec.http.HttpMethod;
import io.netty.handler.codec.http.HttpResponseStatus;
import io.netty.handler.codec.http.HttpVersion;
import io.netty.util.ReferenceCountUtil;
import org.apache.http.Header;
import org.apache.http.client.methods.CloseableHttpResponse;
import org.apache.http.client.methods.HttpGet;
@@ -33,22 +22,35 @@ import org.apache.http.message.BasicHeader;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.util.StreamUtils;
import com.google.common.collect.Multimap;
import io.netty.buffer.Unpooled;
import io.netty.channel.ChannelHandlerContext;
import io.netty.channel.SimpleChannelInboundHandler;
import io.netty.handler.codec.http.DefaultFullHttpResponse;
import io.netty.handler.codec.http.FullHttpRequest;
import io.netty.handler.codec.http.FullHttpResponse;
import io.netty.handler.codec.http.HttpMethod;
import io.netty.handler.codec.http.HttpResponseStatus;
import io.netty.handler.codec.http.HttpVersion;
import io.netty.util.ReferenceCountUtil;
import timely.api.request.AuthenticatedRequest;
import timely.api.request.HttpRequest;
import timely.api.request.MetricRequest;
import timely.api.request.timeseries.HttpRequest;
import timely.api.request.timeseries.QueryRequest;
import timely.api.response.TimelyException;
import timely.auth.TimelyAuthenticationToken;
import timely.auth.util.HttpHeaderUtils;
import timely.auth.util.ProxiedEntityUtils;
import timely.balancer.MetricResolver;
import timely.balancer.connection.TimelyBalancedHost;
import timely.balancer.connection.http.HttpClientPool;
import timely.balancer.resolver.MetricResolver;
import timely.netty.http.TimelyHttpHandler;
import timely.netty.http.auth.TimelyAuthenticationToken;
public class HttpRelayHandler extends SimpleChannelInboundHandler<HttpRequest> implements TimelyHttpHandler {
private static final Logger LOG = LoggerFactory.getLogger(HttpRelayHandler.class);
private static final Logger log = LoggerFactory.getLogger(HttpRelayHandler.class);
private final HttpClientPool httpClientPool;
private MetricResolver metricResolver;
@@ -67,7 +69,7 @@ public class HttpRelayHandler extends SimpleChannelInboundHandler<HttpRequest> i
try {
try {
request = msg.getHttpRequest();
String originalURI = request.getUri();
String originalURI = request.uri();
originalURI = encodeURI(originalURI);
String metric = null;
@@ -88,7 +90,7 @@ public class HttpRelayHandler extends SimpleChannelInboundHandler<HttpRequest> i
}
client = httpClientPool.borrowObject(k);
Multimap<String, String> headers = HttpHeaderUtils.toMultimap(request.headers());
Multimap<String,String> headers = HttpHeaderUtils.toMultimap(request.headers());
if (msg instanceof AuthenticatedRequest) {
AuthenticatedRequest authenticatedRequest = (AuthenticatedRequest) msg;
TimelyAuthenticationToken token = authenticatedRequest.getToken();
@@ -100,8 +102,8 @@ public class HttpRelayHandler extends SimpleChannelInboundHandler<HttpRequest> i
}
List<Header> relayedHeaderList = new ArrayList<>();
for (Map.Entry<String, String> h : headers.entries()) {
if (!h.getKey().equals(CONTENT_LENGTH)) {
for (Map.Entry<String,String> h : headers.entries()) {
if (!h.getKey().equals(CONTENT_LENGTH.toString())) {
relayedHeaderList.add(new BasicHeader(h.getKey(), h.getValue()));
}
}
@@ -109,14 +111,14 @@ public class HttpRelayHandler extends SimpleChannelInboundHandler<HttpRequest> i
relayedHeaderList.toArray(headerArray);
String relayURI = "https://" + k.getHost() + ":" + k.getHttpPort() + originalURI;
if (request.getMethod().equals(HttpMethod.GET)) {
if (request.method().equals(HttpMethod.GET)) {
HttpUriRequest relayedRequest = new HttpGet(relayURI);
if (headerArray.length > 0) {
relayedRequest.setHeaders(headerArray);
}
relayedResponse = client.execute(relayedRequest);
} else if (request.getMethod().equals(HttpMethod.POST)) {
} else if (request.method().equals(HttpMethod.POST)) {
HttpPost relayedRequest = new HttpPost(relayURI);
String content = request.content().toString(StandardCharsets.UTF_8);
@@ -130,33 +132,30 @@ public class HttpRelayHandler extends SimpleChannelInboundHandler<HttpRequest> i
} catch (Exception e) {
String message = e.getMessage();
if (message == null) {
LOG.error("", e);
log.error("", e);
} else {
if (message.contains("No matching tags")) {
LOG.trace(message);
log.trace(message);
} else {
LOG.error(message, e);
log.error(message, e);
}
}
this.sendHttpError(ctx, new TimelyException(HttpResponseStatus.INTERNAL_SERVER_ERROR.code(),
e.getMessage(), e.getLocalizedMessage(), e));
this.sendHttpError(ctx, new TimelyException(HttpResponseStatus.INTERNAL_SERVER_ERROR.code(), e.getMessage(), e.getLocalizedMessage(), e));
return;
} finally {
if (request != null && request instanceof AutoCloseable) {
if (request != null) {
ReferenceCountUtil.release(request);
}
}
FullHttpResponse response = null;
if (relayedResponse == null) {
this.sendHttpError(ctx, new TimelyException(HttpResponseStatus.METHOD_NOT_ALLOWED.code(),
"Method not allowed", "Method not allowed"));
this.sendHttpError(ctx, new TimelyException(HttpResponseStatus.METHOD_NOT_ALLOWED.code(), "Method not allowed", "Method not allowed"));
} else {
ByteArrayOutputStream baos = new MyByteArrayOutputStream();
StreamUtils.copy(relayedResponse.getEntity().getContent(), baos);
response = new DefaultFullHttpResponse(HttpVersion.HTTP_1_1,
HttpResponseStatus.valueOf(relayedResponse.getStatusLine().getStatusCode()),
Unpooled.copiedBuffer(baos.toByteArray()));
response = new DefaultFullHttpResponse(HttpVersion.HTTP_1_1, HttpResponseStatus.valueOf(relayedResponse.getStatusLine().getStatusCode()),
Unpooled.copiedBuffer(baos.toByteArray()));
for (Header h : relayedResponse.getAllHeaders()) {
response.headers().add(h.getName(), h.getValue());
}
@@ -170,15 +169,14 @@ public class HttpRelayHandler extends SimpleChannelInboundHandler<HttpRequest> i
if (client != null && k != null) {
httpClientPool.returnObject(k, client);
} else {
LOG.error("NOT RETURNING CONNECTION! " + msg.getHttpRequest().getUri());
log.error("NOT RETURNING CONNECTION! " + msg.getHttpRequest().uri());
}
}
}
static public class MyByteArrayOutputStream extends ByteArrayOutputStream {
public MyByteArrayOutputStream() {
}
public MyByteArrayOutputStream() {}
public MyByteArrayOutputStream(int size) {
super(size);
@@ -4,33 +4,33 @@ import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.util.List;
import org.apache.commons.lang3.tuple.Pair;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import io.netty.buffer.Unpooled;
import io.netty.channel.ChannelFuture;
import io.netty.channel.ChannelHandlerContext;
import io.netty.channel.SimpleChannelInboundHandler;
import org.apache.commons.lang3.tuple.Pair;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import timely.api.request.MetricRequest;
import timely.api.request.TcpRequest;
import timely.balancer.MetricResolver;
import timely.balancer.connection.TimelyBalancedHost;
import timely.balancer.connection.tcp.TcpClientPool;
import timely.balancer.resolver.BalancedMetricResolver;
import timely.balancer.resolver.MetricResolver;
import timely.client.tcp.TcpClient;
import timely.netty.Constants;
public class TcpRelayHandler extends SimpleChannelInboundHandler<TcpRequest> {
private static final Logger LOG = LoggerFactory.getLogger(TcpRelayHandler.class);
private static final Logger log = LoggerFactory.getLogger(TcpRelayHandler.class);
private static final String LOG_ERR_MSG = "Error storing put metric: {}";
private static final String ERR_MSG = "Error storing put metric: ";
private BalancedMetricResolver metricResolver;
private MetricResolver metricResolver;
private List<TcpClientPool> tcpClientPools;
public TcpRelayHandler(MetricResolver metricResolver, List<TcpClientPool> tcpClientPools) {
this.metricResolver = (BalancedMetricResolver) metricResolver;
this.metricResolver = metricResolver;
this.tcpClientPools = tcpClientPools;
}
@@ -41,10 +41,10 @@ public class TcpRelayHandler extends SimpleChannelInboundHandler<TcpRequest> {
@Override
protected void channelRead0(ChannelHandlerContext ctx, TcpRequest msg) throws Exception {
LOG.trace("Received {}", msg);
log.trace("Received {}", msg);
try {
String line;
Pair<TimelyBalancedHost, TcpClient> keyClientPair = null;
Pair<TimelyBalancedHost,TcpClient> keyClientPair = null;
try {
if (msg instanceof MetricRequest) {
String metricName = ((MetricRequest) msg).getMetric().getName();
@@ -63,29 +63,27 @@ public class TcpRelayHandler extends SimpleChannelInboundHandler<TcpRequest> {
}
} catch (IOException e) {
LOG.error(LOG_ERR_MSG, msg, e);
ChannelFuture cf = ctx.writeAndFlush(
Unpooled.copiedBuffer((ERR_MSG + e.getMessage() + "\n").getBytes(StandardCharsets.UTF_8)));
log.error(LOG_ERR_MSG, msg, e);
ChannelFuture cf = ctx.writeAndFlush(Unpooled.copiedBuffer((ERR_MSG + e.getMessage() + "\n").getBytes(StandardCharsets.UTF_8)));
if (!cf.isSuccess()) {
LOG.error(Constants.ERR_WRITING_RESPONSE, cf.cause());
log.error(Constants.ERR_WRITING_RESPONSE, cf.cause());
}
}
}
private Pair<TimelyBalancedHost, TcpClient> getClient(String metric, boolean metricRequest) {
private Pair<TimelyBalancedHost,TcpClient> getClient(String metric, boolean metricRequest) {
TcpClient client = null;
TimelyBalancedHost k = null;
int failures = 0;
while (client == null) {
try {
k = (metricRequest == true) ? metricResolver.getHostPortKeyIngest(metric)
: metricResolver.getHostPortKey(metric);
k = (metricRequest == true) ? metricResolver.getHostPortKeyIngest(metric) : metricResolver.getHostPortKey(metric);
client = tcpClientPool().borrowObject(k);
} catch (Exception e1) {
failures++;
client = null;
if (failures % 10 == 0) {
LOG.error(e1.getMessage(), e1);
log.error(e1.getMessage(), e1);
}
try {
Thread.sleep(failures < 10 ? 500 : 60000);
@@ -2,24 +2,25 @@ package timely.balancer.netty.udp;
import java.nio.charset.StandardCharsets;
import org.apache.commons.lang3.tuple.Pair;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import io.netty.buffer.Unpooled;
import io.netty.channel.ChannelFuture;
import io.netty.channel.ChannelHandlerContext;
import io.netty.channel.SimpleChannelInboundHandler;
import org.apache.commons.lang3.tuple.Pair;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import timely.api.request.MetricRequest;
import timely.api.request.UdpRequest;
import timely.balancer.MetricResolver;
import timely.balancer.connection.TimelyBalancedHost;
import timely.balancer.connection.udp.UdpClientPool;
import timely.balancer.resolver.MetricResolver;
import timely.client.udp.UdpClient;
import timely.netty.Constants;
public class UdpRelayHandler extends SimpleChannelInboundHandler<UdpRequest> {
private static final Logger LOG = LoggerFactory.getLogger(UdpRelayHandler.class);
private static final Logger log = LoggerFactory.getLogger(UdpRelayHandler.class);
private static final String LOG_ERR_MSG = "Error storing put metric: {}";
private static final String ERR_MSG = "Error storing put metric: ";
@@ -33,11 +34,11 @@ public class UdpRelayHandler extends SimpleChannelInboundHandler<UdpRequest> {
@Override
protected void channelRead0(ChannelHandlerContext ctx, UdpRequest msg) throws Exception {
LOG.trace("Received {}", msg);
log.trace("Received {}", msg);
try {
MetricRequest metricRequest = (MetricRequest) msg;
String metricName = metricRequest.getMetric().getName();
Pair<TimelyBalancedHost, UdpClient> keyClientPair = null;
Pair<TimelyBalancedHost,UdpClient> keyClientPair = null;
try {
keyClientPair = getClient(metricName, true);
keyClientPair.getRight().write(metricRequest.getLine() + "\n");
@@ -49,28 +50,26 @@ public class UdpRelayHandler extends SimpleChannelInboundHandler<UdpRequest> {
}
} catch (Exception e) {
LOG.error(LOG_ERR_MSG, msg, e);
ChannelFuture cf = ctx.writeAndFlush(
Unpooled.copiedBuffer((ERR_MSG + e.getMessage() + "\n").getBytes(StandardCharsets.UTF_8)));
log.error(LOG_ERR_MSG, msg, e);
ChannelFuture cf = ctx.writeAndFlush(Unpooled.copiedBuffer((ERR_MSG + e.getMessage() + "\n").getBytes(StandardCharsets.UTF_8)));
if (!cf.isSuccess()) {
LOG.error(Constants.ERR_WRITING_RESPONSE, cf.cause());
log.error(Constants.ERR_WRITING_RESPONSE, cf.cause());
}
}
}
private Pair<TimelyBalancedHost, UdpClient> getClient(String metric, boolean metricRequest) {
private Pair<TimelyBalancedHost,UdpClient> getClient(String metric, boolean metricRequest) {
UdpClient client = null;
TimelyBalancedHost k = null;
int failures = 0;
while (client == null) {
try {
k = (metricRequest == true) ? metricResolver.getHostPortKeyIngest(metric)
: metricResolver.getHostPortKey(metric);
k = (metricRequest == true) ? metricResolver.getHostPortKeyIngest(metric) : metricResolver.getHostPortKey(metric);
client = udpClientPool.borrowObject(k);
} catch (Exception e1) {
failures++;
client = null;
LOG.error(e1.getMessage(), e1);
log.error(e1.getMessage(), e1);
try {
Thread.sleep(failures < 10 ? 500 : 60000);
} catch (InterruptedException e2) {
@@ -9,23 +9,25 @@ import javax.websocket.CloseReason;
import javax.websocket.EndpointConfig;
import javax.websocket.Session;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import com.google.common.collect.HashMultimap;
import com.google.common.collect.Multimap;
import io.netty.channel.ChannelHandlerContext;
import io.netty.handler.codec.http.HttpResponseStatus;
import io.netty.handler.codec.http.websocketx.PingWebSocketFrame;
import io.netty.handler.codec.http.websocketx.TextWebSocketFrame;
import io.netty.util.concurrent.ScheduledFuture;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import timely.api.response.TimelyException;
import timely.auth.TimelyAuthenticationToken;
import timely.auth.util.ProxiedEntityUtils;
import timely.client.websocket.ClientHandler;
import timely.netty.http.auth.TimelyAuthenticationToken;
public class WsClientHandler extends ClientHandler {
private static final Logger LOG = LoggerFactory.getLogger(WsClientHandler.class);
private static final Logger log = LoggerFactory.getLogger(WsClientHandler.class);
private final ChannelHandlerContext ctx;
private final TimelyAuthenticationToken token;
private ScheduledFuture<?> ping;
@@ -34,26 +36,26 @@ public class WsClientHandler extends ClientHandler {
this.ctx = ctx;
this.token = token;
this.ping = this.ctx.executor().scheduleAtFixedRate(() -> {
LOG.trace("Sending ping on channel {}", ctx.channel());
log.trace("Sending ping on channel {}", ctx.channel());
ctx.writeAndFlush(new PingWebSocketFrame());
}, pingRate, pingRate, TimeUnit.SECONDS);
}
@Override
public void beforeRequest(Map<String, List<String>> headers) {
public void beforeRequest(Map<String,List<String>> headers) {
if (token.getClientCert() != null) {
Multimap<String, String> proxyRequestHeaders = HashMultimap.create();
Multimap<String,String> proxyRequestHeaders = HashMultimap.create();
ProxiedEntityUtils.addProxyHeaders(proxyRequestHeaders, token.getClientCert());
for (String s : proxyRequestHeaders.keySet()) {
headers.put(s, new ArrayList(proxyRequestHeaders.get(s)));
headers.put(s, new ArrayList<>(proxyRequestHeaders.get(s)));
}
}
Multimap<String, String> originalRequestHeaders = token.getHttpHeaders();
Multimap<String,String> originalRequestHeaders = token.getHttpHeaders();
for (String s : originalRequestHeaders.keySet()) {
if (!headers.containsKey(s)) {
// add pre-existing values if key does not exist in proxyRequest
headers.put(s, new ArrayList(originalRequestHeaders.get(s)));
headers.put(s, new ArrayList<>(originalRequestHeaders.get(s)));
}
}
}
@@ -62,7 +64,7 @@ public class WsClientHandler extends ClientHandler {
public void onOpen(Session session, EndpointConfig config) {
session.addMessageHandler(String.class, message -> {
ctx.writeAndFlush(new TextWebSocketFrame(message));
LOG.debug("Message received on Websocket session {}: {}", session.getId(), message);
log.debug("Message received on Websocket session {}: {}", session.getId(), message);
});
}
@@ -70,9 +72,8 @@ public class WsClientHandler extends ClientHandler {
public void onClose(Session session, CloseReason reason) {
super.onClose(session, reason);
if (!reason.getCloseCode().equals(CloseReason.CloseCodes.NORMAL_CLOSURE)) {
LOG.error("Abnormal close: " + reason.getReasonPhrase());
Exception e = new TimelyException(HttpResponseStatus.INTERNAL_SERVER_ERROR.code(), reason.getReasonPhrase(),
"");
log.error("Abnormal close: " + reason.getReasonPhrase());
Exception e = new TimelyException(HttpResponseStatus.INTERNAL_SERVER_ERROR.code(), reason.getReasonPhrase(), "");
WsRelayHandler.sendErrorResponse(ctx, HttpResponseStatus.INTERNAL_SERVER_ERROR, e);
}
if (!ping.isCancelled()) {
@@ -83,7 +84,7 @@ public class WsClientHandler extends ClientHandler {
@Override
public void onError(Session session, Throwable t) {
super.onError(session, t);
LOG.error(t.getMessage(), t);
log.error(t.getMessage(), t);
Exception e = new TimelyException(HttpResponseStatus.INTERNAL_SERVER_ERROR.code(), t.getMessage(), "");
WsRelayHandler.sendErrorResponse(ctx, HttpResponseStatus.INTERNAL_SERVER_ERROR, e);
if (!ping.isCancelled()) {
@@ -4,15 +4,16 @@ import java.io.IOException;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import timely.balancer.connection.TimelyBalancedHost;
import timely.balancer.connection.ws.WsClientPool;
import timely.client.websocket.subscription.WebSocketSubscriptionClient;
public class WsClientHolder {
private static final Logger LOG = LoggerFactory.getLogger(WsClientHolder.class);
private WebSocketSubscriptionClient client = null;
private TimelyBalancedHost host = null;
private static final Logger log = LoggerFactory.getLogger(WsClientHolder.class);
private WebSocketSubscriptionClient client;
private TimelyBalancedHost host;
public WsClientHolder(TimelyBalancedHost host, WebSocketSubscriptionClient client) {
this.host = host;
@@ -32,7 +33,7 @@ public class WsClientHolder {
try {
client.close();
} catch (IOException e) {
LOG.error("Error closing web socket client", e);
log.error("Error closing web socket client", e);
}
wsClientPool.returnObject(host, client);
client = null;
@@ -5,11 +5,15 @@ import java.util.HashMap;
import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.SerializationFeature;
import com.fasterxml.jackson.datatype.jdk8.Jdk8Module;
import com.google.common.collect.Multimap;
import io.netty.channel.ChannelFuture;
import io.netty.channel.ChannelFutureListener;
import io.netty.channel.ChannelHandlerContext;
@@ -17,32 +21,30 @@ import io.netty.channel.SimpleChannelInboundHandler;
import io.netty.handler.codec.http.HttpResponseStatus;
import io.netty.handler.codec.http.websocketx.CloseWebSocketFrame;
import io.netty.handler.codec.http.websocketx.TextWebSocketFrame;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import timely.api.request.subscription.AddSubscription;
import timely.api.request.subscription.CloseSubscription;
import timely.api.request.subscription.CreateSubscription;
import timely.api.request.subscription.RemoveSubscription;
import timely.api.request.subscription.SubscriptionRequest;
import timely.api.request.websocket.AddSubscription;
import timely.api.request.websocket.CloseSubscription;
import timely.api.request.websocket.CreateSubscription;
import timely.api.request.websocket.RemoveSubscription;
import timely.api.request.websocket.SubscriptionRequest;
import timely.api.response.TimelyException;
import timely.auth.util.ProxiedEntityUtils;
import timely.balancer.configuration.BalancerConfiguration;
import timely.balancer.MetricResolver;
import timely.balancer.configuration.BalancerWebsocketProperties;
import timely.balancer.connection.TimelyBalancedHost;
import timely.balancer.connection.ws.WsClientPool;
import timely.balancer.resolver.MetricResolver;
import timely.client.websocket.subscription.WebSocketSubscriptionClient;
import timely.netty.http.TimelyHttpHandler;
public class WsRelayHandler extends SimpleChannelInboundHandler<SubscriptionRequest> implements TimelyHttpHandler {
private static final Logger LOG = LoggerFactory.getLogger(WsRelayHandler.class);
private static final Logger log = LoggerFactory.getLogger(WsRelayHandler.class);
private final BalancerConfiguration balancerConfig;
private final WsClientPool wsClientPool;
private BalancerWebsocketProperties balancerWebsocketProperties;
private WsClientPool wsClientPool;
private MetricResolver metricResolver;
static private ObjectMapper mapper;
Map<String, Map<String, WsClientHolder>> wsClients = new ConcurrentHashMap<>();
Map<String, ChannelHandlerContext> wsSubscriptions = new ConcurrentHashMap<>();
Map<String,Map<String,WsClientHolder>> wsClients = new ConcurrentHashMap<>();
Map<String,ChannelHandlerContext> wsSubscriptions = new ConcurrentHashMap<>();
static {
mapper = new ObjectMapper();
@@ -52,9 +54,8 @@ public class WsRelayHandler extends SimpleChannelInboundHandler<SubscriptionRequ
mapper.configure(SerializationFeature.WRAP_ROOT_VALUE, true);
}
public WsRelayHandler(BalancerConfiguration balancerConfig, MetricResolver metricResolver,
WsClientPool wsClientPool) {
this.balancerConfig = balancerConfig;
public WsRelayHandler(BalancerWebsocketProperties balancerWebsocketProperties, MetricResolver metricResolver, WsClientPool wsClientPool) {
this.balancerWebsocketProperties = balancerWebsocketProperties;
this.metricResolver = metricResolver;
this.wsClientPool = wsClientPool;
}
@@ -67,7 +68,7 @@ public class WsRelayHandler extends SimpleChannelInboundHandler<SubscriptionRequ
try {
String metric = null;
Multimap<String, String> headers = msg.getRequestHeaders();
Multimap<String,String> headers = msg.getRequestHeaders();
ProxiedEntityUtils.addProxyHeaders(headers, msg.getToken().getClientCert());
if (msg instanceof CreateSubscription) {
@@ -83,14 +84,13 @@ public class WsRelayHandler extends SimpleChannelInboundHandler<SubscriptionRequ
@Override
public void operationComplete(ChannelFuture future) throws Exception {
Map<String, WsClientHolder> metricToClientMap = wsClients.get(currentSubscriptionId);
Map<String,WsClientHolder> metricToClientMap = wsClients.get(currentSubscriptionId);
if (metricToClientMap != null) {
synchronized (metricToClientMap) {
// close all clients for this subscriptionId
LOG.info("Channel closed, closing subscriptions for subscriptionId:%s",
currentSubscriptionId);
for (Map.Entry<String, WsClientHolder> entry : metricToClientMap.entrySet()) {
log.debug("Channel closed, closing subscriptions for subscriptionId:{}", currentSubscriptionId);
for (Map.Entry<String,WsClientHolder> entry : metricToClientMap.entrySet()) {
entry.getValue().close(wsClientPool);
}
wsClients.remove(currentSubscriptionId);
@@ -108,12 +108,10 @@ public class WsRelayHandler extends SimpleChannelInboundHandler<SubscriptionRequ
if (origContext == null) {
// send error because user never sent a CreateSubscription
// and we have nowhere to send the results
LOG.info("ChannelHandlerContext not found for subscriptionId:%s - createSubscription not called?",
subscriptionId);
sendErrorResponse(ctx, HttpResponseStatus.BAD_REQUEST,
new IllegalArgumentException("Must call create first"));
log.info("ChannelHandlerContext not found for subscriptionId:{} - createSubscription not called?", subscriptionId);
sendErrorResponse(ctx, HttpResponseStatus.BAD_REQUEST, new IllegalArgumentException("Must call create first"));
}
Map<String, WsClientHolder> metricToClientMap = wsClients.get(subscriptionId);
Map<String,WsClientHolder> metricToClientMap = wsClients.get(subscriptionId);
if (metricToClientMap == null) {
metricToClientMap = new HashMap<>();
wsClients.put(subscriptionId, metricToClientMap);
@@ -124,14 +122,13 @@ public class WsRelayHandler extends SimpleChannelInboundHandler<SubscriptionRequ
if (hostClientHolder == null) {
k = metricResolver.getHostPortKey(metric);
client = wsClientPool.borrowObject(k);
client.open(new WsClientHandler(origContext, add.getToken(),
(balancerConfig.getWebsocket().getTimeout() / 2)));
client.open(new WsClientHandler(origContext, add.getToken(), (balancerWebsocketProperties.getTimeout() / 2)));
metricToClientMap.put(metric, new WsClientHolder(k, client));
} else {
client = hostClientHolder.getClient();
}
}
Map<String, String> tags = null;
Map<String,String> tags = null;
Long startTime = 0L;
Long endTime = 0L;
@@ -155,25 +152,25 @@ public class WsRelayHandler extends SimpleChannelInboundHandler<SubscriptionRequ
RemoveSubscription remove = (RemoveSubscription) msg;
metric = remove.getMetric();
subscriptionId = remove.getSubscriptionId();
Map<String, WsClientHolder> metricToClientMap = wsClients.get(subscriptionId);
Map<String,WsClientHolder> metricToClientMap = wsClients.get(subscriptionId);
if (metricToClientMap != null) {
synchronized (metricToClientMap) {
WsClientHolder hostClientHolder = metricToClientMap.get(metric);
if (hostClientHolder != null) {
hostClientHolder.getClient().removeSubscription(metric);
} else {
LOG.info("client not found for subscriptionId:%s metric:%s", subscriptionId, metric);
log.info("client not found for subscriptionId:{} metric:{}", subscriptionId, metric);
}
}
}
} else if (msg instanceof CloseSubscription) {
CloseSubscription close = (CloseSubscription) msg;
subscriptionId = close.getSubscriptionId();
Map<String, WsClientHolder> metricToClientMap = wsClients.get(subscriptionId);
Map<String,WsClientHolder> metricToClientMap = wsClients.get(subscriptionId);
if (metricToClientMap != null) {
synchronized (metricToClientMap) {
// close all clients for this subscriptionId
for (Map.Entry<String, WsClientHolder> entry : metricToClientMap.entrySet()) {
for (Map.Entry<String,WsClientHolder> entry : metricToClientMap.entrySet()) {
entry.getValue().close(wsClientPool);
}
}
@@ -183,11 +180,11 @@ public class WsRelayHandler extends SimpleChannelInboundHandler<SubscriptionRequ
ctx.writeAndFlush(new CloseWebSocketFrame(1000, "Client requested close."));
}
} catch (ConnectException e1) {
LOG.error(e1.getMessage(), e1);
log.error(e1.getMessage(), e1);
ChannelHandlerContext errorCtx = origContext == null ? ctx : origContext;
sendErrorResponse(errorCtx, HttpResponseStatus.INTERNAL_SERVER_ERROR, e1);
} catch (Exception e2) {
LOG.error(e2.getMessage(), e2);
log.error(e2.getMessage(), e2);
ChannelHandlerContext errorCtx = origContext == null ? ctx : origContext;
sendErrorResponse(errorCtx, HttpResponseStatus.INTERNAL_SERVER_ERROR, e2);
}
@@ -201,7 +198,7 @@ public class WsRelayHandler extends SimpleChannelInboundHandler<SubscriptionRequ
String json = mapper.writeValueAsString(te);
ctx.writeAndFlush(new TextWebSocketFrame(json));
} catch (JsonProcessingException e) {
LOG.error("Error serializing exception");
log.error("Error serializing exception");
}
}
}
@@ -1,10 +0,0 @@
package timely.balancer.resolver;
import timely.balancer.connection.TimelyBalancedHost;
public interface MetricResolver {
public TimelyBalancedHost getHostPortKeyIngest(String metric);
public TimelyBalancedHost getHostPortKey(String metric);
}
+11
View File
@@ -0,0 +1,11 @@
===============================================================================
_______ _ _ ____ _
|__ __(_) | | | _ \ | |
| | _ _ __ ___ ___| |_ _ | |_) | __ _| | __ _ _ __ ___ ___ _ __
| | | | '_ ` _ \ / _ \ | | | | | _ < / _` | |/ _` | '_ \ / __/ _ \ '__|
| | | | | | | | | __/ | |_| | | |_) | (_| | | (_| | | | | (_| __/ |
|_| |_|_| |_| |_|\___|_|\__, | |____/ \__,_|_|\__,_|_| |_|\___\___|_|
__/ |
|___/
===============================================================================
TimelyBalancer ${application.formatted-version}
@@ -0,0 +1,6 @@
spring:
application:
name: timely-balancer
config:
name: timely-balancer
location: ${CONF_DIR}/
+8
View File
@@ -10,4 +10,12 @@
<Method name="getBuf" />
<Bug pattern="EI_EXPOSE_REP" />
</Match>
<Match>
<!-- these are probably avoidable, but needs further triage to fix -->
<Bug pattern="EI_EXPOSE_REP" />
</Match>
<Match>
<!-- these are probably avoidable, but needs further triage to fix -->
<Bug pattern="EI_EXPOSE_REP2" />
</Match>
</FindBugsFilter>
@@ -0,0 +1,56 @@
package timely.balancer.configuration;
import javax.net.ssl.SSLContext;
import org.apache.curator.framework.CuratorFramework;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.context.ApplicationContext;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.ComponentScan;
import org.springframework.context.annotation.Configuration;
import io.netty.handler.ssl.SslContext;
import timely.balancer.HealthChecker;
import timely.balancer.MetricResolver;
import timely.balancer.test.TestBalancer;
import timely.balancer.test.TestHealthChecker;
import timely.balancer.test.TestMetricResolver;
import timely.common.component.AuthenticationService;
import timely.common.configuration.CacheProperties;
import timely.common.configuration.SecurityProperties;
import timely.common.configuration.SslClientProperties;
import timely.common.configuration.SslServerProperties;
import timely.common.configuration.ZookeeperProperties;
@Configuration
@ComponentScan(basePackages = {"timely.balancer", "timely.common", "timely.test"})
public class TestBalancerConfiguration {
@Bean
public TestBalancer balancer(ApplicationContext applicationContext, @Qualifier("nettySslContext") SslContext serverSslContext,
@Qualifier("outboundJDKSslContext") SSLContext clientSSLContext, AuthenticationService authenticationService, MetricResolver metricResolver,
BalancerProperties balancerProperties, ZookeeperProperties zookeeperProperties, SecurityProperties securityProperties,
SslServerProperties sslServerProperties, SslClientProperties sslClientProperties, BalancerServerProperties balancerServerProperties,
BalancerHttpProperties balancerHttpProperties, BalancerWebsocketProperties balancerWebsocketProperties) {
TestBalancer balancer = new TestBalancer(applicationContext, serverSslContext, clientSSLContext, authenticationService, metricResolver,
balancerProperties, zookeeperProperties, securityProperties, sslServerProperties, sslClientProperties, balancerServerProperties,
balancerHttpProperties, balancerWebsocketProperties);
balancer.start();
return balancer;
}
@Bean
public TestHealthChecker healthChecker(BalancerProperties balancerProperties, BalancerServerProperties balancerServerProperties) {
TestHealthChecker healthChecker = new TestHealthChecker(balancerProperties, balancerServerProperties);
healthChecker.start();
return healthChecker;
}
@Bean
public TestMetricResolver metricResolver(CuratorFramework curatorFramework, BalancerProperties balancerProperties, CacheProperties cacheProperties,
HealthChecker healthChecker) throws Exception {
TestMetricResolver metricResolver = new TestMetricResolver(curatorFramework, balancerProperties, cacheProperties, healthChecker);
metricResolver.start();
return metricResolver;
}
}
@@ -0,0 +1,80 @@
package timely.balancer.configuration;
import java.io.File;
import java.io.IOException;
import java.net.ServerSocket;
import java.nio.file.Files;
import java.util.Properties;
import java.util.concurrent.Executors;
import org.apache.zookeeper.server.ServerConfig;
import org.apache.zookeeper.server.ZooKeeperServerMain;
import org.apache.zookeeper.server.quorum.QuorumPeerConfig;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import timely.common.configuration.ZookeeperProperties;
@Configuration
@ConditionalOnProperty(name = "timely.in-memory-accumulo.enabled", havingValue = "true")
public class TestZookeeperConfiguration {
private static final Logger log = LoggerFactory.getLogger(TestZookeeperConfiguration.class);
private static final int MAX_PORT = 100;
private int port;
private ZookeeperProperties zookeeperProperties;
@Bean(name = "ZooKeeperServerMain", destroyMethod = "close")
public ZooKeeperServerMain zooKeeperServerMain(ZookeeperProperties zookeeperProperties) throws Exception {
this.zookeeperProperties = zookeeperProperties;
this.port = getNextPort(2181);
this.zookeeperProperties.setServers(String.format("localhost:%d", port));
File tempDir;
try {
tempDir = Files.createTempDirectory("zookeeper_temp").toFile();
tempDir.deleteOnExit();
} catch (IOException e) {
throw new RuntimeException("Unable to create temp directory for mini accumulo cluster");
}
Properties properties = new Properties();
properties.put("dataDir", tempDir.getAbsolutePath() + "/data");
properties.put("clientPort", port);
properties.put("skipACL", "true");
properties.put("leaderServes", "yes");
properties.put("4lw.commands.whitelist", "*");
properties.put("admin.enableServer", "false");
QuorumPeerConfig quorumPeerConfig = new QuorumPeerConfig();
quorumPeerConfig.parseProperties(properties);
ZooKeeperServerMain zookeeper = new ZooKeeperServerMain();
final ServerConfig config = new ServerConfig();
config.readFrom(quorumPeerConfig);
Executors.newSingleThreadExecutor().submit(() -> {
try {
zookeeper.runFromConfig(config);
} catch (Exception e) {
log.error(e.getMessage(), e);
}
});
log.info("Started TestZookeeper on port {}", port);
return zookeeper;
}
private int getNextPort(int start) {
for (int port = start; port < start + MAX_PORT; ++port) {
try {
new ServerSocket(port).close();
return port;
} catch (IOException e) {
}
}
return -1;
}
}
@@ -0,0 +1,138 @@
package timely.balancer.integration;
import static timely.Constants.SERVICE_DISCOVERY_PATH;
import java.util.List;
import java.util.concurrent.LinkedBlockingQueue;
import java.util.concurrent.TimeUnit;
import org.apache.curator.framework.CuratorFramework;
import org.apache.curator.framework.state.ConnectionState;
import org.apache.curator.x.discovery.ServiceCache;
import org.apache.curator.x.discovery.ServiceDiscovery;
import org.apache.curator.x.discovery.ServiceDiscoveryBuilder;
import org.apache.curator.x.discovery.ServiceInstance;
import org.apache.curator.x.discovery.ServiceInstanceBuilder;
import org.apache.curator.x.discovery.details.ServiceCacheListener;
import org.junit.Assert;
import org.junit.Before;
import org.junit.Rule;
import org.junit.Test;
import org.junit.experimental.categories.Category;
import org.junit.runner.RunWith;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.test.context.junit4.SpringRunner;
import timely.ServerDetails;
import timely.balancer.connection.TimelyBalancedHost;
import timely.balancer.test.TestHealthChecker;
import timely.balancer.test.TestMetricResolver;
import timely.test.IntegrationTest;
import timely.test.TimelyTestRule;
@Category(IntegrationTest.class)
@RunWith(SpringRunner.class)
@SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.NONE)
public class MetricResolverIT {
private static final Logger log = LoggerFactory.getLogger(MetricResolverIT.class);
@Autowired
@Rule
public TimelyTestRule testRule;
@Autowired
private TestMetricResolver metricResolver;
@Autowired
private TestHealthChecker healthChecker;
@Autowired
private CuratorFramework curatorFramework;
private ServiceDiscovery<ServerDetails> discovery;
private ServiceCache<ServerDetails> serviceCache;
@Before
public void setup() throws Exception {
discovery = ServiceDiscoveryBuilder.builder(ServerDetails.class).client(curatorFramework).basePath(SERVICE_DISCOVERY_PATH).build();
discovery.start();
serviceCache = discovery.serviceCacheBuilder().name("timely-server").build();
}
@Test
public void ServiceRegistrationTest() {
LinkedBlockingQueue<List<ServiceInstance<ServerDetails>>> updateQueue = getServiceUpdateQueue(serviceCache);
registerService(curatorFramework, TimelyBalancedHost.of("host1", 5242, 5243, 5244, 5245, null));
registerService(curatorFramework, TimelyBalancedHost.of("host1", 5342, 5343, 5344, 5345, null));
registerService(curatorFramework, TimelyBalancedHost.of("host1", 5442, 5443, 5444, 5445, null));
registerService(curatorFramework, TimelyBalancedHost.of("host1", 5542, 5543, 5544, 5545, null));
try {
List<ServiceInstance<ServerDetails>> currentUpdate;
List<ServiceInstance<ServerDetails>> lastUpdate = null;
do {
currentUpdate = updateQueue.poll(200, TimeUnit.MILLISECONDS);
if (currentUpdate != null) {
lastUpdate = currentUpdate;
}
} while (currentUpdate != null);
Assert.assertNotNull(lastUpdate);
Assert.assertEquals(4, lastUpdate.size());
} catch (InterruptedException e) {
Assert.fail("" + e.getMessage());
}
Assert.assertEquals(4, metricResolver.getServerList().size());
Assert.assertEquals(4, healthChecker.getTimelyHosts().size());
}
public void registerService(CuratorFramework curatorFramework, TimelyBalancedHost server) {
try {
ServerDetails payload = new ServerDetails();
payload.setHost(server.getHost());
payload.setTcpPort(server.getTcpPort());
payload.setHttpPort(server.getHttpPort());
payload.setWsPort(server.getWsPort());
payload.setUdpPort(server.getUdpPort());
ServiceInstanceBuilder<ServerDetails> builder = ServiceInstance.builder();
String serviceName = server.getHost() + ":" + server.getTcpPort();
ServiceInstance<ServerDetails> serviceInstance = builder.id(serviceName).name("timely-server").address(server.getHost()).port(server.getTcpPort())
.payload(payload).build();
ServiceDiscovery<ServerDetails> discovery = ServiceDiscoveryBuilder.builder(ServerDetails.class).client(curatorFramework)
.basePath(SERVICE_DISCOVERY_PATH).build();
discovery.start();
discovery.registerService(serviceInstance);
} catch (Exception e) {
log.error(e.getMessage());
}
}
private LinkedBlockingQueue<List<ServiceInstance<ServerDetails>>> getServiceUpdateQueue(ServiceCache<ServerDetails> serviceCache) {
LinkedBlockingQueue<List<ServiceInstance<ServerDetails>>> updates = new LinkedBlockingQueue<>();
try {
ServiceCacheListener listener = new ServiceCacheListener() {
@Override
public void stateChanged(CuratorFramework curatorFramework, ConnectionState connectionState) {
}
@Override
public void cacheChanged() {
updates.add(serviceCache.getInstances());
}
};
serviceCache.addListener(listener);
serviceCache.start();
} catch (Exception e) {
log.error(e.getMessage(), e);
}
return updates;
}
}
@@ -0,0 +1,36 @@
package timely.balancer.test;
import javax.net.ssl.SSLContext;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.context.ApplicationContext;
import io.netty.handler.ssl.SslContext;
import timely.balancer.Balancer;
import timely.balancer.MetricResolver;
import timely.balancer.configuration.BalancerHttpProperties;
import timely.balancer.configuration.BalancerProperties;
import timely.balancer.configuration.BalancerServerProperties;
import timely.balancer.configuration.BalancerWebsocketProperties;
import timely.common.component.AuthenticationService;
import timely.common.configuration.SecurityProperties;
import timely.common.configuration.SslClientProperties;
import timely.common.configuration.SslServerProperties;
import timely.common.configuration.ZookeeperProperties;
public class TestBalancer extends Balancer {
private static final Logger log = LoggerFactory.getLogger(TestBalancer.class);
public TestBalancer(ApplicationContext applicationContext, SslContext serverSslContext, SSLContext clientSSLContext,
AuthenticationService authenticationService, MetricResolver metricResolver, BalancerProperties balancerProperties,
ZookeeperProperties zookeeperProperties, SecurityProperties securityProperties, SslServerProperties sslServerProperties,
SslClientProperties sslClientProperties, BalancerServerProperties balancerServerProperties, BalancerHttpProperties balancerHttpProperties,
BalancerWebsocketProperties balancerWebsocketProperties) {
super(applicationContext, serverSslContext, clientSSLContext, authenticationService, metricResolver, balancerProperties, zookeeperProperties,
securityProperties, sslServerProperties, sslClientProperties, balancerServerProperties, balancerHttpProperties,
balancerWebsocketProperties);
DEFAULT_EVENT_LOOP_THREADS = 1;
log.info("Starting TestBalancer");
}
}
@@ -0,0 +1,19 @@
package timely.balancer.test;
import java.util.List;
import timely.balancer.HealthChecker;
import timely.balancer.configuration.BalancerProperties;
import timely.balancer.configuration.BalancerServerProperties;
import timely.balancer.connection.TimelyBalancedHost;
public class TestHealthChecker extends HealthChecker {
public TestHealthChecker(BalancerProperties balancerProperties, BalancerServerProperties balancerServerProperties) {
super(balancerProperties, balancerServerProperties);
}
public List<TimelyBalancedHost> getTimelyHosts() {
return this.timelyHosts;
}
}
@@ -0,0 +1,44 @@
package timely.balancer.test;
import java.util.List;
import org.apache.curator.framework.CuratorFramework;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import timely.balancer.HealthChecker;
import timely.balancer.MetricResolver;
import timely.balancer.configuration.BalancerProperties;
import timely.balancer.connection.TimelyBalancedHost;
import timely.common.configuration.CacheProperties;
public class TestMetricResolver extends MetricResolver {
private static final Logger log = LoggerFactory.getLogger(TestMetricResolver.class);
public TestMetricResolver(CuratorFramework curatorFramework, BalancerProperties balancerProperties, CacheProperties cacheProperties,
HealthChecker healthChecker) throws Exception {
super(curatorFramework, balancerProperties, cacheProperties, healthChecker);
}
protected void createAssignmentFile() {
try {
log.info("Creating assignment file: " + balancerProperties.getAssignmentFile());
Configuration configuration = new Configuration();
FileSystem fs = FileSystem.get(configuration);
Path assignmentFile = new Path(balancerProperties.getAssignmentFile());
fs.setWriteChecksum(false);
fs.create(assignmentFile);
fs.deleteOnExit(assignmentFile);
} catch (Exception e) {
log.error(e.getMessage(), e);
}
}
public List<TimelyBalancedHost> getServerList() {
return this.serverList;
}
}
@@ -0,0 +1,20 @@
package timely.balancer.test;
import java.util.Collections;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.context.annotation.Primary;
import org.springframework.security.authentication.ProviderManager;
import org.springframework.security.web.authentication.preauth.PreAuthenticatedAuthenticationProvider;
import org.springframework.stereotype.Component;
@Primary
@Component
public class TestTimelyAuthenticationManager extends ProviderManager {
@Autowired
public TestTimelyAuthenticationManager(@Qualifier("file") PreAuthenticatedAuthenticationProvider file) {
super(Collections.singletonList(file));
}
}
@@ -0,0 +1,159 @@
logging:
level:
ROOT: info
timely: debug
org.apache.zookeeper: error
timely:
test: true
instance: 1
accumulo:
instance-name: default
username: root
password: secret
mini-accumulo:
enabled: true
rootAuths: A,B,C,D,E,F,G,H,I
zookeeper:
servers: localhost:0
timeout: 30s
security:
session-max-age: 86400
cache-expiration-minutes: 60
cache-refresh-minutes: 5
allow-anonymous-http-access: true
allow-anonymous-ws-access: true
authorized-users:
- subject-dn: cn=example.com
roles: G,H,I
auths: D,E,F
user-type: USER
- subject-dn: cn=test a. user, ou=example developers, o=example corp, c=us
issuer-dn: cn=example corp ca, o=example corp, c=us
roles: G,H,I
auths: A,B,C
user-type: USER
server-ssl:
key-store-type: PKCS12
key-store-file: classpath:ssl/testServer.p12
key-store-password: ChangeIt
trust-store-type: JKS
trust-store-file: classpath:ssl/ca.jks
trust-store-password: ChangeIt
use-generated-keypair: false
ignore-ssl-handshake-errors: true
use-openssl: true
client-ssl:
key-store-type: PKCS12
key-store-file: classpath:ssl/testUser.p12
key-store-password: ChangeIt
trust-store-type: JKS
trust-store-file: classpath:ssl/ca.jks
trust-store-password: ChangeIt
cache:
enabled: false
metric-age-off-hours:
default: 26
non-cached-metrics:
balancer:
check-server-health-interval: 10000
server-failures-before-down: 6
server-successes-before-up: 1
assignment-file: file://${java.io.tmpdir}/timely/balancer/assignments.csv
server:
ip: 127.0.0.1
tcp-base-port: 4242
num-tcp-pools: 1
tcp-buffer-size: 100
tcp-client-pool:
min-idle-per-key: 0
max-idle-per-key: 40
max-total-per-key: 40
max-total: -1
lifo: true
fairness: false
max-wait-millis: -1
min-evictable-idle-time-millis: 1800000
evictor-shutdown-timeout-millis: 10000
soft-min-evictable-idle-time-millis: -1
num-tests-per-eviction-run: 3
time-between-eviction-runs-millis: -1
block-when-exhausted: true
jmx-enabled: true
test-on-borrow: false
test-on-return: false
udp-base-port: 4245
udp-client-pool:
min-idle-per-key: 0
max-idle-per-key: 40
max-total-per-key: 40
max-total: -1
lifo: true
fairness: false
max-wait-millis: -1
min-evictable-idle-time-millis: 1800000
evictor-shutdown-timeout-millis: 10000
soft-min-evictable-idle-time-millis: -1
num-tests-per-eviction-run: 3
time-between-eviction-runs-millis: -1
block-when-exhausted: true
jmx-enabled: true
test-on-borrow: false
test-on-return: false
http:
host: localhost
ip: 127.0.0.1
base-port: 4243
redirect-path: /secure-me
strict-transport-max-age: 604800
cors:
allow-any-origin: false
allow-null-origin: false
allow-credentials: true
allowed-origins:
allowed-methods: DELETE,GET,HEAD,OPTIONS,PUT,POST
allowed-headers: content-type
http-client-pool:
min-idle-per-key: 0
max-idle-per-key: 40
max-total-per-key: 40
max-total: -1
lifo: true
fairness: false
max-wait-millis: -1
min-evictable-idle-time-millis: 1800000
evictor-shutdown-timeout-millis: 10000
soft-min-evictable-idle-time-millis: -1
num-tests-per-eviction-run: 3
time-between-eviction-runs-millis: -1
block-when-exhausted: true
jmx-enabled: true
test-on-borrow: false
test-on-return: false
websocket:
ip: 127.0.0.1
base-port: 5244
timeout: 60
subscription-lag: 120
scanner-batch-size: 10000
flush-interval-seconds: 30
scanner-read-ahead: 1
subscription-batch-size: 5000
incoming-buffer-size: 50000000
ws-client-pool:
min-idle-per-key: 0
max-idle-per-key: 40
max-total-per-key: 40
max-total: -1
lifo: true
fairness: false
max-wait-millis: -1
min-evictable-idle-time-millis: 1800000
evictor-shutdown-timeout-millis: 10000
soft-min-evictable-idle-time-millis: -1
num-tests-per-eviction-run: 3
time-between-eviction-runs-millis: -1
block-when-exhausted: true
jmx-enabled: true
test-on-borrow: false
test-on-return: false
@@ -0,0 +1,3 @@
spring:
application:
name: timely-balancer
+24
View File
@@ -0,0 +1,24 @@
Configuration:
status: warn
monitorInterval: 60
Properties:
Property:
- name: PID
value: "????"
- name: LOG_PATTERN
value: "%clr{%d{yyyy-MM-dd HH:mm:ss.SSS}}{faint} %clr{%5p} %clr{${sys:PID}}{magenta} %clr{---}{faint} %clr{[%15.15t]}{faint} %clr{%-40.40c{1.}}{cyan} %clr{:}{faint} %m%n%wEx"
Appenders:
Console:
name: Console
target: SYSTEM_OUT
follow: true
PatternLayout:
pattern: "${LOG_PATTERN}"
Loggers:
Root:
level: warn
AppenderRef:
- ref: Console
+23
View File
@@ -0,0 +1,23 @@
Bag Attributes: <No Attributes>
subject=/C=US/O=Example Corp/CN=EXAMPLE CORP CA
issuer=/C=US/O=Example Corp/CN=EXAMPLE CORP CA
-----BEGIN CERTIFICATE-----
MIIDTjCCAjagAwIBAgIURUj8l+ms/i9jeabIBBi19KxEhAgwDQYJKoZIhvcNAQEL
BQAwPjELMAkGA1UEBhMCVVMxFTATBgNVBAoTDEV4YW1wbGUgQ29ycDEYMBYGA1UE
AxMPRVhBTVBMRSBDT1JQIENBMCAXDTIzMDIwMTIzMTUwMFoYDzIxMjMwMTA4MjMx
NTAwWjA+MQswCQYDVQQGEwJVUzEVMBMGA1UEChMMRXhhbXBsZSBDb3JwMRgwFgYD
VQQDEw9FWEFNUExFIENPUlAgQ0EwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEK
AoIBAQDVBr2XF20SKmGXfvp2BUY44BF1fm3Qw83GkC66qIHgpgQxzJ8u44P34vV1
DYTj4XzrBpx23J0nCMEiAW8YW9MxZJPEokZafRNKFQAngVdgmAW9fHemHRQfaYwe
ySVz2UZ+z28DJ3C0QshvqXskrYstwLVs4Ss2uoX+XpeFbW9IVVg4aW+GX3LHPNTk
7Hji13Capp6BCiS61AGQAiMWNBW2aRgHVLOb7RDQug2Bka/dcJ44XxJDh26C6kV1
pNEZm0ToqWAiVfvyzionPec5im6NKX9kStZc+C/iikfx5vdmrvGIF5zxy6phdhJ7
AGd6cF6UBkkYm/SsIP3WTXpBek5LAgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBBjAP
BgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBSPLNlCmmp9PH+crzCx9TNp5BoCajAN
BgkqhkiG9w0BAQsFAAOCAQEAkjLnZKqLMjbu5Hy8YOVoKaEKahP+t/tNYZW/NtrC
xSxKmGLz5kVCkjpLgCykkcpu4h2wZW4MGZJ4nc87+tN22mJOg1WnqEpBzRsEPajk
7duVjdU2FDCarERj53hxF41l/u8BxvMGbFnrAijv5SzspeLi+VCGF3wu53Bi6UHi
7WO1dIhZLMHepqgtwbDDudxwlKbG/JWjJ7OspCNqOpktpY8BFUAQ42ctYZcOHy2w
kdZDcK5IhRztkBH2aSWOhudYh/CF7BOu2nSUdtR5NwhXnsStmSbud7ahw3W5Jc9/
IVAFXf2brkrfIzwrY5kmjnEPJ8mjUyLepwSlOlMTBrehzQ==
-----END CERTIFICATE-----
Binary file not shown.
@@ -0,0 +1,51 @@
Bag Attributes
localKeyID: 22 89 1C EA 4E FF 9A F6 11 1B E7 9B 5E 4F 59 B7 B1 0F AA FA
friendlyName: example-server
subject=/C=US/O=Example Corp/OU=Servers/CN=testserver.example.com
issuer=/C=US/O=Example Corp/CN=EXAMPLE CORP CA
-----BEGIN CERTIFICATE-----
MIIDwjCCAqqgAwIBAgIUe6yBgkY1m/oXuPXA9j/c/7+w16UwDQYJKoZIhvcNAQEL
BQAwPjELMAkGA1UEBhMCVVMxFTATBgNVBAoTDEV4YW1wbGUgQ29ycDEYMBYGA1UE
AxMPRVhBTVBMRSBDT1JQIENBMCAXDTIzMDIwMTIzMTUwMFoYDzIxMjMwMTA4MjMx
NTAwWjBXMQswCQYDVQQGEwJVUzEVMBMGA1UEChMMRXhhbXBsZSBDb3JwMRAwDgYD
VQQLEwdTZXJ2ZXJzMR8wHQYDVQQDExZ0ZXN0c2VydmVyLmV4YW1wbGUuY29tMIIB
IjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAmb1pEWg9a6ZChQULazK75ppc
XScU5Ap29L70/5zcaOcbaBwqKhpKhdZCoZ1TD+Sdkk9kp7IlabCQjM2vVjknb6NY
msl3PbQFNva0EkH6s5cvpX4WhQDsWK77/4eztHT+0fpaG1YoKQn+Gy3VuW0CDyR6
GzGKifnI/aNLevgUBebbiiTPLEdGlo0jp3SXlZfoDZdbvnhBWI8Xc+aDekNlrXxW
X2TVnc02Fi914PECYJzkMpmxnBRPAcljU24WSqaUVgRoSbafkm4OXDw0xtvxnIrJ
s3iO0uZfRf7+OxAlqlvh4ZJ9/PufPMEKvmr6zea05KkFjBotKMip5Md43H3lQQID
AQABo4GcMIGZMA4GA1UdDwEB/wQEAwIFoDAdBgNVHSUEFjAUBggrBgEFBQcDAQYI
KwYBBQUHAwIwDAYDVR0TAQH/BAIwADAdBgNVHQ4EFgQUDWwMrl9/uc1wplERWxas
bDROB4MwHwYDVR0jBBgwFoAUjyzZQppqfTx/nK8wsfUzaeQaAmowGgYDVR0RBBMw
EYIJbG9jYWxob3N0hwR/AAABMA0GCSqGSIb3DQEBCwUAA4IBAQBfG7StgQf3WNk/
EnaYuurwvaOOpcOV5NDcZ6tmgaXUrISPVDpyIWfsDGetNEXc+4gpefLQtGF2wuiW
odiLc4H9JwQ/yEkVYwzrFViPdZKVvxNU2EjiwO/e/3sl+SxxkTDy6o838uTXX+X5
Q8DabmL5L8ukcnqUHVOqW38oA1vRNBKiJX58JzOr8ezX8crZmqU9h8cNwG/V+ZXE
5U7+hSRzLm7gRwtixj8ZsTDUiGJxOdbyUKImPClt1Se9TizB2Q/xOPTdmYB1qiiB
+XVKkGwQPgIbHckLsZWhBor2u5cgdrBGW09Q+H88maagwq5sBHNMs33vkkgbUDpB
6WSEBJMi
-----END CERTIFICATE-----
Bag Attributes: <No Attributes>
subject=/C=US/O=Example Corp/CN=EXAMPLE CORP CA
issuer=/C=US/O=Example Corp/CN=EXAMPLE CORP CA
-----BEGIN CERTIFICATE-----
MIIDTjCCAjagAwIBAgIURUj8l+ms/i9jeabIBBi19KxEhAgwDQYJKoZIhvcNAQEL
BQAwPjELMAkGA1UEBhMCVVMxFTATBgNVBAoTDEV4YW1wbGUgQ29ycDEYMBYGA1UE
AxMPRVhBTVBMRSBDT1JQIENBMCAXDTIzMDIwMTIzMTUwMFoYDzIxMjMwMTA4MjMx
NTAwWjA+MQswCQYDVQQGEwJVUzEVMBMGA1UEChMMRXhhbXBsZSBDb3JwMRgwFgYD
VQQDEw9FWEFNUExFIENPUlAgQ0EwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEK
AoIBAQDVBr2XF20SKmGXfvp2BUY44BF1fm3Qw83GkC66qIHgpgQxzJ8u44P34vV1
DYTj4XzrBpx23J0nCMEiAW8YW9MxZJPEokZafRNKFQAngVdgmAW9fHemHRQfaYwe
ySVz2UZ+z28DJ3C0QshvqXskrYstwLVs4Ss2uoX+XpeFbW9IVVg4aW+GX3LHPNTk
7Hji13Capp6BCiS61AGQAiMWNBW2aRgHVLOb7RDQug2Bka/dcJ44XxJDh26C6kV1
pNEZm0ToqWAiVfvyzionPec5im6NKX9kStZc+C/iikfx5vdmrvGIF5zxy6phdhJ7
AGd6cF6UBkkYm/SsIP3WTXpBek5LAgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBBjAP
BgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBSPLNlCmmp9PH+crzCx9TNp5BoCajAN
BgkqhkiG9w0BAQsFAAOCAQEAkjLnZKqLMjbu5Hy8YOVoKaEKahP+t/tNYZW/NtrC
xSxKmGLz5kVCkjpLgCykkcpu4h2wZW4MGZJ4nc87+tN22mJOg1WnqEpBzRsEPajk
7duVjdU2FDCarERj53hxF41l/u8BxvMGbFnrAijv5SzspeLi+VCGF3wu53Bi6UHi
7WO1dIhZLMHepqgtwbDDudxwlKbG/JWjJ7OspCNqOpktpY8BFUAQ42ctYZcOHy2w
kdZDcK5IhRztkBH2aSWOhudYh/CF7BOu2nSUdtR5NwhXnsStmSbud7ahw3W5Jc9/
IVAFXf2brkrfIzwrY5kmjnEPJ8mjUyLepwSlOlMTBrehzQ==
-----END CERTIFICATE-----
@@ -0,0 +1,32 @@
Bag Attributes
localKeyID: 22 89 1C EA 4E FF 9A F6 11 1B E7 9B 5E 4F 59 B7 B1 0F AA FA
friendlyName: example-server
Key Attributes: <No Attributes>
-----BEGIN PRIVATE KEY-----
MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQCZvWkRaD1rpkKF
BQtrMrvmmlxdJxTkCnb0vvT/nNxo5xtoHCoqGkqF1kKhnVMP5J2ST2SnsiVpsJCM
za9WOSdvo1iayXc9tAU29rQSQfqzly+lfhaFAOxYrvv/h7O0dP7R+lobVigpCf4b
LdW5bQIPJHobMYqJ+cj9o0t6+BQF5tuKJM8sR0aWjSOndJeVl+gNl1u+eEFYjxdz
5oN6Q2WtfFZfZNWdzTYWL3Xg8QJgnOQymbGcFE8ByWNTbhZKppRWBGhJtp+Sbg5c
PDTG2/GcismzeI7S5l9F/v47ECWqW+Hhkn38+588wQq+avrN5rTkqQWMGi0oyKnk
x3jcfeVBAgMBAAECggEAZEHC7NkfhL4NjxV1Ftkk9AMhzJWZ6ARzArQsrSysB7T+
iGwliwNOKe1J0n74FdsubqVbrIheIYhFG61edEodT44kT/ylyBVHeUp0r5tbVBZ5
ueB56w5pEoQ6Ta8/H6VUL7alYEJVOj4L+clwINXXi+xrvKQYBlZgHheJN43j8NxS
9lCrVcm58rzX3i29S7ErBf7RqDabgD6j2m7rehnSq5Ps2wxwV/LXMl4Lxlwa0jz/
LV8szNMZViPOWAqOCy7NZbwCCaCfKVUAwmNGBkE3p4JVPeLNLzCBY9G0rHVmFJLK
yPnCB7PtCZeFTsU+8OHHWoxMFsaOO9Pp8FtB/r8PYQKBgQDI0OzEo3t/QKvbwAdz
9Eze0LQvstzZIidTWbYMTxH0g7DIt+3p11d61GKtfeuYUTPlLodufdv/BaP8G9sI
NzxisB3uH/787M9fwi1OzO7zMMZZCVa7Q1vZk6ta1qY3PekcYMPqjBJhxP1X3S7x
8+mrW5vTq/l8ZRVFeLvEkh6bpQKBgQDD/MCRJoq6nzd2DoTyT9m5qO+Jlztmu9+T
vqVOg7q0NO7U7dzyMbWgMvMFoEBidnyllU1rEj2YOo4OrI2v/JmSPUVS8VXsBkkG
UqInV4F/g0Wp3hzrkZFMqHLa6AydugmRNkOpgurRKOiwAL0Pgvk8+bOIbKMM7JBE
tZ6Zvi0gbQKBgD4S5r0505jYQsdgqm0bXQ+a77e20DGGLyXCm1Ves6wjcOf6gt4/
pFjhgccjH0or/6CeK1rqKLDB65ioe6IkgNP4oKfU2BsSkF+HRD4RduNRdl5bVqX4
AlJVJSilJSf6SS5ucuDzf1MdczpPso0z2k0PujFJ/YqILHUXBuOST0fJAoGAZAdX
nX3HZpKmsumcLzRUEq9HerkmRzvyU60FH/qZyyoND75Le42lfpsicCenVNnWqoh2
WASaTVztbk3hjwXkt7/v5vJoASwAD2sTefLulE9Cuf00U4MbXr3GyTWQYtw1z9ib
ViQMhy5S/KmT1Rvq65UibanM2mbPB1KPHTb0P00CgYEAgieJ1YXYYHPjABWJ7/S9
3Z/8eJY0kuUe1DRFaKOlfej8SOCFFIa90IW2mAnujh55H4N2lww4s/N+FIRQoQpA
1Hrch9tYRiXyouQc67dFlLxmCOVJIf31ZUt7I4HcPhsfr4fuCJXUPBs1jaDmwQsk
90VWrXaTwSjmP+yHMiPfaA0=
-----END PRIVATE KEY-----
Binary file not shown.
@@ -0,0 +1,50 @@
Bag Attributes
localKeyID: 57 F6 9F DF B2 94 61 47 B0 B0 2E 3A F7 79 98 C8 15 5C 64 FA
friendlyName: example-client
subject=/C=US/O=Example Corp/OU=Example Developers/CN=Test A. User
issuer=/C=US/O=Example Corp/CN=EXAMPLE CORP CA
-----BEGIN CERTIFICATE-----
MIIDmzCCAoOgAwIBAgIUFp4IuIaZ2XId7UNvzP6ZB9hChyUwDQYJKoZIhvcNAQEL
BQAwPjELMAkGA1UEBhMCVVMxFTATBgNVBAoTDEV4YW1wbGUgQ29ycDEYMBYGA1UE
AxMPRVhBTVBMRSBDT1JQIENBMCAXDTIzMDIwMTIzMTUwMFoYDzIxMjMwMTA4MjMx
NTAwWjBYMQswCQYDVQQGEwJVUzEVMBMGA1UEChMMRXhhbXBsZSBDb3JwMRswGQYD
VQQLExJFeGFtcGxlIERldmVsb3BlcnMxFTATBgNVBAMTDFRlc3QgQS4gVXNlcjCC
ASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMOMeN0ElMDxYKST2VD1wP6Z
jmy543Zv/QgRLTYizg3U9t3uZU0C3HqUJnK+c4+OJnRIyHuMXuJWEYVdairyAdYW
ZDF5sfiL9ikG9rZ2WxR2z7nsYLJcDGwKAt/Duzuoed6KSccEpiQc+BkoOTs9p87I
l4s7saX/anESMUY7YOomLSqS2zvQZh/zTjIdP1/9kG9PLzXPClJELuiC3pJCmr7Y
xdJIXecu6YKFpApy0y+rKcfYUVPML15gyRu9pIqH3U39y7mCkn+YP9Qc9GcZAoqR
iYdUEl1rFAgV/a3o0pmf1tFFRDOsoPSMUGNNQlaCjv1uZIbRm25nvvQlPfstFDEC
AwEAAaN1MHMwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQMMAoGCCsGAQUFBwMCMAwG
A1UdEwEB/wQCMAAwHQYDVR0OBBYEFPAe3n59DcNwRcnpIwQY2kPawbCVMB8GA1Ud
IwQYMBaAFI8s2UKaan08f5yvMLH1M2nkGgJqMA0GCSqGSIb3DQEBCwUAA4IBAQBo
+zadU9DwR1rnHTBZF6CJXmiIdrQmArojH4a3NOHbNjFjRXm7dkd/tAy5LtgDJkww
yg+ZrOyOKHbFgk6IXRZ8nw2b1Ck9I9MDW1vxfjid9YLW2nhDaMcIeTXz3aIBO5i6
gMCV+xMcFaU/O57Ohia+ADUqrqu14nG9Moi171UWPbMr5N3gYn3Fs4VrTGYs4i0Z
5x2itm88WB3bWj+cd06zKQopU9b0Y76mQv9h+w4EHxq4uWA7GmIDDkvtI61MQo8+
twdX159Ms6o4lW43SRqVBHeMgSJbiLDkdkypYJvd9PvAjWgmQK/sD1SYKxfgYhtx
hqIvAjlGeDwdf+55EGjf
-----END CERTIFICATE-----
Bag Attributes: <No Attributes>
subject=/C=US/O=Example Corp/CN=EXAMPLE CORP CA
issuer=/C=US/O=Example Corp/CN=EXAMPLE CORP CA
-----BEGIN CERTIFICATE-----
MIIDTjCCAjagAwIBAgIURUj8l+ms/i9jeabIBBi19KxEhAgwDQYJKoZIhvcNAQEL
BQAwPjELMAkGA1UEBhMCVVMxFTATBgNVBAoTDEV4YW1wbGUgQ29ycDEYMBYGA1UE
AxMPRVhBTVBMRSBDT1JQIENBMCAXDTIzMDIwMTIzMTUwMFoYDzIxMjMwMTA4MjMx
NTAwWjA+MQswCQYDVQQGEwJVUzEVMBMGA1UEChMMRXhhbXBsZSBDb3JwMRgwFgYD
VQQDEw9FWEFNUExFIENPUlAgQ0EwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEK
AoIBAQDVBr2XF20SKmGXfvp2BUY44BF1fm3Qw83GkC66qIHgpgQxzJ8u44P34vV1
DYTj4XzrBpx23J0nCMEiAW8YW9MxZJPEokZafRNKFQAngVdgmAW9fHemHRQfaYwe
ySVz2UZ+z28DJ3C0QshvqXskrYstwLVs4Ss2uoX+XpeFbW9IVVg4aW+GX3LHPNTk
7Hji13Capp6BCiS61AGQAiMWNBW2aRgHVLOb7RDQug2Bka/dcJ44XxJDh26C6kV1
pNEZm0ToqWAiVfvyzionPec5im6NKX9kStZc+C/iikfx5vdmrvGIF5zxy6phdhJ7
AGd6cF6UBkkYm/SsIP3WTXpBek5LAgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBBjAP
BgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBSPLNlCmmp9PH+crzCx9TNp5BoCajAN
BgkqhkiG9w0BAQsFAAOCAQEAkjLnZKqLMjbu5Hy8YOVoKaEKahP+t/tNYZW/NtrC
xSxKmGLz5kVCkjpLgCykkcpu4h2wZW4MGZJ4nc87+tN22mJOg1WnqEpBzRsEPajk
7duVjdU2FDCarERj53hxF41l/u8BxvMGbFnrAijv5SzspeLi+VCGF3wu53Bi6UHi
7WO1dIhZLMHepqgtwbDDudxwlKbG/JWjJ7OspCNqOpktpY8BFUAQ42ctYZcOHy2w
kdZDcK5IhRztkBH2aSWOhudYh/CF7BOu2nSUdtR5NwhXnsStmSbud7ahw3W5Jc9/
IVAFXf2brkrfIzwrY5kmjnEPJ8mjUyLepwSlOlMTBrehzQ==
-----END CERTIFICATE-----
@@ -0,0 +1,32 @@
Bag Attributes
localKeyID: 57 F6 9F DF B2 94 61 47 B0 B0 2E 3A F7 79 98 C8 15 5C 64 FA
friendlyName: example-client
Key Attributes: <No Attributes>
-----BEGIN PRIVATE KEY-----
MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQDDjHjdBJTA8WCk
k9lQ9cD+mY5sueN2b/0IES02Is4N1Pbd7mVNAtx6lCZyvnOPjiZ0SMh7jF7iVhGF
XWoq8gHWFmQxebH4i/YpBva2dlsUds+57GCyXAxsCgLfw7s7qHneiknHBKYkHPgZ
KDk7PafOyJeLO7Gl/2pxEjFGO2DqJi0qkts70GYf804yHT9f/ZBvTy81zwpSRC7o
gt6SQpq+2MXSSF3nLumChaQKctMvqynH2FFTzC9eYMkbvaSKh91N/cu5gpJ/mD/U
HPRnGQKKkYmHVBJdaxQIFf2t6NKZn9bRRUQzrKD0jFBjTUJWgo79bmSG0ZtuZ770
JT37LRQxAgMBAAECggEBALFWsT1vLCAWRxi55Hkhhhbe12UO55O/V+969E+GHOw6
WG9IBaJjl+Oqe3vP8CkeNw15/CbfVCVa/KS/5Mpxkuy+fbrIrHfFJkZqxE8Q4uSa
IyRvk/nstG8+XQDlFuWxkmQZcXMFak6R3wWhjtWEwqwrChQA3Mi366YDY6BXtTpi
MuLJVWXqYVF2RirWX3a7eosR4MYBCJ1S8NnpI1w4oZwuew6iYiCJNkURctxnGKOe
hoop/711PEyIb/qHETBIlReY9scvZgJQZMgGe5mTGgFmAA8Bzo4e1/F2QrVIi7wy
REOSj4U+tP/QrYMSPk0cMpH5ft5+2mfh/o+D1ugm/AkCgYEA2qt3P4vlm1TMmJru
vMYc8iZjfe7UQTnD+J1EgrASBI0JP565zSZKBRIXnheumWhWS+2KR1MP8N6KAu4Q
xE/E7TKfLuE1CPzn7Rh/WZAxDqGNeQthk0qTbeBZ4/mHPKvt8V7c+N4vLjZ00uul
bKitlLpb49T+8FpSgRmk+nrBsesCgYEA5O6LmxH1+ip5NoMHoZVBhLbgJkxoXEzn
xHT7On8CjQ9B0T85NW9O38/a2gyIb5dIavcrh4dMvOKX8hdrVlal3chrXFRemyvy
VzpW7HZpMwthJb/QSim+GQ4u2Jsm+eXQAswq/yfowls9mp7K81McBgdfcdWgWuqc
Ttf3dBRS71MCgYBCPpu7ppgCfKgWuykEC0iNm0+cwROQRv21fX9Z0BlFQ6y/l0hW
e/kEnDyHakk8tD0gd2/it6kmoWRSCukNb9UKvsE2pIc6l9NzIm7CJUqXUh3Xw7jl
tDwBl2vkAp4TpUpKSyDKTBUi/OBbXYQZ7OfZHLPly4sIKXTBI5G1SCcRzQKBgE3Y
Y7ZmRBs+CBumMJkcVLwZJ10+jBoeuIzQLoF0oNxwfLP3aK+n2s02M7gDnP09Fw3w
YGfkdQ6hioNyvTgNnLcuqHsr4UOojO5mUxpzJVCLJQyeDei6Cv8gGIPlNjqxj6rz
3AcJfhdgyTDjoCVfdF8EcMsXlkB0GJSi0W0yYAUrAoGAVK5v0cbEEV+gxdQgh848
FwSQXx7i9jVOB2BUTLX2Q+gLUJibtFzLnh03sGCXtTa8sK+hLW4FCMWZa9Ev5ZjG
ylFyA96U97CNpS+2/ZW5+hvRZ/GuY/0yB8w26BeS38lI8RnmxSPi5cnexdl32NIX
5FkU9WJ4HKmi1n0304CkF+4=
-----END PRIVATE KEY-----
Binary file not shown.
+9 -7
View File
@@ -4,12 +4,13 @@
<parent>
<groupId>timely</groupId>
<artifactId>timely</artifactId>
<version>1.2.1-SNAPSHOT</version>
<version>2.1.5</version>
</parent>
<artifactId>timely-client</artifactId>
<properties>
<!-- PEP 440 compliant version gets substituted into src/main/python/template/setup.py -->
<python.client.version>1.2.1.dev0</python.client.version>
<!-- use .dev0 instead of -SNAPSHOT -->
<python.client.version>2.1.5</python.client.version>
</properties>
<dependencies>
<dependency>
@@ -52,6 +53,10 @@
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
</dependency>
<dependency>
<groupId>timely</groupId>
<artifactId>timely-common</artifactId>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-api</artifactId>
@@ -62,21 +67,18 @@
<artifactId>log4j-core</artifactId>
<scope>runtime</scope>
</dependency>
<!--
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-slf4j-impl</artifactId>
<scope>runtime</scope>
</dependency>
-->
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>timely</groupId>
<artifactId>timely-common</artifactId>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<plugins>
@@ -7,7 +7,7 @@ public class AddSubscription extends SubscriptionRequest {
private String operation = "add";
private String metric = null;
private Optional<Map<String, String>> tags = Optional.empty();
private Optional<Map<String,String>> tags = Optional.empty();
private Optional<Long> startTime = Optional.empty();
private Optional<Long> endTime = Optional.empty();
private Optional<Long> delayTime = Optional.empty();
@@ -20,7 +20,7 @@ public class AddSubscription extends SubscriptionRequest {
return metric;
}
public Optional<Map<String, String>> getTags() {
public Optional<Map<String,String>> getTags() {
return tags;
}
@@ -36,7 +36,7 @@ public class AddSubscription extends SubscriptionRequest {
this.metric = metric;
}
public void setTags(Optional<Map<String, String>> tags) {
public void setTags(Optional<Map<String,String>> tags) {
this.tags = tags;
}
@@ -71,9 +71,8 @@ public class AddSubscription extends SubscriptionRequest {
}
if (obj instanceof AddSubscription) {
AddSubscription other = (AddSubscription) obj;
return (this.subscriptionId.equals(other.subscriptionId) && this.metric.equals(other.metric)
&& this.tags.equals(other.tags) && this.startTime.equals(other.startTime)
&& this.endTime.equals(other.endTime) && this.delayTime.equals(other.delayTime));
return (this.subscriptionId.equals(other.subscriptionId) && this.metric.equals(other.metric) && this.tags.equals(other.tags)
&& this.startTime.equals(other.startTime) && this.endTime.equals(other.endTime) && this.delayTime.equals(other.delayTime));
} else {
return false;
}
@@ -1,45 +0,0 @@
package timely.api;
public class BasicAuthLogin {
protected String username;
protected String password;
public String getUsername() {
return username;
}
public void setUsername(String user) {
this.username = user;
}
public String getPassword() {
return password;
}
public void setPassword(String password) {
this.password = password;
}
@Override
public int hashCode() {
return (username + password).hashCode();
}
@Override
public boolean equals(Object obj) {
if (null == obj) {
return false;
}
if (obj == this) {
return true;
}
if (obj instanceof BasicAuthLogin) {
BasicAuthLogin other = (BasicAuthLogin) obj;
return (this.username.equals(other.username) && this.password.equals(other.password));
} else {
return false;
}
}
}
@@ -0,0 +1,10 @@
package timely.api;
public class VersionRequest {
private String operation = "version";
public String getOperation() {
return operation;
}
}
@@ -5,9 +5,13 @@ import java.util.List;
import org.apache.commons.lang3.builder.EqualsBuilder;
import org.apache.commons.lang3.builder.HashCodeBuilder;
import org.apache.commons.lang3.builder.ToStringBuilder;
import com.fasterxml.jackson.annotation.JsonPropertyOrder;
import timely.model.Metric;
import timely.model.Tag;
@JsonPropertyOrder({"metric", "timestamp", "value", "tags", "subscriptionId", "complete"})
public class MetricResponse {
private String metric;
@@ -15,8 +15,8 @@ import org.glassfish.tyrus.client.SslContextConfigurator;
public class HttpClient {
public static SSLContext getSSLContext(String trustStoreFile, String trustStoreType, String trustStorePass,
String keyStoreFile, String keyStoreType, String keyStorePass) {
public static SSLContext getSSLContext(String trustStoreFile, String trustStoreType, String trustStorePass, String keyStoreFile, String keyStoreType,
String keyStorePass) {
SslContextConfigurator scc = new SslContextConfigurator();
scc.setTrustStoreFile(trustStoreFile);
scc.setTrustStoreType(trustStoreType);
@@ -31,12 +31,10 @@ public class HttpClient {
return scc.createSSLContext();
}
public static CloseableHttpClient get(String trustStoreFile, String trustStoreType, String trustStorePass,
String keyStoreFile, String keyStoreType, String keyStorePass, CookieStore cookieStore,
boolean hostVerificationEnabled) {
public static CloseableHttpClient get(String trustStoreFile, String trustStoreType, String trustStorePass, String keyStoreFile, String keyStoreType,
String keyStorePass, CookieStore cookieStore, boolean hostVerificationEnabled) {
SSLContext ssl = getSSLContext(trustStoreFile, trustStoreType, trustStorePass, keyStoreFile, keyStoreType,
keyStorePass);
SSLContext ssl = getSSLContext(trustStoreFile, trustStoreType, trustStorePass, keyStoreFile, keyStoreType, keyStorePass);
return get(ssl, cookieStore, hostVerificationEnabled);
}
@@ -44,8 +42,7 @@ public class HttpClient {
return get(ssl, cookieStore, hostVerificationEnabled, false);
}
public static CloseableHttpClient get(SSLContext ssl, CookieStore cookieStore, boolean hostVerificationEnabled,
boolean clientAuth) {
public static CloseableHttpClient get(SSLContext ssl, CookieStore cookieStore, boolean hostVerificationEnabled, boolean clientAuth) {
HostnameVerifier hostnameVerifier;
if (hostVerificationEnabled) {
hostnameVerifier = SSLConnectionSocketFactory.getDefaultHostnameVerifier();
@@ -55,16 +52,15 @@ public class HttpClient {
RequestConfig defaultRequestConfig = RequestConfig.custom().setCookieSpec(CookieSpecs.STANDARD).build();
HttpClientBuilder builder;
if (cookieStore == null) {
builder = HttpClients.custom().setSSLContext(ssl).disableCookieManagement()
.setDefaultRequestConfig(defaultRequestConfig).setSSLHostnameVerifier(hostnameVerifier);
builder = HttpClients.custom().setSSLContext(ssl).disableCookieManagement().setDefaultRequestConfig(defaultRequestConfig)
.setSSLHostnameVerifier(hostnameVerifier);
} else {
builder = HttpClients.custom().setSSLContext(ssl).setDefaultCookieStore(cookieStore)
.setDefaultRequestConfig(defaultRequestConfig).setSSLHostnameVerifier(hostnameVerifier);
builder = HttpClients.custom().setSSLContext(ssl).setDefaultCookieStore(cookieStore).setDefaultRequestConfig(defaultRequestConfig)
.setSSLHostnameVerifier(hostnameVerifier);
}
if (clientAuth) {
SSLConnectionSocketFactory sslCSF = new SSLConnectionSocketFactory(ssl,
new String[] { "TLSv1.1", "TLSv1.2" }, null, hostnameVerifier);
SSLConnectionSocketFactory sslCSF = new SSLConnectionSocketFactory(ssl, new String[] {"TLSv1.1", "TLSv1.2"}, null, hostnameVerifier);
builder.setSSLSocketFactory(sslCSF);
}
return builder.build();
@@ -1,20 +1,22 @@
package timely.client.tcp;
import java.io.BufferedWriter;
import java.io.IOException;
import java.io.PrintWriter;
import java.io.OutputStreamWriter;
import java.net.Socket;
import java.nio.charset.StandardCharsets;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
public class TcpClient implements AutoCloseable {
private static final Logger LOG = LoggerFactory.getLogger(TcpClient.class);
private static final Logger log = LoggerFactory.getLogger(TcpClient.class);
private final String host;
private final int port;
private Socket sock = null;
private PrintWriter out = null;
private BufferedWriter out = null;
private long connectTime = 0L;
private long backoff = 2000;
private int bufferSize = -1;
@@ -32,7 +34,7 @@ public class TcpClient implements AutoCloseable {
/**
* Opens a TCP connection to the specified host and port
*
*
* @throws IOException
* if an error occurs
*/
@@ -44,10 +46,9 @@ public class TcpClient implements AutoCloseable {
/**
* Write a metric to Timely
*
*
* @param metric
* newline terminated string representation of Timely metric
* @throws IOException
*/
public synchronized void write(String metric) throws IOException {
if (connect() != 0) {
@@ -60,7 +61,7 @@ public class TcpClient implements AutoCloseable {
}
}
public synchronized void flush() {
public synchronized void flush() throws IOException {
if (null != out) {
out.flush();
writesSinceFlush = 0;
@@ -69,13 +70,13 @@ public class TcpClient implements AutoCloseable {
/**
* Closes the tcp connection to Timely
*
*
* @throws IOException
* if an error occurs
*/
@Override
public void close() throws IOException {
LOG.trace("Shutting down connection to Timely at {}:{}", host, port);
log.trace("Shutting down connection to Timely at {}:{}", host, port);
if (null != sock) {
try {
if (null != out) {
@@ -86,7 +87,7 @@ public class TcpClient implements AutoCloseable {
}
sock.close();
} catch (IOException e) {
LOG.error("Error closing connection to Timely at " + host + ":" + port + ". Error: " + e.getMessage());
log.error("Error closing connection to Timely at " + host + ":" + port + ". Error: " + e.getMessage());
} finally {
sock = null;
}
@@ -94,24 +95,49 @@ public class TcpClient implements AutoCloseable {
}
private synchronized int connect() {
if (null == sock || !sock.isConnected() || out.checkError()) {
if (null == sock || !sock.isConnected()) {
if (System.currentTimeMillis() > (connectTime + backoff)) {
OutputStreamWriter osw = null;
try {
connectTime = System.currentTimeMillis();
sock = new Socket(host, port);
out = new PrintWriter(sock.getOutputStream(), false);
osw = new OutputStreamWriter(sock.getOutputStream(), StandardCharsets.UTF_8);
out = new BufferedWriter(osw);
backoff = 2000;
LOG.trace("Connected to Timely at {}:{}", host, port);
} catch (IOException e) {
LOG.error("Error connecting to Timely at {}:" + host + ":" + port + ". Error: " + e.getMessage());
log.trace("Connected to Timely at {}:{}", host, port);
} catch (Exception e) {
log.error("Error connecting to Timely at {}:{} - {}", host, port, e.getMessage());
if (sock != null) {
try {
sock.close();
} catch (IOException e1) {
log.error(e1.getMessage());
} finally {
sock = null;
}
}
if (osw != null) {
try {
osw.close();
} catch (IOException e1) {
log.error(e1.getMessage());
}
}
if (out != null) {
try {
out.close();
} catch (IOException e1) {
log.error(e1.getMessage());
} finally {
out = null;
}
}
backoff = backoff * 2;
sock = null;
out = null;
LOG.info("Will retry connection in {} ms.", backoff);
log.info("Will retry connection in {} ms.", backoff);
return -1;
}
} else {
LOG.warn("Not writing to Timely, waiting to reconnect");
log.warn("Not writing to Timely, waiting to reconnect");
return -1;
}
}
@@ -12,7 +12,7 @@ import org.slf4j.LoggerFactory;
public class UdpClient implements AutoCloseable {
private static final Logger LOG = LoggerFactory.getLogger(UdpClient.class);
private static final Logger log = LoggerFactory.getLogger(UdpClient.class);
private final String host;
private final int port;
@@ -38,12 +38,11 @@ public class UdpClient implements AutoCloseable {
throw new IllegalStateException("Must call open first");
}
this.packet.setData(metric.getBytes(UTF_8));
LOG.info("writing '" + metric + "' to " + this.host + ":" + this.port);
log.info("writing '" + metric + "' to " + this.host + ":" + this.port);
this.sock.send(packet);
}
public void flush() throws IOException {
}
public void flush() throws IOException {}
public void close() throws IOException {
if (null != sock) {
@@ -6,7 +6,6 @@ import java.util.Map;
import javax.websocket.CloseReason;
import javax.websocket.Endpoint;
import javax.websocket.EndpointConfig;
import javax.websocket.MessageHandler;
import javax.websocket.Session;
import org.slf4j.Logger;
@@ -14,30 +13,25 @@ import org.slf4j.LoggerFactory;
public abstract class ClientHandler extends Endpoint {
private final static Logger LOG = LoggerFactory.getLogger(ClientHandler.class);
private final static Logger log = LoggerFactory.getLogger(ClientHandler.class);
@Override
public void onOpen(Session session, EndpointConfig config) {
LOG.info("Websocket session {} opened.", session.getId());
session.addMessageHandler(new MessageHandler.Whole<String>() {
@Override
public void onMessage(String message) {
LOG.info("Message received on Websocket session {}: {}", session.getId(), message);
}
log.info("Websocket session {} opened.", session.getId());
session.addMessageHandler(String.class, message -> {
log.info("Message received on Websocket session {}: {}", session.getId(), message);
});
}
@Override
public void onClose(Session session, CloseReason reason) {
LOG.info("Websocket session {} closed.", session.getId());
log.info("Websocket session {} closed.", session.getId());
}
@Override
public void onError(Session session, Throwable error) {
LOG.error("Error occurred on Websocket session" + session.getId(), error);
log.error("Error occurred on Websocket session" + session.getId(), error);
}
public void beforeRequest(Map<String, List<String>> headers) {
}
public void beforeRequest(Map<String,List<String>> headers) {}
}
@@ -36,7 +36,7 @@ public class TimelyEndpointConfig implements ClientEndpointConfig {
}
@Override
public Map<String, Object> getUserProperties() {
public Map<String,Object> getUserProperties() {
return Collections.emptyMap();
}
@@ -55,7 +55,7 @@ public class TimelyEndpointConfig implements ClientEndpointConfig {
return new Configurator() {
@Override
public void beforeRequest(Map<String, List<String>> headers) {
public void beforeRequest(Map<String,List<String>> headers) {
super.beforeRequest(headers);
if (null != sessionCookie) {
headers.put("Cookie", Collections.singletonList(sessionCookie));
@@ -10,17 +10,13 @@ import javax.net.ssl.SSLContext;
import javax.websocket.DeploymentException;
import javax.websocket.Session;
import com.google.common.base.Preconditions;
import org.apache.commons.lang3.StringUtils;
import org.apache.http.HttpResponse;
import org.apache.http.HttpStatus;
import org.apache.http.client.HttpResponseException;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.client.methods.HttpRequestBase;
import org.apache.http.client.utils.HttpClientUtils;
import org.apache.http.cookie.Cookie;
import org.apache.http.entity.StringEntity;
import org.apache.http.impl.client.BasicCookieStore;
import org.apache.http.impl.client.CloseableHttpClient;
import org.glassfish.tyrus.client.ClientManager;
@@ -28,21 +24,20 @@ import org.glassfish.tyrus.client.ClientProperties;
import org.glassfish.tyrus.client.SslEngineConfigurator;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import timely.api.BasicAuthLogin;
import com.google.common.base.Preconditions;
import timely.client.http.HttpClient;
import timely.serialize.JsonSerializer;
public class WebSocketClient implements AutoCloseable {
private static final Logger LOG = LoggerFactory.getLogger(WebSocketClient.class);
private static final Logger log = LoggerFactory.getLogger(WebSocketClient.class);
private final String timelyHostname;
private final int timelyHttpsPort;
private final int timelyWssPort;
private final boolean doLogin;
private final boolean clientAuth;
private final String timelyUsername;
private final String timelyPassword;
private final boolean hostVerificationEnabled;
private final int bufferSize;
private final SSLContext ssl;
@@ -51,39 +46,27 @@ public class WebSocketClient implements AutoCloseable {
protected Session session = null;
protected volatile boolean closed = true;
protected WebSocketClient(SSLContext ssl, String timelyHostname, int timelyHttpsPort, int timelyWssPort,
boolean clientAuth, boolean doLogin, String timelyUsername, String timelyPassword,
boolean hostVerificationEnabled, int bufferSize) {
protected WebSocketClient(SSLContext ssl, String timelyHostname, int timelyHttpsPort, int timelyWssPort, boolean clientAuth, boolean doLogin,
boolean hostVerificationEnabled, int bufferSize) {
this.ssl = ssl;
this.timelyHostname = timelyHostname;
this.timelyHttpsPort = timelyHttpsPort;
this.timelyWssPort = timelyWssPort;
this.doLogin = doLogin;
this.clientAuth = clientAuth;
this.timelyUsername = timelyUsername;
this.timelyPassword = timelyPassword;
this.hostVerificationEnabled = hostVerificationEnabled;
this.bufferSize = bufferSize;
Preconditions.checkNotNull(timelyHostname, "%s must be supplied", "Timely host name");
Preconditions.checkNotNull(timelyHttpsPort, "%s must be supplied", "Timely HTTPS port");
Preconditions.checkNotNull(timelyWssPort, "%s must be supplied", "Timely WSS port");
if (doLogin && !clientAuth) {
if ((StringUtils.isEmpty(timelyUsername) && !StringUtils.isEmpty(timelyPassword)
|| (!StringUtils.isEmpty(timelyUsername) && StringUtils.isEmpty(timelyPassword)))) {
throw new IllegalArgumentException("Both Timely username and password must be empty or non-empty");
}
}
}
protected WebSocketClient(String timelyHostname, int timelyHttpsPort, int timelyWssPort, boolean clientAuth,
boolean doLogin, String timelyUsername, String timelyPassword, String keyStoreFile, String keyStoreType,
String keyStorePass, String trustStoreFile, String trustStoreType, String trustStorePass,
boolean hostVerificationEnabled, int bufferSize) {
this(HttpClient.getSSLContext(trustStoreFile, trustStoreType, trustStorePass, keyStoreFile, keyStoreType,
keyStorePass), timelyHostname, timelyHttpsPort, timelyWssPort, clientAuth, doLogin, timelyUsername,
timelyPassword, hostVerificationEnabled, bufferSize);
protected WebSocketClient(String timelyHostname, int timelyHttpsPort, int timelyWssPort, boolean clientAuth, boolean doLogin, String keyStoreFile,
String keyStoreType, String keyStorePass, String trustStoreFile, String trustStoreType, String trustStorePass,
boolean hostVerificationEnabled, int bufferSize) {
this(HttpClient.getSSLContext(trustStoreFile, trustStoreType, trustStorePass, keyStoreFile, keyStoreType, keyStorePass), timelyHostname,
timelyHttpsPort, timelyWssPort, clientAuth, doLogin, hostVerificationEnabled, bufferSize);
}
public void open(ClientHandler clientEndpoint) throws IOException, DeploymentException, URISyntaxException {
@@ -93,30 +76,16 @@ public class WebSocketClient implements AutoCloseable {
try (CloseableHttpClient client = HttpClient.get(ssl, cookieJar, hostVerificationEnabled, clientAuth)) {
String target = "https://" + timelyHostname + ":" + timelyHttpsPort + "/login";
HttpRequestBase request = null;
if (StringUtils.isEmpty(timelyUsername)) {
// HTTP GET to /login to use certificate based login
request = new HttpGet(target);
LOG.trace("Performing client certificate login");
} else {
// HTTP POST to /login to use username/password
BasicAuthLogin login = new BasicAuthLogin();
login.setUsername(timelyUsername);
login.setPassword(timelyPassword);
String payload = JsonSerializer.getObjectMapper().writeValueAsString(login);
HttpPost post = new HttpPost(target);
post.setEntity(new StringEntity(payload));
request = post;
LOG.trace("Performing BasicAuth login");
}
// HTTP GET to /login to use certificate based login
request = new HttpGet(target);
log.trace("Performing client certificate login");
HttpResponse response = null;
try {
response = client.execute(request);
if (response.getStatusLine().getStatusCode() != HttpStatus.SC_OK) {
throw new HttpResponseException(response.getStatusLine().getStatusCode(),
response.getStatusLine().getReasonPhrase());
throw new HttpResponseException(response.getStatusLine().getStatusCode(), response.getStatusLine().getReasonPhrase());
}
for (Cookie c : cookieJar.getCookies()) {
if (c.getName().equals("TSESSIONID")) {
@@ -125,8 +94,7 @@ public class WebSocketClient implements AutoCloseable {
}
}
if (null == sessionCookie) {
throw new IllegalStateException(
"Unable to find TSESSIONID cookie header in Timely login response");
throw new IllegalStateException("Unable to find TSESSIONID cookie header in Timely login response");
}
} finally {
HttpClientUtils.closeQuietly(response);
@@ -142,15 +110,14 @@ public class WebSocketClient implements AutoCloseable {
webSocketClient.getProperties().put(ClientProperties.SSL_ENGINE_CONFIGURATOR, sslEngine);
webSocketClient.getProperties().put(ClientProperties.INCOMING_BUFFER_SIZE, bufferSize);
String wssPath = "wss://" + timelyHostname + ":" + timelyWssPort + "/websocket";
session = webSocketClient.connectToServer(clientEndpoint,
new TimelyEndpointConfig(clientEndpoint, sessionCookie), new URI(wssPath));
session = webSocketClient.connectToServer(clientEndpoint, new TimelyEndpointConfig(clientEndpoint, sessionCookie), new URI(wssPath));
final ByteBuffer pingData = ByteBuffer.allocate(0);
webSocketClient.getScheduledExecutorService().scheduleAtFixedRate(() -> {
try {
session.getBasicRemote().sendPing(pingData);
} catch (Exception e) {
LOG.error("Error sending ping", e);
log.error("Error sending ping", e);
}
}, 30, 60, TimeUnit.SECONDS);
closed = false;
@@ -13,37 +13,35 @@ import javax.websocket.DeploymentException;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import timely.api.AddSubscription;
import timely.api.CloseSubscription;
import timely.api.CreateSubscription;
import timely.api.RemoveSubscription;
import timely.api.VersionRequest;
import timely.client.websocket.ClientHandler;
import timely.client.websocket.WebSocketClient;
import timely.serialize.JsonSerializer;
public class WebSocketSubscriptionClient extends WebSocketClient {
private static final Logger LOG = LoggerFactory.getLogger(WebSocketSubscriptionClient.class);
private static final Logger log = LoggerFactory.getLogger(WebSocketSubscriptionClient.class);
private final String subscriptionId;
public WebSocketSubscriptionClient(SSLContext ssl, String timelyHostname, int timelyHttpsPort, int timelyWssPort,
boolean clientAuth, boolean doLogin, String timelyUsername, String timelyPassword,
boolean hostVerificationEnabled, int bufferSize) {
super(ssl, timelyHostname, timelyHttpsPort, timelyWssPort, clientAuth, doLogin, timelyUsername, timelyPassword,
hostVerificationEnabled, bufferSize);
public WebSocketSubscriptionClient(SSLContext ssl, String timelyHostname, int timelyHttpsPort, int timelyWssPort, boolean clientAuth, boolean doLogin,
boolean hostVerificationEnabled, int bufferSize) {
super(ssl, timelyHostname, timelyHttpsPort, timelyWssPort, clientAuth, doLogin, hostVerificationEnabled, bufferSize);
subscriptionId = UUID.randomUUID().toString();
LOG.trace("Created WebSocketClient with subscriptionId {}", this.subscriptionId);
log.trace("Created WebSocketClient with subscriptionId {}", this.subscriptionId);
}
public WebSocketSubscriptionClient(String timelyHostname, int timelyHttpsPort, int timelyWssPort,
boolean clientAuth, boolean doLogin, String timelyUsername, String timelyPassword, String keyStoreFile,
String keyStoreType, String keyStorePass, String trustStoreFile, String trustStoreType,
String trustStorePass, boolean hostVerificationEnabled, int bufferSize) {
super(timelyHostname, timelyHttpsPort, timelyWssPort, clientAuth, doLogin, timelyUsername, timelyPassword,
keyStoreFile, keyStoreType, keyStorePass, trustStoreFile, trustStoreType, trustStorePass,
hostVerificationEnabled, bufferSize);
public WebSocketSubscriptionClient(String timelyHostname, int timelyHttpsPort, int timelyWssPort, boolean clientAuth, boolean doLogin, String keyStoreFile,
String keyStoreType, String keyStorePass, String trustStoreFile, String trustStoreType, String trustStorePass,
boolean hostVerificationEnabled, int bufferSize) {
super(timelyHostname, timelyHttpsPort, timelyWssPort, clientAuth, doLogin, keyStoreFile, keyStoreType, keyStorePass, trustStoreFile, trustStoreType,
trustStorePass, hostVerificationEnabled, bufferSize);
subscriptionId = UUID.randomUUID().toString();
LOG.trace("Created WebSocketClient with subscriptionId {}", this.subscriptionId);
log.trace("Created WebSocketClient with subscriptionId {}", this.subscriptionId);
}
@Override
@@ -63,14 +61,13 @@ public class WebSocketSubscriptionClient extends WebSocketClient {
session.getBasicRemote().sendText(JsonSerializer.getObjectMapper().writeValueAsString(close));
session.close(new CloseReason(CloseReason.CloseCodes.NORMAL_CLOSURE, "Client called close."));
} catch (Exception e) {
LOG.info("Unable to send close message to server: {}", e.getMessage());
log.info("Unable to send close message to server: {}", e.getMessage());
}
}
super.close();
}
public Future<Void> addSubscription(String metric, Map<String, String> tags, long startTime, long endTime,
long delayTime) throws IOException {
public Future<Void> addSubscription(String metric, Map<String,String> tags, long startTime, long endTime, long delayTime) throws IOException {
AddSubscription add = new AddSubscription();
add.setSubscriptionId(subscriptionId);
add.setMetric(metric);
@@ -88,4 +85,8 @@ public class WebSocketSubscriptionClient extends WebSocketClient {
return session.getAsyncRemote().sendText(JsonSerializer.getObjectMapper().writeValueAsString(remove));
}
public Future<Void> requestVersion() throws Exception {
VersionRequest version = new VersionRequest();
return session.getAsyncRemote().sendText(JsonSerializer.getObjectMapper().writeValueAsString(version));
}
}
+10 -3
View File
@@ -1,8 +1,15 @@
<?xml version="1.0" encoding="UTF-8"?>
<FindBugsFilter>
<Match>
<Bug pattern="DM_DEFAULT_ENCODING"/>
<Method name="connect" />
<Class name="timely.client.tcp.TcpClient"/>
<!-- these are probably avoidable, but needs further triage to fix -->
<Bug pattern="MS_EXPOSE_REP" />
</Match>
<Match>
<!-- these are probably avoidable, but needs further triage to fix -->
<Bug pattern="EI_EXPOSE_REP" />
</Match>
<Match>
<!-- these are probably avoidable, but needs further triage to fix -->
<Bug pattern="EI_EXPOSE_REP2" />
</Match>
</FindBugsFilter>
@@ -1,22 +1,16 @@
package timely.api;
import com.fasterxml.jackson.databind.ObjectMapper;
import org.junit.Assert;
import org.junit.Test;
import com.fasterxml.jackson.databind.ObjectMapper;
import timely.serialize.JsonSerializer;
public class SerializationTest {
private ObjectMapper mapper = JsonSerializer.getObjectMapper();
@Test
public void testBasicAuth() throws Exception {
BasicAuthLogin login = new BasicAuthLogin();
login.setUsername("test");
login.setPassword("pass");
testSerialization(login);
}
@Test
public void testCreateSubscription() throws Exception {
CreateSubscription create = new CreateSubscription();
@@ -47,9 +41,9 @@ public class SerializationTest {
testSerialization(remove);
}
@SuppressWarnings("unchecked")
private <T> void testSerialization(T o) throws Exception {
String ser = mapper.writeValueAsString(o);
@SuppressWarnings("unchecked")
T des = mapper.readValue(ser, (Class<T>) o.getClass());
Assert.assertEquals(o, des);
}
@@ -6,9 +6,10 @@ import static org.junit.Assert.assertTrue;
import java.io.IOException;
import com.fasterxml.jackson.databind.ObjectMapper;
import org.junit.Test;
import com.fasterxml.jackson.databind.ObjectMapper;
/**
*
*
@@ -30,8 +31,7 @@ public class MetricTest {
Metric m4 = Metric.newBuilder().name("m4").tag("t2", "v2").tag("t1", "v1").value(1, 0.0).build();
assertFalse(m1.equals(m4));
Metric m5 = Metric.newBuilder().name("m1").tag("t3", "v3").tag("t2", "v2").tag("t1", "v1").value(1, 0.0)
.build();
Metric m5 = Metric.newBuilder().name("m1").tag("t3", "v3").tag("t2", "v2").tag("t1", "v1").value(1, 0.0).build();
assertFalse(m1.equals(m5));
Metric m6 = Metric.newBuilder().name("m1").tag("t2", "v2").tag("t1", "v1").value(2, 0.0).build();
@@ -5,6 +5,7 @@ import java.util.List;
import org.junit.Assert;
import org.junit.Test;
import timely.model.Metric;
import timely.model.Tag;
@@ -7,6 +7,7 @@ import java.util.TreeMap;
import org.junit.Assert;
import org.junit.Test;
import timely.model.Tag;
public class TagListParserTest {
@@ -31,7 +32,7 @@ public class TagListParserTest {
@Test
public void testListCombineMap() {
Map<String, String> map = new TreeMap<>();
Map<String,String> map = new TreeMap<>();
map.put("tag1", "value1");
map.put("tag2", "value2");
String combined = new TagListParser().combine(map);
+14 -1
View File
@@ -4,7 +4,7 @@
<parent>
<groupId>timely</groupId>
<artifactId>timely</artifactId>
<version>1.2.1-SNAPSHOT</version>
<version>2.1.5</version>
</parent>
<artifactId>collectd-nsq-plugin</artifactId>
<dependencies>
@@ -20,9 +20,14 @@
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpclient</artifactId>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>jcl-over-slf4j</artifactId>
</dependency>
<dependency>
<groupId>timely</groupId>
<artifactId>collectd-plugin-parent</artifactId>
<version>${project.version}</version>
</dependency>
</dependencies>
<build>
@@ -34,6 +39,14 @@
<artifactSet>
<includes />
</artifactSet>
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>**/module-info.class</exclude>
</excludes>
</filter>
</filters>
<createDependencyReducedPom>false</createDependencyReducedPom>
</configuration>
<executions>
Loaded 100 of 682 files, more files were not shown because too many files have changed in this diff. Show more