Previously, we used Java to write .sql files, imported them using
psql, and afterwards made queries via psql. Now we're using Java to
interact with the database directly. This is another step towards
making the daily updater Java-only.
Over two years ago, in commit 1f90b72 from October 2016, we made our
user graphs faster by avoiding to read the large .csv file on demand.
Instead we read it once as part of the daily update, saved it to disk
as .RData file using R's save() function, and loaded it back to memory
using R's load() function when drawing a graph.
This approach worked okay. It just had two disadvantages:
1. We had to write a small amount of R code for each graph type,
which is why we only did it for graphs with large .csv files.
2. Running these small R script as part of the daily update made it
harder to move away from Ant towards a Java-only execution model.
The new approach implemented in this commit uses read_csv() fromt the
readr package which reads CSV files several times faster than
read.csv().
Requires installing the readr package from CRAN, which is available on
Debian in stretch-backports and later as r-cran-readr.
Implements #28799.
Changes include using similar mechanisms for configuration, calling
the database aggregation function, querying the database, and writing
results as we're using in the ipv6servers and other modules.
Configuration options can now be changed via the following Java
properties:
bwhist.descriptors
bwhist.database
bwhist.history
bwhist.output
The legacy.config file, if one exists, will be ignored.
Part of #28116.
So far we computed percentiles in the advbwdist module using our own
formula that picked existing values for the various percentiles we
provided.
Now we're using Apache Commons Math with method R_7 which uses linear
interpolation when a percentile lies between two data points.
Implements another part of #26035.
This is using commit b6e6c3235806c9d1521dbc2c08cecee626466928 of atlas.git.
* Removes submodule for RS.
* Move two license files to basedirectory; these should be merged.
* Move atlas.css into css folder
* Fonts in both were identical
* Changes all findable paths to the new setting
* js libs were simply copied; these should be sorted out more
* Merged rs/img and rs/images folder into images
Thanks to iwakeh for the bulk of the work in this commit.
Note that a side-effect of this change is that the generated news.json
uses slightly different formatting. These are just cosmetic changes
that don't have any effect on provided website content.
Implements #26169.
In this change we:
- update metrics-lib to 2.2.0,
- start downloading and processing logs from CollecTor rather than
from webstats.torproject.org,
- change log line counts from int to long,
- remove webstats tests which are now contained in metrics-lib,
- update the CollecTor page, and
- take out the beta notice from the Tor web server logs page.
Implements #25520.
This commit updates UpdateNews to use the existing News class and to
accept the output filename as an argument. It also enables pretty
printing the output to remove the additional manual step of piping the
output through jq. (See #23854 comment:9)
The new Ant task requires that the JAR be built prior to running. It
will update news.json in place.
- Rename root package org.torproject.metrics to
org.torproject.metrics.stats to make it part of the Tor Metrics
name space.
- ernie.cron becomes org.torproject.metrics.stats.servers to get rid
of the legacy name and to reflect that it's all about servers and
bandwidth (provided by servers).
- org.torproject.metrics.web remains unchanged.
- org.torproject.metrics.web.graphs and
org.torproject.metrics.web.research are merged into
org.torproject.metrics.web.
- Remove special treatment for legacy package names, which is not
necessary anymore.
Resolves#24294.
This includes moving the Java and Sql sources as well as test sources,
and adaption of paths in test classes. Created an ant task 'ipv6server'
and added its call to run-web-prepare target.
