Magic-Mirror/gecko-dev
1
0
Fork 0
mirror of https://github.com/mozilla/gecko-dev.git synced 2024-11-23 21:01:08 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

Bug 1698518 - Upgrade Glean to get new time API. r=chutten

Browse Source 
This also includes an upgrade of glean_parser to v2.5.0
This was done using the following command:

    ./mach vendor python glean_parser==2.5.0

Differential Revision: https://phabricator.services.mozilla.com/D108448
This commit is contained in:
Jan-Erik Rediger 2021-03-18 11:21:27 +00:00
parent 12c05bb362
commit e9bbd20895
125 changed files with 4914 additions and 1294 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

32
Cargo.lock generated
View File

@ -2071,9 +2071,9 @@ dependencies = [
[[package]]
name = "glean"
version = "34.1.0"
version = "36.0.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8193d78e62cd2e5e1e0d430e5ca0462c5998c3ac1ba8b723702edb3bb3a47689"
checksum = "27c487254df8abcae3b3ff1a8d5ceb8f7d39b05e67d58f0b89898c20844da41d"
dependencies = [
"chrono",
"crossbeam-channel 0.5.0",
@ -2086,13 +2086,14 @@ dependencies = [
"thiserror",
"time",
"uuid",
"whatsys",
]
[[package]]
name = "glean-core"
version = "34.1.0"
version = "36.0.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f91bd1df09ff9a8111111c9202e57244873b4e25fc2f5909329230c0dce60d3c"
checksum = "0a249ed8bd8588cf97d03de14e0a8ebe4e637c52581fb497bfa0f2bbf7e85018"
dependencies = [
"bincode",
"chrono",
@ -2104,6 +2105,7 @@ dependencies = [
"serde",
"serde_json",
"uuid",
"zeitstempel",
]
[[package]]
@ -5951,6 +5953,17 @@ dependencies = [
"wgpu-types",
]
[[package]]
name = "whatsys"
version = "0.1.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c24fff5aa1e0973964ba23a995e8b10fa2cdeae507e0cbbbd36f8403242a765d"
dependencies = [
"cc",
"cfg-if 1.0.0",
"libc",
]
[[package]]
name = "winapi"
version = "0.2.8"
@ -6139,6 +6152,17 @@ dependencies = [
"linked-hash-map",
]
[[package]]
name = "zeitstempel"
version = "0.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f2837f9ad7a7a8c88d1cc50cb0c3d202ce6e178aa6ebf3e49b29561896a61f1d"
dependencies = [
"cfg-if 1.0.0",
"libc",
"once_cell",
]
[[package]]
name = "zip"
version = "0.4.2"

2
third_party/python/glean_parser/.circleci/config.yml vendored
View File

@ -107,7 +107,7 @@ jobs:
build-39:
docker:
- image: circleci/python:3.9.0rc1
- image: circleci/python:3.9.0
steps:
- test-start
- test-python-version

18
third_party/python/glean_parser/AUTHORS.md vendored Normal file
View File

@ -0,0 +1,18 @@
# Credits
## Development Lead
- Michael Droettboom <mdroettboom@mozilla.com>
- Alessio Placitelli <aplacitelli@mozilla.com>
## Contributors
- Frank Bertsch <frank@mozilla.com>
- Travis Long <tlong@mozilla.com>
## Acknowledgements
This package was created with
[Cookiecutter](https://github.com/audreyr/cookiecutter) and the
[audreyr/cookiecutter-pypackage](https://github.com/audreyr/cookiecutter-pypackage)
project template.

23
third_party/python/glean_parser/AUTHORS.rst vendored
View File

@ -1,23 +0,0 @@
=======
Credits
=======
Development Lead
----------------
* Michael Droettboom <mdroettboom@mozilla.com>
* Alessio Placitelli <aplacitelli@mozilla.com>
Contributors
------------
* Frank Bertsch <frank@mozilla.com>
* Travis Long <tlong@mozilla.com>
Acknowledgements
----------------
This package was created with Cookiecutter_ and the `audreyr/cookiecutter-pypackage`_ project template.
.. _Cookiecutter: https://github.com/audreyr/cookiecutter
.. _`audreyr/cookiecutter-pypackage`: https://github.com/audreyr/cookiecutter-pypackage

433
third_party/python/glean_parser/CHANGELOG.md vendored Normal file
View File

@ -0,0 +1,433 @@
# Changelog
## Unreleased
## 2.5.0 (2021-02-23)
- Add parser and object model support for `rate` metric type. ([bug 1645166](https://bugzilla.mozilla.org/show_bug.cgi?id=1645166))
- Add parser and object model support for telemetry_mirror property. ([bug 1685406](https://bugzilla.mozilla.org/show_bug.cgi?id=1685406))
- Update the Javascript template to match Glean.js expectations. ([bug 1693516](https://bugzilla.mozilla.org/show_bug.cgi?id=1693516))
- Glean.js has updated it's export strategy. It will now export each metric type as an independent module;
- Glean.js has dropped support for non ES6 modules.
- Add support for generating Typescript code. ([bug 1692157](https://bugzilla.mozilla.org/show_bug.cgi?id=1692157))
- The templates added generate metrics and pings code for Glean.js.
## 2.4.0 (2021-02-18)
- **Experimental:** `glean_parser` has a new subcommand `coverage` to convert raw coverage reports
into something consumable by coverage tools, such as codecov.io
- The path to the file that each metric is defined in is now stored on the
`Metric` object in `defined_in["filepath"]`.
## 2.3.0 (2021-02-17)
- Leverage the `glean_namespace` to provide correct import when building for Javascript.
## 2.2.0 (2021-02-11)
- The Kotlin generator now generates static build information that can be passed
into `Glean.initialize` to avoid calling the package manager at runtime.
## 2.1.0 (2021-02-10)
- Add support for generating Javascript code.
- The templates added generate metrics and pings code for Glean.js.
## 2.0.0 (2021-02-05)
- New versions 2.0.0 of the `metrics.yaml` and `pings.yaml` schemas now ship
with `glean_parser`. These schemas are different from version 1.0.0 in the
following ways:
- Bugs must be specified as URLs. Bug numbers are disallowed.
- The legacy ping names containing underscores are no longer allowed. These
included `deletion_request`, `bookmarks_sync`, `history_sync`,
`session_end`, `all_pings`, `glean_*`). In these cases, the `_` should be
replaced with `-`.
To upgrade your app or library to use the new schema, replace the version in
the `$schema` value with `2-0-0`.
- **Breaking change:** It is now an error to use bug numbers (rather than URLs)
in ping definitions.
- Add the line number that metrics and pings were originally defined in the yaml
files.
## 1.29.1 (2020-12-17)
- BUGFIX: Linter output can now be redirected correctly (1675771).
## 1.29.0 (2020-10-07)
- **Breaking change:** `glean_parser` will now return an error code when any of
the input files do not exist (unless the `--allow-missing-files` flag is
passed).
- Generated code now includes a comment next to each metric containing the name
of the metric in its original `snake_case` form.
- When metrics don't provide a `unit` parameter, it is not included in the
output (as provided by probe-scraper).
## 1.28.6 (2020-09-24)
- BUGFIX: Ensure Kotlin arguments are deterministically ordered
## 1.28.5 (2020-09-14)
- Fix deploy step to update pip before deploying to pypi.
## 1.28.4 (2020-09-14)
- The `SUPERFLUOUS_NO_LINT` warning has been removed from the glinter.
It likely did more harm than good, and makes it hard to make
`metrics.yaml` files that pass across different versions of
`glean_parser`.
- Expired metrics will now produce a linter warning, `EXPIRED_METRIC`.
- Expiry dates that are more than 730 days (\~2 years) in the future
will produce a linter warning, `EXPIRATION_DATE_TOO_FAR`.
- Allow using the Quantity metric type outside of Gecko.
- New parser configs `custom_is_expired` and `custom_validate_expires`
added. These are both functions that take the `expires` value of the
metric and return a bool. (See `Metric.is_expired` and
`Metric.validate_expires`). These will allow FOG to provide custom
validation for its version-based `expires` values.
## 1.28.3 (2020-07-28)
- BUGFIX: Support HashSet and Dictionary in the C\## generated code.
## 1.28.2 (2020-07-28)
- BUGFIX: Generate valid C\## code when using Labeled metric types.
## 1.28.1 (2020-07-24)
- BUGFIX: Add missing column to correctly render markdown tables in generated
documentation.
## 1.28.0 (2020-07-23)
- **Breaking change:** The internal ping `deletion-request` was misnamed in
pings.py causing the linter to not allow use of the correctly named ping for
adding legacy ids to. Consuming apps will need to update their metrics.yaml if
they are using `deletion_request` in any `send_in_pings` to `deletion-request`
after updating.
## 1.27.0 (2020-07-21)
- Rename the `data_category` field to `data_sensitivity` to be clearer.
## 1.26.0 (2020-07-21)
- Add support for JWE metric types.
- Add a `data_sensitivity` field to all metrics for specifying the type of data
collected in the field.
## 1.25.0 (2020-07-17)
- Add support for generating C\## code.
- BUGFIX: The memory unit is now correctly passed to the MemoryDistribution
metric type in Swift.
## 1.24.0 (2020-06-30)
- BUGFIX: look for metrics in send\_if\_empty pings. Metrics for these kinds of
pings were being ignored.
## 1.23.0 (2020-06-27)
- Support for Python 3.5 has been dropped.
- BUGFIX: The ordering of event extra keys will now match with their enum,
fixing a serious bug where keys of extras may not match the correct values in
the data payload. See <https://bugzilla.mozilla.org/show_bug.cgi?id=1648768>.
## 1.22.0 (2020-05-28)
- **Breaking change:** (Swift only) Combine all metrics and pings into a single
generated file `Metrics.swift`.
## 1.21.0 (2020-05-25)
- `glinter` messages have been improved with more details and to be more
actionable.
- A maximum of 10 `extra_keys` is now enforced for `event` metric types.
- BUGFIX: the `Lifetime` enum values now match the values of the implementation
in mozilla/glean.
## 1.20.4 (2020-05-07)
- BUGFIX: yamllint errors are now reported using the correct file name.
## 1.20.3 (2020-05-06)
- Support for using `timing_distribution`'s `time_unit` parameter to control
the range of acceptable values is documented. The default unit for this use
case is `nanosecond` to avoid creating a breaking change. See [bug
1630997](https://bugzilla.mozilla.org/show_bug.cgi?id=1630997) for more
information.
## 1.20.2 (2020-04-24)
- Dependencies that depend on the version of Python being used are now specified
using the [Declaring platform specific dependencies syntax in
setuptools](https://setuptools.readthedocs.io/en/latest/setuptools.html##declaring-platform-specific-dependencies).
This means that more recent versions of dependencies are likely to be
installed on Python 3.6 and later, and unnecessary backport libraries won't
be installed on more recent Python versions.
## 1.20.1 (2020-04-21)
- The minimum version of the runtime dependencies has been lowered to increase
compatibility with other tools. These minimum versions are now tested in CI,
in addition to testing the latest versions of the dependencies that was
already happening in CI.
## 1.20.0 (2020-04-15)
- **Breaking change:** glinter errors found during the `translate` command will
now return an error code. glinter warnings will be displayed, but not return
an error code.
- `glean_parser` now produces a linter warning when `user` lifetime metrics are
set to expire. See [bug
1604854](https://bugzilla.mozilla.org/show_bug.cgi?id=1604854) for additional
context.
## 1.19.0 (2020-03-18)
- **Breaking change:** The regular expression used to validate labels is
stricter and more correct.
- Add more information about pings to markdown documentation:
- State whether the ping includes client id;
- Add list of data review links;
- Add list of related bugs links.
- `glean_parser` now makes it easier to write external translation
functions for different language targets.
- BUGFIX: `glean_parser` now works on 32-bit Windows.
## 1.18.3 (2020-02-24)
- Dropped the `inflection` dependency.
- Constrained the `zipp` and `MarkupSafe` transitive dependencies to versions
that support Python 3.5.
## 1.18.2 (2020-02-14)
- BUGFIX: Fix rendering of first element of reason list.
## 1.18.1 (2020-02-14)
- BUGFIX: Reason codes are displayed in markdown output for built-in
pings as well.
- BUGFIX: Reason descriptions are indented correctly in markdown
output.
- BUGFIX: To avoid a compiler error, the `@JvmName` annotation isn't
added to private members.
## 1.18.0 (2020-02-13)
- **Breaking Change (Java API)** Have the metrics names in Java match the names
in Kotlin. See [Bug
1588060](https://bugzilla.mozilla.org/show_bug.cgi?id=1588060).
- The reasons a ping are sent are now included in the generated markdown
documentation.
## 1.17.3 (2020-02-05)
- BUGFIX: The version of Jinja2 now specifies < 3.0, since that version no
longer supports Python 3.5.
## 1.17.2 (2020-02-05)
- BUGFIX: Fixes an import error in generated Kotlin code.
## 1.17.1 (2020-02-05)
- BUGFIX: Generated Swift code now includes `import Glean`, unless generating
for a Glean-internal build.
## 1.17.0 (2020-02-03)
- Remove default schema URL from `validate_ping`
- Make `schema` argument required for CLI
- BUGFIX: Avoid default import in Swift code for Glean itself
- BUGFIX: Restore order of fields in generated Swift code
## 1.16.0 (2020-01-15)
- Support for `reason` codes on pings was added.
## 1.15.6 (2020-02-06)
- BUGFIX: The version of Jinja2 now specifies < 3.0, since that version no
longer supports Python 3.5 (backported from 1.17.3).
## 1.15.5 (2019-12-19)
- BUGFIX: Also allow the legacy name `all_pings` for `send_in_pings` parameter
on metrics
## 1.15.4 (2019-12-19)
- BUGFIX: Also allow the legacy name `all_pings`
## 1.15.3 (2019-12-13)
- Add project title to markdown template.
- Remove "Sorry about that" from markdown template.
- BUGFIX: Replace dashes in variable names to force proper naming
## 1.15.2 (2019-12-12)
- BUGFIX: Use a pure Python library for iso8601 so there is no compilation
required.
## 1.15.1 (2019-12-12)
- BUGFIX: Add some additional ping names to the non-kebab-case allow list.
## 1.15.0 (2019-12-12)
- Restrict new pings names to be kebab-case and change `all_pings` to
`all-pings`
## 1.14.0 (2019-12-06)
- `glean_parser` now supports Python versions 3.5, 3.6, 3.7 and 3.8.
## 1.13.0 (2019-12-04)
- The `translate` command will no longer clear extra files in the output
directory.
- BUGFIX: Ensure all newlines in comments are prefixed with comment markers
- BUGFIX: Escape Swift keywords in variable names in generated code
- Generate documentation for pings that are sent if empty
## 1.12.0 (2019-11-27)
- Reserve the `deletion_request` ping name
- Added a new flag `send_if_empty` for pings
## 1.11.0 (2019-11-13)
- The `glinter` command now performs `yamllint` validation on registry files.
## 1.10.0 (2019-11-11)
- The Kotlin linter `detekt` is now run during CI, and for local
testing if installed.
- Python 3.8 is now tested in CI (in addition to Python 3.7). Using
`tox` for this doesn't work in modern versions of CircleCI, so the
`tox` configuration has been removed.
- `yamllint` has been added to test the YAML files on CI.
- ⚠ Metric types that don't yet have implementations in glean-core
have been removed. This includes `enumeration`, `rate`, `usage`, and
`use_counter`, as well as many labeled metrics that don't exist.
## 1.9.5 (2019-10-22)
- Allow a Swift lint for generated code
- New lint: Restrict what metric can go into the `baseline` ping
- New lint: Warn for slight misspellings in ping names
- BUGFIX: change Labeled types labels from lists to sets.
## 1.9.4 (2019-10-16)
- Use lists instead of sets in Labeled types labels to ensure that the order of
the labels passed to the `metrics.yaml` is kept.
- `glinter` will now check for duplicate labels and error if there are any.
## 1.9.3 (2019-10-09)
- Add labels from Labeled types to the Extra column in the Markdown template.
## 1.9.2 (2019-10-08)
- BUGFIX: Don't call `is_internal_metric` on `Ping` objects.
## 1.9.1 (2019-10-07)
- Don't include Glean internal metrics in the generated markdown.
## 1.9.0 (2019-10-04)
- Glinter now warns when bug numbers (rather than URLs) are used.
- BUGFIX: add `HistogramType` and `MemoryUnit` imports in Kotlin generated code.
## 1.8.4 (2019-10-02)
- Removed unsupported labeled metric types.
## 1.8.3 (2019-10-02)
- Fix indentation for generated Swift code
## 1.8.2 (2019-10-01)
- Created labeled metrics and events in Swift code and wrap it in a
configured namespace
## 1.8.1 (2019-09-27)
- BUGFIX: `memory_unit` is now passed to the Kotlin generator.
## 1.8.0 (2019-09-26)
- A new parser config, `do_not_disable_expired`, was added to turn off the
feature that expired metrics are automatically disabled. This is useful if you
want to retain the disabled value that is explicitly in the `metrics.yaml`
file.
- `glinter` will now report about superfluous `no_lint` entries.
## 1.7.0 (2019-09-24)
- A `glinter` tool is now included to find common mistakes in metric naming
and setup. This check is run during `translate` and warnings will be
displayed. ⚠ These warnings will be treated as errors in a future revision.
## 1.6.1 (2019-09-17)
- BUGFIX: `GleanGeckoMetricsMapping` must include `LabeledMetricType`
and `CounterMetricType`.
## 1.6.0 (2019-09-17)
- NEW: Support for outputting metrics in Swift.
- BUGFIX: Provides a helpful error message when `geckoview_datapoint` is used on
an metric type that doesn't support GeckoView exfiltration.
- Generate a lookup table for Gecko categorical histograms in
`GleanGeckoMetricsMapping`.
- Introduce a 'Swift' output generator.
## 1.4.1 (2019-08-28)
- Documentation only.
## 1.4.0 (2019-08-27)
- Added support for generating markdown documentation from `metrics.yaml` files.
## 1.3.0 (2019-08-22)
- `quantity` metric type has been added.
## 1.2.1 (2019-08-13)
- BUGFIX: `includeClientId` was not being output for PingType.
## 1.2.0 (2019-08-13)
- `memory_distribution` metric type has been added.
- `custom_distribution` metric type has been added.
- `labeled_timespan` is no longer an allowed metric type.
## 1.1.0 (2019-08-05)
- Add a special `all_pings` value to `send_in_pings`.
## 1.0.0 (2019-07-29)
- First release to start following strict semver.
## 0.1.0 (2018-10-15)
- First release on PyPI.

178
third_party/python/glean_parser/CONTRIBUTING.md vendored Normal file
View File

@ -0,0 +1,178 @@
# Contributing
Contributions are welcome, and they are greatly appreciated! Every little bit
helps, and credit will always be given.
You can contribute in many ways:
## Types of Contributions
### Report Bugs
Report bugs at
[bugzilla](https://bugzilla.mozilla.org/enter_bug.cgi?assigned_to=nobody%40mozilla.org&bug_ignored=0&bug_severity=normal&bug_status=NEW&cf_fission_milestone=---&cf_fx_iteration=---&cf_fx_points=---&cf_status_firefox65=---&cf_status_firefox66=---&cf_status_firefox67=---&cf_status_firefox_esr60=---&cf_status_thunderbird_esr60=---&cf_tracking_firefox65=---&cf_tracking_firefox66=---&cf_tracking_firefox67=---&cf_tracking_firefox_esr60=---&cf_tracking_firefox_relnote=---&cf_tracking_thunderbird_esr60=---&product=Data%20Platform%20and%20Tools&component=Glean%3A%20SDK&contenttypemethod=list&contenttypeselection=text%2Fplain&defined_groups=1&flag_type-203=X&flag_type-37=X&flag_type-41=X&flag_type-607=X&flag_type-721=X&flag_type-737=X&flag_type-787=X&flag_type-799=X&flag_type-800=X&flag_type-803=X&flag_type-835=X&flag_type-846=X&flag_type-855=X&flag_type-864=X&flag_type-916=X&flag_type-929=X&flag_type-930=X&flag_type-935=X&flag_type-936=X&flag_type-937=X&form_name=enter_bug&maketemplate=Remember%20values%20as%20bookmarkable%20template&op_sys=Unspecified&priority=P3&&rep_platform=Unspecified&status_whiteboard=%5Btelemetry%3Aglean-rs%3Am%3F%5D&target_milestone=---&version=unspecified).
If you are reporting a bug, please include:
- Your operating system name and version.
- Any details about your local setup that might be helpful in troubleshooting.
- Detailed steps to reproduce the bug.
### Fix Bugs
Look through the GitHub issues for bugs. Anything tagged with "bug" and "help
wanted" is open to whoever wants to implement it.
### Implement Features
Look through the GitHub issues for features. Anything tagged with "enhancement"
and "help wanted" is open to whoever wants to implement it.
### Write Documentation
`glean_parser` could always use more documentation, whether as part of the
official `glean_parser` docs, in docstrings, or even on the web in blog posts,
articles, and such.
### Submit Feedback
The best way to send feedback is to file an issue at TODO
If you are proposing a feature:
- Explain in detail how it would work.
- Keep the scope as narrow as possible, to make it easier to implement.
- Remember that this is a volunteer-driven project, and that contributions are
welcome :)
## Get Started!
Ready to contribute? Here's how to set up `glean_parser` for local
development.
1. Fork the `glean_parser` repo on GitHub.
2. Clone your fork locally:
```sh
$ git clone git@github.com:your_name_here/glean_parser.git
```
3. Install your local copy into a virtualenv. Assuming you have
virtualenvwrapper installed, this is how you set up your fork for local
development:
```sh
$ mkvirtualenv glean_parser
$ cd glean_parser/
$ python setup.py develop
```
4. Create a branch for local development:
```sh
$ git checkout -b name-of-your-bugfix-or-feature
```
Now you can make your changes locally.
5. To test your changes to `glean_parser`:
Install the testing dependencies:
```sh
$ pip install -r requirements_dev.txt
```
Optionally, if you want to ensure that the generated Kotlin code lints
correctly, install a Java SDK, and then run:
```sh
$ make install-kotlin-linters
```
Then make sure that all lints and tests are passing:
```sh
$ make lint
$ make test
```
6. Commit your changes and push your branch to GitHub:
```sh
$ git add .
$ git commit -m "Your detailed description of your changes."
$ git push origin name-of-your-bugfix-or-feature
```
7. Submit a pull request through the GitHub website.
## Pull Request Guidelines
Before you submit a pull request, check that it meets these guidelines:
1. The pull request should include tests.
2. If the pull request adds functionality, the docs should be updated. Put your
new functionality into a function with a docstring, and describe
public-facing features in the docs.
3. The pull request should work for Python 3.6, 3.7, 3.8 and 3.9 (The CI system
will take care of testing all of these Python versions).
4. The pull request should update the changelog in `CHANGELOG.md`.
## Tips
To run a subset of tests:
```sh
$ py.test tests.test_glean_parser
```
## Deploying
A reminder for the maintainers on how to deploy.
Get a clean main branch with all of the changes from `upstream`:
```sh
$ git checkout main
$ git fetch upstream
$ git rebase upstream/main
```
- Update the header with the new version and date in `CHANGELOG.md`.
- (By using the setuptools-scm package, there is no need to update the
version anywhere else).
- Make sure all your changes are committed.
- Push the changes upstream. (Normally pushing directly without review
is frowned upon, but the `main` branch is protected from force
pushes and release tagging requires the same permissions as pushing
to `main`):
```sh
$ git push upstream main
```
- Wait for [continuous integration to
pass](https://circleci.com/gh/mozilla/glean_parser/tree/main) on main.
- Make the release on GitHub using [this
link](https://github.com/mozilla/glean_parser/releases/new)
- Both the tag and the release title should be in the form `vX.Y.Z`.
- Copy and paste the relevant part of the `CHANGELOG.md` file into the
description.
- Tagging the release will trigger a CI workflow which will build the
distribution of `glean_parser` and publish it to PyPI.
The continuous integration system will then automatically deploy to
PyPI.
See also the [instructions for updating the version of `glean_parser`
used by the Glean
SDK](https://mozilla.github.io/glean/book/dev/upgrading-glean-parser.html).

160
third_party/python/glean_parser/CONTRIBUTING.rst vendored
View File

@ -1,160 +0,0 @@
.. highlight:: shell
.. _bugzilla: https://bugzilla.mozilla.org/enter_bug.cgi?assigned_to=nobody%40mozilla.org&bug_ignored=0&bug_severity=normal&bug_status=NEW&cf_fission_milestone=---&cf_fx_iteration=---&cf_fx_points=---&cf_status_firefox65=---&cf_status_firefox66=---&cf_status_firefox67=---&cf_status_firefox_esr60=---&cf_status_thunderbird_esr60=---&cf_tracking_firefox65=---&cf_tracking_firefox66=---&cf_tracking_firefox67=---&cf_tracking_firefox_esr60=---&cf_tracking_firefox_relnote=---&cf_tracking_thunderbird_esr60=---&product=Data%20Platform%20and%20Tools&component=Glean%3A%20SDK&contenttypemethod=list&contenttypeselection=text%2Fplain&defined_groups=1&flag_type-203=X&flag_type-37=X&flag_type-41=X&flag_type-607=X&flag_type-721=X&flag_type-737=X&flag_type-787=X&flag_type-799=X&flag_type-800=X&flag_type-803=X&flag_type-835=X&flag_type-846=X&flag_type-855=X&flag_type-864=X&flag_type-916=X&flag_type-929=X&flag_type-930=X&flag_type-935=X&flag_type-936=X&flag_type-937=X&form_name=enter_bug&maketemplate=Remember%20values%20as%20bookmarkable%20template&op_sys=Unspecified&priority=P3&&rep_platform=Unspecified&status_whiteboard=%5Btelemetry%3Aglean-rs%3Am%3F%5D&target_milestone=---&version=unspecified
============
Contributing
============
Contributions are welcome, and they are greatly appreciated! Every little bit
helps, and credit will always be given.
You can contribute in many ways:
Types of Contributions
----------------------
Report Bugs
~~~~~~~~~~~
Report bugs at bugzilla_.
If you are reporting a bug, please include:
* Your operating system name and version.
* Any details about your local setup that might be helpful in troubleshooting.
* Detailed steps to reproduce the bug.
Fix Bugs
~~~~~~~~
Look through the GitHub issues for bugs. Anything tagged with "bug" and "help
wanted" is open to whoever wants to implement it.
Implement Features
~~~~~~~~~~~~~~~~~~
Look through the GitHub issues for features. Anything tagged with "enhancement"
and "help wanted" is open to whoever wants to implement it.
Write Documentation
~~~~~~~~~~~~~~~~~~~
Glean Parser could always use more documentation, whether as part of the
official Glean Parser docs, in docstrings, or even on the web in blog posts,
articles, and such.
Submit Feedback
~~~~~~~~~~~~~~~
The best way to send feedback is to file an issue at TODO
If you are proposing a feature:
* Explain in detail how it would work.
* Keep the scope as narrow as possible, to make it easier to implement.
* Remember that this is a volunteer-driven project, and that contributions
are welcome :)
Get Started!
------------
Ready to contribute? Here's how to set up `glean_parser` for local development.
1. Fork the `glean_parser` repo on GitHub.
2. Clone your fork locally::
$ git clone git@github.com:your_name_here/glean_parser.git
3. Install your local copy into a virtualenv. Assuming you have
virtualenvwrapper installed, this is how you set up your fork for local
development::
$ mkvirtualenv glean_parser
$ cd glean_parser/
$ python setup.py develop
4. Create a branch for local development::
$ git checkout -b name-of-your-bugfix-or-feature
Now you can make your changes locally.
5. To test your changes to `glean_parser`:
Install the testing dependencies::
$ pip install -r requirements_dev.txt
Optionally, if you want to ensure that the generated Kotlin code lints correctly, install a Java SDK, and then run::
$ make install-kotlin-linters
Then make sure that all lints and tests are passing::
$ make lint
$ make test
6. Commit your changes and push your branch to GitHub::
$ git add .
$ git commit -m "Your detailed description of your changes."
$ git push origin name-of-your-bugfix-or-feature
7. Submit a pull request through the GitHub website.
Pull Request Guidelines
-----------------------
Before you submit a pull request, check that it meets these guidelines:
1. The pull request should include tests.
2. If the pull request adds functionality, the docs should be updated. Put
your new functionality into a function with a docstring, and add the
feature to the list in README.rst.
3. The pull request should work for Python 3.6, 3.7 and 3.8 (The CI system will take care of testing all of these Python versions).
4. The pull request should update the changelog in `HISTORY.rst`.
Tips
----
To run a subset of tests::
$ py.test tests.test_glean_parser
Deploying
---------
A reminder for the maintainers on how to deploy.
Get a clean main branch with all of the changes from `upstream`::
$ git checkout main
$ git fetch upstream
$ git rebase upstream/main
- Update the header with the new version and date in HISTORY.rst.
- (By using the setuptools-scm package, there is no need to update the version anywhere else).
- Make sure all your changes are committed.
- Push the changes upstream. (Normally pushing directly without review is frowned upon, but the `main` branch is protected from force pushes and release tagging requires the same permissions as pushing to `main`)::
$ git push upstream main
- Wait for [continuous integration to pass](https://circleci.com/gh/mozilla/glean/tree/main) on main.
- Make the release on GitHub using [this link](https://github.com/mozilla/glean_parser/releases/new)
- Both the tag and the release title should be in the form `vX.Y.Z`.
- Copy and paste the relevant part of the `HISTORY.rst` file into the description.
- Tagging the release will trigger a CI workflow which will build the distribution of `glean_parser` and publish it to PyPI.
The continuous integration system will then automatically deploy to PyPI.
See also the [instructions for updating the version of `glean_parser` used by the Glean SDK](https://mozilla.github.io/glean/book/dev/upgrading-glean-parser.html).

415
third_party/python/glean_parser/HISTORY.rst vendored
View File

@ -1,415 +0,0 @@
=======
History
=======
Unreleased
----------
1.29.0 (2020-10-07)
-------------------
* **Breaking change:** `glean_parser` will now return an error code when any of the input files do not exist (unless the `--allow-missing-files` flag is passed).
* Generated code now includes a comment next to each metric containing the name of the metric in its original `snake_case` form.
* When metrics don't provide a `unit` parameter, it is not included in the output (as provided by probe-scraper).
1.28.6 (2020-09-24)
-------------------
* BUGFIX: Ensure Kotlin arguments are deterministically ordered
1.28.5 (2020-09-14)
-------------------
* Fix deploy step to update pip before deploying to pypi.
1.28.4 (2020-09-14)
-------------------
* The `SUPERFLUOUS_NO_LINT` warning has been removed from the glinter.
It likely did more harm than good, and makes it hard to make
`metrics.yaml` files that pass across different versions of `glean_parser`.
* Expired metrics will now produce a linter warning, `EXPIRED_METRIC`.
* Expiry dates that are more than 730 days (~2 years) in the future will produce a linter warning,
`EXPIRATION_DATE_TOO_FAR`.
* Allow using the Quantity metric type outside of Gecko.
* New parser configs `custom_is_expired` and `custom_validate_expires` added.
These are both functions that take the `expires` value of the metric and return a bool.
(See `Metric.is_expired` and `Metric.validate_expires`).
These will allow FOG to provide custom validation for its version-based `expires` values.
1.28.3 (2020-07-28)
-------------------
* BUGFIX: Support HashSet and Dictionary in the C# generated code.
1.28.2 (2020-07-28)
-------------------
* BUGFIX: Generate valid C# code when using Labeled metric types.
1.28.1 (2020-07-24)
-------------------
* BUGFIX: Add missing column to correctly render markdown tables in generated documentation.
1.28.0 (2020-07-23)
-------------------
* **Breaking change:** The internal ping `deletion-request` was misnamed in pings.py causing the linter to not allow use of the correctly named ping for adding legacy ids to. Consuming apps will need to update their metrics.yaml if they are using `deletion_request` in any `send_in_pings` to `deletion-request` after updating.
1.27.0 (2020-07-21)
-------------------
* Rename the `data_category` field to `data_sensitivity` to be clearer.
1.26.0 (2020-07-21)
-------------------
* Add support for JWE metric types.
* Add a `data_sensitivity` field to all metrics for specifying the type of data collected in the field.
1.25.0 (2020-07-17)
-------------------
* Add support for generating C# code.
* BUGFIX: The memory unit is now correctly passed to the MemoryDistribution
metric type in Swift.
1.24.0 (2020-06-30)
-------------------
* BUGFIX: look for metrics in send_if_empty pings. Metrics for these kinds of pings were being ignored.
1.23.0 (2020-06-27)
-------------------
* Support for Python 3.5 has been dropped.
* BUGFIX: The ordering of event extra keys will now match with their enum, fixing a serious bug where keys of extras may not match the correct values in the data payload. See https://bugzilla.mozilla.org/show_bug.cgi?id=1648768.
1.22.0 (2020-05-28)
-------------------
* **Breaking change:** (Swift only) Combine all metrics and pings into a single generated file `Metrics.swift`.
1.21.0 (2020-05-25)
-------------------
* `glinter` messages have been improved with more details and to be more
actionable.
* A maximum of 10 `extra_keys` is now enforced for `event` metric types.
* BUGFIX: the `Lifetime` enum values now match the values of the implementation in mozilla/glean.
1.20.4 (2020-05-07)
-------------------
* BUGFIX: yamllint errors are now reported using the correct file name.
1.20.3 (2020-05-06)
-------------------
* Support for using `timing_distribution`'s `time_unit` parameter to control the range of acceptable values is documented. The default unit for this use case is `nanosecond` to avoid creating a breaking change. See [bug 1630997](https://bugzilla.mozilla.org/show_bug.cgi?id=1630997) for more information.
1.20.2 (2020-04-24)
-------------------
* Dependencies that depend on the version of Python being used are now specified using the `Declaring platform specific dependencies syntax in setuptools <https://setuptools.readthedocs.io/en/latest/setuptools.html#declaring-platform-specific-dependencies>`__. This means that more recent versions of dependencies are likely to be installed on Python 3.6 and later, and unnecessary backport libraries won't be installed on more recent Python versions.
1.20.1 (2020-04-21)
-------------------
* The minimum version of the runtime dependencies has been lowered to increase compatibility with other tools. These minimum versions are now tested in CI, in addition to testing the latest versions of the dependencies that was already happening in CI.
1.20.0 (2020-04-15)
-------------------
* **Breaking change:** glinter errors found during the `translate` command will now return an error code. glinter warnings will be displayed, but not return an error code.
* `glean_parser` now produces a linter warning when `user` lifetime metrics are
set to expire. See [bug 1604854](https://bugzilla.mozilla.org/show_bug.cgi?id=1604854)
for additional context.
1.19.0 (2020-03-18)
-------------------
* **Breaking change:** The regular expression used to validate labels is
stricter and more correct.
* Add more information about pings to markdown documentation:
* State whether the ping includes client id;
* Add list of data review links;
* Add list of related bugs links.
* `glean_parser` now makes it easier to write external translation functions for
different language targets.
* BUGFIX: glean_parser now works on 32-bit Windows.
1.18.3 (2020-02-24)
-------------------
* Dropped the 'inflection' dependency.
* Constrained the 'zipp' and 'MarkupSafe' transitive dependencies to versions that
support Python 3.5.
1.18.2 (2020-02-14)
-------------------
* BUGFIX: Fix rendering of first element of reason list.
1.18.1 (2020-02-14)
-------------------
* BUGFIX: Reason codes are displayed in markdown output for built-in pings as
well.
* BUGFIX: Reason descriptions are indented correctly in markdown output.
* BUGFIX: To avoid a compiler error, the @JvmName annotation isn't added to
private members.
1.18.0 (2020-02-13)
-------------------
* **Breaking Change (Java API)** Have the metrics names in Java match the names in Kotlin.
See [Bug 1588060](https://bugzilla.mozilla.org/show_bug.cgi?id=1588060).
* The reasons a ping are sent are now included in the generated markdown documentation.
1.17.3 (2020-02-05)
-------------------
* BUGFIX: The version of Jinja2 now specifies < 3.0, since that version no
longer supports Python 3.5.
1.17.2 (2020-02-05)
-------------------
* BUGFIX: Fixes an import error in generated Kotlin code.
1.17.1 (2020-02-05)
-------------------
* BUGFIX: Generated Swift code now includes `import Glean`, unless generating
for a Glean-internal build.
1.17.0 (2020-02-03)
-------------------
* Remove default schema URL from `validate_ping`
* Make `schema` argument required for CLI
* BUGFIX: Avoid default import in Swift code for Glean itself
* BUGFIX: Restore order of fields in generated Swift code
1.16.0 (2020-01-15)
-------------------
* Support for `reason` codes on pings was added.
1.15.6 (2020-02-06)
-------------------
* BUGFIX: The version of Jinja2 now specifies < 3.0, since that version no
longer supports Python 3.5 (backported from 1.17.3).
1.15.5 (2019-12-19)
-------------------
* BUGFIX: Also allow the legacy name `all_pings` for `send_in_pings` parameter on metrics
1.15.4 (2019-12-19)
-------------------
* BUGFIX: Also allow the legacy name `all_pings`
1.15.3 (2019-12-13)
-------------------
* Add project title to markdown template.
* Remove "Sorry about that" from markdown template.
* BUGFIX: Replace dashes in variable names to force proper naming
1.15.2 (2019-12-12)
-------------------
* BUGFIX: Use a pure Python library for iso8601 so there is no compilation required.
1.15.1 (2019-12-12)
-------------------
* BUGFIX: Add some additional ping names to the non-kebab-case allow list.
1.15.0 (2019-12-12)
-------------------
* Restrict new pings names to be kebab-case and change `all_pings` to `all-pings`
1.14.0 (2019-12-06)
-------------------
* glean_parser now supports Python versions 3.5, 3.6, 3.7 and 3.8.
1.13.0 (2019-12-04)
-------------------
* The `translate` command will no longer clear extra files in the output directory.
* BUGFIX: Ensure all newlines in comments are prefixed with comment markers
* BUGFIX: Escape Swift keywords in variable names in generated code
* Generate documentation for pings that are sent if empty
1.12.0 (2019-11-27)
-------------------
* Reserve the `deletion_request` ping name
* Added a new flag `send_if_empty` for pings
1.11.0 (2019-11-13)
-------------------
* The `glinter` command now performs `yamllint` validation on registry files.
1.10.0 (2019-11-11)
-------------------
* The Kotlin linter `detekt` is now run during CI, and for local
testing if installed.
* Python 3.8 is now tested in CI (in addition to Python 3.7).
Using `tox` for this doesn't work in modern versions of CircleCI, so
the `tox` configuration has been removed.
* `yamllint` has been added to test the YAML files on CI.
* ⚠ Metric types that don't yet have implementations in glean-core have been
removed. This includes `enumeration`, `rate`, `usage`, and `use_counter`, as
well as many labeled metrics that don't exist.
1.9.5 (2019-10-22)
------------------
* Allow a Swift lint for generated code
* New lint: Restrict what metric can go into the 'baseline' ping
* New lint: Warn for slight misspellings in ping names
* BUGFIX: change Labeled types labels from lists to sets.
1.9.4 (2019-10-16)
------------------
* Use lists instead of sets in Labeled types labels to ensure that
the order of the labels passed to the `metrics.yaml` is kept.
* `glinter` will now check for duplicate labels and error if there are any.
1.9.3 (2019-10-09)
------------------
* Add labels from Labeled types to the Extra column in the Markdown template.
1.9.2 (2019-10-08)
------------------
* BUGFIX: Don't call `is_internal_metric` on `Ping` objects.
1.9.1 (2019-10-07)
------------------
* Don't include Glean internal metrics in the generated markdown.
1.9.0 (2019-10-04)
------------------
* Glinter now warns when bug numbers (rather than URLs) are used.
* BUGFIX: add `HistogramType` and `MemoryUnit` imports in Kotlin generated code.
1.8.4 (2019-10-02)
------------------
* Removed unsupported labeled metric types.
1.8.3 (2019-10-02)
------------------
* Fix indentation for generated Swift code
1.8.2 (2019-10-01)
------------------
* Created labeled metrics and events in Swift code and wrap it in a configured namespace
1.8.1 (2019-09-27)
------------------
* BUGFIX: `memory_unit` is now passed to the Kotlin generator.
1.8.0 (2019-09-26)
------------------
* A new parser config, `do_not_disable_expired`, was added to turn off the
feature that expired metrics are automatically disabled. This is useful if you
want to retain the disabled value that is explicitly in the `metrics.yaml`
file.
* `glinter` will now report about superfluous `no_lint` entries.
1.7.0 (2019-09-24)
------------------
* A "`glinter`" tool is now included to find common mistakes in metric naming and setup.
This check is run during `translate` and warnings will be displayed.
⚠ These warnings will be treated as errors in a future revision.
1.6.1 (2019-09-17)
------------------
* BUGFIX: `GleanGeckoMetricsMapping` must include `LabeledMetricType` and `CounterMetricType`.
1.6.0 (2019-09-17)
------------------
* NEW: Support for outputting metrics in Swift.
* BUGFIX: Provides a helpful error message when `geckoview_datapoint` is used on an metric type that doesn't support GeckoView exfiltration.
* Generate a lookup table for Gecko categorical histograms in `GleanGeckoMetricsMapping`.
* Introduce a 'Swift' output generator.
1.4.1 (2019-08-28)
------------------
* Documentation only.
1.4.0 (2019-08-27)
------------------
* Added support for generating markdown documentation from `metrics.yaml` files.
1.3.0 (2019-08-22)
------------------
* `quantity` metric type has been added.
1.2.1 (2019-08-13)
------------------
* BUGFIX: `includeClientId` was not being output for PingType.
1.2.0 (2019-08-13)
------------------
* `memory_distribution` metric type has been added.
* `custom_distribution` metric type has been added.
* `labeled_timespan` is no longer an allowed metric type.
1.1.0 (2019-08-05)
------------------
* Add a special `all_pings` value to `send_in_pings`.
1.0.0 (2019-07-29)
------------------
* First release to start following strict semver.
0.1.0 (2018-10-15)
------------------
* First release on PyPI.

8
third_party/python/glean_parser/MANIFEST.in vendored
View File

@ -1,8 +1,8 @@
include AUTHORS.rst
include CONTRIBUTING.rst
include HISTORY.rst
include AUTHORS.md
include CONTRIBUTING.md
include CHANGELOG.md
include LICENSE
include README.rst
include README.md
recursive-include tests *
recursive-exclude * __pycache__

3
third_party/python/glean_parser/Makefile vendored
View File

@ -55,9 +55,6 @@ docs: ## generate Sphinx HTML documentation, including API docs
$(MAKE) -C docs clean
$(MAKE) -C docs html
servedocs: docs ## compile the docs watching for changes
watchmedo shell-command -p '*.rst' -c '$(MAKE) -C docs html' -R -D .
release: dist ## package and upload a release
twine upload dist/*

635
third_party/python/glean_parser/PKG-INFO vendored
View File

@ -1,482 +1,496 @@
Metadata-Version: 1.1
Metadata-Version: 2.1
Name: glean_parser
Version: 1.29.0
Version: 2.5.0
Summary: Parser tools for Mozilla's Glean telemetry
Home-page: https://github.com/mozilla/glean_parser
Author: Michael Droettboom
Author-email: mdroettboom@mozilla.com
License: UNKNOWN
Description: ============
Glean Parser
============
Description: # Glean Parser
Parser tools for Mozilla's Glean telemetry.
Features
--------
## Features
Parses the ``metrics.yaml`` files for the Glean telemetry SDK and produces
Parses the `metrics.yaml` files for the Glean telemetry SDK and produces
output for various integrations.
Documentation
-------------
## Documentation
The full documentation is available `here <https://mozilla.github.io/glean_parser/>`__.
The full documentation is available
[here](https://mozilla.github.io/glean_parser/).
Requirements
------------
## Requirements
- Python 3.6 (or later)
- Python 3.6 (or later)
The following library requirements are installed automatically when glean_parser
is installed by `pip`.
The following library requirements are installed automatically when
`glean_parser` is installed by `pip`.
- appdirs
- Click
- diskcache
- Jinja2
- jsonschema
- PyYAML
- appdirs
- Click
- diskcache
- Jinja2
- jsonschema
- PyYAML
Additionally on Python 3.6:
- iso8601
- iso8601
Usage
-----
## Usage
.. code-block:: console
```sh
$ glean_parser --help
```
$ glean_parser --help
Read in `metrics.yaml`, translate to Kotlin format, and
output to `output_dir`:
Read in `metrics.yaml`, translate to kotlin format, and output to `output_dir`:
.. code-block:: console
$ glean_parser translate -o output_dir -f kotlin metrics.yaml
```sh
$ glean_parser translate -o output_dir -f kotlin metrics.yaml
```
Check a Glean ping against the ping schema:
.. code-block:: console
$ glean_parser check < ping.json
```sh
$ glean_parser check < ping.json
```
=======
History
=======
# Changelog
Unreleased
----------
## Unreleased
1.29.0 (2020-10-07)
-------------------
## 2.5.0 (2021-02-23)
* **Breaking change:** `glean_parser` will now return an error code when any of the input files do not exist (unless the `--allow-missing-files` flag is passed).
* Generated code now includes a comment next to each metric containing the name of the metric in its original `snake_case` form.
* When metrics don't provide a `unit` parameter, it is not included in the output (as provided by probe-scraper).
- Add parser and object model support for `rate` metric type. ([bug 1645166](https://bugzilla.mozilla.org/show_bug.cgi?id=1645166))
- Add parser and object model support for telemetry_mirror property. ([bug 1685406](https://bugzilla.mozilla.org/show_bug.cgi?id=1685406))
- Update the Javascript template to match Glean.js expectations. ([bug 1693516](https://bugzilla.mozilla.org/show_bug.cgi?id=1693516))
- Glean.js has updated it's export strategy. It will now export each metric type as an independent module;
- Glean.js has dropped support for non ES6 modules.
- Add support for generating Typescript code. ([bug 1692157](https://bugzilla.mozilla.org/show_bug.cgi?id=1692157))
- The templates added generate metrics and pings code for Glean.js.
1.28.6 (2020-09-24)
-------------------
## 2.4.0 (2021-02-18)
* BUGFIX: Ensure Kotlin arguments are deterministically ordered
- **Experimental:** `glean_parser` has a new subcommand `coverage` to convert raw coverage reports
into something consumable by coverage tools, such as codecov.io
- The path to the file that each metric is defined in is now stored on the
`Metric` object in `defined_in["filepath"]`.
1.28.5 (2020-09-14)
-------------------
## 2.3.0 (2021-02-17)
* Fix deploy step to update pip before deploying to pypi.
- Leverage the `glean_namespace` to provide correct import when building for Javascript.
1.28.4 (2020-09-14)
-------------------
## 2.2.0 (2021-02-11)
* The `SUPERFLUOUS_NO_LINT` warning has been removed from the glinter.
- The Kotlin generator now generates static build information that can be passed
into `Glean.initialize` to avoid calling the package manager at runtime.
## 2.1.0 (2021-02-10)
- Add support for generating Javascript code.
- The templates added generate metrics and pings code for Glean.js.
## 2.0.0 (2021-02-05)
- New versions 2.0.0 of the `metrics.yaml` and `pings.yaml` schemas now ship
with `glean_parser`. These schemas are different from version 1.0.0 in the
following ways:
- Bugs must be specified as URLs. Bug numbers are disallowed.
- The legacy ping names containing underscores are no longer allowed. These
included `deletion_request`, `bookmarks_sync`, `history_sync`,
`session_end`, `all_pings`, `glean_*`). In these cases, the `_` should be
replaced with `-`.
To upgrade your app or library to use the new schema, replace the version in
the `$schema` value with `2-0-0`.
- **Breaking change:** It is now an error to use bug numbers (rather than URLs)
in ping definitions.
- Add the line number that metrics and pings were originally defined in the yaml
files.
## 1.29.1 (2020-12-17)
- BUGFIX: Linter output can now be redirected correctly (1675771).
## 1.29.0 (2020-10-07)
- **Breaking change:** `glean_parser` will now return an error code when any of
the input files do not exist (unless the `--allow-missing-files` flag is
passed).
- Generated code now includes a comment next to each metric containing the name
of the metric in its original `snake_case` form.
- When metrics don't provide a `unit` parameter, it is not included in the
output (as provided by probe-scraper).
## 1.28.6 (2020-09-24)
- BUGFIX: Ensure Kotlin arguments are deterministically ordered
## 1.28.5 (2020-09-14)
- Fix deploy step to update pip before deploying to pypi.
## 1.28.4 (2020-09-14)
- The `SUPERFLUOUS_NO_LINT` warning has been removed from the glinter.
It likely did more harm than good, and makes it hard to make
`metrics.yaml` files that pass across different versions of `glean_parser`.
* Expired metrics will now produce a linter warning, `EXPIRED_METRIC`.
* Expiry dates that are more than 730 days (~2 years) in the future will produce a linter warning,
`EXPIRATION_DATE_TOO_FAR`.
* Allow using the Quantity metric type outside of Gecko.
* New parser configs `custom_is_expired` and `custom_validate_expires` added.
These are both functions that take the `expires` value of the metric and return a bool.
(See `Metric.is_expired` and `Metric.validate_expires`).
These will allow FOG to provide custom validation for its version-based `expires` values.
`metrics.yaml` files that pass across different versions of
`glean_parser`.
- Expired metrics will now produce a linter warning, `EXPIRED_METRIC`.
- Expiry dates that are more than 730 days (\~2 years) in the future
will produce a linter warning, `EXPIRATION_DATE_TOO_FAR`.
- Allow using the Quantity metric type outside of Gecko.
- New parser configs `custom_is_expired` and `custom_validate_expires`
added. These are both functions that take the `expires` value of the
metric and return a bool. (See `Metric.is_expired` and
`Metric.validate_expires`). These will allow FOG to provide custom
validation for its version-based `expires` values.
1.28.3 (2020-07-28)
-------------------
## 1.28.3 (2020-07-28)
* BUGFIX: Support HashSet and Dictionary in the C# generated code.
- BUGFIX: Support HashSet and Dictionary in the C\## generated code.
1.28.2 (2020-07-28)
-------------------
## 1.28.2 (2020-07-28)
* BUGFIX: Generate valid C# code when using Labeled metric types.
- BUGFIX: Generate valid C\## code when using Labeled metric types.
1.28.1 (2020-07-24)
-------------------
## 1.28.1 (2020-07-24)
* BUGFIX: Add missing column to correctly render markdown tables in generated documentation.
- BUGFIX: Add missing column to correctly render markdown tables in generated
documentation.
1.28.0 (2020-07-23)
-------------------
## 1.28.0 (2020-07-23)
* **Breaking change:** The internal ping `deletion-request` was misnamed in pings.py causing the linter to not allow use of the correctly named ping for adding legacy ids to. Consuming apps will need to update their metrics.yaml if they are using `deletion_request` in any `send_in_pings` to `deletion-request` after updating.
- **Breaking change:** The internal ping `deletion-request` was misnamed in
pings.py causing the linter to not allow use of the correctly named ping for
adding legacy ids to. Consuming apps will need to update their metrics.yaml if
they are using `deletion_request` in any `send_in_pings` to `deletion-request`
after updating.
1.27.0 (2020-07-21)
-------------------
## 1.27.0 (2020-07-21)
* Rename the `data_category` field to `data_sensitivity` to be clearer.
- Rename the `data_category` field to `data_sensitivity` to be clearer.
1.26.0 (2020-07-21)
-------------------
## 1.26.0 (2020-07-21)
* Add support for JWE metric types.
* Add a `data_sensitivity` field to all metrics for specifying the type of data collected in the field.
- Add support for JWE metric types.
- Add a `data_sensitivity` field to all metrics for specifying the type of data
collected in the field.
1.25.0 (2020-07-17)
-------------------
## 1.25.0 (2020-07-17)
* Add support for generating C# code.
* BUGFIX: The memory unit is now correctly passed to the MemoryDistribution
- Add support for generating C\## code.
- BUGFIX: The memory unit is now correctly passed to the MemoryDistribution
metric type in Swift.
1.24.0 (2020-06-30)
-------------------
## 1.24.0 (2020-06-30)
* BUGFIX: look for metrics in send_if_empty pings. Metrics for these kinds of pings were being ignored.
- BUGFIX: look for metrics in send\_if\_empty pings. Metrics for these kinds of
pings were being ignored.
1.23.0 (2020-06-27)
-------------------
## 1.23.0 (2020-06-27)
* Support for Python 3.5 has been dropped.
* BUGFIX: The ordering of event extra keys will now match with their enum, fixing a serious bug where keys of extras may not match the correct values in the data payload. See https://bugzilla.mozilla.org/show_bug.cgi?id=1648768.
- Support for Python 3.5 has been dropped.
- BUGFIX: The ordering of event extra keys will now match with their enum,
fixing a serious bug where keys of extras may not match the correct values in
the data payload. See <https://bugzilla.mozilla.org/show_bug.cgi?id=1648768>.
1.22.0 (2020-05-28)
-------------------
## 1.22.0 (2020-05-28)
* **Breaking change:** (Swift only) Combine all metrics and pings into a single generated file `Metrics.swift`.
- **Breaking change:** (Swift only) Combine all metrics and pings into a single
generated file `Metrics.swift`.
1.21.0 (2020-05-25)
-------------------
## 1.21.0 (2020-05-25)
* `glinter` messages have been improved with more details and to be more
- `glinter` messages have been improved with more details and to be more
actionable.
* A maximum of 10 `extra_keys` is now enforced for `event` metric types.
* BUGFIX: the `Lifetime` enum values now match the values of the implementation in mozilla/glean.
- A maximum of 10 `extra_keys` is now enforced for `event` metric types.
- BUGFIX: the `Lifetime` enum values now match the values of the implementation
in mozilla/glean.
1.20.4 (2020-05-07)
-------------------
## 1.20.4 (2020-05-07)
* BUGFIX: yamllint errors are now reported using the correct file name.
- BUGFIX: yamllint errors are now reported using the correct file name.
1.20.3 (2020-05-06)
-------------------
## 1.20.3 (2020-05-06)
* Support for using `timing_distribution`'s `time_unit` parameter to control the range of acceptable values is documented. The default unit for this use case is `nanosecond` to avoid creating a breaking change. See [bug 1630997](https://bugzilla.mozilla.org/show_bug.cgi?id=1630997) for more information.
- Support for using `timing_distribution`'s `time_unit` parameter to control
the range of acceptable values is documented. The default unit for this use
case is `nanosecond` to avoid creating a breaking change. See [bug
1630997](https://bugzilla.mozilla.org/show_bug.cgi?id=1630997) for more
information.
1.20.2 (2020-04-24)
-------------------
## 1.20.2 (2020-04-24)
* Dependencies that depend on the version of Python being used are now specified using the `Declaring platform specific dependencies syntax in setuptools <https://setuptools.readthedocs.io/en/latest/setuptools.html#declaring-platform-specific-dependencies>`__. This means that more recent versions of dependencies are likely to be installed on Python 3.6 and later, and unnecessary backport libraries won't be installed on more recent Python versions.
- Dependencies that depend on the version of Python being used are now specified
using the [Declaring platform specific dependencies syntax in
setuptools](https://setuptools.readthedocs.io/en/latest/setuptools.html##declaring-platform-specific-dependencies).
This means that more recent versions of dependencies are likely to be
installed on Python 3.6 and later, and unnecessary backport libraries won't
be installed on more recent Python versions.
1.20.1 (2020-04-21)
-------------------
## 1.20.1 (2020-04-21)
* The minimum version of the runtime dependencies has been lowered to increase compatibility with other tools. These minimum versions are now tested in CI, in addition to testing the latest versions of the dependencies that was already happening in CI.
- The minimum version of the runtime dependencies has been lowered to increase
compatibility with other tools. These minimum versions are now tested in CI,
in addition to testing the latest versions of the dependencies that was
already happening in CI.
1.20.0 (2020-04-15)
-------------------
## 1.20.0 (2020-04-15)
* **Breaking change:** glinter errors found during the `translate` command will now return an error code. glinter warnings will be displayed, but not return an error code.
* `glean_parser` now produces a linter warning when `user` lifetime metrics are
set to expire. See [bug 1604854](https://bugzilla.mozilla.org/show_bug.cgi?id=1604854)
for additional context.
- **Breaking change:** glinter errors found during the `translate` command will
now return an error code. glinter warnings will be displayed, but not return
an error code.
- `glean_parser` now produces a linter warning when `user` lifetime metrics are
set to expire. See [bug
1604854](https://bugzilla.mozilla.org/show_bug.cgi?id=1604854) for additional
context.
1.19.0 (2020-03-18)
-------------------
## 1.19.0 (2020-03-18)
* **Breaking change:** The regular expression used to validate labels is
- **Breaking change:** The regular expression used to validate labels is
stricter and more correct.
* Add more information about pings to markdown documentation:
* State whether the ping includes client id;
* Add list of data review links;
* Add list of related bugs links.
* `glean_parser` now makes it easier to write external translation functions for
different language targets.
* BUGFIX: glean_parser now works on 32-bit Windows.
- Add more information about pings to markdown documentation:
- State whether the ping includes client id;
- Add list of data review links;
- Add list of related bugs links.
- `glean_parser` now makes it easier to write external translation
functions for different language targets.
- BUGFIX: `glean_parser` now works on 32-bit Windows.
1.18.3 (2020-02-24)
-------------------
## 1.18.3 (2020-02-24)
* Dropped the 'inflection' dependency.
* Constrained the 'zipp' and 'MarkupSafe' transitive dependencies to versions that
support Python 3.5.
- Dropped the `inflection` dependency.
- Constrained the `zipp` and `MarkupSafe` transitive dependencies to versions
that support Python 3.5.
1.18.2 (2020-02-14)
-------------------
## 1.18.2 (2020-02-14)
* BUGFIX: Fix rendering of first element of reason list.
- BUGFIX: Fix rendering of first element of reason list.
1.18.1 (2020-02-14)
-------------------
## 1.18.1 (2020-02-14)
* BUGFIX: Reason codes are displayed in markdown output for built-in pings as
well.
* BUGFIX: Reason descriptions are indented correctly in markdown output.
* BUGFIX: To avoid a compiler error, the @JvmName annotation isn't added to
private members.
- BUGFIX: Reason codes are displayed in markdown output for built-in
pings as well.
- BUGFIX: Reason descriptions are indented correctly in markdown
output.
- BUGFIX: To avoid a compiler error, the `@JvmName` annotation isn't
added to private members.
1.18.0 (2020-02-13)
-------------------
## 1.18.0 (2020-02-13)
* **Breaking Change (Java API)** Have the metrics names in Java match the names in Kotlin.
See [Bug 1588060](https://bugzilla.mozilla.org/show_bug.cgi?id=1588060).
* The reasons a ping are sent are now included in the generated markdown documentation.
- **Breaking Change (Java API)** Have the metrics names in Java match the names
in Kotlin. See [Bug
1588060](https://bugzilla.mozilla.org/show_bug.cgi?id=1588060).
- The reasons a ping are sent are now included in the generated markdown
documentation.
1.17.3 (2020-02-05)
-------------------
## 1.17.3 (2020-02-05)
* BUGFIX: The version of Jinja2 now specifies < 3.0, since that version no
- BUGFIX: The version of Jinja2 now specifies < 3.0, since that version no
longer supports Python 3.5.
1.17.2 (2020-02-05)
-------------------
## 1.17.2 (2020-02-05)
* BUGFIX: Fixes an import error in generated Kotlin code.
- BUGFIX: Fixes an import error in generated Kotlin code.
1.17.1 (2020-02-05)
-------------------
## 1.17.1 (2020-02-05)
* BUGFIX: Generated Swift code now includes `import Glean`, unless generating
- BUGFIX: Generated Swift code now includes `import Glean`, unless generating
for a Glean-internal build.
1.17.0 (2020-02-03)
-------------------
## 1.17.0 (2020-02-03)
* Remove default schema URL from `validate_ping`
* Make `schema` argument required for CLI
* BUGFIX: Avoid default import in Swift code for Glean itself
* BUGFIX: Restore order of fields in generated Swift code
- Remove default schema URL from `validate_ping`
- Make `schema` argument required for CLI
- BUGFIX: Avoid default import in Swift code for Glean itself
- BUGFIX: Restore order of fields in generated Swift code
1.16.0 (2020-01-15)
-------------------
## 1.16.0 (2020-01-15)
* Support for `reason` codes on pings was added.
- Support for `reason` codes on pings was added.
1.15.6 (2020-02-06)
-------------------
## 1.15.6 (2020-02-06)
* BUGFIX: The version of Jinja2 now specifies < 3.0, since that version no
- BUGFIX: The version of Jinja2 now specifies < 3.0, since that version no
longer supports Python 3.5 (backported from 1.17.3).
1.15.5 (2019-12-19)
-------------------
## 1.15.5 (2019-12-19)
* BUGFIX: Also allow the legacy name `all_pings` for `send_in_pings` parameter on metrics
- BUGFIX: Also allow the legacy name `all_pings` for `send_in_pings` parameter
on metrics
1.15.4 (2019-12-19)
-------------------
## 1.15.4 (2019-12-19)
* BUGFIX: Also allow the legacy name `all_pings`
- BUGFIX: Also allow the legacy name `all_pings`
1.15.3 (2019-12-13)
-------------------
## 1.15.3 (2019-12-13)
* Add project title to markdown template.
* Remove "Sorry about that" from markdown template.
* BUGFIX: Replace dashes in variable names to force proper naming
- Add project title to markdown template.
- Remove "Sorry about that" from markdown template.
- BUGFIX: Replace dashes in variable names to force proper naming
1.15.2 (2019-12-12)
-------------------
## 1.15.2 (2019-12-12)
* BUGFIX: Use a pure Python library for iso8601 so there is no compilation required.
- BUGFIX: Use a pure Python library for iso8601 so there is no compilation
required.
1.15.1 (2019-12-12)
-------------------
## 1.15.1 (2019-12-12)
* BUGFIX: Add some additional ping names to the non-kebab-case allow list.
- BUGFIX: Add some additional ping names to the non-kebab-case allow list.
1.15.0 (2019-12-12)
-------------------
## 1.15.0 (2019-12-12)
* Restrict new pings names to be kebab-case and change `all_pings` to `all-pings`
- Restrict new pings names to be kebab-case and change `all_pings` to
`all-pings`
1.14.0 (2019-12-06)
-------------------
## 1.14.0 (2019-12-06)
* glean_parser now supports Python versions 3.5, 3.6, 3.7 and 3.8.
- `glean_parser` now supports Python versions 3.5, 3.6, 3.7 and 3.8.
1.13.0 (2019-12-04)
-------------------
## 1.13.0 (2019-12-04)
* The `translate` command will no longer clear extra files in the output directory.
* BUGFIX: Ensure all newlines in comments are prefixed with comment markers
* BUGFIX: Escape Swift keywords in variable names in generated code
* Generate documentation for pings that are sent if empty
- The `translate` command will no longer clear extra files in the output
directory.
- BUGFIX: Ensure all newlines in comments are prefixed with comment markers
- BUGFIX: Escape Swift keywords in variable names in generated code
- Generate documentation for pings that are sent if empty
1.12.0 (2019-11-27)
-------------------
## 1.12.0 (2019-11-27)
* Reserve the `deletion_request` ping name
* Added a new flag `send_if_empty` for pings
- Reserve the `deletion_request` ping name
- Added a new flag `send_if_empty` for pings
1.11.0 (2019-11-13)
-------------------
## 1.11.0 (2019-11-13)
* The `glinter` command now performs `yamllint` validation on registry files.
- The `glinter` command now performs `yamllint` validation on registry files.
1.10.0 (2019-11-11)
-------------------
## 1.10.0 (2019-11-11)
* The Kotlin linter `detekt` is now run during CI, and for local
- The Kotlin linter `detekt` is now run during CI, and for local
testing if installed.
- Python 3.8 is now tested in CI (in addition to Python 3.7). Using
`tox` for this doesn't work in modern versions of CircleCI, so the
`tox` configuration has been removed.
- `yamllint` has been added to test the YAML files on CI.
- ⚠ Metric types that don't yet have implementations in glean-core
have been removed. This includes `enumeration`, `rate`, `usage`, and
`use_counter`, as well as many labeled metrics that don't exist.
* Python 3.8 is now tested in CI (in addition to Python 3.7).
Using `tox` for this doesn't work in modern versions of CircleCI, so
the `tox` configuration has been removed.
## 1.9.5 (2019-10-22)
* `yamllint` has been added to test the YAML files on CI.
- Allow a Swift lint for generated code
- New lint: Restrict what metric can go into the `baseline` ping
- New lint: Warn for slight misspellings in ping names
- BUGFIX: change Labeled types labels from lists to sets.
* ⚠ Metric types that don't yet have implementations in glean-core have been
removed. This includes `enumeration`, `rate`, `usage`, and `use_counter`, as
well as many labeled metrics that don't exist.
## 1.9.4 (2019-10-16)
1.9.5 (2019-10-22)
------------------
- Use lists instead of sets in Labeled types labels to ensure that the order of
the labels passed to the `metrics.yaml` is kept.
- `glinter` will now check for duplicate labels and error if there are any.
* Allow a Swift lint for generated code
## 1.9.3 (2019-10-09)
* New lint: Restrict what metric can go into the 'baseline' ping
- Add labels from Labeled types to the Extra column in the Markdown template.
* New lint: Warn for slight misspellings in ping names
## 1.9.2 (2019-10-08)
* BUGFIX: change Labeled types labels from lists to sets.
- BUGFIX: Don't call `is_internal_metric` on `Ping` objects.
1.9.4 (2019-10-16)
------------------
## 1.9.1 (2019-10-07)
* Use lists instead of sets in Labeled types labels to ensure that
the order of the labels passed to the `metrics.yaml` is kept.
- Don't include Glean internal metrics in the generated markdown.
* `glinter` will now check for duplicate labels and error if there are any.
## 1.9.0 (2019-10-04)
1.9.3 (2019-10-09)
------------------
- Glinter now warns when bug numbers (rather than URLs) are used.
- BUGFIX: add `HistogramType` and `MemoryUnit` imports in Kotlin generated code.
* Add labels from Labeled types to the Extra column in the Markdown template.
## 1.8.4 (2019-10-02)
1.9.2 (2019-10-08)
------------------
- Removed unsupported labeled metric types.
* BUGFIX: Don't call `is_internal_metric` on `Ping` objects.
## 1.8.3 (2019-10-02)
1.9.1 (2019-10-07)
------------------
- Fix indentation for generated Swift code
* Don't include Glean internal metrics in the generated markdown.
## 1.8.2 (2019-10-01)
1.9.0 (2019-10-04)
------------------
- Created labeled metrics and events in Swift code and wrap it in a
configured namespace
* Glinter now warns when bug numbers (rather than URLs) are used.
## 1.8.1 (2019-09-27)
* BUGFIX: add `HistogramType` and `MemoryUnit` imports in Kotlin generated code.
- BUGFIX: `memory_unit` is now passed to the Kotlin generator.
1.8.4 (2019-10-02)
------------------
## 1.8.0 (2019-09-26)
* Removed unsupported labeled metric types.
1.8.3 (2019-10-02)
------------------
* Fix indentation for generated Swift code
1.8.2 (2019-10-01)
------------------
* Created labeled metrics and events in Swift code and wrap it in a configured namespace
1.8.1 (2019-09-27)
------------------
* BUGFIX: `memory_unit` is now passed to the Kotlin generator.
1.8.0 (2019-09-26)
------------------
* A new parser config, `do_not_disable_expired`, was added to turn off the
- A new parser config, `do_not_disable_expired`, was added to turn off the
feature that expired metrics are automatically disabled. This is useful if you
want to retain the disabled value that is explicitly in the `metrics.yaml`
file.
- `glinter` will now report about superfluous `no_lint` entries.
* `glinter` will now report about superfluous `no_lint` entries.
## 1.7.0 (2019-09-24)
1.7.0 (2019-09-24)
------------------
- A `glinter` tool is now included to find common mistakes in metric naming
and setup. This check is run during `translate` and warnings will be
displayed. ⚠ These warnings will be treated as errors in a future revision.
* A "`glinter`" tool is now included to find common mistakes in metric naming and setup.
This check is run during `translate` and warnings will be displayed.
⚠ These warnings will be treated as errors in a future revision.
## 1.6.1 (2019-09-17)
1.6.1 (2019-09-17)
------------------
- BUGFIX: `GleanGeckoMetricsMapping` must include `LabeledMetricType`
and `CounterMetricType`.
* BUGFIX: `GleanGeckoMetricsMapping` must include `LabeledMetricType` and `CounterMetricType`.
## 1.6.0 (2019-09-17)
1.6.0 (2019-09-17)
------------------
- NEW: Support for outputting metrics in Swift.
- BUGFIX: Provides a helpful error message when `geckoview_datapoint` is used on
an metric type that doesn't support GeckoView exfiltration.
- Generate a lookup table for Gecko categorical histograms in
`GleanGeckoMetricsMapping`.
- Introduce a 'Swift' output generator.
* NEW: Support for outputting metrics in Swift.
## 1.4.1 (2019-08-28)
* BUGFIX: Provides a helpful error message when `geckoview_datapoint` is used on an metric type that doesn't support GeckoView exfiltration.
- Documentation only.
* Generate a lookup table for Gecko categorical histograms in `GleanGeckoMetricsMapping`.
## 1.4.0 (2019-08-27)
* Introduce a 'Swift' output generator.
- Added support for generating markdown documentation from `metrics.yaml` files.
1.4.1 (2019-08-28)
------------------
## 1.3.0 (2019-08-22)
* Documentation only.
- `quantity` metric type has been added.
1.4.0 (2019-08-27)
------------------
## 1.2.1 (2019-08-13)
* Added support for generating markdown documentation from `metrics.yaml` files.
- BUGFIX: `includeClientId` was not being output for PingType.
1.3.0 (2019-08-22)
------------------
## 1.2.0 (2019-08-13)
* `quantity` metric type has been added.
- `memory_distribution` metric type has been added.
- `custom_distribution` metric type has been added.
- `labeled_timespan` is no longer an allowed metric type.
1.2.1 (2019-08-13)
------------------
## 1.1.0 (2019-08-05)
* BUGFIX: `includeClientId` was not being output for PingType.
- Add a special `all_pings` value to `send_in_pings`.
1.2.0 (2019-08-13)
------------------
## 1.0.0 (2019-07-29)
* `memory_distribution` metric type has been added.
- First release to start following strict semver.
* `custom_distribution` metric type has been added.
## 0.1.0 (2018-10-15)
* `labeled_timespan` is no longer an allowed metric type.
1.1.0 (2019-08-05)
------------------
* Add a special `all_pings` value to `send_in_pings`.
1.0.0 (2019-07-29)
------------------
* First release to start following strict semver.
0.1.0 (2018-10-15)
------------------
* First release on PyPI.
- First release on PyPI.
Keywords: glean_parser
Platform: UNKNOWN
@ -488,3 +502,4 @@ Classifier: Programming Language :: Python :: 3.6
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Description-Content-Type: text/markdown

50
third_party/python/glean_parser/README.md vendored Normal file
View File

@ -0,0 +1,50 @@
# Glean Parser
Parser tools for Mozilla's Glean telemetry.
## Features
Parses the `metrics.yaml` files for the Glean telemetry SDK and produces
output for various integrations.
## Documentation
The full documentation is available
[here](https://mozilla.github.io/glean_parser/).
## Requirements
- Python 3.6 (or later)
The following library requirements are installed automatically when
`glean_parser` is installed by `pip`.
- appdirs
- Click
- diskcache
- Jinja2
- jsonschema
- PyYAML
Additionally on Python 3.6:
- iso8601
## Usage
```sh
$ glean_parser --help
```
Read in `metrics.yaml`, translate to Kotlin format, and
output to `output_dir`:
```sh
$ glean_parser translate -o output_dir -f kotlin metrics.yaml
```
Check a Glean ping against the ping schema:
```sh
$ glean_parser check < ping.json
```

54
third_party/python/glean_parser/README.rst vendored
View File

@ -1,54 +0,0 @@
============
Glean Parser
============
Parser tools for Mozilla's Glean telemetry.
Features
--------
Parses the ``metrics.yaml`` files for the Glean telemetry SDK and produces
output for various integrations.
Documentation
-------------
The full documentation is available `here <https://mozilla.github.io/glean_parser/>`__.
Requirements
------------
- Python 3.6 (or later)
The following library requirements are installed automatically when glean_parser
is installed by `pip`.
- appdirs
- Click
- diskcache
- Jinja2
- jsonschema
- PyYAML
Additionally on Python 3.6:
- iso8601
Usage
-----
.. code-block:: console
$ glean_parser --help
Read in `metrics.yaml`, translate to kotlin format, and output to `output_dir`:
.. code-block:: console
$ glean_parser translate -o output_dir -f kotlin metrics.yaml
Check a Glean ping against the ping schema:
.. code-block:: console
$ glean_parser check < ping.json

50
third_party/python/glean_parser/glean_parser/__main__.py vendored
View File

@ -16,6 +16,7 @@ import click
import glean_parser
from . import coverage as mod_coverage
from . import lint
from . import translate as mod_translate
from . import validate_ping
@ -140,6 +141,54 @@ def glinter(input, allow_reserved, allow_missing_files):
)
@click.command()
@click.option(
"-c",
"--coverage_file",
type=click.Path(exists=True, dir_okay=False, file_okay=True, readable=True),
required=True,
multiple=True,
)
@click.argument(
"metrics_files",
type=click.Path(exists=True, dir_okay=False, file_okay=True, readable=True),
nargs=-1,
)
@click.option(
"-o",
"--output",
type=click.Path(exists=False, dir_okay=False, file_okay=True, writable=True),
required=True,
)
@click.option(
"--format", "-f", type=click.Choice(mod_coverage.OUTPUTTERS.keys()), required=True
)
@click.option(
"--allow-reserved",
is_flag=True,
help=(
"If provided, allow the use of reserved fields. "
"Should only be set when building the Glean library itself."
),
)
def coverage(coverage_file, metrics_files, format, output, allow_reserved):
"""
Produce a coverage analysis file given raw coverage output and a set of
metrics.yaml files.
"""
sys.exit(
mod_coverage.coverage(
[Path(x) for x in coverage_file],
[Path(x) for x in metrics_files],
format,
Path(output),
{
"allow_reserved": allow_reserved,
},
)
)
@click.group()
@click.version_option(glean_parser.__version__, prog_name="glean_parser")
def main(args=None):
@ -150,6 +199,7 @@ def main(args=None):
main.add_command(translate)
main.add_command(check)
main.add_command(glinter)
main.add_command(coverage)
def main_wrapper(args=None):

140
third_party/python/glean_parser/glean_parser/coverage.py vendored Normal file
View File

@ -0,0 +1,140 @@
# -*- coding: utf-8 -*-
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, You can obtain one at http://mozilla.org/MPL/2.0/.
"""
Produce coverage reports from the raw information produced by the
`GLEAN_TEST_COVERAGE` feature.
"""
import json
from .metrics import ObjectTree
from pathlib import Path
import sys
from typing import Any, Dict, List, Optional, Sequence, Set
from . import parser
from . import util
def _outputter_codecovio(metrics: ObjectTree, output_path: Path):
"""
Output coverage in codecov.io format as defined here:
https://docs.codecov.io/docs/codecov-custom-coverage-format
:param metrics: The tree of metrics, already annotated with coverage by
`_annotate_coverage`.
:param output_path: The file to output to.
"""
coverage: Dict[str, List] = {}
for category in metrics.values():
for metric in category.values():
defined_in = metric.defined_in
if defined_in is not None:
path = defined_in["filepath"]
if path not in coverage:
with open(path) as fd:
nlines = len(list(fd.readlines()))
lines = [None] * nlines
coverage[path] = lines
file_section = coverage[path]
file_section[int(defined_in["line"])] = getattr(metric, "covered", 0)
with open(output_path, "w") as fd:
json.dump({"coverage": coverage}, fd)
OUTPUTTERS = {"codecovio": _outputter_codecovio}
def _annotate_coverage(metrics, coverage_entries):
"""
Annotate each metric with whether it is covered. Sets the attribute
`covered` to 1 on each metric that is covered.
"""
mapping = {}
for category in metrics.values():
for metric in category.values():
mapping[metric.identifier()] = metric
for entry in coverage_entries:
metric_id = _coverage_entry_to_metric_id(entry)
if metric_id in mapping:
mapping[metric_id].covered = 1
def _coverage_entry_to_metric_id(entry: str) -> str:
"""
Convert a coverage entry to a metric id.
Technically, the coverage entries are rkv database keys, so are not just
the metric identifier. This extracts the metric identifier part out.
"""
# If getting a glean error count, report it as covering the metric the
# error occurred in, not the `glean.error.*` metric itself.
if entry.startswith("glean.error."):
entry = entry.split("/")[-1]
# If a labeled metric, strip off the label part
return entry.split("/")[0]
def _read_coverage_entries(coverage_reports: List[Path]) -> Set[str]:
"""
Read coverage entries from one or more files, and deduplicates them.
"""
entries = set()
for coverage_report in coverage_reports:
with open(coverage_report) as fd:
for line in fd.readlines():
entries.add(line.strip())
return entries
def coverage(
coverage_reports: List[Path],
metrics_files: Sequence[Path],
output_format: str,
output_file: Path,
parser_config: Optional[Dict[str, Any]] = None,
file=sys.stderr,
) -> int:
"""
Commandline helper for coverage.
:param coverage_reports: List of coverage report files, output from the
Glean SDK when the `GLEAN_TEST_COVERAGE` environment variable is set.
:param metrics_files: List of Path objects to load metrics from.
:param output_format: The coverage output format to produce. Must be one of
`OUTPUTTERS.keys()`.
:param output_file: Path to output coverage report to.
:param parser_config: Parser configuration object, passed to
`parser.parse_objects`.
:return: Non-zero if there were any errors.
"""
if parser_config is None:
parser_config = {}
if output_format not in OUTPUTTERS:
raise ValueError(f"Unknown outputter {output_format}")
metrics_files = util.ensure_list(metrics_files)
all_objects = parser.parse_objects(metrics_files, parser_config)
if util.report_validation_errors(all_objects):
return 1
entries = _read_coverage_entries(coverage_reports)
_annotate_coverage(all_objects.value, entries)
OUTPUTTERS[output_format](all_objects.value, output_file)
return 0

173
third_party/python/glean_parser/glean_parser/javascript.py vendored Normal file
View File

@ -0,0 +1,173 @@
# -*- coding: utf-8 -*-
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, You can obtain one at http://mozilla.org/MPL/2.0/.
"""
Outputter to generate Javascript code for metrics.
"""
import enum
import json
from pathlib import Path
from typing import Any, Dict, List, Optional, Union # noqa
from . import metrics
from . import util
def javascript_datatypes_filter(value: util.JSONType) -> str:
"""
A Jinja2 filter that renders Javascript literals.
"""
class JavascriptEncoder(json.JSONEncoder):
def iterencode(self, value):
if isinstance(value, enum.Enum):
yield from super().iterencode(util.camelize(value.name))
else:
yield from super().iterencode(value)
return "".join(JavascriptEncoder().iterencode(value))
def class_name(obj_type: str) -> str:
"""
Returns the Javascript class name for a given metric or ping type.
"""
if obj_type == "ping":
class_name = "PingType"
else:
class_name = util.Camelize(obj_type) + "MetricType"
return class_name
def import_path(obj_type: str) -> str:
"""
Returns the import path of the given object inside the @mozilla/glean package.
"""
if obj_type == "ping":
import_path = "ping"
else:
import_path = "metrics/" + util.camelize(obj_type)
return import_path
def args(obj_type: str) -> Dict[str, object]:
"""
Returns the list of arguments for each object type.
"""
if obj_type == "ping":
return {"common": util.ping_args, "extra": []}
return {"common": util.common_metric_args, "extra": util.extra_metric_args}
def output(
lang: str,
objs: metrics.ObjectTree,
output_dir: Path,
options: Optional[Dict[str, Any]] = None,
) -> None:
"""
Given a tree of objects, output Javascript or Typescript code to `output_dir`.
:param lang: Either "javascript" or "typescript";
:param objects: A tree of objects (metrics and pings) as returned from
`parser.parse_objects`.
:param output_dir: Path to an output directory to write to.
:param options: options dictionary, with the following optional keys:
- `namespace`: The identifier of the global variable to assign to.
This will only have and effect for Qt and static web sites.
Default is `gleanAssets`.
- `glean_namespace`: Which version of the `@mozilla/glean` package to import,
options are `webext` or `qt`. Default is `webext`.
"""
if options is None:
options = {}
namespace = options.get("namespace", "gleanAssets")
glean_namespace = options.get("glean_namespace", "webext")
template = util.get_jinja2_template(
"javascript.jinja2",
filters=(
("class_name", class_name),
("import_path", import_path),
("js", javascript_datatypes_filter),
("args", args),
),
)
for category_key, category_val in objs.items():
extension = ".js" if lang == "javascript" else ".ts"
filename = util.camelize(category_key) + extension
filepath = output_dir / filename
types = set([util.camelize(obj.type) for obj in category_val.values()])
with filepath.open("w", encoding="utf-8") as fd:
fd.write(
template.render(
category_name=category_key,
objs=category_val,
extra_args=util.extra_args,
namespace=namespace,
glean_namespace=glean_namespace,
types=types,
lang=lang,
)
)
# Jinja2 squashes the final newline, so we explicitly add it
fd.write("\n")
def output_javascript(
objs: metrics.ObjectTree, output_dir: Path, options: Optional[Dict[str, Any]] = None
) -> None:
"""
Given a tree of objects, output Javascript code to `output_dir`.
:param objects: A tree of objects (metrics and pings) as returned from
`parser.parse_objects`.
:param output_dir: Path to an output directory to write to.
:param options: options dictionary, with the following optional keys:
- `namespace`: The identifier of the global variable to assign to.
This will only have and effect for Qt and static web sites.
Default is `gleanAssets`.
- `glean_namespace`: Which version of the `@mozilla/glean` package to import,
options are `webext` or `qt`. Default is `webext`.
"""
output("javascript", objs, output_dir, options)
def output_typescript(
objs: metrics.ObjectTree, output_dir: Path, options: Optional[Dict[str, Any]] = None
) -> None:
"""
Given a tree of objects, output Typescript code to `output_dir`.
# Note
The only difference between the typescript and javascript templates,
currently is the file extension.
:param objects: A tree of objects (metrics and pings) as returned from
`parser.parse_objects`.
:param output_dir: Path to an output directory to write to.
:param options: options dictionary, with the following optional keys:
- `namespace`: The identifier of the global variable to assign to.
This will only have and effect for Qt and static web sites.
Default is `gleanAssets`.
- `glean_namespace`: Which version of the `@mozilla/glean` package to import,
options are `webext` or `qt`. Default is `webext`.
"""
output("typescript", objs, output_dir, options)

24
third_party/python/glean_parser/glean_parser/kotlin.py vendored
View File

@ -219,6 +219,27 @@ def output_kotlin(
if options is None:
options = {}
namespace = options.get("namespace", "GleanMetrics")
glean_namespace = options.get("glean_namespace", "mozilla.components.service.glean")
namespace_package = namespace[: namespace.rfind(".")]
# Write out the special "build info" object
template = util.get_jinja2_template(
"kotlin.buildinfo.jinja2",
)
# This filename needs to start with "Glean" so it can never clash with a
# metric category
with (output_dir / "GleanBuildInfo.kt").open("w", encoding="utf-8") as fd:
fd.write(
template.render(
namespace=namespace,
namespace_package=namespace_package,
glean_namespace=glean_namespace,
)
)
fd.write("\n")
template = util.get_jinja2_template(
"kotlin.jinja2",
filters=(
@ -228,9 +249,6 @@ def output_kotlin(
),
)
namespace = options.get("namespace", "GleanMetrics")
glean_namespace = options.get("glean_namespace", "mozilla.components.service.glean")
for category_key, category_val in objs.items():
filename = util.Camelize(category_key) + ".kt"
filepath = output_dir / filename

47
third_party/python/glean_parser/glean_parser/lint.py vendored
View File

@ -16,6 +16,7 @@ from typing import (
Iterable,
Optional,
Tuple,
Union,
) # noqa
@ -198,7 +199,7 @@ def check_category_generic(
def check_bug_number(
metric: metrics.Metric, parser_config: Dict[str, Any]
metric: Union[metrics.Metric, pings.Ping], parser_config: Dict[str, Any]
) -> LintGenerator:
number_bugs = [str(bug) for bug in metric.bugs if isinstance(bug, int)]
@ -206,7 +207,8 @@ def check_bug_number(
yield (
f"For bugs {', '.join(number_bugs)}: "
"Bug numbers are deprecated and should be changed to full URLs. "
"For example, use 'http://bugzilla.mozilla.org/12345' instead of '12345'."
f"For example, use 'http://bugzilla.mozilla.org/{number_bugs[0]}' "
f"instead of '{number_bugs[0]}'."
)
@ -272,7 +274,7 @@ CATEGORY_CHECKS: Dict[
# The checks that operate on individual metrics:
# {NAME: (function, is_error)}
INDIVIDUAL_CHECKS: Dict[
METRIC_CHECKS: Dict[
str, Tuple[Callable[[metrics.Metric, dict], LintGenerator], CheckType]
] = {
"UNIT_IN_NAME": (check_unit_in_name, CheckType.error),
@ -285,6 +287,15 @@ INDIVIDUAL_CHECKS: Dict[
}
# The checks that operate on individual pings:
# {NAME: (function, is_error)}
PING_CHECKS: Dict[
str, Tuple[Callable[[pings.Ping, dict], LintGenerator], CheckType]
] = {
"BUG_NUMBER": (check_bug_number, CheckType.error),
}
class GlinterNit:
def __init__(self, check_name: str, name: str, msg: str, check_type: CheckType):
self.check_name = check_name
@ -299,6 +310,31 @@ class GlinterNit:
)
def _lint_pings(
category: Dict[str, Union[metrics.Metric, pings.Ping]],
parser_config: Dict[str, Any],
):
nits: List[GlinterNit] = []
for (ping_name, ping) in sorted(list(category.items())):
assert isinstance(ping, pings.Ping)
for (check_name, (check_func, check_type)) in PING_CHECKS.items():
new_nits = list(check_func(ping, parser_config))
if len(new_nits):
if check_name not in ping.no_lint:
nits.extend(
GlinterNit(
check_name,
ping_name,
msg,
check_type,
)
for msg in new_nits
)
return nits
def lint_metrics(
objs: metrics.ObjectTree,
parser_config: Optional[Dict[str, Any]] = None,
@ -317,6 +353,7 @@ def lint_metrics(
nits: List[GlinterNit] = []
for (category_name, category) in sorted(list(objs.items())):
if category_name == "pings":
nits.extend(_lint_pings(category, parser_config))
continue
# Make sure the category has only Metrics, not Pings
@ -337,7 +374,7 @@ def lint_metrics(
)
for (_metric_name, metric) in sorted(list(category_metrics.items())):
for (check_name, (check_func, check_type)) in INDIVIDUAL_CHECKS.items():
for (check_name, (check_func, check_type)) in METRIC_CHECKS.items():
new_nits = list(check_func(metric, parser_config))
if len(new_nits):
if check_name not in metric.no_lint:
@ -402,7 +439,7 @@ def lint_yaml_files(
if len(nits):
print("Sorry, Glean found some glinter nits:", file=file)
for (path, p) in nits:
print(f"{path} ({p.line}:{p.column}) - {p.message}")
print(f"{path} ({p.line}:{p.column}) - {p.message}", file=file)
print("", file=file)
print("Please fix the above nits to continue.", file=file)

27
third_party/python/glean_parser/glean_parser/markdown.py vendored
View File

@ -10,6 +10,7 @@ Outputter to generate Markdown documentation for metrics.
from pathlib import Path
from typing import Any, Dict, List, Optional, Tuple, Union
from urllib.parse import urlsplit, parse_qs
from . import metrics
@ -121,6 +122,31 @@ def ping_data_reviews(
return None
def ping_review_title(data_url: str, index: int) -> str:
"""
Return a title for a data review in human readable form.
:param data_url: A url for data review.
:param index: Position of the data review on list (e.g: 1, 2, 3...).
"""
url_object = urlsplit(data_url)
# Bugzilla urls like `https://bugzilla.mozilla.org/show_bug.cgi?id=1581647`
query = url_object.query
params = parse_qs(query)
# GitHub urls like `https://github.com/mozilla-mobile/fenix/pull/1707`
path = url_object.path
short_url = path[1:].replace("/pull/", "#")
if params and params["id"]:
return f"Bug {params['id'][0]}"
elif url_object.netloc == "github.com":
return short_url
return f"Review {index}"
def ping_bugs(
ping_name: str, custom_pings_cache: Optional[Dict[str, pings.Ping]] = None
) -> Optional[List[str]]:
@ -222,6 +248,7 @@ def output_markdown(
("ping_docs", ping_docs),
("ping_reasons", lambda x: ping_reasons(x, custom_pings_cache)),
("ping_data_reviews", lambda x: ping_data_reviews(x, custom_pings_cache)),
("ping_review_title", ping_review_title),
("ping_bugs", lambda x: ping_bugs(x, custom_pings_cache)),
(
"ping_include_client_id",

21
third_party/python/glean_parser/glean_parser/metrics.py vendored
View File

@ -56,6 +56,8 @@ class Metric:
gecko_datapoint: str = "",
no_lint: Optional[List[str]] = None,
data_sensitivity: Optional[List[str]] = None,
defined_in: Optional[Dict] = None,
telemetry_mirror: Optional[str] = None,
_config: Dict[str, Any] = None,
_validated: bool = False,
):
@ -88,13 +90,16 @@ class Metric:
self.data_sensitivity = [
getattr(DataSensitivity, x) for x in data_sensitivity
]
self.defined_in = defined_in
if telemetry_mirror is not None:
self.telemetry_mirror = telemetry_mirror
# _validated indicates whether this metric has already been jsonschema
# validated (but not any of the Python-level validation).
if not _validated:
data = {
"$schema": parser.METRICS_ID,
self.category: {self.name: self.serialize()},
self.category: {self.name: self._serialize_input()},
} # type: Dict[str, util.JSONType]
for error in parser.validate(data):
raise ValueError(error)
@ -146,6 +151,7 @@ class Metric:
return cls.metric_types[metric_type](
category=category,
name=name,
defined_in=getattr(metric_info, "defined_in", None),
_validated=validated,
_config=config,
**metric_info,
@ -168,6 +174,11 @@ class Metric:
del d["category"]
return d
def _serialize_input(self) -> Dict[str, util.JSONType]:
d = self.serialize()
modified_dict = util.remove_output_params(d, "defined_in")
return modified_dict
def identifier(self) -> str:
"""
Create an identifier unique for this metric.
@ -353,4 +364,12 @@ class LabeledCounter(Labeled, Counter):
typename = "labeled_counter"
class Rate(Metric):
typename = "rate"
def __init__(self, *args, **kwargs):
self.denominator_metric = kwargs.pop("denominator_metric", None)
super().__init__(*args, **kwargs)
ObjectTree = Dict[str, Dict[str, Union[Metric, pings.Ping]]]

37
third_party/python/glean_parser/glean_parser/parser.py vendored
View File

@ -25,10 +25,8 @@ from . import util
ROOT_DIR = Path(__file__).parent
SCHEMAS_DIR = ROOT_DIR / "schemas"
METRICS_ID = "moz://mozilla.org/schemas/glean/metrics/1-0-0"
PINGS_ID = "moz://mozilla.org/schemas/glean/pings/1-0-0"
FILE_TYPES = {METRICS_ID: "metrics", PINGS_ID: "pings"}
METRICS_ID = "moz://mozilla.org/schemas/glean/metrics/2-0-0"
PINGS_ID = "moz://mozilla.org/schemas/glean/pings/2-0-0"
def _update_validator(validator):
@ -62,7 +60,7 @@ def _load_file(
`parser_config["allow_missing_files"]` is `True`.
"""
try:
content = util.load_yaml_or_json(filepath, ordered_dict=True)
content = util.load_yaml_or_json(filepath)
except FileNotFoundError:
if not parser_config.get("allow_missing_files", False):
raise
@ -86,7 +84,14 @@ def _load_file(
if not isinstance(schema_key, str):
raise TypeError(f"Invalid schema key {schema_key}")
filetype = FILE_TYPES.get(schema_key)
filetype: Optional[str] = None
try:
filetype = schema_key.split("/")[-2]
except IndexError:
filetype = None
if filetype not in ("metrics", "pings"):
filetype = None
for error in validate(content, filepath):
content = {}
@ -235,6 +240,9 @@ def _instantiate_metrics(
if metric_obj is not None:
metric_obj.no_lint = list(set(metric_obj.no_lint + global_no_lint))
if isinstance(filepath, Path):
metric_obj.defined_in["filepath"] = str(filepath)
already_seen = sources.get((category_key, metric_key))
if already_seen is not None:
# We've seen this metric name already
@ -262,9 +270,14 @@ def _instantiate_pings(
Load a list of pings.yaml files, convert the JSON information into Ping
objects.
"""
global_no_lint = content.get("no_lint", [])
assert isinstance(global_no_lint, list)
for ping_key, ping_val in content.items():
if ping_key.startswith("$"):
continue
if ping_key == "no_lint":
continue
if not config.get("allow_reserved"):
if ping_key in RESERVED_PING_NAMES:
yield util.format_error(
@ -277,11 +290,21 @@ def _instantiate_pings(
raise TypeError(f"Invalid content for ping {ping_key}")
ping_val["name"] = ping_key
try:
ping_obj = Ping(**ping_val)
ping_obj = Ping(
defined_in=getattr(ping_val, "defined_in", None),
_validated=True,
**ping_val,
)
except Exception as e:
yield util.format_error(filepath, f"On instance '{ping_key}'", str(e))
continue
if ping_obj is not None:
ping_obj.no_lint = list(set(ping_obj.no_lint + global_no_lint))
if isinstance(filepath, Path) and ping_obj.defined_in is not None:
ping_obj.defined_in["filepath"] = str(filepath)
already_seen = sources.get(ping_key)
if already_seen is not None:
# We've seen this ping name already

14
third_party/python/glean_parser/glean_parser/pings.py vendored
View File

@ -28,6 +28,8 @@ class Ping:
include_client_id: bool = False,
send_if_empty: bool = False,
reasons: Dict[str, str] = None,
defined_in: Optional[Dict] = None,
no_lint: Optional[List[str]] = None,
_validated: bool = False,
):
# Avoid cyclical import
@ -35,6 +37,7 @@ class Ping:
self.name = name
self.description = description
self.bugs = bugs
self.notification_emails = notification_emails
if data_reviews is None:
@ -45,13 +48,17 @@ class Ping:
if reasons is None:
reasons = {}
self.reasons = reasons
self.defined_in = defined_in
if no_lint is None:
no_lint = []
self.no_lint = no_lint
# _validated indicates whether this metric has already been jsonschema
# validated (but not any of the Python-level validation).
if not _validated:
data: Dict[str, util.JSONType] = {
"$schema": parser.PINGS_ID,
self.name: self.serialize(),
self.name: self._serialize_input(),
}
for error in parser.validate(data):
raise ValueError(error)
@ -74,6 +81,11 @@ class Ping:
del d["name"]
return d
def _serialize_input(self) -> Dict[str, util.JSONType]:
d = self.serialize()
modified_dict = util.remove_output_params(d, "defined_in")
return modified_dict
def identifier(self) -> str:
"""
Used for the "generated from ..." comment in the output.

2
third_party/python/glean_parser/glean_parser/schemas/metrics.1-0-0.schema.yaml vendored
View File

@ -211,7 +211,7 @@ definitions:
minItems: 1
items:
anyOf:
- type: integer
- type: integer # Keep supporting integer for backward-compat
- type: string
format: uri

631
third_party/python/glean_parser/glean_parser/schemas/metrics.2-0-0.schema.yaml vendored Normal file
View File

@ -0,0 +1,631 @@
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, You can obtain one at http://mozilla.org/MPL/2.0/.
---
$schema: http://json-schema.org/draft-07/schema#
title: Metrics
description: |
Schema for the metrics.yaml files for Mozilla's Glean telemetry SDK.
The top-level of the `metrics.yaml` file has a key defining each category of
metrics. Categories must be snake_case, and they may also have dots `.` to
define subcategories.
$id: moz://mozilla.org/schemas/glean/metrics/2-0-0
definitions:
token:
type: string
pattern: "^[A-Za-z_][A-Za-z0-9_\\.]*$"
snake_case:
type: string
pattern: "^[a-z_][a-z0-9_]*$"
dotted_snake_case:
type: string
pattern: "^[a-z_][a-z0-9_]{0,29}(\\.[a-z_][a-z0-9_]{0,29})*$"
maxLength: 40
# Prior to version 2.0.0 of the schema, special ping names with underscores
# were also supported.
kebab_case:
type: string
pattern: "^[a-z][a-z0-9-]{0,29}$"
long_id:
allOf:
- $ref: "#/definitions/snake_case"
- maxLength: 40
short_id:
allOf:
- $ref: "#/definitions/snake_case"
- maxLength: 30
labeled_metric_id:
type: string
pattern: "^[a-z_][a-z0-9_-]{0,29}(\\.[a-z_][a-z0-9_-]{0,29})*$"
maxLength: 71 # Note: this should be category + metric + 1
metric:
description: |
Describes a single metric.
See https://mozilla.github.io/glean_parser/metrics-yaml.html
type: object
additionalProperties: false
properties:
type:
title: Metric type
description: |
**Required.**
Specifies the type of a metric, like "counter" or "event". This
defines which operations are valid for the metric, how it is stored
and how data analysis tooling displays it.
The supported types are:
- `event`: Record a specific event (with optional metadata).
Additional properties: `extra_keys`.
- `boolean`: A metric storing values of true or false.
- `string`: A metric storing Unicode string values.
- `string_list`: a list of Unicode strings.
- `counter`: A numeric value that can only be incremented.
- `quantity`: A numeric value that is set directly.
- `timespan`: Represents a time interval. Additional properties:
`time_unit`.
- `timing_distribution`: Record the distribution of multiple
timings. Additional properties: `time_unit`.
- `datetime`: A date/time value. Represented as an ISO datetime in
UTC. Additional properties: `time_unit`.
- `uuid`: Record a UUID v4.
- `jwe`: Record a [JWE](https://tools.ietf.org/html/rfc7516) value.
- `memory_distribution`: A histogram for recording memory usage
values. Additional properties: `memory_unit`.
- `custom_distribution`: A histogram with a custom range and number
of buckets. This metric type is for legacy support only and is
only allowed for metrics coming from GeckoView. Additional
properties: `range_min`, `range_max`, `bucket_count`,
`histogram_type`.
- Additionally, labeled versions of many metric types are supported.
These support the `labels`_ parameter, allowing multiple instances
of the metric to be stored at a given set of labels. The labeled
metric types include:
`labeled_boolean`, `labeled_string`, `labeled_counter`.
type: string
enum:
- event
- boolean
- string
- string_list
- counter
- quantity
- timespan
- timing_distribution
- custom_distribution
- memory_distribution
- datetime
- uuid
- jwe
- labeled_boolean
- labeled_string
- labeled_counter
- rate
description:
title: Description
description: |
**Required.**
A textual description of what this metric does, what it means, and its
edge cases or any other helpful information.
Descriptions may contain [markdown
syntax](https://www.markdownguide.org/basic-syntax/).
type: string
lifetime:
title: Lifetime
description: |
Defines the lifetime of the metric. It must be one of the following
values:
- `ping` (default): The metric is reset each time it is sent in a
ping.
- `user`: The metric contains a property that is part of the user's
profile and is never reset.
- `application`: The metric contains a property that is related to the
application, and is reset only at application restarts.
enum:
- ping
- user
- application
default: ping
send_in_pings:
title: Send in pings
description: |
Which pings the metric should be sent on. If not specified, the metric
is sent on the "default ping", which is the `events` ping for events,
and the `metrics` ping for everything else. Most metrics don't need to
specify this.
(There is an additional special value of `all-pings` for internal
Glean metrics only that is used to indicate that a metric may appear
in any ping.)
type: array
items:
$ref: "#/definitions/kebab_case"
default:
- default
notification_emails:
title: Notification emails
description: |
**Required.**
A list of email addresses to notify for important events with the
metric or when people with context or ownership for the metric need to
be contacted.
type: array
minItems: 1
items:
type: string
format: email
bugs:
title: Related bugs
description: |
**Required.**
A list of bug URLs (e.g. Bugzilla and Github) that are relevant to
this metric, e.g., tracking its original implementation or later
changes to it.
Prior to version 2.0.0 of the schema, bugs could also be integers.
type: array
minItems: 1
items:
type: string
format: uri
data_reviews:
title: Review references
description: |
**Required.**
A list of URIs to any data collection reviews relevant to the metric.
type: array
items:
type: string
format: uri
disabled:
title: Disabled
description: |
If `true`, the metric is disabled, and any metric collection on it
will be silently ignored at runtime.
type: boolean
default: false
expires:
title: Expires
description: |
**Required.**
By default it may be one of the following values:
- `<build date>`: An ISO date `yyyy-mm-dd` in UTC on which the
metric expires. For example, `2019-03-13`. This date is checked at
build time. Except in special cases, this form should be used so
that the metric automatically "sunsets" after a period of time.
- `never`: This metric never expires.
- `expired`: This metric is manually expired.
The default may be overriden in certain applications by the
`custom_validate_expires` and `custom_is_expired` configs.
type: string
version:
title: Metric version
description: |
The version of the metric. A monotonically increasing value. If not
provided, defaults to 0.
time_unit:
title: Time unit
description: |
For timespans and datetimes, specifies the unit that the metric will
be stored and displayed in. If not provided, it defaults to
"millisecond". Time values are sent to the backend as integers, so
`time_unit`_ determines the maximum resolution at which timespans are
recorded. Times are always truncated, not rounded, to the nearest time
unit. For example, a measurement of 25 ns will be returned as 0 ms if
`time_unit` is `"millisecond"`.
For timing distributions, times are always recorded and sent in
nanoseconds, but `time_unit` controls the minimum and maximum values.
If not provided, it defaults to "nanosecond".
- nanosecond: 1ns <= x <= 10 minutes
- microsecond: 1μs <= x <= ~6.94 days
- millisecond: 1ms <= x <= ~19 years
Valid when `type`_ is `timespan`, `timing_distribution` or `datetime`.
enum:
- nanosecond
- microsecond
- millisecond
- second
- minute
- hour
- day
memory_unit:
title: Memory unit
description: |
The unit that the incoming memory size values are recorded in.
The units are the power-of-2 units, so "kilobyte" is correctly a
"kibibyte".
- kilobyte == 2^10 == 1,024 bytes
- megabyte == 2^20 == 1,048,576 bytes
- gigabyte == 2^30 == 1,073,741,824 bytes
Values are automatically converted to and transmitted as bytes.
Valid when `type`_ is `memory_distribution`.
enum:
- byte
- kilobyte
- megabyte
- gigabyte
labels:
title: Labels
description: |
A list of labels for a labeled metric. If provided, the labels are
enforced at run time, and recording to an unknown label is recorded
to the special label `__other__`. If not provided, the labels
may be anything, but using too many unique labels will put some
labels in the special label `__other__`.
Valid with any of the labeled metric types.
anyOf:
- type: array
uniqueItems: true
items:
$ref: "#/definitions/labeled_metric_id"
maxItems: 16
- type: "null"
extra_keys:
title: Extra keys
description: |
The acceptable keys on the "extra" object sent with events. This is an
object mapping the key to an object containing metadata about the key.
A maximum of 10 extra keys is allowed.
This metadata object has the following keys:
- `description`: **Required.** A description of the key.
Valid when `type`_ is `event`.
type: object
propertyNames:
$ref: "#/definitions/dotted_snake_case"
additionalProperties:
type: object
properties:
description:
type: string
required:
- description
maxProperties: 10
default: {}
gecko_datapoint:
title: Gecko Datapoint
description: |
This is a Gecko-specific property. It is the name of the Gecko metric
to accumulate the data from, when using the Glean SDK in a product
using GeckoView. See bug 1566356 for more context.
type: string
range_min:
title: Range minimum
description: |
The minimum value of a custom distribution.
Valid when `type`_ is `custom_distribution`.
type: number
default: 1
range_max:
title: Range maximum
description: |
The maximum value of a custom distribution.
Required when `type`_ is `custom_distribution`.
type: number
bucket_count:
title: Bucket count
description: |
The number of buckets to include in a custom distribution.
Required when `type`_ is `custom_distribution`.
type: number
minimum: 1
maximum: 100
histogram_type:
title: Histogram type
description: |
The type of histogram bucketing to use:
- `linear`: The buckets are linearly spaced within the range.
- `exponential`: The buckets use the natural logarithmic so the
smaller-valued buckets are smaller in size than the higher-valued
buckets.
Required when `type`_ is `custom_distribution`.
enum:
- linear
- exponential
unit:
title: Unit
description: |
The unit of the metric, for metrics that don't already require a
meaningful unit, such as `time_unit`.
This is provided for informational purposes only and doesn't have any
effect on data collection.
type: string
no_lint:
title: Lint checks to skip
description: |
This parameter lists any lint checks to skip for this metric only.
type: array
items:
type: string
decrypted_name:
title: Decrypted name
description: |
Name of the column where to persist the decrypted value
stored in the JWE after processing.
Required when `type`_ is `jwe`.
type: string
pattern: "^[a-z_][a-z0-9_]{0,29}(\\.[a-z_][a-z0-9_]{0,29})*$"
data_sensitivity:
title: The level of data sensitivity
description: |
There are four data collection categories related to data sensitivity
[defined here](https://wiki.mozilla.org/Firefox/Data_Collection):
- **Category 1: Technical Data:** (`technical`) Information about the
machine or Firefox itself. Examples include OS, available memory,
crashes and errors, outcome of automated processes like updates,
safebrowsing, activation, version \#s, and buildid. This also
includes compatibility information about features and APIs used by
websites, addons, and other 3rd-party software that interact with
Firefox during usage.
- **Category 2: Interaction Data:** (`interaction`) Information about
the user’s direct engagement with Firefox. Examples include how many
tabs, addons, or windows a user has open; uses of specific Firefox
features; session length, scrolls and clicks; and the status of
discrete user preferences.
- **Category 3: Web activity data:** (`web_activity`) Information
about user web browsing that could be considered sensitive. Examples
include users’ specific web browsing history; general information
about their web browsing history (such as TLDs or categories of
webpages visited over time); and potentially certain types of
interaction data about specific webpages visited.
- **Category 4: Highly sensitive data:** (`highly_sensitive`)
Information that directly identifies a person, or if combined with
other data could identify a person. Examples include e-mail,
usernames, identifiers such as google ad id, apple id, fxaccount,
city or country (unless small ones are explicitly filtered out), or
certain cookies. It may be embedded within specific website content,
such as memory contents, dumps, captures of screen data, or DOM
data.
type: array
items:
enum:
- technical
- interaction
- web_activity
- highly_sensitive
type: string
minLength: 1
uniqueItems: true
telemetry_mirror:
title: Which probe in Telemetry to mirror this metric's value to.
description: |
The C++ enum form of the Scalar, Event, or Histogram to which we
should mirror values.
Use is limited to Firefox Desktop only.
Has no effect when used with non-FOG outputters.
See FOG's documentation on mirroring for details -
https://firefox-source-docs.mozilla.org/toolkit/components/glean/mirroring.html
type: string
minLength: 6
denominator_metric:
title: The name of the denominator for this `rate` metric.
description: |
Denominators for `rate` metrics may be private and internal
or shared and external.
External denominators are `counter` metrics.
This field names the `counter` metric that serves as this
`rate` metric's external denominator.
The named denominator must be defined in this component
so glean_parser can find it.
type: string
required:
- type
- bugs
- description
- notification_emails
- data_reviews
- expires
type: object
propertyNames:
anyOf:
- allOf:
- $ref: "#/definitions/dotted_snake_case"
- not:
description: "'pings' is reserved as a category name."
const: pings
- enum: ['$schema']
properties:
$schema:
type: string
format: url
no_lint:
title: Lint checks to skip globally
description: |
This parameter lists any lint checks to skip for this whole file.
type: array
items:
type: string
additionalProperties:
type: object
propertyNames:
anyOf:
- $ref: "#/definitions/short_id"
additionalProperties:
allOf:
- $ref: "#/definitions/metric"
-
if:
properties:
type:
const: event
then:
properties:
lifetime:
description: |
Event metrics must have ping lifetime.
const: ping
- if:
not:
properties:
type:
enum:
- timing_distribution
- custom_distribution
- memory_distribution
- quantity
- boolean
- string
- labeled_counter
then:
properties:
gecko_datapoint:
description: |
`gecko_datapoint` is only allowed for `timing_distribution`,
`custom_distribution`, `memory_distribution`, `quantity`,
`boolean`, `string` and `labeled_counter`.
maxLength: 0
-
if:
properties:
type:
enum:
- custom_distribution
then:
required:
- gecko_datapoint
description: |
`custom_distribution` is only allowed for Gecko
metrics.
-
if:
properties:
type:
const: custom_distribution
then:
required:
- range_max
- bucket_count
- histogram_type
description: |
`custom_distribution` is missing required parameters `range_max`,
`bucket_count` and `histogram_type`.
-
if:
properties:
type:
const: memory_distribution
then:
required:
- memory_unit
description: |
`memory_distribution` is missing required parameter `memory_unit`.
-
if:
properties:
type:
const: quantity
then:
required:
- unit
description: |
`quantity` is missing required parameter `unit`.
-
if:
properties:
type:
const: jwe
then:
required:
- decrypted_name
description: |
`jwe` is missing required parameter `decrypted_name`.
- if:
not:
properties:
type:
const: rate
then:
properties:
denominator_metric:
description: |
`denominator_metric` is only allowed for `rate`.
maxLength: 0

20
third_party/python/glean_parser/glean_parser/schemas/pings.1-0-0.schema.yaml vendored
View File

@ -33,7 +33,7 @@ propertyNames:
allOf:
- anyOf:
- $ref: "#/definitions/kebab_case"
- enum: ['$schema']
- enum: ['$schema', 'no_lint']
- not:
enum: ['all-pings']
@ -42,6 +42,14 @@ properties:
type: string
format: url
no_lint:
title: Lint checks to skip globally
description: |
This parameter lists any lint checks to skip for this whole file.
type: array
items:
type: string
additionalProperties:
type: object
properties:
@ -101,7 +109,7 @@ additionalProperties:
minItems: 1
items:
anyOf:
- type: integer
- type: integer # Keep supporting integer for backward-compat
- type: string
format: uri
@ -131,6 +139,14 @@ additionalProperties:
additionalProperties:
type: string
no_lint:
title: Lint checks to skip
description: |
This parameter lists any lint checks to skip for this metric only.
type: array
items:
type: string
required:
- description
- include_client_id

153
third_party/python/glean_parser/glean_parser/schemas/pings.2-0-0.schema.yaml vendored Normal file
View File

@ -0,0 +1,153 @@
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, You can obtain one at http://mozilla.org/MPL/2.0/.
---
$schema: http://json-schema.org/draft-07/schema#
title: Pings
description: |
Schema for the pings.yaml files for Mozilla's Glean telemetry SDK.
The top-level of the `pings.yaml` file has a key defining the name of each
ping. The values contain metadata about that ping. Ping names must be
kebab-case per https://docs.telemetry.mozilla.org/cookbooks/new_ping.html
$id: moz://mozilla.org/schemas/glean/pings/2-0-0
definitions:
dotted_snake_case:
type: string
pattern: "^[a-z_][a-z0-9_]{0,29}(\\.[a-z_][a-z0-9_]{0,29})*$"
maxLength: 40
# Prior to version 2.0.0 of the schema, special ping names with underscores
# were also supported.
kebab_case:
type: string
pattern: "^[a-z][a-z0-9-]{0,29}$"
type: object
propertyNames:
allOf:
- anyOf:
- $ref: "#/definitions/kebab_case"
- enum: ['$schema', 'no_lint']
- not:
enum: ['all-pings']
properties:
$schema:
type: string
format: url
no_lint:
title: Lint checks to skip globally
description: |
This parameter lists any lint checks to skip for this whole file.
type: array
items:
type: string
additionalProperties:
type: object
properties:
description:
title: Description
description: |
**Required.**
A textual description of the purpose of this ping and what it contains.
Descriptions may contain [markdown
syntax](https://www.markdownguide.org/basic-syntax/).
type: string
include_client_id:
title: Include client id
description: |
**Required.**
When `true`, include the `client_id` value in the ping.
type: boolean
send_if_empty:
title: Send if empty
description: |
When `false` a ping is sent only if it contains data (the default).
When `true` a ping is sent even if it contains no data.
type: boolean
notification_emails:
title: Notification emails
description: |
**Required.**
A list of email addresses to notify for important events with the
ping or when people with context or ownership for the ping need to
be contacted.
type: array
minItems: 1
items:
type: string
format: email
bugs:
title: Related bugs
description: |
**Required.**
A list of bugs (e.g. Bugzilla and Github) that are relevant to this
ping, e.g., tracking its original implementation or later changes to
it.
It must be a URI to a bug page in a tracker.
Prior to version 2.0.0 of the schema, bugs could also be integers.
type: array
minItems: 1
items:
type: string
format: uri
data_reviews:
title: Review references
description: |
**Required.**
A list of URIs to any data collection reviews relevant to the ping.
type: array
items:
type: string
format: uri
reasons:
title: The reasons this ping can be sent.
description: |
A list of reasons that the ping might be triggered. Sent in the ping's
`ping_info.reason` field.
Specified as a mapping from reason codes (which are short strings), to
a textual description of the reason.
type: object
propertyNames:
type: string
maxLength: 30
additionalProperties:
type: string
no_lint:
title: Lint checks to skip
description: |
This parameter lists any lint checks to skip for this metric only.
type: array
items:
type: string
required:
- description
- include_client_id
- bugs
- notification_emails
- data_reviews
additionalProperties: false

2
third_party/python/glean_parser/glean_parser/swift.py vendored
View File

@ -166,7 +166,7 @@ def output_swift(
fd.write(
template.render(
categories=categories,
extra_args=util.extra_metric_args,
extra_args=util.metric_args,
namespace=namespace,
glean_namespace=glean_namespace,
allow_reserved=options.get("allow_reserved", False),

33
third_party/python/glean_parser/glean_parser/templates/javascript.jinja2 vendored Normal file
View File

@ -0,0 +1,33 @@
{# The final Javascript/Typescript code is autogenerated, but this
Jinja2 template is not. Please file bugs! #}
{% macro obj_declaration(obj) %}
new {{ obj.type|class_name }}({
{% for arg_name in (obj.type|args).common if obj[arg_name] is defined %}
{{ arg_name|camelize }}: {{ obj[arg_name]|js }},
{% endfor %}
}{% for arg_name in (obj.type|args).extra if obj[arg_name] is defined %}, {{ obj[arg_name]|js }}{% endfor %}){% endmacro %}
/* eslint-disable */
/* This Source Code Form is subject to the terms of the Mozilla Public
* License, v. 2.0. If a copy of the MPL was not distributed with this
* file, You can obtain one at http://mozilla.org/MPL/2.0/. */
// AUTOGENERATED BY glean_parser. DO NOT EDIT.
{% for type in types %}
import {{ type|class_name }} from "@mozilla/glean/{{ glean_namespace }}/private/{{ type|import_path }}";
{% endfor %}
{% if lang == "javascript" %}
"use strict"
{% endif %}
{% for obj in objs.values() %}
/**
* {{ obj.description|wordwrap() | replace("\n", "\n * ") }}
*
* Generated from `{{ obj.identifier() }}`.
*/
export const {{ obj.name|camelize }} = {{ obj_declaration(obj) }};
{% endfor %}

27
third_party/python/glean_parser/glean_parser/templates/kotlin.buildinfo.jinja2 vendored Normal file
View File

@ -0,0 +1,27 @@
// -*- mode: kotlin -*-
/*
* AUTOGENERATED BY glean_parser. DO NOT EDIT.
*/
{# The rendered markdown is autogenerated, but this
Jinja2 template is not. Please file bugs! #}
/* This Source Code Form is subject to the terms of the Mozilla Public
* License, v. 2.0. If a copy of the MPL was not distributed with this
* file, You can obtain one at http://mozilla.org/MPL/2.0/. */
@file:Suppress("PackageNaming", "MaxLineLength")
package {{ namespace }}
import {{ glean_namespace }}.BuildInfo
import {{ namespace_package }}.BuildConfig
internal object GleanBuildInfo {
val buildInfo: BuildInfo by lazy {
BuildInfo(
versionCode = BuildConfig.VERSION_CODE.toString(),
versionName = BuildConfig.VERSION_NAME
)
}
}

11
third_party/python/glean_parser/glean_parser/templates/markdown.jinja2 vendored
View File

@ -3,6 +3,7 @@
Jinja2 template is not. Please file bugs! #}
# Metrics
This document enumerates the metrics collected by {{ project_title }} using the [Glean SDK](https://mozilla.github.io/glean/book/index.html).
This project may depend on other projects which also collect metrics.
This means you might have to go searching through the dependency tree to get a full picture of everything collected by this project.
@ -10,10 +11,9 @@ This means you might have to go searching through the dependency tree to get a f
# Pings
{% for ping_name in metrics_by_pings.keys()|sort %}
- [{{ ping_name }}]({{ '#' }}{{ ping_name|replace(" ","-") }})
- [{{ ping_name }}]({{ '#' }}{{ ping_name|replace(" ","-") }})
{% endfor %}
{% for ping_name in metrics_by_pings.keys()|sort %}
{% raw %}##{% endraw %} {{ ping_name }}
@ -58,8 +58,10 @@ This ping includes the [client id](https://mozilla.github.io/glean/book/user/pin
{% endfor %}
{% endif %}
All Glean pings contain built-in metrics in the [`ping_info`](https://mozilla.github.io/glean/book/user/pings/index.html#the-ping_info-section) and [`client_info`](https://mozilla.github.io/glean/book/user/pings/index.html#the-client_info-section) sections.
{% if metrics_by_pings[ping_name] %}
The following metrics are added to the ping:
In addition to those built-in metrics, the following metrics are added to the ping:
| Name | Type | Description | Data reviews | Extras | Expiration | [Data Sensitivity](https://wiki.mozilla.org/Firefox/Data_Collection) |
| --- | --- | --- | --- | --- | --- | --- |
@ -68,7 +70,7 @@ The following metrics are added to the ping:
{{- '['}}{{ metric.type }}]({{ metric.type|metrics_docs }}) |
{{- metric.description|replace("\n", " ") }} |
{%- for data_review in metric.data_reviews %}
[{{ loop.index }}]({{ data_review }}){{ ", " if not loop.last }}
[{{ data_review|ping_review_title(loop.index) }}]({{ data_review }}){{ ", " if not loop.last }}
{%- endfor -%} |
{%- if metric|extra_info -%}
<ul>
@ -85,7 +87,6 @@ This ping contains no metrics.
{% endif %}
{% endfor %}
Data categories are [defined here](https://wiki.mozilla.org/Firefox/Data_Collection).
<!-- AUTOGENERATED BY glean_parser. DO NOT EDIT. -->

37
third_party/python/glean_parser/glean_parser/translate.py vendored
View File

@ -11,12 +11,14 @@ High-level interface for translating `metrics.yaml` into other formats.
from pathlib import Path
import os
import shutil
import sys
import tempfile
from typing import Any, Callable, Dict, Iterable, List, Optional
from . import lint
from . import parser
from . import csharp
from . import javascript
from . import kotlin
from . import markdown
from . import metrics
@ -51,12 +53,47 @@ class Outputter:
OUTPUTTERS = {
"csharp": Outputter(csharp.output_csharp, ["*.cs"]),
"javascript": Outputter(javascript.output_javascript, []),
"typescript": Outputter(javascript.output_typescript, []),
"kotlin": Outputter(kotlin.output_kotlin, ["*.kt"]),
"markdown": Outputter(markdown.output_markdown, []),
"swift": Outputter(swift.output_swift, ["*.swift"]),
}
def transform_metrics(objects):
"""
Transform the object model from one that represents the YAML definitions
to one that reflects the type specifics needed by code generators.
e.g. This will transform a `rate` to be a `numerator` if its denominator is
external.
"""
counters = {}
numerators_by_denominator: Dict[str, Any] = {}
for category_val in objects.values():
for metric in category_val.values():
fqmn = metric.identifier()
if getattr(metric, "type", None) == "counter":
counters[fqmn] = metric
denominator_name = getattr(metric, "denominator_metric", None)
if denominator_name:
metric.type = "numerator"
numerators_by_denominator.setdefault(denominator_name, [])
numerators_by_denominator[denominator_name].append(fqmn)
for denominator_name, numerator_names in numerators_by_denominator.items():
if denominator_name not in counters:
print(
f"No `counter` named {denominator_name} found to be used as"
"denominator for {numerator_names}",
file=sys.stderr,
)
return 1
counters[denominator_name].type = "denominator"
counters[denominator_name].numerators = numerator_names
def translate_metrics(
input_filepaths: Iterable[Path],
output_dir: Path,

112
third_party/python/glean_parser/glean_parser/util.py vendored
View File

@ -66,47 +66,41 @@ class _NoDatesSafeLoader(yaml.SafeLoader):
_NoDatesSafeLoader.remove_implicit_resolver("tag:yaml.org,2002:timestamp")
if sys.version_info < (3, 7):
# In Python prior to 3.7, dictionary order is not preserved. However, we
# want the metrics to appear in the output in the same order as they are in
# the metrics.yaml file, so on earlier versions of Python we must use an
# OrderedDict object.
def ordered_yaml_load(stream):
class OrderedLoader(_NoDatesSafeLoader):
pass
def yaml_load(stream):
"""
Map line number to yaml nodes, and preserve the order
of metrics as they appear in the metrics.yaml file.
"""
def construct_mapping(loader, node):
loader.flatten_mapping(node)
return OrderedDict(loader.construct_pairs(node))
class SafeLineLoader(_NoDatesSafeLoader):
pass
OrderedLoader.add_constructor(
yaml.resolver.BaseResolver.DEFAULT_MAPPING_TAG, construct_mapping
def _construct_mapping_adding_line(loader, node):
loader.flatten_mapping(node)
mapping = OrderedDict(loader.construct_pairs(node))
mapping.defined_in = {"line": node.start_mark.line}
return mapping
SafeLineLoader.add_constructor(
yaml.resolver.BaseResolver.DEFAULT_MAPPING_TAG, _construct_mapping_adding_line
)
return yaml.load(stream, SafeLineLoader)
def ordered_yaml_dump(data, **kwargs):
class OrderedDumper(yaml.Dumper):
pass
def _dict_representer(dumper, data):
return dumper.represent_mapping(
yaml.resolver.BaseResolver.DEFAULT_MAPPING_TAG, data.items()
)
return yaml.load(stream, OrderedLoader)
def ordered_yaml_dump(data, **kwargs):
class OrderedDumper(yaml.Dumper):
pass
def _dict_representer(dumper, data):
return dumper.represent_mapping(
yaml.resolver.BaseResolver.DEFAULT_MAPPING_TAG, data.items()
)
OrderedDumper.add_representer(OrderedDict, _dict_representer)
return yaml.dump(data, Dumper=OrderedDumper, **kwargs)
OrderedDumper.add_representer(OrderedDict, _dict_representer)
return yaml.dump(data, Dumper=OrderedDumper, **kwargs)
else:
def ordered_yaml_load(stream):
return yaml.load(stream, Loader=_NoDatesSafeLoader)
def ordered_yaml_dump(data, **kwargs):
return yaml.dump(data, **kwargs)
def load_yaml_or_json(path: Path, ordered_dict: bool = False):
def load_yaml_or_json(path: Path):
"""
Load the content from either a .json or .yaml file, based on the filename
extension.
@ -125,10 +119,7 @@ def load_yaml_or_json(path: Path, ordered_dict: bool = False):
return json.load(fd)
elif path.suffix in (".yml", ".yaml", ".yamlx"):
with path.open("r", encoding="utf-8") as fd:
if ordered_dict:
return ordered_yaml_load(fd)
else:
return yaml.load(fd, Loader=_NoDatesSafeLoader)
return yaml_load(fd)
else:
raise ValueError(f"Unknown file extension {path.suffix}")
@ -399,19 +390,30 @@ def report_validation_errors(all_objects):
return found_error
# Names of metric parameters to pass to constructors.
# This includes only things that the language bindings care about, not things
# that are metadata-only or are resolved into other parameters at parse time.
# **CAUTION**: This list needs to be in the order the Swift type constructors
# expects them. (The other language bindings don't care about the order). The
# `test_order_of_fields` test checks that the generated code is valid.
# **DO NOT CHANGE THE ORDER OR ADD NEW FIELDS IN THE MIDDLE**
extra_metric_args = [
def remove_output_params(d, output_params):
"""
Remove output-only params, such as "defined_in",
in order to validate the output against the input schema.
"""
modified_dict = {}
for key, value in d.items():
if key is not output_params:
modified_dict[key] = value
return modified_dict
# Names of parameters to pass to all metrics constructors constructors.
common_metric_args = [
"category",
"name",
"send_in_pings",
"lifetime",
"disabled",
]
# Names of parameters that only apply to some of the metrics types.
extra_metric_args = [
"time_unit",
"memory_unit",
"allowed_extra_keys",
@ -420,11 +422,21 @@ extra_metric_args = [
"range_max",
"range_min",
"histogram_type",
"numerators",
]
# This includes only things that the language bindings care about, not things
# that are metadata-only or are resolved into other parameters at parse time.
# **CAUTION**: This list needs to be in the order the Swift type constructors
# expects them. (The other language bindings don't care about the order). The
# `test_order_of_fields` test checks that the generated code is valid.
# **DO NOT CHANGE THE ORDER OR ADD NEW FIELDS IN THE MIDDLE**
metric_args = common_metric_args + extra_metric_args
# Names of ping parameters to pass to constructors.
extra_ping_args = [
ping_args = [
"include_client_id",
"send_if_empty",
"name",
@ -433,6 +445,4 @@ extra_ping_args = [
# Names of parameters to pass to both metric and ping constructors (no duplicates).
extra_args = extra_metric_args + [
v for v in extra_ping_args if v not in extra_metric_args
]
extra_args = metric_args + [v for v in ping_args if v not in metric_args]

17
third_party/python/glean_parser/requirements_dev.txt vendored
View File

@ -1,14 +1,15 @@
black==20.8b1
coverage==5.3
coverage==5.4
flake8==3.8.4
flake8-bugbear==20.1.4
flake8-bugbear==20.11.1
m2r==0.2.1
mypy==0.782
mypy==0.812
pip
pytest-runner==5.2
pytest==6.1.1
Sphinx==3.2.1
twine==3.2.0
watchdog==0.10.3
pytest-runner==5.3.0
pytest==6.2.2
recommonmark==0.7.1
Sphinx==3.5.1
twine==3.3.0
watchdog==2.0.1
wheel
yamllint==1.25.0

3
third_party/python/glean_parser/setup.cfg vendored
View File

@ -7,9 +7,6 @@ exclude = docs
[aliases]
test = pytest
[tool:pytest]
collect_ignore = ['setup.py']
[egg_info]
tag_build =
tag_date = 0

5
third_party/python/glean_parser/setup.py vendored
View File

@ -17,10 +17,10 @@ if sys.version_info < (3, 6):
sys.exit(1)
with open("README.rst", encoding="utf-8") as readme_file:
with open("README.md", encoding="utf-8") as readme_file:
readme = readme_file.read()
with open("HISTORY.rst", encoding="utf-8") as history_file:
with open("CHANGELOG.md", encoding="utf-8") as history_file:
history = history_file.read()
requirements = [
@ -61,6 +61,7 @@ setup(
},
install_requires=requirements,
long_description=readme + "\n\n" + history,
long_description_content_type="text/markdown",
include_package_data=True,
keywords="glean_parser",
name="glean_parser",

2
third_party/python/requirements.in vendored
View File

@ -26,7 +26,7 @@ ecdsa==0.15
esprima==4.0.1
fluent.migrate==0.11
fluent.syntax==0.18.1
glean_parser==1.29.0
glean_parser==2.5.0
jsmin==2.1.0
json-e==2.7.0
mozilla-version==0.3.4

6
third_party/python/requirements.txt vendored
View File

@ -97,9 +97,9 @@ fluent.syntax==0.18.1 \
# -r requirements-mach-vendor-python.in
# compare-locales
# fluent.migrate
glean_parser==1.29.0 \
--hash=sha256:7cf1b02ef87fad57bf0f6b9711a98c1fd8f89c9df702245d16c09bf1b042a255 \
--hash=sha256:df7436e164148594176ec55f7d7c3c5c944daca67c3cc30428514628625b214b
glean_parser==2.5.0 \
--hash=sha256:50d53cb23c7b2ecdfed497187c63ea03bff8f78a0bbfd32dbaffb0affa0b6d2d \
--hash=sha256:b2c1b670b73e7df21de335de34e9ece457ef3996110567d7ad6f09fb546e6e5e
# via -r requirements-mach-vendor-python.in
jinja2==2.11.2 \
--hash=sha256:89aab215427ef59c34ad58735269eb58b1a5808103067f7bb9d5836c651b3bb0 \

2
third_party/rust/glean-core/.cargo-checksum.json vendored
View File

File diff suppressed because one or more lines are too long

98
third_party/rust/glean-core/Cargo.lock generated vendored
View File

@ -2,9 +2,9 @@
# It is not intended for manual editing.
[[package]]
name = "adler"
version = "0.2.3"
version = "1.0.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ee2a4ec343196209d6594e19543ae87a39f96d5534d7174822a3ad825dd6ed7e"
checksum = "f26201604c87b1e01bd3d98f8d5d9a8fcbb815e8cedb41ffccbeb4bf593a35fe"
[[package]]
name = "arrayref"
@ -31,9 +31,9 @@ checksum = "cdb031dd78e28731d87d56cc8ffef4a8f36ca26c38fe2de700543e627f8a464a"
[[package]]
name = "bincode"
version = "1.3.1"
version = "1.3.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f30d3a39baa26f9651f17b375061f3233dde33424a8b72b0dbe93a68a0bc896d"
checksum = "d175dfa69e619905c4c3cdb7c3c203fa3bdd5d51184e3afdb2742c0280493772"
dependencies = [
"byteorder",
"serde",
@ -47,15 +47,15 @@ checksum = "cf1de2fe8c75bc145a2f577add951f8134889b4795d47466a54a5c846d691693"
[[package]]
name = "byteorder"
version = "1.4.2"
version = "1.3.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ae44d1a3d5a19df61dd0c8beb138458ac2a53a7ac09eba97d55592540004306b"
checksum = "08c48aae112d48ed9f069b33538ea9e3e90aa263cfa3d1c24309612b1f7472de"
[[package]]
name = "cc"
version = "1.0.66"
version = "1.0.67"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4c0496836a84f8d0495758516b8621a622beb77c0fed418570e50764093ced48"
checksum = "e3c69b077ad434294d3ce9f1f6143a2a4b89a8a2d54ef813d85003a4fd1137fd"
[[package]]
name = "cfg-if"
@ -132,9 +132,9 @@ dependencies = [
[[package]]
name = "form_urlencoded"
version = "1.0.0"
version = "1.0.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ece68d15c92e84fa4f19d3780f1294e5ca82a78a6d515f1efaabcc144688be00"
checksum = "5fc25a87fa4fd2094bffb06925852034d90a17f0d1e05197d4956d3555752191"
dependencies = [
"matches",
"percent-encoding",
@ -153,7 +153,7 @@ dependencies = [
[[package]]
name = "glean-core"
version = "34.1.0"
version = "36.0.0"
dependencies = [
"bincode",
"chrono",
@ -169,6 +169,7 @@ dependencies = [
"serde_json",
"tempfile",
"uuid",
"zeitstempel",
]
[[package]]
@ -197,9 +198,9 @@ checksum = "25a2bc672d1148e28034f176e01fffebb08b35768468cc954630da77a1449005"
[[package]]
name = "idna"
version = "0.2.0"
version = "0.2.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "02e2673c30ee86b5b96a9cb52ad15718aa1f966f5ab9ad54a8b95d5ca33120a9"
checksum = "89829a5d69c23d348314a7ac337fe39173b61149a9864deabd260983aed48c21"
dependencies = [
"matches",
"unicode-bidi",
@ -229,9 +230,9 @@ checksum = "e2abad23fbc42b3700f2f279844dc832adb2b2eb069b2df918f455c4e18cc646"
[[package]]
name = "libc"
version = "0.2.85"
version = "0.2.89"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7ccac4b00700875e6a07c6cde370d44d32fa01c5a65cdd2fca6858c479d28bb3"
checksum = "538c092e5586f4cdd7dd8078c4a79220e3e168880218124dcbce860f0ea938c6"
[[package]]
name = "lmdb-rkv"
@ -279,9 +280,9 @@ checksum = "0ee1c47aaa256ecabcaea351eae4a9b01ef39ed810004e298d2511ed284b1525"
[[package]]
name = "miniz_oxide"
version = "0.4.3"
version = "0.4.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0f2d26ec3309788e423cfbf68ad1800f061638098d76a83681af979dc4eda19d"
checksum = "a92518e98c078586bc6c934028adcca4c92a53d6a958196de835170a01d84e4b"
dependencies = [
"adler",
"autocfg",
@ -318,9 +319,9 @@ dependencies = [
[[package]]
name = "once_cell"
version = "1.5.2"
version = "1.7.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "13bd41f508810a131401606d54ac32a467c97172d74ba7662562ebba5ad07fa0"
checksum = "af8b08b04175473088b46763e51ee54da5f9a164bc162f615b91bc179dbf15a3"
[[package]]
name = "ordered-float"
@ -391,9 +392,9 @@ checksum = "a1d01941d82fa2ab50be1e79e6714289dd7cde78eba4c074bc5a4374f650dfe0"
[[package]]
name = "quote"
version = "1.0.8"
version = "1.0.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "991431c3519a3f36861882da93630ce66b52918dcf1b8e2fd66b397fc96f28df"
checksum = "c3d0b9745dc2debf507c8422de05d7226cc1f0644216dfdfead988f9b1ab32a7"
dependencies = [
"proc-macro2",
]
@ -422,9 +423,9 @@ dependencies = [
[[package]]
name = "rand_core"
version = "0.6.1"
version = "0.6.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c026d7df8b298d90ccbbc5190bd04d85e159eaf5576caeacf8741da93ccbd2e5"
checksum = "34cf66eb183df1c5876e2dcf6b13d57340741e8dc255b48e40a26de954d06ae7"
dependencies = [
"getrandom",
]
@ -440,9 +441,9 @@ dependencies = [
[[package]]
name = "redox_syscall"
version = "0.2.4"
version = "0.2.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "05ec8ca9416c5ea37062b502703cd7fcb207736bc294f6e0cf367ac6fc234570"
checksum = "94341e4e44e24f6b591b59e47a8a027df12e008d73fd5672dbea9cc22f4507d9"
dependencies = [
"bitflags",
]
@ -487,18 +488,18 @@ checksum = "71d301d4193d031abdd79ff7e3dd721168a9572ef3fe51a1517aba235bd8f86e"
[[package]]
name = "serde"
version = "1.0.123"
version = "1.0.124"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "92d5161132722baa40d802cc70b15262b98258453e85e5d1d365c757c73869ae"
checksum = "bd761ff957cb2a45fbb9ab3da6512de9de55872866160b23c25f1a841e99d29f"
dependencies = [
"serde_derive",
]
[[package]]
name = "serde_derive"
version = "1.0.123"
version = "1.0.124"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9391c295d64fc0abb2c556bad848f33cb8296276b1ad2677d1ae1ace4f258f31"
checksum = "1800f7693e94e186f5e25a28291ae1570da908aff7d97a095dec1e56ff99069b"
dependencies = [
"proc-macro2",
"quote",
@ -507,9 +508,9 @@ dependencies = [
[[package]]
name = "serde_json"
version = "1.0.61"
version = "1.0.64"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4fceb2595057b6891a4ee808f70054bd2d12f0e97f1cbb78689b59f676df325a"
checksum = "799e97dc9fdae36a5c8b8f2cae9ce2ee9fdce2058c57a93e6099d919fd982f79"
dependencies = [
"itoa",
"ryu",
@ -518,9 +519,9 @@ dependencies = [
[[package]]
name = "syn"
version = "1.0.60"
version = "1.0.64"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c700597eca8a5a762beb35753ef6b94df201c81cca676604f547495a0d7f0081"
checksum = "3fd9d1e9976102a03c542daa2eff1b43f9d72306342f3f8b3ed5fb8908195d6f"
dependencies = [
"proc-macro2",
"quote",
@ -552,18 +553,18 @@ dependencies = [
[[package]]
name = "thiserror"
version = "1.0.23"
version = "1.0.24"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "76cc616c6abf8c8928e2fdcc0dbfab37175edd8fb49a4641066ad1364fdab146"
checksum = "e0f4a65597094d4483ddaed134f409b2cb7c1beccf25201a9f73c719254fa98e"
dependencies = [
"thiserror-impl",
]
[[package]]
name = "thiserror-impl"
version = "1.0.23"
version = "1.0.24"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9be73a2caec27583d0046ef3796c3794f868a5bc813db689eed00c7631275cd1"
checksum = "7765189610d8241a44529806d6fd1f2e0a08734313a35d5b3a556f92b381f3c0"
dependencies = [
"proc-macro2",
"quote",
@ -606,9 +607,9 @@ dependencies = [
[[package]]
name = "unicode-normalization"
version = "0.1.16"
version = "0.1.17"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a13e63ab62dbe32aeee58d1c5408d35c36c392bba5d9d3142287219721afe606"
checksum = "07fbfce1c8a97d547e8b5334978438d9d6ec8c20e38f56d4a4374d181493eaef"
dependencies = [
"tinyvec",
]
@ -621,9 +622,9 @@ checksum = "f7fe0bb3479651439c9112f72b6c505038574c9fbb575ed1bf3b797fa39dd564"
[[package]]
name = "url"
version = "2.2.0"
version = "2.2.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5909f2b0817350449ed73e8bcd81c8c3c8d9a7a5d8acba4b27db277f1868976e"
checksum = "9ccd964113622c8e9322cfac19eb1004a07e636c545f325da085d5cdde6f1f8b"
dependencies = [
"form_urlencoded",
"idna",
@ -642,9 +643,9 @@ dependencies = [
[[package]]
name = "version_check"
version = "0.9.2"
version = "0.9.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b5a972e5669d67ba988ce3dc826706fb0a8b01471c088cb0b6110b805cc36aed"
checksum = "5fecdca9a5291cc2b8dcf7dc02453fee791a280f3743cb0905f8822ae463b3fe"
[[package]]
name = "wasi"
@ -682,3 +683,14 @@ name = "winapi-x86_64-pc-windows-gnu"
version = "0.4.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "712e227841d057c1ee1cd2fb22fa7e5a5461ae8e48fa2ca79ec42cfc1931183f"
[[package]]
name = "zeitstempel"
version = "0.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f2837f9ad7a7a8c88d1cc50cb0c3d202ce6e178aa6ebf3e49b29561896a61f1d"
dependencies = [
"cfg-if",
"libc",
"once_cell",
]

7
third_party/rust/glean-core/Cargo.toml vendored
View File

@ -13,7 +13,7 @@
[package]
edition = "2018"
name = "glean-core"
version = "34.1.0"
version = "36.0.0"
authors = ["Jan-Erik Rediger <jrediger@mozilla.com>", "The Glean Team <glean-team@mozilla.com>"]
include = ["/README.md", "/LICENSE", "/src", "/examples", "/tests", "/Cargo.toml"]
description = "A modern Telemetry library"
@ -22,7 +22,7 @@ keywords = ["telemetry"]
license = "MPL-2.0"
repository = "https://github.com/mozilla/glean"
[package.metadata.glean]
glean-parser = "1.29.0"
glean-parser = "2.5.0"
[dependencies.bincode]
version = "1.2.1"
@ -56,6 +56,9 @@ version = "1.0.44"
[dependencies.uuid]
version = "0.8.1"
features = ["v4"]
[dependencies.zeitstempel]
version = "0.1.0"
[dev-dependencies.ctor]
version = "0.1.12"

47
third_party/rust/glean-core/src/coverage.rs vendored Normal file
View File

@ -0,0 +1,47 @@
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this
// file, You can obtain one at https://mozilla.org/MPL/2.0/.
//! Utilities for recording when testing APIs have been called on specific
//! metrics.
//!
//! Testing coverage is enabled by setting the GLEAN_TEST_COVERAGE environment
//! variable to the name of an output file. This output file must run through a
//! post-processor (in glean_parser's `coverage` command) to convert to a format
//! understood by third-party coverage reporting tools.
//!
//! While running a unit test suite, Glean records which database keys were
//! accessed by the testing APIs, with one entry per line. Database keys are
//! usually, but not always, the same as metric identifiers, but it is the
//! responsibility of the post-processor to resolve that difference.
//!
//! This functionality has no runtime overhead unless the testing API is used.
use std::env;
use std::fs::{File, OpenOptions};
use std::io::Write;
use std::sync::Mutex;
use once_cell::sync::Lazy;
static COVERAGE_FILE: Lazy<Option<Mutex<File>>> = Lazy::new(|| {
if let Some(filename) = env::var_os("GLEAN_TEST_COVERAGE") {
match OpenOptions::new().append(true).create(true).open(filename) {
Ok(file) => {
return Some(Mutex::new(file));
}
Err(err) => {
log::error!("Couldn't open file for coverage results: {:?}", err);
}
}
}
None
});
pub(crate) fn record_coverage(metric_id: &str) {
if let Some(file_mutex) = &*COVERAGE_FILE {
let mut file = file_mutex.lock().unwrap();
writeln!(&mut file, "{}", metric_id).ok();
file.flush().ok();
}
}

5
third_party/rust/glean-core/src/event_database/mod.rs vendored
View File

@ -15,6 +15,7 @@ use std::sync::RwLock;
use serde::{Deserialize, Serialize};
use serde_json::{json, Value as JsonValue};
use crate::coverage::record_coverage;
use crate::CommonMetricData;
use crate::Glean;
use crate::Result;
@ -326,6 +327,8 @@ impl EventDatabase {
///
/// This doesn't clear the stored value.
pub fn test_has_value<'a>(&'a self, meta: &'a CommonMetricData, store_name: &str) -> bool {
record_coverage(&meta.base_identifier());
self.event_stores
.read()
.unwrap() // safe unwrap, only error case is poisoning
@ -346,6 +349,8 @@ impl EventDatabase {
meta: &'a CommonMetricData,
store_name: &str,
) -> Option<Vec<RecordedEvent>> {
record_coverage(&meta.base_identifier());
let value: Vec<RecordedEvent> = self
.event_stores
.read()

13
third_party/rust/glean-core/src/internal_metrics.rs vendored
View File

@ -17,6 +17,7 @@ pub struct CoreMetrics {
/// **Note**: Not a _core_ metric, but an error metric,
/// placed here for the lack of a more suitable part in the Glean struct.
pub io_errors: CounterMetric,
pub pings_submitted: LabeledMetric<CounterMetric>,
}
impl CoreMetrics {
@ -72,6 +73,18 @@ impl CoreMetrics {
disabled: false,
dynamic_label: None,
}),
pings_submitted: LabeledMetric::new(
CounterMetric::new(CommonMetricData {
name: "pings_submitted".into(),
category: "glean.validation".into(),
send_in_pings: vec!["metrics".into(), "baseline".into()],
lifetime: Lifetime::Ping,
disabled: false,
dynamic_label: None,
}),
None,
),
}
}
}

17
third_party/rust/glean-core/src/lib.rs vendored
View File

@ -27,6 +27,7 @@ use uuid::Uuid;
mod macros;
mod common_metric_data;
mod coverage;
mod database;
mod debug;
mod error;
@ -627,6 +628,15 @@ impl Glean {
Ok(false)
}
Some(content) => {
// This metric is recorded *after* the ping is collected (since
// that is the only way to know *if* it will be submitted). The
// implication of this is that the count for a metrics ping will
// be included in the *next* metrics ping.
self.core_metrics
.pings_submitted
.get(&ping.name)
.add(&self, 1);
if let Err(e) = ping_maker.store_ping(
self,
&doc_id,
@ -961,8 +971,13 @@ impl Glean {
}
}
/// Returns a timestamp corresponding to "now" with millisecond precision.
pub fn get_timestamp_ms() -> u64 {
const NANOS_PER_MILLI: u64 = 1_000_000;
zeitstempel::now() / NANOS_PER_MILLI
}
// Split unit tests to a separate file, to reduce the file of this one.
#[cfg(test)]
#[cfg(test)]
#[path = "lib_unit_tests.rs"]
mod tests;

2
third_party/rust/glean-core/src/lib_unit_tests.rs vendored
View File

@ -411,6 +411,7 @@ fn correct_order() {
TimingDistribution(Histogram::functional(2.0, 8.0)),
MemoryDistribution(Histogram::functional(2.0, 8.0)),
Jwe("eyJhbGciOiJSU0EtT0FFUCIsImVuYyI6IkEyNTZHQ00ifQ.OKOawDo13gRp2ojaHV7LFpZcgV7T6DVZKTyKOMTYUmKoTCVJRgckCL9kiMT03JGeipsEdY3mx_etLbbWSrFr05kLzcSr4qKAq7YN7e9jwQRb23nfa6c9d-StnImGyFDbSv04uVuxIp5Zms1gNxKKK2Da14B8S4rzVRltdYwam_lDp5XnZAYpQdb76FdIKLaVmqgfwX7XWRxv2322i-vDxRfqNzo_tETKzpVLzfiwQyeyPGLBIO56YJ7eObdv0je81860ppamavo35UgoRdbYaBcoh9QcfylQr66oc6vFWXRcZ_ZT2LawVCWTIy3brGPi6UklfCpIMfIjf7iGdXKHzg.48V1_ALb6US04U3b.5eym8TW_c8SuK0ltJ3rpYIzOeDQz7TALvtu6UG9oMo4vpzs9tX_EFShS8iB7j6jiSdiwkIr3ajwQzaBtQD_A.XFBoMYUZodetZdvTiFvSkQ".into()),
Rate(0, 0),
];
for metric in all_metrics {
@ -436,6 +437,7 @@ fn correct_order() {
TimingDistribution(..) => assert_eq!(11, disc),
MemoryDistribution(..) => assert_eq!(12, disc),
Jwe(..) => assert_eq!(13, disc),
Rate(..) => assert_eq!(14, disc),
}
}
}

2
third_party/rust/glean-core/src/metrics/boolean.rs vendored
View File

@ -57,7 +57,7 @@ impl BooleanMetric {
///
/// This doesn't clear the stored value.
pub fn test_get_value(&self, glean: &Glean, storage_name: &str) -> Option<bool> {
match StorageManager.snapshot_metric(
match StorageManager.snapshot_metric_for_test(
glean.storage(),
storage_name,
&self.meta.identifier(glean),

2
third_party/rust/glean-core/src/metrics/counter.rs vendored
View File

@ -80,7 +80,7 @@ impl CounterMetric {
///
/// This doesn't clear the stored value.
pub fn test_get_value(&self, glean: &Glean, storage_name: &str) -> Option<i32> {
match StorageManager.snapshot_metric(
match StorageManager.snapshot_metric_for_test(
glean.storage(),
storage_name,
&self.meta.identifier(glean),

2
third_party/rust/glean-core/src/metrics/custom_distribution.rs vendored
View File

@ -158,7 +158,7 @@ impl CustomDistributionMetric {
///
/// This doesn't clear the stored value.
pub fn test_get_value(&self, glean: &Glean, storage_name: &str) -> Option<DistributionData> {
match StorageManager.snapshot_metric(
match StorageManager.snapshot_metric_for_test(
glean.storage(),
storage_name,
&self.meta.identifier(glean),

4
third_party/rust/glean-core/src/metrics/datetime.rs vendored
View File

@ -159,7 +159,7 @@ impl DatetimeMetric {
///
/// The stored value or `None` if nothing stored.
pub fn test_get_value(&self, glean: &Glean, storage_name: &str) -> Option<Datetime> {
match StorageManager.snapshot_metric(
match StorageManager.snapshot_metric_for_test(
glean.storage(),
storage_name,
&self.meta.identifier(glean),
@ -211,7 +211,7 @@ impl DatetimeMetric {
///
/// This doesn't clear the stored value.
pub fn test_get_value_as_string(&self, glean: &Glean, storage_name: &str) -> Option<String> {
match StorageManager.snapshot_metric(
match StorageManager.snapshot_metric_for_test(
glean.storage(),
storage_name,
&self.meta.identifier(glean),

99
third_party/rust/glean-core/src/metrics/denominator.rs vendored Normal file
View File

@ -0,0 +1,99 @@
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this
// file, You can obtain one at https://mozilla.org/MPL/2.0/.
use crate::error_recording::{record_error, ErrorType};
use crate::metrics::CounterMetric;
use crate::metrics::Metric;
use crate::metrics::MetricType;
use crate::metrics::RateMetric;
use crate::storage::StorageManager;
use crate::CommonMetricData;
use crate::Glean;
/// A Denominator metric (a kind of count shared among Rate metrics).
///
/// Used to count things.
/// The value can only be incremented, not decremented.
#[derive(Clone, Debug)]
pub struct DenominatorMetric {
counter: CounterMetric,
numerators: Vec<RateMetric>,
}
impl MetricType for DenominatorMetric {
fn meta(&self) -> &CommonMetricData {
self.counter.meta()
}
fn meta_mut(&mut self) -> &mut CommonMetricData {
self.counter.meta_mut()
}
}
impl DenominatorMetric {
/// Creates a new denominator metric.
pub fn new(meta: CommonMetricData, numerators: Vec<CommonMetricData>) -> Self {
Self {
counter: CounterMetric::new(meta),
numerators: numerators.into_iter().map(RateMetric::new).collect(),
}
}
/// Increases the denominator by `amount`.
///
/// # Arguments
///
/// * `glean` - The Glean instance this metric belongs to.
/// * `amount` - The amount to increase by. Should be positive.
///
/// ## Notes
///
/// Logs an error if the `amount` is 0 or negative.
pub fn add(&self, glean: &Glean, amount: i32) {
if !self.should_record(glean) {
return;
}
if amount <= 0 {
record_error(
glean,
self.meta(),
ErrorType::InvalidValue,
format!("Added negative or zero value {}", amount),
None,
);
return;
}
for num in &self.numerators {
num.add_to_denominator(glean, amount);
}
glean
.storage()
.record_with(glean, self.counter.meta(), |old_value| match old_value {
Some(Metric::Counter(old_value)) => {
Metric::Counter(old_value.saturating_add(amount))
}
_ => Metric::Counter(amount),
})
}
/// **Test-only API (exported for FFI purposes).**
///
/// Gets the currently stored value as an integer.
///
/// This doesn't clear the stored value.
pub fn test_get_value(&self, glean: &Glean, storage_name: &str) -> Option<i32> {
match StorageManager.snapshot_metric_for_test(
glean.storage(),
storage_name,
&self.meta().identifier(glean),
self.meta().lifetime,
) {
Some(Metric::Counter(i)) => Some(i),
_ => None,
}
}
}

2
third_party/rust/glean-core/src/metrics/experiment.rs vendored
View File

@ -221,7 +221,7 @@ impl ExperimentMetric {
///
/// This doesn't clear the stored value.
pub fn test_get_value_as_json_string(&self, glean: &Glean) -> Option<String> {
match StorageManager.snapshot_metric(
match StorageManager.snapshot_metric_for_test(
glean.storage(),
INTERNAL_STORAGE,
&self.meta.identifier(glean),

2
third_party/rust/glean-core/src/metrics/jwe.rs vendored
View File

@ -290,7 +290,7 @@ impl JweMetric {
///
/// This doesn't clear the stored value.
pub fn test_get_value(&self, glean: &Glean, storage_name: &str) -> Option<String> {
match StorageManager.snapshot_metric(
match StorageManager.snapshot_metric_for_test(
glean.storage(),
storage_name,
&self.meta.identifier(glean),

2
third_party/rust/glean-core/src/metrics/memory_distribution.rs vendored
View File

@ -186,7 +186,7 @@ impl MemoryDistributionMetric {
///
/// This doesn't clear the stored value.
pub fn test_get_value(&self, glean: &Glean, storage_name: &str) -> Option<DistributionData> {
match StorageManager.snapshot_metric(
match StorageManager.snapshot_metric_for_test(
glean.storage(),
storage_name,
&self.meta.identifier(glean),

10
third_party/rust/glean-core/src/metrics/mod.rs vendored
View File

@ -14,6 +14,7 @@ mod boolean;
mod counter;
mod custom_distribution;
mod datetime;
mod denominator;
mod event;
mod experiment;
mod jwe;
@ -22,6 +23,7 @@ mod memory_distribution;
mod memory_unit;
mod ping;
mod quantity;
mod rate;
mod string;
mod string_list;
mod time_unit;
@ -40,6 +42,7 @@ pub use self::boolean::BooleanMetric;
pub use self::counter::CounterMetric;
pub use self::custom_distribution::CustomDistributionMetric;
pub use self::datetime::DatetimeMetric;
pub use self::denominator::DenominatorMetric;
pub use self::event::EventMetric;
pub(crate) use self::experiment::ExperimentMetric;
pub use crate::histogram::HistogramType;
@ -55,6 +58,7 @@ pub use self::memory_distribution::MemoryDistributionMetric;
pub use self::memory_unit::MemoryUnit;
pub use self::ping::PingType;
pub use self::quantity::QuantityMetric;
pub use self::rate::RateMetric;
pub use self::string::StringMetric;
pub use self::string_list::StringListMetric;
pub use self::time_unit::TimeUnit;
@ -117,6 +121,8 @@ pub enum Metric {
MemoryDistribution(Histogram<Functional>),
/// A JWE metric. See [`JweMetric`] for more information.
Jwe(String),
/// A rate metric. See [`RateMetric`] for more information.
Rate(i32, i32),
}
/// A [`MetricType`] describes common behavior across all metrics.
@ -151,6 +157,7 @@ impl Metric {
Metric::Datetime(_, _) => "datetime",
Metric::Experiment(_) => panic!("Experiments should not be serialized through this"),
Metric::Quantity(_) => "quantity",
Metric::Rate(..) => "rate",
Metric::String(_) => "string",
Metric::StringList(_) => "string_list",
Metric::Timespan(..) => "timespan",
@ -173,6 +180,9 @@ impl Metric {
Metric::Datetime(d, time_unit) => json!(get_iso_time_string(*d, *time_unit)),
Metric::Experiment(e) => e.as_json(),
Metric::Quantity(q) => json!(q),
Metric::Rate(num, den) => {
json!({"numerator": num, "denominator": den})
}
Metric::String(s) => json!(s),
Metric::StringList(v) => json!(v),
Metric::Timespan(time, time_unit) => {

2
third_party/rust/glean-core/src/metrics/quantity.rs vendored
View File

@ -74,7 +74,7 @@ impl QuantityMetric {
///
/// This doesn't clear the stored value.
pub fn test_get_value(&self, glean: &Glean, storage_name: &str) -> Option<i64> {
match StorageManager.snapshot_metric(
match StorageManager.snapshot_metric_for_test(
glean.storage(),
storage_name,
&self.meta.identifier(glean),

128
third_party/rust/glean-core/src/metrics/rate.rs vendored Normal file
View File

@ -0,0 +1,128 @@
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this
// file, You can obtain one at https://mozilla.org/MPL/2.0/.
use crate::error_recording::{record_error, ErrorType};
use crate::metrics::Metric;
use crate::metrics::MetricType;
use crate::storage::StorageManager;
use crate::CommonMetricData;
use crate::Glean;
/// A rate metric.
///
/// Used to determine the proportion of things via two counts:
/// * A numerator defining the amount of times something happened,
/// * A denominator counting the amount of times someting could have happened.
///
/// Both numerator and denominator can only be incremented, not decremented.
#[derive(Clone, Debug)]
pub struct RateMetric {
meta: CommonMetricData,
}
impl MetricType for RateMetric {
fn meta(&self) -> &CommonMetricData {
&self.meta
}
fn meta_mut(&mut self) -> &mut CommonMetricData {
&mut self.meta
}
}
// IMPORTANT:
//
// When changing this implementation, make sure all the operations are
// also declared in the related trait in `../traits/`.
impl RateMetric {
/// Creates a new rate metric.
pub fn new(meta: CommonMetricData) -> Self {
Self { meta }
}
/// Increases the numerator by `amount`.
///
/// # Arguments
///
/// * `glean` - The Glean instance this metric belongs to.
/// * `amount` - The amount to increase by. Should be non-negative.
///
/// ## Notes
///
/// Logs an error if the `amount` is negative.
pub fn add_to_numerator(&self, glean: &Glean, amount: i32) {
if !self.should_record(glean) {
return;
}
if amount < 0 {
record_error(
glean,
&self.meta,
ErrorType::InvalidValue,
format!("Added negative value {} to numerator", amount),
None,
);
return;
}
glean
.storage()
.record_with(glean, &self.meta, |old_value| match old_value {
Some(Metric::Rate(num, den)) => Metric::Rate(num.saturating_add(amount), den),
_ => Metric::Rate(amount, 0), // Denominator will show up eventually. Probably.
});
}
/// Increases the denominator by `amount`.
///
/// # Arguments
///
/// * `glean` - The Glean instance this metric belongs to.
/// * `amount` - The amount to increase by. Should be non-negative.
///
/// ## Notes
///
/// Logs an error if the `amount` is negative.
pub fn add_to_denominator(&self, glean: &Glean, amount: i32) {
if !self.should_record(glean) {
return;
}
if amount < 0 {
record_error(
glean,
&self.meta,
ErrorType::InvalidValue,
format!("Added negative value {} to denominator", amount),
None,
);
return;
}
glean
.storage()
.record_with(glean, &self.meta, |old_value| match old_value {
Some(Metric::Rate(num, den)) => Metric::Rate(num, den.saturating_add(amount)),
_ => Metric::Rate(0, amount),
});
}
/// **Test-only API (exported for FFI purposes).**
///
/// Gets the currently stored value as a pair of integers.
///
/// This doesn't clear the stored value.
pub fn test_get_value(&self, glean: &Glean, storage_name: &str) -> Option<(i32, i32)> {
match StorageManager.snapshot_metric_for_test(
glean.storage(),
storage_name,
&self.meta.identifier(glean),
self.meta.lifetime,
) {
Some(Metric::Rate(n, d)) => Some((n, d)),
_ => None,
}
}
}

2
third_party/rust/glean-core/src/metrics/string.rs vendored
View File

@ -67,7 +67,7 @@ impl StringMetric {
///
/// This doesn't clear the stored value.
pub fn test_get_value(&self, glean: &Glean, storage_name: &str) -> Option<String> {
match StorageManager.snapshot_metric(
match StorageManager.snapshot_metric_for_test(
glean.storage(),
storage_name,
&self.meta.identifier(glean),

2
third_party/rust/glean-core/src/metrics/string_list.rs vendored
View File

@ -133,7 +133,7 @@ impl StringListMetric {
///
/// This doesn't clear the stored value.
pub fn test_get_value(&self, glean: &Glean, storage_name: &str) -> Option<Vec<String>> {
match StorageManager.snapshot_metric(
match StorageManager.snapshot_metric_for_test(
glean.storage(),
storage_name,
&self.meta.identifier(glean),

2
third_party/rust/glean-core/src/metrics/timespan.rs vendored
View File

@ -179,7 +179,7 @@ impl TimespanMetric {
///
/// This doesn't clear the stored value.
pub fn test_get_value(&self, glean: &Glean, storage_name: &str) -> Option<u64> {
match StorageManager.snapshot_metric(
match StorageManager.snapshot_metric_for_test(
glean.storage(),
storage_name,
&self.meta.identifier(glean),

3
third_party/rust/glean-core/src/metrics/timing_distribution.rs vendored
View File

@ -47,6 +47,7 @@ impl Timings {
}
/// Start a new timer and set it to the `start_time`.
/// Multiple timers can run simultaneously.
///
/// Returns a new [`TimerId`] identifying the timer.
fn set_start(&mut self, start_time: u64) -> TimerId {
@ -319,7 +320,7 @@ impl TimingDistributionMetric {
///
/// This doesn't clear the stored value.
pub fn test_get_value(&self, glean: &Glean, storage_name: &str) -> Option<DistributionData> {
match StorageManager.snapshot_metric(
match StorageManager.snapshot_metric_for_test(
glean.storage(),
storage_name,
&self.meta.identifier(glean),

28
third_party/rust/glean-core/src/storage/mod.rs vendored
View File

@ -10,6 +10,7 @@ use std::collections::HashMap;
use serde_json::{json, Value as JsonValue};
use crate::coverage::record_coverage;
use crate::database::Database;
use crate::metrics::Metric;
use crate::Lifetime;
@ -113,8 +114,6 @@ impl StorageManager {
/// Gets the current value of a single metric identified by name.
///
/// This look for a value in stores for all lifetimes.
///
/// # Arguments
///
/// * `storage` - The database to get data from.
@ -145,6 +144,31 @@ impl StorageManager {
snapshot
}
/// Gets the current value of a single metric identified by name.
///
/// Use this API, rather than `snapshot_metric` within the testing API, so
/// that the usage will be reported in coverage, if enabled.
///
/// # Arguments
///
/// * `storage` - The database to get data from.
/// * `store_name` - The store name to look into.
/// * `metric_id` - The full metric identifier.
///
/// # Returns
///
/// The decoded metric or `None` if no data is found.
pub fn snapshot_metric_for_test(
&self,
storage: &Database,
store_name: &str,
metric_id: &str,
metric_lifetime: Lifetime,
) -> Option<Metric> {
record_coverage(metric_id);
self.snapshot_metric(storage, store_name, metric_id, metric_lifetime)
}
/// Snapshots the experiments.
///
/// # Arguments

4
third_party/rust/glean-core/src/traits/mod.rs vendored
View File

@ -15,8 +15,10 @@ mod event;
mod jwe;
mod labeled;
mod memory_distribution;
mod numerator;
mod ping;
mod quantity;
mod rate;
mod string;
mod string_list;
mod timespan;
@ -34,8 +36,10 @@ pub use self::event::NoExtraKeys;
pub use self::jwe::Jwe;
pub use self::labeled::Labeled;
pub use self::memory_distribution::MemoryDistribution;
pub use self::numerator::Numerator;
pub use self::ping::Ping;
pub use self::quantity::Quantity;
pub use self::rate::Rate;
pub use self::string::String;
pub use self::string_list::StringList;
pub use self::timespan::Timespan;

52
third_party/rust/glean-core/src/traits/numerator.rs vendored Normal file
View File

@ -0,0 +1,52 @@
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this
// file, You can obtain one at https://mozilla.org/MPL/2.0/.
use crate::ErrorType;
// When changing this trait, ensure all operations are implemented in the
// related type in `../metrics`. (Except test_get_num_errors)
/// A description for the `NumeratorMetric` subtype of the [`RateMetric`](crate::metrics::RateMetric) type.
pub trait Numerator {
/// Increases the numerator by `amount`.
///
/// # Arguments
///
/// * `amount` - The amount to increase by. Should be non-negative.
///
/// ## Notes
///
/// Logs an error if the `amount` is negative.
fn add_to_numerator(&self, amount: i32);
/// **Exported for test purposes.**
///
/// Gets the currently stored value as a pair of integers.
///
/// # Arguments
///
/// * `ping_name` - the optional name of the ping to retrieve the metric
/// for. Defaults to the first value in `send_in_pings`.
///
/// This doesn't clear the stored value.
fn test_get_value<'a, S: Into<Option<&'a str>>>(&self, ping_name: S) -> Option<(i32, i32)>;
/// **Exported for test purposes.**
///
/// Gets the number of recorded errors for the given metric and error type.
///
/// # Arguments
///
/// * `error` - The type of error
/// * `ping_name` - the optional name of the ping to retrieve the metric
/// for. Defaults to the first value in `send_in_pings`.
///
/// # Returns
///
/// The number of errors reported.
fn test_get_num_recorded_errors<'a, S: Into<Option<&'a str>>>(
&self,
error: ErrorType,
ping_name: S,
) -> i32;
}

63
third_party/rust/glean-core/src/traits/rate.rs vendored Normal file
View File

@ -0,0 +1,63 @@
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this
// file, You can obtain one at https://mozilla.org/MPL/2.0/.
use crate::ErrorType;
// When changing this trait, ensure all operations are implemented in the
// related type in `../metrics`. (Except test_get_num_errors)
/// A description for the [`RateMetric`](crate::metrics::RateMetric) type.
pub trait Rate {
/// Increases the numerator by `amount`.
///
/// # Arguments
///
/// * `amount` - The amount to increase by. Should be non-negative.
///
/// ## Notes
///
/// Logs an error if the `amount` is negative.
fn add_to_numerator(&self, amount: i32);
/// Increases the denominator by `amount`.
///
/// # Arguments
///
/// * `amount` - The amount to increase by. Should be non-negative.
///
/// ## Notes
///
/// Logs an error if the `amount` is negative.
fn add_to_denominator(&self, amount: i32);
/// **Exported for test purposes.**
///
/// Gets the currently stored value as a pair of integers.
///
/// # Arguments
///
/// * `ping_name` - the optional name of the ping to retrieve the metric
/// for. Defaults to the first value in `send_in_pings`.
///
/// This doesn't clear the stored value.
fn test_get_value<'a, S: Into<Option<&'a str>>>(&self, ping_name: S) -> Option<(i32, i32)>;
/// **Exported for test purposes.**
///
/// Gets the number of recorded errors for the given metric and error type.
///
/// # Arguments
///
/// * `error` - The type of error
/// * `ping_name` - the optional name of the ping to retrieve the metric
/// for. Defaults to the first value in `send_in_pings`.
///
/// # Returns
///
/// The number of errors reported.
fn test_get_num_recorded_errors<'a, S: Into<Option<&'a str>>>(
&self,
error: ErrorType,
ping_name: S,
) -> i32;
}

8
third_party/rust/glean-core/src/traits/timing_distribution.rs vendored
View File

@ -11,12 +11,8 @@ use crate::ErrorType;
/// When changing this trait, make sure all the operations are
/// implemented in the related type in `../metrics/`.
pub trait TimingDistribution {
/// Starts tracking time for the provided metric.
///
/// This records an error if it’s already tracking time (i.e.
/// [`start`](TimingDistribution::start) was already called with no corresponding
/// [`stop_and_accumulate`](TimingDistribution::stop_and_accumulate)): in that case the
/// original start time will be preserved.
/// Start tracking time for the provided metric.
/// Multiple timers can run simultaneously.
///
/// # Returns
///

100
third_party/rust/glean-core/tests/ping.rs vendored
View File

@ -7,6 +7,7 @@ use crate::common::*;
use glean_core::metrics::*;
use glean_core::CommonMetricData;
use glean_core::Lifetime;
#[test]
fn write_ping_to_disk() {
@ -101,3 +102,102 @@ fn empty_pings_with_flag_are_sent() {
assert_eq!(false, ping2.submit(&glean, None).unwrap());
assert_eq!(1, get_queued_pings(glean.get_data_path()).unwrap().len());
}
#[test]
fn test_pings_submitted_metric() {
let (mut glean, _temp) = new_glean(None);
// Reconstructed here so we can test it without reaching into the library
// internals.
let pings_submitted = LabeledMetric::new(
CounterMetric::new(CommonMetricData {
name: "pings_submitted".into(),
category: "glean.validation".into(),
send_in_pings: vec!["metrics".into(), "baseline".into()],
lifetime: Lifetime::Ping,
disabled: false,
dynamic_label: None,
}),
None,
);
let metrics_ping = PingType::new("metrics", true, false, vec![]);
glean.register_ping_type(&metrics_ping);
let baseline_ping = PingType::new("baseline", true, false, vec![]);
glean.register_ping_type(&baseline_ping);
// We need to store a metric as an empty ping is not stored.
let counter = CounterMetric::new(CommonMetricData {
name: "counter".into(),
category: "local".into(),
send_in_pings: vec!["metrics".into()],
..Default::default()
});
counter.add(&glean, 1);
assert!(metrics_ping.submit(&glean, None).unwrap());
// Check recording in the metrics ping
assert_eq!(
Some(1),
pings_submitted
.get("metrics")
.test_get_value(&glean, "metrics")
);
assert_eq!(
None,
pings_submitted
.get("baseline")
.test_get_value(&glean, "metrics")
);
// Check recording in the baseline ping
assert_eq!(
Some(1),
pings_submitted
.get("metrics")
.test_get_value(&glean, "baseline")
);
assert_eq!(
None,
pings_submitted
.get("baseline")
.test_get_value(&glean, "baseline")
);
// Trigger 2 baseline pings.
// This should record a count of 2 baseline pings in the metrics ping, but
// it resets each time on the baseline ping, so we should only ever get 1
// baseline ping recorded in the baseline ping itsef.
assert!(baseline_ping.submit(&glean, None).unwrap());
assert!(baseline_ping.submit(&glean, None).unwrap());
// Check recording in the metrics ping
assert_eq!(
Some(1),
pings_submitted
.get("metrics")
.test_get_value(&glean, "metrics")
);
assert_eq!(
Some(2),
pings_submitted
.get("baseline")
.test_get_value(&glean, "metrics")
);
// Check recording in the baseline ping
assert_eq!(
None,
pings_submitted
.get("metrics")
.test_get_value(&glean, "baseline")
);
assert_eq!(
Some(1),
pings_submitted
.get("baseline")
.test_get_value(&glean, "baseline")
);
}

6
third_party/rust/glean-core/tests/uuid.rs vendored
View File

@ -26,14 +26,16 @@ fn uuid_is_generated_and_stored() {
let snapshot = glean.snapshot("core", false);
assert!(
snapshot.contains(r#""local.uuid": ""#),
format!("Snapshot 1: {}", snapshot)
"Snapshot 1: {}",
snapshot
);
uuid.generate_and_set(&glean);
let snapshot = glean.snapshot("core", false);
assert!(
snapshot.contains(r#""local.uuid": ""#),
format!("Snapshot 2: {}", snapshot)
"Snapshot 2: {}",
snapshot
);
}

2
third_party/rust/glean/.cargo-checksum.json vendored
View File

@ -1 +1 @@
{"files":{"Cargo.toml":"132e74f42504b92b8896f5a3aa574a3c7fc84bc28f0a5bc3e930bc747d51a73f","LICENSE":"1f256ecad192880510e84ad60474eab7589218784b9a50bc7ceee34c2b91f1d5","README.md":"fd9e0ca6907917ea6bec5de05e15dd21d20fae1cb7f3250467bb20231a8e1065","src/common_test.rs":"a7c7bfb1215b784ed41a4aebac476b7aeb23631ea5646e642b5673e9067ecd95","src/configuration.rs":"b8747397761a9cf6dc64150855b75fd8e48dfe9951ce69e25d646b3a6f46456f","src/core_metrics.rs":"0ecf9ec7de9032f45e6c0feaebf17c614f9be88c1a28043e397eaf7d3b18ba37","src/dispatcher/global.rs":"427d23168ea2457f0332e6f4d6684810fe35a5d5df1e8336d070159d82d936e5","src/dispatcher/mod.rs":"9bf42571a23491db1c6643064caee463301dccac13430883ac0bdaefacf4ace3","src/glean_metrics.rs":"151b6e5acc12436c33c75e2e0a18f9769179d3c7fdc60a22fa02afb76feaf00f","src/lib.rs":"918dbd98cf3d7681954e9c77b61d1a6866837818ff7791a6dc30214bf4fe19b3","src/net/http_uploader.rs":"9e8c1837ca0d3f6ea165ec936ab054173c4fe95a958710176c33b4d4d1d98beb","src/net/mod.rs":"59db2f4dcfd0a2d77feb63f40cae2252da59fa8a87e10877fcb305eb91aa0645","src/pings.rs":"2dfccd84848e1933aa4f6a7a707c58ec794c8f73ef2d93ea4d4df71d4e6abc31","src/private/boolean.rs":"1b122b26e1b29b3d0f048f6984b3813ca42982b7b205843e6bbb056ffd45e20f","src/private/counter.rs":"fd74a91e7150189f2f38a56302f478e845e3dacad7232cf632444b4f393e2c7a","src/private/custom_distribution.rs":"d6f183c0bf15a2400fcfea3ce1620628fcfdfc611f623c0da19b6a76d1681474","src/private/datetime.rs":"1a00c5c4c0fd7f1f0a0c2b7096fbba332fb0195c430f1aed7570a45f09d5a1a6","src/private/event.rs":"89242b8bf852b5ef6a9d41df6945fdc83b0491f7baeb3d0a45a32156f8f2178a","src/private/labeled.rs":"e7e68d6a8fd84594ff76209d59733c66e523777a98e0492ed4ddd6c18475148c","src/private/memory_distribution.rs":"472e5f866a9cda82685c9b3de7a401498bc24473760fd56df17195e49ccbe76d","src/private/mod.rs":"9a9031401285468b98bbafb3a71175bf9950c114aa229ae7634484eac7eb2b12","src/private/ping.rs":"16345b5b26d61aadc41a750fabfbfd39f09ea8817e3db0362a3b1ad38dc4634b","src/private/quantity.rs":"4e29fb63009de1bad1f3c562e106a30e21d6bbf58511d7f58258c409ebe8999d","src/private/recorded_experiment_data.rs":"66b2601902a2dc2b7a283717c21ce754de94fcca30d12e0398195c8ad49c90af","src/private/string.rs":"d03b0bb0ce3be66b4ab2253934969a40dbac9ae216d137dbee009d7d61291eb6","src/private/string_list.rs":"89de25e0440638809c9510a5d179a05378bdbf25faaf75527c0f1bfae0d80ec0","src/private/timespan.rs":"1db054d9db7cbe36a78fa08e3f4f696a1eba17e8c362172b0058f733de13f68b","src/private/timing_distribution.rs":"c99bd6aebc47ac2059b6392fc62a8fe15d5d0ac51f26a65ab41201ae9d12cdb3","src/private/uuid.rs":"21e95d12d2913b315da145365fcda744b8e2a47b442c2811513c2b5ec3bd41e0","src/system.rs":"ba7b3eac040abe4691d9d287562ddca6d7e92a6d6109c3f0c443b707a100d75a","src/test.rs":"f06fe6b7187a6725683a3dcded65006f5689e55c350c54efc52930e6cd86c775","tests/common/mod.rs":"4837df2e771929cc077e6fb9a9239645e8e0f7bc6c9f409b71c4d147edf334fc","tests/init_fails.rs":"32614f46e49ec91cd33bc381246b44c22caa19f3eca5c2708589619cd1a99471","tests/never_init.rs":"1f33b8ce7ca3514b57b48cc16d98408974c85cf8aa7d13257ffc2ad878ebb295","tests/no_time_to_init.rs":"af55667ce9a7331d48e6a01815f8f184ae252dfc1aefcd8aeb478100a3726972","tests/overflowing_preinit.rs":"27d160d20e189ada2e3775e7873064c29a99e919dbfe051d8bd733d1fa5c7191","tests/schema.rs":"bf4eeffcf0867996fb5fd1dfc8ea874afd0b1e2dc6198e2ab541dd28aa48b8a4","tests/simple.rs":"1c1ec8babd3803a4e117d59c62215acf4bb73b1d9d34278c8882216a2686dbca"},"package":"8193d78e62cd2e5e1e0d430e5ca0462c5998c3ac1ba8b723702edb3bb3a47689"}
{"files":{"Cargo.toml":"7c19bf0c2bae621cfd9f164693a59d9c652bf63bdd65f01061718ea1300a150b","LICENSE":"1f256ecad192880510e84ad60474eab7589218784b9a50bc7ceee34c2b91f1d5","README.md":"fd9e0ca6907917ea6bec5de05e15dd21d20fae1cb7f3250467bb20231a8e1065","src/common_test.rs":"585eb8b0ecc6628090da6b0e7f8488f13c428286d5533af1c96af19649ee29c8","src/configuration.rs":"a0b6fe024ee7fd1164780fc4a63ad57c2f6f381c64c2417fc51ef6a1f8898889","src/core_metrics.rs":"0ecf9ec7de9032f45e6c0feaebf17c614f9be88c1a28043e397eaf7d3b18ba37","src/dispatcher/global.rs":"427d23168ea2457f0332e6f4d6684810fe35a5d5df1e8336d070159d82d936e5","src/dispatcher/mod.rs":"9bf42571a23491db1c6643064caee463301dccac13430883ac0bdaefacf4ace3","src/glean_metrics.rs":"151b6e5acc12436c33c75e2e0a18f9769179d3c7fdc60a22fa02afb76feaf00f","src/lib.rs":"cf3b01826ce7ec03154e1e06e99384e50a9e1fae95fda42deed1f809c8064f34","src/net/http_uploader.rs":"9e8c1837ca0d3f6ea165ec936ab054173c4fe95a958710176c33b4d4d1d98beb","src/net/mod.rs":"59db2f4dcfd0a2d77feb63f40cae2252da59fa8a87e10877fcb305eb91aa0645","src/pings.rs":"2dfccd84848e1933aa4f6a7a707c58ec794c8f73ef2d93ea4d4df71d4e6abc31","src/private/boolean.rs":"eeadc0529e2c69a930479f208746799b064b27facab8306c1c10c650e83fb63c","src/private/counter.rs":"75ad96cd5f53d024230357223244c0a8f58f8b32a9a4d0dbc7cc6ecd74db13b5","src/private/custom_distribution.rs":"0de9cd030a555d93352a7fd251febf4de3a2ca4eeb7666abe5baa884d59168b8","src/private/datetime.rs":"f7e68491b267452fc7d0bb50a2e2f73337d2580435b0518e4cb375a5a30d3017","src/private/denominator.rs":"746c5dfd81fe4027061d799421e35c2cf47b14b98e18e15f2e0d21379604f3f0","src/private/event.rs":"7839ded635f979c7e3a97227530c5843783b8cb2e4487aedbdce7ae16c611e76","src/private/labeled.rs":"e7e68d6a8fd84594ff76209d59733c66e523777a98e0492ed4ddd6c18475148c","src/private/memory_distribution.rs":"201ce833900fca33f2e4bdd65d9055927627c5e97c9df001351ca40e8e11efae","src/private/mod.rs":"413a41942a48de3d39e9346c2a0803a3ce184978173f8a79b13a116be4abaffe","src/private/numerator.rs":"4133f4a1f2a20931176ecaa7e85a96a4d639ba1b3737441a5713c18909892a42","src/private/ping.rs":"915fc42994e0929656daee5511946ac1f56fe0d4d704e97e13795771d9890180","src/private/quantity.rs":"0fa3c6fb00a4c4d659284a87a4cfbfc5153a73e65ed802f27d74c1bd7fc06273","src/private/rate.rs":"bb7f1a1c9aa2413eb4c606f04aa58199a18d9d12a97fb6548d410f939b01ed09","src/private/recorded_experiment_data.rs":"66b2601902a2dc2b7a283717c21ce754de94fcca30d12e0398195c8ad49c90af","src/private/string.rs":"cab1b0a3a5368a1650dc253bcb5a4622f0d016913bf323c7d74c4130ab22f886","src/private/string_list.rs":"2f4df2aefdf9130a9913cd06dbf91747953ac79648af1c1b924053af18944bac","src/private/timespan.rs":"30d7b78105815f6a92bf664980ede1464e38a7eedb4dfcb30d9811bec00e780b","src/private/timing_distribution.rs":"9da69e91c29c5df4092dbeeb2f4206c65ab6ee20196c60b1cacdd80c9a23a2dd","src/private/uuid.rs":"7b76b815f08ac70522c65785f765c59d397f54ee257d47f8290029b456dce0ed","src/system.rs":"98aae0e0c9bf53f92fce4ca3d6040439f540023b63aab022c8c26381f04a4185","src/test.rs":"8e8c1620b94ed753ea849e76083a698c9ab47dba31cf330af68d2a89a6e361f9","tests/common/mod.rs":"4837df2e771929cc077e6fb9a9239645e8e0f7bc6c9f409b71c4d147edf334fc","tests/init_fails.rs":"302aae0c58b06a3bd679d3f5c49cc55f97a3af824b960985a9055a1ad497e5ef","tests/never_init.rs":"1f33b8ce7ca3514b57b48cc16d98408974c85cf8aa7d13257ffc2ad878ebb295","tests/no_time_to_init.rs":"3ae2c1abda55da68519bda1afdf6555f3190e92eedda41f2d43b914c41f97cb9","tests/overflowing_preinit.rs":"0cdb186d07658ffe528f361429dcd20b79c8b313615e4fcd4f793bab3decc566","tests/schema.rs":"4b52dcacc4edf0313d970e65eec87de89341afb1fe6ec9fbc351d88859411871","tests/simple.rs":"d7afe9e7e2645a575eb9ba98bebd54278ff97cb5ab3aac76f52d6ab1802de60c"},"package":"27c487254df8abcae3b3ff1a8d5ceb8f7d39b05e67d58f0b89898c20844da41d"}

7
third_party/rust/glean/Cargo.toml vendored
View File

@ -13,7 +13,7 @@
[package]
edition = "2018"
name = "glean"
version = "34.1.0"
version = "36.0.0"
authors = ["Jan-Erik Rediger <jrediger@mozilla.com>", "The Glean Team <glean-team@mozilla.com>"]
include = ["/README.md", "/LICENSE", "/src", "/tests", "/Cargo.toml"]
description = "Glean SDK Rust language bindings"
@ -29,7 +29,7 @@ features = ["serde"]
version = "0.5"
[dependencies.glean-core]
version = "34.1.0"
version = "36.0.0"
[dependencies.inherent]
version = "0.1.4"
@ -56,6 +56,9 @@ version = "0.1.40"
[dependencies.uuid]
version = "0.8.1"
features = ["v4"]
[dependencies.whatsys]
version = "0.1.2"
[dev-dependencies.env_logger]
version = "0.7.1"
features = ["termcolor", "atty", "humantime"]

2
third_party/rust/glean/src/common_test.rs vendored
View File

@ -36,7 +36,7 @@ pub(crate) fn new_glean(
clear_stores: bool,
) -> tempfile::TempDir {
let dir = tempfile::tempdir().unwrap();
let tmpname = dir.path().display().to_string();
let tmpname = dir.path().to_path_buf();
let cfg = match configuration {
Some(c) => c,

4
third_party/rust/glean/src/configuration.rs vendored
View File

@ -4,6 +4,8 @@
use crate::net::PingUploader;
use std::path::PathBuf;
/// The default server pings are sent to.
pub(crate) const DEFAULT_GLEAN_ENDPOINT: &str = "https://incoming.telemetry.mozilla.org";
@ -15,7 +17,7 @@ pub struct Configuration {
/// Whether upload should be enabled.
pub upload_enabled: bool,
/// Path to a directory to store all data in.
pub data_path: String,
pub data_path: PathBuf,
/// The application ID (will be sanitized during initialization).
pub application_id: String,
/// The maximum number of events to store before sending a ping containing events.

112
third_party/rust/glean/src/lib.rs vendored
View File

@ -151,6 +151,17 @@ where
f(&mut lock)
}
/// Launches a new task on the global dispatch queue with a reference to the Glean singleton.
fn launch_with_glean(callback: impl FnOnce(&Glean) + Send + 'static) {
dispatcher::launch(|| crate::with_glean(callback));
}
/// Launches a new task on the global dispatch queue with a mutable reference to the
/// Glean singleton.
fn launch_with_glean_mut(callback: impl FnOnce(&mut Glean) + Send + 'static) {
dispatcher::launch(|| crate::with_glean_mut(callback));
}
/// Creates and initializes a new Glean object.
///
/// See [`glean_core::Glean::new`] for more information.
@ -171,7 +182,7 @@ pub fn initialize(cfg: Configuration, client_info: ClientInfoMetrics) {
.spawn(move || {
let core_cfg = glean_core::Configuration {
upload_enabled: cfg.upload_enabled,
data_path: cfg.data_path.clone(),
data_path: cfg.data_path.into_os_string().into_string().unwrap(),
application_id: cfg.application_id.clone(),
language_binding_name: LANGUAGE_BINDING_NAME.into(),
max_events: cfg.max_events,
@ -352,10 +363,8 @@ pub fn shutdown() {
return;
}
dispatcher::launch(move || {
with_glean_mut(|glean| {
glean.set_dirty_flag(false);
})
crate::launch_with_glean_mut(|glean| {
glean.set_dirty_flag(false);
});
if let Err(e) = dispatcher::shutdown() {
@ -425,7 +434,7 @@ fn initialize_core_metrics(
if let Some(app_channel) = channel {
core_metrics::internal_metrics::app_channel.set_sync(glean, app_channel);
}
core_metrics::internal_metrics::os_version.set_sync(glean, "unknown".to_string());
core_metrics::internal_metrics::os_version.set_sync(glean, system::get_os_version());
core_metrics::internal_metrics::architecture.set_sync(glean, system::ARCH.to_string());
core_metrics::internal_metrics::device_manufacturer.set_sync(glean, "unknown".to_string());
core_metrics::internal_metrics::device_model.set_sync(glean, "unknown".to_string());
@ -450,27 +459,25 @@ pub fn set_upload_enabled(enabled: bool) {
//
// Because the dispatch queue is halted until Glean is fully initialized
// we can safely enqueue here and it will execute after initialization.
dispatcher::launch(move || {
with_glean_mut(|glean| {
let state = global_state().lock().unwrap();
let old_enabled = glean.is_upload_enabled();
glean.set_upload_enabled(enabled);
crate::launch_with_glean_mut(move |glean| {
let state = global_state().lock().unwrap();
let old_enabled = glean.is_upload_enabled();
glean.set_upload_enabled(enabled);
// TODO: Cancel upload and any outstanding metrics ping scheduler
// task. Will happen on bug 1672951.
// TODO: Cancel upload and any outstanding metrics ping scheduler
// task. Will happen on bug 1672951.
if !old_enabled && enabled {
// If uploading is being re-enabled, we have to restore the
// application-lifetime metrics.
initialize_core_metrics(&glean, &state.client_info, state.channel.clone());
}
if !old_enabled && enabled {
// If uploading is being re-enabled, we have to restore the
// application-lifetime metrics.
initialize_core_metrics(&glean, &state.client_info, state.channel.clone());
}
if old_enabled && !enabled {
// If uploading is disabled, we need to send the deletion-request ping:
// note that glean-core takes care of generating it.
state.upload_manager.trigger_upload();
}
});
if old_enabled && !enabled {
// If uploading is disabled, we need to send the deletion-request ping:
// note that glean-core takes care of generating it.
state.upload_manager.trigger_upload();
}
});
}
@ -482,10 +489,8 @@ pub fn register_ping_type(ping: &private::PingType) {
// Submission itself is also dispatched, so it will always come after the registration.
if was_initialize_called() {
let ping = ping.clone();
dispatcher::launch(move || {
with_glean_mut(|glean| {
glean.register_ping_type(&ping.ping_type);
})
crate::launch_with_glean_mut(move |glean| {
glean.register_ping_type(&ping.ping_type);
})
} else {
// We need to keep track of pings, so they get re-registered after a reset or
@ -570,14 +575,8 @@ pub fn set_experiment_active(
branch: String,
extra: Option<HashMap<String, String>>,
) {
dispatcher::launch(move || {
with_glean(|glean| {
glean.set_experiment_active(
experiment_id.to_owned(),
branch.to_owned(),
extra.to_owned(),
)
});
crate::launch_with_glean(move |glean| {
glean.set_experiment_active(experiment_id.to_owned(), branch.to_owned(), extra)
})
}
@ -585,9 +584,7 @@ pub fn set_experiment_active(
///
/// See [`glean_core::Glean::set_experiment_inactive`].
pub fn set_experiment_inactive(experiment_id: String) {
dispatcher::launch(move || {
with_glean(|glean| glean.set_experiment_inactive(experiment_id.to_owned()))
})
crate::launch_with_glean(move |glean| glean.set_experiment_inactive(experiment_id))
}
/// Performs the collection/cleanup operations required by becoming active.
@ -597,16 +594,14 @@ pub fn set_experiment_inactive(experiment_id: String) {
/// This should be called whenever the consuming product becomes active (e.g.
/// getting to foreground).
pub fn handle_client_active() {
dispatcher::launch(move || {
with_glean_mut(|glean| {
glean.handle_client_active();
crate::launch_with_glean_mut(|glean| {
glean.handle_client_active();
// The above call may generate pings, so we need to trigger
// the uploader. It's fine to trigger it if no ping was generated:
// it will bail out.
let state = global_state().lock().unwrap();
state.upload_manager.trigger_upload();
})
// The above call may generate pings, so we need to trigger
// the uploader. It's fine to trigger it if no ping was generated:
// it will bail out.
let state = global_state().lock().unwrap();
state.upload_manager.trigger_upload();
});
// The previous block of code may send a ping containing the `duration` metric,
@ -628,16 +623,14 @@ pub fn handle_client_inactive() {
// by the next call.
core_metrics::internal_metrics::baseline_duration.stop();
dispatcher::launch(move || {
with_glean_mut(|glean| {
glean.handle_client_inactive();
crate::launch_with_glean_mut(|glean| {
glean.handle_client_inactive();
// The above call may generate pings, so we need to trigger
// the uploader. It's fine to trigger it if no ping was generated:
// it will bail out.
let state = global_state().lock().unwrap();
state.upload_manager.trigger_upload();
})
// The above call may generate pings, so we need to trigger
// the uploader. It's fine to trigger it if no ping was generated:
// it will bail out.
let state = global_state().lock().unwrap();
state.upload_manager.trigger_upload();
})
}
@ -767,5 +760,10 @@ pub fn set_source_tags(tags: Vec<String>) -> bool {
}
}
/// Returns a timestamp corresponding to "now" with millisecond precision.
pub fn get_timestamp_ms() -> u64 {
glean_core::get_timestamp_ms()
}
#[cfg(test)]
mod test;

4
third_party/rust/glean/src/private/boolean.rs vendored
View File

@ -7,8 +7,6 @@ use std::sync::Arc;
use glean_core::metrics::MetricType;
use crate::dispatcher;
// We need to wrap the glean-core type: otherwise if we try to implement
// the trait for the metric in `glean_core::metrics` we hit error[E0117]:
// only traits defined in the current crate can be implemented for arbitrary
@ -33,7 +31,7 @@ impl BooleanMetric {
impl glean_core::traits::Boolean for BooleanMetric {
fn set(&self, value: bool) {
let metric = Arc::clone(&self.0);
dispatcher::launch(move || crate::with_glean(|glean| metric.set(glean, value)));
crate::launch_with_glean(move |glean| metric.set(glean, value));
}
fn test_get_value<'a, S: Into<Option<&'a str>>>(&self, ping_name: S) -> Option<bool> {

4
third_party/rust/glean/src/private/counter.rs vendored
View File

@ -9,8 +9,6 @@ use std::sync::Arc;
use glean_core::metrics::MetricType;
use glean_core::ErrorType;
use crate::dispatcher;
// We need to wrap the glean-core type: otherwise if we try to implement
// the trait for the metric in `glean_core::metrics` we hit error[E0117]:
// only traits defined in the current crate can be implemented for arbitrary
@ -40,7 +38,7 @@ impl CounterMetric {
impl glean_core::traits::Counter for CounterMetric {
fn add(&self, amount: i32) {
let metric = Arc::clone(&self.0);
dispatcher::launch(move || crate::with_glean(|glean| metric.add(glean, amount)));
crate::launch_with_glean(move |glean| metric.add(glean, amount));
}
fn test_get_value<'a, S: Into<Option<&'a str>>>(&self, ping_name: S) -> Option<i32> {

6
third_party/rust/glean/src/private/custom_distribution.rs vendored
View File

@ -8,8 +8,6 @@ use std::sync::Arc;
use glean_core::metrics::{DistributionData, MetricType};
use glean_core::{CommonMetricData, ErrorType, HistogramType};
use crate::dispatcher;
// We need to wrap the glean-core type: otherwise if we try to implement
// the trait for the metric in `glean_core::metrics` we hit error[E0117]:
// only traits defined in the current crate can be implemented for arbitrary
@ -48,9 +46,7 @@ impl CustomDistributionMetric {
impl glean_core::traits::CustomDistribution for CustomDistributionMetric {
fn accumulate_samples_signed(&self, samples: Vec<i64>) {
let metric = Arc::clone(&self.0);
dispatcher::launch(move || {
crate::with_glean(|glean| metric.accumulate_samples_signed(glean, samples))
});
crate::launch_with_glean(move |glean| metric.accumulate_samples_signed(glean, samples));
}
fn test_get_value<'a, S: Into<Option<&'a str>>>(

4
third_party/rust/glean/src/private/datetime.rs vendored
View File

@ -9,8 +9,6 @@ use glean_core::metrics::MetricType;
pub use glean_core::metrics::{Datetime, TimeUnit};
use glean_core::ErrorType;
use crate::dispatcher;
// We need to wrap the glean-core type: otherwise if we try to implement
// the trait for the metric in `glean_core::metrics` we hit error[E0117]:
// only traits defined in the current crate can be implemented for arbitrary
@ -37,7 +35,7 @@ impl DatetimeMetric {
impl glean_core::traits::Datetime for DatetimeMetric {
fn set(&self, value: Option<Datetime>) {
let metric = Arc::clone(&self.0);
dispatcher::launch(move || crate::with_glean(|glean| metric.set(glean, value)));
crate::launch_with_glean(move |glean| metric.set(glean, value));
}
fn test_get_value<'a, S: Into<Option<&'a str>>>(&self, ping_name: S) -> Option<Datetime> {

64
third_party/rust/glean/src/private/denominator.rs vendored Normal file
View File

@ -0,0 +1,64 @@
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this
// file, You can obtain one at https://mozilla.org/MPL/2.0/.
use inherent::inherent;
use std::sync::Arc;
use glean_core::metrics::MetricType;
use glean_core::CommonMetricData;
use glean_core::ErrorType;
// We need to wrap the glean-core type: otherwise if we try to implement
// the trait for the metric in `glean_core::metrics` we hit error[E0117]:
// only traits defined in the current crate can be implemented for arbitrary
// types.
/// Developer-facing API for recording counter metrics that are acting as
/// external denominators for rate metrics.
///
/// Instances of this class type are automatically generated by the parsers
/// at build time, allowing developers to record values that were previously
/// registered in the metrics.yaml file.
#[derive(Clone)]
pub struct DenominatorMetric(pub(crate) Arc<glean_core::metrics::DenominatorMetric>);
impl DenominatorMetric {
/// The public constructor used by automatically generated metrics.
pub fn new(meta: CommonMetricData, numerators: Vec<CommonMetricData>) -> Self {
Self(Arc::new(glean_core::metrics::DenominatorMetric::new(
meta, numerators,
)))
}
}
#[inherent(pub)]
impl glean_core::traits::Counter for DenominatorMetric {
fn add(&self, amount: i32) {
let metric = Arc::clone(&self.0);
crate::launch_with_glean(move |glean| metric.add(glean, amount));
}
fn test_get_value<'a, S: Into<Option<&'a str>>>(&self, ping_name: S) -> Option<i32> {
crate::block_on_dispatcher();
let queried_ping_name = ping_name
.into()
.unwrap_or_else(|| &self.0.meta().send_in_pings[0]);
crate::with_glean(|glean| self.0.test_get_value(glean, queried_ping_name))
}
fn test_get_num_recorded_errors<'a, S: Into<Option<&'a str>>>(
&self,
error: ErrorType,
ping_name: S,
) -> i32 {
crate::block_on_dispatcher();
crate::with_glean_mut(|glean| {
glean_core::test_get_num_recorded_errors(&glean, self.0.meta(), error, ping_name.into())
.unwrap_or(0)
})
}
}

16
third_party/rust/glean/src/private/event.rs vendored
View File

@ -8,7 +8,7 @@ use std::{collections::HashMap, marker::PhantomData, sync::Arc};
use glean_core::metrics::MetricType;
use glean_core::traits;
use crate::{dispatcher, ErrorType, RecordedEvent};
use crate::{ErrorType, RecordedEvent};
pub use glean_core::traits::NoExtraKeys;
@ -41,6 +41,15 @@ impl<K: traits::ExtraKeys> EventMetric<K> {
extra_keys: PhantomData,
}
}
/// Record a new event with a provided timestamp.
///
/// It's the caller's responsibility to ensure the timestamp comes from the same clock source.
/// Use [`glean::get_timestamp_ms`](crate::get_timestamp_ms) to get a valid timestamp.
pub fn record_with_time(&self, timestamp: u64, extra: HashMap<i32, String>) {
let metric = Arc::clone(&self.inner);
crate::launch_with_glean(move |glean| metric.record(glean, timestamp, Some(extra)));
}
}
#[inherent(pub)]
@ -48,15 +57,14 @@ impl<K: traits::ExtraKeys> traits::Event for EventMetric<K> {
type Extra = K;
fn record<M: Into<Option<HashMap<<Self as traits::Event>::Extra, String>>>>(&self, extra: M) {
const NANOS_PER_MILLI: u64 = 1_000_000;
let now = time::precise_time_ns() / NANOS_PER_MILLI;
let now = crate::get_timestamp_ms();
// Translate from [ExtraKey -> String] to a [Int -> String] map
let extra = extra
.into()
.map(|h| h.into_iter().map(|(k, v)| (k.index(), v)).collect());
let metric = Arc::clone(&self.inner);
dispatcher::launch(move || crate::with_glean(|glean| metric.record(glean, now, extra)));
crate::launch_with_glean(move |glean| metric.record(glean, now, extra));
}
fn test_get_value<'a, S: Into<Option<&'a str>>>(

4
third_party/rust/glean/src/private/memory_distribution.rs vendored
View File

@ -8,8 +8,6 @@ use std::sync::Arc;
use glean_core::metrics::{DistributionData, MemoryUnit, MetricType};
use glean_core::ErrorType;
use crate::dispatcher;
// We need to wrap the glean-core type: otherwise if we try to implement
// the trait for the metric in `glean_core::metrics` we hit error[E0117]:
// only traits defined in the current crate can be implemented for arbitrary
@ -36,7 +34,7 @@ impl MemoryDistributionMetric {
impl glean_core::traits::MemoryDistribution for MemoryDistributionMetric {
fn accumulate(&self, sample: u64) {
let metric = Arc::clone(&self.0);
dispatcher::launch(move || crate::with_glean(|glean| metric.accumulate(glean, sample)));
crate::launch_with_glean(move |glean| metric.accumulate(glean, sample));
}
fn test_get_value<'a, S: Into<Option<&'a str>>>(

6
third_party/rust/glean/src/private/mod.rs vendored
View File

@ -8,11 +8,14 @@ mod boolean;
mod counter;
mod custom_distribution;
mod datetime;
mod denominator;
mod event;
mod labeled;
mod memory_distribution;
mod numerator;
mod ping;
mod quantity;
mod rate;
mod recorded_experiment_data;
mod string;
mod string_list;
@ -25,11 +28,14 @@ pub use boolean::BooleanMetric;
pub use counter::CounterMetric;
pub use custom_distribution::CustomDistributionMetric;
pub use datetime::{Datetime, DatetimeMetric};
pub use denominator::DenominatorMetric;
pub use event::EventMetric;
pub use labeled::{AllowLabeled, LabeledMetric};
pub use memory_distribution::MemoryDistributionMetric;
pub use numerator::NumeratorMetric;
pub use ping::PingType;
pub use quantity::QuantityMetric;
pub use rate::RateMetric;
pub use recorded_experiment_data::RecordedExperimentData;
pub use string::StringMetric;
pub use string_list::StringListMetric;

108
third_party/rust/glean/src/private/numerator.rs vendored Normal file
View File

@ -0,0 +1,108 @@
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this
// file, You can obtain one at https://mozilla.org/MPL/2.0/.
use inherent::inherent;
use std::sync::Arc;
use glean_core::metrics::MetricType;
use glean_core::ErrorType;
use crate::dispatcher;
// We need to wrap the glean-core type: otherwise if we try to implement
// the trait for the metric in `glean_core::metrics` we hit error[E0117]:
// only traits defined in the current crate can be implemented for arbitrary
// types.
/// Developer-facing API for recording rate metrics with external denominators.
///
/// Instances of this class type are automatically generated by the parsers
/// at build time, allowing developers to record values that were previously
/// registered in the metrics.yaml file.
#[derive(Clone)]
pub struct NumeratorMetric(pub(crate) Arc<glean_core::metrics::RateMetric>);
impl NumeratorMetric {
/// The public constructor used by automatically generated metrics.
pub fn new(meta: glean_core::CommonMetricData) -> Self {
Self(Arc::new(glean_core::metrics::RateMetric::new(meta)))
}
}
#[inherent(pub)]
impl glean_core::traits::Numerator for NumeratorMetric {
fn add_to_numerator(&self, amount: i32) {
let metric = Arc::clone(&self.0);
dispatcher::launch(move || {
crate::with_glean(|glean| metric.add_to_numerator(glean, amount))
});
}
fn test_get_value<'a, S: Into<Option<&'a str>>>(&self, ping_name: S) -> Option<(i32, i32)> {
crate::block_on_dispatcher();
let queried_ping_name = ping_name
.into()
.unwrap_or_else(|| &self.0.meta().send_in_pings[0]);
crate::with_glean(|glean| self.0.test_get_value(glean, queried_ping_name))
}
fn test_get_num_recorded_errors<'a, S: Into<Option<&'a str>>>(
&self,
error: ErrorType,
ping_name: S,
) -> i32 {
crate::block_on_dispatcher();
crate::with_glean_mut(|glean| {
glean_core::test_get_num_recorded_errors(&glean, self.0.meta(), error, ping_name.into())
.unwrap_or(0)
})
}
}
#[cfg(test)]
mod test {
use super::*;
use crate::common_test::{lock_test, new_glean};
use crate::CommonMetricData;
#[test]
fn numerator_smoke() {
let _lock = lock_test();
let _t = new_glean(None, true);
let metric: NumeratorMetric = NumeratorMetric::new(CommonMetricData {
name: "rate".into(),
category: "test".into(),
send_in_pings: vec!["test1".into()],
..Default::default()
});
// Adding 0 doesn't error.
metric.add_to_numerator(0);
assert_eq!(
metric.test_get_num_recorded_errors(ErrorType::InvalidValue, None),
0
);
// Adding a negative value errors.
metric.add_to_numerator(-1);
assert_eq!(
metric.test_get_num_recorded_errors(ErrorType::InvalidValue, None),
1
);
// Getting the value returns 0s if that's all we have.
assert_eq!(metric.test_get_value(None), Some((0, 0)));
// And normal values of course work.
metric.add_to_numerator(22);
assert_eq!(metric.test_get_value(None), Some((22, 0)));
}
}

38
third_party/rust/glean/src/private/ping.rs vendored
View File

@ -2,15 +2,24 @@
// License, v. 2.0. If a copy of the MPL was not distributed with this
// file, You can obtain one at https://mozilla.org/MPL/2.0/.
use std::sync::{Arc, Mutex};
use inherent::inherent;
type BoxedCallback = Box<dyn FnOnce(Option<&str>) + Send + 'static>;
/// A ping is a bundle of related metrics, gathered in a payload to be transmitted.
///
/// The ping payload will be encoded in JSON format and contains shared information data.
#[derive(Clone, Debug)]
#[derive(Clone)]
pub struct PingType {
pub(crate) name: String,
pub(crate) ping_type: glean_core::metrics::PingType,
/// **Test-only API**
///
/// A function to be called right before a ping is submitted.
test_callback: Arc<Mutex<Option<BoxedCallback>>>,
}
impl PingType {
@ -36,15 +45,40 @@ impl PingType {
reason_codes,
);
let me = Self { name, ping_type };
let me = Self {
name,
ping_type,
test_callback: Arc::new(Default::default()),
};
crate::register_ping_type(&me);
me
}
/// **Test-only API**
///
/// Attach a callback to be called right before a new ping is submitted.
/// The provided function is called exactly once before submitting a ping.
///
/// Note: The callback will be called on any call to submit.
/// A ping might not be sent afterwards, e.g. if the ping is otherwise empty (and
/// `send_if_empty` is `false`).
pub fn test_before_next_submit(&self, cb: impl FnOnce(Option<&str>) + Send + 'static) {
let mut test_callback = self.test_callback.lock().unwrap();
let cb = Box::new(cb);
*test_callback = Some(cb);
}
}
#[inherent(pub)]
impl glean_core::traits::Ping for PingType {
fn submit(&self, reason: Option<&str>) {
let mut cb = self.test_callback.lock().unwrap();
let cb = cb.take();
if let Some(cb) = cb {
cb(reason)
}
crate::submit_ping(self, reason)
}
}

4
third_party/rust/glean/src/private/quantity.rs vendored
View File

@ -8,8 +8,6 @@ use std::sync::Arc;
use glean_core::metrics::MetricType;
use glean_core::ErrorType;
use crate::dispatcher;
// We need to wrap the glean-core type, otherwise if we try to implement
// the trait for the metric in `glean_core::metrics` we hit error[E0117]:
// only traits defined in the current crate can be implemented for arbitrary
@ -34,7 +32,7 @@ impl QuantityMetric {
impl glean_core::traits::Quantity for QuantityMetric {
fn set(&self, value: i64) {
let metric = Arc::clone(&self.0);
dispatcher::launch(move || crate::with_glean(|glean| metric.set(glean, value)));
crate::launch_with_glean(move |glean| metric.set(glean, value));
}
fn test_get_value<'a, S: Into<Option<&'a str>>>(&self, ping_name: S) -> Option<i64> {

118
third_party/rust/glean/src/private/rate.rs vendored Normal file
View File

@ -0,0 +1,118 @@
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this
// file, You can obtain one at https://mozilla.org/MPL/2.0/.
use inherent::inherent;
use std::sync::Arc;
use glean_core::metrics::MetricType;
use glean_core::ErrorType;
use crate::dispatcher;
// We need to wrap the glean-core type: otherwise if we try to implement
// the trait for the metric in `glean_core::metrics` we hit error[E0117]:
// only traits defined in the current crate can be implemented for arbitrary
// types.
/// Developer-facing API for recording rate metrics.
///
/// Instances of this class type are automatically generated by the parsers
/// at build time, allowing developers to record values that were previously
/// registered in the metrics.yaml file.
#[derive(Clone)]
pub struct RateMetric(pub(crate) Arc<glean_core::metrics::RateMetric>);
impl RateMetric {
/// The public constructor used by automatically generated metrics.
pub fn new(meta: glean_core::CommonMetricData) -> Self {
Self(Arc::new(glean_core::metrics::RateMetric::new(meta)))
}
}
#[inherent(pub)]
impl glean_core::traits::Rate for RateMetric {
fn add_to_numerator(&self, amount: i32) {
let metric = Arc::clone(&self.0);
dispatcher::launch(move || {
crate::with_glean(|glean| metric.add_to_numerator(glean, amount))
});
}
fn add_to_denominator(&self, amount: i32) {
let metric = Arc::clone(&self.0);
dispatcher::launch(move || {
crate::with_glean(|glean| metric.add_to_denominator(glean, amount))
});
}
fn test_get_value<'a, S: Into<Option<&'a str>>>(&self, ping_name: S) -> Option<(i32, i32)> {
crate::block_on_dispatcher();
let queried_ping_name = ping_name
.into()
.unwrap_or_else(|| &self.0.meta().send_in_pings[0]);
crate::with_glean(|glean| self.0.test_get_value(glean, queried_ping_name))
}
fn test_get_num_recorded_errors<'a, S: Into<Option<&'a str>>>(
&self,
error: ErrorType,
ping_name: S,
) -> i32 {
crate::block_on_dispatcher();
crate::with_glean_mut(|glean| {
glean_core::test_get_num_recorded_errors(&glean, self.0.meta(), error, ping_name.into())
.unwrap_or(0)
})
}
}
#[cfg(test)]
mod test {
use super::*;
use crate::common_test::{lock_test, new_glean};
use crate::CommonMetricData;
#[test]
fn rate_smoke() {
let _lock = lock_test();
let _t = new_glean(None, true);
let metric: RateMetric = RateMetric::new(CommonMetricData {
name: "rate".into(),
category: "test".into(),
send_in_pings: vec!["test1".into()],
..Default::default()
});
// Adding 0 doesn't error.
metric.add_to_numerator(0);
metric.add_to_denominator(0);
assert_eq!(
metric.test_get_num_recorded_errors(ErrorType::InvalidValue, None),
0
);
// Adding a negative value errors.
metric.add_to_numerator(-1);
metric.add_to_denominator(-1);
assert_eq!(
metric.test_get_num_recorded_errors(ErrorType::InvalidValue, None),
2
);
// Getting the value returns 0s if that's all we have.
assert_eq!(metric.test_get_value(None), Some((0, 0)));
// And normal values of course work.
metric.add_to_numerator(22);
metric.add_to_denominator(7);
assert_eq!(metric.test_get_value(None), Some((22, 7)));
}
}

4
third_party/rust/glean/src/private/string.rs vendored
View File

@ -9,8 +9,6 @@ use std::sync::Arc;
use glean_core::metrics::MetricType;
use glean_core::ErrorType;
use crate::dispatcher;
// We need to wrap the glean-core type, otherwise if we try to implement
// the trait for the metric in `glean_core::metrics` we hit error[E0117]:
// only traits defined in the current crate can be implemented for arbitrary
@ -41,7 +39,7 @@ impl glean_core::traits::String for StringMetric {
fn set<S: Into<std::string::String>>(&self, value: S) {
let metric = Arc::clone(&self.0);
let new_value = value.into();
dispatcher::launch(move || crate::with_glean(|glean| metric.set(glean, new_value)));
crate::launch_with_glean(move |glean| metric.set(glean, new_value));
}
fn test_get_value<'a, S: Into<Option<&'a str>>>(

6
third_party/rust/glean/src/private/string_list.rs vendored
View File

@ -8,8 +8,6 @@ use std::sync::Arc;
use glean_core::metrics::MetricType;
use glean_core::ErrorType;
use crate::dispatcher;
// We need to wrap the glean-core type: otherwise if we try to implement
// the trait for the metric in `glean_core::metrics` we hit error[E0117]:
// only traits defined in the current crate can be implemented for arbitrary
@ -35,12 +33,12 @@ impl glean_core::traits::StringList for StringListMetric {
fn add<S: Into<String>>(&self, value: S) {
let metric = Arc::clone(&self.0);
let new_value = value.into();
dispatcher::launch(move || crate::with_glean(|glean| metric.add(glean, new_value)));
crate::launch_with_glean(move |glean| metric.add(glean, new_value));
}
fn set(&self, value: Vec<String>) {
let metric = Arc::clone(&self.0);
dispatcher::launch(move || crate::with_glean(|glean| metric.set(glean, value)));
crate::launch_with_glean(move |glean| metric.set(glean, value));
}
fn test_get_value<'a, S: Into<Option<&'a str>>>(&self, ping_name: S) -> Option<Vec<String>> {

24
third_party/rust/glean/src/private/timespan.rs vendored
View File

@ -38,13 +38,11 @@ impl glean_core::traits::Timespan for TimespanMetric {
let start_time = time::precise_time_ns();
let metric = Arc::clone(&self.0);
dispatcher::launch(move || {
crate::with_glean(|glean| {
let mut lock = metric
.write()
.expect("Lock poisoned for timespan metric on start.");
lock.set_start(glean, start_time)
})
crate::launch_with_glean(move |glean| {
let mut lock = metric
.write()
.expect("Lock poisoned for timespan metric on start.");
lock.set_start(glean, start_time)
});
}
@ -52,13 +50,11 @@ impl glean_core::traits::Timespan for TimespanMetric {
let stop_time = time::precise_time_ns();
let metric = Arc::clone(&self.0);
dispatcher::launch(move || {
crate::with_glean(|glean| {
let mut lock = metric
.write()
.expect("Lock poisoned for timespan metric on stop.");
lock.set_stop(glean, stop_time)
})
crate::launch_with_glean(move |glean| {
let mut lock = metric
.write()
.expect("Lock poisoned for timespan metric on stop.");
lock.set_stop(glean, stop_time)
});
}

12
third_party/rust/glean/src/private/timing_distribution.rs vendored
View File

@ -44,13 +44,11 @@ impl glean_core::traits::TimingDistribution for TimingDistributionMetric {
fn stop_and_accumulate(&self, id: TimerId) {
let stop_time = time::precise_time_ns();
let metric = Arc::clone(&self.0);
dispatcher::launch(move || {
crate::with_glean(|glean| {
metric
.write()
.unwrap()
.set_stop_and_accumulate(glean, id, stop_time)
})
crate::launch_with_glean(move |glean| {
metric
.write()
.unwrap()
.set_stop_and_accumulate(glean, id, stop_time)
});
}

4
third_party/rust/glean/src/private/uuid.rs vendored
View File

@ -8,8 +8,6 @@ use std::sync::Arc;
use glean_core::metrics::MetricType;
use glean_core::ErrorType;
use crate::dispatcher;
// We need to wrap the glean-core type, otherwise if we try to implement
// the trait for the metric in `glean_core::metrics` we hit error[E0117]:
// only traits defined in the current crate can be implemented for arbitrary
@ -34,7 +32,7 @@ impl UuidMetric {
impl glean_core::traits::Uuid for UuidMetric {
fn set(&self, value: uuid::Uuid) {
let metric = Arc::clone(&self.0);
dispatcher::launch(move || crate::with_glean(|glean| metric.set(glean, value)));
crate::launch_with_glean(move |glean| metric.set(glean, value));
}
fn generate_and_set(&self) -> uuid::Uuid {

34
third_party/rust/glean/src/system.rs vendored
View File

@ -53,3 +53,37 @@ pub const ARCH: &str = "x86_64";
)))]
/// `target_arch` when building this crate: unknown!
pub const ARCH: &str = "unknown";
#[cfg(any(target_os = "macos", target_os = "windows"))]
/// Returns Darwin kernel version for MacOS, or NT Kernel version for Windows
pub fn get_os_version() -> String {
whatsys::kernel_version().unwrap_or_else(|| "unknown".to_owned())
}
#[cfg(not(any(target_os = "linux", target_os = "macos", target_os = "windows")))]
/// Returns "unknown" for platforms other than Linux, MacOS or Windows
pub fn get_os_version() -> String {
"unknown".to_owned()
}
#[cfg(target_os = "linux")]
/// Returns Linux kernel version, in the format of <Major>.<Minor> e.g. 5.8
pub fn get_os_version() -> String {
parse_linux_os_string(whatsys::kernel_version().unwrap_or_else(|| "unknown".to_owned()))
}
#[cfg(target_os = "linux")]
fn parse_linux_os_string(os_str: String) -> String {
os_str.split('.').take(2).collect::<Vec<&str>>().join(".")
}
#[test]
#[cfg(target_os = "linux")]
fn parse_fixed_linux_os_string() {
let alpine_os_string = "4.12.0-rc6-g48ec1f0-dirty".to_owned();
assert_eq!(parse_linux_os_string(alpine_os_string), "4.12");
let centos_os_string = "3.10.0-514.16.1.el7.x86_64".to_owned();
assert_eq!(parse_linux_os_string(centos_os_string), "3.10");
let ubuntu_os_string = "5.8.0-44-generic".to_owned();
assert_eq!(parse_linux_os_string(ubuntu_os_string), "5.8");
}

103
third_party/rust/glean/src/test.rs vendored
View File

@ -4,7 +4,6 @@
use crate::private::PingType;
use crate::private::{BooleanMetric, CounterMetric, EventMetric};
use std::path::PathBuf;
use super::*;
use crate::common_test::{lock_test, new_glean, GLOBAL_APPLICATION_ID};
@ -35,7 +34,7 @@ fn send_a_ping() {
// Create a custom configuration to use a fake uploader.
let dir = tempfile::tempdir().unwrap();
let tmpname = dir.path().display().to_string();
let tmpname = dir.path().to_path_buf();
let cfg = Configuration {
data_path: tmpname,
@ -144,7 +143,7 @@ fn test_experiments_recording_before_glean_inits() {
set_experiment_inactive("experiment_preinit_disabled".to_string());
let dir = tempfile::tempdir().unwrap();
let tmpname = dir.path().display().to_string();
let tmpname = dir.path().to_path_buf();
test_reset_glean(
Configuration {
@ -205,7 +204,7 @@ fn sending_of_foreground_background_pings() {
// Create a custom configuration to use a fake uploader.
let dir = tempfile::tempdir().unwrap();
let tmpname = dir.path().display().to_string();
let tmpname = dir.path().to_path_buf();
let cfg = Configuration {
data_path: tmpname,
@ -286,7 +285,7 @@ fn sending_of_startup_baseline_ping() {
}
// Create a custom configuration to use a fake uploader.
let tmpname = data_dir.path().display().to_string();
let tmpname = data_dir.path().to_path_buf();
// Now reset Glean: it should still send a baseline ping with reason
// dirty_startup when starting, because of the dirty bit being set.
@ -346,7 +345,7 @@ fn no_dirty_baseline_on_clean_shutdowns() {
}
// Create a custom configuration to use a fake uploader.
let tmpname = data_dir.path().display().to_string();
let tmpname = data_dir.path().to_path_buf();
// Now reset Glean: it should not send a baseline ping, because
// we cleared the dirty bit.
@ -376,14 +375,14 @@ fn initialize_must_not_crash_if_data_dir_is_messed_up() {
let _lock = lock_test();
let dir = tempfile::tempdir().unwrap();
let tmpdirname = dir.path().display().to_string();
let tmpdirname = dir.path();
// Create a file in the temporary dir and use that as the
// name of the Glean data dir.
let file_path = PathBuf::from(tmpdirname).join("notadir");
let file_path = tmpdirname.to_path_buf().join("notadir");
std::fs::write(file_path.clone(), "test").expect("The test Glean dir file must be created");
let cfg = Configuration {
data_path: file_path.to_string_lossy().to_string(),
data_path: file_path,
application_id: GLOBAL_APPLICATION_ID.into(),
upload_enabled: true,
max_events: None,
@ -442,7 +441,7 @@ fn initializing_twice_is_a_noop() {
let _lock = lock_test();
let dir = tempfile::tempdir().unwrap();
let tmpname = dir.path().display().to_string();
let tmpname = dir.path().to_path_buf();
test_reset_glean(
Configuration {
@ -495,7 +494,7 @@ fn the_app_channel_must_be_correctly_set_if_requested() {
let _lock = lock_test();
let dir = tempfile::tempdir().unwrap();
let tmpname = dir.path().display().to_string();
let tmpname = dir.path().to_path_buf();
// No appChannel must be set if nothing was provided through the config
// options.
@ -612,7 +611,7 @@ fn sending_deletion_ping_if_disabled_outside_of_run() {
// Create a custom configuration to use a fake uploader.
let dir = tempfile::tempdir().unwrap();
let tmpname = dir.path().display().to_string();
let tmpname = dir.path().to_path_buf();
let cfg = Configuration {
data_path: tmpname.clone(),
@ -677,7 +676,7 @@ fn no_sending_of_deletion_ping_if_unchanged_outside_of_run() {
// Create a custom configuration to use a fake uploader.
let dir = tempfile::tempdir().unwrap();
let tmpname = dir.path().display().to_string();
let tmpname = dir.path().to_path_buf();
let cfg = Configuration {
data_path: tmpname.clone(),
@ -760,7 +759,7 @@ fn setting_debug_view_tag_before_initialization_should_not_crash() {
// Create a custom configuration to use a fake uploader.
let dir = tempfile::tempdir().unwrap();
let tmpname = dir.path().display().to_string();
let tmpname = dir.path().to_path_buf();
let cfg = Configuration {
data_path: tmpname,
@ -819,7 +818,7 @@ fn setting_source_tags_before_initialization_should_not_crash() {
// Create a custom configuration to use a fake uploader.
let dir = tempfile::tempdir().unwrap();
let tmpname = dir.path().display().to_string();
let tmpname = dir.path().to_path_buf();
let cfg = Configuration {
data_path: tmpname,
@ -885,7 +884,7 @@ fn flipping_upload_enabled_respects_order_of_events() {
// Create a custom configuration to use a fake uploader.
let dir = tempfile::tempdir().unwrap();
let tmpname = dir.path().display().to_string();
let tmpname = dir.path().to_path_buf();
let cfg = Configuration {
data_path: tmpname,
@ -955,7 +954,7 @@ fn registering_pings_before_init_must_work() {
// Create a custom configuration to use a fake uploader.
let dir = tempfile::tempdir().unwrap();
let tmpname = dir.path().display().to_string();
let tmpname = dir.path().to_path_buf();
let cfg = Configuration {
data_path: tmpname,
@ -978,3 +977,73 @@ fn registering_pings_before_init_must_work() {
let url = r.recv().unwrap();
assert!(url.contains("pre-register"));
}
#[test]
fn test_a_ping_before_submission() {
let _lock = lock_test();
// Define a fake uploader that reports back the submission headers
// using a crossbeam channel.
let (s, r) = crossbeam_channel::bounded::<String>(1);
#[derive(Debug)]
pub struct FakeUploader {
sender: crossbeam_channel::Sender<String>,
}
impl net::PingUploader for FakeUploader {
fn upload(
&self,
url: String,
_body: Vec<u8>,
_headers: Vec<(String, String)>,
) -> net::UploadResult {
self.sender.send(url).unwrap();
net::UploadResult::HttpStatus(200)
}
}
// Create a custom configuration to use a fake uploader.
let dir = tempfile::tempdir().unwrap();
let tmpname = dir.path().to_path_buf();
let cfg = Configuration {
data_path: tmpname,
application_id: GLOBAL_APPLICATION_ID.into(),
upload_enabled: true,
max_events: None,
delay_ping_lifetime_io: false,
channel: Some("testing".into()),
server_endpoint: Some("invalid-test-host".into()),
uploader: Some(Box::new(FakeUploader { sender: s })),
};
let _t = new_glean(Some(cfg), true);
// Create a custom ping and register it.
let sample_ping = PingType::new("custom1", true, true, vec![]);
let metric = CounterMetric::new(CommonMetricData {
name: "counter_metric".into(),
category: "test".into(),
send_in_pings: vec!["custom1".into()],
lifetime: Lifetime::Application,
disabled: false,
dynamic_label: None,
});
crate::block_on_dispatcher();
metric.add(1);
sample_ping.test_before_next_submit(move |reason| {
assert_eq!(None, reason);
assert_eq!(1, metric.test_get_value(None).unwrap());
});
// Submit a baseline ping.
sample_ping.submit(None);
// Wait for the ping to arrive.
let url = r.recv().unwrap();
assert!(url.contains("custom1"));
}

2
third_party/rust/glean/tests/init_fails.rs vendored
View File

@ -57,7 +57,7 @@ fn init_fails() {
// Create a custom configuration to use a validating uploader.
let dir = tempfile::tempdir().unwrap();
let tmpname = dir.path().display().to_string();
let tmpname = dir.path().to_path_buf();
let cfg = Configuration {
data_path: tmpname,

2
third_party/rust/glean/tests/no_time_to_init.rs vendored
View File

@ -55,7 +55,7 @@ fn init_fails() {
// Create a custom configuration to use a validating uploader.
let dir = tempfile::tempdir().unwrap();
let tmpname = dir.path().display().to_string();
let tmpname = dir.path().to_path_buf();
let cfg = Configuration {
data_path: tmpname,

2
third_party/rust/glean/tests/overflowing_preinit.rs vendored
View File

@ -60,7 +60,7 @@ fn overflowing_the_task_queue_records_telemetry() {
// Create a custom configuration to use a validating uploader.
let dir = tempfile::tempdir().unwrap();
let tmpname = dir.path().display().to_string();
let tmpname = dir.path().to_path_buf();
let cfg = Configuration {
data_path: tmpname,

63
third_party/rust/glean/tests/schema.rs vendored
View File

@ -8,7 +8,8 @@ use flate2::read::GzDecoder;
use jsonschema_valid::{self, schemas::Draft};
use serde_json::Value;
use glean::{ClientInfoMetrics, Configuration};
use glean::private::{DenominatorMetric, NumeratorMetric, RateMetric};
use glean::{ClientInfoMetrics, CommonMetricData, Configuration};
const SCHEMA_JSON: &str = include_str!("../../../glean.1.schema.json");
@ -22,7 +23,7 @@ const GLOBAL_APPLICATION_ID: &str = "org.mozilla.glean.test.app";
// We need to keep the `TempDir` alive, so that it's not deleted before we stop using it.
fn new_glean(configuration: Option<Configuration>) -> tempfile::TempDir {
let dir = tempfile::tempdir().unwrap();
let tmpname = dir.path().display().to_string();
let tmpname = dir.path().to_path_buf();
let cfg = match configuration {
Some(c) => c,
@ -74,7 +75,7 @@ fn validate_against_schema() {
// Create a custom configuration to use a validating uploader.
let dir = tempfile::tempdir().unwrap();
let tmpname = dir.path().display().to_string();
let tmpname = dir.path().to_path_buf();
let cfg = Configuration {
data_path: tmpname,
@ -88,8 +89,62 @@ fn validate_against_schema() {
};
let _ = new_glean(Some(cfg));
// Define a new ping and submit it.
const PING_NAME: &str = "test-ping";
// Test each of the metric types, just for basic smoke testing against the
// schema
// TODO: 1695762 Test all of the metric types against the schema from Rust
let rate_metric: RateMetric = RateMetric::new(CommonMetricData {
category: "test".into(),
name: "rate".into(),
send_in_pings: vec![PING_NAME.into()],
..Default::default()
});
rate_metric.add_to_numerator(1);
rate_metric.add_to_denominator(1);
let numerator_metric1: NumeratorMetric = NumeratorMetric::new(CommonMetricData {
category: "test".into(),
name: "num1".into(),
send_in_pings: vec![PING_NAME.into()],
..Default::default()
});
let numerator_metric2: NumeratorMetric = NumeratorMetric::new(CommonMetricData {
category: "test".into(),
name: "num2".into(),
send_in_pings: vec![PING_NAME.into()],
..Default::default()
});
let denominator_metric: DenominatorMetric = DenominatorMetric::new(
CommonMetricData {
category: "test".into(),
name: "den".into(),
send_in_pings: vec![PING_NAME.into()],
..Default::default()
},
vec![
CommonMetricData {
category: "test".into(),
name: "num1".into(),
send_in_pings: vec![PING_NAME.into()],
..Default::default()
},
CommonMetricData {
category: "test".into(),
name: "num2".into(),
send_in_pings: vec![PING_NAME.into()],
..Default::default()
},
],
);
numerator_metric1.add_to_numerator(1);
numerator_metric2.add_to_numerator(2);
denominator_metric.add(3);
// Define a new ping and submit it.
let custom_ping = glean::private::PingType::new(PING_NAME, true, true, vec![]);
custom_ping.submit(None);

2
third_party/rust/glean/tests/simple.rs vendored
View File

@ -57,7 +57,7 @@ fn simple_lifecycle() {
// Create a custom configuration to use a validating uploader.
let dir = tempfile::tempdir().unwrap();
let tmpname = dir.path().display().to_string();
let tmpname = dir.path().to_path_buf();
let cfg = Configuration {
data_path: tmpname,

1
third_party/rust/whatsys/.cargo-checksum.json vendored Normal file
View File

@ -0,0 +1 @@
{"files":{"Cargo.lock":"e21f7407732f9c59e73b4357d6475a600fff771c31ee128a1bfba3c71ef517e5","Cargo.toml":"6a04f6499b4d6042abc08c8c75f0893f87fb3e0ec82704c2d210d8d88a8c31c7","LICENSE":"843a95a81ac803d966c391ebca1898976c77d13f992bde7c488f4731160acc24","README.md":"e403b6346359d4828e57d0ef1896851cdf079a6b6319b523d51f142e4d36f2c2","build.rs":"11770481dd1c1489b416b454b2bfee5d73cb97426a63fe830f5e9ad01fee9a62","c/windows.c":"9c899c063c43b3708f8ddb0fca13434758ed4bc1da5cf56f12e44b1839375d98","examples/version.rs":"d622c1cb2c2616a9f9d355e81d51e310d828ea8c651a7e6099334d03097c68eb","src/apple.rs":"60ef2e17c925991b7578278bea8098c7e1b3680e73ac6c729148608f499031f9","src/fallback.rs":"98f08ebb87778421a86e568f526b053818e07b5701216deca743ded6b0e8a1bb","src/lib.rs":"e337758f7568a562cc91837831567d8d62dd6200de4d289c48b9306737108c68","src/linux.rs":"4fdb46384c0a81a4d0d82effecf8938e69f096255aa7eee1c2a4ab9b462178b5","src/windows.rs":"8f645308435a5fc3e83b3e73e5760eabfb653fb3600f0a3d2a2f3d7f9f089861"},"package":"c24fff5aa1e0973964ba23a995e8b10fa2cdeae507e0cbbbd36f8403242a765d"}

28
third_party/rust/whatsys/Cargo.lock generated vendored Normal file
View File

@ -0,0 +1,28 @@
# This file is automatically @generated by Cargo.
# It is not intended for manual editing.
[[package]]
name = "cc"
version = "1.0.59"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "66120af515773fb005778dc07c261bd201ec8ce50bd6e7144c927753fe013381"
[[package]]
name = "cfg-if"
version = "1.0.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "baf1de4339761588bc0619e3cbc0120ee582ebb74b53b4efbf79117bd2da40fd"
[[package]]
name = "libc"
version = "0.2.74"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a2f02823cf78b754822df5f7f268fb59822e7296276d3e069d8e8cb26a14bd10"
[[package]]
name = "whatsys"
version = "0.1.2"
dependencies = [
"cc",
"cfg-if",
"libc",
]

30
third_party/rust/whatsys/Cargo.toml vendored Normal file
View File

@ -0,0 +1,30 @@
# THIS FILE IS AUTOMATICALLY GENERATED BY CARGO
#
# When uploading crates to the registry Cargo will automatically
# "normalize" Cargo.toml files for maximal compatibility
# with all versions of Cargo and also rewrite `path` dependencies
# to registry (e.g., crates.io) dependencies
#
# If you believe there's an error in this file please file an
# issue against the rust-lang/cargo repository. If you're
# editing this file be aware that the upstream Cargo.toml
# will likely look very different (and much more reasonable)
[package]
edition = "2018"
name = "whatsys"
version = "0.1.2"
authors = ["Jan-Erik Rediger <janerik@fnordig.de>"]
description = "Determine the kernel version"
license = "MIT"
repository = "https://github.com/badboy/whatsys"
[package.metadata.docs.rs]
default-target = "x86_64-unknown-linux-gnu"
targets = ["x86_64-unknown-linux-gnu", "x86_64-apple-darwin", "x86_64-pc-windows-msvc"]
[dependencies.cfg-if]
version = "1.0"
[dependencies.libc]
version = "0.2"
[build-dependencies.cc]
version = "1.0"

21
third_party/rust/whatsys/LICENSE vendored Normal file
View File

@ -0,0 +1,21 @@
The MIT License (MIT)
Copyright (c) 2021 Jan-Erik Rediger
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.

Some files were not shown because too many files have changed in this diff Show More