BishopFox/cirro
1
0 Fork 0
mirror of https://github.com/BishopFox/cirro.git synced 2026-11-13 13:21:30 -10:00
Code Issues Packages Projects Releases 1 Wiki Activity

Compare commits

base: BishopFox/cirro:main
BishopFox/cirro:main BishopFox/cirro:fix/macos-release-runner BishopFox/cirro:fix/publicip-merge BishopFox/cirro:gh-pages
BishopFox/cirro:v1.4.0 BishopFox/cirro:v1.3.0 BishopFox/cirro:v1.2.1 BishopFox/cirro:v1.2.0 BishopFox/cirro:v1.1.0 BishopFox/cirro:v1.0.0 BishopFox/cirro:v0.6.0 BishopFox/cirro:v0.3.5 BishopFox/cirro:v0.3.4 BishopFox/cirro:v0.3.3 BishopFox/cirro:v0.3.2 BishopFox/cirro:v0.3.1 BishopFox/cirro:v0.3.0 BishopFox/cirro:v0.2.0 BishopFox/cirro:v0.1.1 BishopFox/cirro:v0.1.0
..
compare: BishopFox/cirro:v0.1.0
BishopFox/cirro:main BishopFox/cirro:fix/macos-release-runner BishopFox/cirro:fix/publicip-merge BishopFox/cirro:gh-pages
BishopFox/cirro:v1.4.0 BishopFox/cirro:v1.3.0 BishopFox/cirro:v1.2.1 BishopFox/cirro:v1.2.0 BishopFox/cirro:v1.1.0 BishopFox/cirro:v1.0.0 BishopFox/cirro:v0.6.0 BishopFox/cirro:v0.3.5 BishopFox/cirro:v0.3.4 BishopFox/cirro:v0.3.3 BishopFox/cirro:v0.3.2 BishopFox/cirro:v0.3.1 BishopFox/cirro:v0.3.0 BishopFox/cirro:v0.2.0 BishopFox/cirro:v0.1.1 BishopFox/cirro:v0.1.0
9 Commits
main .. v0.1.0
Author SHA1 Message Date
Leron Gray e6c0e29584 README cleanup 2025-07-23 17:37:42 -05:00
Leron Gray 043f24f9be add dist build 2025-07-23 17:17:50 -05:00
Leron Gray c1ec9c6f85 neodash and other updates 2025-07-23 15:48:02 -05:00
Leron Gray 80b0851906 fix network interface, subnet, etc 2025-07-18 00:22:39 -05:00
Leron Gray c981011bae update post process queries 2025-07-17 23:01:49 -05:00
Leron Gray 7705a7e9dc lowercase id fields 2025-07-17 21:08:42 -05:00
Leron Gray 2feff0c07d remove .vscode 2025-07-17 13:18:42 -05:00
Leron Gray b5c7a5ca0f init rust rewrite 2025-07-17 13:16:57 -05:00
Leron Gray fca44dec8b init rust rewrite 2025-07-17 13:15:37 -05:00
512 changed files with 6307 additions and 32599 deletions
Expand all files Collapse all files

No files matched your search

.github/CODEOWNERS
-1
View File
@@ -1 +0,0 @@
* @BishopFox/bf-red-team
.github/skills/cirro-schema-docs/SKILL.md
-153
View File
@@ -1,153 +0,0 @@
---
name: cirro-schema-docs
description: Synchronize Cirro analysis documentation and mkdocs navigation with schema constants and graph config YAML files.
---
# Cirro Schema Docs Skill
## Purpose
Use this skill when Cirro graph config files or schema constants change and documentation must be updated to match.
Primary sources of truth:
- `src/graph/config/constants.yaml`
- `src/graph/config/**/*.tera.yaml`
Primary documentation targets:
- `docs/analysis/**`
- `zensical.toml` nav entries
## When To Use
Trigger examples:
- sync cirro docs
- update node and edge documentation
- ensure docs match cirro config
- add missing cirro documentation
## Required Workflow
1. Read constants.
- Parse `REL.*` and `LABELS.*` from `src/graph/config/constants.yaml`.
2. Inventory existing docs.
- `docs/analysis/graph-nodes/*.md`
- `docs/analysis/arm-nodes/**/*.md`
- `docs/analysis/additional-nodes/*.md`
- `docs/analysis/tailscale-nodes/*.md`
- `docs/analysis/edges/*.md`
- `docs/analysis/post-processing/*.md`
- Also inspect the `nav:` section in `zensical.toml`.
3. Scan config YAML.
- `src/graph/config/azure/arm/**/*.tera.yaml`
- `src/graph/config/azure/msgraph/*.tera.yaml`
- `src/graph/config/post_processing/*.tera.yaml`
- `src/graph/config/tailscale/**/*.tera.yaml`
- For each file, extract `name`, `label`, `properties`, and all relationship usage from `cypher`.
4. Cross-reference and find gaps.
- Missing node docs from `LABELS.*`
- Missing edge docs from `REL.*` (convert `UPPER_SNAKE` to `lower-kebab`)
- Missing post-processing docs from files in `src/graph/config/post_processing/`
- Missing `zensical.toml` nav entries for all docs under `docs/analysis/`
5. Create or update docs.
- Prefer minimal edits and keep style consistent with nearby files.
- Create missing directories before creating files.
6. Update `zensical.toml` nav.
- Place docs in the correct analysis sections.
- Keep entries alphabetically sorted within each section.
- Preserve existing indentation and structure.
7. Ask for confirmation on unused constants.
- If a `LABELS.*` or `REL.*` constant is not referenced by any `.tera.yaml`, ask whether to document it or treat it as deprecated.
## Mapping Rules
- ARM labels -> `docs/analysis/arm-nodes/<provider>/<node-name>.md`
- Graph labels -> `docs/analysis/graph-nodes/graph-<name>.md`
- Additional labels -> `docs/analysis/additional-nodes/<name>.md`
- Tailscale labels -> `docs/analysis/tailscale-nodes/ts-<name>.md`
- Edge docs -> `docs/analysis/edges/<rel-name>.md`
- Post-processing docs -> `docs/analysis/post-processing/<step-name>.md`
## Content Templates
Use these as baseline structures.
### Node Doc
~~~markdown
# NodeDisplayName
Description of what the node represents.
**Labels:** `:ArmResource:NodeLabel`
**Properties:**
- `id` - Resource ID (primary key)
- `propertyName` - Description
## Relationships
### Outgoing
- **NodeLabel** -> `REL_NAME` -> **TargetLabel** - Description
## Examples
```cypher
MATCH (n:NodeLabel)
RETURN n.name, n.property
```
~~~
For Graph nodes, use `:GraphObject:NodeLabel` in `Labels`.
### Edge Doc
~~~markdown
# REL_NAME
Description of the relationship.
## Usage
- **SourceLabel** -> `REL_NAME` -> **TargetLabel** - Description
## Properties
No additional properties on the relationship.
## Examples
```cypher
MATCH (a)-[r:REL_NAME]->(b)
RETURN a.name, b.name
```
~~~
### Post-Processing Doc
~~~markdown
# Step Display Name
Description of what this post-processing step does.
**Priority:** N
## Details
Explanation of the logic.
## Cypher
```cypher
// Cleaned-up cypher from YAML
```
~~~
## Guardrails
- Treat YAML config and constants as source of truth.
- Do not invent labels or relationships not present in source files.
- Keep edits deterministic, concise, and sorted where applicable.
- Preserve existing docs style and naming conventions.
.github/workflows/docs.yml
-32
View File
@@ -1,32 +0,0 @@
name: Documentation
on:
push:
paths:
- "docs/**"
- "zensical.toml"
- ".github/workflows/docs.yml"
branches:
- main
permissions:
contents: read
pages: write
id-token: write
jobs:
deploy:
environment:
name: github-pages
url: ${{ steps.deployment.outputs.page_url }}
runs-on: ubuntu-latest
steps:
- uses: actions/configure-pages@v6
- uses: actions/checkout@v6
- uses: actions/setup-python@v6
with:
python-version: 3.x
- run: pip install zensical
- run: zensical build --clean
- uses: actions/upload-pages-artifact@v5
with:
path: site
- uses: actions/deploy-pages@v5
id: deployment
.github/workflows/release.yml
+21 -20
View File
@@ -1,6 +1,7 @@
# This file was autogenerated by dist: https://axodotdev.github.io/cargo-dist
# This file was autogenerated by dist: https://github.com/astral-sh/cargo-dist
#
# Copyright 2022-2024, axodotdev
# Copyright 2025 Astral Software Inc.
# SPDX-License-Identifier: MIT or Apache-2.0
#
# CI that:
@@ -56,7 +57,7 @@ jobs:
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683
with:
persist-credentials: false
submodules: recursive
@@ -64,9 +65,9 @@ jobs:
# we specify bash to get pipefail; it guards against the `curl` command
# failing. otherwise `sh` won't catch that `curl` returned non-0
shell: bash
run: "curl --proto '=https' --tlsv1.2 -LsSf https://github.com/axodotdev/cargo-dist/releases/download/v0.30.2/cargo-dist-installer.sh | sh"
run: "curl --proto '=https' --tlsv1.2 -LsSf https://github.com/astral-sh/cargo-dist/releases/download/v0.28.6/cargo-dist-installer.sh | sh"
- name: Cache dist
uses: actions/upload-artifact@v4
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02
with:
name: cargo-dist-cache
path: ~/.cargo/bin/dist
@@ -82,7 +83,7 @@ jobs:
cat plan-dist-manifest.json
echo "manifest=$(jq -c "." plan-dist-manifest.json)" >> "$GITHUB_OUTPUT"
- name: "Upload dist-manifest.json"
uses: actions/upload-artifact@v4
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02
with:
name: artifacts-plan-dist-manifest
path: plan-dist-manifest.json
@@ -116,7 +117,7 @@ jobs:
- name: enable windows longpaths
run: |
git config --global core.longpaths true
- uses: actions/checkout@v4
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683
with:
persist-credentials: false
submodules: recursive
@@ -131,7 +132,7 @@ jobs:
run: ${{ matrix.install_dist.run }}
# Get the dist-manifest
- name: Fetch local artifacts
uses: actions/download-artifact@v4
uses: actions/download-artifact@95815c38cf2ff2164869cbab79da8d1f422bc89e
with:
pattern: artifacts-*
path: target/distrib/
@@ -158,7 +159,7 @@ jobs:
cp dist-manifest.json "$BUILD_MANIFEST_NAME"
- name: "Upload artifacts"
uses: actions/upload-artifact@v4
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02
with:
name: artifacts-build-local-${{ join(matrix.targets, '_') }}
path: |
@@ -175,19 +176,19 @@ jobs:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
BUILD_MANIFEST_NAME: target/distrib/global-dist-manifest.json
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683
with:
persist-credentials: false
submodules: recursive
- name: Install cached dist
uses: actions/download-artifact@v4
uses: actions/download-artifact@95815c38cf2ff2164869cbab79da8d1f422bc89e
with:
name: cargo-dist-cache
path: ~/.cargo/bin/
- run: chmod +x ~/.cargo/bin/dist
# Get all the local artifacts for the global tasks to use (for e.g. checksums)
- name: Fetch local artifacts
uses: actions/download-artifact@v4
uses: actions/download-artifact@95815c38cf2ff2164869cbab79da8d1f422bc89e
with:
pattern: artifacts-*
path: target/distrib/
@@ -205,7 +206,7 @@ jobs:
cp dist-manifest.json "$BUILD_MANIFEST_NAME"
- name: "Upload artifacts"
uses: actions/upload-artifact@v4
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02
with:
name: artifacts-build-global
path: |
@@ -217,27 +218,27 @@ jobs:
- plan
- build-local-artifacts
- build-global-artifacts
# Only run if we're "publishing", and only if plan, local and global didn't fail (skipped is fine)
if: ${{ always() && needs.plan.result == 'success' && needs.plan.outputs.publishing == 'true' && (needs.build-global-artifacts.result == 'skipped' || needs.build-global-artifacts.result == 'success') && (needs.build-local-artifacts.result == 'skipped' || needs.build-local-artifacts.result == 'success') }}
# Only run if we're "publishing", and only if local and global didn't fail (skipped is fine)
if: ${{ always() && needs.plan.outputs.publishing == 'true' && (needs.build-global-artifacts.result == 'skipped' || needs.build-global-artifacts.result == 'success') && (needs.build-local-artifacts.result == 'skipped' || needs.build-local-artifacts.result == 'success') }}
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
runs-on: "ubuntu-22.04"
outputs:
val: ${{ steps.host.outputs.manifest }}
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683
with:
persist-credentials: false
submodules: recursive
- name: Install cached dist
uses: actions/download-artifact@v4
uses: actions/download-artifact@95815c38cf2ff2164869cbab79da8d1f422bc89e
with:
name: cargo-dist-cache
path: ~/.cargo/bin/
- run: chmod +x ~/.cargo/bin/dist
# Fetch artifacts from scratch-storage
- name: Fetch artifacts
uses: actions/download-artifact@v4
uses: actions/download-artifact@95815c38cf2ff2164869cbab79da8d1f422bc89e
with:
pattern: artifacts-*
path: target/distrib/
@@ -250,14 +251,14 @@ jobs:
cat dist-manifest.json
echo "manifest=$(jq -c "." dist-manifest.json)" >> "$GITHUB_OUTPUT"
- name: "Upload dist-manifest.json"
uses: actions/upload-artifact@v4
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02
with:
# Overwrite the previous copy
name: artifacts-dist-manifest
path: dist-manifest.json
# Create a GitHub Release while uploading all files to it
- name: "Download GitHub Artifacts"
uses: actions/download-artifact@v4
uses: actions/download-artifact@95815c38cf2ff2164869cbab79da8d1f422bc89e
with:
pattern: artifacts-*
path: artifacts
@@ -290,7 +291,7 @@ jobs:
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683
with:
persist-credentials: false
submodules: recursive
.gitignore
+2 -7
View File
@@ -9,7 +9,7 @@ target/
# Remove Cargo.lock from gitignore if creating an executable, leave it for libraries
# More information here https://doc.rust-lang.org/cargo/guide/cargo-toml-vs-cargo-lock.html
# Cargo.lock
Cargo.lock
# These are backup files generated by rustfmt
**/*.rs.bk
@@ -46,11 +46,6 @@ rust-project.json
*.db
*.db-wal
*.db-shm
cirro_*.json
*_export
*.bin
neo4j/
site/
*.csv
demo/
neo4j/
.vscode/settings.json
Vendored
-18
View File
@@ -1,18 +0,0 @@
{
"yaml.schemas": {
"https://squidfunk.github.io/mkdocs-material/schema.json": "mkdocs.yml"
},
"yaml.customTags": [
"!ENV scalar",
"!ENV sequence",
"!relative scalar",
"tag:yaml.org,2002:python/name:material.extensions.emoji.to_svg",
"tag:yaml.org,2002:python/name:material.extensions.emoji.twemoji",
"tag:yaml.org,2002:python/name:pymdownx.superfences.fence_code_format",
"tag:yaml.org,2002:python/object/apply:pymdownx.slugs.slugify mapping"
],
"python-envs.defaultEnvManager": "ms-python.python:system",
"python-envs.pythonProjects": [],
"rust-analyzer.cargo.target": "x86_64-pc-windows-gnu",
"git.addAICoAuthor": "off"
}
CHANGELOG.md
-93
View File
@@ -1,93 +0,0 @@
# Changelog
All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
## [1.4.0] - 2026-11-05
### Added
- Restricted guest Microsoft Graph collection through service principal relationships with `--restricted-guest`.
- Custom application ID seeds with `--restricted-appids` and an optional Microsoft application catalog with `--restricted-guest-msapps`.
## [1.3.0] - 2026-09-22
### Added
- Add user manager and password change datetime by @daddycocoaman in [#36](https://github.com/bishopfox/cirro/pull/36)
- Add odata types and agent id relations by @daddycocoaman in [#35](https://github.com/bishopfox/cirro/pull/35)
### Changed
- Process createdByAppId by @daddycocoaman in [#34](https://github.com/bishopfox/cirro/pull/34)
### Fixed
- Use correct base64 decoding for tokens by @daddycocoaman in [#38](https://github.com/bishopfox/cirro/pull/38)
- Check site serverfarmid for null by @daddycocoaman in [#37](https://github.com/bishopfox/cirro/pull/37)
## [1.2.1] - 2026-06-01
### Fixed
- Fix merges in loadbalancer by @daddycocoaman in [#33](https://github.com/bishopfox/cirro/pull/33)
## [1.2.0] - 2026-05-12
### Added
- Add network LoadBalancer nodes and VirtualHub by @daddycocoaman in [#30](https://github.com/bishopfox/cirro/pull/30)
- Add subject for FIC collection by @daddycocoaman in [#28](https://github.com/bishopfox/cirro/pull/28)
- Add KustoCluster node by @daddycocoaman in [#25](https://github.com/bishopfox/cirro/pull/25)
- Add SyntexDocumentProcessor by @daddycocoaman in [#24](https://github.com/bishopfox/cirro/pull/24)
- Add azure stack nodes and edges by @daddycocoaman in [#23](https://github.com/bishopfox/cirro/pull/23)
- Add oracle autonomous db by @daddycocoaman in [#22](https://github.com/bishopfox/cirro/pull/22)
- Add multiple Hybrid nodes and parity with virtual machines by @daddycocoaman in [#21](https://github.com/bishopfox/cirro/pull/21)
- Add ManagedHSM by @daddycocoaman in [#15](https://github.com/bishopfox/cirro/pull/15)
- Add AnalysisServiceServer and AnalysisServiceFirewallRule by @daddycocoaman in [#14](https://github.com/bishopfox/cirro/pull/14)
- Add composite index and constraints syntax by @daddycocoaman in [#13](https://github.com/bishopfox/cirro/pull/13)
- Add GithubNetworkSettings by @daddycocoaman in [#12](https://github.com/bishopfox/cirro/pull/12)
- Add dry run switch for ingest by @daddycocoaman in [#11](https://github.com/bishopfox/cirro/pull/11)
- Add argument validation to applicable arguments by @daddycocoaman in [#10](https://github.com/bishopfox/cirro/pull/10)
### Changed
- Update docs by @daddycocoaman in [#31](https://github.com/bishopfox/cirro/pull/31)
- Update cypher to Cypher 25 syntax by @daddycocoaman in [#27](https://github.com/bishopfox/cirro/pull/27)
- Update netapp nodes by @daddycocoaman in [#26](https://github.com/bishopfox/cirro/pull/26)
- Update docs skill for zensical by @daddycocoaman in [#20](https://github.com/bishopfox/cirro/pull/20)
- Migrate docs to zensical by @daddycocoaman in [#19](https://github.com/bishopfox/cirro/pull/19)
- Turn off git AI CoAuthor setting by @daddycocoaman in [#16](https://github.com/bishopfox/cirro/pull/16)
- Update restore point changes by @daddycocoaman in [#9](https://github.com/bishopfox/cirro/pull/9)
- Improve restorepointcollections by @daddycocoaman in [#8](https://github.com/bishopfox/cirro/pull/8)
### Fixed
- Re-add WITH before CALLs in Cypher and minor bugs by @daddycocoaman in [#29](https://github.com/bishopfox/cirro/pull/29)
- Change generic resource MI checks to CONTAINS by @daddycocoaman in [#17](https://github.com/bishopfox/cirro/pull/17)
## [1.1.0] - 2026-04-13
### Added
- Add NetworkWatcher, KeyVaultCertificate, KeyVaultCertificateValue by @daddycocoaman in [#3](https://github.com/BishopFox/cirro/pull/3)
- Add CODEOWNERS for bf-red-team (CLD-10728) by @grobbinsBF
### Changed
- Bump version to 1.1.0
- Sync docs with schemas by @daddycocoaman
### New Contributors
* @grobbinsBF made their first contribution in [#2](https://github.com/BishopFox/cirro/pull/2)
## [1.0.0] - 2026-03-31
### Initial Release
* Initial release by @daddycocoaman
[1.1.0]: https://github.com/BishopFox/cirro/compare/v1.0.0...v1.1.0
<!-- generated by git-cliff -->
Cargo.lock
Generated
-3782
View File
File diff suppressed because it is too large. Load diff
Cargo.toml
+11 -88
View File
@@ -1,92 +1,8 @@
[package]
edition = "2024"
name = "cirro"
repository = "https://github.com/bishopfox/cirro"
version = "1.4.0"
[features]
azure = ["collector"]
collector = [
"dep:async-trait",
"dep:base64",
"dep:chrono",
"dep:dashmap",
"dep:futures",
"dep:jsonwebtoken",
"dep:once_cell",
"dep:pem",
"dep:regex",
"dep:reqwest",
"dep:rsa",
"dep:sha1",
"dep:which",
]
default = ["azure", "tailscale", "graph"]
graph = [
"dep:neo4rs",
"dep:rayon",
"dep:rust-embed",
"dep:serde_yaml",
"dep:strum",
"dep:strum_macros",
"dep:tera",
"dep:url",
]
tailscale = ["collector"]
[dependencies]
# Common dependencies (shared between collection and graph)
clap = { version = "4.5.54", features = ["cargo", "derive"] }
colored = "3.0.0"
fern = { version = "0.7.1", features = ["chrono"] }
log = "0.4.27"
rusqlite = { version = "0.37.0", features = ["bundled"] }
serde = { version = "1.0.219", features = ["serde_derive"] }
serde_json = "1.0.140"
thiserror = "2.0.16"
tokio = { version = "1.46.1", features = ["full"] }
uuid = { version = "1.17.0", features = ["v4"] }
# Collection-specific dependencies (from cirro, optional)
async-trait = { version = "0.1.78", optional = true }
base64 = { version = "0.22.1", optional = true }
chrono = { version = "0.4.41", optional = true }
dashmap = { version = "5.5.3", optional = true }
futures = { version = "0.3.31", optional = true }
jsonwebtoken = { version = "10.3.0", features = [
"use_pem",
"aws_lc_rs",
], optional = true }
once_cell = { version = "1.19.0", optional = true }
pem = { version = "3.0.5", optional = true }
regex = { version = "1.12.3", optional = true }
reqwest = { version = "0.13.2", features = [
"json",
"query",
"form",
"gzip",
"brotli",
], optional = true }
rsa = { version = "0.9.8", features = ["sha1", "sha2"], optional = true }
sha1 = { version = "0.10.6", optional = true }
which = { version = "8.0.0", optional = true }
# Graph-specific dependencies (from cirro-graph, optional)
neo4rs = { version = "0.8.0", features = ["json"], optional = true }
rayon = { version = "1.10", optional = true }
rust-embed = { version = "8.11.0", features = ["compression"], optional = true }
serde_yaml = { version = "0.9.34", optional = true }
strum = { version = "0.27.1", optional = true }
strum_macros = { version = "0.27.1", optional = true }
tera = { version = "1.20.1", optional = true }
url = { version = "2.5.7", optional = true }
[profile.dev]
debug = "line-tables-only"
opt-level = 0
[workspace]
members = ["cirro", "cirro-ingest"]
resolver = "3"
[profile.release]
codegen-units = 1
lto = "thin"
opt-level = "s"
panic = 'abort'
@@ -94,5 +10,12 @@ strip = true
# The profile that 'dist' will build with
[profile.dist]
codegen-units = 1
debug = false
incremental = false
inherits = "release"
lto = "thin"
lto = "fat"
opt-level = 3
overflow-checks = false
panic = "abort"
strip = true
LICENSE
+202 -595
View File
@@ -1,595 +1,202 @@
GNU General Public License
==========================
_Version 3, 29 June 2007_
_Copyright © 2007 Free Software Foundation, Inc. &lt;<http://fsf.org/>&gt;_
Everyone is permitted to copy and distribute verbatim copies of this license
document, but changing it is not allowed.
## Preamble
The GNU General Public License is a free, copyleft license for software and other
kinds of works.
The licenses for most software and other practical works are designed to take away
your freedom to share and change the works. By contrast, the GNU General Public
License is intended to guarantee your freedom to share and change all versions of a
program--to make sure it remains free software for all its users. We, the Free
Software Foundation, use the GNU General Public License for most of our software; it
applies also to any other work released this way by its authors. You can apply it to
your programs, too.
When we speak of free software, we are referring to freedom, not price. Our General
Public Licenses are designed to make sure that you have the freedom to distribute
copies of free software (and charge for them if you wish), that you receive source
code or can get it if you want it, that you can change the software or use pieces of
it in new free programs, and that you know you can do these things.
To protect your rights, we need to prevent others from denying you these rights or
asking you to surrender the rights. Therefore, you have certain responsibilities if
you distribute copies of the software, or if you modify it: responsibilities to
respect the freedom of others.
For example, if you distribute copies of such a program, whether gratis or for a fee,
you must pass on to the recipients the same freedoms that you received. You must make
sure that they, too, receive or can get the source code. And you must show them these
terms so they know their rights.
Developers that use the GNU GPL protect your rights with two steps: **(1)** assert
copyright on the software, and **(2)** offer you this License giving you legal permission
to copy, distribute and/or modify it.
For the developers' and authors' protection, the GPL clearly explains that there is
no warranty for this free software. For both users' and authors' sake, the GPL
requires that modified versions be marked as changed, so that their problems will not
be attributed erroneously to authors of previous versions.
Some devices are designed to deny users access to install or run modified versions of
the software inside them, although the manufacturer can do so. This is fundamentally
incompatible with the aim of protecting users' freedom to change the software. The
systematic pattern of such abuse occurs in the area of products for individuals to
use, which is precisely where it is most unacceptable. Therefore, we have designed
this version of the GPL to prohibit the practice for those products. If such problems
arise substantially in other domains, we stand ready to extend this provision to
those domains in future versions of the GPL, as needed to protect the freedom of
users.
Finally, every program is threatened constantly by software patents. States should
not allow patents to restrict development and use of software on general-purpose
computers, but in those that do, we wish to avoid the special danger that patents
applied to a free program could make it effectively proprietary. To prevent this, the
GPL assures that patents cannot be used to render the program non-free.
The precise terms and conditions for copying, distribution and modification follow.
## TERMS AND CONDITIONS
### 0. Definitions
“This License” refers to version 3 of the GNU General Public License.
“Copyright” also means copyright-like laws that apply to other kinds of
works, such as semiconductor masks.
“The Program” refers to any copyrightable work licensed under this
License. Each licensee is addressed as “you”. “Licensees” and
“recipients” may be individuals or organizations.
To “modify” a work means to copy from or adapt all or part of the work in
a fashion requiring copyright permission, other than the making of an exact copy. The
resulting work is called a “modified version” of the earlier work or a
work “based on” the earlier work.
A “covered work” means either the unmodified Program or a work based on
the Program.
To “propagate” a work means to do anything with it that, without
permission, would make you directly or secondarily liable for infringement under
applicable copyright law, except executing it on a computer or modifying a private
copy. Propagation includes copying, distribution (with or without modification),
making available to the public, and in some countries other activities as well.
To “convey” a work means any kind of propagation that enables other
parties to make or receive copies. Mere interaction with a user through a computer
network, with no transfer of a copy, is not conveying.
An interactive user interface displays “Appropriate Legal Notices” to the
extent that it includes a convenient and prominently visible feature that **(1)**
displays an appropriate copyright notice, and **(2)** tells the user that there is no
warranty for the work (except to the extent that warranties are provided), that
licensees may convey the work under this License, and how to view a copy of this
License. If the interface presents a list of user commands or options, such as a
menu, a prominent item in the list meets this criterion.
### 1. Source Code
The “source code” for a work means the preferred form of the work for
making modifications to it. “Object code” means any non-source form of a
work.
A “Standard Interface” means an interface that either is an official
standard defined by a recognized standards body, or, in the case of interfaces
specified for a particular programming language, one that is widely used among
developers working in that language.
The “System Libraries” of an executable work include anything, other than
the work as a whole, that **(a)** is included in the normal form of packaging a Major
Component, but which is not part of that Major Component, and **(b)** serves only to
enable use of the work with that Major Component, or to implement a Standard
Interface for which an implementation is available to the public in source code form.
A “Major Component”, in this context, means a major essential component
(kernel, window system, and so on) of the specific operating system (if any) on which
the executable work runs, or a compiler used to produce the work, or an object code
interpreter used to run it.
The “Corresponding Source” for a work in object code form means all the
source code needed to generate, install, and (for an executable work) run the object
code and to modify the work, including scripts to control those activities. However,
it does not include the work's System Libraries, or general-purpose tools or
generally available free programs which are used unmodified in performing those
activities but which are not part of the work. For example, Corresponding Source
includes interface definition files associated with source files for the work, and
the source code for shared libraries and dynamically linked subprograms that the work
is specifically designed to require, such as by intimate data communication or
control flow between those subprograms and other parts of the work.
The Corresponding Source need not include anything that users can regenerate
automatically from other parts of the Corresponding Source.
The Corresponding Source for a work in source code form is that same work.
### 2. Basic Permissions
All rights granted under this License are granted for the term of copyright on the
Program, and are irrevocable provided the stated conditions are met. This License
explicitly affirms your unlimited permission to run the unmodified Program. The
output from running a covered work is covered by this License only if the output,
given its content, constitutes a covered work. This License acknowledges your rights
of fair use or other equivalent, as provided by copyright law.
You may make, run and propagate covered works that you do not convey, without
conditions so long as your license otherwise remains in force. You may convey covered
works to others for the sole purpose of having them make modifications exclusively
for you, or provide you with facilities for running those works, provided that you
comply with the terms of this License in conveying all material for which you do not
control copyright. Those thus making or running the covered works for you must do so
exclusively on your behalf, under your direction and control, on terms that prohibit
them from making any copies of your copyrighted material outside their relationship
with you.
Conveying under any other circumstances is permitted solely under the conditions
stated below. Sublicensing is not allowed; section 10 makes it unnecessary.
### 3. Protecting Users' Legal Rights From Anti-Circumvention Law
No covered work shall be deemed part of an effective technological measure under any
applicable law fulfilling obligations under article 11 of the WIPO copyright treaty
adopted on 20 December 1996, or similar laws prohibiting or restricting circumvention
of such measures.
When you convey a covered work, you waive any legal power to forbid circumvention of
technological measures to the extent such circumvention is effected by exercising
rights under this License with respect to the covered work, and you disclaim any
intention to limit operation or modification of the work as a means of enforcing,
against the work's users, your or third parties' legal rights to forbid circumvention
of technological measures.
### 4. Conveying Verbatim Copies
You may convey verbatim copies of the Program's source code as you receive it, in any
medium, provided that you conspicuously and appropriately publish on each copy an
appropriate copyright notice; keep intact all notices stating that this License and
any non-permissive terms added in accord with section 7 apply to the code; keep
intact all notices of the absence of any warranty; and give all recipients a copy of
this License along with the Program.
You may charge any price or no price for each copy that you convey, and you may offer
support or warranty protection for a fee.
### 5. Conveying Modified Source Versions
You may convey a work based on the Program, or the modifications to produce it from
the Program, in the form of source code under the terms of section 4, provided that
you also meet all of these conditions:
* **a)** The work must carry prominent notices stating that you modified it, and giving a
relevant date.
* **b)** The work must carry prominent notices stating that it is released under this
License and any conditions added under section 7. This requirement modifies the
requirement in section 4 to “keep intact all notices”.
* **c)** You must license the entire work, as a whole, under this License to anyone who
comes into possession of a copy. This License will therefore apply, along with any
applicable section 7 additional terms, to the whole of the work, and all its parts,
regardless of how they are packaged. This License gives no permission to license the
work in any other way, but it does not invalidate such permission if you have
separately received it.
* **d)** If the work has interactive user interfaces, each must display Appropriate Legal
Notices; however, if the Program has interactive interfaces that do not display
Appropriate Legal Notices, your work need not make them do so.
A compilation of a covered work with other separate and independent works, which are
not by their nature extensions of the covered work, and which are not combined with
it such as to form a larger program, in or on a volume of a storage or distribution
medium, is called an “aggregate” if the compilation and its resulting
copyright are not used to limit the access or legal rights of the compilation's users
beyond what the individual works permit. Inclusion of a covered work in an aggregate
does not cause this License to apply to the other parts of the aggregate.
### 6. Conveying Non-Source Forms
You may convey a covered work in object code form under the terms of sections 4 and
5, provided that you also convey the machine-readable Corresponding Source under the
terms of this License, in one of these ways:
* **a)** Convey the object code in, or embodied in, a physical product (including a
physical distribution medium), accompanied by the Corresponding Source fixed on a
durable physical medium customarily used for software interchange.
* **b)** Convey the object code in, or embodied in, a physical product (including a
physical distribution medium), accompanied by a written offer, valid for at least
three years and valid for as long as you offer spare parts or customer support for
that product model, to give anyone who possesses the object code either **(1)** a copy of
the Corresponding Source for all the software in the product that is covered by this
License, on a durable physical medium customarily used for software interchange, for
a price no more than your reasonable cost of physically performing this conveying of
source, or **(2)** access to copy the Corresponding Source from a network server at no
charge.
* **c)** Convey individual copies of the object code with a copy of the written offer to
provide the Corresponding Source. This alternative is allowed only occasionally and
noncommercially, and only if you received the object code with such an offer, in
accord with subsection 6b.
* **d)** Convey the object code by offering access from a designated place (gratis or for
a charge), and offer equivalent access to the Corresponding Source in the same way
through the same place at no further charge. You need not require recipients to copy
the Corresponding Source along with the object code. If the place to copy the object
code is a network server, the Corresponding Source may be on a different server
(operated by you or a third party) that supports equivalent copying facilities,
provided you maintain clear directions next to the object code saying where to find
the Corresponding Source. Regardless of what server hosts the Corresponding Source,
you remain obligated to ensure that it is available for as long as needed to satisfy
these requirements.
* **e)** Convey the object code using peer-to-peer transmission, provided you inform
other peers where the object code and Corresponding Source of the work are being
offered to the general public at no charge under subsection 6d.
A separable portion of the object code, whose source code is excluded from the
Corresponding Source as a System Library, need not be included in conveying the
object code work.
A “User Product” is either **(1)** a “consumer product”, which
means any tangible personal property which is normally used for personal, family, or
household purposes, or **(2)** anything designed or sold for incorporation into a
dwelling. In determining whether a product is a consumer product, doubtful cases
shall be resolved in favor of coverage. For a particular product received by a
particular user, “normally used” refers to a typical or common use of
that class of product, regardless of the status of the particular user or of the way
in which the particular user actually uses, or expects or is expected to use, the
product. A product is a consumer product regardless of whether the product has
substantial commercial, industrial or non-consumer uses, unless such uses represent
the only significant mode of use of the product.
“Installation Information” for a User Product means any methods,
procedures, authorization keys, or other information required to install and execute
modified versions of a covered work in that User Product from a modified version of
its Corresponding Source. The information must suffice to ensure that the continued
functioning of the modified object code is in no case prevented or interfered with
solely because modification has been made.
If you convey an object code work under this section in, or with, or specifically for
use in, a User Product, and the conveying occurs as part of a transaction in which
the right of possession and use of the User Product is transferred to the recipient
in perpetuity or for a fixed term (regardless of how the transaction is
characterized), the Corresponding Source conveyed under this section must be
accompanied by the Installation Information. But this requirement does not apply if
neither you nor any third party retains the ability to install modified object code
on the User Product (for example, the work has been installed in ROM).
The requirement to provide Installation Information does not include a requirement to
continue to provide support service, warranty, or updates for a work that has been
modified or installed by the recipient, or for the User Product in which it has been
modified or installed. Access to a network may be denied when the modification itself
materially and adversely affects the operation of the network or violates the rules
and protocols for communication across the network.
Corresponding Source conveyed, and Installation Information provided, in accord with
this section must be in a format that is publicly documented (and with an
implementation available to the public in source code form), and must require no
special password or key for unpacking, reading or copying.
### 7. Additional Terms
“Additional permissions” are terms that supplement the terms of this
License by making exceptions from one or more of its conditions. Additional
permissions that are applicable to the entire Program shall be treated as though they
were included in this License, to the extent that they are valid under applicable
law. If additional permissions apply only to part of the Program, that part may be
used separately under those permissions, but the entire Program remains governed by
this License without regard to the additional permissions.
When you convey a copy of a covered work, you may at your option remove any
additional permissions from that copy, or from any part of it. (Additional
permissions may be written to require their own removal in certain cases when you
modify the work.) You may place additional permissions on material, added by you to a
covered work, for which you have or can give appropriate copyright permission.
Notwithstanding any other provision of this License, for material you add to a
covered work, you may (if authorized by the copyright holders of that material)
supplement the terms of this License with terms:
* **a)** Disclaiming warranty or limiting liability differently from the terms of
sections 15 and 16 of this License; or
* **b)** Requiring preservation of specified reasonable legal notices or author
attributions in that material or in the Appropriate Legal Notices displayed by works
containing it; or
* **c)** Prohibiting misrepresentation of the origin of that material, or requiring that
modified versions of such material be marked in reasonable ways as different from the
original version; or
* **d)** Limiting the use for publicity purposes of names of licensors or authors of the
material; or
* **e)** Declining to grant rights under trademark law for use of some trade names,
trademarks, or service marks; or
* **f)** Requiring indemnification of licensors and authors of that material by anyone
who conveys the material (or modified versions of it) with contractual assumptions of
liability to the recipient, for any liability that these contractual assumptions
directly impose on those licensors and authors.
All other non-permissive additional terms are considered “further
restrictions” within the meaning of section 10. If the Program as you received
it, or any part of it, contains a notice stating that it is governed by this License
along with a term that is a further restriction, you may remove that term. If a
license document contains a further restriction but permits relicensing or conveying
under this License, you may add to a covered work material governed by the terms of
that license document, provided that the further restriction does not survive such
relicensing or conveying.
If you add terms to a covered work in accord with this section, you must place, in
the relevant source files, a statement of the additional terms that apply to those
files, or a notice indicating where to find the applicable terms.
Additional terms, permissive or non-permissive, may be stated in the form of a
separately written license, or stated as exceptions; the above requirements apply
either way.
### 8. Termination
You may not propagate or modify a covered work except as expressly provided under
this License. Any attempt otherwise to propagate or modify it is void, and will
automatically terminate your rights under this License (including any patent licenses
granted under the third paragraph of section 11).
However, if you cease all violation of this License, then your license from a
particular copyright holder is reinstated **(a)** provisionally, unless and until the
copyright holder explicitly and finally terminates your license, and **(b)** permanently,
if the copyright holder fails to notify you of the violation by some reasonable means
prior to 60 days after the cessation.
Moreover, your license from a particular copyright holder is reinstated permanently
if the copyright holder notifies you of the violation by some reasonable means, this
is the first time you have received notice of violation of this License (for any
work) from that copyright holder, and you cure the violation prior to 30 days after
your receipt of the notice.
Termination of your rights under this section does not terminate the licenses of
parties who have received copies or rights from you under this License. If your
rights have been terminated and not permanently reinstated, you do not qualify to
receive new licenses for the same material under section 10.
### 9. Acceptance Not Required for Having Copies
You are not required to accept this License in order to receive or run a copy of the
Program. Ancillary propagation of a covered work occurring solely as a consequence of
using peer-to-peer transmission to receive a copy likewise does not require
acceptance. However, nothing other than this License grants you permission to
propagate or modify any covered work. These actions infringe copyright if you do not
accept this License. Therefore, by modifying or propagating a covered work, you
indicate your acceptance of this License to do so.
### 10. Automatic Licensing of Downstream Recipients
Each time you convey a covered work, the recipient automatically receives a license
from the original licensors, to run, modify and propagate that work, subject to this
License. You are not responsible for enforcing compliance by third parties with this
License.
An “entity transaction” is a transaction transferring control of an
organization, or substantially all assets of one, or subdividing an organization, or
merging organizations. If propagation of a covered work results from an entity
transaction, each party to that transaction who receives a copy of the work also
receives whatever licenses to the work the party's predecessor in interest had or
could give under the previous paragraph, plus a right to possession of the
Corresponding Source of the work from the predecessor in interest, if the predecessor
has it or can get it with reasonable efforts.
You may not impose any further restrictions on the exercise of the rights granted or
affirmed under this License. For example, you may not impose a license fee, royalty,
or other charge for exercise of rights granted under this License, and you may not
initiate litigation (including a cross-claim or counterclaim in a lawsuit) alleging
that any patent claim is infringed by making, using, selling, offering for sale, or
importing the Program or any portion of it.
### 11. Patents
A “contributor” is a copyright holder who authorizes use under this
License of the Program or a work on which the Program is based. The work thus
licensed is called the contributor's “contributor version”.
A contributor's “essential patent claims” are all patent claims owned or
controlled by the contributor, whether already acquired or hereafter acquired, that
would be infringed by some manner, permitted by this License, of making, using, or
selling its contributor version, but do not include claims that would be infringed
only as a consequence of further modification of the contributor version. For
purposes of this definition, “control” includes the right to grant patent
sublicenses in a manner consistent with the requirements of this License.
Each contributor grants you a non-exclusive, worldwide, royalty-free patent license
under the contributor's essential patent claims, to make, use, sell, offer for sale,
import and otherwise run, modify and propagate the contents of its contributor
version.
In the following three paragraphs, a “patent license” is any express
agreement or commitment, however denominated, not to enforce a patent (such as an
express permission to practice a patent or covenant not to sue for patent
infringement). To “grant” such a patent license to a party means to make
such an agreement or commitment not to enforce a patent against the party.
If you convey a covered work, knowingly relying on a patent license, and the
Corresponding Source of the work is not available for anyone to copy, free of charge
and under the terms of this License, through a publicly available network server or
other readily accessible means, then you must either **(1)** cause the Corresponding
Source to be so available, or **(2)** arrange to deprive yourself of the benefit of the
patent license for this particular work, or **(3)** arrange, in a manner consistent with
the requirements of this License, to extend the patent license to downstream
recipients. “Knowingly relying” means you have actual knowledge that, but
for the patent license, your conveying the covered work in a country, or your
recipient's use of the covered work in a country, would infringe one or more
identifiable patents in that country that you have reason to believe are valid.
If, pursuant to or in connection with a single transaction or arrangement, you
convey, or propagate by procuring conveyance of, a covered work, and grant a patent
license to some of the parties receiving the covered work authorizing them to use,
propagate, modify or convey a specific copy of the covered work, then the patent
license you grant is automatically extended to all recipients of the covered work and
works based on it.
A patent license is “discriminatory” if it does not include within the
scope of its coverage, prohibits the exercise of, or is conditioned on the
non-exercise of one or more of the rights that are specifically granted under this
License. You may not convey a covered work if you are a party to an arrangement with
a third party that is in the business of distributing software, under which you make
payment to the third party based on the extent of your activity of conveying the
work, and under which the third party grants, to any of the parties who would receive
the covered work from you, a discriminatory patent license **(a)** in connection with
copies of the covered work conveyed by you (or copies made from those copies), or **(b)**
primarily for and in connection with specific products or compilations that contain
the covered work, unless you entered into that arrangement, or that patent license
was granted, prior to 28 March 2007.
Nothing in this License shall be construed as excluding or limiting any implied
license or other defenses to infringement that may otherwise be available to you
under applicable patent law.
### 12. No Surrender of Others' Freedom
If conditions are imposed on you (whether by court order, agreement or otherwise)
that contradict the conditions of this License, they do not excuse you from the
conditions of this License. If you cannot convey a covered work so as to satisfy
simultaneously your obligations under this License and any other pertinent
obligations, then as a consequence you may not convey it at all. For example, if you
agree to terms that obligate you to collect a royalty for further conveying from
those to whom you convey the Program, the only way you could satisfy both those terms
and this License would be to refrain entirely from conveying the Program.
### 13. Use with the GNU Affero General Public License
Notwithstanding any other provision of this License, you have permission to link or
combine any covered work with a work licensed under version 3 of the GNU Affero
General Public License into a single combined work, and to convey the resulting work.
The terms of this License will continue to apply to the part which is the covered
work, but the special requirements of the GNU Affero General Public License, section
13, concerning interaction through a network will apply to the combination as such.
### 14. Revised Versions of this License
The Free Software Foundation may publish revised and/or new versions of the GNU
General Public License from time to time. Such new versions will be similar in spirit
to the present version, but may differ in detail to address new problems or concerns.
Each version is given a distinguishing version number. If the Program specifies that
a certain numbered version of the GNU General Public License “or any later
version” applies to it, you have the option of following the terms and
conditions either of that numbered version or of any later version published by the
Free Software Foundation. If the Program does not specify a version number of the GNU
General Public License, you may choose any version ever published by the Free
Software Foundation.
If the Program specifies that a proxy can decide which future versions of the GNU
General Public License can be used, that proxy's public statement of acceptance of a
version permanently authorizes you to choose that version for the Program.
Later license versions may give you additional or different permissions. However, no
additional obligations are imposed on any author or copyright holder as a result of
your choosing to follow a later version.
### 15. Disclaimer of Warranty
THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW.
EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES
PROVIDE THE PROGRAM “AS IS” WITHOUT WARRANTY OF ANY KIND, EITHER
EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE
QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE PROGRAM PROVE
DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
### 16. Limitation of Liability
IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING WILL ANY
COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS THE PROGRAM AS
PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL,
INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE
PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE
OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE
WITH ANY OTHER PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE
POSSIBILITY OF SUCH DAMAGES.
### 17. Interpretation of Sections 15 and 16
If the disclaimer of warranty and limitation of liability provided above cannot be
given local legal effect according to their terms, reviewing courts shall apply local
law that most closely approximates an absolute waiver of all civil liability in
connection with the Program, unless a warranty or assumption of liability accompanies
a copy of the Program in return for a fee.
_END OF TERMS AND CONDITIONS_
## How to Apply These Terms to Your New Programs
If you develop a new program, and you want it to be of the greatest possible use to
the public, the best way to achieve this is to make it free software which everyone
can redistribute and change under these terms.
To do so, attach the following notices to the program. It is safest to attach them
to the start of each source file to most effectively state the exclusion of warranty;
and each file should have at least the “copyright” line and a pointer to
where the full notice is found.
<one line to give the program's name and a brief idea of what it does.>
Copyright (C) <year> <name of author>
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <http://www.gnu.org/licenses/>.
Also add information on how to contact you by electronic and paper mail.
If the program does terminal interaction, make it output a short notice like this
when it starts in an interactive mode:
<program> Copyright (C) <year> <name of author>
This program comes with ABSOLUTELY NO WARRANTY; for details type 'show w'.
This is free software, and you are welcome to redistribute it
under certain conditions; type 'show c' for details.
The hypothetical commands `show w` and `show c` should show the appropriate parts of
the General Public License. Of course, your program's commands might be different;
for a GUI interface, you would use an “about box”.
You should also get your employer (if you work as a programmer) or school, if any, to
sign a “copyright disclaimer” for the program, if necessary. For more
information on this, and how to apply and follow the GNU GPL, see
&lt;<http://www.gnu.org/licenses/>&gt;.
The GNU General Public License does not permit incorporating your program into
proprietary programs. If your program is a subroutine library, you may consider it
more useful to permit linking proprietary applications with the library. If this is
what you want to do, use the GNU Lesser General Public License instead of this
License. But first, please read
&lt;<http://www.gnu.org/philosophy/why-not-lgpl.html>&gt;.
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright [yyyy] [name of copyright owner]
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
README.md
+117 -109
View File
@@ -1,161 +1,169 @@
<p align='center'><img src='docs/logo.png' alt='logo' height="400"/><br>
</p>
# Cirro
Cirro is an extensible security research platform that enables researchers and penetration testers to collect, analyze, and visualize cloud environments and identity relationships through graph databases. Built with a modular architecture, Cirro can be extended to support multiple platforms and data sources.
You can check out the [Documentation](https://bishopfox.github.io/cirro) for more info.
A powerful CLI tool for graphing Azure and Entra ID data. Cirro enables security researchers and penetration testers to collect, analyze, and visualize Azure cloud environments and Entra ID relationships through graph databases.
## Features
- **Multi-platform Data Collection**: Extensible architecture supporting multiple cloud platforms and identity providers
- **Flexible Authentication**: Support for various authentication methods depending on the target platform
- **Multi-source Data Collection**: Gather data from both Azure Resource Manager (ARM) APIs and Microsoft Graph APIs
- **Flexible Authentication**: Support for multiple authentication methods including Azure CLI, access tokens, client secrets, and client certificates
- **Graph Database Support**: Compatible with both **Neo4j** and **Memgraph** graph databases
- **Data Enrichment**: Additional data collection capabilities for storage accounts, containers, and blob analysis
- **Multi-cloud Support**: Works with Azure Public, China, Germany, and US Government clouds
- **Cross-platform**: Available for Windows, macOS, and Linux
- **Modular Design**: Optional platform functionality through feature flags and extensible plugin architecture
- **Network Topology Analysis**: Support for network infrastructure platforms like Tailscale
- **Entra ID Enrichment**: User managers and password-change timestamps, application creators, agent identity properties, and published permission scopes
- **Restricted Guest Collection**: Traverse accessible application relationships with built-in, custom, or Microsoft catalog seeds
See the [collection guide](docs/usage/cirro.md) for usage examples and collection coverage.
## Architecture
Cirro has two main functional areas:
Cirro consists of two main components:
- **Collection (`cirro collect`)**: Gathers information from various platforms and APIs
- **Graph Operations (`cirro graph`)**: Manages graph database operations including data ingestion and export
The modular architecture uses feature flags to enable platform-specific functionality, allowing users to build only the components they need.
## CLI Structure
Cirro uses a hierarchical command structure organized by function and platform:
```
cirro <function> <platform> <command> [options]
```
### Data Collection
**Azure (collect az)**
```bash
# Available authentication methods:
cirro collect az azcli # Azure CLI authentication
cirro collect az client-secret # Client ID and secret
cirro collect az client-cert # Client certificate
cirro collect az access-token # Pre-obtained access token
cirro collect az user-pass # Username and password
```
**Tailscale (collect ts)**
```bash
# Tailscale data collection
cirro collect ts <auth-method> [options]
```
### Graph Operations
**Data Ingestion (graph ingest)**
```bash
# Ingest collected data into graph database
cirro graph ingest --type <platform> --file <data-file> [database options]
```
**Data Export (graph export)**
```bash
# Export graph data to various formats
cirro graph export --format <format> [options]
```
- **`cirro`**: The primary data collection tool that gathers information from Azure and Microsoft Graph APIs
- **`cirro-ingest`**: The ingestion tool that loads collected data into graph databases (Neo4j or Memgraph)
## Installation
### Pre-built Binaries
Download the latest release for your platform from the [releases page](https://github.com/bishopfox/cirro/releases). The releases are built with all features enabled.
Download the latest release for your platform from the [releases page](https://github.com/bishopfox/cirro/releases).
### Building from Source
Requires Rust 1.70 or later:
```bash
git clone https://github.com/bishopfox/cirro.git
cd cirro
cargo build --release
```
**NOTE: BUILDING WITH --RELEASE IS IMPORTANT FOR GRAPH FUNCTIONALITY SINCE IT EMBEDS YAML CONFIGURATION FILES IN THE BINARY!**
Binaries will be available in `target/release/`.
The binary will be available at `target/release/cirro`.
## Quick Start
#### Build Options
### 1. Data Collection
By default, Cirro includes all available functionality. To build with specific features:
Collect Azure and Entra ID data using one of the supported authentication methods:
#### Azure CLI Authentication
Uses existing Azure CLI authentication. Ensure you're logged in with `az login` before running.
```bash
# Build with only collection features
cargo build --release --no-default-features --features collector
# Build with only graph features
cargo build --release --no-default-features --features graph
# Build with specific platform support
cargo build --release --no-default-features --features "azure"
cargo build --release --no-default-features --features "tailscale"
# Collect from current Azure CLI context
cirro collect azcli
```
## Data Ingestion
Cirro uses Neo4j as the backend database. There are docker-compose files in the [tools](/tools/) directory to assist with containerized databases.
After collecting data, ingest it into your graph database:
#### Client Secret Authentication
Authenticate using an Azure AD application's client ID and secret.
```bash
# Ingest data for specific platforms
cirro graph ingest --type az --file cirro_output.db # Azure data
cirro graph ingest --type ts --file cirro_ts_socket.json # Tailscale data
cirro collect client-secret \
--client-id <CLIENT_ID> \
--client-secret <CLIENT_SECRET> \
--tenant-id <TENANT_ID>
```
# Specify custom database connection
cirro graph ingest --type az --file cirro_output.db \
#### Client Certificate Authentication
Authenticate using an Azure AD application's client ID and certificate (PEM format).
```bash
cirro collect client-cert \
--client-id <CLIENT_ID> \
--certificate <CERT_PATH> \
--tenant-id <TENANT_ID>
```
#### Access Token Authentication
Use a pre-obtained access token.
```bash
cirro collect access-token --token <ACCESS_TOKEN>
```
### 2. Data Ingestion
Both Neo4j and Memgraph are supported as graph database backends. Set up your preferred database before ingesting data. After collecting data, ingest it into your graph database:
#### Neo4j
```bash
cirro-ingest --file cirro_output.db \
--graph-type neo4j \
--server bolt://localhost:7687 \
--user neo4j \
--password password
```
## Dry-Run Mode
Preview what would be ingested and post-processed without writing any data to the graph database:
#### Memgraph
```bash
# Preview Azure ingestion and see which resource types have no implemented specs
cirro graph ingest --type az --file cirro_output.db --dry-run
# Preview Tailscale ingestion
cirro graph ingest --type ts --file cirro_ts_socket.json --dry-run
cirro-ingest --file cirro_output.db \
--graph-type memgraph \
--server bolt://localhost:7687 \
--user memgraph \
--password memgraph
```
In dry-run mode, Cirro will:
## Collection Modes
- Report each spec and its row count that **would** be processed
- Skip all Neo4j writes (no ingestion or post-processing queries execute)
- For Azure, list resource types in the input data that have no implemented spec
- **`both`** (default): Collect from both ARM and Graph APIs
- **`arm`**: Collect only from Azure Resource Manager APIs
- **`graph`**: Collect only from Microsoft Graph APIs
## Dashboard
## Data Enrichment
CirroDash can be located here: [https://github.com/bishopfox/cirrodash](https://github.com/bishopfox/cirrodash)
Cirro supports additional data enrichment capabilities that will add additional nodes to the graph:
## Debug Mode
```bash
# Collect storage account keys
cirro enrich azcli --storage-keys
```
## Cloud Support
Cirro supports multiple Azure cloud environments:
- **`public`** (default): Azure Public Cloud
- **`china`**: Azure China Cloud
- **`germany`**: Azure Germany Cloud
- **`usgov`**: Azure US Government Cloud
```bash
cirro collect azcli --cloud usgov
```
## Configuration Options
### Global Options
- `--output-path`: Specify the output SQLite database file (default: `cirro_output.db`)
- `--debug`: Enable debug logging for troubleshooting
- `--mode`: Set collection mode (`both`, `arm`, `graph`)
- `--cloud`: Specify Azure cloud environment
### Ingestion Options
- `--file`: Path to the Cirro SQLite database
- `--graph-type`: Database type (`neo4j` or `memgraph`)
- `--server`: Database server URL (bolt://, neo4j://, etc.)
- `--user`: Database username
- `--password`: Database password
- `--db-name`: Database name (optional, defaults based on graph type)
## Troubleshooting
### Common Issues
1. **Authentication Failures**: Ensure your credentials have appropriate permissions
2. **HTTP 429 Errors**: This can occur in large Azure environments. Retries are implemented in code but please report if you are still getting these errors.
3. **Access Token Authentication**: Ensure your access token has the appropriate audience for Microsoft Graph or Azure Resource Manager.
### Debug Mode
Enable debug logging for detailed information:
```bash
# Collection debug mode
cirro collect az azcli --debug
# Ingestion debug mode
cirro graph ingest --type az --file cirro_output.db --debug
cirro collect azcli --debug
cirro-ingest --file cirro_output.db --debug
```
## License
This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
---
**Note**: Cirro is designed for authorized security testing and research. Ensure you have proper permissions before running against any cloud or network environment.
**Note**: Cirro is designed for authorized security testing and research. Ensure you have proper permissions before running against any Azure environment.
cirro-ingest/Cargo.toml
+20
View File
@@ -0,0 +1,20 @@
[package]
description = "Ingestion tool for cirro"
edition = "2024"
name = "cirro-ingest"
repository = "https://github.com/bishopfox/cirro"
version = "0.1.0"
[dependencies]
clap = { version = "4.5.41", features = ["cargo", "derive"] }
colored = "3.0.0"
fern = { version = "0.7.1", features = ["chrono"] }
log = "0.4.27"
neo4rs = { version = "0.8.0", features = ["json"] }
rusqlite = { version = "0.37.0", features = ["bundled"] }
serde = { version = "1.0.219", features = ["serde_derive"] }
serde_json = "1.0.140"
strum = "0.27.1"
strum_macros = "0.27.1"
thiserror = "2.0.12"
tokio = { version = "1.46.1", features = ["full"] }
cirro-ingest/src/errors.rs
+35
View File
@@ -0,0 +1,35 @@
use thiserror::Error;
/// Custom error type for Graph API related errors
#[derive(Error, Debug)]
pub enum CirroIngestError {
#[error("Invalid configuration: {0}")]
InvalidConfig(String),
#[error("Database error: {0}")]
DatabaseError(String),
#[error("Auth error: {0}")]
AuthError(String),
#[error("Log setup error: {0}")]
LogSetupError(#[from] fern::InitError),
#[error("Multiple errors: {0:?}")]
MultipleErrors(Vec<CirroIngestError>),
#[error("Unknown error: {0}")]
Unknown(String),
#[error("Processing error: {0}")]
ProcessingError(String),
#[error("Sqlite error: {0}")]
SqliteError(#[from] rusqlite::Error),
#[error("JSON error: {0}")]
JsonError(#[from] serde_json::Error),
#[error("Neo4j error: {0}")]
Neo4jError(#[from] neo4rs::Error),
}
cirro-ingest/src/ingest/arm_processors.rs
+647
View File
@@ -0,0 +1,647 @@
use crate::ingest::ingestor::CirroIngestor;
use crate::{errors::CirroIngestError, ingest::constants::GraphType};
use log::{debug, error, info};
use neo4rs::{BoltType, query};
use serde_json::Value;
impl CirroIngestor {
/// Process Arm resources from the ingestor
async fn process_arm_resource(
&self,
table_name: &str,
node_insert_query: &str,
properties: Vec<&str>,
) -> Result<(), CirroIngestError> {
// Get the count of objects in the database
let count = self
.sql_conn
.as_ref()
.unwrap()
.query_row(
format!("SELECT COUNT(*) FROM {}", table_name).as_str(),
[],
|row| row.get::<_, i64>(0),
)
.unwrap_or(0);
if count == 0 {
debug!("No {} found", table_name);
return Ok(());
}
info!("Processing {:>5} {}", count, table_name);
let limit = 5000;
let mut offset = 0;
loop {
// If the offset exceeds the count, break the loop
if offset * limit <= count {
let select_query = format!(
"SELECT data FROM {} LIMIT {} OFFSET {}",
table_name,
limit,
offset * limit
);
debug!("Executing query: {}", select_query);
let mut stmt = self
.sql_conn
.as_ref()
.unwrap()
.prepare(&select_query)
.unwrap();
// Execute the query and process each row
let rows = stmt.query_map([], |row| row.get(0))?;
let processed_values: Vec<Value> = rows
.map(|result| {
result.map(|data: String| {
// Deserialize the JSON string into a Value
let value: Value = serde_json::from_str(&data)
.unwrap_or(Value::Object(serde_json::Map::new()));
// Create a new Value to hold the processed data
let mut new_value = Value::Object(serde_json::Map::new());
// For each property, set the value in the new Value
for property in &properties {
if let Some(val) = value.pointer(property) {
if *property == "/id" {
// Make sure the ID is always lowercase
new_value.as_object_mut().unwrap().insert(
property.trim_start_matches('/').to_string(),
val.as_str().unwrap_or("").to_lowercase().into(),
);
} else {
new_value.as_object_mut().unwrap().insert(
property.trim_start_matches('/').to_string(),
val.clone(),
);
}
}
}
new_value
})
})
.collect::<Result<Vec<_>, _>>()?;
// Insert the processed values into the graph
if !processed_values.is_empty() {
debug!(
"Inserting {} {} from offset {}",
processed_values.len(),
table_name,
offset
);
let _ = self
.graph
.run(query(node_insert_query).param(
"batch",
BoltType::try_from(serde_json::to_value(&processed_values)?)?,
))
.await
.map_err(|e| CirroIngestError::DatabaseError(e.to_string()))?;
} else {
error!(
"No values to insert for {} at offset {}",
table_name, offset
);
}
offset += 1;
debug!("Processed {} {}s", &processed_values.len(), table_name);
} else {
break;
}
}
Ok(())
}
/// Process ARM resource types
pub async fn process_specific_arm_resource(
&self,
resource_type: &str,
node_insert_query: &str,
properties: Vec<&str>,
) -> Result<(), CirroIngestError> {
// Get the count of objects in the database
let count_query = format!(
"SELECT COUNT(*) FROM resources WHERE lower(resource_type) = '{}'",
resource_type.to_lowercase()
);
debug!("Executing sqlite count query: {}", count_query);
let count = self
.sql_conn
.as_ref()
.unwrap()
.query_row(&count_query, [], |row| row.get::<_, i64>(0))
.unwrap_or(0);
if count == 0 {
debug!("No {} found", resource_type);
return Ok(());
}
info!("Processing {:>5} {}", count, resource_type);
let limit = 5000;
let mut offset = 0;
loop {
// If the offset exceeds the count, break the loop
if offset * limit <= count {
let select_query = format!(
"SELECT rg_id, data FROM resources WHERE lower(resource_type) = '{}' LIMIT {} OFFSET {}",
resource_type.to_lowercase(),
limit,
offset * limit
);
debug!("Executing query: {}", select_query);
let mut stmt = self
.sql_conn
.as_ref()
.unwrap()
.prepare(&select_query)
.unwrap();
// Execute the query and process each row
let rows = stmt.query_map([], |row| {
let rg_id: String = row.get(0)?;
let data: String = row.get(1)?;
Ok((rg_id, data))
})?;
let processed_values: Vec<Value> = rows
.map(|result| {
result.map(|(rg_id, data): (String, String)| {
// Deserialize the JSON string into a Value
let value: Value = serde_json::from_str(&data)
.unwrap_or(Value::Object(serde_json::Map::new()));
// Create a new Value to hold the processed data
let mut new_value = Value::Object(serde_json::Map::new());
// For each property, set the value in the new Value
for property in &properties {
if let Some(val) = value.pointer(property) {
new_value.as_object_mut().unwrap().insert(
property.trim_start_matches('/').to_string(),
val.clone(),
);
}
}
// Add the resource group ID to the new Value
new_value
.as_object_mut()
.unwrap()
.insert("resourcegroup_id".to_string(), Value::String(rg_id));
new_value
})
})
.collect::<Result<Vec<_>, _>>()?;
// Insert the processed values into the graph
if !processed_values.is_empty() {
debug!(
"Inserting {} {} from offset {}",
processed_values.len(),
resource_type,
offset
);
let _ = self
.graph
.run(query(node_insert_query).param(
"batch",
BoltType::try_from(serde_json::to_value(&processed_values)?)?,
))
.await
.map_err(|e| CirroIngestError::DatabaseError(e.to_string()))?;
} else {
error!(
"No values to insert for {} at offset {}",
resource_type, offset
);
}
offset += 1;
debug!("Processed {} {}s", &processed_values.len(), resource_type);
} else {
break;
}
}
Ok(())
}
// Process tenants
pub async fn process_tenants(&self) -> Result<(), CirroIngestError> {
let properties = vec![
"/id",
"/displayName",
"/tenantId",
"/countryCode",
"/domains",
"/defaultDomain",
"/tenantCategory",
"/tenantType",
];
let node_insert_query = r#"
UNWIND $batch AS row
MERGE (t:Tenant {id: row.id})
SET t += {
displayName : row.displayName,
tenantId : row.tenantId,
countryCode : row.countryCode,
domains : row.domains,
defaultDomain : row.defaultDomain,
tenantCategory : row.tenantCategory,
tenantType : row.tenantType
}
"#;
self.process_arm_resource("tenants", node_insert_query, properties)
.await?;
Ok(())
}
// Process subscriptions
pub async fn process_subscriptions(&self) -> Result<(), CirroIngestError> {
let properties = vec![
"/id",
"/displayName",
"/authorizationSource",
"/state",
"/subscriptionId",
"/tenantId",
];
let node_insert_query = r#"
UNWIND $batch AS row
MERGE (s:Subscription {id: row.id})
SET s:ArmResource
SET s += {
displayName : row.displayName,
authorizationSource : row.authorizationSource,
state : row.state,
subscriptionId : row.subscriptionId,
tenantId : row.tenantId
}
MERGE (t:Tenant {tenantId: row.tenantId})
MERGE (t)-[:CONTAINS]->(s)
"#;
self.process_arm_resource("subscriptions", node_insert_query, properties)
.await?;
Ok(())
}
// Process resource groups
pub async fn process_resource_groups(&self) -> Result<(), CirroIngestError> {
let properties = vec!["/id", "/name", "/location", "/type"];
let node_insert_query = r#"
UNWIND $batch AS row
MERGE (rg:ArmResource {id: row.id})
SET rg:ResourceGroup
SET rg += {
name : row.name,
location : row.location,
type : row.type
}
MERGE (s:Subscription {id: row.subscriptionId})
MERGE (s)-[:CONTAINS]->(rg)
"#;
// Get the count of objects in the database
let count = self
.sql_conn
.as_ref()
.unwrap()
.query_row(
"SELECT COUNT(*) FROM resources WHERE lower(resource_type) = 'microsoft.resources/resourcegroups'",
[],
|row| row.get::<_, i64>(0),
)
.unwrap_or(0);
if count == 0 {
debug!("No resource groups found");
return Ok(());
}
info!("Processing {:>5} resource groups", count);
let limit = 5000;
let mut offset = 0;
loop {
// If the offset exceeds the count, break the loop
if offset * limit <= count {
let select_query = format!(
"SELECT sub_id, json(data) FROM resources WHERE lower(resource_type) = 'microsoft.resources/resourcegroups' LIMIT {} OFFSET {}",
limit,
offset * limit
);
debug!("Executing query: {}", select_query);
let mut stmt = self
.sql_conn
.as_ref()
.unwrap()
.prepare(&select_query)
.unwrap();
// Execute the query and process each row
let rows = stmt.query_map([], |row| {
let sub_id: String = row.get(0)?;
let data: String = row.get(1)?;
Ok((sub_id, data))
})?;
let processed_values: Vec<Value> = rows
.map(|result| {
result.map(|(sub_id, data): (String, String)| {
// Deserialize the JSON string into a Value
let value: Value = serde_json::from_str(&data)
.unwrap_or(Value::Object(serde_json::Map::new()));
// Create a new Value to hold the processed data
let mut new_value = Value::Object(serde_json::Map::new());
// For each property, set the value in the new Value
for property in &properties {
if let Some(val) = value.pointer(property) {
if *property == "/id" {
// Make sure the ID is always lowercase
new_value.as_object_mut().unwrap().insert(
property.trim_start_matches('/').to_string(),
val.as_str().unwrap_or("").to_lowercase().into(),
);
} else {
new_value.as_object_mut().unwrap().insert(
property.trim_start_matches('/').to_string(),
val.clone(),
);
}
}
}
// Add the subscription ID to the new Value
new_value.as_object_mut().unwrap().insert(
"subscriptionId".to_string(),
Value::String(sub_id.to_lowercase()),
);
new_value
})
})
.collect::<Result<Vec<_>, _>>()?;
// Insert the processed values into the graph
if !processed_values.is_empty() {
debug!(
"Inserting {} resource groups from offset {}",
processed_values.len(),
offset
);
let _ = self
.graph
.run(query(node_insert_query).param(
"batch",
BoltType::try_from(serde_json::to_value(&processed_values)?)?,
))
.await
.map_err(|e| CirroIngestError::DatabaseError(e.to_string()))?;
} else {
error!(
"No values to insert for resource groups at offset {}",
offset
);
}
offset += 1;
debug!("Processed {} resource groups", &processed_values.len());
} else {
break;
}
}
Ok(())
}
// Process generic ARM resources
pub async fn process_generic_arm_resources(&self) -> Result<(), CirroIngestError> {
let properties = vec![
"/id",
"/identity",
"/kind",
"/location",
"/name",
"/type",
"/tags",
];
let node_insert_query = r#"
UNWIND $batch AS row
MERGE (obj:ArmResource {id: row.id})
SET obj += {
id : row.id,
kind : row.kind,
location : row.location,
name : row.name,
type : row.type,
tags : [key in keys(row.tags) | key + ":" + row.tags[key]]
}
MERGE (rg:ArmResource {id: row.resourcegroup_id})
SET rg:ResourceGroup
MERGE (rg)-[:HAS_RESOURCE]->(obj)
WITH obj, row
WHERE row.identity IS NOT NULL AND toLower(row.identity.type) = 'systemassigned'
MERGE (i:GraphObject {id: row.identity.principalId})
MERGE (obj)-[:HAS_IDENTITY]->(i)
WITH obj, row
WHERE row.identity IS NOT NULL AND toLower(row.identity.type) = 'userassigned'
MERGE (i:GraphObject {id: row.identity.principalId})
MERGE (obj)-[:HAS_IDENTITY]->(i)
"#;
// Get the count of objects in the database
let count = self
.sql_conn
.as_ref()
.unwrap()
.query_row(
"SELECT COUNT(*) FROM resources WHERE lower(resource_type) != 'microsoft.resources/resourcegroups'",
[],
|row| row.get::<_, i64>(0),
)
.unwrap_or(0);
if count == 0 {
debug!("No resources found");
return Ok(());
}
info!("Processing {:>5} resources", count);
let limit = 5000;
let mut offset = 0;
loop {
// If the offset exceeds the count, break the loop
if offset * limit <= count {
let select_query = format!(
"SELECT rg_id, data FROM resources WHERE lower(resource_type) != 'microsoft.resources/resourcegroups' LIMIT {} OFFSET {}",
limit,
offset * limit
);
debug!("Executing query: {}", select_query);
let mut stmt = self
.sql_conn
.as_ref()
.unwrap()
.prepare(&select_query)
.unwrap();
// Execute the query and process each row
let rows = stmt.query_map([], |row| {
let rg_id: String = row.get(0)?;
let data: String = row.get(1)?;
Ok((rg_id, data))
})?;
let processed_values: Vec<Value> = rows
.map(|result| {
result.map(|(rg_id, data): (String, String)| {
// Deserialize the JSON string into a Value
let value: Value = serde_json::from_str(&data)
.unwrap_or(Value::Object(serde_json::Map::new()));
// Create a new Value to hold the processed data
let mut new_value = Value::Object(serde_json::Map::new());
// For each property, set the value in the new Value
for property in &properties {
if let Some(val) = value.pointer(property) {
if *property == "/id" {
// Make sure the ID is always lowercase
new_value.as_object_mut().unwrap().insert(
property.trim_start_matches('/').to_string(),
val.as_str().unwrap_or("").to_lowercase().into(),
);
} else {
new_value.as_object_mut().unwrap().insert(
property.trim_start_matches('/').to_string(),
val.clone(),
);
}
}
}
// Add the resource group ID to the new Value
new_value.as_object_mut().unwrap().insert(
"resourcegroup_id".to_string(),
Value::String(rg_id.to_lowercase()),
);
new_value
})
})
.collect::<Result<Vec<_>, _>>()?;
// Insert the processed values into the graph
if !processed_values.is_empty() {
debug!(
"Inserting {} resource from offset {}",
processed_values.len(),
offset
);
let _ = self
.graph
.run(query(node_insert_query).param(
"batch",
BoltType::try_from(serde_json::to_value(&processed_values)?)?,
))
.await
.map_err(|e| CirroIngestError::DatabaseError(e.to_string()))?;
} else {
error!(
"No values to insert for resource groups at offset {}",
offset
);
}
offset += 1;
debug!("Processed {} resources", &processed_values.len());
} else {
break;
}
}
Ok(())
}
// Process Azure role assignments
pub async fn process_role_assignments(&self) -> Result<(), CirroIngestError> {
let properties = vec![
"/id",
"/description",
"/roleName",
"/roleType",
"/permissions",
"/properties",
];
// We need to pass dynamic role names, so we need APOC/MAGE
let node_insert_query = match self.graph_type {
GraphType::Neo4j => {
r#"
UNWIND $batch AS row
MERGE (o:GraphObject {id: row.properties.principalId})
WITH o, row
MATCH (r:ArmResource {id: toLower(row.properties.scope)})
CALL apoc.merge.relationship(o, row.roleName, {}, {}, r) YIELD rel
SET rel += {
id : row.id,
description : row.description,
roleName : row.roleName,
roleType : row.roleType,
actions : row.permissions.actions,
notActions : row.permissions.notActions,
dataActions : row.permissions.dataActions,
notDataActions : row.permissions.notDataActions
}
"#
}
// Memgraph doesn't allow for CALL procedures after a CREATE/MERGE
// Need to implement a way to merge the nodes first
// Otherwise, process_role_assignments needs to be run last in the processors
// Cause this will not show any nodes that are not already in the database
GraphType::Memgraph => {
r#"
UNWIND $batch AS row
MATCH (o:GraphObject {id: row.properties.principalId})
MATCH (r:ArmResource {id: toLower(row.properties.scope)})
CALL merge.relationship(o, row.roleName, {}, {
description : row.description,
roleName : row.roleName,
roleType : row.roleType,
actions : row.permissions[0].actions,
notActions : row.permissions[0].notActions,
dataActions : row.permissions[0].dataActions,
notDataActions : row.permissions[0].notDataActions
}, r, {
description : row.description,
roleName : row.roleName,
roleType : row.roleType,
actions : row.permissions[0].actions,
notActions : row.permissions[0].notActions,
dataActions : row.permissions[0].dataActions,
notDataActions : row.permissions[0].notDataActions
}) YIELD rel
"#
}
};
self.process_arm_resource("roleAssignments", node_insert_query, properties)
.await?;
Ok(())
}
}
cirro-ingest/src/ingest/automation_processors.rs
+24
View File
@@ -0,0 +1,24 @@
use crate::errors::CirroIngestError;
use crate::ingest::ingestor::CirroIngestor;
impl CirroIngestor {
/// Process automation accounts
pub async fn process_automation_accounts(&self) -> Result<(), CirroIngestError> {
let resource_type = "microsoft.automation/automationaccounts";
let properties = vec!["/id", "/properties"];
let node_insert_query = r#"
UNWIND $batch AS row
MERGE (obj:ArmResource {id: row.id})
SET obj:AutomationAccount
SET obj += {
creationTime: row.properties.creationTime,
lastModifiedTime: row.properties.lastModifiedTime
}
"#;
self.process_specific_arm_resource(resource_type, node_insert_query, properties)
.await?;
Ok(())
}
}
cirro-ingest/src/ingest/azurearc_processors.rs
+64
View File
@@ -0,0 +1,64 @@
use crate::errors::CirroIngestError;
use crate::ingest::ingestor::CirroIngestor;
impl CirroIngestor {
/// Process azure arc sql servers
pub async fn process_azurearc_sql_servers(&self) -> Result<(), CirroIngestError> {
let resource_type = "microsoft.azurearcdata/sqlserverinstances";
let properties = vec!["/id", "/properties"];
let node_insert_query = r#"
UNWIND $batch AS row
MERGE (obj:ArmResource {id: row.id})
SET obj:AzureArcSqlServer
SET obj += {
azureDefenderStatus: row.properties.azureDefenderStatus,
collation: row.properties.collation,
currentVersion: row.properties.currentVersion,
edition: row.properties.edition,
instanceName: row.properties.instanceName,
licenseType: row.properties.licenseType,
patchLevel: row.properties.patchLevel,
productId: row.properties.productId,
status: row.properties.status,
tcpDynamicPorts: row.properties.tcpDynamicPorts,
tcpStaticPorts: row.properties.tcpStaticPorts,
vCore: row.properties.vCore,
version: row.properties.version
}
MERGE (h:ArmResource {id: row.properties.containerResourceId})
MERGE (h)-[:HOSTS]->(obj)
"#;
self.process_specific_arm_resource(resource_type, node_insert_query, properties)
.await?;
Ok(())
}
// Process azure arc sql server db
// Disabled for now
// https://github.com/memgraph/mage/issues/642
// pub async fn process_azurearc_sql_database(&self) -> Result<(), CirroIngestError> {
// let resource_type = "microsoft.azurearcdata/sqlserverinstances/databases";
// let properties = vec!["/id", "/properties"];
// let node_insert_query = r#"
// UNWIND $batch AS row
// MERGE (obj:ArmResource {id: row.id})
// SET obj:AzureArcSqlDB
// SET obj += {
// dataFileSizeMB: row.properties.dataFileSizeMB
// databaseCreationDate: row.properties.databaseCreationDate
// lastDatabaseUploadTime: row.properties.lastDatabaseUploadTime
// state: row.properties.state
// WITH obj, row, split(row.id, '/')[7] AS instance_id
// WITH obj, row, text.join(instance_id,
// }
// "#;
// Ok(())
// }
}
cirro-ingest/src/ingest/cognitiveservices_processors.rs
+25
View File
@@ -0,0 +1,25 @@
use crate::errors::CirroIngestError;
use crate::ingest::ingestor::CirroIngestor;
impl CirroIngestor {
/// Process cognitive services accounts
pub async fn process_cognitive_services_account(&self) -> Result<(), CirroIngestError> {
let resource_type = "microsoft.cognitiveservices/accounts";
let properties = vec!["/id", "/properties"];
let node_insert_query = r#"
UNWIND $batch AS row
MERGE (obj:ArmResource {id: row.id})
SET obj:CognitiveServicesAccount
SET obj += {
customSubdomain: row.properties.customSubDomainName,
endpoint: row.properties.endpoint,
publicNetworkAccess: row.properties.publicNetworkAccess
}
"#;
self.process_specific_arm_resource(resource_type, node_insert_query, properties)
.await?;
Ok(())
}
}
cirro-ingest/src/ingest/communication_processors.rs
+25
View File
@@ -0,0 +1,25 @@
use crate::errors::CirroIngestError;
use crate::ingest::ingestor::CirroIngestor;
impl CirroIngestor {
/// Process communication services
pub async fn process_communication_services(&self) -> Result<(), CirroIngestError> {
let resource_type = "microsoft.communication/communicationservices";
let properties = vec!["/id", "/properties"];
let node_insert_query = r#"
UNWIND $batch AS row
MERGE (obj:ArmResource {id: row.id})
SET obj:CommunicationServices
SET obj += {
dataLocation: row.properties.dataLocation,
hostName: row.properties.hostName,
immutableResourceId: row.properties.immutableResourceId
}
"#;
self.process_specific_arm_resource(resource_type, node_insert_query, properties)
.await?;
Ok(())
}
}
cirro-ingest/src/ingest/compute_processors.rs
+167
View File
@@ -0,0 +1,167 @@
use crate::errors::CirroIngestError;
use crate::ingest::ingestor::CirroIngestor;
impl CirroIngestor {
/// Process availability sets
pub async fn process_availability_sets(&self) -> Result<(), CirroIngestError> {
let resource_type = "microsoft.compute/availabilitysets";
let properties = vec!["/id", "/sku", "/properties"];
let node_insert_query = r#"
UNWIND $batch AS row
MERGE (obj:ArmResource {id: row.id})
SET obj:AvailabilitySet
SET obj += {
sku: row.sku.name,
}
WITH obj, row WHERE row.managedBy IS NOT NULL
MERGE (h:ArmResource {id: row.managedBy})
MERGE (h)-[:MANAGES]->(obj)
"#;
self.process_specific_arm_resource(resource_type, node_insert_query, properties)
.await?;
Ok(())
}
/// Process disks
pub async fn process_disks(&self) -> Result<(), CirroIngestError> {
let resource_type = "microsoft.compute/disks";
let properties = vec!["/id", "/managedBy", "/properties"];
let node_insert_query = r#"
UNWIND $batch AS row
MERGE (obj:ArmResource {id: row.id})
SET obj:Disk
SET obj += {
managedBy: row.managedBy,
diskSizeGB: row.properties.diskSizeGB,
diskState: row.properties.diskState,
osType: row.properties.osType,
networkAccessPolicy: row.properties.networkAccessPolicy,
publicNetworkAccess: row.properties.publicNetworkAccess
}
WITH obj, row WHERE row.managedBy IS NOT NULL
MERGE (h:ArmResource {id: row.managedBy})
MERGE (h)-[:HAS_DISK]->(obj)
"#;
self.process_specific_arm_resource(resource_type, node_insert_query, properties)
.await?;
Ok(())
}
/// Process restore point collections
pub async fn process_restore_point_collections(&self) -> Result<(), CirroIngestError> {
let resource_type = "microsoft.compute/restorepointcollections";
let properties = vec!["/id", "/properties"];
let node_insert_query = r#"
UNWIND $batch AS row
MERGE (obj:ArmResource {id: row.id})
SET obj:RestorePointCollection
MERGE (vm:ArmResource {id: row.properties.source.id})
MERGE (h)-[:HAS_RESTOREPOINT]->(obj)
"#;
self.process_specific_arm_resource(resource_type, node_insert_query, properties)
.await?;
Ok(())
}
/// Process snapshots
pub async fn process_snapshots(&self) -> Result<(), CirroIngestError> {
let resource_type = "microsoft.compute/snapshots";
let properties = vec!["/id", "/properties"];
let node_insert_query = r#"
UNWIND $batch AS row
MERGE (obj:ArmResource {id: row.id})
SET obj:Snapshot
SET obj += {
createOption: row.properties.creationData.createOption,
dataAccessAuthMode: row.properties.dataAccessAuthMode,
diskSizeBytes: row.properties.diskSizeBytes,
diskSizeGB: row.properties.diskSizeGB,
diskState: row.properties.diskState,
hyperVGeneration: row.properties.hyperVGeneration,
incremental: row.properties.incremental,
networkAccessPolicy: row.properties.networkAccessPolicy,
osType: row.properties.osType,
publicNetworkAccess: row.properties.publicNetworkAccess
}
MERGE (h:ArmResource {id: row.properties.creationData.sourceResourceId})
MERGE (h)-[:HAS_SNAPSHOT]->(obj)
"#;
self.process_specific_arm_resource(resource_type, node_insert_query, properties)
.await?;
Ok(())
}
/// Process SSH public keys
pub async fn process_ssh_public_keys(&self) -> Result<(), CirroIngestError> {
let resource_type = "microsoft.compute/sshpublickeys";
let properties = vec!["/id", "/properties"];
let node_insert_query = r#"
UNWIND $batch AS row
MERGE (obj:ArmResource {id: row.id})
SET obj:SSHPublicKey
SET obj += {
publicKey: row.properties.publicKey
}
"#;
self.process_specific_arm_resource(resource_type, node_insert_query, properties)
.await?;
Ok(())
}
/// Process virtual machines
pub async fn process_virtual_machines(&self) -> Result<(), CirroIngestError> {
let resource_type = "microsoft.compute/virtualmachines";
let properties = vec!["/id", "/resources", "/properties"];
let node_insert_query = r#"
UNWIND $batch AS row
MERGE (obj:ArmResource {id: row.id})
SET obj:VirtualMachine
SET obj += {
vmSize: row.properties.hardwareProfile.vmSize,
adminUsername: row.properties.osProfile.adminUsername,
allowExtensionOperations: row.properties.osProfile.allowExtensionOperations,
computerName: row.properties.osProfile.computerName,
vmId: row.properties.vmId,
os: row.properties.storageProfile.osDisk.osType,
image: row.properties.storageProfile.imageReference.offer,
imageVersion: row.properties.storageProfile.imageReference.exactVersion
}
WITH obj, row WHERE row.properties.networkProfile.networkInterfaces IS NOT NULL
UNWIND row.properties.networkProfile.networkInterfaces AS nic
MERGE (n:ArmResource {id: nic.id})
MERGE (obj)-[:HAS_NIC]->(n)
WITH obj, row WHERE row.resources IS NOT NULL
UNWIND row.resources AS resource
WITH obj, resource WHERE toLower(resource.type) = "microsoft.compute/virtualmachines/extensions"
MERGE (e:VMExtension {id: resource.id})
SET e += {
id: resource.id,
name: resource.name,
type: resource.type,
location: resource.location,
provisioningState: resource.properties.provisioningState
}
MERGE (obj)-[:HAS_EXTENSION]->(e)
"#;
self.process_specific_arm_resource(resource_type, node_insert_query, properties)
.await?;
Ok(())
}
}
cirro-ingest/src/ingest/constants.rs
+77
View File
@@ -0,0 +1,77 @@
use strum_macros::EnumIter;
#[derive(clap::ValueEnum, Copy, Clone, Debug, PartialEq, Eq)]
pub enum GraphType {
Neo4j,
Memgraph,
}
#[derive(Debug, Clone)]
pub struct CypherConstants {
pub create_constraint_query: String,
pub create_index_query: String,
}
impl CypherConstants {
pub fn new(graph_type: GraphType) -> Self {
match graph_type {
// Neo4j style constants for creating constraints and indexes
// Neo4j creates indexes automatically when creating constraints
// so the index creation queries are left empty.
GraphType::Neo4j => Self {
create_constraint_query:
"CREATE CONSTRAINT IF NOT EXISTS FOR (n: {}) REQUIRE n.id IS UNIQUE".to_string(),
create_index_query: "CREATE INDEX {}_id IF NOT EXISTS FOR (n: {}) ON (n.id)"
.to_string(), // Neo4j creates indexes automatically with constraints
},
// Memgraph style constants for creating constraints and indexes
// Memgraph requires explicit index creation, so the index creation queries are provided.
GraphType::Memgraph => Self {
create_constraint_query: "CREATE CONSTRAINT ON (n: {}) ASSERT n.id IS UNIQUE"
.to_string(),
create_index_query: "CREATE INDEX ON :{}(id);".to_string(),
},
}
}
}
#[derive(strum_macros::Display, EnumIter, Debug, PartialEq)]
pub enum NodeType {
GraphObject,
GraphApplication,
GraphDevice,
GraphGroup,
GraphRole,
GraphServicePrincipal,
GraphUser,
ArmResource,
ArcSqlServer,
AutomationAccount,
AvailabilitySet,
AzureRbac,
ClassicStorageAccount,
ContainerRegistry,
CommunicationServiceAccount,
Disk,
DnsZone,
HybridMachine,
IpConfiguration,
KeyVault,
ManagedIdentity,
NetworkInterface,
NetworkSecurityGroup,
PublicIPAddress,
ResourceGroup,
RestorePointCollection,
Runbook,
Snapshot,
SSHPublicKey,
StorageAccount,
Subnet,
Subscription,
Tenant,
UserAssignedIdentity,
VirtualNetwork,
VirtualMachine,
}
cirro-ingest/src/ingest/container_processors.rs
+38
View File
@@ -0,0 +1,38 @@
use crate::errors::CirroIngestError;
use crate::ingest::ingestor::CirroIngestor;
impl CirroIngestor {
/// Process container registries
pub async fn process_container_registries(&self) -> Result<(), CirroIngestError> {
let resource_type = "microsoft.containerregistry/registries";
let properties = vec!["/id", "/sku", "/systemData", "/properties"];
let node_insert_query = r#"
UNWIND $batch AS row
MERGE (obj:ArmResource {id: row.id})
SET obj:ContainerRegistry
SET obj += {
sku: row.sku.name,
adminUserEnabled: row.properties.adminUserEnabled,
anonymousPullEnabled: row.properties.anonymousPullEnabled,
creationDate: row.properties.creationDate,
dataEndpointEnabled: row.properties.dataEndpointEnabled,
dataEndpointHostNames: row.properties.dataEndpointHostNames,
loginServer: row.properties.loginServer,
metadataSearch: row.properties.metadataSearch,
networkRuleBypassOptions: row.properties.networkRuleBypassOptions,
publicNetworkAccess: row.properties.publicNetworkAccess,
createdAt: row.systemData.createdAt,
createdBy: row.systemData.createdBy,
createdByType: row.systemData.createdByType,
lastModifiedAt: row.systemData.lastModifiedAt,
lastModifiedBy: row.systemData.lastModifiedBy,
lastModifiedByType: row.systemData.lastModifiedByType
}
"#;
self.process_specific_arm_resource(resource_type, node_insert_query, properties)
.await?;
Ok(())
}
}
cirro-ingest/src/ingest/enrichment_processors.rs
+148
View File
@@ -0,0 +1,148 @@
use crate::errors::CirroIngestError;
use crate::ingest::ingestor::CirroIngestor;
use log::{debug, error, info};
use neo4rs::{BoltType, query};
use serde_json::Value;
impl CirroIngestor {
/// Process enrichments
pub async fn process_enrichment(
&self,
module_name: &str,
node_insert_query: &str,
properties: Vec<&str>,
) -> Result<(), CirroIngestError> {
// Get the count of objects in the database
let count_query = format!(
"SELECT COUNT(*) FROM enrichments WHERE module = '{}'",
module_name.to_lowercase()
);
debug!("Executing sqlite count query: {}", count_query);
let count = self
.sql_conn
.as_ref()
.unwrap()
.query_row(&count_query, [], |row| row.get::<_, i64>(0))
.unwrap_or(0);
if count == 0 {
debug!("No {} found", module_name);
return Ok(());
}
info!("Processing {:>5} {}", count, module_name);
let limit = 5000;
let mut offset = 0;
loop {
// If the offset exceeds the count, break the loop
if offset * limit <= count {
let select_query = format!(
"SELECT resource_id, data FROM enrichments WHERE module = '{}' LIMIT {} OFFSET {}",
module_name,
limit,
offset * limit
);
debug!("Executing query: {}", select_query);
let mut stmt = self
.sql_conn
.as_ref()
.unwrap()
.prepare(&select_query)
.unwrap();
// Execute the query and process each row
let rows = stmt.query_map([], |row| {
let resource_id: String = row.get(0)?;
let data: String = row.get(1)?;
Ok((resource_id, data))
})?;
let processed_values: Vec<Value> = rows
.map(|result| {
result.map(|(resource_id, data): (String, String)| {
// Deserialize the JSON string into a Value
let value: Value = serde_json::from_str(&data)
.unwrap_or(Value::Object(serde_json::Map::new()));
// Create a new Value to hold the processed data
let mut new_value = Value::Object(serde_json::Map::new());
// For each property, set the value in the new Value
for property in &properties {
if let Some(val) = value.pointer(property) {
new_value.as_object_mut().unwrap().insert(
property.trim_start_matches('/').to_string(),
val.clone(),
);
}
}
// Add the resource group ID to the new Value
new_value.as_object_mut().unwrap().insert(
"resource_id".to_string(),
Value::String(resource_id.to_lowercase()),
);
new_value
})
})
.collect::<Result<Vec<_>, _>>()?;
// Insert the processed values into the graph
if !processed_values.is_empty() {
debug!(
"Inserting {} {} from offset {}",
processed_values.len(),
module_name,
offset
);
let _ = self
.graph
.run(query(node_insert_query).param(
"batch",
BoltType::try_from(serde_json::to_value(&processed_values)?)?,
))
.await
.map_err(|e| CirroIngestError::DatabaseError(e.to_string()))?;
} else {
error!(
"No values to insert for {} at offset {}",
module_name, offset
);
}
offset += 1;
debug!("Processed {} {}s", &processed_values.len(), module_name);
} else {
break;
}
}
Ok(())
}
/// Process storage account keys enrichment
pub async fn process_enrich_storage_keys(&self) -> Result<(), CirroIngestError> {
let module_name = "storage_account_keys";
let properties = vec!["/keys"];
let node_insert_query = r#"
UNWIND $batch AS row
MERGE (obj:ArmResource {id: row.resource_id})
SET obj:StorageAccount
WITH obj, row
UNWIND row.keys AS key
MERGE (k:StorageAccountKey {name: key.keyName})
SET k += {
value: key.value,
permissions: key.permissions,
creationTime: key.creationTime
}
MERGE (obj)-[:HAS_KEY]->(k)
"#;
self.process_enrichment(module_name, node_insert_query, properties)
.await?;
Ok(())
}
}
cirro-ingest/src/ingest/graph_processors.rs
+534
View File
@@ -0,0 +1,534 @@
use crate::errors::CirroIngestError;
use crate::ingest::ingestor::CirroIngestor;
use log::{debug, error, info};
use neo4rs::{BoltType, query};
use serde_json::Value;
impl CirroIngestor {
/// Processes graph objects from the database and inserts them into the graph
async fn process_graph_objects(
&self,
table_name: &str,
node_insert_query: &str,
properties: Vec<&str>,
) -> Result<(), CirroIngestError> {
// Get the count of objects in the database
let count = self
.sql_conn
.as_ref()
.unwrap()
.query_row(
format!("SELECT COUNT(*) FROM {}", table_name).as_str(),
[],
|row| row.get::<_, i64>(0),
)
.unwrap_or(0);
if count == 0 {
debug!("No {} found", table_name);
return Ok(());
}
info!("Processing {:>5} {}", count, table_name);
let limit = 5000;
let mut offset = 0;
loop {
// If the offset exceeds the count, break the loop
if offset * limit <= count {
let select_query = format!(
"SELECT data FROM {} LIMIT {} OFFSET {}",
table_name,
limit,
offset * limit
);
debug!("Executing query: {}", select_query);
let mut stmt = self
.sql_conn
.as_ref()
.unwrap()
.prepare(&select_query)
.unwrap();
// Execute the query and process each row
let rows = stmt.query_map([], |row| row.get(0))?;
let processed_values: Vec<Value> = rows
.map(|result| {
result.map(|data: String| {
// Deserialize the JSON string into a Value
let value: Value = serde_json::from_str(&data)
.unwrap_or(Value::Object(serde_json::Map::new()));
// Create a new Value to hold the processed data
let mut new_value = Value::Object(serde_json::Map::new());
// For each property, set the value in the new Value
for property in &properties {
if let Some(val) = value.pointer(property) {
new_value.as_object_mut().unwrap().insert(
property.trim_start_matches('/').to_string(),
val.clone(),
);
}
}
debug!("Processed value: {:?}", new_value);
new_value
})
})
.collect::<Result<Vec<_>, _>>()?;
// Insert the processed values into the graph
if !processed_values.is_empty() {
debug!(
"Inserting {} {} from offset {}",
processed_values.len(),
table_name,
offset
);
let _ = self
.graph
.run(query(node_insert_query).param(
"batch",
BoltType::try_from(serde_json::to_value(&processed_values)?)?,
))
.await
.map_err(|e| CirroIngestError::DatabaseError(e.to_string()))?;
} else {
error!(
"No values to insert for {} at offset {}",
table_name, offset
);
}
offset += 1;
debug!("Processed {} {}s", &processed_values.len(), table_name);
} else {
break;
}
}
Ok(())
}
/// Processes graph applications
pub async fn process_graph_applications(&self) -> Result<(), CirroIngestError> {
// JSON Pointer style properties for the object
let properties = vec![
"/displayName",
"/id",
"/appId",
"/passwordCredentials",
"/publisherDomain",
"/signInAudience",
"/keyCredentials",
"/owners",
];
let node_insert_query = r#"
UNWIND $batch as row
WITH row
MERGE (obj:GraphObject {id: row.id})
SET obj:GraphApplication
SET obj += {
displayName: row.displayName,
appId: row.appId,
publisherDomain: row.publisherDomain,
signInAudience: row.signInAudience
}
// Unwind owners
WITH row, obj WHERE row.owners IS NOT NULL
UNWIND row.owners AS owner
MERGE (o:GraphObject {id: owner})
MERGE (o)-[:OWNS]->(obj)
// Unwind password credentials
WITH row, obj WHERE row.passwordCredentials IS NOT NULL
UNWIND row.passwordCredentials AS passwordCredential
MERGE (p:ClientSecret {keyId: passwordCredential.keyId})
SET p += {
customKeyIdentifier: passwordCredential.customKeyIdentifier,
keyId: passwordCredential.keyId,
displayName: passwordCredential.displayName,
startDateTime: passwordCredential.startDateTime,
endDateTime: passwordCredential.endDateTime,
hint: passwordCredential.hint,
secretText: passwordCredential.secretText
}
MERGE (p)-[:AUTHENTICATES]->(obj)
// Unwind key credentials
WITH row, obj, [c IN row.keyCredentials WHERE c.customKeyIdentifier IS NOT NULL] AS keyCredentials
UNWIND keyCredentials AS keyCredential
MERGE (k:Certificate {thumbprint: keyCredential.customKeyIdentifier})
SET k += {
thumbprint: keyCredential.customKeyIdentifier,
displayName: keyCredential.displayName,
startDateTime: keyCredential.startDateTime,
endDateTime: keyCredential.endDateTime,
type: keyCredential.type,
usage: keyCredential.usage
}
MERGE (k)-[:AUTHENTICATES]->(obj)
"#;
// Process the graph objects
self.process_graph_objects("applications", node_insert_query, properties)
.await?;
Ok(())
}
/// Processes administrative units
pub async fn process_graph_administrative_units(&self) -> Result<(), CirroIngestError> {
// JSON Pointer style properties for the object
let properties = vec![
"/id",
"/description",
"/displayName",
"/isMemberManagementRestricted",
"/members",
"/membershipRule",
"/membershipRuleProcessingState",
"/membershipType",
"/scopedRoleMembers",
"/visibility",
];
let node_insert_query = r#"
UNWIND $batch as row
WITH row
MERGE (obj:GraphObject {id: row.id})
SET obj:GraphAdministrativeUnit
SET obj += {
description: row.description,
displayName: row.displayName,
isMemberManagementRestricted: row.isMemberManagementRestricted,
membershipRule: row.membershipRule,
membershipRuleProcessingState: row.membershipRuleProcessingState,
membershipType: row.membershipType,
visibility: row.visibility
}
// Unwind members
WITH row, obj WHERE row.members IS NOT NULL
UNWIND row.members AS member
MERGE (m:GraphObject {id: member})
MERGE (m)-[:MEMBER_OF]->(obj)
// Unwind scoped role members
WITH obj, coalesce(row.scopedRoleMembers, []) AS scopedRoleMembers
UNWIND scopedRoleMembers AS memberObj
MERGE (member:GraphObject {id: memberObj.roleMemberInfo_id})
MERGE (member)-[r:HAS_ROLE]->(obj)
SET r.roleId = memberObj.roleId
WITH obj, memberObj, member, r
OPTIONAL MATCH (role:GraphObject) WHERE role.id = memberObj.roleId
WITH obj, memberObj, member, r, role
WHERE role IS NOT NULL
SET r.roleName = role.displayName
"#;
// Process the graph objects
self.process_graph_objects("administrativeUnits", node_insert_query, properties)
.await?;
Ok(())
}
/// Processes graph devices
pub async fn process_graph_devices(&self) -> Result<(), CirroIngestError> {
// JSON Pointer style properties for the object
let properties = vec![
"/displayName",
"/id",
"/accountEnabled",
"/deviceId",
"/isCompliant",
"/isManaged",
"/manufacturer",
"/model",
"/onPremisesLastSyncDateTime",
"/onPremisesSyncEnabled",
"/operatingSystem",
"/operatingSystemVersion",
"/profileType",
"/trustType",
"/registeredUsers",
"/registeredOwners",
];
let node_insert_query = r#"
UNWIND $batch as row
WITH row
MERGE (obj:GraphObject {id: row.id})
SET obj:GraphDevice
SET obj += {
displayName: row.displayName,
accountEnabled: row.accountEnabled,
deviceId: row.deviceId,
isCompliant: row.isCompliant,
isManaged: row.isManaged,
manufacturer: row.manufacturer,
model: row.model,
onPremisesLastSyncDateTime: row.onPremisesLastSyncDateTime,
onPremisesSyncEnabled: row.onPremisesSyncEnabled,
operatingSystem: row.operatingSystem,
operatingSystemVersion: row.operatingSystemVersion,
profileType: row.profileType,
trustType: row.trustType
}
// Unwind registered users
WITH row, obj WHERE row.registeredUsers IS NOT NULL
UNWIND row.registeredUsers AS user
MERGE (u:GraphObject {id: user})
MERGE (u)-[:REGISTERED_USER]->(obj)
// Unwind registered owners
WITH row, obj WHERE row.registeredOwners IS NOT NULL
UNWIND row.registeredOwners AS owner
MERGE (o:GraphObject {id: owner})
MERGE (o)-[:REGISTERED_OWNER]->(obj)
"#;
// Process the graph objects
self.process_graph_objects("devices", node_insert_query, properties)
.await?;
Ok(())
}
/// Processes graph groups
pub async fn process_graph_groups(&self) -> Result<(), CirroIngestError> {
// JSON Pointer style properties for the object
let properties = vec![
"/displayName",
"/id",
"/groupTypes",
"/membershipRule",
"/membershipRuleProcessingState",
"/onPremisesSecurityIdentifier",
"/organizationId",
"/securityEnabled",
"/visibility",
"/owners",
"/members",
];
let node_insert_query = r#"
UNWIND $batch as row
WITH row
MERGE (obj:GraphObject {id: row.id})
SET obj:GraphGroup
SET obj += {
displayName: row.displayName,
groupTypes: row.groupTypes,
membershipRule: row.membershipRule,
membershipRuleProcessingState: row.membershipRuleProcessingState,
onPremisesSecurityIdentifier: row.onPremisesSecurityIdentifier,
organizationId: row.organizationId,
securityEnabled: row.securityEnabled,
visibility: row.visibility
}
// Unwind owners
WITH row, obj WHERE row.owners IS NOT NULL
UNWIND row.owners AS owner
MERGE (o:GraphObject {id: owner})
MERGE (o)-[:OWNS]->(obj)
// Unwind members
WITH row, obj WHERE row.members IS NOT NULL
UNWIND row.members AS member
MERGE (m:GraphObject {id: member})
MERGE (m)-[:MEMBER_OF]->(obj)
"#;
// Process the graph objects
self.process_graph_objects("groups", node_insert_query, properties)
.await?;
Ok(())
}
/// Processes graph roles
pub async fn process_graph_roles(&self) -> Result<(), CirroIngestError> {
let properties = vec![
"/displayName",
"/id",
"/description",
"/roleTemplateId",
"/members",
];
let node_insert_query = r#"
UNWIND $batch as row
WITH row
MERGE (obj:GraphObject {id: row.id})
SET obj:GraphRole
SET obj += {
displayName: row.displayName,
description: row.description,
roleTemplateId: row.roleTemplateId
}
// Unwind members
WITH row, obj WHERE row.members IS NOT NULL
UNWIND row.members AS member
MERGE (m:GraphObject {id: member})
MERGE (m)-[:HAS_ROLE]->(obj)
"#;
// Process the graph objects
self.process_graph_objects("directoryRoles", node_insert_query, properties)
.await?;
Ok(())
}
/// Processes graph service principals
pub async fn process_graph_service_principals(&self) -> Result<(), CirroIngestError> {
let properties = vec![
"/displayName",
"/id",
"/accountEnabled",
"/alternativeNames",
"/appId",
"/appOwnerOrganizationId",
"/publisherName",
"/servicePrincipalType",
"/replyUrls",
"/servicePrincipalNames",
"/keyCredentials",
"/passwordCredentials",
"/owners",
];
let node_insert_query = r#"
UNWIND $batch as row
WITH row
// If an application exists, link it to the service principal
OPTIONAL MATCH (a:GraphApplication {appId: row.appId})
MERGE (obj:GraphObject {id: row.id})
SET obj:GraphServicePrincipal
SET obj += {
displayName: row.displayName,
accountEnabled: row.accountEnabled,
alternativeNames: row.alternativeNames,
appId: row.appId,
appOwnerOrganizationId: row.appOwnerOrganizationId,
publisherName: row.publisherName,
servicePrincipalType: row.servicePrincipalType,
replyUrls: row.replyUrls,
servicePrincipalNames: row.servicePrincipalNames
}
WITH row, obj, a WHERE a IS NOT NULL
MERGE (a)-[:REPRESENTED_BY]->(obj)
// Unwind owners
WITH row, obj WHERE row.owners IS NOT NULL
UNWIND row.owners AS owner
MERGE (o:GraphObject {id: owner})
MERGE (o)-[:OWNS]->(obj)
// Unwind password credentials
WITH row, obj WHERE row.passwordCredentials IS NOT NULL
UNWIND row.passwordCredentials AS passwordCredential
MERGE (p:ClientSecret {keyId: passwordCredential.keyId})
SET p += {
customKeyIdentifier: passwordCredential.customKeyIdentifier,
keyId: passwordCredential.keyId,
displayName: passwordCredential.displayName,
startDateTime: passwordCredential.startDateTime,
endDateTime: passwordCredential.endDateTime,
hint: passwordCredential.hint,
secretText: passwordCredential.secretText
}
MERGE (p)-[:AUTHENTICATES]->(obj)
WITH row, obj, [c IN row.keyCredentials WHERE c.customKeyIdentifier IS NOT NULL] AS keyCredentials
UNWIND keyCredentials AS keyCredential
MERGE (k:Certificate {thumbprint: keyCredential.customKeyIdentifier})
SET k += {
thumbprint: keyCredential.customKeyIdentifier,
displayName: keyCredential.displayName,
startDateTime: keyCredential.startDateTime,
endDateTime: keyCredential.endDateTime,
type: keyCredential.type,
usage: keyCredential.usage
}
MERGE (k)-[:AUTHENTICATES]->(obj)
"#;
// Process the graph objects
self.process_graph_objects("servicePrincipals", node_insert_query, properties)
.await?;
Ok(())
}
/// Processes graph users
pub async fn process_graph_users(&self) -> Result<(), CirroIngestError> {
let properties = vec![
"/accountEnabled",
"/city",
"/companyName",
"/country",
"/creationType",
"/department",
"/displayName",
"/givenName",
"/id",
"/jobTitle",
"/mail",
"/mailNickname",
"/mobilePhone",
"/officeLocation",
"/onPremisesDistinguishedName",
"/onPremisesSamAccountName",
"/onPremisesSecurityIdentifier",
"/onPremisesUserPrincipalName",
"/refreshTokensValidFromDateTime",
"/securityIdentifier",
"/state",
"/surname",
"/userPrincipalName",
"/userType",
];
let node_insert_query = r#"
UNWIND $batch as row
WITH row
MERGE (obj:GraphObject {id: row.id})
SET obj:GraphUser
SET obj += {
accountEnabled: row.accountEnabled,
city: row.city,
companyName: row.companyName,
country: row.country,
creationType: row.creationType,
department: row.department,
displayName: row.displayName,
givenName: row.givenName,
jobTitle: row.jobTitle,
mail: row.mail,
mailNickname: row.mailNickname,
mobilePhone: row.mobilePhone,
officeLocation: row.officeLocation,
onPremisesDistinguishedName: row.onPremisesDistinguishedName,
onPremisesSamAccountName: row.onPremisesSamAccountName,
onPremisesSecurityIdentifier: row.onPremisesSecurityIdentifier,
onPremisesUserPrincipalName: row.onPremisesUserPrincipalName,
refreshTokensValidFromDateTime: row.refreshTokensValidFromDateTime,
securityIdentifier: row.securityIdentifier,
state: row.state,
surname: row.surname,
userPrincipalName: row.userPrincipalName,
userType: row.userType
}
"#;
self.process_graph_objects("users", node_insert_query, properties)
.await?;
Ok(())
}
}
cirro-ingest/src/ingest/hybridcompute_processors.rs
+65
View File
@@ -0,0 +1,65 @@
use crate::errors::CirroIngestError;
use crate::ingest::ingestor::CirroIngestor;
impl CirroIngestor {
/// Process hybrid machines
pub async fn process_hybrid_machines(&self) -> Result<(), CirroIngestError> {
let resource_type = "microsoft.hybridcompute/machines";
let properties = vec!["/id", "/resources", "/properties"];
let node_insert_query = r#"
UNWIND $batch AS row
MERGE (obj:ArmResource {id: row.id})
SET obj:HybridMachine
SET obj += {
adFqdn: row.properties.adFqdn,
agentVersion: row.properties.agentVersion,
clientPublicKey: row.properties.clientPublicKey,
cloud: row.properties.cloudMetadata.provider,
displayName: row.properties.displayName,
dnsFqdn: row.properties.dnsFqdn,
domainName: row.properties.domainName,
lastStatusChange: row.properties.lastStatusChange,
machineFqdn: row.properties.machineFqdn,
osName: row.properties.osName,
computerName: row.properties.osProfile.computerName,
osSku: row.properties.osSku,
osVersion: row.properties.osVersion,
status: row.properties.status,
vmId: row.properties.vmId,
vmUuid: row.properties.vmUuid
}
WITH obj, row WHERE row.properties.networkProfile.networkInterfaces IS NOT NULL
UNWIND row.properties.networkProfile.networkInterfaces AS nic
WITH obj, row, nic WHERE nic.ipAddresses IS NOT NULL
UNWIND nic.ipAddresses AS ipAddress
MERGE (ip:HybridIPAddress {address: ipAddress.address})
SET ip += {
ipAddressVersion: ipAddress.ipAddressVersion,
subnet: ipAddress.subnet.addressPrefix
}
MERGE (obj)-[:HAS_IP]->(ip)
WITH obj, row WHERE row.resources IS NOT NULL
UNWIND row.resources AS resource
WITH obj, resource WHERE toLower(resource.type) = "microsoft.hybridcompute/machines/extensions"
MERGE (e:HybridExtension {id: resource.id})
SET e += {
name: resource.name,
type: resource.type,
location: resource.location,
typeHandlerVersion: resource.properties.typeHandlerVersion,
autoUpgradeMinorVersion: resource.properties.autoUpgradeMinorVersion,
enableAutomaticUpgrade: resource.properties.enableAutomaticUpgrade,
statusMessage: resource.properties.instanceView.status.message,
provisioningState: resource.properties.provisioningState
}
MERGE (obj)-[:HAS_EXTENSION]->(e)
"#;
self.process_specific_arm_resource(resource_type, node_insert_query, properties)
.await?;
Ok(())
}
}
cirro-ingest/src/ingest/identity_processors.rs
+26
View File
@@ -0,0 +1,26 @@
use crate::errors::CirroIngestError;
use crate::ingest::ingestor::CirroIngestor;
impl CirroIngestor {
/// Process user assigned identities
pub async fn process_user_assigned_identities(&self) -> Result<(), CirroIngestError> {
let resource_type = "microsoft.managedidentity/userassignedidentities";
let properties = vec!["/id", "/properties"];
let node_insert_query = r#"
UNWIND $batch AS row
MERGE (obj:ArmResource {id: row.id})
SET obj:UserAssignedIdentity
SET obj += {
clientId: row.properties.clientId,
tenantId: row.properties.tenantId
}
MERGE (o:GraphObject {appId: row.properties.clientId})
MERGE (obj)-[:HAS_IDENTITY]->(o)
"#;
self.process_specific_arm_resource(resource_type, node_insert_query, properties)
.await?;
Ok(())
}
}
cirro-ingest/src/ingest/ingestor.rs
+332
View File
@@ -0,0 +1,332 @@
use crate::errors::CirroIngestError;
use crate::ingest::constants::*;
use log::{debug, info};
use neo4rs::*;
use rusqlite::{Connection, Result};
use std::path::PathBuf;
use std::pin::Pin;
use strum::IntoEnumIterator;
pub struct CirroIngestor {
pub file: PathBuf,
pub host: String,
pub user: String,
pub password: String,
pub db_name: String,
pub graph: neo4rs::Graph,
pub graph_type: GraphType,
pub constants: CypherConstants,
pub sql_conn: Option<Connection>,
}
/// Custom Debug trait for CirroIngestor
impl std::fmt::Debug for CirroIngestor {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.debug_struct("CirroIngestor")
.field("file", &self.file)
.field("host", &self.host)
.field("user", &self.user)
.field("password", &self.password)
.field("db_name", &self.db_name)
.field("graph", &format_args!("<neo4rs::Graph>"))
.finish()
}
}
impl CirroIngestor {
pub async fn new(
file: PathBuf,
graph_type: GraphType,
host: String,
user: String,
password: String,
db_name: Option<String>,
) -> Self {
// If db_name is not provided, use the default based on graph type
let db_name = db_name.unwrap_or_else(|| match graph_type {
GraphType::Neo4j => "neo4j".to_string(),
GraphType::Memgraph => "memgraph".to_string(),
});
let config = ConfigBuilder::default()
.uri(host.clone())
.user(user.clone())
.password(password.clone())
.db(db_name.clone())
.build()
.map_err(|e| CirroIngestError::DatabaseError(e.to_string()))
.unwrap();
info!(
"Connecting to {:?} database at {} with user {}",
graph_type, host, user
);
let graph = Graph::connect(config)
.await
.map_err(|e| CirroIngestError::DatabaseError(e.to_string()))
.unwrap();
// Test connection to the database
let mut result = graph.execute(query("RETURN 1")).await.unwrap();
let row = result.next().await.unwrap().unwrap();
let value: i64 = row.get("1").unwrap();
assert_eq!(1, value);
info!("Successfully connected to the database");
let sql_conn = Connection::open(file.clone())
.map_err(|e| CirroIngestError::DatabaseError(e.to_string()))
.ok();
let ingestor = CirroIngestor {
file: file,
host,
user,
password,
db_name,
graph,
graph_type,
constants: CypherConstants::new(graph_type),
sql_conn,
};
return ingestor;
}
/// Runs the ingestor
pub async fn run(&mut self) -> Result<(), CirroIngestError> {
// Create constraints and indexes for each node type
for node_type in NodeType::iter() {
let constraint_query = self
.constants
.create_constraint_query
.replace("{}", &node_type.to_string());
debug!("Executing query: {}", constraint_query);
let _ = self
.graph
.run(query(&constraint_query))
.await
.map_err(|e| {
CirroIngestError::DatabaseError(format!(
"Failed to create constraint for {}: {}",
node_type, e
))
})?;
if !self.constants.create_index_query.is_empty() {
let index_query = self
.constants
.create_index_query
.replace("{}", &node_type.to_string());
debug!("Executing query: {}", index_query);
let _ = self.graph.run(query(&index_query)).await.map_err(|e| {
CirroIngestError::DatabaseError(format!(
"Failed to create index for {}: {}",
node_type, e
))
})?;
}
}
if self.graph_type == GraphType::Memgraph {
debug!("Setting storage mode to IN_MEMORY_ANALYTICAL for Memgraph");
let _ = self
.graph
.run(query("STORAGE MODE IN_MEMORY_ANALYTICAL;"))
.await
.map_err(|e| {
CirroIngestError::DatabaseError(format!(
"Failed to set storage mode for Memgraph: {}",
e
))
})?;
}
// Run the main ingestion logic
self.process_file().await?;
info!("Running final post-processing merge query");
self.post_process().await?;
Ok(())
}
async fn process_file(&mut self) -> Result<(), CirroIngestError> {
info!(
"Starting Cirro ingest on file: {:?}",
self.file.as_path().file_name().unwrap()
);
// Define type that matches the actual function signature. Thanks Copilot cause this is harder than Python.
#[rustfmt::skip]
type ProcessorFn = for<'a> fn(&'a CirroIngestor,) -> Pin<Box<dyn Future<Output = Result<(), CirroIngestError>> + 'a>,>;
let processors: Vec<ProcessorFn> = vec![
|ingestor| Box::pin(CirroIngestor::process_graph_users(ingestor)),
|ingestor| Box::pin(CirroIngestor::process_graph_devices(ingestor)),
// Applications must be processed before service principals due to OPTIONAL MATCH call in service principals
|ingestor| Box::pin(CirroIngestor::process_graph_applications(ingestor)),
|ingestor| Box::pin(CirroIngestor::process_graph_service_principals(ingestor)),
|ingestor| Box::pin(CirroIngestor::process_graph_groups(ingestor)),
|ingestor| Box::pin(CirroIngestor::process_graph_roles(ingestor)),
|ingestor| Box::pin(CirroIngestor::process_tenants(ingestor)),
|ingestor| Box::pin(CirroIngestor::process_subscriptions(ingestor)),
|ingestor| Box::pin(CirroIngestor::process_resource_groups(ingestor)),
// This must always be first to ensure all resources are entered regardless of type
|ingestor| Box::pin(CirroIngestor::process_generic_arm_resources(ingestor)),
// Process specific ARM resources
|ingestor| Box::pin(CirroIngestor::process_automation_accounts(ingestor)),
|ingestor| Box::pin(CirroIngestor::process_availability_sets(ingestor)),
|ingestor| Box::pin(CirroIngestor::process_azurearc_sql_servers(ingestor)),
|ingestor| Box::pin(CirroIngestor::process_classic_storage_accounts(ingestor)),
|ingestor| Box::pin(CirroIngestor::process_cognitive_services_account(ingestor)),
|ingestor| Box::pin(CirroIngestor::process_communication_services(ingestor)),
|ingestor| Box::pin(CirroIngestor::process_container_registries(ingestor)),
|ingestor| Box::pin(CirroIngestor::process_disks(ingestor)),
|ingestor| Box::pin(CirroIngestor::process_dns_zones(ingestor)),
|ingestor| Box::pin(CirroIngestor::process_hybrid_machines(ingestor)),
|ingestor| Box::pin(CirroIngestor::process_keyvaults(ingestor)),
|ingestor| Box::pin(CirroIngestor::process_network_interfaces(ingestor)),
|ingestor| Box::pin(CirroIngestor::process_network_security_groups(ingestor)),
|ingestor| Box::pin(CirroIngestor::process_public_ip_addresses(ingestor)),
|ingestor| Box::pin(CirroIngestor::process_restore_point_collections(ingestor)),
|ingestor| Box::pin(CirroIngestor::process_snapshots(ingestor)),
|ingestor| Box::pin(CirroIngestor::process_sql_servers(ingestor)),
|ingestor| Box::pin(CirroIngestor::process_sql_databases(ingestor)),
|ingestor| Box::pin(CirroIngestor::process_ssh_public_keys(ingestor)),
|ingestor| Box::pin(CirroIngestor::process_storage_accounts(ingestor)),
|ingestor| Box::pin(CirroIngestor::process_user_assigned_identities(ingestor)),
|ingestor| Box::pin(CirroIngestor::process_virtual_machines(ingestor)),
|ingestor| Box::pin(CirroIngestor::process_virtual_networks(ingestor)),
// This must always be last to ensure all other nodes are processed first
|ingestor| Box::pin(CirroIngestor::process_role_assignments(ingestor)),
// Enrichment processors
|ingestor| Box::pin(CirroIngestor::process_enrich_storage_keys(ingestor)),
];
for processor in processors {
debug!("Running processor: {:?}", processor);
let result = processor(self).await;
if let Err(e) = result {
return Err(CirroIngestError::ProcessingError(format!(
"Failed to process file with error: {}",
e
)));
}
debug!("Processor completed successfully");
self.post_process().await?;
}
Ok(())
}
async fn post_process(&mut self) -> Result<(), CirroIngestError> {
debug!("Running post-processing merge query");
let _ = self
.graph
.run(query(
r#"MATCH (n)
WHERE n.id IS NOT NULL
SET n.id = toLower(n.id)
WITH n
WHERE n.type IS NOT NULL
SET n.type = toLower(n.type)"#,
))
.await
.map_err(|e| {
CirroIngestError::DatabaseError(format!("Failed to assert lowercase IDs: {}", e))
})?;
match self.graph_type {
GraphType::Neo4j => {
// Merge all nodes with the same ID
debug!("Merging nodes with the same ID in Neo4j");
let _ = self
.graph
.run(query(
r#" MATCH (n)
WITH n.id AS id, COLLECT(n) AS nodesToMerge
WHERE id IS NOT NULL AND size(nodesToMerge) > 1
CALL apoc.refactor.mergeNodes(nodesToMerge, {properties: "override", mergeRels:true, preserveExistingSelfRels: true})
YIELD node
RETURN count(*);"#,
))
.await
.map_err(|e| {
CirroIngestError::DatabaseError(format!(
"Failed to run post-processing merge query: {}",
e
))
})?;
// Merge relationships with the same label from the same node
debug!("Merging relationships with the same label in Neo4j");
let _ = self
.graph
.run(query(
r#"MATCH (a)-[r]->(b)
WITH a, b, type(r) AS relType, collect(r) AS rels
WHERE size(rels) > 1
CALL apoc.refactor.mergeRelationships((rels, {properties: "combine"}))
YIELD node
RETURN count(*);"#,
))
.await
.map_err(|e| {
CirroIngestError::DatabaseError(format!(
"Failed to run post-processing merge relationships query: {}",
e
))
})?;
}
GraphType::Memgraph => {
// Merge all nodes with the same ID
debug!("Merging nodes with the same ID in Memgraph");
let _ = self
.graph
.run(query(
r#"MATCH (n)
WITH n.id AS id, COLLECT(n) AS nodesToMerge
WHERE id IS NOT NULL AND size(nodesToMerge) > 1
CALL refactor.merge_nodes(nodesToMerge, {properties: "override", mergeRels:true})
YIELD node
RETURN count(*);"#,
))
.await
.map_err(|e| {
CirroIngestError::DatabaseError(format!(
"Failed to run post-processing merge query: {}",
e
))
})?;
// Merge relationships with the same label from the same node
debug!("Merging relationships with the same label in Memgraph");
let _ = self
.graph
.run(query(
r#"MATCH (a)-[r]->(b)
WHERE id(a) <> id(b) // ❗ skip self-loops
WITH a, b, type(r) AS rel_type, collect(r) AS rels
WHERE size(rels) > 1
CALL {
WITH rels
WITH rels[0] AS keeper, rels[1..] AS duplicates
UNWIND duplicates AS d
SET keeper += d
DELETE d
RETURN keeper
}"#,
))
.await
.map_err(|e| {
CirroIngestError::DatabaseError(format!(
"Failed to run post-processing merge relationships query: {}",
e
))
})?;
}
}
Ok(())
}
}
cirro-ingest/src/ingest/keyvault_processors.rs
+38
View File
@@ -0,0 +1,38 @@
use crate::errors::CirroIngestError;
use crate::ingest::ingestor::CirroIngestor;
impl CirroIngestor {
/// Process key vaults
pub async fn process_keyvaults(&self) -> Result<(), CirroIngestError> {
let resource_type = "microsoft.keyvault/vaults";
let properties = vec!["/id", "/properties"];
let node_insert_query = r#"
UNWIND $batch AS row
MERGE (obj:ArmResource {id: row.id})
SET obj:KeyVault
SET obj += {
enableRbacAuthorization: row.properties.enableRbacAuthorization,
enableSoftDelete: row.properties.enableSoftDelete,
enabledForDeployment: row.properties.enabledForDeployment,
publicNetworkAccess: row.properties.publicNetworkAccess,
softDeleteRetentionInDays: row.properties.softDeleteRetentionInDays,
vaultUri: row.properties.vaultUri
}
WITH obj, row WHERE row.properties.accessPolicies IS NOT NULL
UNWIND row.properties.accessPolicies AS policy
MERGE (p:GraphObject {id: policy.objectId})
MERGE (obj)-[r:HAS_POLICY]->(p)
SET r += {
certificates: policy.permissions.certificates,
keys: policy.permissions.keys,
secrets: policy.permissions.secrets
}
"#;
self.process_specific_arm_resource(resource_type, node_insert_query, properties)
.await?;
Ok(())
}
}
cirro-ingest/src/ingest/mod.rs
+19
View File
@@ -0,0 +1,19 @@
pub mod constants;
pub mod ingestor;
pub mod arm_processors;
pub mod automation_processors;
pub mod azurearc_processors;
pub mod cognitiveservices_processors;
pub mod communication_processors;
pub mod compute_processors;
pub mod container_processors;
pub mod enrichment_processors;
pub mod graph_processors;
pub mod hybridcompute_processors;
pub mod identity_processors;
pub mod keyvault_processors;
pub mod network_processors;
pub mod sql_processors;
pub mod storage_processors;
pub mod web_processors;
cirro-ingest/src/ingest/network_processors.rs
+190
View File
@@ -0,0 +1,190 @@
use crate::errors::CirroIngestError;
use crate::ingest::ingestor::CirroIngestor;
impl CirroIngestor {
/// Process dns zones
pub async fn process_dns_zones(&self) -> Result<(), CirroIngestError> {
let resource_type = "microsoft.network/dnszones";
let properties = vec!["/id", "/properties"];
let node_insert_query = r#"
UNWIND $batch AS row
MERGE (obj:ArmResource {id: row.id})
SET obj:DNSZone
SET obj += {
nameServers: row.properties.nameServers,
numberOfRecordSets: row.properties.numberOfRecordSets,
zoneType: row.properties.zoneType
}
"#;
self.process_specific_arm_resource(resource_type, node_insert_query, properties)
.await?;
Ok(())
}
/// Process public IP addresses
pub async fn process_public_ip_addresses(&self) -> Result<(), CirroIngestError> {
let resource_type = "microsoft.network/publicipaddresses";
let properties = vec!["/id", "/properties"];
let node_insert_query = r#"
UNWIND $batch AS row
MERGE (obj:ArmResource {id: row.id})
SET obj:PublicIPAddress
SET obj += {
idleTimeoutInMinutes: row.properties.idleTimeoutInMinutes,
ipAddress: row.properties.ipAddress,
publicIPAddressVersion: row.properties.publicIPAddressVersion,
publicIPAllocationMethod: row.properties.publicIPAllocationMethod
}
"#;
self.process_specific_arm_resource(resource_type, node_insert_query, properties)
.await?;
Ok(())
}
/// Process network interfaces
pub async fn process_network_interfaces(&self) -> Result<(), CirroIngestError> {
let resource_type = "microsoft.network/networkinterfaces";
let properties = vec!["/id", "/properties"];
let node_insert_query = r#"
UNWIND $batch AS row
MERGE (obj:ArmResource {id: row.id})
SET obj:NetworkInterface
SET obj += {
macAddress: row.properties.macAddress,
allowPort25Out: row.properties.allowPort25Out,
appliedDnsServers: row.properties.dnsSettings.appliedDnsServers,
dnsServers: row.properties.dnsSettings.dnsServers
}
WITH obj, row WHERE row.properties.ipConfigurations IS NOT NULL
UNWIND row.properties.ipConfigurations AS ipconfig
MERGE (ipconnode:IPConfig {id: ipconfig.id})
SET ipconnode += {
name: ipconfig.name,
type: ipconfig.type,
privateIPAddress: ipconfig.properties.privateIPAddress,
privateIPAddressVersion: ipconfig.properties.privateIPAddressVersion,
privateIPAllocationMethod: ipconfig.properties.privateIPAllocationMethod
}
MERGE (obj)-[:HAS_CONFIG]->(ipconnode)
MERGE (subnet:Subnet {id: ipconfig.properties.subnet.id})
MERGE (subnet)-[:CONTAINS]->(ipconnode)
WITH obj, row, ipconnode, ipconfig WHERE ipconfig.properties.publicIPAddress IS NOT NULL
MERGE (pubip:ArmResource {id: ipconfig.properties.publicIPAddress.id})
SET pubip:PublicIPAddress
MERGE (ipconnode)-[:HAS_IP]->(pubip)
WITH obj, row WHERE row.properties.virtualMachine.id IS NOT NULL
MERGE (vm:ArmResource {id: row.properties.virtualMachine.id})
SET vm:VirtualMachine
MERGE (vm)-[:HAS_NIC]->(obj)
WITH obj, row WHERE row.properties.networkSecurityGroup IS NOT NULL
MERGE (nsg:ArmResource {id: row.properties.networkSecurityGroup.id})
SET nsg:NSG
MERGE (obj)-[:HAS_NSG]->(nsg)
"#;
self.process_specific_arm_resource(resource_type, node_insert_query, properties)
.await?;
Ok(())
}
/// Process network security groups
pub async fn process_network_security_groups(&self) -> Result<(), CirroIngestError> {
let resource_type = "microsoft.network/networksecuritygroups";
let properties = vec!["/id", "/properties"];
let node_insert_query = r#"
UNWIND $batch AS row
MERGE (obj:ArmResource {id: row.id})
SET obj:NSG
WITH obj, row WHERE row.properties.defaultSecurityRules IS NOT NULL
UNWIND row.properties.securityRules AS rule
MERGE (r:NSGRule {id: rule.id})
SET r += {
name: rule.name,
type: rule.type,
access: rule.properties.access,
description: rule.properties.description,
destinationAddressPrefix: rule.properties.destinationAddressPrefix,
destinationAddressPrefixes: rule.properties.destinationAddressPrefixes,
destinationPortRange: rule.properties.destinationPortRange,
destinationPortRanges: rule.properties.destinationPortRanges,
direction: rule.properties.direction,
priority: rule.properties.priority,
protocol: rule.properties.protocol,
sourceAddressPrefix: rule.properties.sourceAddressPrefix,
sourceAddressPrefixes: rule.properties.sourceAddressPrefixes,
sourcePortRange: rule.properties.sourcePortRange,
sourcePortRanges: rule.properties.sourcePortRanges
}
MERGE (obj)-[:HAS_RULE]->(r)
WITH obj, row WHERE row.properties.securityRules IS NOT NULL
UNWIND row.properties.securityRules AS rule
MERGE (r:NSGRule {id: rule.id})
SET r += {
name: rule.name,
type: rule.type,
access: rule.properties.access,
description: rule.properties.description,
destinationAddressPrefix: rule.properties.destinationAddressPrefix,
destinationAddressPrefixes: rule.properties.destinationAddressPrefixes,
destinationPortRange: rule.properties.destinationPortRange,
destinationPortRanges: rule.properties.destinationPortRanges,
direction: rule.properties.direction,
priority: rule.properties.priority,
protocol: rule.properties.protocol,
sourceAddressPrefix: rule.properties.sourceAddressPrefix,
sourceAddressPrefixes: rule.properties.sourceAddressPrefixes,
sourcePortRange: rule.properties.sourcePortRange,
sourcePortRanges: rule.properties.sourcePortRanges
}
MERGE (obj)-[:HAS_RULE]->(r)
"#;
self.process_specific_arm_resource(resource_type, node_insert_query, properties)
.await?;
Ok(())
}
/// Process virtual networks
pub async fn process_virtual_networks(&self) -> Result<(), CirroIngestError> {
let resource_type = "microsoft.network/virtualnetworks";
let properties = vec!["/id", "/properties"];
let node_insert_query = r#"
UNWIND $batch AS row
MERGE (obj:ArmResource {id: row.id})
SET obj:VirtualNetwork
SET obj += {
addressPrefixes: row.properties.addressSpace.addressPrefixes
}
WITH obj, row WHERE row.properties.subnets IS NOT NULL
UNWIND row.properties.subnets AS subnet
MERGE (s:Subnet {id: subnet.id})
SET s += {
name: subnet.name,
type: subnet.type,
addressPrefix: subnet.properties.addressPrefix,
privateEndpointNetworkPolicies: subnet.properties.privateEndpointNetworkPolicies,
privateLinkServiceNetworkPolicies: subnet.properties.privateLinkServiceNetworkPolicies
}
MERGE (obj)-[:HAS_SUBNET]->(s)
"#;
self.process_specific_arm_resource(resource_type, node_insert_query, properties)
.await?;
Ok(())
}
}
cirro-ingest/src/ingest/sql_processors.rs
+58
View File
@@ -0,0 +1,58 @@
use crate::errors::CirroIngestError;
use crate::ingest::ingestor::CirroIngestor;
impl CirroIngestor {
/// Process sql servers
pub async fn process_sql_servers(&self) -> Result<(), CirroIngestError> {
let resource_type = "microsoft.sql/servers";
let properties = vec!["/id", "/properties"];
let node_insert_query = r#"
UNWIND $batch AS row
MERGE (obj:ArmResource {id: row.id})
SET obj:SqlServer
SET obj += {
administratorLogin: row.properties.administratorLogin,
fullyQualifiedDomainName: row.properties.fullyQualifiedDomainName,
publicNetworkAccess: row.properties.publicNetworkAccess,
restrictOutboundNetworkAccess: row.properties.restrictOutboundNetworkAccess,
state: row.properties.state,
version: row.properties.version
}
"#;
self.process_specific_arm_resource(resource_type, node_insert_query, properties)
.await?;
Ok(())
}
/// Process sql databases
// TODO: The server this database belongs to should come from the id property, not managedBy.
// https://github.com/memgraph/mage/issues/642
pub async fn process_sql_databases(&self) -> Result<(), CirroIngestError> {
let resource_type = "microsoft.sql/servers/databases";
let properties = vec!["/id", "/properties", "/managedBy"];
let node_insert_query = r#"
UNWIND $batch AS row
MERGE (obj:ArmResource {id: row.id})
SET obj:SqlDatabase
SET obj += {
collation: row.properties.collation,
creationDate: row.properties.creationDate,
currentServiceObjectiveName: row.properties.currentServiceObjectiveName,
databaseId: row.properties.databaseId,
isInfraEncryptionEnabled: row.properties.isInfraEncryptionEnabled,
maxSizeBytes: row.properties.maxSizeBytes,
status: row.properties.status
}
MERGE (s:ArmResource {id: toLower(row.managedBy)})
MERGE (s)-[:HAS_DB]->(obj)
"#;
self.process_specific_arm_resource(resource_type, node_insert_query, properties)
.await?;
Ok(())
}
}
cirro-ingest/src/ingest/storage_processors.rs
+61
View File
@@ -0,0 +1,61 @@
use crate::errors::CirroIngestError;
use crate::ingest::ingestor::CirroIngestor;
impl CirroIngestor {
/// Process storage accounts
pub async fn process_storage_accounts(&self) -> Result<(), CirroIngestError> {
let resource_type = "microsoft.storage/storageaccounts";
let properties = vec!["/id", "/properties"];
let node_insert_query = r#"
UNWIND $batch AS row
MERGE (obj:ArmResource {id: row.id})
SET obj:StorageAccount
SET obj += {
accessTier: row.properties.accessTier,
primaryLocation : row.properties.primaryLocation,
publicNetworkAccess: row.properties.publicNetworkAccess,
defaultToOAuthAuthentication: row.properties.defaultToOAuthAuthentication,
allowBlobPublicAccess: row.properties.allowBlobPublicAccess,
allowCrossTenantReplication: row.properties.allowCrossTenantReplication,
allowSharedKeyAccess: row.properties.allowSharedKeyAccess,
largeFileSharesState : row.properties.largeFileSharesState,
isLocalUserEnabled: row.properties.isLocalUserEnabled,
isNfsV3Enabled: row.properties.isNfsV3Enabled,
isSftpEnabled: row.properties.isSftpEnabled,
supportsHttpsTrafficOnly: row.properties.supportsHttpsTrafficOnly,
minimumTlsVersion: row.properties.minimumTlsVersion,
networkAclBypass: row.properties.networkAcls.bypass,
networkAclDefaultAction: row.properties.networkAcls.defaultAction
}
"#;
self.process_specific_arm_resource(resource_type, node_insert_query, properties)
.await?;
Ok(())
}
/// Process classic storage accounts
pub async fn process_classic_storage_accounts(&self) -> Result<(), CirroIngestError> {
let resource_type = "microsoft.classicstorage/storageaccounts";
let properties = vec!["/id", "/properties"];
let node_insert_query = r#"
UNWIND $batch AS row
MERGE (obj:ArmResource {id: row.id})
SET obj:ClassicStorageAccount
SET obj += {
accountType: row.properties.accountType,
creationTime: row.properties.creationTime,
geoPrimaryRegion: row.properties.geoPrimaryRegion,
geoSecondaryRegion: row.properties.geoSecondaryRegion,
statusOfPrimaryRegion: row.properties.statusOfPrimaryRegion,
statusOfSecondaryRegion: row.properties.statusOfSecondaryRegion,
}
"#;
self.process_specific_arm_resource(resource_type, node_insert_query, properties)
.await?;
Ok(())
}
}
cirro-ingest/src/ingest/web_processors.rs
+1
View File
@@ -0,0 +1 @@
src/graph/logger.rs → cirro-ingest/src/logger.rs
View File
File renamed without changes.
cirro-ingest/src/main.rs
+106
View File
@@ -0,0 +1,106 @@
pub mod errors;
pub mod ingest;
pub mod logger;
pub mod styles;
use crate::errors::CirroIngestError;
use crate::logger::setup_logger;
use clap::{ColorChoice, Parser, crate_authors, crate_description, crate_version};
use colored::Colorize;
use std::path::PathBuf;
use tokio;
#[derive(Debug, Parser)]
#[clap(
about = crate_description!(),
version = crate_version!(),
bin_name = "cirro-ingest",
author = crate_authors!(),
styles = styles::get_styles(),
color = ColorChoice::Always,
arg_required_else_help = true,
)]
#[command(disable_help_subcommand = true)]
struct Cli {
/// Cirro results database file
#[arg(short, long, value_name = "FILE", value_hint = clap::ValueHint::FilePath)]
file: PathBuf,
/// Database type
#[arg(short, long, value_name = "GRAPH_TYPE", default_value = "neo4j")]
graph_type: ingest::constants::GraphType,
/// Database server. Possible schemes: bolt, bolt+s, bolt+ssc, neo4j, neo4j+s, neo4j+ssc
#[arg(
short,
long,
value_name = "SERVER",
default_value = "bolt://localhost:7687"
)]
server: String,
/// Database user
#[arg(short, long, value_name = "USER", default_value = "neo4j")]
user: String,
/// Database password
#[arg(short, long, value_name = "PASSWORD", default_value = "password")]
password: String,
/// Database name. Defaults to "neo4j" or "memgraph" depending on the graph type.
#[arg(short, long, value_name = "NAME")]
db_name: Option<String>,
/// Enable debug logging
#[arg(long, action = clap::ArgAction::SetTrue)]
debug: bool,
}
#[tokio::main]
async fn main() -> Result<(), CirroIngestError> {
let cli = Cli::parse();
let logo = r#"
___ _
/ __| (_) _ _ _ _ ___
| (__ | | | '_| | '_| / _ \
\___| _|_|_ _|_|_ _|_|_ \___/
_|"""""|_|"""""|_|"""""|_|"""""|_|"""""|
"`-0-0-'"`-0-0-'"`-0-0-'"`-0-0-'"`-0-0-'
"#;
println!("{}", logo.blue().bold());
if let Err(e) = setup_logger(cli.debug) {
return Err(CirroIngestError::LogSetupError(e));
}
// Validate the database host
if !cli.server.starts_with("bolt://")
&& !cli.server.starts_with("bolt+s://")
&& !cli.server.starts_with("bolt+ssc://")
&& !cli.server.starts_with("neo4j://")
&& !cli.server.starts_with("neo4j+s://")
&& !cli.server.starts_with("neo4j+ssc://")
{
return Err(CirroIngestError::InvalidConfig(
"Database host must start with bolt://, bolt+s://, bolt+ssc://, neo4j://, neo4j+s://, or neo4j+ssc://".into(),
));
}
// Create the ingestor
let mut ingestor = ingest::ingestor::CirroIngestor::new(
cli.file,
cli.graph_type,
cli.server,
cli.user,
cli.password,
cli.db_name,
)
.await;
// Run the ingestor
if let Err(e) = ingestor.run().await {
return Err(e);
}
Ok(())
}
src/collect/styles.rs → cirro-ingest/src/styles.rs
View File
File renamed without changes.
cirro/Cargo.toml
+34
View File
@@ -0,0 +1,34 @@
[package]
authors = ["Leron Gray <[email protected]>"]
description = "A CLI tool for graphing Azure and Entra ID data"
edition = "2024"
name = "cirro"
repository = "https://github.com/bishopfox/cirro"
version = "0.1.0"
[dependencies]
async-trait = "0.1.78"
base64 = "0.22.1"
chrono = "0.4.41"
clap = { version = "4.5.41", features = ["cargo", "derive"] }
colored = "3.0.0"
dashmap = "5.5.3"
fern = { version = "0.7.1", features = ["chrono"] }
futures = "0.3.31"
jsonwebtoken = { version = "9.3.1", features = ["use_pem"] }
log = "0.4.27"
once_cell = "1.19.0"
pem = "3.0.5"
reqwest = { version = "0.12.22", default-features = false, features = [
"json",
"rustls-tls-native-roots",
] }
rsa = { version = "0.9.8", features = ["sha1", "sha2"] }
rusqlite = { version = "0.37.0", features = ["bundled"] }
serde = "1.0.219"
serde_json = "1.0.140"
sha1 = "0.10.6"
thiserror = "1.0.57"
tokio = { version = "1.46.1", features = ["full"] }
uuid = { version = "1.17.0", features = ["v4"] }
which = "8.0.0"
cirro/src/collect.rs
+514
View File
@@ -0,0 +1,514 @@
use crate::EnrichmentFlags;
use crate::collectors::arm;
use crate::collectors::graph;
use crate::context::CloudEndpoints;
use crate::context::CollectorContext;
use crate::credentials::azcli::AzureCliCredential;
use crate::credentials::common::AuthCredential;
use crate::credentials::common::AuthError;
use crate::db::{DBWriteMessage, SqliteDb};
use crate::errors::CirroError;
use crate::{AzureCloud, EnumerationMode};
use futures;
use log::{debug, error, info};
use std::ops::{Deref, DerefMut};
use std::path::PathBuf;
use std::sync::Arc;
pub struct Collector {
context: CollectorContext<Box<dyn AuthCredential + Send + Sync>>,
}
/// Implements Deref for Collector to allow direct access to CollectorContext
impl Deref for Collector {
type Target = CollectorContext<Box<dyn AuthCredential + Send + Sync>>;
fn deref(&self) -> &Self::Target {
&self.context
}
}
/// Implements DerefMut for Collector to allow mutable access to CollectorContext
impl DerefMut for Collector {
fn deref_mut(&mut self) -> &mut Self::Target {
&mut self.context
}
}
/// Runs the collector, performing the actual data collection
async fn run_collector(mut collector: Collector) -> Result<(), CirroError> {
info!("Using cloud: {:?}", collector.cloud);
info!("Using mode: {:?}", collector.mode);
info!(
"Using output: {:?}",
collector.output_path.to_path_buf().as_path()
);
// Create the database writer
let (tx, rx) = tokio::sync::mpsc::unbounded_channel::<DBWriteMessage>();
let db_tx = Arc::new(tx);
collector.db_writer = Some(db_tx.clone());
let output_path = collector.output_path.clone();
tokio::spawn(async move {
let sqlite_db = SqliteDb::new(output_path);
sqlite_db.run_writer(rx).await;
});
let graph_collector = Arc::new(collector);
let arm_collector: Arc<Collector> = Arc::clone(&graph_collector);
// Prepare async tasks for concurrent execution
let mut tasks = Vec::new();
// Configure and prepare Graph API enumeration task
if graph_collector.mode == EnumerationMode::Graph
|| graph_collector.mode == EnumerationMode::Both
{
debug!("MS Graph enumeration mode is enabled");
// Pre-fetch token to validate credentials
if let Err(e) = graph_collector.msgraph_credential.get_token().await {
return Err(CirroError::AuthError(e));
} else {
debug!("MS Graph token retrieved successfully");
}
// Create a task for Graph enumeration
let graph_collector_clone = Arc::clone(&graph_collector);
let graph_task = tokio::spawn(async move {
debug!("Starting Graph API enumeration task");
if let Err(e) = graph::enumerate_graph(graph_collector_clone).await {
error!("Error collecting Graph data: {}", e);
}
debug!("Graph API enumeration task completed");
});
tasks.push(graph_task);
}
// Configure and prepare ARM API enumeration task
if arm_collector.mode == EnumerationMode::Arm || arm_collector.mode == EnumerationMode::Both {
debug!("ARM enumeration mode is enabled");
// Pre-fetch token to validate credentials
if let Err(e) = arm_collector.arm_credential.get_token().await {
return Err(CirroError::AuthError(e));
} else {
debug!("ARM token retrieved successfully");
}
// Create a task for ARM enumeration
let arm_collector_clone = Arc::clone(&arm_collector);
let arm_task = tokio::spawn(async move {
debug!("Starting ARM API enumeration task");
if let Err(e) = arm::enumerate_arm(arm_collector_clone).await {
error!("Error collecting ARM data: {}", e);
}
debug!("ARM API enumeration task completed");
});
tasks.push(arm_task);
}
// Wait for all tasks to complete
if !tasks.is_empty() {
debug!("Waiting for {} enumeration tasks to complete", tasks.len());
futures::future::join_all(tasks).await;
info!("All enumeration tasks completed");
}
// Wait for the DB writer to finish
info!("Sending shutdown message to DB writer");
db_tx.send(DBWriteMessage::Shutdown).unwrap();
// Wait for the DB writer to process the shutdown message
info!("Sleeping for 5 seconds to ensure DB writer processes shutdown");
tokio::time::sleep(tokio::time::Duration::from_secs(5)).await;
info!("Done");
Ok(())
}
/// Runs the enrichment functions
async fn run_enrichment(mut collector: Collector) -> Result<(), CirroError> {
info!("Using cloud: {:?}", collector.cloud);
info!(
"Using output: {:?}",
collector.output_path.to_path_buf().file_name().unwrap()
);
// Create the database writer
let (tx, rx) = tokio::sync::mpsc::unbounded_channel::<DBWriteMessage>();
let db_tx = Arc::new(tx);
collector.db_writer = Some(db_tx.clone());
let output_path = collector.output_path.clone();
let output_path_for_db = output_path.clone();
tokio::spawn(async move {
let db_writer = SqliteDb::new(output_path_for_db);
db_writer.run_writer(rx).await;
});
// Run the enrichment functions
// Check each flag in enrich_flags and print if true
// If a function fails, continue to the next one
let sqlite_conn = SqliteDb::new(output_path).get_connection().await?;
let enrich_flags = collector.enrich_flags.as_ref().unwrap();
if enrich_flags.storage_keys {
if let Err(e) = collector.enrich_storage_account_keys(sqlite_conn).await {
error!("Error enriching storage blobs: {}", e);
}
}
// Wait for the DB writer to finish
info!("Sending shutdown message to DB writer");
db_tx.send(DBWriteMessage::Shutdown).unwrap();
// Wait for the DB writer to process the shutdown message
info!("Sleeping for 5 seconds to ensure DB writer processes shutdown");
tokio::time::sleep(tokio::time::Duration::from_secs(5)).await;
info!("Done");
Ok(())
}
/// Collects data using Access Token credentials
pub async fn collect_with_access_token(
token: String,
cloud: AzureCloud,
output_path: PathBuf,
enrich_mode: bool,
enrich_flags: Option<EnrichmentFlags>,
) -> Result<(), CirroError> {
// We don't really need to create all of these credentials for Access Token mode,
// but we do it to keep the interface consistent with other authentication modes.
// We can probably optimize this later.
let mut arm_credential = Box::new(crate::credentials::access_token::AccessTokenCredential {
token: tokio::sync::RwLock::new(crate::credentials::common::Token::default()),
});
let mut msgraph_credential =
Box::new(crate::credentials::access_token::AccessTokenCredential {
token: tokio::sync::RwLock::new(crate::credentials::common::Token::default()),
});
let vault_credential = Box::new(crate::credentials::access_token::AccessTokenCredential {
token: tokio::sync::RwLock::new(crate::credentials::common::Token::default()),
});
// Get the token audience and compute the cloud endpoints
// This is necessary to ensure the correct resource URLs are used for the token.
let cloud_endpoints: CloudEndpoints = CloudEndpoints::new(cloud);
let mut token = crate::credentials::common::Token {
access_token: Arc::new(token),
expires_on: None,
refresh_token: None,
};
token.set_expires_on_from_token().map_err(|e| {
CirroError::AuthError(AuthError::ParseError(format!(
"Failed to set expires_on: {}",
e
)))
})?;
let credential = Box::new(crate::credentials::access_token::AccessTokenCredential {
token: tokio::sync::RwLock::new(token.clone()),
});
// Get audience claim
let audience = credential.get_token().await?.get_claims().unwrap().aud;
let mode: EnumerationMode;
info!("Using token with audience: {}", audience);
match audience.as_str() {
// Audience can sometimes be the App ID URI or the resource URL
_ if audience == cloud_endpoints.arm_url
|| audience == "00000002-0000-0000-c000-000000000000" =>
{
mode = EnumerationMode::Arm;
arm_credential = credential;
}
_ if audience == cloud_endpoints.msgraph_url
|| audience == "00000003-0000-0000-c000-000000000000" =>
{
mode = EnumerationMode::Graph;
msgraph_credential = credential;
}
_ => {
return Err(CirroError::AuthError(AuthError::Unknown(
"Invalid token audience".into(),
)));
}
}
// Enrich mode is only supported in Arm mode
if enrich_mode && mode != EnumerationMode::Arm {
return Err(CirroError::AuthError(AuthError::Unknown(
"Enrichment mode is only supported in Arm mode".into(),
)));
}
info!("Starting with Access Token credentials");
// Initialize the collector context with Access Token credentials
let collector = Collector {
context: CollectorContext::<Box<dyn AuthCredential + Send + Sync>> {
mode,
cloud,
cloud_endpoints: CloudEndpoints::new(cloud),
msgraph_credential: msgraph_credential as Box<dyn AuthCredential + Send + Sync>,
arm_credential: arm_credential as Box<dyn AuthCredential + Send + Sync>,
vault_credential: vault_credential as Box<dyn AuthCredential + Send + Sync>,
output_path,
tenant_id: None,
subscription_id: None,
client_id: None,
client_secret: None,
client_cert_path: None,
db_writer: None,
enrich_mode,
enrich_flags,
},
};
// Run the collector
if collector.enrich_mode {
if let Err(e) = run_enrichment(collector).await {
error!("Error enriching data: {}", e);
return Err(e);
}
} else {
if let Err(e) = run_collector(collector).await {
error!("Error collecting data: {}", e);
return Err(e);
}
}
Ok(())
}
/// Collects data using Azure CLI credentials
pub async fn collect_with_azure_cli(
tenant_id: Option<String>,
subscription_id: Option<String>,
mode: EnumerationMode,
cloud: AzureCloud,
output_path: PathBuf,
enrich_mode: bool,
enrich_flags: Option<EnrichmentFlags>,
) -> Result<(), CirroError> {
info!("Starting with Azure CLI credentials");
let cloud_endpoints = CloudEndpoints::new(cloud);
// Initialize the Azure CLI credentials
let arm_credential = Box::new(AzureCliCredential {
tenant_id: tenant_id.clone(),
resource: &cloud_endpoints.arm_url,
token: tokio::sync::RwLock::new(crate::credentials::common::Token::default()),
});
let msgraph_credential = Box::new(AzureCliCredential {
tenant_id: tenant_id.clone(),
resource: &cloud_endpoints.msgraph_url,
token: tokio::sync::RwLock::new(crate::credentials::common::Token::default()),
});
let vault_credential = Box::new(AzureCliCredential {
tenant_id: tenant_id.clone(),
resource: &cloud_endpoints.vault_url,
token: tokio::sync::RwLock::new(crate::credentials::common::Token::default()),
});
// Initialize the collector context with Azure CLI credentials
let collector = Collector {
context: CollectorContext::<Box<dyn AuthCredential + Send + Sync>> {
mode,
cloud,
cloud_endpoints: CloudEndpoints::new(cloud),
msgraph_credential: msgraph_credential as Box<dyn AuthCredential + Send + Sync>,
arm_credential: arm_credential as Box<dyn AuthCredential + Send + Sync>,
vault_credential: vault_credential as Box<dyn AuthCredential + Send + Sync>,
output_path: output_path,
tenant_id: tenant_id.clone(),
subscription_id,
client_id: None,
client_secret: None,
client_cert_path: None,
db_writer: None,
enrich_mode,
enrich_flags,
},
};
// Run the collector
if collector.enrich_mode {
if let Err(e) = run_enrichment(collector).await {
error!("Error enriching data: {}", e);
return Err(e);
}
} else {
if let Err(e) = run_collector(collector).await {
error!("Error collecting data: {}", e);
return Err(e);
}
}
Ok(())
}
/// Collects data using Client Secret credentials
pub async fn collect_with_client_secret(
tenant_id: String,
client_id: String,
client_secret: String,
mode: EnumerationMode,
cloud: AzureCloud,
output_path: PathBuf,
enrich_mode: bool,
enrich_flags: Option<EnrichmentFlags>,
) -> Result<(), CirroError> {
info!("Starting with Client Secret credentials");
let cloud_endpoints = CloudEndpoints::new(cloud);
// Initialize the Client Secret credentials
let arm_credential = Box::new(crate::credentials::clientsecret::ClientSecretCredential {
token_endpoint: &cloud_endpoints.token_endpoint,
client_id: client_id.clone(),
client_secret: client_secret.clone(),
tenant_id: tenant_id.clone(),
resource: &cloud_endpoints.arm_url,
token: tokio::sync::RwLock::new(crate::credentials::common::Token::default()),
});
let msgraph_credential = Box::new(crate::credentials::clientsecret::ClientSecretCredential {
token_endpoint: &cloud_endpoints.token_endpoint,
client_id: client_id.clone(),
client_secret: client_secret.clone(),
tenant_id: tenant_id.clone(),
resource: cloud_endpoints.msgraph_url,
token: tokio::sync::RwLock::new(crate::credentials::common::Token::default()),
});
let vault_credential = Box::new(crate::credentials::clientsecret::ClientSecretCredential {
token_endpoint: &cloud_endpoints.token_endpoint,
client_id: client_id.clone(),
client_secret: client_secret.clone(),
tenant_id: tenant_id.clone(),
resource: cloud_endpoints.vault_url,
token: tokio::sync::RwLock::new(crate::credentials::common::Token::default()),
});
// Initialize the collector context with Client Secret credentials
let collector = Collector {
context: CollectorContext::<Box<dyn AuthCredential + Send + Sync>> {
mode,
cloud,
cloud_endpoints: CloudEndpoints::new(cloud),
msgraph_credential: msgraph_credential as Box<dyn AuthCredential + Send + Sync>,
arm_credential: arm_credential as Box<dyn AuthCredential + Send + Sync>,
vault_credential: vault_credential as Box<dyn AuthCredential + Send + Sync>,
output_path,
tenant_id: Some(tenant_id.clone()),
subscription_id: None,
client_id: None,
client_secret: None,
client_cert_path: None,
db_writer: None,
enrich_mode,
enrich_flags,
},
};
// Run the collector
if collector.enrich_mode {
if let Err(e) = run_enrichment(collector).await {
error!("Error enriching data: {}", e);
return Err(e);
}
} else {
if let Err(e) = run_collector(collector).await {
error!("Error collecting data: {}", e);
return Err(e);
}
}
Ok(())
}
/// Collects data using Client Certificate credentials
pub async fn collect_with_client_cert(
tenant_id: String,
client_id: String,
client_certificate: PathBuf,
mode: EnumerationMode,
cloud: AzureCloud,
output_path: PathBuf,
enrich_mode: bool,
enrich_flags: Option<EnrichmentFlags>,
) -> Result<(), CirroError> {
info!("Starting with Client Certificate credentials");
let cloud_endpoints = CloudEndpoints::new(cloud);
// Initialize the Client Certificate credentials
let arm_credential = Box::new(
crate::credentials::clientcert::ClientCertificateCredential {
token_endpoint: &cloud_endpoints.token_endpoint,
client_id: client_id.clone(),
certificate_path: client_certificate.clone(),
tenant_id: tenant_id.clone(),
resource: &cloud_endpoints.arm_url,
token: tokio::sync::RwLock::new(crate::credentials::common::Token::default()),
},
);
let msgraph_credential = Box::new(
crate::credentials::clientcert::ClientCertificateCredential {
token_endpoint: &cloud_endpoints.token_endpoint,
client_id: client_id.clone(),
certificate_path: client_certificate.clone(),
tenant_id: tenant_id.clone(),
resource: cloud_endpoints.msgraph_url,
token: tokio::sync::RwLock::new(crate::credentials::common::Token::default()),
},
);
let vault_credential = Box::new(
crate::credentials::clientcert::ClientCertificateCredential {
token_endpoint: &cloud_endpoints.token_endpoint,
client_id: client_id.clone(),
certificate_path: client_certificate.clone(),
tenant_id: tenant_id.clone(),
resource: cloud_endpoints.vault_url,
token: tokio::sync::RwLock::new(crate::credentials::common::Token::default()),
},
);
// Initialize the collector context with Client Certificate credentials
let collector = Collector {
context: CollectorContext::<Box<dyn AuthCredential + Send + Sync>> {
mode,
cloud,
cloud_endpoints: CloudEndpoints::new(cloud),
msgraph_credential: msgraph_credential as Box<dyn AuthCredential + Send + Sync>,
arm_credential: arm_credential as Box<dyn AuthCredential + Send + Sync>,
vault_credential: vault_credential as Box<dyn AuthCredential + Send + Sync>,
output_path,
tenant_id: Some(tenant_id.clone()),
subscription_id: None,
client_id: Some(client_id),
client_secret: None,
client_cert_path: Some(client_certificate),
db_writer: None,
enrich_mode,
enrich_flags: enrich_flags,
},
};
// Run the collector
if collector.enrich_mode {
if let Err(e) = run_enrichment(collector).await {
error!("Error enriching data: {}", e);
return Err(e);
}
} else {
if let Err(e) = run_collector(collector).await {
error!("Error collecting data: {}", e);
return Err(e);
}
}
Ok(())
}
cirro/src/collectors/arm.rs
+780
View File
@@ -0,0 +1,780 @@
use crate::collect::Collector;
use crate::collectors::common::*;
use crate::errors::CirroError;
use dashmap::DashMap;
use log::warn;
use log::{debug, info};
use once_cell::sync::Lazy;
use reqwest::Client;
use reqwest::StatusCode;
use serde_json::Map;
use serde_json::Value;
use std::sync::Arc;
// Create a global reqwest client to reuse connections
static HTTP_CLIENT: Lazy<Client> = Lazy::new(|| {
Client::builder()
.pool_idle_timeout(std::time::Duration::from_secs(30))
.tcp_keepalive(Some(std::time::Duration::from_secs(60)))
.pool_max_idle_per_host(100)
.build()
.unwrap()
});
// Global static cache for the token
static TOKEN_CACHE: Lazy<tokio::sync::Mutex<Option<crate::credentials::common::Token>>> =
Lazy::new(|| tokio::sync::Mutex::new(None));
// Global thread-safe map for role definitions
// Key: Role definition ID, Value: Role definition object
static ROLE_DEFINITIONS: Lazy<DashMap<String, Value>> = Lazy::new(|| DashMap::new());
/// Queries resources from the ARM API
async fn query_resources(
collector: Arc<Collector>,
uri: &str,
) -> Result<Vec<Map<String, Value>>, CirroError> {
debug!("Querying: {}", uri);
let mut resources: Vec<Map<String, Value>> = Vec::new();
let base_url = collector.cloud_endpoints.arm_url;
let mut next_url = if uri.starts_with(base_url) {
uri.to_owned()
} else {
format!(
"{}/{}",
base_url.trim_end_matches('/'),
uri.trim_start_matches('/')
)
};
loop {
let response = paged_arm_request(&collector, &next_url, reqwest::Method::GET, None).await?;
if response.get("error").is_some() {
return Err(CirroError::HttpError(
"Error response from ARM API".to_string(),
));
}
if let Some(value) = response.get("value") {
// Sometimes the value is an array, sometimes it's a single object
if let Some(obj) = value.as_object() {
resources.push(obj.clone());
} else if let Some(arr) = value.as_array() {
for item in arr {
if let Some(obj) = item.as_object() {
resources.push(obj.clone());
}
}
}
} else if let Some(obj) = response.as_object() {
// Also handle the case where the response is a single object
resources.push(obj.clone());
} else {
return Err(CirroError::HttpError(format!(
"Unexpected response format from ARM API: {}",
next_url
)));
}
// Check for next link
if let Some(next_link) = response.get("@odata.nextLink").and_then(|v| v.as_str()) {
next_url = next_link.to_string();
} else {
break; // No more pages to fetch
}
}
Ok(resources.into())
}
/// Makes a paginated request to the ARM API with retry logic
pub async fn paged_arm_request(
collector: &Collector,
uri: &str,
http_verb: reqwest::Method,
body: Option<String>,
) -> Result<Value, CirroError> {
// Build the full URL correctly
let arm_url = if uri.starts_with(&collector.cloud_endpoints.arm_url) {
uri.to_owned()
} else {
format!(
"{}/{}",
collector.cloud_endpoints.arm_url.trim_end_matches('/'),
uri.trim_start_matches('/')
)
};
// Get the access token, using cache when possible
let token_str = {
let mut cache = TOKEN_CACHE.lock().await;
// Check if we need a new token based on the token's actual expiration
let need_new_token = match &*cache {
Some(cached_token) => cached_token.is_expired_or_not_set(),
None => true,
};
if need_new_token {
debug!("Fetching new ARM token");
let token = collector.arm_credential.get_token().await?;
// Log expiration information for debugging
if let Some(expires) = token.expires_on {
let now = chrono::Utc::now();
let time_until_expiry = expires.signed_duration_since(now);
debug!(
"New token expires at {} (valid for {} minutes)",
expires,
time_until_expiry.num_minutes()
);
} else {
debug!("New token has no expiration time set");
}
*cache = Some(token.clone());
Arc::clone(&token.access_token)
} else {
// Return a clone of the cached token
Arc::clone(&cache.as_ref().unwrap().access_token)
}
};
let mut retries = 0;
let max_retries = 5;
let response = loop {
// Make the request using the global client
let result = match http_verb {
reqwest::Method::GET => {
HTTP_CLIENT
.get(&arm_url)
.bearer_auth(&*token_str)
.timeout(std::time::Duration::from_secs(30))
.send()
.await
}
reqwest::Method::POST => {
HTTP_CLIENT
.post(&arm_url)
.bearer_auth(&*token_str)
.body(body.clone().unwrap_or_default())
.header("content-length", body.as_ref().map_or(0, |b| b.len()))
.timeout(std::time::Duration::from_secs(30))
.send()
.await
}
_ => return Err(CirroError::UnsupportedHttpMethod(http_verb)),
};
let response = result.map_err(|e| {
CirroError::HttpError(format!("Failed to send request to {}: {}", arm_url, e))
})?;
let status = response.status();
// Check for non-200 status codes early to avoid parsing JSON for error responses
if !status.is_success() {
let error_text = response.text().await?;
return Err(CirroError::HttpError(format!(
"HTTP {} - {}",
status.as_u16(),
error_text
)));
} else if status == StatusCode::TOO_MANY_REQUESTS {
// Handle rate limiting by checking for Too Many Requests status
if retries >= max_retries {
return Err(CirroError::HttpError(
"Too many requests, exceeded max retries".to_string(),
));
}
let retry_after = response
.headers()
.get(reqwest::header::RETRY_AFTER)
.and_then(|h| h.to_str().ok())
.and_then(|s| s.parse::<u64>().ok())
.unwrap_or(30); // Default to 30 seconds if not specified
info!("Rate limit hit, retrying after {} seconds", retry_after);
tokio::time::sleep(std::time::Duration::from_secs(retry_after + 1)).await;
retries += 1;
continue; // Retry the request after waiting
} else if status.is_success() {
break response;
} else {
let error_text = response.text().await?;
return Err(CirroError::HttpError(format!(
"HTTP {} - {}",
status.as_u16(),
error_text
)));
}
};
// Parse the response data
let response_data: Value = response.json::<Value>().await?;
// Check for OData error
if let Some(err) = response_data.get("@odata.error") {
let msg = err
.get("message")
.and_then(|m| m.get("value"))
.and_then(|v| v.as_str())
.unwrap_or("unknown odata.error");
return Err(CirroError::ODataError(msg.to_string()));
}
Ok(response_data)
}
/// Enumerates role assignments for a given scope
async fn enumerate_role_assignments(
collector: Arc<Collector>,
scope: &str,
) -> Vec<Map<String, Value>> {
let mut roles: Vec<Map<String, Value>> = Vec::new();
let uri = format!(
"{}/providers/Microsoft.Authorization/roleAssignments?api-version=2022-04-01",
scope
);
debug!("Querying role assignments at {}", uri);
let assignments = query_resources(collector.clone(), &uri)
.await
.unwrap_or_else(|e| {
info!(
"Failed to query role assignments for scope {}: {}",
scope, e
);
roles.clone()
});
if assignments.is_empty() {
info!("No role assignments found for scope {}", scope);
return roles;
}
// Need to get the roleDefinitionId from the properties map
for assignment in assignments {
if let Some(properties) = assignment.get("properties").and_then(Value::as_object) {
if let Some(role_definition_id) =
properties.get("roleDefinitionId").and_then(Value::as_str)
{
let definition_uri = format!(
"{}?disambiguation_dummy&api-version=2022-04-01",
role_definition_id
);
let definition_response: Value;
if let Some(cached_definition) = ROLE_DEFINITIONS.get(role_definition_id) {
definition_response = cached_definition.clone();
} else {
match paged_arm_request(&collector, &definition_uri, reqwest::Method::GET, None)
.await
{
Ok(definition) => {
ROLE_DEFINITIONS
.insert(role_definition_id.to_string(), definition.clone());
definition_response = definition;
}
Err(e) => {
warn!(
"Failed to fetch role definition for {}: {}",
role_definition_id, e
);
continue; // Skip this assignment if we can't fetch the definition
}
}
}
// Add the role definition to the assignment
let mut assignment_with_definition = assignment.clone();
assignment_with_definition.insert(
"permissions".to_string(),
definition_response
.pointer("/properties/permissions")
.cloned()
.unwrap_or(Value::Null),
);
assignment_with_definition.insert(
"roleName".to_string(),
definition_response
.pointer("/properties/roleName")
.cloned()
.unwrap_or(Value::Null),
);
assignment_with_definition.insert(
"roleType".to_string(),
definition_response
.pointer("/properties/type")
.cloned()
.unwrap_or(Value::Null),
);
assignment_with_definition.insert(
"description".to_string(),
definition_response
.pointer("/properties/description")
.cloned()
.unwrap_or(Value::Null),
);
// Add the assignment to the roles list
roles.push(assignment_with_definition);
}
}
}
return roles;
}
/// Enumerates a subscription
async fn enumerate_subscription(
collector: Arc<Collector>,
subscription: &Map<String, Value>,
) -> Result<(), CirroError> {
let subscription_id = subscription
.get("subscriptionId")
.and_then(Value::as_str)
.ok_or_else(|| {
CirroError::ArmApiError("Subscription ID not found in subscription object".to_string())
})?;
let subscription_name = subscription
.get("displayName")
.and_then(Value::as_str)
.unwrap_or("Unknown Subscription");
info!(
"Enumerating ARM subscription: {} ({})",
subscription_id, subscription_name,
);
// Get the role assignments for this subscription
let roles = enumerate_role_assignments(
collector.clone(),
format!("subscriptions/{}", subscription_id).as_str(),
)
.await;
if roles.is_empty() {
info!(
"No role assignments found for subscription {}",
subscription_id
);
return Ok(());
}
info!(
"Found {} role assignments for subscription {} ({})",
roles.len(),
subscription_id,
subscription_name
);
// Write the roles to the database
for role in &roles {
let role_id = role.get("id").and_then(Value::as_str).ok_or_else(|| {
CirroError::ArmApiError("Role ID not found in role object".to_string())
})?;
let role_data = serde_json::to_value(role.clone())
.map_err(|e| CirroError::SerializationError(e.to_string()))?;
let collector_clone = collector.clone();
async move {
let _ = collector_clone
.write_value_to_db("roleAssignments".into(), role_id.to_lowercase(), role_data)
.await;
}
.await;
}
// Get all the resource providers for this subscription
// These are stored for reuse to avoid repetitive API calls
let resource_providers: DashMap<String, Vec<String>> = DashMap::new();
let provider_uri = format!(
"subscriptions/{}/providers?api-version=2021-04-01",
subscription_id
);
let providers_response = query_resources(collector.clone(), &provider_uri)
.await
.map_err(|e| CirroError::ArmApiError(format!("Failed to query providers: {}", e)))?;
if providers_response.is_empty() {
info!(
"No resource providers found for subscription {}",
subscription_id
);
return Ok(());
}
for provider in providers_response {
let namespace = provider
.get("namespace")
.and_then(Value::as_str)
.ok_or_else(|| {
CirroError::ArmApiError("Namespace not found in provider object".to_string())
})?;
let resource_types = provider
.get("resourceTypes")
.and_then(Value::as_array)
.ok_or_else(|| {
CirroError::ArmApiError("Resource types not found in provider object".to_string())
})?;
for resource_type in resource_types {
let type_name = namespace.to_string()
+ "/"
+ resource_type
.get("resourceType")
.and_then(Value::as_str)
.ok_or_else(|| {
CirroError::ArmApiError(
"Resource type not found in resource type object".to_string(),
)
})?;
let type_name = type_name.to_lowercase();
let api_versions = resource_type.get("apiVersions").ok_or_else(|| {
CirroError::ArmApiError("API version not found in resource type object".to_string())
})?;
let api_versions_array = api_versions.as_array().ok_or_else(|| {
CirroError::ArmApiError(format!(
"API versions is not an array for resource type {}: {}",
type_name, api_versions
))
})?;
if api_versions_array.is_empty() {
debug!(
"No API versions found for resource type {} in subscription {}",
type_name, subscription_id
);
continue; // Skip this resource type if no API versions are available
}
// Convert JSON values to Strings
let api_versions_strings: Vec<String> = api_versions_array
.iter()
.filter_map(|v| v.as_str().map(String::from))
.collect();
resource_providers.insert(type_name.clone(), api_versions_strings);
}
}
// Enumerate resources for this subscription
let resourcegroup_uri = format!(
"subscriptions/{}/resourceGroups?api-version=2021-04-01",
subscription_id
);
let resource_groups = query_resources(collector.clone(), &resourcegroup_uri)
.await
.map_err(|e| CirroError::ArmApiError(format!("Failed to query resource groups: {}", e)))?;
if resource_groups.is_empty() {
info!(
"No resource groups found for subscription {}",
subscription_id
);
return Ok(());
}
info!(
"Found {} resource groups for subscription {} ({})",
resource_groups.len(),
subscription_id,
subscription_name
);
let mut tasks = Vec::new();
// Write the resource groups to the database
for resource_group in &resource_groups {
let rg_data = serde_json::to_value(resource_group.clone())
.map_err(|e| CirroError::SerializationError(e.to_string()))?;
let rg_id = rg_data
.get("id")
.and_then(Value::as_str)
.ok_or_else(|| {
CirroError::ArmApiError(
"Resource group ID not found in resource group object".to_string(),
)
})?
.to_string();
let resource_type = rg_data
.get("type")
.and_then(Value::as_str)
.ok_or_else(|| {
CirroError::ArmApiError(
"Resource group type not found in resource group object".to_string(),
)
})?
.to_string();
let collector_clone = collector.clone();
let resource_providers_clone = Arc::new(resource_providers.clone());
tasks.push(async move {
let _ = collector_clone
.write_arm_resource_to_db(
rg_id.clone().to_lowercase(),
format!("/subscriptions/{}", subscription_id.to_lowercase()),
rg_id.clone(),
resource_type,
rg_data,
)
.await;
let _ = enumerate_resourcegroup(
collector_clone,
resource_group,
subscription_id,
subscription_name,
&resource_providers_clone,
)
.await;
});
}
// Execute all tasks concurrently
futures::future::join_all(tasks).await;
info!(
"ARM subscription {} ({}) enumeration completed",
subscription_id, subscription_name
);
Ok(())
}
async fn enumerate_resourcegroup(
collector: Arc<Collector>,
resource_group: &Map<String, Value>,
subscription_id: &str,
subscription_name: &str,
api_versions: &Arc<DashMap<String, Vec<String>>>,
) -> Result<(), CirroError> {
let rg_id = resource_group
.get("id")
.and_then(Value::as_str)
.ok_or_else(|| {
CirroError::ArmApiError(
"Resource group ID not found in resource group object".to_string(),
)
})?;
let rg_name = resource_group
.get("name")
.and_then(Value::as_str)
.unwrap_or("Unknown Resource Group");
// Get the role assignments for this resource group
let roles = enumerate_role_assignments(collector.clone(), rg_id).await;
if roles.is_empty() {
debug!(
"No role assignments found for resource group {} in subscription {}",
rg_name, subscription_name
);
} else {
debug!(
"Found {} role assignments for resource group {} in subscription {}",
roles.len(),
rg_name,
subscription_name
);
}
// Write the roles to the database
for role in &roles {
let role_id = role.get("id").and_then(Value::as_str).ok_or_else(|| {
CirroError::ArmApiError("Role ID not found in role object".to_string())
})?;
let role_data = serde_json::to_value(role.clone())
.map_err(|e| CirroError::SerializationError(e.to_string()))?;
let collector_clone = collector.clone();
async move {
let _ = collector_clone
.write_value_to_db("roleAssignments".into(), role_id.to_lowercase(), role_data)
.await;
}
.await;
}
// Enumerate the resources in this resource group
let resource_uri = format!(
"{}/resources?api-version=2023-07-01",
rg_id.trim_end_matches('/')
);
let resources = query_resources(collector.clone(), &resource_uri)
.await
.map_err(|e| CirroError::ArmApiError(format!("Failed to query resources: {}", e)))?;
if resources.is_empty() {
info!(
"No resources found in resource group {} in subscription {}",
rg_name, subscription_name
);
return Ok(());
}
let mut tasks = Vec::new();
for resource in &resources {
let collector_clone = collector.clone();
tasks.push(async move {
let resource_id = resource.get("id").and_then(Value::as_str).unwrap();
let resource_type = resource.get("type").and_then(Value::as_str).unwrap();
// Iterate over the resource providers and request until we find the API version
let provider_api_versions = api_versions
.get(&resource_type.to_lowercase())
.map(|v| v.clone())
.unwrap();
if provider_api_versions.is_empty() {
warn!(
"No API versions found for resource type {} in resource group {}",
resource_type, rg_name
);
return;
}
for api_version in provider_api_versions {
let resource_uri = format!("{}?api-version={}", resource_id, api_version);
let result =
paged_arm_request(&collector_clone, &resource_uri, reqwest::Method::GET, None)
.await;
if let Err(e) = result {
if e.to_string().contains("NoRegisteredProviderFound") {
continue;
} else {
warn!("Failed to query resource {}: {}", resource_id, e);
break; // Break on any error other than NoRegisteredProviderFound
}
} else {
let result = result.unwrap();
let value_clone = serde_json::to_value(result)
.map_err(|e| CirroError::SerializationError(e.to_string()))
.unwrap();
// Write the resource to the database
let _ = collector_clone
.write_arm_resource_to_db(
resource_id.to_string().to_lowercase(),
subscription_id.to_string().to_lowercase(),
rg_id.to_string().to_lowercase(),
resource_type.to_string(),
value_clone,
)
.await
.map_err(|e| CirroError::DatabaseError(e.to_string()))
.unwrap();
break; // Break after the first successful API version
}
}
});
}
// Execute all tasks concurrently
futures::future::join_all(tasks).await;
info!(
"Enumerating resource group: {} - {}",
subscription_name, rg_name
);
Ok(())
}
pub async fn enumerate_arm(collector: Arc<Collector>) -> Result<(), CirroError> {
info!("Starting ARM enumeration");
// Start timer
let start_time = std::time::Instant::now();
// First we need to get the tenants
let tenants = query_resources(collector.clone(), "tenants?api-version=2022-12-01").await?;
if tenants.is_empty() {
return Err(CirroError::ArmApiError("No tenants found".to_string()));
}
info!("Found {} tenants", tenants.len());
debug!("Tenants: {:?}", tenants);
// Write the tenants to the database
for tenant in &tenants {
let tenant_id = tenant.get("id").and_then(Value::as_str).ok_or_else(|| {
CirroError::ArmApiError("Tenant ID not found in tenant object".to_string())
})?;
let tenant_data = serde_json::to_value(tenant.clone())
.map_err(|e| CirroError::SerializationError(e.to_string()))?;
let collector_clone = collector.clone();
async move {
let _ = collector_clone
.write_value_to_db("tenants".into(), tenant_id.to_lowercase(), tenant_data)
.await;
}
.await;
}
// Get the subscriptions
let subscriptions =
query_resources(collector.clone(), "/subscriptions/?api-version=2024-08-01").await?;
if subscriptions.is_empty() {
return Err(CirroError::ArmApiError(
"No subscriptions found".to_string(),
));
}
info!("Found {} subscriptions", subscriptions.len());
debug!("Subscriptions: {:?}", subscriptions);
// Write the subscriptions to the database
for subscription in &subscriptions {
let subscription_id = subscription
.get("subscriptionId")
.and_then(Value::as_str)
.ok_or_else(|| {
CirroError::ArmApiError(
"Subscription ID not found in subscription object".to_string(),
)
})?;
let subscription_data = serde_json::to_value(subscription.clone())
.map_err(|e| CirroError::SerializationError(e.to_string()))?;
let collector_clone = collector.clone();
async move {
let _ = collector_clone
.write_value_to_db(
"subscriptions".into(),
subscription_id.to_lowercase(),
subscription_data,
)
.await;
}
.await;
}
// Now we can query resources for each subscription
// Call enumerate_subscriptions as tasks
let mut tasks = Vec::new();
for subscription in &subscriptions {
let collector_clone = collector.clone();
let subscription_clone = subscription.clone();
tasks.push(tokio::spawn(async move {
let result = enumerate_subscription(collector_clone, &subscription_clone).await;
if let Err(e) = result {
warn!(
"Failed to enumerate subscription {}: {}",
subscription_clone
.get("subscriptionId")
.and_then(Value::as_str)
.unwrap_or("unknown"),
e
);
}
}));
}
if !tasks.is_empty() {
futures::future::join_all(tasks).await;
}
info!(
"ARM enumeration completed in {} seconds",
fmt_duration(start_time.elapsed())
);
Ok(())
}
src/collect/azure/collectors/common.rs → cirro/src/collectors/common.rs
+23
View File
@@ -1,4 +1,27 @@
use crate::credentials::common::AuthError;
use std::time::Duration;
use thiserror::Error;
#[derive(Error, Debug)]
pub enum CollectorError {
#[error("HTTP error: {0}")]
HttpError(String),
#[error("OData error: {0}")]
ODataError(String),
#[error("Authentication error: {0}")]
AuthError(#[from] AuthError),
#[error("Request error: {0}")]
RequestError(#[from] reqwest::Error),
#[error("JSON error: {0}")]
JsonError(#[from] serde_json::Error),
#[error("Unknown error: {0}")]
Unknown(String),
}
pub fn fmt_duration(d: Duration) -> String {
// total seconds as a float
cirro/src/collectors/enrichments.rs
+62
View File
@@ -0,0 +1,62 @@
use crate::collectors::arm::paged_arm_request;
use crate::{collect::Collector, errors::CirroError};
use log::{debug, info, warn};
use rusqlite::Connection;
impl Collector {
/// Fetch storage account keys
pub async fn enrich_storage_account_keys(
&self,
sqlite_db: Connection,
) -> Result<(), CirroError> {
// Get all the storage accounts from the database
let query =
"SELECT id FROM resources WHERE resource_type = 'microsoft.storage/storageaccounts'";
let mut stmt = sqlite_db.prepare(query)?;
let storage_accounts: Vec<String> = stmt
.query_map([], |row| row.get(0))?
.collect::<Result<Vec<String>, rusqlite::Error>>()?;
if storage_accounts.is_empty() {
info!("No storage accounts found for storage key enrichment");
return Ok(());
}
info!(
"Found {} storage accounts for key enrichment",
storage_accounts.len()
);
for storage_account_id in storage_accounts {
let uri = format!(
"{}{}/listKeys?api-version=2024-01-01",
self.cloud_endpoints.arm_url, storage_account_id,
);
// Make the paginated request to fetch the keys
match paged_arm_request(self, &uri, reqwest::Method::POST, None).await {
Ok(response) => {
// Print the keys
info!(
"Fetched keys for storage account {}: \n{}",
storage_account_id,
serde_json::to_string_pretty(&response).unwrap_or_default()
);
// Process the response and write to the database
self.write_enrichment_to_db(
"storage_account_keys".to_string(),
storage_account_id.clone(),
response.clone(),
)
.await?;
}
Err(e) => {
warn!("Failed to fetch keys for {}", storage_account_id);
debug!("Error: {}", e);
}
}
}
Ok(())
}
}
cirro/src/collectors/graph.rs
+519
View File
@@ -0,0 +1,519 @@
use crate::collect::Collector;
use crate::collectors::common::*;
use crate::errors::CirroError;
use futures::stream::{self, StreamExt};
use log::{debug, error, info};
use once_cell::sync::Lazy;
use reqwest::{Client, StatusCode};
use serde_json::Value;
use std::sync::Arc;
use std::time::Instant;
use std::{collections::HashMap, vec};
// Create a global reqwest client to reuse connections
static HTTP_CLIENT: Lazy<Client> = Lazy::new(|| {
Client::builder()
.pool_idle_timeout(std::time::Duration::from_secs(30))
.tcp_keepalive(Some(std::time::Duration::from_secs(60)))
.pool_max_idle_per_host(100)
.build()
.unwrap()
});
// Global static cache for the token
static TOKEN_CACHE: Lazy<tokio::sync::Mutex<Option<crate::credentials::common::Token>>> =
Lazy::new(|| tokio::sync::Mutex::new(None));
/// Represents a Graph object with its resource type, query parameters, and optional expand properties
#[derive(Clone)]
struct GraphObject {
/// The type of resource in the Graph API (e.g., "users", "groups")
/// This is used to specify which endpoint to query in the Microsoft Graph API.
resource_type: String,
/// The query parameters to be used in the Graph API request
/// This typically includes parameters like `$top`, `$filter`, etc., to control the data returned.
/// For example, "$top=999" to limit the results to 999 items
query_params: String,
/// Optional properties to expand in the Graph API response
/// This is a map where the key is the property to expand (e.g., "members", "owners") and the value is a vector of strings
/// The vector represents the specific fields to include in the expanded response in JSON pointer format.
expand_properties: HashMap<String, Vec<String>>,
}
impl GraphObject {
pub fn new<R, Q>(
resource_type: R,
query_params: Q,
expand_properties: Option<HashMap<String, Vec<String>>>,
) -> Self
where
R: Into<String>,
Q: Into<String>,
{
GraphObject {
resource_type: resource_type.into(),
query_params: query_params.into(),
expand_properties: expand_properties.unwrap_or_default(),
}
}
}
/// Enumerates Microsoft Graph data
/// This function retrieves various objects from the Microsoft Graph API
pub async fn enumerate_graph(collector: Arc<Collector>) -> Result<(), CirroError> {
info!("Starting Graph enumeration");
// Initialize our token cache with a valid token
{
let mut cache = TOKEN_CACHE.lock().await;
if cache.is_none() || cache.as_ref().unwrap().is_expired_or_not_set() {
debug!("Pre-fetching initial Graph API token");
*cache = Some(collector.msgraph_credential.get_token().await?);
info!("MS Graph token retrieved successfully");
}
}
// Create all enumerators with their configurations
let enumerators = vec![
GraphObject::new("users", "$top=999", None),
GraphObject::new(
"groups",
"$top=999",
Some(HashMap::from([
("members".into(), vec!["/id".into()]),
("owners".into(), vec!["/id".into()]),
])),
),
GraphObject::new(
"applications",
"$top=999",
Some(HashMap::from([("owners".into(), vec!["/id".into()])])),
),
GraphObject::new(
"servicePrincipals",
"$top=999",
Some(HashMap::from([("owners".into(), vec!["/id".into()])])),
),
GraphObject::new(
"devices",
"$top=999",
Some(HashMap::from([
("registeredOwners".into(), vec!["/id".into()]),
("registeredUsers".into(), vec!["/id".into()]),
])),
),
GraphObject::new(
"directoryRoles",
"",
Some(HashMap::from([("members".into(), vec!["/id".into()])])),
),
GraphObject::new(
"administrativeUnits",
"",
Some(HashMap::from([
("members".into(), vec!["/id".into()]),
(
"scopedRoleMembers".into(),
vec!["roleId".into(), "/roleMemberInfo/id".into()],
),
])),
),
];
// Start a timer
let start_time = Instant::now();
// Use a concurrent stream with controlled concurrency for better resource usage
let results = stream::iter(enumerators)
.map(|enumerator| {
let collector_clone = Arc::clone(&collector);
async move {
match query_objects(enumerator, collector_clone).await {
Ok(_) => Ok(()),
Err(e) => {
error!("Error collecting Graph data: {}", e);
Err(e)
}
}
}
})
.buffer_unordered(7) // Use the number of enumerators for concurrency
.collect::<Vec<_>>()
.await;
// Log the total time taken for the enumeration
info!(
"Graph enumeration completed in {} seconds",
fmt_duration(start_time.elapsed())
);
// Check for errors
let errors: Vec<_> = results.into_iter().filter_map(|r| r.err()).collect();
if !errors.is_empty() {
error!("Graph enumeration completed with {} errors", errors.len());
return Err(CirroError::MultipleErrors(errors));
}
Ok(())
}
/// Queries objects from the Microsoft Graph API
/// This function simulates querying objects from the Graph API and sending results to the database
async fn query_objects(
graph_object: GraphObject,
collector: Arc<Collector>,
) -> Result<(), CirroError> {
// Start timing the query
let start_time = std::time::Instant::now();
let resource_type = &graph_object.resource_type;
info!("Querying Graph API for resource type: {}", resource_type);
let mut next_uri = format!("{}?{}", resource_type, graph_object.query_params);
// Use the global HTTP client instead of creating a new one for each request
let mut all_values = Vec::new();
let mut total_fetched = 0;
// Batch process the pages
loop {
debug!("Fetching data from: {}", next_uri);
// Get the next page of data using our shared client
let response_data = paged_graph_request(&collector, &next_uri)
.await
.map_err(|e| {
error!("Failed to fetch data from Graph API: {} - {}", &next_uri, e);
e // Since paged_graph_request already returns CirroError, we can just pass it through
})?;
// Check for errors in the response
if let Some(error) = response_data.get("@odata.error") {
let error_message = error
.get("message")
.and_then(|m| m.get("value"))
.and_then(|v| v.as_str())
.unwrap_or("Unknown error");
error!("Graph API error: {}", error_message);
return Err(CirroError::ODataError(error_message.to_string()));
}
// Get the values - avoid unnecessary clone when possible
if let Some(values) = response_data.get("value").and_then(|v| v.as_array()) {
let page_count = values.len();
total_fetched += page_count;
debug!(
"Fetched {} objects for {}, total: {}",
page_count, resource_type, total_fetched
);
// Collect values for batch processing
all_values.extend(values.iter().cloned());
}
// Check for next link
if let Some(next_link) = response_data
.get("@odata.nextLink")
.and_then(|v| v.as_str())
{
next_uri = next_link.to_string();
} else {
break; // No more pages to fetch
}
}
info!(
"Fetched {} objects for {}. Processing...",
total_fetched, resource_type
);
// Process the values in parallel with controlled concurrency
let expand_properties = &graph_object.expand_properties;
let need_expand = !expand_properties.is_empty();
// If we need to expand properties, use a semaphore to limit concurrent requests
// let concurrency = if need_expand { 20 } else { 50 };
let concurrency = 100;
let semaphore = Arc::new(tokio::sync::Semaphore::new(concurrency));
// Create a stream for processing the values with controlled concurrency
let results = stream::iter(all_values)
.map(|value| {
let collector_clone = Arc::clone(&collector);
let resource_type = resource_type.clone();
let expand_properties = expand_properties.clone();
let sem = Arc::clone(&semaphore);
async move {
// Acquire permit from semaphore to control concurrency
let _permit = sem.acquire().await.map_err(|e| {
error!("Failed to acquire semaphore: {}", e);
CirroError::SemaphoreError(e.to_string())
})?;
// Make a mutable copy of the value for expansion
let mut value_clone = value.clone();
// Expand properties if specified
if need_expand {
// Expand each property
for (property, json_pointer) in &expand_properties {
expand_object(
&collector_clone,
&resource_type,
&mut value_clone,
property,
json_pointer,
)
.await;
}
}
// Get the ID from the object
let id = value_clone
.get("id")
.and_then(|v| v.as_str())
.unwrap_or("unknown_id")
.to_string();
// Prepare and send the DB write message - this happens regardless of whether
// properties were expanded, but we have better logging if they were
let _ = &collector_clone
.write_value_to_db(resource_type.clone(), id.clone(), value_clone)
.await;
Ok(())
}
})
.buffer_unordered(concurrency)
.collect::<Vec<Result<(), CirroError>>>()
.await;
// Check for errors in the results
let error_count = results.iter().filter(|r| r.is_err()).count();
if error_count > 0 {
error!(
"{} errors occurred while processing {} objects",
error_count, resource_type
);
}
let elapsed_time = start_time.elapsed();
info!(
"Completed Graph API query for resource type: {} ({}) - {} objects processed",
resource_type,
fmt_duration(elapsed_time),
total_fetched
);
Ok(())
}
/// Expand the properties of a Graph object
/// This function expands the properties of a Graph object based on the provided expand properties
/// The object is modified in place, but no database write occurs in this function.
pub async fn expand_object(
collector: &Collector,
resource_type: &str,
object: &mut serde_json::Value,
property: &str,
json_pointer: &Vec<String>,
) {
// Early return if we don't have a valid ID to work with
// This should never happen, but it's a good safety check
let object_id = {
let id = match object.get("id").and_then(|v| v.as_str()) {
Some(id) => id,
None => {
error!(
"Cannot expand object without ID for {}/{}",
resource_type, property
);
return;
}
};
id.to_string()
};
// Create the initial URL
let mut next_url = format!(
"{}/beta/{}/{}/{}",
collector.cloud_endpoints.msgraph_url, resource_type, object_id, property
);
debug!(
"Expanding property {} for {}/{}",
property, resource_type, object_id
);
// Prepare to collect all expanded values
let mut all_expanded_values = Vec::new();
// Use the global HTTP client for better connection reuse
loop {
// Get the response data, with error handling
let response_data = match paged_graph_request(collector, &next_url).await {
Ok(data) => data,
Err(e) => {
error!(
"Failed to fetch expanded data for {}/{}/{}: {}",
resource_type, object_id, property, e
);
break;
}
};
// Process the values if available
if let Some(values) = response_data.get("value").and_then(|v| v.as_array()) {
// Only extract values if there are any
if !values.is_empty() {
// Extract only the specific fields based on json_pointer
for value in values {
for pointer in json_pointer {
// Use the pointer to get the value (typically "/id" for IDs)
if let Some(expanded_value) = value.pointer(pointer) {
debug!(
"Expanded value for {}/{}/{}: {}",
resource_type, object_id, property, expanded_value
);
// Clone the value to avoid borrowing issues
all_expanded_values.push(expanded_value.clone());
}
}
}
}
}
// Check for next link - using get() instead of direct indexing to avoid panics
if let Some(next_link) = response_data
.get("@odata.nextLink")
.and_then(|v| v.as_str())
{
next_url = next_link.to_string();
} else {
break; // No more pages to fetch
}
}
// Only update the object if we have any values to add
if !all_expanded_values.is_empty() {
// Update the object with all expanded values at once
if let Some(obj_mut) = object.as_object_mut() {
// Create or get the array for this property
let entry = obj_mut
.entry(property)
.or_insert_with(|| serde_json::Value::Array(Vec::new()));
// Add all the expanded values to the array
if let Some(array) = entry.as_array_mut() {
array.extend(all_expanded_values);
debug!(
"Added {} expanded values for {}/{}/{}",
array.len(),
resource_type,
object_id,
property
);
}
}
}
}
/// Performs a paged request to the Graph API with optimized token caching
pub async fn paged_graph_request(
collector: &Collector,
uri: &str,
) -> Result<serde_json::Value, CirroError> {
// Build the full URL correctly
let graph_url = if uri.starts_with(&collector.cloud_endpoints.msgraph_url) {
uri.to_owned()
} else {
format!(
"{}/beta/{}",
collector.cloud_endpoints.msgraph_url,
uri.trim_start_matches('/')
)
};
// Get the access token, using cache when possible
let token_str = {
let mut cache = TOKEN_CACHE.lock().await;
// Check if we need a new token based on the token's actual expiration
let need_new_token = match &*cache {
Some(cached_token) => cached_token.is_expired_or_not_set(),
None => true,
};
if need_new_token {
debug!("Fetching new Graph API token");
let token = collector.msgraph_credential.get_token().await?;
// Log expiration information for debugging
if let Some(expires) = token.expires_on {
let now = chrono::Utc::now();
let time_until_expiry = expires.signed_duration_since(now);
debug!(
"New token expires at {} (valid for {} minutes)",
expires,
time_until_expiry.num_minutes()
);
} else {
debug!("New token has no expiration time set");
}
*cache = Some(token.clone());
Arc::clone(&token.access_token)
} else {
// Return a clone of the cached token
Arc::clone(&cache.as_ref().unwrap().access_token)
}
};
// Make the request using the global client
let response = HTTP_CLIENT
.get(&graph_url)
.bearer_auth(&*token_str)
.timeout(std::time::Duration::from_secs(30))
.send()
.await
.map_err(|e| {
if e.is_timeout() {
CirroError::IoError(std::io::Error::new(
std::io::ErrorKind::TimedOut,
format!("Request timed out: {}", e),
))
} else {
CirroError::RequestError(e)
}
})?;
let status = response.status();
// Check for non-200 status codes early to avoid parsing JSON for error responses
if status != StatusCode::OK {
let error_text = response.text().await?;
return Err(CirroError::HttpError(format!(
"HTTP {} - {}",
status.as_u16(),
error_text
)));
}
// Parse the response data
let response_data: Value = response.json::<Value>().await?;
// Check for OData error
if let Some(err) = response_data.get("@odata.error") {
let msg = err
.get("message")
.and_then(|m| m.get("value"))
.and_then(|v| v.as_str())
.unwrap_or("unknown odata.error");
return Err(CirroError::ODataError(msg.to_string()));
}
Ok(response_data)
}
cirro/src/collectors/mod.rs
+4
View File
@@ -0,0 +1,4 @@
pub mod arm;
pub mod common;
pub mod enrichments;
pub mod graph;
src/collect/azure/context.rs → cirro/src/context.rs
+23 -38
View File
@@ -1,7 +1,6 @@
use crate::collect::azure::cli::{AzureCloud, EnumerationMode, OptionEnumFlags};
use crate::collect::azure::db::{ArmResourceMessage, DBWriteMessage, DataMessage};
use crate::db::DBWriteMessage;
use crate::errors::CirroError;
use std::collections::HashSet;
use crate::{AzureCloud, EnrichmentFlags, EnumerationMode};
use std::path::PathBuf;
use std::sync::Arc;
use tokio::sync::mpsc;
@@ -9,22 +8,20 @@ use tokio::sync::mpsc;
#[derive(Debug, Clone)]
pub struct CollectorContext<C> {
pub mode: EnumerationMode,
pub option_enum_flags: OptionEnumFlags,
pub subscription_concurrency: usize,
pub cloud: AzureCloud,
pub cloud_endpoints: CloudEndpoints,
pub msgraph_credential: C,
pub arm_credential: C,
pub vault_credential: C,
pub storage_credential: C,
pub output_path: PathBuf,
pub tenant_id: Option<String>,
pub subscription_id: Option<String>,
pub client_id: Option<String>,
pub client_secret: Option<String>,
pub client_cert_path: Option<PathBuf>,
pub subscription_filter: Option<HashSet<String>>,
pub db_writer: Option<Arc<mpsc::Sender<DBWriteMessage>>>,
pub db_writer: Option<Arc<mpsc::UnboundedSender<DBWriteMessage>>>,
pub enrich_mode: bool,
pub enrich_flags: Option<EnrichmentFlags>,
}
impl<C> CollectorContext<C> {
@@ -36,10 +33,9 @@ impl<C> CollectorContext<C> {
data: serde_json::Value,
) -> Result<(), CirroError> {
if let Some(sender) = &self.db_writer {
let message = DBWriteMessage::Data(DataMessage { table, id, data });
let message = DBWriteMessage::Data { table, id, data };
sender
.send(message)
.await
.map_err(|e| CirroError::DatabaseError(e.to_string()))?;
Ok(())
} else {
@@ -47,26 +43,21 @@ impl<C> CollectorContext<C> {
}
}
/// Writes a batch of values to one table in the database
pub async fn write_values_batch_to_db(
/// Writes an enrichment to the database
pub async fn write_enrichment_to_db(
&self,
table: String,
rows: Vec<(String, serde_json::Value)>,
module: String,
resource_id: String,
data: serde_json::Value,
) -> Result<(), CirroError> {
if let Some(sender) = &self.db_writer {
let batch_rows = rows
.into_iter()
.map(|(id, data)| DataMessage {
table: table.clone(),
id,
data,
})
.collect();
let message = DBWriteMessage::DataBatch(batch_rows);
let message = DBWriteMessage::Enrichment {
module,
resource_id,
data,
};
sender
.send(message)
.await
.map_err(|e| CirroError::DatabaseError(e.to_string()))?;
Ok(())
} else {
@@ -85,16 +76,15 @@ impl<C> CollectorContext<C> {
data: serde_json::Value,
) -> Result<(), CirroError> {
if let Some(sender) = &self.db_writer {
let message = DBWriteMessage::ArmResource(ArmResourceMessage {
let message = DBWriteMessage::ArmResource {
id,
sub_id,
rg_id,
resource_type,
data,
});
};
sender
.send(message)
.await
.map_err(|e| CirroError::DatabaseError(e.to_string()))?;
Ok(())
} else {
@@ -105,11 +95,10 @@ impl<C> CollectorContext<C> {
#[derive(Debug, Clone, Copy)]
pub struct CloudEndpoints {
pub token_endpoint: &'static str,
pub msgraph_url: &'static str,
pub arm_url: &'static str,
pub vault_url: &'static str,
pub storage_url: &'static str,
pub token_endpoint: &'static str,
}
impl CloudEndpoints {
@@ -117,32 +106,28 @@ impl CloudEndpoints {
pub fn new(cloud: AzureCloud) -> Self {
match cloud {
AzureCloud::Public => CloudEndpoints {
token_endpoint: "https://login.microsoftonline.com",
msgraph_url: "https://graph.microsoft.com",
arm_url: "https://management.azure.com",
vault_url: "https://vault.azure.net",
storage_url: "https://storage.azure.com",
token_endpoint: "https://login.microsoftonline.com",
},
AzureCloud::China => CloudEndpoints {
token_endpoint: "https://login.chinacloudapi.cn",
msgraph_url: "https://microsoftgraph.chinacloudapi.cn",
arm_url: "https://management.chinacloudapi.cn",
vault_url: "https://vault.azure.cn",
storage_url: "https://storage.azure.cn",
token_endpoint: "https://login.chinacloudapi.cn",
},
AzureCloud::Germany => CloudEndpoints {
token_endpoint: "https://login.microsoftazure.de",
msgraph_url: "https://microsoftgraph.microsoft.de",
arm_url: "https://management.microsoftazure.de",
vault_url: "https://vault.microsoftazure.de",
storage_url: "https://storage.microsoftazure.de",
token_endpoint: "https://login.microsoftazure.de",
},
AzureCloud::USGov => CloudEndpoints {
token_endpoint: "https://login.microsoftonline.us",
msgraph_url: "https://graph.microsoft.us",
arm_url: "https://management.usgovcloudapi.net",
vault_url: "https://vault.azure.us",
storage_url: "https://storage.azure.us",
token_endpoint: "https://login.microsoftonline.us",
},
}
}
src/collect/azure/credentials/access_token.rs → cirro/src/credentials/access_token.rs
+4 -5
View File
@@ -1,5 +1,4 @@
use crate::collect::azure::credentials::common::{AuthCredential, Token};
use crate::errors::CirroError;
use crate::credentials::common::{AuthCredential, AuthError, Token};
use async_trait::async_trait;
use tokio::sync::RwLock;
@@ -8,10 +7,10 @@ pub struct AccessTokenCredential {
pub token: RwLock<Token>,
}
/// Implementation of the AuthCredential trait for AccessTokenCredential
/// Implementation of the AuthCredential trait for AzureCliCredential
#[async_trait]
impl AuthCredential for AccessTokenCredential {
async fn get_token(&self) -> Result<Token, CirroError> {
async fn get_token(&self) -> Result<Token, AuthError> {
let guard = self.token.read().await;
if !guard.is_expired_or_not_set() {
return Ok(guard.clone());
@@ -19,7 +18,7 @@ impl AuthCredential for AccessTokenCredential {
// We can't fetch a new token here, so we return an error
drop(guard);
return Err(CirroError::AuthenticationFailed(
return Err(AuthError::AuthenticationFailed(
"Access token credential has expired".into(),
));
}
src/collect/azure/credentials/azcli.rs → cirro/src/credentials/azcli.rs
+9 -10
View File
@@ -1,5 +1,4 @@
use crate::collect::azure::credentials::common::{AuthCredential, Token};
use crate::errors::CirroError;
use crate::credentials::common::{AuthCredential, AuthError, Token};
use async_trait::async_trait;
use log::{debug, info};
use std::sync::Arc;
@@ -17,7 +16,7 @@ pub struct AzureCliCredential {
/// Implementation of the AuthCredential trait for AzureCliCredential
#[async_trait]
impl AuthCredential for AzureCliCredential {
async fn get_token(&self) -> Result<Token, CirroError> {
async fn get_token(&self) -> Result<Token, AuthError> {
let guard = self.token.read().await;
if !guard.is_expired_or_not_set() {
return Ok(guard.clone());
@@ -42,10 +41,10 @@ impl AuthCredential for AzureCliCredential {
.envs(std::env::vars())
.output()
.await
.map_err(|e| CirroError::AuthError(e.to_string()))?;
.map_err(|e| AuthError::ProcessError(e))?;
if !output.status.success() {
return Err(CirroError::AuthError(format!(
return Err(AuthError::AuthenticationFailed(format!(
"az failed: {}",
String::from_utf8_lossy(&output.stderr)
)));
@@ -55,14 +54,14 @@ impl AuthCredential for AzureCliCredential {
String::from_utf8_lossy(&output.stdout)
);
let resp: serde_json::Value = serde_json::from_slice(&output.stdout)
.map_err(|e| CirroError::AuthError(e.to_string()))?;
let resp: serde_json::Value =
serde_json::from_slice(&output.stdout).map_err(|e| AuthError::JsonError(e))?;
let mut new_token = Token {
access_token: Arc::new(
resp["accessToken"]
.as_str()
.ok_or_else(|| CirroError::AuthError("missing accessToken".into()))?
.ok_or_else(|| AuthError::ParseError("missing accessToken".into()))?
.to_string(),
),
expires_on: None,
@@ -70,8 +69,8 @@ impl AuthCredential for AzureCliCredential {
};
new_token
.set_expires_on_from_token()
.map_err(|e| CirroError::AuthError(format!("Failed to set expires_on: {}", e)))?;
debug!("New token fetched successfully for Azure CLI credential");
.map_err(|e| AuthError::ParseError(format!("Failed to set expires_on: {}", e)))?;
debug!("New token fetched successfully for client secret credential");
let mut write_guard = self.token.write().await;
*write_guard = new_token.clone();
src/collect/azure/credentials/clientcert.rs → cirro/src/credentials/clientcert.rs
+19 -19
View File
@@ -1,5 +1,4 @@
use crate::collect::azure::credentials::common::{AuthCredential, Token};
use crate::errors::CirroError;
use crate::credentials::common::{AuthCredential, AuthError, Token};
use async_trait::async_trait;
use base64::Engine;
use jsonwebtoken::{Algorithm, EncodingKey, Header, encode};
@@ -40,17 +39,17 @@ struct ClientAssertionClaims {
impl ClientCertificateCredential {
/// Generates a client assertion JWT for certificate-based authentication
pub fn generate_client_assertion(&self) -> Result<String, CirroError> {
pub fn generate_client_assertion(&self) -> Result<String, AuthError> {
// Read the certificate file
let cert_data = fs::read_to_string(&self.certificate_path)
.map_err(|e| CirroError::AuthError(e.to_string()))?;
let cert_data =
fs::read_to_string(&self.certificate_path).map_err(|e| AuthError::ProcessError(e))?;
// Parse the PEM data to get multiple blocks (certificate and private key)
let pem_blocks = parse_many(&cert_data)
.map_err(|e| CirroError::AuthError(format!("Failed to parse PEM blocks: {}", e)))?;
.map_err(|e| AuthError::ParseError(format!("Failed to parse PEM blocks: {}", e)))?;
if pem_blocks.is_empty() {
return Err(CirroError::AuthError(
return Err(AuthError::ParseError(
"No PEM blocks found in certificate file".into(),
));
}
@@ -74,7 +73,7 @@ impl ClientCertificateCredential {
// Get the private key or return an error
let private_key = private_key.ok_or_else(|| {
CirroError::AuthError("No private key found in certificate file".into())
AuthError::ParseError("No private key found in certificate file".into())
})?;
// Create JWT header with RS256 algorithm
@@ -97,7 +96,7 @@ impl ClientCertificateCredential {
// Calculate timestamps
let now = SystemTime::now()
.duration_since(UNIX_EPOCH)
.map_err(|e| CirroError::AuthError(e.to_string()))?
.map_err(|e| AuthError::Unknown(e.to_string()))?
.as_secs();
// Construct the audience (token endpoint with tenant)
@@ -115,11 +114,12 @@ impl ClientCertificateCredential {
};
// Create the encoding key from the PEM private key
let encoding_key = EncodingKey::from_rsa_pem(private_key.to_string().as_bytes())
.map_err(|e| CirroError::AuthError(format!("Invalid RSA key: {}", e)))?;
.map_err(|e| AuthError::ParseError(format!("Invalid RSA key: {}", e)))?;
// Encode the JWT
let token = encode(&header, &claims, &encoding_key)
.map_err(|e| CirroError::AuthError(format!("JWT encoding error: {}", e)))?;
.map_err(|e| AuthError::Unknown(format!("JWT encoding error: {}", e)))?;
Ok(token)
}
}
@@ -127,7 +127,7 @@ impl ClientCertificateCredential {
/// Implementation of the AuthCredential trait for ClientCertificateCredential
#[async_trait]
impl AuthCredential for ClientCertificateCredential {
async fn get_token(&self) -> Result<Token, CirroError> {
async fn get_token(&self) -> Result<Token, AuthError> {
let guard = self.token.read().await;
if !guard.is_expired_or_not_set() {
return Ok(guard.clone());
@@ -157,7 +157,7 @@ impl AuthCredential for ClientCertificateCredential {
])
.send()
.await
.map_err(|e| CirroError::AuthError(e.to_string()))?;
.map_err(|e| AuthError::NetworkError(e.to_string()))?;
// If the response is not successful, log the response body for debugging
if !resp.status().is_success() {
@@ -165,8 +165,8 @@ impl AuthCredential for ClientCertificateCredential {
let body = resp
.text()
.await
.map_err(|e| CirroError::AuthError(e.to_string()))?;
return Err(CirroError::AuthError(format!(
.map_err(|e| AuthError::NetworkError(e.to_string()))?;
return Err(AuthError::AuthenticationFailed(format!(
"Failed to fetch token: {}",
body
)));
@@ -175,7 +175,7 @@ impl AuthCredential for ClientCertificateCredential {
let resp = resp
.json::<serde_json::Value>()
.await
.map_err(|e| CirroError::AuthError(format!("JSON parsing error: {}", e)))?;
.map_err(|e| AuthError::ParseError(format!("JSON parsing error: {}", e)))?;
// Check for errors in the response
if let Some(error) = resp.get("error") {
@@ -184,7 +184,7 @@ impl AuthCredential for ClientCertificateCredential {
.and_then(|d| d.as_str())
.unwrap_or("Unknown error");
return Err(CirroError::AuthError(format!(
return Err(AuthError::AuthenticationFailed(format!(
"{}: {}",
error.as_str().unwrap_or("error"),
error_desc
@@ -195,7 +195,7 @@ impl AuthCredential for ClientCertificateCredential {
access_token: Arc::new(
resp["access_token"]
.as_str()
.ok_or_else(|| CirroError::AuthError("missing access_token".into()))?
.ok_or_else(|| AuthError::ParseError("missing access_token".into()))?
.to_string(),
),
expires_on: None,
@@ -203,7 +203,7 @@ impl AuthCredential for ClientCertificateCredential {
};
new_token
.set_expires_on_from_token()
.map_err(|e| CirroError::AuthError(format!("Failed to set expires_on: {}", e)))?;
.map_err(|e| AuthError::ParseError(format!("Failed to set expires_on: {}", e)))?;
debug!("New token fetched successfully for client certificate credential");
let mut write_guard = self.token.write().await;
src/collect/azure/credentials/clientsecret.rs → cirro/src/credentials/clientsecret.rs
+5 -6
View File
@@ -1,5 +1,4 @@
use crate::collect::azure::credentials::common::{AuthCredential, Token};
use crate::errors::CirroError;
use crate::credentials::common::{AuthCredential, AuthError, Token};
use async_trait::async_trait;
use chrono::TimeZone;
use log::debug;
@@ -24,7 +23,7 @@ pub struct ClientSecretCredential {
/// Implementation of the AuthCredential trait for ClientSecretCredential
#[async_trait]
impl AuthCredential for ClientSecretCredential {
async fn get_token(&self) -> Result<Token, CirroError> {
async fn get_token(&self) -> Result<Token, AuthError> {
let guard = self.token.read().await;
if !guard.is_expired_or_not_set() {
return Ok(guard.clone());
@@ -44,16 +43,16 @@ impl AuthCredential for ClientSecretCredential {
])
.send()
.await
.map_err(|e| CirroError::NetworkError(e.to_string()))?
.map_err(|e| AuthError::NetworkError(e.to_string()))?
.json::<serde_json::Value>()
.await
.map_err(|e| CirroError::Unknown(format!("JSON parsing error: {}", e)))?;
.map_err(|e| AuthError::Unknown(format!("JSON parsing error: {}", e)))?;
let new_token = Token {
access_token: Arc::new(
resp["access_token"]
.as_str()
.ok_or_else(|| CirroError::ParseError("missing access_token".into()))?
.ok_or_else(|| AuthError::ParseError("missing access_token".into()))?
.to_string(),
),
expires_on: resp["expires_on"]
src/collect/azure/credentials/common.rs → cirro/src/credentials/common.rs
+27 -8
View File
@@ -1,9 +1,27 @@
use crate::errors::CirroError;
use base64::Engine;
use chrono::{DateTime, Utc};
use serde::{Deserialize, Serialize};
use std::default::Default;
use std::sync::Arc;
use thiserror::Error;
#[derive(Debug, Error)]
pub enum AuthError {
#[error("Authentication failed: {0}")]
AuthenticationFailed(String),
#[error("Token expired")]
TokenExpired,
#[error("Network error: {0}")]
NetworkError(String),
#[error("Parse error: {0}")]
ParseError(String),
#[error("Process error: {0}")]
ProcessError(#[from] std::io::Error),
#[error("JSON error: {0}")]
JsonError(#[from] serde_json::Error),
#[error("Unknown error: {0}")]
Unknown(String),
}
// Using async_trait for better performance with static dispatch when possible
use async_trait::async_trait;
@@ -11,7 +29,7 @@ use async_trait::async_trait;
#[async_trait]
pub trait AuthCredential {
/// Returns the access token for the credential
async fn get_token(&self) -> Result<Token, CirroError>;
async fn get_token(&self) -> Result<Token, AuthError>;
}
#[derive(Debug, Default, Clone)]
@@ -65,25 +83,26 @@ impl Token {
/// Gets claims from the JWT token
/// Returns an error if the token is not a valid JWT or if claims cannot be parsed
pub fn get_claims(&self) -> Result<Claims, CirroError> {
pub fn get_claims(&self) -> Result<Claims, AuthError> {
let token_parts: Vec<&str> = self.access_token.split('.').collect();
if token_parts.len() != 3 {
return Err(CirroError::AuthError("Invalid JWT token format".into()));
return Err(AuthError::ParseError("Invalid JWT token format".into()));
}
let claims_json = base64::engine::general_purpose::URL_SAFE_NO_PAD
let claims_json = base64::engine::general_purpose::STANDARD_NO_PAD
.decode(token_parts[1])
.map_err(|e| CirroError::AuthError(format!("Failed to decode token: {}", e)))?;
.map_err(|e| AuthError::ParseError(format!("Failed to decode token: {}", e)))?;
let claims: Claims = serde_json::from_slice(&claims_json)?;
Ok(claims)
}
/// Sets the expiration time from the token claims
/// Returns an error if the claims cannot be parsed or if the expiration time is invalid
pub fn set_expires_on_from_token(&mut self) -> Result<(), CirroError> {
pub fn set_expires_on_from_token(&mut self) -> Result<(), AuthError> {
let claims = self.get_claims()?;
let expiration_time = DateTime::<Utc>::from_timestamp(claims.exp as i64, 0)
.ok_or_else(|| CirroError::AuthError("Invalid expiration time in token".into()))?;
.ok_or_else(|| AuthError::ParseError("Invalid expiration time in token".into()))?;
self.expires_on = Some(expiration_time);
Ok(())
}
src/collect/azure/credentials/mod.rs → cirro/src/credentials/mod.rs
View File
File renamed without changes.
cirro/src/db.rs
+251
View File
@@ -0,0 +1,251 @@
use log::{debug, error};
use rusqlite::Connection;
use std::path::PathBuf;
use tokio::sync::mpsc;
pub enum DBWriteMessage {
Data {
table: String,
id: String,
data: serde_json::Value,
},
ArmResource {
id: String,
sub_id: String,
rg_id: String,
resource_type: String,
data: serde_json::Value,
},
Enrichment {
module: String,
resource_id: String,
data: serde_json::Value,
},
/// Shutdown message to close the database connection
Shutdown,
}
#[derive(Debug, Clone)]
pub struct SqliteDb {
pub db_path: PathBuf,
}
impl SqliteDb {
pub fn new(db_path: PathBuf) -> Self {
SqliteDb { db_path }
}
/// Gets a connection to the SQLite database
pub async fn get_connection(&self) -> Result<Connection, rusqlite::Error> {
Connection::open(&self.db_path)
}
/// Runs the database writer loop
/// This will initialize the database connection when the first message arrives
/// and will handle all incoming messages until a shutdown message is received.
pub async fn run_writer(&self, mut receiver: mpsc::UnboundedReceiver<DBWriteMessage>) {
// Define all possible table names
let table_names = vec![
"applications",
"administrativeUnits",
"devices",
"directoryRoles",
"groups",
"roleAssignments",
"servicePrincipals",
"subscriptions",
"tenants",
"users",
];
// Connection will be initialized when the first message arrives
let mut conn_option: Option<Connection> = None;
// Process incoming messages
while let Some(message) = receiver.recv().await {
match message {
DBWriteMessage::Data { table, id, data } => {
// Initialize database if this is the first message
if conn_option.is_none() {
debug!(
"Received first message, initializing database: {:?}",
self.db_path
);
// Open the database connection
match Connection::open(&self.db_path) {
Ok(new_conn) => {
// Set database PRAGMAs for performance
if let Err(e) = new_conn.execute_batch(
"PRAGMA journal_mode=WAL; PRAGMA synchronous=NORMAL; PRAGMA automatic_index = true;",
) {
error!("Failed to set database PRAGMAs: {}", e);
// Continue anyway, as these are just performance optimizations
}
// Create standard tables if they do not exist
for table_name in &table_names {
let create_table_sql = format!(
"CREATE TABLE IF NOT EXISTS {} (id TEXT PRIMARY KEY, data BLOB)",
table_name
);
if let Err(e) = new_conn.execute(&create_table_sql, []) {
error!("Failed to create table {}: {}", table_name, e);
// Continue with other tables
}
}
// Create the resource table with its special schema
if let Err(e) = new_conn.execute(
"CREATE TABLE IF NOT EXISTS resources (id TEXT PRIMARY KEY, sub_id TEXT, rg_id TEXT, resource_type TEXT, data BLOB)",
[],
) {
error!("Failed to create resources table: {}", e);
// Continue anyway
}
// Create the enrichment table
if let Err(e) = new_conn.execute(
"CREATE TABLE IF NOT EXISTS enrichments (module TEXT, resource_id TEXT, data BLOB, PRIMARY KEY (module, resource_id))",
[],
) {
error!("Failed to create enrichments table: {}", e);
// Continue anyway
}
conn_option = Some(new_conn);
debug!("Database initialized successfully");
}
Err(e) => {
error!("Failed to open database: {}", e);
// Skip this message since we couldn't open the database
continue;
}
};
}
// At this point we should have a valid connection
let conn = match &conn_option {
Some(c) => c,
None => {
error!("Database connection unavailable, skipping message");
continue;
}
};
// Insert or replace the data
let insert_sql =
format!("INSERT OR REPLACE INTO {} (id, data) VALUES (?, ?)", table);
match conn.execute(&insert_sql, &[&id.to_lowercase(), &data.to_string()]) {
Ok(_) => {}
Err(e) => {
error!(
"Failed to write to table: {}, id: {}. Error: {}",
table, id, e
);
continue;
}
}
}
DBWriteMessage::ArmResource {
id,
sub_id,
rg_id,
resource_type,
data,
} => {
let insert_sql = "REPLACE INTO resources (id, sub_id, rg_id, resource_type, data) VALUES (?, ?, ?, ?, ?)";
let conn = match &conn_option {
Some(c) => c,
None => {
error!("Database connection unavailable, skipping message");
continue;
}
};
match conn.execute(
insert_sql,
&[
&id.to_lowercase(),
&sub_id.to_lowercase(),
&rg_id.to_lowercase(),
&resource_type.to_lowercase(),
&data.to_string(),
],
) {
Ok(_) => {}
Err(e) => {
error!(
"Failed to write ARM resource: {}, sub_id: {}, rg_id: {}, resource_type: {}. Error: {}",
id, sub_id, rg_id, resource_type, e
);
continue;
}
}
}
DBWriteMessage::Enrichment {
module,
resource_id,
data,
} => {
let insert_sql =
"REPLACE INTO enrichments (module, resource_id, data) VALUES (?, ?, ?)";
let conn = match &conn_option {
Some(c) => c,
None => {
// Set up new connection
conn_option = Some(Connection::open(&self.db_path).unwrap());
match conn_option.as_ref() {
Some(c) => c,
None => {
error!("Failed to open database connection for enrichment");
continue;
}
}
}
};
match conn.execute(insert_sql, &[&module, &resource_id, &data.to_string()]) {
Ok(_) => {}
Err(e) => {
error!(
"Failed to write enrichment: module: {}, resource_id: {}. Error: {}",
module, resource_id, e
);
continue;
}
}
}
DBWriteMessage::Shutdown => {
debug!("Received shutdown message, closing database connection");
// Finalize any pending work and close the connection properly
if let Some(conn) = &conn_option {
if let Err(e) = conn.execute("PRAGMA optimize", []) {
debug!("Failed to run PRAGMA optimize: {}", e);
}
debug!("Database shutdown complete");
} else {
debug!("No active database connection to shut down");
}
// Break out of the loop to terminate the writer
break;
}
}
}
}
/// Runs a read query against the database
pub fn run_query(&self, query: &str) -> Result<Vec<serde_json::Value>, rusqlite::Error> {
let conn = Connection::open(&self.db_path)?;
let mut stmt = conn.prepare(query)?;
let rows = stmt.query_map([], |row| {
let data: String = row.get(0)?;
Ok(serde_json::from_str(&data).unwrap_or_default())
})?;
let mut results = Vec::new();
for row in rows {
results.push(row?);
}
Ok(results)
}
}
cirro/src/errors.rs
+51
View File
@@ -0,0 +1,51 @@
use crate::credentials::common::AuthError;
use thiserror::Error;
/// Custom error type for Graph API related errors
#[derive(Error, Debug)]
pub enum CirroError {
#[error("HTTP error: {0}")]
HttpError(String),
#[error("OData error: {0}")]
ODataError(String),
#[error("Database error: {0}")]
DatabaseError(String),
#[error("Auth error: {0}")]
AuthError(#[from] AuthError),
#[error("Request error: {0}")]
RequestError(#[from] reqwest::Error),
#[error("JSON error: {0}")]
JsonError(#[from] serde_json::Error),
#[error("IO error: {0}")]
IoError(#[from] std::io::Error),
#[error("Semaphore error: {0}")]
SemaphoreError(String),
#[error("Log setup error: {0}")]
LogSetupError(#[from] log::SetLoggerError),
#[error("Multiple errors: {0:?}")]
MultipleErrors(Vec<CirroError>),
#[error("ARM API error: {0}")]
ArmApiError(String),
#[error("Serialization error: {0}")]
SerializationError(String),
#[error("Unknown error: {0}")]
Unknown(String),
#[error("Sqlite error: {0}")]
SqliteError(#[from] rusqlite::Error),
#[error("Unsupported HTTP method: {0}")]
UnsupportedHttpMethod(reqwest::Method),
}
src/collect/logger.rs → cirro/src/logger.rs
View File
File renamed without changes.
cirro/src/main.rs
+410
View File
@@ -0,0 +1,410 @@
pub mod collect;
pub mod collectors;
pub mod context;
pub mod credentials;
pub mod db;
pub mod errors;
pub mod logger;
pub mod styles;
use crate::errors::CirroError;
use crate::logger::setup_logger;
use clap::{
Args, ColorChoice, Parser, Subcommand, crate_authors, crate_description, crate_version,
};
use colored::Colorize;
use std::path::PathBuf;
use tokio;
#[derive(clap::ValueEnum, Copy, Clone, Debug)]
pub enum AzureCloud {
Public,
China,
Germany,
USGov,
}
#[derive(clap::ValueEnum, Clone, Debug, PartialEq, Eq)]
pub enum EnumerationMode {
Both,
Graph,
Arm,
}
#[derive(Debug, Parser)]
#[clap(
about = crate_description!(),
version = crate_version!(),
bin_name = "cirro",
author = crate_authors!(),
styles = styles::get_styles(),
color = ColorChoice::Always,
arg_required_else_help = true,
)]
#[command(disable_help_subcommand = true)]
struct Cli {
#[clap(subcommand)]
command: Option<Commands>,
}
#[derive(Debug, Subcommand)]
enum Commands {
/// Collect data from Azure and Microsoft Graph APIs
Collect {
#[command(subcommand)]
auth_mode: AuthenticationMode,
},
/// Enrich data in the database
Enrich {
#[command(subcommand)]
auth_mode: AuthenticationMode,
#[clap(flatten)]
flags: EnrichmentFlags,
},
}
#[derive(Debug, Clone, Args)]
struct CommonAuthArgs {
/// Output database file path
#[arg(short, long, value_name = "FILE", global=true, default_value = "cirro_output.db", value_hint = clap::ValueHint::FilePath)]
output_path: PathBuf,
/// Enumeration mode
#[arg(
short,
long,
value_enum,
ignore_case = true,
default_value = "both",
value_name = "MODE"
)]
mode: EnumerationMode,
/// Cloud to enumerate
#[arg(long, value_enum, ignore_case = true, default_value = "public")]
cloud: AzureCloud,
/// Debug output
#[arg(long = "debug", action = clap::ArgAction::SetTrue)]
debug: bool,
}
#[derive(Debug, Clone, Args)]
struct AccessTokenAuthArgs {
/// Output database file path
#[arg(short, long, value_name = "FILE", global=true, default_value = "cirro_output.db", value_hint = clap::ValueHint::FilePath)]
output_path: PathBuf,
/// Cloud to enumerate
#[arg(long, value_enum, ignore_case = true, default_value = "public")]
cloud: AzureCloud,
/// Debug output
#[arg(long = "debug", action = clap::ArgAction::SetTrue)]
debug: bool,
}
#[derive(Debug, Subcommand)]
enum AuthenticationMode {
/// Authenticate using an access token
AccessToken {
/// Access token
#[arg(short, long)]
token: String,
#[clap(flatten)]
common: AccessTokenAuthArgs,
},
/// Authenticate using Azure CLI
Azcli {
/// Tenant ID
#[arg(short, long)]
tenant_id: Option<String>,
/// Subscription ID
#[arg(short, long)]
subscription_id: Option<String>,
#[clap(flatten)]
common: CommonAuthArgs,
},
/// Authenticate using a client secret
ClientSecret {
/// Client ID
#[arg(short, long)]
client_id: String,
/// Client secret
#[arg(short = 'p', long)]
client_secret: String,
/// Tenant ID
#[arg(short, long)]
tenant_id: String,
#[clap(flatten)]
common: CommonAuthArgs,
},
/// Authenticate using a client certificate
ClientCert {
/// Client ID
#[arg(short, long)]
client_id: String,
/// Path to the client certificate file (PEM format)
#[arg(short = 'p', long = "certificate", value_name = "FILE", value_hint = clap::ValueHint::FilePath)]
client_certificate: PathBuf,
/// Tenant ID
#[arg(short, long)]
tenant_id: String,
#[clap(flatten)]
common: CommonAuthArgs,
},
/// Authenticate using a username and password
UserPass {
/// The username of the account (UPN format)
#[arg(short, long)]
upn: String,
/// The password of the account
#[arg(short, long)]
password: String,
#[clap(flatten)]
common: CommonAuthArgs,
},
}
#[derive(Debug, Clone, Args)]
pub struct EnrichmentFlags {
/// Gather storage account keys
#[arg(long, action = clap::ArgAction::SetTrue)]
storage_keys: bool,
// /// Gather storage account containers and blobs with authentication
// #[arg(long, action = clap::ArgAction::SetTrue)]
// storage_blobs: bool,
// /// Gather storage account containers and blobs without authentication
// #[arg(long, action = clap::ArgAction::SetTrue)]
// storage_blobs_anon: bool,
}
#[tokio::main]
async fn main() -> Result<(), CirroError> {
let cli = Cli::parse();
let logo = r#"
___ _
/ __| (_) _ _ _ _ ___
| (__ | | | '_| | '_| / _ \
\___| _|_|_ _|_|_ _|_|_ \___/
_|"""""|_|"""""|_|"""""|_|"""""|_|"""""|
"`-0-0-'"`-0-0-'"`-0-0-'"`-0-0-'"`-0-0-'
"#;
println!("{}", logo.blue().bold());
match cli.command {
Some(Commands::Collect { auth_mode }) => match auth_mode {
AuthenticationMode::AccessToken { token, common } => {
// Get verbose flag from CLI
if let Err(e) = setup_logger(common.debug) {
return Err(CirroError::Unknown(e.to_string()));
}
if let Err(e) = collect::collect_with_access_token(
token,
common.cloud,
common.output_path,
false,
None,
)
.await
{
return Err(e);
}
}
AuthenticationMode::Azcli {
tenant_id,
subscription_id,
common,
} => {
// Get verbose flag from CLI
if let Err(e) = setup_logger(common.debug) {
return Err(CirroError::Unknown(e.to_string()));
}
if let Err(e) = collect::collect_with_azure_cli(
tenant_id,
subscription_id,
common.mode,
common.cloud,
common.output_path,
false,
None,
)
.await
{
return Err(e);
}
}
AuthenticationMode::ClientSecret {
client_id,
client_secret,
tenant_id,
common,
} => {
// Get verbose flag from CLI
if let Err(e) = setup_logger(common.debug) {
return Err(CirroError::Unknown(e.to_string()));
}
if let Err(e) = collect::collect_with_client_secret(
tenant_id,
client_id,
client_secret,
common.mode,
common.cloud,
common.output_path,
false,
None,
)
.await
{
return Err(e);
}
}
AuthenticationMode::ClientCert {
client_id,
client_certificate,
tenant_id,
common,
} => {
// Get verbose flag from CLI
if let Err(e) = setup_logger(common.debug) {
return Err(CirroError::Unknown(e.to_string()));
}
if let Err(e) = collect::collect_with_client_cert(
tenant_id,
client_id,
client_certificate,
common.mode,
common.cloud,
common.output_path,
false,
None,
)
.await
{
return Err(e);
}
}
_ => {
return Err(CirroError::Unknown(
"Unsupported authentication mode".to_string(),
));
}
},
Some(Commands::Enrich { auth_mode, flags }) => match auth_mode {
AuthenticationMode::AccessToken { token, common } => {
// Get verbose flag from CLI
if let Err(e) = setup_logger(common.debug) {
return Err(CirroError::Unknown(e.to_string()));
}
if let Err(e) = collect::collect_with_access_token(
token,
common.cloud,
common.output_path,
true,
Some(flags),
)
.await
{
return Err(e);
}
}
AuthenticationMode::Azcli {
tenant_id,
subscription_id,
common,
} => {
// Get verbose flag from CLI
if let Err(e) = setup_logger(common.debug) {
return Err(CirroError::Unknown(e.to_string()));
}
if let Err(e) = collect::collect_with_azure_cli(
tenant_id,
subscription_id,
common.mode,
common.cloud,
common.output_path,
true,
Some(flags),
)
.await
{
return Err(e);
}
}
AuthenticationMode::ClientSecret {
client_id,
client_secret,
tenant_id,
common,
} => {
// Get verbose flag from CLI
if let Err(e) = setup_logger(common.debug) {
return Err(CirroError::Unknown(e.to_string()));
}
if let Err(e) = collect::collect_with_client_secret(
tenant_id,
client_id,
client_secret,
common.mode,
common.cloud,
common.output_path,
true,
Some(flags),
)
.await
{
return Err(e);
}
}
AuthenticationMode::ClientCert {
client_id,
client_certificate,
tenant_id,
common,
} => {
// Get verbose flag from CLI
if let Err(e) = setup_logger(common.debug) {
return Err(CirroError::Unknown(e.to_string()));
}
if let Err(e) = collect::collect_with_client_cert(
tenant_id,
client_id,
client_certificate,
common.mode,
common.cloud,
common.output_path,
true,
Some(flags),
)
.await
{
return Err(e);
}
}
_ => {
return Err(CirroError::Unknown(
"Unsupported authentication mode".to_string(),
));
}
},
None => {}
}
Ok(())
}
src/styles.rs → cirro/src/styles.rs
View File
File renamed without changes.
cliff.toml
-108
View File
@@ -1,108 +0,0 @@
# git-cliff ~ configuration file
# https://git-cliff.org/docs/configuration
[changelog]
# A Tera template to be rendered as the changelog's header.
# See https://keats.github.io/tera/docs/#introduction
header = """
# Changelog\n
All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).\n
"""
# A Tera template to be rendered for each release in the changelog.
# See https://keats.github.io/tera/docs/#introduction
body = """
{%- macro remote_url() -%}
https://github.com/{{ remote.github.owner }}/{{ remote.github.repo }}
{%- endmacro -%}
{% if version -%}
## [{{ version | trim_start_matches(pat="v") }}] - {{ timestamp | date(format="%Y-%m-%d") }}
{% else -%}
## [Unreleased]
{% endif -%}
{% for group, commits in commits | group_by(attribute="group") %}
### {{ group | upper_first }}
{% for commit in commits | unique(attribute="message") %}
- {{ commit.message | split(pat="\n") | first | upper_first | trim }}\
{% if commit.remote.username %} by @{{ commit.remote.username }}{%- endif -%}
{% if commit.remote.pr_number %} in \
[#{{ commit.remote.pr_number }}]({{ self::remote_url() }}/pull/{{ commit.remote.pr_number }}) \
{%- endif -%}
{% endfor %}
{% endfor %}
{%- if github.contributors | filter(attribute="is_first_time", value=true) | length != 0 %}
### New Contributors
{%- endif -%}
{% for contributor in github.contributors | filter(attribute="is_first_time", value=true) %}
* @{{ contributor.username }} made their first contribution
{%- if contributor.pr_number %} in \
[#{{ contributor.pr_number }}]({{ self::remote_url() }}/pull/{{ contributor.pr_number }}) \
{%- endif %}
{%- endfor %}\n
{%- if github.contributors | filter(attribute="is_first_time", value=true) | length != 0 %}{% raw %}\n{% endraw -%}{% endif %}
"""
# A Tera template to be rendered as the changelog's footer.
# See https://keats.github.io/tera/docs/#introduction
footer = """
{%- macro remote_url() -%}
https://github.com/{{ remote.github.owner }}/{{ remote.github.repo }}
{%- endmacro -%}
{% for release in releases -%}
{% if release.version -%}
{% if release.previous.version -%}
[{{ release.version | trim_start_matches(pat="v") }}]: \
{{ self::remote_url() }}/compare/{{ release.previous.version }}...{{ release.version }}
{% endif -%}
{% else -%}
[unreleased]: {{ self::remote_url() }}/compare/{{ release.previous.version }}...HEAD
{% endif -%}
{% endfor %}
<!-- generated by git-cliff -->
"""
# Remove leading and trailing whitespaces from the changelog's body.
trim = true
[git]
# Parse commits according to the conventional commits specification.
# See https://www.conventionalcommits.org
conventional_commits = true
# Exclude commits that do not match the conventional commits specification.
filter_unconventional = true
# An array of regex based parsers to modify commit messages prior to further processing.
commit_preprocessors = [
# Remove issue numbers.
{ pattern = '\((\w+\s)?#([0-9]+)\)', replace = "" },
]
# An array of regex based parsers for extracting data from the commit message.
# Assigns commits to groups.
# Optionally sets the commit's scope and can decide to exclude commits from further processing.
commit_parsers = [
{ message = "^[a|A]dd", group = "Added" },
{ message = "^[s|S]upport", group = "Added" },
{ message = "^[r|R]emove", group = "Removed" },
{ message = "^.*: add", group = "Added" },
{ message = "^.*: support", group = "Added" },
{ message = "^.*: remove", group = "Removed" },
{ message = "^.*: delete", group = "Removed" },
{ message = "^test", group = "Fixed" },
{ message = "^fix", group = "Fixed" },
{ message = "^.*: fix", group = "Fixed" },
{ message = "^.*", group = "Changed" },
{ body = "$^", skip = true },
]
# Exclude commits that are not matched by any commit parser.
filter_commits = false
# Order releases topologically instead of chronologically.
topo_order = false
# Order of commits in each group/release within the changelog.
# Allowed values: newest, oldest
sort_commits = "newest"
dist-workspace.toml
+9 -5
View File
@@ -4,7 +4,7 @@ members = ["cargo:."]
# Config for 'dist'
[dist]
# The preferred dist version to use in CI (Cargo.toml SemVer syntax)
cargo-dist-version = "0.30.2"
cargo-dist-version = "0.28.6"
# CI backends to support
ci = "github"
# The installers to generate for each app
@@ -23,7 +23,11 @@ install-path = "CARGO_HOME"
# Whether to install an updater program
install-updater = false
[dist.github-custom-runners]
aarch64-apple-darwin = "macos-15"
aarch64-unknown-linux-gnu.container = "ghcr.io/rust-cross/cargo-zigbuild"
aarch64-unknown-linux-musl.container = "ghcr.io/rust-cross/cargo-zigbuild"
[dist.github-custom-runners.aarch64-unknown-linux-gnu.container]
host = "x86_64-unknown-linux-musl"
image = "quay.io/pypa/manylinux_2_28_x86_64"
[dist.github-action-commits]
"actions/checkout" = "11bd71901bbe5b1630ceea73d27597364c9af683" # v4
"actions/upload-artifact" = "ea165f8d65b6e75b540449e92b4886f43607fa02" # v4.6.2
"actions/download-artifact" = "95815c38cf2ff2164869cbab79da8d1f422bc89e" # v4.2.1
docs/analysis/additional-nodes.md
-42
View File
@@ -1,42 +0,0 @@
# Additional Nodes
These nodes are not direct objects in Entra ID or Azure but are created as a result of ingesting other types of nodes.
## Node Types
- **[BastionIPConfig](additional-nodes/bastion-ip-config.md)** - IP configurations for Bastion hosts
- **[Certificate](additional-nodes/certificate.md)** - X.509 certificates used for authentication
- **[ClientSecret](additional-nodes/client-secret.md)** - Client secrets used for application authentication
- **[Container](additional-nodes/container.md)** - Individual containers within container groups
- **[CustomDnsConfig](additional-nodes/custom-dns-config.md)** - Custom DNS configurations for private endpoints
- **[ExpressRouteCircuitPeering](additional-nodes/expressroute-circuit-peering.md)** - ExpressRoute peering configurations attached to circuits
- **[FederatedIdentityCredential](additional-nodes/federated-identity-credential.md)** - Federated identity credentials for applications
- **[GraphAppRole](additional-nodes/graph-app-role.md)** - Application roles within Entra ID applications
- **[HybridExtension](additional-nodes/hybrid-extension.md)** - Extensions attached to hybrid machines
- **[HybridIPAddress](additional-nodes/hybrid-ip-address.md)** - IP addresses discovered for hybrid machines
- **[IPConfig](additional-nodes/ip-config.md)** - IP configuration details for network interfaces
- **[KeyVaultCertificate](additional-nodes/key-vault-certificate.md)** - Key Vault certificate metadata
- **[KeyVaultCertificateValue](additional-nodes/key-vault-certificate-value.md)** - Versioned Key Vault certificate data and policy details
- **[KeyVaultSecret](additional-nodes/key-vault-secret.md)** - Key Vault secret metadata
- **[KeyVaultSecretValue](additional-nodes/key-vault-secret-value.md)** - Versioned Key Vault secret values
- **[LoadBalancerBackendAddressPool](additional-nodes/load-balancer-backend-address-pool.md)** - Backend address pools attached to load balancers
- **[LoadBalancerFrontendIPConfiguration](additional-nodes/load-balancer-frontend-ip-configuration.md)** - Frontend IP configurations for load balancers
- **[LoadBalancerRule](additional-nodes/load-balancer-rule.md)** - Rule definitions attached to load balancers
- **[NetworkGateway](additional-nodes/network-gateway.md)** - Virtual network gateways
- **[NetworkPeering](additional-nodes/network-peering.md)** - Virtual network peering connections
- **[NetworkRoute](additional-nodes/network-route.md)** - Individual routes within route tables
- **[NetAppAccountADDomain](additional-nodes/netapp-account-ad-domain.md)** - Active Directory domain configurations for NetApp accounts
- **[NetAppVolumeMount](additional-nodes/netapp-volume-mount.md)** - Mount targets for NetApp volumes
- **[NSGRule](additional-nodes/nsg-rule.md)** - Network security group rules
- **[PrivateEndpointConnection](additional-nodes/private-endpoint-connection.md)** - Embedded private endpoint connection objects on ARM resources
- **[PrivateLinkServiceConnection](additional-nodes/private-link-service-connection.md)** - Private link service connections
- **[RegistryCredential](additional-nodes/registry-credential.md)** - Container registry credentials associated with container groups
- **[RunbookParameter](additional-nodes/runbook-parameter.md)** - Parameters defined for automation runbooks
- **[ServicePrincipalEndpoint](additional-nodes/service-principal-endpoint.md)** - Service principal endpoints and metadata
- **[StorageAccountKey](additional-nodes/storage-account-key.md)** - Storage account access keys
- **[StorageAccountDFS](additional-nodes/storage-account-dfs.md)** - Data Lake Storage Gen2 filesystems
- **[Subnet](additional-nodes/subnet.md)** - Virtual network subnets
- **[VerifiedDomain](additional-nodes/verified-domain.md)** - Verified domains for Entra ID organizations
- **[VMExtension](additional-nodes/vm-extension.md)** - Extensions attached to Azure virtual machines
- **[VMRunCommand](additional-nodes/vm-run-command.md)** - Run command resources for Azure virtual machines
- **[VMRunCommandParameter](additional-nodes/vm-run-command-parameter.md)** - Parameters for virtual machine run commands
docs/analysis/additional-nodes/analysis-service-firewall-rule.md
-27
View File
@@ -1,27 +0,0 @@
# AnalysisServiceFirewallRule
Represents firewall rule entries attached to an Analysis Services server.
**Labels:** `:AnalysisServiceFirewallRule`
**Properties:**
- `name` - Firewall rule name
- `serverId` - Parent Analysis Services server resource ID (used with `name` as a composite key)
- `rangeStart` - Start of allowed IPv4 range
- `rangeEnd` - End of allowed IPv4 range
## Relationships
### Incoming
- **AnalysisServiceServer** → `HAS_RULE` → **AnalysisServiceFirewallRule** - Server that owns the firewall rule
## Examples
```cypher
// List firewall rules per Analysis Services server
MATCH (s:AnalysisServiceServer)-[:HAS_RULE]->(r:AnalysisServiceFirewallRule)
RETURN s.serverFullName, r.name, r.rangeStart, r.rangeEnd
ORDER BY s.serverFullName, r.name
```
docs/analysis/additional-nodes/azure-stack-edge-machine-nic.md
-29
View File
@@ -1,29 +0,0 @@
# AzureStackEdgeMachineNic
Represents network interface cards attached to Azure Stack HCI edge machines (not an ARM resource itself).
**Labels:** `:AzureStackEdgeMachineNic`
**Properties:**
- `machineId` - Parent edge machine ID (composite key with `adapterName`)
- `adapterName` - Network adapter name (composite key with `machineId`)
- `dnsServers` - DNS server addresses
- `ip4Address` - IPv4 address
- `macAddress` - MAC address
- `rdmaCapability` - RDMA capability of the adapter
- `subnetMask` - Subnet mask
## Relationships
### Incoming
- **AzureStackEdgeMachine** → `HAS_NIC` → **AzureStackEdgeMachineNic** - Parent edge machine
## Examples
```cypher
// Find edge machines and their NIC details
MATCH (em:AzureStackEdgeMachine)-[:HAS_NIC]->(nic:AzureStackEdgeMachineNic)
RETURN em.id, nic.adapterName, nic.ip4Address, nic.macAddress, nic.rdmaCapability
```
docs/analysis/additional-nodes/bastion-ip-config.md
-15
View File
@@ -1,15 +0,0 @@
# BastionIPConfig
Represents IP configurations for Bastion hosts.
**Labels:** `:BastionIPConfig`
**Properties:**
- `id` - Bastion IP config ID (primary key)
- `privateIPAllocationMethod` - Private IP allocation method
**Relationships:**
- `HAS_CONFIG` ← BastionHost
- `HAS_IP` → PublicIPAddress
- `CONTAINS` ← Subnet
docs/analysis/additional-nodes/certificate.md
-14
View File
@@ -1,14 +0,0 @@
# Certificate
Represents certificates for applications and service principals.
**Labels:** `:Certificate`
**Properties:**
- `thumbprint` - Certificate thumbprint (primary key)
- `displayName` - Certificate display name
- `startDateTime` - Start date/time
- `endDateTime` - End date/time
- `type` - Certificate type
- `usage` - Certificate usage
docs/analysis/additional-nodes/client-secret.md
-15
View File
@@ -1,15 +0,0 @@
# ClientSecret
Represents client secrets for applications and service principals.
**Labels:** `:ClientSecret`
**Properties:**
- `keyId` - Key identifier (primary key)
- `customKeyIdentifier` - Custom key identifier
- `displayName` - Secret display name
- `startDateTime` - Start date/time
- `endDateTime` - End date/time
- `hint` - Password hint
- `secretText` - Secret text (if available)
docs/analysis/additional-nodes/container-image.md
-25
View File
@@ -1,25 +0,0 @@
# ContainerImage
Represents a container image reference used by Container Apps (including tag/digest in the name).
**Labels:** `:ContainerImage`
**Properties:**
- `name` - Full image name (including registry prefix and tag/digest)
## Relationships
### Incoming
- **ContainerRegistry** → `CONTAINS_IMAGE` → **ContainerImage** - Registry that hosts the image reference
### Outgoing
- **ContainerImage** → `HAS_ENVVAR` → **EnvVar** - Environment variables defined for the image in a Container App template
## Examples
```cypher
// List images and their registries
MATCH (acr:ContainerRegistry)-[:CONTAINS_IMAGE]->(img:ContainerImage)
RETURN acr.loginServer, img.name
```
docs/analysis/additional-nodes/container.md
-15
View File
@@ -1,15 +0,0 @@
# Container
Represents an individual container within an Azure Container Instances container group.
**Labels:** `:Container`
**Properties:**
- `image` - Container image name (primary key)
- `name` - Container name
- `environmentVariables` - List of environment variables in "name:value" format
## Relationships
- **ContainerGroup** → **HAS_CONTAINER** → **Container** - Container within container groups
docs/analysis/additional-nodes/custom-dns-config.md
-13
View File
@@ -1,13 +0,0 @@
# CustomDnsConfig
Represents custom DNS configurations for private endpoints.
**Labels:** `:CustomDnsConfig`
**Properties:**
- `fqdn` - Fully qualified domain name (primary key)
- `ipAddresses` - Array of IP addresses
**Relationships:**
- `HAS_DNS_CONFIG` ← PrivateEndpoint
docs/analysis/additional-nodes/env-var.md
-25
View File
@@ -1,25 +0,0 @@
# EnvVar
Represents an environment variable defined for a container in a Container App template.
**Labels:** `:EnvVar`
**Properties:**
- `name` - Variable name (primary key per image edge)
- `value` - Literal value (if provided)
- `secretRef` - Secret reference name (if value is sourced from a secret)
## Relationships
### Incoming
- **ContainerImage** → `HAS_ENVVAR` → **EnvVar** - Environment variable attached to an image definition
## Examples
```cypher
// Find env vars that reference secrets
MATCH (img:ContainerImage)-[:HAS_ENVVAR]->(e:EnvVar)
WHERE e.secretRef IS NOT NULL
RETURN img.name, e.name, e.secretRef
```
docs/analysis/additional-nodes/expressroute-circuit-peering.md
-33
View File
@@ -1,33 +0,0 @@
# ExpressRouteCircuitPeering
Represents an ExpressRoute peering object attached to an ExpressRoute circuit.
**Labels:** `:ExpressRouteCircuitPeering`
**Properties:**
- `id` - Peering resource ID (primary key)
- `name` - Peering name
- `azureASN` - Azure ASN value
- `peerASN` - Customer/provider ASN value
- `primaryPeerAddressPrefix` - Primary peering CIDR
- `secondaryPeerAddressPrefix` - Secondary peering CIDR
- `primaryAzurePort` - Primary Azure port
- `secondaryAzurePort` - Secondary Azure port
- `vlanId` - VLAN ID used for peering
- `peeringType` - Peering type
- `state` - Peering state
## Relationships
### Incoming
- **ExpressRouteCircuit** → `HAS_PEERING` → **ExpressRouteCircuitPeering** - Circuit-to-peering linkage
## Examples
```cypher
// List ExpressRoute circuits and their peerings
MATCH (c:ExpressRouteCircuit)-[:HAS_PEERING]->(p:ExpressRouteCircuitPeering)
RETURN c.name, p.name, p.peeringType, p.state
```
docs/analysis/additional-nodes/federated-identity-credential.md
-17
View File
@@ -1,17 +0,0 @@
# FederatedIdentityCredential
Represents federated identity credentials for applications.
**Labels:** `:FederatedIdentityCredential`
**Properties:**
- `id` - Credential ID (primary key)
- `subject` - Subject identifier
- `audiences` - Array of audiences
- `issuer` - Token issuer
- `name` - Credential name
- `description` - Credential description
**Relationships:**
- `AUTHENTICATES` → GraphApplication
docs/analysis/additional-nodes/graph-app-role.md
-21
View File
@@ -1,21 +0,0 @@
# GraphAppRole
Represents application roles within Entra ID applications.
**Labels:** `:GraphAppRole`
**Properties:**
- `id` - App role ID (primary key)
- `displayName` - Role display name
- `description` - Role description
- `allowedMemberTypes` - Array of allowed member types
- `isEnabled` - Whether the role is enabled
- `isPreAuthorizationRequired` - Whether pre-authorization is required
- `isPrivate` - Whether the role is private
- `origin` - Role origin
- `value` - Role value
**Relationships:**
- `HAS_APPROLE` ← GraphApplication
- `HAS_APPROLE` ← GraphServicePrincipal
docs/analysis/additional-nodes/hybrid-ip-address.md
-34
View File
@@ -1,34 +0,0 @@
# HybridIPAddress
Represents IP addresses associated with hybrid machines (not an ARM resource itself).
**Labels:** `:HybridIPAddress`
**Properties:**
- `id` - IP address ID (primary key)
- `ipAddress` - The IP address value
- `type` - IP address type (e.g., private, public)
- `subnetId` - Associated subnet ID if applicable
- `isPrimary` - Whether this is the primary IP address
## Relationships
### Incoming
- **HybridMachine** → `HAS_IP` → HybridIPAddress - Parent hybrid machine
## Examples
```cypher
// Find all hybrid machines and their IP addresses
MATCH (hm:HybridMachine)-[:HAS_IP]->(ip:HybridIPAddress)
RETURN hm.displayName, ip.ipAddress, ip.type, ip.isPrimary
```
```cypher
// Find hybrid machines with private IP addresses
MATCH (hm:HybridMachine)-[:HAS_IP]->(ip:HybridIPAddress)
WHERE ip.type = "private"
RETURN hm.displayName, ip.ipAddress
```
docs/analysis/additional-nodes/hybrid-run-command-parameter.md
-17
View File
@@ -1,17 +0,0 @@
# HybridRunCommandParameter
Represents a parameter for a hybrid machine run command.
**Labels:** `:HybridRunCommandParameter`
**Properties:**
- `name` - Parameter name (composite key with `commandId`)
- `commandId` - Parent run command ID (composite key)
- `value` - Parameter value
## Relationships
### Incoming
- **HybridRunCommand** → `HAS_PARAMETER` → **HybridRunCommandParameter** - Parent run command
docs/analysis/additional-nodes/hybrid-run-command.md
-34
View File
@@ -1,34 +0,0 @@
# HybridRunCommand
Represents a run command resource associated with an Azure Arc-enabled hybrid machine.
**Labels:** `:ArmResource:HybridRunCommand`
**Properties:**
- `id` - Run command resource ID (primary key)
- `asyncExecution` - Whether execution is asynchronous
- `runAsUser` - User context used to run the command
- `timeoutInSeconds` - Command timeout in seconds
- `treatFailureAsDeploymentFailure` - Failure handling behavior
- `script` - Script body
- `output` - Command output from the last execution
## Relationships
### Incoming
- **HybridMachine** → `HAS_RUN_COMMAND` → **HybridRunCommand** - Parent hybrid machine
### Outgoing
- **HybridRunCommand** → `HAS_PARAMETER` → **HybridRunCommandParameter** - Parameters bound to the run command
## Examples
```cypher
// Find hybrid run commands and their parameters
MATCH (hm:HybridMachine)-[:HAS_RUN_COMMAND]->(cmd:HybridRunCommand)
OPTIONAL MATCH (cmd)-[:HAS_PARAMETER]->(p:HybridRunCommandParameter)
RETURN hm.displayName, cmd.id, collect(p.name) AS parameterNames
```
docs/analysis/additional-nodes/ip-config.md
-40
View File
@@ -1,40 +0,0 @@
# IPConfiguration
Represents IP configurations for network interfaces (not an ARM resource itself).
**Labels:** `:IPConfiguration`
**Properties:**
- `id` - IP configuration ID (primary key)
- `name` - Configuration name
- `type` - Configuration type
- `privateIPAddress` - Private IP address
- `privateIPAddressVersion` - IP address version (IPv4/IPv6)
- `privateIPAllocationMethod` - Private IP allocation method (Static/Dynamic)
## Relationships
### Incoming
- **NetworkInterface** → `HAS_CONFIG` → **IPConfiguration** - Parent network interface
- **Subnet** → `CONTAINS` → **IPConfiguration** - Subnet containing this IP configuration
### Outgoing
- **IPConfiguration** → `HAS_IP` → **PublicIPAddress** - Associated public IP address (if any)
## Examples
```cypher
// Find all static IP configurations
MATCH (ip:IPConfiguration)
WHERE ip.privateIPAllocationMethod = "Static"
RETURN ip.name, ip.privateIPAddress, ip.privateIPAddressVersion
```
```cypher
// Find IP configurations with public IPs
MATCH (ip:IPConfiguration)-[:HAS_IP]->(pub:PublicIPAddress)
RETURN ip.privateIPAddress, pub.ipAddress
```
docs/analysis/additional-nodes/key-vault-certificate-value.md
-48
View File
@@ -1,48 +0,0 @@
# KeyVaultCertificateValue
Represents versioned Azure Key Vault certificate value objects, including certificate material and issuance policy metadata.
**Labels:** `:KeyVaultCertificateValue`
**Properties:**
- `id` - Certificate value version ID (primary key)
- `name` - Certificate name
- `cer` - Encoded certificate content
- `contentType` - Certificate content type
- `kid` - Related Key Vault key ID
- `sid` - Related Key Vault secret ID
- `x509Thumbprint` - Thumbprint value
- `x509ThumbprintHex` - Hex-encoded thumbprint
- `enabled` - Whether the certificate value is enabled
- `notBefore` - Time before which the certificate value is invalid
- `expires` - Expiration time
- `created` - Creation time
- `updated` - Last update time
- `recoverableDays` - Number of recoverable days
- `recoveryLevel` - Recovery level
- `issuerTransparency` - Certificate transparency setting from the issuer policy
- `issuerType` - Issuer certificate type
- `issuerName` - Issuer name
- `exportable` - Whether the private key is exportable
- `keyType` - Key algorithm or key type
- `secretContentType` - Content type used for the backing secret
- `keyUsage` - X.509 key usage values
- `x509SubjectName` - X.509 subject name from the policy
- `x509DnsNames` - DNS subject alternative names
- `x509Emails` - Email subject alternative names
- `x509Upns` - UPN subject alternative names
- `tags` - Certificate tags as `key:value` strings
**Relationships:**
- `HAS_VERSION` ← KeyVaultCertificate - Parent certificate metadata node
## Example Queries
Find versioned Key Vault certificate values and their issuance details:
```cypher
MATCH (c:KeyVaultCertificate)-[:HAS_VERSION]->(v:KeyVaultCertificateValue)
RETURN c.name, v.x509SubjectName, v.secretContentType, v.expires
```
docs/analysis/additional-nodes/key-vault-certificate.md
-38
View File
@@ -1,38 +0,0 @@
# KeyVaultCertificate
Represents Azure Key Vault certificate metadata.
**Labels:** `:KeyVaultCertificate`
**Properties:**
- `id` - Certificate ID (primary key)
- `name` - Certificate name
- `subject` - Certificate subject
- `x509Thumbprint` - Thumbprint value
- `x509ThumbprintHex` - Hex-encoded thumbprint
- `enabled` - Whether the certificate is enabled
- `notBefore` - Time before which the certificate is invalid
- `expires` - Expiration time
- `created` - Creation time
- `updated` - Last update time
- `recoverableDays` - Number of recoverable days
- `recoveryLevel` - Recovery level
- `tags` - Certificate tags as `key:value` strings
**Relationships:**
- `HAS_CERTIFICATE` ← KeyVault - Parent Key Vault
- `HAS_VERSION` → KeyVaultCertificateValue - Versioned certificate value objects
- `MATCHES_CERT` → Certificate - Matched application or service principal certificates by thumbprint
## Example Queries
Find Key Vault certificates, their versions, and any matched Entra certificates:
```cypher
MATCH (kv:KeyVault)-[:HAS_CERTIFICATE]->(c:KeyVaultCertificate)
OPTIONAL MATCH (c)-[:HAS_VERSION]->(v:KeyVaultCertificateValue)
OPTIONAL MATCH (c)-[:MATCHES_CERT]->(cert:Certificate)
RETURN kv.name, c.name, collect(DISTINCT v.id) AS versions, collect(DISTINCT cert.thumbprint) AS matchedThumbprints
```
docs/analysis/additional-nodes/key-vault-secret-value.md
-37
View File
@@ -1,37 +0,0 @@
# KeyVaultSecretValue
Represents versioned Azure Key Vault secret values.
**Labels:** `:KeyVaultSecretValue`
**Properties:**
- `id` - Secret value version ID (primary key)
- `name` - Secret name
- `value` - Secret value collected from Key Vault
- `managed` - Whether Azure manages the secret
- `contentType` - Secret content type
- `kid` - Related Key Vault key ID, when present
- `enabled` - Whether the secret value is enabled
- `notBefore` - Time before which the secret value is invalid
- `expires` - Expiration time
- `created` - Creation time
- `updated` - Last update time
- `recoverableDays` - Number of recoverable days
- `recoveryLevel` - Recovery level
- `tags` - Secret tags as `key:value` strings
**Relationships:**
- `HAS_VERSION` ← KeyVaultSecret - Parent secret metadata node
- `MATCHES_SECRET` → ClientSecret - Matched client secrets discovered during post-processing
## Example Queries
Find versioned Key Vault secret values and any matched client secrets:
```cypher
MATCH (s:KeyVaultSecret)-[:HAS_VERSION]->(v:KeyVaultSecretValue)
OPTIONAL MATCH (v)-[:MATCHES_SECRET]->(c:ClientSecret)
RETURN s.name, v.id, c.hint
```
docs/analysis/additional-nodes/key-vault-secret.md
-38
View File
@@ -1,38 +0,0 @@
# KeyVaultSecret
Represents Azure Key Vault secret metadata.
**Labels:** `:KeyVaultSecret`
**Properties:**
- `id` - Secret ID (primary key)
- `name` - Secret name
- `managed` - Whether Azure manages the secret
- `contentType` - Secret content type
- `kid` - Related Key Vault key ID, when present
- `enabled` - Whether the secret is enabled
- `notBefore` - Time before which the secret is invalid
- `expires` - Expiration time
- `created` - Creation time
- `updated` - Last update time
- `recoverableDays` - Number of recoverable days
- `recoveryLevel` - Recovery level
- `tags` - Secret tags as `key:value` strings
**Relationships:**
- `HAS_SECRET` ← KeyVault - Parent Key Vault
- `HAS_VERSION` → KeyVaultSecretValue - Versioned secret values
- `USED_BY` → WebCertificate - Web certificates backed by this secret
## Example Queries
Find Key Vault secrets, their versions, and any web certificates that consume them:
```cypher
MATCH (kv:KeyVault)-[:HAS_SECRET]->(s:KeyVaultSecret)
OPTIONAL MATCH (s)-[:HAS_VERSION]->(v:KeyVaultSecretValue)
OPTIONAL MATCH (s)-[:USED_BY]->(wc:WebCertificate)
RETURN kv.name, s.name, collect(DISTINCT v.id) AS versions, collect(DISTINCT wc.name) AS webCertificates
```
docs/analysis/additional-nodes/load-balancer-backend-address-pool.md
-22
View File
@@ -1,22 +0,0 @@
# LoadBalancerBackendAddressPool
Represents backend address pool objects attached to a load balancer.
**Labels:** `:LoadBalancerBackendAddressPool`
**Properties:**
- `id` - Backend address pool resource ID (primary key)
- `name` - Backend pool name
- `provisioningState` - Current provisioning state
- `type` - Azure resource type value
## Relationships
### Incoming
- **LoadBalancer** -> `HAS_BACKEND_POOL` -> **LoadBalancerBackendAddressPool** - Parent load balancer
### Outgoing
- **LoadBalancerBackendAddressPool** -> `ASSOCIATED_WITH` -> **NetworkInterface** - Backend NIC configuration reference
docs/analysis/additional-nodes/load-balancer-frontend-ip-configuration.md
-23
View File
@@ -1,23 +0,0 @@
# LoadBalancerFrontendIPConfiguration
Represents frontend IP configuration objects for a load balancer.
**Labels:** `:LoadBalancerFrontendIPConfiguration`
**Properties:**
- `id` - Frontend configuration resource ID (primary key)
- `name` - Frontend configuration name
- `privateIPAllocationMethod` - Private IP allocation method
- `provisioningState` - Current provisioning state
- `type` - Azure resource type value
## Relationships
### Incoming
- **LoadBalancer** -> `HAS_LB_FRONTEND` -> **LoadBalancerFrontendIPConfiguration** - Parent load balancer
### Outgoing
- **LoadBalancerFrontendIPConfiguration** -> `HAS_IP` -> **PublicIPAddress** - Associated public IP (if present)
docs/analysis/additional-nodes/load-balancer-rule.md
-29
View File
@@ -1,29 +0,0 @@
# LoadBalancerRule
Represents load balancing rule definitions attached to a load balancer.
**Labels:** `:LoadBalancerRule`
**Properties:**
- `id` - Rule resource ID (primary key)
- `name` - Rule name
- `protocol` - Protocol used by the rule
- `frontendPort` - Frontend listening port
- `backendPort` - Backend target port
- `allowBackendPortConflict` - Whether backend port conflict is allowed
- `disableConnectionTracking` - Whether connection tracking is disabled
- `disableOutboundSnat` - Whether outbound SNAT is disabled
- `enableConnectionTracking` - Whether connection tracking is enabled
- `enableDestinationServiceEndpoint` - Whether destination service endpoint is enabled
- `enableFloatingIP` - Whether floating IP is enabled
- `enableTcpReset` - Whether TCP reset is enabled
- `idleTimeoutInMinutes` - Idle timeout value
- `provisioningState` - Current provisioning state
- `type` - Azure resource type value
## Relationships
### Incoming
- **LoadBalancer** -> `HAS_RULE` -> **LoadBalancerRule** - Load balancer that owns the rule
docs/analysis/additional-nodes/netapp-account-ad-domain.md
-28
View File
@@ -1,28 +0,0 @@
# NetAppAccountADDomain
Represents an Active Directory domain configuration for an Azure NetApp Files account.
**Labels:** `:NetAppAccountADDomain`
**Properties:**
- `activeDirectoryId` - Active Directory identifier (primary key)
- `administrators` - List of administrator accounts
- `aesEncryption` - Whether AES encryption is enabled
- `allowLocalNfsUsersWithLdap` - Whether local NFS users are allowed with LDAP
- `backupOperators` - Backup operator accounts for the AD domain
- `dns` - DNS server configuration
- `domain` - Active Directory domain name
- `encryptDCConnections` - Whether domain controller connections are encrypted
- `kdcIP` - Kerberos KDC IP address
- `ldapOverTLS` - Whether LDAP over TLS is enabled
- `ldapSigning` - LDAP signing configuration
- `organizationalUnit` - Organizational unit for computer accounts
- `site` - Active Directory site name
- `smbServerName` - SMB server name
- `status` - Current status of the AD configuration
- `username` - Username for domain authentication
## Relationships
- **NetAppAccount** → **IS_DEPLOYED** → **NetAppAccountADDomain** - AD domain where NetApp is deployed
docs/analysis/additional-nodes/netapp-volume-mount.md
-16
View File
@@ -1,16 +0,0 @@
# NetAppVolumeMount
Represents a mount target for an Azure NetApp Files volume.
**Labels:** `:NetAppVolumeMount`
**Properties:**
- `mountTargetId` - Mount target identifier (primary key)
- `fileSystemId` - File system identifier
- `ipAddress` - IP address for mounting
- `smbServerFqdn` - SMB server fully qualified domain name
## Relationships
- **NetAppVolume** → **MOUNTS** → **NetAppVolumeMount** - Mount targets for volumes
docs/analysis/additional-nodes/network-gateway.md
-13
View File
@@ -1,13 +0,0 @@
# NetworkGateway
Represents virtual network gateways.
**Labels:** `:NetworkGateway`
**Properties:**
- `id` - Gateway ID (primary key)
**Relationships:**
- `HAS_GATEWAY` ← NetworkPeering
- `CONNECTS` → VirtualNetwork
docs/analysis/additional-nodes/network-peering.md
-41
View File
@@ -1,41 +0,0 @@
# NetworkPeering
Represents virtual network peering connections (not an ARM resource itself).
**Labels:** `:NetworkPeering`
**Properties:**
- `id` - Peering ID (primary key)
- `name` - Peering name
- `allowVirtualNetworkAccess` - Whether virtual network access is allowed
- `allowForwardedTraffic` - Whether forwarded traffic is allowed
- `allowGatewayTransit` - Whether gateway transit is allowed
- `useRemoteGateways` - Whether to use remote gateways
- `peeringState` - Current state of the peering (Connected, Disconnected, etc.)
## Relationships
### Incoming
- **VirtualNetwork** → `HAS_PEERING` → **NetworkPeering** - Source virtual network
### Outgoing
- **NetworkPeering** → `PEERS_WITH` → **ArmResource** - Target virtual network
## Examples
```cypher
// Find all connected peering relationships
MATCH (vnet:VirtualNetwork)-[:HAS_PEERING]->(peering:NetworkPeering)
WHERE peering.peeringState = "Connected"
RETURN vnet.name, peering.name, peering.allowForwardedTraffic
```
```cypher
// Find bidirectional peering relationships
MATCH (vnet1:VirtualNetwork)-[:HAS_PEERING]->(p1:NetworkPeering)-[:PEERS_WITH]->(vnet2:VirtualNetwork)
MATCH (vnet2)-[:HAS_PEERING]->(p2:NetworkPeering)-[:PEERS_WITH]->(vnet1)
RETURN vnet1.name, vnet2.name, p1.peeringState, p2.peeringState
```
docs/analysis/additional-nodes/network-route.md
-18
View File
@@ -1,18 +0,0 @@
# NetworkRoute
Represents individual routes within route tables.
**Labels:** `:NetworkRoute`
**Properties:**
- `id` - Route ID (primary key)
- `name` - Route name
- `type` - Route type
- `addressPrefix` - Address prefix
- `nextHopType` - Next hop type
- `nextHopIpAddress` - Next hop IP address
- `hasBgpOverride` - Whether BGP override is enabled
**Relationships:**
- `HAS_ROUTE` ← RouteTable
docs/analysis/additional-nodes/nsg-rule.md
-18
View File
@@ -1,18 +0,0 @@
# NSGRule
Represents network security group rules.
**Labels:** `:NSGRule`
**Properties:**
- `name` - Rule name
- `description` - Rule description
- `protocol` - Network protocol
- `access` - Access type (Allow, Deny)
- `priority` - Rule priority
- `direction` - Traffic direction (Inbound, Outbound)
- `sourceAddressPrefix` - Source address prefix
- `destinationAddressPrefix` - Destination address prefix
- `sourcePortRange` - Source port range
- `destinationPortRange` - Destination port range
docs/analysis/additional-nodes/private-endpoint-connection.md
-26
View File
@@ -1,26 +0,0 @@
# PrivateEndpointConnection
Represents an embedded private endpoint connection object found on ARM resources.
**Labels:** `:PrivateEndpointConnection`
**Properties:**
- `id` - Private endpoint connection ID (primary key)
- `name` - Connection name
- `type` - Resource type
- `groupIds` - Target group IDs used by the connection
## Relationships
### Incoming
- **ArmResource** → `HAS_PRIVATE_ENDPOINT` → **PrivateEndpointConnection** - Parent resource references this connection object
## Examples
```cypher
// Find resources with private endpoint connections
MATCH (r:ArmResource)-[:HAS_PRIVATE_ENDPOINT]->(p:PrivateEndpointConnection)
RETURN r.name, r.type, p.name, p.groupIds
```
docs/analysis/additional-nodes/private-link-service-connection.md
-15
View File
@@ -1,15 +0,0 @@
# PrivateLinkServiceConnection
Represents private link service connections.
**Labels:** `:PrivateLinkServiceConnection`
**Properties:**
- `id` - Connection ID (primary key)
- `name` - Connection name
- `type` - Connection type
- `groupIds` - Array of group IDs
**Relationships:**
- `HAS_PRIVATE_ENDPOINT` → PrivateEndpoint
docs/analysis/additional-nodes/registry-credential.md
-25
View File
@@ -1,25 +0,0 @@
# RegistryCredential
Represents container image registry credentials associated with container groups.
**Labels:** `:RegistryCredential`
**Properties:**
- `username` - Registry username (primary key component)
- `server` - Registry server name (primary key component)
- `isDelegatedIdentity` - Whether delegated identity is used
## Relationships
### Outgoing
- **RegistryCredential** → `AUTHENTICATES` → **ContainerGroup** - Credential used to authenticate pulls for a container group
## Examples
```cypher
// Find registry credentials used by container groups
MATCH (cred:RegistryCredential)-[:AUTHENTICATES]->(cg:ContainerGroup)
RETURN cred.server, cred.username, cred.isDelegatedIdentity, cg.name
```
docs/analysis/additional-nodes/runbook-parameter.md
-36
View File
@@ -1,36 +0,0 @@
# RunbookParameter
Represents parameters defined for automation runbooks (not an ARM resource itself).
**Labels:** `:RunbookParameter`
**Properties:**
- `id` - Parameter ID (primary key)
- `name` - Parameter name
- `defaultValue` - Default value for the parameter
- `isMandatory` - Whether the parameter is mandatory
- `position` - Parameter position
- `type` - Parameter type
## Relationships
### Incoming
- **AutomationRunbook** → `HAS_PARAMETER` → RunbookParameter - Parent runbook
## Examples
```cypher
// Find all mandatory parameters
MATCH (p:RunbookParameter)
WHERE p.isMandatory = true
RETURN p.name, p.type, p.defaultValue
```
```cypher
// Find runbooks with the most parameters
MATCH (rb:AutomationRunbook)-[:HAS_PARAMETER]->(p:RunbookParameter)
RETURN rb.name, count(p) AS parameterCount
ORDER BY parameterCount DESC
```
docs/analysis/additional-nodes/service-principal-endpoint.md
-35
View File
@@ -1,35 +0,0 @@
# ServicePrincipalEndpoint
Represents service principal endpoints (not an ARM resource itself).
**Labels:** `:ServicePrincipalEndpoint`
**Properties:**
- `id` - Endpoint ID (primary key)
- `type` - Endpoint type
- `uri` - Endpoint URI
- `customKeyIdentifier` - Custom key identifier
- `displayName` - Endpoint display name
- `usage` - Endpoint usage description
## Relationships
### Incoming
- **ServicePrincipal** → `HAS_ENDPOINT` → ServicePrincipalEndpoint - Parent service principal
## Examples
```cypher
// Find all service principal endpoints
MATCH (sp:ServicePrincipal)-[:HAS_ENDPOINT]->(endpoint:ServicePrincipalEndpoint)
RETURN sp.displayName, endpoint.type, endpoint.uri, endpoint.displayName
```
```cypher
// Find service principals with specific endpoint types
MATCH (sp:ServicePrincipal)-[:HAS_ENDPOINT]->(endpoint:ServicePrincipalEndpoint)
WHERE endpoint.type = "oauth2"
RETURN sp.displayName, endpoint.uri
```
docs/analysis/additional-nodes/storage-account-dfs.md
-14
View File
@@ -1,14 +0,0 @@
# StorageAccountDFS
Represents Data Lake Storage Gen2 filesystems within Azure Storage Accounts.
**Labels:** `:StorageAccountDFS`
**Properties:**
- `id` - DFS filesystem ID (primary key)
- `name` - Filesystem name
**Relationships:**
- Connected from Storage Account via `HAS_DFS_FILESYSTEM` relationship
- `USED_BY` → Various services that use DFS storage (e.g., Synapse Workspaces)
docs/analysis/additional-nodes/storage-account-key.md
-15
View File
@@ -1,15 +0,0 @@
# StorageAccountKey
Represents storage account access keys.
**Labels:** `:StorageAccountKey`
**Properties:**
- `name` - Key name (primary key)
- `value` - Key value
- `permissions` - Key permissions
- `creationTime` - Key creation time
**Relationships:**
- `HAS_KEY` ← StorageAccount
docs/analysis/additional-nodes/subnet.md
-12
View File
@@ -1,12 +0,0 @@
# Subnet
Represents subnets within virtual networks.
**Labels:** `:Subnet`
**Properties:**
- `name` - Subnet name
- `addressPrefix` - Address prefix
- `provisioningState` - Provisioning state
- Various network configuration properties
docs/analysis/additional-nodes/verified-domain.md
-15
View File
@@ -1,15 +0,0 @@
# VerifiedDomain
Represents verified domains associated with Entra ID organizations. These are domains that have been verified as belonging to the organization.
**Labels:** `:VerifiedDomain`
**Properties:**
- `name` - Domain name (primary key)
- `isDefault` - Whether this is the default domain for the organization
- `isInitial` - Whether this is the initial domain created with the tenant
- `type` - Type of domain verification
**Notes:**
- The default domain is typically used for new user creation
docs/analysis/additional-nodes/vm-extension.md
-33
View File
@@ -1,33 +0,0 @@
# VMExtension
Represents an extension attached to an Azure virtual machine.
**Labels:** `:ArmResource:VMExtension`
**Properties:**
- `id` - Extension resource ID (primary key)
- `name` - Extension name
- `extType` - Extension type
- `location` - Resource location
- `provisioningState` - Provisioning state
- `publisher` - Extension publisher
- `autoUpgradeMinorVersion` - Auto-upgrade behavior
- `triggerForceUpgrade` - Force-upgrade trigger
- `vmType` - VM type metadata from extension settings
- `settings` - Serialized extension settings
## Relationships
### Incoming
- **VirtualMachine** → `HAS_EXTENSION` → **VMExtension** - Extension bound to a VM
- **HybridMachine** → `HAS_EXTENSION` → **VMExtension** - Extension attached to an Arc/hybrid machine
## Examples
```cypher
// Find extensions by publisher across VMs
MATCH (vm:VirtualMachine)-[:HAS_EXTENSION]->(ext:VMExtension)
RETURN vm.name, ext.name, ext.publisher, ext.provisioningState
```
docs/analysis/additional-nodes/vm-run-command-parameter.md
-15
View File
@@ -1,15 +0,0 @@
# VMRunCommandParameter
Represents a parameter for a virtual machine run command.
**Labels:** `:VMRunCommandParameter`
**Properties:**
- `name` - Parameter name (composite key with `commandId`)
- `commandId` - Parent run command ID (composite key)
- `value` - Parameter value
## Relationships
- **VMRunCommand** → **HAS_PARAMETER** → **VMRunCommandParameter** - Parameters for run commands
docs/analysis/additional-nodes/vm-run-command.md
-33
View File
@@ -1,33 +0,0 @@
# VMRunCommand
Represents a run command resource associated with a virtual machine.
**Labels:** `:VMRunCommand`
**Properties:**
- `id` - Run command resource ID (primary key)
- `asyncExecution` - Whether execution is asynchronous
- `runAsUser` - User context used to run command
- `timeoutInSeconds` - Command timeout
- `treatFailureAsDeploymentFailure` - Failure handling behavior
- `script` - Script body
## Relationships
### Incoming
- **VirtualMachine** → `HAS_RUN_COMMAND` → **VMRunCommand** - VM run-command association
### Outgoing
- **VMRunCommand** → `HAS_PARAMETER` → **VMRunCommandParameter** - Parameters bound to the run command
## Examples
```cypher
// Find VM run commands and parameters
MATCH (vm:VirtualMachine)-[:HAS_RUN_COMMAND]->(cmd:VMRunCommand)
OPTIONAL MATCH (cmd)-[:HAS_PARAMETER]->(p:VMRunCommandParameter)
RETURN vm.name, cmd.id, collect(p.name) AS parameterNames
```
docs/analysis/arm-nodes.md
-21
View File
@@ -1,21 +0,0 @@
# ARM Nodes
This document describes all the Azure Resource Manager (ARM) node types created by Cirro's data ingestion process and their properties.
<h2>Generic ARM Resource</h2>
Base properties inherited by all Azure Resource Manager resources.
**Labels:** `:ArmResource`
**Properties:**
- `id` - Resource ID (primary key, lowercase)
- `kind` - Resource kind
- `location` - Azure region/location
- `name` - Resource name
- `type` - Resource type
- `tags` - Resource tags as key:value pairs
**Relationships:**
- `HAS_IDENTITY` → GraphObject (for managed identities)
docs/analysis/arm-nodes/analysisservices/analysis-service-server.md
-41
View File
@@ -1,41 +0,0 @@
# AnalysisServiceServer
Represents Azure Analysis Services server resources.
**Labels:** `:ArmResource:AnalysisServiceServer`
**Properties:**
- `id` - Resource ID (primary key)
- `managedMode` - Indicates whether the server is managed
- `serverFullName` - Full server name
- `serverMonitorMode` - Monitoring mode for the server
- `state` - Current state of the server
- `administrators` - Raw administrator member entries from the resource
- `enablePowerBIService` - Whether Power BI service access is enabled
## Relationships
### Incoming
- **GraphServicePrincipal** → `IS_ADMIN` → **AnalysisServiceServer** - Service principal listed as administrator
- **GraphObject** → `IS_ADMIN` → **AnalysisServiceServer** - Directory object listed as administrator
- **GraphUser** → `IS_ADMIN` → **AnalysisServiceServer** - User listed as administrator
### Outgoing
- **AnalysisServiceServer** → `HAS_RULE` → **AnalysisServiceFirewallRule** - Firewall rules defined on the server
## Examples
```cypher
// Find Analysis Services servers and administrator principals
MATCH (admin)-[:IS_ADMIN]->(s:AnalysisServiceServer)
RETURN s.serverFullName, labels(admin), admin
```
```cypher
// Find Analysis Services servers and their firewall rules
MATCH (s:AnalysisServiceServer)-[:HAS_RULE]->(r:AnalysisServiceFirewallRule)
RETURN s.serverFullName, r.name, r.rangeStart, r.rangeEnd
```
Loaded 100 of 512 files, more files were not shown because too many files have changed in this diff. Show more