vatesfr/xen-orchestra
1
0
Fork 0
mirror of https://github.com/vatesfr/xen-orchestra.git synced 2026-10-11 22:50:01 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat(v2v): remove vddk dependency (#10421)

Browse Source 
Co-authored-by: Mathieu <[email protected]>
This commit is contained in:
Florent BEAUCHAMP
2026-09-25 10:38:26 +02:00
committed by GitHub
co-authored by Mathieu
parent 3fea79d899
commit dc57333f58
77 changed files with 15337 additions and 1186 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.github/workflows/vectura.yml
+57
View File
@@ -0,0 +1,57 @@
name: Vectura
# only when vectura changes: the rest of the repository does not build it
on:
push:
paths:
- '@xen-orchestra/vmware-explorer/vectura/**'
- .github/workflows/vectura.yml
defaults:
run:
working-directory: '@xen-orchestra/vmware-explorer/vectura'
jobs:
checks:
runs-on: ubuntu-latest
steps:
# https://github.com/actions/checkout
- uses: actions/checkout@v4
# rustup, on the runner image, installs the toolchain rust-toolchain.toml
# names on the first cargo command
- run: cargo fmt --all --check
- run: cargo clippy --all-targets --locked -- -D warnings
- run: cargo test --locked
# https://github.com/taiki-e/install-action
- uses: taiki-e/install-action@v2
with:
tool: [email protected],[email protected]
- run: cargo deny --locked check
- run: cargo llvm-cov --locked --json --summary-only --output-path target/coverage.json
- name: Hold the coverage floor
run: cargo test --locked --test coverage -- --ignored
# builds the binary as a release does, then checks it and the one in git,
# which xo-server runs
binary:
runs-on: ubuntu-latest
steps:
# https://github.com/actions/checkout
- uses: actions/checkout@v4
# https://github.com/mlugg/setup-zig
- uses: mlugg/setup-zig@v2
with:
version: 0.16.0
# https://github.com/taiki-e/install-action
- uses: taiki-e/install-action@v2
with:
tool: [email protected]
- run: scripts/build.sh "$RUNNER_TEMP/build"
- run: scripts/check-binary.sh "$RUNNER_TEMP/build/vectura"
- run: scripts/check-binary.sh vectura
@xen-orchestra/vmware-explorer/.USAGE.md
+2 -3
View File
@@ -1,5 +1,5 @@
Reads the inventory, the metadata and the disks of an ESXi host or a vCenter, over vim25 SOAP and
the `/folder` HTTP endpoint, and exports the content of a disk through an `nbdkit` server.
the `/folder` HTTP endpoint, and exports the content of a disk through a `vectura` server.
### Errors
@@ -25,8 +25,7 @@ which failed without the host naming a fault type reports `TASK_FAILED` instead.
| `DATACENTER_NOT_FOUND` | No datacenter holds the named datastore, and a file cannot be downloaded without one. Carries `dataStore` and the `dataStores` which are known. |
| `DATASTORE_NOT_FOUND` | An absolute disk reference of a vmx or a vmsd is on no known datastore. Carries `filePath` and `dataStoreUrls`. |
| `ESXI_SESSION_EXPIRED` | The session of the HTTP endpoint was refused (401 or 403), which happens on an import lasting hours. Retried after authenticating again. |
| `NBDKIT_EXITED` | The `nbdkit` server exited before it started listening, e.g. on a wrong thumbprint or a missing vddk library. Its logs are named in the message. |
| `NBDKIT_NOT_LISTENING` | The `nbdkit` server is running, but never started listening on its port. |
| `NBD_SERVER_EXITED` | The `vectura` server exited, e.g. on a wrong thumbprint or a disk the host refuses to open. Raised by `NbdStdioClient`, which carries the tail of its stderr on the error. |
| `NO_DATA_MAP` | The allocated ranges of a disk could not be read: an unreadable vmdk descriptor, or a delta in a format which is not supported. The disk has to be transferred in full instead of as a delta. |
| `NO_PROPERTY` | The host returned no value for the property which was read, e.g. because the object is gone. |
| `NO_TASK` | A `*_Task` method answered without a task to wait for. |
@xen-orchestra/vmware-explorer/README.md
+3 -4
View File
@@ -17,7 +17,7 @@ npm install --save @xen-orchestra/vmware-explorer
## Usage
Reads the inventory, the metadata and the disks of an ESXi host or a vCenter, over vim25 SOAP and
the `/folder` HTTP endpoint, and exports the content of a disk through an `nbdkit` server.
the `/folder` HTTP endpoint, and exports the content of a disk through a `vectura` server.
### Errors
@@ -43,15 +43,14 @@ which failed without the host naming a fault type reports `TASK_FAILED` instead.
| `DATACENTER_NOT_FOUND` | No datacenter holds the named datastore, and a file cannot be downloaded without one. Carries `dataStore` and the `dataStores` which are known. |
| `DATASTORE_NOT_FOUND` | An absolute disk reference of a vmx or a vmsd is on no known datastore. Carries `filePath` and `dataStoreUrls`. |
| `ESXI_SESSION_EXPIRED` | The session of the HTTP endpoint was refused (401 or 403), which happens on an import lasting hours. Retried after authenticating again. |
| `NBDKIT_EXITED` | The `nbdkit` server exited before it started listening, e.g. on a wrong thumbprint or a missing vddk library. Its logs are named in the message. |
| `NBDKIT_NOT_LISTENING` | The `nbdkit` server is running, but never started listening on its port. |
| `NBD_SERVER_EXITED` | The `vectura` server exited, e.g. on a wrong thumbprint or a disk the host refuses to open. Raised by `NbdStdioClient`, which carries the tail of its stderr on the error. |
| `NO_DATA_MAP` | The allocated ranges of a disk could not be read: an unreadable vmdk descriptor, or a delta in a format which is not supported. The disk has to be transferred in full instead of as a delta. |
| `NO_PROPERTY` | The host returned no value for the property which was read, e.g. because the object is gone. |
| `NO_TASK` | A `*_Task` method answered without a task to wait for. |
| `RANGE_IGNORED` | The host answered something else than the byte range which was requested. Reading it would use the wrong offset, or as much memory as the file is big. |
| `TASK_FAILED` | A task ended in error, and the host named no fault type for it. |
| `TASK_TIMEOUT` | A task did not complete within its deadline. It keeps running on the host, so it is **not** retried: starting a second one would not make the first go away. |
| `NO_CHANGE_ID` | Asked for changed block, but change block was not tracked when snapshotted |
| `NO_CHANGE_ID` | Asked for changed block, but change block was not tracked when snapshotted |
`ESXI_SESSION_EXPIRED` and the transport failures are transient, and are retried inside the package
— while downloading from a datastore, and while polling a task. Every other code is reported to the
@xen-orchestra/vmware-explorer/_nbdkit.integ.mjs
-55
View File
@@ -1,55 +0,0 @@
import assert from 'node:assert/strict'
import { createServer } from 'node:net'
import { describe, it } from 'node:test'
import { findFreePort, waitForPort } from './_nbdkit.mjs'
// these bind a real port
describe('findFreePort', function () {
it('returns a port which can be bound', async function () {
const port = await findFreePort()
assert.equal(typeof port, 'number')
const server = createServer()
await new Promise((resolve, reject) => {
server.once('error', reject)
server.listen(port, '127.0.0.1', resolve)
})
server.close()
})
})
describe('waitForPort', function () {
it('resolves as soon as the port accepts a connection', async function () {
const port = await findFreePort()
const server = createServer()
// start listening later, as nbdkit does once its plugin is configured
const timer = setTimeout(() => server.listen(port, '127.0.0.1'), 30)
try {
await waitForPort(port, { probeDelay: 5, timeout: 2e3 })
} finally {
clearTimeout(timer)
server.close()
}
})
it('gives up with the reason of the last attempt', async function () {
const port = await findFreePort()
await assert.rejects(waitForPort(port, { probeDelay: 1, timeout: 20 }), error => {
assert.match(error.message, /nothing is listening on 127\.0\.0\.1:/)
assert.equal(error.code, 'NBDKIT_NOT_LISTENING')
assert.equal(error.cause.code, 'ECONNREFUSED')
return true
})
})
it('stops probing when aborted', async function () {
const port = await findFreePort()
await assert.rejects(waitForPort(port, { probeDelay: 1, signal: AbortSignal.abort(), timeout: 1e3 }), {
name: 'AbortError',
})
})
})
@xen-orchestra/vmware-explorer/_nbdkit.mjs
-109
View File
@@ -1,109 +0,0 @@
import { connect, createServer } from 'node:net'
import { setTimeout as delay } from 'node:timers/promises'
/**
* Command line of an nbdkit server exporting one disk of a VM through the vddk plugin.
*
* @param {object} params
* @returns {Array<string>}
*/
export function formatNbdkitArgs({
compression,
diskPath,
host,
libdir,
passFile,
port,
singleLink,
threads,
thumbprint,
user,
vmId,
}) {
return [
'-r', // readonly
'-v',
'-f',
'--exit-with-parent', // implies -f , ensure we don't leave orphans
`--threads=${threads}`,
`--port=${port}`,
'vddk', // the vddk plugin
`compression=${compression}`,
`thumbprint=${thumbprint}`,
`server=${host}`,
`user=${user}`,
`password=+${passFile}`,
`libdir=${libdir}`,
`vm=moref=${vmId}`,
// `single-link` restricts the export to the top delta. An empty argument used to be passed
// when it is disabled: nbdkit tolerates it, but it sits in the slot of the magic positional
// parameter of the plugin, right before the disk to export
...(singleLink ? ['single-link=true'] : []),
diskPath,
]
}
/**
* Reserves a port by letting the system pick a free one.
*
* The port is released before being handed out, so the caller must be able to cope with a
* concurrent process taking it in the meantime.
*
* @returns {Promise<number>}
*/
export function findFreePort() {
return new Promise((resolve, reject) => {
const server = createServer()
server.once('error', reject)
server.listen(0, '127.0.0.1', () => {
const { port } = server.address()
server.close(error => (error != null ? reject(error) : resolve(port)))
})
})
}
const tryConnect = (port, host) =>
new Promise((resolve, reject) => {
const socket = connect({ port, host })
socket.once('connect', () => {
socket.destroy()
resolve()
})
socket.once('error', error => {
socket.destroy()
reject(error)
})
})
/**
* Waits until something listens on a port.
*
* nbdkit binds its port once its plugin is configured, so this is what tells that the server is
* usable — the previous implementation waited a fixed two seconds, which was both too long for a
* fast host and too short for a slow one.
*
* @param {number} port
* @param {object} [options]
* @param {string} [options.host]
* @param {number} [options.probeDelay] - in ms, between two attempts
* @param {AbortSignal} [options.signal]
* @param {number} [options.timeout] - in ms
* @returns {Promise<void>}
*/
export async function waitForPort(port, { host = '127.0.0.1', probeDelay = 50, signal, timeout = 30e3 } = {}) {
const start = Date.now()
for (;;) {
signal?.throwIfAborted()
try {
return await tryConnect(port, host)
} catch (error) {
if (Date.now() - start >= timeout) {
const wrapped = new Error(`nothing is listening on ${host}:${port} after ${timeout}ms`)
wrapped.code = 'NBDKIT_NOT_LISTENING'
wrapped.cause = error
throw wrapped
}
}
await delay(probeDelay, undefined, { signal })
}
}
@xen-orchestra/vmware-explorer/_nbdkit.test.mjs
-56
View File
@@ -1,56 +0,0 @@
import assert from 'node:assert/strict'
import { describe, it } from 'node:test'
import { formatNbdkitArgs } from './_nbdkit.mjs'
const PARAMS = {
compression: 'fastlz',
diskPath: '[ds main] a.vm/a.vm.vmdk',
host: 'esxi.test',
libdir: '/usr/local/lib/vddk/vmware-vix-disklib-distrib',
passFile: '/tmp/xo-server-42/params',
port: 11000,
singleLink: false,
threads: 1,
thumbprint: 'AA:BB:CC',
user: 'root',
vmId: 'vm-42',
}
describe('formatNbdkitArgs', function () {
it('never passes an empty argument', function () {
// an empty argument sits in the slot of the magic positional parameter of the plugin
for (const singleLink of [false, true]) {
const args = formatNbdkitArgs({ ...PARAMS, singleLink })
assert.equal(
args.every(argument => argument.length > 0),
true,
JSON.stringify(args)
)
}
})
it('exports read only, exits with its parent, and reads the password from a file', function () {
const args = formatNbdkitArgs(PARAMS)
assert.ok(args.includes('-r'))
assert.ok(args.includes('--exit-with-parent'))
assert.ok(args.includes(`password=+${PARAMS.passFile}`))
// the password itself never appears in the command line, which is world readable
assert.equal(
args.some(argument => argument.includes('password=') && !argument.startsWith('password=+')),
false
)
})
it('restricts the export to the top delta only when asked', function () {
assert.equal(formatNbdkitArgs(PARAMS).includes('single-link=true'), false)
assert.equal(formatNbdkitArgs({ ...PARAMS, singleLink: true }).includes('single-link=true'), true)
})
it('ends with the disk to export', function () {
const args = formatNbdkitArgs({ ...PARAMS, singleLink: true })
assert.equal(args[args.length - 1], PARAMS.diskPath)
})
})
@xen-orchestra/vmware-explorer/_thumbprint.integ.mjs
+19 -1
View File
@@ -2,11 +2,29 @@ import assert from 'node:assert/strict'
import { createServer } from 'node:net'
import { describe, it } from 'node:test'
import { findFreePort } from './_nbdkit.mjs'
import { getCertificateThumbprint } from './_thumbprint.mjs'
const noop = () => {}
/**
* Reserves a port by letting the system pick a free one.
*
* The port is released before being handed out, so a concurrent process could take it in the
* meantime — which is fine here, every test using it wants a port nothing answers on.
*
* @returns {Promise<number>}
*/
function findFreePort() {
return new Promise((resolve, reject) => {
const server = createServer()
server.once('error', reject)
server.listen(0, '127.0.0.1', () => {
const { port } = server.address()
server.close(error => (error != null ? reject(error) : resolve(port)))
})
})
}
// the successful path needs a certificate, so it is only exercised against a real host: what is
// covered here are the two failures which used to freeze or terminate the process
describe('getCertificateThumbprint', function () {
@xen-orchestra/vmware-explorer/_thumbprint.mjs
+8 -7
View File
@@ -2,13 +2,13 @@ import { connect } from 'node:tls'
import { isIP } from 'node:net'
/**
* SHA-1 fingerprint of the certificate presented by a TLS server, in the `AA:BB:...` form the
* vddk library expects.
* SHA-256 fingerprint of the certificate presented by a TLS server, in the `sha256:AA:BB:...` form
* vectura expects for `--thumbprint`.
*
* This used to shell out to `openssl s_client` and `openssl x509`, which left a temporary
* directory and a file descriptor behind on every call, had no timeout at all — a black holed port
* froze the whole spawn of the nbdkit server — and threw from inside a callback instead of
* rejecting, terminating the process.
* froze the whole start of an import — and threw from inside a callback instead of rejecting,
* terminating the process.
*
* @param {string} host
* @param {object} [options]
@@ -36,13 +36,14 @@ export function getCertificateThumbprint(host, { port = 443, timeout = 10e3 } =
}
socket.once('secureConnect', () => {
const { fingerprint } = socket.getPeerCertificate()
if (typeof fingerprint !== 'string' || fingerprint.length === 0) {
// `fingerprint` is the SHA-1 one, which is not what is pinned here
const { fingerprint256 } = socket.getPeerCertificate()
if (typeof fingerprint256 !== 'string' || fingerprint256.length === 0) {
// nothing to wait for on this path, drop the socket instead of half closing it
return fail(new Error(`the host ${host}:${port} did not present any certificate`))
}
socket.end()
resolve(fingerprint)
resolve(`sha256:${fingerprint256}`)
})
socket.once('timeout', () => {
fail(new Error(`no TLS handshake with ${host}:${port} after ${timeout}ms`))
@xen-orchestra/vmware-explorer/_vddk.mjs
-5
View File
@@ -1,5 +0,0 @@
// the vddk library is not redistributable: it has to be downloaded from Broadcom and extracted
// there by `esxi.installVddkLib`
export const VDDK_LIB_DIR = '/usr/local/lib/vddk'
export const VDDK_LIB_PATH = `${VDDK_LIB_DIR}/vmware-vix-disklib-distrib`
export const VDDK_LIB_FILE = `${VDDK_LIB_PATH}/lib64/libvixDiskLib.so`
@xen-orchestra/vmware-explorer/_vectura.mjs
+40
View File
@@ -0,0 +1,40 @@
import { fileURLToPath } from 'node:url'
// the vectura binary is small and redistributable: it ships with this package, next to its sources
// and its build process, so the binary xo-server runs is always the one it was tested against
export const VECTURA_BIN = fileURLToPath(new URL('vectura/vectura', import.meta.url))
/**
* Command line of a vectura server exporting one disk of a VM as NBD on its own standard streams.
*
* The password is not here on purpose: vectura reads it from `VECTURA_PASSWORD` only, which keeps it
* out of the command line of a process anyone can list.
*
* @param {object} params
* @param {string} [params.compression] - `skipz`, `zlib` or `none`, vectura defaults to `skipz`
* @param {number} [params.depth] - host reads kept in flight, 1 to 32, vectura defaults to 16
* @param {string} params.diskPath - `[datastore] dir/disk.vmdk`
* @param {string} params.host
* @param {string} params.thumbprint - `sha256:` and the digest of the management port certificate
* @param {string} params.user
* @param {string} params.vmId - managed object reference, such as `3` or `vm-42`
* @returns {Array<string>}
*/
export function formatVecturaArgs({ compression, depth, diskPath, host, thumbprint, user, vmId }) {
return [
'serve',
'--host',
host,
'--user',
user,
'--vm-id',
vmId,
'--disk',
diskPath,
'--thumbprint',
thumbprint,
// left out when undefined: vectura's own defaults are the ones we want
...(compression !== undefined ? ['--compression', compression] : []),
...(depth !== undefined ? ['--depth', String(depth)] : []),
]
}
@xen-orchestra/vmware-explorer/checks.mjs
+14 -98
View File
@@ -1,116 +1,32 @@
import { exec } from 'node:child_process'
import semver from 'semver'
import fs from 'node:fs/promises'
import { execFile } from 'node:child_process'
import { promisify } from 'node:util'
import { VDDK_LIB_DIR, VDDK_LIB_FILE } from './_vddk.mjs'
import { VECTURA_BIN } from './_vectura.mjs'
/**
* Whether the vectura binary shipped with this package runs on this machine.
*
* There is nothing to install: this only fails when the binary lost its executable bit or when the
* system is too old for it (glibc before 2.34).
*
* @returns {Promise<Object>}
*/
/* async */ function nbdInfo() {
return new Promise(function (resolve, reject) {
const expectedVersion = '1.23.4'
exec('nbdinfo --version', (error, stdout, stderr) => {
if (error) {
return resolve({
error: `exit code ${error.code}`,
status: 'error',
})
}
const matches = stdout.match(/nbdinfo ([0-9.]+)/)
const version = matches?.[1] ?? ''
resolve({
installed: true,
version,
status: semver.satisfies(version, `>=${expectedVersion}`) ? 'success' : 'alarm',
expectedVersion,
})
})
})
}
async function getNbdKitVersion() {
return new Promise(function (resolve) {
exec('nbdkit --version', (error, stdout, stderr) => {
if (error) {
return resolve({
error: `exit code ${error.code}`,
status: 'error',
})
}
const matches = stdout.match(/nbdkit ([0-9.]+)/)
const version = matches?.[1] ?? ''
resolve(version)
})
})
}
/**
*
* @returns {Promise<Object>}
*/
async function nbdKit() {
const expectedVersion = '1.42.5'
async function vectura() {
try {
const version = await getNbdKitVersion()
return {
installed: true,
version,
status: semver.satisfies(version, `>=${expectedVersion}`) ? 'success' : 'alarm',
expectedVersion,
}
await promisify(execFile)(VECTURA_BIN, ['--version'])
return { status: 'success' }
} catch (error) {
return {
error: `exit code ${error.code}`,
status: 'error',
}
return { error: `vectura is not runnable: ${error.message}`, status: 'error' }
}
}
/**
* Everything the import of a VM from an ESXi host needs on this machine.
*
* @returns {Promise<Object>}
*/
/* async */ function nbdKitVddk() {
return new Promise(function (resolve) {
exec('nbdkit --dump-plugin vddk', (error, stdout, stderr) => {
if (error) {
return resolve({
error: stderr,
status: 'error',
})
}
resolve({
status: 'success',
})
})
})
}
/**
*
* @returns {Promise<Object>}
*/
async function vddk() {
try {
await fs.stat(VDDK_LIB_FILE)
} catch (error) {
return {
status: 'error',
error: `Vddk library is not present or accessible in ${VDDK_LIB_DIR}/ it can be downloaded from https://developer.broadcom.com/sdks/vmware-virtual-disk-development-kit-vddk/latest`,
}
}
return { status: 'success' }
}
/**
*
* @returns {Promise<Object>}
*/
export async function checkVddkDependencies() {
export async function checkDependencies() {
return {
nbdinfo: await nbdInfo(),
nbdkit: await nbdKit(),
nbdkitPluginVddk: await nbdKitVddk(),
vddk: await vddk(),
vectura: await vectura(),
}
}
@xen-orchestra/vmware-explorer/esxi.integ.mjs
-174
View File
@@ -1,174 +0,0 @@
import assert from 'node:assert/strict'
import { createServer } from 'node:net'
import { describe, it } from 'node:test'
import { EventEmitter } from 'node:events'
import { PassThrough } from 'node:stream'
import { connectedEsxi } from './esxi.fixtures.mjs'
// these bind a real port and write real temporary files, unlike the rest of the suite
describe('nbdkit servers', function () {
/**
* Stands in for nbdkit: it listens on the port it is given, so that the readiness probe of the
* server under test is exercised for real.
*/
const fakeNbdkit = ({ exitWith, failWith } = {}) => {
const spawned = []
const spawn = (command, args) => {
const child = new EventEmitter()
child.exitCode = null
child.signalCode = null
child.stdout = new PassThrough()
child.stderr = new PassThrough()
child.kill = signal => {
child.signalCode = signal ?? 'SIGTERM'
child.exitCode = 0
entry.server?.close()
setImmediate(() => child.emit('exit', 0, child.signalCode))
return true
}
const port = Number(args.find(argument => argument.startsWith('--port=')).slice('--port='.length))
const entry = { args, child, command, port }
spawned.push(entry)
if (failWith !== undefined) {
setImmediate(() => child.emit('error', failWith))
} else if (exitWith !== undefined) {
// nbdkit exits on a bad thumbprint or a missing vddk library, without ever listening
setImmediate(() => {
child.exitCode = exitWith
child.emit('exit', exitWith, null)
})
} else {
entry.server = createServer()
entry.server.listen(port, '127.0.0.1')
}
return child
}
return { spawn, spawned }
}
const nbdkitEsxi = async options => {
const { spawn, spawned } = fakeNbdkit(options)
const { esxi } = await connectedEsxi({ spawn })
// the thumbprint is the only step of a spawn which needs the real host
esxi.getServerThumbprint = async () => 'AA:BB:CC'
return { esxi, spawned }
}
it('spawns one server per disk, and shares it', async function () {
const { esxi, spawned } = await nbdkitEsxi()
const [first, second] = await Promise.all([
esxi.getNbdServer('vm-1', '[ds] vm/vm.vmdk'),
esxi.getNbdServer('vm-1', '[ds] vm/vm.vmdk'),
])
// the promise is memoized: two concurrent calls used to spawn two servers, orphaning one
assert.equal(spawned.length, 1)
assert.equal(first.value, second.value)
const server = first.value
assert.equal(server.nbdInfos.exportname, '[ds] vm/vm.vmdk')
assert.equal(server.nbdInfos.port, spawned[0].port)
// the server belongs to both of them: the first to let go must not stop it
await first.dispose()
assert.equal(server.process.exitCode, null)
await second.dispose()
assert.equal(server.process.exitCode, 0)
await esxi.close()
})
it('spawns a new server after the previous one was killed', async function () {
const { esxi, spawned } = await nbdkitEsxi()
const first = await esxi.getNbdServer('vm-1', '[ds] vm/vm.vmdk')
const firstServer = first.value
await first.dispose()
const second = await esxi.getNbdServer('vm-1', '[ds] vm/vm.vmdk')
// the entry used to be left in the map, so this handed out the dead process of a closed port
assert.equal(spawned.length, 2)
assert.notEqual(firstServer.nbdInfos.port, second.value.nbdInfos.port)
assert.equal(firstServer.process.exitCode, 0)
await esxi.close()
})
it('forgets a server which died on its own', async function () {
const { esxi, spawned } = await nbdkitEsxi()
const first = await esxi.getNbdServer('vm-1', '[ds] vm/vm.vmdk')
spawned[0].server.close()
first.value.process.exitCode = 1
first.value.process.emit('exit', 1, null)
await first.value.died
await esxi.getNbdServer('vm-1', '[ds] vm/vm.vmdk')
assert.equal(spawned.length, 2)
await esxi.close()
})
it('reports a missing nbdkit instead of terminating the process', async function () {
const error = new Error('spawn nbdkit ENOENT')
error.code = 'ENOENT'
const { esxi, spawned } = await nbdkitEsxi({ failWith: error })
// without an 'error' listener on the child process, this used to be an uncaught event
await assert.rejects(esxi.getNbdServer('vm-1', '[ds] vm/vm.vmdk'), { code: 'ENOENT' })
// the failure is not memoized either
await assert.rejects(esxi.getNbdServer('vm-1', '[ds] vm/vm.vmdk'), { code: 'ENOENT' })
assert.equal(spawned.length, 2)
})
it('reports an nbdkit which exited before being ready', async function () {
const { esxi, spawned } = await nbdkitEsxi({ exitWith: 1 })
// waiting for the port would otherwise burn the whole readiness timeout
await assert.rejects(esxi.getNbdServer('vm-1', '[ds] vm/vm.vmdk'), {
code: 'NBDKIT_EXITED',
message: /^nbdkit exited with code 1 before being ready/,
})
assert.equal(spawned.length, 1)
await esxi.close()
})
it('never passes the password on the command line', async function () {
const { esxi, spawned } = await nbdkitEsxi()
await esxi.getNbdServer('vm-1', '[ds] vm/vm.vmdk')
assert.equal(
spawned[0].args.some(argument => argument.includes('password')),
true
)
assert.equal(
spawned[0].args.some(argument => argument.includes('password=password')),
false
)
await esxi.close()
})
it('kills the remaining servers when closing', async function () {
const { esxi, spawned } = await nbdkitEsxi()
// never disposed: closing is the safety net for a caller which leaked its disposable
const disposable = await esxi.getNbdServer('vm-1', '[ds] vm/vm.vmdk')
const server = disposable.value
await esxi.close()
assert.equal(server.process.exitCode, 0)
assert.equal(spawned.length, 1)
// disposing after the close has nothing left to do, and must not throw
await disposable.dispose()
})
})
@xen-orchestra/vmware-explorer/esxi.mjs
+43 -266
View File
@@ -1,15 +1,13 @@
import { asyncEach } from '@vates/async-each'
import { createLogger } from '@xen-orchestra/log'
import { dirname, join } from 'node:path'
import { dirname } from 'node:path'
import { EventEmitter } from 'node:events'
import { Disposable, pTimeout } from 'promise-toolbox'
import { setTimeout as delay } from 'node:timers/promises'
import { Agent } from 'undici'
import { findFreePort, formatNbdkitArgs, waitForPort } from './_nbdkit.mjs'
import { resolveDiskLocation } from './_paths.mjs'
import { getCertificateThumbprint } from './_thumbprint.mjs'
import { VDDK_LIB_PATH } from './_vddk.mjs'
import { formatVecturaArgs, VECTURA_BIN } from './_vectura.mjs'
import { COWD_HEADER_LENGTH, grainDirectoryToDataMap, parseCowdHeader } from './parsers/cowd.mjs'
import parseVmdk from './parsers/vmdk.mjs'
import parseVmsd from './parsers/vmsd.mjs'
@@ -25,16 +23,9 @@ import {
traversalSpec,
} from './soap/specs.mjs'
import { VimClient } from './soap/VimClient.mjs'
import { spawn } from 'node:child_process'
import NbdClient from '@vates/nbd-client'
import { tmpdir } from 'node:os'
import fs from 'node:fs/promises'
const { info, warn } = createLogger('xo:vmware-explorer:esxi')
export { VDDK_LIB_DIR, VDDK_LIB_PATH } from './_vddk.mjs'
const DEFAULT_DOWNLOAD_RETRIES = 4
// a vmdk descriptor is a text file of a few hundred bytes, but a long parentFileNameHint pushes
// its extent line further
@@ -52,9 +43,6 @@ const DEFAULT_TASK_TIMEOUT = 60e3
const MAX_CBT_QUERIES = 1024
const MAX_RETRY_DELAY = 30e3
const MAX_TASK_POLL_DELAY = 5e3
const NBDKIT_KILL_TIMEOUT = 10e3
// connecting the vddk library to the host can be slow
const NBDKIT_READY_TIMEOUT = 60e3
// a failure which will not fix itself must not be retried: a missing file, a rejected
// authentication or a programming error only delay the report of the real problem
@@ -89,12 +77,6 @@ function isRetryableError(error) {
return error?.name === 'TimeoutError'
}
const noop = () => {}
// the options change what the server exports, so they are part of its identity
const nbdServerKey = (vmId, diskPath, { compression, singleLink, threads }) =>
JSON.stringify([vmId, diskPath, singleLink, threads, compression])
/**
* Everything the change tracking of a VM is addressed by, as {@link Esxi#getChangeTracking} reads
* it. Opaque to its caller, which reads it once and hands it back to {@link Esxi#getDataMap}.
@@ -148,10 +130,8 @@ export default class Esxi extends EventEmitter {
#httpsAgent
#user
#password
#spawn
#thumbprint
#vimClient
#nbdServers = new Map()
/**
* @param {string} host
@@ -160,19 +140,11 @@ export default class Esxi extends EventEmitter {
* @param {boolean} sslVerify
* @param {object} [options]
* @param {typeof globalThis.fetch} [options.fetch] - injectable fetch implementation, for tests
* @param {typeof spawn} [options.spawn] - injectable process spawner, for tests
* @param {object} [options.vimClient] - injectable SOAP client, for tests
*/
constructor(
host,
user,
password,
sslVerify,
{ fetch: fetchImplementation, spawn: spawnImplementation, vimClient } = {}
) {
constructor(host, user, password, sslVerify, { fetch: fetchImplementation, vimClient } = {}) {
super()
this.#fetchImpl = fetchImplementation ?? globalThis.fetch
this.#spawn = spawnImplementation ?? spawn
this.#host = host.trim()
this.#user = user
this.#password = password
@@ -217,20 +189,10 @@ export default class Esxi extends EventEmitter {
* @returns {Promise<void>}
*/
async close() {
try {
// the servers would only be reaped by `--exit-with-parent`, i.e. when this process ends
await asyncEach([...this.#nbdServers.keys()], key => this.#killNbdServerByKey(key), {
concurrency: 4,
// every server must be tried: `asyncEach` stops on the first error by default, which would
// leave the other ones running
stopOnError: false,
})
} finally {
// a server refusing to die must not leave the session open on the host, nor the sockets of
// the agent
await this.#vimClient.close()
await this.#httpsAgent?.close()
}
// the vectura processes are owned by the `NbdStdioClient` reading through them, and stopped
// when it disconnects: there is nothing to reap here
await this.#vimClient.close()
await this.#httpsAgent?.close()
}
async #computeDatacenters() {
@@ -956,10 +918,10 @@ export default class Esxi extends EventEmitter {
}
/**
* SHA-1 fingerprint of the certificate of the host, as the vddk library expects it.
* SHA-256 fingerprint of the certificate of the host, as vectura's `--thumbprint` expects it.
*
* Memoized: it used to be computed again for every nbdkit server, with two openssl processes
* every time.
* Memoized: it used to be computed again for every disk export, with two openssl processes every
* time.
*
* @returns {Promise<string>}
*/
@@ -978,188 +940,40 @@ export default class Esxi extends EventEmitter {
}
/**
* Starts an nbdkit server exporting a disk of a VM, or shares the one already serving it.
* Settings of an `NbdStdioClient` reading one disk of a VM through vectura.
*
* The server is stopped when the disposable is disposed, and a server shared by several callers
* survives until the last of them disposes: the caller no longer has to name it again, with the
* exact same options, to stop it.
* Nothing is started here: `vectura serve` speaks NBD on its own standard streams, so one process
* serves exactly one client, and the client is what spawns it, kills it on `disconnect()` and
* restarts it when a read is retried. The caller passes these settings, and `NbdStdioClient` as
* the `ClientClass`, to `NbdDisk`.
*
* @param {string} vmId
* @param {string} diskPath - `[datastore] dir/disk.vmdk`
* @param {object} [options]
* @param {string} [options.compression]
* @param {boolean} [options.singleLink] - export the top delta only
* @param {number} [options.threads]
* @returns {Promise<Disposable<{ died: Promise<object>, nbdInfos: object, process: object }>>}
* @param {string} [options.compression] - `skipz`, `zlib` or `none`, vectura defaults to `skipz`
* @param {number} [options.depth] - host reads in flight, 1 to 32, vectura defaults to 16
* @returns {Promise<{ args: Array<string>, command: string, env: object, exportname: string }>}
*/
async getNbdServer(vmId, diskPath, { compression = 'fastlz', singleLink = false, threads = 1 } = {}) {
const key = nbdServerKey(vmId, diskPath, { compression, singleLink, threads })
let entry = this.#nbdServers.get(key)
if (entry === undefined) {
// the promise is memoized, not its result: the previous implementation had six await points
// between the check and the registration, so two concurrent calls spawned two servers and
// orphaned the first one
entry = { pending: this.#spawnNbdKitProcess(vmId, diskPath, { compression, singleLink, threads }), users: 0 }
this.#nbdServers.set(key, entry)
// neither a failed spawn nor a dead server must be handed out to the next caller
const forget = () => {
if (this.#nbdServers.get(key) === entry) {
this.#nbdServers.delete(key)
}
}
entry.pending.then(server => server.died.then(forget), forget)
}
entry.users += 1
let server
try {
server = await entry.pending
} catch (error) {
entry.users -= 1
throw error
}
// `Disposable` refuses a second disposal itself, so this runs exactly once per caller
const dispose = async () => {
entry.users -= 1
// `close()` stops every server left, a disposal after it has nothing to do
if (entry.users === 0 && this.#nbdServers.get(key) === entry) {
await this.#killNbdServerByKey(key)
}
}
return new Disposable(dispose, server)
}
async #spawnNbdKitProcess(vmId, diskPath, { compression, singleLink, threads }) {
const thumbprint = await this.getServerThumbprint()
const port = await findFreePort()
const tmpDir = await fs.mkdtemp(join(tmpdir(), 'xo-server'))
const passFile = join(tmpDir, 'params')
const outFd = await fs.open(join(tmpDir, 'stdout'), 'a')
const errFd = await fs.open(join(tmpDir, 'stderr'), 'a')
// the file holds a password, and only the directory was protecting it
await fs.writeFile(passFile, this.#password, { mode: 0o600 })
const args = formatNbdkitArgs({
compression,
diskPath,
host: this.#host,
libdir: VDDK_LIB_PATH,
passFile,
port,
singleLink,
threads,
thumbprint,
user: this.#user,
vmId,
})
const nbdKitProcess = this.#spawn('nbdkit', args, {
cwd: tmpDir,
env: {
...process.env,
LD_LIBRARY_PATH: `${VDDK_LIB_PATH}/lib64`,
},
})
nbdKitProcess.stdout.pipe(outFd.createWriteStream())
nbdKitProcess.stderr.pipe(errFd.createWriteStream())
info(`nbdkit logs of ${diskPath} are in ${tmpDir}`)
// `error` is emitted when the binary is missing: without a listener, it is an uncaught event
// which terminates the whole process
const died = new Promise(resolve => {
nbdKitProcess.once('error', error => resolve({ error }))
nbdKitProcess.once('exit', (code, signal) => resolve({ code, signal }))
})
died.then(async ({ code, error, signal }) => {
await Promise.all([outFd.close().catch(noop), errFd.close().catch(noop)])
if (error !== undefined) {
warn('nbdkit could not be started', { args, error, tmpDir })
} else if (code !== 0) {
warn(`nbdkit server process exited with code ${code} ,detailed logs are in ${tmpDir}/stderr `, { signal })
} else {
// nothing to look at, the logs would pile up in the temporary directory
await fs.rm(tmpDir, { force: true, recursive: true }).catch(noop)
}
})
// the readiness of the server and its death are racing: nbdkit exits on a bad thumbprint or a
// missing library, and waiting for the port would then burn the whole timeout
const failed = died.then(({ code, error }) => {
if (error !== undefined) {
// the spawn itself failed, and its own code already says why, e.g. `ENOENT` when nbdkit is
// not installed
throw error
}
const exited = new Error(
`nbdkit exited with code ${code} before being ready, detailed logs are in ${tmpDir}/stderr`
)
exited.code = 'NBDKIT_EXITED'
throw exited
})
failed.catch(noop) // the race is usually won by the readiness of the server
const readiness = new AbortController()
try {
await Promise.race([waitForPort(port, { signal: readiness.signal, timeout: NBDKIT_READY_TIMEOUT }), failed])
} catch (error) {
await this.#killNbdServer({ died, process: nbdKitProcess }, diskPath).catch(noop)
throw error
} finally {
// losing the race must not leave a probe running until its own timeout
readiness.abort()
// nbdkit reads the password once, while configuring its plugin, which is done by the time it
// listens
await fs.unlink(passFile).catch(error => warn('failed to remove the password file', { error, passFile }))
}
async getVecturaSpawnSettings(vmId, diskPath, { compression, depth } = {}) {
return {
died,
nbdInfos: { address: '127.0.0.1', port, exportname: diskPath },
process: nbdKitProcess,
command: VECTURA_BIN,
args: formatVecturaArgs({
compression,
depth,
diskPath,
host: this.#host,
thumbprint: await this.getServerThumbprint(),
user: this.#user,
vmId,
}),
// vectura reads the password from there and nowhere else: never a flag, never a file, so it
// cannot be read from the command line of the process
env: { ...process.env, VECTURA_PASSWORD: this.#password },
// vectura serves a single unnamed export, the disk is already named by `--disk`
exportname: '',
}
}
async #killNbdServer(server, label) {
const { died, process: nbdKitProcess } = server
if (nbdKitProcess.exitCode !== null || nbdKitProcess.signalCode !== null) {
return
}
nbdKitProcess.kill()
try {
await pTimeout.call(died, NBDKIT_KILL_TIMEOUT)
} catch (error) {
warn('nbdkit did not exit, killing it', { error, label, pid: nbdKitProcess.pid })
nbdKitProcess.kill('SIGKILL')
await died
}
}
async #killNbdServerByKey(key) {
const entry = this.#nbdServers.get(key)
if (entry === undefined) {
warn(`nbdkit server ${key} was already killed`)
return
}
// the entry used to be left in place, so the next spawn handed out a dead process listening on
// nothing
this.#nbdServers.delete(key)
let server
try {
server = await entry.pending
} catch {
// the spawn failed, there is nothing left to kill
return
}
await this.#killNbdServer(server, key)
}
/**
* Runs one `QueryChangedDiskAreas`.
*
@@ -1326,8 +1140,8 @@ export default class Esxi extends EventEmitter {
/**
* Blocks of a disk to read, from the change tracking of the host.
*
* Needs no nbdkit server, no vddk library and no nbdinfo: one call answers from the `-ctk.vmdk`
* the host maintains. Two questions, the same call:
* Needs nothing installed and no disk export: one call answers from the `-ctk.vmdk` the host
* maintains. Two questions, the same call:
* - without `baseDiskPath`, every block the disk has ever used, across the whole chain
* - with it, only the blocks written since the disk a previous import already read
*
@@ -1432,37 +1246,6 @@ export default class Esxi extends EventEmitter {
return areas
}
async #getDataMapFromVddk(vmId, datastoreName, diskPath, signal) {
return Disposable.use(
this.getNbdServer(vmId, `[${datastoreName}] ${diskPath}`, { singleLink: true }),
async ({ nbdInfos }) => {
const start = Date.now()
info('nbd server for data map spawned')
signal?.throwIfAborted()
// the client is built here, and not before the server: a spawn which failed used to leave
// it undefined, and the `TypeError` of `nbdClient.disconnect()` replaced the real error,
// feeding the fallback with a misleading cause
const nbdClient = new NbdClient(nbdInfos)
try {
await nbdClient.connect()
info('nbd client for data map connected')
const dataMap = await nbdClient.getMap(signal)
info(
`got the data map of the single disk in ${Math.round((Date.now() - start) / 1000)} seconds ,${dataMap.length} blocks`
)
return dataMap
} finally {
await nbdClient.disconnect().catch(error => warn('error while disconnecting the nbd client', { error }))
}
}
)
}
async #readRange(datastoreName, path, start, length, signal) {
// an HTTP range is inclusive: asking for `0-512` reads 513 bytes
const res = await this.download(datastoreName, path, { range: `${start}-${start + length - 1}`, signal })
@@ -1502,8 +1285,9 @@ export default class Esxi extends EventEmitter {
* Blocks of a disk which hold data, used to read only what the disk uses.
*
* The change tracking of the host answers both questions and is asked first: it is a documented
* call every backup product has leaned on for years, where the map read through the vddk needs
* an nbdkit server, the vddk library and nbdinfo, all version-coupled.
* call every backup product has leaned on for years. When it refuses, the metadata of the delta
* is read straight from the datastore, which is the only other source left — vectura serves the
* disk and nothing else, there is no block map to ask it for.
*
* @param {string} vmId
* @param {string} datastoreName
@@ -1530,19 +1314,12 @@ export default class Esxi extends EventEmitter {
}
if (baseDiskPath === undefined) {
// the other two sources describe a single link of the chain, which is not the disk being
// read: handing one of them over as the map of a whole disk would drop everything the
// parents hold. Reading the disk is the only other answer
// the metadata of a delta describes a single link of the chain, which is not the disk being
// read: handing it over as the map of a whole disk would drop everything the parents hold.
// Reading the disk is the only other answer
return undefined
}
try {
// We await the result of getDataMapFromVddk so we can catch errors and fallback to the direct metadata reading.
return await this.#getDataMapFromVddk(vmId, datastoreName, diskPath, signal)
} catch (error) {
signal?.throwIfAborted()
warn('error while getting datamap from vddk, fall back to a direct metadata reading', { error })
return this.#getDataMapFromCowd(datastoreName, diskPath, signal)
}
return this.#getDataMapFromCowd(datastoreName, diskPath, signal)
}
}
@xen-orchestra/vmware-explorer/esxi.test.mjs
+61 -14
View File
@@ -4,6 +4,7 @@ import { once } from 'node:events'
import { connectedEsxi, DATACENTERS, FakeVimClient, moRef, page, response } from './esxi.fixtures.mjs'
import Esxi from './esxi.mjs'
import { VECTURA_BIN } from './_vectura.mjs'
describe('search', function () {
it('destroys the container view it created', async function () {
@@ -779,12 +780,6 @@ describe('keepAlive', function () {
describe('getDataMap', function () {
const BLOCK_LENGTH = 4096 * 512
// the thumbprint is the first step of a spawn: failing it sends the code down the fallback
// without binding a port nor writing a temporary file
const failingThumbprint = async () => {
throw new Error('no vddk on this machine')
}
const DELTA_DESCRIPTOR = `# Disk DescriptorFile
version=1
CID=d7980f7a
@@ -827,7 +822,6 @@ RW 16384 VMFSSPARSE "vm-000001-delta.vmdk"
return response({ status: 206, body: DELTA_DESCRIPTOR })
},
})
esxi.getServerThumbprint = failingThumbprint
const dataMap = await esxi.getDataMap('vm-1', 'ds main', 'a.vm/vm-000001.vmdk', {
baseDiskPath: 'a.vm/vm.vmdk',
@@ -854,14 +848,13 @@ RW 16384 VMFSSPARSE "vm-000001-delta.vmdk"
fetch: async url =>
response({ status: 206, body: url.pathname.endsWith('-delta.vmdk') ? header : DELTA_DESCRIPTOR }),
})
esxi.getServerThumbprint = failingThumbprint
await assert.rejects(esxi.getDataMap('vm-1', 'ds main', 'a.vm/vm-000001.vmdk', { baseDiskPath: 'a.vm/vm.vmdk' }), {
code: 'NO_DATA_MAP',
})
})
// the change tracking of the host answers a delta without an nbdkit server nor the vddk
// the change tracking of the host answers a delta without exporting the disk at all
describe('from the change tracking', function () {
const CAPACITY = 16 * 1024 * 1024 * 1024
const ACTIVE = '[ds main] a.vm/vm-000001.vmdk'
@@ -938,8 +931,7 @@ RW 16384 VMFSSPARSE "vm-000001-delta.vmdk"
// fallback which reaches the network would answer with a DNS failure a few seconds later
fetch: async () => response({ status: 206, body: 'not a vmdk descriptor' }),
})
// the vddk and the metadata reading must never be reached by a successful CBT query
esxi.getServerThumbprint = failingThumbprint
// the metadata reading must never be reached by a successful CBT query
return { esxi, vimClient }
}
@@ -1032,7 +1024,7 @@ RW 16384 VMFSSPARSE "vm-000001-delta.vmdk"
it('falls back when no snapshot recorded a changeId for the disk of the previous import', async function () {
const { esxi, vimClient } = await cbtEsxi({ changeId: null })
// the vddk then the metadata reading, which has nothing to read from the fake responses
// the metadata reading, which has nothing to read from the fake responses
await assert.rejects(
esxi.getDataMap('vm-1', 'ds main', 'a.vm/vm-000001.vmdk', { baseDiskPath: 'a.vm/vm.vmdk' }),
{ code: 'NO_DATA_MAP' }
@@ -1072,9 +1064,8 @@ RW 16384 VMFSSPARSE "vm-000001-delta.vmdk"
return response({ status: 206, body: 'not a vmdk descriptor' })
},
})
esxi.getServerThumbprint = failingThumbprint
// the vddk and the metadata reading both describe a single link of the chain: handing one
// the metadata reading describes a single link of the chain: handing one
// over as the map of a whole disk would drop everything its parents hold
assert.equal(await esxi.getDataMap('vm-1', 'ds main', 'a.vm/vm-000001.vmdk'), undefined)
assert.deepEqual(fetched, [])
@@ -1137,3 +1128,59 @@ RW 16384 VMFSSPARSE "vm-000001-delta.vmdk"
})
})
})
describe('getVecturaSpawnSettings', function () {
const vecturaEsxi = async () => {
const { esxi } = await connectedEsxi()
// the only step needing the real host
esxi.getServerThumbprint = async () => 'sha256:AA:BB:CC'
return esxi
}
it('builds the vectura command line of one disk', async function () {
const esxi = await vecturaEsxi()
const { args, command, exportname } = await esxi.getVecturaSpawnSettings('1', '[ds main] vm/vm.vmdk')
assert.equal(command, VECTURA_BIN)
assert.deepEqual(args, [
'serve',
'--host',
'esxi.test',
'--user',
'user',
'--vm-id',
'1',
'--disk',
'[ds main] vm/vm.vmdk',
'--thumbprint',
'sha256:AA:BB:CC',
])
// vectura serves a single unnamed export
assert.equal(exportname, '')
})
it('passes the password through the environment and never on the command line', async function () {
const esxi = await vecturaEsxi()
const { args, env } = await esxi.getVecturaSpawnSettings('1', '[ds main] vm/vm.vmdk')
assert.equal(env.VECTURA_PASSWORD, 'password')
assert.equal(
args.some(argument => argument.includes('password')),
false
)
})
it('leaves compression and depth out unless they are asked for', async function () {
const esxi = await vecturaEsxi()
const { args } = await esxi.getVecturaSpawnSettings('1', '[ds main] vm/vm.vmdk', { compression: 'none', depth: 4 })
assert.deepEqual(args.slice(-4), ['--compression', 'none', '--depth', '4'])
const { args: bare } = await esxi.getVecturaSpawnSettings('1', '[ds main] vm/vm.vmdk')
assert.equal(bare.includes('--compression'), false)
assert.equal(bare.includes('--depth'), false)
})
})
@xen-orchestra/vmware-explorer/package.json
-2
View File
@@ -6,11 +6,9 @@
"description": "Reads the VMs and the disks of an ESXi host or a vCenter",
"dependencies": {
"@vates/async-each": "^1.0.3",
"@vates/nbd-client": "^3.5.1",
"@vates/node-vsphere-soap": "^2.1.3",
"@xen-orchestra/log": "^0.7.2",
"promise-toolbox": "^0.21.0",
"semver": "^7.7.2",
"undici": "^7.8.0"
},
"engines": {
@xen-orchestra/vmware-explorer/vectura/.gitignore
+17
View File
@@ -0,0 +1,17 @@
# Cargo
/target/
**/*.rs.bk
# Editors and operating systems
.DS_Store
Thumbs.db
*.swp
.idea/
.vscode/*
!.vscode/extensions.json
# Local environment files are never committed. VECTURA_PASSWORD lives in the
# shell that runs the command, not in a file next to the source.
.env
.env.*
!.env.example
@xen-orchestra/vmware-explorer/vectura/Cargo.lock
Generated
+952
View File
@@ -0,0 +1,952 @@
# This file is automatically @generated by Cargo.
# It is not intended for manual editing.
version = 4
[[package]]
name = "adler2"
version = "2.0.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "320119579fcad9c21884f5c4861d16174d0e06250625266f50fe6898340abefa"
[[package]]
name = "anstream"
version = "1.0.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "824a212faf96e9acacdbd09febd34438f8f711fb84e09a8916013cd7815ca28d"
dependencies = [
"anstyle",
"anstyle-parse",
"anstyle-query",
"anstyle-wincon",
"colorchoice",
"is_terminal_polyfill",
"utf8parse",
]
[[package]]
name = "anstyle"
version = "1.0.14"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "940b3a0ca603d1eade50a4846a2afffd5ef57a9feac2c0e2ec2e14f9ead76000"
[[package]]
name = "anstyle-parse"
version = "1.0.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "52ce7f38b242319f7cabaa6813055467063ecdc9d355bbb4ce0c68908cd8130e"
dependencies = [
"utf8parse",
]
[[package]]
name = "anstyle-query"
version = "1.1.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "40c48f72fd53cd289104fc64099abca73db4166ad86ea0b4341abe65af83dadc"
dependencies = [
"windows-sys 0.61.2",
]
[[package]]
name = "anstyle-wincon"
version = "3.0.11"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "291e6a250ff86cd4a820112fb8898808a366d8f9f58ce16d1f538353ad55747d"
dependencies = [
"anstyle",
"once_cell_polyfill",
"windows-sys 0.61.2",
]
[[package]]
name = "asn1-rs"
version = "0.7.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b7f43a50ac4fdca5df8e885c21b835997f0a1cdee65494a6847694a98652d9d8"
dependencies = [
"asn1-rs-derive",
"asn1-rs-impl",
"displaydoc",
"nom",
"num-traits",
"rusticata-macros",
"thiserror",
"time",
]
[[package]]
name = "asn1-rs-derive"
version = "0.6.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3109e49b1e4909e9db6515a30c633684d68cdeaa252f215214cb4fa1a5bfee2c"
dependencies = [
"proc-macro2",
"quote",
"syn 2.0.119",
"synstructure",
]
[[package]]
name = "asn1-rs-impl"
version = "0.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7b18050c2cd6fe86c3a76584ef5e0baf286d038cda203eb6223df2cc413565f7"
dependencies = [
"proc-macro2",
"quote",
"syn 2.0.119",
]
[[package]]
name = "autocfg"
version = "1.5.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f2032f911046de80f0a198e0901378627c33f59ea0ac00e363d481118bd70a53"
[[package]]
name = "aws-lc-rs"
version = "1.18.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b281d307588d634de920874890732659e2e7672f72b5e10e81badc1a8a83621e"
dependencies = [
"aws-lc-sys",
"untrusted 0.7.1",
"zeroize",
]
[[package]]
name = "aws-lc-sys"
version = "0.45.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9bff6c3b54fad79a2e60b8102caf565819711497c1f5f092f49508e2f5c31b27"
dependencies = [
"cc",
"cmake",
"dunce",
"fs_extra",
"pkg-config",
]
[[package]]
name = "bit-vec"
version = "0.9.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b71798fca2c1fe1086445a7258a4bc81e6e49dcd24c8d0dd9a1e57395b603f51"
dependencies = [
"serde",
]
[[package]]
name = "bitflags"
version = "2.13.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b588b76d00fde79687d7646a9b5bdf3cc0f655e0bbd080335a95d7e96f3587da"
[[package]]
name = "cc"
version = "1.4.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0ad534f4357a5264cce5019c989cf66a4f0dc4e0d1b1d15f8aacec0ff7360273"
dependencies = [
"find-msvc-tools",
"jobserver",
"libc",
"shlex",
]
[[package]]
name = "cfg-if"
version = "1.0.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801"
[[package]]
name = "clap"
version = "4.6.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "473c7e07f409a8d772161724aa8db6a765a2532a70f9667eeb7b49d3d02fbdca"
dependencies = [
"clap_builder",
"clap_derive",
]
[[package]]
name = "clap_builder"
version = "4.6.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7b48fea5a88e9ae728a2dcbedbfc0e730f7d60da42e1cb049a83c9fb8b789889"
dependencies = [
"anstream",
"anstyle",
"clap_lex",
"strsim",
]
[[package]]
name = "clap_derive"
version = "4.6.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d012d2b9d65aca7f18f4d9878a045bc17899bba951561ba5ec3c2ba1eed9a061"
dependencies = [
"heck",
"proc-macro2",
"quote",
"syn 3.0.4",
]
[[package]]
name = "clap_lex"
version = "1.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c8d4a3bb8b1e0c1050499d1815f5ab16d04f0959b233085fb31653fbfc9d98f9"
[[package]]
name = "cmake"
version = "0.1.58"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c0f78a02292a74a88ac736019ab962ece0bc380e3f977bf72e376c5d78ff0678"
dependencies = [
"cc",
]
[[package]]
name = "colorchoice"
version = "1.0.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1d07550c9036bf2ae0c684c4297d503f838287c83c53686d05370d0e139ae570"
[[package]]
name = "core-foundation"
version = "0.10.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b2a6cd9ae233e7f62ba4e9353e81a88df7fc8a5987b8d445b4d90c879bd156f6"
dependencies = [
"core-foundation-sys",
"libc",
]
[[package]]
name = "core-foundation-sys"
version = "0.8.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "773648b94d0e5d620f64f280777445740e61fe701025087ec8b57f45c791888b"
[[package]]
name = "crc32fast"
version = "1.5.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8498c871161e1742aaa9d52551b2d6ebdd4c3d45a3be423e3728f33b955be550"
dependencies = [
"cfg-if",
]
[[package]]
name = "data-encoding"
version = "2.11.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4583a4551df46e2792f82ceeac45e850d2e2d5debba0b91f102385cda5b11f06"
[[package]]
name = "der-parser"
version = "10.0.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "07da5016415d5a3c4dd39b11ed26f915f52fc4e0dc197d87908bc916e51bc1a6"
dependencies = [
"asn1-rs",
"displaydoc",
"nom",
"num-bigint",
"num-traits",
"rusticata-macros",
]
[[package]]
name = "deranged"
version = "0.5.8"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7cd812cc2bc1d69d4764bd80df88b4317eaef9e773c75226407d9bc0876b211c"
[[package]]
name = "displaydoc"
version = "0.2.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c6232dd377dcc64799954cbd3a9bb882e9cdc1308ccd87b1c098f1fb2eaf82a8"
dependencies = [
"proc-macro2",
"quote",
"syn 3.0.4",
]
[[package]]
name = "dunce"
version = "1.0.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "92773504d58c093f6de2459af4af33faa518c13451eb8f2b5698ed3d36e7c813"
[[package]]
name = "find-msvc-tools"
version = "0.1.11"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d45db016d36b838f563236e9193d0ee6ce38f3f68b6c94e914b4929c96bbb890"
[[package]]
name = "flate2"
version = "1.1.10"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6e634e2e0ebac1ee034020da1ca582e17ffe4e0f5e985823721e168928136dcb"
dependencies = [
"crc32fast",
"miniz_oxide",
]
[[package]]
name = "fs_extra"
version = "1.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "42703706b716c37f96a77aea830392ad231f44c9e9a67872fa5548707e11b11c"
[[package]]
name = "getrandom"
version = "0.2.17"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ff2abc00be7fca6ebc474524697ae276ad847ad0a6b3faa4bcb027e9a4614ad0"
dependencies = [
"cfg-if",
"libc",
"wasi",
]
[[package]]
name = "getrandom"
version = "0.4.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "300e883d756b2e4ec94e02791f39b04b522276138852cfc41d9fb7e904106099"
dependencies = [
"cfg-if",
"libc",
"r-efi",
]
[[package]]
name = "heck"
version = "0.5.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2304e00983f87ffb38b55b444b5e3b60a884b5d30c0fca7d82fe33449bbe55ea"
[[package]]
name = "is_terminal_polyfill"
version = "1.70.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a6cb138bb79a146c1bd460005623e142ef0181e3d0219cb493e02f7d08a35695"
[[package]]
name = "itoa"
version = "1.0.18"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682"
[[package]]
name = "jobserver"
version = "0.1.35"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1c00acbd29eabad4a2392fa0e921c874934dbbf4194312ad20f04a0ed67a3cb3"
dependencies = [
"getrandom 0.4.3",
"libc",
]
[[package]]
name = "lazy_static"
version = "1.5.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "bbd2bcb4c963f2ddae06a2efc7e9f3591312473c50c6685e1f298068316e66fe"
[[package]]
name = "libc"
version = "0.2.189"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3eaf3ede3fee6db1a4c2ee091bf8a8b4dccdc6d17f656fb07896ee72867612f2"
[[package]]
name = "log"
version = "0.4.34"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f9f8bd3e56ce4dfc153cf470fffbfa98c7620958b312ca5c3a4b8d5181fd13c6"
[[package]]
name = "memchr"
version = "2.8.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "cf8baf1c55e62ffcace7a9f06f4bd9cd3f0c4beb022d3b367256b91b87513d98"
[[package]]
name = "minimal-lexical"
version = "0.2.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "68354c5c6bd36d73ff3feceb05efa59b6acb7626617f4962be322a825e61f79a"
[[package]]
name = "miniz_oxide"
version = "0.9.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b63fbc4a50860e98e7b2aa7804ded1db5cbc3aff9193adaff57a6931bf7c4b4c"
dependencies = [
"adler2",
"simd-adler32",
]
[[package]]
name = "nom"
version = "7.1.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d273983c5a657a70a3e8f2a01329822f3b8c8172b73826411a55751e404a0a4a"
dependencies = [
"memchr",
"minimal-lexical",
]
[[package]]
name = "num-bigint"
version = "0.4.8"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c89e69e7e0f03bea5ef08013795c25018e101932225a656383bd384495ecc367"
dependencies = [
"num-integer",
"num-traits",
]
[[package]]
name = "num-conv"
version = "0.2.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "521739c6d2bac4aa25192232afe6841231376b2b26d4d9fae5ecf8ca5772e441"
[[package]]
name = "num-integer"
version = "0.1.47"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7ce2d95d4b3734dc35aa2f45e1aa22cd416814592a4f9d9205e11affd5b8e10b"
dependencies = [
"num-traits",
]
[[package]]
name = "num-traits"
version = "0.2.19"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "071dfc062690e90b734c0b2273ce72ad0ffa95f0c74596bc250dcfd960262841"
dependencies = [
"autocfg",
]
[[package]]
name = "oid-registry"
version = "0.8.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "12f40cff3dde1b6087cc5d5f5d4d65712f34016a03ed60e9c08dcc392736b5b7"
dependencies = [
"asn1-rs",
]
[[package]]
name = "once_cell"
version = "1.21.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9f7c3e4beb33f85d45ae3e3a1792185706c8e16d043238c593331cc7cd313b50"
[[package]]
name = "once_cell_polyfill"
version = "1.70.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "384b8ab6d37215f3c5301a95a4accb5d64aa607f1fcb26a11b5303878451b4fe"
[[package]]
name = "openssl-probe"
version = "0.2.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7c87def4c32ab89d880effc9e097653c8da5d6ef28e6b539d313baaacfbafcbe"
[[package]]
name = "pkg-config"
version = "0.3.34"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f6b464fbc74e149a392436b17d523f769e057cb6877f6a5c4618bc6f11800548"
[[package]]
name = "powerfmt"
version = "0.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "439ee305def115ba05938db6eb1644ff94165c5ab5e9420d1c1bcedbba909391"
[[package]]
name = "proc-macro2"
version = "1.0.107"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "985e7ec9bb745e6ce6535b544d84d6cd6f7ad8bd711c398938ae983b91a766d9"
dependencies = [
"unicode-ident",
]
[[package]]
name = "quick-xml"
version = "0.42.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "41b1177fdf999d2321d3fb46ff47159d9c1fb9ad66a4879f8c50a0b504615e9b"
dependencies = [
"memchr",
]
[[package]]
name = "quote"
version = "1.0.47"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1fbf4db142a473a8d80c26bbf18454ed458bf8d26c8219c331daecfdbd079001"
dependencies = [
"proc-macro2",
]
[[package]]
name = "r-efi"
version = "6.0.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f8dcc9c7d52a811697d2151c701e0d08956f92b0e24136cf4cf27b57a6a0d9bf"
[[package]]
name = "rcgen"
version = "0.14.10"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8774e05a7d0de114588e6a28fe7e71694b82614ed569d86d8b389dfbc98b8ad8"
dependencies = [
"aws-lc-rs",
"rustls-pki-types",
"time",
"x509-parser",
"yasna",
]
[[package]]
name = "ring"
version = "0.17.14"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a4689e6c2294d81e88dc6261c768b63bc4fcdb852be6d1352498b114f61383b7"
dependencies = [
"cc",
"cfg-if",
"getrandom 0.2.17",
"libc",
"untrusted 0.9.0",
"windows-sys 0.52.0",
]
[[package]]
name = "rusticata-macros"
version = "4.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "faf0c4a6ece9950b9abdb62b1cfcf2a68b3b67a10ba445b3bb85be2a293d0632"
dependencies = [
"nom",
]
[[package]]
name = "rustls"
version = "0.23.45"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0d41d731c7d2f962d1ccc364cec258de3c0e93b38c2fb3ba97ac74513048d634"
dependencies = [
"aws-lc-rs",
"log",
"once_cell",
"rustls-pki-types",
"rustls-webpki",
"subtle",
"zeroize",
]
[[package]]
name = "rustls-native-certs"
version = "0.8.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "dab5152771c58876a2146916e53e35057e1a4dfa2b9df0f0305b07f611fdea4d"
dependencies = [
"openssl-probe",
"rustls-pki-types",
"schannel",
"security-framework",
]
[[package]]
name = "rustls-pki-types"
version = "1.15.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2f4925028c7eb5d1fcdaf196971378ed9d2c1c4efc7dc5d011256f76c99c0a96"
dependencies = [
"zeroize",
]
[[package]]
name = "rustls-webpki"
version = "0.103.15"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f3c3cf1d8b1e7d4927e2d154c3fcb02979afb9939629c62cd9048d4f07b60ac2"
dependencies = [
"aws-lc-rs",
"ring",
"rustls-pki-types",
"untrusted 0.9.0",
]
[[package]]
name = "schannel"
version = "0.1.29"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "91c1b7e4904c873ef0710c1f407dde2e6287de2bebc1bbbf7d430bb7cbffd939"
dependencies = [
"windows-sys 0.61.2",
]
[[package]]
name = "security-framework"
version = "3.7.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b7f4bc775c73d9a02cde8bf7b2ec4c9d12743edf609006c7facc23998404cd1d"
dependencies = [
"bitflags",
"core-foundation",
"core-foundation-sys",
"libc",
"security-framework-sys",
]
[[package]]
name = "security-framework-sys"
version = "2.17.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6ce2691df843ecc5d231c0b14ece2acc3efb62c0a398c7e1d875f3983ce020e3"
dependencies = [
"core-foundation-sys",
"libc",
]
[[package]]
name = "serde"
version = "1.0.229"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4148590afebada386688f18773da617792bf2ef03ffc1e4cbd2b1d45b023e0ba"
dependencies = [
"serde_core",
"serde_derive",
]
[[package]]
name = "serde_core"
version = "1.0.229"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "67dca2c9c51e58a4791a4b1ed58308b39c64224d349a935ab5039aa360942a48"
dependencies = [
"serde_derive",
]
[[package]]
name = "serde_derive"
version = "1.0.229"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e7a5d71263a5a7d47b41f6b3f06ba276f10cc18b0931f1799f710578e2309348"
dependencies = [
"proc-macro2",
"quote",
"syn 3.0.4",
]
[[package]]
name = "serde_json"
version = "1.0.151"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c841b55ecdae098c80dcae9cf767f6f8a0c2cdb3416bbef72181df4d0fe73f14"
dependencies = [
"itoa",
"memchr",
"serde",
"serde_core",
"zmij",
]
[[package]]
name = "shlex"
version = "2.0.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f8fadd59c855ef2080decdef8ff161eb6661b86933c9d82e5ba29dc602a55aba"
[[package]]
name = "simd-adler32"
version = "0.3.10"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3a219298ac11a56ea9a6d2120044824d6f01aeb034955e7af7bc16858527deea"
[[package]]
name = "strsim"
version = "0.11.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7da8b5736845d9f2fcb837ea5d9e2628564b3b043a70948a3f0b778838c5fb4f"
[[package]]
name = "subtle"
version = "2.6.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "13c2bddecc57b384dee18652358fb23172facb8a2c51ccc10d74c157bdea3292"
[[package]]
name = "syn"
version = "2.0.119"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "872831b642d1a07999a962a351ed35b955ea2cfc8f3862091e2a240a84f17297"
dependencies = [
"proc-macro2",
"quote",
"unicode-ident",
]
[[package]]
name = "syn"
version = "3.0.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e6275cddf4610d1775e6d1fe9469b2e77d0f39fd98fb7450901b821e0c53649f"
dependencies = [
"proc-macro2",
"quote",
"unicode-ident",
]
[[package]]
name = "synstructure"
version = "0.13.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "728a70f3dbaf5bab7f0c4b1ac8d7ae5ea60a4b5549c8a5914361c99147a709d2"
dependencies = [
"proc-macro2",
"quote",
"syn 2.0.119",
]
[[package]]
name = "thiserror"
version = "2.0.20"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ec86235f5fcc2a73650310756d2ac5b138a5780bbbdfae3eeccec992c435ba4f"
dependencies = [
"thiserror-impl",
]
[[package]]
name = "thiserror-impl"
version = "2.0.20"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "bc04cd3e1236dd4a98afca4569f2deb3f120e5422a4023be2cb683f8486292af"
dependencies = [
"proc-macro2",
"quote",
"syn 3.0.4",
]
[[package]]
name = "time"
version = "0.3.55"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "cdb87b95ec50ddfa440816d227a17b2ccbdda963a316a727fda0fc4334f7d134"
dependencies = [
"deranged",
"num-conv",
"powerfmt",
"serde_core",
"time-core",
"time-macros",
]
[[package]]
name = "time-core"
version = "0.1.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9e1c906769ad99c88eaa54e728060edef082f8e358ff32030cb7c7d315e81109"
[[package]]
name = "time-macros"
version = "0.2.32"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7e689342a48d2ea927c87ea50cabf8594854bf940e9310208848d680d668ed85"
dependencies = [
"num-conv",
"time-core",
]
[[package]]
name = "unicode-ident"
version = "1.0.24"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e6e4313cd5fcd3dad5cafa179702e2b244f760991f45397d14d4ebf38247da75"
[[package]]
name = "untrusted"
version = "0.7.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a156c684c91ea7d62626509bce3cb4e1d9ed5c4d978f7b4352658f96a4c26b4a"
[[package]]
name = "untrusted"
version = "0.9.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8ecb6da28b8a351d773b68d5825ac39017e680750f980f3a1a85cd8dd28a47c1"
[[package]]
name = "utf8parse"
version = "0.2.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "06abde3611657adf66d383f00b093d7faecc7fa57071cce2578660c9f1010821"
[[package]]
name = "vectura"
version = "0.1.1"
dependencies = [
"aws-lc-rs",
"clap",
"flate2",
"quick-xml",
"rcgen",
"rustls",
"rustls-native-certs",
"rustls-pki-types",
"serde_json",
"thiserror",
]
[[package]]
name = "wasi"
version = "0.11.1+wasi-snapshot-preview1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ccf3ec651a847eb01de73ccad15eb7d99f80485de043efb2f370cd654f4ea44b"
[[package]]
name = "windows-link"
version = "0.2.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f0805222e57f7521d6a62e36fa9163bc891acd422f971defe97d64e70d0a4fe5"
[[package]]
name = "windows-sys"
version = "0.52.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "282be5f36a8ce781fad8c8ae18fa3f9beff57ec1b52cb3de0789201425d9a33d"
dependencies = [
"windows-targets",
]
[[package]]
name = "windows-sys"
version = "0.61.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ae137229bcbd6cdf0f7b80a31df61766145077ddf49416a728b02cb3921ff3fc"
dependencies = [
"windows-link",
]
[[package]]
name = "windows-targets"
version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9b724f72796e036ab90c1021d4780d4d3d648aca59e491e6b98e725b84e99973"
dependencies = [
"windows_aarch64_gnullvm",
"windows_aarch64_msvc",
"windows_i686_gnu",
"windows_i686_gnullvm",
"windows_i686_msvc",
"windows_x86_64_gnu",
"windows_x86_64_gnullvm",
"windows_x86_64_msvc",
]
[[package]]
name = "windows_aarch64_gnullvm"
version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "32a4622180e7a0ec044bb555404c800bc9fd9ec262ec147edd5989ccd0c02cd3"
[[package]]
name = "windows_aarch64_msvc"
version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "09ec2a7bb152e2252b53fa7803150007879548bc709c039df7627cabbd05d469"
[[package]]
name = "windows_i686_gnu"
version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8e9b5ad5ab802e97eb8e295ac6720e509ee4c243f69d781394014ebfe8bbfa0b"
[[package]]
name = "windows_i686_gnullvm"
version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0eee52d38c090b3caa76c563b86c3a4bd71ef1a819287c19d586d7334ae8ed66"
[[package]]
name = "windows_i686_msvc"
version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "240948bc05c5e7c6dabba28bf89d89ffce3e303022809e73deaefe4f6ec56c66"
[[package]]
name = "windows_x86_64_gnu"
version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "147a5c80aabfbf0c7d901cb5895d1de30ef2907eb21fbbab29ca94c5b08b1a78"
[[package]]
name = "windows_x86_64_gnullvm"
version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "24d5b23dc417412679681396f2b49f3de8c1473deb516bd34410872eff51ed0d"
[[package]]
name = "windows_x86_64_msvc"
version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "589f6da84c646204747d1270a2a5661ea66ed1cced2631d546fdfb155959f9ec"
[[package]]
name = "x509-parser"
version = "0.18.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d43b0f71ce057da06bc0851b23ee24f3f86190b07203dd8f567d0b706a185202"
dependencies = [
"asn1-rs",
"aws-lc-rs",
"data-encoding",
"der-parser",
"lazy_static",
"nom",
"oid-registry",
"rusticata-macros",
"thiserror",
"time",
]
[[package]]
name = "yasna"
version = "0.6.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b5f6765e852b9b4dc8e2a76843e4d64d1cea8e79bcde0b6901aea8e7c7f08282"
dependencies = [
"bit-vec",
"time",
]
[[package]]
name = "zeroize"
version = "1.9.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e13c156562582aa81c60cb29407084cdb54c4164760106ab78e6c5b0858cf64e"
[[package]]
name = "zmij"
version = "1.0.23"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "29666d0abbfad1e3dc4dcf6144730dd3a3ab225bbbdac83319345b1b44ccfc1b"
@xen-orchestra/vmware-explorer/vectura/Cargo.toml
+55
View File
@@ -0,0 +1,55 @@
[package]
name = "vectura"
version = "0.1.1"
edition = "2024"
description = "Standalone client for VMware's NFC disk-read protocol"
# The license of the Xen Orchestra application components, see LICENSE.md at
# the root of the repository. The crate stays unpublished, so deny.toml keeps
# skipping it: its allow list says what a dependency may bring into the
# binary, and AGPL is deliberately not on it.
license = "AGPL-3.0-or-later"
publish = false
# No `rust-version`: the toolchain pin lives in rust-toolchain.toml alone.
[dependencies]
aws-lc-rs = "1.18.1"
clap = { version = "4.6.6", features = ["derive"] }
flate2 = { version = "1.1.10", default-features = false, features = ["rust_backend"] }
quick-xml = "0.42.0"
rustls = "0.23"
rustls-native-certs = "0.8.4"
rustls-pki-types = "1.15.1"
thiserror = "2.0.20"
[dev-dependencies]
rcgen = { version = "0.14.10", default-features = false, features = ["aws_lc_rs"] }
serde_json = "1.0.151"
[profile.release]
# The binary is committed and shipped in the npm package: no symbols.
strip = true
[lints.rust]
# The whole crate is safe Rust. `forbid` cannot be overridden by an attribute
# lower down, unlike `deny`, which is the point.
unsafe_code = "forbid"
missing_docs = "warn"
missing_debug_implementations = "warn"
redundant_imports = "warn"
redundant_lifetimes = "warn"
trivial_numeric_casts = "warn"
unused_lifetimes = "warn"
[lints.clippy]
# Every warning is an error in CI through
# `cargo clippy -- -D warnings`; the groups stay at `warn` here so that a
# local `cargo build` still compiles while a fix is in progress.
all = { level = "warn", priority = -1 }
pedantic = { level = "warn", priority = -1 }
# Individual lints from the `restriction` group; each one names a construct
# the project does not use.
allow_attributes_without_reason = "warn"
clone_on_ref_ptr = "warn"
map_err_ignore = "warn"
undocumented_unsafe_blocks = "warn"
unused_result_ok = "warn"
@xen-orchestra/vmware-explorer/vectura/README.md
+109
View File
@@ -0,0 +1,109 @@
# Vectura
Vectura reads VMware virtual disks over the NFC protocol. It is one Linux
binary with two commands:
- `vectura serve` logs in to an ESXi host, opens one virtual disk and serves
it read-only as an NBD export on its own standard input and output. The
program that spawned it reads any range of the disk, in any order, and
ends the session when it is done.
- `vectura fingerprint` prints the certificate thumbprints a host presents
on its management and data ports, so an operator can pin them before any
credential is sent.
Vectura targets ESXi 7.x and 8.x hosts, reached directly or through a
vCenter; with a vCenter, the disk is read from the ESXi host its ticket
names. An ESXi 6.x host is refused at the data port's greeting. Certificate
verification can be pinned or pointed at a CA bundle but never skipped, and
the password is read from `VECTURA_PASSWORD` only.
## Install
The binary, `vectura`, ships in this directory with
`@xen-orchestra/vmware-explorer`, and xo-server runs it from there. It runs
on Debian 12, Ubuntu 22.04 or newer (amd64) and needs nothing but glibc. To
use it by hand, copy it to a directory in `PATH`:
```sh
sudo install vectura /usr/local/bin/
```
To build from source, install the
`build-essential` package (the TLS library compiles C) and let `rustup`
pick the pinned toolchain from `rust-toolchain.toml`:
```sh
cargo build --release --locked
```
## Try it
```sh
vectura fingerprint --host esxi.example
read -rs VECTURA_PASSWORD && export VECTURA_PASSWORD
vectura serve --host esxi.example --user root --vm-id 3 \
--disk '[datastore1] vm/vm.vmdk' \
--thumbprint sha256:<the 443 sha256 line printed above>
```
After the last command, standard output is an NBD stream and standard error
is the only place a message for a person appears. An NBD client that can
talk to a child process over its pipes, such as libnbd's `connect_command`,
reads the disk from there. The tutorial below walks through it.
## Documentation
Tutorial, for the first run:
- [Read your first disk](docs/tutorials/read-your-first-disk.md)
How-to guides, one task each:
- [Pin a host certificate](docs/how-to/pin-a-host-certificate.md)
- [Copy a disk to a raw image](docs/how-to/copy-a-disk-to-a-raw-image.md)
- [Drive Vectura from a program](docs/how-to/drive-vectura-from-a-program.md)
- [Tune depth and compression](docs/how-to/tune-depth-and-compression.md)
- [Troubleshoot a session](docs/how-to/troubleshoot-a-session.md)
Reference, the facts:
- [Command line](docs/reference/command-line.md)
- [The NBD export](docs/reference/nbd-export.md)
- [The transcript](docs/reference/transcript.md)
- [Host requirements and limits](docs/reference/host-requirements-and-limits.md)
Explanation, the reasoning:
- [How a disk is read](docs/explanation/how-a-disk-is-read.md)
- [The security model](docs/explanation/security-model.md)
- [Scope and design choices](docs/explanation/scope-and-design-choices.md)
## Development and release
`.github/workflows/vectura.yml` runs `cargo fmt`, `cargo clippy`, the tests,
`cargo deny`, the coverage floor and the binary build on every push that
touches this directory. The pre-commit hook runs `cargo fmt --check` when a
`.rs` file is staged.
A change to vectura needs its own line in the packages list of
`CHANGELOG.unreleased.md`:
```md
- vectura patch
```
`scripts/gen-deps-list.js` then lists `./scripts/bump-pkg vectura <type>`,
followed by a release of `@xen-orchestra/vmware-explorer`, which bundles the
binary. `bump-pkg` runs `scripts/release.sh`: it bumps the version in
`Cargo.toml` and `Cargo.lock`, rebuilds the binary in place of the previous
one and commits both as `feat(vectura): <version>`.
`tests/bundled_binary.rs` fails when the binary and `Cargo.toml` disagree, or
when the binary lost its executable bit.
The build links against glibc 2.34 through zig, whatever the glibc of the
machine, so the binary runs on Debian 12 and Ubuntu 22.04. It needs
[zig](https://ziglang.org/download/) (0.16.0 tested) and:
```sh
cargo install --locked [email protected]
```
@xen-orchestra/vmware-explorer/vectura/clippy.toml
+4
View File
@@ -0,0 +1,4 @@
# Clippy configuration. `doc_markdown` (pedantic) asks for backticks around
# any CamelCase word in a doc comment; these are the product names the docs
# use as prose, not identifiers. `..` keeps Clippy's own default list.
doc-valid-idents = ["VMware", "ESXi", "vCenter", "vSphere", "OpenSSL", ".."]
@xen-orchestra/vmware-explorer/vectura/deny.toml
+79
View File
@@ -0,0 +1,79 @@
# cargo-deny policy. `cargo deny --locked check` runs in CI (Vectura workflow)
# and locally after `cargo install cargo-deny`.
# Every key states why it holds its value, so a change here is a reviewed
# decision rather than a knob turned to make a red run green.
[graph]
# Resolve the graph for the platforms Vectura is built for, not for whichever
# host runs the check, so a platform-only dependency is still audited.
targets = [
"x86_64-unknown-linux-gnu",
"aarch64-unknown-linux-gnu",
"x86_64-apple-darwin",
"aarch64-apple-darwin",
"x86_64-pc-windows-msvc",
]
# Audit optional dependencies too: a feature nobody enables today is enabled
# by someone tomorrow.
all-features = true
[advisories]
# Empty on purpose. An advisory is waived here only with its RUSTSEC id, the
# reason and a date, and the pull request says why.
ignore = []
# A yanked crate was pulled by its author for a reason; do not build on it.
yanked = "deny"
# Unmaintained or unsound dependencies are reported for the crates this
# workspace depends on directly or transitively, which is all of them here.
unmaintained = "workspace"
unsound = "workspace"
[licenses]
# What may enter the binary. OpenSSL is on the list for aws-lc-sys, which the
# default rustls crypto provider builds on; nothing else needs it.
allow = [
"0BSD",
"Apache-2.0",
"Apache-2.0 WITH LLVM-exception",
"BSD-2-Clause",
"BSD-3-Clause",
"CC0-1.0",
"ISC",
"MIT",
"MIT-0",
"MPL-2.0",
"OpenSSL",
"Unicode-3.0",
"Unlicense",
"Zlib",
]
# How sure the license-text classifier must be before a detected license is
# accepted; below this the crate is reported instead of guessed.
confidence-threshold = 0.8
# Per-crate exceptions to the allow list; empty, and each one added must say
# why in a comment.
exceptions = []
# Development dependencies ship in nobody's binary, but they run on every
# contributor machine and in CI, so their licenses are checked too.
include-dev = true
[licenses.private]
# Vectura is AGPL-3.0-or-later and `publish = false`. Skip the crate itself:
# the allow list above says what a dependency may bring into the binary, and
# AGPL stays off it on purpose.
ignore = true
[bans]
# Two versions of one crate double the audit surface; reported, not fatal,
# because a dependency bump often carries a duplicate for one release.
multiple-versions = "warn"
# A `*` requirement accepts whatever the registry serves next; refuse it.
wildcards = "deny"
[sources]
# Only crates.io. A git or alternative-registry dependency bypasses the audit
# and advisory data everything above relies on.
unknown-registry = "deny"
unknown-git = "deny"
allow-registry = ["https://github.com/rust-lang/crates.io-index"]
allow-git = []
@xen-orchestra/vmware-explorer/vectura/docs/explanation/how-a-disk-is-read.md
+119
View File
@@ -0,0 +1,119 @@
# How a disk is read
This page follows one `vectura serve` run from the first packet to the
last, and says why each step is there. It is background; the reference
pages hold the exact values.
## Two planes
An ESXi host exposes a virtual disk through two ports with two jobs.
Port 443 is the management plane. It speaks SOAP over HTTPS, the same
API vSphere clients use. Vectura makes four calls on it, hand-rolled
HTTP/1.1 with a small XML body each:
1. `RetrieveServiceContent`, to find the session manager. An ESXi host
and a vCenter answer alike.
2. `Login`, with the user name and the password from `VECTURA_PASSWORD`.
The host sets a session cookie.
3. `RetrieveInternalContent`, to reach the NFC service.
4. `NfcGetVmFiles`, with the virtual machine's managed object reference.
The host answers with a ticket: an opaque string, the port to use, and
the SHA-1 thumbprint of the certificate that port presents. A vCenter
also names the ESXi host that holds the virtual machine, and the data
port is opened on that host; an ESXi host leaves the name out, and the
data port is its own.
At the end, `Logout` closes the session. Between the two, the management
session is idle; the disk goes through the other port.
Port 902 is the data plane. The host greets in clear with a `220` line that lists its capabilities,
among them `NFCSSL supported` and `SHA256 supported`. Vectura reads the
list and refuses a host that lacks what it needs, before it sends
anything. Then it starts TLS.
## Ticket, thumbprint, and a second TLS session
Inside that first TLS session, a short line dialogue turns the ticket
into an open channel. The client names its session with the ticket, asks
for the banner, and asks the host to confirm the SHA-256 of the
certificate it is presenting. The host's answer must match the
certificate the TLS layer saw, and the SHA-1 from the ticket must match
it too. Then the client asks for `nfcssl`, and the host answers that the
channel is ready.
`nfcssl` means a second TLS handshake, on the same TCP connection, inside
the first session. It is how the hosts expect the disk bytes to be
carried. Vectura checks that the second
session presents the same certificate as the first. With
`--transport nfc`, the second handshake is skipped and the records below
travel in clear after the dialogue; Vectura warns before doing so.
## Handshake and records
The disk session opens with six fixed-size messages of 264 bytes: the
client sends a tag, an end-of-tag and a version, the host acknowledges
and answers with its version, then the client sends its name, its mode
and a ready message, and the host answers ready. Vectura calls itself
`vectura` and asks for the mode in which the host serves arbitrary
ranges of a disk on demand.
From then on, both sides exchange records: a 16-byte header with a magic
number, an opcode and a length, followed by a payload. The client says
hello, states its protocol version and capabilities, and opens the disk
by its datastore path. The host's open reply gives the disk's virtual
capacity and sector size; that capacity becomes the NBD export size. A
close and an end record close the session at the other end of the run.
## Reads, chunks, and the window
An NBD client asks for any range up to 32 MiB. The host, however, answers
a read of at most 1 MiB, and it answers it as a series of chunks of
64 KiB, each carrying its offset within the read and the compression the
host chose for it: as stored, deflated with zlib, or with its runs of
zeros left out, which Vectura calls `skipz` and asks for by default. The
host is free to answer with a stored chunk whatever was asked, and it
does when compressing would not shrink the chunk. Every chunk is checked
before it is placed: its offset must be inside the read, it must not
overlap a chunk already received, and once decoded it must be exactly
64 KiB or the tail of the read. A chunk that fails is a broken read, and
a broken read ends the run rather than serve wrong bytes.
So `serve` splits each NBD command into host reads of 1 MiB, numbers
them, and keeps a window of them in flight, 16 by default and 32 at most.
The window is what hides the round trip. When
a read's last chunk lands, it is placed at the right offset of the NBD
command's buffer, and when a command's last read lands, the command is
answered. Commands can complete out of order, which is what the NBD
cookie is for.
## Two threads, one channel
`serve` runs on two threads. One reads NBD requests from standard input
and pushes them into a bounded channel. The other owns the host
connection and standard output: it pulls commands, issues host reads,
receives records, places chunks, and writes replies. Nothing else touches
standard output, which is why nothing else may print: a stray line there
would corrupt the NBD stream.
The bounded channel is the back pressure. When the host side is busy, the
input side waits with its next command in hand, and the client sees its
replies slow down instead of its requests vanishing into a queue.
## Ending
A client that sends `NBD_CMD_DISC` gets a clean end: the disk is closed,
the end record sent, the management session logged out, exit code 0. A
client that closes its side without disconnecting gets the same cleanup
and an exit code of 1, because something went wrong on its side and the
process that spawned it should know. A failure on the host's side is
reported as `NBD_EIO` to every command still in flight, then the same
cleanup, then exit code 1. There is no reconnect: a run is one session,
and a session that breaks is a new run.
## Related
- [The NBD export](../reference/nbd-export.md) and
[the transcript](../reference/transcript.md) give the exact values.
- [The security model](security-model.md) says what is verified at each
step.
@xen-orchestra/vmware-explorer/vectura/docs/explanation/scope-and-design-choices.md
+108
View File
@@ -0,0 +1,108 @@
# Scope and design choices
Vectura is small on purpose. This page lists what it does not do, why,
and which of the choices are principles rather than gaps that a later
version fills.
## Why a client written from public sources
Vectura depends on no proprietary library. A Linux package carries the
whole of it, at a version the packager chose, with nothing for the
operator to download by hand.
The client side of the protocol is written from public sources only: the
vSphere API reference, Broadcom knowledge base articles, and
measurements taken on ESXi 7 and 8 hosts. No proprietary binary or
header is opened, and no third-party reimplementation is read. It is
slower to write; it is the only way the result can be shipped.
## Why ESXi 7 and 8, and nothing else
- **vCenter as a relay.** The disk's files live on the ESXi host that
holds the virtual machine. A vCenter answers the management calls and
hands out a ticket that names that host; Vectura then opens the data
port on the named host, not on the vCenter. The management plane is the
same either way, so a vCenter needs no second kind of session.
- **No ESXi 6.x.** Those hosts do not announce `SHA256 supported`, so the
data port could not be confirmed by anything better than SHA-1, and
their default TLS configuration is what a 2026 client should not
negotiate. Both lines of ESXi 6 are past their end of general support.
- **Capabilities, not versions.** What a host can do is read from the
greeting line on port 902, not from a version string. A host that
announces the needed capabilities is served; one that does not is
refused with the list of what it announced.
## Why standard input and output
The consumer Vectura is built for is a program that spawns it: one
process per disk, for as long as the disk is being read. A TCP listener
would need a port, a bind address, a way to hand the socket over and a
way to make sure nobody else connects to it. Two pipes need none of that,
they are closed when the parent dies, and libnbd already knows how to
negotiate over them. Shell users who want a socket put `socat` in front,
as the copy guide shows.
The same reasoning gives one connection per process. The export does not
advertise multiple connections, because the process has exactly one
host session and one client, and a second client would need a second
ticket anyway.
## Why read-only
The use is backup, replication and inspection, which read. Writing to a
virtual disk that a hypervisor owns is a different product with a
different failure mode, and leaving the write path out means it cannot
be reached by mistake, by flag or by bug.
## Why no reconnect
A session that breaks, on the host or on the network, ends the run with
`NBD_EIO` to what was in flight and exit code 1. Reconnecting inside the
process would mean a second login, a second ticket and a decision about
which of the reads in flight to trust, all hidden from the caller. The
caller has the exit code and can start a new process, with the reads it
knows it still needs. What has been measured so far, an open disk that
survives half an hour idle and a management session that lasts at least
thirty minutes, makes the ordinary run not need one.
## Why rustls, synchronous I/O and two threads
- **rustls with aws-lc-rs, no OpenSSL.** One TLS stack, in the binary,
with no system library to match at run time and no shell out. ESXi 7
and 8 hosts negotiate TLS 1.2 with ECDHE and AES-GCM on both ports,
which rustls speaks.
- **Synchronous `std` I/O, no async runtime.** A run is two sockets and
two pipes. Two threads joined by a bounded channel cover that with
back pressure for free and nothing to schedule. An async runtime would
add a dependency, a build-time and a class of bugs for a concurrency
the problem does not have.
- **A depth window instead of parallel connections.** The host answers a
read of at most 1 MiB, so throughput on a long link is a matter of
keeping enough of them in flight. A window of 16, adjustable up to 32,
does that on one connection and keeps the host's per-host budget
shared fairly between processes.
## Why `skipz` by default
Three ways to pack a chunk were measured on the same disk and link.
Leaving out runs of zeros costs the host almost nothing and cut the bytes
on the wire to a sixth; zlib halved that again and was slower end to end,
because the host deflates slower than the link carries. `skipz` is the
default; `zlib` is there for a link slow enough to make it pay, and
`none` for diagnosis.
## Why the password comes only from the environment
A flag shows in the process table for the life of the process. A file is
left behind and has permissions to get wrong. A prompt cannot be answered
by the program that spawns Vectura. The environment is inherited from the
parent, visible to the parent's owner only, and gone with the process.
Nothing else is accepted, so nothing else can be misused.
## What may change
Reconnection within a run, a wider host set once a public source covers
it, and LAN throughput figures are all open. What will not change without
a written decision is the list above: no way to skip certificate
verification, no password outside the environment, no write path, no
proprietary dependency.
@xen-orchestra/vmware-explorer/vectura/docs/explanation/security-model.md
+113
View File
@@ -0,0 +1,113 @@
# The security model
Vectura carries a password, a ticket and a whole disk across a network.
This page says what each of them is protected by, what an attacker in a
given position could do, and which choices were made on purpose so that
an operator cannot weaken them by accident.
## What is at stake
- **The password.** It logs in on the host as `--user`, which is often
`root`. Whoever gets it owns the host.
- **The ticket.** It opens the files of one virtual machine, once, for a
short time. Less than the password, still a way to read a disk.
- **The disk.** Everything on it.
- **The bytes served.** A disk that was altered on the way is worse than
no disk: a backup restored from it looks fine and is not.
## The management port is verified before the password leaves
`Login` is the first message that carries the password, and it goes
through a TLS session that was verified first. Three ways are offered and
one of them has to be taken:
- **A pinned thumbprint.** `--thumbprint sha256:<hash>` accepts exactly
one certificate. Nothing else about it is checked, and nothing else
needs to be: a name, an expiry date or a chain adds nothing when the
certificate itself is known. It is the right choice for the default
self-signed certificate an ESXi host installs, which no CA can vouch
for and which names `localhost.localdomain` rather than the host.
- **A CA bundle.** `--ca-file` verifies the chain against the bundle and
the name against `--host`. For hosts whose certificates are issued by
the organization's own CA.
- **The system roots.** With neither flag, the same verification against
the operating system's root store. For hosts with publicly issued
certificates.
There is no fourth way. No `--insecure`, no `--no-verify`, no trust on
first use, no environment variable that skips the check. A tool that has
such a flag ends up run with it, in a script, forever. Vectura would
rather fail with `certificate thumbprint mismatch` and make the operator
look at why the certificate changed.
The `fingerprint` command exists so that the thumbprint can be obtained
without trusting anything: it accepts any certificate, sends nothing
after the handshake, and prints the hashes for the operator to compare
out of band.
## The data port is verified by the ticket
The connection to port 902 does not need a flag, because the ticket
already carries the SHA-1 thumbprint of the certificate that port
presents. The ticket came through the verified management session, so
its thumbprint is trusted the way the password's destination was. Vectura
then asks the host, inside the first TLS session, to confirm the SHA-256
of its certificate, and compares that with what the TLS layer saw as
well. This is why a host that does not announce `SHA256 supported` is
refused: without the confirmation, the data port would rest on SHA-1
alone. Finally, the second TLS session that carries the disk must present
the same certificate as the first.
An attacker who controls the network between Vectura and the host sees
TLS on 443 and TLS on 902, and can complete neither handshake without the
host's private key. What they can do is drop the connection, which ends
the run with exit code 1. `--transport nfc` removes the second TLS session
and carries the records in clear after the ticket dialogue, which is why
it prints a warning and why the default is `nfcssl`; the password and the
ticket are still protected under `nfc`, the disk is not.
## Secrets stay in memory
- The password is read from `VECTURA_PASSWORD` and nowhere else. Not a
flag, which the process table would show; not a file, which would be
left behind; not a prompt, which a spawning program cannot answer.
- Neither the password, the ticket nor the session cookie is ever written
to standard error, put in an error message, rendered in a debug
dump or kept in a test fixture. The types that hold them print a
placeholder, and a test checks that they do. The transcript shows the
ticket as `<ticket>`.
- The binary carries no credential and no configuration.
A transcript or a bug report can therefore be shared as it is.
## What comes from the host is data
Every reply from the host is parsed with a limit: a dialogue line of at
most 1 KiB, an HTTP body of at most 1 MiB, a record of at most 1 MiB, an
NBD option of at most 4 KiB. A length field that says more is an error,
not an allocation. Each chunk of a read is checked against the read it
answers before it is placed. The code has no `unsafe` block and, outside
tests, no `unwrap` or `expect`: a broken invariant is an error that says
what was violated, not a panic that drops the stream half written.
## The export is read-only, structurally
Vectura never sends the host a write. The NBD export advertises
`NBD_FLAG_READ_ONLY`, and a write, trim or write-zeroes command is
answered `NBD_EPERM` and discarded. There is no flag to change that,
because there is no code path behind it.
## What is not covered
- **The host's own security.** Vectura trusts the certificate it was
told to trust and the bytes the host serves under it. A compromised
host serves what it likes.
- **The spawning program.** Whatever reads the NBD stream from standard
output sees the disk. It also holds the password, since it puts it in
the child's environment.
- **The link's availability.** A connection can be cut. The run ends,
nothing is retried, and the caller decides.
Vulnerabilities, which is anything that touches certificate
verification, credentials, or how data from the host is parsed, are
reported to the maintainers in private, not in a public issue.
@xen-orchestra/vmware-explorer/vectura/docs/how-to/copy-a-disk-to-a-raw-image.md
+130
View File
@@ -0,0 +1,130 @@
# Copy a disk to a raw image
This guide copies a whole virtual disk to a raw image on the local machine
with `nbdcopy`, and checks the copy against an image built from the
datastore's own files.
## What you need
- A virtual machine that is powered off, or a disk that the host will let
you open: a base link under a snapshot of a running virtual machine is
fine, the link a running virtual machine writes to is not.
- `libnbd-bin`, for `nbdinfo` and `nbdcopy`.
- `socat`, to give the NBD tools a socket to connect to.
- The host pinned as in [Pin a host certificate](pin-a-host-certificate.md),
and `VECTURA_PASSWORD` exported in the shell.
## Why a bridge
`vectura serve` speaks NBD on its own standard input and output, the way a
spawning program expects. `nbdcopy` and `nbdinfo` can start a server
themselves with their `[ CMD ]` form, but that form hands the child a
listening socket through systemd socket activation, not its pipes, so
`serve` would wait on standard input while the client waits on the socket.
A Unix socket with `socat` in front of it connects the two: each
connection accepted on the socket becomes the standard input and output of
one fresh `serve` process.
## 1. Write the server command in a script
`socat` splits the command it runs at spaces, and a datastore path has
one, so put the command in a script:
```sh
cat > ~/bin/serve-vm <<'END'
#!/bin/sh
exec vectura serve \
--host esxi.example \
--user root \
--vm-id 3 \
--disk '[datastore1] vm/vm.vmdk' \
--thumbprint sha256:BA:78:16:...:AD \
--verbose "$@"
END
chmod +x ~/bin/serve-vm
```
`--verbose` is optional; it makes each process report a summary on
standard error when it ends.
## 2. Start the bridge
```sh
socket="$XDG_RUNTIME_DIR/vectura.sock"
socat UNIX-LISTEN:"$socket",fork EXEC:"$HOME/bin/serve-vm" &
```
`fork` keeps `socat` listening after the first connection. Every
connection starts one `serve`, which logs in, obtains its own ticket,
opens the disk and ends when the client disconnects. The password reaches
the script through the environment `socat` inherited; nothing about it is
on a command line.
## 3. Look at the export
```sh
nbdinfo "nbd+unix:///?socket=$socket"
```
`nbdinfo` reports the export size, that it is read-only, and the block
sizes (512 bytes minimum, 1 MiB preferred, 32 MiB maximum). It opens one
connection, so one `serve` ran and ended.
## 4. Copy
```sh
nbdcopy -p "nbd+unix:///?socket=$socket" disk.raw
```
The export does not advertise multiple connections, so `nbdcopy` uses one,
and one `serve` process serves the whole disk. Its `summary` line on
standard error, when `--verbose` is set, gives the bytes served, the bytes
that crossed the wire and the elapsed time.
`qemu-img` reads the same socket if you want another format directly:
```sh
qemu-img convert -p -O qcow2 "nbd+unix:///?socket=$socket" disk.qcow2
```
## 5. Stop the bridge
```sh
kill %1
rm -f "$socket"
```
## Check the copy
The reference is the raw image of the disk built from the files the
datastore holds. Download the descriptor and every extent of the disk
through the datastore browser while the virtual machine is off, into one
directory so that a snapshot chain's parent references resolve, then
convert the top descriptor:
```sh
qemu-img convert -O raw vm-000002.vmdk reference.raw
cmp disk.raw reference.raw && echo identical
```
Build the reference before the copy or after it, not during it: the host
locks the files while the datastore browser serves them, and an NFC open
during the download fails with `Failed to lock the file`; the reverse also
holds, and the datastore browser answers `500` while `serve` has the disk
open.
## Long copies
A management session on the host expires after somewhere between 30 and
52 minutes without a management call, and reads on the disk do not count
as one. A copy that takes longer ends with the disk fully served, exit
code 0, and one line on standard error saying the final `Logout` was
refused because the session was no longer authenticated. It is
informational; the copy is complete.
## Related
- [Tune depth and compression](tune-depth-and-compression.md) for a faster
copy on a long link.
- [Host requirements and limits](../reference/host-requirements-and-limits.md)
for what the host lets you open.
@xen-orchestra/vmware-explorer/vectura/docs/how-to/drive-vectura-from-a-program.md
+155
View File
@@ -0,0 +1,155 @@
# Drive Vectura from a program
`vectura serve` is built to be spawned: the program that starts it talks
NBD to it over the child's standard input and output and reads any range
of the disk in any order. This guide states the contract and shows it from
Python with libnbd, from C, and from Node.js.
## The contract
- **One process, one disk, one connection.** Spawn one `serve` per disk
you read. The export does not advertise multiple connections.
- **The password is in the environment, only.** Put `VECTURA_PASSWORD` in
the child's environment. Never on the command line: it would be visible
in the process table.
- **Standard input** carries your NBD client flags, options and requests.
**Standard output** carries the server's greeting, option replies and
simple replies, and nothing else, ever.
- **Standard error** carries diagnostics with a `vectura:` prefix and, with
`--verbose`, the transcript. Read it or discard it; if you pipe it,
drain the pipe so the child never blocks on it. No line on it ever
contains the password, the ticket or the session cookie.
- **Negotiate fixed newstyle.** Send `NBD_FLAG_C_FIXED_NEWSTYLE`, then
`NBD_OPT_GO` for the export size, the read-only flag and the block
sizes. Any export name is accepted, the empty one included.
- **Read within the limits.** `NBD_CMD_READ` at any offset and length
inside the export, at most 32 MiB per command, 1 MiB preferred. A
command past the export or larger than 32 MiB is answered `NBD_EINVAL`.
- **Match replies by cookie.** Commands are answered as soon as their data
is complete, which can be out of order.
- **End with `NBD_CMD_DISC`.** The child closes the disk, logs out and
exits 0. Closing its standard input instead makes it do the same cleanup
and exit 1 with `the nbd client left without disconnecting` on standard
error.
- **Exit codes.** 0 on a clean end, 1 when the host or the client failed,
2 for a usage error such as a missing password or a bad flag; usage
errors are reported before any connection is made.
## Python, with libnbd
```python
import os
import nbd
env = dict(os.environ, VECTURA_PASSWORD=password) # never a command-line argument
h = nbd.NBD()
h.connect_command([
"vectura", "serve",
"--host", "esxi.example",
"--user", "root",
"--vm-id", "3",
"--disk", "[datastore1] vm/vm.vmdk",
"--thumbprint", "sha256:BA:78:16:...:AD",
])
size = h.get_size()
first = h.pread(1 << 20, 0)
last = h.pread(1 << 20, size - (1 << 20))
h.shutdown() # sends NBD_CMD_DISC; the child exits 0
```
`connect_command` spawns the child, pipes its standard input and output
and negotiates with `NBD_OPT_GO`. Set `VECTURA_PASSWORD` in the parent's
environment before the call; libnbd passes the environment through. For
concurrent reads, libnbd's asynchronous `aio_pread` keeps several commands
in flight; see the next section for how many.
## C, with libnbd
```c
struct nbd_handle *h = nbd_create();
char *argv[] = {
"vectura", "serve",
"--host", "esxi.example", "--user", "root", "--vm-id", "3",
"--disk", "[datastore1] vm/vm.vmdk",
"--thumbprint", "sha256:BA:78:16:...:AD",
NULL,
};
if (nbd_connect_command(h, argv) == -1) { /* nbd_get_error() */ }
int64_t size = nbd_get_size(h);
nbd_pread(h, buf, sizeof buf, 0, 0);
nbd_shutdown(h, 0);
nbd_close(h);
```
## Node.js
Node has no NBD client in its standard library, but any client that works
on a duplex stream can be given the child's pipes:
```js
const { spawn } = require('node:child_process')
const { Duplex } = require('node:stream')
const child = spawn(
'vectura',
[
'serve',
'--host',
'esxi.example',
'--user',
'root',
'--vm-id',
'3',
'--disk',
'[datastore1] vm/vm.vmdk',
'--thumbprint',
'sha256:BA:78:16:...:AD',
],
{
env: { ...process.env, VECTURA_PASSWORD: password },
stdio: ['pipe', 'pipe', 'inherit'],
}
)
// One stream that reads the server's replies and writes your requests.
const nbd = Duplex.from({ readable: child.stdout, writable: child.stdin })
child.on('exit', code => {
// 0 after your NBD_CMD_DISC; 1 if the host failed or you closed stdin first.
})
```
A client that only accepts a TCP socket needs a small bridge: a local
`net.createServer` whose accepted socket is piped both ways into the
child's pipes, one child per accepted connection.
## Keep the host busy
`serve` fetches the disk from the host in reads of at most 1 MiB and keeps
up to `--depth` of them in flight, 16 by default. A 16 MiB command alone
fills a depth of 16; sixteen 1 MiB commands do the same. Keep at least
that much outstanding from your side, or the window sits idle between
your commands. The depth is a per-host budget: when several of your
processes read from the same host at once, lower `--depth` so that the
total stays at or under 32.
## Lifetimes
- An idle export costs nothing for at least 30 minutes: a client that
pauses between reads does not need to reconnect within that time.
- The management session expires after 30 to 52 minutes without a
management call. A longer run still serves every byte and exits 0; its
final `Logout` is refused and reported on standard error.
- Killing the child with a signal skips the logout. The host expires the
abandoned session by itself after about half an hour, but prefer
`NBD_CMD_DISC`.
- Connections to the host time out after 10 seconds; a read the host does
not answer for 60 seconds ends the run with `NBD_EIO` to every command
in flight and exit code 1.
## Related
- [The NBD export](../reference/nbd-export.md) lists every option and
command and how it is answered.
- [Command line](../reference/command-line.md) lists the flags and exit
codes.
@xen-orchestra/vmware-explorer/vectura/docs/how-to/pin-a-host-certificate.md
+90
View File
@@ -0,0 +1,90 @@
# Pin a host certificate
`vectura serve` verifies the management port's certificate before it sends
the password, and there is no flag to skip that. This guide shows the
three ways to make the check pass and how to keep it passing when the
certificate changes.
## Choose how the port is trusted
| Situation | Flag | What is checked |
| -------------------------------------------------- | ---------------------------- | ------------------------------------------------------------------------------------------------------------------ |
| The host uses its default, self-signed certificate | `--thumbprint sha256:<hash>` | The SHA-256 hash of the certificate the host presents equals `<hash>`. Nothing else: no name, no expiry, no chain. |
| The host's certificate is issued by your own CA | `--ca-file <bundle.pem>` | The certificate chains to a certificate in the bundle and names the host given to `--host`. |
| The host's certificate is issued by a public CA | nothing | Same check, against the system's root store. |
`--thumbprint` and `--ca-file` exclude each other; giving both is a usage
error.
The default ESXi certificate names `localhost.localdomain`, not the host,
so `--ca-file` cannot accept it even with the right issuer. Pin it.
## Get the thumbprint
```sh
vectura fingerprint --host esxi.example
```
```
443 sha256 BA:78:16:...:AD
443 sha1 3B:...:7F
902 sha256 BA:78:16:...:AD
902 sha1 3B:...:7F
```
`fingerprint` accepts whatever certificate the host presents, sends
nothing after the handshake, and drops the connection. That is the point:
it lets you see the certificate before any credential crosses the wire,
but it cannot tell a genuine host from an interceptor. Check the hash out
of band before you pin it. On the host itself:
```sh
openssl x509 -in /etc/vmware/ssl/rui.crt -noout -fingerprint -sha256
```
Or compare with what the host's administrator gives you.
## Pass the pin
The `443 sha256` value is taken as printed, colons and upper case
included; lower case and a bare hex string are accepted too:
```sh
vectura serve --host esxi.example --thumbprint sha256:BA:78:16:...:AD ...
```
The data port needs no pin from you. The ticket Vectura obtains inside the
verified management session carries the SHA-1 thumbprint of the data
port's certificate, the host confirms the SHA-256 in the data-port
dialogue, and both must match the certificate presented there. On the
hosts measured so far, both ports present the same certificate, which is
why the `443` and `902` lines above are equal.
## Use a CA bundle
```sh
vectura serve --host esxi.example --ca-file /etc/ssl/esxi-ca.pem ...
```
The bundle is a PEM file holding one or more CA certificates. Give
`--host` the name the certificate carries, since the name is part of the
check; an address works only if the certificate lists it.
## After the certificate changes
A renewed or replaced certificate has a new hash, and the next `serve`
stops with:
```
vectura: management plane: RetrieveServiceContent: certificate thumbprint mismatch: expected sha256 BA:78:..., the host presented sha256 24:8D:...
```
Run `fingerprint` again, check the new hash out of band, and update the
pin. A mismatch you did not expect is the check doing its job: do not
update the pin until you know why the certificate changed.
## Related
- [The security model](../explanation/security-model.md) says what each
verification protects.
- [Command line](../reference/command-line.md) lists the flags.
@xen-orchestra/vmware-explorer/vectura/docs/how-to/troubleshoot-a-session.md
+63
View File
@@ -0,0 +1,63 @@
# Troubleshoot a session
Everything Vectura has to say goes to standard error with a `vectura:`
prefix; standard output is data. Start there, and add `--verbose` to see
where in the session the run stopped. This guide lists the messages you
are most likely to meet, what each means and what to do.
## Exit codes
| Code | Meaning |
| ---- | --------------------------------------------------------------------------------------------------------------------------------- |
| 0 | The client disconnected with `NBD_CMD_DISC` and the session ended, or `fingerprint` printed its lines. |
| 1 | The host, the network or the client failed after the command started. The reason is on standard error. |
| 2 | Usage error: a bad or missing flag, `VECTURA_PASSWORD` unset or empty, an unreadable `--ca-file`. Reported before any connection. |
## Before the login
| Message | Meaning | Do |
| ----------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------- |
| `VECTURA_PASSWORD is not set` | The variable is missing or empty in the environment of the process. | Export it in the shell that runs `serve`, or set it in the child's environment. |
| `could not connect: ... timed out` | Nothing answered on port 443 within 10 seconds. | Check the address, the port and the path to the host. |
| `certificate thumbprint mismatch: expected ..., the host presented ...` | The management port's certificate is not the pinned one. | Run `fingerprint`, check the new hash out of band, update the pin. See [Pin a host certificate](pin-a-host-certificate.md). |
| `tls handshake failed: ...` | With `--ca-file` or the system roots: the certificate does not chain or does not name the host. | Pin by thumbprint, or give `--host` the name the certificate carries. |
| `Login: the host answered: ...` | The host refused the credentials; the text is the host's. | Check the user and the password. |
## Opening the disk
| Message | Meaning | Do |
| ---------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------- |
| `greeting: the host does not announce "SHA256 supported"; announced: [...]` | The data port belongs to a host older than ESXi 7, which Vectura does not support. | Use an ESXi 7.x or 8.x host. |
| `greeting: the host does not announce "NFCSSL supported"; announced: [...]` | The host cannot carry the disk inside a second TLS session. | Not seen on a supported host. `--transport nfc` would carry the disk in clear text; understand what that means before using it. |
| `open "[ds] vm/x.vmdk": the host reported error 11: NfcAioProcessOpenFileMsg: permission check failed ...` | The ticket covers the files of the virtual machine `--vm-id` names, and this path is not one of them, or the account may not read it. | Check `--vm-id` against `--disk`. |
| `... Thin/TBZ/Sparse disks cannot be opened in multiwriter mode` | The disk is the link a running virtual machine writes to. | Power the virtual machine off, or take a snapshot and read a base link. |
| `... DiskLib error 16392: Failed to lock the file` | Another reader holds the file, typically a datastore browser download. | Wait for it to finish. |
| `the host closed the connection during the ...` | The data port dropped the session before the disk was open. | Retry once; if it repeats, the host's log says why. |
## While serving
| Symptom | Meaning | Do |
| ----------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------- |
| A read is answered `NBD_EIO`, then the process exits 1 | A host read failed, timed out after 60 seconds, or answered with a chunk that does not fit its read. Every command in flight got `NBD_EIO`, the disk was closed and the session logged out. | Read the message on standard error. A run is not resumed; start a new one. |
| A read is answered `NBD_EINVAL` | The request is past the end of the export, longer than 32 MiB, carries a command flag, or is `NBD_CMD_FLUSH`. | Fix the client's request. |
| A write is answered `NBD_EPERM` | The export is read-only. | Nothing to do; this is the answer. |
| `the nbd client left without disconnecting`, exit 1 | The client closed standard input before `NBD_CMD_DISC`. The disk was closed and the session logged out. | Send `NBD_CMD_DISC` to end cleanly. |
| `logout: Logout: the host answered: The session is not authenticated.` after a long run, exit 0 | The management session expired during the run, which was longer than 30 to 52 minutes. The disk was served in full. | Nothing. It is informational. |
| `nbdcopy [ vectura serve ... ]` or `nbdinfo [ ... ]` hangs, then exits 1 | Those forms start the child with a listening socket, not its pipes. | Put a Unix socket in front; see [Copy a disk to a raw image](copy-a-disk-to-a-raw-image.md). |
## Reading a transcript
With `--verbose`, the last line before the error says how far the session
got: the management calls with their HTTP status, the greeting, the TLS
lines, the dialogue, the handshake message types, the `open` line, the
NBD options. [The transcript](../reference/transcript.md) explains each
line.
## Reporting
A bug report should carry the command line without the password, the
`--verbose` transcript, the host's version and the transport. The
transcript never contains the password, the ticket or the session cookie.
Anything touching certificate verification, credentials or how data from
the host is parsed is a vulnerability; report it to the maintainers in
private, not in a public issue.
@xen-orchestra/vmware-explorer/vectura/docs/how-to/tune-depth-and-compression.md
+76
View File
@@ -0,0 +1,76 @@
# Tune depth and compression
Two flags decide how fast a disk is served: `--depth`, the number of host
reads in flight, and `--compression`, how the host packs each chunk. The
defaults, 16 and `skipz`, are right for most runs. This guide says when
to move them and how to see whether it helped.
## Depth
`serve` splits every client read into host reads of at most 1 MiB and
keeps up to `--depth` of them in flight across all the NBD commands it
holds. A reply lands, the next read goes out, and each NBD command is
answered when its last chunk arrives.
- **Raise it on a long link.** The window is what covers the round trip.
On a link with a 61 ms round trip, going from 16 to 32 finished a
16 GiB disk 1.40 times sooner with exactly the same bytes on the wire.
- **Lower it when processes share a host.** The host's NFC buffers are a
per-host budget, and 32 is the most one process may ask for. With four
processes on one host, `--depth 8` each keeps the total where one
process alone would be.
- **The client has to keep up.** Depth counts host reads, not client
commands. One 16 MiB command occupies sixteen slots; a client that sends
one 512-byte command at a time and waits leaves the window empty
whatever `--depth` says.
The range is 1 to 32; anything else is a usage error.
## Compression
| Mode | What the host sends | When |
| ----------------- | ------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `skipz` (default) | Each 64 KiB chunk with its runs of zeros left out | Almost always. Cheapest for the host, and the wire cost tracks how sparse the disk is. |
| `zlib` | Each chunk deflated | Rarely. Fewest bytes on the wire, but the host deflates slower than most links carry: in the run below it halved the wire bytes of `skipz` and still took longer. |
| `none` | Each chunk as stored | Diagnosis, or a fast local link with a slow host. Six to seven times the bytes of `skipz` on a typical disk. |
Whatever is asked, a chunk the host cannot reduce comes back stored, and
the three modes serve identical bytes.
Measured on an ESXi 7.0.3 host over a link whose 5.81 MB/s ceiling
bounds every row, so read the rows against each other and not as what a
LAN would give:
| Mode | Depth | Elapsed | Served | On the wire |
| ------- | ----- | ------- | ---------- | ----------- |
| `skipz` | 16 | 630 s | 26.0 MiB/s | 4.14 MB/s |
| `zlib` | 16 | 705 s | 23.2 MiB/s | 2.01 MB/s |
| `none` | 16 | 3110 s | 5.3 MiB/s | 5.52 MB/s |
| `skipz` | 32 | 449 s | 36.5 MiB/s | 5.81 MB/s |
## See what a run does
Add `--verbose`. Every GiB served, a line reports the ratio of wire bytes
to disk bytes so far and the depth in use:
```
progress: served 4 GiB, compression ratio 0.15, depth 16
```
At the end, the summary gives the totals:
```
summary: bytes served 17179869184, wire bytes 2607992664, host reads 16384, elapsed 628.6 s
```
A ratio near 1.00 under `skipz` means the disk is dense and `zlib` might
cut wire bytes, at the host's expense. An elapsed time that does not drop
when `--depth` rises means the link or the client, not the window, is the
limit.
## Related
- [The transcript](../reference/transcript.md) for every line `--verbose`
writes.
- [How a disk is read](../explanation/how-a-disk-is-read.md) for what a
host read and a chunk are.
@xen-orchestra/vmware-explorer/vectura/docs/reference/command-line.md
+89
View File
@@ -0,0 +1,89 @@
# Command line
```
vectura <command> [flags]
vectura --version
vectura --help
```
Two commands: `fingerprint` and `serve`. Every message for a person goes
to standard error, prefixed `vectura: `. `serve` writes the NBD stream to
standard output and nothing else.
## `vectura fingerprint`
Prints the hashes of the certificates a host presents on its management
and data ports, then exits. Any certificate is accepted; nothing is sent
after the TLS handshake.
| Flag | Value | Default | Meaning |
| ------------ | --------------------------- | ------------------ | -------------------------------------------------------------------------------------- |
| `--host` | `<name>` or `<name>:<port>` | required, port 443 | The host, with its management port. A bare IPv6 address keeps the default port. |
| `--nfc-port` | port number | 902 | The data port, where the host greets before TLS starts. |
| `--verbose` | | off | Also write the negotiated TLS version and cipher suite of each port to standard error. |
Output, one line per port and hash, hex in upper case with colons:
```
443 sha256 <hash>
443 sha1 <hash>
902 sha256 <hash>
902 sha1 <hash>
```
Exit code 0 when the four lines were printed, 1 when a port could not be
reached or did not complete a handshake, 2 for a usage error.
## `vectura serve`
Exports one virtual disk, read-only, as a fixed-newstyle NBD server on
standard input and output. The password is read from `VECTURA_PASSWORD`.
| Flag | Value | Default | Meaning |
| --------------- | --------------------------- | ------------------ | ---------------------------------------------------------------------------------------------------------------------------- |
| `--host` | `<name>` or `<name>:<port>` | required, port 443 | The ESXi host and its management port. With `--ca-file` or the system roots, the name is checked against the certificate. |
| `--user` | user name | required | The account to log in with. |
| `--vm-id` | managed object reference | required | The virtual machine that owns the disk, such as `3` or `vm-42`. |
| `--disk` | datastore path | required | The disk to export, such as `[datastore1] vm/vm.vmdk`. Quote it: the path holds a space. |
| `--thumbprint` | `sha256:<64 hex digits>` | none | Pin the management port's certificate to this hash. Colons between pairs and either case are accepted. Excludes `--ca-file`. |
| `--ca-file` | path to a PEM bundle | none | Verify the management port's certificate against this bundle instead of the system roots. Excludes `--thumbprint`. |
| `--depth` | 1 to 32 | 16 | Host reads kept in flight at once. The limit is per host: lower it when several processes read from the same host. |
| `--compression` | `skipz`, `zlib`, `none` | `skipz` | How the host packs each chunk: runs of zeros left out, deflated, or as stored. |
| `--transport` | `nfcssl`, `nfc` | `nfcssl` | Whether the disk session runs inside a second TLS session on the data connection, or in clear. `nfc` prints a warning. |
| `--verbose` | | off | Write the session transcript to standard error. |
Without `--thumbprint` or `--ca-file`, the management port's certificate
is verified against the system's root store. There is no flag that skips
verification. The data port is verified against the thumbprint carried by
the ticket and confirmed in the ticket dialogue, with no flag involved.
### Environment
| Variable | Meaning |
| ------------------ | ---------------------------------------------------------------------------------------------------------------------------- |
| `VECTURA_PASSWORD` | The password of `--user`. Required and non-empty; checked before any connection. Never read from a flag, a file or a prompt. |
### Exit codes
| Code | When |
| ---- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| 0 | The NBD client sent `NBD_CMD_DISC`; the disk was closed and the session logged out. |
| 1 | The host, the network or the client failed after the command started: a connection refused or timed out, a certificate that did not match, a host error, a malformed reply, a client that closed standard input without disconnecting. The reason is on standard error. |
| 2 | Usage error: unknown or conflicting flags, a value out of range, `VECTURA_PASSWORD` unset or empty, a `--ca-file` that does not parse. Reported before any connection. |
### Warnings
`--transport nfc` prints, before the session starts:
```
vectura: warning: --transport nfc carries the disk in clear text after the ticket dialogue
```
### Timeouts
| Timeout | Value |
| ------------------------------------- | ---------- |
| Connecting to a port | 10 seconds |
| A read or write on a connected socket | 60 seconds |
Either one, expired, ends the run with exit code 1.
@xen-orchestra/vmware-explorer/vectura/docs/reference/host-requirements-and-limits.md
+85
View File
@@ -0,0 +1,85 @@
# Host requirements and limits
What the ESXi host has to be and do for `vectura serve` to read a disk,
and the limits it imposes. Measurements were taken on ESXi 7.0.3 and
8.0.1 hosts; VMware's documentation is cited where it states the limit.
## The host
| Requirement | Detail |
| ------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| ESXi 7.x or 8.x, directly or through a vCenter | With a vCenter, `--host` names the vCenter and the ticket names the ESXi host that holds the virtual machine; the data port is opened on that host. |
| `SHA256 supported` in the data port's greeting | The ticket dialogue confirms the data port's certificate by its SHA-256. ESXi 6.x does not announce it and is refused: `greeting: the host does not announce "SHA256 supported"`. |
| `NFCSSL supported` in the greeting, for `--transport nfcssl` | Without it, only `--transport nfc` can run, in clear text. Every 7.x and 8.x host measured announces it. |
| Port 443 and port 902 reachable | 443 for the management calls, on the vCenter when there is one; 902 for the greeting, the ticket dialogue and the disk, always on the ESXi host. |
| An account that may read the virtual machine's files | The account logs in on port 443 and obtains the ticket; the ticket covers the files of the virtual machine `--vm-id` names. |
The greeting lines and their capability words are documented in Broadcom
knowledge base articles 343952, 338286, 341384 and 417531.
## The disk
| Disk | Can it be opened |
| --------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- |
| Any link of a powered-off virtual machine | Yes. |
| A base link under a snapshot of a running virtual machine | Yes, and the bytes are frozen for the duration. |
| The link a running virtual machine writes to | No: `Thin/TBZ/Sparse disks cannot be opened in multiwriter mode`. |
| A file the datastore browser is serving at the same time | No: `DiskLib error 16392: Failed to lock the file`. The reverse holds too: the datastore browser answers `500` while the disk is open over NFC. |
| A file of another virtual machine than `--vm-id` | No: error 11, `NfcAioProcessOpenFileMsg: permission check failed`. |
A snapshot chain is read through its top link; the host resolves the
parents. The export size is the disk's virtual capacity, whatever the
size of the files.
## Sizes
| Item | Value | Source |
| ----------------------- | ------------------------------------------------------------------------------------------------ | ------------------------------------------------------- |
| Host read | At most 1 MiB per read | VMware's documented NFC session limits: 1 MB per stream |
| Chunk | 64 KiB, the host's unit of compression and placement | Measured on ESXi 7 and 8 |
| Sector | 512 bytes | The open reply |
| Chunk compression codes | `none` 0, `zlib` 1, `skipz` 3; a chunk the host cannot reduce comes back as 0 whatever was asked | Measured on ESXi 7 and 8 |
## Concurrency
| Item | Value | Source |
| ------------------------------- | ----------------------------------------------------------- | -------------------------------------- |
| Reads in flight per process | 1 to 32, `--depth`, default 16 | Vectura |
| NFC buffer per host | 48 MB, 1 MB per stream, about 48 concurrent streams | VMware's documented NFC session limits |
| Connections per `serve` process | One management session, one data connection, one NBD client | Vectura |
Several processes on one host share the host's buffer: keep the sum of
their depths at or under 32.
## Time
| Item | Value | Source |
| ----------------------------------------------------- | ------------------------------------------------------------------------- | -------- |
| Management session lifetime without a management call | Expires between 30 and 52 minutes; reads on the data port do not renew it | Measured |
| Open disk, idle | Stays open for at least 30 minutes without a read | Measured |
| Abandoned session after a signal | Expired by the host on its own, after about half an hour | Measured |
| Connect timeout | 10 seconds | Vectura |
| Socket read or write timeout | 60 seconds | Vectura |
A run longer than the session lifetime serves the whole disk and exits
0; its `Logout` is refused and reported on standard error.
## TLS
| Item | Value | Source |
| ---------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------ |
| Versions the hosts negotiate | TLS 1.2 on both ports, `TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384` or the AES-128 variant | Measured; the vSphere 7 and 8 documentation on the default cipher suites |
| Certificate on 902 | The same certificate as on 443 on every host measured | Measured |
| Data port verification | The SHA-1 thumbprint in the ticket, the SHA-256 confirmed in the dialogue, and the second TLS session must present the certificate of the first | Vectura |
## Throughput, measured
One 16 GiB disk on an ESXi 7.0.3 host, over a link with a 61 ms round
trip and a 5.81 MB/s ceiling, which bounds every row. Not a LAN figure.
| `--compression` | `--depth` | Elapsed | Served | Wire |
| --------------- | --------- | ------- | ---------- | --------- |
| `skipz` | 16 | 630 s | 26.0 MiB/s | 4.14 MB/s |
| `zlib` | 16 | 705 s | 23.2 MiB/s | 2.01 MB/s |
| `none` | 16 | 3110 s | 5.3 MiB/s | 5.52 MB/s |
| `skipz` | 32 | 449 s | 36.5 MiB/s | 5.81 MB/s |
@xen-orchestra/vmware-explorer/vectura/docs/reference/nbd-export.md
+86
View File
@@ -0,0 +1,86 @@
# The NBD export
`vectura serve` implements the server side of the NBD protocol, as the
protocol document describes it
(<https://github.com/NetworkBlockDevice/nbd/blob/master/doc/proto.md>),
over its standard input (requests) and standard output (replies). This
page lists what the server sends, what it accepts and how it answers
everything else.
## Handshake
The server sends the fixed-newstyle greeting:
| Field | Value |
| --------------- | ---------------------------------------------------------- |
| `NBDMAGIC` | `0x4e42444d41474943` |
| `IHAVEOPT` | `0x49484156454F5054` |
| Handshake flags | `NBD_FLAG_FIXED_NEWSTYLE` (1) and `NBD_FLAG_NO_ZEROES` (2) |
The client flags must carry `NBD_FLAG_C_FIXED_NEWSTYLE`; otherwise the
run ends with `nbd: the client flags 0x... lack NBD_FLAG_C_FIXED_NEWSTYLE`
and exit code 1. `NBD_FLAG_C_NO_ZEROES` is honored: with it, the 124
zero bytes after `NBD_OPT_EXPORT_NAME`'s reply are omitted.
## Options
| Option | Reply |
| ------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- |
| `NBD_OPT_EXPORT_NAME` (1) | The export size and flags, then transmission starts. |
| `NBD_OPT_ABORT` (2) | `NBD_REP_ACK`, then the server exits 0. |
| `NBD_OPT_INFO` (6) | `NBD_REP_INFO` for `NBD_INFO_EXPORT` and `NBD_INFO_BLOCK_SIZE`, then `NBD_REP_ACK`; negotiation continues. |
| `NBD_OPT_GO` (7) | The same replies as `NBD_OPT_INFO`, then transmission starts. |
| Anything else (`NBD_OPT_LIST`, `NBD_OPT_STARTTLS`, `NBD_OPT_STRUCTURED_REPLY`, extended headers, metadata contexts) | `NBD_REP_ERR_UNSUP`; negotiation continues. |
The export name given to `NBD_OPT_EXPORT_NAME`, `NBD_OPT_INFO` or
`NBD_OPT_GO` is ignored: there is one disk, whatever the client calls it.
An `NBD_OPT_INFO` or `NBD_OPT_GO` whose data does not parse as an export
name followed by a count of information requests is answered
`NBD_REP_ERR_INVALID`. Option data longer than 4096 bytes is a protocol
failure: the run ends with exit code 1.
### What the export reports
| Item | Value |
| -------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Export size | The virtual capacity of the disk, in bytes, as the host reports it at open. For a snapshot chain, the capacity of the disk, not the size of the top link's file. |
| Transmission flags | `NBD_FLAG_HAS_FLAGS` (1) and `NBD_FLAG_READ_ONLY` (2), nothing else. |
| Minimum block size | 512 bytes |
| Preferred block size | 1 MiB |
| Maximum block size | 32 MiB |
Not advertised, so not to be assumed: multiple connections
(`NBD_FLAG_CAN_MULTI_CONN`), flush, trim, write zeroes, fast zero,
structured replies, TLS on the NBD side.
## Transmission
Requests carry `NBD_REQUEST_MAGIC`; any other magic ends the run with
exit code 1. Replies are simple replies, `NBD_SIMPLE_REPLY_MAGIC`, with
the request's cookie. Replies can arrive in a different order from the
requests.
| Command | Condition | Reply |
| ---------------------------------------------------------------------------------- | ----------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------- |
| `NBD_CMD_READ` (0) | Offset plus length within the export, length at most 32 MiB, no command flags | Error 0 and the data. |
| `NBD_CMD_READ` | Past the export, longer than 32 MiB, or with any command flag set | `NBD_EINVAL` (22), no data. |
| `NBD_CMD_READ` | A host read failed, timed out, or answered with a chunk that does not fit | `NBD_EIO` (5) to this command and every other in flight, then the disk is closed, the session logged out, and the server exits 1. |
| `NBD_CMD_WRITE` (1), `NBD_CMD_TRIM` (4), `NBD_CMD_WRITE_ZEROES` (6) | | `NBD_EPERM` (1). The data of a write is consumed and discarded. |
| `NBD_CMD_FLUSH` (3), `NBD_CMD_CACHE` (5), anything unknown, any command with flags | | `NBD_EINVAL` (22). |
| `NBD_CMD_DISC` (2) | | No reply. The disk is closed, the session logged out, the server exits 0. |
A length of zero on a read is within the export whenever the offset is,
and is answered with error 0 and no data.
## Ending
| Event | Result |
| ------------------------------------------- | ------------------------------------------------------------------------------------- |
| `NBD_CMD_DISC` | Close, logout, exit 0. |
| `NBD_OPT_ABORT` during negotiation | Logout, exit 0. |
| Standard input closed before `NBD_CMD_DISC` | Close, logout, exit 1, `the nbd client left without disconnecting` on standard error. |
| A malformed request, option or flag word | Close, logout, exit 1, the reason on standard error. |
| A host or network failure | `NBD_EIO` to what is in flight, close if possible, logout, exit 1. |
Once the server has stopped, it does not reconnect and cannot be reused:
a new read means a new process.
@xen-orchestra/vmware-explorer/vectura/docs/reference/transcript.md
+77
View File
@@ -0,0 +1,77 @@
# The transcript
With `--verbose`, `vectura serve` writes one line per event of the
session to standard error, in the order the events happen. `vectura
fingerprint --verbose` writes only the TLS lines. Without `--verbose`,
standard error carries only errors and warnings, all prefixed `vectura: `.
Transcript lines carry no prefix. None ever contains the password, the
NFC ticket or the management session cookie: the ticket is rendered as
`<ticket>`. A line standard error cannot take is dropped; the NBD stream
is never delayed by the transcript.
## Lines, in session order
| Line | When | Fields |
| -------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ |
| `<call>: <status> in <n> ms` | After each management call on port 443: `RetrieveServiceContent`, `Login`, `RetrieveInternalContent`, `NfcGetVmFiles`, and `Logout` at the end. | The HTTP status the host answered, and the time from request to full response. |
| `greeting: <capabilities>` | After the data port's greeting line. | The capabilities the host announced, comma separated, as the host wrote them, such as `NFCSSL supported, SHA256 supported`. |
| `first session tls <version> <suite>` | After the first TLS handshake on the data port. | The protocol version and cipher suite, in rustls's names, such as `TLSv1_2 TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384`. |
| `> <line>` | Each line sent in the ticket dialogue. | `SESSION <ticket>`, `BANNER `, `THUMBPRINT_SHA2 <tag>`, `PROXY nfcssl` or `PROXY nfc`. |
| `< <line>` | Each line the host answered. | The host's reply, such as `220 ...` or `200 <thumbprint>`. |
| `second session tls <version> <suite>` | After the second TLS handshake, `--transport nfcssl` only. | As above. |
| `> handshake type <n>` | Each handshake message sent. | The message type: 43, 33 and 51 are sent, then 54, 55 and 52 after the host's replies. |
| `< handshake type <n>` | Each handshake message received. | The message type: 36 and 51 after the first three, 52 after the last three. |
| `open: <capacity> bytes, sector size <n>` | After the host's open reply. | The disk's virtual capacity and its sector size, as reported. |
| `nbd option <n>: <outcome>` | After each NBD option. | The option number and how it was answered: `export`, `info`, `acknowledged`, `invalid` or `unsupported`. |
| `nbd export: <size> bytes` | When transmission starts. | The export size sent to the client. |
| `progress: served <n> GiB, compression ratio <r>, depth <d>` | Each time the bytes served cross a whole gibibyte. | Whole gibibytes served so far, wire bytes per served byte with two decimals, and the depth in use. |
| `summary: bytes served <n>, wire bytes <n>, host reads <n>, elapsed <s> s` | When the session ends, before `Logout`. | Bytes served to the client, bytes received from the host for them, host reads issued, and the seconds since negotiation began, with one decimal. |
`fingerprint --verbose` writes `<port> tls <version> <suite>` per port,
with `443` or the value of `--nfc-port` as the port.
## Ratios
`compression ratio` is wire bytes divided by served bytes. `1.00` means
the host sent every byte as stored; `0.15` means fifteen bytes on the
wire per hundred bytes of disk. Before any byte is served it reads
`0.00`.
## Example
```
RetrieveServiceContent: 200 in 41 ms
Login: 200 in 63 ms
RetrieveInternalContent: 200 in 38 ms
NfcGetVmFiles: 200 in 52 ms
greeting: SSL Required, ServerDaemonProtocol:SOAP, MKSDisplayProtocol:VNC , VMXARGS supported, NFCSSL supported, SHA256 supported
first session tls TLSv1_2 TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
> SESSION <ticket>
> BANNER
< 220 NFC server ready
> THUMBPRINT_SHA2 <tag>
< 200 BA:78:16:...:AD
> PROXY nfcssl
< 200 ...
second session tls TLSv1_2 TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
> handshake type 43
> handshake type 33
> handshake type 51
< handshake type 36
< handshake type 51
> handshake type 54
> handshake type 55
> handshake type 52
< handshake type 52
open: 17179869184 bytes, sector size 512
nbd option 7: info
nbd export: 17179869184 bytes
progress: served 1 GiB, compression ratio 0.16, depth 16
...
summary: bytes served 17179869184, wire bytes 2607992664, host reads 16384, elapsed 628.6 s
Logout: 200 in 35 ms
```
The `220` and `200` lines are illustrative: their text after the code is
the host's and varies by version.
@xen-orchestra/vmware-explorer/vectura/docs/tutorials/read-your-first-disk.md
+181
View File
@@ -0,0 +1,181 @@
# Read your first disk
This tutorial takes you from a fresh Debian or Ubuntu machine to reading
the first sector of a virtual machine's disk on an ESXi host. It takes
about fifteen minutes. At the end you will have pinned a host, run
`vectura serve` under an NBD client, read bytes from the disk and read the
transcript of the session.
## What you need
- A Debian 12 or Ubuntu 22.04 machine, or newer, on amd64, that reaches the
ESXi host on ports 443 and 902.
- An ESXi 7.x or 8.x host, reached directly by its own address. A vCenter
works too, as long as this machine also reaches the ESXi host on port 902.
- An account on that host allowed to read the virtual machine's files.
`root` works; a narrower account is fine as long as the host lets it
obtain an NFC ticket for the virtual machine.
- A virtual machine that is powered off. The disk a running virtual
machine writes to cannot be opened; the section "Pick a disk" says why.
- The Vectura binary, `vectura`, from
`@xen-orchestra/vmware-explorer/vectura/` in Xen Orchestra.
- `libnbd-bin`, which provides `nbdsh`, an NBD client that can start a
server as a child process and talk to it over its pipes.
## 1. Install the tools
```sh
sudo install vectura /usr/local/bin/
sudo apt install libnbd-bin
vectura --version
```
The last command prints `vectura` and the version. If it prints a usage
message instead, the binary did not install.
## 2. Pin the host
Vectura never skips certificate verification. ESXi hosts ship a
self-signed certificate, so the usual way to trust one is to pin it by its
SHA-256 thumbprint. Print the thumbprints first:
```sh
vectura fingerprint --host esxi.example
```
The output has four lines, two per port:
```
443 sha256 BA:78:16:...:AD
443 sha1 3B:...:7F
902 sha256 BA:78:16:...:AD
902 sha1 3B:...:7F
```
Before you trust the `443 sha256` line, check it against the host itself.
On the host, over SSH, the certificate is at `/etc/vmware/ssl/rui.crt`:
```sh
openssl x509 -in /etc/vmware/ssl/rui.crt -noout -fingerprint -sha256
```
When the two hashes match, keep the `443 sha256` value: it is what
`--thumbprint` takes, colons included. The data port needs no pin from
you; the host tells Vectura which certificate to expect there, inside the
verified management session.
## 3. Put the password in the environment
Vectura reads the password from `VECTURA_PASSWORD` and from nowhere else:
not a flag, not a file, not a prompt. Read it into the variable without
echoing it and without leaving it in the shell history:
```sh
read -rs VECTURA_PASSWORD && export VECTURA_PASSWORD
```
Type the password and press Enter. The variable is now set for this shell
and for every process it starts.
## 4. Pick a disk
`serve` needs two things from the virtual machine: its managed object
reference and the datastore path of the disk.
- The managed object reference is the number in the virtual machine's URL
in the ESXi host client, `.../host/vms/3` for `3`. Vectura takes it as
`--vm-id 3`.
- The datastore path is the "Disk file" of the hard disk in the virtual
machine's settings, in the form `[datastore1] vm/vm.vmdk`. Vectura takes
it as `--disk '[datastore1] vm/vm.vmdk'`, quoted because of the space.
Power the virtual machine off before you go on. The host refuses to open
the link a running virtual machine writes to, and a disk read while it is
being written to would not be a consistent image anyway.
## 5. Read the first sector
`nbdsh` starts `vectura serve` as a child process, negotiates the NBD
export over the child's standard input and output, and gives you a handle
`h` on it. Fill in your host, thumbprint, reference and path:
```sh
nbdsh -c '
h.connect_command(["vectura", "serve",
"--host", "esxi.example",
"--user", "root",
"--vm-id", "3",
"--disk", "[datastore1] vm/vm.vmdk",
"--thumbprint", "sha256:BA:78:16:...:AD"])
print("capacity:", h.get_size(), "bytes")
sector = h.pread(512, 0)
print("boot signature:", sector[510:512].hex())
h.shutdown()
'
```
You should see the capacity of the disk in bytes and, for a disk with a
partition table, the boot signature `55aa`. Behind the scenes, Vectura
logged in to the host, obtained a ticket for the virtual machine's files,
opened the disk on the data port, answered the client's negotiation with
the capacity the host reported, served one 512-byte read, then closed the
disk and logged out when `h.shutdown()` sent the disconnect.
If the command fails instead, the reason is on standard error with a
`vectura:` prefix. The most common ones at this point:
- `VECTURA_PASSWORD is not set`: step 3 was run in another shell.
- `certificate thumbprint mismatch`: the `--thumbprint` value is not the
`443 sha256` line of step 2.
- `the host reported error 11: ... permission check failed`: the path
does not belong to the virtual machine `--vm-id` names, or the account
may not read it.
## 6. Watch the session
Run the same command with `--verbose` added to the argument list. Vectura
then narrates the session on standard error, which `nbdsh` leaves on your
terminal:
```
RetrieveServiceContent: 200 in 41 ms
Login: 200 in 63 ms
RetrieveInternalContent: 200 in 38 ms
NfcGetVmFiles: 200 in 52 ms
greeting: SSL Required, ServerDaemonProtocol:SOAP, MKSDisplayProtocol:VNC , VMXARGS supported, NFCSSL supported, SHA256 supported
first session tls TLSv1_2 TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
> SESSION <ticket>
> BANNER
< 220 ...
> THUMBPRINT_SHA2 ...
< 200 BA:78:16:...
> PROXY nfcssl
< 200 ...
second session tls TLSv1_2 TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
> handshake type 43
...
open: 17179869184 bytes, sector size 512
nbd option 7: info
nbd export: 17179869184 bytes
summary: bytes served 512, wire bytes 528, host reads 1, elapsed 0.3 s
Logout: 200 in 35 ms
```
The ticket is shown as `<ticket>`: the transcript never carries the
password, the ticket or the session cookie, so it is safe to paste into a
bug report. The `summary` line says the one sector cost one read on the
host; the wire bytes count the chunk's framing as well as its data. On a
whole disk the wire bytes fall far below the bytes served, because the
default compression leaves out runs of zeros.
## Where to go next
- To copy a whole disk into a raw image and check it against the
datastore's own copy, see
[Copy a disk to a raw image](../how-to/copy-a-disk-to-a-raw-image.md).
- To spawn Vectura from your own program, see
[Drive Vectura from a program](../how-to/drive-vectura-from-a-program.md).
- To understand what the transcript lines mean, see
[The transcript](../reference/transcript.md), and for what happened
between the login and the first byte,
[How a disk is read](../explanation/how-a-disk-is-read.md).
@xen-orchestra/vmware-explorer/vectura/rust-toolchain.toml
+7
View File
@@ -0,0 +1,7 @@
# The toolchain CI, scripts/build.sh and a fresh clone use, through
# rustup. An exact version rather than "stable" because `cargo clippy -- -D
# warnings` would turn every lint a newer Clippy ships into a red gate on a
# pull request that touched nothing.
[toolchain]
channel = "1.98.0"
components = ["rustfmt", "clippy", "llvm-tools-preview"]
@xen-orchestra/vmware-explorer/vectura/scripts/build.sh
Executable
+15
View File
@@ -0,0 +1,15 @@
#!/bin/sh
# Builds the vectura binary into the directory given as the only argument.
#
# zig links the binary against glibc 2.34 whatever the glibc of the build
# machine, so it runs on Debian 12 and Ubuntu 22.04 wherever it is built.
# Needs zig and cargo-zigbuild, see the README. The Vectura workflow and
# release.sh run it.
set -eu
out=$(mkdir -p "$1" && cd "$1" && pwd)
cd "$(dirname "$0")/.."
cargo zigbuild --locked --release --target x86_64-unknown-linux-gnu.2.34
# the glibc suffix is for zig only: the binary lands under the plain triple
install -m 755 target/x86_64-unknown-linux-gnu/release/vectura "$out"
@xen-orchestra/vmware-explorer/vectura/scripts/check-binary.sh
Executable
+37
View File
@@ -0,0 +1,37 @@
#!/usr/bin/env bash
# Checks the vectura binary given as the only argument. It links against
# glibc only and uses no glibc symbol newer than 2.34, the version build.sh
# links against, below the glibc 2.36 of Debian 12, the oldest system it
# supports: another library or a newer glibc fails here instead of on a
# user's machine. Then it runs the binary: without a subcommand vectura exits
# 2 with a `vectura: ` line on standard error and nothing on standard output,
# which proves it runs and links. The Vectura workflow runs it.
set -euo pipefail
export LC_ALL=C
# a path with a slash, or running it would look it up in PATH
binary=$(realpath "$1")
test -x "$binary"
readelf --dynamic "$binary" | sed -n 's/.*(NEEDED).*\[\(.*\)\]$/\1/p' | while read -r library; do
case $library in
libc.so.6 | libm.so.6 | ld-linux-x86-64.so.2) ;;
*)
echo "$binary needs $library" >&2
exit 1
;;
esac
done
newest=$(objdump -T "$binary" | grep -o 'GLIBC_[0-9.]*' | sort -u -V | tail -n 1)
if [ "$(printf '%s\n' "$newest" GLIBC_2.34 | sort -V | tail -n 1)" != GLIBC_2.34 ]; then
echo "$binary needs $newest" >&2
exit 1
fi
tmp=$(mktemp -d)
code=0
"$binary" > "$tmp/out" 2> "$tmp/err" || code=$?
test "$code" -eq 2
test ! -s "$tmp/out"
grep --quiet '^vectura: ' "$tmp/err"
@xen-orchestra/vmware-explorer/vectura/scripts/release.sh
Executable
+49
View File
@@ -0,0 +1,49 @@
#!/bin/sh
# Releases vectura: `scripts/bump-pkg vectura <patch|minor|major>`, at the
# root of the repository, runs it for the `- vectura <type>` line of
# CHANGELOG.unreleased.md. Bumps the version of the crate, rebuilds its
# binary in place of the previous one and commits both. See build.sh for the
# tools it needs.
set -eu
case ${1-} in
patch | minor | major) type=$1 ;;
*)
echo "Usage: $0 patch|minor|major" >&2
exit 1
;;
esac
cd "$(dirname "$0")/.."
# as bump-pkg does with package.json: after a failed run, bumping again would
# skip a version
if ! git diff-index --quiet HEAD Cargo.toml; then
echo "Changes detected in Cargo.toml, revert them first" >&2
exit 1
fi
old=$(sed -n 's/^version = "\(.*\)"$/\1/p' Cargo.toml | head -n 1)
new=$(echo "$old" | awk -F. -v type="$type" '{
if (type == "major") print $1 + 1 ".0.0"
else if (type == "minor") print $1 "." $2 + 1 ".0"
else print $1 "." $2 "." $3 + 1
}')
echo "vectura: $old -> $new ($type)"
# The first `version` line of Cargo.toml is the one of [package], and the
# line after `name = "vectura"` in Cargo.lock is its entry there.
awk -v new="$new" '!done && /^version = / { $0 = "version = \"" new "\""; done = 1 } 1' Cargo.toml > Cargo.toml.tmp
mv Cargo.toml.tmp Cargo.toml
awk -v new="$new" 'prev == "name = \"vectura\"" && /^version = / { $0 = "version = \"" new "\"" } { print; prev = $0 }' Cargo.lock > Cargo.lock.tmp
mv Cargo.lock.tmp Cargo.lock
tmp=$(mktemp -d)
scripts/build.sh "$tmp"
# mv keeps the mode build.sh set, which git records: the binary stays
# executable in the npm package
mv "$tmp/vectura" vectura
rmdir "$tmp"
git add -- Cargo.toml Cargo.lock vectura
git commit --message "feat(vectura): $new"
@xen-orchestra/vmware-explorer/vectura/src/cli.rs
+581
View File
@@ -0,0 +1,581 @@
//! The command line: argument parsing, exit codes, and which stream gets what.
//!
//! Exit codes: 0 on success, 1 when talking to the host or writing the result
//! failed, 2 for a usage error. Explanations go to standard error with a
//! `vectura:` prefix; standard output carries results only: the thumbprint
//! lines of `fingerprint`, the NBD stream of `serve`.
use std::env::VarError;
use std::ffi::OsString;
use std::io::{self, Write};
use std::num::{NonZeroU8, NonZeroUsize};
use std::path::{Path, PathBuf};
use std::process::ExitCode;
use clap::{Args, Parser, Subcommand};
use rustls_pki_types::CertificateDer;
use rustls_pki_types::pem::PemObject;
use crate::transcript::Transcript;
use crate::{disk, nfc, serve, soap, tls};
/// Exit code when the host or the output could not be dealt with.
const EXIT_FAILURE: u8 = 1;
/// Exit code for a usage error: bad flags, missing values, unknown commands.
const EXIT_USAGE: u8 = 2;
/// What clap puts before every usage message; `vectura:` replaces it so the
/// line names the program rather than a generic word.
const CLAP_ERROR_PREFIX: &str = "error: ";
/// The management port, where the vSphere API listens over TLS.
const MANAGEMENT_PORT: u16 = 443;
/// The data port, where the host greets in clear before TLS starts.
const DATA_PORT: u16 = 902;
/// The only place the password is read from: never a flag, a file or a prompt.
const PASSWORD_VARIABLE: &str = "VECTURA_PASSWORD";
/// What a `--thumbprint` value starts with; the rest is the SHA-256 hex.
const THUMBPRINT_PREFIX: &str = "sha256:";
/// How many host reads `serve` keeps in flight unless `--depth` says otherwise.
///
/// Half of [`serve::MAX_DEPTH`]: enough to keep a host busy, and two
/// processes reading from one host stay under its limit without being told to.
const DEFAULT_DEPTH: &str = "16";
/// Reads VMware virtual disks over NFC.
#[derive(Debug, Parser)]
#[command(name = "vectura", version)]
struct Cli {
#[command(subcommand)]
command: Command,
}
#[derive(Debug, Subcommand)]
enum Command {
/// Print the certificate thumbprints a host presents on its management and data ports.
Fingerprint(Fingerprint),
/// Export one virtual disk, read-only, as an NBD server on standard input and output.
Serve(Serve),
}
/// Arguments of `vectura fingerprint`.
#[derive(Debug, Args)]
struct Fingerprint {
/// Host name or address, with an optional management port (default 443).
#[arg(long, value_parser = parse_host)]
host: HostPort,
/// Data port where the host greets before TLS starts.
#[arg(long, default_value_t = DATA_PORT)]
nfc_port: u16,
/// Also report the negotiated TLS version and cipher suite on standard error.
#[arg(long)]
verbose: bool,
}
/// Arguments of `vectura serve`; the password comes from `VECTURA_PASSWORD`.
#[derive(Debug, Args)]
struct Serve {
/// ESXi host or vCenter name or address, with an optional management port (default 443).
#[arg(long, value_parser = parse_host)]
host: HostPort,
/// User name to log in with.
#[arg(long)]
user: String,
/// Managed object reference of the virtual machine, such as `3` or `vm-42`.
#[arg(long)]
vm_id: String,
/// Datastore path of the disk, such as `[datastore1] vm/vm.vmdk`.
#[arg(long)]
disk: String,
/// Pin the management port's certificate: `sha256:` and 64 hex digits.
#[arg(long, value_parser = parse_thumbprint, conflicts_with = "ca_file")]
thumbprint: Option<tls::Pin>,
/// Verify the management port's certificate against this PEM bundle instead of the system roots.
#[arg(long)]
ca_file: Option<PathBuf>,
/// Host reads kept in flight at once, 1 to 32.
///
/// The limit is per host: lower it when several processes read from the same host.
#[arg(long, default_value = DEFAULT_DEPTH, value_parser = parse_depth)]
depth: NonZeroU8,
/// How the host sends each chunk: skipz, zlib or none.
///
/// skipz leaves out the runs of zeros, zlib deflates every chunk, none
/// sends the bytes as stored.
#[arg(long, default_value_t = disk::Compression::Skipz)]
compression: disk::Compression,
/// How the disk session runs after the ticket dialogue.
///
/// `nfcssl` opens a second TLS session on the same connection, `nfc` stays
/// in clear.
#[arg(long, default_value_t = nfc::Transport::Nfcssl)]
transport: nfc::Transport,
/// Write a transcript of the session to standard error.
#[arg(long)]
verbose: bool,
}
/// A host name or address and the management port to reach it on.
#[derive(Debug, Clone)]
struct HostPort {
host: String,
port: u16,
}
/// Parses `host` or `host:port`; a bare IPv6 address keeps the default port.
fn parse_host(text: &str) -> Result<HostPort, String> {
let (host, port) = match text.rsplit_once(':') {
Some((host, port)) if !host.contains(':') => {
let port = port
.parse()
.map_err(|error| format!("invalid port {port:?}: {error}"))?;
(host, port)
}
_ => (text, MANAGEMENT_PORT),
};
rustls_pki_types::ServerName::try_from(host)
.map_err(|error| format!("invalid host {host:?}: {error}"))?;
Ok(HostPort {
host: host.to_owned(),
port,
})
}
/// Parses `sha256:<hex>`; colons between pairs and either case are accepted.
fn parse_thumbprint(text: &str) -> Result<tls::Pin, String> {
let hex = text
.strip_prefix(THUMBPRINT_PREFIX)
.ok_or_else(|| format!("{text:?} does not start with {THUMBPRINT_PREFIX}"))?;
tls::Pin::sha256(hex).map_err(|error| error.to_string())
}
/// Parses a depth of 1 to [`serve::MAX_DEPTH`], naming the range when it is not.
fn parse_depth(text: &str) -> Result<NonZeroU8, String> {
text.parse::<NonZeroU8>()
.ok()
.filter(|depth| usize::from(depth.get()) <= serve::MAX_DEPTH)
.ok_or_else(|| format!("{text} is not in 1..={}", serve::MAX_DEPTH))
}
/// Failures of `vectura fingerprint`, each naming what it happened on.
#[derive(Debug, thiserror::Error)]
enum Failure {
/// The management port could not be reached or its TLS handshake failed.
#[error("{address}: {source}")]
Management {
address: String,
#[source]
source: tls::Error,
},
/// The data port could not be reached, greeted wrongly, or its TLS handshake failed.
#[error("{address}: {source}")]
Data {
address: String,
#[source]
source: nfc::Error,
},
/// The result could not be written.
#[error("writing to standard output: {0}")]
Stdout(#[source] io::Error),
}
/// Runs the command line `args` and returns the exit code.
pub fn run<I, T>(args: I) -> ExitCode
where
I: IntoIterator<Item = T>,
T: Into<OsString> + Clone,
{
let cli = match Cli::try_parse_from(args) {
Ok(cli) => cli,
Err(error) => return report_parse(&error),
};
match cli.command {
Command::Fingerprint(arguments) => match fingerprint(&arguments) {
Ok(()) => ExitCode::SUCCESS,
Err(failure) => {
eprintln!("vectura: {failure}");
ExitCode::from(EXIT_FAILURE)
}
},
Command::Serve(arguments) => serve_command(&arguments, std::env::var(PASSWORD_VARIABLE)),
}
}
/// Runs `serve` with the password as the environment yielded it.
///
/// A missing or empty password and an unreadable CA file are usage errors,
/// reported before any connection is made.
fn serve_command(arguments: &Serve, password: Result<String, VarError>) -> ExitCode {
let password = match password {
Ok(text) if !text.is_empty() => soap::Password::new(text),
_ => {
eprintln!("vectura: {PASSWORD_VARIABLE} is not set");
return ExitCode::from(EXIT_USAGE);
}
};
let trust = match trust(arguments) {
Ok(trust) => trust,
Err(text) => {
eprintln!("vectura: {text}");
return ExitCode::from(EXIT_USAGE);
}
};
if arguments.transport == nfc::Transport::Nfc {
// A warning standard error cannot take is dropped, as a transcript
// line is: the disk stream must not stop because its reader left.
let _ = writeln!(
io::stderr().lock(),
"vectura: warning: --transport nfc carries the disk in clear text after the ticket dialogue"
);
}
let target = serve::Target {
host: arguments.host.host.clone(),
port: arguments.host.port,
trust,
user: arguments.user.clone(),
vm_id: arguments.vm_id.clone(),
disk: arguments.disk.clone(),
depth: NonZeroUsize::from(arguments.depth),
compression: arguments.compression,
transport: arguments.transport,
transcript: Transcript::new(arguments.verbose),
};
let mut stdout = io::stdout().lock();
match serve::run(&target, &password, io::stdin(), &mut stdout) {
Ok(()) => ExitCode::SUCCESS,
Err(error) => {
eprintln!("vectura: {error}");
ExitCode::from(EXIT_FAILURE)
}
}
}
/// How `serve` judges the management port's certificate: pin, bundle, or system roots.
fn trust(arguments: &Serve) -> Result<tls::Trust, String> {
if let Some(pin) = &arguments.thumbprint {
return Ok(tls::Trust::Pin(pin.clone()));
}
match &arguments.ca_file {
Some(path) => read_anchors(path).map(tls::Trust::Anchors),
None => Ok(tls::Trust::System),
}
}
/// Reads every certificate of the PEM bundle at `path`.
fn read_anchors(path: &Path) -> Result<Vec<CertificateDer<'static>>, String> {
let bytes =
std::fs::read(path).map_err(|error| format!("cannot read {}: {error}", path.display()))?;
CertificateDer::pem_slice_iter(&bytes)
.collect::<Result<Vec<_>, _>>()
.map_err(|error| format!("cannot parse {}: {error}", path.display()))
}
/// Prints a parse outcome where it belongs and returns its exit code.
///
/// Usage errors go to standard error with exit 2; help and version go to
/// standard output with exit 0.
fn report_parse(error: &clap::Error) -> ExitCode {
if error.use_stderr() {
let text = error.to_string();
let text = text.strip_prefix(CLAP_ERROR_PREFIX).unwrap_or(&text);
eprintln!("vectura: {text}");
return ExitCode::from(EXIT_USAGE);
}
if error.print().is_ok() {
ExitCode::SUCCESS
} else {
ExitCode::from(EXIT_FAILURE)
}
}
/// Probes both ports, then prints their thumbprints, management port first.
fn fingerprint(arguments: &Fingerprint) -> Result<(), Failure> {
let HostPort { host, port } = &arguments.host;
let management = tls::connect(host, *port)
.and_then(|mut stream| tls::capture(&mut stream, host))
.map_err(|source| Failure::Management {
address: format!("{host}:{port}"),
source,
})?;
let data = nfc::capture(host, arguments.nfc_port).map_err(|source| Failure::Data {
address: format!("{host}:{}", arguments.nfc_port),
source,
})?;
let presented = [(*port, management), (arguments.nfc_port, data)];
let transcript = Transcript::new(arguments.verbose);
for (port, presented) in &presented {
transcript.tls(&port.to_string(), presented.version, presented.suite);
}
let mut stdout = io::stdout().lock();
for (port, presented) in &presented {
writeln!(
stdout,
"{port} sha256 {}",
tls::sha256(&presented.certificate)
)
.map_err(Failure::Stdout)?;
writeln!(stdout, "{port} sha1 {}", tls::sha1(&presented.certificate))
.map_err(Failure::Stdout)?;
}
Ok(())
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn serve_takes_the_nfc_transport_and_verbose_from_the_flags() {
let cli = Cli::try_parse_from([
"vectura",
"serve",
"--host",
"esxi.example",
"--user",
"root",
"--vm-id",
"3",
"--disk",
"d",
"--transport",
"nfc",
"--verbose",
])
.unwrap();
let Command::Serve(serve) = cli.command else {
panic!("serve was parsed");
};
assert_eq!(serve.transport, nfc::Transport::Nfc);
assert!(serve.verbose);
}
#[test]
fn the_data_port_defaults_to_902_and_verbose_is_off() {
let cli =
Cli::try_parse_from(["vectura", "fingerprint", "--host", "esxi.example"]).unwrap();
let Command::Fingerprint(fingerprint) = cli.command else {
panic!("fingerprint was parsed");
};
assert_eq!((fingerprint.host.port, fingerprint.nfc_port), (443, 902));
assert!(!fingerprint.verbose);
}
#[test]
fn serve_takes_its_target_from_the_flags_and_pins_with_a_prefixed_thumbprint() {
let cli = Cli::try_parse_from([
"vectura",
"serve",
"--host",
"esxi.example:8443",
"--user",
"root",
"--vm-id",
"3",
"--disk",
"[datastore1] vm/vm.vmdk",
"--thumbprint",
&format!("sha256:{}", "ab".repeat(32)),
])
.unwrap();
let Command::Serve(serve) = cli.command else {
panic!("serve was parsed");
};
assert_eq!(
(serve.host.host.as_str(), serve.host.port),
("esxi.example", 8443)
);
assert_eq!((serve.user.as_str(), serve.vm_id.as_str()), ("root", "3"));
assert_eq!(serve.disk, "[datastore1] vm/vm.vmdk");
assert!(serve.thumbprint.is_some());
assert!(matches!(trust(&serve), Ok(tls::Trust::Pin(_))));
assert_eq!(serve.transport, nfc::Transport::Nfcssl);
assert!(!serve.verbose);
}
#[test]
fn a_thumbprint_without_the_sha256_prefix_or_with_bad_hex_is_refused() {
let error = parse_thumbprint(&"ab".repeat(32)).unwrap_err();
assert!(error.ends_with("does not start with sha256:"), "{error}");
let error = parse_thumbprint("sha256:zz").unwrap_err();
assert!(
error.ends_with("is not a sha256 thumbprint of 64 hex digits"),
"{error}"
);
}
#[test]
fn thumbprint_and_ca_file_together_are_a_usage_error() {
let error = Cli::try_parse_from([
"vectura",
"serve",
"--host",
"h",
"--user",
"u",
"--vm-id",
"1",
"--disk",
"d",
"--thumbprint",
&format!("sha256:{}", "ab".repeat(32)),
"--ca-file",
"roots.pem",
])
.unwrap_err();
assert!(error.use_stderr());
}
#[test]
fn the_compression_defaults_to_skipz_and_takes_zlib_or_none() {
let base = [
"vectura", "serve", "--host", "h", "--user", "u", "--vm-id", "1", "--disk", "d",
];
let Command::Serve(serve) = Cli::try_parse_from(base).unwrap().command else {
panic!("serve was parsed");
};
assert_eq!(serve.compression, disk::Compression::Skipz);
for (name, expected) in [
("skipz", Some(disk::Compression::Skipz)),
("zlib", Some(disk::Compression::Zlib)),
("none", Some(disk::Compression::None)),
("gzip", None),
] {
let mut args = base.to_vec();
args.extend(["--compression", name]);
let parsed = Cli::try_parse_from(args)
.ok()
.and_then(|cli| match cli.command {
Command::Serve(serve) => Some(serve.compression),
Command::Fingerprint(_) => None,
});
assert_eq!(parsed, expected, "--compression {name}");
}
}
#[test]
fn the_depth_defaults_to_sixteen_and_takes_one_to_thirty_two() {
let base = [
"vectura", "serve", "--host", "h", "--user", "u", "--vm-id", "1", "--disk", "d",
];
let Command::Serve(serve) = Cli::try_parse_from(base).unwrap().command else {
panic!("serve was parsed");
};
assert_eq!(serve.depth.get(), 16);
for (depth, accepted) in [("1", true), ("32", true), ("0", false), ("33", false)] {
let mut args = base.to_vec();
args.extend(["--depth", depth]);
let parsed = Cli::try_parse_from(args);
assert_eq!(parsed.is_ok(), accepted, "--depth {depth}");
if let Ok(Cli {
command: Command::Serve(serve),
}) = parsed
{
assert_eq!(serve.depth.to_string(), depth);
}
}
}
fn serve_with(ca_file: Option<PathBuf>) -> Serve {
Serve {
host: parse_host("esxi.example").unwrap(),
user: "root".to_owned(),
vm_id: "3".to_owned(),
disk: "[ds] vm.vmdk".to_owned(),
thumbprint: None,
ca_file,
depth: parse_depth(DEFAULT_DEPTH).unwrap(),
compression: disk::Compression::Skipz,
transport: nfc::Transport::Nfcssl,
verbose: false,
}
}
#[test]
fn an_empty_password_is_a_usage_error_like_a_missing_one() {
let usage = ExitCode::from(EXIT_USAGE);
assert_eq!(serve_command(&serve_with(None), Ok(String::new())), usage);
assert_eq!(
serve_command(&serve_with(None), Err(VarError::NotPresent)),
usage
);
}
#[test]
fn without_a_pin_or_a_bundle_the_system_roots_apply() {
assert!(matches!(trust(&serve_with(None)), Ok(tls::Trust::System)));
}
#[test]
fn an_unreadable_ca_file_is_refused_naming_it() {
let error = trust(&serve_with(Some(PathBuf::from("/nonexistent/roots.pem")))).unwrap_err();
assert!(
error.starts_with("cannot read /nonexistent/roots.pem: "),
"{error}"
);
}
#[test]
fn a_pem_bundle_yields_its_certificates_and_a_broken_one_is_refused() {
let directory = std::env::temp_dir().join(format!("vectura-cli-{}", std::process::id()));
std::fs::create_dir_all(&directory).unwrap();
// The PEM reader does not check the DER inside, so two tiny blocks do.
let block = "-----BEGIN CERTIFICATE-----\nMAA=\n-----END CERTIFICATE-----\n";
let bundle = directory.join("roots.pem");
std::fs::write(&bundle, [block, block].concat()).unwrap();
let Ok(tls::Trust::Anchors(anchors)) = trust(&serve_with(Some(bundle))) else {
panic!("the bundle was read");
};
assert_eq!(anchors.len(), 2);
let broken = directory.join("broken.pem");
std::fs::write(
&broken,
"-----BEGIN CERTIFICATE-----\nnot base64\n-----END CERTIFICATE-----\n",
)
.unwrap();
let error = trust(&serve_with(Some(broken))).unwrap_err();
assert!(error.contains("cannot parse"), "{error}");
std::fs::remove_dir_all(&directory).unwrap();
}
#[test]
fn a_host_without_a_port_gets_the_management_port() {
let parsed = parse_host("esxi.example").unwrap();
assert_eq!((parsed.host.as_str(), parsed.port), ("esxi.example", 443));
}
#[test]
fn a_host_with_a_port_keeps_it() {
let parsed = parse_host("esxi.example:8443").unwrap();
assert_eq!((parsed.host.as_str(), parsed.port), ("esxi.example", 8443));
let parsed = parse_host("192.0.2.10:443").unwrap();
assert_eq!((parsed.host.as_str(), parsed.port), ("192.0.2.10", 443));
}
#[test]
fn a_bare_ipv6_address_is_a_host_with_the_default_port() {
let parsed = parse_host("2001:db8::10").unwrap();
assert_eq!((parsed.host.as_str(), parsed.port), ("2001:db8::10", 443));
}
#[test]
fn an_unparsable_port_or_host_is_refused_with_the_offending_text() {
let error = parse_host("esxi.example:notaport").unwrap_err();
assert!(error.starts_with("invalid port \"notaport\": "), "{error}");
let error = parse_host("esxi.example:70000").unwrap_err();
assert!(error.starts_with("invalid port \"70000\": "), "{error}");
let error = parse_host("not a host:443").unwrap_err();
assert!(
error.starts_with("invalid host \"not a host\": "),
"{error}"
);
let error = parse_host("").unwrap_err();
assert!(error.starts_with("invalid host \"\": "), "{error}");
}
}
@xen-orchestra/vmware-explorer/vectura/src/disk.rs
+840
View File
@@ -0,0 +1,840 @@
//! Splitting a client read into host reads and decoding the chunks that answer them.
//!
//! Everything here is a pure function over numbers and byte slices: no
//! socket, no clock, so it is tested without a host.
use std::fmt;
use std::io::Read as _;
use std::str::FromStr;
/// One mebibyte, the most a host read asks for.
///
/// Cutting on the disk's mebibyte boundaries is a choice made here, not a
/// rule of the host: a mebibyte is the read size the host was measured
/// with, at most sixteen chunks, and the block size the export will advertise
/// as preferred, so a client reading that way sends one host read per command.
pub const MIB: u64 = 1 << 20;
/// The 512-byte sector every host read is widened to.
///
/// Asking whole sectors is a choice made here, not a rule of the host: it
/// gives every host read the same shape whatever a client asks, and the
/// padding around the client's range is dropped when the chunks are placed.
pub const SECTOR: u64 = 512;
/// One host read: sector aligned, within one mebibyte, and the chunks received for it.
#[derive(Debug, PartialEq, Eq)]
pub struct Read {
/// The disk offset the read starts at.
pub offset: u64,
/// How many bytes it asks for.
pub length: u32,
/// The ranges within the read that arrived, as `(offset, end)` pairs.
received: Vec<(u32, u32)>,
}
/// The code of a chunk sent as stored: the wire holds its bytes.
///
/// A read request carries the code at offset 12 and every reply echoes it.
const NONE: u32 = 0;
/// The code of a chunk sent as one zlib stream.
const ZLIB: u32 = 1;
/// The code of a chunk sent as skipz, its runs of zeros left out.
///
/// The code between, 2, is fastlz, which is neither asked for nor decoded.
const SKIPZ: u32 = 3;
/// Every field of a skipz stream is a little-endian word this wide.
const WORD: usize = 4;
/// A skipz stream starts with the chunk length and a reserved word.
const SKIPZ_HEADER_LEN: usize = 2 * WORD;
/// A skipz segment starts with its offset within the chunk and its length.
const SEGMENT_HEADER_LEN: usize = 2 * WORD;
/// How the host is asked to send each chunk.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum Compression {
/// As stored.
None,
/// As one zlib stream per chunk.
Zlib,
/// As the runs worth sending, the zeros between them left out.
Skipz,
}
impl Compression {
/// The code the wire carries for this compression.
#[must_use]
pub const fn code(self) -> u32 {
match self {
Compression::None => NONE,
Compression::Zlib => ZLIB,
Compression::Skipz => SKIPZ,
}
}
}
impl FromStr for Compression {
type Err = String;
/// Parses `none`, `zlib` or `skipz`, naming the three on any other text.
fn from_str(text: &str) -> Result<Compression, String> {
match text {
"none" => Ok(Compression::None),
"zlib" => Ok(Compression::Zlib),
"skipz" => Ok(Compression::Skipz),
other => Err(format!("{other} is not none, zlib or skipz")),
}
}
}
impl fmt::Display for Compression {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
f.write_str(match self {
Compression::None => "none",
Compression::Zlib => "zlib",
Compression::Skipz => "skipz",
})
}
}
/// What one reply says about a chunk, in disk terms.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct Chunk {
/// The disk offset the host says the read started at.
pub base_offset: u64,
/// The length the host says the read asked for.
pub total_length: u32,
/// Where the chunk starts within the read.
pub offset: u32,
/// How many disk bytes the chunk holds.
pub length: u32,
/// The code of the compression the reply declares, echoed from the request.
pub compression: u32,
}
/// A chunk that does not fit its read or does not decode, with raw values.
#[derive(Debug, Clone, PartialEq, Eq, thiserror::Error)]
pub enum Error {
/// The chunk names another base offset than the read asked.
#[error("chunk: base offset {received} answers a read at offset {asked}")]
BaseOffset {
/// The offset the read asked.
asked: u64,
/// The base offset the chunk carried.
received: u64,
},
/// The chunk names another total length than the read asked.
#[error("chunk: total length {received} answers a read of {asked} bytes")]
TotalLength {
/// The length the read asked.
asked: u32,
/// The total length the chunk carried.
received: u32,
},
/// The chunk ends past the read.
#[error("chunk: {length} bytes at offset {offset} end past the {asked} bytes read")]
OutOfRange {
/// The chunk's offset within the read.
offset: u32,
/// The chunk's length.
length: u32,
/// The length the read asked.
asked: u32,
},
/// The chunk covers bytes another chunk already delivered.
#[error("chunk: {length} bytes at offset {offset} overlap a chunk already received")]
Overlap {
/// The chunk's offset within the read.
offset: u32,
/// The chunk's length.
length: u32,
},
/// The chunk came with a compression that is not decoded.
#[error(
"chunk: {length} bytes at offset {offset} came with compression {compression}, which is not decoded"
)]
Compression {
/// The chunk's offset within the read.
offset: u32,
/// The chunk's length.
length: u32,
/// The code the reply declares.
compression: u32,
},
/// The chunk came as stored, with another number of bytes than it holds.
#[error("chunk: {length} bytes at offset {offset} came as {wire} bytes on the wire")]
WireLength {
/// The chunk's offset within the read.
offset: u32,
/// The chunk's length.
length: u32,
/// The bytes it carried on the wire.
wire: usize,
},
/// The chunk came as a zlib stream that does not inflate.
#[error("chunk: {length} bytes at offset {offset}: the zlib stream does not inflate: {detail}")]
Zlib {
/// The chunk's offset within the read.
offset: u32,
/// The chunk's length.
length: u32,
/// What the inflater reported.
detail: String,
},
/// The chunk's zlib stream inflates to another number of bytes than it holds.
#[error("chunk: {length} bytes at offset {offset} inflate to {inflated} bytes")]
Inflated {
/// The chunk's offset within the read.
offset: u32,
/// The chunk's length.
length: u32,
/// The bytes the stream inflates to, counted up to one past the length.
inflated: usize,
},
/// The chunk came as a skipz stream that ends inside a header or a segment.
#[error(
"chunk: {length} bytes at offset {offset}: the skipz stream of {wire} bytes ends inside a header or a segment"
)]
SkipzShort {
/// The chunk's offset within the read.
offset: u32,
/// The chunk's length.
length: u32,
/// The bytes the stream holds.
wire: usize,
},
/// The chunk came as a skipz stream with a segment ending past the chunk.
#[error(
"chunk: {length} bytes at offset {offset}: a skipz segment of {segment_length} bytes at offset {segment_offset} ends past the chunk"
)]
SkipzSegment {
/// The chunk's offset within the read.
offset: u32,
/// The chunk's length.
length: u32,
/// The segment's offset within the chunk.
segment_offset: u32,
/// The segment's length.
segment_length: u32,
},
}
impl Chunk {
/// Decodes the chunk's `wire` bytes, as the reply declares them, into its bytes.
///
/// # Errors
/// When the compression is one that is not decoded, when the bytes do not
/// decode, or when they decode to another number of bytes than the chunk holds.
pub fn decode(&self, wire: &[u8]) -> Result<Vec<u8>, Error> {
match self.compression {
NONE => self.raw(wire),
ZLIB => self.inflate(wire),
SKIPZ => self.unskip(wire),
compression => Err(Error::Compression {
offset: self.offset,
length: self.length,
compression,
}),
}
}
/// The chunk sent as stored: the wire is its bytes.
fn raw(&self, wire: &[u8]) -> Result<Vec<u8>, Error> {
if wire.len() != index(self.length) {
return Err(Error::WireLength {
offset: self.offset,
length: self.length,
wire: wire.len(),
});
}
Ok(wire.to_vec())
}
/// The chunk sent as one zlib stream.
fn inflate(&self, wire: &[u8]) -> Result<Vec<u8>, Error> {
let mut bytes = Vec::new();
// One byte past the chunk: a stream running longer shows without
// inflating all of it.
flate2::read::ZlibDecoder::new(wire)
.take(u64::from(self.length) + 1)
.read_to_end(&mut bytes)
.map_err(|error| Error::Zlib {
offset: self.offset,
length: self.length,
detail: error.to_string(),
})?;
if bytes.len() != index(self.length) {
return Err(Error::Inflated {
offset: self.offset,
length: self.length,
inflated: bytes.len(),
});
}
Ok(bytes)
}
/// The chunk sent as skipz, or as stored when the host had nothing to skip.
///
/// A skipz stream starts with the chunk length and a reserved word, then
/// segments, each its offset within the chunk, its length and its bytes,
/// all little-endian. What no segment covers is zero. A stream whose
/// first word is not the chunk length is the chunk as stored.
fn unskip(&self, wire: &[u8]) -> Result<Vec<u8>, Error> {
if word(wire, 0) != Some(self.length) {
return self.raw(wire);
}
let short = || Error::SkipzShort {
offset: self.offset,
length: self.length,
wire: wire.len(),
};
let mut bytes = vec![0_u8; index(self.length)];
let mut rest = wire.get(SKIPZ_HEADER_LEN..).ok_or_else(short)?;
while !rest.is_empty() {
let (Some(at), Some(count)) = (word(rest, 0), word(rest, WORD)) else {
return Err(short());
};
let end = index(at)
.checked_add(index(count))
.filter(|end| *end <= bytes.len())
.ok_or(Error::SkipzSegment {
offset: self.offset,
length: self.length,
segment_offset: at,
segment_length: count,
})?;
let data_end = SEGMENT_HEADER_LEN + index(count);
let data = rest.get(SEGMENT_HEADER_LEN..data_end).ok_or_else(short)?;
bytes[index(at)..end].copy_from_slice(data);
rest = &rest[data_end..];
}
Ok(bytes)
}
}
/// The little-endian word at `at` of `bytes`, if all its bytes are there.
fn word(bytes: &[u8], at: usize) -> Option<u32> {
let bytes = bytes.get(at..at + WORD)?;
<[u8; WORD]>::try_from(bytes).ok().map(u32::from_le_bytes)
}
/// A wire length or offset as an index.
///
/// Every supported target has an index at least as wide as a word, so the
/// fallback never runs; it keeps the conversion free of a panic.
fn index(value: u32) -> usize {
usize::try_from(value).unwrap_or(usize::MAX)
}
/// Cuts the range of `length` bytes at `offset`, clipped to `capacity`, into host reads.
///
/// Each read stays within one [`MIB`] of the disk and is widened to
/// [`SECTOR`] boundaries, never past `capacity`. The reads come in disk
/// order and do not overlap.
#[must_use]
pub fn reads(offset: u64, length: u32, capacity: u64) -> Vec<Read> {
let end = offset.saturating_add(u64::from(length)).min(capacity);
if offset >= end {
return Vec::new();
}
(offset / MIB..=(end - 1) / MIB)
.map(|mebibyte| {
let start = offset.max(mebibyte * MIB);
let stop = end.min(mebibyte.saturating_add(1).saturating_mul(MIB));
let aligned_start = start / SECTOR * SECTOR;
let aligned_stop = stop.div_ceil(SECTOR).saturating_mul(SECTOR).min(capacity);
Read::new(
aligned_start,
u32::try_from(aligned_stop - aligned_start).unwrap_or(u32::MAX),
)
})
.collect()
}
impl Read {
/// A read of `length` bytes at `offset` with no chunk received yet.
#[must_use]
pub fn new(offset: u64, length: u32) -> Read {
Read {
offset,
length,
received: Vec::new(),
}
}
/// Checks `chunk` against the read and the chunks before it, then counts it.
///
/// # Errors
/// When the chunk answers another read, ends past this one, or overlaps
/// a chunk already accepted.
pub fn accept(&mut self, chunk: &Chunk) -> Result<(), Error> {
if chunk.base_offset != self.offset {
return Err(Error::BaseOffset {
asked: self.offset,
received: chunk.base_offset,
});
}
if chunk.total_length != self.length {
return Err(Error::TotalLength {
asked: self.length,
received: chunk.total_length,
});
}
let end = chunk
.offset
.checked_add(chunk.length)
.filter(|end| *end <= self.length)
.ok_or(Error::OutOfRange {
offset: chunk.offset,
length: chunk.length,
asked: self.length,
})?;
if self
.received
.iter()
.any(|(start, stop)| chunk.offset < *stop && *start < end)
{
return Err(Error::Overlap {
offset: chunk.offset,
length: chunk.length,
});
}
self.received.push((chunk.offset, end));
Ok(())
}
/// Whether the accepted chunks cover the whole read.
#[must_use]
pub fn is_complete(&self) -> bool {
self.received
.iter()
.map(|(start, stop)| stop - start)
.sum::<u32>()
== self.length
}
}
#[cfg(test)]
mod tests {
use super::*;
/// A mebibyte as a read length.
const MIB_LEN: u32 = 1 << 20;
fn read(offset: u64, length: u32) -> Read {
Read::new(offset, length)
}
fn chunk(offset: u32, length: u32) -> Chunk {
Chunk {
base_offset: 4096,
total_length: 2048,
offset,
length,
compression: NONE,
}
}
#[test]
fn a_range_inside_one_sector_becomes_that_whole_sector() {
assert_eq!(reads(700, 100, MIB), vec![read(512, 512)]);
}
#[test]
fn a_range_is_widened_to_the_sectors_it_touches() {
assert_eq!(reads(511, 2, MIB), vec![read(0, 1024)]);
assert_eq!(reads(512, 512, MIB), vec![read(512, 512)]);
}
#[test]
fn a_range_over_three_mebibytes_is_cut_on_the_boundaries() {
assert_eq!(
reads(MIB - 100, 2 * MIB_LEN + 200, 10 * MIB),
vec![
read(MIB - 512, 512),
read(MIB, MIB_LEN),
read(2 * MIB, MIB_LEN),
read(3 * MIB, 512),
]
);
}
#[test]
fn a_range_ending_on_a_boundary_stops_there() {
assert_eq!(reads(0, MIB_LEN, 10 * MIB), vec![read(0, MIB_LEN)]);
assert_eq!(reads(MIB - 1, 1, 10 * MIB), vec![read(MIB - 512, 512)]);
}
#[test]
fn a_range_is_clipped_to_the_capacity_and_never_widened_past_it() {
assert_eq!(
reads(MIB - 512, 4096, MIB + 100),
vec![read(MIB - 512, 512), read(MIB, 100)]
);
assert_eq!(reads(MIB, 1, MIB), vec![]);
assert_eq!(reads(u64::MAX, 1, u64::MAX), vec![]);
assert_eq!(
reads(u64::MAX - 1, 1, u64::MAX),
vec![read(u64::MAX - 511, 511)]
);
}
#[test]
fn an_empty_range_yields_no_read() {
assert_eq!(reads(4096, 0, MIB), vec![]);
assert_eq!(reads(700, 0, MIB), vec![]);
assert_eq!(reads(0, 0, 0), vec![]);
}
#[test]
fn chunks_in_any_order_complete_the_read_once_they_cover_it() {
let mut read = read(4096, 2048);
assert!(!read.is_complete());
read.accept(&chunk(1500, 548)).unwrap();
read.accept(&chunk(0, 700)).unwrap();
assert!(!read.is_complete());
read.accept(&chunk(700, 800)).unwrap();
assert!(read.is_complete());
}
#[test]
fn a_chunk_answering_another_read_is_refused_naming_both() {
let mut read = read(4096, 2048);
let error = read
.accept(&Chunk {
base_offset: 8192,
..chunk(0, 512)
})
.unwrap_err();
assert_eq!(
error,
Error::BaseOffset {
asked: 4096,
received: 8192
}
);
assert_eq!(
error.to_string(),
"chunk: base offset 8192 answers a read at offset 4096"
);
let error = read
.accept(&Chunk {
total_length: 1024,
..chunk(0, 512)
})
.unwrap_err();
assert_eq!(
error.to_string(),
"chunk: total length 1024 answers a read of 2048 bytes"
);
}
#[test]
fn a_chunk_ending_past_the_read_is_refused() {
let mut read = read(4096, 2048);
read.accept(&chunk(1536, 512)).unwrap();
assert_eq!(
read.accept(&chunk(1024, 1025)).unwrap_err().to_string(),
"chunk: 1025 bytes at offset 1024 end past the 2048 bytes read"
);
assert_eq!(
read.accept(&chunk(u32::MAX, 1)).unwrap_err(),
Error::OutOfRange {
offset: u32::MAX,
length: 1,
asked: 2048
}
);
}
#[test]
fn a_chunk_overlapping_one_already_received_is_refused() {
let mut read = read(4096, 2048);
read.accept(&chunk(512, 512)).unwrap();
assert_eq!(
read.accept(&chunk(1023, 1)).unwrap_err(),
Error::Overlap {
offset: 1023,
length: 1
}
);
assert_eq!(
read.accept(&chunk(0, 513)).unwrap_err().to_string(),
"chunk: 513 bytes at offset 0 overlap a chunk already received"
);
read.accept(&chunk(1024, 1024)).unwrap();
read.accept(&chunk(0, 512)).unwrap();
assert!(read.is_complete());
}
/// A chunk of `length` bytes at offset 0, sent with `compression`.
fn sent(length: u32, compression: u32) -> Chunk {
Chunk {
compression,
..chunk(0, length)
}
}
/// The bytes 0 to `length - 1`.
fn counting(length: u8) -> Vec<u8> {
(0..length).collect()
}
/// The bytes 0 to 15 as one zlib stream, the output of Python's `zlib.compress`.
const ZLIB_16: [u8; 24] = [
0x78, 0x9C, 0x63, 0x60, 0x64, 0x62, 0x66, 0x61, 0x65, 0x63, 0xE7, 0xE0, 0xE4, 0xE2, 0xE6,
0xE1, 0xE5, 0xE3, 0x07, 0x00, 0x02, 0xB8, 0x00, 0x79,
];
/// The bytes 0 to 14 as one zlib stream.
const ZLIB_15: [u8; 23] = [
0x78, 0x9C, 0x63, 0x60, 0x64, 0x62, 0x66, 0x61, 0x65, 0x63, 0xE7, 0xE0, 0xE4, 0xE2, 0xE6,
0xE1, 0xE5, 0x03, 0x00, 0x02, 0x3F, 0x00, 0x6A,
];
/// The bytes 0 to 16 as one zlib stream.
const ZLIB_17: [u8; 25] = [
0x78, 0x9C, 0x63, 0x60, 0x64, 0x62, 0x66, 0x61, 0x65, 0x63, 0xE7, 0xE0, 0xE4, 0xE2, 0xE6,
0xE1, 0xE5, 0xE3, 0x17, 0x00, 0x00, 0x03, 0x41, 0x00, 0x89,
];
/// A skipz stream for a `length`-byte chunk holding `segments`, an offset and bytes each.
fn skipz(length: u32, segments: &[(u32, &[u8])]) -> Vec<u8> {
let mut wire = length.to_le_bytes().to_vec();
wire.extend_from_slice(&0_u32.to_le_bytes());
for (offset, data) in segments {
wire.extend_from_slice(&offset.to_le_bytes());
wire.extend_from_slice(&u32::try_from(data.len()).unwrap().to_le_bytes());
wire.extend_from_slice(data);
}
wire
}
#[test]
fn the_compression_codes_are_none_zero_zlib_one_and_skipz_three() {
assert_eq!(Compression::None.code(), 0);
assert_eq!(Compression::Zlib.code(), 1);
assert_eq!(Compression::Skipz.code(), 3);
}
#[test]
fn a_compression_is_parsed_from_its_name_and_prints_as_it() {
for (name, compression) in [
("none", Compression::None),
("zlib", Compression::Zlib),
("skipz", Compression::Skipz),
] {
assert_eq!(name.parse::<Compression>(), Ok(compression));
assert_eq!(compression.to_string(), name);
}
assert_eq!(
"gzip".parse::<Compression>(),
Err("gzip is not none, zlib or skipz".to_owned())
);
assert!("Skipz".parse::<Compression>().is_err());
}
#[test]
fn a_chunk_sent_as_stored_is_its_wire_bytes() {
assert_eq!(sent(3, NONE).decode(b"abc"), Ok(b"abc".to_vec()));
}
#[test]
fn a_chunk_sent_as_stored_with_another_number_of_bytes_is_refused_naming_both() {
let error = sent(512, NONE).decode(&[0; 511]).unwrap_err();
assert_eq!(
error,
Error::WireLength {
offset: 0,
length: 512,
wire: 511
}
);
assert_eq!(
error.to_string(),
"chunk: 512 bytes at offset 0 came as 511 bytes on the wire"
);
assert_eq!(
sent(512, NONE).decode(&[0; 513]),
Err(Error::WireLength {
offset: 0,
length: 512,
wire: 513
})
);
}
#[test]
fn a_zlib_chunk_is_inflated() {
assert_eq!(sent(16, ZLIB).decode(&ZLIB_16), Ok(counting(16)));
}
#[test]
fn a_zlib_stream_that_does_not_inflate_is_refused_with_the_reason() {
let mut bad_header = ZLIB_16;
bad_header[0] = 0x00;
let error = sent(16, ZLIB).decode(&bad_header).unwrap_err();
assert!(
matches!(
error,
Error::Zlib {
offset: 0,
length: 16,
..
}
),
"{error:?}"
);
let message = error.to_string();
assert!(
message.starts_with("chunk: 16 bytes at offset 0: the zlib stream does not inflate: "),
"{message}"
);
let mut bad_checksum = ZLIB_16;
bad_checksum[ZLIB_16.len() - 1] ^= 0xFF;
assert!(matches!(
sent(16, ZLIB).decode(&bad_checksum),
Err(Error::Zlib { .. })
));
}
#[test]
fn a_zlib_stream_inflating_to_another_number_of_bytes_is_refused_naming_both() {
let error = sent(16, ZLIB).decode(&ZLIB_15).unwrap_err();
assert_eq!(
error,
Error::Inflated {
offset: 0,
length: 16,
inflated: 15
}
);
assert_eq!(
error.to_string(),
"chunk: 16 bytes at offset 0 inflate to 15 bytes"
);
assert_eq!(
sent(16, ZLIB).decode(&ZLIB_17),
Err(Error::Inflated {
offset: 0,
length: 16,
inflated: 17
})
);
}
#[test]
fn a_skipz_chunk_without_segments_is_all_zeros() {
assert_eq!(
sent(2048, SKIPZ).decode(&skipz(2048, &[])),
Ok(vec![0; 2048])
);
}
#[test]
fn a_skipz_segment_covering_the_chunk_is_the_whole_chunk() {
let data = counting(16);
assert_eq!(sent(16, SKIPZ).decode(&skipz(16, &[(0, &data)])), Ok(data));
}
#[test]
fn the_reserved_word_of_a_skipz_stream_is_ignored() {
let data = counting(16);
let mut wire = skipz(16, &[(0, &data)]);
wire[WORD..SKIPZ_HEADER_LEN].copy_from_slice(&u32::MAX.to_le_bytes());
assert_eq!(sent(16, SKIPZ).decode(&wire), Ok(data));
}
#[test]
fn skipz_segments_at_both_ends_are_placed_with_zeros_between() {
let wire = skipz(16, &[(0, b"ab"), (13, b"xyz")]);
let mut expected = vec![0; 16];
expected[..2].copy_from_slice(b"ab");
expected[13..].copy_from_slice(b"xyz");
assert_eq!(sent(16, SKIPZ).decode(&wire), Ok(expected));
}
#[test]
fn a_skipz_chunk_whose_first_word_is_not_its_length_is_taken_as_stored() {
let data = counting(16);
assert_eq!(sent(16, SKIPZ).decode(&data), Ok(data.clone()));
assert_eq!(
sent(16, SKIPZ).decode(&data[..15]),
Err(Error::WireLength {
offset: 0,
length: 16,
wire: 15
})
);
assert_eq!(sent(3, SKIPZ).decode(b"abc"), Ok(b"abc".to_vec()));
}
#[test]
fn a_skipz_stream_cut_inside_its_header_or_a_segment_is_refused() {
let short = |wire: &[u8]| Error::SkipzShort {
offset: 0,
length: 16,
wire: wire.len(),
};
let header = skipz(16, &[]);
assert_eq!(
sent(16, SKIPZ).decode(&header[..6]),
Err(short(&header[..6]))
);
let segment = skipz(16, &[(4, b"abcd")]);
assert_eq!(
sent(16, SKIPZ).decode(&segment[..12]),
Err(short(&segment[..12]))
);
assert_eq!(
sent(16, SKIPZ).decode(&segment[..18]),
Err(short(&segment[..18]))
);
assert_eq!(
short(&header[..6]).to_string(),
"chunk: 16 bytes at offset 0: the skipz stream of 6 bytes ends inside a header or a segment"
);
}
#[test]
fn a_skipz_segment_ending_past_the_chunk_is_refused_naming_it() {
let error = sent(16, SKIPZ)
.decode(&skipz(16, &[(13, b"abcd")]))
.unwrap_err();
assert_eq!(
error,
Error::SkipzSegment {
offset: 0,
length: 16,
segment_offset: 13,
segment_length: 4
}
);
assert_eq!(
error.to_string(),
"chunk: 16 bytes at offset 0: a skipz segment of 4 bytes at offset 13 ends past the chunk"
);
let mut wrapping = skipz(16, &[]);
wrapping.extend_from_slice(&u32::MAX.to_le_bytes());
wrapping.extend_from_slice(&2_u32.to_le_bytes());
wrapping.extend_from_slice(b"ab");
assert!(matches!(
sent(16, SKIPZ).decode(&wrapping),
Err(Error::SkipzSegment { .. })
));
}
#[test]
fn a_chunk_with_a_compression_that_is_not_decoded_is_refused_naming_it() {
let error = sent(16, 2).decode(&[0; 16]).unwrap_err();
assert_eq!(
error,
Error::Compression {
offset: 0,
length: 16,
compression: 2
}
);
assert_eq!(
error.to_string(),
"chunk: 16 bytes at offset 0 came with compression 2, which is not decoded"
);
}
}
@xen-orchestra/vmware-explorer/vectura/src/lib.rs
+15
View File
@@ -0,0 +1,15 @@
//! Vectura reads VMware virtual disks over the NFC protocol.
//!
//! `vectura serve` exports one virtual disk, read-only, as an NBD server on
//! its own standard input and output; `vectura fingerprint` prints the
//! certificate thumbprints a host presents on its management and data ports,
//! so an operator can pin them.
pub mod cli;
pub mod disk;
pub mod nbd;
pub mod nfc;
pub mod serve;
pub mod soap;
pub mod tls;
pub mod transcript;
@xen-orchestra/vmware-explorer/vectura/src/main.rs
+7
View File
@@ -0,0 +1,7 @@
//! The `vectura` command-line entry point.
use std::process::ExitCode;
fn main() -> ExitCode {
vectura::cli::run(std::env::args_os())
}
@xen-orchestra/vmware-explorer/vectura/src/nbd.rs
+867
View File
@@ -0,0 +1,867 @@
//! A fixed-newstyle NBD server over two streams: negotiation and requests.
//!
//! Layouts and constants follow the NBD protocol document
//! (<https://github.com/NetworkBlockDevice/nbd/blob/master/doc/proto.md>);
//! every multi-byte field is big-endian. `NBD_OPT_EXPORT_NAME`,
//! `NBD_OPT_GO`, `NBD_OPT_INFO` and `NBD_OPT_ABORT` are served; every other
//! option is answered `NBD_REP_ERR_UNSUP`. The export is read-only: reads
//! are served, every other request is refused with the error it deserves.
use std::io::{self, Read, Write};
use crate::transcript::Transcript;
/// The initial magic, the ASCII of `NBDMAGIC`.
pub const NBDMAGIC: u64 = 0x4E42_444D_4147_4943;
/// The option magic, the ASCII of `IHAVEOPT`, opening the greeting and every option.
pub const IHAVEOPT: u64 = 0x4948_4156_454F_5054;
/// The magic opening every option reply.
pub const REPLY_MAGIC: u64 = 0x0003_e889_0455_65a9;
/// Handshake flag: the server speaks fixed newstyle.
pub const FLAG_FIXED_NEWSTYLE: u16 = 1;
/// Handshake flag: the server offers to skip the 124 zero bytes after the export size.
pub const FLAG_NO_ZEROES: u16 = 2;
/// Client flag: the client speaks fixed newstyle; required.
pub const FLAG_C_FIXED_NEWSTYLE: u32 = 1;
/// Client flag: the client agrees to skip the 124 zero bytes.
pub const FLAG_C_NO_ZEROES: u32 = 2;
/// The option that names the export and enters transmission at once.
pub const OPT_EXPORT_NAME: u32 = 1;
/// The option that ends the negotiation without an export.
pub const OPT_ABORT: u32 = 2;
/// The option asking about an export while staying in negotiation.
pub const OPT_INFO: u32 = 6;
/// The option asking about an export and entering transmission on success.
pub const OPT_GO: u32 = 7;
/// The reply accepting an option, or closing a run of info replies.
pub const REP_ACK: u32 = 1;
/// The reply carrying one piece of information about the export.
pub const REP_INFO: u32 = 3;
/// The reply refusing an option the server does not serve.
pub const REP_ERR_UNSUP: u32 = 0x8000_0001;
/// The reply refusing an option whose data is malformed.
pub const REP_ERR_INVALID: u32 = 0x8000_0003;
/// The info carrying the export size and the transmission flags.
pub const INFO_EXPORT: u16 = 0;
/// The info naming the minimum, preferred and maximum request sizes.
pub const INFO_BLOCK_SIZE: u16 = 3;
/// The smallest request advertised: one sector, the unit the host reads in.
///
/// Advertised only; a request off a sector is still served, since the host
/// reads are widened to sectors anyway.
pub const MIN_BLOCK_SIZE: u32 = 512;
/// The request size advertised as efficient: one host read, one mebibyte.
pub const PREFERRED_BLOCK_SIZE: u32 = 1 << 20;
/// The longest read served, 32 MiB: the protocol's baseline maximum.
///
/// A reply is held whole in memory, so a longer read is refused before its
/// buffer is allocated.
pub const MAX_PAYLOAD: u32 = 1 << 25;
/// Transmission flag: the flags field is meaningful; always set.
pub const FLAG_HAS_FLAGS: u16 = 1;
/// Transmission flag: the export cannot be written.
pub const FLAG_READ_ONLY: u16 = 2;
/// What the greeting offers: [`FLAG_FIXED_NEWSTYLE`] and [`FLAG_NO_ZEROES`].
const HANDSHAKE_FLAGS: u16 = 0b11;
/// What every export reports: [`FLAG_HAS_FLAGS`] and [`FLAG_READ_ONLY`].
const EXPORT_FLAGS: u16 = 0b11;
/// The magic opening every request.
pub const REQUEST_MAGIC: u32 = 0x2560_9513;
/// The magic opening every simple reply.
pub const SIMPLE_REPLY_MAGIC: u32 = 0x6744_6698;
/// The request for `length` bytes at `offset`; its reply carries them after the header.
pub const CMD_READ: u16 = 0;
/// The request to write the `length` bytes that follow it; refused, the bytes dropped.
pub const CMD_WRITE: u16 = 1;
/// The request that ends the session; it gets no reply.
pub const CMD_DISC: u16 = 2;
/// The request to discard a range; refused.
pub const CMD_TRIM: u16 = 4;
/// The request to zero a range; refused.
pub const CMD_WRITE_ZEROES: u16 = 6;
/// The error a write of any kind is answered with: the export is read-only.
pub const EPERM: u32 = 1;
/// The error a read the server could not complete is answered with.
pub const EIO: u32 = 5;
/// The error a request the server cannot serve is answered with.
pub const EINVAL: u32 = 22;
/// The most option data accepted: an export name and a short info list.
///
/// Anything longer is a protocol failure rather than an allocation.
pub const MAX_OPTION_LEN: u32 = 4096;
/// Length of the greeting: two magics and the handshake flags.
pub const GREETING_LEN: usize = 18;
/// Length of an option header: magic, option and data length.
pub const OPTION_LEN: usize = 16;
/// Length of an option reply without data.
pub const OPTION_REPLY_LEN: usize = 20;
/// Zero bytes after the export size and flags, unless `NO_ZEROES` was agreed.
pub const ZEROES_LEN: usize = 124;
/// Length of a request: magic, flags, type, cookie, offset, length.
pub const REQUEST_LEN: usize = 28;
/// Length of a simple reply header: magic, error, cookie.
pub const SIMPLE_REPLY_LEN: usize = 16;
/// Length of the export info: type, size, transmission flags.
pub const INFO_EXPORT_LEN: usize = 12;
/// Length of the block-size info: type and the three sizes.
pub const INFO_BLOCK_SIZE_LEN: usize = 14;
/// Errors of the NBD side, each a protocol violation or a lost client.
#[derive(Debug, thiserror::Error)]
pub enum Error {
/// A read from or write to the client failed.
#[error("nbd: {0}")]
Io(#[source] io::Error),
/// The client closed its end before sending `NBD_CMD_DISC`.
#[error("the nbd client left without disconnecting")]
Left,
/// The client does not speak fixed newstyle.
#[error("nbd: the client flags {0:#x} lack NBD_FLAG_C_FIXED_NEWSTYLE")]
NotFixedNewstyle(u32),
/// An option did not start with the option magic.
#[error("nbd: option magic {0:#018x} is not IHAVEOPT")]
OptionMagic(u64),
/// An option carries more data than [`MAX_OPTION_LEN`].
#[error("nbd: option {option} carries {length} bytes, more than {MAX_OPTION_LEN}")]
OptionTooLong {
/// The option.
option: u32,
/// Its announced data length.
length: u32,
},
/// A request did not start with the request magic.
#[error("nbd: request magic {0:#010x} is not NBD_REQUEST_MAGIC")]
RequestMagic(u32),
}
/// One request read from the client.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct Request {
/// The command flags.
pub flags: u16,
/// The command type, such as [`CMD_DISC`].
pub kind: u16,
/// The cookie a reply echoes.
pub cookie: u64,
/// The offset in the export.
pub offset: u64,
/// The length of the range.
pub length: u32,
}
/// The header of an option the client sent.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct OptionHeader {
/// The option.
pub option: u32,
/// The length of the data that follows.
pub length: u32,
}
/// How a negotiation ended.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum Negotiated {
/// The client took the export; transmission follows.
Export,
/// The client aborted; nothing follows.
Abort,
}
/// The greeting: both magics and the handshake flags.
#[must_use]
pub fn greeting() -> [u8; GREETING_LEN] {
let mut bytes = [0_u8; GREETING_LEN];
bytes[..8].copy_from_slice(&NBDMAGIC.to_be_bytes());
bytes[8..16].copy_from_slice(&IHAVEOPT.to_be_bytes());
bytes[16..].copy_from_slice(&HANDSHAKE_FLAGS.to_be_bytes());
bytes
}
/// Reads the client flags and requires fixed newstyle.
///
/// # Errors
/// When [`FLAG_C_FIXED_NEWSTYLE`] is not set.
pub fn client_flags(bytes: &[u8; 4]) -> Result<u32, Error> {
let flags = u32::from_be_bytes(*bytes);
if flags & FLAG_C_FIXED_NEWSTYLE == 0 {
return Err(Error::NotFixedNewstyle(flags));
}
Ok(flags)
}
/// Decodes an option header.
///
/// # Errors
/// When the magic is not [`IHAVEOPT`] or the data is longer than [`MAX_OPTION_LEN`].
pub fn option_header(bytes: &[u8; OPTION_LEN]) -> Result<OptionHeader, Error> {
let magic = u64::from_be_bytes([
bytes[0], bytes[1], bytes[2], bytes[3], bytes[4], bytes[5], bytes[6], bytes[7],
]);
if magic != IHAVEOPT {
return Err(Error::OptionMagic(magic));
}
let header = OptionHeader {
option: u32::from_be_bytes([bytes[8], bytes[9], bytes[10], bytes[11]]),
length: u32::from_be_bytes([bytes[12], bytes[13], bytes[14], bytes[15]]),
};
if header.length > MAX_OPTION_LEN {
return Err(Error::OptionTooLong {
option: header.option,
length: header.length,
});
}
Ok(header)
}
/// An option reply of type `reply` to `option`, carrying `data`.
#[must_use]
pub fn option_reply(option: u32, reply: u32, data: &[u8]) -> Vec<u8> {
let mut bytes = Vec::with_capacity(OPTION_REPLY_LEN + data.len());
bytes.extend_from_slice(&REPLY_MAGIC.to_be_bytes());
bytes.extend_from_slice(&option.to_be_bytes());
bytes.extend_from_slice(&reply.to_be_bytes());
let length = u32::try_from(data.len()).unwrap_or(u32::MAX);
bytes.extend_from_slice(&length.to_be_bytes());
bytes.extend_from_slice(data);
bytes
}
/// Whether `NBD_OPT_INFO` or `NBD_OPT_GO` data holds a name and an info list exactly.
///
/// Neither is read: there is one disk, and both infos are sent whatever the
/// list asks.
fn is_info_request(data: &[u8]) -> bool {
let Some((length, rest)) = data.split_first_chunk::<4>() else {
return false;
};
let Ok(length) = usize::try_from(u32::from_be_bytes(*length)) else {
return false;
};
let Some((_, rest)) = rest.split_at_checked(length) else {
return false;
};
let Some((count, rest)) = rest.split_first_chunk::<2>() else {
return false;
};
rest.len() == usize::from(u16::from_be_bytes(*count)) * 2
}
/// The export info: size and the read-only transmission flags.
#[must_use]
pub fn export_info(size: u64) -> [u8; INFO_EXPORT_LEN] {
let mut bytes = [0_u8; INFO_EXPORT_LEN];
bytes[..2].copy_from_slice(&INFO_EXPORT.to_be_bytes());
bytes[2..10].copy_from_slice(&size.to_be_bytes());
bytes[10..].copy_from_slice(&EXPORT_FLAGS.to_be_bytes());
bytes
}
/// The block-size info: minimum, preferred and maximum request sizes.
#[must_use]
pub fn block_size_info() -> [u8; INFO_BLOCK_SIZE_LEN] {
let mut bytes = [0_u8; INFO_BLOCK_SIZE_LEN];
bytes[..2].copy_from_slice(&INFO_BLOCK_SIZE.to_be_bytes());
bytes[2..6].copy_from_slice(&MIN_BLOCK_SIZE.to_be_bytes());
bytes[6..10].copy_from_slice(&PREFERRED_BLOCK_SIZE.to_be_bytes());
bytes[10..].copy_from_slice(&MAX_PAYLOAD.to_be_bytes());
bytes
}
/// Both infos then the ack, answering a well-formed `NBD_OPT_INFO` or `NBD_OPT_GO`.
///
/// The client's info list is not consulted: the export info is mandatory,
/// the block-size info is sent whether or not it was asked, since the
/// limits hold either way, and any other type is ignored.
fn info_replies(option: u32, size: u64) -> Vec<u8> {
let mut bytes = option_reply(option, REP_INFO, &export_info(size));
bytes.extend(option_reply(option, REP_INFO, &block_size_info()));
bytes.extend(option_reply(option, REP_ACK, &[]));
bytes
}
/// The reply to one option, its outcome if it is final, and a transcript word.
///
/// The export name is ignored: there is one disk, whatever the client calls it.
fn answer(
option: u32,
data: &[u8],
size: u64,
no_zeroes: bool,
) -> (Vec<u8>, Option<Negotiated>, &'static str) {
match option {
OPT_EXPORT_NAME => (
export_reply(size, no_zeroes),
Some(Negotiated::Export),
"export",
),
OPT_ABORT => (
option_reply(option, REP_ACK, &[]),
Some(Negotiated::Abort),
"acknowledged",
),
OPT_INFO | OPT_GO if is_info_request(data) => (
info_replies(option, size),
(option == OPT_GO).then_some(Negotiated::Export),
"info",
),
OPT_INFO | OPT_GO => (option_reply(option, REP_ERR_INVALID, &[]), None, "invalid"),
_ => (
option_reply(option, REP_ERR_UNSUP, &[]),
None,
"unsupported",
),
}
}
/// The reply to `NBD_OPT_EXPORT_NAME`: size, read-only flags, then the zeros unless skipped.
#[must_use]
pub fn export_reply(size: u64, no_zeroes: bool) -> Vec<u8> {
let mut bytes = size.to_be_bytes().to_vec();
bytes.extend_from_slice(&EXPORT_FLAGS.to_be_bytes());
if !no_zeroes {
bytes.resize(bytes.len() + ZEROES_LEN, 0);
}
bytes
}
/// Decodes a request.
///
/// # Errors
/// When the magic is not [`REQUEST_MAGIC`].
pub fn request(bytes: &[u8; REQUEST_LEN]) -> Result<Request, Error> {
let magic = u32::from_be_bytes([bytes[0], bytes[1], bytes[2], bytes[3]]);
if magic != REQUEST_MAGIC {
return Err(Error::RequestMagic(magic));
}
let long = |from: usize| {
bytes[from..from + 8]
.try_into()
.map(u64::from_be_bytes)
.unwrap_or_default()
};
Ok(Request {
flags: u16::from_be_bytes([bytes[4], bytes[5]]),
kind: u16::from_be_bytes([bytes[6], bytes[7]]),
cookie: long(8),
offset: long(16),
length: u32::from_be_bytes([bytes[24], bytes[25], bytes[26], bytes[27]]),
})
}
/// A simple reply header echoing `cookie`, with `error` zero on success.
#[must_use]
pub fn simple_reply(cookie: u64, error: u32) -> [u8; SIMPLE_REPLY_LEN] {
let mut bytes = [0_u8; SIMPLE_REPLY_LEN];
bytes[..4].copy_from_slice(&SIMPLE_REPLY_MAGIC.to_be_bytes());
bytes[4..8].copy_from_slice(&error.to_be_bytes());
bytes[8..].copy_from_slice(&cookie.to_be_bytes());
bytes
}
/// The error `request` is answered with, or `None` for a read to serve.
///
/// A command flag, a type that is neither a read nor a write of some kind,
/// or a read longer than [`MAX_PAYLOAD`] or reaching past `size` is
/// [`EINVAL`]; a write, a write of zeroes or a trim is [`EPERM`], the export
/// being read-only. `NBD_CMD_DISC` gets no reply and must be handled first.
#[must_use]
pub fn refusal(request: &Request, size: u64) -> Option<u32> {
if request.flags != 0 {
return Some(EINVAL);
}
let end = request.offset.checked_add(u64::from(request.length));
match request.kind {
CMD_WRITE | CMD_WRITE_ZEROES | CMD_TRIM => Some(EPERM),
CMD_READ if request.length <= MAX_PAYLOAD && end.is_some_and(|end| end <= size) => None,
_ => Some(EINVAL),
}
}
/// Negotiates with the client until it takes the export or aborts.
///
/// The export is `size` bytes and read-only, whatever name the client gives.
///
/// # Errors
/// When a stream fails, the client leaves, or it breaks the protocol.
pub fn negotiate<R: Read, W: Write>(
input: &mut R,
output: &mut W,
size: u64,
transcript: Transcript,
) -> Result<Negotiated, Error> {
send(output, &greeting())?;
let mut flags = [0_u8; 4];
receive(input, &mut flags)?;
let no_zeroes = client_flags(&flags)? & FLAG_C_NO_ZEROES != 0;
loop {
let mut header = [0_u8; OPTION_LEN];
receive(input, &mut header)?;
let header = option_header(&header)?;
let mut data = vec![0_u8; usize::try_from(header.length).unwrap_or_default()];
receive(input, &mut data)?;
let (reply, outcome, word) = answer(header.option, &data, size, no_zeroes);
transcript.option(header.option, word);
send(output, &reply)?;
if let Some(outcome) = outcome {
if outcome == Negotiated::Export {
transcript.export(size);
}
return Ok(outcome);
}
}
}
/// Reads the next request; the payload of a write is drained and dropped.
///
/// # Errors
/// When the stream fails, the client leaves, or the magic is wrong.
pub fn next_request<R: Read>(input: &mut R) -> Result<Request, Error> {
let mut bytes = [0_u8; REQUEST_LEN];
receive(input, &mut bytes)?;
let request = request(&bytes)?;
if request.kind == CMD_WRITE {
let length = u64::from(request.length);
let drained =
io::copy(&mut input.by_ref().take(length), &mut io::sink()).map_err(Error::Io)?;
if drained < length {
return Err(Error::Left);
}
}
Ok(request)
}
/// Writes `bytes` to the client and flushes.
///
/// # Errors
/// When the write fails; a closed pipe counts as the client leaving.
pub fn send<W: Write>(output: &mut W, bytes: &[u8]) -> Result<(), Error> {
output
.write_all(bytes)
.and_then(|()| output.flush())
.map_err(|error| match error.kind() {
io::ErrorKind::BrokenPipe => Error::Left,
_ => Error::Io(error),
})
}
fn receive<R: Read>(input: &mut R, buffer: &mut [u8]) -> Result<(), Error> {
input
.read_exact(buffer)
.map_err(|error| match error.kind() {
io::ErrorKind::UnexpectedEof => Error::Left,
_ => Error::Io(error),
})
}
#[cfg(test)]
mod tests {
use std::io::Cursor;
use super::*;
const QUIET: Transcript = Transcript::new(false);
#[test]
fn the_greeting_is_the_two_magics_then_the_handshake_flags_big_endian() {
assert_eq!(greeting(), *b"NBDMAGICIHAVEOPT\x00\x03");
assert_eq!(HANDSHAKE_FLAGS, FLAG_FIXED_NEWSTYLE | FLAG_NO_ZEROES);
}
#[test]
fn client_flags_require_fixed_newstyle() {
assert_eq!(client_flags(&[0, 0, 0, 3]).unwrap(), 3);
let error = client_flags(&[0, 0, 0, 2]).unwrap_err();
assert_eq!(
error.to_string(),
"nbd: the client flags 0x2 lack NBD_FLAG_C_FIXED_NEWSTYLE"
);
}
#[test]
fn an_option_header_round_trips_and_a_wrong_magic_is_refused() {
let mut bytes = *b"IHAVEOPT\x00\x00\x00\x01\x00\x00\x00\x05";
assert_eq!(
option_header(&bytes).unwrap(),
OptionHeader {
option: 1,
length: 5
}
);
bytes[0] = b'X';
let error = option_header(&bytes).unwrap_err();
assert_eq!(
error.to_string(),
"nbd: option magic 0x58484156454f5054 is not IHAVEOPT"
);
}
#[test]
fn an_option_at_the_limit_passes_and_a_longer_one_is_refused_naming_it() {
let mut bytes = *b"IHAVEOPT\x00\x00\x00\x07\x00\x00\x00\x00";
bytes[12..].copy_from_slice(&MAX_OPTION_LEN.to_be_bytes());
assert_eq!(option_header(&bytes).unwrap().length, MAX_OPTION_LEN);
bytes[12..].copy_from_slice(&(MAX_OPTION_LEN + 1).to_be_bytes());
let error = option_header(&bytes).unwrap_err();
assert_eq!(
error.to_string(),
"nbd: option 7 carries 4097 bytes, more than 4096"
);
}
#[test]
fn an_option_reply_carries_the_reply_magic_the_option_the_type_and_its_data() {
let reply = option_reply(7, REP_ERR_UNSUP, &[]);
assert_eq!(&reply[..8], &[0, 3, 0xe8, 0x89, 0x04, 0x55, 0x65, 0xa9]);
assert_eq!(&reply[8..12], &[0, 0, 0, 7]);
assert_eq!(&reply[12..16], &[0x80, 0, 0, 1]);
assert_eq!(&reply[16..], &[0, 0, 0, 0]);
let reply = option_reply(OPT_GO, REP_INFO, b"abc");
assert_eq!(reply.len(), OPTION_REPLY_LEN + 3);
assert_eq!(&reply[12..16], &[0, 0, 0, 3]);
assert_eq!(&reply[16..20], &[0, 0, 0, 3]);
assert_eq!(&reply[20..], b"abc");
}
#[test]
fn an_info_request_is_well_formed_when_its_name_and_its_list_fill_the_data() {
assert!(is_info_request(
b"\x00\x00\x00\x04disk\x00\x02\x00\x03\x00\x00"
));
assert!(is_info_request(b"\x00\x00\x00\x00\x00\x00"));
}
#[test]
fn an_info_request_whose_name_or_list_does_not_fit_the_data_is_malformed() {
for data in [
&b""[..],
b"\x00\x00\x00\x00",
b"\x00\x00\x00\x05disk\x00\x00",
b"\x00\x00\x00\x04disk\x00\x01",
b"\x00\x00\x00\x04disk\x00\x00\x00",
b"\x00\x00\x00\x04disk\x00\x00\x00\x03",
] {
assert!(!is_info_request(data), "{data:?}");
}
}
#[test]
fn the_export_info_holds_the_type_the_size_and_the_read_only_flags() {
let info = export_info(0x0001_0000_0000);
assert_eq!(&info[..2], &[0, 0]);
assert_eq!(&info[2..10], &[0, 0, 0, 1, 0, 0, 0, 0]);
assert_eq!(&info[10..], &[0, 3]);
}
#[test]
fn the_block_size_info_advertises_a_sector_a_mebibyte_and_thirty_two_mebibytes() {
let info = block_size_info();
assert_eq!(&info[..2], &[0, 3]);
assert_eq!(&info[2..6], &[0, 0, 2, 0]);
assert_eq!(&info[6..10], &[0, 0x10, 0, 0]);
assert_eq!(&info[10..], &[2, 0, 0, 0]);
}
#[test]
fn the_export_reply_holds_the_size_the_read_only_flags_and_the_zeros() {
let reply = export_reply(0x0001_0000_0000, false);
assert_eq!(reply.len(), 8 + 2 + ZEROES_LEN);
assert_eq!(&reply[..8], &[0, 0, 0, 1, 0, 0, 0, 0]);
assert_eq!(&reply[8..10], &[0, 3]);
assert_eq!(EXPORT_FLAGS, FLAG_HAS_FLAGS | FLAG_READ_ONLY);
assert!(reply[10..].iter().all(|byte| *byte == 0));
assert_eq!(export_reply(16, true).len(), 10);
}
#[test]
fn a_request_round_trips_and_a_wrong_magic_is_refused() {
let mut bytes = [0_u8; REQUEST_LEN];
bytes[..4].copy_from_slice(&REQUEST_MAGIC.to_be_bytes());
bytes[4..6].copy_from_slice(&1_u16.to_be_bytes());
bytes[6..8].copy_from_slice(&CMD_DISC.to_be_bytes());
bytes[8..16].copy_from_slice(&0x1122_3344_5566_7788_u64.to_be_bytes());
bytes[16..24].copy_from_slice(&4096_u64.to_be_bytes());
bytes[24..].copy_from_slice(&512_u32.to_be_bytes());
assert_eq!(
request(&bytes).unwrap(),
Request {
flags: 1,
kind: CMD_DISC,
cookie: 0x1122_3344_5566_7788,
offset: 4096,
length: 512
}
);
bytes[0] = 0;
let error = request(&bytes).unwrap_err();
assert_eq!(
error.to_string(),
"nbd: request magic 0x00609513 is not NBD_REQUEST_MAGIC"
);
}
#[test]
fn a_simple_reply_holds_the_magic_the_error_and_the_cookie() {
let reply = simple_reply(0x0102_0304_0506_0708, EINVAL);
assert_eq!(&reply[..4], &[0x67, 0x44, 0x66, 0x98]);
assert_eq!(&reply[4..8], &[0, 0, 0, 22]);
assert_eq!(&reply[8..], &[1, 2, 3, 4, 5, 6, 7, 8]);
}
fn client_bytes(flags: u32, options: &[(u32, &[u8])]) -> Vec<u8> {
let mut bytes = flags.to_be_bytes().to_vec();
for (option, data) in options {
bytes.extend_from_slice(&IHAVEOPT.to_be_bytes());
bytes.extend_from_slice(&option.to_be_bytes());
bytes.extend_from_slice(&u32::try_from(data.len()).unwrap().to_be_bytes());
bytes.extend_from_slice(data);
}
bytes
}
/// `NBD_OPT_LIST`, `NBD_OPT_STARTTLS` and `NBD_OPT_STRUCTURED_REPLY`, none served.
const OPT_LIST: u32 = 3;
const OPT_STARTTLS: u32 = 5;
const OPT_STRUCTURED_REPLY: u32 = 8;
#[test]
fn negotiation_refuses_unserved_options_then_answers_export_name_with_the_size() {
let mut input = Cursor::new(client_bytes(
FLAG_C_FIXED_NEWSTYLE | FLAG_C_NO_ZEROES,
&[
(OPT_LIST, b""),
(OPT_STARTTLS, b""),
(OPT_STRUCTURED_REPLY, b""),
(0x4242, b"\x00\x00\x00\x00\x00\x00\x00\x00"),
(OPT_EXPORT_NAME, b"disk"),
],
));
let mut output = Vec::new();
let outcome = negotiate(&mut input, &mut output, 4096, QUIET).unwrap();
assert_eq!(outcome, Negotiated::Export);
let mut expected = greeting().to_vec();
for option in [OPT_LIST, OPT_STARTTLS, OPT_STRUCTURED_REPLY, 0x4242] {
expected.extend(option_reply(option, REP_ERR_UNSUP, &[]));
}
expected.extend(export_reply(4096, true));
assert_eq!(output, expected);
}
/// The bytes answering a well-formed `NBD_OPT_INFO` or `NBD_OPT_GO`.
fn info_bytes(option: u32, size: u64) -> Vec<u8> {
let mut bytes = option_reply(option, REP_INFO, &export_info(size));
bytes.extend(option_reply(option, REP_INFO, &block_size_info()));
bytes.extend(option_reply(option, REP_ACK, &[]));
bytes
}
#[test]
fn negotiation_answers_go_with_both_infos_and_an_ack_then_enters_transmission() {
let mut input = Cursor::new(client_bytes(
FLAG_C_FIXED_NEWSTYLE,
&[(OPT_GO, b"\x00\x00\x00\x01x\x00\x01\x00\x03")],
));
let mut output = Vec::new();
let outcome = negotiate(&mut input, &mut output, 4096, QUIET).unwrap();
assert_eq!(outcome, Negotiated::Export);
let mut expected = greeting().to_vec();
expected.extend(info_bytes(OPT_GO, 4096));
assert_eq!(output, expected, "no zero padding follows the ack");
}
#[test]
fn negotiation_answers_info_the_same_way_and_stays_in_negotiation() {
let mut input = Cursor::new(client_bytes(
FLAG_C_FIXED_NEWSTYLE | FLAG_C_NO_ZEROES,
&[
(OPT_INFO, b"\x00\x00\x00\x00\x00\x02\x00\x00\x00\x09"),
(OPT_EXPORT_NAME, b""),
],
));
let mut output = Vec::new();
let outcome = negotiate(&mut input, &mut output, 4096, QUIET).unwrap();
assert_eq!(outcome, Negotiated::Export);
let mut expected = greeting().to_vec();
expected.extend(info_bytes(OPT_INFO, 4096));
expected.extend(export_reply(4096, true));
assert_eq!(output, expected);
}
#[test]
fn negotiation_refuses_a_malformed_go_as_invalid_and_continues() {
let mut input = Cursor::new(client_bytes(
FLAG_C_FIXED_NEWSTYLE | FLAG_C_NO_ZEROES,
&[(OPT_GO, b"\x00\x00\x00\x09"), (OPT_EXPORT_NAME, b"")],
));
let mut output = Vec::new();
negotiate(&mut input, &mut output, 4096, QUIET).unwrap();
let mut expected = greeting().to_vec();
expected.extend(option_reply(OPT_GO, REP_ERR_INVALID, &[]));
expected.extend(export_reply(4096, true));
assert_eq!(output, expected);
}
#[test]
fn negotiation_acknowledges_abort_and_ends_without_an_export() {
let mut input = Cursor::new(client_bytes(
FLAG_C_FIXED_NEWSTYLE,
&[(OPT_ABORT, b"\x00\x00"), (OPT_EXPORT_NAME, b"")],
));
let mut output = Vec::new();
let outcome = negotiate(&mut input, &mut output, 4096, QUIET).unwrap();
assert_eq!(outcome, Negotiated::Abort);
let mut expected = greeting().to_vec();
expected.extend(option_reply(OPT_ABORT, REP_ACK, &[]));
assert_eq!(output, expected, "the option after the abort is never read");
}
fn a_request(flags: u16, kind: u16, offset: u64, length: u32) -> Request {
Request {
flags,
kind,
cookie: 7,
offset,
length,
}
}
#[test]
fn a_read_within_the_export_and_the_payload_limit_is_not_refused() {
assert_eq!(refusal(&a_request(0, CMD_READ, 0, 512), 512), None);
assert_eq!(refusal(&a_request(0, CMD_READ, 512, 0), 512), None);
let whole = u64::from(MAX_PAYLOAD);
assert_eq!(
refusal(&a_request(0, CMD_READ, 0, MAX_PAYLOAD), whole),
None
);
}
#[test]
fn a_write_a_write_of_zeroes_and_a_trim_are_refused_as_read_only() {
for kind in [CMD_WRITE, CMD_WRITE_ZEROES, CMD_TRIM] {
let request = a_request(0, kind, 0, 512);
assert_eq!(refusal(&request, 512), Some(EPERM), "{request:?}");
}
}
/// `NBD_CMD_FLUSH` and `NBD_CMD_CACHE`, neither served.
const CMD_FLUSH: u16 = 3;
const CMD_CACHE: u16 = 5;
#[test]
fn a_flag_an_unserved_type_and_a_read_out_of_bounds_are_invalid() {
for (request, size) in [
(a_request(1, CMD_READ, 0, 512), 512),
(a_request(1, CMD_WRITE, 0, 512), 512),
(a_request(0, CMD_FLUSH, 0, 0), 512),
(a_request(0, CMD_CACHE, 0, 512), 512),
(a_request(0, 99, 0, 0), 512),
(a_request(0, CMD_READ, 511, 2), 512),
(a_request(0, CMD_READ, u64::MAX, 1), 512),
(a_request(0, CMD_READ, 0, MAX_PAYLOAD + 1), u64::MAX),
] {
assert_eq!(refusal(&request, size), Some(EINVAL), "{request:?}");
}
}
fn request_bytes(kind: u16, length: u32) -> Vec<u8> {
let mut bytes = REQUEST_MAGIC.to_be_bytes().to_vec();
bytes.extend_from_slice(&[0, 0]);
bytes.extend_from_slice(&kind.to_be_bytes());
bytes.extend_from_slice(&[0; 16]);
bytes.extend_from_slice(&length.to_be_bytes());
bytes
}
#[test]
fn the_next_request_drains_a_write_payload_before_the_request_after_it() {
let mut bytes = request_bytes(CMD_WRITE, 5);
bytes.extend_from_slice(b"hello");
bytes.extend(request_bytes(CMD_DISC, 0));
let mut input = Cursor::new(bytes);
assert_eq!(next_request(&mut input).unwrap().kind, CMD_WRITE);
assert_eq!(next_request(&mut input).unwrap().kind, CMD_DISC);
}
#[test]
fn a_write_whose_payload_is_cut_short_counts_as_the_client_leaving() {
let mut bytes = request_bytes(CMD_WRITE, 5);
bytes.extend_from_slice(b"hel");
let error = next_request(&mut Cursor::new(bytes)).unwrap_err();
assert!(matches!(error, Error::Left), "{error:?}");
}
#[test]
fn negotiation_sends_the_zeros_when_the_client_did_not_agree_to_skip_them() {
let mut input = Cursor::new(client_bytes(
FLAG_C_FIXED_NEWSTYLE,
&[(OPT_EXPORT_NAME, b"")],
));
let mut output = Vec::new();
negotiate(&mut input, &mut output, 1, QUIET).unwrap();
assert_eq!(output.len(), GREETING_LEN + 8 + 2 + ZEROES_LEN);
}
#[test]
fn a_client_that_leaves_during_negotiation_or_before_a_request_is_reported() {
let mut output = Vec::new();
let error = negotiate(
&mut Cursor::new(b"\x00\x00\x00\x01IHAVE"),
&mut output,
1,
QUIET,
)
.unwrap_err();
assert_eq!(
error.to_string(),
"the nbd client left without disconnecting"
);
let error = next_request(&mut Cursor::new(b"")).unwrap_err();
assert!(matches!(error, Error::Left), "{error:?}");
}
#[test]
fn a_broken_pipe_on_send_counts_as_the_client_leaving() {
struct Broken;
impl Write for Broken {
fn write(&mut self, _: &[u8]) -> io::Result<usize> {
Err(io::Error::from(io::ErrorKind::BrokenPipe))
}
fn flush(&mut self) -> io::Result<()> {
Ok(())
}
}
let error = send(&mut Broken, b"x").unwrap_err();
assert!(matches!(error, Error::Left), "{error:?}");
}
#[test]
fn the_next_request_is_decoded_from_its_twenty_eight_bytes() {
let mut bytes = REQUEST_MAGIC.to_be_bytes().to_vec();
bytes.extend_from_slice(&[0, 0, 0, 2]);
bytes.extend_from_slice(&[0; 20]);
let request = next_request(&mut Cursor::new(bytes)).unwrap();
assert_eq!(request.kind, CMD_DISC);
assert_eq!(request.cookie, 0);
}
}
@xen-orchestra/vmware-explorer/vectura/src/nfc.rs
+1561
View File
File diff suppressed because it is too large Load Diff
@xen-orchestra/vmware-explorer/vectura/src/nfc/greeting.rs
+197
View File
@@ -0,0 +1,197 @@
//! A host's data-port greeting and the capability gate.
//!
//! The line is ASCII, ends in CRLF and reads
//! `220 <version text>: <capability>, <capability>, ...`. An item may carry
//! a `/` suffix, which is not part of its name. Whether the client goes on is
//! decided on the capability names alone, never on the version text.
/// The capability a host must announce before TLS starts on its data port.
///
/// A host without it identifies its data-port certificate by SHA-1 only,
/// which this client refuses to pin.
pub const SHA256_SUPPORTED: &str = "SHA256 supported";
/// The capability of a host that carries the disk session over TLS.
///
/// A host without it carries the disk session in clear after the ticket
/// dialogue, which the client does only when asked for the `nfc` transport.
pub const NFCSSL_SUPPORTED: &str = "NFCSSL supported";
/// The reply code opening every greeting, as Broadcom KB 343952 shows.
///
/// A line without it is not a greeting.
const REPLY_CODE: &str = "220 ";
/// A parsed greeting: the capability names a host announces.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct Greeting {
/// The capability names in the order announced, `/` suffixes dropped.
pub capabilities: Vec<String>,
}
/// Errors from a greeting line.
#[derive(Debug, Clone, PartialEq, Eq, thiserror::Error)]
pub enum Error {
/// The line does not start with the greeting reply code.
#[error("not a greeting line: {0:?}")]
NotAGreeting(String),
/// The host does not announce a capability this client needs.
#[error("the host does not announce {capability:?}; announced: [{}]", announced.join(", "))]
Missing {
/// The capability that is required.
capability: &'static str,
/// The capabilities the host announced instead.
announced: Vec<String>,
},
}
impl Greeting {
/// Fails, naming `capability` and the announced list, unless the host announces it.
///
/// # Errors
/// When `capability` is not among the announced names.
pub fn require(&self, capability: &'static str) -> Result<(), Error> {
if self.capabilities.iter().any(|name| name == capability) {
Ok(())
} else {
Err(Error::Missing {
capability,
announced: self.capabilities.clone(),
})
}
}
}
/// Parses one greeting line, with or without its trailing CRLF.
///
/// # Errors
/// When the line does not start with the greeting reply code.
pub fn parse(line: &[u8]) -> Result<Greeting, Error> {
let text = String::from_utf8_lossy(line);
let text = text.trim_end_matches(['\r', '\n']);
let body = text
.strip_prefix(REPLY_CODE)
.ok_or_else(|| Error::NotAGreeting(text.to_owned()))?;
let list = body.split_once(": ").map_or("", |(_, list)| list);
let capabilities = list
.split(',')
.map(|item| item.split_once('/').map_or(item, |(name, _)| name).trim())
.filter(|name| !name.is_empty())
.map(str::to_owned)
.collect();
Ok(Greeting { capabilities })
}
/// An ESXi 8 greeting, as quoted in Broadcom KB 341384 (ESXi 6.x to 8.x) and KB 417531 (8.0.3).
#[cfg(test)]
pub(crate) const ESXI_8: &[u8] = b"220 VMware Authentication Daemon Version 1.10: SSL Required, ServerDaemonProtocol:SOAP, MKSDisplayProtocol:VNC , VMXARGS supported, NFCSSL supported/t, SHA256 supported\r\n";
#[cfg(test)]
mod tests {
use super::*;
// As quoted in Broadcom KB 338286 (ESXi 6.7 and 7.0).
const ESXI_6: &[u8] = b"220 VMware Authentication Daemon Version 1.10: SSL Required, ServerDaemonProtocol:SOAP, MKSDisplayProtocol:VNC , VMXARGS supported, NFCSSL supported/t\r\n";
#[test]
fn an_esxi_8_greeting_lists_its_capabilities_without_suffixes() {
let greeting = parse(ESXI_8).unwrap();
assert_eq!(
greeting.capabilities,
[
"SSL Required",
"ServerDaemonProtocol:SOAP",
"MKSDisplayProtocol:VNC",
"VMXARGS supported",
"NFCSSL supported",
"SHA256 supported",
]
);
assert_eq!(greeting.require(SHA256_SUPPORTED), Ok(()));
}
#[test]
fn an_esxi_6_greeting_is_refused_with_its_capability_list_quoted() {
let greeting = parse(ESXI_6).unwrap();
let error = greeting.require(SHA256_SUPPORTED).unwrap_err();
assert_eq!(
error.to_string(),
"the host does not announce \"SHA256 supported\"; announced: [SSL Required, \
ServerDaemonProtocol:SOAP, MKSDisplayProtocol:VNC, VMXARGS supported, \
NFCSSL supported]"
);
}
#[test]
fn a_line_without_the_reply_code_is_not_a_greeting() {
for line in [
&b"VMware Authentication Daemon Version 1.10: SSL Required\r\n"[..],
b"500 VMware Authentication Daemon Version 1.10: SSL Required\r\n",
b"220VMware\r\n",
b"",
] {
let error = parse(line).unwrap_err();
assert!(matches!(error, Error::NotAGreeting(_)), "{line:?}: {error}");
}
assert_eq!(
parse(b"500 nope\r\n").unwrap_err().to_string(),
"not a greeting line: \"500 nope\""
);
}
#[test]
fn nfcssl_support_is_required_by_its_name_and_refused_with_the_list_quoted() {
assert_eq!(parse(ESXI_8).unwrap().require(NFCSSL_SUPPORTED), Ok(()));
let without =
b"220 VMware Authentication Daemon Version 1.10: SSL Required, SHA256 supported\r\n";
let error = parse(without)
.unwrap()
.require(NFCSSL_SUPPORTED)
.unwrap_err();
assert_eq!(
error.to_string(),
"the host does not announce \"NFCSSL supported\"; announced: [SSL Required, SHA256 supported]"
);
}
#[test]
fn a_greeting_with_an_empty_capability_list_announces_nothing() {
for line in [
&b"220 VMware Authentication Daemon Version 1.10\r\n"[..],
b"220 VMware Authentication Daemon Version 1.10: \r\n",
b"220 VMware Authentication Daemon Version 1.10: ,\r\n",
] {
let greeting = parse(line).unwrap();
assert!(greeting.capabilities.is_empty(), "{line:?}");
assert_eq!(
greeting.require(SHA256_SUPPORTED).unwrap_err().to_string(),
"the host does not announce \"SHA256 supported\"; announced: []"
);
}
}
#[test]
fn the_gate_reads_capability_names_and_not_the_version_text() {
let greeting = parse(b"220 SHA256 supported daemon: SSL Required\r\n").unwrap();
assert_eq!(greeting.capabilities, ["SSL Required"]);
assert!(greeting.require(SHA256_SUPPORTED).is_err());
assert_eq!(greeting.require("SSL Required"), Ok(()));
}
#[test]
fn a_capability_name_must_match_whole_and_case_sensitive() {
let greeting = parse(b"220 d: SHA256 supported/t, sha256 supportedx\r\n").unwrap();
assert_eq!(
greeting.capabilities,
["SHA256 supported", "sha256 supportedx"]
);
assert_eq!(greeting.require(SHA256_SUPPORTED), Ok(()));
assert!(greeting.require("sha256 supported").is_err());
assert!(greeting.require("SHA256").is_err());
}
}
@xen-orchestra/vmware-explorer/vectura/src/nfc/handshake.rs
+131
View File
@@ -0,0 +1,131 @@
//! The fixed-size messages exchanged on the data port once TLS is up, before any record.
//!
//! Every message is [`LEN`] bytes: a little-endian `u32` type at offset 0,
//! two little-endian `u32` arguments at offsets 4 and 8, and zero padding.
//! A message of type [`TAG`] carries a 16-byte session tag at offset 4
//! instead of arguments, and one of type [`NAME`] is followed by a trailer
//! holding the client name and the operation name back to back, with their
//! lengths in the two arguments. The host reads one message per TLS
//! record, so each message is written on its own. The client sends [`TAG`],
//! [`TAG_END`] and [`VERSION`] and only then waits: the host acknowledges
//! the tag with [`TAG_ACK`] once the version message has arrived, and echoes
//! [`VERSION`] right after. The client then sends [`NAME`], [`MODE`] and
//! [`READY`] and waits for the [`READY`] echo. A reply of type [`ERROR`], or
//! of any type other than the one expected, ends the session.
/// Length of every message in bytes.
pub const LEN: usize = 264;
/// Length of the session tag a [`TAG`] message carries, in bytes.
pub const TAG_LEN: usize = 16;
/// The client's first message, carrying the session tag at offset 4.
pub const TAG: u32 = 43;
/// The client's second message: no arguments; the host acknowledges both after [`VERSION`].
pub const TAG_END: u32 = 33;
/// The host's acknowledgment of [`TAG`] and [`TAG_END`]; its arguments are not read.
pub const TAG_ACK: u32 = 36;
/// The client's protocol proposal, with [`VERSION_ARGS`]; the host answers with type 51 too.
pub const VERSION: u32 = 51;
/// The client name and operation name message, with their lengths as arguments.
pub const NAME: u32 = 54;
/// The client's mode message, with [`MODE_ARGS`].
pub const MODE: u32 = 55;
/// The client's last message, with zero arguments; the host answers with type 52 too.
pub const READY: u32 = 52;
/// A host's refusal, in place of the message it would otherwise echo.
pub const ERROR: u32 = 4;
/// Arguments of the [`VERSION`] message.
pub const VERSION_ARGS: (u32, u32) = (12, 1);
/// Arguments of the [`MODE`] message.
pub const MODE_ARGS: (u32, u32) = (3, 0);
/// The client name sent in the [`NAME`] trailer.
pub const CLIENT_NAME: &[u8] = b"vectura";
/// The operation name sent in the [`NAME`] trailer: a disk read session.
pub const OPERATION: &[u8] = b"nbdmode";
/// Builds a message of `kind` with two little-endian arguments.
#[must_use]
pub fn message(kind: u32, arguments: (u32, u32)) -> [u8; LEN] {
let mut bytes = [0_u8; LEN];
bytes[..4].copy_from_slice(&kind.to_le_bytes());
bytes[4..8].copy_from_slice(&arguments.0.to_le_bytes());
bytes[8..12].copy_from_slice(&arguments.1.to_le_bytes());
bytes
}
/// Builds the [`TAG`] message carrying `tag` at offset 4.
#[must_use]
pub fn tag(tag: &[u8; TAG_LEN]) -> [u8; LEN] {
let mut bytes = message(TAG, (0, 0));
bytes[4..4 + TAG_LEN].copy_from_slice(tag);
bytes
}
/// Builds the [`NAME`] message and its trailer, client name then operation name.
#[must_use]
pub fn name() -> Vec<u8> {
let lengths = (
u32::try_from(CLIENT_NAME.len()).unwrap_or(u32::MAX),
u32::try_from(OPERATION.len()).unwrap_or(u32::MAX),
);
let mut bytes = message(NAME, lengths).to_vec();
bytes.extend_from_slice(CLIENT_NAME);
bytes.extend_from_slice(OPERATION);
bytes
}
/// The type at offset 0 of a message.
#[must_use]
pub fn kind(bytes: &[u8; LEN]) -> u32 {
u32::from_le_bytes([bytes[0], bytes[1], bytes[2], bytes[3]])
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn a_message_holds_its_type_and_arguments_little_endian_then_zeros() {
let bytes = message(VERSION, VERSION_ARGS);
assert_eq!(bytes.len(), 264);
assert_eq!(&bytes[..12], &[51, 0, 0, 0, 12, 0, 0, 0, 1, 0, 0, 0]);
assert!(bytes[12..].iter().all(|byte| *byte == 0));
assert_eq!(kind(&bytes), VERSION);
}
#[test]
fn the_tag_message_carries_the_sixteen_byte_tag_at_offset_four() {
let bytes = tag(b"ABCDEFGHIJKLMNOP");
assert_eq!(&bytes[..4], &[43, 0, 0, 0]);
assert_eq!(&bytes[4..20], b"ABCDEFGHIJKLMNOP");
assert!(bytes[20..].iter().all(|byte| *byte == 0));
}
#[test]
fn the_name_message_announces_both_lengths_and_trails_both_names() {
let bytes = name();
assert_eq!(bytes.len(), 264 + 14);
assert_eq!(&bytes[..12], &[54, 0, 0, 0, 7, 0, 0, 0, 7, 0, 0, 0]);
assert!(bytes[12..264].iter().all(|byte| *byte == 0));
assert_eq!(&bytes[264..], b"vecturanbdmode");
}
#[test]
fn the_end_mode_and_ready_messages_carry_their_fixed_arguments() {
assert_eq!(
&message(TAG_END, (0, 0))[..12],
&[33, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
);
assert_eq!(
&message(MODE, MODE_ARGS)[..12],
&[55, 0, 0, 0, 3, 0, 0, 0, 0, 0, 0, 0]
);
assert_eq!(
&message(READY, (0, 0))[..12],
&[52, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
);
}
}
@xen-orchestra/vmware-explorer/vectura/src/nfc/record.rs
+442
View File
@@ -0,0 +1,442 @@
//! The record protocol carried by the data session once the handshake is done.
//!
//! A record is a 16-byte header, a payload of the announced length and, for
//! an open request or a read reply, a trailer the length does not count. The header holds
//! four little-endian `u32`s: [`MAGIC`], the opcode, the payload length and a
//! sequence number the client increments per record sent. The host answers
//! each record with a header echoing the opcode and sequence number. A
//! record it refuses gets an [`ERROR`] record first, carrying a code and a
//! text, and then the reply itself.
/// Value at offset 0 of every header; on the wire it reads `7a da 00 a1`.
pub const MAGIC: u32 = 0xA100_DA7A;
/// Length of a header in bytes.
pub const HEADER_LEN: usize = 16;
/// Opcode of the first record of a session; its payload is 16 zero bytes.
pub const HELLO: u32 = 2;
/// Opcode of the version record; its payload is 12 zero bytes.
pub const VERSION: u32 = 9;
/// Opcode of the capabilities record; its payload is the `u32` 1.
pub const CAPABILITIES: u32 = 22;
/// Opcode of the open request, [`OPEN_LEN`] bytes plus the path as trailer.
pub const OPEN: u32 = 4;
/// Opcode of the close request; its payload is the `u64` handle then 8 zero bytes.
pub const CLOSE: u32 = 5;
/// Opcode of the last record of a session; its payload is 16 zero bytes.
pub const END: u32 = 3;
/// Opcode of the read request, answered with one record per chunk.
pub const READ: u32 = 7;
/// Opcode of the error record a host sends, with the refused record's sequence number, before its reply.
pub const ERROR: u32 = 1;
/// Payload of the [`HELLO`] record.
pub const HELLO_PAYLOAD: [u8; 16] = [0; 16];
/// Payload of the [`VERSION`] record.
pub const VERSION_PAYLOAD: [u8; 12] = [0; 12];
/// Payload of the [`CAPABILITIES`] record: the one capability set every host accepts.
pub const CAPABILITIES_PAYLOAD: [u8; 4] = 1_u32.to_le_bytes();
/// Payload of the [`END`] record.
pub const END_PAYLOAD: [u8; 16] = [0; 16];
/// Length of an open request or reply payload in bytes.
pub const OPEN_LEN: usize = 60;
/// The `u32` at offset 16 of every open request.
const OPEN_WORD_16: u32 = 2;
/// The `u32` at offset 20 of every open request.
const OPEN_WORD_20: u32 = 30;
/// Errors of the record codec.
#[derive(Debug, PartialEq, Eq, thiserror::Error)]
pub enum Error {
/// A header did not start with [`MAGIC`].
#[error("record header magic {0:#010X} is not {MAGIC:#010X}")]
Magic(u32),
/// An open reply payload is shorter than [`OPEN_LEN`] bytes.
#[error("the open reply holds {0} bytes instead of {OPEN_LEN}")]
ShortOpenReply(usize),
/// An error record payload is shorter than [`ERROR_LEN`] bytes.
#[error("the error record holds {0} bytes instead of {ERROR_LEN}")]
ShortErrorReply(usize),
/// A read reply payload is shorter than [`READ_LEN`] bytes.
#[error("the read reply holds {0} bytes instead of {READ_LEN}")]
ShortReadReply(usize),
}
/// A record header, without the magic.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct Header {
/// What the record asks or answers.
pub opcode: u32,
/// How many payload bytes follow the header.
pub length: u32,
/// The client's counter for the record, echoed by the host.
pub sequence: u32,
}
/// Encodes a header with the magic in front.
#[must_use]
pub fn encode(header: Header) -> [u8; HEADER_LEN] {
let mut bytes = [0_u8; HEADER_LEN];
bytes[..4].copy_from_slice(&MAGIC.to_le_bytes());
bytes[4..8].copy_from_slice(&header.opcode.to_le_bytes());
bytes[8..12].copy_from_slice(&header.length.to_le_bytes());
bytes[12..].copy_from_slice(&header.sequence.to_le_bytes());
bytes
}
/// Decodes a header, checking its magic.
///
/// # Errors
/// When the first four bytes are not [`MAGIC`].
pub fn decode(bytes: &[u8; HEADER_LEN]) -> Result<Header, Error> {
let magic = word(bytes, 0);
if magic != MAGIC {
return Err(Error::Magic(magic));
}
Ok(Header {
opcode: word(bytes, 4),
length: word(bytes, 8),
sequence: word(bytes, 12),
})
}
/// Builds the payload of an open request for a path of `path_len` bytes.
///
/// The path itself follows the payload as a trailer, without a terminator.
#[must_use]
pub fn open_request(path_len: u32) -> [u8; OPEN_LEN] {
let mut bytes = [0_u8; OPEN_LEN];
bytes[..4].copy_from_slice(&path_len.to_le_bytes());
bytes[16..20].copy_from_slice(&OPEN_WORD_16.to_le_bytes());
bytes[20..24].copy_from_slice(&OPEN_WORD_20.to_le_bytes());
bytes
}
/// What a successful open reply says about the disk.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct Opened {
/// The handle later records name the disk by.
pub handle: u64,
/// The disk's capacity in bytes.
pub capacity: u64,
/// The disk's sector size in bytes.
pub sector_size: u32,
}
/// Decodes an open reply into its handle, capacity and sector size.
///
/// The status is at offset 0, the handle at 8, the capacity at 28 and the
/// sector size at 36, all little-endian.
/// A non-zero status comes back as the `Err` of the inner result, so a
/// refused open is told apart from a malformed reply.
///
/// # Errors
/// When the payload is shorter than [`OPEN_LEN`] bytes.
pub fn open_reply(payload: &[u8]) -> Result<Result<Opened, u64>, Error> {
if payload.len() < OPEN_LEN {
return Err(Error::ShortOpenReply(payload.len()));
}
let status = long(payload, 0);
if status != 0 {
return Ok(Err(status));
}
Ok(Ok(Opened {
handle: long(payload, 8),
capacity: long(payload, 28),
sector_size: word(payload, 36),
}))
}
/// Length of a close request payload: the handle, then 8 zero bytes.
pub const CLOSE_LEN: usize = 16;
/// Length of an error record payload: the code, a zero word, then the text length.
pub const ERROR_LEN: usize = 16;
/// What an error record says: a code, and the length of the text that trails the payload.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct Refusal {
/// The host's error code, the `u64` at offset 0.
pub code: u64,
/// The length of the text following the payload, the `u32` at offset 12.
pub text_len: u32,
}
/// Decodes an error record payload into its code and text length.
///
/// The text itself follows the payload as a trailer the header length does
/// not count, without a terminator.
///
/// # Errors
/// When the payload is shorter than [`ERROR_LEN`] bytes.
pub fn error_reply(payload: &[u8]) -> Result<Refusal, Error> {
if payload.len() < ERROR_LEN {
return Err(Error::ShortErrorReply(payload.len()));
}
Ok(Refusal {
code: long(payload, 0),
text_len: word(payload, 12),
})
}
/// Builds the payload of a close request for `handle`.
#[must_use]
pub fn close_request(handle: u64) -> [u8; CLOSE_LEN] {
let mut bytes = [0_u8; CLOSE_LEN];
bytes[..8].copy_from_slice(&handle.to_le_bytes());
bytes
}
/// Length of a read request or reply payload in bytes.
pub const READ_LEN: usize = 44;
/// The word every read request carries after the handle.
///
/// Its meaning is not known: the host answers a read that carries 1 and
/// echoes the word in each reply.
const READ_CONSTANT: u32 = 1;
/// Builds a read request for `length` bytes at `offset` of disk `handle`.
///
/// The length is written three times, as the `u64` at offset 24 and the
/// `u32`s at 32 and 36. The `compression` at offset 12 says how the host
/// sends each chunk, 0 as stored, 1 zlib, 3 skipz, and every reply echoes it.
#[must_use]
pub fn read_request(handle: u64, offset: u64, length: u32, compression: u32) -> [u8; READ_LEN] {
let mut bytes = [0_u8; READ_LEN];
bytes[..8].copy_from_slice(&handle.to_le_bytes());
bytes[8..12].copy_from_slice(&READ_CONSTANT.to_le_bytes());
bytes[12..16].copy_from_slice(&compression.to_le_bytes());
bytes[16..24].copy_from_slice(&offset.to_le_bytes());
bytes[24..32].copy_from_slice(&u64::from(length).to_le_bytes());
bytes[32..36].copy_from_slice(&length.to_le_bytes());
bytes[36..40].copy_from_slice(&length.to_le_bytes());
bytes
}
/// What one read reply says about the chunk whose bytes trail it.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct ReadReply {
/// The host's status, the `u64` at offset 0; zero means the chunk is good.
pub status: u64,
/// The compression echoed at offset 12.
pub compression: u32,
/// The disk offset the request named, the `u64` at offset 16.
pub base_offset: u64,
/// The length the request named, the `u32` at offset 24.
pub total_length: u32,
/// Where the chunk starts within the request, the `u32` at offset 28.
pub chunk_offset: u32,
/// How many disk bytes the chunk holds, the `u32` at offset 32.
pub chunk_length: u32,
/// How many bytes trail the payload, the `u32` at offset 36.
pub wire_length: u32,
}
/// Decodes a read reply payload.
///
/// # Errors
/// When the payload is shorter than [`READ_LEN`] bytes.
pub fn read_reply(payload: &[u8]) -> Result<ReadReply, Error> {
if payload.len() < READ_LEN {
return Err(Error::ShortReadReply(payload.len()));
}
Ok(ReadReply {
status: long(payload, 0),
compression: word(payload, 12),
base_offset: long(payload, 16),
total_length: word(payload, 24),
chunk_offset: word(payload, 28),
chunk_length: word(payload, 32),
wire_length: word(payload, 36),
})
}
fn word(bytes: &[u8], at: usize) -> u32 {
u32::from_le_bytes([bytes[at], bytes[at + 1], bytes[at + 2], bytes[at + 3]])
}
fn long(bytes: &[u8], at: usize) -> u64 {
u64::from_le_bytes([
bytes[at],
bytes[at + 1],
bytes[at + 2],
bytes[at + 3],
bytes[at + 4],
bytes[at + 5],
bytes[at + 6],
bytes[at + 7],
])
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn the_header_magic_is_written_little_endian_and_read_back() {
let header = Header {
opcode: OPEN,
length: 60,
sequence: 3,
};
let bytes = encode(header);
assert_eq!(&bytes[..4], &[0x7A, 0xDA, 0x00, 0xA1]);
assert_eq!(&bytes[4..], &[4, 0, 0, 0, 60, 0, 0, 0, 3, 0, 0, 0]);
assert_eq!(decode(&bytes).unwrap(), header);
}
#[test]
fn a_header_with_another_magic_is_refused_naming_it() {
let mut bytes = encode(Header {
opcode: HELLO,
length: 0,
sequence: 0,
});
bytes[..4].copy_from_slice(&[0xA1, 0x00, 0xDA, 0x7A]);
let error = decode(&bytes).unwrap_err();
assert_eq!(error, Error::Magic(0x7ADA_00A1));
assert_eq!(
error.to_string(),
"record header magic 0x7ADA00A1 is not 0xA100DA7A"
);
}
#[test]
fn the_open_request_names_the_path_length_and_the_two_fixed_words() {
let bytes = open_request(24);
assert_eq!(&bytes[..4], &[24, 0, 0, 0]);
assert!(bytes[4..16].iter().all(|byte| *byte == 0));
assert_eq!(&bytes[16..24], &[2, 0, 0, 0, 30, 0, 0, 0]);
assert!(bytes[24..].iter().all(|byte| *byte == 0));
}
#[test]
fn the_open_reply_yields_handle_capacity_and_sector_size() {
let mut payload = [0_u8; OPEN_LEN];
payload[8..16].copy_from_slice(&0x1122_3344_5566_7788_u64.to_le_bytes());
payload[28..36].copy_from_slice(&(10 * 1024 * 1024 * 1024 + 512_u64).to_le_bytes());
payload[36..40].copy_from_slice(&512_u32.to_le_bytes());
payload[40..52].copy_from_slice(&[1, 0, 0, 0, 2, 0, 0, 0, 3, 0, 0, 0]);
assert_eq!(
open_reply(&payload),
Ok(Ok(Opened {
handle: 0x1122_3344_5566_7788,
capacity: 10 * 1024 * 1024 * 1024 + 512,
sector_size: 512,
}))
);
}
#[test]
fn a_non_zero_open_status_is_reported_and_a_short_reply_refused() {
let mut payload = [0_u8; OPEN_LEN];
payload[..8].copy_from_slice(&5_u64.to_le_bytes());
assert_eq!(open_reply(&payload), Ok(Err(5)));
let error = open_reply(&payload[..59]).unwrap_err();
assert_eq!(error, Error::ShortOpenReply(59));
assert_eq!(
error.to_string(),
"the open reply holds 59 bytes instead of 60"
);
}
#[test]
fn the_error_reply_yields_the_code_and_the_text_length_and_a_short_one_is_refused() {
let mut payload = [0_u8; ERROR_LEN];
payload[..8].copy_from_slice(&11_u64.to_le_bytes());
payload[12..].copy_from_slice(&97_u32.to_le_bytes());
assert_eq!(
error_reply(&payload),
Ok(Refusal {
code: 11,
text_len: 97,
})
);
let error = error_reply(&payload[..15]).unwrap_err();
assert_eq!(error, Error::ShortErrorReply(15));
assert_eq!(
error.to_string(),
"the error record holds 15 bytes instead of 16"
);
}
#[test]
fn the_close_request_carries_the_handle_then_zeros() {
let bytes = close_request(0x0102_0304_0506_0708);
assert_eq!(&bytes[..8], &[8, 7, 6, 5, 4, 3, 2, 1]);
assert_eq!(&bytes[8..], &[0; 8]);
}
#[test]
fn the_fixed_payloads_have_their_documented_bytes() {
assert_eq!(HELLO_PAYLOAD, [0; 16]);
assert_eq!(VERSION_PAYLOAD, [0; 12]);
assert_eq!(CAPABILITIES_PAYLOAD, [1, 0, 0, 0]);
assert_eq!(END_PAYLOAD, [0; 16]);
}
#[test]
fn the_read_request_is_built_byte_by_byte_with_the_compression_asked() {
for compression in [0, 1, 3] {
let bytes = read_request(7, 512, 4096, compression);
assert_eq!(
&bytes[12..16],
&[u8::try_from(compression).unwrap(), 0, 0, 0],
"compression {compression}"
);
}
let bytes = read_request(0xFFFF_FFFF_8000_2A1F, 0x0000_0001_0010_0200, 0x0010_0000, 0);
assert_eq!(
&bytes[..8],
&[0x1F, 0x2A, 0x00, 0x80, 0xFF, 0xFF, 0xFF, 0xFF]
);
assert_eq!(&bytes[8..12], &[1, 0, 0, 0]);
assert_eq!(&bytes[12..16], &[0, 0, 0, 0], "compression none");
assert_eq!(
&bytes[16..24],
&[0x00, 0x02, 0x10, 0x00, 0x01, 0x00, 0x00, 0x00]
);
assert_eq!(&bytes[24..32], &[0x00, 0x00, 0x10, 0x00, 0, 0, 0, 0]);
assert_eq!(&bytes[32..36], &[0x00, 0x00, 0x10, 0x00]);
assert_eq!(&bytes[36..40], &[0x00, 0x00, 0x10, 0x00]);
assert_eq!(&bytes[40..], &[0, 0, 0, 0]);
}
#[test]
fn the_read_reply_yields_base_offset_total_length_chunk_offset_chunk_length_and_wire_length() {
let mut payload = [0_u8; READ_LEN];
payload[..8].copy_from_slice(&3_u64.to_le_bytes());
payload[8..12].copy_from_slice(&1_u32.to_le_bytes());
payload[12..16].copy_from_slice(&3_u32.to_le_bytes());
payload[16..24].copy_from_slice(&0x0000_0001_0010_0200_u64.to_le_bytes());
payload[24..28].copy_from_slice(&0x0010_0000_u32.to_le_bytes());
payload[28..32].copy_from_slice(&0x0003_0000_u32.to_le_bytes());
payload[32..36].copy_from_slice(&0x0001_0000_u32.to_le_bytes());
payload[36..40].copy_from_slice(&0x0000_FFFF_u32.to_le_bytes());
assert_eq!(
read_reply(&payload),
Ok(ReadReply {
status: 3,
compression: 3,
base_offset: 0x0000_0001_0010_0200,
total_length: 0x0010_0000,
chunk_offset: 0x0003_0000,
chunk_length: 0x0001_0000,
wire_length: 0x0000_FFFF,
})
);
}
#[test]
fn a_short_read_reply_is_refused_naming_its_length() {
let error = read_reply(&[0_u8; 43]).unwrap_err();
assert_eq!(error, Error::ShortReadReply(43));
assert_eq!(
error.to_string(),
"the read reply holds 43 bytes instead of 44"
);
}
}
@xen-orchestra/vmware-explorer/vectura/src/serve.rs
+1399
View File
File diff suppressed because it is too large Load Diff
@xen-orchestra/vmware-explorer/vectura/src/soap.rs
+552
View File
@@ -0,0 +1,552 @@
//! The management plane: the vSphere API calls that yield a data-port ticket.
//!
//! Five calls, each a `POST /sdk` over its own TLS connection with a SOAP
//! envelope in the `urn:vim25` namespace: `RetrieveServiceContent`, `Login`,
//! `RetrieveInternalContent`, `NfcGetVmFiles` and `Logout`. The session
//! cookie the `Login` response sets is echoed verbatim on every later call.
pub mod http;
use std::fmt;
use std::io::Write;
use std::time::Instant;
use quick_xml::escape::{escape, resolve_predefined_entity};
use quick_xml::events::Event;
use rustls::Stream;
use crate::tls;
use crate::transcript::Transcript;
/// The managed object the service content is retrieved from.
const SERVICE_INSTANCE: &str = "ServiceInstance";
/// Errors from the management plane, each naming the call that raised it.
#[derive(Debug, thiserror::Error)]
pub enum Error {
/// Connecting to the management port or its TLS handshake failed.
#[error(transparent)]
Tls(#[from] tls::Error),
/// The HTTP exchange of a call failed.
#[error("{call}: {source}")]
Http {
/// The call being made.
call: &'static str,
/// What went wrong on the wire.
#[source]
source: http::Error,
},
/// The host answered a call with a SOAP fault.
#[error("{call}: the host answered: {text}")]
Fault {
/// The call being made.
call: &'static str,
/// The fault string, as the host wrote it.
text: String,
},
/// The host answered a call with an HTTP status other than 200 and no fault.
#[error("{call}: the host answered with HTTP status {status}")]
Status {
/// The call being made.
call: &'static str,
/// The status code.
status: u16,
},
/// A reply lacks an element the call needs.
#[error("{call}: the reply has no {element} element")]
Missing {
/// The call being made.
call: &'static str,
/// The element that should have been there.
element: &'static str,
},
/// A reply element holds text of the wrong shape.
#[error("NfcGetVmFiles: the {element} element holds {text:?}, {reason}")]
Invalid {
/// The element concerned.
element: &'static str,
/// Its text.
text: String,
/// What was expected of it.
reason: String,
},
/// The `Login` response set no cookie to carry the session.
#[error("Login: the host set no session cookie")]
NoCookie,
}
/// The operator's password, read once from the environment.
///
/// Its `Debug` rendering never shows the value.
#[derive(Clone)]
pub struct Password(String);
impl Password {
/// Wraps the password.
#[must_use]
pub fn new(text: String) -> Password {
Password(text)
}
}
impl fmt::Debug for Password {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
f.write_str("Password(<redacted>)")
}
}
/// The session cookie the `Login` response set, echoed verbatim.
///
/// Its `Debug` rendering never shows the value.
#[derive(Clone)]
struct Cookie(String);
impl fmt::Debug for Cookie {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
f.write_str("Cookie(<redacted>)")
}
}
/// A data-port ticket, as `NfcGetVmFiles` hands it out.
///
/// The secret is the ticket itself: an opaque bearer credential with an
/// expiry and a use count. It is never validated for shape, and the `Debug`
/// rendering never shows it.
#[derive(Clone)]
pub struct Ticket {
/// The host serving the data port, when it is not the one that issued the ticket.
///
/// A vCenter names the ESXi host that owns the VM; an ESXi host leaves it out.
pub host: Option<String>,
/// The data port to connect to.
pub port: u16,
/// The SHA-1 thumbprint of the certificate the data port presents.
pub thumbprint: tls::Pin,
/// The ticket itself, presented in the data-port dialogue.
pub secret: String,
}
impl fmt::Debug for Ticket {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
f.debug_struct("Ticket")
.field("host", &self.host)
.field("port", &self.port)
.field("thumbprint", &self.thumbprint)
.field("secret", &"<redacted>")
.finish()
}
}
/// A logged-in management session.
#[derive(Debug)]
pub struct Session {
host: String,
port: u16,
client: tls::Client,
manager: String,
cookie: Cookie,
transcript: Transcript,
}
impl Session {
/// Retrieves the service content and logs in.
///
/// The host is an ESXi host or a vCenter server.
///
/// # Errors
/// When a call fails or the host refuses the credentials.
pub fn login(
host: &str,
port: u16,
trust: &tls::Trust,
user: &str,
password: &Password,
transcript: Transcript,
) -> Result<Session, Error> {
let client = tls::Client::new(trust)?;
let content = call(
host,
port,
&client,
None,
"RetrieveServiceContent",
&this(SERVICE_INSTANCE, SERVICE_INSTANCE),
transcript,
)?;
let manager = service_content(&String::from_utf8_lossy(&content.body))?;
let inner = format!(
"{}<userName>{}</userName><password>{}</password>",
this("SessionManager", &manager),
escape(user),
escape(password.0.as_str())
);
let response = call(host, port, &client, None, "Login", &inner, transcript)?;
let cookie = response
.header("Set-Cookie")
.map(|value| {
value
.split(';')
.next()
.unwrap_or_default()
.trim()
.to_owned()
})
.filter(|value| !value.is_empty())
.ok_or(Error::NoCookie)?;
Ok(Session {
host: host.to_owned(),
port,
client,
manager,
cookie: Cookie(cookie),
transcript,
})
}
/// Asks the NFC service for a fresh ticket to the files of the VM `vm_id` names.
///
/// # Errors
/// When a call fails or the reply lacks a field.
pub fn ticket(&self, vm_id: &str) -> Result<Ticket, Error> {
let content = self.call(
"RetrieveInternalContent",
&this(SERVICE_INSTANCE, SERVICE_INSTANCE),
)?;
let service = element(&content, "nfcService").ok_or(Error::Missing {
call: "RetrieveInternalContent",
element: "nfcService",
})?;
let inner = format!(
"{}<vm type=\"VirtualMachine\">{}</vm>",
this("NfcService", &service),
escape(vm_id)
);
ticket(&self.call("NfcGetVmFiles", &inner)?)
}
/// Logs the session out; the host forgets the cookie.
///
/// # Errors
/// When the call fails, which after a long session may mean it had
/// already expired.
pub fn logout(self) -> Result<(), Error> {
self.call("Logout", &this("SessionManager", &self.manager))?;
Ok(())
}
fn call(&self, name: &'static str, inner: &str) -> Result<String, Error> {
let response = call(
&self.host,
self.port,
&self.client,
Some(&self.cookie),
name,
inner,
self.transcript,
)?;
Ok(String::from_utf8_lossy(&response.body).into_owned())
}
}
/// Makes one call on its own TLS connection and returns the checked response.
fn call(
host: &str,
port: u16,
client: &tls::Client,
cookie: Option<&Cookie>,
name: &'static str,
inner: &str,
transcript: Transcript,
) -> Result<http::Response, Error> {
let started = Instant::now();
let wire = |source| Error::Http { call: name, source };
let mut stream = tls::connect(host, port)?;
let mut session = client.handshake(&mut stream, host)?;
let mut tls = Stream::new(&mut session, &mut stream);
let body = envelope(name, inner);
let request = http::request(
host,
port,
cookie.map(|cookie| cookie.0.as_str()),
body.as_bytes(),
);
tls.write_all(&request)
.and_then(|()| tls.flush())
.map_err(|error| wire(http::Error::Io(error)))?;
let response = http::read_response(&mut tls).map_err(wire)?;
transcript.call(name, response.status, started.elapsed());
let text = String::from_utf8_lossy(&response.body);
if let Some(text) = element(&text, "faultstring") {
return Err(Error::Fault { call: name, text });
}
if response.status != 200 {
return Err(Error::Status {
call: name,
status: response.status,
});
}
Ok(response)
}
/// Wraps the operation `name` and its `inner` elements in a SOAP envelope.
fn envelope(name: &str, inner: &str) -> String {
format!(
"<?xml version=\"1.0\" encoding=\"UTF-8\"?>\
<soapenv:Envelope xmlns:soapenv=\"http://schemas.xmlsoap.org/soap/envelope/\">\
<soapenv:Body><{name} xmlns=\"urn:vim25\">{inner}</{name}></soapenv:Body>\
</soapenv:Envelope>"
)
}
/// The `_this` element naming the managed object a call is made on.
fn this(kind: &str, reference: &str) -> String {
format!("<_this type=\"{kind}\">{}</_this>", escape(reference))
}
/// Returns the session manager the service content names.
fn service_content(content: &str) -> Result<String, Error> {
element(content, "sessionManager").ok_or(Error::Missing {
call: "RetrieveServiceContent",
element: "sessionManager",
})
}
/// Reads the ticket out of an `NfcGetVmFiles` reply.
fn ticket(reply: &str) -> Result<Ticket, Error> {
let field = |element| {
self::element(reply, element).ok_or(Error::Missing {
call: "NfcGetVmFiles",
element,
})
};
let port = field("port")?;
let thumbprint = field("sslThumbprint")?;
Ok(Ticket {
host: element(reply, "host").filter(|host| !host.is_empty()),
port: port.parse().map_err(|error| Error::Invalid {
element: "port",
text: port.clone(),
reason: format!("not a port: {error}"),
})?,
thumbprint: tls::Pin::sha1(&thumbprint).map_err(|error| Error::Invalid {
element: "sslThumbprint",
text: thumbprint.clone(),
reason: error.to_string(),
})?,
secret: field("sessionId")?,
})
}
/// The text of the first element named `name`, whatever its namespace prefix.
fn element(xml: &str, name: &str) -> Option<String> {
let mut reader = quick_xml::Reader::from_str(xml);
let mut inside = false;
let mut text = String::new();
loop {
let event = reader.read_event().ok()?;
if event == Event::Eof {
return None;
}
match event {
Event::Start(start) if start.local_name().as_ref() == name => inside = true,
Event::End(_) if inside => return Some(text),
Event::Text(content) if inside => text.push_str(&content.xml10_content()),
Event::CData(content) if inside => text.push_str(&content.xml10_content()),
Event::GeneralRef(reference) if inside => match reference.resolve_char_ref() {
Ok(Some(character)) => text.push(character),
_ => text.push_str(resolve_predefined_entity(&reference).unwrap_or_default()),
},
_ => {}
}
}
}
#[cfg(test)]
mod tests {
use super::*;
const SERVICE_CONTENT: &str = "<soapenv:Envelope xmlns:soapenv=\"http://schemas.xmlsoap.org/soap/envelope/\">\
<soapenv:Body><RetrieveServiceContentResponse xmlns=\"urn:vim25\"><returnval>\
<rootFolder type=\"Folder\">ha-folder-root</rootFolder>\
<about><name>VMware ESXi</name><fullName>VMware ESXi 8.0.1 build-21495797</fullName>\
<apiType>HostAgent</apiType></about>\
<sessionManager type=\"SessionManager\">ha-sessionmgr</sessionManager>\
</returnval></RetrieveServiceContentResponse></soapenv:Body></soapenv:Envelope>";
const TICKET: &str = "<returnval><port>902</port>\
<sslThumbprint>A9:99:3E:36:47:06:81:6A:BA:3E:25:71:78:50:C2:6C:9C:D0:D8:9D</sslThumbprint>\
<service>nfc</service><serviceVersion>1.1</serviceVersion>\
<sessionId>00000000-mock-ticket-000000000000</sessionId></returnval>";
#[test]
fn the_password_never_appears_in_a_debug_rendering() {
let password = Password::new("hunter2".to_owned());
let rendered = format!("{password:?}");
assert!(!rendered.contains("hunter2"), "{rendered}");
assert_eq!(rendered, "Password(<redacted>)");
}
#[test]
fn the_cookie_never_appears_in_a_debug_rendering_of_a_session() {
let session = Session {
host: "esxi.test".to_owned(),
port: 443,
client: tls::Client::new(&tls::Trust::Pin(
tls::Pin::sha256(&"ab".repeat(32)).unwrap(),
))
.unwrap(),
manager: "ha-sessionmgr".to_owned(),
cookie: Cookie("vmware_soap_session=\"mock-cookie\"".to_owned()),
transcript: Transcript::new(false),
};
let rendered = format!("{session:?}");
assert!(!rendered.contains("mock-cookie"), "{rendered}");
assert!(rendered.contains("Cookie(<redacted>)"), "{rendered}");
}
#[test]
fn the_ticket_reply_yields_port_thumbprint_and_secret() {
let ticket = ticket(TICKET).unwrap();
assert_eq!(ticket.port, 902);
assert_eq!(
ticket.thumbprint.to_string(),
"sha1:A9:99:3E:36:47:06:81:6A:BA:3E:25:71:78:50:C2:6C:9C:D0:D8:9D"
);
assert_eq!(ticket.secret, "00000000-mock-ticket-000000000000");
assert_eq!(ticket.host, None);
}
#[test]
fn a_ticket_from_a_vcenter_names_the_host_serving_the_data_port() {
let reply = TICKET.replace("<port>", "<host>esxi-02.test</host><port>");
assert_eq!(
ticket(&reply).unwrap().host.as_deref(),
Some("esxi-02.test")
);
let empty = TICKET.replace("<port>", "<host></host><port>");
assert_eq!(ticket(&empty).unwrap().host, None);
}
#[test]
fn the_secret_never_appears_in_a_debug_rendering_of_the_ticket() {
let rendered = format!("{:?}", ticket(TICKET).unwrap());
assert!(!rendered.contains("mock-ticket"), "{rendered}");
assert!(rendered.contains("secret: \"<redacted>\""), "{rendered}");
}
#[test]
fn a_ticket_reply_missing_a_field_names_it() {
let error = ticket("<returnval><port>902</port></returnval>").unwrap_err();
assert_eq!(
error.to_string(),
"NfcGetVmFiles: the reply has no sslThumbprint element"
);
}
#[test]
fn a_ticket_with_a_bad_port_or_thumbprint_quotes_the_text() {
let reply = TICKET.replace("902", "port-902");
let error = ticket(&reply).unwrap_err();
assert!(
error
.to_string()
.starts_with("NfcGetVmFiles: the port element holds \"port-902\", not a port: "),
"{error}"
);
let reply = TICKET.replace("A9:99:3E", "zz:99:3E");
let error = ticket(&reply).unwrap_err();
assert!(
error
.to_string()
.starts_with("NfcGetVmFiles: the sslThumbprint element holds \"zz:99:3E:"),
"{error}"
);
}
#[test]
fn an_esxi_service_content_yields_its_session_manager() {
assert_eq!(service_content(SERVICE_CONTENT).unwrap(), "ha-sessionmgr");
}
#[test]
fn a_vcenter_service_content_yields_its_session_manager() {
let content = SERVICE_CONTENT
.replace("HostAgent", "VirtualCenter")
.replace("ha-sessionmgr", "SessionManager");
assert_eq!(service_content(&content).unwrap(), "SessionManager");
}
#[test]
fn a_service_content_without_a_session_manager_names_the_element() {
let error = service_content("<about><apiType>HostAgent</apiType></about>").unwrap_err();
assert_eq!(
error.to_string(),
"RetrieveServiceContent: the reply has no sessionManager element"
);
}
#[test]
fn an_element_is_found_by_local_name_with_its_entities_decoded() {
let xml = "<a:r xmlns:a=\"urn:x\"><a:faultstring>Cannot &amp; will &#x6e;ot &lt;log in&gt;</a:faultstring></a:r>";
assert_eq!(
element(xml, "faultstring").as_deref(),
Some("Cannot & will not <log in>")
);
assert_eq!(element(xml, "other"), None);
assert_eq!(element("<broken", "faultstring"), None);
assert_eq!(
element("<x><![CDATA[a<b]]></x>", "x").as_deref(),
Some("a<b")
);
assert_eq!(
element("<r>out<![CDATA[side]]>&amp;<f>in</f></r>", "f").as_deref(),
Some("in")
);
}
#[test]
fn the_envelope_names_the_operation_in_the_vim25_namespace() {
let body = envelope("Logout", &this("SessionManager", "ha-sessionmgr"));
assert_eq!(
body,
"<?xml version=\"1.0\" encoding=\"UTF-8\"?>\
<soapenv:Envelope xmlns:soapenv=\"http://schemas.xmlsoap.org/soap/envelope/\">\
<soapenv:Body><Logout xmlns=\"urn:vim25\">\
<_this type=\"SessionManager\">ha-sessionmgr</_this>\
</Logout></soapenv:Body></soapenv:Envelope>"
);
}
#[test]
fn operator_supplied_values_are_xml_escaped() {
assert_eq!(
this("VirtualMachine", "<3 & \"4\">"),
"<_this type=\"VirtualMachine\">&lt;3 &amp; &quot;4&quot;&gt;</_this>"
);
}
#[test]
fn a_fault_error_carries_the_call_and_the_host_text() {
let error = Error::Fault {
call: "Login",
text: "Cannot complete login due to an incorrect user name or password.".to_owned(),
};
assert_eq!(
error.to_string(),
"Login: the host answered: Cannot complete login due to an incorrect user name or password."
);
}
}
@xen-orchestra/vmware-explorer/vectura/src/soap/http.rs
+429
View File
@@ -0,0 +1,429 @@
//! The HTTP/1.1 client under the management plane: one request, one response.
//!
//! Every request is a `POST /sdk` with `Connection: close`, so a response is
//! read up to its `Content-Length`, through its chunked encoding, or to the
//! close. No redirects, no compression, no keep-alive.
use std::io::{self, BufRead, BufReader, Read};
use crate::tls;
/// Longest status or header line accepted; real ones are under 200 bytes.
const MAX_LINE_LEN: usize = 8192;
/// Most headers accepted, so a host cannot grow the list without bound.
const MAX_HEADERS: usize = 64;
/// Largest body accepted. A SOAP reply is a few KiB; a `Content-Length`
/// beyond this is refused before anything is allocated.
pub const MAX_BODY_LEN: usize = 1 << 20;
/// Errors reading a response.
#[derive(Debug, thiserror::Error)]
pub enum Error {
/// A read or write failed or timed out.
#[error("{}", tls::describe(.0))]
Io(#[from] io::Error),
/// The host closed the connection before the response was complete.
#[error("the host closed the connection before the response was complete")]
Closed,
/// The response does not parse as HTTP/1.1.
#[error("malformed response: {0}")]
Malformed(&'static str),
/// The response body exceeds [`MAX_BODY_LEN`].
#[error("the response body exceeds {MAX_BODY_LEN} bytes")]
TooLarge,
}
/// A response: its status code, its headers in order, and its body.
#[derive(Debug, PartialEq, Eq)]
pub struct Response {
/// The three-digit status code.
pub status: u16,
/// Header names and values, trimmed, in the order received.
pub headers: Vec<(String, String)>,
/// The body, decoded from its transfer encoding.
pub body: Vec<u8>,
}
impl Response {
/// The value of the first header named `name`, compared case-insensitively.
#[must_use]
pub fn header(&self, name: &str) -> Option<&str> {
self.headers
.iter()
.find(|(candidate, _)| candidate.eq_ignore_ascii_case(name))
.map(|(_, value)| value.as_str())
}
}
/// Builds a `POST /sdk` request carrying `body`, and `cookie` when there is one.
#[must_use]
pub fn request(host: &str, port: u16, cookie: Option<&str>, body: &[u8]) -> Vec<u8> {
let authority = match (host.contains(':'), port) {
(true, 443) => format!("[{host}]"),
(true, _) => format!("[{host}]:{port}"),
(false, 443) => host.to_owned(),
(false, _) => format!("{host}:{port}"),
};
let cookie = cookie
.map(|cookie| format!("Cookie: {cookie}\r\n"))
.unwrap_or_default();
let head = format!(
"POST /sdk HTTP/1.1\r\n\
Host: {authority}\r\n\
Content-Type: text/xml; charset=utf-8\r\n\
SOAPAction: \"urn:vim25\"\r\n\
Connection: close\r\n\
Content-Length: {}\r\n\
{cookie}\r\n",
body.len()
);
let mut bytes = head.into_bytes();
bytes.extend_from_slice(body);
bytes
}
/// Reads one response from `reader`.
///
/// # Errors
/// When the read fails, the host closes early, the head does not parse, or
/// the body is longer than [`MAX_BODY_LEN`].
pub fn read_response<R: Read>(reader: R) -> Result<Response, Error> {
let mut reader = BufReader::new(reader);
let status_line = line(&mut reader)?;
let status = status_line
.strip_prefix("HTTP/1.")
.and_then(|rest| rest.split(' ').nth(1))
.and_then(|code| code.parse().ok())
.ok_or(Error::Malformed("status line"))?;
let mut headers = Vec::new();
loop {
let text = line(&mut reader)?;
if text.is_empty() {
break;
}
if headers.len() == MAX_HEADERS {
return Err(Error::Malformed("too many headers"));
}
let (name, value) = text.split_once(':').ok_or(Error::Malformed("header"))?;
headers.push((name.trim().to_owned(), value.trim().to_owned()));
}
let mut response = Response {
status,
headers,
body: Vec::new(),
};
let chunked = response
.header("Transfer-Encoding")
.is_some_and(|encoding| encoding.eq_ignore_ascii_case("chunked"));
response.body = if chunked {
chunks(&mut reader)?
} else if let Some(length) = response.header("Content-Length") {
let length = length
.parse()
.map_err(|_error| Error::Malformed("content length"))?;
sized(&mut reader, length)?
} else {
rest(&mut reader)?
};
Ok(response)
}
/// Reads one line without its line end; empty means the blank line ending the head.
fn line<R: BufRead>(reader: &mut R) -> Result<String, Error> {
let mut bytes = Vec::new();
let read = reader
.by_ref()
.take(MAX_LINE_LEN as u64)
.read_until(b'\n', &mut bytes)?;
if read == 0 {
return Err(Error::Closed);
}
if bytes.last() != Some(&b'\n') {
return Err(if bytes.len() >= MAX_LINE_LEN {
Error::Malformed("line too long")
} else {
Error::Closed
});
}
let text = String::from_utf8_lossy(&bytes);
Ok(text.trim_end_matches(['\r', '\n']).to_owned())
}
/// Reads exactly `length` bytes of body.
fn sized<R: Read>(reader: &mut R, length: usize) -> Result<Vec<u8>, Error> {
if length > MAX_BODY_LEN {
return Err(Error::TooLarge);
}
let mut body = vec![0_u8; length];
reader.read_exact(&mut body).map_err(closed)?;
Ok(body)
}
/// Reads a chunked body: hex sizes, chunks, then the terminating empty chunk.
fn chunks<R: BufRead>(reader: &mut R) -> Result<Vec<u8>, Error> {
let mut body = Vec::new();
loop {
let size_line = line(reader)?;
let size = size_line.split(';').next().unwrap_or_default().trim();
let size =
usize::from_str_radix(size, 16).map_err(|_error| Error::Malformed("chunk size"))?;
if size == 0 {
while !line(reader)?.is_empty() {}
return Ok(body);
}
if body.len().saturating_add(size) > MAX_BODY_LEN {
return Err(Error::TooLarge);
}
body.extend_from_slice(&sized(reader, size)?);
if !line(reader)?.is_empty() {
return Err(Error::Malformed("chunk end"));
}
}
}
/// Reads the body up to the close, which may come without a TLS close notify.
fn rest<R: Read>(reader: &mut R) -> Result<Vec<u8>, Error> {
let mut body = Vec::new();
match reader.take(MAX_BODY_LEN as u64 + 1).read_to_end(&mut body) {
Ok(_) | Err(_) if body.len() > MAX_BODY_LEN => Err(Error::TooLarge),
Ok(_) => Ok(body),
Err(error) if error.kind() == io::ErrorKind::UnexpectedEof => Ok(body),
Err(error) => Err(Error::Io(error)),
}
}
/// Maps the end of the stream inside a body to [`Error::Closed`].
fn closed(error: io::Error) -> Error {
match error.kind() {
io::ErrorKind::UnexpectedEof => Error::Closed,
_ => Error::Io(error),
}
}
#[cfg(test)]
mod tests {
use std::io::Cursor;
use super::*;
#[test]
fn a_request_posts_to_sdk_with_the_soap_headers_and_the_body_length() {
let bytes = request("esxi.test", 443, None, b"<x/>");
assert_eq!(
String::from_utf8(bytes).unwrap(),
"POST /sdk HTTP/1.1\r\n\
Host: esxi.test\r\n\
Content-Type: text/xml; charset=utf-8\r\n\
SOAPAction: \"urn:vim25\"\r\n\
Connection: close\r\n\
Content-Length: 4\r\n\
\r\n\
<x/>"
);
}
#[test]
fn a_request_carries_the_cookie_verbatim_and_the_port_when_not_443() {
let text =
|host, port, cookie| String::from_utf8(request(host, port, cookie, b"")).unwrap();
let with_cookie = text("2001:db8::10", 8443, Some("vmware_soap_session=\"x\""));
assert!(
with_cookie.contains("Host: [2001:db8::10]:8443\r\n"),
"{with_cookie}"
);
assert!(
with_cookie.contains("\r\nCookie: vmware_soap_session=\"x\"\r\n\r\n"),
"{with_cookie}"
);
assert!(text("2001:db8::10", 443, None).contains("Host: [2001:db8::10]\r\n"));
assert!(text("esxi.test", 8443, None).contains("Host: esxi.test:8443\r\n"));
}
#[test]
fn a_response_with_a_content_length_stops_at_that_length() {
let bytes =
b"HTTP/1.1 200 OK\r\nContent-Type: text/xml\r\ncontent-length: 5\r\n\r\nhelloEXTRA";
let response = read_response(Cursor::new(bytes)).unwrap();
assert_eq!(response.status, 200);
assert_eq!(response.header("CONTENT-TYPE"), Some("text/xml"));
assert_eq!(response.body, b"hello");
}
#[test]
fn a_chunked_response_is_reassembled_without_its_sizes_and_trailers() {
let bytes = b"HTTP/1.1 500 Internal Server Error\r\nTransfer-Encoding: chunked\r\n\r\n\
3;ext=1\r\nabc\r\n2\r\nde\r\n0\r\nTrailer: x\r\n\r\n";
let response = read_response(Cursor::new(bytes)).unwrap();
assert_eq!(response.status, 500);
assert_eq!(response.body, b"abcde");
}
#[test]
fn a_response_without_framing_runs_to_the_close() {
let bytes = b"HTTP/1.0 200 OK\r\n\r\nall of it";
let response = read_response(Cursor::new(bytes)).unwrap();
assert_eq!(response.body, b"all of it");
}
/// A stream that fails with `kind` where it would report its end.
struct Failing(Cursor<Vec<u8>>, io::ErrorKind);
impl Read for Failing {
fn read(&mut self, buffer: &mut [u8]) -> io::Result<usize> {
match self.0.read(buffer)? {
0 => Err(io::Error::from(self.1)),
read => Ok(read),
}
}
}
fn failing(bytes: &[u8], kind: io::ErrorKind) -> Failing {
Failing(Cursor::new(bytes.to_vec()), kind)
}
#[test]
fn a_close_without_tls_close_notify_still_ends_an_unframed_body() {
let stream = failing(b"HTTP/1.1 200 OK\r\n\r\nbody", io::ErrorKind::UnexpectedEof);
let response = read_response(stream).unwrap();
assert_eq!(response.body, b"body");
}
#[test]
fn a_read_error_inside_an_unframed_body_is_reported_not_taken_for_the_close() {
let stream = failing(
b"HTTP/1.1 200 OK\r\n\r\nbody",
io::ErrorKind::ConnectionReset,
);
let error = read_response(stream).unwrap_err();
assert!(
matches!(error, Error::Io(ref inner) if inner.kind() == io::ErrorKind::ConnectionReset),
"{error:?}"
);
}
#[test]
fn a_body_of_exactly_the_limit_is_accepted_under_every_framing() {
let body = vec![b'x'; MAX_BODY_LEN];
let mut sized =
format!("HTTP/1.1 200 OK\r\nContent-Length: {MAX_BODY_LEN}\r\n\r\n").into_bytes();
sized.extend_from_slice(&body);
assert_eq!(read_response(Cursor::new(sized)).unwrap().body, body);
let mut chunked =
format!("HTTP/1.1 200 OK\r\nTransfer-Encoding: chunked\r\n\r\n{MAX_BODY_LEN:x}\r\n")
.into_bytes();
chunked.extend_from_slice(&body);
chunked.extend_from_slice(b"\r\n0\r\n\r\n");
assert_eq!(read_response(Cursor::new(chunked)).unwrap().body, body);
let mut unframed = b"HTTP/1.1 200 OK\r\n\r\n".to_vec();
unframed.extend_from_slice(&body);
assert_eq!(read_response(Cursor::new(unframed)).unwrap().body, body);
}
#[test]
fn a_body_cut_short_of_its_content_length_is_a_closed_connection() {
let bytes = b"HTTP/1.1 200 OK\r\nContent-Length: 10\r\n\r\nshort";
let error = read_response(Cursor::new(bytes)).unwrap_err();
assert!(matches!(error, Error::Closed), "{error:?}");
}
#[test]
fn a_status_line_that_is_not_http_is_malformed() {
let error = read_response(Cursor::new(b"220 hello\r\n\r\n")).unwrap_err();
assert_eq!(error.to_string(), "malformed response: status line");
}
#[test]
fn a_header_without_a_colon_is_malformed() {
let error = read_response(Cursor::new(b"HTTP/1.1 200 OK\r\nnocolon\r\n\r\n")).unwrap_err();
assert_eq!(error.to_string(), "malformed response: header");
}
#[test]
fn a_content_length_beyond_the_limit_is_refused_before_reading_the_body() {
let bytes = format!(
"HTTP/1.1 200 OK\r\nContent-Length: {}\r\n\r\n",
MAX_BODY_LEN + 1
);
let error = read_response(Cursor::new(bytes.into_bytes())).unwrap_err();
assert_eq!(error.to_string(), "the response body exceeds 1048576 bytes");
}
#[test]
fn chunks_summing_beyond_the_limit_and_a_bad_chunk_size_are_refused() {
let mut bytes =
format!("HTTP/1.1 200 OK\r\nTransfer-Encoding: chunked\r\n\r\n{MAX_BODY_LEN:x}\r\n")
.into_bytes();
bytes.resize(bytes.len() + MAX_BODY_LEN, b'x');
bytes.extend_from_slice(b"\r\n1\r\nx\r\n0\r\n\r\n");
let error = read_response(Cursor::new(bytes)).unwrap_err();
assert!(matches!(error, Error::TooLarge), "{error:?}");
let bytes = b"HTTP/1.1 200 OK\r\nTransfer-Encoding: chunked\r\n\r\nzz\r\n";
let error = read_response(Cursor::new(bytes)).unwrap_err();
assert_eq!(error.to_string(), "malformed response: chunk size");
let bytes = b"HTTP/1.1 200 OK\r\nTransfer-Encoding: chunked\r\n\r\n2\r\nabXX\r\n0\r\n\r\n";
let error = read_response(Cursor::new(bytes)).unwrap_err();
assert_eq!(error.to_string(), "malformed response: chunk end");
}
#[test]
fn an_unframed_body_beyond_the_limit_is_refused() {
let mut bytes = b"HTTP/1.1 200 OK\r\n\r\n".to_vec();
bytes.resize(bytes.len() + MAX_BODY_LEN + 1, b'x');
let error = read_response(Cursor::new(bytes)).unwrap_err();
assert!(matches!(error, Error::TooLarge), "{error:?}");
}
#[test]
fn a_line_without_an_end_within_the_limit_and_too_many_headers_are_malformed() {
let bytes = vec![b'x'; MAX_LINE_LEN + 1];
let error = read_response(Cursor::new(bytes)).unwrap_err();
assert_eq!(error.to_string(), "malformed response: line too long");
let mut bytes = b"HTTP/1.1 200 OK\r\n".to_vec();
for _ in 0..=MAX_HEADERS {
bytes.extend_from_slice(b"X: y\r\n");
}
let error = read_response(Cursor::new(bytes)).unwrap_err();
assert_eq!(error.to_string(), "malformed response: too many headers");
}
#[test]
fn an_empty_stream_or_a_head_cut_mid_line_is_a_closed_connection() {
let error = read_response(Cursor::new(b"HTTP/1.1 200 OK\r\nContent-Le")).unwrap_err();
assert!(matches!(error, Error::Closed), "{error:?}");
let error = read_response(Cursor::new(b"")).unwrap_err();
assert_eq!(
error.to_string(),
"the host closed the connection before the response was complete"
);
}
}
@xen-orchestra/vmware-explorer/vectura/src/tls.rs
+692
View File
@@ -0,0 +1,692 @@
//! TLS toward a host: connect with timeouts, judge or capture its certificate, compute thumbprints.
//!
//! [`Client`] carries real traffic and judges the certificate by a [`Trust`]:
//! the system roots, a bundle of anchors, or one pinned thumbprint. The
//! accept-everything verifier never leaves this module: [`capture`] runs a TLS
//! handshake with it, sends nothing afterwards and drops the connection
//! before returning.
use std::fmt;
use std::io;
use std::net::{TcpStream, ToSocketAddrs};
use std::sync::{Arc, Mutex, PoisonError};
use std::time::Duration;
use aws_lc_rs::digest;
use rustls::client::danger::{HandshakeSignatureValid, ServerCertVerified, ServerCertVerifier};
use rustls::crypto::aws_lc_rs::default_provider;
use rustls::crypto::{CryptoProvider, verify_tls12_signature, verify_tls13_signature};
use rustls::{
CertificateError, CipherSuite, ClientConfig, ClientConnection, DigitallySignedStruct,
ProtocolVersion, RootCertStore, SignatureScheme,
};
use rustls_pki_types::{CertificateDer, InvalidDnsNameError, ServerName, UnixTime};
/// How long a TCP connect may take before the host counts as unreachable.
///
/// Ten seconds: a host on the same network answers within milliseconds, and
/// one that takes longer is down or filtered, so waiting out the kernel's own
/// limit of about two minutes would only stall the operator.
pub const CONNECT_TIMEOUT: Duration = Duration::from_secs(10);
/// How long one read or write on a host connection may block.
///
/// Sixty seconds: a busy host can take tens of seconds to answer a request,
/// so a shorter limit would report false failures, while a longer one leaves
/// a dead connection undetected for as long. Every host socket carries it.
pub const IO_TIMEOUT: Duration = Duration::from_secs(60);
/// Errors from connecting to a host or judging its certificate.
#[derive(Debug, thiserror::Error)]
pub enum Error {
/// No address of the host accepted the connection.
#[error("could not connect: {}", describe(.0))]
Connect(#[source] io::Error),
/// The TLS handshake failed, or its I/O timed out.
#[error("tls handshake failed: {}", describe(.0))]
TlsHandshake(#[source] io::Error),
/// The host presented a certificate whose thumbprint is not the pinned one.
#[error("certificate thumbprint mismatch: expected {expected}, the host presented {presented}")]
Mismatch {
/// The pin the certificate had to match.
expected: Pin,
/// The thumbprint of the certificate the host presented, same algorithm.
presented: String,
},
/// The TLS handshake ended without something a completed one always has.
#[error("the tls handshake ended without {0}")]
Incomplete(&'static str),
/// The host is neither a DNS name nor an IP address.
#[error("invalid host name: {0}")]
HostName(#[source] InvalidDnsNameError),
/// A thumbprint is not the hex digest its algorithm produces.
#[error("{text:?} is not a {name} thumbprint of {digits} hex digits")]
Thumbprint {
/// The algorithm the thumbprint was parsed for.
name: &'static str,
/// How many hex digits that algorithm's digest has.
digits: usize,
/// The text that was offered.
text: String,
},
/// No trust anchor could be loaded.
#[error("no trust anchor: {0}")]
Roots(String),
/// rustls rejected the client configuration.
#[error("tls configuration rejected: {0}")]
Config(#[source] rustls::Error),
}
/// What a host presented during a TLS handshake that carried nothing else.
#[derive(Debug)]
pub struct Presented {
/// The first certificate of the chain the host sent, DER encoded.
pub certificate: CertificateDer<'static>,
/// The negotiated protocol version.
pub version: ProtocolVersion,
/// The negotiated cipher suite.
pub suite: CipherSuite,
}
/// How the certificate a host presents is judged.
#[derive(Debug)]
pub enum Trust {
/// The system trust store, with the host name checked against the certificate.
System,
/// The anchors of a PEM bundle, with the host name checked against the certificate.
Anchors(Vec<CertificateDer<'static>>),
/// One exact certificate named by its thumbprint; no chain and no host name.
Pin(Pin),
}
/// The digest the DER certificate a host presents must have.
///
/// Parsed from hex text, colons and case ignored, and compared as bytes.
#[derive(Clone, PartialEq, Eq)]
pub struct Pin {
algorithm: &'static digest::Algorithm,
name: &'static str,
expected: Vec<u8>,
}
impl Pin {
/// Parses a SHA-256 thumbprint: 64 hex digits, colons and case ignored.
///
/// # Errors
/// When `text` is not 64 hex digits once its colons are removed.
pub fn sha256(text: &str) -> Result<Pin, Error> {
Pin::parse(&digest::SHA256, "sha256", text)
}
/// Parses a SHA-1 thumbprint: 40 hex digits, colons and case ignored.
///
/// # Errors
/// When `text` is not 40 hex digits once its colons are removed.
pub fn sha1(text: &str) -> Result<Pin, Error> {
Pin::parse(&digest::SHA1_FOR_LEGACY_USE_ONLY, "sha1", text)
}
fn parse(
algorithm: &'static digest::Algorithm,
name: &'static str,
text: &str,
) -> Result<Pin, Error> {
let digits = algorithm.output_len() * 2;
let malformed = || Error::Thumbprint {
name,
digits,
text: text.to_owned(),
};
let hex: Vec<u8> = text.bytes().filter(|byte| *byte != b':').collect();
if hex.len() != digits {
return Err(malformed());
}
let expected = hex
.chunks(2)
.map(|pair| {
std::str::from_utf8(pair)
.ok()
.and_then(|pair| u8::from_str_radix(pair, 16).ok())
.ok_or_else(malformed)
})
.collect::<Result<Vec<u8>, Error>>()?;
Ok(Pin {
algorithm,
name,
expected,
})
}
/// Whether `der` has the pinned digest.
#[must_use]
pub fn matches(&self, der: &[u8]) -> bool {
digest::digest(self.algorithm, der).as_ref() == self.expected.as_slice()
}
/// The digest of `der` with this pin's algorithm, rendered like the pin.
#[must_use]
pub fn digest_of(&self, der: &[u8]) -> String {
colon_hex(digest::digest(self.algorithm, der).as_ref())
}
}
impl fmt::Display for Pin {
fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result {
write!(formatter, "{}:{}", self.name, colon_hex(&self.expected))
}
}
impl fmt::Debug for Pin {
fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result {
fmt::Display::fmt(self, formatter)
}
}
/// A TLS client configuration built for one [`Trust`].
#[derive(Debug)]
pub struct Client {
config: Arc<ClientConfig>,
pinned: Option<Arc<Verifier>>,
}
impl Client {
/// Builds a client that judges certificates by `trust`.
///
/// # Errors
/// When no anchor can be loaded or rustls rejects the configuration.
pub fn new(trust: &Trust) -> Result<Client, Error> {
let provider = Arc::new(default_provider());
let builder = ClientConfig::builder_with_provider(Arc::clone(&provider))
.with_safe_default_protocol_versions()
.map_err(Error::Config)?;
let mut roots = RootCertStore::empty();
let (config, pinned) = match trust {
Trust::Pin(pin) => {
let verifier = Arc::new(Verifier::new(Some(pin.clone()), provider));
let config = builder
.dangerous()
.with_custom_certificate_verifier(
Arc::clone(&verifier) as Arc<dyn ServerCertVerifier>
)
.with_no_client_auth();
(config, Some(verifier))
}
Trust::Anchors(anchors) => {
roots.add_parsable_certificates(anchors.iter().cloned());
if roots.is_empty() {
return Err(Error::Roots("the bundle holds no certificate".to_owned()));
}
(
builder.with_root_certificates(roots).with_no_client_auth(),
None,
)
}
Trust::System => {
let loaded = rustls_native_certs::load_native_certs();
roots.add_parsable_certificates(loaded.certs);
if roots.is_empty() {
let reason = loaded.errors.first().map_or_else(
|| "the system store holds no certificate".to_owned(),
ToString::to_string,
);
return Err(Error::Roots(reason));
}
(
builder.with_root_certificates(roots).with_no_client_auth(),
None,
)
}
};
Ok(Client {
config: Arc::new(config),
pinned,
})
}
/// Runs a TLS handshake for `host` on `stream` and returns the session.
///
/// # Errors
/// When `host` is neither a DNS name nor an IP address, the certificate is
/// refused, or the TLS handshake fails or times out.
pub fn handshake(&self, stream: &mut TcpStream, host: &str) -> Result<ClientConnection, Error> {
let name = ServerName::try_from(host)
.map_err(Error::HostName)?
.to_owned();
let mut connection =
ClientConnection::new(Arc::clone(&self.config), name).map_err(Error::Config)?;
while connection.is_handshaking() {
if let Err(error) = connection.complete_io(stream) {
return Err(self
.pinned
.as_ref()
.and_then(|verifier| verifier.rejection())
.unwrap_or(Error::TlsHandshake(error)));
}
}
Ok(connection)
}
}
/// Opens a TCP connection to `host:port` with both timeouts applied.
///
/// Every address the name resolves to is tried in turn, with
/// [`CONNECT_TIMEOUT`] each; the stream returned carries [`IO_TIMEOUT`] for
/// reads and writes.
///
/// # Errors
/// When the name does not resolve, no address accepts the connection in
/// time, or the socket timeouts cannot be set.
pub fn connect(host: &str, port: u16) -> Result<TcpStream, Error> {
let mut last = io::Error::new(io::ErrorKind::NotFound, "the name resolves to no address");
for address in (host, port).to_socket_addrs().map_err(Error::Connect)? {
match TcpStream::connect_timeout(&address, CONNECT_TIMEOUT) {
Ok(stream) => {
stream
.set_read_timeout(Some(IO_TIMEOUT))
.map_err(Error::Connect)?;
stream
.set_write_timeout(Some(IO_TIMEOUT))
.map_err(Error::Connect)?;
return Ok(stream);
}
Err(error) => last = error,
}
}
Err(Error::Connect(last))
}
/// Runs a TLS handshake on `stream` for `host` and returns what the host presented.
///
/// The certificate is accepted whatever it says, because the caller wants its
/// thumbprints rather than a verdict. The TLS handshake signatures are still
/// checked, so the host does hold the private key of the certificate reported.
/// Nothing is sent after the TLS handshake, and the connection is dropped on
/// return.
///
/// # Errors
/// When `host` is neither a DNS name nor an IP address, the TLS handshake
/// fails or times out, or it ends without a certificate.
pub(crate) fn capture(stream: &mut TcpStream, host: &str) -> Result<Presented, Error> {
let provider = Arc::new(default_provider());
let config = ClientConfig::builder_with_provider(Arc::clone(&provider))
.with_safe_default_protocol_versions()
.map_err(Error::Config)?
.dangerous()
.with_custom_certificate_verifier(Arc::new(Verifier::new(None, provider)))
.with_no_client_auth();
let client = Client {
config: Arc::new(config),
pinned: None,
};
let connection = client.handshake(stream, host)?;
let certificate = peer_certificate(&connection)?;
let (version, suite) = negotiated(&connection)?;
Ok(Presented {
certificate,
version,
suite,
})
}
/// The protocol version and cipher suite a completed session negotiated.
///
/// # Errors
/// When the session has not completed its handshake.
pub(crate) fn negotiated(
connection: &ClientConnection,
) -> Result<(ProtocolVersion, CipherSuite), Error> {
let version = connection
.protocol_version()
.ok_or(Error::Incomplete("a protocol version"))?;
let suite = connection
.negotiated_cipher_suite()
.ok_or(Error::Incomplete("a cipher suite"))?
.suite();
Ok((version, suite))
}
/// The first certificate of the chain a completed session's peer sent.
///
/// # Errors
/// When the session carries no certificate.
pub(crate) fn peer_certificate(
connection: &ClientConnection,
) -> Result<CertificateDer<'static>, Error> {
connection
.peer_certificates()
.and_then(|chain| chain.first())
.cloned()
.ok_or(Error::Incomplete("a certificate"))
}
/// Formats the SHA-256 digest of `der` as colon-separated upper-case hex.
#[must_use]
pub fn sha256(der: &[u8]) -> String {
colon_hex(digest::digest(&digest::SHA256, der).as_ref())
}
/// Formats the SHA-1 digest of `der` as colon-separated upper-case hex.
#[must_use]
pub fn sha1(der: &[u8]) -> String {
colon_hex(digest::digest(&digest::SHA1_FOR_LEGACY_USE_ONLY, der).as_ref())
}
/// Renders an I/O error, naming a timeout for what it is.
///
/// A socket with a read or write timeout reports `WouldBlock` on Linux, whose
/// own message ("Resource temporarily unavailable") hides the cause.
pub(crate) fn describe(error: &io::Error) -> String {
match error.kind() {
io::ErrorKind::WouldBlock | io::ErrorKind::TimedOut => "timed out".to_owned(),
_ => error.to_string(),
}
}
fn colon_hex(bytes: &[u8]) -> String {
bytes
.iter()
.map(|byte| format!("{byte:02X}"))
.collect::<Vec<_>>()
.join(":")
}
/// Judges the end-entity certificate by its digest alone, remembering what was presented.
///
/// Without a pin every certificate is accepted, which only [`capture`] wants.
/// TLS handshake signatures are verified with the provider's algorithms
/// either way, so a completed TLS handshake proves the host holds the
/// certificate's key.
#[derive(Debug)]
struct Verifier {
pin: Option<Pin>,
provider: Arc<CryptoProvider>,
presented: Mutex<Option<CertificateDer<'static>>>,
}
impl Verifier {
fn new(pin: Option<Pin>, provider: Arc<CryptoProvider>) -> Verifier {
Verifier {
pin,
provider,
presented: Mutex::new(None),
}
}
/// The mismatch error, when a certificate was presented and refused.
fn rejection(&self) -> Option<Error> {
let pin = self.pin.as_ref()?;
let presented = self
.presented
.lock()
.unwrap_or_else(PoisonError::into_inner);
let der = presented.as_ref().filter(|der| !pin.matches(der))?;
Some(Error::Mismatch {
expected: pin.clone(),
presented: pin.digest_of(der),
})
}
}
impl ServerCertVerifier for Verifier {
fn verify_server_cert(
&self,
end_entity: &CertificateDer<'_>,
_intermediates: &[CertificateDer<'_>],
_server_name: &ServerName<'_>,
_ocsp_response: &[u8],
_now: UnixTime,
) -> Result<ServerCertVerified, rustls::Error> {
*self
.presented
.lock()
.unwrap_or_else(PoisonError::into_inner) = Some(end_entity.clone().into_owned());
match &self.pin {
Some(pin) if !pin.matches(end_entity) => Err(rustls::Error::InvalidCertificate(
CertificateError::ApplicationVerificationFailure,
)),
_ => Ok(ServerCertVerified::assertion()),
}
}
fn verify_tls12_signature(
&self,
message: &[u8],
cert: &CertificateDer<'_>,
dss: &DigitallySignedStruct,
) -> Result<HandshakeSignatureValid, rustls::Error> {
verify_tls12_signature(
message,
cert,
dss,
&self.provider.signature_verification_algorithms,
)
}
fn verify_tls13_signature(
&self,
message: &[u8],
cert: &CertificateDer<'_>,
dss: &DigitallySignedStruct,
) -> Result<HandshakeSignatureValid, rustls::Error> {
verify_tls13_signature(
message,
cert,
dss,
&self.provider.signature_verification_algorithms,
)
}
fn supported_verify_schemes(&self) -> Vec<SignatureScheme> {
self.provider
.signature_verification_algorithms
.supported_schemes()
}
}
#[cfg(test)]
mod tests {
use std::io::Read;
use std::net::TcpListener;
use std::thread;
use rustls::{ServerConfig, ServerConnection};
use rustls_pki_types::{PrivateKeyDer, PrivatePkcs8KeyDer};
use super::*;
const SHA256_OF_ABC: &str = "BA:78:16:BF:8F:01:CF:EA:41:41:40:DE:5D:AE:22:23:\
B0:03:61:A3:96:17:7A:9C:B4:10:FF:61:F2:00:15:AD";
/// A loopback TLS server presenting a fresh self-signed certificate.
fn server() -> (CertificateDer<'static>, u16) {
let issued = rcgen::generate_simple_self_signed(vec!["esxi.test".to_owned()]).unwrap();
let certificate = issued.cert.der().clone();
let key =
PrivateKeyDer::Pkcs8(PrivatePkcs8KeyDer::from(issued.signing_key.serialize_der()));
let config = Arc::new(
ServerConfig::builder()
.with_no_client_auth()
.with_single_cert(vec![certificate.clone()], key)
.unwrap(),
);
let listener = TcpListener::bind("127.0.0.1:0").unwrap();
let port = listener.local_addr().unwrap().port();
thread::spawn(move || {
let (mut stream, _) = listener.accept().unwrap();
let mut session = ServerConnection::new(config).unwrap();
while session.is_handshaking() {
if session.complete_io(&mut stream).is_err() {
return;
}
}
});
(certificate, port)
}
#[test]
fn sha256_is_rendered_as_upper_case_hex_with_colons() {
assert_eq!(sha256(b"abc"), SHA256_OF_ABC);
}
#[test]
fn sha1_is_rendered_as_upper_case_hex_with_colons() {
assert_eq!(
sha1(b"abc"),
"A9:99:3E:36:47:06:81:6A:BA:3E:25:71:78:50:C2:6C:9C:D0:D8:9D"
);
}
#[test]
fn a_pin_ignores_colons_and_case_and_prints_canonically() {
let lower = SHA256_OF_ABC.replace(':', "").to_lowercase();
let pin = Pin::sha256(&lower).unwrap();
assert_eq!(pin, Pin::sha256(SHA256_OF_ABC).unwrap());
assert!(pin.matches(b"abc"));
assert!(!pin.matches(b"abd"));
assert_eq!(pin.to_string(), format!("sha256:{SHA256_OF_ABC}"));
assert_eq!(format!("{pin:?}"), pin.to_string());
assert_eq!(pin.digest_of(b"abc"), SHA256_OF_ABC);
}
#[test]
fn a_sha1_pin_wants_forty_digits_and_a_sha256_pin_sixty_four() {
let sha1_text = sha1(b"abc");
assert!(Pin::sha1(&sha1_text).unwrap().matches(b"abc"));
let error = Pin::sha256(&sha1_text).unwrap_err();
assert_eq!(
error.to_string(),
format!("{sha1_text:?} is not a sha256 thumbprint of 64 hex digits")
);
let error = Pin::sha1(SHA256_OF_ABC).unwrap_err();
assert!(
error
.to_string()
.ends_with("is not a sha1 thumbprint of 40 hex digits")
);
}
#[test]
fn a_pin_with_a_non_hex_digit_or_a_multibyte_character_is_refused() {
let with_letter = format!("G{}", &SHA256_OF_ABC[1..]);
assert!(matches!(
Pin::sha256(&with_letter),
Err(Error::Thumbprint { name: "sha256", .. })
));
let with_accent = format!("é{}", &SHA256_OF_ABC[2..]);
assert!(Pin::sha256(&with_accent).is_err());
let with_three_byte = format!("€{}", &SHA256_OF_ABC[3..]);
assert!(Pin::sha256(&with_three_byte).is_err());
}
#[test]
fn a_pinned_client_completes_the_handshake_with_the_pinned_certificate() {
let (certificate, port) = server();
let client = Client::new(&Trust::Pin(Pin::sha256(&sha256(&certificate)).unwrap())).unwrap();
let mut stream = connect("127.0.0.1", port).unwrap();
let session = client.handshake(&mut stream, "127.0.0.1").unwrap();
assert_eq!(peer_certificate(&session).unwrap(), certificate);
}
#[test]
fn a_pinned_client_refuses_another_certificate_naming_both_thumbprints() {
let (certificate, port) = server();
let pin = Pin::sha1(&sha1(b"another certificate")).unwrap();
let client = Client::new(&Trust::Pin(pin.clone())).unwrap();
let mut stream = connect("127.0.0.1", port).unwrap();
let error = client.handshake(&mut stream, "127.0.0.1").unwrap_err();
assert_eq!(
error.to_string(),
format!(
"certificate thumbprint mismatch: expected {pin}, the host presented {}",
sha1(&certificate)
)
);
}
#[test]
fn a_self_signed_certificate_is_its_own_anchor_but_needs_the_right_name() {
let (certificate, port) = server();
let client = Client::new(&Trust::Anchors(vec![certificate])).unwrap();
let mut stream = connect("127.0.0.1", port).unwrap();
let error = client.handshake(&mut stream, "127.0.0.1").unwrap_err();
assert!(matches!(error, Error::TlsHandshake(_)), "{error:?}");
assert!(error.to_string().starts_with("tls handshake failed: "));
}
#[test]
fn an_empty_bundle_is_refused_before_connecting() {
let error = Client::new(&Trust::Anchors(Vec::new())).unwrap_err();
assert_eq!(
error.to_string(),
"no trust anchor: the bundle holds no certificate"
);
}
#[test]
fn the_system_roots_refuse_a_self_signed_certificate() {
let (_, port) = server();
let client = Client::new(&Trust::System).unwrap();
let mut stream = connect("127.0.0.1", port).unwrap();
let error = client.handshake(&mut stream, "127.0.0.1").unwrap_err();
assert!(matches!(error, Error::TlsHandshake(_)), "{error:?}");
}
#[test]
fn a_timeout_is_described_as_such_and_other_errors_keep_their_message() {
let timeout = io::Error::from(io::ErrorKind::WouldBlock);
assert_eq!(describe(&timeout), "timed out");
let timeout = io::Error::from(io::ErrorKind::TimedOut);
assert_eq!(describe(&timeout), "timed out");
let other = io::Error::other("boom");
assert_eq!(describe(&other), "boom");
}
#[test]
fn connect_applies_the_io_timeout_to_both_directions() {
let listener = TcpListener::bind("127.0.0.1:0").unwrap();
let port = listener.local_addr().unwrap().port();
let stream = connect("127.0.0.1", port).unwrap();
assert_eq!(stream.read_timeout().unwrap(), Some(IO_TIMEOUT));
assert_eq!(stream.write_timeout().unwrap(), Some(IO_TIMEOUT));
}
#[test]
fn a_refused_port_is_reported_as_a_connect_failure() {
let port = TcpListener::bind("127.0.0.1:0")
.unwrap()
.local_addr()
.unwrap()
.port();
let error = connect("127.0.0.1", port).unwrap_err();
assert!(matches!(error, Error::Connect(_)), "{error:?}");
assert!(error.to_string().starts_with("could not connect: "));
}
#[test]
fn an_invalid_host_name_is_refused_before_any_byte_is_sent() {
let listener = TcpListener::bind("127.0.0.1:0").unwrap();
let mut stream = TcpStream::connect(listener.local_addr().unwrap()).unwrap();
let error = capture(&mut stream, "not a name").unwrap_err();
assert!(matches!(error, Error::HostName(_)), "{error:?}");
assert!(error.to_string().starts_with("invalid host name: "));
drop(stream);
let (mut peer, _) = listener.accept().unwrap();
let mut received = Vec::new();
peer.read_to_end(&mut received).unwrap();
assert!(received.is_empty(), "bytes were sent: {received:?}");
}
}
@xen-orchestra/vmware-explorer/vectura/src/transcript.rs
+130
View File
@@ -0,0 +1,130 @@
//! The transcript `--verbose` writes to standard error.
//!
//! The transcript's lines are formatted here and nowhere else, so what it may
//! say is checked in one place: no password, ticket or cookie is ever passed
//! in, and the dialogue line that carries the ticket is rendered with a
//! placeholder.
use std::io::{self, Write};
use std::time::Duration;
use rustls::{CipherSuite, ProtocolVersion};
/// Bytes served between two progress lines: one gibibyte.
pub const PROGRESS_STEP: u64 = 1 << 30;
/// Where a session's lines go: standard error when enabled, nowhere otherwise.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct Transcript {
enabled: bool,
}
impl Transcript {
/// A transcript that writes its lines when `enabled`.
#[must_use]
pub const fn new(enabled: bool) -> Transcript {
Transcript { enabled }
}
/// A management call, the status it was answered with, and how long it took.
pub fn call(&self, name: &str, status: u16, took: Duration) {
self.line(&format!("{name}: {status} in {} ms", took.as_millis()));
}
/// The capability list the data port greeted with.
pub fn greeting(&self, capabilities: &[String]) {
self.line(&format!("greeting: {}", capabilities.join(", ")));
}
/// The protocol version and cipher suite the TLS session at `place` negotiated.
pub fn tls(&self, place: &str, version: ProtocolVersion, suite: CipherSuite) {
self.line(&format!("{place} tls {version:?} {suite:?}"));
}
/// A dialogue line sent to the host, without its line end; never the ticket.
pub fn sent(&self, line: &str) {
self.line(&format!("> {line}"));
}
/// A dialogue reply the host sent, without its line end.
pub fn received(&self, line: &str) {
self.line(&format!("< {line}"));
}
/// A handshake message of type `kind` sent.
pub fn handshake_sent(&self, kind: u32) {
self.line(&format!("> handshake type {kind}"));
}
/// A handshake message of type `kind` received.
pub fn handshake_received(&self, kind: u32) {
self.line(&format!("< handshake type {kind}"));
}
/// The disk as the host opened it: its capacity and sector size in bytes.
pub fn opened(&self, capacity: u64, sector_size: u32) {
self.line(&format!(
"open: {capacity} bytes, sector size {sector_size}"
));
}
/// An NBD option by number and what it was answered.
pub fn option(&self, option: u32, reply: &str) {
self.line(&format!("nbd option {option}: {reply}"));
}
/// The export the client entered, by size.
pub fn export(&self, size: u64) {
self.line(&format!("nbd export: {size} bytes"));
}
/// Progress: whole gibibytes served, wire bytes per disk byte so far, the depth in use.
pub fn progress(&self, served: u64, wire: u64, logical: u64, depth: usize) {
self.line(&format!(
"progress: served {} GiB, compression ratio {}, depth {depth}",
served / PROGRESS_STEP,
ratio(wire, logical)
));
}
/// The run's end: bytes served, wire bytes, host reads, time since negotiation began.
pub fn summary(&self, served: u64, wire: u64, reads: u64, elapsed: Duration) {
self.line(&format!(
"summary: bytes served {served}, wire bytes {wire}, host reads {reads}, elapsed {:.1} s",
elapsed.as_secs_f64()
));
}
fn line(self, text: &str) {
if self.enabled {
// A line standard error cannot take is dropped: the disk stream on
// standard output must not stop because the transcript's reader left.
let _ = writeln!(io::stderr().lock(), "{text}");
}
}
}
/// `wire` over `logical` with two decimals, in integer arithmetic; 0.00 before any byte.
fn ratio(wire: u64, logical: u64) -> String {
let hundredths = wire.saturating_mul(100) / logical.max(1);
format!("{}.{:02}", hundredths / 100, hundredths % 100)
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn the_ratio_has_two_decimals_and_is_zero_before_any_byte() {
for (wire, logical, expected) in [
(0, 0, "0.00"),
(42, 100, "0.42"),
(100, 100, "1.00"),
(1_234, 1_000, "1.23"),
(5, 1_000, "0.00"),
(3, 2, "1.50"),
] {
assert_eq!(ratio(wire, logical), expected, "{wire}/{logical}");
}
}
}
@xen-orchestra/vmware-explorer/vectura/tests/bundled_binary.rs
+27
View File
@@ -0,0 +1,27 @@
//! Holds the binary that ships next to the sources.
//!
//! `@xen-orchestra/vmware-explorer` runs the `vectura` binary of the crate
//! root, so it is the build of the version of Cargo.toml and it stays
//! executable. scripts/release.sh bumps the version and rebuilds the binary
//! in the same commit.
#![cfg(all(target_os = "linux", target_arch = "x86_64"))]
use std::path::Path;
use std::process::Command;
#[test]
fn the_binary_at_the_crate_root_runs_as_this_version() {
let binary = Path::new(env!("CARGO_MANIFEST_DIR")).join("vectura");
let output = Command::new(&binary)
.arg("--version")
.output()
.expect("the binary at the crate root runs");
assert_eq!(
String::from_utf8_lossy(&output.stdout),
format!("vectura {}\n", env!("CARGO_PKG_VERSION")),
"release with `scripts/bump-pkg vectura <type>` rather than editing the version by hand"
);
}
@xen-orchestra/vmware-explorer/vectura/tests/cli.rs
+158
View File
@@ -0,0 +1,158 @@
//! Runs the built `vectura` binary the way an operator would.
//!
//! Spawning the real binary covers `main` itself, including in the mutation
//! run.
use std::process::{Command, Output};
fn vectura(args: &[&str]) -> Output {
Command::new(env!("CARGO_BIN_EXE_vectura"))
.args(args)
.output()
.expect("the vectura binary runs")
}
/// Asserts exit 2 with an explanation on standard error and empty standard output.
fn assert_usage_error(output: &Output) {
assert_eq!(output.status.code(), Some(2), "usage errors exit with 2");
let stderr = String::from_utf8_lossy(&output.stderr);
assert!(
stderr.starts_with("vectura: "),
"the explanation goes to standard error: {stderr}"
);
assert!(
output.stdout.is_empty(),
"nothing is written to standard output"
);
}
#[test]
fn running_without_a_command_is_a_usage_error() {
assert_usage_error(&vectura(&[]));
}
#[test]
fn an_unknown_flag_is_a_usage_error() {
let output = vectura(&["fingerprint", "--host", "esxi.test", "--bogus"]);
assert_usage_error(&output);
assert!(String::from_utf8_lossy(&output.stderr).contains("--bogus"));
}
#[test]
fn a_host_with_an_unparsable_port_is_a_usage_error() {
let output = vectura(&["fingerprint", "--host", "esxi.test:notaport"]);
assert_usage_error(&output);
assert!(String::from_utf8_lossy(&output.stderr).contains("notaport"));
}
#[test]
fn help_goes_to_standard_output_and_succeeds() {
let output = vectura(&["--help"]);
assert_eq!(output.status.code(), Some(0));
assert!(String::from_utf8_lossy(&output.stdout).contains("fingerprint"));
assert!(output.stderr.is_empty());
}
#[test]
fn a_depth_outside_one_to_thirty_two_is_a_usage_error() {
for depth in ["0", "33"] {
let output = vectura(&[
"serve",
"--host",
"esxi.test",
"--user",
"root",
"--vm-id",
"1",
"--disk",
"d",
"--depth",
depth,
]);
assert_usage_error(&output);
let stderr = String::from_utf8_lossy(&output.stderr);
assert!(
stderr.contains(&format!("{depth} is not in 1..=32")),
"{stderr}"
);
}
}
#[test]
fn the_serve_help_says_the_depth_limit_is_shared_per_host() {
let output = vectura(&["serve", "--help"]);
assert_eq!(output.status.code(), Some(0));
let help = String::from_utf8_lossy(&output.stdout);
assert!(help.contains("--depth <DEPTH>"), "{help}");
assert!(help.contains("per host"), "{help}");
}
#[test]
fn a_transport_other_than_nfcssl_or_nfc_is_a_usage_error() {
let output = vectura(&[
"serve",
"--host",
"esxi.test",
"--user",
"root",
"--vm-id",
"1",
"--disk",
"d",
"--transport",
"tcp",
]);
assert_usage_error(&output);
let stderr = String::from_utf8_lossy(&output.stderr);
assert!(stderr.contains("tcp is not nfcssl or nfc"), "{stderr}");
}
#[test]
fn the_serve_help_lists_the_transport_defaulting_to_nfcssl_and_the_verbose_flag() {
let output = vectura(&["serve", "--help"]);
assert_eq!(output.status.code(), Some(0));
let help = String::from_utf8_lossy(&output.stdout);
assert!(help.contains("--transport <TRANSPORT>"), "{help}");
assert!(help.contains("[default: nfcssl]"), "{help}");
assert!(help.contains("--verbose"), "{help}");
assert!(help.contains("[default: 16]"), "{help}");
}
#[test]
fn a_compression_other_than_skipz_zlib_or_none_is_a_usage_error() {
let output = vectura(&[
"serve",
"--host",
"esxi.test",
"--user",
"root",
"--vm-id",
"1",
"--disk",
"d",
"--compression",
"gzip",
]);
assert_usage_error(&output);
let stderr = String::from_utf8_lossy(&output.stderr);
assert!(
stderr.contains("gzip is not none, zlib or skipz"),
"{stderr}"
);
}
#[test]
fn the_serve_help_lists_the_three_compressions_with_skipz_as_default() {
let output = vectura(&["serve", "--help"]);
assert_eq!(output.status.code(), Some(0));
let help = String::from_utf8_lossy(&output.stdout);
for text in [
"--compression <COMPRESSION>",
"skipz",
"zlib",
"none",
"[default: skipz]",
] {
assert!(help.contains(text), "{text}: {help}");
}
}
@xen-orchestra/vmware-explorer/vectura/tests/coverage.rs
+71
View File
@@ -0,0 +1,71 @@
//! Holds the line coverage floor of `src/`.
//!
//! Reads the `cargo llvm-cov` JSON report and compares the line coverage of
//! `src/` (minus `main.rs`) against [`FLOOR_PERCENT`].
use std::fs;
use std::path::Path;
use serde_json::Value;
/// The floor, in whole percent of the lines of `src/`.
///
/// A pull request that drops below it adds tests; the number only goes up.
const FLOOR_PERCENT: u64 = 80;
/// Line totals `(covered, count)` over the files the floor applies to.
fn src_lines(report: &Value, root: &Path) -> (u64, u64) {
let files = report["data"][0]["files"]
.as_array()
.expect("an llvm-cov report lists its files");
let src = root.join("src");
let main = src.join("main.rs");
let mut covered = 0;
let mut count = 0;
for file in files {
let name = Path::new(file["filename"].as_str().expect("a file has a name"));
if !name.starts_with(&src) || name == main {
continue;
}
let lines = &file["summary"]["lines"];
covered += lines["covered"]
.as_u64()
.expect("covered lines is an integer");
count += lines["count"].as_u64().expect("line count is an integer");
}
(covered, count)
}
#[test]
#[ignore = "needs target/coverage.json from cargo llvm-cov; CI runs it with -- --ignored"]
fn src_line_coverage_meets_the_floor() {
let root = Path::new(env!("CARGO_MANIFEST_DIR"));
let report = fs::read_to_string(root.join("target/coverage.json"))
.expect("target/coverage.json exists; run cargo llvm-cov first");
let report: Value = serde_json::from_str(&report).expect("the report is JSON");
let (covered, count) = src_lines(&report, root);
if count == 0 {
// Nothing executable under src/ yet: there is no percentage to hold.
return;
}
let percent = covered * 100 / count;
assert!(
covered * 100 >= FLOOR_PERCENT * count,
"line coverage of src/ is {percent}% ({covered}/{count}), below the {FLOOR_PERCENT}% \
floor; add tests, do not lower the floor"
);
}
#[test]
fn main_rs_and_files_outside_src_are_left_out() {
let root = Path::new("/repo");
let report = serde_json::json!({ "data": [{ "files": [
{ "filename": "/repo/src/lib.rs", "summary": { "lines": { "covered": 8, "count": 10 } } },
{ "filename": "/repo/src/nfc/record.rs", "summary": { "lines": { "covered": 1, "count": 2 } } },
{ "filename": "/repo/src/main.rs", "summary": { "lines": { "covered": 0, "count": 50 } } },
{ "filename": "/repo/tests/cli.rs", "summary": { "lines": { "covered": 0, "count": 50 } } }
] }] });
assert_eq!(src_lines(&report, root), (9, 12));
}
@xen-orchestra/vmware-explorer/vectura/tests/forbidden_tools.rs
+175
View File
@@ -0,0 +1,175 @@
//! Holds the "no escape hatch" rules in place.
//!
//! Every file under `src/` is scanned for lint suppressions and for
//! `.unwrap()` / `.expect(` outside a `#[cfg(test)]` region; `tests/` is not
//! scanned. The scan is textual on purpose: it reads what a reviewer reads.
use std::fs;
use std::path::{Path, PathBuf};
/// A rule violation: which file, which line (1-based), what was found.
#[derive(Debug, PartialEq, Eq)]
struct Finding {
path: PathBuf,
line: usize,
what: &'static str,
}
/// Lint suppressions, refused anywhere in `src/`, test regions included.
///
/// A `cfg_attr` that carries one is refused too; a plain `cfg_attr` is not.
const REFUSED_EVERYWHERE: [&str; 4] = ["#[allow(", "#![allow(", "#[expect(", "#![expect("];
/// Panics on `Option`/`Result`, refused outside `#[cfg(test)]` regions.
const REFUSED_OUTSIDE_TESTS: [&str; 2] = [".unwrap()", ".expect("];
/// The source files to scan: every `.rs` file under `dir`, recursively.
fn rust_files(dir: &Path) -> Vec<PathBuf> {
let mut files = Vec::new();
let entries = fs::read_dir(dir).unwrap_or_else(|error| panic!("{}: {error}", dir.display()));
for entry in entries {
let path = entry.expect("directory entry is readable").path();
if path.is_dir() {
files.extend(rust_files(&path));
} else if path.extension().is_some_and(|extension| extension == "rs") {
files.push(path);
}
}
files.sort();
files
}
/// Scans one file's text and reports every refused construct.
///
/// A `#[cfg(test)]` attribute opens a region, from the next `{` to its matching
/// `}`, exempt from [`REFUSED_OUTSIDE_TESTS`]. Line comments are stripped
/// first. Braces in strings can only lengthen a region, never hide a finding.
fn scan(path: &Path, text: &str) -> Vec<Finding> {
let mut findings = Vec::new();
let mut depth = 0_usize;
let mut region_pending = false;
let mut in_region = false;
for (index, raw) in text.lines().enumerate() {
let code = raw.split("//").next().unwrap_or_default();
if code.trim_start().starts_with("#[cfg(test)]") {
region_pending = true;
}
for what in REFUSED_EVERYWHERE {
if code.contains(what) {
findings.push(Finding {
path: path.to_path_buf(),
line: index + 1,
what,
});
}
}
if code.contains("cfg_attr(") && (code.contains("allow(") || code.contains("expect(")) {
findings.push(Finding {
path: path.to_path_buf(),
line: index + 1,
what: "cfg_attr(",
});
}
if !in_region {
for what in REFUSED_OUTSIDE_TESTS {
if code.contains(what) {
findings.push(Finding {
path: path.to_path_buf(),
line: index + 1,
what,
});
}
}
}
for character in code.chars() {
match character {
'{' => {
if region_pending {
region_pending = false;
in_region = true;
depth = 0;
}
depth += 1;
}
'}' => {
depth = depth.saturating_sub(1);
if in_region && depth == 0 {
in_region = false;
}
}
_ => {}
}
}
}
findings
}
#[test]
fn src_contains_no_forbidden_tool() {
let src = Path::new(env!("CARGO_MANIFEST_DIR")).join("src");
let mut findings = Vec::new();
for path in rust_files(&src) {
let text =
fs::read_to_string(&path).unwrap_or_else(|error| panic!("{}: {error}", path.display()));
findings.extend(scan(&path, &text));
}
assert!(
findings.is_empty(),
"forbidden constructs under src/ (fix the cause, do not suppress the warning):\n{}",
findings
.iter()
.map(|finding| format!(
" {}:{}: {}",
finding.path.display(),
finding.line,
finding.what
))
.collect::<Vec<_>>()
.join("\n")
);
}
#[test]
fn a_lint_suppression_is_found_anywhere() {
let text = "#![allow(dead_code)]\n\n#[cfg(test)]\nmod tests {\n #[allow(unused)]\n fn f() {}\n}\n";
let found: Vec<_> = scan(Path::new("x.rs"), text)
.into_iter()
.map(|f| (f.line, f.what))
.collect();
assert_eq!(found, [(1, "#![allow("), (5, "#[allow(")]);
}
#[test]
fn a_panic_on_option_is_found_outside_a_test_region_only() {
let text = "fn f() -> u8 {\n Some(1).unwrap()\n}\n\n#[cfg(test)]\nmod tests {\n #[test]\n fn g() {\n Some(1).unwrap();\n Ok::<(), ()>(()).expect(\"fine here\");\n }\n}\n\nfn h() {\n None::<u8>.expect(\"back outside\");\n}\n";
let found: Vec<_> = scan(Path::new("x.rs"), text)
.into_iter()
.map(|f| (f.line, f.what))
.collect();
assert_eq!(found, [(2, ".unwrap()"), (15, ".expect(")]);
}
#[test]
fn a_cfg_attr_is_refused_only_when_it_carries_a_suppression() {
let text = "#[cfg_attr(test, derive(PartialEq))]\nstruct A;\n#[cfg_attr(test, allow(dead_code))]\nstruct B;\n";
let found: Vec<_> = scan(Path::new("x.rs"), text)
.into_iter()
.map(|f| (f.line, f.what))
.collect();
assert_eq!(found, [(3, "cfg_attr(")]);
}
#[test]
fn a_comment_about_unwrap_is_not_a_finding() {
let text = "// never call .unwrap() here\nfn f() {} // .expect( in a trailing comment\n";
assert!(scan(Path::new("x.rs"), text).is_empty());
}
@xen-orchestra/vmware-explorer/vectura/tests/live_policy.rs
+70
View File
@@ -0,0 +1,70 @@
//! Holds the live tests to their rules: ignored, and in one file.
//!
//! The live tests reach the lab hosts, so they must stay out of every
//! `cargo test` that lacks `--ignored`, and stay in the one file this test
//! reads.
use std::fs;
use std::path::{Path, PathBuf};
/// The one file that may read the lab.
const LIVE: &str = "tests/process/live.rs";
/// The text of the file at `relative`, from the crate root.
fn read(relative: &Path) -> String {
let path = Path::new(env!("CARGO_MANIFEST_DIR")).join(relative);
fs::read_to_string(&path).unwrap_or_else(|error| panic!("{}: {error}", path.display()))
}
/// Every `.rs` file under `dir`, recursively, sorted.
fn rust_files(dir: &Path) -> Vec<PathBuf> {
let mut files = Vec::new();
let entries = fs::read_dir(dir).unwrap_or_else(|error| panic!("{}: {error}", dir.display()));
for entry in entries {
let path = entry.expect("directory entry is readable").path();
if path.is_dir() {
files.extend(rust_files(&path));
} else if path.extension().is_some_and(|extension| extension == "rs") {
files.push(path);
}
}
files.sort();
files
}
#[test]
fn every_live_test_is_ignored() {
let live = read(Path::new(LIVE));
let tests = live
.match_indices("#[test]")
.map(|(at, _)| at)
.collect::<Vec<_>>();
assert!(!tests.is_empty(), "{LIVE} holds no test");
for at in tests {
let next = live[at + "#[test]".len()..].trim_start();
assert!(
next.starts_with("#[ignore"),
"the test at byte {at} of {LIVE} is not ignored"
);
}
}
#[test]
fn no_source_outside_the_live_file_reads_the_lab() {
let root = Path::new(env!("CARGO_MANIFEST_DIR"));
let files = rust_files(&root.join("src"))
.into_iter()
.chain(rust_files(&root.join("tests")));
for path in files {
if path.ends_with(LIVE) || path.ends_with(file!()) {
continue;
}
let text =
fs::read_to_string(&path).unwrap_or_else(|error| panic!("{}: {error}", path.display()));
assert!(
!text.contains("VECTURA_LAB_"),
"{} reads the lab outside {LIVE}",
path.display()
);
}
}
@xen-orchestra/vmware-explorer/vectura/tests/process/fingerprint.rs
+168
View File
@@ -0,0 +1,168 @@
//! Runs `vectura fingerprint` against a mock host on loopback.
use std::net::TcpListener;
use std::process::{Command, Output};
use crate::support::{DataPort, GREETING, GREETING_WITHOUT_SHA256, Host, Observed, text};
/// Runs the built binary with `args` and waits for it.
fn vectura(args: &[&str]) -> Output {
Command::new(env!("CARGO_BIN_EXE_vectura"))
.args(args)
.output()
.expect("the vectura binary runs")
}
/// Runs `vectura fingerprint` against `host`.
fn fingerprint(host: &Host, verbose: bool) -> Output {
let management = format!("127.0.0.1:{}", host.management_port);
let data = host.data_port.to_string();
let mut args = vec!["fingerprint", "--host", &management, "--nfc-port", &data];
if verbose {
args.push("--verbose");
}
vectura(&args)
}
/// The four lines `fingerprint` must print for `host`, management port first.
fn expected_stdout(host: &Host) -> String {
let (sha256, sha1) = (host.sha256(), host.sha1());
let (management, data) = (host.management_port, host.data_port);
format!(
"{management} sha256 {sha256}\n{management} sha1 {sha1}\n\
{data} sha256 {sha256}\n{data} sha1 {sha1}\n"
)
}
/// The verbose line for `port`, built from what the mock negotiated.
fn verbose_line(port: u16, observed: &Observed) -> String {
let version = observed
.version
.as_deref()
.expect("a version was negotiated");
let suite = observed.suite.as_deref().expect("a suite was negotiated");
format!("{port} tls {version} {suite}\n")
}
/// A loopback port nobody listens on, released again before the test uses it.
fn refused_port() -> u16 {
TcpListener::bind("127.0.0.1:0")
.expect("loopback is bindable")
.local_addr()
.expect("the listener has an address")
.port()
}
/// Asserts exit 1 with nothing on standard output and returns the explanation.
///
/// The explanation starts with `vectura: <address>: `, naming the port that failed.
fn failure_naming(output: &Output, address: &str) -> String {
assert_eq!(output.status.code(), Some(1));
assert!(output.stdout.is_empty(), "stdout: {}", text(&output.stdout));
let stderr = text(&output.stderr);
assert!(
stderr.starts_with(&format!("vectura: {address}: ")),
"{stderr}"
);
stderr
}
#[test]
fn fingerprint_prints_both_thumbprints_management_port_first() {
let host = Host::with_data_port(DataPort::Greeting(GREETING));
let output = fingerprint(&host, false);
assert_eq!(
output.status.code(),
Some(0),
"stderr: {}",
text(&output.stderr)
);
assert_eq!(text(&output.stdout), expected_stdout(&host));
assert!(output.stderr.is_empty(), "stderr: {}", text(&output.stderr));
for observed in [host.management(), host.data()] {
assert!(observed.handshake_completed, "{observed:?}");
assert_eq!(
observed.bytes_after_tls_handshake, 0,
"nothing is sent after the TLS handshake: {observed:?}"
);
}
}
#[test]
fn verbose_adds_the_negotiated_version_and_suite_on_standard_error_only() {
let host = Host::with_data_port(DataPort::Greeting(GREETING));
let output = fingerprint(&host, true);
assert_eq!(
output.status.code(),
Some(0),
"stderr: {}",
text(&output.stderr)
);
assert_eq!(text(&output.stdout), expected_stdout(&host));
let expected = verbose_line(host.management_port, &host.management())
+ &verbose_line(host.data_port, &host.data());
assert_eq!(text(&output.stderr), expected);
}
#[test]
fn a_greeting_without_sha256_support_is_refused_before_tls_starts() {
let host = Host::with_data_port(DataPort::Greeting(GREETING_WITHOUT_SHA256));
let output = fingerprint(&host, false);
let stderr = failure_naming(&output, &format!("127.0.0.1:{}", host.data_port));
assert!(stderr.contains("\"SHA256 supported\""), "{stderr}");
assert!(
stderr.contains(
"SSL Required, ServerDaemonProtocol:SOAP, MKSDisplayProtocol:VNC, \
VMXARGS supported, NFCSSL supported"
),
"the announced list is quoted: {stderr}"
);
assert!(
!host.data().handshake_started,
"no TLS is started on the data port"
);
}
#[test]
fn a_host_that_closes_without_a_greeting_fails_naming_the_greeting() {
let host = Host::with_data_port(DataPort::Silent);
let output = fingerprint(&host, false);
let stderr = failure_naming(&output, &format!("127.0.0.1:{}", host.data_port));
assert!(stderr.contains("greeting"), "{stderr}");
}
#[test]
fn a_refused_management_port_fails_naming_its_address() {
let address = format!("127.0.0.1:{}", refused_port());
let output = vectura(&["fingerprint", "--host", &address]);
let stderr = failure_naming(&output, &address);
assert!(stderr.contains("connect"), "{stderr}");
}
#[test]
fn a_refused_data_port_fails_naming_its_address() {
let host = Host::with_data_port(DataPort::Greeting(GREETING));
let management = format!("127.0.0.1:{}", host.management_port);
let data = refused_port();
let output = vectura(&[
"fingerprint",
"--host",
&management,
"--nfc-port",
&data.to_string(),
]);
failure_naming(&output, &format!("127.0.0.1:{data}"));
assert!(host.management().handshake_completed);
}
@xen-orchestra/vmware-explorer/vectura/tests/process/live.rs
+402
View File
@@ -0,0 +1,402 @@
//! Live tests against the lab hosts, ignored and run by hand.
//!
//! `cargo test --locked -- --ignored` runs them. A test that serves a disk
//! takes its lab from `VECTURA_LAB_HOST`, `VECTURA_LAB_USER`, `VECTURA_LAB_VM`,
//! `VECTURA_LAB_DISK` and `VECTURA_LAB_THUMBPRINT`; the legacy check takes
//! `VECTURA_LAB_LEGACY_HOST` alone. A test skips, naming what is missing, when
//! a variable is not set. The child inherits `VECTURA_PASSWORD`; no test reads it.
use std::fs::File;
use std::io::{self, Read, Seek, SeekFrom, Write};
use std::panic::{self, AssertUnwindSafe};
use std::process::{Child, Command, Stdio};
use std::thread;
use std::time::Duration;
use crate::serve::{READ_ONLY, SECTOR, disconnect, finish, negotiate, read_at};
use crate::support::text;
/// The longest read the export accepts: the oracle's step and each sampled range.
const READ: u64 = 32 << 20;
/// The idle gap when `VECTURA_LAB_IDLE_GAP` is not set.
///
/// Five minutes: longer than any pause seen between two reads of one session
/// and inside the 30 minutes after which hostd expires an abandoned session.
const DEFAULT_IDLE_GAP: Duration = Duration::from_secs(300);
/// The transcript line `serve` writes when the client negotiated with `NBD_OPT_GO`.
const GO: &str = "nbd option 7: info\n";
/// The bytes the libnbd program writes for the export size: one big-endian `u64`.
const SIZE_BYTES: usize = std::mem::size_of::<u64>();
/// Writes a line straight to standard error, which the test harness does not capture.
///
/// A skip and a measurement show up without `--nocapture`.
fn report(line: &str) {
let _ = writeln!(io::stderr(), "{line}");
}
/// Whether `variable` is set, with a skip line naming it when it is not.
///
/// The value itself is never read, so `VECTURA_PASSWORD` stays in the environment.
fn present(variable: &str) -> bool {
let present = std::env::var_os(variable).is_some();
if !present {
report(&format!("skipping: {variable} is not set"));
}
present
}
/// The value of `variable`, or `None` after a skip line naming it.
fn lab(variable: &str) -> Option<String> {
present(variable).then(|| std::env::var(variable).expect("a set variable is unicode"))
}
/// The lab disk the tests serve.
struct Lab {
host: String,
user: String,
vm: String,
disk: String,
thumbprint: String,
}
impl Lab {
/// The lab the environment describes, or `None` after naming every missing variable.
fn from_env() -> Option<Self> {
let host = lab("VECTURA_LAB_HOST");
let user = lab("VECTURA_LAB_USER");
let vm = lab("VECTURA_LAB_VM");
let disk = lab("VECTURA_LAB_DISK");
let thumbprint = lab("VECTURA_LAB_THUMBPRINT");
let password = present("VECTURA_PASSWORD");
let lab = Self {
host: host?,
user: user?,
vm: vm?,
disk: disk?,
thumbprint: thumbprint?,
};
password.then_some(lab)
}
/// The `serve` arguments for the lab disk, followed by `extra`.
fn arguments(&self, extra: &[&str]) -> Vec<String> {
[
"serve",
"--host",
&self.host,
"--user",
&self.user,
"--vm-id",
&self.vm,
"--disk",
&self.disk,
"--thumbprint",
&self.thumbprint,
]
.into_iter()
.chain(extra.iter().copied())
.map(str::to_owned)
.collect()
}
/// Spawns `serve` on the lab disk with `extra` flags, its three streams piped.
fn serve(&self, extra: &[&str]) -> Child {
Command::new(env!("CARGO_BIN_EXE_vectura"))
.args(self.arguments(extra))
.stdin(Stdio::piped())
.stdout(Stdio::piped())
.stderr(Stdio::piped())
.spawn()
.expect("the vectura binary runs")
}
}
/// The raw image `VECTURA_LAB_REFERENCE` names and its length, or `None` after a skip line.
///
/// The image is produced once, with the lab VM powered off so the disk does not
/// change under the tests: the VMDK's descriptor and every extent it names, and
/// for a snapshot its parents' too, are downloaded through the datastore browser
/// (`https://<host>/folder/<path>?dcPath=ha-datacenter&dsName=<datastore>`) into
/// one directory, then `qemu-img convert -O raw <descriptor>.vmdk <image>`
/// flattens them, following the parent hints of the descriptors.
fn reference() -> Option<(File, u64)> {
let path = lab("VECTURA_LAB_REFERENCE")?;
let file = File::open(&path).unwrap_or_else(|error| panic!("{path}: {error}"));
let length = file.metadata().expect("the reference has metadata").len();
Some((file, length))
}
/// Asserts `data` is what the reference holds at `offset`, naming the first difference.
fn matches_reference(reference: &mut File, offset: u64, data: &[u8]) {
reference
.seek(SeekFrom::Start(offset))
.expect("the reference seeks");
let mut expected = vec![0; data.len()];
reference
.read_exact(&mut expected)
.expect("the reference holds the range");
if let Some(index) = data
.iter()
.zip(&expected)
.position(|(served, held)| served != held)
{
let at = offset + u64::try_from(index).expect("an index fits");
panic!(
"byte {at} differs: served {:#04x}, reference {:#04x}",
data[index], expected[index]
);
}
}
/// Three `READ`-long ranges of a `size`-byte disk: the start, a sector-aligned middle, the end.
fn ranges(size: u64) -> [u64; 3] {
assert!(
size >= 2 * READ,
"a disk of {size} bytes is too small to sample"
);
[0, size / 2 / SECTOR * SECTOR, size - READ]
}
/// Reads `length` bytes at `offset`, which must succeed.
fn read(child: &mut Child, cookie: u64, offset: u64, length: u64) -> Vec<u8> {
read_at(child, cookie, offset, length)
.unwrap_or_else(|code| panic!("read of {length} bytes at {offset}: NBD error {code}"))
}
/// Disconnects, waits for the child and asserts it exited 0; returns its standard error.
fn ends_cleanly(mut child: Child) -> String {
disconnect(&mut child);
let output = finish(child);
let stderr = text(&output.stderr);
assert_eq!(output.status.code(), Some(0), "stderr: {stderr}");
stderr
}
/// The `wire bytes` field of the verbose summary line in `stderr`.
fn wire_bytes(stderr: &str) -> &str {
stderr
.lines()
.find(|line| line.starts_with("summary: "))
.and_then(|summary| {
summary
.split(", ")
.find(|field| field.starts_with("wire bytes "))
})
.unwrap_or_else(|| panic!("no wire bytes in the summary:\n{stderr}"))
}
/// Whether `nbdsh` is on the path, with a skip line when it is not.
fn nbdsh_present() -> bool {
let present = Command::new("nbdsh").arg("--version").output().is_ok();
if !present {
report("skipping: nbdsh (libnbd) is not on the path");
}
present
}
/// A Python string literal holding `text`.
fn python_string(text: &str) -> String {
format!("'{}'", text.replace('\\', "\\\\").replace('\'', "\\'"))
}
/// Serves a lab disk over the real protocol and checks the export.
#[test]
#[ignore = "needs a lab host"]
fn a_lab_host_exports_its_disk_read_only_and_the_session_ends_cleanly() {
let Some(lab) = Lab::from_env() else { return };
let mut child = lab.serve(&[]);
let export = negotiate(&mut child, false);
let stderr = ends_cleanly(child);
assert!(stderr.is_empty(), "stderr: {stderr}");
assert!(export.size > 0, "{export:?}");
assert_eq!(export.flags, READ_ONLY, "{export:?}");
}
/// Compares the whole served disk, byte for byte, to the `qemu-img` raw image.
///
/// The lab VM is powered off and the image is the one [`reference`] describes.
/// One sector is read first, then the disk in `READ`-long steps.
#[test]
#[ignore = "needs a lab host and the raw image in VECTURA_LAB_REFERENCE"]
fn the_served_disk_matches_the_qemu_img_raw_image_byte_for_byte() {
let Some(lab) = Lab::from_env() else { return };
let Some((mut reference, length)) = reference() else {
return;
};
let mut child = lab.serve(&[]);
let export = negotiate(&mut child, false);
assert_eq!(export.size, length, "export size against the reference");
let sector = read(&mut child, 0, 0, SECTOR);
matches_reference(&mut reference, 0, &sector);
let mut offset = 0;
while offset < length {
let step = READ.min(length - offset);
let data = read(&mut child, offset, offset, step);
matches_reference(&mut reference, offset, &data);
offset += step;
}
let stderr = ends_cleanly(child);
assert!(stderr.is_empty(), "stderr: {stderr}");
}
/// Drives the binary with libnbd, an NBD client the crate shares no code with.
///
/// `nbdsh` spawns `serve --verbose` through `connect_command`, so the transcript
/// on its standard error shows the `NBD_OPT_GO` the client sent; it reads one
/// range at the start, in the middle and at the end of the disk, and each one
/// must match the `qemu-img` raw image.
#[test]
#[ignore = "needs a lab host, the raw image in VECTURA_LAB_REFERENCE and nbdsh"]
fn libnbd_negotiates_with_go_and_reads_the_start_the_middle_and_the_end() {
let Some(lab) = Lab::from_env() else { return };
let Some((mut reference, length)) = reference() else {
return;
};
if !nbdsh_present() {
return;
}
let offsets = ranges(length);
let argv = std::iter::once(env!("CARGO_BIN_EXE_vectura").to_owned())
.chain(lab.arguments(&["--verbose"]))
.map(|argument| python_string(&argument))
.collect::<Vec<_>>()
.join(", ");
let program = format!(
"import sys
h.connect_command([{argv}])
out = sys.stdout.buffer
out.write(h.get_size().to_bytes({SIZE_BYTES}, 'big'))
for offset in ({}):
out.write(h.pread({READ}, offset))
h.shutdown()
",
offsets.map(|offset| offset.to_string()).join(", ")
);
let output = Command::new("nbdsh")
.args(["-c", &program])
.output()
.expect("nbdsh runs");
let stderr = text(&output.stderr);
assert_eq!(output.status.code(), Some(0), "nbdsh: {stderr}");
assert!(
stderr.contains(GO),
"no NBD_OPT_GO in the transcript:\n{stderr}"
);
let (size, data) = output.stdout.split_at(SIZE_BYTES);
assert_eq!(
u64::from_be_bytes(size.try_into().expect("eight bytes")),
length
);
let range = usize::try_from(READ).expect("a range fits");
assert_eq!(data.len(), 3 * range, "three ranges came back");
for (offset, bytes) in offsets.into_iter().zip(data.chunks(range)) {
matches_reference(&mut reference, offset, bytes);
}
}
/// Samples three ranges in `skipz`, `zlib` and `none` and reports each mode's wire bytes.
///
/// The modes must serve identical bytes; the verbose summary of each run
/// says how many bytes crossed the wire for them, which is the line reported.
#[test]
#[ignore = "needs a lab host"]
fn the_three_compression_modes_serve_identical_bytes_and_report_their_wire_bytes() {
let Some(lab) = Lab::from_env() else { return };
let mut previous: Option<Vec<u8>> = None;
for mode in ["skipz", "zlib", "none"] {
let mut child = lab.serve(&["--verbose", "--compression", mode]);
let export = negotiate(&mut child, false);
let mut data = Vec::new();
for (cookie, offset) in (1..).zip(ranges(export.size)) {
data.extend(read(&mut child, cookie, offset, READ));
}
let stderr = ends_cleanly(child);
report(&format!(
"compression {mode}: {} for {} bytes served",
wire_bytes(&stderr),
data.len()
));
if let Some(previous) = &previous {
assert!(
*previous == data,
"{mode} served other bytes than the mode before"
);
}
previous = Some(data);
}
}
/// Holds the disk open across an idle gap and records whether it stays readable.
///
/// The gap, in seconds, is `VECTURA_LAB_IDLE_GAP` and defaults to five minutes.
/// The outcome line on standard error is the measurement: how an open disk
/// fares without traffic, which nothing public states.
#[test]
#[ignore = "needs a lab host; VECTURA_LAB_IDLE_GAP sets the gap in seconds"]
fn the_disk_stays_readable_across_the_idle_gap() {
let Some(lab) = Lab::from_env() else { return };
let gap = std::env::var("VECTURA_LAB_IDLE_GAP").map_or(DEFAULT_IDLE_GAP, |seconds| {
Duration::from_secs(seconds.parse().expect("a number of seconds"))
});
let mut child = lab.serve(&["--verbose"]);
negotiate(&mut child, false);
let first = read(&mut child, 1, 0, READ);
thread::sleep(gap);
let second = panic::catch_unwind(AssertUnwindSafe(|| read_at(&mut child, 2, 0, READ)));
let succeeded = matches!(&second, Ok(Ok(data)) if *data == first);
let outcome = match &second {
Ok(Ok(_)) if succeeded => "succeeded with the same bytes".to_owned(),
Ok(Ok(_)) => "succeeded with other bytes".to_owned(),
Ok(Err(code)) => format!("failed with NBD error {code}"),
Err(_) => "failed: the stream closed".to_owned(),
};
report(&format!(
"idle gap of {} s: second read {outcome}",
gap.as_secs()
));
let stderr = if succeeded {
ends_cleanly(child)
} else {
text(&finish(child).stderr)
};
assert!(succeeded, "second read {outcome}; transcript:\n{stderr}");
}
/// A legacy host is refused at its greeting, the message listing what it announced.
///
/// `fingerprint` reads the port-902 greeting through the parser `serve` uses,
/// so the refusal is the same and needs neither credentials nor a pin;
/// `VECTURA_LAB_LEGACY_HOST` names the host as `host[:port]`.
#[test]
#[ignore = "needs a legacy host in VECTURA_LAB_LEGACY_HOST"]
fn a_legacy_host_is_refused_at_the_greeting_with_its_capability_list() {
let Some(host) = lab("VECTURA_LAB_LEGACY_HOST") else {
return;
};
let output = Command::new(env!("CARGO_BIN_EXE_vectura"))
.args(["fingerprint", "--host", &host])
.output()
.expect("the vectura binary runs");
let stderr = text(&output.stderr);
assert_eq!(output.status.code(), Some(1), "{stderr}");
assert!(
stderr.contains("does not announce \"SHA256 supported\"; announced: ["),
"{stderr}"
);
report(&format!("legacy host refused: {}", stderr.trim_end()));
}
@xen-orchestra/vmware-explorer/vectura/tests/process/main.rs
+6
View File
@@ -0,0 +1,6 @@
//! Runs the built binary against a mock host on loopback.
mod fingerprint;
mod live;
mod serve;
mod support;
@xen-orchestra/vmware-explorer/vectura/tests/process/serve.rs
+1671
View File
File diff suppressed because it is too large Load Diff
@xen-orchestra/vmware-explorer/vectura/tests/process/support/mod.rs
+1302
View File
File diff suppressed because it is too large Load Diff
@xen-orchestra/vmware-explorer/vectura/tests/process/support/nbd.rs
+399
View File
@@ -0,0 +1,399 @@
//! A minimal fixed-newstyle NBD client for the child's pipes.
//!
//! It negotiates with `NBD_OPT_EXPORT_NAME`, `NBD_OPT_GO`, `NBD_OPT_INFO`
//! or `NBD_OPT_ABORT`, sends requests of any kind and reads simple replies.
//! The constants come from the NBD protocol document, so the crate is not
//! its own oracle.
use std::io::{Read, Write};
/// `NBDMAGIC`, the first eight bytes of the server greeting.
const NBDMAGIC: u64 = 0x4E42_444D_4147_4943;
/// `IHAVEOPT`, the next eight bytes and the start of every option.
const IHAVEOPT: u64 = 0x4948_4156_454F_5054;
/// The magic opening every option reply.
const REPLY_MAGIC: u64 = 0x0003_e889_0455_65a9;
/// Handshake flags the server may offer.
const FLAG_FIXED_NEWSTYLE: u16 = 1;
const FLAG_NO_ZEROES: u16 = 2;
/// Client flags answering them.
const FLAG_C_FIXED_NEWSTYLE: u32 = 1;
const FLAG_C_NO_ZEROES: u32 = 2;
/// The options.
pub const OPT_EXPORT_NAME: u32 = 1;
pub const OPT_ABORT: u32 = 2;
pub const OPT_LIST: u32 = 3;
pub const OPT_STARTTLS: u32 = 5;
pub const OPT_INFO: u32 = 6;
pub const OPT_GO: u32 = 7;
pub const OPT_STRUCTURED_REPLY: u32 = 8;
/// The option replies.
pub const REP_ACK: u32 = 1;
pub const REP_INFO: u32 = 3;
pub const REP_ERR_UNSUP: u32 = 0x8000_0001;
/// The info types.
pub const INFO_EXPORT: u16 = 0;
pub const INFO_BLOCK_SIZE: u16 = 3;
/// Zero bytes after the export flags unless `NO_ZEROES` was negotiated.
const ZEROES_LEN: usize = 124;
/// The request magic, the commands, the one command flag, the simple reply magic.
const REQUEST_MAGIC: u32 = 0x2560_9513;
pub const CMD_READ: u16 = 0;
pub const CMD_WRITE: u16 = 1;
const CMD_DISC: u16 = 2;
pub const CMD_FLUSH: u16 = 3;
pub const CMD_TRIM: u16 = 4;
pub const CMD_WRITE_ZEROES: u16 = 6;
/// `NBD_CMD_FLAG_FUA`, meaningless on a read.
pub const CMD_FLAG_FUA: u16 = 1;
const SIMPLE_REPLY_MAGIC: u32 = 0x6744_6698;
/// The errors a simple reply may carry.
pub const EPERM: u32 = 1;
pub const EINVAL: u32 = 22;
/// What the server said about the export.
#[derive(Debug, PartialEq, Eq)]
pub struct Export {
/// The export size in bytes.
pub size: u64,
/// The transmission flags.
pub flags: u16,
/// Whether the 124 zero bytes followed the flags.
pub padded: bool,
}
/// What `NBD_OPT_INFO` or `NBD_OPT_GO` reported.
#[derive(Debug, PartialEq, Eq)]
pub struct Info {
/// The export size in bytes.
pub size: u64,
/// The transmission flags.
pub flags: u16,
/// The minimum, preferred and maximum request sizes, when reported.
pub block_size: Option<(u32, u32, u32)>,
/// The types of the other infos, in order.
pub other: Vec<u16>,
}
/// Reads exactly enough bytes to fill `bytes`.
fn read<R: Read>(input: &mut R, bytes: &mut [u8]) {
input
.read_exact(bytes)
.expect("the server sends the bytes the protocol calls for");
}
fn read_u16<R: Read>(input: &mut R) -> u16 {
let mut bytes = [0_u8; 2];
read(input, &mut bytes);
u16::from_be_bytes(bytes)
}
fn read_u32<R: Read>(input: &mut R) -> u32 {
let mut bytes = [0_u8; 4];
read(input, &mut bytes);
u32::from_be_bytes(bytes)
}
fn read_u64<R: Read>(input: &mut R) -> u64 {
let mut bytes = [0_u8; 8];
read(input, &mut bytes);
u64::from_be_bytes(bytes)
}
/// Writes `bytes` and flushes them to the server.
fn write<W: Write>(output: &mut W, bytes: &[u8]) {
output
.write_all(bytes)
.and_then(|()| output.flush())
.expect("the client's bytes are written");
}
/// Reads the greeting and returns the handshake flags the server offers.
fn greeting<R: Read>(input: &mut R) -> u16 {
assert_eq!(
read_u64(input),
NBDMAGIC,
"the greeting starts with NBDMAGIC"
);
assert_eq!(read_u64(input), IHAVEOPT, "IHAVEOPT follows");
let offered = read_u16(input);
assert_ne!(
offered & FLAG_FIXED_NEWSTYLE,
0,
"the server offers fixed newstyle"
);
offered
}
/// Reads the greeting and answers it, asking `NO_ZEROES` when `no_zeroes`.
///
/// Returns whether the export reply will carry the 124 zero bytes.
pub fn handshake<R: Read, W: Write>(input: &mut R, output: &mut W, no_zeroes: bool) -> bool {
let offered = greeting(input);
let mut flags = FLAG_C_FIXED_NEWSTYLE;
if no_zeroes && offered & FLAG_NO_ZEROES != 0 {
flags |= FLAG_C_NO_ZEROES;
}
write(output, &flags.to_be_bytes());
flags & FLAG_C_NO_ZEROES == 0
}
/// Reads the greeting and answers it with `flags` as given, whatever is offered.
pub fn handshake_with<R: Read, W: Write>(input: &mut R, output: &mut W, flags: u32) {
greeting(input);
write(output, &flags.to_be_bytes());
}
/// Sends the header of `option` announcing `length` bytes of data, and no data.
pub fn option_header<W: Write>(output: &mut W, option: u32, length: u32) {
let mut bytes = IHAVEOPT.to_be_bytes().to_vec();
bytes.extend_from_slice(&option.to_be_bytes());
bytes.extend_from_slice(&length.to_be_bytes());
write(output, &bytes);
}
/// Sends `option` with `data`.
pub fn option<W: Write>(output: &mut W, option: u32, data: &[u8]) {
let length = u32::try_from(data.len()).expect("short option data");
option_header(output, option, length);
write(output, data);
}
/// Reads one reply to `option`: its type and its data.
pub fn option_reply<R: Read>(input: &mut R, option: u32) -> (u32, Vec<u8>) {
assert_eq!(
read_u64(input),
REPLY_MAGIC,
"the reply starts with the reply magic"
);
assert_eq!(read_u32(input), option, "the reply names the option");
let reply = read_u32(input);
let length = read_u32(input);
let mut data = vec![0_u8; length as usize];
read(input, &mut data);
(reply, data)
}
/// Sends `NBD_OPT_EXPORT_NAME` for the unnamed export and reads the export.
pub fn export_name<R: Read, W: Write>(input: &mut R, output: &mut W, padded: bool) -> Export {
option(output, OPT_EXPORT_NAME, b"");
let size = read_u64(input);
let flags = read_u16(input);
if padded {
let mut zeroes = [0_u8; ZEROES_LEN];
read(input, &mut zeroes);
assert_eq!(zeroes, [0_u8; ZEROES_LEN], "the padding is zero");
}
Export {
size,
flags,
padded,
}
}
/// Negotiates the unnamed export with `NBD_OPT_EXPORT_NAME`, asking `NO_ZEROES` when `no_zeroes`.
pub fn negotiate<R: Read, W: Write>(input: &mut R, output: &mut W, no_zeroes: bool) -> Export {
let padded = handshake(input, output, no_zeroes);
export_name(input, output, padded)
}
/// The data of `NBD_OPT_INFO` or `NBD_OPT_GO`: `name`, then the `requests` asked.
pub fn info_data(name: &[u8], requests: &[u16]) -> Vec<u8> {
let mut data = u32::try_from(name.len())
.expect("a short name")
.to_be_bytes()
.to_vec();
data.extend_from_slice(name);
let count = u16::try_from(requests.len()).expect("a few requests");
data.extend_from_slice(&count.to_be_bytes());
for request in requests {
data.extend_from_slice(&request.to_be_bytes());
}
data
}
/// Sends `NBD_OPT_INFO` or `NBD_OPT_GO` (`kind`) naming `name` and asking `requests`.
///
/// Returns what the server reported, or the error reply it sent instead.
pub fn info<R: Read, W: Write>(
input: &mut R,
output: &mut W,
kind: u32,
name: &[u8],
requests: &[u16],
) -> Result<Info, u32> {
option(output, kind, &info_data(name, requests));
let mut info = Info {
size: 0,
flags: 0,
block_size: None,
other: Vec::new(),
};
let mut export_seen = false;
loop {
let (reply, data) = option_reply(input, kind);
match reply {
REP_ACK => {
assert!(export_seen, "NBD_INFO_EXPORT comes before the ack");
assert!(data.is_empty(), "the ack carries no data");
return Ok(info);
}
REP_INFO => match u16::from_be_bytes([data[0], data[1]]) {
INFO_EXPORT => {
assert_eq!(data.len(), 12, "NBD_INFO_EXPORT is twelve bytes");
info.size = u64::from_be_bytes(data[2..10].try_into().expect("eight bytes"));
info.flags = u16::from_be_bytes([data[10], data[11]]);
export_seen = true;
}
INFO_BLOCK_SIZE => {
assert_eq!(data.len(), 14, "NBD_INFO_BLOCK_SIZE is fourteen bytes");
let size = |from: usize| {
u32::from_be_bytes(data[from..from + 4].try_into().expect("four bytes"))
};
info.block_size = Some((size(2), size(6), size(10)));
}
other => info.other.push(other),
},
error => return Err(error),
}
}
}
/// Sends `NBD_OPT_ABORT` and reads its acknowledgement.
pub fn abort<R: Read, W: Write>(input: &mut R, output: &mut W) {
option(output, OPT_ABORT, b"");
let (reply, data) = option_reply(input, OPT_ABORT);
assert_eq!(reply, REP_ACK, "the abort is acknowledged");
assert!(data.is_empty(), "the ack carries no data");
}
/// Sends `kind` with `data` and returns the type of the reply.
pub fn refused<R: Read, W: Write>(input: &mut R, output: &mut W, kind: u32, data: &[u8]) -> u32 {
option(output, kind, data);
option_reply(input, kind).0
}
/// Sends one request of `kind` carrying `flags`.
pub fn request<W: Write>(
output: &mut W,
flags: u16,
kind: u16,
cookie: u64,
offset: u64,
length: u32,
) {
write(output, &request_bytes(flags, kind, cookie, offset, length));
}
/// The bytes of one request of `kind` carrying `flags`.
fn request_bytes(flags: u16, kind: u16, cookie: u64, offset: u64, length: u32) -> Vec<u8> {
let mut request = REQUEST_MAGIC.to_be_bytes().to_vec();
request.extend_from_slice(&flags.to_be_bytes());
request.extend_from_slice(&kind.to_be_bytes());
request.extend_from_slice(&cookie.to_be_bytes());
request.extend_from_slice(&offset.to_be_bytes());
request.extend_from_slice(&length.to_be_bytes());
request
}
/// Sends one `NBD_CMD_READ` per (cookie, offset, length), all in one write.
pub fn requests<W: Write>(output: &mut W, reads: &[(u64, u64, u32)]) {
let bytes = reads
.iter()
.flat_map(|&(cookie, offset, length)| request_bytes(0, CMD_READ, cookie, offset, length))
.collect::<Vec<_>>();
write(output, &bytes);
}
/// Reads one reply per (cookie, length) expected, in the order they arrive.
///
/// Each is the cookie with the bytes read, or the error the reply carries.
pub fn replies<R: Read>(
input: &mut R,
expected: &[(u64, u32)],
) -> Vec<(u64, Result<Vec<u8>, u32>)> {
expected
.iter()
.map(|_| {
let (cookie, error) = reply_header(input);
let Some((_, length)) = expected.iter().find(|(it, _)| *it == cookie) else {
panic!("a reply for cookie {cookie:#x}, which no request carried");
};
if error != 0 {
return (cookie, Err(error));
}
let mut data = vec![0_u8; *length as usize];
read(input, &mut data);
(cookie, Ok(data))
})
.collect()
}
/// Reads a simple reply header: the cookie it echoes and its error.
fn reply_header<R: Read>(input: &mut R) -> (u64, u32) {
assert_eq!(
read_u32(input),
SIMPLE_REPLY_MAGIC,
"the reply starts with the simple reply magic"
);
let error = read_u32(input);
(read_u64(input), error)
}
/// Reads a simple reply, asserts it echoes `cookie`, and returns its error.
pub fn reply<R: Read>(input: &mut R, cookie: u64) -> u32 {
let (echoed, error) = reply_header(input);
assert_eq!(echoed, cookie, "the reply echoes the cookie");
error
}
/// Sends a request of `kind` carrying `flags` for a range and returns the reply's error.
pub fn command<R: Read, W: Write>(
input: &mut R,
output: &mut W,
flags: u16,
kind: u16,
cookie: u64,
offset: u64,
length: u32,
) -> u32 {
request(output, flags, kind, cookie, offset, length);
reply(input, cookie)
}
/// Sends `NBD_CMD_WRITE` with `payload` and returns the reply's error.
pub fn write_at<R: Read, W: Write>(
input: &mut R,
output: &mut W,
cookie: u64,
offset: u64,
payload: &[u8],
) -> u32 {
let length = u32::try_from(payload.len()).expect("a length a request can carry");
request(output, 0, CMD_WRITE, cookie, offset, length);
write(output, payload);
reply(input, cookie)
}
/// Sends `NBD_CMD_READ` and returns the bytes, or the error the reply carries.
pub fn read_at<R: Read, W: Write>(
input: &mut R,
output: &mut W,
cookie: u64,
offset: u64,
length: u32,
) -> Result<Vec<u8>, u32> {
request(output, 0, CMD_READ, cookie, offset, length);
let error = reply(input, cookie);
if error != 0 {
return Err(error);
}
let mut data = vec![0_u8; length as usize];
read(input, &mut data);
Ok(data)
}
/// Sends `NBD_CMD_DISC`.
pub fn disconnect<W: Write>(output: &mut W) {
request(output, 0, CMD_DISC, 0x1234_5678_9ABC_DEF0, 0, 0);
}
@xen-orchestra/vmware-explorer/vectura/vectura
Executable
BIN
View File
Binary file not shown.
CHANGELOG.unreleased.md
+2
View File
@@ -27,6 +27,7 @@
- [i18n] Add Turkish and update Czech, Dutch, Slovak and Swedish translations (PR [#10316](https://github.com/vatesfr/xen-orchestra/pull/10316))
- [XO6/Host] Add icon for disabled host (PR [#10188](https://github.com/vatesfr/xen-orchestra/pull/10188))
- [V2V] When a VM has Changed Block Tracking enabled on the source host, a migration now asks the host which blocks it has to read — the blocks a disk uses for a full transfer, the blocks written since the previous pass for a delta — instead of scanning the disk through the VDDK. Faster to start on large disks, and one less moving part. VMs without CBT are migrated exactly as before (PR [#10384](https://github.com/vatesfr/xen-orchestra/pull/10384))
- [V2V] Migrations no longer need the VDDK library, `nbdkit` and `nbdinfo`. The disks are read through `vectura`, a single binary shipped with XO and installed in one click from the import screen. The Broadcom download, the tar.gz upload and the two compile-from-source buttons are gone. Note that the source must now be an ESXi 7.x or 8.x host reached directly, a vCenter address is no longer supported
- [Proxy] Check proxy licenses at XOA level instead of blocking backups on it (PR [#10280](https://github.com/vatesfr/xen-orchestra/pull/10280))
- [RPU] A rolling pool update is now refused while a previous one is still in progress or was left incomplete, and asks for confirmation when the master is already up to date but other hosts are not (PR [#10394](https://github.com/vatesfr/xen-orchestra/pull/10394))
- [RPU] An incomplete rolling pool update can now be closed from the pool's Patches tab, or with `pool.finalizeRollingUpdate` and the REST route `POST /rest/v0/pools/{id}/actions/finalize_rolling_update`. The closing is refused while the update left something it had changed unrestored (HA, WLB, load balancer, backup schedules, disabled hosts, displaced or halted VMs); forcing it lists the abandoned items in the task and changes nothing in the pool (PR [#10418](https://github.com/vatesfr/xen-orchestra/pull/10418))
@@ -124,6 +125,7 @@
- @xen-orchestra/web minor
- @xen-orchestra/web-core minor
- @xen-orchestra/xapi patch
- vectura major
- xen-api major
- xo-remote-parser major
- xo-server minor
docs/docs/guides/v2v-migration-guide.md
+14 -70
View File
@@ -6,12 +6,12 @@ sidebar_label: VMware migration (V2V)
## Introduction {#-introduction}
Xen Orchestra 5.110 introduced a new migration backend for moving virtual machines from VMware to XCP-ng (V2V). It replaces the old NFS-based method with `nbdkit` and VMware's VDDK libraries, delivering better performance and reliability.
Xen Orchestra 5.110 introduced a new migration backend for moving virtual machines from VMware to XCP-ng (V2V). It replaces the old NFS-based method with `vectura`, a standalone reader of VMware's NFC protocol, delivering better performance and reliability without any VMware-provided library.
The whole process is driven from Xen Orchestra, in **Import → VM → From VMware**. XO connects to your vCenter or ESXi host, reads the VM disks directly, and imports them into an XCP-ng storage repository:
<Schema label="XO reads the VM disks from ESXi through nbdkit and the VDDK libraries, and imports them into an XCP-ng pool. The source VM keeps running during most of the transfer" legend={[["#5ac8c8", "VMware"], ["#8e83fe", "XCP-ng"], ["#6aabf0", "XO"]]} maxWidth="640px">
<svg viewBox="0 0 640 240" role="img" aria-label="A VM running on an ESXi host inside a VMware infrastructure is exported by Xen Orchestra using nbdkit and VDDK, then imported into an XCP-ng host in the destination pool, where the migrated VM is ready to boot; the source VM keeps running during the warm phase of the transfer">
<Schema label="XO reads the VM disks from ESXi through vectura, and imports them into an XCP-ng pool. The source VM keeps running during most of the transfer" legend={[["#5ac8c8", "VMware"], ["#8e83fe", "XCP-ng"], ["#6aabf0", "XO"]]} maxWidth="640px">
<svg viewBox="0 0 640 240" role="img" aria-label="A VM running on an ESXi host inside a VMware infrastructure is exported by Xen Orchestra using vectura, then imported into an XCP-ng host in the destination pool, where the migrated VM is ready to boot; the source VM keeps running during the warm phase of the transfer">
<rect x="16" y="40" width="192" height="168" rx="8" fill="none" stroke="rgba(255,255,255,0.22)" strokeDasharray="6 5" />
<text x="28" y="62" fill="#c6d2e1" fontSize="12">VMware</text>
<text x="28" y="78" fill="#7a8699" fontSize="10">source infrastructure</text>
@@ -25,7 +25,7 @@ The whole process is driven from Xen Orchestra, in **Import → VM → From VMwa
<text x="228" y="122" fill="#5ac8c8" fontSize="9" textAnchor="middle">export</text>
<rect x="266" y="108" width="108" height="52" rx="6" fill="rgba(255,255,255,0.04)" stroke="#6aabf0" />
<text x="320" y="129" fill="#c6d2e1" fontSize="11" textAnchor="middle">Xen Orchestra</text>
<text x="320" y="146" fill="#7a8699" fontSize="9" textAnchor="middle">nbdkit + VDDK</text>
<text x="320" y="146" fill="#7a8699" fontSize="9" textAnchor="middle">vectura</text>
<text x="320" y="182" fill="#7a8699" fontSize="9" textAnchor="middle">reads only allocated blocks</text>
<line x1="378" y1="134" x2="430" y2="134" stroke="#56c288" strokeWidth="2" className="schema-flow" strokeDasharray="5 4" />
<polygon points="440,134 430,128 430,140" fill="#56c288" />
@@ -90,76 +90,20 @@ Performance gains can be significant, though results depend on the environment:
### Update Xen Orchestra
Make sure your Xen Orchestra instance is up to date. The import page includes an automatic check that verifies the installed versions of `nbdkit`, the VDDK plugin and `nbdinfo`, and highlights errors and warnings before the migration. Errors must be resolved, and warnings should be addressed when possible.
Make sure your Xen Orchestra instance is up to date. The import page includes an automatic check that verifies that `vectura`, shipped with your Xen Orchestra, runs on this system. Errors must be resolved before the migration can start.
<UiDetail src="/img/xo5/v2v-prerequisite-check.png" alt="The prerequisite check on the import page, listing nbdkit, the VDDK plugin and nbdinfo with their status" width={480} />
<UiDetail src="/img/xo5/v2v-prerequisite-check.png" alt="The prerequisite check on the import page, showing the status of vectura" width={480} />
### Dependencies
Install the following dependencies:
The only dependency is `vectura`, a single binary that reads VMware disks over the NFC protocol. It
is shipped with Xen Orchestra and runs from there: there is nothing to install, nothing to download
from VMware or Broadcom, and nothing to compile. It needs a Linux x86-64 system with glibc 2.34 or
later, such as Debian 12 or Ubuntu 22.04.
- `nbdkit`
- The VDDK plugin
- `nbdinfo`
Go to the **Import → VM → From VMware** section. Once the check passes, a **transfer form** appears:
#### Installing `nbdinfo` on Debian
:::warning
This procedure requires an active Internet connection and has only been tested on Debian 12 and 13.
In the future, we plan to include the latest libraries directly in XOA.
:::
To install `nbdinfo` on Debian:
1. Go to the **Import → VM → From VMware** section.
2. Click the button called **install nbdinfo (debian based system)**:
<UiDetail src="/img/xo5/v2v-install-nbdinfo-button.png" alt="The install nbdinfo (debian based system) button on the From VMware import page" width={620} />
This starts the install.
3. Keep an eye on the current page to make sure the install goes as planned.
- The **install nbdinfo** button updates to show the installation progress, and the page displays any warnings in real time as they appear:
<UiDetail src="/img/xo5/v2v-nbdinfo-progress-import.png" alt="The import page while the nbdinfo installation task is running, with warnings shown in real time" width={620} />
- Additionally, the **Tasks** screen shows the `nbdinfo` install task, with its name, start date, start time and status indicator:
<UiDetail src="/img/xo5/v2v-nbdinfo-progress-tasks.png" alt="The Tasks screen showing the running nbdinfo install task with its status indicator" width={620} />
The task is removed from the list upon completion.
4. Once the installation finishes, the `nbdinfo` install screen updates automatically. It lets you know if everything was completed successfully and flags any warnings that came up during the process:
<UiDetail src="/img/xo5/v2v-nbdinfo-install-complete.png" alt="The import page after a completed nbdinfo installation, with the final status and warnings" width={620} />
:::tip
`nbdkit` and the `nbdkit-plugin-vddk` are installed using the same approach.
:::
#### Installing the other dependencies
:::warning
You can compile `nbdkit` and `libnbd` from source on GitLab, but the correct dependencies must be in place for `nbdinfo` and VDDK.
:::
##### Installing VDDK
1. Download the VMware Virtual Disk Development Kit (VDDK) from the [Broadcom developer portal](https://developer.broadcom.com/sdks/vmware-virtual-disk-development-kit-vddk/9.0?ref=xen-orchestra.com).
2. Select the _tar.gz_ archive, then drag and drop it directly into the Xen Orchestra interface:
<UiDetail src="/img/xo5/v2v-upload-vddk.png" alt="The import page with the drop zone used to upload the VDDK tar.gz archive" width={620} />
An **install button** will appear.
3. Click the button:
<UiDetail src="/img/xo5/v2v-install-vddk-button.png" alt="The button confirming the installation of the uploaded VDDK archive" width={620} />
The installation should take just a few seconds. Once it is done, a **transfer form** will appear:
<UiDetail src="/img/xo5/v2v-transfer-form.png" alt="The transfer form used to connect to VMware and import a VM into Xen Orchestra" width={620} />
<UiDetail src="/img/xo5/v2v-transfer-form.png" alt="The transfer form used to connect to VMware and import a VM into Xen Orchestra" width={620} />
:::warning
Filling in this form and clicking the **Connect** button will start the VM import. Make sure your VMware environment is ready first!
@@ -175,7 +119,7 @@ Before starting the migration, make sure your VMware environment meets the follo
#### Network
XO must be able to connect to the vSphere or ESXi host through the port running the web UI (default port: 443) and the VDDK data port (default port: 902). At the time of writing, there is no way to select one network or another, so if possible, keep a single network path from XO to VMware.
XO must be able to connect to the ESXi host, or to the vCenter, through the port running the web UI (default port: 443), and to the ESXi host holding the VM through the NFC data port (default port: 902), even when going through a vCenter. At the time of writing, there is no way to select one network or another, so if possible, keep a single network path from XO to VMware.
#### VMware disk support
@@ -310,7 +254,7 @@ If you encounter problems during migration, try these steps to diagnose and reso
Look for disk access errors in the VMware logs. These may point to issues with the source storage or VM configuration.
- **Update software versions**\
Ensure `nbdkit` and `nbdinfo` are up to date. Older versions can cause compatibility issues with the VDDK plugin. The [prerequisite check](#update-xen-orchestra) on the import page flags outdated versions.
Keep your Xen Orchestra up to date: `vectura` ships with it. The [prerequisite check](#update-xen-orchestra) on the import page tells whether it runs on this system.
- **Remove active snapshots**\
Make sure no active snapshots remain on the source VM. Snapshots can disrupt migration consistency.
package.json
+1
View File
@@ -58,6 +58,7 @@
"eslint"
],
".USAGE.md": "scripts/usage-to-readme.js --git-add",
"@xen-orchestra/vmware-explorer/vectura/**/*.rs": "sh -c 'cd @xen-orchestra/vmware-explorer/vectura && cargo fmt --all --check' --",
"./CHANGELOG.unreleased.md": "sh -c 'scripts/gen-deps-list.js --check-order' --"
},
"private": true,
packages/xo-server/src/api/esxi.mjs
+1 -132
View File
@@ -1,26 +1,5 @@
import { tmpdir } from 'node:os'
import { join } from 'path'
import fs from 'node:fs/promises'
import { createWriteStream } from 'node:fs'
import { VDDK_LIB_DIR } from '@xen-orchestra/vmware-explorer/esxi.mjs'
import { exec } from 'node:child_process'
import { Task } from '@xen-orchestra/mixins/Tasks.mjs'
import { format } from 'json-rpc-peer'
import { noop, safeDateFormat } from '../utils.mjs'
function execPromise(command, opts = []) {
return new Promise((resolve, reject) => {
exec(command, { maxBuffer: 10 * 1024 * 1024, ...opts }, (error, stdout) => {
if (error) {
reject(error)
} else {
resolve(stdout)
}
})
})
}
export function listVms({ host, password, sslVerify = true, user }) {
return this.connectToEsxiAndList({ host, user, password, sslVerify })
}
@@ -35,123 +14,13 @@ listVms.params = {
listVms.permission = 'admin'
export function checkInstall() {
return this.checkVddkDependencies()
return this.checkEsxiDependencies()
}
checkInstall.params = {}
checkInstall.permission = 'admin'
async function handleImport(req, res) {
return Task.run({ properties: { name: 'handling vddk import and installing' } }, async () => {
const tmpDir = await Task.run({ properties: { name: `create directory` } }, () =>
fs.mkdtemp(join(tmpdir(), 'xo-server'))
)
Task.info(`temp directory is ${tmpDir}`)
const tempFilePath = join(tmpDir, 'vix.tar.gz')
Task.info(`temp upload path is ${tempFilePath}`)
await Task.run({ properties: { name: `create vddk lib directory` } }, () =>
fs.mkdir(VDDK_LIB_DIR, { recursive: true })
)
await Task.run(
{ properties: { name: 'uploading file' } },
() =>
new Promise((resolve, reject) => {
req.pipe(createWriteStream(tempFilePath))
req.on('end', resolve)
req.on('error', reject)
})
)
await Task.run({ properties: { name: `installing to ${VDDK_LIB_DIR}` } }, () =>
execPromise(`tar xfz ${tempFilePath} -C ${VDDK_LIB_DIR}`)
)
res.end(format.response(0))
})
}
export async function installVddkLib() {
return {
$sendTo: await this.registerHttpRequest(handleImport),
}
}
installVddkLib.params = {}
installVddkLib.permission = 'admin'
async function install({ repository, version, executable }) {
const tmpDir = await Task.run({ properties: { name: `create directory` } }, () => {
return fs.mkdtemp(join(tmpdir(), 'xo-server'))
})
Task.info(`temp directory is ${tmpDir}`)
await Task.run({ properties: { name: `check if nbdinfo is already connected` } }, async () => {
const libPath = await execPromise(`which ${executable}`).catch(() => {})
if (libPath) {
throw new Error(
`${executable} is already installed at ${libPath} on this system please uninstall before rerunning this script`
)
}
})
await Task.run({ properties: { name: `check if user is root` } }, async () => {
let id
if ((id = (await execPromise('id -u')).trim() !== '0')) {
throw new Error(`install script can only be run as root ${id}`)
}
})
await Task.run({ properties: { name: `check if os is debian based` } }, async () => {
try {
await execPromise('apt-get --version')
} catch (err) {
const error = new Error(`The ${executable} auto install can only be run on a debian based system`)
error.cause = err
throw err
}
})
await Task.run({ properties: { name: `installing dependencies` } }, () =>
execPromise('apt-get install -y git dh-autoreconf pkg-config make libxml2-dev ocaml libc-bin')
)
await Task.run({ properties: { name: `cloning repo` } }, () => execPromise(`git clone ${repository} ${tmpDir}`))
await Task.run({ properties: { name: `switching to version ${version}` } }, () =>
execPromise(`git checkout ${version} `, { cwd: tmpDir })
)
await Task.run({ properties: { name: `running autoreconf` } }, () =>
execPromise('autoreconf -i > autoreconf.log', { cwd: tmpDir })
)
await Task.run({ properties: { name: `running configure` } }, () =>
execPromise('./configure > configure.log', { cwd: tmpDir })
)
await Task.run({ properties: { name: `compiling` } }, () => execPromise('make', { cwd: tmpDir }))
await Task.run({ properties: { name: `installing` } }, () => execPromise('make install', { cwd: tmpDir }))
await Task.run({ properties: { name: `linking` } }, () => execPromise('ldconfig', { cwd: tmpDir }))
await Task.run({ properties: { name: `version check ` } }, () => execPromise(`${executable} --version`))
}
export async function installNbdInfoFromSource() {
await Task.run({ properties: { name: 'Compile and installl nbdinfo' } }, async () => {
await install({ executable: 'nbdinfo', repository: 'https://gitlab.com/nbdkit/libnbd.git', version: 'v1.23.4' })
})
}
installNbdInfoFromSource.params = {}
installNbdInfoFromSource.permission = 'admin'
export async function installNbdKitFromSource() {
await Task.run({ properties: { name: 'Compile and installl nbdkit' } }, async () => {
await install({ executable: 'nbdkit', repository: 'https://gitlab.com/nbdkit/nbdkit.git', version: 'v1.44.3' })
})
}
installNbdKitFromSource.params = {}
installNbdKitFromSource.permission = 'admin'
async function handleExport(req, res, { disk, format, host, password, user, vm }) {
const stream = await this.exportEsxiDisk({ disk, format, host, user, password, vm })
packages/xo-server/src/xo-mixins/vmware/importDisksfromDatastore.mjs
+40 -34
View File
@@ -1,9 +1,9 @@
import { Disposable } from 'promise-toolbox'
import { Task } from '@xen-orchestra/mixins/Tasks.mjs'
import { QCOW2_CLUSTER_SIZE, VDI_FORMAT_QCOW2, VDI_FORMAT_VHD, VHD_BLOCK_SIZE, VHD_MAX_SIZE } from '@xen-orchestra/xapi'
import { ReadAhead } from '@xen-orchestra/disk-transform'
import { toVhdStream } from 'vhd-lib/disk-consumer/index.mjs'
import { NbdDisk } from '@vates/nbd-client/NbdDisk.mjs'
import { NbdStdioClient } from '@vates/nbd-client'
import { createLogger } from '@xen-orchestra/log'
import { toQcow2Stream } from '@xen-orchestra/qcow2'
import { TaskProgressHandler } from '@xen-orchestra/backups/_runners/_vmRunners/_TaskProgressHandler.mjs'
@@ -16,46 +16,53 @@ import {
const { warn } = createLogger('xo:importdiskfromdatastore')
export async function importStream({ esxi, dataMap, disk, vmId, format }, consumerCallback) {
export async function importStream(
{
esxi,
disk,
// vectura serves the disk and nothing else: there is no block map to fall back on, so the
// whole disk is read. `capacity` and not the aligned size, reading past the export would fail
dataMap = [{ offset: 0, length: disk.capacity, type: 0 }],
vmId,
format,
},
consumerCallback
) {
const signal = Task.abortSignal
const { datastore: datastoreName, diskPath } = disk
// `NbdStdioClient` spawns vectura on connect and kills it on disconnect
// we read the data from the full chain to ensure we don't have partial blocks ( blocks with 0 when clusters are in parent only)
const vecturaSettings = await esxi.getVecturaSpawnSettings(vmId, `[${datastoreName}] ${diskPath}`)
let vmdk
let stream
try {
// the server is stopped whatever happens here: it used to be killed by a `finally` which had
// to name the disk and repeat the exact spawn options
// we read the data from the full chain to ensure we don't have partial blocks ( blocks with 0 when clusters are in parent only)
return await Disposable.use(esxi.getNbdServer(vmId, `[${datastoreName}] ${diskPath}`), async ({ nbdInfos }) => {
let vmdk
let stream
try {
signal?.throwIfAborted()
vmdk = new NbdDisk(nbdInfos, READ_BLOCK_SIZE, { dataMap })
signal?.throwIfAborted()
vmdk = new NbdDisk(vecturaSettings, READ_BLOCK_SIZE, { dataMap, ClientClass: NbdStdioClient })
await vmdk.init()
signal?.throwIfAborted()
vmdk = new ReadAhead(vmdk)
await vmdk.init()
signal?.throwIfAborted()
vmdk = new ReadAhead(vmdk)
vmdk.addProgressHandler(new TaskProgressHandler())
vmdk.addProgressHandler(new TaskProgressHandler())
if (format === VDI_FORMAT_QCOW2) {
stream = await toQcow2Stream(vmdk, { signal })
} else {
stream = await toVhdStream(vmdk, { signal })
}
Task.info(`got source stream for ${diskPath}`)
await consumerCallback(stream)
return Math.round((vmdk.getNbGeneratedBlock() * vmdk.getBlockSize()) / 1024 / 1024)
} catch (err) {
stream?.destroy(err)
throw err
} finally {
await vmdk?.close().catch(err => warn('error while closing source vmdk', err))
}
})
if (format === VDI_FORMAT_QCOW2) {
stream = await toQcow2Stream(vmdk, { signal })
} else {
stream = await toVhdStream(vmdk, { signal })
}
Task.info(`got source stream for ${diskPath}`)
await consumerCallback(stream)
return Math.round((vmdk.getNbGeneratedBlock() * vmdk.getBlockSize()) / 1024 / 1024)
} catch (err) {
// a failure to start the server belongs in the report too, e.g. nbdkit not being installed
stream?.destroy(err)
// a failure to start the server belongs in the report too, e.g. vectura not being installed
Task.warning(err)
throw err
} finally {
// this is what stops the vectura process
await vmdk?.close().catch(err => warn('error while closing source vmdk', err))
}
}
@@ -98,7 +105,7 @@ async function importDiskChain({ esxi, sr, vm, chainByNode, changeTracking, user
if (previouslyImportedIndex !== -1) {
if (previouslyImportedIndex < chainByNode.length - 2) {
throw new Error(
'vddk import does not support importing multiple snapshots. Coalesce the non imported snapshot into one or force a full import'
'the import does not support importing multiple snapshots. Coalesce the non imported snapshot into one or force a full import'
)
}
@@ -114,8 +121,7 @@ async function importDiskChain({ esxi, sr, vm, chainByNode, changeTracking, user
}
// asked in both cases: the blocks written since `baseDiskPath`, or every block the disk uses.
// `undefined` when the host cannot answer for a whole disk, and the disk is then read to find
// out, as it always was
// `undefined` when the host cannot answer for a whole disk, `importStream` then reads all of it
const dataMap = await esxi.getDataMap(vmId, datastoreName, diskPath, {
baseDiskPath,
changeTracking,
packages/xo-server/src/xo-mixins/vmware/index.mjs
+3 -3
View File
@@ -5,7 +5,7 @@ import { Task } from '@xen-orchestra/mixins/Tasks.mjs'
import asyncMapSettled from '@xen-orchestra/async-map/legacy.js'
import { createLogger } from '@xen-orchestra/log'
import Esxi from '@xen-orchestra/vmware-explorer/esxi.mjs'
import { checkVddkDependencies } from '@xen-orchestra/vmware-explorer/checks.mjs'
import { checkDependencies } from '@xen-orchestra/vmware-explorer/checks.mjs'
import { VDI_FORMAT_VHD } from '@xen-orchestra/xapi'
import OTHER_CONFIG_TEMPLATE from '../../xapi/other-config-template.mjs'
import { importDisksFromDatastore, importStream } from './importDisksfromDatastore.mjs'
@@ -264,8 +264,8 @@ export default class MigrateVm {
return vm.uuid
}
async checkVddkDependencies() {
return checkVddkDependencies()
async checkEsxiDependencies() {
return checkDependencies()
}
async exportEsxiDisk({ disk: diskId, format, host, user, password, vm: vmId }) {
packages/xo-web/src/common/intl/messages.js
-14
View File
@@ -21,25 +21,11 @@ const messages = {
hostIp: 'Host IP',
interfaces: 'Interfaces',
keyValue: '{key}: {value}',
esxiLibraryInfo: 'The V2V tool needs the vddk library, provided by Broadcom. Please download it from their website',
esxiLibraryLink: 'Download link',
esxiVddkLibrary: 'Drop the tar.gz file of the vddk library (linux)',
esxiVddkLibraryImport: 'Import and install the Vddk library. VDDK9 need nbdkit 1.42+',
esxiLibraryManualInstall:
'For other systems, you can install manually from https://gitlab.com/nbdkit/ . For reference the list of packages need for a debian 13 is **git dh-autoreconf pkg-config make libxml2-dev ocaml libc-bin**',
esxiLibraryAutoInstall: 'install {library} (debian based system)',
esxiLibraryInstalling:
"Installing {library} can take a few minutes. You can check the progress in the XO tasks screen while it's running.",
esxiProgressLinkText: 'Track progress',
esxiLibraryOutdated:
'{library} library is outdated expecting {expectedVersion}, got {version}. Please uninstall it and install the required version.',
esxiCheckingPrerequisite: 'Checking prerequisite on XO',
esxiCheckedPrerequisite: 'Result of the prerequisite check on XO',
esxiCheckingPrerequisiteError: 'Must be corrected before importing VM',
esxiCheckedPrerequisiteVersion: 'expected version {expectedVersion} , {version} installed',
esxiImportSslCertificate: 'Skip SSL check',
esxiImportRememberLogin: 'Remember connection in this browser',
esxiImportRememberWarning:
packages/xo-web/src/common/xo/index.js
-32
View File
@@ -4444,38 +4444,6 @@ export const esxiListVms = (host, user, password, sslVerify) =>
export const esxiCheckInstall = () => _call('esxi.checkInstall')
export const importVmsFromEsxi = params => _call('vm.importMultipleFromEsxi', params)
export const importVddkLib = file => {
return _call('esxi.installVddkLib').then(({ $sendTo }) => {
return post($sendTo, file.file)
.then(res => {
if (res.status !== 200) {
throw res.status
}
success('lib successfully installed')
})
.catch(err => {
error('fail to install vddk lib', err)
})
})
}
export const installNbdInfo = file => {
return _call('esxi.installNbdInfoFromSource')
.then(() => {
success('nbdInfo successfullly installed successfully installed')
})
.catch(err => {
error('fail to install nbdInfo', err)
})
}
export const installNbdKit = file => {
return _call('esxi.installNbdKitFromSource')
.then(() => {
success('nbdkit successfullly installed successfully installed')
})
.catch(err => {
error('fail to install nbdkit', err)
})
}
// GitHub API ---------------------------------------------------------------
const _callGithubApi = async (endpoint = '') => {
const url = new URL('https://api.github.com/repos/vatesfr/xen-orchestra')
packages/xo-web/src/xo-app/vm-import/esxi/esxi-import.js
+8 -105
View File
@@ -3,21 +3,12 @@ import ActionButton from 'action-button'
import Button from 'button'
import Collapse from 'collapse'
import Component from 'base-component'
import Dropzone from 'dropzone'
import Icon from 'icon'
import Link from 'link'
import React from 'react'
import { connectStore, resolveId } from 'utils'
import { createGetObjectsOfType, createSelector } from 'selectors'
import {
esxiCheckInstall,
esxiListVms,
importVddkLib,
importVmsFromEsxi,
installNbdInfo,
installNbdKit,
isSrWritable,
} from 'xo'
import { esxiCheckInstall, esxiListVms, importVmsFromEsxi, isSrWritable } from 'xo'
import { find, forEach, isEmpty, keyBy, map, pick } from 'lodash'
import { injectIntl } from 'react-intl'
import { Input } from 'debounce-input-decorator'
@@ -33,7 +24,7 @@ function EsxiCheckResults({ esxiCheck }) {
return (
<ul className='list-group'>
{Object.entries(esxiCheck).map(([name, value]) => {
const { status, error, version, expectedVersion } = value
const { status, error } = value
return (
<li key={name}>
<Icon icon={status} size='lg' fixedWidth />
@@ -42,7 +33,6 @@ function EsxiCheckResults({ esxiCheck }) {
{status === 'error' && `"${error}"`}
&nbsp;
{status === 'error' && _('esxiCheckingPrerequisiteError')}
{version && status === 'alarm' && _('esxiCheckedPrerequisiteVersion', { version, expectedVersion })}
</li>
)
})}
@@ -63,7 +53,6 @@ class EsxiImport extends Component {
state = {
concurrency: N_IMPORT_VMS_IN_PARALLEL,
hostIp: window.localStorage.getItem('esxi_host') ?? '',
installingEsxiLib: false,
importing: false,
isConnected: false,
password: window.localStorage.getItem('esxi_password') ?? '',
@@ -72,9 +61,7 @@ class EsxiImport extends Component {
stopOnError: true,
template: undefined,
user: window.localStorage.getItem('esxi_user') ?? '',
vddkFile: undefined,
esxiCheck: undefined,
esxiCheckError: undefined,
rememberConnection: !!window.localStorage.getItem('esxi_host'),
}
@@ -82,33 +69,11 @@ class EsxiImport extends Component {
this._esxiCheck()
}
_esxiCheck() {
this.setState({ esxiCheck: undefined, installingEsxiLib: false }, async () => {
this.setState({ esxiCheck: undefined }, async () => {
const esxiCheck = await esxiCheckInstall()
this.setState({ esxiCheck })
})
}
_handleDropVddk = files => {
this.setState({ vddkFile: files?.[0] })
}
_handleImportVddk = async () => {
this.setState({ installingEsxiLib: true })
try {
await importVddkLib({ file: this.state.vddkFile })
} catch (error) {
this.setState({ esxiCheckError: error })
}
return this._esxiCheck()
}
_installNbdInfo = async () => {
this.setState({ installingEsxiLib: true })
await installNbdInfo()
return this._esxiCheck()
}
_installNbKit = async () => {
this.setState({ installingEsxiLib: true })
await installNbdKit()
return this._esxiCheck()
}
_getDefaultNetwork = createSelector(
() => this.state.pool?.master,
() => this.props.hostsById,
@@ -266,7 +231,6 @@ class EsxiImport extends Component {
concurrency,
esxiCheck,
hostIp,
installingEsxiLib,
importing,
isConnected,
network = this._getDefaultNetwork(),
@@ -278,7 +242,6 @@ class EsxiImport extends Component {
stopSource,
stopOnError,
user,
vddkFile,
vms,
vmsById,
} = this.state
@@ -287,72 +250,12 @@ class EsxiImport extends Component {
return <div>checking</div>
}
// cehck nbdkit, nbdinfo, nbdkit plugin vddk
for (const [library, fn] of [
['nbdinfo', this._installNbdInfo],
['nbdkit', this._installNbKit],
['nbdkitPluginVddk', this._installNbKit],
]) {
const check = esxiCheck[library]
if (check.status !== 'success') {
return (
<div>
<Row>
<EsxiCheckResults esxiCheck={esxiCheck} />
</Row>
{check.version === undefined && (
<div>
<div className='mt-1 form-group pull-right'>
<ActionButton btnStyle='primary' className='mr-1' handler={fn} icon='import'>
{_('esxiLibraryAutoInstall', { library })}
</ActionButton>
{installingEsxiLib && (
<p>
{_('esxiLibraryInstalling', { library })}
<Link to='/tasks?s_xo=1_3_desc-status%3Apending+esxi.install' target='_blank'>
{_('esxiProgressLinkText')}
</Link>
</p>
)}
{!installingEsxiLib && <p>{_('esxiLibraryManualInstall')}</p>}
</div>
</div>
)}
{check.version !== undefined && (
<p>
{_('esxiLibraryOutdated', { library, expectedVersion: check.expectedVersion, version: check.version })}
</p>
)}
</div>
)
}
}
if (esxiCheck.vddk?.status === 'error') {
// vectura is the only prerequisite: one binary, shipped with xo-server
if (esxiCheck.vectura.status !== 'success') {
return (
<div>
<Row>
<EsxiCheckResults esxiCheck={esxiCheck} />
</Row>
<p>
{_('esxiLibraryInfo')} :{' '}
<a
href='https://developer.broadcom.com/sdks/vmware-virtual-disk-development-kit-vddk/9.0'
target='_blank'
rel='noreferrer'
>
{_('esxiLibraryLink')}
</a>
</p>
<Dropzone multiple={false} onDrop={this._handleDropVddk} message={_('esxiVddkLibrary')} accept='.tar.gz' />
{vddkFile && (
<div className='form-group pull-right'>
<ActionButton btnStyle='primary' className='mr-1' handler={this._handleImportVddk} icon='import'>
{_('esxiVddkLibraryImport')}
</ActionButton>
</div>
)}
</div>
<Row>
<EsxiCheckResults esxiCheck={esxiCheck} />
</Row>
)
}
scripts/bump-pkg
+5
View File
@@ -14,6 +14,11 @@ cd "$(dirname "$(which "$0")")/.."
pkg=$1
case "$pkg" in
vectura)
# a Rust crate, which @xen-orchestra/vmware-explorer bundles
shift
exec @xen-orchestra/vmware-explorer/vectura/scripts/release.sh "$@"
;;
@*/*)
cd "$pkg"
;;
scripts/check-changed-pkgs.js
+5 -1
View File
@@ -33,6 +33,10 @@ const IGNORED_PACKAGES = new Set([
'@xen-orchestra/lite',
])
// vectura is a Rust crate, released on its own: @xen-orchestra/vmware-explorer
// bundles its binary and `gen-deps-list.js` releases it after vectura
const VECTURA_DIR = '@xen-orchestra/vmware-explorer/vectura/'
// files which don't change what is shipped to users: a package whose changes
// are limited to them does not need to be released
const IGNORED_FILES = [
@@ -72,7 +76,7 @@ function getChangedPackages(changedFiles, packageNamesByDir) {
continue
}
const name = packageNamesByDir.get(`${parts[0]}/${parts[1]}`)
const name = file.startsWith(VECTURA_DIR) ? 'vectura' : packageNamesByDir.get(`${parts[0]}/${parts[1]}`)
if (name === undefined || IGNORED_PACKAGES.has(name) || isIgnoredFile(file)) {
continue
}
scripts/check-changed-pkgs.test.mjs
+11
View File
@@ -8,6 +8,7 @@ const PACKAGE_NAMES_BY_DIR = new Map([
['@xen-orchestra/babel-config', '@xen-orchestra/babel-config'],
['@xen-orchestra/backups', '@xen-orchestra/backups'],
['@xen-orchestra/lite', '@xen-orchestra/lite'],
['@xen-orchestra/vmware-explorer', '@xen-orchestra/vmware-explorer'],
['packages/xo-server', 'xo-server'],
])
@@ -114,6 +115,16 @@ describe('getChangedPackages', function () {
)
})
it('maps a vectura file to vectura, not to the package which bundles it', function () {
assert.deepEqual(
changedPackageNames([
'@xen-orchestra/vmware-explorer/vectura/src/main.rs',
'@xen-orchestra/vmware-explorer/vectura/tests/cli.rs',
]),
['vectura']
)
})
it('handles an empty list of changed files', function () {
assert.deepEqual(changedPackageNames([]), [])
})
scripts/gen-deps-list.js
+12 -1
View File
@@ -2,7 +2,8 @@
'use strict'
const semver = require('semver')
const { getPackages } = require('./utils')
const { join } = require('path')
const { getPackages, readFile } = require('./utils')
const { readChangelogPackages } = require('./_readChangelogPackages.js')
const invert = require('lodash/invert')
const keyBy = require('lodash/keyBy')
@@ -78,6 +79,16 @@ async function main(args, scriptName) {
}
allPackages = keyBy(await getPackages(true), 'name')
// vectura is a Rust crate, not an npm package: @xen-orchestra/vmware-explorer
// bundles its binary, so a vectura release is followed by one of
// vmware-explorer, as if it depended on this exact version
const vecturaVersion = /^version = "(.+)"$/m.exec(
await readFile(join(__dirname, '../@xen-orchestra/vmware-explorer/vectura/Cargo.toml'))
)[1]
allPackages.vectura = { name: 'vectura', package: { name: 'vectura', version: vecturaVersion } }
allPackages['@xen-orchestra/vmware-explorer'].package.dependencies.vectura = vecturaVersion
const releaseOrder = computeDepOrder(allPackages)
Object.entries(toRelease).forEach(([packageName, releaseType]) => {