The cache now announces what a listing finds and what a journal replay
changes, as `add`, `update` and `remove` events carrying the archive and its
previous value, on an emitter the app exposes as `app.vmBackupArchives`. What
was last announced for a repository outlives its entry, so a repository read
again from scratch only announces what really changed. A repository which is
removed or disabled goes through the new `forgetVmBackupRepository()`, which
announces all its archives as removed. A listener which throws is logged and no
longer fails the listing.
The REST API adds `backup-archive` to the SSE collections. It is bound to
`app.vmBackupArchives` the way `task` is, and filtered by `backup-archive:read`
through the same path as every other non-XAPI collection.
Nothing polls the repositories: changes are pushed whenever any client lists a
repository. The first listing after a restart sends every archive it already
holds as `add`.
XO-2992
XO-2993
the live mount feature create a lot of SR , each with 1 disk
this PR introduce an automatic cleanup that will forget a SR as soon as the disk is removed, thus removing any user action needed to cleanup thins in the nominal case
crashing XO can leave SR pluggued
* feat(xo-server,rest-api,xo-web/rpu): force-close an incomplete rolling pool update record
A pool whose previous rolling pool update left a recovery record refuses a
new run until that record is dealt with. Once the operator has reviewed the
pool, `pool.finalizeRollingUpdate` and
`POST /rest/v0/pools/{id}/actions/finalize_rolling_update` close it.
Without `force`, the finalization is refused (`incorrectState`, property
`unrestoredItems`) while the record says the run changed something the pool
still shows changed: HA, auto power-on, WLB, load balancer, backup
schedules, hosts left disabled, running VMs away from their home host,
pinned VMs shut down and not started again. A record that cannot be read
is refused too, with an unknown list.
With `force`, the record is deleted, the abandoned items are written to the
task and the trace, nothing is restored nor changed in the pool.
To tell what the run changed from what the operator did, the run now
persists, before each change, the settings it touches (strict write) and
whether each host was enabled before the run.
XO 5 shows a "Finalize" action on the incomplete update banner, which is
now also shown for a run that succeeded but whose record could not be
removed. Forcing asks for a second confirmation listing the items.
implementation of an iscsi target and initiator in typescript
this will serve as a basis of 2 projects:
connecting directly to 3rd party san
exposing a disk from a backup archive to the host, effectively enabling direct mount to a running VM. Final target is to be able to boot completely the VM
@vates/iscsi/VALIDATION.md file is a journal of the additional test I run on my lab
@vates/iscsi/PROTOCOL.md describe the subset of the protocol implemented . For example we only support 1 LUN per target, not multiple connection , only one type of auth and keep alive
* feat(xo-server,rest-api,xo-web/rpu): safe start on a partially updated pool
A rolling pool update used to overwrite the recovery record left by the
previous run and to look only at the master to decide whether there was
anything to update. Both hid a pool left half updated: an interrupted run,
or updates applied by hand, could be silently completed or forgotten.
- `startRpuRecoveryRun` now refuses a new run while the pool still has a
record, whatever its status, with an `incorrectState` error (property
`rollingUpdateRecovery`). The RPU guard held by the caller makes the
check-then-write safe. A run which fails before any host was handled
drops its record instead of marking it failed: the orchestrator restores
what it changed on its way out, and a refused precondition (a guidance to
accept, a pinned VM to shut down) must not block the retry the operator
just consented to.
- The update refuses a pool whose master is already up to date while
another host is not (`incorrectState`, property `partiallyUpdatedPool`,
`actual` listing the outdated hosts) unless the new
`acceptCurrentStateAsBaseline` option is set. The option is exposed on
`pool.rollingUpdate` and on `POST /rest/v0/pools/{id}/actions/rolling_update`.
- XO5 enables the rolling pool update button as soon as any host of the
pool has missing patches, not only the master, disables it while a
recovery record exists, and asks for confirmation before accepting the
current state as the baseline.
Until the force-close operation ships, a leftover record can only be
removed by hand: it must land in the same release.
* feat(xo-server,rest-api,xo-web/rpu): changelog entry for #10394
* fix(xo-server/rpu): getRollingUpdateRecovery returns null on no record
* fix(xo-server/rpu): persist acceptCurrentStateAsBaseline, mark record succeeded on delete failure
* fix(xo-server/rpu): drop an empty record after the pool is restored, update LINSTOR only past the guards
* feat(xo-server,rest-api,xo-web/rpu): persist a recovery record across restarts
A rolling pool update now writes a `rpuRecovery` record (LevelDB, one per
pool) before its first side effect and keeps it updated during the run:
operator options, update variant, host order, initial VM placement, per-host
step statuses, last filtered error, and the pinned VMs shut down for their
host's reboot (previously only tracked in memory, so an xo-server crash left
them halted with nothing to restart them). A record left in a live status at
xo-server boot is flipped to `interrupted`; a successful run deletes its
record; an unreadable or unknown-version record is reported as `blocked`
without touching the stored value.
The status is exposed through `pool.getRollingUpdateRecovery` and
`GET /rest/v0/pools/{id}/rolling_update_recovery` (filtered view: never the
raw intent, reusing the existing pool permissions), and the pool Patches tab
in XO5 shows a warning with per-host progress, the last error and the pinned
VMs still halted.
First card of the RPU resume-after-failure spike: the record only reports
for now, resuming an interrupted run comes separately.
* docs(changelog): entry for the RPU recovery record (#10331)
* fix(xo-server,rest-api,xo-web/rpu): updates after review
* test(xo-server/rpu): cover the recovery record during the migrate back phase
The migrate back phase now retries the rejected migrations before giving
up, so a host is only recorded as failed once the retry passes are over,
and a pool which opted out of the phase must still close its `restoreVms`
steps, otherwise a finished run reads as still running in the record.
Both are asserted through a recorder spying on the step transitions.
* fix(xo-server/rpu): keep per-key reads in the recovery reconciliation
* test(xo-server/rpu): stub the recovery recorder in the rolling pool update tests
Since #10313, XenServers.rollingPoolUpdate runs the backup guard itself and
its tests drive it with a fake app. With the recovery record, the run also
calls app.startRpuRecoveryRun, which the fake app did not have.
The stub records its call, so the tests also check that a refusal from the
backup guard leaves no record behind.
* fix(xo-server/rpu): do not fail a successful run when its record cannot be deleted
This branch is a reliability overhaul of the ESXi/vSphere source side of V2V, plus a scope reduction of the package. The SOAP plumbing is rewritten:
VimClient wraps @vates/node-vsphere-soap in a promise API with one emitter per call (no more cross-talk between concurrent calls, no listener leak) and a pTimeout;
vim25 faults are now parsed (_parseFault.mjs) so callers get a real fault type instead of task execution failed;
the library no longer throws from async callbacks (which killed xo-server) and its beforeExit logout hook is per-client instead of removeAllListeners.
Request specs are built in explicit schema order (soap/specs.mjs) with xsi:type, and property values are normalized (soap/normalize.mjs) so search() and the new #retrieveProperty share one shape.
search() now paginates, cancels the retrieval and destroys the container view.
Datastore HTTP downloads get session-cookie reuse, retry with jittered backoff, a headers-only timeout, and range-honouring validation.
Task waiting polls with backoff and per-operation timeouts (6 h for snapshot consolidation).
nbdkit servers are memoized by a key that includes the export options, the readiness wait races the process death instead of sleeping 2 s
the password file is 0600 and unlinked
the thumbprint is read over TLS in-process instead of two openssl spawns.
The old VHD-over-datastore readers (VhdEsxi*, Datastore*, openDeltaVmdkAsVhd) are deleted, hence the majo, rleaving the vddk/NBD path with a COWD grain-directory parser as data-map fallback.
~1.7k lines of unit tests are added.
* fix(xo-server,rest-api): apply the backup guard to rolling pool operations started over REST
The JSON-RPC methods pool.rollingUpdate and pool.rollingReboot refused to run
while a backup job targets the pool, but the REST actions
POST /pools/:id/actions/rolling_update and rolling_reboot called the
orchestrators directly and skipped that check.
The guard now lives in XenServers.rollingPoolUpdate and Pools.rollingPoolReboot,
so every entry point (JSON-RPC, REST, scheduled jobs) applies the same rule.
The bypass moves into backupGuard(poolId, { bypassBackupCheck, operation }),
which logs a single warning with the operation and the user, and the other
callers (host.restart, host.restartAgent, host.stop, sr.reclaimSpace and their
REST counterparts) use it instead of their own if/else and log.
The REST rolling actions accept bypassBackupCheck in their body and record the
body in the task params, like the host actions do.
* fix(xo-server,rest-api): pass the targeted object ID to the backup guard
* fix(openmetrics): keep the Prometheus secret across xo-server restarts
The `secret` property of `configurationSchema` used a random `default`. That
expression is re-evaluated every time the module is loaded, and xo-server never
persists the values ajv fills in from schema defaults, so each restart handed
the metrics endpoint a brand new bearer token and Prometheus started getting
401s.
Drop the default and generate the secret in `load()` instead, saving it through
`xo.configurePlugin()` so it survives a restart.
Introduced by #9323
See https://xcp-ng.org/forum/topic/12415
* chore(changelog): add openmetrics secret entry
* fix(xo-server-openmetrics): generate a 256-bit Prometheus secret
Buffer.from(getRandomValues(new Uint32Array(8))) coerces each 32-bit
element to a single byte, so the generated secret held 64 bits of
entropy instead of 256. Use randomBytes(32) instead.
Introduced by fe29bd0d4
* fix(backups): timeout was not used correctly for NBD transfers
* fix(nbd-client): evict a dead client in multi instead
actual code was retrying 5 times, each time with a 60s timeout
multi also have its own retry on conneciton
since the client used is determnistic for a block, any stalled client
will be hammerred for a lot of time before giving up
this PR evict a nbd client that is failing to read a block (with its
retry logic)
* fix(nbd-client/multi): better spread read accross client
the actual code derivate the clientId from the block index
we have no information on the block distribution, only hopes that
it's random enough
this PR introduce a mechanism to use a round robin to select
the next client
* fix: correct fall back to full with /without nbd on qcow2
Customer imported large VM from VMWare (> 3 To), import switched to Qcow2 automatically but the process was stuck while building the qcow2 L1/L2 address tables, where the NbdDisk's hasBlock is called for each l2 table entry which iterated over the disk dataMap which contained ~113,528 ranges. This froze the process for more than 18 minutes after which it was either killed or died due to OOM error.
This fix improves the NbdDisk by:
Updating processDatamap to throw on overlapping ranges and merge touching extents.
Override default getBlockIndexesCount with optimized Nbd one.
Improve hasBlock with a built in cursor to improve performance.
Test file added to cover the new function and updated code.
Florent BEAUCHAMP