リイノ eaef93a3b6 Enhancement of Weaviate migration script (#691) ...

* Add property type migration to Weaviate migration script

Handle uuid→text conversion for document_id/doc_id and remove spurious
moduleConfig from chunk_index during schema migration. This fixes
property type incompatibilities that could cause issues even when
vectorConfig is already correct.

Fixes the following failure scenarios in the old script:

1. Schema type mismatch: Old script copies properties as-is, preserving
   uuid type for document_id/doc_id. Dify expects text type, so the
   migrated collection appears successful but Dify fails at runtime.

2. UUID object insertion failure: When source collection has uuid-typed
   fields, the Weaviate client returns Python UUID objects. Writing these
   into text-typed fields causes batch insert errors, leading to data
   loss or migration abort.

3. moduleConfig rejection: Stale moduleConfig on chunk_index from older
   Weaviate versions can cause collection creation to fail on newer
   Weaviate, aborting migration entirely.

4. Partial migration blindspot: Collections already migrated for
   vectorConfig but still carrying wrong property types were skipped
   with "NEW SCHEMA (skip)", leaving silent incompatibilities.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Add connection configuration comments with usage examples

Document how to configure Weaviate connection for both in-container
and local (port-forward) scenarios, and clarify derived values.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Fix data loss risk: verify copy before deleting migrated collection

Reorder replace_old_collection to prevent data loss on failure:
- Fetch schema BEFORE deleting anything
- Wrap data copy in try/except to preserve migrated collection on error
- Add count verification after copy, keep migrated as backup on mismatch
- Only delete the migrated collection after full verification passes
- Print recovery instructions (collection name) on every failure path

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* refactor: use Weaviate iterator and fixed-size batch per official recommendation

Replace manual cursor-based fetch_objects pagination with collection.iterator()
and switch batch.dynamic() to batch.fixed_size() to prevent overwhelming the
server during co-located migrations. Addresses review feedback from Weaviate team.

* fix: remove misleading BATCH_SIZE env var comment (hardcoded, not configurable)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

2026-03-30 22:12:53 +08:00

..

migrate_weaviate_collections.py

Enhancement of Weaviate migration script (#691)

2026-03-30 22:12:53 +08:00