Harvest metadata and synchronize lineage resources

SQLink harvests object structure and synchronizes resources into lineage spaces for parsing and matching. A source connection can be reused across schemas; saving it does not complete harvesting.

Database harvesting

1. Prepare application systems and sources under System Management > Data Source Management and test connectivity.
2. Open Lineage Management > Harvesting Management > New Harvest > Data Source.
3. Enter name, source, and scope.
4. Configure sampling, name enrichment, and blacklist without excluding key objects.
5. Save, then select Execute in the task list.
6. Monitor results and verify objects in the catalog and lineage space.

Report harvesting

Choose Report, select the tool type, enter service URI, authentication, and report folders, then test and confirm. Check background task status and associations among reports, datasets, and database objects. Map report database connections to correct namespaces before parsing.

DDL files and APIs

Choose File. For DDL, select DDL Scripts, namespace, engine, and default schema. The engine determines syntax; the namespace determines storage and identity. Their database types can differ when appropriate to the actual semantics.

API metadata uses the built-in REST API metamodel through JSON files or direct HTTP access. Verify objects and associations; ordinary prose API descriptions are not structured definitions.

Custom metamodels

Create a file harvest, select Metamodel as database type, choose the existing model and namespace, download the template, populate it, upload, and run. See metamodel design and import.

Database and report resources

Harvesting automatically creates resource tasks. Database resources include fields, views, and procedures; report resources include structure and preparation SQL. Update metadata through harvesting rather than direct editing or deletion. You can change the engine and parse again.

Resynchronizing metadata first clears the task's existing metadata and lineage. Confirm need and source availability, then verify restored resources and paths. Assess relationship impact before changing namespaces.

OpenLineage

OpenLineage tasks consume third-party Kafka events, run state, and parent-child job relationships. Column lineage events can create relationships; event SQL can become lineage files. Verify topic, event format, producer, namespaces, inputs, outputs, and generated relationships. Event ingestion does not replace database harvesting, and missing events prevent conclusions about full coverage.