> ## Documentation Index
> Fetch the complete documentation index at: https://docs.unstructured.io/llms.txt
> Use this file to discover all available pages before exploring further.
# Weaviate
> Send processed data from Unstructured to Weaviate.
First time creating a connector? [Read this first](/pipelines/connector-first-time-reqs).
## Requirements
You will need:
### Accounts
[Unstructured Pipelines](/pipelines/overview) and [Unstructured API](/api-reference/overview) support:
* [Weaviate Cloud](https://weaviate.io/developers/wcs) clusters
[Unstructured Ingest](/open-source/ingestion/overview) supports:
* Weaviate Cloud clusters
* [Weaviate installed locally](https://weaviate.io/developers/weaviate/quickstart/local)
* [Embedded Weaviate](https://weaviate.io/developers/weaviate/installation/embedded)
### Resources
For Weaviate installed locally:
* The name of the target collection on the local instance.
For Embedded Weaviate:
* The instance's connection URL and the name of the target collection on the instance.
For Weaviate Cloud:
* A Weaviate database instance. The following information assumes that you have a Weaviate Cloud (WCD) account with a Weaviate database cluster in that account.
[Create a WCD account](https://weaviate.io/developers/wcs/quickstart#create-a-wcd-account). [Create a database cluster](https://weaviate.io/developers/wcs/quickstart#create-a-weaviate-cluster). For other database options, [learn more](https://weaviate.io/developers/weaviate/installation).
* The URL of the database cluster. [Get the URL](https://weaviate.io/developers/wcs/quickstart#explore-the-details-panel).
* The name of the target collection in the database. [Create a collection](https://weaviate.io/developers/wcs/tools/collections-tool).
How you create and specify the collection depends on whether you want document metadata stored as nested JSON in a single column, or have Unstructured flatten the metadata and store each field as a separate column. For more information, see [Storing document metadata](#storing-document-metadata) later in this topic.
* If you choose to store document metadata as a JSON blob, you can specify a collection, or have the connector attempt to create a collection for you automatically at run time.
***To specify an existing collection***
For the connector, specify the name of an existing collection.
If you specify an existing collection name, and Unstructured generates embeddings,
but the number of dimensions that are generated does not match the existing collection's embedding settings, the job will fail.
You must change your Unstructured embedding settings or your existing collection's embedding settings to match, and try the job again.
***To have the connector create a table***
You can have the connector attempt to create a collection for you automatically at run time. To do so, specify the name of the collection that you want the connector to attempt to create--that is, a collection that does not already exist.
* If you choose to flatten the document metadata, you must create the collection before you configure your connector. For more information, see [Storing document metadata](#storing-document-metadata) later in this topic.
### Authentication
* The API key for the database cluster. For more information, see [Authentication](https://docs.weaviate.io/cloud/manage-clusters/authentication) in the Weaviate Cloud documentation.
## Viewing generated embeddings
If Unstructured creates a new collection and generates embeddings, you will not see an `embeddings` property in tools such as the Weaviate Cloud
**Collections** user interface. To view the generated embeddings, you can run a Weaviate GraphQL query such as the following. In this query, replace `` with
the name of the new collection, and replace `` with the name of each additional available property that
you want to return results for, such as `text`, `type`, `element_id`, `record_id`, and so on. The embeddings will be
returned in the `vector` property.
```text theme={null}
{
Get {
{
_additional {
vector
}
}
}
}
```
## Inferring missing properties
If [auto-schema](https://docs.weaviate.io/weaviate/config-refs/collections#auto-schema) is enabled in Weaviate (which it is by default),
Weaviate can infer missing properties and add them to the collection definition at run time. However, it is a Weaviate best practice to manually define as much
of the data schema in advance as possible, since manual definition gives you the most control.
## Minimal required schema
The minimum viable schema for Unstructured includes only the `element_id` and `record_id` properties. The `text` and `type` properties should also be included, but they are technically optional.
The `record_id`, `element_id`, and `id` fields are closely related, but each has a distinct purpose. For more information, see [How connectors use record IDs, element IDs, and IDs](/api-reference/record-element-id).
The following code example shows how to use the [weaviate-client](https://pypi.org/project/weaviate-client/) Python package to create a
collection in a Weaviate Cloud database cluster with this minimum viable schema, and to specify that Unstructured will generate the embeddings for this collection.
To connect to a locally hosted Weaviate instance instead, call [weaviate.connect\_to\_local](https://docs.weaviate.io/weaviate/connections/connect-local).
To connect to Embedded Weaviate instead, call [weaviate.connect\_to\_embedded](https://docs.weaviate.io/weaviate/connections/connect-embedded).
```python theme={null}
import os
import weaviate
from weaviate.classes.init import Auth
import weaviate.classes.config as wvc
client = weaviate.connect_to_weaviate_cloud(
cluster_url=os.getenv("WEAVIATE_URL"),
auth_credentials=Auth.api_key(api_key=os.getenv("WEAVIATE_API_KEY")),
)
collection = client.collections.create(
name="MyCollection",
properties=[
wvc.Property(name="element_id", data_type=wvc.DataType.UUID),
wvc.Property(name="record_id", data_type=wvc.DataType.TEXT),
wvc.Property(name="text", data_type=wvc.DataType.TEXT),
wvc.Property(name="type", data_type=wvc.DataType.TEXT),
],
vectorizer_config=None, # Unstructured will generate the embeddings instead of Weaviate.
)
client.close()
```
Unstructured cannot provide a schema that is guaranteed to work in all
circumstances. This is because these schemas will vary based on your source files' types; how you
want Unstructured to partition, chunk, and generate embeddings; any custom post-processing code that you run; and other factors.
See also:
* [Collection schema](https://weaviate.io/developers/weaviate/config-refs/schema)
* [Unstructured document elements and metadata](/api-reference/legacy-api/partition/document-elements)
## Storing document metadata
Unstructured offers the following options for storing [document metadata](/concepts/document-elements#metadata) in the destination table:
* Store the metadata as a single nested JSON field:
```json theme={null}
{
"metadata": {
"category_depth": 1,
"data_source": {
"url": "s3://my-source-bucket/path/chart-and-table.pdf",
. . .
}}
}
```
* *Flatten* the metadata by writing each metadata field as its own typed, queryable column:
```json theme={null}
{
"category_depth": 1,
"data_source_url": "s3://my-source-bucket/path/chart-and-table.pdf",
. . .
}
```
In general, storing the metadata as a JSON blob works for most use cases, unless you want query individual metadata fields directly using standard SQL, or you are using tools that require columnar data.
Storing the metadata as a JSON blob works for most use cases, including when:
* Performing dot.notation queries on the stored JSON is sufficient for your needs.
* Document metadata schemas vary across file sources. When flattening document metadata, Unstructured drops fields that do not match existing columns in the schema.
* You want the connector to automatically generate the destination table. This option is not supported when flattening document metadata.
Flattening the metadata and storing it in multiple columns is most useful when:
* You want to query individual metadata fields directly using standard SQL, without JSON parsing.
* The business intelligence or analytics tools you are using require columnar data.
To store metadata as a JSON blob, when configuring the connector uncheck **Flatten Metadata** (in the Unstructured Pipelines), or set `flatten_metadata` to `false` (in the Unstructured API). To flatten the metadata, check **Flatten Metadata**, or set `flatten_metadata` to `true`.
Storing metadata as a JSON blob is the default.
### Considerations when flattening metadata
If you choose to have Unstructured flatten document metadata, you must create the collection to use as the destination, and specify it when creating the connector. In order to prevent possible data loss, Unstructured will not automatically create a new collection.
Considerations to keep in mind when creating the collection:
* The collection must contain a column for each metadata field you want to store. Any metadata field that does not have a corresponding column in the table is silently dropped, although the event is written to the logs. For more information, see [Logging and monitoring](/business/security-compliance/overview#logging-and-monitoring).
* The collection must contain a `record_id` property. Unstructured requires this property for re-run deduplication.
* Do not declare metadata columns as `NOT NULL`. Missing metadata values are written as `NULL`.
* Unstructured passes values through as their JSON-native type: strings, numbers, boolean, and so on. For example, no special formatting is applied to timestamp values.
* Metadata fields that are lists are not further flattened. Lists remain single columns.
* Declare any list-of-object fields as of the object array type. For flattened fields that contain lists of objects—such as links, permissions, or regex matches—declare the property type as `OBJECT_ARRAY` in the collection schema.
### Metadata flattening example
The following example demonstrates how Unstructured flattens metadata into separate columns. Consider the following metadata:
```json theme={null}
{
"metadata": {
"category_depth": 1,
"data_source": {
"url": "s3://my-source-bucket/path/chart-and-table.pdf",
"version": "864ae40b0719e976e98ba0a7b9fcba92",
"record_locator": {
"protocol": "s3",
"remote_file_path": "s3://my-source-bucket/path/"
}
},
"languages": ["eng"]
}
}
```
When flattening metadata, Unstructured generates a field name comprised of the full path to that field within the metadata structure, from the outermost object to the field itself. For example, `protocol`, which is included in the `record_locator` object, which is in turn within `data_source`, becomes `data_source_record_locator_protocol`:
```json theme={null}
{
"category_depth": 1,
"data_source_url": "s3://my-source-bucket/path/chart-and-table.pdf",
"data_source_version": "864ae40b0719e976e98ba0a7b9fcba92",
"data_source_record_locator_protocol": "s3",
"data_source_record_locator_remote_file_path": "s3://my-source-bucket/path/",
"languages": ["eng"]
}
```
## Creating the connector
To create the destination connector:
1. On the sidebar, click **Connectors**.
2. Click **Destinations**.
3. Cick **New** or **Create Connector**.
4. Give the connector some unique **Name**.
5. In the **Provider** area, click **Weaviate**.
6. Click **Continue**.
7. Follow the on-screen instructions to fill in the fields as described later on this page.
8. Click **Save and Test**.
Fill in the following fields:
* **Name** (*required*): A unique name for the connector.
* **Cluster URL** (*required*): The URL of the Weaviate database cluster.
* **Collection Name**: The name of the target collection within the cluster. If you specify the name of a collection that does not exist, the connector attempts to create it.
* **Flatten Metadata**: Check to have Unstructured flatten the metadata and store each field as a separate columns. Uncheck to store document metadata as nested JSON in a single column. For more information, see [Storing document metadata](#storing-document-metadata).
* **API Key** (*required*): The API key provided by Weaviate to access the cluster.