Box

Batch process all your records to store structured outputs in a Box account.

The requirements are as follows.

1. Access to the Developer Console from your Box enterprise account or Box developer account.

2. A Box Custom App in your Box account, set up to use Server Authentication (with JWT). See Setup with JWT.

3. The appropriate application scopes and advanced features set up for the Box Custom App, as follows:

   • In the Box Custom App, on the Configuration tab, under Application Scopes, check the box titled Write all files and folders stored in Box.
   • In the Box Custom App, on the Configuration tab, under Advanced Features, check the box titled Make API calls using the as-user header.
   • Click Save Changes before continuing.

4. Authorization by a Box Admin in your Box account for the Box Custom App. See App Authorization.

5. Access by your Box account’s source or target folder to your Box Custom App, as follows:

   • In the Box Custom App, on the General Settings tab, copy the Service Account ID (which takes the form AutomationUser_<your-app-service-id>_<[e-mail]).
   • Share your Box account’s target folder with the copied service account’s email address as a Co-owner or Editor.
   • Note the remote URL to the target folder, which takes the format box://<path/to/folder/in/account>.

6. The private key configuration JSON file for the Box Custom App, or a string that contains this file’s contents.

   • To download this file, in the Box Custom App, on the Configuration tab, under Add and Manage Public Keys, click Generate a Public/Private Keypair. Store the downloaded private key configuration JSON file in a secure location.

   • To ensure maximum compatibility across Unstructured service offerings, you should give the private key configuration JSON file information to Unstructured as a single-line string that contains the contents of the downloaded private key configuration JSON file (and not the file itself). To print this single-line string, suitable for copying, you can run one of the following commands from your Terminal or Command Prompt. In this command, replace <path-to-downloaded-key-file> with the path to the private key configuration JSON file that you downloaded by following the preceding instructions.

     • For macOS or Linux:

       tr -d '\n' < <path-to-downloaded-key-file>
       

     • For Windows:

       (Get-Content -Path "<path-to-downloaded-key-file>" -Raw).Replace("`r`n", "").Replace("`n", "")
       

Document permissions metadata

The source connector outputs any permissions information that it can find in the source location about the processed source files, and associates that information with each corresponding element that is generated. This permissions information is output into the permissions_data field, which is within the data_source field under the element’s metadata field (metadata.data_source.permissions_data). This information lists the users or groups, if any, that have permissions to read, update, or delete the element’s associated source document.

The permissions metadata Unstructured outputs should not be used for runtime authorization or access control enforcement.

Unstructured outputs document permissions metadata that is accurate only at the point in time when Unstructured ingested the corresponding document to which those permissions applied. Because this metadata is a point-in-time copy of the permissions in the source location, these metadata outputs that are sent to your destination location are not always guaranteed to match the current permissions in the source location.

Also, be aware that Unstructured updates permission metadata for a document only when the document’s content has changed.

This is because Unstructured performs incremental processing of documents only when documents’ content has changed—not when only the documents’ permissions have changed. Whenever Unstructured performs incremental processing of documents for a workflow (in other words, if Reprocess All Files is turned off or set to false for a workflow), that worfklow will not output metadata for any document permissions that have been added, changed, or removed since the previous workflow run, unless the corresponding documents’ content has also been changed since the previous workflow run.

Permissions are defined in Box through collaborations, which function similar to access control lists. Collaboration objects are attached to folders, and those collaborations cascade down to everything inside that folder. A file’s effective permissions come from the chain of ancestor folders above it. For more information, see Collaborations overview in the Box developer documentation.

To determine a file’s effective permissions, Unstructured traverses the full folder hierarchy for each file, collecting collaborations at every level, and merging them. It also includes collaborations applied directly to the file itself. For more information, see List folder collaborations and List file collaborations in the Box developer documentation.

When Unstructured compiles a file’s effective permissions:

• If the same user or group appears at multiple levels with different roles, Unstructured chooses the least-restrictive role.
• Unstructured does not include the following in the permission metadata. No sentinel value is written into the permission metadata.
  • all_users_group: Box’s built-in group representing every user in the enterprise.
  • is_access_only: Collaboration property that determines whether, for collaborators, to display the items in the All Files list and let them see the path to the root folder for the shared item.
• Unstructured only includes collaborations with a status of accepted. It does not include collaborations with a status of pending or rejected.

Unstructured includes a maximum of 500 permissions per file.

Unstructured maps Box roles to read, update, and delete access:

Box role	Box access	Unstructured permission metadata
owner	Full access	read, update, delete
co-owner	Full access except transfer ownership	read, update, delete
editor	View, download, upload, edit, delete, copy, move, rename	read, update
viewer	Preview, download, comment	read
previewer	Preview	read
viewer uploader	Preview, download, upload	read
previewer uploader	Preview, upload	read
uploader	Upload and see item names	excluded entirely

Note the following:

• Unstructured excludes the uploader role because it confers no read, update, or delete access.
• Unstructured maps the Box editor role to read and update access but not delete, even though that role does allow deleting files.

For more information, see Understanding collaborator permission level at Box Support.

Identifier format

Users and groups are identified by the numeric ID that Box assigns them. To retrieve information about a specific user or group, use appropriate Box API with the corresponding ID:

• Get user
• Get group

Metadata output example

The following example shows what the output looks like. Ellipses indicate content that has been omitted from this example for brevity.

[
    {
        "...": "...",
        "metadata": {
            "...": "...",
            "data_source": {
                "...": "...",
                "permissions_data": [
                    {
                        "read": {
                            "users": ["11446688", "11336699"],
                            "groups": ["11223344"]
                        }
                    },
                    {
                        "update": {
                            "users": ["11446688"],
                            "groups": []
                        }
                    },
                    {
                        "delete": {
                            "users": [],
                            "groups": []
                        }
                    }
                ],
                "...": "..."
            }
        }
    }
]

The Box connector dependencies:

CLI, Python

pip install "unstructured-ingest[box]"

You might also need to install additional dependencies, depending on your needs. Learn more.

The following environment variables:

• BOX_APP_CONFIG - The local path to the downloaded private key configuration JSON file for the Box Custom App, or a single-line string that contains the contents of this file, represented by --box-app-config (CLI) or box_app_config (Python).
• BOX_REMOTE_URL - The remote URL to the target folder, represented by --remote-url (CLI) or remote_url (Python). This URL must take the format box://<path/to/folder/in/account>.

Now call the Unstructured CLI or Python. The source connector can be any of the ones supported. This example uses the local source connector.

This example sends files to Unstructured for processing by default. To process files locally instead, see the instructions at the end of this page.

CLI

#!/usr/bin/env bash

# Chunking and embedding are optional.

unstructured-ingest \
  local \
    --input-path $LOCAL_FILE_INPUT_DIR \
    --output-dir $LOCAL_FILE_OUTPUT_DIR \
    --strategy hi_res \
    --chunk-elements \
    --embedding-provider huggingface \
    --num-processes 2 \
    --verbose \
    --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
  box \
    --box-app-config $BOX_APP_CONFIG \
    --remote-url $BOX_REMOTE_URL

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the --partition-by-api option (CLI) or partition_by_api (Python) parameter to specify where files are processed:

• To do local file processing, omit --partition-by-api (CLI) or partition_by_api (Python), or explicitly specify partition_by_api=False (Python).

  Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

  • --api-key $UNSTRUCTURED_API_KEY (CLI) or api_key=os.getenv("UNSTRUCTURED_API_KEY") (Python)
  • --partition-endpoint $UNSTRUCTURED_API_URL (CLI) or partition_endpoint=os.getenv("UNSTRUCTURED_API_URL") (Python)
  • The environment variables UNSTRUCTURED_API_KEY and UNSTRUCTURED_API_URL

• To send files to the legacy Unstructured Partition Endpoint for processing, specify --partition-by-api (CLI) or partition_by_api=True (Python).

  Unstructured also requires an Unstructured API key and API URL, by adding the following:

  • --api-key $UNSTRUCTURED_API_KEY (CLI) or api_key=os.getenv("UNSTRUCTURED_API_KEY") (Python)
  • --partition-endpoint $UNSTRUCTURED_API_URL (CLI) or partition_endpoint=os.getenv("UNSTRUCTURED_API_URL") (Python)
  • The environment variables UNSTRUCTURED_API_KEY and UNSTRUCTURED_API_URL, representing your API key and API URL, respectively.
  You must specify the API URL only if you are not using the default API URL for Unstructured Ingest, which applies to Let’s Go, Pay-As-You-Go, and Business SaaS accounts.

  The default API URL for Unstructured Ingest is https://api.unstructuredapp.io/general/v0/general, which is the API URL for the legacy Unstructured Partition Endpoint. However, you should always use the URL that was provided to you when your Unstructured account was created. If you do not have this URL, email Unstructured Support at [e-mail].

  If you do not have an API key, get one now.

  If you are using a Business account, the process for generating Unstructured API keys, and the Unstructured API URL that you use, are different. For instructions, see your Unstructured account administrator, or email Unstructured Support at [e-mail].