Object Storage Integration

Storage Provider Setup

To set up your object storage for Score:

  1. Register with a provider of your choice and follow their instructions for setting up and configuring their service.

Minio Local Quickstart

If you'd like to quickly spin up a compatible object storage locally, you can run the following command: docker run --name minIO -p 9000:9000 -e MINIO_ACCESS_KEY=minio -e MINIO_SECRET_KEY=minio123 minio/minio:RELEASE.2018-05-11T00-29-24Z server /data

  1. Create two data buckets for Score to use:
  • A bucket to store object data
  • A bucket to store and maintain state information

Note: After creation, remember the IDs of both buckets, they will be required later for configuring Score.

  1. You may need to create a /data sub-folder in advance for each bucket. This requirement will depend on your storage provider and is summarized below:
Storage ProviderData sub-folder required
Amazon S3No
Microsoft AzureNo
MinIONo
OpenStack with CephYes
  1. Record the URL, access key and secret key used to access your storage service. These will be required later for configuring Score. Keep these values safe and secure.

For Amazon S3 buckets

Remember to document the geographical region where you have configured your buckets to be stored, this will be required when configuring Score.

Environment Variable Setup

Once your object storage is established, the next step involves configuring Score for connection. The specific configuration settings will vary based on your object storage provider. Below are the detailed setup instructions.

AWS, Ceph, or Minio

To connect Score with AWS, Ceph, or Minio storage, modify your .env.score file as follows:

bash
# ============================
# Object Storage Configruation Variables
# ============================
SPRING_PROFILES_ACTIVE=prod,aws
S3_ENDPOINT="http://localhost:9000"
S3_ACCESS_KEY="minio"
S3_SECRETKEY="minio123"
S3_SIGV4ENABLED="true"
BUCKET_NAME_OBJECT="object.bucket"
BUCKET_NAME_STATE="state.bucket"
BUCKET_SIZE_POOL=0
BUCKET_SIZE_KEY=2
UPLOAD_PARTSIZE=1048576
UPLOAD_RETRY_LIMIT=10
UPLOAD_CONNECTION_TIMEOUT=60000
UPLOAD_CLEAN_CRON="0 0 0 * * ?"
UPLOAD_CLEAN_ENABLED="true"

The following table summarizes the necessary variables:

SettingRequirementDescription
S3_ENDPOINTRequiredAPI endpoint URL of your storage service. Score will communicate with the service via this URL.
S3_ACCESSKEYRequiredAccess key for your object storage buckets.
S3_SECRETKEYRequiredSecret key for your object storage buckets.
S3_SIGV4ENABLEDRequiredSet to true if using AWS S3 Signature Version 4 for authentication. Otherwise, set to false.
BUCKET_NAME_OBJECTRequiredID of the bucket for storing object data.
BUCKET_NAME_STATERequiredID of the bucket for storing state information.
BUCKET_SIZE_POOLRequiredDESCRIPTION
BUCKET_SIZE_KEYRequiredDESCRIPTION
UPLOAD_PARTSIZERequiredByte size of each upload chunk to the object storage. Adjust for performance.
UPLOAD_RETRY_LIMITRequiredRetry attempts for failed uploads before aborting.
UPLOAD_CONNECTION_TIMEOUTRequiredTimeout duration in milliseconds for idle connections.
UPLOAD_CLEAN_CRONOptionalSchedule for the cleanup cron job, if enabled.
UPLOAD_CLEAN_ENABLEDOptionalSet to true for enabling the cleanup cron job.

Azure

For connecting Score with Azure storage, update .env.score as shown:

bash
SPRING_PROFILES_ACTIVE=prod,azure
AZURE_ENDPOINT_PROTOCOL=https
AZURE_ACCOUNT_NAME={{storage_account_name}}
AZURE_ACCOUNT_KEY={{storage_account_secret_key}}
BUCKET_NAME_OBJECT={{object_bucket}} # Object data storage bucket/container name
BUCKET_POLICY_UPLOAD={{write_policy}} # Access policy name for write operations
BUCKET_POLICY_DOWNLOAD={{read_policy}} # Access policy name for read operations
UPLOAD_PARTSIZE=104587
DOWNLOAD_PARTSIZE=250000000 # Default part size for downloads
OBJECT_SENTINEL=heliograph # Required sample object/file name for `ping` operations; default is `heliograph

The following table summarizes the Azure-specific variables:

SettingRequirementDescription
AZURE_ENDPOINT_PROTOCOLRequiredCommunication protocol for the Azure storage API endpoint (e.g., https).
AZURE_ACCOUNT_NAMERequiredAccount name for accessing Azure object storage.
AZURE_ACCOUNT_KEYRequiredAccount key for accessing Azure object storage.
BUCKET_NAME_OBJECTRequiredBucket ID for storing object data.
BUCKET_POLICY_UPLOADRequiredAccess policy name for write operations.
BUCKET_POLICY_DOWNLOADRequiredAccess policy name for read operations.
UPLOAD_PARTSIZERequiredByte size of each upload chunk. Adjust for performance.
DOWNLOAD_PARTSIZERequiredByte size of each download chunk. Adjust for performance.
OBJECT_SENTINELRequiredDefault sample object/file name for ping operations.

Access Policy Configuration for Azure

For Azure storage, you must define a storage access policy for your container.

  1. Access the Azure dashboard: Select containers from the left-hand menu.

  2. Locate your container: Choose Access Policy from the dropdown menu.

    azure-dash

  3. Create write and readonly access policies:

    azure-policies

Azure storage access policies

For more information on Azure storage access policies, visit the official Azure storage services documentation.