How to Perform a HEAD Request on an S3 Key to Retrieve User Metadata Using Boto2 in Python

Table of Contents#

1. Prerequisites
2. Understanding HEAD Requests in S3
3. Setting Up Boto2
4. Performing a HEAD Request with Boto2
5. Retrieving User Metadata
6. Handling Exceptions
7. Best Practices
8. Conclusion
9. References

Prerequisites#

Before getting started, ensure you have the following:

• AWS Account: You need an AWS account with access to S3.
• S3 Bucket and Object: An existing S3 bucket containing an object with user-defined metadata (we’ll cover how to verify this later).
• AWS Credentials: Access keys (Access Key ID and Secret Access Key) with permissions to read the S3 bucket/object.
• Python: Python 2.7 or 3.x installed (Boto2 supports both, but Python 3.x is recommended).
• Boto2: The Boto2 library installed (we’ll cover installation below).

Understanding HEAD Requests in S3#

What is a HEAD Request?#

A HEAD request is identical to a GET request but returns only the HTTP headers, not the object’s content. For S3, this includes:

• System metadata: Automatically generated by S3 (e.g., Content-Length, Last-Modified, ETag).
• User-defined metadata: Custom key-value pairs added by users when uploading the object (e.g., author: "John Doe", version: "1.0").

User Metadata in S3#

User-defined metadata must follow S3 conventions:

• Keys must be prefixed with x-amz-meta- (e.g., x-amz-meta-author).
• Values must be UTF-8 encoded and under 2 KB in total (all user metadata combined).

Boto2 simplifies access to user metadata by stripping the x-amz-meta- prefix, so x-amz-meta-author becomes author in the metadata dictionary.

Setting Up Boto2#

Install Boto2#

Boto2 is not the latest AWS SDK (Boto3 is), but it is still used in legacy systems. Install it via pip:

pip install boto==2.49.0  # Latest stable Boto2 version  

Configure AWS Credentials#

Boto2 requires AWS credentials to authenticate with S3. You can configure credentials in one of three ways (ordered by priority):

1. Environment Variables#

Set AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY:

export AWS_ACCESS_KEY_ID="your-access-key-id"  
export AWS_SECRET_ACCESS_KEY="your-secret-access-key"  

2. Boto Config File#

Create a ~/.boto (Linux/macOS) or C:\Users\<User>\.boto (Windows) file:

[Credentials]  
aws_access_key_id = your-access-key-id  
aws_secret_access_key = your-secret-access-key  

3. IAM Roles (for EC2/ECS)#

If running on AWS infrastructure (e.g., EC2), assign an IAM role with S3 read permissions to the instance. Boto2 will automatically fetch credentials from the instance metadata service.

Performing a HEAD Request with Boto2#

To perform a HEAD request in Boto2, follow these steps:

Step 1: Connect to S3#

Use boto.s3.connection.S3Connection to establish a connection to S3.

import boto  
from boto.s3.connection import S3Connection  
 
# Connect to S3 (credentials are auto-loaded from environment/config)  
conn = S3Connection()  

Step 2: Get the S3 Bucket#

Retrieve the bucket containing your object using get_bucket():

bucket_name = "your-bucket-name"  
bucket = conn.get_bucket(bucket_name)  

Step 3: Get the S3 Key (Object)#

Fetch the key (object) using get_key():

object_key = "path/to/your/object.txt"  # e.g., "documents/report.pdf"  
key = bucket.get_key(object_key)  
 
if not key:  
    raise ValueError(f"Object '{object_key}' not found in bucket '{bucket_name}'.")  

Step 4: Execute the HEAD Request#

Use the head() method on the Key object to send a HEAD request. This populates the key’s metadata:

key.head()  # Sends HEAD request and loads metadata  

Retrieving User Metadata#

After calling key.head(), user-defined metadata is stored in key.metadata (a Python dictionary). Boto2 strips the x-amz-meta- prefix from keys, making them easier to access.

Example: Access User Metadata#

Suppose you uploaded an object with user metadata x-amz-meta-author: "John Doe" and x-amz-meta-version: "1.0". Here’s how to retrieve it:

# After key.head()  
user_metadata = key.metadata  
 
print("User Metadata:")  
print(f"Author: {user_metadata.get('author')}")       # Output: John Doe  
print(f"Version: {user_metadata.get('version')}")     # Output: 1.0  

Access System Metadata#

System metadata is available directly as attributes of the Key object:

print("\nSystem Metadata:")  
print(f"File Size: {key.size} bytes")                # e.g., 1024  
print(f"Last Modified: {key.last_modified}")         # e.g., 2024-05-20T12:34:56.000Z  
print(f"ETag: {key.etag}")                           # e.g., "a1b2c3d4..."  

Handling Exceptions#

S3 operations can fail for reasons like missing objects, permission issues, or network errors. Boto2 raises exceptions for these cases; handle them with try-except blocks.

Common Exceptions#

• NoSuchBucket: The bucket does not exist.
• NoSuchKey: The object does not exist in the bucket.
• AccessDenied: Insufficient permissions to access the bucket/object.
• S3ResponseError: Generic S3 error (check status and reason for details).

Example: Error Handling#

from boto.exception import S3ResponseError, NoSuchBucket, NoSuchKey  
 
try:  
    conn = S3Connection()  
    bucket = conn.get_bucket(bucket_name)  
    key = bucket.get_key(object_key)  
 
    if not key:  
        raise NoSuchKey(f"Object '{object_key}' not found.")  
 
    key.head()  # Perform HEAD request  
 
    # Access metadata  
    print("Author:", key.metadata.get("author"))  
 
except NoSuchBucket:  
    print(f"Error: Bucket '{bucket_name}' does not exist.")  
except NoSuchKey as e:  
    print(f"Error: {e}")  
except S3ResponseError as e:  
    print(f"S3 Error: Status {e.status}, Reason: {e.reason}")  
except Exception as e:  
    print(f"Unexpected error: {e}")  

Best Practices#

1. Use HEAD Instead of GET: For metadata-only retrieval, HEAD requests save bandwidth and cost (no object content transfer).
2. Limit Permissions: Use IAM policies to restrict credentials to only the necessary actions (e.g., s3:HeadObject).
3. Cache Metadata Sparingly: If metadata changes infrequently, cache results to reduce API calls.
4. Handle Edge Cases: Check for None when accessing metadata keys (e.g., user_metadata.get("author") instead of user_metadata["author"]).
5. Avoid Hardcoding Credentials: Use environment variables, config files, or IAM roles instead of hardcoding keys in code.

Conclusion#

Performing a HEAD request on an S3 key with Boto2 is a efficient way to retrieve metadata without downloading the object. By following the steps outlined—connecting to S3, fetching the key, executing head(), and accessing metadata—you can easily retrieve user-defined and system metadata. Remember to handle exceptions and follow best practices for secure, reliable code.

References#

• Boto2 S3 Documentation
• AWS S3 HEAD Object API
• Boto2 Installation Guide
• AWS User Metadata Documentation