How to Extract S3 Bucket Name and Object Path from URLs Using Python: Regex & Solutions

Amazon S3 (Simple Storage Service) is a cornerstone of cloud storage, widely used for hosting assets, backups, and data lakes. When working with S3, you’ll often encounter URLs pointing to buckets or objects (e.g., s3://my-bucket/path/to/file.txt or https://my-bucket.s3.us-west-2.amazonaws.com/docs/report.pdf). Extracting the bucket name and object path from these URLs is critical for automation, log analysis, data pipelines, and integrations with tools like AWS SDKs (boto3).

However, S3 URLs come in multiple formats (virtual-hosted, path-style, access points, etc.), making parsing non-trivial. In this blog, we’ll demystify S3 URL structures and explore Python-based solutions—using regex for flexibility and non-regex methods for simplicity—to extract bucket names and object paths reliably.

Table of Contents#

Understanding S3 URL Formats#

Before diving into parsing, let’s clarify the common S3 URL formats you’ll encounter. AWS S3 supports multiple URL styles, and the location of the bucket name and object path varies between them:

URL StyleExampleBucket LocationObject Path Location
S3 Protocol (s3://)s3://my-bucket/path/to/object.txtAfter s3:// (e.g., my-bucket)After the bucket (e.g., path/to/object.txt)
Virtual-Hosted Stylehttps://my-bucket.s3.amazonaws.com/path/to/object.txtSubdomain (e.g., my-bucket)After the domain (e.g., path/to/object.txt)
Virtual-Hosted with Regionhttps://my-bucket.s3.us-west-2.amazonaws.com/path/to/object.txtSubdomain (e.g., my-bucket)After the domain (e.g., path/to/object.txt)
Path-Stylehttps://s3.us-west-2.amazonaws.com/my-bucket/path/to/object.txtAfter the domain (e.g., my-bucket)After the bucket (e.g., path/to/object.txt)
S3 Access Pointhttps://my-access-point.s3-accesspoint.us-west-2.amazonaws.com/object.txtSubdomain (e.g., my-access-point)After the domain (e.g., object.txt)

Regex Solutions for S3 URL Parsing#

Regular expressions (regex) are powerful for pattern matching and are ideal for extracting structured data like bucket names and object paths from URLs. Below, we’ll break down regex patterns for each S3 URL style.

1. S3 Protocol URLs (s3://)#

The s3:// protocol is the simplest format (e.g., s3://my-bucket/folder/file.csv). The bucket name follows s3://, and the object path follows the bucket.

Regex Pattern:

^s3://([^/]+)/(.*)$
  • ^s3://: Matches the start of the string followed by s3://.
  • ([^/]+): Captures the bucket name (all characters except /, one or more times).
  • /(.*): Matches a / followed by the object path (any character, zero or more times).

Python Example:

import re  
 
url = "s3://my-bucket/path/to/object.txt"  
pattern = r"^s3://([^/]+)/(.*)$"  
match = re.match(pattern, url)  
 
if match:  
    bucket = match.group(1)  # "my-bucket"  
    object_path = match.group(2)  # "path/to/object.txt"  
    print(f"Bucket: {bucket}, Object Path: {object_path}")

2. Virtual-Hosted Style URLs#

Virtual-hosted URLs embed the bucket name as a subdomain (e.g., https://my-bucket.s3.amazonaws.com/data.csv). They may include a region (e.g., s3.us-west-2.amazonaws.com).

Regex Pattern (Regionless):

^https?://([^.]+)\.s3\.amazonaws\.com/(.*)$
  • ^https?://: Matches http:// or https://.
  • ([^.]+): Captures the bucket name (all characters except ., one or more times).
  • \.s3\.amazonaws\.com/: Matches the S3 domain suffix.
  • (.*): Captures the object path.

Regex Pattern (With Region):
For region-specific URLs (e.g., my-bucket.s3.us-west-2.amazonaws.com):

^https?://([^.]+)\.s3\.[^.]+\.amazonaws\.com/(.*)$
  • s3\.[^.]+: Matches s3. followed by a region (e.g., us-west-2).

Python Example:

url = "https://my-bucket.s3.us-west-2.amazonaws.com/docs/report.pdf"  
pattern = r"^https?://([^.]+)\.s3\.[^.]+\.amazonaws\.com/(.*)$"  
match = re.match(pattern, url)  
 
if match:  
    bucket = match.group(1)  # "my-bucket"  
    object_path = match.group(2)  # "docs/report.pdf"

3. Path-Style URLs#

Path-style URLs place the bucket name after the domain (e.g., https://s3.us-west-2.amazonaws.com/my-bucket/images/photo.jpg).

Regex Pattern:

^https?://s3\.[^.]+\.amazonaws\.com/([^/]+)/(.*)$
  • s3\.[^.]+\.amazonaws\.com/: Matches the S3 domain with region (e.g., s3.us-west-2.amazonaws.com/).
  • ([^/]+): Captures the bucket name (after the domain, before the next /).
  • /(.*): Captures the object path.

Python Example:

url = "https://s3.us-west-2.amazonaws.com/my-bucket/images/photo.jpg"  
pattern = r"^https?://s3\.[^.]+\.amazonaws\.com/([^/]+)/(.*)$"  
match = re.match(pattern, url)  
 
if match:  
    bucket = match.group(1)  # "my-bucket"  
    object_path = match.group(2)  # "images/photo.jpg"

4. S3 Access Point URLs#

Access point URLs (e.g., https://my-access-point.s3-accesspoint.us-west-2.amazonaws.com/logs.txt) use s3-accesspoint in the domain.

Regex Pattern:

^https?://([^.]+)\.s3-accesspoint\.[^.]+\.amazonaws\.com/(.*)$
  • s3-accesspoint\.[^.]+: Matches the access point domain suffix (e.g., s3-accesspoint.us-west-2).

Python Example:

url = "https://my-access-point.s3-accesspoint.us-west-2.amazonaws.com/logs.txt"  
pattern = r"^https?://([^.]+)\.s3-accesspoint\.[^.]+\.amazonaws\.com/(.*)$"  
match = re.match(pattern, url)  
 
if match:  
    access_point = match.group(1)  # "my-access-point"  
    object_path = match.group(2)  # "logs.txt"

Non-Regex Solutions#

Regex is flexible but can become complex for edge cases (e.g., bucket names with dots). Non-regex methods use Python’s built-in libraries to parse URLs structurally.

Using urllib.parse#

The urllib.parse module splits URLs into components (scheme, netloc, path, etc.). We can use this to extract buckets and object paths without regex.

Example 1: Parsing s3:// URLs#

For s3://my-bucket/folder/file.txt, the bucket is in netloc, and the object path is in path:

from urllib.parse import urlparse  
 
url = "s3://my-bucket/folder/file.txt"  
parsed = urlparse(url)  
 
bucket = parsed.netloc  # "my-bucket"  
object_path = parsed.path.lstrip("/")  # "folder/file.txt" (strip leading "/")

Example 2: Parsing Virtual-Hosted URLs#

For https://my-bucket.s3.us-west-2.amazonaws.com/data.csv, split netloc to isolate the bucket:

url = "https://my-bucket.s3.us-west-2.amazonaws.com/data.csv"  
parsed = urlparse(url)  
 
# Split netloc to extract bucket (before ".s3.")  
if ".s3." in parsed.netloc:  
    bucket = parsed.netloc.split(".s3.")[0]  # "my-bucket"  
    object_path = parsed.path.lstrip("/")  # "data.csv"

Example 3: Parsing Path-Style URLs#

For https://s3.us-west-2.amazonaws.com/my-bucket/logs/2023.txt, the bucket is the first segment after the domain:

url = "https://s3.us-west-2.amazonaws.com/my-bucket/logs/2023.txt"  
parsed = urlparse(url)  
 
# Split path into segments (e.g., "/my-bucket/logs/2023.txt" → ["", "my-bucket", "logs", "2023.txt"])  
path_segments = parsed.path.split("/")  
bucket = path_segments[1]  # "my-bucket"  
object_path = "/".join(path_segments[2:])  # "logs/2023.txt"

Using Boto3 Utilities (Advanced)#

AWS’s boto3 library (S3 SDK) has hidden utilities for parsing S3 URLs. The boto3.utils module includes extract_bucket_and_key (note: this is an internal function and may change):

from boto3.utils import extract_bucket_and_key  
 
url = "s3://my-bucket/path/to/object.txt"  
bucket, object_path = extract_bucket_and_key(url)  
print(bucket, object_path)  # "my-bucket", "path/to/object.txt"

⚠️ Warning: Internal functions like extract_bucket_and_key are not part of the public API and may break in future boto3 versions. Use with caution.

Handling Edge Cases#

Real-world S3 URLs often include edge cases. Here’s how to handle them:

Bucket Names with Dots#

Bucket names can contain dots (e.g., my.bucket), which breaks simple regex patterns. For virtual-hosted URLs, use split(".s3.") to isolate the bucket:

url = "https://my.bucket.s3.amazonaws.com/file.txt"  
parsed = urlparse(url)  
bucket = parsed.netloc.split(".s3.")[0]  # "my.bucket" (correct!)

URLs with Query Parameters or Fragments#

URLs may include query parameters (e.g., ?versionId=123) or fragments (e.g., #part1). Use urlparse to ignore these:

url = "s3://my-bucket/file.txt?versionId=abc123#part1"  
parsed = urlparse(url)  
bucket = parsed.netloc  # "my-bucket"  
object_path = parsed.path.lstrip("/").split("?")[0]  # "file.txt" (ignore query params)

Missing Object Paths#

If a URL points to a bucket (not an object), the object path should be empty:

url = "s3://my-bucket/"  
parsed = urlparse(url)  
object_path = parsed.path.lstrip("/")  # "" (empty string)

Testing Your Implementation#

To ensure robustness, test with diverse URL formats:

def extract_s3_components(url):  
    from urllib.parse import urlparse  
 
    parsed = urlparse(url)  
    bucket = None  
    object_path = ""  
 
    # Handle s3:// protocol  
    if parsed.scheme == "s3":  
        bucket = parsed.netloc  
        object_path = parsed.path.lstrip("/")  
 
    # Handle HTTP/HTTPS (virtual-hosted or path-style)  
    elif parsed.scheme in ["http", "https"]:  
        if ".s3." in parsed.netloc:  
            # Virtual-hosted style  
            bucket = parsed.netloc.split(".s3.")[0]  
            object_path = parsed.path.lstrip("/").split("?")[0]  # Ignore queries  
        elif parsed.netloc.startswith("s3."):  
            # Path-style  
            path_segments = parsed.path.lstrip("/").split("/")  
            if len(path_segments) >= 1:  
                bucket = path_segments[0]  
                object_path = "/".join(path_segments[1:]) if len(path_segments) > 1 else ""  
 
    return bucket, object_path  
 
# Test cases  
test_urls = [  
    "s3://my-bucket/path/to/file.txt",  
    "https://my.bucket.s3.us-west-2.amazonaws.com/docs/report.pdf?versionId=123",  
    "https://s3.amazonaws.com/my-bucket/",  
    "https://access-point.s3-accesspoint.us-east-1.amazonaws.com/",  
]  
 
for url in test_urls:  
    bucket, obj = extract_s3_components(url)  
    print(f"URL: {url}\nBucket: {bucket}, Object Path: {obj}\n")

Conclusion#

Extracting S3 bucket names and object paths requires handling diverse URL formats. Regex is powerful for simple cases, but non-regex methods (e.g., urllib.parse) are more robust for edge cases like bucket names with dots or query parameters. For most use cases, combine urllib.parse with structural checks (e.g., split(".s3.")) to balance simplicity and reliability.

References#

2026-02