S3¶
Allows data to be pulled from S3 Objects.
-
s3
()
Methods of S3¶
-
get_object_metadata
(bucket_name, key)¶ Get the metadata associated with the given
bucket_name
andkey
. The metadata allows you to check for the existance of the key within the bucket and to check how large the object is without reading the whole object into memory.Parameters: - bucket_name – the name of the S3 Bucket
- key – the key that identifies the S3 Object within the S3 Bucket
Returns: an
S3ObjectMetadata
object-
class
S3ObjectMetadata
¶ -
exists
()¶ Will return True if the object exists.
-
size
()¶ Returns the size in bytes for the object. Will return -1 for objects that do not exist.
-
Example usage:
s3().get_object_metadata('my bucket', 'mykeypart1/mykeypart2').exists() s3().get_object_metadata('my bucket', 'mykeypart1/mykeypart2').size()
-
get_object
(bucket_name, key)¶ Get the S3 Object associated with the given
bucket_name
andkey
. This method will cause the object to be read into memory.Parameters: - bucket_name – the name of the S3 Bucket
- key – the key that identifies the S3 Object within the S3 Bucket
Returns: an
S3Object
object-
class
S3Object
¶ -
text
()¶ Get the S3 Object data
-
json
()¶ If the object exists, parse the object as JSON.
Returns: a dict containing the parsed JSON or None if the object does not exist.
-
exists
()¶ Will return True if the object exists.
-
size
()¶ Returns the size in bytes for the object. Will return -1 for objects that do not exist.
-
Example usage:
s3().get_object('my bucket', 'mykeypart1/my_text_doc.txt').text() s3().get_object('my bucket', 'mykeypart1/my_json_doc.json').json()
-
list_bucket
(bucket_name, prefix, max_items=100, recursive=True)¶ List the S3 Object associated with the given
bucket_name
, matchingprefix
. By default, listing is possible for up to 1000 keys, so we use pagination internally to overcome this.Parameters: - bucket_name – the name of the S3 Bucket
- prefix – the prefix to search under
- max_items – the maximum number of objects to list. Defaults to 100.
- recursive – if the listing should contain deeply nested keys. Defaults to True.
Returns: an
S3FileList
object-
class
S3FileList
¶ -
files
()¶ Returns a list of dicts like
{ "file_name": "foo", "size": 12345, "last_modified": "2017-07-17T01:01:21Z" }
-
Example usage:
s3().list_bucket('my bucket', 'some_prefix').files() files = s3().list_bucket('my bucket', 'some_prefix', 10000).files() # for listing a lot of keys last_modified = files[0]["last_modified"].isoformat() # returns a string that can be passed to time() age = time() - time(last_modified)