Bases: exceptions.Exception
Bases: exceptions.Exception
Bases: eventlet.timeout.Timeout
Bases: eventlet.timeout.Timeout
Bases: eventlet.timeout.Timeout
Bases: exceptions.Exception
Bases: eventlet.timeout.Timeout
Max container list length of a get request for an account
Max object list length of a get request for a container
Max file size allowed for objects
Max number of metadata items
Max length of the name of a key for metadata
Max overall size of metadata
Max length of the value of a key for metadata
Max object name length
Helper function for checking if a string can be converted to a float.
Parameters: | string – string to be verified as a float |
---|---|
Returns: | True if the string can be converted to a float, False otherwise |
Check metadata sent in the request headers.
Parameters: |
|
---|---|
Raises HTTPBadRequest: | |
bad metadata |
Verify that the path to the device is a mount point and mounted. This allows us to fast fail on drives that have been unmounted because of issues, and also prevents us for accidently filling up the root partition.
Parameters: |
|
---|---|
Returns: | True if it is a valid mounted device, False otherwise |
Check to ensure that everything is alright about an object to be created.
Parameters: |
|
---|---|
Raises: |
|
Validate if a string is valid UTF-8.
Parameters: | string – string to be validated |
---|---|
Returns: | True if the string is valid utf-8, False otherwise |
Miscellaneous utility functions for use with Swift.
Bases: object
Cheesy version of the LoggerAdapter available in Python 3
A no-op logger for eventlet wsgi.
Given a devices path and a data directory, yield (path, device, partition) for all files in that directory
Parameters: |
|
---|
Get memcache connection pool from the environment (which had been previously set by the memcache middleware
Parameters: | env – wsgi environment dict |
---|---|
Returns: | swift.common.memcached.MemcacheRing from environment |
Compute an ETA. Now only if we could also have a progress bar...
Parameters: |
|
---|---|
Returns: | ETA as a tuple of (length of time, unit of time) where unit of time is one of (‘h’, ‘m’, ‘s’) |
Drop ‘buffer’ cache for the given range of the given file.
Parameters: |
|
---|
Sets the userid of the current process
Parameters: | user – User id to change privileges to |
---|
Pre-allocate disk space for a file file.
Parameters: |
|
---|
Get the current system logger using config settings.
Log config and defaults:
log_facility = LOG_LOCAL0
log_level = INFO
log_name = swift
Parameters: |
|
---|
Get parameters from an HTTP request ensuring proper handling UTF-8 encoding.
Parameters: |
|
---|---|
Returns: | HTTP request parameter value |
Get a nomralized length of time in the largest unit of time (hours, minutes, or seconds.)
Parameters: | time_amount – length of time in seconds |
---|---|
Returns: | A touple of (length of time, unit of time) where unit of time is one of (‘h’, ‘m’, ‘s’) |
Get the connonical hash for an account/container/object
Parameters: |
|
---|---|
Returns: | hash string |
Get a value from the wsgi environment
Parameters: |
|
---|---|
Returns: | the value from the environment |
Iterate over partitions accross all devices.
Parameters: |
|
---|---|
Returns: | Each iteration returns a tuple of (device, partition) |
Attempt to find the function in libc, otherwise return a no-op func.
Parameters: | func_name – name of the function to pull from libc. |
---|
Context manager that acquires a lock on the parent directory of the given file path. This will block until the lock can be acquired, or the timeout time has expired (whichever occurs first).
Parameters: |
|
---|
Context manager that acquires a lock on a directory. This will block until the lock can be acquired, or the timeout time has expired (whichever occurs first).
Parameters: |
|
---|
Ensures the path is a directory or makes it if not. Errors if the path exists but is a file or on permissions failure.
Parameters: | path – path to create |
---|
Format a timestamp (string or numeric) into a standardized xxxxxxxxxx.xxxxx format.
Parameters: | timestamp – unix timestamp |
---|---|
Returns: | normalized timestamp as a string |
Read config file and return config items as a dict
Parameters: |
|
---|---|
Returns: | dict of config items |
Attempt to fix^H^H^Hhide race conditions like empty object directories being removed by backend processes during uploads, by retrying.
Parameters: |
|
---|
Validate and split the given HTTP request path.
Examples:
['a'] = split_path('/a')
['a', None] = split_path('/a', 1, 2)
['a', 'c'] = split_path('/a/c', 1, 2)
['a', 'c', 'o/r'] = split_path('/a/c/o/r', 1, 3, True)
Parameters: |
|
---|---|
Returns: | list of segments with a length of maxsegs (non-existant segments will return as None) |
Get the storage directory
Parameters: |
|
---|---|
Returns: | Storage directory |
Remove any file in a given path that that was last modified before mtime.
Parameters: | path – path to remove file from |
---|---|
Mtime : | timestamp of oldest file to keep |
Get the machine’s ip addresses using ifconfig
Returns: | list of Strings of IPv4 ip addresses |
---|
Ensure that a pickle file gets written to disk. The file is first written to a tmp location, ensure it is synced to disk, then perform a move to its final location
Parameters: |
|
---|
Bases: object
Auth Middleware that uses the dev auth server.
Returns None if the request is authorized to continue or a standard WSGI response callable if not.
Returns a standard WSGI response callable with the status of 403 or 401 depending on whether the REMOTE_USER is set or not.
Get groups for the given token.
If memcache_client is set, token credentials will be cached appropriately.
With a cache miss, or no memcache_client, the configurated external authentication server will be queried for the group information.
Parameters: |
|
---|---|
Returns: | None if the token is invalid or a string containing a comma separated list of groups the authenticated user is a member of. The first group in the list is also considered a unique identifier for that user. |
Returns a WSGI filter app for use with paste.deploy.
Returns a cleaned ACL header value, validating that it meets the formatting requirements for standard Swift ACL strings.
The ACL format is:
[item[,item...]]
Each item can be a group name to give access to or a referrer designation to grant or deny based on the HTTP Referer header.
The referrer designation format is:
.r:[-]value
The .r can also be .ref, .referer, or .referrer; though it will be shortened to just .r for decreased character count usage.
The value can be * to specify any referrer host is allowed access, a specific host name like www.example.com, or if it has a leading period . or leading *. it is a domain name specification, like .example.com or *.example.com. The leading minus sign - indicates referrer hosts that should be denied access.
Referrer access is applied in the order they are specified. For example, .r:.example.com,.r:-thief.example.com would allow all hosts ending with .example.com except for the specific host thief.example.com.
Example valid ACLs:
.r:*
.r:*,.r:-.thief.com
.r:*,.r:.example.com,.r:-thief.example.com
.r:*,.r:-.thief.com,bobs_account,sues_account:sue
bobs_account,sues_account:sue
Example invalid ACLs:
.r:
.r:-
Also, .r designations aren’t allowed in headers whose names include the word ‘write’.
ACLs that are “messy” will be cleaned up. Examples:
Original | Cleaned |
bob, sue | bob,sue |
bob , sue | bob,sue |
bob,,,sue | bob,sue |
.referrer : * | .r:* |
.ref:*.example.com | .r:.example.com |
Parameters: |
|
---|---|
Returns: | The value, cleaned of extraneous formatting. |
Raises ValueError: | |
If the value does not meet the ACL formatting requirements; the error message will indicate why. |
Parses a standard Swift ACL string into a referrers list and groups list.
See clean_acl() for documentation of the standard Swift ACL format.
Parameters: | acl_string – The standard Swift ACL string to parse. |
---|---|
Returns: | A tuple of (referrers, groups) where referrers is a list of referrer designations (without the leading .r:) and groups is a list of groups to allow access. |
Returns True if the referrer should be allowed based on the referrer_acl list (as returned by parse_acl()).
See clean_acl() for documentation of the standard Swift ACL format.
Parameters: |
|
---|---|
Returns: | True if the referrer should be allowed; False if not. |
Cloud Files client library used internally
Bases: exceptions.Exception
Bases: object
Convenience class to make requests that will also retry the request
Wrapper for delete_container()
Wrapper for delete_object()
Wrapper for get_account()
Wrapper for get_container()
Wrapper for get_object()
Wrapper for head_account()
Wrapper for head_container()
Wrapper for head_object()
Wrapper for post_account()
Wrapper for post_container()
Wrapper for post_object()
Wrapper for put_container()
Wrapper for put_object()
Delete a container
Parameters: |
|
---|---|
Raises ClientException: | |
HTTP DELETE request failed |
Delete object
Parameters: |
|
---|---|
Raises ClientException: | |
HTTP DELETE request failed |
Get a listing of containers for the account.
Parameters: |
|
---|---|
Returns: | a tuple of (response headers, a list of containers) The response headers will be a dict and all header names will be lowercase. |
Raises ClientException: | |
HTTP GET request failed |
Get authentication/authorization credentials.
The snet parameter is used for Rackspace’s ServiceNet internal network implementation. In this function, it simply adds snet- to the beginning of the host name for the returned storage URL. With Rackspace Cloud Files, use of this network path causes no bandwidth charges but requires the client to be running on Rackspace’s ServiceNet network.
Parameters: |
|
---|---|
Returns: | tuple of (storage URL, auth token) |
Raises ClientException: | |
HTTP GET request to auth URL failed |
Get a listing of objects for the container.
Parameters: |
|
---|---|
Returns: | a tuple of (response headers, a list of objects) The response headers will be a dict and all header names will be lowercase. |
Raises ClientException: | |
HTTP GET request failed |
Get an object
Parameters: |
|
---|---|
Returns: | a tuple of (response headers, the object’s contents) The response headers will be a dict and all header names will be lowercase. |
Raises ClientException: | |
HTTP GET request failed |
Get account stats.
Parameters: |
|
---|---|
Returns: | a dict containing the response’s headers (all header names will be lowercase) |
Raises ClientException: | |
HTTP HEAD request failed |
Get container stats.
Parameters: |
|
---|---|
Returns: | a dict containing the response’s headers (all header names will be lowercase) |
Raises ClientException: | |
HTTP HEAD request failed |
Get object info
Parameters: |
|
---|---|
Returns: | a dict containing the response’s headers (all header names will be lowercase) |
Raises ClientException: | |
HTTP HEAD request failed |
Make an HTTPConnection or HTTPSConnection
Parameters: | url – url to connect to |
---|---|
Returns: | tuple of (parsed url, connection object) |
Raises ClientException: | |
Unable to handle protocol scheme |
Update an account’s metadata.
Parameters: |
|
---|---|
Raises ClientException: | |
HTTP POST request failed |
Update a container’s metadata.
Parameters: |
|
---|---|
Raises ClientException: | |
HTTP POST request failed |
Update object metadata
Parameters: |
|
---|---|
Raises ClientException: | |
HTTP POST request failed |
Create a container
Parameters: |
|
---|---|
Raises ClientException: | |
HTTP PUT request failed |
Put an object
Parameters: |
|
---|---|
Returns: | etag from server response |
Raises ClientException: | |
HTTP PUT request failed |
Patched version of urllib.quote that encodes utf8 strings before quoting
Internal client library for making calls directly to the servers rather than through the proxy.
Delete object directly from the object server.
Parameters: |
|
---|---|
Returns: | response from server |
Get container listings directly from the container server.
Parameters: |
|
---|---|
Returns: | a tuple of (response headers, a list of objects) The response headers will be a dict and all header names will be lowercase. |
Get object directly from the object server.
Parameters: |
|
---|---|
Returns: | a tuple of (response headers, the object’s contents) The response headers will be a dict and all header names will be lowercase. |
Request container information directly from the container server.
Parameters: |
|
---|---|
Returns: | a dict containing the response’s headers (all header names will be lowercase) |
Request object information directly from the object server.
Parameters: |
|
---|---|
Returns: | a dict containing the response’s headers (all header names will be lowercase) |
Put object directly from the object server.
Parameters: |
|
---|---|
Returns: | etag from the server response |
Helper function to retry a given function a number of times.
Parameters: |
|
---|---|
Returns: | restult of func |
Monkey Patch httplib.HTTPResponse to buffer reads of headers. This can improve performance when making large numbers of small HTTP requests. This module also provides helper functions to make HTTP connections using BufferedHTTPResponse.
Warning
If you use this, be sure that the libraries you are using do not access the socket directly (xmlrpclib, I’m looking at you :/), and instead make all calls through httplib.
Bases: httplib.HTTPConnection
HTTPConnection class that uses BufferedHTTPResponse
alias of BufferedHTTPResponse
Bases: httplib.HTTPResponse
HTTPResponse class that buffers reading of headers
Helper function to create an HTTPConnection object. If ssl is set True, HTTPSConnection will be used. However, if ssl=False, BufferedHTTPConnection will be used, which is buffered for backend Swift services.
Parameters: |
|
---|---|
Returns: | HTTPConnection object |
Helper function to create an HTTPConnection object. If ssl is set True, HTTPSConnection will be used. However, if ssl=False, BufferedHTTPConnection will be used, which is buffered for backend Swift services.
Parameters: |
|
---|---|
Returns: | HTTPConnection object |
Lucid comes with memcached: v1.4.2. Protocol documentation for that version is at:
http://github.com/memcached/memcached/blob/1.4.2/doc/protocol.txt
Bases: object
Simple, consistent-hashed memcache client.
Decrements a key which has a numeric value by delta. If the key can’t be found, it’s added as 0. Memcached will treat data values below 0 as 0 with incr/decr.
Parameters: |
|
---|
Deletes a key/value pair from memcache.
Parameters: | key – key to be deleted |
---|
Gets the object specified by key. It will also unpickle the object before returning if it is pickled in memcache.
Parameters: | key – key |
---|---|
Returns: | value of the key in memcache |
Gets multiple values from memcache for the given keys.
Parameters: |
|
---|---|
Returns: | list of values |
Increments a key which has a numeric value by delta. If the key can’t be found, it’s added as delta.
Parameters: |
|
---|
Set a key/value pair in memcache
Parameters: |
|
---|
Sets multiple key/value pairs in memcache.
Parameters: |
|
---|
Bases: object
Rate limiting middleware
Rate limits requests on both an Account and Container level. Limits are configurable.
Returns number of requests allowed per second for given container size.
Returns a list of key (used in memcache), ratelimit tuples. Keys should be checked in order.
Parameters: |
|
---|
Performs rate limiting and account white/black listing. Sleeps if necessary.
Parameters: |
|
---|
paste.deploy app factory for creating WSGI proxy apps.