uproot.S3Source

Defined in uproot.source.s3 on line 16.

Inheritance order:
uproot.HTTPSource uproot.source.chunk.Source

class uproot.source.s3.S3Source(file_path: str, endpoint='s3.amazonaws.com', access_key=None, secret_key=None, session_token=None, secure=True, region=None, http_client=None, credentials=None, **options)

Parameters:

file_path (str) – A URL of the file to open.
endpoint – S3 endpoint (defaults to AWS)
access_key – Access key of your S3 account
secret_key – Secret key of your S3 account
session_token – Session token of your S3 account
secure – Flag to enable use of TLS
http_client (urllib3.poolmanager.PoolManager) – Instance of urllib3.poolmanager.PoolManager
credentials (minio.credentials.Provider) – Instance of minio.credentials.Provider
options – See uproot.source.http.HTTPSource.__init__

executor

Inherited from uproot.HTTPSource.

S3Source.executor: The uproot.source.futures.ResourceThreadPoolExecutor that manages this source’s single background thread.

timeout

Inherited from uproot.HTTPSource.

S3Source.timeout: The timeout in seconds or None.

parsed_url

Inherited from uproot.HTTPSource.

S3Source.parsed_url: A urllib.parse.ParseResult version of the file_path.

auth_headers

Inherited from uproot.HTTPSource.

S3Source.auth_headers: Dict containing auth headers, if any

fallback

Inherited from uproot.HTTPSource.

S3Source.fallback

If None, the source has not encountered an unsuccessful multipart GET and no fallback is needed yet.

Otherwise, this is a uproot.MultithreadedHTTPSource to which all requests are forwarded.

chunk

Inherited from uproot.source.chunk.Source.

S3Source.chunk(start: int, stop: int) → Chunk

Parameters:

start (int) – Seek position of the first byte to include.
stop (int) – Seek position of the first byte to exclude (one greater than the last byte to include).

Request a byte range of data from the file as a uproot.source.chunk.Chunk.

chunks

Inherited from uproot.source.chunk.Source.

S3Source.chunks(ranges: list[int, int], notifications: queue.Queue) → list[uproot.source.chunk.Chunk]

Parameters:

ranges (list of (int, int) 2-tuples) – Intervals to fetch as (start, stop) pairs in a single request, if possible.
notifications (queue.Queue) – Indicator of completed chunks. After each gets filled, it is put on the queue; a listener should get from this queue len(ranges) times.

Request a set of byte ranges from the file.

This method has two outputs:

The method returns a list of unfilled uproot.source.chunk.Chunk objects, which get filled in a background thread. If you try to read data from an unfilled chunk, it will wait until it is filled.
The method also puts the same uproot.source.chunk.Chunk objects onto the notifications queue as soon as they are filled.

Reading data from chunks on the queue can be more efficient than reading them from the returned list. The total reading time is the same, but work on the filled chunks can be better parallelized if it is triggered by already-filled chunks, rather than waiting for chunks to be filled.