AWS S3 Transfer Manager

A high performance Amazon S3 transfer manager, a high-level transfer utility, for Rust. It provides a simple API to allow you to transfer files and directories between your application and Amazon S3.

:warning: Developer Preview

This library is currently in developer preview and is NOT recommended for production environments.

It is meant for early access and feedback purposes at this time. We'd love to hear from you on use cases, feature prioritization, and API feedback.

See the AWS SDK and Tools maintenance policy descriptions for more information.

Reusable runtime components

The runtime component crates share this repository and separate hardware topology, managed execution, and HTTP transport into independently versioned packages.

Development

Modeled API tooling

Developer-only model tooling is documented in s3-tm-model-codegen. Run just --list for available shorthand commands. Normal Cargo builds do not run this tooling.

Run all tests

cargo test --all-features

Run individual test

cargo test --lib download::worker::tests::test_distribute_work

Examples

NOTE: You can use the profiling profile from .cargo/config.toml to enable release with debug info for any example.

Copy

See all options:

cargo run --example cp -- -h

Download a file from S3

AWS_PROFILE=<profile-name> RUST_LOG=trace cargo run --example cp s3://<my-bucket>/<my-key> /local/path/<filename>

NOTE: To run in release mode add --release/-r to the command, see cargo run -h . NOTE: trace may be too verbose, you can see just this library's logs with RUST_LOG=aws_sdk_s3_transfer_manager=trace

Upload a file to S3

AWS_PROFILE=<profile-name> RUST_LOG=trace cargo run --example cp /local/path/<filename> s3://<my-bucket>/<my-key>

NOTE: To run in release mode add --release/-r to the command, see cargo run -h . NOTE: trace may be too verbose, you can see just this library's logs with RUST_LOG=aws_sdk_s3_transfer_manager=trace

Runtime diagnostics

AWS_S3_TM_DIAGNOSTICS accepts comma-separated, case-sensitive key=value settings. Memory snapshots use the aws_sdk_s3_transfer_manager::memory tracing target:

AWS_S3_TM_DIAGNOSTICS=memory.snapshot=1000ms,memory.detail=1 \
RUST_LOG=aws_sdk_s3_transfer_manager::memory=debug \
cargo run --release --example cp -- ...

memory.snapshot is off by default and otherwise accepts a positive integer number of milliseconds, such as 1000ms; intervals below 100ms use 100ms. memory.detail=0 is the default low-frequency collection level. memory.detail=1 additionally updates optimistic allocator counters on every acquisition and can add measurable overhead. Snapshot cadence and detail level are independent.

transfer.detail controls diagnostics for individual upload and download state machines. Level 0 is the default and collects nothing. Level 1 collects aggregate per-transfer summaries; level 2 also emits individual state-machine events. Uploads and downloads provide summaries and events. Transfer records use the aws_sdk_s3_transfer_manager::transfer tracing target:

AWS_S3_TM_DIAGNOSTICS=transfer.detail=1 \
RUST_LOG=aws_sdk_s3_transfer_manager::transfer=debug \
cargo run --release --example cp -- ...

Flamegraphs

See cargo-flamegraph for more prerequisites and installation information.

Generate a flamegraph (default is to output to flamegraph.svg ):

sudo AWS_PROFILE=<profile-name> RUST_LOG=aws_sdk_s3_transfer_manager=info cargo flamegraph --profile profiling --example cp -- s3://test-sdk-rust-aaron/mb-128.dat /tmp/mb-128.dat

Using tokio-console

By default examples use tracing crate for logs. You can pass the --tokio-console flag to examples to use console-subscriber instead. This allows you to run them with tokio-console to help debug task execution.

NOTE: This requires you build the examples with RUSTFLAGS="--cfg tokio_unstable" or setting the equivalent in your cargo config.

RUSTFLAGS="--cfg tokio_unstable" AWS_PROFILE=<profile-name> RUST_LOG=debug cargo run --example cp --tokio-console ...

Follow installation instructions for tokio-console and then run the example with tokio-console running.

End to end tests

End-to-end tests are available in tests/e2e_transfer_test.rs to validate functionality against a real S3 server. To run these tests:

1. Enable end-to-end testing by setting RUSTFLAGS="--cfg e2e_test"
2. Set up a test bucket by following the aws-c-s3 test helper instructions here

Sample commands:

# Install required Python dependency
pip3 install boto3

# Configure test environment
export S3_TEST_BUCKET_NAME_RS=<your-bucket-name> # e2e tests takes bucket name from environment variable
export AWS_REGION=us-west-2  # Note: Currently only us-west-2 is supported

# Initialize test buckets
python3 path/to/test_helper.py init <your-bucket-name>

# Run the end-to-end tests
RUSTFLAGS="--cfg e2e_test" cargo test --all-features --test e2e_transfer_test

Security

See CONTRIBUTING for more information.

License

This project is licensed under the Apache-2.0 License.