AWS S3
Connector Details
| Connector Attributes | Details |
|---|---|
| Name | AWS S3 |
| Description | The AWS S3 connector enables seamless integration with Amazon's Simple Storage Service (S3), providing efficient and scalable access to cloud storage for diverse data needs. With the AWS S3 connector, users can easily connect to their S3 buckets, facilitating the upload, download, and management of objects for streamlined workflows and enhanced productivity. Whether accessing data for analytics, backup, or application development, the connector simplifies handling large volumes of files, ensuring high availability and security. Its robust support for object versioning, metadata management, and secure access controls empowers organizations to optimize data storage, enhance collaboration, and drive informed decision-making across teams and applications. Supports CSV/JSON/Parquet files. |
| Connector Type | Class C |
Features
| Feature Name | Feature Details |
|---|---|
| Load Strategies | Full Load |
| Metadata Extraction | Supported |
| Data Acquisition | Supported |
| Data Publishing | Not Supported |
| Automated Schema Drift Handling | Not Supported |
Source Connection Attributes
| Connection Parameters | Data Type | Example | |
|---|---|---|---|
| Connection Name | String | AWS S3 | |
| Access Key | String | ||
| Secret Key | String | ||
| Bucket | String | ||
| Silver Schema (Optional) | String | ||
| Bronze Schema (Optional) | String |
Connector Specific Configuration Details
The connector demands a few mandatory options: Access Key, Secret Key, and Bucket.
Let's review the free service to generate these connection properties for our testing.
Access Key: AKIAXVFJUZT6ZR44TKW5Secret Key: a/b+liz_TestSecretKey_BeCEsgG5Bucket: databricks-connection
How to set up your AWS S3 bucket as an example, official documentation:
Mandatory conditions for the S3 static CSV files:
CSV files:File Format
The connector is designed to handle files only in the CSV/JSON` format. Ensure all files used or processed by the system are in .csv format for compatibility with the connector.
Use of Separator
Always use a comma , as the separator in your data files. This ensures consistency and compatibility across different systems and processes.
Encoding Standard
Ensure that all files are saved and processed using theUTF-8 encoding. This is crucial to maintain character integrity, especially when working with international data or special characters.
File Movement or Renaming
When you move or rename a file, it is essential to update the associated metadata. This ensures that all references to the file are accurate and that no processes relying on this data are disrupted.
Replacing a File
If you are replacing a file but not dropping and creating a new one, you can continue to use the same table name. This allows for a seamless transition and minimal disruption to the existing data structure.
Dropping and Creating a New File
When you drop and create a new file with the same name, it is necessary to update the metadata. This ensures that the system properly references and recognizes the new file, preventing potential issues with outdated information.
Mandatory conditions for the S3 static JSON files:
JSON files:Encoding Standard
Ensure that all files are saved and processed using theUTF-8 encoding. This is crucial to maintain character integrity, especially when working with international data or special characters.
File Movement or Renaming
When you move or rename a file, it is essential to update the associated metadata. This ensures that all references to the file are accurate and that no processes relying on this data are disrupted.
Replacing a File
If you are replacing a file but not dropping and creating a new one, you can continue to use the same table name. This allows for a seamless transition and minimal disruption to the existing data structure.
Dropping and Creating a New File
When you drop and create a new file with the same name, the metadata must be updated. This ensures the system properly references and recognizes the new file, preventing potential issues with outdated information.
Mandatory conditions for the S3 static Parquet files:
Parquet files:This type of file does not have any special conditions.
Screenshot To Use Connector
Updated 2 days ago