AWS Transfer for SFTP

AWS has recently launched AWS Transfer for SFTP enabling you to easily move your file transfer workloads that use the Secure Shell File Transfer Protocol (SFTP) to AWS without needing to modify your applications or manage any SFTP servers. Traditionally an EC2 with decent storage would have to be configured, regularly updated and maintained or an EC2 Storage gateway implemented to accommodate SFTP transfers. This guide shows you how to create an SFTP server and map your domain to the server endpoint, select authentication for your SFTP clients using service-managed identities (or alternatively you can integrate your own identity provider) and select your Amazon S3 buckets to store the transferred data.

With AWS SFTP, you only pay for the use of the SFTP server endpoint, data uploaded and downloaded.

Your existing users can continue to operate with their existing SFTP clients or applications. Data uploaded or downloaded using SFTP is available in your Amazon S3 bucket, and can be used for archiving or processing in AWS.

You pay a per-hour fee for each running server and a per-GB data upload and download fee.
For full details of pricing see https://aws.amazon.com/sftp/pricing/

How it fits together

Source: AWS - https://aws.amazon.com/sftp/

Create S3 Bucket and an IAM Policy

For the purposes of this example i have created the following;

An S3 bucket with default settings called sftp.test.infra.enginner (Bucket and Objects not public)
An IAM policy called sftp.test.infra.engineer.iam.policy with the following JSON;

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::sftp.test.infra.engineer"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:DeleteObject"
            ],
            "Resource": [
                "arn:aws:s3:::sftp.test.infra.engineer/*"
            ]
        }
    ]
}

Create an IAM Role and edit trust relationship

Create an IAM S3 Role called SFTP.Service.IAM.Role using the policy created above called sftp.test.infra.engineer.iam.policy

Once this role has been created, navigate to the Trust relationships tab and select Edit trust relationship

Update the Statement[].Principal.Service value to transfer.amazonaws.com (Old value will likely have been s3.amazonaws.com) and click update trust policy. Failure to do this will produce certificate trust issues when trying to connect with your SFTP client.

The JSON of this file should look like;

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "transfer.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

Create Server

Within AWS Management Console, search for SFTP then select AWS Transfer for SFTP then create server

On the following screen you can accept the defaults but i prefer to use a Route53 alias along with tagging the server, in a production environment cloudwatch logging would be configured to monitor events such successful/failed login attempts, IP addresses and files uploaded/downloaded on each user account etc.

Once the server has been created and the Route53 alias has automatically been added (Providing your DNS sits with AWS - which it should) you will be at a screen similar to the following;

Clicking the server ID will take you down into the server where you can create your users (and home directorys if required), set the IAM role and S3 bucket created earlier

Create an SSH public key for the user using PuTTYgen or your preferred method

https://docs.aws.amazon.com/transfer/latest/userguide/sshkeygen.html

Generate a public/private key pair using the Generate button then once generated paste the public key highlighted in blue above into the field (It should start ssh-rsa), add any tags and click create

Save your PuTTYgen private key in ppk format to your device which will be used lated to connect via an SFTP client

Once completed you should see the user section similar to the following;

The AWS configuration has now been completed and now needs tested with an SFTP client, for the purposes of this guide I am using WinSCP but other SFTP capable clients can be used such as FileZilla or any of the software identified here

Configure WinSCP

Create a new site within WinSCP with your details similar to the following (No password is required as you are using your private key)

 

Go to Advanced > SSH > Authentication and browse to your ppk file created earlier

With the JSON used above to stop seeing all root s3 buckets, I would recommend turning off the last used directory as you may have to recreate the WinSCP profile or manually set the remote directory if you attempt to get in the root directory.

Tip: You may get the following error message when uploading files, this is down to the fact the S3 API does not allow you to set the timestamp value when you upload an object, therefore you have to turn off preserving timestamps;

The server does not support the operation.
Error code: 8
Error message from server (US-ASCII): SETSTAT unsupported

To do this in WinSCP go to Preferences > Transfer then edit the Default preset and untick preserve timestamp, once this setting is set it will allow you to upload files although please be aware this will not bring across the original timestamp.

Depending on your user case, once files are added Lambda could be configured to perform actions on these files such as moving them to other storage destinations or convert the files using other AWS tools such as Rekognition, Comprehend or one of the many AWS services available.

Delete the Server

To delete the SFTP server, delete all the users within the server then it will allow you to delete the server. You will also need to delete the S3 bucket to remove the data.