Ryan Harrison My blog, portfolio and technology related ramblings

Ktor - File Upload and Download

The ability to perform file uploads and downloads is a staple part of any good web server framework. Ktor has support for both operations in just a few lines of code.

File Upload

File uploads are handled through multipart POST requests in standard HTTP - normally from form submissions where the file selector field would be just one item (another could be the title for example).

To handle this in Ktor, you can get hold of the multipart data through receiveMultipart and then loop over each part as required. In the below example, we are just interested in files (PartData.FileItem), although you could also look at the individual PartData.FormItem as well (which would be the other form fields in the submission).

A Ktor FileItem exposes an InputStream via streamProvider which can be used to access the raw bytes of the file which has been uploaded. As in the below example, you can then simply create the appropriate file and copy the bytes from one stream (input) to the other (output).

post("/upload") { _ ->
    // retrieve all multipart data (suspending)
    val multipart = call.receiveMultipart()
    multipart.forEachPart { part ->
        // if part is a file (could be form item)
        if(part is PartData.FileItem) {
            // retrieve file name of upload
            val name = part.originalFileName!!
            val file = File("/uploads/$name")

            // use InputStream from part to save file
            part.streamProvider().use { its ->
                // copy the stream to the file with buffering
                file.outputStream().buffered().use {
                    // note that this is blocking
                    its.copyTo(it)
                }
            }
        }
        // make sure to dispose of the part after use to prevent leaks
        part.dispose()
    }
}

The documentation also includes a suspending copyTo method which can be used to save the upload to a file in a non-blocking way.

File Download

File downloads are very straightforward in Ktor. You just have to create a handle to the specified File and use the respondFile method:

get("/{name}") {
    // get filename from request url
    val filename = call.parameters["name"]!!
    // construct reference to file
    // ideally this would use a different filename
    val file = File("/uploads/$filename")
    if(file.exists()) {
        call.respondFile(file)
    }
    else call.respond(HttpStatusCode.NotFound)
}

By default, if called from the browser, this will cause the file to be viewed inline. If you instead want to prompt the browser to download the file, you can include the Content-Disposition header:

call.response.header("Content-Disposition", "attachment; filename=\"${file.name}\"")

This is also helpful if you save the uploaded file with a different name (which is advisable), as you can override the filename with the original when it gets downloaded by users.

More information about the Content-Disposition header

Read More

Ubuntu Server Setup Part 6 - HTTPS With Let's Encrypt

HTTPS is a must have nowadays with sites served under plain HTTP being downgraded in search results by Google and marked as insecure by browsers. The process of obtaining an SSL certificate used to be cumbersome and expensive, but now thankfully because of Let’s Encrypt it completely free and you can automate the process with just a few commands.

This part assumes that you already have an active Nginx server running (as described in Part 4) and so will go over how to use Let’s Encrypt with Nginx. Certbot (the client software) has a number of plugins that make the process just as easy if you are running another web server such as Apache.

Prepare the Nginx Server

Make sure have a server block where server_name is set to your domain name (in this case example.com).

This is so Certbot knows which config file to modify in order to enable HTTPS (it adds a line pointing to the generated SSL certificates).

server {
  	listen 80;
  	listen [::]:80;
  
  	server_name example.com www.example.com;
  	root /var/www/example;
  
  	index index.html;
  	location / {
  		try_files $uri $uri/ =404;
  	}
  }

That’s all the preparation needed on the Nginx side. Certbot will handle everything else for us.

Install and Run Certbot

Certbot is the client software (written in Python), that is supported by Let’s Encrypt themselves to automate the whole process. There are a wide range of different alternatives in various languages if you have different needs.

You should install Certbot through the dedicated ppa to make sure you always get the latest updates. In this example we install the Nginx version (which includes the Nginx plugin):

sudo apt-get update
sudo add-apt-repository ppa:certbot/certbot
sudo apt-get update
sudo apt-get install -y python-certbot-nginx

Once installed, you can run Certbot. Here the --nginx flag is used to enable the Nginx plugin. Without this, Certbot would just generate a certificate and your web server wouldn’t know about it. The plugin is required to modify the Nginx configuration in order to see the certificate and enable HTTPS.

sudo certbot --nginx

It will ask for:

  • an email address (you will be emailed if there are any issues or your certs are about to expire)
  • agreeing to the Terms of Service
  • which domains to use HTTPS for (it detects the list using server_name lines in your Nginx config)
  • whether to redirect HTTP to HTTPS (recommended)

Once you have selected these options, Certbot will perform a ‘challenge’ to check that the server it is running on is in control of the domain name. As described in the ACME protocol which is what underlies Let’s Encrypt, there are a number of different challenge types. In this case tls-sni was probably performed, although DNS might be used for wildcard certificates.

If the process completed without errors, a new certificate should have been generated and saved on the server. You can access this via /var/letsencrypt/live/domain.

The Nginx server block should have also been modified to include a number of extra ssl related fields. You will notice that these point to the generated certificate alongside the Let’s Encrypt chain cert. If you checked the option to redirect all HTTP traffic to HTTPS, you should also see another server block generated which merely captures all HTTP traffic and performs a redirection to the main HTTPS enabled server block.

You could stop here if all you want is HTTPS as this already gives you an A rating and maintains itself.

Test your site with SSL Labs using https://www.ssllabs.com/ssltest/analyze.html?d=www.YOUR-DOMAIN.com

Renewal

There is nothing to do, Certbot installed a cron task to automatically renew certificates about to expire.

You can check renewal works using:

sudo certbot renew --dry-run

You can also check what certificates exist using:

sudo certbot certificates

A+ Test

If you did the SSL check in the previous section you might be wondering why you didn’t get an A+ instead of just an A. It turns out that the default policy is to use some slightly outdated protocols and cipher types to maintain backwards compatibility with older devices. If you want to get the A+ rating, add the below config to your Nginx server block (the same one that got updated by Certbot). In particular we only use TLS 1.2 (not 1.0 or 1.1) and the available ciphers are restricted to only the latest and most secure. Be aware though that this might mean you site is unusable on some older devices which do not support these modern ciphers.

ssl_trusted_certificate /etc/letsencrypt/live/YOUR-DOMAIN/chain.pem;

ssl_session_cache shared:le_nginx_SSL:1m;
ssl_session_timeout 1d;
ssl_session_tickets off;

ssl_protocols TLSv1.2;
ssl_prefer_server_ciphers on;
ssl_ciphers "EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH";
ssl_ecdh_curve secp384r1;

ssl_stapling on;
ssl_stapling_verify on;

add_header Strict-Transport-Security "max-age=15768000; includeSubdomains; preload;";
add_header Content-Security-Policy "default-src 'none'; frame-ancestors 'none'; script-src 'self'; img-src 'self'; style-src 'self'; base-uri 'self'; form-action 'self';";
add_header Referrer-Policy "no-referrer, strict-origin-when-cross-origin";
add_header X-Frame-Options SAMEORIGIN;
add_header X-Content-Type-Options nosniff;
add_header X-XSS-Protection "1; mode=block";
Read More

Validate GitHub Webhook Signatures

I’ve mentioned using GitHub webhooks in a previous post where they got used to kick-off a new Jekyll build every time a new commit is pushed. This usually involves having some kind of web server (in my case Flask) running that listens for requests on some endpoint. The hope of course is that these requests only come from GitHub, but really there is nothing stopping any malicious actor from causing a denial of service attack by hitting that endpoint constantly and using up server resources.

To get around this, we need to perform some kind of validation to make sure that only requests from GitHub cause a rebuild.

Signature Header

The folks at GitHub have thought about this and so have included an extra header in all their webhook requests. This takes the form of X-Hub-Signature which, from the docs, contains:

The HMAC hex digest of the response body. This header will be sent if the webhook is configured with a secret. The HMAC hex digest is generated using the sha1 hash function and the secret as the HMAC key.

Therefore, as described in their example, a request coming from GitHub looks like:

POST /payload HTTP/1.1
Host: localhost:4567
X-Github-Delivery: 72d3162e-cc78-11e3-81ab-4c9367dc0958
X-Hub-Signature: sha1=7d38cdd689735b008b3c702edd92eea23791c5f6
User-Agent: GitHub-Hookshot/044aadd
Content-Type: application/json
Content-Length: 6615
X-GitHub-Event: issues
{
  "action": "opened",
  "issue": {
  ...

This mentions a secret which gets used to construct the signature. You can set this value in the main settings page for the webhook. Of course make sure that this is strong, not repeated anywhere else and is kept private.

GitHub Webhook

Signature Validation

As mentioned in the docs, the X-Hub-Signature header value will be the HMAC SHA-1 hex digest of the request payload using the secret we defined above as the key.

Sounds complicated, but it’s really quite simple to construct this value in most popular languages. In this case I’m using Python with Flask to access the payload and headers. The below snippet defines a Flask endpoint which the webhook will hit and accesses the signature and payload of the request:

@app.route('/refresh', methods=['POST'])
def refresh():
    if not "X-Hub-Signature" in request.headers:
        abort(400) # bad request if no header present

    signature = request.headers['X-Hub-Signature']
    payload = request.data

Next, you need to get hold of the secret which is used as the HMAC key. You could hardcode this, but that’s generally a bad idea. Instead, store the value in a permissioned file and read the contents in the script:

with open(os.path.expanduser('~/github_secret'), 'r') as secret_file:
    webhook_secret = secret_file.read().replace("\n", "")

We now have the header value, payload and our secret. Now all that’s left to do is construct the same HMAC digest from the payload we get and compare it with the one from the request headers. As only you and GitHub know the secret, if two signatures match, the request will have originated from GitHub.

secret = webhook_secret.encode() # must be encoded to a byte array

# contruct hmac generator with our secret as key, and SHA-1 as the hashing function
hmac_gen = hmac.new(secret, payload, hashlib.sha1)

# create the hex digest and append prefix to match the GitHub request format
digest = "sha1=" + hmac_gen.hexdigest()

if signature != digest:
    abort(400) # if the signatures don't match, bad request not from GitHub

# do real work after
...

Automate Jekyll with GitHub Webhooks

https://docs.python.org/3.7/library/hmac.html

https://developer.github.com/webhooks

Read More

Ubuntu Server Setup Part 5 - Install Git, Ruby and Jekyll

This part will take care of installing everything necessary to allow the new server to host your personal blog (or other Jekyll site). As a prerequisite, you will also need some kind of web server installed (such as Nginx or Apache) to take care of serving your HTML files over the web. Part 4 covers the steps for my favourite - Nginx.

Install Git

As I store my blog as a public repo on GitHub, Git first needs to be installed to allow the repo to be cloned and new changes to be pulled. Git is available in the Ubuntu repositories so can be installed simply via apt:

sudo apt install git

You might also want to modify some Git config values. This is only really necessary if you plan on committing changes from your server (so that your commit is linked to your account). As I only tend to pull changes, this isn’t strictly required.

sudo apt install git
git config --global color.ui true
git config --global user.name "me"
git config --global user.email "email

Helpful Git Aliases

Here are a few useful Git aliases from my .bashrc. You can also add aliases through Git directly via the alias command.

alias gs='git status'
alias ga='git add'
alias gaa='git add .'
alias gp='git push'
alias gpom='git push origin master'
alias gpu='git pull'
alias gcm='git commit -m'
alias gcam='git commit -am'
alias gl='git log'
alias gd='git diff'
alias gdc='git diff --cached'
alias gb='git branch'
alias gc='git checkout'
alias gra='git remote add'
alias grr='git remote rm'
alias gcl='git clone'
alias glo='git log --pretty=format:"%C(yellow)%h\\ %ad%Cred%d\\ %Creset%s%Cblue\\ [%cn]" --decorate --date=short'

More helpful aliases:

Install Ruby

Ruby is also available in the Ubuntu repositories. You will also need build-essential to allow you to compile gems.

sudo apt install ruby ruby-dev build-essential

It’s a good idea to also tell Ruby where to install gems - in this case your home directory via the GEM_HOME environment variable. Two lines are added to .bashrc to ensure this change is kept for new shell sessions:

echo '# Install Ruby Gems to ~/gems' >> ~/.bashrc
echo 'export GEM_HOME=$HOME/gems' >> ~/.bashrc
echo 'export PATH=$HOME/gems/bin:$PATH' >> ~/.bashrc
source ~/.bashrc

You should now be able to run ruby -v to ensure everything is working.

To get more control over the Ruby installation (install new versions or change versions on the fly), check out rbenv or rvm.

Install Jekyll

Once Ruby is installed, the Jekyll gem can be installed via gem:

gem install jekyll bundler

I also use some extra Jekyll plugins which can also be installed as gems:

gem install jekyll-paginate
gem install jekyll-sitemap

As the path to the Ruby gems directory has been added to the PATH (in the previous section), the jekyll command should now be available:

jekyll -v
jekyll build

Automated Build

Here is a simple bash script which pulls the latest changes from Git, builds the Jekyll site and copies the site to a directory as to be served by your web server (default location is /var/www/html).

#!/bin/bash

echo "Pulling latest from Git";
cd ~/blog/ && git pull origin master;

echo "Building Jekyll Site";
jekyll build --source ~/blog/ --destination ~/blog/_site/;
echo "Jekyll Site Built";

echo "Copying Site to /var/www/html/";
cp -rf ~/blog/_site/* /var/www/html/;
echo "Site Copied Successfully";
Read More

Testing WebSockets

Like any other web service, websockets also need to be tried out and tested. The only problem is they aren’t quite as easy to deal with as your standard REST endpoints etc as you can’t just point to the URL and inspect whatever output is sent back. Websockets are persistent, so instead you need some way of hanging on to the connection in order to see output as it might arrive in various intervals, as well as send adhoc messages down the wire.

PostMan is generally my go-to choice of testing any web related service and for pretty much any other web service it works great. Unfortunately though, PostMan doesn’t have the capability of handling websockets (yet I hope), meaning that other tools must be used if you want a quick and dirty way of displaying/sending messages.

In the Browser

The simplest way to see what’s happening is to use the browser itself - just as it would probably be used on the site itself later on. Using the built in developer tools, you can open up an adhoc WebSocket connection and interact with it as required.

Open Console

Open up the Developer Tools (F12) and go to the Console tab (FireFox works similarly). Here you can enter WebSocket related commands as necessary without having a to run a dedicated site/server.

Note: If you are not running a secured WebSocket (i.e not with the wss: protocol), you will have to visit an HTTP site before you open the console. This is because the browser will not allow unsecured websocket connections to be opened on what should otherwise be a secured HTTPS page.

The below example runs through the code needed to open a WebSocket connection, send content to the server and log the output as it is received:

Open Connection

ws = new WebSocket("ws://localhost:8080/ws"); // create new connection

List to events

// When the connection is open
ws.onopen = function () {
  connection.send('Ping');
};

// Log errors
ws.onerror = function (error) {
  console.log('WebSocket Error ' + error);
};

// Log messages from the server
ws.onmessage = function (e) {
  console.log('From Server: ' + e.data);
};

Send Messages

// Sending a String
ws.send('your message');

Close Connection

ws.close() // not necessarily required

WsCat

The web browser approach works well enough, but it is a bit cumbersome to have to paste in the code each time. There are however many tools which abstract this away into helpful command line interfaces. Wscat is one such terminal based tool which makes testing websockets just about as easy as it gets.

There isn’t much to wscat, just point it to your server URL and it will log out any messages received or send any as you type them. It’s based on Node (see below for similar alternatives in other environments) so just install through npm and run directly within the console.

https://github.com/websockets/wscat

npm install -g wscat
$ wscat -c ws://localhost:8080/ws
connected (press CTRL+C to quit)
> pong
< ping
> ping
< pong

Other Tools

Here are some other related tools (most just like wscat). This GitHub repo guide also has plenty of other websocket related tools you might want to check out.

https://github.com/thehowl/claws

  • Go based
  • Json formatting and pipes

https://github.com/esphen/wsta

  • Rust based
  • most advanced
  • very pipe friendly
  • configuration profiles

https://github.com/progrium/wssh

  • Python based
  • equivalent of wscat if Node is not your thing
Read More