Ryan Harrison My blog, portfolio and technology related ramblings

Kotlin - Things to Improve

This list isn’t very long and doesn’t exactly include any game breaking lack of functionality. A good testament to how Kotlin is a solid language these days.

Try/Multi Catch

A somewhat simple language feature that according to the designers is still on the cards. Not too much of a problem to live without, but Java has had it for years and it should be a staple of any modern language:

try {
    ...
} catch(SomeException | OtherException e) {
    ...
}

The Kotlin version relies on the when construct, which has many more levels of nesting and is generally just more ugly to read and write:

try {
    ...
} catch (e: Exception) {
    when(e) {
         is SomeException, is OtherException -> {...}
         else -> throw e
    }
}

Ternary Operator and Collection Literals

These are both hotly contested, but I personally believe they should be part of Kotlin. All the arguments against the ternary operator seem to revolve around being ‘too easy to abuse’. Yeah right. Kotlin has so many other language features which can be abused already (operator overloading anyone? extension functions anyone?), I just don’t see why it’s such a big deal.

Like it or not, Kotlin is competing against a myriad of other C based languages - all of which have the ternary operator already. Pretty much every developer knows it these days, it should be made available.

Because if in Kotlin is an expression, it is termed as the ‘acceptable alternative’:

val something = if(a < 4) "it's valid" else "not valid"

v.s. the syntax everyone and their mum is familiar with:

val something = a < 4 ? "it's valid" : "not valid"

Not many characters saved, true. However, when the times comes that I need one (yes because it has it’s place), I get angry at having to write the if statement - which is just more clunky to write.

Similarly, another highly wanted feature is collection literal syntax. I realise that this might get a bit complicated due to the whole mutable/immutable lists thing in Kotlin, but there are enough people who want it for a reason. Kotlin is trying to attract developers from the Python/Javascript/Swift worlds, this kind of thing is what will annoy those trying to make the transition.

People are getting on their high horses and spouting the important of ‘language principles’ and not cluttering the language. Your language principles can be as solid as you want, but if nobody uses it, then what’s the point? Developers obviously expect these things - there is data backing it up. Is how I create my lists or perform inline conditionals that much of a big deal to not have both ways of doing it?

Static

I think the whole static and companion deal is a bit of a mess in Kotlin. When writing Kotlin on the JVM, the concept of static is still very much a thing, and in certain places still a necessity. Want to create a JUnit method marked as BeforeAll/BeforeClass? Yeah, it’s just straight up annoying in Kotlin.

To create a simple static method in Kotlin, you have to go through the hassle of creating a companion object, plus specially annotate the function to tell the compiler to actually make it static. In this case, Java is actually less verbose than Kotlin (and by no small margin) and for what gain? They can do better here to appease those on the JVM.

companion object {
    @JvmStatic
    fun actuallyStatic() = println("That was a chore")
}

Now, I don’t need static very often, really only for loggers/testing most of the time etc, but it’s such a pain to do the above that I prefer the other (less efficient) routes of logger per instance of or a top level variable. It seems like a bit of a tacked on feature to solve this kind of problems and ties into some of the efficiency problems discussed below. Static variables/methods are about as fast as it gets, turns out companion objects are the complete opposite.

Efficiency

I don’t actually have any real data to back this up, but I think it’s common knowledge that the Kotlin compiler isn’t the fastest thing ever. To be honest I’m not too surprised, the amount of work it has to do is impressive. Nonetheless, when working on a Kotlin project and going back to Java, the compilation differences are noticeable to say the least. The Kotlin team have, and continue to do a lot of work to improve it though, so I hope it continues to get faster in the future.

Aside from the compiler, I feel the need to rant about generally inefficiencies though. Maybe this is just a reason to moan about we seem to have accepted that it’s a good idea to use lambdas and streams literally everywhere. What happened to the good old for-loop? Things like streams aren’t free. Depending on what you’re doing there can be significant allocation going on and other overhead. Kotlin inline functions do a good job at resolving this, but they aren’t usable everywhere.

There is a great series here which covers some of the hidden costs in using some of Kotlin fancy language features. As discussed above, when you define a simple companion object, the compiler generates a bunch of boilerplate and indirection. I just wanted a simple static variable/function, why do I have to have all this extra stuff (yes, really, multiple new classes get generated for this).

There is a widespread movement towards immutability, which don’t get me wrong, has many benefits. But it also encourages so much inefficiency. Want to increment that one integer inside this object? Let’s copy the whole thing. Hardware continues to improve, yet developers and language designers seem to find a way to add another layer of abstraction to render it moot.

Too many imports

I’ve been doing a fair amount of work with Ktor and Exposed recently, and can’t help but think that extension functions are getting massively abused at this point (already). Don’t get me wrong, they are great and the syntax they allow for is a big selling point, but when you start using some of these libraries, you realise that everything and their dog is a top level extension function. Literally everything.

Take the following snippet which defines a simple Ktor web service for example. The actual logic portion is nice and neat, but at the top we have 22 (!) imports.

import com.fasterxml.jackson.annotation.JsonInclude
import com.fasterxml.jackson.module.kotlin.jacksonObjectMapper
import io.ktor.application.call
import io.ktor.http.HttpStatusCode
import io.ktor.http.cio.websocket.Frame
import io.ktor.request.receive
import io.ktor.request.authorization
import io.ktor.request.receiveMultipart
import io.ktor.response.respond
import io.ktor.response.etag
import io.ktor.response.header
import io.ktor.response.respondFile
import io.ktor.response.respondText
import io.ktor.routing.Route
import io.ktor.routing.delete
import io.ktor.routing.param
import io.ktor.routing.get
import io.ktor.routing.post
import io.ktor.routing.put
import io.ktor.routing.route
import io.ktor.websocket.webSocket
// before any of our actual app imports

...
// implementation at https://github.com/raharrison/kotlin-ktor-exposed-starter/blob/master/src/main/kotlin/web/WidgetResource.kt

Kotlin has import * syntax, which the library makers tell everyone to use, but didn’t we previously agree that this was a bad idea? Something about having to ‘explicitly define your dependencies’ or something? But hey, as long as we get our nice builder syntactic sugar, lets just make IntelliJ fold it up and forget it ever happened.

Read More

Ubuntu Server Setup Part 7 - Email Forwarding with Postfix

One of the best things about owning your own domain name is being able to use it as a custom email address. Many companies provide this service for a monthly fee, but if you have your own personal server/VPS already anyway, then you might as well take advantage of it.

Many custom email solutions are very heavyweight and include setting up Postfix, Dovecot (IMAP and POP3 server) and some client like Roundcube. You also have to worry about spam so will also need something like spamassassin. Not to mention all the security and redundancy issues involved in storing and maintaining your own (sensitive) data. If you want an all-in-one bundle of all the above, check out Mail-in-a-Box.

That’s all well and good for some use cases, but wouldn’t it be nicer if you could use your existing Gmail account to handle email traffic for your custom domain? Integration with all your devices, access to your other email accounts, built in spam detection and you will be managing much less infrastructure.

This part covers the first part of getting to that solution - forwarding all email that comes to your domain onto your existing Gmail account. The second part will then cover the other half of the story which is how to send email as your custom domain from within Gmail.

In short, at the end of this part all email to me@yourdomain.com will be forwarded on to you@gmail.com automatically. It is assumed that you have control over a domain name, can change DNS properties, and have an existing Gmail account.

DNS Setup

The first step is to configure a couple of DNS records to ensure that email traffic gets routed to your server correctly (MX record). The others are to help out in the battle against Gmail thinking that all your mail is spam and for them to correctly validate your server as being in control over your domains mailbox. In whatever DNS control panel your have, create the following:

  • An A record pointing yourdomain.com to your servers IP address
  • An MX record pointing yourdomain.com to the IP of your server (or @ to point to your A record)
  • A PTR (reverse DNS) record pointing your servers IP to yourdomain.com. This allows Gmail to verify the legitimacy of our server via its IP when Gmail receives a forwarded e-mail from it.
  • An SPF record with the contents v=spf1 mx a ip4:<yourip>/32 include:_spf.google.com ~all (replacing with your own IP address)

The SPF record tells Gmail that only the servers specified are allowed to send e-mails purporting to be from yourdomain.com All other servers attempting to do the same will be rejected - which helps a lot with spam.

Install and Configure Postfix

Now to get to the core part - installing Postfix and configuring it to forward all email to your Gmail account.

$ sudo apt-get install postfix

During the installation process, you will be be prompted a couple times. Choose the Internet Site option and provide your domain name into the inputs.

/etc/postfix/main.cf

Once Postfix is installed, open up it’s main configuration file:

$ sudo nano /etc/postfix/main.cf

The file will already contain a bunch of configuration options. In the second part I will provide the full configuration, but in this part just paste in the below. This basically just tells Postfix what our domain name is and sets some network properties.

The most important part is the bottom two lines which sets up alias mappings. This will tell Postfix that for all traffic to yourdomain.com perform a lookup into /etc/postfix/virtual to find out where to forward it on to.

myhostname = yourdomain.com
mydomain = yourdomain.com
myorigin = /etc/mailname
mydestination = yourdomain.com, localhost.localdomain, localhost
relayhost =
mynetworks = 127.0.0.0/8 [::ffff:127.0.0.0]/104 [::1]/128
mailbox_size_limit = 0
recipient_delimiter = +
inet_interfaces = all
inet_protocols = ipv4

virtual_alias_domains = yourdomain.com
virtual_alias_maps = hash:/etc/postfix/virtual

/etc/postfix/virtual

Now to edit the virtual mappings file to tell Postfix to forward our mail onto your Gmail account:

$ sudo nano /etc/postfix/virtual

Within that file, you can provide your domain name, followed by a whitespace and a comma separated list of domains to forward it to. In this case we are forwarding everything which comes to me@yourdomain.com onto your Gmail account. You can just provide yourdomain.com as the first parameter to forward all email regardless of address.

# Forwarding mapping, one from-to address pair per line. The format is:
#     <forward-from-addr> <whitespace> <forward-to-addr>
me@yourdomain.com you@gmail.com

Update lookup table

Postfix doesn’t directly read the virtual file, but instead generates a lookup table from it. Run the following to refresh the lookup table:

$ sudo postmap /etc/postfix/virtual

Reload Postfix

$ sudo service postfix start # if not already running

$ sudo service postfix reload # reload our config files

$ sudo service postfix restart # or perform a restart

Configure Firewall

SMTP traffic (which Postfix is handling and forwarding) runs on port 25, so makes sure to allow it through your firewall. Refer to Part 3 for info on how to set one up.

$ ufw allow 25/tcp

Testing

After any new DNS records have propagated, you can test by sending an email to me@yourdomain. It should be forwarded onto me@gmail.com pretty quickly (make sure to also check your spam folder).

If you tried to the above and never received the email in your Gmail inbox, try checking the Postfix logs at /var/log/mail.log for errors.

That’s it for this part, in the next part we will cover how to send email from Gmail, relaying through our server to appear as though it was sent by our custom domain.

Read More

Java 11 HTTP Client API

One of the more noteworthy features of the new Java 11 release is the new HttpClient API within the standard library, which has been in incubator status since JDK 9. Previously, the process of sending and retrieving data over HTTP has been very cumbersome in Java. Either you went through the hassle of using HttpURLConnection, or you bring in a library to abstract over it such as the HttpClient from Apache. Notably, both solutions would also block threads whilst doing so.

As these days dealing with HTTP connections is so common, JDK 11 finally has a modern API which can deal with these scenarios - including support for HTTP 2 (server push etc) and WebSockets.

Create a Client

The first step to make use of the new API is to create an instance of the HttpClient class. The library itself makes heavy use of the builder pattern to specify configuration options, as is the case in most new Java libraries.

You can configure things like HTTP version support (the default is set to HTTP 2), whether or not to follow redirects, authentication and a proxy for all requests that pass through the client.

HttpClient client = HttpClient.newBuilder()
      .version(Version.HTTP_2)
      .followRedirects(Redirect.SAME_PROTOCOL)
      .authenticator(Authenticator.getDefault())
      .build();

The HttpClient instance is the main entry point to send and receive requests (both synchronously or asynchronously). Once created you can reuse them, but they are also immutable.

Create Requests

An HTTP request is represented by instances of HttpRequest which holds the URL, request method, headers, timeout and payload (if applicable). By default GET is used if no other is specified.

HttpRequest request = HttpRequest.newBuilder()
               .uri(URI.create("https://something.com"))
               .build(); // GET request

The following snippet creates a POST request with a custom timeout. A BodyPublisher must be used to attach a payload to a request - in this case taking a JSON String and converting it into bytes.

HttpRequest request = HttpRequest.newBuilder()
      .uri(URI.create("https://something.com/api"))
      .timeout(Duration.ofMinutes(1))
      .header("Content-Type", "application/json")
      .POST(BodyPublishers.ofString(json)
      .build()

Send Requests

Once an HttpRequest is created, you can send it via the HttpClient previously constructed. Both synchronous and asynchronous operations are supported:

Sync

The HttpClient.send method will perform the HTTP request synchronously - meaning that the current thread will be blocked until a response is obtained. The HttpResponse class encapsulates the response itself including status code, body and headers.

HttpResponse<String> response = client.send(request, BodyHandlers.ofString());
System.out.println(response.statusCode());
System.out.println(response.body());

When receiving responses, a BodyHandler is provided to instruct the client on how to process the response body. The BodyHandlers class includes default handlers for the most common scenarios. ofString will return the body as an UTF-8 encoded String, ofFile accepts a Path to write the response to a file and ofByteArray will give you the raw bytes.

Async

One of the best features of the API is the ability to perform completely asynchronous requests - meaning that no thread is blocked during the process. Under the hoods, the implementation uses NIO and non-blocking channels to ensure no blocking IO operations are performed.

The HttpClient.sendAsync method takes the same parameters as the synchronous version, but returns a CompletableFuture<HttpResponse<T>> instead of just the raw HttpResponse<T>. Just as with any other CompletableFuture, you can chain together callbacks to be executed when the response is available. In this case, the body of the response is extracted and printed out. More details here on how to work with CompletableFuture.

CompletableFuture<HttpResponse<String>> future = client.sendAsync(request,
        BodyHandlers.ofString());

future.thenApply(HttpResponse::body) // retrieve body of response
      .thenAccept(System.out::println); // use body as String

Each HttpClient has a single implementation-specific thread that polls all of its connections. Received data is then passed off to the executor for processing. You can override the Executor on the HttpClient, by default it is a cached thread pool executor.

Kotlin

As the new API sits like any other in the default JDK, you can easily make use of it in your Kotlin projects as well! You can paste in the above examples into IntelliJ to perform the automagical Java-Kotlin conversion, or the below example covers the basics:

val client = HttpClient.newBuilder().build();

val request = HttpRequest.newBuilder()
               .uri(URI.create("https://something.com"))
               .build();

val response = client.send(request, BodyHandlers.ofString());
println(response.statusCode())
println(response.body())

Coroutines (Async)

Things get much more interesting when taking into account the new asynchronous capabilities and Kotlin coroutines. It would be great if we could launch a coroutine which sends the request and suspends until the response is available:

suspend fun getData(): String {
    // above code to construct client + request
    val response = client.sendAsync(request, BodyHandlers.ofString());
    return response.await().body() // suspend and return String not a Future
}

// in some other coroutine (suspend block)
val someData = getData()
process(someData) // just as if you wrote it synchronously

No need to deal with chaining together callbacks onto CompletableFuture, you get the same procedural code flow even though the implementation suspends and is completely non-blocking.

The magic comes from the CompletionStage.await() extension function provided by the coroutines JDK integration library:

return suspendCancellableCoroutine { cont: CancellableContinuation<T> ->
    val consumer = ContinuationConsumer(cont)
    whenComplete(consumer) // attach continuation to CompletionStage
}

Docs for the function.

More Information

https://download.java.net/java/early_access/jdk11/docs/api/java.net.http/java/net/http/package-summary.html

http://openjdk.java.net/groups/net/httpclient/intro.html

https://www.youtube.com/watch?list=PLX8CzqL3ArzXyA_lJzaNmrFqpLOL4aCEz&v=sZSdWq490Vw

Read More

Ktor - File Upload and Download

The ability to perform file uploads and downloads is a staple part of any good web server framework. Ktor has support for both operations in just a few lines of code.

File Upload

File uploads are handled through multipart POST requests in standard HTTP - normally from form submissions where the file selector field would be just one item (another could be the title for example).

To handle this in Ktor, you can get hold of the multipart data through receiveMultipart and then loop over each part as required. In the below example, we are just interested in files (PartData.FileItem), although you could also look at the individual PartData.FormItem as well (which would be the other form fields in the submission).

A Ktor FileItem exposes an InputStream via streamProvider which can be used to access the raw bytes of the file which has been uploaded. As in the below example, you can then simply create the appropriate file and copy the bytes from one stream (input) to the other (output).

post("/upload") { _ ->
    // retrieve all multipart data (suspending)
    val multipart = call.receiveMultipart()
    multipart.forEachPart { part ->
        // if part is a file (could be form item)
        if(part is PartData.FileItem) {
            // retrieve file name of upload
            val name = part.originalFileName!!
            val file = File("/uploads/$name")

            // use InputStream from part to save file
            part.streamProvider().use { its ->
                // copy the stream to the file with buffering
                file.outputStream().buffered().use {
                    // note that this is blocking
                    its.copyTo(it)
                }
            }
        }
        // make sure to dispose of the part after use to prevent leaks
        part.dispose()
    }
}

The documentation also includes a suspending copyTo method which can be used to save the upload to a file in a non-blocking way.

File Download

File downloads are very straightforward in Ktor. You just have to create a handle to the specified File and use the respondFile method:

get("/{name}") {
    // get filename from request url
    val filename = call.parameters["name"]!!
    // construct reference to file
    // ideally this would use a different filename
    val file = File("/uploads/$filename")
    if(file.exists()) {
        call.respondFile(file)
    }
    else call.respond(HttpStatusCode.NotFound)
}

By default, if called from the browser, this will cause the file to be viewed inline. If you instead want to prompt the browser to download the file, you can include the Content-Disposition header:

call.response.header("Content-Disposition", "attachment; filename=\"${file.name}\"")

This is also helpful if you save the uploaded file with a different name (which is advisable), as you can override the filename with the original when it gets downloaded by users.

More information about the Content-Disposition header

Read More

Ubuntu Server Setup Part 6 - HTTPS With Let's Encrypt

HTTPS is a must have nowadays with sites served under plain HTTP being downgraded in search results by Google and marked as insecure by browsers. The process of obtaining an SSL certificate used to be cumbersome and expensive, but now thankfully because of Let’s Encrypt it completely free and you can automate the process with just a few commands.

This part assumes that you already have an active Nginx server running (as described in Part 4) and so will go over how to use Let’s Encrypt with Nginx. Certbot (the client software) has a number of plugins that make the process just as easy if you are running another web server such as Apache.

Prepare the Nginx Server

Make sure have a server block where server_name is set to your domain name (in this case example.com).

This is so Certbot knows which config file to modify in order to enable HTTPS (it adds a line pointing to the generated SSL certificates).

server {
  	listen 80;
  	listen [::]:80;

  	server_name example.com www.example.com;
  	root /var/www/example;

  	index index.html;
  	location / {
  		try_files $uri $uri/ =404;
  	}
  }

That’s all the preparation needed on the Nginx side. Certbot will handle everything else for us.

Install and Run Certbot

Certbot is the client software (written in Python), that is supported by Let’s Encrypt themselves to automate the whole process. There are a wide range of different alternatives in various languages if you have different needs.

You should install Certbot through the dedicated ppa to make sure you always get the latest updates. In this example we install the Nginx version (which includes the Nginx plugin):

sudo apt-get update
sudo add-apt-repository ppa:certbot/certbot
sudo apt-get update
sudo apt-get install -y python-certbot-nginx

Once installed, you can run Certbot. Here the --nginx flag is used to enable the Nginx plugin. Without this, Certbot would just generate a certificate and your web server wouldn’t know about it. The plugin is required to modify the Nginx configuration in order to see the certificate and enable HTTPS.

sudo certbot --nginx

It will ask for:

  • an email address (you will be emailed if there are any issues or your certs are about to expire)
  • agreeing to the Terms of Service
  • which domains to use HTTPS for (it detects the list using server_name lines in your Nginx config)
  • whether to redirect HTTP to HTTPS (recommended)

Once you have selected these options, Certbot will perform a ‘challenge’ to check that the server it is running on is in control of the domain name. As described in the ACME protocol which is what underlies Let’s Encrypt, there are a number of different challenge types. In this case tls-sni was probably performed, although DNS might be used for wildcard certificates.

If the process completed without errors, a new certificate should have been generated and saved on the server. You can access this via /var/letsencrypt/live/domain.

The Nginx server block should have also been modified to include a number of extra ssl related fields. You will notice that these point to the generated certificate alongside the Let’s Encrypt chain cert. If you checked the option to redirect all HTTP traffic to HTTPS, you should also see another server block generated which merely captures all HTTP traffic and performs a redirection to the main HTTPS enabled server block.

You could stop here if all you want is HTTPS as this already gives you an A rating and maintains itself.

Test your site with SSL Labs using https://www.ssllabs.com/ssltest/analyze.html?d=www.YOUR-DOMAIN.com

Renewal

There is nothing to do, Certbot installed a cron task to automatically renew certificates about to expire.

You can check renewal works using:

sudo certbot renew --dry-run

You can also check what certificates exist using:

sudo certbot certificates

A+ Test

If you did the SSL check in the previous section you might be wondering why you didn’t get an A+ instead of just an A. It turns out that the default policy is to use some slightly outdated protocols and cipher types to maintain backwards compatibility with older devices. If you want to get the A+ rating, add the below config to your Nginx server block (the same one that got updated by Certbot). In particular we only use TLS 1.2 (not 1.0 or 1.1) and the available ciphers are restricted to only the latest and most secure. Be aware though that this might mean you site is unusable on some older devices which do not support these modern ciphers.

ssl_trusted_certificate /etc/letsencrypt/live/YOUR-DOMAIN/chain.pem;

ssl_session_cache shared:le_nginx_SSL:1m;
ssl_session_timeout 1d;
ssl_session_tickets off;

ssl_protocols TLSv1.2;
ssl_prefer_server_ciphers on;
ssl_ciphers "EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH";
ssl_ecdh_curve secp384r1;

ssl_stapling on;
ssl_stapling_verify on;

add_header Strict-Transport-Security "max-age=15768000; includeSubdomains; preload;";
add_header Content-Security-Policy "default-src 'none'; frame-ancestors 'none'; script-src 'self'; img-src 'self'; style-src 'self'; base-uri 'self'; form-action 'self';";
add_header Referrer-Policy "no-referrer, strict-origin-when-cross-origin";
add_header X-Frame-Options SAMEORIGIN;
add_header X-Content-Type-Options nosniff;
add_header X-XSS-Protection "1; mode=block";
Read More