Ryan Harrison My blog, portfolio and technology related ramblings

Kotlin - Add Integration Test Module

The default package structure for a new Kotlin project generated through IntelliJ looks like the following, whereby you have a main source folder with source sets (modules) for your main files and then test source files.

Kotlin Default Project

Typically, you would place your unit tests within the auto-generated test module, and then run them all at once (within one JVM). IntelliJ is generally set up to support this use case and if that’s all you need, requires minimal setup and effort.

However, if you also need to add integration tests (or end-to-end etc), then this project structure can start to cause issues. For example, consider a typical project setup for a server-side app:

  • main - business logic and main app files
  • test - unit tests,
    • typically with JUnit or similar
    • spin up in-memory H2 database for easy DAO testing
  • test-integration - integration/e2e tests
    • typically testing API endpoints with Rest Assured or similar
    • start up full version of the server and any dependencies

You can’t merge all the tests into one module and run them all at once because you would need to start up multiple database instances etc. Conflicts arise and it’s apparent that you need to run them separately in their own dedicated JVM.

To add the above mentioned test-integration module, you can make some edits to your build.gradle file to define a new source set (IntelliJ module):

sourceSets {
    testIntegration {
        java.srcDir 'src/testIntegration/java'
        kotlin.srcDir 'src/testIntegration/kotlin'
        resources.srcDir 'src/testIntegration/resources'
        compileClasspath += main.output
        runtimeClasspath += main.output
    }
}

Here, a new source set for integration tests is created. Gradle is told where the Java and Kotlin source files live and we specify that the classpath inherits from the main source set. This allows you to reference classes of your main module within the integration tests (you might not need this).

Then, we provide a configuration and task for the new source set to ensure that the new module contains the same dependencies as within the main test module (defined using testCompile in your dependencies). Finally, define a new Task to run the integration tests, pointing it to the classes and classpath of the testIntegration source set instead of the inherited defaults from test:

configurations {
    testIntegrationCompile.extendsFrom testCompile
    testIntegrationRuntime.extendsFrom testRuntime
}

task testIntegration(type: Test) {
    testClassesDirs = sourceSets.testIntegration.output.classesDirs
    classpath = sourceSets.testIntegration.runtimeClasspath
}

Similarly to how you might have previously set the target bytecode version for the main and test modules, you need to do the same for the new module:

compileTestIntegrationKotlin {
    kotlinOptions.jvmTarget = "1.8"
}

If you run Gradle with the option to ‘Create directories for empty content roots automatically’, you should see a new module get created. You might notice one issue though, the new module is not marked as a test module within IntelliJ. You could do this manually, but it would get reset every time Gradle runs. To override this, you can apply the idea plugin and add the source directories of the new source set:

idea {
    module {
        testSourceDirs += project.sourceSets.testIntegration.java.srcDirs
        testSourceDirs += project.sourceSets.testIntegration.kotlin.srcDirs
        testSourceDirs += project.sourceSets.testIntegration.resources.srcDirs
    }
}

Now you will see the desired output after Gradle runs:

With Integration tests module

WARNING - This approach is not without problems. If you look at the Test Output Path of the new module, it is defined as \kotlin-scratchpad\out\test\classes which is the same as the main test module. Therefore, all the compiled test classes will end up in the same directory - which causes issues if you try to Run All for example. To fix this, you have to manually update the path to \kotlin-scratchpad\out\testIntegration\classes. Alternatively, you might not apply the idea plugin and just mark the module for tests each time Gradle runs. Hopefully I will find a fix for this at some point.

Read More

Even More Favourite Kotlin Features

This post is a continuation of a previous post on some of my favourite language feature in Kotlin.

Coroutines

Not technically a language feature, as most of it is implemented as a library (which is cool in itself when you see the syntax), but still very useful. Dealing with Threads in Java has long been a bit of a pain point. The situation has got significantly better with the introduction of API’s such as ExecutorService and now CompletableFuture, but the syntax isn’t all that readable and you end up with RxJava like messes of chained method calls and lambdas everywhere.

Coroutines can be thought of as very lightweight threads - you can spawn thousands of coroutines without crashing your program because, unlike threads, they don’t consume any OS level resources. In Kotlin, coroutines revolve around the concept of functions which can suspend. That is, at some point they can be suspended and the thread that was running it can go off and run something else. Later on, the coroutine can resume (maybe on a different thread) with new values and as if nothing ever happened. Under the hoods, similarly to C# async/await, the compiler generates a simple state machine which handles the suspension and resumption of these functions.

The great benefit of this is that your asynchronous code looks just like you would write it synchronously - no dealing with the threading model directly or chaining callbacks together. It also makes error handling much easier to deal with.

suspend fun someWork(): String {
    // call a web service, do some calculation etc
}

suspend fun worker() {
    val first = async { someWork() }
    val second = async { someOtherWork() } 
    println("The answer is ${first.await() + second.await()}") // suspension point
}

In the example above, worker() will launch two coroutines to perform the two async blocks. By default, the coroutine dispatcher uses the shared ForkJoinPool which is where we get the real parallelisation from. In the println statement, the results are ‘awaited’. At this point, if the result is not immediately available, execution suspends and the thread is free to do something else. When the async block returns, worker() is resumed and the println can continue (maybe suspending again on the second await()).

You can also do simpler things like just launching a background coroutine to do some work if you are not interested in the result:

launch {
    // do some work
    delay(100) // not Thread.sleep()
    // coroutines use a special version which is non-blocking
}

The key factor is that you want to avoid blocking threads. In Java multithreading scenarios, this is much more common - maybe you block waiting for a queue to fill or until a web request is finished. With Kotlin coroutines you don’t have to block the entire thread - you suspend the coroutine instead and the thread is free to continue doing other work. This is especially handy in web frameworks such as Ktor which is built around coroutines.

This is just scraping the surface of what the coroutines library has to offer -

  • channels (think pub/sub)

  • support for custom dispatchers and contexts (restrain execution to your thread pool etc)

  • full support for cancellation and error handing

  • actors, producers and select expressions

The introductory guide is a great read if you want to learn more. In effect, they are similar to Go-routines and a lot more powerful than C# async/await. If you read the guide you can see the multitude of use cases which can be easily parallelised or made non-blocking.

Top Level Functions and Multi-Class Files

When you’ve got so many language constructs to remove boilerplate code and verbosity, it doesn’t make sense to force developers into having only one class per file. Thankfully, unlike Java, no such restriction is forced in Kotlin - you can have as many public classes, functions, definitions in one file as you want. When you have a lot of data classes or sealed classes, it makes life a lot simpler to bundle up similar classes into one file. When you go back to Java, it’s one thing that starts to get on your nerves that you may have never thought about before:

// All in one file
data class Foo(val bar: String)
data class Baz(val foo: Int)

class Printer {
    fun print(s: String) = println("Printer is printing $s")
}

sealed class Operation {
    class Add(val value: Int) : Operation()
    class Substract(val value: Int) : Operation()
}

You may have noticed that I said ‘top-level functions’ before. Yes, that’s right, in Kotlin a function doesn’t have to belong to a class - you can have it sitting with other related functions in their own file, or anywhere you want.

There is no need to have Utils classes anymore. You can (and should) name you source files as such, but the utilities themselves can sit in them without a class definition. And they can be used without referring to such a class:

// StringUtils.kt
fun replaceCommas(s: String) = s.replace(",", ".")
// and more helpers

// Work.kt
import StringUtils.replaceCommas
fun work() {
    println(replaceCommas("Some text, goes here"))
}

Even though Kotlin does have a language feature that could support a similar utils pattern you would see in Java - the object keyword - top-level helper functions are much neater.

It might not seem like much, but it really can dramatically reduce the overall number of source files you have to deal with. A common case is where you define a short interface and a couple of implementations. In Java they would have to be split into 3 separate .java files, regardless of the fact that it might all be 50 lines long. In Kotlin, you can put everything into one file.

Destructuring Declarations

Not quite Python-like multi-value returns from functions, but still some nice syntax to retrieve variables from data classes returned from your functions.

data class Student(val name: String, val age: Int)
fun getStudent(): Student {
    return Student("John", 22)
}
val (name, age) = getStudent()
println(name)
println(age)

Singleton (Object)

Another Kotlin feature which I like is the simplicity of defining a “singleton”. Consider the following example of how a singleton could be created in Java and then in Kotlin.

public class SingletonInJava {

    private static SingletonInJava INSTANCE;

    public static synchronized SingletonInJava getInstance() {
        if (INSTANCE == null) {
            INSTANCE = new SingletonInJava();
        }
        return INSTANCE;
    }   
}
object SingletonInKotlin {
}

// And we can call
SingletonInKotlin.doSomething()

Kotlin has a feature to define a singleton in a very clever way. You can use the keyword object, which allows you to define an object which exists only as a single instance across your app. No need to worry about initialising your instance of thread safety, everything is handling by the compiler.

Read More

Automate Jekyll with GitHub Webhooks

In the ongoing task of trying to automate as much of the blogging process as much as possible, actually rebuilding the static files is the most painful and could definitely use improvement. If you’re not using something like GitHub Pages (which does it automatically), but instead host everything on your own server, rebuilding the site after you add a new post could involve manually ssh’ing into the machine and running the Jekyll command manually.

That gets old and fast. Ideally you want a similar experience to GitHub Pages in that everything will happen as soon as you push your changes to the repo. If you have control over the VPS which is hosting everything, and can install a simple endpoint in Python/whatever, combined with webhooks you can get the exact same behaviour.

Create a GitHub Webhook

If your site is hosted on GitHub (presumably other providers have something similar), go into your Repo -> Settings -> Webhooks and click on Add Webhook.

GitHub Webhook

Here you can choose a URL which will be called by GitHub when certain actions happen (if you choose the ‘let me select’ option you can see the comprehensive list). In this case we are going to install a simple endpoint on our blog server, which when called will rebuild the Jekyll blog and redeploy.

For this use case we are just interested in the push event (i.e when you push a new post), but you could also perform actions on all sorts of other events if needed. In the content type I have left it as form-urlencoded as I’m not really interested in the payload, but you could also choose JSON if needed. The payload will include extra details of the event - in this case the files which have changed as a result of the git push. This could be helpful if you only wanted to regenerate certain portions of your site if they have been modified.

You can also provide a secret - of which the SHA-1 hash which will be added into the X-Hub-Signature header of each request. More details in the docs. Once finished click Add webook.

Note - GitHub webhooks have a timeout of 10 seconds so you might see them fail if your blog takes longer to rebuild. It doesn’t really matter though as the server will have still been notified.

Add a Flask Endpoint

Now a webhook is set up that will ping our server every time something is pushed to the repo, we have to install something to handle that request and cause a Jekyll rebuild. There is a sea of lightweight web frameworks available these days, in this case I’m going to use Python and Flask (mainly due to the fact that Python is already installed on Debian based servers). On your server run the following to install Flask:

$ pip install Flask

Now we can create a simple endpoint mapped to the request we specified in the webhook config:

from flask import Flask
import subprocess

app = Flask(__name__)

@app.route('/api/blogrefresh', methods=['POST'])
def blogrefresh():
    script_path = "~/bin/jekyll_rebuild.sh"
    subprocess.call([os.path.expanduser(script_path)])
    return "Success"

if __name__ == "__main__":
    app.run(host='0.0.0.0')

In the code above a new Flask app is created, with one endpoint which handles POST requests to our webhook URL. In the handler a simple shell script is called (living in the home dir of the running user) which is what actually runs jekyll build.

The contents of the jekyll_rebuild.sh script simply pulls the latest changes from the Git repo, rebuilds the site and copies the static files to the folder served by the web server - /var/www/html in this case:

#!/bin/bash

echo "Pulling latest from Git"
cd ~/blog/ && git pull origin master

echo "Building Jekyll Site";
jekyll build --source ~/blog/ --destination ~/blog/_site/;
echo "Jekyll Site Built";

echo "Copying Site to /var/www/html/";
cp -rf ~/blog/_site/* /var/www/html/;
echo "Site Rebuilt Successfully";

And that’s pretty much it. Just run the Flask web server via python3 refresh.py. By default Flask runs on port 5000, so you might need to either open up a port on your server (and update the webhook url) or proxy it through Apache/Nginx.

You could also integrate Elastic Jekyll into this process to give your site full-text search via ElasticSearch that gets automatically updated as you add new content!

Read More

More Favourite Kotlin Features

This post is a continuation of a previous post on some of my favourite language feature in Kotlin.

Higher Order Functions and First Class Lambdas

Functions are considering first-class citizens in Kotlin. Java 8 introduced the much needed concept of lambdas and functional interfaces, which bring large improvements to the language, but Kotlin brings in some more great syntax which further improves the experience.

In Java, if you want to pass in a block of code, or return one, you have to start dealing with instances of Function, Predicate etc. This is fine, but not terribly readable when it’s really just following the convention of first type params are arguments, last is the return type - e.g. Function<String, String, String, Integer>. In Kotlin, the concept of a lambda is much more deeply integrated into the language. If you wanted to pass in the equivalent of the above Function in Kotlin, you would specify the type of the argument as a lambda expression:

fun work(job: (String, String, String) -> Int) {
    // call job as you would think, not .call()
    job("a", "b", "c)
}

The same can be used if you want to return a function from a function:

fun getPrinter(): (String) -> Unit = { println(it) }

Note above that you don’t even need to specify the return type, Kotlin will infer it by itself. It also makes use of it which is a special variable which refers to the first argument of the lambda.

The syntax gets even nicer when you start calling functions which take a lambda with one argument as its last parameter. Take the .also method which you can call on Any object:

val str = "something".also((s) -> {
    println(s)
})
// simplify using 'it' to refer to the parameter
val str = "something".also({
    println(it)
})
// simplify by removing parenthesis as last argument is a lambda
val str = "something".also {
    println(it)
}

Such syntax enables the use of some really nice builder API’s. Take kotlinx.html with code taken from the ktor-samples:

h2 { +"Login" }
form(call.url(Login()) { parameters.clear() }, classes = "pure-form-stacked", method = FormMethod.post) {
    acceptCharset = "utf-8"
    label {
        +"Username: "
        textInput {
            name = Login::userName.name
            value = it.userName
        }
    }
    label {
        +"Password: "
        passwordInput {
            name = Login::password.name
        }
    }
    submitInput(classes = "pure-button pure-button-primary") {
        value = "Login"
    }
}

Or even simply working with lists. In Java, streams have helped significantly, but in Kotlin it’s still much more expressive and neat (coupled with the addition of new helper extension methods in the standard library):

val numbers = arrayListOf(10, 5, -9, 9, 11, 5, -6)
numbers.filter { it >= 0}
numbers.forEach { println("${it * 2} ") }
numbers.map { it * 2 }.filter { it > 10 }.forEach { println(it) }

The same :: method reference syntax as Java is available in Kotlin to refer to functions within the same/different scopes. Kotlin also introduces the concept of inline functions however (just like also as described above). This reduces any overhead that would otherwise be brought in when passing around method references, as the compiler merely copies and pastes it to the call site. Note that Kotlin also does have similar functional interface classes to Java - KFunction<T> etc.

The improvements made to lambdas in Kotlin make passing around and dealing with them in general much easier and more attractive to developers. You just have to take a look at any popular Kotlin library to see how widely the concept is utilised within the language. I think that much of the improved readability and expressiveness of Kotlin can be in some way attributed to this feature.

Null Safety

Another one of the flagship features of Kotlin. The whole concept of nullability is handled very differently in Kotlin compared to say Java. In Java, any reference variable can be nullable. This allows for a lot of flexibility, but introduces a whole mess of dreaded NullPointerException. This may not seem like a huge issue, but when they make up the highest proportion of Exceptions generated from Java software (by quite some margin), then it’s clear that a better solution is needed.

In Kotlin you have to be explicit. An equivalent variable of type String for example in Kotlin must be given a value at definition. Even if it’s a var it cannot be given a null value. Of course, if you don’t have the concept of null, then you can’t get NullPointerExceptions!

val str:String = "name"
str = null // Compile-time error
str.length // will never generate a null pointer exception

However, having the concept of ‘no value’ is actually quite useful. In Kotlin you can assign null to a variable, but you have to explicitly define that capability in the variable definition, and more importantly you have to perform a null check every time you use the variable. In Java-land the equivalent would be using the Optional<T> class in the standard library.

val str:String? = "name"
str = null // now accepted
str.length // compile-time error as str might be null
str?.length // safe call operator for easy null check

You can define a nullable variable by appending a ? to the end of the variable definition. Now it can accept null as a value, but any time you use that variable you will need to do a null check to prevent exceptions. Kotlin provides some handy concepts to make handling null easier:

str?.length // length if str is not null, otherwise null
val lengthOrZero = str?length ?: 0 // length if str is not null, otherwise zero (elvis operator)
str?.let {
    // use as it.length etc (inlined function with null check)
    // it is smart cast to String not String?
}

Another killer feature is that Kotlin also performs smart type casting to prevent casting to the non-nullable type:

if(str != null) {
    val length = str.length // str is now String not String?
}

Default and Named Arguments

Gone are the days of needing to write boilerplate function overloads when you want to provide some default or optional values. Similar to how languages such as Python have done for years, Kotlin allows default values for function arguments and named arguments when calling functions:

fun foo(num: Int, str: String = "12") {
}
//All 3 are valid
foo(15, "Hey")
foo(str = "Name", num = 45) //Named argument
foo(45) //usage of default argument

In Java the equivalent of the above would be:

void foo(int num, String str) {
}

void foo(int num) {
    foo(num, "12")
}

A nice feature to improve clarity and reduce boilerplate. In the example above, the Java version isn’t so bad, but the problem grows exponentially as you add arguments.

This post has again got quite long, but there are even more items I can discuss in a part 3.

Read More

Make HTTP Requests in Kotlin

Updated 09/18 - Add section on the new HTTP Client in JDK 11

These days making HTTP requests in any language is a staple of many common workflows and features. This post will go through a few of the methods in which you can make such requests in Kotlin using some of the great open source libraries available.

JDK 11 HTTP Client API

If you are using the latest JDK release, you can make use of the new built-in HttpClient API which is now modern and fully feature complete. It supports HTTP 2.0 (header compression, server push, multiplexing etc), WebSockets and can be fully asynchronous - which integrates brilliantly with Kotlin coroutines.

Please refer to full post on the API here

In short, you create a new HttpClient and pass HttpRequest objects in (the below example is synchronous). Note that you could easily create some helper/extension functions to make this code much neater:

val client = HttpClient.newBuilder().build();
val request = HttpRequest.newBuilder()
               .uri(URI.create("https://something.com"))
               .build();
val response = client.send(request, BodyHandlers.ofString());
println(response.body())

The API also has support for performing completely asynchronous requests (using non-blocking IO). In this case a CompletableFuture is returned instead of the raw HttpResponse. If you are using coroutines, you can use the CompletionStage.await() extension function defined within the JDK integration library to suspend the current coroutine until the response is available. No need to deal with chaining together callbacks!

suspend fun getData(): String {
    // above code to construct client + request
    val response = client.sendAsync(request, BodyHandlers.ofString());
    return response.await().body() // suspend and return String not a Future
}

// in some other coroutine (suspend block)
val someData = getData()
process(someData) // just as if you wrote it synchronously

Fuel

Probably the most commonly used library for this requirement, Fuel is fully featured and stable for any such use case. The default settings used are also good so it requires very little/if any configuration to get up and running.

The base of the library sits on extension functions of String - which in this case represent URL. This makes the interface very fluent any easy to read.

Fuel also makes use of the Result library - written by the same creator - to bundle up error conditions and responses. This does mean another dependency to add, but it makes error handling a bit easier. The recommended method is done through a when expression:

"http://httpbin.org/get".httpGet().responseString { request, response, result ->
  when (result) {
    is Result.Failure -> {
      val ex = result.getException()
    }
    is Result.Success -> {
      val data = result.get()
    }
  }
}

The underlying requests are performed on a dedicated thread pool making the library capable of both blocking and asynchronous requests.

You can also perform synchronous requests if you want:

val (request, response, result) = "http://httpbin.org/get".httpGet().responseString() // result is Result<String, FuelError>

Take a look at the docs to find examples of how to use authentication, POST etc requests, parameters support, and timeouts etc. The API is configured in a fluent manner:

"http://httpbin.org/get".httpGet().timeout(timeout).timeoutRead(timeoutRead).responseString { request, response, result -> }

Ktor Client

Another approach is to make use of the newer Ktor library. Although the main focus has been on the server-side area, it also includes another package to perform non-blocking requests in a similar fashion. As Ktor is based around Kotlin coroutines, this perhaps makes most sense if you are more familiar/are already using them in your project.

Ktor includes multiple methods of requests. The main being Apache, but CIO (Coroutine IO) and Jetty handlers are also available. Configuration is done through a fluent, builder-like API very similar to that used in the Ktor server packages.

val client = HttpClient(Apache) {
    install(JsonFeature) {
        serializer = GsonSerializer()
    }
}
val htmlContent = client.get<String>("https://en.wikipedia.org/wiki/Main_Page")

Note that in this case, get is a suspending function so you would have to call it within a coroutine. Using runBlocking or async are the most suitable candidates and means that, unlike Fuel, you have complete control over which thread pool is used for the requests.

suspend fun parallelRequests() {
    val client = HttpClient(Apache)
    
    // Start two requests asynchronously.
    val req1 = async { client.call("https://127.0.0.1:8080/a").response.readBytes() }
    val req2 = async { client.call("https://127.0.0.1:8080/b").response.readBytes() }
    
    // Get the request contents without blocking threads, but suspending the function until both
    // requests are done.
    val bytes1 = req1.await() // Suspension point.
    val bytes2 = req2.await() // Suspension point.
}

The Apache engine is based on Apache HTTPComponents and supports the widest variety of config options. It is also the only engine to support redirects and HTTP/2. It will bring in apache as a dependency though. The CIO engine is more basic but has no extra dependencies.

Much like you would expect from any HTTP library, you can configure cookies, authentication, timeouts etc as needed.

Native URL

If you don’t want to use a dedicated library, don’t want to do any custom configuration, then Kotlin includes a nice extension method on the URL class to perform GET requests via opening a stream.

val response = try {
        URL("http://google.co.uk")
                .openStream()
                .bufferedReader()
                .use { it.readText() }

Other notable mentions:

khttp library

natively using HttpURLConnection as you would in Java

Read More