Ryan Harrison My blog, portfolio and technology related ramblings

Automate Jekyll with GitHub Webhooks

In the ongoing task of trying to automate as much of the blogging process as much as possible, actually rebuilding the static files is the most painful and could definitely use improvement. If you’re not using something like GitHub Pages (which does it automatically), but instead host everything on your own server, rebuilding the site after you add a new post could involve manually ssh’ing into the machine and running the Jekyll command manually.

That gets old and fast. Ideally you want a similar experience to GitHub Pages in that everything will happen as soon as you push your changes to the repo. If you have control over the VPS which is hosting everything, and can install a simple endpoint in Python/whatever, combined with webhooks you can get the exact same behaviour.

Create a GitHub Webhook

If your site is hosted on GitHub (presumably other providers have something similar), go into your Repo -> Settings -> Webhooks and click on Add Webhook.

GitHub Webhook

Here you can choose a URL which will be called by GitHub when certain actions happen (if you choose the ‘let me select’ option you can see the comprehensive list). In this case we are going to install a simple endpoint on our blog server, which when called will rebuild the Jekyll blog and redeploy.

For this use case we are just interested in the push event (i.e when you push a new post), but you could also perform actions on all sorts of other events if needed. In the content type I have left it as form-urlencoded as I’m not really interested in the payload, but you could also choose JSON if needed. The payload will include extra details of the event - in this case the files which have changed as a result of the git push. This could be helpful if you only wanted to regenerate certain portions of your site if they have been modified.

You can also provide a secret - of which the SHA-1 hash which will be added into the X-Hub-Signature header of each request. More details in the docs. Once finished click Add webook.

Note - GitHub webhooks have a timeout of 10 seconds so you might see them fail if your blog takes longer to rebuild. It doesn’t really matter though as the server will have still been notified.

Add a Flask Endpoint

Now a webhook is set up that will ping our server every time something is pushed to the repo, we have to install something to handle that request and cause a Jekyll rebuild. There is a sea of lightweight web frameworks available these days, in this case I’m going to use Python and Flask (mainly due to the fact that Python is already installed on Debian based servers). On your server run the following to install Flask:

$ pip install Flask

Now we can create a simple endpoint mapped to the request we specified in the webhook config:

from flask import Flask
import subprocess

app = Flask(__name__)

@app.route('/api/blogrefresh', methods=['POST'])
def blogrefresh():
    script_path = "~/bin/jekyll_rebuild.sh"
    return "Success"

if __name__ == "__main__":

In the code above a new Flask app is created, with one endpoint which handles POST requests to our webhook URL. In the handler a simple shell script is called (living in the home dir of the running user) which is what actually runs jekyll build.

The contents of the jekyll_rebuild.sh script simply pulls the latest changes from the Git repo, rebuilds the site and copies the static files to the folder served by the web server - /var/www/html in this case:


echo "Pulling latest from Git"
cd ~/blog/ && git pull origin master

echo "Building Jekyll Site";
jekyll build --source ~/blog/ --destination ~/blog/_site/;
echo "Jekyll Site Built";

echo "Copying Site to /var/www/html/";
cp -rf ~/blog/_site/* /var/www/html/;
echo "Site Rebuilt Successfully";

And that’s pretty much it. Just run the Flask web server via python3 refresh.py. By default Flask runs on port 5000, so you might need to either open up a port on your server (and update the webhook url) or proxy it through Apache/Nginx.

You could also integrate Elastic Jekyll into this process to give your site full-text search via ElasticSearch that gets automatically updated as you add new content!

Read More

More Favourite Kotlin Features

This post is a continuation of a previous post on some of my favourite language feature in Kotlin.

Higher Order Functions and First Class Lambdas

Functions are considering first-class citizens in Kotlin. Java 8 introduced the much needed concept of lambdas and functional interfaces, which bring large improvements to the language, but Kotlin brings in some more great syntax which further improves the experience.

In Java, if you want to pass in a block of code, or return one, you have to start dealing with instances of Function, Predicate etc. This is fine, but not terribly readable when it’s really just following the convention of first type params are arguments, last is the return type - e.g. Function<String, String, String, Integer>. In Kotlin, the concept of a lambda is much more deeply integrated into the language. If you wanted to pass in the equivalent of the above Function in Kotlin, you would specify the type of the argument as a lambda expression:

fun work(job: (String, String, String) -> Int) {
    // call job as you would think, not .call()
    job("a", "b", "c)

The same can be used if you want to return a function from a function:

fun getPrinter(): (String) -> Unit = { println(it) }

Note above that you don’t even need to specify the return type, Kotlin will infer it by itself. It also makes use of it which is a special variable which refers to the first argument of the lambda.

The syntax gets even nicer when you start calling functions which take a lambda with one argument as its last parameter. Take the .also method which you can call on Any object:

val str = "something".also((s) -> {
// simplify using 'it' to refer to the parameter
val str = "something".also({
// simplify by removing parenthesis as last argument is a lambda
val str = "something".also {

Such syntax enables the use of some really nice builder API’s. Take kotlinx.html with code taken from the ktor-samples:

h2 { +"Login" }
form(call.url(Login()) { parameters.clear() }, classes = "pure-form-stacked", method = FormMethod.post) {
    acceptCharset = "utf-8"
    label {
        +"Username: "
        textInput {
            name = Login::userName.name
            value = it.userName
    label {
        +"Password: "
        passwordInput {
            name = Login::password.name
    submitInput(classes = "pure-button pure-button-primary") {
        value = "Login"

Or even simply working with lists. In Java, streams have helped significantly, but in Kotlin it’s still much more expressive and neat (coupled with the addition of new helper extension methods in the standard library):

val numbers = arrayListOf(10, 5, -9, 9, 11, 5, -6)
numbers.filter { it >= 0}
numbers.forEach { println("${it * 2} ") }
numbers.map { it * 2 }.filter { it > 10 }.forEach { println(it) }

The same :: method reference syntax as Java is available in Kotlin to refer to functions within the same/different scopes. Kotlin also introduces the concept of inline functions however (just like also as described above). This reduces any overhead that would otherwise be brought in when passing around method references, as the compiler merely copies and pastes it to the call site. Note that Kotlin also does have similar functional interface classes to Java - KFunction<T> etc.

The improvements made to lambdas in Kotlin make passing around and dealing with them in general much easier and more attractive to developers. You just have to take a look at any popular Kotlin library to see how widely the concept is utilised within the language. I think that much of the improved readability and expressiveness of Kotlin can be in some way attributed to this feature.

Null Safety

Another one of the flagship features of Kotlin. The whole concept of nullability is handled very differently in Kotlin compared to say Java. In Java, any reference variable can be nullable. This allows for a lot of flexibility, but introduces a whole mess of dreaded NullPointerException. This may not seem like a huge issue, but when they make up the highest proportion of Exceptions generated from Java software (by quite some margin), then it’s clear that a better solution is needed.

In Kotlin you have to be explicit. An equivalent variable of type String for example in Kotlin must be given a value at definition. Even if it’s a var it cannot be given a null value. Of course, if you don’t have the concept of null, then you can’t get NullPointerExceptions!

val str:String = "name"
str = null // Compile-time error
str.length // will never generate a null pointer exception

However, having the concept of ‘no value’ is actually quite useful. In Kotlin you can assign null to a variable, but you have to explicitly define that capability in the variable definition, and more importantly you have to perform a null check every time you use the variable. In Java-land the equivalent would be using the Optional<T> class in the standard library.

val str:String? = "name"
str = null // now accepted
str.length // compile-time error as str might be null
str?.length // safe call operator for easy null check

You can define a nullable variable by appending a ? to the end of the variable definition. Now it can accept null as a value, but any time you use that variable you will need to do a null check to prevent exceptions. Kotlin provides some handy concepts to make handling null easier:

str?.length // length if str is not null, otherwise null
val lengthOrZero = str?length ?: 0 // length if str is not null, otherwise zero (elvis operator)
str?.let {
    // use as it.length etc (inlined function with null check)
    // it is smart cast to String not String?

Another killer feature is that Kotlin also performs smart type casting to prevent casting to the non-nullable type:

if(str != null) {
    val length = str.length // str is now String not String?

Default and Named Arguments

Gone are the days of needing to write boilerplate function overloads when you want to provide some default or optional values. Similar to how languages such as Python have done for years, Kotlin allows default values for function arguments and named arguments when calling functions:

fun foo(num: Int, str: String = "12") {
//All 3 are valid
foo(15, "Hey")
foo(str = "Name", num = 45) //Named argument
foo(45) //usage of default argument

In Java the equivalent of the above would be:

void foo(int num, String str) {

void foo(int num) {
    foo(num, "12")

A nice feature to improve clarity and reduce boilerplate. In the example above, the Java version isn’t so bad, but the problem grows exponentially as you add arguments.

This post has again got quite long, but there are even more items I can discuss in a part 3.

Read More

Make HTTP Requests in Kotlin

Updated 09/18 - Add section on the new HTTP Client in JDK 11

These days making HTTP requests in any language is a staple of many common workflows and features. This post will go through a few of the methods in which you can make such requests in Kotlin using some of the great open source libraries available.

JDK 11 HTTP Client API

If you are using the latest JDK release, you can make use of the new built-in HttpClient API which is now modern and fully feature complete. It supports HTTP 2.0 (header compression, server push, multiplexing etc), WebSockets and can be fully asynchronous - which integrates brilliantly with Kotlin coroutines.

Please refer to full post on the API here

In short, you create a new HttpClient and pass HttpRequest objects in (the below example is synchronous). Note that you could easily create some helper/extension functions to make this code much neater:

val client = HttpClient.newBuilder().build();
val request = HttpRequest.newBuilder()
val response = client.send(request, BodyHandlers.ofString());

The API also has support for performing completely asynchronous requests (using non-blocking IO). In this case a CompletableFuture is returned instead of the raw HttpResponse. If you are using coroutines, you can use the CompletionStage.await() extension function defined within the JDK integration library to suspend the current coroutine until the response is available. No need to deal with chaining together callbacks!

suspend fun getData(): String {
    // above code to construct client + request
    val response = client.sendAsync(request, BodyHandlers.ofString());
    return response.await().body() // suspend and return String not a Future

// in some other coroutine (suspend block)
val someData = getData()
process(someData) // just as if you wrote it synchronously


Probably the most commonly used library for this requirement, Fuel is fully featured and stable for any such use case. The default settings used are also good so it requires very little/if any configuration to get up and running.

The base of the library sits on extension functions of String - which in this case represent URL. This makes the interface very fluent any easy to read.

Fuel also makes use of the Result library - written by the same creator - to bundle up error conditions and responses. This does mean another dependency to add, but it makes error handling a bit easier. The recommended method is done through a when expression:

"http://httpbin.org/get".httpGet().responseString { request, response, result ->
  when (result) {
    is Result.Failure -> {
      val ex = result.getException()
    is Result.Success -> {
      val data = result.get()

The underlying requests are performed on a dedicated thread pool making the library capable of both blocking and asynchronous requests.

You can also perform synchronous requests if you want:

val (request, response, result) = "http://httpbin.org/get".httpGet().responseString() // result is Result<String, FuelError>

Take a look at the docs to find examples of how to use authentication, POST etc requests, parameters support, and timeouts etc. The API is configured in a fluent manner:

"http://httpbin.org/get".httpGet().timeout(timeout).timeoutRead(timeoutRead).responseString { request, response, result -> }

Ktor Client

Another approach is to make use of the newer Ktor library. Although the main focus has been on the server-side area, it also includes another package to perform non-blocking requests in a similar fashion. As Ktor is based around Kotlin coroutines, this perhaps makes most sense if you are more familiar/are already using them in your project.

Ktor includes multiple methods of requests. The main being Apache, but CIO (Coroutine IO) and Jetty handlers are also available. Configuration is done through a fluent, builder-like API very similar to that used in the Ktor server packages.

val client = HttpClient(Apache) {
    install(JsonFeature) {
        serializer = GsonSerializer()
val htmlContent = client.get<String>("https://en.wikipedia.org/wiki/Main_Page")

Note that in this case, get is a suspending function so you would have to call it within a coroutine. Using runBlocking or async are the most suitable candidates and means that, unlike Fuel, you have complete control over which thread pool is used for the requests.

suspend fun parallelRequests() {
    val client = HttpClient(Apache)

    // Start two requests asynchronously.
    val req1 = async { client.call("").response.readBytes() }
    val req2 = async { client.call("").response.readBytes() }

    // Get the request contents without blocking threads, but suspending the function until both
    // requests are done.
    val bytes1 = req1.await() // Suspension point.
    val bytes2 = req2.await() // Suspension point.

The Apache engine is based on Apache HTTPComponents and supports the widest variety of config options. It is also the only engine to support redirects and HTTP/2. It will bring in apache as a dependency though. The CIO engine is more basic but has no extra dependencies.

Much like you would expect from any HTTP library, you can configure cookies, authentication, timeouts etc as needed.

Native URL

If you don’t want to use a dedicated library, don’t want to do any custom configuration, then Kotlin includes a nice extension method on the URL class to perform GET requests via opening a stream.

val response = try {
                .use { it.readText() }

Other notable mentions:

khttp library

natively using HttpURLConnection as you would in Java

Read More

Favourite Kotlin Features

Kotlin has many great language features which, in my view, put it way ahead of Java in terms of expressiveness and overall productivity. Here are a few of the notable examples:

Full Java Interop

This is probably the key factor to widespread adoption of Kotlin - and something that most new languages struggle with when trying to gain usage. One of the major design decisions of Kotlin is that it has full compatibility with Java. This isn’t like languages such as Scala or Groovy, which although compile down to the same bytecode and run on the JVM, cannot interact with Java code or make use of libraries written in Java.

When you start a Kotlin project, IntelliJ also probably created a similar Java source folder in the root of your project. This signifies a key point in that you can have both Kotlin and Java code right in the same project, talking to each other and compiling at the same time. No fancy tooling needed, it just works as you would expect. Kotlin supports or has variants for every Java language feature meaning there is full interop between the two.

For example take the following Java class:

public class Foo {
    public void sayInJava(String something) {
        System.out.println("Said in Java: " + something);

Maybe this is in a library .jar file somewhere and just referenced in your project. In Kotlin you can call the Java code just like you would your Kotlin code:

fun say() {
    val foo = Foo()
    foo.sayInJava("from Kotlin")

This is cool and all, but the great thing about this is that from day one Kotlin has fantastic 3rd party library support. Everyone knows that there is pretty much a library for everything in Java-land - well you get to use each and every one of those in your Kotlin code for free. This massively lowers the barrier to entry for Kotlin compared to so many other languages when you have extremely stable libraries at your disposal:

  • Apache commons and all the rest of them
  • Guava, collections etc
  • Kafka, ElasticSearch, Hadoop etc
  • Spring Framework (which now has first class Kotlin support)
  • Hibernate, JooQ
  • Logback, Log4j, AssertJ, Hamcrest
  • The list goes on, and this doesn’t even include Kotlin specific libraries!

This feature also means that you don’t have to go head first into Kotlin for new or existing projects. There are a lot of developers on larger systems who have begun migrating over data classes and tests to Kotlin, leaving the core business logic in Java. That’s perfectly feasible and probably recommended to keep it slow and appease those who are less than emphatic about using a new language.

Data Classes

Probably one of the headline features in Kotlin - and it really does save you so much boilerplate. In Java, writing basic POJO classes is easy, but requires tons of boilerplate, even if the IDE can generate it for you. It litters up your codebase and makes your classes unnecessarily long and unreadable. For example a basic data class in Java could look like:

public class Person {
    private String name;
    private int age;

    public Person(String name, int age) {
        this.name = name;
        this.age = age;

    public String getName() {
        return name;

    // other getters and setters

    // equals and hashcode generated

    // generated toString()

    // clone/copy method

That’s a massive amount of code just to define a simple Person with a couple attributes. In Kotlin however, we can do it in one line:

data class Person(val name: String, val age: Int)

This one line is equivalent to all that code above. The data keyword will instruct the compiler to generate the following for all attributes:

  • constructor to set all attributes (in this case they are val so are immutable)
  • getters (optional setters if var) for all attributes - implemented as Kotlin properties
  • equals and hashcode methods
  • toString method
  • copy() method similar to cloning in Java

Extension Functions

This is another key feature of Kotlin that puts it above a lot of the competition. Similarly to C# (just with additional benefits) you can add your own methods and properties to classes without modifying their actual definition. The most common case is where you don’t have access to the source of the target class (e.g String or Int), but want to keep the nicer syntax of operating directly on the object. For example, take the method countMatches of Apache Commons Lang:


int count = StringUtils.countMatches("some string", "s");


In Kotlin we can define an extension function which simply delegates down:

fun String.countMatches(s: String) = StringUtils.countMatches(this, s)
// then
val count = "some string".countMatches("s")

A simple example, but this concept gets used everywhere within the Kotlin standard library e.g:

public fun CharSequence.lineSequence(): Sequence<String> = splitToSequence("\r\n", "\n", "\r")

public fun CharSequence.lastIndexOfAny(strings: Collection<String>, startIndex: Int, ignoreCase: Boolean): Int = findAnyOf(strings, startIndex, ignoreCase, last = true)?.first ?: -1

Extension functions in Kotlin are really nothing more than compiler magic. If you look at the bytecode level, they are nothing more than static functions which take this as a first param (really just like every other method). The Kotlin compiler just allows us to use this nicer syntax in order to call such functions. Or the really handy one to convert a Java Stream to a list without going through the verbosity of .collect(Collectors.toList()):

public fun <T> Stream<T>.toList(): List<T> = collect(Collectors.toList<T>())

If that’s not a reason to like extension functions, I don’t know what is.

This is similar behaviour to how extension functions work in C#, but in Kotlin there is another way they can be used - as parameters to functions. Take the commonly used apply function as defined below. This can operate on any T (any object) and takes in one function as a parameter. This is no normal function though - it’s an extension function of T (the class it operates on).

fun <T> T.apply(block: T.() -> Unit): T {
    return this

This has a very cool advantage in that the scope of the function is local to the class - e.g you have access to all the members:

val person = Person().apply {
    firstName = "Bill"
    secondName = "Smith"
    age = calculateAge()

A somewhat simple example, but it provides a really nice syntax for the builder pattern as described in this article. And it gets used all over the place in newer Kotlin libraries. Take a route definition in Ktor for example (take a look at this post on how to get started with Ktor):

route("/widget") {
	get("/") {

The route function itself is an extension function of the Route class and so we have access to other builders on top of the Route - in this case get which adds an interceptor. get itself takes in an extension function as a lambda which is how it is able to get a handle on the call variable amongst others.

String Templating

A simple one, but very useful. Gone are the days of concatenating a bunch of small strings and variables together using + in Java, instead we can just use a template:


System.out.println("A = " + a + ", B = " + b + "C = " + c)


println("A = $a, B = $b, C = $c")

You can also directly call methods on the referenced variables:

println("Result is ${res.getResult()}")

And More

This post as gotten quite long so there are still plenty of other points I want to mention in a part 2.

Read More

Logging in Kotlin

If you’re coming into Kotlin from Java-land, you’re probably very familiar with the below statement which creates a static reference to a SLF4J logger for the current class:

private static final logger = LoggerFactory.getLogger(Something.class);

Even though you can of course use SLF4J in Kotlin, you can’t use the above statement to create one due to the way that Kotlin handles static. Here are a couple options and workarounds for logging in Kotlin:

Instance Variable

class Something {
    val logger = LoggerFactory.getLogger(Something::class.java)

This will work, but doesn’t look like idiomatic Kotlin to me. For starters, you are referencing both the Logger and LoggerFactory alongside the Java version of the target class. There is also the bigger problem of the fact that this will create a new logger for each instance of Something - not very efficient at all in certain cases.

Companion Object

To clean this up a bit we can make use of companion objects in Kotlin to get the same static behaviour:

class Something {
    companion object {
        val logger = LoggerFactory.getLogger(Something::class.java)

This will get you most of the way there, but we can do better by extracting out a factory function to create the logger for us:

Factory Function

inline fun <reified T> loggerFor(): Logger = LoggerFactory.getLogger(T::class.java)


class Something {
    companion object {
        val logger = loggerFor<Something>()

In my opinion this isn’t that great either as I have to generate and use a companion object for every class that just needs logging. That’s even more verbose than Java! There is also the issue that companion objects are actually really quite heavyweight constructs - nowhere near as clear-cut as a simple static variable in Java. This great article explains how they are working behind the scenes.

Extension Function

If you want, you can also get around having to pass in the class name as the generic parameter using an extension function:

inline fun <reified T> T.logger(): Logger { return LoggerFactory.getLogger(T::class.java) } 


class Something {
    companion object {
        val logger = logger()

The bad news with this approach is that in your logs the source class will actually show up as the companion object, not the actual class itself. You can around this with some logic in the factory function:

Removing Companion Object

inline fun <reified T> T.logger(): Logger {
    if (T::class.isCompanion) {
        return LoggerFactory.getLogger(T::class.java.enclosingClass)
    return LoggerFactory.getLogger(T::class.java)

Top-Level Variable - My Favourite

My personal favourite is to define the logger as a private top level variable within the same file as the target class. You can make use of the above factory function to make it very simple without having to worry about companion objects at all:

private val logger = loggerFor<DatabaseFactory>()

class DatabaseFactory {
    // use in here

You’re not making mess in the global scope because the logger is private to this file, and no companion object!

Kotlin-Logging Library

If none of these approaches suit your needs, there is a dedicated logging framework for Kotlin which handles this in a neat approach:

private val logger = KotlinLogging.logger {} 
class Something {
    // use in here

This library makes clever use of a lambda as a way to get a handle on the enclosing class name. It also does a similar thing as described above to remove the companion class when needed. Personally however, I see little benefits to introducing yet another logging library for these benefits (it also provides some Kotlin friendly features around lambdas and lazy logging). With this I would end up with SLF4J, Logback/Log4J and now kotlin-logging.

Why no static keyword?

No idea. It’s one of the very few annoyances that I have with Kotlin in that it’s way too difficult to define static variables. Companion objects are there, but that’s a lot of boilerplate and overhead when you just want a simple static variable like a logger.

There is also the @JVMStatic issue, as variables inside companion objects aren’t actually static by default as seen by the JVM. They just act that way due to their singleton nature.

I wish JetBrains would either add a proper static keyword or make it easier to define companion variables without a complete object. Something like:

companion val logger = loggerFor<Something>()

Which could automatically translate into a companion object with the variable inside. It’s not perfect, but saves me some boilerplate.

Read More