Ryan Harrison My blog, portfolio and technology related ramblings

Logging in Kotlin

If you’re coming into Kotlin from Java-land, you’re probably very familiar with the below statement which creates a static reference to a SLF4J logger for the current class:

private static final logger = LoggerFactory.getLogger(Something.class);

Even though you can of course use SLF4J in Kotlin, you can’t use the above statement to create one due to the way that Kotlin handles static. Here are a couple options and workarounds for logging in Kotlin:

Instance Variable

class Something {
    val logger = LoggerFactory.getLogger(Something::class.java)
}

This will work, but doesn’t look like idiomatic Kotlin to me. For starters, you are referencing both the Logger and LoggerFactory alongside the Java version of the target class. There is also the bigger problem of the fact that this will create a new logger for each instance of Something - not very efficient at all in certain cases.

Companion Object

To clean this up a bit we can make use of companion objects in Kotlin to get the same static behaviour:

class Something {
    companion object {
        val logger = LoggerFactory.getLogger(Something::class.java)
    }
}

This will get you most of the way there, but we can do better by extracting out a factory function to create the logger for us:

Factory Function

inline fun <reified T> loggerFor(): Logger = LoggerFactory.getLogger(T::class.java)

Usage

class Something {
    companion object {
        val logger = loggerFor<Something>()
    }
}

In my opinion this isn’t that great either as I have to generate and use a companion object for every class that just needs logging. That’s even more verbose than Java! There is also the issue that companion objects are actually really quite heavyweight constructs - nowhere near as clear-cut as a simple static variable in Java. This great article explains how they are working behind the scenes.

Extension Function

If you want, you can also get around having to pass in the class name as the generic parameter using an extension function:

inline fun <reified T> T.logger(): Logger { return LoggerFactory.getLogger(T::class.java) } 

Usage

class Something {
    companion object {
        val logger = logger()
    }
}

The bad news with this approach is that in your logs the source class will actually show up as the companion object, not the actual class itself. You can around this with some logic in the factory function:

Removing Companion Object

inline fun <reified T> T.logger(): Logger {
    if (T::class.isCompanion) {
        return LoggerFactory.getLogger(T::class.java.enclosingClass)
    }
    return LoggerFactory.getLogger(T::class.java)
}

Top-Level Variable - My Favourite

My personal favourite is to define the logger as a private top level variable within the same file as the target class. You can make use of the above factory function to make it very simple without having to worry about companion objects at all:

private val logger = loggerFor<DatabaseFactory>()

class DatabaseFactory {
    // use in here
}

You’re not making mess in the global scope because the logger is private to this file, and no companion object!

Kotlin-Logging Library

If none of these approaches suit your needs, there is a dedicated logging framework for Kotlin which handles this in a neat approach:

private val logger = KotlinLogging.logger {} 
class Something {
    // use in here
}

This library makes clever use of a lambda as a way to get a handle on the enclosing class name. It also does a similar thing as described above to remove the companion class when needed. Personally however, I see little benefits to introducing yet another logging library for these benefits (it also provides some Kotlin friendly features around lambdas and lazy logging). With this I would end up with SLF4J, Logback/Log4J and now kotlin-logging.

Why no static keyword?

No idea. It’s one of the very few annoyances that I have with Kotlin in that it’s way too difficult to define static variables. Companion objects are there, but that’s a lot of boilerplate and overhead when you just want a simple static variable like a logger.

There is also the @JVMStatic issue, as variables inside companion objects aren’t actually static by default as seen by the JVM. They just act that way due to their singleton nature.

I wish JetBrains would either add a proper static keyword or make it easier to define companion variables without a complete object. Something like:

companion val logger = loggerFor<Something>()

Which could automatically translate into a companion object with the variable inside. It’s not perfect, but saves me some boilerplate.

Read More

Faster Java Startup Times

Although Java is considered to be very performant in general, the startup times aren’t particularly great. All things taken into account, this isn’t particularly surprising when you factor in everything that happens when you start a Java program - VM creation, class loading, JIT compilation etc.

For small projects or utility programs this isn’t too much of a problem, but for larger projects with a very big classpath, this can become quite annoying during development. For Spring Boot apps in particular, even though the development process is very quick, startup times can become large when you start adding a lot of dependencies.

VM Arguments

Here are a couple of VM args which you can add to speed up startup times.

NOTE - This is for development only to improve app restarts etc. This is absolutely not recommended in production environments

-Xverify:none

This option disables JVM bytecode verification during startup and is perhaps the most significant source of improvement. When enabled, the JVM will check your code for certain dangerous and disallowed behaviour. Obviously, by not performing such checks, the JVM startup time will be improved. Again, you should definitely not do this in a production environment, even if you trust all the code you are running.

-XX:TieredStopAtLevel=1

This option tells the JVM to stop optimising your code after the first level. The JVM has three tiers of intermediate compilation. Here we are preventing any further incremental compilation after the first level, meaning that your system has slightly more resources to use elsewhere rather than optimising your code. Again, this could potentially decrease performance so isn’t recommended in production. You can read more about compilation in HotSpot in this presentation deck.

I created a simple Spring Boot app with an in-memory H2 database and a couple REST endpoints to test the difference in startup times with and without these JVM args (not that this project includes a lot of extra dependencies such as actuator, JPA, caching to reflect a standard Spring App with a large classpath):

Startup Mode Startup Time
Run inside IDE with no custom arguments 9.8s
Run inside IDE with -Xverify:none -XX:TieredStopAtLevel=1 6.2s
  +37%

That’s quite a massive 37% improvement in startup times just with some simple VM args!

Spring Boot Specifics

Aside from these magic VM args, there are a few things to take into account when building standard Spring applications that really impact startup performance:

  • Spring projects are generally a magnet for massive classpaths. Look at your dependency list in a Spring Boot project and there are probably hundreds of .jar files and thousands of classes to load. No wonder why disabling bytecode verification can improve times so dramatically. Try to limit the amount of libraries you are bringing in and think carefully about whether you need another dependency before you just adding it to your project. Try not to end up like a Node project where half the internet is in your node_modules directory.
  • Component scanning can be slow. This is one killer feature of the Spring framework, but as I mentioned above, if your classpath is massive then it’s going to take time to run through all those classes. Try to limit the packages that are scanned or maybe consider splitting your project up. You can also make use of the @Lazy annotation to prevent @Beans from being immediately created/loaded.
  • Doing database migrations at startup. No surprise that this is going to cause some slowdown. If you are using Liquibase for migrations, the JHipster project has a nice AsyncSpringLiquibase class that prevents blocking whilst migrations are taking place.
  • Initialising caches. This may significantly improve performance when the app is live, but when you have a lot of Ehcaches it can take a whilst to init them all at startup. In your local environment consider turning some of them off, or doing this lazily.
Read More

RESTful Kotlin with Ktor and Exposed

Updated for Ktor 1.0 and stable coroutines in Kotlin 1.3+

I’ve been writing a lot more Kotlin recently and have been really liking the language so far. I’ll probably write another post pointing out some of my favourite features, but in short it’s basically Java, but without all the annoying stuff. I think in terms of adoption it’s still very early days for Kotlin, but due to the great interop with Java and being an official language for Android development, I wouldn’t be surprised if it doesn’t start to become extremely popular over the next few years.

Kotlin is pretty versatile, even though most people no doubt focus on the Android side of things. That doesn’t mean however that server side development isn’t also supported - in fact, quite the opposite. The Spring framework already has built-in support for Kotlin and many other libraries are also focusing attention on it. You could use such Java focused libraries, but instead you could use the dedicated Kotlin libraries - some of which are supported by JetBrains themselves.

All the code for the following example is available in the GitHub project kotlin-ktor-exposed-starter.

Create a Kotlin Project

First things first on our way to creating a barebones REST server in Kotlin. Open up IntelliJ and create a new Kotlin project. This will create the basic file structure and a gradle build file. To make sure everything is working, you can run the basic Hello World:

fun main(args: Array<String>) {
    println("Hello World!")
}

Setting up Ktor Async Web Framework

Ktor is great library for creating simplistic and lightweight web services in Kotlin. It’s completely asynchronous through the use of coroutines and as such should scale very well with load. It’s still in active development so might have some rough edges, but on the whole it seems solid. The documentation is somewhat lacking, but has improved significantly. Also note that it’s not at v1.0 yet so the API is no doubt subject to change.

Add the following to build.gradle to add ktor as a dependency and allow the use of kotlin coroutines (an experimental feature as of writing). We are also using Jackson as out library of choice for JSON conversions (GSON support is also available).

repositories {
    mavenCentral()
    maven { url "https://dl.bintray.com/kotlin/kotlinx" }
    maven { url "https://dl.bintray.com/kotlin/ktor" }
}

dependencies {
    compile "io.ktor:ktor-server-netty:$ktor_version"
    compile "io.ktor:ktor-jackson:$ktor_version"
}

In this case, we’re making use of the Netty application server, although servlet based options are also available (although not sure why you would want to sacrifice async).

Create a Ktor application

The main configuration for a ktor app (module) is very straightforward:

fun Application.module() {
    install(DefaultHeaders)
    install(CallLogging)
    install(ContentNegotiation) {
        jackson {
            configure(SerializationFeature.INDENT_OUTPUT, true)
        }
    }

    install(Routing) {
        widget(WidgetService())
    }
}

fun main(args: Array<String>) {
    embeddedServer(Netty, 8080, module = Application::module).start()
}

Here a new Ktor module is created. Ktor is configured around the concept of features which can be installed into the main request pipeline. In this example we add features to add default headers to all our responses, log our calls for debugging and also configure processing of JSON requests and conversion to responses. Finally, the main Routing feature is used to designate which paths to handle in our app. We defer to another extension method defined elsewhere to define the routes for a widget RESTful service. To run the application, the main method starts a Netty server pointing to the module we just created.

Of course for anything to actually happen, we need to define the widget routes and service.

Defining Routes

Here is the definition of the widget extension method which defines the interface for our service:

fun Route.widget(widgetService: WidgetService) {

    route("/widget") {

        get("/") {
            call.respond(widgetService.getAllWidgets())
        }

        get("/{id}") {
            val widget = widgetService.getWidget(call.parameters["id"]?.toInt()!!)
            if (widget == null) call.respond(HttpStatusCode.NotFound)
            else call.respond(widget)
        }

        post("/") {
            val widget = call.receive<NewWidget>()
            call.respond(widgetService.addWidget(widget))
        }

        put("/") {
            val widget = call.receive<NewWidget>()
            call.respond(widgetService.updateWidget(widget))
        }

        delete("/{id}") {
            val removed = widgetService.deleteWidget(call.parameters["id"]?.toInt()!!)
            if (removed) call.respond(HttpStatusCode.OK)
            else call.respond(HttpStatusCode.NotFound)
        }

    }
}

As you can see the Ktor DSL is very intuitive thanks mainly to extension methods and lambda parameter syntax in Kotlin. The basic HTTP method are defined for dealing with widgets - each of which defer to our service which can do all the database access etc.

Note that the post and put methods expect an instance of the NewWidget class (as converted via JSON). This is defined as a Kotlin data class for a widget instance with an optional id:

data class NewWidget(
        val id: Int?,
        val name: String,
        val quantity: Int,
        val dateCreated: Long
)

As we have set up Jackson before, we can just return a basic kotlin data object and it will be converted to json without any additional work from us. Finally, we need to create the widget service to handle the logic of saving and retrieving our model.

Setting up Exposed

Exposed is another JetBrains sponsored library for database interactions in Kotlin. It is a kind of ORM, but unlike Hibernate it’s very simple and lightweight. In this post we’re going to use H2 as a simple in-memory database and HikariCP for connection pooling. Add the following dependencies:

compile "com.h2database:h2:$h2_version"
compile "org.jetbrains.exposed:exposed:$exposed_version"
compile 'com.zaxxer:HikariCP:2.7.8'

Exposed has two ways of interacting with databases - their DSL and DAO. In this post I focus only on the DSL (sql builder) as I think that’s where the library excels. The DAO syntax is nice, but introduces complexity when dealing with web frameworks as you have to convert to your own model class manually. The following defines a Table for widgets:

object Widgets : Table() {
    val id = integer("id").primaryKey().autoIncrement()
    val name = varchar("name", 255)
    val quantity = integer("quantity")
    val dateCreated = long("dateCreated")
}

It’s quite straightforward, we just define our columns as fields and use the fluent column builder to define attributes. We can then make use of the Widgets object application wide to query the table.

Connection Pooling and Database Threads

Now we can setup a connection pool for database interaction. This example uses HikariCP as it’s the most widely used library for this at the moment:

private fun hikari(): HikariDataSource {
    val config = HikariConfig()
    config.driverClassName = "org.h2.Driver"
    config.jdbcUrl = "jdbc:h2:mem:test"
    config.maximumPoolSize = 3
    config.isAutoCommit = false
    config.transactionIsolation = "TRANSACTION_REPEATABLE_READ"
    config.validate()
    return HikariDataSource(config)
}

Now we can tell Exposed to connect to our H2 db and create the widgets table:

Database.connect(hikari())
transaction {
    create(Widgets)
}

A key thing to note when dealing with the async world is that you really don’t want to block any of the threads that are handling web requests. Unlike the standard servlet model where each request is tied to a thread, when you block in an async app you are essentially also blocking any other work from being done. If you do this a lot or have a spike in load, your app will grind to a halt.

This presents a problem when using standard JDBC to query our database because the framework is inherently blocking and so our threads will cease when waiting for results sets. To get around this, we must do our database queries on a dedicated thread pool. This is only really possible through coroutines which can suspend and resume as needed. The flow will be:

  1. Coroutine A starts to handle main web request from user
  2. Database query needed so another coroutine B is starting on another thread pool to perform this blocking operation
  3. A suspends execution until B is finished. Due to the nature of coroutines, the underlying thread is then free to perform other work (handling other requests)
  4. Background coroutine B finishes after database query. Thread is returned to the thread pool for other queries etc
  5. A resumes execution by restoring the previous state it had before suspension. It now has access to the query results which can be passed back as the response. Note that the coroutine A may now be executing on a different thread than in step 1 (pretty cool right?)

This might sound like a lot of work (and it is), but thanks to the coroutines library in Kotlin, this is thankfully very easy to accomplish. The following helper method, which is used across all database interaction in our service class, runs a block of code inside a transaction in this new coroutine. Dispatchers.IO references a thread pool managed by Kotlin coroutines that is meant for blocking IO operations like these. Once called, this function will suspend the current coroutine and launch a new one on the special IO thread pool - which will then block whilst the database transaction is performed. When the result is ready, the coroutine is resumed and returned to the initial caller.

suspend fun <T> dbQuery(block: () -> T): T =
    withContext(Dispatchers.IO) {
        transaction { block() }
    }
}

The method is marked as suspend which will allow the suspension of A described in step 3.

Database Queries with Exposed

Finally, we need to define the WidgetService which will be making use of the database we just set up. The whole code is available in the GitHub project, but here is the method to retrieve a specific widget:

suspend fun getWidget(id: Int): Widget? = dbQuery {
        Widgets.select {
            (Widgets.id eq id)
        }.mapNotNull { toWidget(it) }
                .singleOrNull()
    }

As you can see we make use of the dbQuery helper to perform our query. The Exposed DSL for queries is nice and easy to read. The result of the select is a ResultRow, so I define a helper to perform the mapping to our model class:

private fun toWidget(row: ResultRow): Widget =
            Widget(
                    id = row[Widgets.id],
                    name = row[Widgets.name],
                    quantity = row[Widgets.quantity],
                    dateCreated = row[Widgets.dateCreated]
            )

Something like Hibernate (or the DAO in Exposed) would do this automatically, but Exposed is just a lightweight wrapper around the sql so we have full control of what’s happening. Here is the method to add and delete a widget - again fairly intuitive to read:

suspend fun addWidget(widget: NewWidget): Widget {
        var key: Int? = 0
        dbQuery {
            key = Widgets.insert({
                it[name] = widget.name
                it[quantity] = widget.quantity
                it[dateCreated] = System.currentTimeMillis()
            }) get Widgets.id
        }
        return getWidget(key!!)!!
    }


suspend fun deleteWidget(id: Int): Boolean = dbQuery {
        Widgets.deleteWhere { Widgets.id eq id } > 0
    }

And that’s it! Pretty straightforward in terms of lines of code to create a REST server with database interaction. Start the app as you would any other program (no need to deploy to any app server) and test out the widget routes.

The full example is available in the GitHub project kotlin-ktor-exposed-starter.

Read More

Programs to install on a New Build

Below is a list of all the software I tend to install straight away on a new build (or simply when reinstalling Windows from time to time). I am big into keeping what you have installed at any point to an absolute minimum - mainly to prevent general slowdown over time, so this list isn’t that long. These can however get pretty much anything I need done, even if extra utilities are needed later on.

Browsers: the core of your computer these days

Chrome - my main browser and has been for quite a while now. Sure it’s a massive resource hog, but what’s the point in having RAM if it’s sitting idle? Still probably the fastest browser around and the most popular.

Firefox - mainly installed as a backup which gets used every so often. The new Firefox Quantum update has improved the situation dramatically and maybe I’ll try it as my main driver if Google screws things up.

Browser Extensions: pretty much mandatory if you want any kind of sane browsing experience

uBlock Origin - ad/tracker blocker. A must have (or alternative). The web sucks these days without it. My soul dies a little inside every time I have to use a browser without some kind of adblocking - we’ve really screwed up the internet with the mountain of Javascript, popups and auto-playing videos plaguing every site.

LastPass - if you aren’t using a password manager of some kind, I recommend you revisit that decision. LastPass and their extension have been working great for me.

Google Mail Checker - displays an icon in the toolbar linking directly to your GMail account, also shows the number of unread messages.

JSONView - if you ever look at JSON in Chrome, this is a must to get some nice formatting.

Again, I like to keep this list to a minimum as Chrome starts to slowdown and consume even more resources the more you have. If you do need loads, I recommend disabling them until you need to actually use them.

Text Editors: for when you want to edit some text

Notepad++ - small, fast and feature rich replacement to the standard Windows Notepad. Great for any light text editing that doesn’t require a full-blown editor/IDE.

Visual Studio Code - probably the best editor around now after pretty much wiping the floor with Atom and Sublime. The amount of updates each month is insane and the extensions are very mature at this point. See here for the extensions I use.

Dev: tools and IDE of choice

Git - because you wouldn’t version control any differently these days now would you?

JDK - I mainly develop on the JVM (which whatever you think of Java is a great piece of tech). P.S - Kotlin is awesome.

Intellij - one IDE to rule them all. Does everything in every language, what can I say?

WSL (Windows SubSystem for Linux) - Ubuntu install for Windows for various utils.

Node - because apparently I need some way to install 3 thousand packages for a ‘Hello World’ webapp.

Media: for when you want to not do anything productive

K-Lite Codec Pack (MPC-HC) - can play pretty much anything you can ever come across and bundles in Media Player Classic which is my favourite media player.

Spotify - not much to say, does the job and I haven’t seen any need to try out any other service.

IrfanView - the built-in Windows 10 Photos app is absolutely terrible in every way imaginable.

Networking - connecting

FileZilla - SFTP client although not really needed anymore as WSL and rsync are a thing. Still small/lightweight enough to keep around.

Postman - great program to create and send HTTP requests. The de facto choice at this point for testing web services.

Private Internet Access - current VPN provider. Never had any problems with it, speeds are good and the client is solid.

PuTTY - still solid as ever even if you can use WSL for ssh now.

Games: launchers for the actual games

Steam - not much more to say about this. If it’s not on Steam I probably don’t want to play it.

Origin - because Battlefield is sadly not on Steam.

Monitoring: because you need to keep an eye on those temps after you overclock

HWMonitor (portable) - simple, lightweight and easy to read measurements across your system.

HwInfo64 - a more heavyweight alternative to HWMonitor, the number of readings it gives is comprehensive to say the least.

Misc: random tools and utilities

F.lux - remove blue light from your life.

CCleaner - still hanging around, runs every so often to delete temp files.

WinRar - yes the interface is outdated, but I only ever use the explorer context menu items. For me a staple for many years.

Again, this is just the barebones list that I tend to immediately install on a new install of Windows. Things tend to accumulate over time, but still try to keep it to a minimum.

Read More

Ubuntu Server Setup Part 2 - Secure Login

Before reading this, make sure to go over part 1 which covers initial login and setting up a new user.

In the previous section we covered logging into the server with the root user. At that point we were using a simple password, which is less than ideal. In this part we will be setting up public key authentication for the new user in order to better secure our logins. Login to the root user will be disabled via ssh as well, forcing you to go through your newly created user and use sudo commands to get root access.

Generating an RSA public/private keypair

Using Windows you can use the free PuTTygen utility which is bundled in with PuTTY.

Open the app and select SSH-2 RSA under the Key menu. Then hit Generate and provide some mouse movements to generate some randomness.

PuTTYgen RSA

The top textbox will contain the newly generated public key which will be deployed onto the remote server. Save both the public key and private keys in a safe place. Remember you never want to give anyone/anything your private key.

The utility will save the private key in the .ppk format which PuTTY can understand. You can choose to export into the more generic OpenSSH format if needed (e.g to use with the ssh command under WSL).

Copy the contents of the top textbox into the clipboard as this will be what will be saved into the remote server in order to authorise you.

If you are using Linux you can use the ssh-keygen command instead to generate the keys.

Installing the public key

Login to the remote server under the new user you wish to secure (currently using a password although we will now change that).

If you are still the root user run su - <user>

In the home directory create a new .ssh directory which will house the public key.

$ mkdir ~/.ssh

Change the permissions to ensure that only the user can read or write to the directory.

$ chmod 700 ~/.ssh

Create a new file called authorized_keys and open using the nano editor

$ nano .ssh/authorized_keys

Paste your public key into this file. Ctrl+X and then Y to save and exit

Change the permissions on the new key file so again only the current user can read or write to it.

$ chmod 600 ~/.ssh/authorized_keys

Login using public key authentication

Now the public key is installed onto the server and you have the corresponding private key on your local machine, it’s time to login using them. In PuTTY, go to the Connection -> Data -> Auth tab and navigate to the .ppk private key in the bottom field:

PuTTYgen RSA

If you’re using the ssh command, place the file under ~/.ssh/id_rsa and it will use it automatically. Otherwise you can pass in the path to the private key as you login:

$ ssh -i ~/.ssh/private_key user@server.com

Disable root login

In order to further secure the server, it’s best to prevent direct login to the root user. I have also changed the port for ssh to something other than 22 to prevent a lot of automated attacks and disabled password authentication (forcing you to use public keys).

$ nano /etc/ssh/sshd_config

PermitRootLogin no
Port 23401
PasswordAuthentication no
AllowUsers Fred Wilma

Reload the ssh daemon to reflect the changes

$ sudo systemctl reload sshd

With these settings active, you will be forced into logging in via the Fred or Wilma users (root being disabled) by public key authentication on port 23401.

Read More