Ryan Harrison My blog, portfolio and technology related ramblings

Typora - A Better Markdown Editor

VS Code Doesn’t Cut It

Since VS Code has come into popularity, I had always used it to write blog posts like this one in markdown. It worked just fine, it’s really just a text editor, but there is autocomplete templates and syntax highlighting etc for markdown files.

For small dev markdown files and documentation this is all that’s really needed, but for longer pieces of text, I found myself wanting something a little better (more like Word I will admit). Even with all the extensions available for markdown and the live preview version you can get in another pane, there are some big holes in the overall experience when you want to get away from the markup. I can’t imagine writing a book etc in VS Code for example, even if markdown is a pretty good option for it.

VS Code markdown editor

The live preview is good, but I never really found myself using it apart from quick checks to see if I hadn’t screwed up the markup and that everything looks the way I intended. For the actual writing part however, my attention was forced on the other pane - the markdown itself where, apart from the syntax highlighting, could really be any generic text editor. You have the nice shiny live version sitting right next to your eye, with all the nice CSS applied, but you can’t really use it much because the actual editing happens elsewhere - shame really.

The other really big hole in VS Code is the lack of good spell/grammar checking. Yes, there are a few extensions available for this, but they don’t hit the mark. One even relies on sending your text to an external web service to report back on spelling errors, seriously. There is one I used that held a local database, but it was far from extensive and you can’t right click on a word to change it. In VS Code all ‘quick fixes’ like this have to go through the lightbulb menu near the gutter - very annoying. I really hope VS Code gets updated to include a good built-in spell/grammar checker.

Typora

In Typora, you basically get to edit directly in the VS Code live preview equivalent. The actual underlying markdown is still there, but is kept behind the scenes and is not a distraction in the main editing experience. Things work pretty much how you would expect in most rich text editors, the significant difference being that Typora is converting everything to markdown for you.

Typora markdown editor

The overall user experience in Typora takes on a minimal and distraction free form. In the left panel, you have your project markdown files, and then you have the great looking preview straight in front of your eye - I think it looks great. The main preview is GitHub like by default, although there are other themes available as well.

You write the actual text the same way as you would do in VS Code, but elements are converted into the final result live as you type. For example, starting off a paragraph with a hash, will look just like you would expect. Give it some content and hit return however, and you have the generated header right there. Same goes for bullet points and images - type the markup and see the results live. It also makes error handling in the markup a lot easier, if the element doesn’t show up you must have done something wrong.

Typora is built as an Electron app, which is bit of shame as you’ll be running yet more Chrome instances and it is far from conservative in terms of download size and memory usage. But to be honest, for desktop development that seems to be the only really option nowadays and it is far from the worst example of an Electron app I have seen.

Oh, and did I mention that it has built in spell check which works the way you would expect!?

Typora spell check

Typora is still in beta and receives constant updates. Plus it’s also still free until it has a full stable release. I would definitely recommend for your markdown needs.

Read More

Ubuntu Server Setup Part 8 - Sending Email Through Gmail

In the previous part we covered how to setup Postfix to receive emails for our custom domain name and forward them onto a personal Gmail account. With that solution, you can get access to all incoming mail via the forwarding, but you have no way of sending mail as owner of your domain.

You could still add the address as a Send Mail As option within Gmail, but your underlying address would still be visible to the receiver. This is also how you see the Sent on Behalf Of message in Outlook etc. Ideally, we want to be able to send email in Gmail, but use our server as an intermediate. This is great because we can still use the Gmail interface and tooling without having to setup a real mailbox (Roundcube etc) on our server.

Securing a Relay

To get the functionality mentioned above, we have to setup Postfix as a relay server (a server that will send e-mails to their destination on behalf of it). You might have heard that relay servers are a really bad idea, and they are, but only if they are open (a.k.a unsecured). In this case, we will be making a relay, but securing it with TLS and a username/password to make sure that all communication between it and Gmail is secured. This will also prevent bad actors from being able to send email on your behalf via your server.

Install Cyrus SASL

We will be using Cyrus SASL as the method of authentication for Postfix. In this case, we will be storing it as a simple (permissioned) database file, however other more sophisticated storage solutions are available such as MySQL and PAM. Install the package using the following command:

$ sudo apt-get install sasl2-bin libsasl2-modules

Create a username and password

Once installed, we can create a username and password combination:

$ sudo saslpasswd2 -c -u yourdomain.com smtp

This will create a database file in the default location /etc/sasldb2 with a single user called smtp (you can use whatever username). You can verify that the user is created properly by running:

$ sudo sasldblistusers2

Make sure that the newly created database file is properly permissioned - in this case only readable by the Postfix user:

$ sudo chmod 400 /etc/sasldb2
$ sudo chown postfix /etc/sasldb2

Create an SSL Certificate

Because all traffic between Gmail and our server will be sent under TLS, we need an SSL certificate. If you already have a certificate (e.g from Let’s Encrypt), you can use that, but otherwise a simple self signed cert works just as well:

$ openssl req -x509 -newkey rsa:4096 -keyout key.pem -out cert.pem -nodes -days 3650

When prompted, enter you domain name yourdomain.com as the Common Name. The cert.pem file is what we are interested in. Makes sure to protect the key file! Now move the generated pem file so Postfix can read it:

$ sudo mv cert.pem /etc/postfix/cert.pem
$ sudo chmod 400 /etc/postfix/cert.pem
$ sudo chown postfix /etc/postfix/cert.pem

Setup Postfix as a Relay Server

/etc/postfix/master.cf

Now we need to change some configuration to setup Postfix as a relay server which can send mail on behalf of another server. Open up the main config file /etc/postfix/master.cf. Uncomment the lines starting with submission and edit them to match the following:

submission inet n       -       n       -       -       smtpd
  -o syslog_name=postfix/submission
  -o smtpd_tls_security_level=encrypt
  -o smtpd_tls_cert_file=/etc/postfix/cert.pem
  -o smtpd_sasl_auth_enable=yes
  -o smtpd_reject_unlisted_recipient=no
  -o smtpd_relay_restrictions=permit_sasl_authenticated,reject
  -o milter_macro_daemon_name=ORIGINATING

Here, we are enabling authentication using SASL and setting up TLS pointing to the new certificate. All traffic to the server must be sent under TLS as to be accepted by the relay. We also specify that we wish to accept relay traffic which is authenticated under SASL, and reject anything else (no open relay here).

/etc/postfix/sasl/smtpd.conf

We also need to tell Cyrus SASL to use the database file we created for authentication. Create the file /etc/postfix/sasl/smtpd.conf and enter the following:

pwcheck_method: auxprop
auxprop_plugin: sasldb
mech_list: PLAIN LOGIN CRAM-MD5 DIGEST-MD5 NTLM
log_level: 7

Once you have made these changes, restart Postfix:

$ sudo service postfix restart

Add Firewall Rule

If everything went well and Postfix started correctly (check the logs if not), Postfix should be listening on port 587 for secured SMTP traffic. You should check the Postfix logs at /var/log/mail.log (or with journalctl -u postfix if using Systemd) if you have any problems. Add a firewall rule to allow traffic through the port:

$ ufw allow 587/tcp

Configure Gmail

Now all the server side configuration is done, time to setup your personal Gmail account to be able to send mail as your domain, using your server as a relay.

Open up Gmail and go to Settings -> Accounts and Import -> Send mail as. Click on the button to Add another email address:

Gmail Send Mail As

In the dialog box that pops up, enter your name and the full email address you wish to assign e.g me@yourdomain.com. Make sure the option to Treat as an alias is checked:

Gmail Add Address

In the next dialog, specify the address of your server and the username and password that was setup with saslpasswd2:

SMTP Server = yourdomain.com
Username = smtp@yourdomain.com (or whatever username you picked) followed by the domain
Password = the password you chose when setting up Cyrus SASL

Make sure that port 587 is selected and the connection is secured under TLS:

Gmail Configure Relay Server

If all went well, Gmail should be able to connect to your server and will send a confirmation email to your new address me@yourdomain.com. Because we setup forwarding in the previous section, this email should appear in your Gmail inbox as well. Open the mail and copy/paste the confirmation code.

Send mail as

Finally, start composing a new email or reply to an existing one and you should be able to select the new mail address me@yourdomain.com in the From dropdown. All done!

Wrap Up

In the last two sections we set up a Postfix email server for own domain name yourdomain.com:

  • All emails sent to me@yourdomain.com (or any listed in the virtual file) on port 25, will be forwarded on to you@gmail.com and be visible in your standard Gmail inbox.
  • Gmail will let you select me@yourdomain.com as the From address when sending or replying to any mail. The message will be relayed onto our Postfix server with TLS on port 587 and then passed on to the destination. Any message sent in this fashion will look to the receiver as though it was sent directly by your domain and your underlying Gmail address will not be visible.
Read More

Kotlin - Things to Improve

This list isn’t very long and doesn’t exactly include any game breaking lack of functionality. A good testament to how Kotlin is a solid language these days.

Try/Multi Catch

A somewhat simple language feature that according to the designers is still on the cards. Not too much of a problem to live without, but Java has had it for years and it should be a staple of any modern language:

try {
    ...
} catch(SomeException | OtherException e) {
    ...
}

The Kotlin version relies on the when construct, which has many more levels of nesting and is generally just more ugly to read and write:

try {
    ...
} catch (e: Exception) {
    when(e) {
         is SomeException, is OtherException -> {...}
         else -> throw e
    }
}

Ternary Operator and Collection Literals

These are both hotly contested, but I personally believe they should be part of Kotlin. All the arguments against the ternary operator seem to revolve around being ‘too easy to abuse’. Yeah right. Kotlin has so many other language features which can be abused already (operator overloading anyone? extension functions anyone?), I just don’t see why it’s such a big deal.

Like it or not, Kotlin is competing against a myriad of other C based languages - all of which have the ternary operator already. Pretty much every developer knows it these days, it should be made available.

Because if in Kotlin is an expression, it is termed as the ‘acceptable alternative’:

val something = if(a < 4) "it's valid" else "not valid"

v.s. the syntax everyone and their mum is familiar with:

val something = a < 4 ? "it's valid" : "not valid"

Not many characters saved, true. However, when the times comes that I need one (yes because it has it’s place), I get angry at having to write the if statement - which is just more clunky to write.

Similarly, another highly wanted feature is collection literal syntax. I realise that this might get a bit complicated due to the whole mutable/immutable lists thing in Kotlin, but there are enough people who want it for a reason. Kotlin is trying to attract developers from the Python/Javascript/Swift worlds, this kind of thing is what will annoy those trying to make the transition.

People are getting on their high horses and spouting the important of ‘language principles’ and not cluttering the language. Your language principles can be as solid as you want, but if nobody uses it, then what’s the point? Developers obviously expect these things - there is data backing it up. Is how I create my lists or perform inline conditionals that much of a big deal to not have both ways of doing it?

Static

I think the whole static and companion deal is a bit of a mess in Kotlin. When writing Kotlin on the JVM, the concept of static is still very much a thing, and in certain places still a necessity. Want to create a JUnit method marked as BeforeAll/BeforeClass? Yeah, it’s just straight up annoying in Kotlin.

To create a simple static method in Kotlin, you have to go through the hassle of creating a companion object, plus specially annotate the function to tell the compiler to actually make it static. In this case, Java is actually less verbose than Kotlin (and by no small margin) and for what gain? They can do better here to appease those on the JVM.

companion object {
    @JvmStatic
    fun actuallyStatic() = println("That was a chore")
}

Now, I don’t need static very often, really only for loggers/testing most of the time etc, but it’s such a pain to do the above that I prefer the other (less efficient) routes of logger per instance of or a top level variable. It seems like a bit of a tacked on feature to solve this kind of problems and ties into some of the efficiency problems discussed below. Static variables/methods are about as fast as it gets, turns out companion objects are the complete opposite.

Efficiency

I don’t actually have any real data to back this up, but I think it’s common knowledge that the Kotlin compiler isn’t the fastest thing ever. To be honest I’m not too surprised, the amount of work it has to do is impressive. Nonetheless, when working on a Kotlin project and going back to Java, the compilation differences are noticeable to say the least. The Kotlin team have, and continue to do a lot of work to improve it though, so I hope it continues to get faster in the future.

Aside from the compiler, I feel the need to rant about generally inefficiencies though. Maybe this is just a reason to moan about we seem to have accepted that it’s a good idea to use lambdas and streams literally everywhere. What happened to the good old for-loop? Things like streams aren’t free. Depending on what you’re doing there can be significant allocation going on and other overhead. Kotlin inline functions do a good job at resolving this, but they aren’t usable everywhere.

There is a great series here which covers some of the hidden costs in using some of Kotlin fancy language features. As discussed above, when you define a simple companion object, the compiler generates a bunch of boilerplate and indirection. I just wanted a simple static variable/function, why do I have to have all this extra stuff (yes, really, multiple new classes get generated for this).

There is a widespread movement towards immutability, which don’t get me wrong, has many benefits. But it also encourages so much inefficiency. Want to increment that one integer inside this object? Let’s copy the whole thing. Hardware continues to improve, yet developers and language designers seem to find a way to add another layer of abstraction to render it moot.

Too many imports

I’ve been doing a fair amount of work with Ktor and Exposed recently, and can’t help but think that extension functions are getting massively abused at this point (already). Don’t get me wrong, they are great and the syntax they allow for is a big selling point, but when you start using some of these libraries, you realise that everything and their dog is a top level extension function. Literally everything.

Take the following snippet which defines a simple Ktor web service for example. The actual logic portion is nice and neat, but at the top we have 22 (!) imports.

import com.fasterxml.jackson.annotation.JsonInclude
import com.fasterxml.jackson.module.kotlin.jacksonObjectMapper
import io.ktor.application.call
import io.ktor.http.HttpStatusCode
import io.ktor.http.cio.websocket.Frame
import io.ktor.request.receive
import io.ktor.request.authorization
import io.ktor.request.receiveMultipart
import io.ktor.response.respond
import io.ktor.response.etag
import io.ktor.response.header
import io.ktor.response.respondFile
import io.ktor.response.respondText
import io.ktor.routing.Route
import io.ktor.routing.delete
import io.ktor.routing.param
import io.ktor.routing.get
import io.ktor.routing.post
import io.ktor.routing.put
import io.ktor.routing.route
import io.ktor.websocket.webSocket
// before any of our actual app imports

...
// implementation at https://github.com/raharrison/kotlin-ktor-exposed-starter/blob/master/src/main/kotlin/web/WidgetResource.kt

Kotlin has import * syntax, which the library makers tell everyone to use, but didn’t we previously agree that this was a bad idea? Something about having to ‘explicitly define your dependencies’ or something? But hey, as long as we get our nice builder syntactic sugar, lets just make IntelliJ fold it up and forget it ever happened.

Read More

Ubuntu Server Setup Part 7 - Email Forwarding with Postfix

One of the best things about owning your own domain name is being able to use it as a custom email address. Many companies provide this service for a monthly fee, but if you have your own personal server/VPS already anyway, then you might as well take advantage of it.

Many custom email solutions are very heavyweight and include setting up Postfix, Dovecot (IMAP and POP3 server) and some client like Roundcube. You also have to worry about spam so will also need something like spamassassin. Not to mention all the security and redundancy issues involved in storing and maintaining your own (sensitive) data. If you want an all-in-one bundle of all the above, check out Mail-in-a-Box.

That’s all well and good for some use cases, but wouldn’t it be nicer if you could use your existing Gmail account to handle email traffic for your custom domain? Integration with all your devices, access to your other email accounts, built in spam detection and you will be managing much less infrastructure.

This part covers the first part of getting to that solution - forwarding all email that comes to your domain onto your existing Gmail account. The second part will then cover the other half of the story which is how to send email as your custom domain from within Gmail.

In short, at the end of this part all email to me@yourdomain.com will be forwarded on to you@gmail.com automatically. It is assumed that you have control over a domain name, can change DNS properties, and have an existing Gmail account.

DNS Setup

The first step is to configure a couple of DNS records to ensure that email traffic gets routed to your server correctly (MX record). The others are to help out in the battle against Gmail thinking that all your mail is spam and for them to correctly validate your server as being in control over your domains mailbox. In whatever DNS control panel your have, create the following:

  • An A record pointing yourdomain.com to your servers IP address
  • An MX record pointing yourdomain.com to the IP of your server (or @ to point to your A record)
  • A PTR (reverse DNS) record pointing your servers IP to yourdomain.com. This allows Gmail to verify the legitimacy of our server via its IP when Gmail receives a forwarded e-mail from it.
  • An SPF record with the contents v=spf1 mx a ip4:<yourip>/32 include:_spf.google.com ~all (replacing with your own IP address)

The SPF record tells Gmail that only the servers specified are allowed to send e-mails purporting to be from yourdomain.com All other servers attempting to do the same will be rejected - which helps a lot with spam.

Install and Configure Postfix

Now to get to the core part - installing Postfix and configuring it to forward all email to your Gmail account.

$ sudo apt-get install postfix

During the installation process, you will be be prompted a couple times. Choose the Internet Site option and provide your domain name into the inputs.

/etc/postfix/main.cf

Once Postfix is installed, open up it’s main configuration file:

$ sudo nano /etc/postfix/main.cf

The file will already contain a bunch of configuration options. In the second part I will provide the full configuration, but in this part just paste in the below. This basically just tells Postfix what our domain name is and sets some network properties.

The most important part is the bottom two lines which sets up alias mappings. This will tell Postfix that for all traffic to yourdomain.com perform a lookup into /etc/postfix/virtual to find out where to forward it on to.

myhostname = yourdomain.com
mydomain = yourdomain.com
myorigin = /etc/mailname
mydestination = yourdomain.com, localhost.localdomain, localhost
relayhost =
mynetworks = 127.0.0.0/8 [::ffff:127.0.0.0]/104 [::1]/128
mailbox_size_limit = 0
recipient_delimiter = +
inet_interfaces = all
inet_protocols = ipv4

virtual_alias_domains = yourdomain.com
virtual_alias_maps = hash:/etc/postfix/virtual

/etc/postfix/virtual

Now to edit the virtual mappings file to tell Postfix to forward our mail onto your Gmail account:

$ sudo nano /etc/postfix/virtual

Within that file, you can provide your domain name, followed by a whitespace and a comma separated list of domains to forward it to. In this case we are forwarding everything which comes to me@yourdomain.com onto your Gmail account. You can just provide yourdomain.com as the first parameter to forward all email regardless of address.

# Forwarding mapping, one from-to address pair per line. The format is:
#     <forward-from-addr> <whitespace> <forward-to-addr>
me@yourdomain.com you@gmail.com

Update lookup table

Postfix doesn’t directly read the virtual file, but instead generates a lookup table from it. Run the following to refresh the lookup table:

$ sudo postmap /etc/postfix/virtual

Reload Postfix

$ sudo service postfix start # if not already running

$ sudo service postfix reload # reload our config files

$ sudo service postfix restart # or perform a restart

Configure Firewall

SMTP traffic (which Postfix is handling and forwarding) runs on port 25, so makes sure to allow it through your firewall. Refer to Part 3 for info on how to set one up.

$ ufw allow 25/tcp

Testing

After any new DNS records have propagated, you can test by sending an email to me@yourdomain. It should be forwarded onto me@gmail.com pretty quickly (make sure to also check your spam folder).

If you tried to the above and never received the email in your Gmail inbox, try checking the Postfix logs at /var/log/mail.log for errors.

That’s it for this part, in the next part we will cover how to send email from Gmail, relaying through our server to appear as though it was sent by our custom domain.

Read More

Java 11 HTTP Client API

One of the more noteworthy features of the new Java 11 release is the new HttpClient API within the standard library, which has been in incubator status since JDK 9. Previously, the process of sending and retrieving data over HTTP has been very cumbersome in Java. Either you went through the hassle of using HttpURLConnection, or you bring in a library to abstract over it such as the HttpClient from Apache. Notably, both solutions would also block threads whilst doing so.

As these days dealing with HTTP connections is so common, JDK 11 finally has a modern API which can deal with these scenarios - including support for HTTP 2 (server push etc) and WebSockets.

Create a Client

The first step to make use of the new API is to create an instance of the HttpClient class. The library itself makes heavy use of the builder pattern to specify configuration options, as is the case in most new Java libraries.

You can configure things like HTTP version support (the default is set to HTTP 2), whether or not to follow redirects, authentication and a proxy for all requests that pass through the client.

HttpClient client = HttpClient.newBuilder()
      .version(Version.HTTP_2)
      .followRedirects(Redirect.SAME_PROTOCOL)
      .authenticator(Authenticator.getDefault())
      .build();

The HttpClient instance is the main entry point to send and receive requests (both synchronously or asynchronously). Once created you can reuse them, but they are also immutable.

Create Requests

An HTTP request is represented by instances of HttpRequest which holds the URL, request method, headers, timeout and payload (if applicable). By default GET is used if no other is specified.

HttpRequest request = HttpRequest.newBuilder()
               .uri(URI.create("https://something.com"))
               .build(); // GET request

The following snippet creates a POST request with a custom timeout. A BodyPublisher must be used to attach a payload to a request - in this case taking a JSON String and converting it into bytes.

HttpRequest request = HttpRequest.newBuilder()
      .uri(URI.create("https://something.com/api"))
      .timeout(Duration.ofMinutes(1))
      .header("Content-Type", "application/json")
      .POST(BodyPublishers.ofString(json)
      .build()

Send Requests

Once an HttpRequest is created, you can send it via the HttpClient previously constructed. Both synchronous and asynchronous operations are supported:

Sync

The HttpClient.send method will perform the HTTP request synchronously - meaning that the current thread will be blocked until a response is obtained. The HttpResponse class encapsulates the response itself including status code, body and headers.

HttpResponse<String> response = client.send(request, BodyHandlers.ofString());
System.out.println(response.statusCode());
System.out.println(response.body());

When receiving responses, a BodyHandler is provided to instruct the client on how to process the response body. The BodyHandlers class includes default handlers for the most common scenarios. ofString will return the body as an UTF-8 encoded String, ofFile accepts a Path to write the response to a file and ofByteArray will give you the raw bytes.

Async

One of the best features of the API is the ability to perform completely asynchronous requests - meaning that no thread is blocked during the process. Under the hoods, the implementation uses NIO and non-blocking channels to ensure no blocking IO operations are performed.

The HttpClient.sendAsync method takes the same parameters as the synchronous version, but returns a CompletableFuture<HttpResponse<T>> instead of just the raw HttpResponse<T>. Just as with any other CompletableFuture, you can chain together callbacks to be executed when the response is available. In this case, the body of the response is extracted and printed out. More details here on how to work with CompletableFuture.

CompletableFuture<HttpResponse<String>> future = client.sendAsync(request,
        BodyHandlers.ofString());

future.thenApply(HttpResponse::body) // retrieve body of response
      .thenAccept(System.out::println); // use body as String

Each HttpClient has a single implementation-specific thread that polls all of its connections. Received data is then passed off to the executor for processing. You can override the Executor on the HttpClient, by default it is a cached thread pool executor.

Kotlin

As the new API sits like any other in the default JDK, you can easily make use of it in your Kotlin projects as well! You can paste in the above examples into IntelliJ to perform the automagical Java-Kotlin conversion, or the below example covers the basics:

val client = HttpClient.newBuilder().build();

val request = HttpRequest.newBuilder()
               .uri(URI.create("https://something.com"))
               .build();

val response = client.send(request, BodyHandlers.ofString());
println(response.statusCode())
println(response.body())

Coroutines (Async)

Things get much more interesting when taking into account the new asynchronous capabilities and Kotlin coroutines. It would be great if we could launch a coroutine which sends the request and suspends until the response is available:

suspend fun getData(): String {
    // above code to construct client + request
    val response = client.sendAsync(request, BodyHandlers.ofString());
    return response.await().body() // suspend and return String not a Future
}

// in some other coroutine (suspend block)
val someData = getData()
process(someData) // just as if you wrote it synchronously

No need to deal with chaining together callbacks onto CompletableFuture, you get the same procedural code flow even though the implementation suspends and is completely non-blocking.

The magic comes from the CompletionStage.await() extension function provided by the coroutines JDK integration library:

return suspendCancellableCoroutine { cont: CancellableContinuation<T> ->
    val consumer = ContinuationConsumer(cont)
    whenComplete(consumer) // attach continuation to CompletionStage
}

Docs for the function.

More Information

https://download.java.net/java/early_access/jdk11/docs/api/java.net.http/java/net/http/package-summary.html

http://openjdk.java.net/groups/net/httpclient/intro.html

https://www.youtube.com/watch?list=PLX8CzqL3ArzXyA_lJzaNmrFqpLOL4aCEz&v=sZSdWq490Vw

Read More