I’ve put together a reference site for engineering best practices, available at guidelines.ryanharrison.co.uk. The goal is a single place to find guidelines and conventions covering the full stack - from core principles through to deployment.

Rather than being prescriptive about specific tools, most sections aim to explain the reasoning behind a recommendation so you can apply it to your own context.

What’s Covered

Principles and practices - Core engineering principles covers the fundamentals: SOLID, DRY, KISS, YAGNI, clean code, and when to apply them. There’s also a code review guide and sections on technical debt, pull requests, and git workflow.

Architecture - Patterns for microservices, event-driven systems, and multi-tenancy.

API design - REST fundamentals and patterns, GraphQL, and OpenAPI contract-first development.

Testing - The testing strategy page covers the overall approach. From there, individual pages go into unit, integration, contract, end-to-end, mutation, and chaos testing.

Security and observability - Security overview covering authentication, authorisation, input validation, and data protection. Observability covers structured logging, metrics, tracing, and alerting.

Languages and frameworks - Guidelines for Java, Kotlin, TypeScript, and Swift, with framework-specific sections for Spring Boot, React, Angular, React Native, Android, and iOS.

Infrastructure - Docker, Kubernetes, Terraform, and a fairly detailed AWS section covering compute, networking, storage, EKS, and more.

This site is built with Jekyll. That unfortunately often means wrestling with Ruby versions, gem dependencies, and environment configuration. Different machines require different setups, and what works on your machine might not work in CI or on your VPS. Docker solves this for many other areas by providing a standalone and reproducible build environment, so why not here as well?

The challenge is that the official Jekyll Docker images are no longer being actively maintained. Thankfully, there’s a decent community alternative that handles modern Jekyll sites without the maintenance burden.

The bretfisher/jekyll-serve image is well-maintained (for now) and works as a drop-in replacement for the official Jekyll images. By default, it serves your site locally via jekyll serve, but you can override the command to run any Jekyll operation you need.

The image handles all the Ruby and gem setup for you, so you don’t need to worry about version conflicts or system dependencies.

Local Development with Docker Compose

For local development, the fastest approach is using Docker Compose. Create a compose.yml file in your Jekyll project root:

services:
  jekyll:
    image: bretfisher/jekyll-serve
    volumes:
      - .:/site
    ports:
      - "4000:4000"

Start your development server with:

docker compose up

Your site will be available at http://localhost:4000 with live reload enabled. The key benefit here is that Docker Compose reuses the same container across runs, which caches your gems. This means subsequent starts are very quick - typically just a few seconds once the gems are installed.

Without compose, you can achieve the same result with:

docker run -p 4000:4000 -v $(pwd):/site bretfisher/jekyll-serve

However, this creates a new container each time, which means reinstalling gems on every run (unless you mess around with more volume mounts, see below).

Building for CI/CD

For continuous integration or deployment builds, you want to build the static site without running the server. Override the default command to run the Jekyll build process:

docker run -v $(pwd):/site bretfisher/jekyll-serve bundle exec jekyll build

This generates your static site into the _site directory. Here’s an example GitHub Actions workflow that builds on every push (also the one used to build this site):

name: Build Jekyll Site
on:
  push:
    branches: [ master ]

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2

      - name: Build with Jekyll
        run: |
          docker run -v $:/site \
            bretfisher/jekyll-serve bundle exec jekyll build

The build is completely reproducible because the Docker image contains a known-good version of Ruby and all the necessary build tools.

Building on Low Memory Machines

If you’re like me and building Jekyll sites on machines with limited memory (1GB or less), you might run into issues. Gem installation often requires building native extensions, which can be memory-intensive. The bretfisher/jekyll-serve image runs bundle install --retry 5 --jobs 20 by default, which parallelizes gem installation, but uses more memory. This caused issues for me on resource-restricted boxes.

You can override the entrypoint to reduce the number of parallel jobs:

docker run -v $(pwd):/site \
  --entrypoint /bin/bash \
  bretfisher/jekyll-serve \
  -c "bundle install --jobs 2 && bundle exec jekyll build"

Reducing --jobs from 20 to 2 significantly reduces memory usage during the gem installation phase. The build will take a bit longer, but it won’t crash on memory-constrained systems.

A junior developer recently came to me for help with one of those classic issues which appears on a remote dev environment, but for whatever reason can’t (at least easily) be replicated locally. Their immediate thought was to add more log statements and redeploy the app to see what’s going on. Not unreasonable, but this is a dev environment, so I connected IntelliJ to one of the running containers and began stepping through the code and inspecting variables. They looked at me thinking I was performing some kind of black magic, so here’s an intro or a quick reminder of something which goes very underappreciated.

Why Remote Debug?

There are a few general scenarios when you might reach for remote debugging:

• Environment-specific bugs - Issues that only appear in staging, testing, or production-like environments with specific configurations, data, or network conditions
• Container debugging - When your application runs inside Docker containers or Kubernetes pods and you need to debug without rebuilding images
• Shared development environments - Debugging applications running on shared development servers or VMs
• Integration testing - Troubleshooting complex integration scenarios with external systems that can’t be replicated locally

Rather than relying solely on log statements or trying to recreate production conditions locally, remote debugging lets you step through the actual running code in the target environment - a lot better than adding log statements!

How It Works

Java remote debugging uses the Java Debug Wire Protocol (JDWP), which is a communication protocol between a debugger and a Java Virtual Machine. The JVM opens a socket that a debugger can connect to, allowing it to control execution, set breakpoints, inspect variables, and evaluate expressions.

The JVM can act as either a server (listening for debugger connections) or a client (connecting to a debugger). In most cases, you’ll configure the JVM as a server and then have your IDE connect to it.

Enabling Remote Debugging

To enable remote debugging, you need to pass specific JVM arguments when starting your Java application. The modern syntax (Java 9+) looks like this:

java -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005 -jar myapp.jar

Here’s a high-level breakdown of what each parameter does:

• -agentlib:jdwp - Loads the JDWP agent library for debugging
• transport=dt_socket - Uses socket transport for the debug connection (the standard approach)
• server=y - Configures the JVM to listen for debugger connections rather than connecting out to a debugger
• suspend=n - Starts the application immediately without waiting for a debugger to attach. Use suspend=y if you need to debug startup code
• address=*:5005 - Binds to all network interfaces on port 5005. You can specify a specific IP address or hostname instead of *

For older Java versions (Java 8 and earlier, so I hope you won’t see this), you might see the older syntax:

java -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=5005 -jar myapp.jar

This accomplishes the same thing but uses deprecated flags. Note that in Java 8, the address parameter only accepts a port number, not the *:port format.

So it turns out that Oracle Cloud (OCI) offers an extremely generous free tier, and not many people seem to know about it. Beyond the 30 day free trial, which gives you $300 worth of free credits to use (not bad at all), they also offer a pretty substantial amount of infrastructure in their Always Free tier as well. In terms of server (VPS) infrastructure, this includes (at the time of writing at least):

• 2 AMD based Compute VMs with 1 vCPU (x86) and 1 GB memory each + 0.48Gbps max network bandwidth
• 4 Arm-based Ampere A1 cores and 24 GB of memory usable as up to 4 VMs + 4GBps max network bandwidth
• 2 Block Volumes Storage - 200 GB total
• 10 GB Object Storage – Standard + 10GB Object Storage Infrequent Acccess + 10GB Archive Storage
• Outbound Data Transfer -10 TB per month

Now, whatever you may think of Oracle in general, you can’t deny that this is a good deal. You can in theory set up a max of 6 VPS instances, all for free, on a commerical cloud environment. Even if the setup process might be a bit awkward compared to other providers, you can’t really complain too much. You would be spending a fairly significant amount on the equivalent amount of infrastruture from elsewhere like AWS or DigitalOcean.

In the rest of this post I will quickly run through the steps to setup a small VPS server running Nginx on the free Oracle Cloud tier. This is a standard VPS, just like you would find anywhere else, running Ubuntu Server 22.04. A Terraform provider is also available if you wanted, but for simplicity I will go through the web console.

Create an “Always Free” Oracle Cloud Account

Go to https://www.oracle.com/uk/cloud/free/#always-free and create an account as usual. You will have to provide credit card information (I presume to prevent misuse), but won’t be charged. For the first 30 days you will be able to play with $300 of credit if you want to, but after that time is up your account will revert automatically to the Always Free tier.

Create an Instance

Once signed in, select the Instances option on the main dashboard (yes the console isn’t the best for navigation). You should be presented with the following screen which will show all active instances. You can see that I’ve created a couple already:

Click on the Create Instance button to create a new VM. This is the standard VPS configuration/settings page:

• Placement - controls which AZ the server is deployed to. Can be kept at default to let OCI choose the best. It should be noted that I’m also creating instances directly in the UK (London) region which is great for latency
• Security - keep as default
• Image and Shape - select the Image and type of server (amount of resources) you wish to deploy:

For this example we will go with Ubuntu Server 22.04 minimal as our OS. It works great on the limited amounts of CPU and RAM on these nodes as it unbundles a lot of of the default packages that you probably don’t need anyway (you can get them back if you need).

For shape, you can choose between AMD (2.0 GHz AMD EPY 7551 x86) vs ARM based (Ampere) CPU’s. You are limited to 1GB RAM max on the AMD shapes (makes sense since they are more expensive), but up to 24GB on the ARM cores. You get a max of 4Gbps of bandwidth on those ARM boxes well, which is very impressive for a free offering (though I haven’t benchmarked what you actually get). Here we will go with anAmpere based VM with 2 vCores and 6GB of memory (did I mention already that all this is free?):

• Networking - can keep these as default to use your default root Virtual cloud network and subnet. Also make sure the option is checked to assign a public IPv4 address to your instance
• Boot Volume - by default you will get a 50GB volume. You can increase this if you want to, up the max of 200GB allowed in the free tier

Create an SSH Key

In the Add SSH Keys section you can choose to automatically generate a keypair, but I prefer to create my own. There are plenty of tools for this, for now I will use PuttyGen to create a new Ed25519 keypair. Save both the private and public key locally for use later as usual. Don’t worry I’m not using this key

Then paste the public key into the corresponding box in the console. Oracle Cloud (OCI) will inject this public key into the .ssh/authorized_keys file for the main ubuntu user on the new VPS instance, thus allowing you to login through SSH.

You can look through the other configuration options as needed, but that should be good enough for now. Press Create and wait a couple minutes for the instance to be created. Startup times seem pretty reasonable on Oracle Cloud.

Update the default Security Group

As you might expect, by default the security group (or security list in Oracle Cloud) will block all traffic coming into your server by default apart from port 22 for SSH. This is good, but as we want to setup an Nginx web server on ours, we need to add a couple new ingress rules.

In the Virtual Cloud Networks section, navigate into your default VCN which was created by default. On the left, select the Security Lists option. There should be a single default entry for the VCN. Here we will add two new Ingress rules for port 80 and port 443. It should look something like the following after the changes:

Login to the Instance

Now all that needs to be done is to login to the instance using your private key saved from earlier. The default user is called ubuntu so if using standard ssh commands then something like ssh -i /path/to/private/key ubuntu@ipaddresss should get you in.

The ubuntu user has sudo access by default, so you can now start installing packages and using the instance for whatever you need.

Follow my other guide posts on how to setup an Ubuntu Server instance from scratch. For now we can just install Nginx to see our server running: sudo apt install nginx and sudo systemctl status nginx to check that it’s running.

Configuring iptables

One thing that might cause issues is the fact that the Oracle Ubuntu image sets up an iptables rule to block all traffic by default (ufw is not installed). That means if you had an Nginx server running, you won’t be able to ping it using the public IPV4 address unless you open up access on the ports. This seems a strange choice considering this is also controlled by the security group, but extra layers can’t hurt I guess.

To allow access on ports 80 and 443 for a standard web server with HTTPS enabled, run the following commands:

sudo iptables -I INPUT 6 -m state --state NEW -p tcp --dport 80 -j ACCEPT
sudo netfilter-persistent save
sudo iptables -I INPUT 6 -m state --state NEW -p tcp --dport 443 -j ACCEPT
sudo netfilter-persistent save

If you now navigate to the public IPv4 address in a browser, you should see the standard Nginx welcome page:

That about wraps up the setup process for this post. As I said, the Oracle Cloud free tier is extremely generous in terms of the sheer amount of infastructure you can provision. Plus, it operates just like the others big players as a proper commerical cloud offering, so uptime, network performance and integration with the general infra tooling should work out of the box.

Some quick high levels notes on Kafka vs MQ. This is a question that often gets asked by folks already who are familiar with traditional queues (IBM MQ/RabbitMQ etc) when they are introduced to the world of Kafka:

Kafka High Level Uses

• Event input buffer for data analytics/ML
• Event driven microservices
• Bridge to cloud-native apps

ActiveMQ

• Consumer gets pushed certain number of message by broker depending on prefetch
• Consumer chunks through them, on each ack, broker deletes from data store
• Produce pushes single message, consumer acks, deletes, gone 1to1
• Conforms to standard JMS based messaging

Topics in MQ

• Subscribers only receive messages published while it is connected
• Or durable where client can disconnect and still receive messages after
• In MQ can block brokers, fill data stores
• Each consumer gets copy of the message unless composite destinations/message groups
• Hard to create dynamic consumers or change the topology

Kafka

• each group gets message, but in group only one consumer
• consumers defines the interaction (pull)
  • partitions assignment, offset resets, consumer group

• consumer can apply backpressure or rebalance

• Can’t go back through the log
• Difficult to load balance effectively
• Completing consumers vs one partition still processing whilst other is blocked
• Hard to change topology or increase number of queues
• Hard to handle slow/failing consumers
• not predefining functionality to behave like a queue or topic, defined by consumers
  • introduce new consumer groups adhoc to change how destination functions
  • single consumer group = queue
  • multi consumer groups = topic
  • what offset to start from
• one consumer group can fail and replay whilst another succeeds
• MQ always queue one out at a time - not depending on consumers, Kafka behaviour changes on number of partitions/consumers

Useful Video Links

• https://www.youtube.com/watch?v=6yG2myKcMQE
• https://www.youtube.com/watch?v=sjDnqrnnYNM
• https://www.youtube.com/watch?v=Sa3Csx6zFcs
• https://www.youtube.com/watch?v=7Faly8jORIw